Nickapedia

To start off, let me preface this by saying two things:

1. This is funny because of the predicament and not the players. In no way am I implying that anyone is incompetent.
2. I have changed the names and story slightly to protect the innocent.

So I am having a normal day; the standard eight hours of meetings mixed well with eight more hours of managing the team and writing design/implementation plans, when into my cube walks Alex. Alex is a business analyst who is working on a new customer process.

This new process takes our existing platform and allows customers to avoid needing an email address or internet access to interact. Alex tells me that they had just purchased a new domain name and need to know the next steps for setting up an email domain and server to handle emails (using corporate email cluster is NOT an option).

Turns out the existing platform requires email as a primary key. Which means that even though the customer doesn’t need one, the database/process does. So the solution Alex brings me is to setup an email server and every time an account is created they will create an email for the customer on this new domain. Then they use the existing email notification components (of which there are many) without change. He asks me to setup an email server to receive these emails and immediately delete them upon arrival (since they are useless).

So I ask the most logical question: “Why do these emails need to go somewhere if you are just going to delete them?”. Why would we setup email infrastructure ($$$) and dependencies for something that has no value?

He answers: “Because the system mustn’t get any bounce backs or errors with delivery”

So I ask the new logical question: “Why even send these emails since no one gets them?”

The next reply: “Because we may actually populate their real email later and need to have the process send an email just in case”

Now I am really irked. The code is to complex to work around, the schema can’t be changed, and the process has to account for multiple possibilities. I ask my final question: “Why not only send an email if the domain isn’t the spoof domain?”

Alex looks at me and says: “Oh….. yeah that would work”, says thanks, and walks away.

And I am left wondering what just happened and why the infrastructure guy had to recommend a few lines of code to avoid a new email server.
Funny but true.

In my career I was a late starter and fast riser. It wasn’t too long ago I was plugging in monitors and crawling under desks (a job I still highly respect). I owe a lot of my success to my supportive Wife who while she was raising my children, let me spend hours reading complete histories of LAN switching or reference book on WAN protocols. Not to mention every RFC under the sun. I owe a lot to my aggressive nature of never losing a fight whether it was an annoying Outlook bug, or an 801.X config that wouldn’t work right. But I also owe a lot to the technologies that I invested my time and effort behind who ultimately made my career. I bet on certain companies and technologies and they came through for me time and again. And so in this short blog post, I want to say thanks. So here are my thanks to the top 4 companies that have made me valuable by staying valuable in the information technology arena.

1. VMware

My first experience with VMware was very early on from listening to a Dell solution architect talk about where they were going. This was back when vMotion was just a rumor and still seemed impossible. I went on to work for a company that did a big virtualization shoot-out. At the end of the shoot-out my recommendation was VMware. I was strongly impressed by their focus on stability and running at the enterprise level. Unfortunately, the company chose another virtualization product(I left shortly after). After I left they found out the hard way what an enterprise virtual product needs and within a year they switched to VMware after going through a very painful experience with the other product.
The next company needed me for specific skills but when problem arose with their *free* virtual development solution, I pitched an idea to convert everything to a VMware platform. I must have had to pitch it a hundred times and I even called one meeting with every development team in one big room to explain how VMware works. After selling it for months I finally got the funds and approval. It was a complete success and allowed me to move production infrastructure services like Exchange, LCS 2005, SQL servers, data warehousing, and file servers down the road. I was able to be the guy that got my company from 15% to 80% virtualized within three years. When we needed to be agile with environments or cheap(aka consolidated) with hardware purchases I had changed the culture to make this possible. But I owe my thanks to everyone at VMware for not only maintaining a stable product, but having incredible support, education services, aggressive improvement cycles, and strong community support. I would not be where I am now without the guys and gals at VMware doing what they do.

2. Dell

Whether it was their servers, switches, or workstations I have been involved in Dell shops since my first IT job. They have been consistent with their excellent documentation, strong support, and great product. My favorite part about Dell is their consistent commitment to integrating with major players like EMC and VMware. When it came down to solutions that needed to work with SAN and virtualization options I had, Dell had already put in the work to make sure they were the logical option. I have had a very difficult time not choosing Dell in datacenter projects that involved VMware. I recently made my first trip to a Dell Executive Briefing and left very impressed with their commitment to really providing value and not just selling a product. Dell for me is a company designed to be invested in the long-term benefits and not short-term margins.

3. Cisco

My first praise for Cisco is that their certification programs still really matter. In a world of paper certs the Cisco network programs are still are a great way to prove your worth and actually study for comprehension. I could go on about stable and feature rich routing and switching but the one place where Cisco really paid off for me was their ASA firewall line. I put my reputation on the line pushing to change my current employer’s firewall devices to Cisco solutions. The things I was able to do with multi-tier VPN, securing site-to-site tunnels, network segmentation, troubleshooting, and improving security blew all expectations out of the water. Shortly after I was able to implement unified wireless solutions that solidified Cisco as an integral part of our infrastructure. It is tough to be at the top and still provide consistent value over many different business lines. Cisco gets kudos for giving me tools to be productive.

4. EMC

While not the cheapest storage option at first glance EMC does bring one consistent thing to the table every time, you can bet your job on it. I have never met people from company where they passionately believe in their product like the EMC folks I have run into. And EMC is definitely in this list for the early adoption of VMware (in integration and corporate stock) which made it a de facto choice early on. Every time I have a design to build, I evaluate a product for what it can do for me.  And EMC consistently provides storage products that not only do what is needed but open up possibilities that you hadn’t thought of. Similar to Cisco this is a company that could very easily surf on market-share. But they are aggressive at staying on top and in turn has paid off for me.

So just to clarify, I am not paid by any of the above. I am just thanking these firms for making my investment in them pay me back. And I know that someone could argue that X firm could have done Y for me in the same place. But, in my particular case I have benefited greatly from four companies and just wanted to say thanks where no one else might have.

Who knows what tomorrow brings? Maybe VMware will start giving free licensing to Osama Bin Laden or EMC starts stealing candy from orphan babies in Calcutta. But at least right now these guys are on my list.

So thanks

At my current employer we use a custom built ETL process for building business reporting and analysis data. Originally this started as a medium-sized Dell server with a full rack of local storage. As the criticality and scale of this resource grew, it outgrew the hardware it was on.  The key to this server was that the build process ran overnight and the server was accessed by multiple departments throughout the day.  This left very little time for hardware maintenance. I had helped move all development environment servers to a VMware cluster months before. Using this momentum I pitched the idea of solving the criticality and scalability with a VMware-based solution. The argument was four-fold:

1. The company wanted to avoid the licensing and hardware expense of moving to a Microsoft clustering solution.
2. VMware HA provided resumption of services in the case of a hardware failure and hardware maintenance would not require downtime.
3. The RTO was satisfied by an automatic HA fail over.
4. The additional cost of VMware licenses and new hosts would be spread over future planned provisioning and actually reduce costs by introducing consolidation.

After playing the part of VMware sales rep. I was able to get endorsements from the CTO, Data Services, and Executive groups. This would be the first time we would attempt to put a business critical service on a VMware platform. I was the only individual in IT at the time that had any exposure to VMware and needless to say my reputation and job was likely on the line.  The success (and to a greater extent stability) of the development environments was the key to get my foot in the door but now it was time to change our culture.

We opted to purchase additional Dell 2950 servers and EMC Clariion enclosures and dedicate a single DAE of fibre-channel 146GB x 15 for database drives. We did a few weeks of testing SQL loads that matched the ETL process (at the time) and determined that the optimal configuration was a single RAID10 of 14 disks with 3 separate VMDKs for data, log, and temp db.
The critical decision was “how do we move it?”.  At the time, P2V of SQL, Active Directory, and Exchange was not very common. I pitched the idea of a cold P2V using VMware converter. My justification was that the virtual could be brought up in parallel and everything tested completely. It took a few meetings to explain the process and give everyone a level of comfort
The P2V was successful and the virtualization has proved to be extremely valuable to the management of this server since. Here are just a few examples of VMware features that we have leveraged:

• Storage VMotion – Moving the OS drive
• Snapshots – Before a critical change
• VMotion – Ability to add to, repair, or restructure hardware without having to involve a outage
• Scale Up & Scale Out – Again without impacting service availability
• DRS – Resource balancing servers of mixed workloads

Our company IT culture has changed in many ways over the last few years. VMware has become a central part in our Infrastructure foundation and has brought both stability and agility. We have virtualized Exchange, OCS, web servers, application databases, and many other services sucessfully since. Now we have ambitious goals of leveraging advanced SOA-based platforms coupled with business continuity and scalability using a VMware infrastructure. Looking back, this initial server was our first step.

I will be adding more posts soon on lessons we learned post-virtualization. If you have comments or even disagreements please leave comments below.

I am working on a NLB with Windows 2008 servers and Exchange 2007 Hub Transport and Client Access roles right now.

There is a lot of good information out there but I thought I would list a couple links I found relevant to VMware and Exchange 2007 SP1:

• VMware & NLB
• MS NLB Team Blog
• Load Balancing Exchange 2007 Client Access Servers using Windows Network Load-Balancing Technology
• Implementing a NLB Cluster
• Deployment Options for Hub Transport Servers
• How to Configure the Availability Service for Network Load Balanced Computers

If you have good recommendations let me know and I will add them here.

Technorati Tags: VMware, Exchange 2007, NLB

I have a telephony engineer on my team who often uses the word “Kluge” to explain really poorly thought out engineering choices.  Today he sent me to a blog that goes into detail on what a “Kluge” really is.

I wrote this tool out a of this simple request:  Why can’t a user’s membership to a domain group expire like their domain accounts can?

At my current employer we frequently need to grant temporary access for a few hours or days to resources.  However, this functionality is not built into Active Directory by default. The issue is that when you grant someone temporary membership to a group there is a real problem about removing that membership. Temporary access is based on a human element instead of an automated process.

As an example, what if your Director of Development comes to you and wants you to grant access to a individual. He/She needs this person to have VC access for group of VMware servers while another person is out sick. So you grant them access, a week goes by and you forget. Now you have an extra user with access they should not have.

But if when you added that user to the group, what if  you could assign a day and time when they lose that access? You could set and forget. This would eliminate auditing and having to keep reminders on these requests.

This tool is two parts. A PHP front-end that is used for submitting requests and a VBS script on the back-end for processing, logging, and alerting on requests. This tool provides the following:

• Granularity on expiration down to the minute
• Email alerts to requester and user-defined list (ISO, Managers, admins, etc) for processed additions and removals
• Request form auto populates with users and groups from domain (no typing)
• Uses DHTLM Calendar for Date/Time picking

Instructions for installation are found inside the README.
You can find the tool here: Domain Group Expiration Tool

If you found this useful or have questions drop me an email or post your comments here.

Over the last couple months I was tasked with finding a way to move my team to a project management process based around Agile methods. Every other part of IT had moved to Scrum and were having positive results.

As I did research into Scrum and spoke with different ScrumMasters I discovered that it was not  incompatible with an operations based group even though it may not be as easily consumable. I decided to take the core values of Agile and write a proposal that was Scrum-like is almost all of its aspects. But at the same time I wanted to maintain focus on specific goals that integrate into things like Visible Ops and teams with >33% dedication to support/maintenance.

A large portion of this document is written with specific challenges in mind for my group. I also consider myself a complete rookie in the arena of Agile but I can see the posibilities of cross-pollination and team-based accountability having significant benefits for an operations team.

You can download the Word 2007 document here:
InfraScrum – Agile Methodology Applied to Infrastructure Operations

**** Update ****

So even with all the work on document above I ended up implementing a hybrid of Scrum and Kanban.  The results have been great and I will try and write a post about it soon.

How to convert Unix Timestamps to Date/Time in Excel.

=timestampcell / 86400 + 25569

Right click on the cell with the formula above Format Cell and choose Date or Time as the Format.

So here is the back story on this idea/script:

I was at my VMware Fast-track class in Dallas when the instructors mentioned an interesting story. They said they knew of a IT shop that had a pretty strict policy on labeling their servers.  That was fine and good pre-virtualization. But after moving to a VMotion-enabled VMware cluster they had the problem of their servers not matching their labels. So did they change the policy? Nope, they got magnetic labels and had an admin run down to the server room every time they used VMotion. True?  Maybe, but it got me thinking.

Why not use the front panel LCD on newer Dell servers to list what VM’s are residing on the host? I pitched the idea to the instructors and they thought it was spiffy. A couple months later I have finally given it a shot. I debated whether to try and write this in C but opted to use Bash because it is easier to implement and I am so rusty at C it isn’t funny(and I should be spending those hours on Powershell).

To start off here are the prereqs:

1. Dell OpenManage installed and running on the ESX host (needed for ipmi drivers)
2. ipmitool 1.8.10 installed (SCP over, ./configure, make, make install….)
3. My lcd_update.sh script

I wrote the script to only use what was available on the ESX 3.5 host(besides prereqs). I place mine in the /root folder and put “/root/lcd_update >> /dev/null &” into my /etc/rc.d/rc.local file to make it start on boot.
If you prefer you can just start it manually with nohup.

The script is pretty simple. It takes inventory of what VM’s are running and lists them on the LCD using IPMI. It keeps a semaphore file in /etc to keep track (index) of what VM it listed last. The Dell LCD can only display 14 characters (unless someone out there figured something else out) so if your VM name is longer it will be truncated. Overall the effect is pretty slick and with the way it is timed it appears to be a scrolling marquee of server names. Also, if you host has no VM’s or there is a problem with vmware-cmd it will just list the ESX hostname (it will trim a FQDN).

I am sure major improvements can be made to this and I am planning on writing one for server health and hosts in maintenance mode. This is more of a ‘can it be done’ utility and my first *public* script for something VMware related.

***Update*** v .2

I changed the script to use vmware-vim-cmd instead of vmware-cmd.  This should give a list of VM names instead of just taking the name from the VMX file (incase they don’t match). Kudos to Duncan Epping’s post @ Yellow-Bricks which gave me the idea.

***Update***

Scott(@DellTechCenter) from www.delltechcenter.com has posted a video of the output: http://www.youtube.com/watch?v=Slm3MDMD7Dc

Just found out yesterday that the newest kid is a boy. Hasn’t really sunk in.

After 3 daughters you kind of get in a groove and know what to expect. I am pretty excited (either way this was the last one) but kind of nervous too.

So much more reflects on me as an example for him as a man. Plus there is the fact that he will be the first Weaver male in 26 years (since my brother)

Good news either way. I am already planning on how to train him on FPS games