Nickapedia- abstract your problems away -

This post will cover so much of the last few months of my life that I don’t even know where to start. I am sitting here in the Las Vegas airport trying to remember all the details of how I got to this spot. It really started at the beginning of March when I moved to the vSpecialist Technical Enablement team. The TE team is tasked with arming the front-line vSpec with information, tools, messaging, and more to do their jobs more effectively.

Another aspect of the TE team is they usually provide many of the keynote and session demos for the conference and partner events like EMC World or VMworld. Also, the TE guys have been running the EMC Hands On Labs for over a year now at events across the US.

Within a week after joining I had a discussion with my boss, Chris Horn, about the how the existing Hands On Labs (which he pioneered and maintained) could be improved and expanded for EMC World. The goal was to do something much more like the experience the VMworld labs provide. Chris had been talking with the EMC Demo Cloud team about using cloud-hosted labs they could provide to the users in Las Vegas at EMC World. After a few long conversations I committed to write the lab automation that would take a session in the Demo Cloud and automate it straight to a user at a desk across the country. Not a small feat and honestly I realized shortly after that it would require a massive amount of design and work.

Step 1: Design

I had multiple aspects to look at to accomplish what I wanted. My goal was to be as completely hands off as possible. I am a firm believer that Cloud automation should just work to a point that it can be forgotten about. This meant I had to adopt patterns that allowed for failure and self-healed without human intervention.

I also had to consider the fact that a 200 seat lab would need to have multiple people running the workflow and I would have little to no time to train anyone on how to use the system. This meant control interfaces must be simple to understand and bulletproof.

This system would communicate via SOAP to a Cloud automation system in Durham, NC. It would have to not only inventory what types and sessions were available, but gather relevant info on the session (type, IPs, etc) and also revert used sessions for reuse by the next user. This connection had to be robust and self-healing also as it was a critical failure point for the whole system.

For the influx of lab users I had to design a object structure representing the lab user, the lab sessions, and the metadata I wanted to capture for reporting both during and after. This would be critical for post-show stats and live stats I planned on showing during.

I had to write a registration system that would allow the user to choose a lab out of a dynamic set that could be managed on the fly. This would have to be separate from the main system to allow it to work regardless of whether the main *machine* was up or down while allowing handoff to the machine when it was up.

I also had to write a lab console that would work on a Wyse think client and would provide a robust experience. This would have to support some sort of lab *guide* in some format and support RDP for connections into the lab VMs in their session. This lab guide should allow for automatic loading, the ability to change content on the fly, and be easy to read. I also needed to add interactive buttons for asking for help “Raise Hand” and detecting when the user was finished “Finish Lab” to allow the Machine to invoke a reset of the lab session they were using. The RDP connection had to be robust also allowing for slow connections or having to reconnect the user should a network blip occur. The lab console had to inherit the user and lab session information, update it with what occurred during the lab and return a completed lab session detail that could be used for analysis later. This lab console had to auto-detect and auto-login to the Machine allowing for dynamic naming to meet any floor plan and any number of clients (within reason).

I needed console applications for registration, lab assignment, admin functions(controlling the Machine), and more. I also wanted to have display reporting

And on top of off the above I needed to be able to test the entire thing without having 1,000+ people to register or 200 WYSE terminals.

I named this project: Bottle Rocket, and got started on March 21st 2011.

Step 2: The Work

In between March 21st and EMC World 2011 I spent an average of 80 hours a week working on Bottle Rocket. I made a couple videos of my progress for my teammates since I was locked away.

Bottle Rocket Assignment Console from Nicholas Weaver on Vimeo.

Bottle Rocket Registration Console from Nicholas Weaver on Vimeo.

Bottle Rocket Lab Console Walkthrough from Nicholas Weaver on Vimeo.

Also in the time period I built my Vplex Geo Maps/vMotion mashup application. Here is a quick video I made on this. This video was early in the development and you can see the final version when the Chad’s World (Live) it was featured in is released to view.

Vplex Map Demo from Nicholas Weaver on Vimeo.

The videos above are actually alpha versions and I added more stuff to most of them (like ‘Raise Hand’ on the console).

I wrote a ton of testing tools including lab console simulators and registration simulators. I actually simulated 1,000 lab consoles and several thousand registrations live against the Demo Cloud from my home office. This allowed me to stress test each component and be sure things would not fall over on the floor. I also built in a “Sim Mode” into the lab console software where the consoles could actually simulate as if a user was sitting at the desk and work the lab steps. This would allow us to simulate stress on the Wyse terminals on site if needed.

Another piece I wrote that I thought I would not need was a Demo Cloud provisioning tool. I had used this for my testing environment but when we had issues early on (see below) I ended up using this to maintain the quota of needed lab session automatically. You basically told this program what you wanted, and it attempted to provision to your level while adjusting to the rate the Demo Cloud would allow and healing from connection issues or errors. A real slick fire and go to sleep tool.

The last couple weeks before EMC World I spent in the lab with the TE team in RTP, NC. While I had been furiously coding, my team had been making the magic happen by building 19(!) different labs around EMC products. They also had enlisted multiple individuals across EMC to build lab guides for each one. This effort was massive and even more time consuming than my own. The greatest part is that these labs are now able to be used for other events down the road.

I ending up before EMC world with 37 different unique code projects including Bottle Rocket and all its components and the Keynote demos I was involved in. Some of the Keynote stuff was: a crazy fast VM cloning tool based on Virtual Selection, a RDP window matrix of up to 100 real RDP windows borderless on a single screen. And a SQL Performance dashboard that showed real time transactions, MB/s, and depth queue stats. I also built an Isilon VSA (internal only) and new UBER VNX in this period for the lab and my team. I am pretty sure I have never created such a massive about of personal output in my life in such a short time.

Step 3 : The Show

I showed up to EMC World early to help setup. We had 36 hours of lab guide building madness converting a ton of word documents over to the format my custom viewer uses (sorry guys). We had a dead Cisco C-series blade we fixed with duct tape. And we had a 48 hour marathon to try and fix a late occurring bug in the Demo Cloud that would mean changing core functionality in my code. During this period Chris Horn didn’t sleep for at least 40 hours and Simon Seagrave almost the same. We ended up launching Monday without having the bug fixed. I had to rewrite my code that morning to work completely differently and pray it worked. Monday ended up being a rough day. We got almost 200 labs done but we realized that the bug HAD to be fixed to do what we really wanted to do.

All night Monday and early Tuesday morning the Demo Cloud folks and my fellow vSpecialist, Clint Kitson, worked on an idea Clint had on how to bypass the error. This would mean going back to the original code and plan. And I almost did a back flip when Tuesday morning testing worked. I quickly reverted the code and fired up the system. And Tuesday was when I saw all the hard work and planning pay off. I watched as the system just worked. No intervention and no management needed. People registered, were walked in, sat down to a deployed lab, and watched as the labs reset after each user got up. My stats screens looked awesome also showing live views of progress and breakdowns in the lab (video below). I ended up updating several console tools based on suggestions from the vLab crew and even writing a couple new ones on the fly.

Here is a video of the final stats screens from the end of day on Thursday. We reached over 1,300 labs and 3,000+ VM’s. All of it hosted 2,000+ miles away inside an EMC datacenter. I think this really sets the standard for within EMC on how an interactive experience can be provided from a cloud,

EMC World vLab Final Stats from Nicholas Weaver on Vimeo.

The Chad’s World Live keynote was also just as aggressive and crazy. I will let Chad cover the behind the scenes on this. But I have new respect for live productions and antacids.

All in all EMC World was a major success. Both as a large team and personally I believe we pushed boundaries and did things we have never done before. I threw together a quick video of the entire EMC World experience from my lowly viewpoint. It is long at about 18min but definitely has some cool behinds-the-scenes shots and after-the-show fun.

EMC World 2011 from Nicholas Weaver on Vimeo.

To be honest I don’t know what is next for me. This was just a monster effort I am now on mandatory down-time. I am going to take some time to maybe go outside for a bit and keep away from the keyboard. I wouldn’t trade the experience of the last few months for anything but I also wouldn’t recommend doing it in such a short time to anyone.

Comments and questions are always welcome,

.nick