abstract your problems away
I got some feedback that some users running Snow Leopard were crashing every time they started UNF.
Turns out there was a method being used on a Cocoa control that required changes made in the 10.7 SDK. I rewrote the function to not need the method and have released a new version (version 1 build 701). All download links are updated but you can download the new version here:
UBER Network Fuser 1.701 Download
Lion users don’t need to update.
Thanks,
.nick
A few months ago I bought my first Mac. I had been a hardcore Windows workstation kind of guy. And being a C# coder in the late hours of the night it was all I needed. But, after seeing my former teammates (vSpecs) using their sexy Mac Books I finally made the switch. I picked up a Mac Book Air and from the minute I started using it, fell in love with everything about it.
Because of this switch I have come to learn the things that the Mac (and Lion) do really well with some things while Windows does better with others. For almost all of my UBER projects I use my beefy Windows workstation running VMware Workstation to setup my lab/dev environments.
But, recently I started trying to move over to my Mac and using VMware Fusion to build lab virtual machines for developing. And to my surprise I found that things in the Fusion world aren’t quite the same as for Workstation. Namely, the networking options are rather limited. By default you only get the choices of three networks 1) DHCP w/ NAT 2) DHCP and 3) Bridging to a physical interface. For me that meant that I couldn’t get a network without DHCP (important if you are testing it) or create multiple isolated networks like I could on Workstation.
So I went digging and found that at one point someone has written some slick scripts to allow for custom multi-network setup but it stopped being updated after Fusion 2. I also found that VMware has some KB articles on how to hack your way to adding networks. But neither was very easy to do or dealt with modifying the VM’s well.
So I decided to fill this gap myself. Over the Christmas holiday I worked furiously to make this and now I am proud to present the next UBER release and my first project in the Manic Innovation Challenge: UBER Network Fuser (UNF)
UNF is a native Mac OSX application supported on Snow Leopard and Lion that allows you to add additional networks to VMware Fusion, customize their settings, and easily change network selection for any Fusion VM’s. I designed it to be simple to use and be similiar to how VMware Workstation network editor works.
Here is the full feature list:
I created a video to demonstrate how it works below:
And to download (free as always) use the link below:
Download UBER Network Fuser 1.0 – DMG for Snow Leopard & Lion (Updated link to new 1.701 version)
This is my first release in the Manic Innovation Challenge. It is written in Objective C & C and uses the Cocoa and Security frameworks. Challenge-wise it was definitely an experience learning and writing an app in Objective C in few weeks. But, it was crazy fun and I have all kinds of cool ideas for the Mac now.
Also, this marks post #100 for me in the 2.3 years I have been running Nickapedia.com. Being that it is also my birthday I am considering today a good day.
As always please test and play with it and let me know with some comments below.
.nick
Ok, this is a real quick example of someone else’s idea. Namely Christofer Hoff of Juniper (infamous blog & twitter account) who in response to my tweet about my MacBook and a webserver (I am a Mac-newbie). Suggested I try using Node.js.
At first I was a little taken back. I think @Beaker thought I didn’t get the point (and I don’t more than I do).
So I *cheated* and just turned on Web Sharing with the good tips from my other twitter buddies. But the Node.js idea bugged me because technically Chris is right. It should work fine.
So I did it. I installed the Node.js package for OSX (been using the Linux version) and fired up TextEdit and wrote this simple bit of code:
var http = require('http'); var fs = require('fs'); var path = require('path'); var rootpath = "/Users/lynxbat/Webroot"; console.log('*Starting'); http.createServer(function (request, response) { console.log('Server started'); var pathToFile = rootpath + request.url; console.log ('File was requested: ' + pathToFile); path.exists(pathToFile, function(exists) { if (exists) { fs.readFile(pathToFile, function(error, content) { if (error) { response.writeHead(500); response.end(); } else { response.writeHead(200, { 'Content-Type': 'application/x-gzip' }); response.end(content); }}); } else { response.writeHead(404); response.end(); }}); }).listen(8010); console.log('Server running at http://127.0.0.1:8010/');
I fired up the server:
node webserver.js
And opened my Safari browser to request a file I had dropped in my root directory.
Which works perfectly. Output on cli:
*Starting Server running at http://127.0.0.1:8010/ Server started File was requested: /Users/lynxbat/Webroot/node-v0.6.6.pkg
Now in my case I am using the Webserver to hand out packages for installing as a local mirror for some VM’s I am building (big demo for Jan). So I hardcoded the MIME type and left response encoding unset. If you were looking to properly serve you should perhaps add some logic for switching types and responses based on the file extension. Kind of like a proper web server works.
So Chris was right. This works. Though I think proving him right probably isn’t a good habit to pick up.
.nick
From the beginning I knew some UBERAlign users would want to go into power user mode. The UBERAlign Console was designed to allow for easy use for the average Joe. But, there are people out there with the desire, guts, and ability to script and automate that want more. So this post will inform you on two other options for UBERAlign.
Each vAligner is a Ubuntu Linux VM. On the VM is a set of binary files that do all work. One is a startup file for initializing, one is a daemon for accepting new jobs via the REST API, and the final is the actual magic behind the scenes. From the beginning UBERAlign was designed to be run from the command line. In fact back months ago the vSpecialist actually got a copy of this to try out and help me test. So for those that do not want to use the Console here are some reasons and instructions on how to run alignment, reclaim, and alignment+reclaim jobs manually.
Some of the reasons you may want to do this:
So here are the steps:
Now that you have everything attached we are running to run a job. If you run ‘./uberalign’ you will see a printout of the command syntax like what follows:
./uberalign MODE DEVICE [OFFSET] [ID] [NAME]
MODE = The mode you want to run in and is formatted. Syntax: -r:[a|r|s|c]
DEVICE = The hardware device your VMDK is mounted on(normally /dev/sdb): Syntax: /dev/sdb
OFFSET = The target offset you want to align to. This defaults to 2048 sectors if not specified.
ID / NAME = Only useful for the API leave these off or use for logging purposes.
Example:
./uberalign –r:a /dev/sdb 2048
Once you run this command the uberalign program will process and log to the screen it’s progress while it works. When it completes, disconnect your VMDK from the vAligner and check your VMDK within a VM. It is as simple as that.
The UBERAlign vAligner comes with a REST API that the UBERConsole uses for managing. While the state machine, orchestration for vSphere, and GUI are all a part of the console; you can still use the vAligner API to integrate into any existing processes you may have. Here is a short guide to the REST API. All data is return in JSON format. The first section explains the API get/put and the second explains the object structure in a pseudo format.
Methods
Root Path: http://<valigner IP>/uberalign/api/
Get Current State (GET)
Path: /uberalign/api/state
Parameter: n/a
Return: UAState
Description: Returns the current state of the vAligner
Get Current Job (GET)
Path: /uberalign/api/job/current
Parameter: n/a
Return: UAJob
Description: Returns the current job of the vAligner. On first boot this is an rather blank object.
Get All Completed(Historical) Jobs (GET)
Path: /uberalign/api/job
Parameter: n/a
Return: UAJob[]
Description: Returns an array with all completed jobs. This list will include all jobs still the vAligner was first deployed. Can be useful for historical purposes.
Get Session (GET)
Path: /uberalign/api/session
Parameter: n/a
Return: UASessionLock
Description: Returns the session lock information. This object contains information on the current lock status of the vAligner.
Lock vAligner (PUT)
Path: /uberalign/api/session/lock
Parameter: UASessionLock
Return: string (“true” | “false”)
Description: Used to lock a vAligner for 30 seconds. A locked vAligner will not accept new jobs from another source. Once a job is started it will not accept a new job until idle again. Use this method to maintain a lock on a vAligner while using it. Must specify a unique GUID that must match the GUID in the job ticket. GUID is a string and should be unique and not change for the duration of the session.
Submit New Job Ticket (PUT)
Path: /uberalign/api/job/new
Parameter: UAJobTicket
Return: string (“true” | “false”)
Description: Used to submit a new job ticket to the vAligner. A job ticket goes into a queue and is picked up by the daemon <15 seconds. The submitted UAJobTicket must have the same GUID as the lock and a lock must exist. Also the vAligner state must either be in Idle(0) or IdleWithError(9). You can use both the Get Current State and Get Current Job to watch for a job starting. The Current Job returns the UAJob object which will have a Name and ID that will match the UAJobTicket Name and GUID fields you submitted.
Object Definitions:
Class UAState
{
string ip;
string errmsg;
string mac;
States state;
}Class UAJob
{
string id;
string name;
string disk;
UADiskDetails diskdetails;
int offset;
States type;
string errmsg;
string currentstep;
string laststep;
double duration;
UAPartition[] partitions;
string msg;
string timestamp;
string completetimestamp;
}Class UAPartition
{
string _system;
string _aligned;
string _id;
string _device;
string _offsetdiff;
string _boot;
string _start;
string _blocks;
string _end;
}Class UADiskDetails
{
string _cyclinders;
string _size;
string _sectorsize;
string _totalsectors;
string _sectoratrack;
string _heads;
string _diskserial;
}Class UASessionLock
{
string guid;
string ip;
int timestamp;
bool locked;
int secondsleft;
}Class UAJobTicket
{
States type;
double offset;
string id;
string name;
string guid;
}Enum States
{
Idle = 0,
AlignNoZero = 1,
AlignWithZero = 2,
Simulate = 3,
CheckOnly = 4,
ZeroOnly = 5,
GrowOnly = 6,
IdleWithError = 9
}
The objects above are simplified from my C# class structure.
This workflow for a job submission would look like this:
I know what some of you are going to say now: “But, Nick – how can I use the API now?”. The answer is provided by the vSpecialist rock-star Clint Kitson in his EMC Community release today. Clint built an awesome example of using Powershell to integrate and control UBERAlign via the REST API. This is so cool because if you already have scripting/code toolsets you use for automation, you can integrate UBERAlign right in. Check out more on this in Clint’ post here.
This is a basic overview and I am sure will probably just lead to more questions 
. Please feel free to play around with both the CLI and API and post questions/comments below.
Thanks,
.nick
NOTICE: For more info on UBERAlign Advanced (API / CLI / Powershell) features also read the new post here: http://nickapedia.com/2011/11/07/for-advanced-users-uberalign-api-cli-powershell/
You know how in cartoons they show a small snowball rolling down a hill until it grows into a massive beast of a snow boulder?
Well, that is kind of how my most recent UBER project has gone. I know it is a been a little while since I have released a tool for the community and I am hopefully making that up with my newest creation: UBERAlign.
The idea of creating this came from the lack of a decent free alignment tool out there for VMware admins. Most every other one at there was either something you had to purchase or you had to be a customer of the vender to get access to it. And even after getting access these tools were either (in my opinion) limited in what they did, how they did it, or had become obsolete in a console-less vSphere 5 architecture.
For those they don’t know, alignment with Virtual Machine disks on top of Storage Arrays has been a performance issue for a long time. I won’t go into long detail explaining the problem or the benefits to alignment. There are great posts by Duncan (http://www.yellow-bricks.com/2010/04/08/aligning-your-vms-virtual-harddisks/) and Kevin(http://www.blueshiftblog.com/?p=300) on what the issues are and some of the tools available.
So my goal in creating UBERAlign was to once and for all create a free and powerful tool resolve this issue for everyone in the VMware community. Along the way I ended up probably overdoing the way I went about it. Turns out there are issues with aligning Windows boot disks, drive letter mappings getting lost on changes, current tools requiring copying of the whole disk, and tools frying NTFS metadata. I ended up deep in the weeds learning more than I have wanted to about NTFS, MBR/GPT, and disk logical formats. I also ended spending days writing data movement algorithms on my whiteboard over and over and I tried to solve specific problems with aligning a VM. And along the way I realized I had an opportunity to solve another issue that did not have a good universal answer.
So let me cut the background story down and get right to the details.
Presenting UBERAlign, a tool for VMware Virtual Machine alignment and Space Reclamation
Features:
So as you can see UBERAlign got to be a bit of a beast along the way.
UBERAlign comes in two pieces. The UBERAlign Console which is a graphical interface meant to be run on a Windows XP/7/Server system with .Net 4.0 or greater. And the UBERAlign vAligner vAppliance which is deployed from OVA into a vSphere environment. The console connect to the vSphere via SOAP and to up to 6 vAligners via REST interface. The way it works is: each vAligner can process VM’s on the storage the vSphere Host it lives on has access to. So you should spread vAligners across clusters and make sure one is on any vSphere Host that has local storage you want to access.
These features are meant to make the life of a VMware admin much easier when taking on this kind of task. But, let me cover some prereqs and how-to information.
Prerequisites/Tips/Caveats:
specifying Thin for the disk format.*DISCLAIMER*
Ok, before I go any further I want to mention one important thing. UBERAlign is an experimental tool and carries no support from EMC Corporation or myself (Nicholas Weaver). It is being released in beta state and while it does have functions that allow for failback you should only perform operations on Virtual Machines for which you have a solid backup. Also, I recommend you test in your lab thoroughly to make sure you understand it fully. You accept full responsibility when you use this tool.
*DISCLAIMER*
Ok, now that that is out of the way here are a few videos showing off the console and some of the cool things you can do (these are some quick/rough cuts):
And now for the downloads:
(New links are HTTP, no more FTP issues)
This was a long process for me. It was something I started in June and had to delay because of VMworld 2011 and side projects. It is my last tool as a vSpecialist (more on my move here) and I hope it helps the VMware community somehow. The good news is this was designed from the beginning to be hypervisor agnostic so watch for a Hyper-V and Xen version in the future.
I also want to thank Larry Whitlock (EMC vSpec) who was the primary tester during the harder part of this. Without his help I would not have gotten anywhere.
Questions, comments, and critisism are all welcomed.
.nick