February 4, 2012

For Advanced Users : UBERAlign API / CLI / Powershell

From the beginning I knew some UBERAlign users would want to go into power user mode. The UBERAlign Console was designed to allow for easy use for the average Joe. But, there are people out there with the desire, guts, and ability to script and automate that want more. So this post will inform you on two other options for UBERAlign.

1. The CLI

Each vAligner is a Ubuntu Linux VM. On the VM is a set of binary files that do all work. One is a startup file for initializing, one is a daemon for accepting new jobs via the REST API, and the final is the actual magic behind the scenes. From the beginning UBERAlign was designed to be run from the command line. In fact back months ago the vSpecialist actually got a copy of this to try out and help me test. So for those that do not want to use the Console here are some reasons and instructions on how to run alignment, reclaim, and alignment+reclaim jobs manually.

Some of the reasons you may want to do this:

  1. Hate MS Windows – Since the console is a .Net WPF app some Mac users (@mcowger) have already asked how to skip using a Windows VM.
  2. VM size is too big – If the VM is more than 50% of the size of the datastore then a snapshot of it can potentially cause an out of space issue if it grows to full size (which and align can do). So if you have a good backup or are using array-based snapshots why use VMware snapshots? The CLI allows you to skip having to use a VMware snapshot which is required for using the Console.
  3. Aligns disks not attached to a VM -  The manually method just needs a valid VMDK file. Whether from a backup or anywhere. The Console requires a VM. Technically you can use tricks with snap and copy similar to the way VMware Converter works with online VM’s if you want.

So here are the steps:

  1. You must have the vAligner running and it must be on a vSphere host than can reach the datastore your VMDK is on.
  2. Make sure that any secondary disks are removed from the vAligner from a previous job.
  3. Make sure no UBERAlign Consoles are using this vAligner. Easy way is to disconnect the NIC and use the console if you are worried. If you console is showing weird FD0 garbage, grab a new copy of the vAligner OVA as this was patched.
  4. Add a Hard Disk to the vAligner (while it is running) to SCSI 0:1. Make sure and choose an existing disk and point it to the VMDK you want to run against.
  5. Login to the vAligner with root/UBERAlign.
  6. Make sure at least 15-20 seconds have passed since step 4. From the command line run: ‘fdisk –lu’ and make sure /dev/sdb is showing. Confirm the partition layout is as expected.
  7. Change directory to the UBERAlign binaries with ‘cd /opt/uberalign/bin’

Now that you have everything attached we are running to run a job. If you run ‘./uberalign’ you will see a printout of the command syntax like what follows:

./uberalign MODE  DEVICE  [OFFSET]  [ID]  [NAME]

MODE = The mode you want to run in and is formatted. Syntax:  -r:[a|r|s|c]

  • [a] = Align & Reclaim
  • [r] = Align only
  • [s] =  Simulate
  • [c] = Check alignment only
  • [z] = Reclaim only

DEVICE = The hardware device your VMDK is mounted on(normally /dev/sdb): Syntax: /dev/sdb

OFFSET = The target offset you want to align to. This defaults to 2048 sectors if not specified.

ID / NAME = Only useful for the API leave these off or use for logging purposes.

Example:

./uberalign –r:a /dev/sdb 2048

Once you run this command the uberalign program will process and log to the screen it’s progress while it works. When it completes, disconnect your VMDK from the vAligner and check your VMDK within a VM. It is as simple as that.

2. The API

The UBERAlign vAligner comes with a REST API that the UBERConsole uses for managing. While the state machine, orchestration for vSphere, and GUI are all a part of the console; you can still use the vAligner API to integrate into any existing processes you may have. Here is a short guide to the REST API. All data is return in JSON format. The first section explains the API get/put and the second explains the object structure in a pseudo format.

Methods

Root Path: http://<valigner IP>/uberalign/api/

Get Current State (GET)
Path: /uberalign/api/state
Parameter: n/a
Return: UAState
Description: Returns the current state of the vAligner

Get Current Job (GET)
Path: /uberalign/api/job/current
Parameter: n/a
Return: UAJob
Description: Returns the current job of the vAligner. On first boot this is an rather blank object.

Get All Completed(Historical) Jobs (GET)
Path: /uberalign/api/job
Parameter: n/a
Return: UAJob[]
Description: Returns an array with all completed jobs. This list will include all jobs still the vAligner was first deployed. Can be useful for historical purposes.

Get Session (GET)
Path: /uberalign/api/session
Parameter: n/a
Return: UASessionLock
Description: Returns the session lock information. This object contains information on the current lock status of the vAligner.

Lock vAligner (PUT)
Path: /uberalign/api/session/lock
Parameter: UASessionLock
Return: string (“true” | “false”)
Description: Used to lock a vAligner for 30 seconds. A locked vAligner will not accept new jobs from another source. Once a job is started it will not accept a new job until idle again. Use this method to maintain a lock on a vAligner while using it. Must specify a unique GUID that must match the GUID in the job ticket. GUID is a string and should be unique and not change for the duration of the session.

Submit New Job Ticket (PUT)
Path: /uberalign/api/job/new
Parameter: UAJobTicket
Return: string (“true” | “false”)
Description: Used to submit a new job ticket to the vAligner. A job ticket goes into a queue and is picked up by the daemon <15 seconds. The submitted UAJobTicket must have the same GUID as the lock and a lock must exist. Also the vAligner state must either be in Idle(0) or IdleWithError(9). You can use both the Get Current State and Get Current Job to watch for a job starting. The Current Job returns the UAJob object which will have a Name and ID that will match the  UAJobTicket Name and GUID fields you submitted.

 

Object Definitions:

Class UAState
{
string ip;
string errmsg;
string mac;
States state;
}

Class UAJob
{
string id;
string name;
string disk;
UADiskDetails diskdetails;
int offset;
States type;
string errmsg;
string currentstep;
string laststep;
double duration;
UAPartition[] partitions;
string msg;
string timestamp;
string completetimestamp;
}

Class UAPartition
{
string _system;
string _aligned;
string _id;
string _device;
string _offsetdiff;
string _boot;
string _start;
string _blocks;
string _end;
}

Class UADiskDetails
{
string _cyclinders;
string _size;
string _sectorsize;
string _totalsectors;
string _sectoratrack;
string _heads;
string _diskserial;
}

Class UASessionLock
{
string guid;
string ip;
int timestamp;
bool locked;
int secondsleft;
}

Class UAJobTicket
{
States type;
double offset;
string id;
string name;
string guid;
}

Enum States
{
Idle = 0,
AlignNoZero = 1,
AlignWithZero = 2,
Simulate = 3,
CheckOnly = 4,
ZeroOnly = 5,
GrowOnly = 6,
IdleWithError = 9
}

The objects above are simplified from my C# class structure.

This workflow for a job submission would look like this:

  1. Check vAligner state (is idle?)
  2. Lock vAligner (and continue to re-lock <30s until done with it)
  3. Mount Disk using vSphere API/Manually
  4. Create and submit new Job Ticket
  5. Watch vAligner State and Current Job to watch status
  6. After seeing the job completes (state=Idle or IdleWithError) un-mount disk

 

 

I know what some of you are going to say now: “But, Nick – how can I use the API now?”. The answer is provided by the vSpecialist rock-star Clint Kitson in his EMC Community release today. Clint built an awesome example of using Powershell to integrate and control UBERAlign via the REST API. This is so cool because if you already have scripting/code toolsets you use for automation, you can integrate UBERAlign right in. Check out more on this in Clint’ post here.

This is a basic overview and I am sure will probably just lead to more questions Smile. Please feel free to play around with both the CLI and API and post questions/comments below.

Thanks,

.nick

More PS Fun : Fix vCenter Startup w/ PowerShell

More to come on this, but I am working more hands-on lately in my role as a vSpecialist. As part of this I am going to make an effort to post every *shortcut* I come up with along the way. And I love shortcuts…

So here is another PowerShell trick. As many of you have experienced. If you have a local SQL server installation (express or not) of vCenter you will sometimes have vCenter not start correctly on boot. This is do to a race condition between the SQL Server services and vCenter. If SQL doesn’t beat vCenter then vCenter will not start. The correct way to fix this is to setup the Dependencies with the vCenter VPXD service to wait till SQL has done its business first. There are actually several good articles out there on how to do this manually.

I don’t like to anything manually, so here is a PowerShell script to auto-detect and configure vCenter startup for local SQL installs. It is setup to work with SQL server or SQL server express. I think I got all bases covered but you can easily customize as you like (or use on other services). [Read more...]

Don’t Drop the box : Windows/Mac scripts for Dropbox on VPN

esuslogo101409I love Dropbox. I mean really. I use it for sharing among my team, my family, and as a direct code working area. I can open up Visual Studio on one laptop, move to another, and go back to my workstation without even thinking about syncing or anything. I am a paying customer and feel like I get every dime in increased productivity.

But, one side effect of using Dropbox is that it is sometimes blocked by internal IT Departments because of it’s File-Sharing profile. I am not going to argue against this,  even though DLP is a better way to go in my opinion. If only there was a way to be able to access your Dropbox while obeying company policy of keeping it off your corporate networks… [Read more...]

UBERShell : Ninja Scripting

Every time I release a tool or write a blog post there is a voice in the back of my head saying, “Dude, no one is going to like this”. I think this is the first time that voice never said a word. From the beginning it was more of a question of how to make this work the way I wanted. And when I started to be able to play with it myself, I was blown away with how lucky I got in the final product. [Read more...]

Celerra VSA – UBER : Smaller, Faster, Easier, Geekier

**** Go here for new version of UBER VSA ****

Have you tried the EMC Celerra VSA before? If so, then forget everything you experienced. If not, then welcome and let me introduce you to easiest way to test enterprise-level features with NAS and virtualization.

I won’t go through the history or the advantages of the Celerra VSA because Chad Sakac does a much better job here, and here.

I am a greenhorn at the EMC vSpecialist team. I have just barely three notches on my belt (if notches represent months). So everything is new to me. I am constantly bothering my direct manager with questions like: “Why do we do that?” or “How did this come about?”. This is because I am looking at everything and trying to understand the why behind it all. And this is part of the reason I ended up diving under the hood of the Celerra VSA; to find out what made it tick and why it wouldn’t tick faster.

But first a quick background story: over the last month or so I have been doing my first tour of duty. I did my first customer presentation to a huge company (very near the top of the Fortune 500 list). I got involved in a proof of concept at another major client near Washington D.C. And I even got involved in helping design labs for VMware Tech Summit and major demos for EMCWorld (more on that after the big day). I have never been surrounded by such a quantity of talented (and cool) people or challenged like this before in my life. Needless to say I am loving every minute of it.

During this first tour I had the opportunity to install the Celerra VSA for the first time. And I will be honest: I was disappointed in it. I don’t like things that take so long. One of the reasons I write in so many scripting/development languages is because I hate repetition. So even before I started setting up my first VSA, I looked at the manual which, while VERY well-written, was way above my tolerance for length. Then and there I made it a personal goal to cut those steps down to as few as possible.

But, back to my first time to build…
An interesting thing happened the first time I did it. It was faster, way faster. In fact the Senior guy who I was assisting took a look to check my work and was sure they were broken. In his words: “It is too fast, it must not be actually doing anything.” But, after a little testing he confirmed that it was working perfectly and quickly. He asked me: “What did you do?”. I looked at him, blinked, and said: “I have no idea…”

I tend to venture off the beaten path in a lot of things. Evidently me skipping around trying to figure *why* each step needed to be done (for my future script) resulted in me fixing an issue. Problem was I had no earthly idea what I did or how.

So fast forward to now- I have spent the last good chunk of April figuring out the how and why of the Celerra VSA. I’ve not only figured out the speed tweak steps. But also I discovered memory improvements, boot speed improvements, and even finished a complete automation of the configuration process.

So in summary here are the new features of what I have dubbed the Celerra VSA –UBER Edition:

1. Completely automated install: On first boot a configuration wizard will ask a few questions and automatically reboot. On second boot everything is configured and ready to use. Also configuration takes care of ALL identity issues with generating MAC addresses. Average run through from first boot to complete is <2 minutes for me.

2. Much, much, much faster. You have to try it to see it. No, seriously it evens runs great on my Core Duo.

3. Low memory requirements: It will now run with 1024 MB of RAM (it is set this way by default). If you need to setup replication it may need more but will use much more efficiently.

4. You should not need to login to the console for anything. All configurations can be done through the Control Station web console (hint: Tools -> Wizards).

******* WARNING *******

Do NOT run a Celerra VSA on ESX on VMware Workstation. You will always get horrible performance this way. On a laptop or desktop use it directly on VMware Workstation 7 with the workstation version. On a server compatible with ESX(i) 4u1 use the OVA version.

************************

The Celerra VSA – Uber comes in two versions:

1.       Celerra VSA – UBER for ESX (OVA) (download link)

2.       Celerra VSA – UBER for Workstation/Fusion (ZIP) (download link)

I am not going to lie; this was at least 40-50 hours of late night work trying to reverse engineer something I never used as a customer. It was difficult and damaging to my ego in many spots. So I ask one simple favor from you. Try it out…

Obviously I think I made something pretty darn cool. But, what do you think? If you see an issue or have a suggestion please post a comment letting me know. I am curious what others notice and if they see the same improvements I do.

I hope this helps someone out in their labs somewhere. Please drop a comment if it does.

.nick

****** Update ******

Here is a video on the installation process:

New Utility: Domain Group Expiration Tool

Typically Active Directory is managed using th...
Image via Wikipedia

I wrote this tool out a of this simple request:  Why can’t a user’s membership to a domain group expire like their domain accounts can?

At my current employer we frequently need to grant temporary access for a few hours or days to resources.  However, this functionality is not built into Active Directory by default. The issue is that when you grant someone temporary membership to a group there is a real problem about removing that membership. Temporary access is based on a human element instead of an automated process.

As an example, what if your Director of Development comes to you and wants you to grant access to a individual. He/She needs this person to have VC access for group of VMware servers while another person is out sick. So you grant them access, a week goes by and you forget. Now you have an extra user with access they should not have.

But if when you added that user to the group, what if  you could assign a day and time when they lose that access? You could set and forget. This would eliminate auditing and having to keep reminders on these requests.

This tool is two parts. A PHP front-end that is used for submitting requests and a VBS script on the back-end for processing, logging, and alerting on requests. This tool provides the following:

  • Granularity on expiration down to the minute
  • Email alerts to requester and user-defined list (ISO, Managers, admins, etc) for processed additions and removals
  • Request form auto populates with users and groups from domain (no typing)
  • Uses DHTLM Calendar for Date/Time picking

Instructions for installation are found inside the README.
You can find the tool here: Domain Group Expiration Tool

dget

If you found this useful or have questions drop me an email or post your comments here.