May 21, 2012

Manic Innovation Challenge(Feb) : UBER Twitter Stats

Continuing my pursuit of the Manic Innovation Challenge I am proud to release my newest *dumb* idea: UBER Twitter Stats!

NEW UPDATE – Based on some tips (thanks: Brian Katz @bmkatz) some of the details haved changed below to make UBER Twitter Stats work a little easier. The same old command style still works. But the newer one is much easier.

animalWritten in 100% Ruby and running in the cloud, UBER Twitter allows you to ask me (technically my cloud-like proxy @myubertwit) for interesting recent stats about your Twitter account. It is really quite simple.

Send a tweet to my app account (@myubertwit) with the text: “<command>”. With the command being one of the following:

  1. My Word Count – Will reply with the top 20 words you used recently. This automatically strips out very common words. Shortcut: ‘mwc’
  2. Mention Word Count – This will reply with the same as the above but for tweets that mention or are to you. Shortct: ‘mmwc’
  3. Who I Mention – This will list the top 20 people you talk to or mention in your recent tweets. Shortcut: ‘wim’
  4. Who Mentions Me – This will reply back with the top 20 people who have mentioned you the most lately. Shortcut: ‘wmm’

If you get stuck just send @myubertwit ’help’ to get the instructions back in a tweet.

Depending on timing response may take up to a minute. Also, 90% of this work was written while I was chilling in a cigar bar with some friends. So while I do have *some* error handling, it ain’t much Smile. It should handle Twitter API limits and does a decent job of trying to iterate and gather as many tweets as possible. For some high volume users (Like @Beaker) it gets pretty close to a 1,000 tweets per request to analyze.

I WILL be releasing the source for this soon. I have to strip out some private stuff first. And this week I will be attending VMware Partner Exchange, Cloud Connect, and one other event along with a ton of meetings. Which means it might be a week or two before I have time to post to Github for everyone.

Feedback is king – let me know if you like it or if you have any good ideas to add!

.nick

Quick Update : UNF version 1.701

I got some feedback that some users running Snow Leopard were crashing every time they started UNF.

Turns out there was a method being used on a Cocoa control that required changes made in the 10.7 SDK. I rewrote the function to not need the method and have released a new version (version 1 build 701). All download links are updated but you can download the new version here:

UBER Network Fuser 1.701 Download

Lion users don’t need to update.

Thanks,

.nick

Breaking new ground : An UBER Tool for the Mac

Screen Shot 2012-01-10 at 12.45.45 AMA few months ago I bought my first Mac. I had been a hardcore Windows workstation kind of guy. And being a C# coder in the late hours of the night it was all I needed. But, after seeing my former teammates (vSpecs) using their sexy Mac Books I finally made the switch. I picked up a Mac Book Air and from the minute I started using it, fell in love with everything about it.

Because of this switch I have come to learn the things that the Mac (and Lion) do really well with some things while Windows does better with others. For almost all of my UBER projects I use my beefy Windows workstation running VMware Workstation to setup my lab/dev environments.

But, recently I started trying to move over to my Mac and using VMware Fusion to build lab virtual machines for developing. And to my surprise I found that things in the Fusion world aren’t quite the same as for Workstation. Namely, the networking options are rather limited. By default you only get the choices of three networks 1) DHCP w/ NAT  2) DHCP and 3) Bridging to a physical interface. For me that meant that I couldn’t get a network without DHCP (important if you are testing it) or create multiple isolated networks like I could on Workstation.

So I went digging and found that at one point someone has written some slick scripts to allow for custom multi-network setup but it stopped being updated after Fusion 2. I also found that VMware has some KB articles on how to hack your way to adding networks. But neither was very easy to do or dealt with modifying the VM’s well.

So I decided to fill this gap myself. Over the Christmas holiday I worked furiously to make this and now I am proud to present the next UBER release and my first project in the Manic Innovation Challenge: UBER Network Fuser (UNF)

UNF is a native Mac OSX application supported on Snow Leopard and Lion that allows you to add additional networks to VMware Fusion, customize their settings, and easily change network selection for any Fusion VM’s. I designed it to be simple to use and be similiar to how VMware Workstation network editor works.

Screen Shot 2012-01-10 at 12.46.01 AMHere is the full feature list:

  1. Allows up to 10 additional custom networks (total of 12)
  2. You can enable/disable DHCP, NAT, virtual host adaptor on any network
  3. Protects and provides rollback of default Fusion settings
  4. Allows dynamic changes of network membership with Fusion VM’s
  5. You may alias networks with custom names (even Workstation doesn’t do this)
  6. Tested and confirmed that VLAN tagging works as expected within private networks
  7. Saves configuration per user (names, paths, etc)
  8. Integrates with Apple’s Security framework for elevated privileges when needed

 

I created a video to demonstrate how it works below:

And to download (free as always) use the link below:

Download UBER Network Fuser 1.0 – DMG for Snow Leopard & Lion (Updated link to new 1.701 version)

 

This is my first release in the Manic Innovation Challenge. It is written in Objective C & C and uses the Cocoa and Security frameworks. Challenge-wise it was definitely an experience learning and writing an app in Objective C in  few weeks. But, it was crazy fun and I have all kinds of cool ideas for the Mac now.

Also, this marks post #100 for me in the 2.3 years I have been running Nickapedia.com. Being that it is also my birthday I am considering today a good day.

As always please test and play with it and let me know with some comments below.

.nick

Proving a another guy’s point : MacBook Air & Node.js as a Webserver

Ok, this is a real quick example of someone else’s idea. Namely Christofer Hoff of Juniper (infamous blog & twitter account) who in response to my tweet about my MacBook and a webserver (I am a Mac-newbie). Suggested I try using Node.js.

beaker_being_beaker

At first I was a little taken back. I think @Beaker thought I didn’t get the point (and I don’t more than I do).

beaker_being_beaker2

So I *cheated* and just turned on Web Sharing with the good tips from my other twitter buddies. But the Node.js idea bugged me because technically Chris is right. It should work fine.

So I did it. I installed the Node.js package for OSX (been using the Linux version) and fired up TextEdit and wrote this simple bit of code:

var http = require('http');
var fs = require('fs');
var path = require('path');
var rootpath = "/Users/lynxbat/Webroot";
 
console.log('*Starting');
http.createServer(function (request, response) {
 console.log('Server started');
 var pathToFile = rootpath + request.url;
 console.log ('File was requested: ' + pathToFile);
 path.exists(pathToFile, function(exists) {
 if (exists) {
  fs.readFile(pathToFile, function(error, content) {
   if (error) {
    response.writeHead(500);
    response.end();
   } else {
    response.writeHead(200, { 'Content-Type': 'application/x-gzip' });
    response.end(content);
   }});
   } else {
     response.writeHead(404);
     response.end();
   }});
}).listen(8010);
console.log('Server running at http://127.0.0.1:8010/');

I fired up the server:

node webserver.js

And opened my Safari browser to request a file I had dropped in my root directory.

Screen Shot 2011-12-19 at 4.18.55 PM

Which works perfectly. Output on cli:

*Starting
Server running at http://127.0.0.1:8010/
Server started
File was requested: /Users/lynxbat/Webroot/node-v0.6.6.pkg

Now in my case I am using the Webserver to hand out packages for installing as a local mirror for some VM’s I am building (big demo for Jan). So I hardcoded the MIME type and left response encoding unset. If you were looking to properly serve you should perhaps add some logic for switching types and responses based on the file extension. Kind of like a proper web server works.

So Chris was right. This works. Though I think proving him right probably isn’t a good habit to pick up.

.nick

Now for something completely different : Ubuntu 11.10, KVM, & VLAN trunking

Sometimes the path to learn something means using very different tools along the way. In my case, I have been learning more on the developing virtual network world along with some of the new DevOps toolsets popping up.

As part of this I have started using KVM as a hypervisor on an Ubuntu 11.10 platform in a portion of my home lab. I have learned quickly that getting something simple done in vSphere can be a bit of a chore in the KVM world. But on the flipside, KVM has been a fun learning experience in understanding virtualization in a more raw format.

One of these challenges I have decided to share is a simple one. I was wanting to play with the Dell-created DevOps deployment tool: Crowbar. Crowbar is an wrapper for OpsCode Chef Server. While a pretty slick little utility to research in the cloud deployment and automation space; one glaring problem is it is designed to run on Dell PowerEdge servers. Since I don’t have PowerEdge servers lying around anymore I needed to run this in a virtual machine. This in itself isn’t a huge problem as a virtual machine can pretty much match most of the logical hardware pieces needed. But, the one problem I ran into was that Crowbar out of the box likes to have a couple interfaces with the ability to tag VLANs itself. In a bare-metal world connecting 802.1q trunk ports to a server is pretty common. And even in the VMware vSphere world you can create trunk portgroups for a guest VM using documented methods(See VGT). But the problem I ran into is how do I pass a trunk port through in KVM/QEMU when using the default bridge-utils setup (not using ovswitch in this case)? [Read more...]

For Advanced Users : UBERAlign API / CLI / Powershell

From the beginning I knew some UBERAlign users would want to go into power user mode. The UBERAlign Console was designed to allow for easy use for the average Joe. But, there are people out there with the desire, guts, and ability to script and automate that want more. So this post will inform you on two other options for UBERAlign.

1. The CLI

Each vAligner is a Ubuntu Linux VM. On the VM is a set of binary files that do all work. One is a startup file for initializing, one is a daemon for accepting new jobs via the REST API, and the final is the actual magic behind the scenes. From the beginning UBERAlign was designed to be run from the command line. In fact back months ago the vSpecialist actually got a copy of this to try out and help me test. So for those that do not want to use the Console here are some reasons and instructions on how to run alignment, reclaim, and alignment+reclaim jobs manually.

Some of the reasons you may want to do this:

  1. Hate MS Windows – Since the console is a .Net WPF app some Mac users (@mcowger) have already asked how to skip using a Windows VM.
  2. VM size is too big – If the VM is more than 50% of the size of the datastore then a snapshot of it can potentially cause an out of space issue if it grows to full size (which and align can do). So if you have a good backup or are using array-based snapshots why use VMware snapshots? The CLI allows you to skip having to use a VMware snapshot which is required for using the Console.
  3. Aligns disks not attached to a VM -  The manually method just needs a valid VMDK file. Whether from a backup or anywhere. The Console requires a VM. Technically you can use tricks with snap and copy similar to the way VMware Converter works with online VM’s if you want.

So here are the steps:

  1. You must have the vAligner running and it must be on a vSphere host than can reach the datastore your VMDK is on.
  2. Make sure that any secondary disks are removed from the vAligner from a previous job.
  3. Make sure no UBERAlign Consoles are using this vAligner. Easy way is to disconnect the NIC and use the console if you are worried. If you console is showing weird FD0 garbage, grab a new copy of the vAligner OVA as this was patched.
  4. Add a Hard Disk to the vAligner (while it is running) to SCSI 0:1. Make sure and choose an existing disk and point it to the VMDK you want to run against.
  5. Login to the vAligner with root/UBERAlign.
  6. Make sure at least 15-20 seconds have passed since step 4. From the command line run: ‘fdisk –lu’ and make sure /dev/sdb is showing. Confirm the partition layout is as expected.
  7. Change directory to the UBERAlign binaries with ‘cd /opt/uberalign/bin’

Now that you have everything attached we are running to run a job. If you run ‘./uberalign’ you will see a printout of the command syntax like what follows:

./uberalign MODE  DEVICE  [OFFSET]  [ID]  [NAME]

MODE = The mode you want to run in and is formatted. Syntax:  -r:[a|r|s|c]

  • [a] = Align & Reclaim
  • [r] = Align only
  • [s] =  Simulate
  • [c] = Check alignment only
  • [z] = Reclaim only

DEVICE = The hardware device your VMDK is mounted on(normally /dev/sdb): Syntax: /dev/sdb

OFFSET = The target offset you want to align to. This defaults to 2048 sectors if not specified.

ID / NAME = Only useful for the API leave these off or use for logging purposes.

Example:

./uberalign –r:a /dev/sdb 2048

Once you run this command the uberalign program will process and log to the screen it’s progress while it works. When it completes, disconnect your VMDK from the vAligner and check your VMDK within a VM. It is as simple as that.

2. The API

The UBERAlign vAligner comes with a REST API that the UBERConsole uses for managing. While the state machine, orchestration for vSphere, and GUI are all a part of the console; you can still use the vAligner API to integrate into any existing processes you may have. Here is a short guide to the REST API. All data is return in JSON format. The first section explains the API get/put and the second explains the object structure in a pseudo format.

Methods

Root Path: http://<valigner IP>/uberalign/api/

Get Current State (GET)
Path: /uberalign/api/state
Parameter: n/a
Return: UAState
Description: Returns the current state of the vAligner

Get Current Job (GET)
Path: /uberalign/api/job/current
Parameter: n/a
Return: UAJob
Description: Returns the current job of the vAligner. On first boot this is an rather blank object.

Get All Completed(Historical) Jobs (GET)
Path: /uberalign/api/job
Parameter: n/a
Return: UAJob[]
Description: Returns an array with all completed jobs. This list will include all jobs still the vAligner was first deployed. Can be useful for historical purposes.

Get Session (GET)
Path: /uberalign/api/session
Parameter: n/a
Return: UASessionLock
Description: Returns the session lock information. This object contains information on the current lock status of the vAligner.

Lock vAligner (PUT)
Path: /uberalign/api/session/lock
Parameter: UASessionLock
Return: string (“true” | “false”)
Description: Used to lock a vAligner for 30 seconds. A locked vAligner will not accept new jobs from another source. Once a job is started it will not accept a new job until idle again. Use this method to maintain a lock on a vAligner while using it. Must specify a unique GUID that must match the GUID in the job ticket. GUID is a string and should be unique and not change for the duration of the session.

Submit New Job Ticket (PUT)
Path: /uberalign/api/job/new
Parameter: UAJobTicket
Return: string (“true” | “false”)
Description: Used to submit a new job ticket to the vAligner. A job ticket goes into a queue and is picked up by the daemon <15 seconds. The submitted UAJobTicket must have the same GUID as the lock and a lock must exist. Also the vAligner state must either be in Idle(0) or IdleWithError(9). You can use both the Get Current State and Get Current Job to watch for a job starting. The Current Job returns the UAJob object which will have a Name and ID that will match the  UAJobTicket Name and GUID fields you submitted.

 

Object Definitions:

Class UAState
{
string ip;
string errmsg;
string mac;
States state;
}

Class UAJob
{
string id;
string name;
string disk;
UADiskDetails diskdetails;
int offset;
States type;
string errmsg;
string currentstep;
string laststep;
double duration;
UAPartition[] partitions;
string msg;
string timestamp;
string completetimestamp;
}

Class UAPartition
{
string _system;
string _aligned;
string _id;
string _device;
string _offsetdiff;
string _boot;
string _start;
string _blocks;
string _end;
}

Class UADiskDetails
{
string _cyclinders;
string _size;
string _sectorsize;
string _totalsectors;
string _sectoratrack;
string _heads;
string _diskserial;
}

Class UASessionLock
{
string guid;
string ip;
int timestamp;
bool locked;
int secondsleft;
}

Class UAJobTicket
{
States type;
double offset;
string id;
string name;
string guid;
}

Enum States
{
Idle = 0,
AlignNoZero = 1,
AlignWithZero = 2,
Simulate = 3,
CheckOnly = 4,
ZeroOnly = 5,
GrowOnly = 6,
IdleWithError = 9
}

The objects above are simplified from my C# class structure.

This workflow for a job submission would look like this:

  1. Check vAligner state (is idle?)
  2. Lock vAligner (and continue to re-lock <30s until done with it)
  3. Mount Disk using vSphere API/Manually
  4. Create and submit new Job Ticket
  5. Watch vAligner State and Current Job to watch status
  6. After seeing the job completes (state=Idle or IdleWithError) un-mount disk

 

 

I know what some of you are going to say now: “But, Nick – how can I use the API now?”. The answer is provided by the vSpecialist rock-star Clint Kitson in his EMC Community release today. Clint built an awesome example of using Powershell to integrate and control UBERAlign via the REST API. This is so cool because if you already have scripting/code toolsets you use for automation, you can integrate UBERAlign right in. Check out more on this in Clint’ post here.

This is a basic overview and I am sure will probably just lead to more questions Smile. Please feel free to play around with both the CLI and API and post questions/comments below.

Thanks,

.nick