Thursday, 5 August 2010

Finding 'idle' VM's in your infrastructure

Having run into a few 'high ready time' alerts recently, and suffering through a perceived lack of performance on a number of VM's, I ran through our datacentre's to see if I could find any wasteful VM's. These had been assigned 2 or 4 vCPU's and were doing nothing with them, due mostly to client requirements and/or specifications. Dropping these down to use less vCPU’s actually made some of the VM’s faster.


Function Get-UnderUsedVM($VCServer,$FQDNCSVLocation)
#Finds all VM's with 'idle' vCPU's, that could be downgraded to less vCPU assigned
$VC = Connect-VIServer $VCServer
$Results = @()
$DCName = Get-Datacenter | Sort-Object
ForEach ($DC in $DCName)
        $ClusterName = Get-Datacenter -Name $DC.Name | get-cluster | Sort-Object
        Foreach ($ClusterEntity in $ClusterName)
        $StatVM = get-cluster -Name $ClusterEntity | get-vm | Sort-Object
        ForEach ($VM in $StatVM)
            $Adatarow = "" | Select Datacenter,'Cluster Name','VM Name','Assigned CPUs','CPU Used',MhzPerCPU
            $Adatarow.Datacenter = $DC.Name
            $VMView = get-vm -Name $VM | get-view
            $Adatarow.'Cluster Name' = $ClusterEntity
            $Adatarow.'VM Name' = $VMView.Name
            $Adatarow.'Assigned CPUs' = $VMView.Config.Hardware.NumCPU
            $Adatarow.'CPU Used' = $VMView.Summary.QuickStats.OverallCpuUsage
            $Adatarow.MhzPerCPU = [int]$VMView.Summary.QuickStats.OverallCpuUsage / $VMView.Config.Hardware.NumCPU
            $Results += $ADataRow
$Results | Select Datacenter,'Cluster Name','VM Name','Assigned CPUs','CPU Used',MhzPerCPU | Where { $_.'Assigned CPUs' -gt 1} | Where { $_.MhzPerCPU -lt 1000} | export-csv $FQDNCSVLocation -notype

If you can make it run faster, have any suggestions, or had this help you please leave a comment below.

Friday, 30 July 2010

Multi-VM Storage Migration Powershell Script

We had to migrate 100-odd VM's from Germany to London for a relocation, and were using a small HP MSA to do this work. Not wanting to take up overtime (which is money!) we automated the whole process via Powershell (and converted the VM's to sparse/thin disks at the same time).

Wrapped up into a function, the script takes input from a CSV that is structured as follows (can be exported using the "Get-VM" command):












We 'balanced the VM's, hence the "0" in the CSV (and provision for it in the script). The function is as simple as running the following on a machine with Powershell installed:

start powershell d:\clients\XXX\Migrate-Thin.ps1 Datastore_1.csv Datastore_1

The script loops through each VM in turn, keeping inside the Virtual Centre maximums on SVMotion jobs, and migrating them from the existing storage to the new. We did all 100-odd VM's overnight, shipped them to London and had them running again on the new storage within 48 hours, all thinly provisioned as well.


Function Migrate-Thin($CSVFile,$DataStoreName)


$FatVMList = Import-CSV $CSVFile

foreach ($FatVM in $FatVMList)


if ($FatVM.Name -ne "0")


connect-viserver MyVCCServer -User ChangeMe\MeToo -Password MyPassWord

$VM = Get-VM $FatVM.Name

$vmView = $VM | Get-View -Property Name

echo $vmView.Name

$dsView = Get-Datastore -Name $DataStoreName | Get-View -Property Name

$spec = New-Object VMware.Vim.VirtualMachineRelocateSpec

$spec.datastore = $dsView.MoRef

$spec.transform = "sparse"

$vmView.RelocateVM($spec, $null)

disconnect-viserver -confirm:$False




$RunningSnapins = Get-PSSnapin

$LoadVMSnapin = $True

foreach($Snapin in $RunningSnapins)


if($ -eq "VMware.VimAutomation.Core")


$LoadVMSnapin = $False



if($LoadVMSnapin -eq $True){

$AddTheSnapin = add-PSSnapin VMware.VimAutomation.Core


$CSV = $args[0]

$DS = $args[1]

Migrate-Thin $CSV $DS

Thursday, 29 July 2010

Useful commands on ESX

After running into serious issues with stale VCB snapshots not committing properly, I'm making a note of useful command for future reference, as I can never remember these!

You can find all the snapshots on all the datastores visible from one host running this on the service console:

find /vmfs/volumes/ -name "*delta*" -type f -print0 | xargs -0 du --human-readable --total
Useful to see only the lines that you need from a .vmx or .vmdk file.

alias sgrep='egrep -i --color "scsi[0-9]+:[0-9]+.present | scsi.*filename | vmdk | parent | CID"'
To see the space available on all the Datastores

vdf -h
To see the space available in the Datastore we are located

vdf -h .
Is there a missing file? Look for it in all the Datastores.

find /vmfs/volumes/ -iname "*FILENAME*"

To see the evolution of the modifications on the files of the VM

watch -d "ls -lt *.vmdk ;date"
To have a nice colorful prompt
PS1='\[\e[0;31m\]\u@\h \W]$ \[\e[m\]\[\e[0;32m\]' ; export PS1

Thanks to Ruben @ VMUtils for this

Wednesday, 28 July 2010

Changing the ESX license server

I had to change and reset the license server on a number of hosts, and thanks to the brilliant Hugo Peeters, and some assistance from one of our scripting guru's here at work, I knocked up some Powershell to reach out to a cluster and check the status of it's license server. If the service is stopped, or not responding the correct server, it is corrected and restarted. It's also nicely wrapped into a function :)

Function Set-ESXLicenseHost


$ESXHosts = get-vmhost

ForEach ($VMHost in $ESXHosts)


$SI = Get-View ServiceInstance

$LM = Get-View $SI.Content.LicenseManager

$VMHostView = Get-VMHost $VMHost


$VMHostRef = $VMHostView.MoRef

$LicUse = $LM.QueryLicenseUsage($VMHostRef)


If ($LicUse.sourceAvailable -ne 'True')


write-output "Not Available - resetting service"

$hostref = ($VMHost


$LicServer = “INT01VCC01”

$licsrc = New-Object VMware.Vim.LicenseServerSource

$licsrc.LicenseServer = $LicServer





write-output "Available"




Tuesday, 27 July 2010

No Performance Stat History In Virtual Center 4.0

Before Christmas we moved our VCC over from a SQL2000 to SQL2005 x64 cluster. This was to support the recent upgrade to vSphere 4, as well as move the system from an ageing pair of Windows 2003 servers to new HP DL380 G6 systems running Windows 2008. After many nights of issues we got everything moved, and thought we had put it all to bed...

A call was logged today and passed to the team I work in to look at an issue with historical Performance stats not running in one of our VCC's. We had a serious issue on Friday that was solved by the KB 1000125, and I thought this may have been the cause. Taking a look, it seems that actually none of the SQL jobs had  been migrated when the database came over. Our standard backup and tuning jobs apply to all non-system DB's, so these were OK but there were no VS4 specific jobs enabled for this server. I followed KB 1004382 to add the scripts back in, and kicked off the first job.

After a waiting a while, I checked - Row Count on VPX_HIST_STAT1 = 62953799

Guess I'll leave it a while then.