ESX Script Extracts and Examples

Daily ESX Up/Down Check

Simple script that runs every morning just before I start work to provide a basic sanity check email to glance through over the first tea of the day.

#############################################################################################
#
#  ESX Checker
#
#############################################################################################
#
# By Simon Strutt
#
# Version 1 - Aug 10
# - Initial creation
#
# Version 2 - Oct 10
# - Fixed VIServer handling (was never good but stuffed by upgrade to PowerCLI 4.1)
# - Improved email in case of vCentre connection failure
#
# Version 2.1 - Nov 10
# - Minor text changes
#
# Version 2.2 - Nov 10
# - Added InvalidLogon catch to vCentre logon to prevent a/c lockout
#
# Version 3 - Dec 10
# - Truncated displayed ESX hostname (stripped domain)
# - Added "since <datetime>" info to ESX's that aren't in a good state
# - Bugfix: Incorrect email config used when VC list config file load failed
#
# Version 3.1 Jan 11
# - Workaround: Powershell/MS DateTime object returns incorrect (US) date format
#
# Version 3.2 Jan 11
# - Bugfix: Extraneous " hrs" on end of v3.1 fixed datetimes 
#
#############################################################################################

$DateFormat = "%R hrs, %a %d %b %Y"

$EmailBody = "Results of ESX status check as of " + (get-date -uFormat "$DateFormat.`n`n")
$EmailRecipients = "person1@domain.com,person2@domain.com"
$EmailSender = "VI-Mgmt@domain.com"

$UserFile = "User.fil"
$PassFile = "Pass.fil"

$smtp = New-Object Net.Mail.SmtpClient -arg "smtp-server.domain.com"
$VCAlertToSend = 0
$ESXAlertToSend = 0
$start = Get-Date

try {
    $ToCheck = Import-CSV "ESX-Check.csv"
} catch {
    $EmailBody = "ERROR: Failed to load config file, no checks have been done.`n"
    $EmailBody += $_
    $smtp.Send($EmailSender, $EmailRecipients, "ESX-Check: Check failed", $EmailBody)
    Exit
}

$pass = Get-Content $PassFile -errorAction Stop | ConvertTo-SecureString 
$user = Get-Content $UserFile -errorAction Stop
$cred = New-Object System.Management.Automation.PsCredential($user, $pass)

foreach ($vc in $ToCheck) {
    Write-Host "Checking ESXs on" $vc.vc
    $VChasBadESX = 0
    try {
        $VCconn = Connect-VIServer -Server $vc.vc -Credential $cred -NotDefault -errorAction "Stop"
    } catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidLogin] {
        $EmailBody += ($vc.vc).ToUpper() + "`t" + "Unable to connect to vCentre, invalid logon error !!`n`n"
        $EmailBody += "Abandoning further script processing in order to prevent potential account lockout.`n"
        $VCAlertToSend = 1
        Break
    } catch {
        $EmailBody += ($vc.vc).ToUpper() + "`t" + "Unable to connect to vCentre, ESX's not checked !!`n"
        $VCAlertToSend = 1
        Continue
    }
    $ESXs = Get-VMHost -Server $vc.vc | Where {$_.State -ne "Connected"} | Sort -property Name
    if ($ESXs) {
        foreach ($ESX in $ESXs) {
            # Attempt to work out when ESX went into its bad state
            Switch ($ESX.State) {
                NotResponding {
                    $events = Get-VIEvent -Entity $ESX -Types Error | Where {$_.fullFormattedMessage -like "*is not responding*"}
                }
                Maintenance {
                    $events = Get-VIEvent -Entity $ESX -MaxSamples 1000 | Where {$_.fullFormattedMessage -like "*has entered maintenance mode*"}
                }
                Disconnected {
                    $events = Get-VIEvent -Entity $ESX -MaxSamples 1000 | Where {$_.fullFormattedMessage -like "*Disconnected from*"}
                }
                Default {
                    $events = 0
                }
            }
            If ($events) {
                $SinceString = " since " + (Get-Date $events[0].CreatedTime -uFormat $DateFormat)
            } else {
                $SinceString = ""
            }
                    
            $EmailBody += $vc.vc + "`t" + ($ESX.Name.Split(".")[0]).ToUpper() + "`t" + $ESX.State + $SinceString + "`n"
            $ESXAlertToSend = 1
            $VChasBadESX = 1
        }
    }
    if (!$VChasBadESX) {
        $EmailBody += $vc.vc + "`t" + "All good`n"
    }
    Disconnect-VIServer -Server $VCconn -Confirm:$false
}

$end = Get-Date

$EmailBody += "`n`n`nCheck started      : " + (Get-Date $start -uFormat $DateFormat) + "`n"
$EmailBody += "Check finished     : " + (Get-Date $end -uFormat $DateFormat) + "`n" 
$EmailBody += "Generated by script: " + ($MyInvocation.MyCommand.Name) + "`n"
$EmailBody += "Sent from machine  : $env:computername"

$EmailBody
Write-Host "Sending email..."
if ($VCAlertToSend) {
    $smtp.Send($EmailSender, $EmailRecipients, "ESX-Check: Unable to connect to some/all vCentres", $EmailBody)
} elseif ($ESXAlertToSend) {
    $smtp.Send($EmailSender, $EmailRecipients, "ESX-Check: Some ESX's are NOT CONNECTED", $EmailBody)
} else {
    $smtp.Send($EmailSender, $EmailRecipients, "ESX-Check: All connected OK", $EmailBody)
}
if (-not $?) {
    Write-Host "SMTP send failed!!"
    Start-Sleep(30000)
}

This above script uses two files ($UserFile and $PassFile) to provide authentication credentials to the script without needing the password to exist in unencrypted plain text. To create your own see the script at the bottom of the Store Password Securely section.

Alternatively, remove/comment out the lines that reference the $UserFile and $PassFile variables and use the following line to create the $cred variable instead...

$cred = Get-Credential

ESX NIC Info

Provides a list of all vmnic speeds for ESX's managed by the vCentre you're connected to.

$ESXs = Get-VMHost

Foreach ($esx in $ESXs) {
    Write-Host $esx.Name $esx.State
    $pNICs = (Get-VMHost -Name "MyESX*" | Get-View).Config.Network.Pnic

    $result = @{}
    Foreach ($pNIC in $pNICs) {
        $result[$pNIC.Device] = $pNIC.LinkSpeed.SpeedMB
    }
    $result = $result.GetEnumerator() | Sort-Object -Property Name
    $result
    Write-Host
}

Discovered Networks Hint

This function provides the discovered network hints for the network interface its passed. Bear in mind that its just a hint, for an ESX to be aware of a particular vLAN it needs to see traffic. If there's no traffic (eg a newly set-up ESX with no VMs) it will show nothing.

Adapted from the following article on the VMware site blog http://blogs.vmware.com/vipowershell/2010/02/how-to-find-out-what-vlans-your-esx-hosts-can-really-see.html

function Get-ObservedIPRange {
	param(
		[Parameter(Mandatory=$true,ValueFromPipeline=$true,HelpMessage="Physical NIC from Get-VMHostNetworkAdapter")]
		[VMware.VimAutomation.Client20.Host.NIC.PhysicalNicImpl]
		$Nic
	)

	process {
		$hostView = Get-VMHost -Id $Nic.VMHostId | Get-View -Property ConfigManager
		$ns = Get-View $hostView.ConfigManager.NetworkSystem
		$hints = $ns.QueryNetworkHint($Nic.Name)

		foreach ($hint in $hints) {
			foreach ($subnet in $hint.subnet) {
				$observed = New-Object -TypeName PSObject
				$observed | Add-Member -MemberType NoteProperty -Name Device -Value $Nic.Name
				$observed | Add-Member -MemberType NoteProperty -Name VlanId -Value $subnet.VlanId
				$observed | Add-Member -MemberType NoteProperty -Name IPSubnet -Value $subnet.IPSubnet
				$observed | Add-Member -MemberType NoteProperty -Name BitRatePerSec -Value $nic.BitRatePerSec
				Write-Output $observed
			}
		}
	}
}

# Example use:

$result = Get-VMHost MyESX* | Get-VMHostNetworkAdapter | Where {$_.Name -Match ".*vmnic*"} | Get-ObservedIPRange | Sort-Object -Property Device, VlanId
$result | Export-Csv -path ESX-vLANs-Observed.csv

Add VLANs/PortGroups to Standard Switch

Simple script to add new port groups to an ESX. Create a CSV file in the format as show in the example below. Then update the $esx and $vSwitch variables as appropriate in the script and run in a PowerCLI session connected to the vCenter.

$PG_List = "ESX-Add-PortGroups.csv"        # CSV to list new port groups in

$esx = "esx-name*"                         # ESX to add port groups to
$vSwitch = "vSwitch0"                      # vSwitch (standard not dvSwitch) on ESX to add port groups to

$Switch = Get-VirtualSwitch -Name $vSwitch -VMHost (Get-VMHost $esx)

try {
    $PGs = Import-CSV $PG_List
} catch {
   Write-Host("ERROR: Failed to load list of PortGroup's")
   Exit
}

foreach ($pg in $PGs) {
    Write-Host ("Adding VLAN " + $pg.VLAN + " " + $pg.Name)
    New-VirtualPortGroup -VirtualSwitch $Switch -Name $pg.Name -VLanId $pg.VLAN
}

Example contents of CSV file

VLAN	Name
101	Testing
102	Development
103	ABC Production

ESX CDP Info

Adapted from posted by LucD on VMware forum http://communities.vmware.com/message/977487

Get-VMHost | Sort -Property Name | %{Get-View $_.ID} | %{$esxname = $_.Name; Get-View $_.ConfigManager.NetworkSystem} | %{
  foreach($physnic in $_.NetworkInfo.Pnic){
    $pnicInfo = $_.QueryNetworkHint($physnic.Device)
    foreach($hint in $pnicInfo){
      Write-Host $esxname $physnic.Device $hint.connectedSwitchPort.DevId $hint.connectedSwitchPort.PortId
    }
  }
}

ESX Log Tail

Note that the below only seems to work for ESX3.5, I gave up trying to get it work for both v3 and v4, its a lot easier to just enable ESXi Tech Support Mode and tail the log

Pointless for ESX (you can just use tail from the Service Console on the appropriate log), but a godsend it you're using ESXi (and haven't got access to tail the ESX logs).

$PollInterval = 1000     # msec (default - can be overrided by user)

#Get ESX to poll
$ESXs = Get-VMHost | Sort-Object -Property Name
Write-Host "`nWhich ESX's log do you want to tail?"
$num = 0
foreach ($esx in $ESXs) {
    Write-Host "[$num]" $ESXs[$num].Name
    $num++
}
$num = 0
$num = Read-Host "? [$num] "
$ESX = Get-VMHost $ESXs[$num]
Write-Host " "

# Get Log keys to look at
$keys = Get-LogType -VMHost $ESX
Write-Host "`nWhich log do you want to tail?"
$num = 0
foreach ($key in $keys) {
    Write-Host "[$num]" $keys[$num].Key
    $num++
}
$num = 0
$num = Read-Host "? [$num] "
$logKey = $keys[$num].Key

# Set polling interval
$PollInterval = Read-Host "`nWhat polling interval do you want to use (msec)? [$PollInterval]"

Write-Host "`nStarting log tail (press Ctrl+C to escape)...`n"

# First pass of log
$ESXLog = Get-Log $LogKey -VMHost $ESX

#Display last $lines
$LineNo = $ESXLog.LastLineNum - $lines
While ($LineNo -le $ESXLog.LastLineNum) {
    Write-Host $ESXlog.Entries[$LineNo]
    $LineNo++
}

#Polling loop
While (1) {
    Start-Sleep -Milliseconds $PollInterval
    try {
        $ESXLog = Get-Log $logKey -VMHost $ESX -StartLineNum $LineNo
    } catch {
        if (Select-String -InputObject $_ -pattern "The 0 argument is less than the minimum allowed range of 1" -Quiet) {
            Write-Host "ESX log is rolling over..."
            $ESXLog = Get-Log $logKey -VMHost $ESX
        } else {
            Write-Host "UNEXPECTED ERROR: $_"
            Exit
        }
    }
    $ESXlog.Entries
    $LineNo = $ESXLog.LastLineNum
}

ESX BIOS, NIC and HBA Driver Versions

Getting BIOS, NIC and HBA driver versions is dependant on what's reported to the ESX by the hardware vendor's CIM provider. This script was written using various HP servers and may well only work for them.

Over time I've found the method below to be unreliable, as the data available from vCentre varies in quality, an alternative (which is more reliable, but requires the use of external libraries) can be found here - http://vblog.strutt.org.uk/2012/04/esx-hba-and-nic-driverfirmware-versions/

# ===============================================================
# ESX Inventory Getter
# ===============================================================
# Simon Strutt        November 2010
# ===============================================================
#
# Version 1
# - Initial Creation
#
# Version 2 - Dec 2010
# - Added BiosVer, CpuCores, CpuModel, HbaModel, NicModel
# - Bugfix: Corrected VC connection handling
# - Removed dependency on obsoleted properties (from upgrade to PowerCLI v4.1.1)
#
# Limitations
# - Tested on HP DL380 (G7), BL465 (G7), BL495 (G6), BL685 (G5)
# - Supports 1 distinct HBA model, 2 distinct NIC models
#
# ================================================================

$start = Get-Date
$OutputFile = "ESXs.csv"
$UserFile = "User.fil"
$PassFile = "Pass.fil"                           # Encrypted file to store password in
$Results = @()

$VCs = @()
$VCs += "vCentreA"                               # Hostname of Virtual Center
$VCs += "vCentreB"                               # Hostname of Virtual Center (repeat if you have more VC's, delete if you've only one)

Start-Transcript -Path ESX-Inventory.log
Write-Host "Started script run at $start"

# Function-U-like ===================================================================================

Function Get-NicDriverVersion ($view, $driver) {
    # Gets the CIM provided driver version (tallies up driver name with CIM software component)
    $result = ($view.Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo | Where {$_.Name -like $driver + " driver*"} | Get-Unique).Name
    $result = ([regex]::Matches($result, "(\b\d)(.*)(?=\s)"))[0].Value         # HP regex to extract "2.102.440.0" for example
    $result
}

# =================================================================================================== 

# Load password credential from encrypted file
$pass = Get-Content $PassFile | ConvertTo-SecureString
$user = Get-Content $UserFile
$cred = New-Object System.Management.Automation.PsCredential($user, $pass)

foreach ($vc in $VCs) {
    
    # Connect to VC
    try {
        Log("Connecting to " + $vc)
        $VCconn = Connect-VIServer -Server $vc -Credential $cred -errorAction "Stop"
    } catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidLogin] {
        Log("Unable to connect to vCentre, invalid logon error !!")
        Log("Abandoning further script processing in order to prevent potential account lockout.")
        Break
    } catch {
        Log("Unable to connect to vCentre - " + $_)
        Continue
    }
    
    # Get ESX objects
    Log("Getting list of ESXs to check on " + ($vc) + "...")
    $ESXs = Get-VMHost -Server $vc | Sort -property Name
    Log("Got list of " + ($ESXs.Count) + " ESXs to check") 
    
    ForEach ($ESX in $ESXs) {
        $row = "" | Select VC, Cluster, Name, IP, Version, Make, Model, BiosVer, CpuCores, CpuModel, HbaModel, HbaDriver, HbaDriverVer, Nic1Model, Nic1Driver, Nic1DriverVer, Nic2Model, Nic2Driver, Nic2DriverVer
        Log($ESX.Name)
        
        # Store objects which get re-used for efficiency
        $ESXview = Get-View -VIObject $ESX
        $VMHostNetworkAdapter = Get-VMHostNetworkAdapter -VMHost $esx 
        
        # Get the basics
        $row.VC = $vc
        $row.Cluster = $ESX.Parent
        $row.Name = $ESX.Name.Split(".")[0]
        $row.IP =  ($VMHostNetworkAdapter | Where {$_.ManagementTrafficEnabled -eq "True" -or $_.DeviceName -like "vswif*" -or $_.Name -eq "vmk0"}).IP
        $row.Version = $ESX.Version + " (" + $ESX.Build + ")"
        $row.Make = $ESX.Manufacturer
        $row.Model = $ESX.Model
        
        # Now onto the more ESX version / hardware specific stuff (new versions of hardware will require further work below)
        
        # BIOS
        if ($ESXView.Hardware.BiosInfo) {     # Works on some systems 
            $row.BiosVer = $ESXview.Hardware.BiosInfo.BiosVersion + " " + $ESXview.Hardware.BiosInfo.ReleaseDate.ToString("yyyy-MM-dd")   # Need date for HP servers as they use same version no for diff versions!
        } else {
            $row.BiosVer = ($ESXview.Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo | Where {$_.Name -like "*BIOS*"}).Name
            $row.BiosVer = ([regex]::Matches($row.BiosVer, "[A-Z]\d{2} 20\d{2}-\d{2}-\d{2}"))[0].Value           # HP regex to extract "A19 2010-09-30" for example
        }
        
        # CPU info
        $row.CpuCores = $ESXview.Hardware.CpuInfo.NumCpuCores.ToString() + " (" + $ESXview.Hardware.CpuPkg.count + "x" + $ESXview.Hardware.CpuPkg[0].ThreadId.count + ")"
        $row.CpuModel = $ESX.ExtensionData.Summary.Hardware.CpuModel
        
        # HBA info (script assumes only one HBA model in use)
        $row.HbaModel = ($ESX.ExtensionData.Config.StorageDevice.HostBusAdapter | Where {$_.Key -like "*FibreChannel*"})[0].Model
        $row.HbaDriver = ($ESX.ExtensionData.Config.StorageDevice.HostBusAdapter | Where {$_.Key -like "*FibreChannel*"})[0].Driver     # Includes version for ESX3
        $row.HbaDriverVer = ($ESXview.Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo | Where {$_.Name -like "*" + $row.HbaDriver + "*"}).Name
        $row.HbaDriverVer = ([regex]::Matches($row.HbaDriverVer, "(\b\d)(.*?)(?=\s)"))[0].Value
        
        # NIC info (script only supports two distinct NIC types)
        $nics = $ESXview.Hardware.PciDevice | Where {$_.ClassId -eq 512} | Select VendorName, DeviceName, Id | Sort-Object -Property DeviceName -Unique
        if ($nics.count) {
            $row.Nic1Model = $nics[0].VendorName + " " + $nics[0].Devicename
            $row.Nic2Model = $nics[1].VendorName + " " + $nics[1].Devicename
            
            # Use PCI ID to match up NIC hardware type with driver name
            $row.Nic1Driver = ($VMHostNetworkAdapter | Where {$_.ExtensionData.Pci -eq $nics[0].Id}).ExtensionData.Driver
            $row.Nic2Driver = ($VMHostNetworkAdapter | Where {$_.ExtensionData.Pci -eq $nics[1].Id}).ExtensionData.Driver
            
            $row.Nic1DriverVer = Get-NicDriverVersion $ESXview $row.Nic1Driver
            $row.Nic2DriverVer = Get-NicDriverVersion $ESXview $row.Nic2Driver
         } else {
            # Annoyingly, $nics is not an array if there's only one NIC type, hence seperate handling
            $row.Nic1Model = $nics.VendorName + " " + $nics.Devicename
            $row.Nic1Driver = ($VMHostNetworkAdapter | Where {$_.ExtensionData.Pci -eq $nics.Id}).ExtensionData.Driver
            $row.Nic1DriverVer = Get-NicDriverVersion $ESXview $row.Nic1Driver
        }
        
        $Results = $Results + $row
    }
    Disconnect-VIServer -Server $VCconn -Confirm:$false
}
$Results | Format-Table *
Log("Writing results to $OutputFile...")
$Results | Export-Csv -path $OutputFile -NoTypeInformation

Log("All completed")
Stop-Transcript

This above script uses two files ($UserFile and $PassFile) to provide authentication credentials to the script without needing the password to exist in unencrypted plain text. To create your own see the script at the bottom of the Store Password Securely section.

Alternatively, remove/comment out the lines that reference the $UserFile and $PassFile variables and use the following line to create the $cred variable instead...

$cred = Get-Credential