Tuesday, January 24, 2017

VMware vSphere 6.0 PSC and SSO Domain useful resources

I do not have real numbers but it seems obvious and logical that SMB and midrange customers are adopting the latest VMware software much quicker then large enterprise customers. To be more precise, they are probably already running vSphere 6.0 and planing to upgrade to 6.5 now or soon. Some of them just waiting for 6.5 U1 which is expected soon.

On the other hand, the largest VMware customers are logically more conservative and starting migrations from vSphere 5.5 to 6.0 just now, in time of writing this article (beginning of 2017). These large customers have significantly larger scale therefore their PSC/SSO topology is much more complex.

During last few weeks I have discussed some vSphere 6 PSC/vCenter topology design decision points with these customers and I have decided to write down blog post about few useful, publicly available, resources / documents for such discussions.

First and foremost,  FAQ below is the most comprehensive VMware KB article about this topic.

FAQ: VMware Platform Services Controller in vSphere 6.0 (2113115)
https://kb.vmware.com/kb/2113115

The most surprised information, even for long time VMware customers, are following two Q&A's from FAQ above.

Q: Can I merge two vSphere Domains together?
A: No, there is no way to merge two vSphere domains together.

Q: Can I get Enhanced Linked Mode (ELM) between two, separate vSphere domains?
A: No, Enhanced Linked Mode requires that all PSCs be in the same domain and replicating. Since two separate vSphere Domains do not have a means of replicating, the new APIs that provide ELM cannot display the contents of both domains.

What does it mean?
Well, if you have multiple independent vSphere 5.5 SSO domains and you want to merge them, you have to do it in vSphere 5.5 before upgrade to 6.0 because you will not be able to do so in vSphere 6 and later.
Note: I do not know how it will change in longer term but it is the true even for vSphere 6.5 which is the latest version in time of writing this blog post.

Q: One of my customers asked me if the same vSphere SSO name (vsphere.local) in their two separate datacenters means that it is the same vSphere domain.
A: No. If you do not have replication between domains, there are not the same domain even they have the same name.

Another good question, you have to ask yourselves is, if you should or should not merge your vSphere domains. The typical reason for single vSphere domain is requirement for Enhanced Linked Mode (ELM). What Enhanced Linked Mode will give you? Below are several benefits of ELM:
  • You can log in to all linked vCenter Server systems simultaneously with a single user name and password.
  • With Enhanced Linked Mode, you can view and search across all linked vCenter Server systems. This mode replicates roles, permissions, licenses, and other key data across systems.
  • You can view and search the inventories of all linked vCenter Server systems within the vSphere Web Client.
  • Roles, permission, licenses, tags, and policies are replicated across linked vCenter Server systems.
  • You can use WebClient GUI to do cross vCenter vMotion

However, any technology has some limits. In case of vSphere, we should always look at vSphere Configuration Maximums. The relevant information from configuration maximums are

  • Maximum PSCs per vSphere Domain - 8
  • Maximum PSCs per site, behind a load balancer - 4
  • Maximum number of VMware Solutions connected to a single PSC - 4
  • Maximum number of VMware Solutions in a vSphere Domain - 10
What are VMware Solutions?

A VMware Solution is defined as a product that creates a Machine Account and one or more Solution User (a collection of vSphere services) within the VMware Directory Service when the product is joined to the PSC, thus the vSphere Domain. The Machine Account and Solution User(s) are used to broker and secure communication between other Solutions available within the vSphere environment. In order to count against these maximums, the Machine Account and Solution Users must befully integrated with all of the PSC's available feature sets (Identity Management and Authentication Brokering, Certificate Management, Licensing, etc.) such that the product makes full use of the PSC. At this time, only vCenter Server is defined as a fully integrated solution and counts against these maximums. Partially integrated solutions, such as vCenter Site Recovery Manager, vCloud Director vRealize Orchestrator, vRealize Automation Center, and vRealize Operations, do not count against these defined maximums.
So vCenters are the only solutions which counts into maximum of 10 VMware solutions. 

Now, when you know if you really need and want to merge vSphere domains it must be done in vSphere 5.5 because in vSphere 6 it is not possible.

I was asked by one of my customers, where is written that vSphere domain merging is supported and how it can be done.

Bellow are two very nice blog post written by blogger Thom Greene ...

Merging SSO Domains in vCenter 5.5 part 1: Why?
https://www.thomgreene.com/blog/2016/11/4/merging-sso-domains-in-vcenter-55-part-1-why

Merging SSO Domains in vCenter Server 5.5 pt 2: How?
https://www.thomgreene.com/blog/2016/11/7/merging-sso-domains-in-vcenter-server-55-pt-2-how

and very detailed blog post of Andreas Peetz referred by Thom in his posts.

Re-pointing vCenter Server 5.5: A Survival Guide to KB2033620
https://www.v-front.de/2016/03/re-pointing-vcenter-server-55-survival.html

... but resources above are not VMware official documents so where are VMware official documents? Andreas' blog posts are referring to following VMware KB's

Migrating two VMware vCenter Single Sign-On embedded VMware vCenter Servers in the same VMware vCenter Single Sign-On domain (2130433)
https://kb.vmware.com/kb/2130433

How to repoint and re-register vCenter Server 5.1 / 5.5 and components (2033620)
https://kb.vmware.com/kb/2033620

VMware vCenter Server 5.1/5.5 fails to start after re-registering with vCenter Single Sign-On (2048753)
https://kb.vmware.com/kb/2048753

Old but still informative blog post ... vSphere Datacenter Design – vCenter Architecture Changes in vSphere 6.0 – Part 1

Useful VMware KB article before upgrade to vSphere 6.5

I have just found following very useful VMware KB articles and blog posts which should be read before any vSphere 6.5 upgrade and design refresh.

Update sequence for vSphere 6.5 and its compatible VMware products (2147289)
https://kb.vmware.com/kb/2147289 

Important information before upgrading to vSphere 6.5 (2147548)
https://kb.vmware.com/kb/2147548

Best practices for upgrading to vCenter Server 6.5 (2147686)
https://kb.vmware.com/kb/2147686

Platform Services Controller Topology Decision Tree
https://blogs.vmware.com/vsphere/2016/04/platform-services-controller-topology-decision-tree.html

Reconfigure a Standalone vCenter Server with an Embedded Platform Services Controller to a vCenter Server with an External Platform Services Controller
link

How to repoint vCenter Server 6.x between External PSC within a site (2113917)
https://kb.vmware.com/kb/2113917


Wednesday, January 11, 2017

Using esxtop to identify storage performance issues for ESX / ESXi

ESXi performance are exposing to administrators through vSphere Clients. You can see real-time performance statistics which are collected in 5 minute intervals where each interval consists of fifteen 20 seconds samples. It is obvious that 20 second sample is pretty large for storage performance where we are working in mili or even micro second scale.
20 seconds contains 20,000 milliseconds
Let's be clear here, we will never have full visibility but smaller monitoring sample will give as better clue what is really happening inside the system. It is similar to microscope device.

The smallest monitoring samples can be achieved by ESXi utility ESXTOP. The default esxtop delay between monitoring points (sample) is 5 seconds. However, it can be lowered up to 2 seconds by parameter -d 2

For real analytics the esxtop data must be exprted to external file. In esxtop terminology it is batch mode and it is achieved by parameter -b 

Another important factor is what statistics (metrics) we are going to collect. The best is to collect all statistics because during performance analytics you have to correlate multiple values against each other. It is achieved by parameter -a

And last parameter is -n which defines how many iterations you want to perform in batch mode. So in example below we will have 30 iterations with delay between each other 2 seconds. So we will do total monitoring for 60 seconds.

esxtop -b -a -d 2 -n 30 > esxtop-data.csv

For all esxtop parameters see screenshot below.

 [root@esx11:~] esxtop -h  
 usage: esxtop [-h] [-v] [-b] [-l] [-s] [-a] [-c config file] [-R vm-support-dir-path]   
         [-d delay] [-n iterations]  
        [-export-entity entity-file] [-import-entity entity-file]   
        -h prints this help menu.  
        -v prints version.  
        -b enables batch mode.  
        -l locks the esxtop objects to those available in the first snapshot.  
        -s enables secure mode.  
        -a show all statistics.  
        -c sets the esxtop configuration file, which by default is .esxtop60rc  
        -R enables replay mode.  
        -d sets the delay between updates in seconds.  
        -n runs esxtop for only n iterations. Use "-n infinity" to run esxtop forever.  
        -----Experimental Features-------------  
        -export-entity writes the entity ids into a file, which can be modified  
         to select interesting entities.  
        -import-entity reads the file of selected entities. If this opion   
         is used, esxtop only shows the data for the selected entities.  

It is important to know, that esxtop will give you significantly more statistics you can see in vSphere Client level. That's another important benefit of esxtop. But each benefit has also some drawbacks or impact. The impact is, that single esxtop output line can have several thousands statistic counters. For example ESXi 6.0 host with just 2 running VMs in my home lab has 27,314 counters. My customer's product ESXi host has over 330,000 counters! So the output file can be pretty large in case you run it for 24 hours. Count on it.

In the file are very interesting counters. Following counters for physical disk devices are the most interesting
### Reponse times
Average Guest MilliSec/Command
Average Kernel MilliSec/Command
Average Queue MilliSec/Command
Average Queue MilliSec/Read
Average Driver MilliSec/Command
Average Driver MilliSec/Write
### Queue
Adapter Q Depth
### IOPS
Reads/sec
Writes/sec
Commands/sec
### MB/s
MBytes Read/sec
MBytes Written/sec"
### Split commands
Split Commands/sec
### SCSI Reservations
Reserves/sec
Failed Reserves/sec
Conflicts/sec
### Failures
Failed Commands/sec
Failed Reads/sec
Failed Writes/sec
Failed Bytes Read/sec
Failed Bytes Written/sec
Aborts/sec
Resets/sec
Some of above counters are not available in vSphere Client but the big benefit is that esxtop will give you data in 2 second interval which is much better granularity.

I hear your questions - So what now? How to analyze esxtop output file?
Well, you can replay it back in esxtop or you can use any of following tools

  • VisualEsxtop
  • perfmon
  • excel
  • esxplot
To be honest, none of tools above fulfilled my requirements therefore I'm writing my own python script for esxtop output analysis.

I will blog about it in next post when script will be good enough for public usage and published on github.

Stay tuned.

Tuesday, December 20, 2016

ESXi sched-stats command

Frank Denneman has shared on twitter very interesting ESXi command to show CPU scheduling statistics and information.

@FrankDenneman tweet

There are not so much information about this command so one have to rely on command help ...

[root@esx01:~] sched-stats -h
Usage:
-c   : use vsi-cache instead of live kernel
-t   : specify the output type from the following list
          :    vcpu-state-times
          :    vcpu-run-times
          :    vcpu-state-counts
          :    vcpu-run-states
          :    vcpu-alloc
          :    vcpu-migration-stats
          :    vcpu-load
          :    vcpu-relations
          :    vcpu-comminfo
          :    ncpus
          :    cpu
          :    pcpu-stats
          :    pcpu-load
          :    overhead-histo
          :    sys-service-stats
          :    run-state-histo
          :    wait-state-histo
          :    coSched-stats
          :    groups
          :    worldlet-state-times
          :    worldlet-state-counts
          :    worldlet-mig-state
          :    worldlet-load
          :    worldlet-relations
          :    worldlet-comminfo
          :    power-pstates
          :    power-cstates
          :    numa-clients
          :    numa-migration
          :    numa-cnode
          :    numa-pnode
          :    numa-global
-f          : ignore version check
-w          : only show stats of the specified world
-p          : only show stats of the specified pcpu
-m          : only show stats of the specified module
-r          : reset scheduler statistics
-s : 1 to enable advanced cpu sched stats gathering, 0 to disable.
-l , : comma separated list of ids to restrict the report to;
                (not supported by all reports)
-k          : check the correctness of scheduling stats
-v          : verbose
-h          : print friendly help message

Note:
Sched-stats reads the stats data from vmkernel for each vcpu one
by one via the VSI interface. Since the scheduling stats may
continue to change during the VSI calls, what's reported by
sched-stats is not a consistent snapshot of the kernel stats.
But the inconsistency is expected to be small.

Sunday, December 11, 2016

The OVF package is invalid or could not be read.

I have just tried to deploy NSX Manager 6.2.4 virtual appliance downloaded from VMware site through WebClient. Following error message popup ...
"The OVF package is invalid or could not be read."
It sounds like corrupted file but it is very rare as it was successfully downloaded  directly from my.vmware.com.

I have double checked download and quickly realize what is wrong. OVF file should be XML file with references to vmdk files (disk images) but this OVF file was just a single large file. The single large file is typically OVA which is an archive of OVF.  OVA is nothing else then zipped TAR (aka tgz). I tried to rename file to .tgz, extract the file and voila ... there was real OVF and VMDK file. See. screenshot below

OVF directory

Deployment of real OVF (ovf descriptor + vmdk files) was successful so it is workaround #1.

Workaround #2  is just renaming file extension of downloaded OVF file (VMware-NSX-Manager-6.2.4-4292526.ovf) to OVA (VMware-NSX-Manager-6.2.4-4292526.ova) and deploy OVF/OVA again.

Hope this helps others struggling with virtual appliance deployment when OVF and OVA file extensions will be interchanged.

Note: I have double checked my.vmware.com and NSX Manager 6.2.4 is downloadable as file with .OVA extension so it was most probably already identified and repaired by VMware or something else weird happened during my download several days ago. 

Friday, November 25, 2016

PowerCLI script to report VMtools version(s)

This week I had a talk with one of my customers about VMtools versions and benefits of VMtools updates/upgrades. Few months ago I wrote another blog post discussing centralized version of VMtools repository (aka shared productLocker) so you can read it here.

Nevertheless, in large environments with multiple vCenters and thousands of VMs it is pretty handy to have single report with all VMtools versions to be able to asset and plan potential VMtools update/upgrade.

The problem with VMtools versions is that APIs reports just a single integer number of VMtools identification like 9354, 10246, etc. These numbers are also visible as Tools version on NGC/VI Client. However, vSphere admins are more familiar with human readable VMtools version also visible in guest OS which looks like 8.3.19, 9.4.6, 10.0.8, etc. For all VMtools version numbers and mapping you can check text file at https://packages.vmware.com/tools/versions

I have invested an hour or so to write simple PowerCLI script reporting VMtools identification and also human readable version for all VMs in particular vCenter or even across multiple vCenters. I have also included VM hardware version which can be handy information as well.

So initial PowerCLI code is below and the latest version will always appear on github here.

 ######################################################################################################################################  
 # Author: David Pasek  
 # E-mail: david.pasek@gmail.com  
 # Twitter: david_pasek  
 # Creation Date: 2016-11-25  
 #  
 # Use case:  
 #  Key use case of this script is to report VMtools from all VMs in vCenter  
 #  
 # Disclaimer:  
 #  Use it on your own risk. Author is not responsible for any impacts caused by this script.   
 ######################################################################################################################################  
 #   
 # CHANGE FOLLOWING VARIABLES BASED ON YOUR SPECIFIC REQUIREMENTS   
 # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv  
 #  
 # Report type - table, grid, file, csv-file  
  $REPORT_TYPE = "csv-file"  
 # Report file name without file extension. Extension is automatically added. File is created in current working directory.   
  $REPORT_FILE_NAME = "report-vmtools"  
 ######################################################################################################################################  
 Clear-Host  
 # We need VMware PowerCLI snapin  
 $o = Add-PSSnapin VMware.VimAutomation.Core  
 $o = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false  
 # Connect to vCenter  
 Write-Host "Connecting to vCenter ..."  
 $VC = Read-Host "Enter one vCentre Server or multiple vCenter servers delimited by comma."  
 Write-Host "Enter vCenter credentials ..."  
 $CRED = Get-Credential  
 Connect-VIServer -Server $VC -Credential $CRED -ErrorAction Stop | Out-Null  
 # Add new property (ToolsVersion) to VM  
 New-VIProperty -Name ToolsVersion -ObjectType VirtualMachine -ValueFromExtensionProperty 'Config.tools.ToolsVersion' -Force | Out-Null  
 # Initalize report  
 $Report = @()  
 foreach ($vm in Get-VM) {  
  # Numbers mapping is from https://packages.vmware.com/tools/versions  
  Switch ($vm.ToolsVersion) {  
      7302 {$GuestToolsVersion = "7.4.6"}  
      7303 {$GuestToolsVersion = "7.4.7"}  
   7304 {$GuestToolsVersion = "7.4.8"}  
   8192 {$GuestToolsVersion = "8.0.0"}  
   8194 {$GuestToolsVersion = "8.0.2"}  
   8195 {$GuestToolsVersion = "8.0.3"}  
   8196 {$GuestToolsVersion = "8.0.4"}  
   8197 {$GuestToolsVersion = "8.0.5"}  
   8198 {$GuestToolsVersion = "8.0.6"}  
   8199 {$GuestToolsVersion = "8.0.7"}  
   8290 {$GuestToolsVersion = "8.3.2"}  
   8295 {$GuestToolsVersion = "8.3.7"}  
   8300 {$GuestToolsVersion = "8.3.12"}  
   8305 {$GuestToolsVersion = "8.3.17"}  
   8306 {$GuestToolsVersion = "8.3.18"}  
   8307 {$GuestToolsVersion = "8.3.19"}  
   8384 {$GuestToolsVersion = "8.6.0"}  
   8389 {$GuestToolsVersion = "8.6.5"}  
   8394 {$GuestToolsVersion = "8.6.10"}  
   8395 {$GuestToolsVersion = "8.6.11"}  
   8396 {$GuestToolsVersion = "8.6.12"}  
   8397 {$GuestToolsVersion = "8.6.13"}  
   8398 {$GuestToolsVersion = "8.6.14"}  
   8399 {$GuestToolsVersion = "8.6.15"}  
   8400 {$GuestToolsVersion = "8.6.16"}  
   8401 {$GuestToolsVersion = "8.6.17"}  
   9216 {$GuestToolsVersion = "9.0.0"}  
   9217 {$GuestToolsVersion = "9.0.1"}  
   9221 {$GuestToolsVersion = "9.0.5"}  
   9226 {$GuestToolsVersion = "9.0.10"}  
   9227 {$GuestToolsVersion = "9.0.11"}  
   9228 {$GuestToolsVersion = "9.0.12"}  
   9229 {$GuestToolsVersion = "9.0.13"}  
   9231 {$GuestToolsVersion = "9.0.15"}  
   9232 {$GuestToolsVersion = "9.0.16"}  
   9233 {$GuestToolsVersion = "9.0.17"}  
   9344 {$GuestToolsVersion = "9.4.0"}  
   9349 {$GuestToolsVersion = "9.4.5"}  
   9350 {$GuestToolsVersion = "9.4.6"}  
   9354 {$GuestToolsVersion = "9.4.10"}  
   9355 {$GuestToolsVersion = "9.4.11"}  
   9356 {$GuestToolsVersion = "9.4.12"}  
   9359 {$GuestToolsVersion = "9.4.15"}  
   9536 {$GuestToolsVersion = "9.10.0"}  
   9537 {$GuestToolsVersion = "9.10.1"}  
   9541 {$GuestToolsVersion = "9.10.5"}  
   10240 {$GuestToolsVersion = "10.0.0"}  
   10245 {$GuestToolsVersion = "10.0.5"}  
   10246 {$GuestToolsVersion = "10.0.6"}  
   10247 {$GuestToolsVersion = "10.0.8"}  
   10249 {$GuestToolsVersion = "10.0.9"}  
   10252 {$GuestToolsVersion = "10.0.12"}  
   10272 {$GuestToolsVersion = "10.1.0"}  
   0   {$GuestToolsVersion = "Not installed"}  
   2147483647 {$GuestToolsVersion = "3rd party - guest managed"}  
      default {$GuestToolsVersion = "Unknown"}  
      }  
  $vminfo = New-Object -Type PSObject -Property @{  
             Name = $vm.Name  
     VMhardwareVersion = $vm.Version  
           ToolsVersion = $vm.ToolsVersion  
           GuestToolsVersion = $GuestToolsVersion  
      }  
  $Report += $vminfo  
 }  
 # Show report  
 Switch ($REPORT_TYPE) {  
      "grid"   { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Out-GridView }  
   "file"   { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Out-File -FilePath "$REPORT_FILE_NAME.txt" }  
   "csv-file" { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | export-csv "$REPORT_FILE_NAME.csv" }  
   default  { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Format-Table }  
 }  
 Disconnect-VIserver -Server $VC -Force -Confirm:$false  

The script can be configured for different report types by changing variable $REPORT_TYPE


Following report types are supported:
  • Standard PowerShell table report to output terminal (table)
  • PowerShell GridView (grid)
  • Standard text file including same content as table (file)
  • Comma separated values file (csv-file)
Different sample reports are visible in screenshots below

CSV-FILE
FILE
GRID
Hope this helps some other folks in VMware community. Any feedback or feature requests are welcome.

Tuesday, October 18, 2016

vSphere 6.5 announced so what is coming?

vSphere 6.5 has been announced on VMworld 2016 so you can ask yourself what it brings and why consider upgrade or at least upgrade plan.

It is obvious and expected that almost all vSphere 6.5 scalability limits will be increased. Configuration maximums like hosts per vCenter, powered on VMs per vCenter, hosts per cluster, VMs per cluster, vCenters in linked mode, etc are expected to increase. Theses limits are no longer limits for me but if you need it, just wait for vSphere 6.5 GA and double check well known document vsphere-65-configuration-maximums.pdf

However, vSphere users are usually looking for new features. So here they are ...

vCenter features
  • vCenter Server Appliance (aka VCSA) will be recommended as "First Choice" because the coolest new features are available just in VCSA. 
  • Platform Service Controller (PSC) will have out-of-the-box high availability for VCSA using PSC's. You will be able to achieve PSC RTO lower then 5 minutes. << UPDATE: unfortunately, this feature was not released in vSphere 6.5 release so let's hope it will be released in future vSphere 6.5 Updates.  
  • VCSA supports native High Availability support of vCenter service with RTO lower then 15 minutes.
  • VCSA has embedded vCenter Appliance Monitoring and Management to gain visibility into VCSA performance and capacity management including embedded vPostgreSQL database service.
  • VMware Update Manager (VUM) is be fully integrated into VCSA.
  • File level vCenter Server Backup and Restore is complementary backup method to existing VDP image backup. It will be possible to restore vCenter file level backup to fresh VCSA.
  • Content Libraries in vSphere 6.5 have additional features including the option to mount an ISO from a Content Library, update existing templates, and apply guest OS Customization Specifications during VM deployments. If Content Libraries reside on VCSA then you can also make use of vCenter HA, and native Backup and Restore, both new features to vSphere 6.5 mentioned above.
vSphere HA Cluster features
  • vSphere HA cluster wide restart ordering ability with intra-app dependencies during failover. It allows multi-tier application consistency during VM fail-overs.  It is also known as "vSphere HA Orchestrated Restart" because you can create VM to VM dependencies which will force specified VMs to perform HA restarts before others. You can also choose in your vSphere HA settings, when the next VM should begin restarting. At the power-on initiated command, when resources allocated, VMware Tools heartbeats,  etc. You can also set additional timeouts and delays if needed.
  • vSphere HA Admission Control - default Admission Control policy has changed from Slot Policy (Default until 6.5), to ‘Cluster Resource Percentage’. Any time you add or remove a host from the cluster, the failover capacity percentages will update, and the amount of resources required on each host will also be updated automatically.
  • Proactive HA - it integrates with the Server vendor’s monitoring software, via a Web Client plugin, which will pass detailed server health status/alerts to DRS, and DRS will react based on the health state of the host’s hardware. Yes, even the name is "Proactive HA" it is DRS functionality. Confusing? The name was chosen because it has positive impact on availability.
vSphere DRS Cluster features
  • Predictive DRS - it integrates DRS with vROps to provide placement and balancing decisions.
  • Network-Aware DRS - DRS takes physical NIC utilization in to consideration. Once a target host has been chosen for placement/load-balancing, DRS will then check to see if that host’s network is saturated (default is 80% utilization of connected uplinks, but can be configured with ‘NetworkAwareDrsSaturationThresholdPercent’. If the host is considered saturated, it will use a different target host
  • DRS Additional Option : VM Distribution - even distribution of VMs across cluster
  • DRS Additional Option : Memory Metric for Load Balancing - usage of active versus consumed memory for DRS recommendations
  • DRS Additional Option : CPU Over-Commitment - limit the number of vCPUs per pCPU in particular DRS cluster. Specific vCPU:pCPU ratio is set as advanced DRS option MaxVcpusPerClusterPct.
ESXi features and improvements
  • ESXi is pretty stable and best in class hypervisor. However even in this component you can expect some improvements.  For example I/O improvements because of RDMA / PVRDMA. PVRDMA (para-virtualized RDMA) is industry first virtualized RDMA and it allows virtualization of applications which require ultra low latency. And it supports live vMotion which SR-IOV does not.
  • ESXi core storage improvements - Support for 4K Native Drives in 512e mode, SE Sparse Default for VMFS, Automatic Space Reclamation, Support for 1024 devices and 4096 paths (versus 256 and 1024 in the previous versions)
vSphere management features
  • Auto Deploy and Image Builder will be full integrated into WebClient and Host profiles will be improved to smoothly support auto deploy.
  • vSphere Web Client usability and performance will be improved again. It is pretty important because C# client is not available for vSphere 6.5 so vSphere admins will rely on web client. HTML5-based vSphere Client should be included in 6.5 release.
  • Content library improvements - mount ISO directly from content library, customization during VM deployment, improved scale and performance, high availability along with VCSA
  • vSphere 6.5 introduces new REST-based APIs for VM Management
Storage related features
  • VVOLs 2.0 will bring data protection and replication along with support for MSCS, Oracle RAC, NFS 4.1 and SMP-FT.
  • VSAN - Virtual SAN iSCSI Service
  • VSAN - 2-Node Direct Connect with witness Traffic Separation for ROBO
  • VSAN - 512e drive support (still waiting for 4K native support)
Security related features
  • VM Encryption will be new feature to protect your VM data with tenants keys. It enables encryption on a per VM as well as per VMDK basis. It can be integrated with 3rd party Key Management Servers (KMS).
  • vSphere 6.5 also delivers enhanced audit-quality logging capabilities that provide more forensic information about user actions.
WebClient related features
  • In vSphere 6.5, the vSphere Web Client will have no dependency on Client Integration Plug-in (as it exists before).  For the Use Window Session Authentication functionality, you will need the new slimmed down Enhanced Authentication Plug-in, but the other functions (File upload/download, Deploy OVA/OVF) are replicated without CIP.
Conclusion

vSphere 6 is already very mature virtualization platform but vSphere 6.5 brings some very interesting enterprise features if you ask me. The most interesting features for me personally are
  • VCSA and PSC high availability
  • VVOLs 2.0
  • VM Encryption
  • REST-based APIs for vSphere Management
but all other features are cool and very handy as well. 

It is very common practice to wait for Update 1 before upgrading production environments but our labs and test environments are good candidates for vSphere 6.5 release when available. I'm eagerly waiting for GA.

Other related blog posts and resources:

Monday, October 17, 2016

VMware SIOC quick configuration in datacenter scale

I'm currently troubleshooting one weird high kernel latency (KAVG) issue and there is a suspicion that the issue can be somehow related to VMware SIOC which is widely use in customer's environment. To confirm or disprove the issue is really related to SIOC we can simply disable SIOC on all datastores and observe if it has positive impact on kernel latency.

Customer has lot of production datastores grouped in datastore clusters so following PowerCLI one liners can help with quick configuration and validation of SIOC settings across whole datacenter.

SIOC current state for all datastores in datastore clusters
 Get-DatastoreCluster | Get-Datastore | select-object name,type,StorageIOControlEnabled | Format-List -Property *  

Disable SIOC for all datastores in datastore clusters
 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $false | select-object name,type,StorageIOControlEnabled

Enable SIOC for all datastores in datastore clusters
 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $true | select-object name,type,StorageIOControlEnabled  

Thanks PowerCLI!