Wednesday, January 13, 2016

Don't use 4K Native drives for VMware vSphere ESXi nor VSAN

First of all, let's be absolutely clear. Disks with 4K sector size are not currently supported by VMware. See VMware KB- Support statement for 512e and 4K Native drives for VMware vSphere and VSAN (2091600)

UPDATE: vSphere 6.5 and VSAN 6.5 introduced 512e support so 4K native drives with 512 emulation (512e) are supported. In other words, 4K native drives without 512e are still not supported. 

IMPORTANT STATEMENTS FROM KB

Does current GA version of vSphere and VSAN support 4K Native drives?
No. 4K Native drives are not supported in current GA releases of vSphere and VSAN.

Does current GA version of vSphere and VSAN support 512e drives?
No. 512e drives are not supported with the current versions of vSphere and VSAN due to potential performance issues when using these drives. 

Therefore only 512n (native) drives are supported on any ESXi (5.x, 6.x) at the moment.

It is usually not big deal on shared storage systems (aka disk arrays) because logical volumes are virtually emulated and sector size is usually 512 by default or configurable (512 or 4K). 

I thought that it is the same with RAID controllers because virtual volumes are also emulated by RAID controller. However, I have just recently learned that it is not true. At least not for all RAID controllers. For example DELL PERC H730 (the best RAID controller DELL currently offers) doesn't allow you to choose sector size for virtual volume. Instead, sector size is passed from physical disks to operating system - ESXi hypervisor in our case.

Here is one real customer story with 4K native disks. 

The customer was not able to create datastore on some disks. The error message was ...
Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastoresystem" on ESXi "esxi-test" failed.  
The error message is depicted on screenshot below.



Server Model: PowerEdge R530 – System Revision I
Operating System: VMware ESXi 5.5.0 build-3343343
BIOS Version: 1.5.4
Lifecycle Controller Firmware: 2.21.21.21
RAID Controller: Perc H730Mini 

RAID1 (SYSTEM)
Physical Disk 0:1:0 Online 0 278.88 GB Not Capable SAS HDD No
Physical Disk 0:1:1 Online 1 278.88 GB Not Capable SAS HDD No
DATASTORE – successfully created during ESXi installation  – OK

RAID1 (DATA)
Physical Disk 0:1:2 Online 2 558.38 GB Not Capable SAS HDD No
Physical Disk 0:1:3 Online 3 558.38 GB Not Capable SAS HDD No
DATASTORE – Datastore cannot be created because 4K - FATAL ISSUE

RAID5 (DATA)
Physical Disk 0:1:4 Online 4 1862.50 GB Not Capable SAS HDD No
Physical Disk 0:1:5 Online 5 1862.50 GB Not Capable SAS HDD No
Physical Disk 0:1:6 Online 6 1862.50 GB Not Capable SAS HDD No
DATASTORE – datastore can be created but not officially supported by VMware because 512e – RISK

RAID Controller list of disks

T17: C0:PD   Flags    State Type Size          S N F P Vendor   Product          Rev  P C ID SAS Addr         Port Phy DevH WU BFw  BRev
T17: C0:------------------------------------------------------------------------------------------------------------------------------
T17: C0:0    f1400005 00020 00   22ecb25b      0 0 0 1 TOSHIBA  AL13SXB30EN      DK02 0 0 500003969802a076 03   04  0010   1  NA   NA - 512b (512 Native)
T17: C0:1    f1400005 00020 00   22ecb25b      0 0 0 1 TOSHIBA  AL13SXB30EN      DK02 0 0 500003969802a062 00   00  000a   1  NA   NA - 512b (512 Native)
T17: C0:2    f1400005 00020 00   8bba5f5       0 0 0 1 HGST     HUC156060CS4204  EK11 0 0 5000cca059596e59 05   06  000e   1  NA   NA - 4kn (4k Native)
T17: C0:3    f1400005 00020 00   8bba5f5       0 0 0 1 HGST     HUC156060CS4204  EK11 0 0 5000cca0595aa1cd 02   02  000c   1  NA   NA - 4kn (4k Native)
T17: C0:4    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3a8efd 04   05  000d   1  NA   NA - 512e (512 Emulation)
T17: C0:5    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3ae545 01   01  000b   1  NA   NA - 512e (512 Emulation)
T17: C0:6    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3b0661 06   07  000f   1  NA   NA - 512e (512 Emulation)
T17: C0:20   01400005 00020 0d   0             0 0 0 0 DP       BP13G+           2.23 0 0 524180704c645200 00   08  0009   0  NA   NA

T17: C0:100  00400005 00020 03   0             0 0 0 0 LSI      SMP/SGPIO/SEP    4402 0 0                0 00   ff  ffff   0  NA   NA

CONCLUSION AND  LESSONS LEARNED
It is obvious that VMware will support 4K disks sometimes in the future because industry is moving there but if you are planning to use directly attached disks choose disks with 512 sector. It is ESXi limitation at the moment. VMware VSAN is also impacted by this limitation because VSAN relies on ESXi.

Update 2016-01-21:
I have just received following question from one reader ...
"How can I list physical disks connected to internal PERC8?"
Unfortunately I don't have access to any 13G Dell server but I did a research and there should be three available methods.

Method 1/ racadm
If you have DRAC (Dell Remote Access/management Card) you can leverage racadm.
Based on racadm documentation it should be possible
  • Storage.PhysicalDisk.BlockSizeInBytes (Read Only) Description This is readonly attribute. This property indicates the logical block size of the physical drive that this virtual disk belongs to. Legal Values Values: 512 or 4096
Method 2/ Export PERC Raid Controller Log with Dell Support Live Image Version 2.0

http://de.community.dell.com/techcenter/support-services/w/wiki/369.export-perc-raid-controller-log-with-dell-support-live-image-version-2-0-englisch

Method 3/ perccli
Dell has utility called perccli. You can check perccli documentation for all details but there is command for viewing physical drive details for the specified slot in the controller.

  • Syntax is perccli /c0/e32/s4 show all

Downside of this method is that perccli binaries exist just for windows or linux so you cannot use it directly from ESXi and you have to boot for example Linux live CD.

Method 4/ megacli (not supported)

Third method is leveraging LSI megacli utility. Dell PERC is manufactured by LSI so it should work. LSI has megacli VIB for ESXi but it is not officially supported by VMware nor Dell.
See details at http://de.community.dell.com/techcenter/support-services/w/wiki/909.how-to-install-megacli-on-esxi-5-x




Tuesday, January 12, 2016

Datacenter Infrastructure Architectural Rules

It is always more complex but in general following rules applies to any datacenter infrastructure architecture transforming to cloud principles ...

Compute Rule
Compute performance is relatively cheap, but CPU context switching is pricey.
In other words, vCPU/pCPU ratio drives your consolidation.

Storage Rule
Storage capacity is relatively cheap, but I/O performance and response time is pricey.
In other words, storage performance drives your consolidation.

Network Rule
Network bandwidth is relatively cheap, but latency is pricey.
In other words, network latency drives your datacenter consolidation, geo clustering, DR architecture and hybrid cloud considerations.

Operational Rule
The infrastructure resources (compute/storage/network) are relatively cheap, but human resources are pricey.
In other words, automate as much as possible and keep it as simple as possible.