Sunday, June 25, 2017

Start order of software services in VMware vCenter Server Appliance 6.0 U2

vCenter Server Appliance 6.0 U2 services are started in the following order ...

  1. vmafdd (VMware Authentication Framework)
  2. vmware-rhttpproxy (VMware HTTP Reverse Proxy)
  3. vmdird (VMware Directory Service)
  4. vmcad (VMware Certificate Service)
  5. vmware-sts-idmd (VMware Identity Management Service)
  6. vmware-stsd (VMware Security Token Service)
  7. vmware-cm (VMware Component Manager)
  8. vmware-cis-license (VMware License Service)
  9. vmware-psc-client (VMware Platform Services Controller Client)
  10. vmware-sca (VMware Service Control Agent)
  11. applmgmt (VMware Appliance Management Service)
  12. vmware-netdumper (VMware vSphere ESXi Dump Collector)
  13. vmware-syslog (VMware Common Logging Service)
  14. vmware-syslog-health (VMware Syslog Health Service)
  15. vmware-vapi-endpoint (VMware vAPI Endpoint)
  16. vmware-vpostgres (VMware Postgres)
  17. vmware-invsvc (VMware Inventory Service)
  18. vmware-mbcs (VMware Message Bus Configuration Service)
  19. vmware-vpxd (VMware vCenter Server)
  20. vmware-eam (VMware ESX Agent Manager)
  21. vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
  22. vmware-sps (VMware vSphere Profile-Driven Storage Service)
  23. vmware-vdcs (VMware Content Library Service)
  24. vmware-vpx-workflow (VMware vCenter Workflow Manager)
  25. vmware-vsan-health (VMware VSAN Health Service)
  26. vmware-vsm (VMware vService Manager)
  27. vsphere-client ()
  28. vmware-perfcharts (VMware Performance Charts)
  29. vmware-vws (VMware System and Hardware Health Manager) 

Thursday, June 22, 2017

CLI for VMware Virtual Distributed Switch

A few weeks ago I have been asked by one of my customers if VMware Virtual Distributed Switch (aka VDS) supports Cisco like command line interface. The key idea behind was to integrate vSphere switch with open-source tool Network Tracking Database (NetDB) which they use for tracking MAC addresses within their network. I have been told by customer that NetDB can telnet/ssh to Cisco switches and do screen scraping so would not it be cool to have the most popular switch CLI commands for VDS? These commands are

  • show mac-address-table
  • show interface status
The official answer is NO, but wait a minute. Almost anything is possible with VMware API. So my solution is leveraging VMware's vSphere Perl SDK to pull information out of Distributed Virtual Switches. I have prepared PERL script which currently supports two commands mentioned above. It goes through all VMware Distributed Switches on single vCenter. Script along with shell wrappers are available on GITHUB here

See screenshots below to get an idea what script does.

The output of the command --username readonly --password readonly --cmd show-port-status
looks as depicted in screenshot below.

and output of the command --username readonly --password readonly --cmd show-mac-address-table
Now, I'm working on telnet daemon which will simulate remotely accessible switch CLI. It will just call PERL scripts above but that is what network administrators want for their day to day operations.

Tuesday, June 13, 2017

Storage DRS integration with storage profiles

This is a very quick blog post. In vSphere 6.0, VMware has introduced Storage DRS integration with storage profiles (aka SPBM - Storage Policy Based Management).

Here is the link to official documentation.

Generally, it is about SDRS advanced option EnforceStorageProfiles. Advanced option EnforceStorageProfiles takes one of these integer values, 0,1 or 2 where the default value is 0.

  • When option is set to 0, it indicates that there is NO storage profile or policy enforcement on the SDRS cluster.
  • When option is set to 1, it indicates that there is storage profile or policy SOFT enforcement on the SDRS cluster. It is analogous with DRS soft rules. SDRS will comply with storage profile/policy in the optimum level. However if required, SDRS will violate the storage profile compliant.
  • When option is set to 2, it indicates that there is storage profile or policy HARD enforcement on the SDRS cluster. It is analogous with DRS hard rules. In any case, SDRS will not violate the storage profile or policy compliant.

Please note that at the time of writing this post, SDRS Storage Profiles Enforcement works only during initial placement and NOT for already provisioned VMs during load balancing. Therefore, when iVM Storage Policy is changed for particular VM, SDRS will not make it automatically compliant nor throw any recommendation.

Another limitation is that vCloud Director (vCD) backed by SDRS cluster does NOT support  Soft (1) or Hard (2) storage profile enforcements. vCloud Director (vCD) will work well with Default (0) option

Relevant references to other resources:

Wednesday, June 07, 2017

VMware Photon OS with PowerCLI

Photon OS is linux distribution maintained by VMware with multiple benefits for virtualized form factor, therefore any virtual appliance should be based on Photon OS.

I have recently tried to play with Photon OS and here are some my notes.

IP Settings

Network configuration files are in directory
IP settings are leased from DHCP by default. It is configured in file  /etc/systemd/network/

File contains following config
To use static IP settings it is good to move DHCP config file down in alphabetical order and create config file with static IP settings.
mv 99-dhcp-en.networkcp
file  /etc/systemd/network/ should looks similar to

Network can be restarted by command
systemctl restart systemd-networkd
and network settings can be checked by command


Package management

Photon OS uses TDNF  (Tiny DNF) package manager. It is based on Fedora's DNF.  This is a development by VMware that comes with compatible repository and package management capabilities. Note that not every dnf command is available but the basic ones are there.

  • tdnf install libxml2
  • tdnf install openssl-devel
  • tdnf install binutils
  • tdnf install pkg-config
  • tdnf perl-Crypt-SSLeay
  • tdnf install cpan
  • tdnf libuuid-devel
  • tdnf install make
Update of the whole operating system can be done by command
tdnf update

Log Management

You will not find typical linux /var/log/messages
Instead, journald is used and you have to use command journalctl

Equivalent to tail -f /var/log/messages is
journalctl -f 

System services

System services are control by command systemctl

To check service status use
systemctl status docker
To start service use
systemctl start docker
To enable service after system start use
systemctl enable docker

Docker and containerized PowerCLI

One of key use cases for Photon OS is to be a docker host, therefore, docker is preinstalled in Photon OS. You can see further Docker information by command
docker info
If Docker is running on your system, you can very quickly spin up docker container. Let's use example of containerized PowerCLI. To download container image from DockerHup use command
docker pull vmware/powerclicore
to check all downloaded images use the command
docker images -a   
 root@photon-machine [ ~ ]# docker images -a    
 REPOSITORY      TAG         IMAGE ID      CREATED       SIZE  
 vmware/powerclicore  latest       a8e3349371c5    6 weeks ago     610 MB  
 root@photon-machine [ ~ ]#   

Now you can run powercli container interactively (-i) and in allocated pseudo-TTY (-t). Option -rm stands for "Automatically remove the container when it exits".
docker run --rm -it vmware/powerclicore 
 root@photon-machine [ ~ ]# docker run --rm -it --name powercli vmware/powerclicore         
 Copyright (C) Microsoft Corporation. All rights reserved.erclicore --name powercl  
      Welcome to VMware vSphere PowerCLI!  
 Log in to a vCenter Server or ESX host:       Connect-VIServer  
 To find out what commands are available, type:    Get-VICommand  
 Once you've connected, display all virtual machines: Get-VM  
     Copyright (C) VMware, Inc. All rights reserved.  
 Loading personal and system profiles took 3083ms.  
 PS /powershell#   

Now you can use PowerCLI running on linux container. The very first PowerCLI command is usually Connect-VIServer, but you can get following warning and error messages

 PS /powershell> Connect-VIServer                                                                         
 cmdlet Connect-VIServer at command pipeline position 1  
 Supply values for the following parameters:  
 Specify Credential  
 Please specify server credential  
 User: cdave  
 Password for user cdave: *********  
 WARNING: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you'd like to connect once or to add  
  a permanent exception for this server.  
 Connect-VIServer : 06/07/2017 19:25:44     Connect-VIServer          An error occurred while sending the request.       
 At line:1 char:1  
 + Connect-VIServer  
 + ~~~~~~~~~~~~~~~~  
   + CategoryInfo     : NotSpecified: (:) [Connect-VIServer], ViError  
   + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_Exception,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer  
 PS /powershell>   

To solve the problem you have to adjust PowerCLI configuration by
Set-PowerCLIConfiguration -InvalidCertificateAction ignore -confirm:$false -scope All
The command above changes PowerCLI configuration for all users.

To use other docker commands you can open another ssh session, and for example list running containers

 root@photon-machine [ ~ ]# docker ps -a     
 CONTAINER ID    IMAGE         COMMAND       CREATED       STATUS       PORTS        NAMES  
 6ecccf77891e    vmware/powerclicore  "powershell"    7 minutes ago    Up 7 minutes              powercli  
 root@photon-machine [ ~ ]#   

... or issue any other docker command.

That's cool, isn't it?

Tuesday, June 06, 2017

VMware VVOLs scalability

I'm personally a big fan of VMware Virtual Volumes concept. If you are not familiar with VVOLs check this blog post with the recording of VMworld session and read VMware KB Understanding Virtual Volumes (VVols) in VMware vSphere 6.0

We all know that the devil is always in details. The same is true with VVOLs. VMware prepared the conceptual framework but implementation always depends on storage vendors thus it vary around storage products.

Recently, I have had VVOLs discussion with one of my customers and he was claiming that their particular storage vendor supports a very small number of VVOLs. That discussion inspired me to do some research.

Please, note that numbers bellow are valid at the moment of writing this article. You should always check current status with your particular storage vendor.

Vendor / Storage ArrayMaximum VVOLs / Snapshots or Clones
DELL / Compellent SC 80002,000 / TBD
EMC / Unity 3009,000 / TBD
EMC / Unity 4009,000 / TBD
EMC / Unity 50013,500 / TBD
EMC / Unity 60030,000 / TBD
EMC / VMAX 364,000 / TBD
Hitachi / VSP G2002,000 / 100,000
Hitachi / VSP G4004,000 / 100,000
Hitachi / VSP G6004,000 / 100,000
Hitachi / VSP G80016,000 / 100,000
Hitachi / VSP G100064,000 / 1,000,000

Numbers above are very important because single VM have minimally 3 VVOLs (home, data, swap) and usually even more (snapshot) or more data disks. If you will assume 10 VVOls for single VM you will end up with just 200 VMs on Dell Compellent or Hitachi VSP G200. On the other hand, EMC Unity 600 would give you up to 3,000 VMs which is not bad and enterprise storage systems (EMC VMAX and Hitachi G1000) would give you up to 6,400 VMs which is IMHO very good scalability.

So as always, it really depends on what storage system do you have or planning to buy.

If you know numbers for other storage systems, please share it in comments below this blog post.

Wednesday, May 31, 2017

vROps & vSphere Tags, Custom Attributes

As many of my customers started to recently customize their vROps and together we are working on various use-cases I find it useful to summarize my notes here and possibly help others during their investigation and customization.

This time I will focus on custom descriptions for the objects in vROps. When you are providing an access to vRealize Operations to your company management, many times they are not familiar with IT naming convention and it is very hard for them to analyze why some object is marked as red and if it is important at all.

We've been thinking this through with David for a bit and there are two very easy alternatives to tackle this use case. vSphere Tags and Custom Attributes in vSphere. In the following lines I will explain step-by-step procedure to use these and tackle possible problems you might hit on the way.

1) Create preferred description in vSphere. For Custom Attributes can be used local (object based) or global definitions - both works fine. At the end of this article you can see how the vSphere Tags and Custom Attributes looks like and what is better to cover your specific use-case.

2) Afterwards switch to vROps and check, if the metric is being propagated to the object. Bear in mind that it might take couple of minutes for metric to be collected.

3) After the metric being available you can start working with it for example in your Views. For this post I've created couple of Tags on my vCenter appliance called APPL_vCenter; therefore selecting Virtual machine as a subject of view creation is logical choice.

4) Now the tricky part I had personally a problem (I would like to thank our great vROps consultant Oleg Ulyanov for helping me out) was that the metric was simply not available in a view. The thing here is that if you have big environment with hundreds of VMs, vROps will randomly chose few (I think the number was 5) and based on those 5 show a merge of available metrics. If you would be lucky as me and APPL_vCenter would not be among them, Tags will not be available. To force vROps to use specific machine, you can use the square next to the Metrics/Properties button.

In newly opened Window you can filter out a VM you want.

5) Afterwards just chose the VM you've created Tag on (in my case again APPL_vCenter) and metric should be now visible.

6) In the final screenshot I would like to compare both solutions - vSphere Tags and Custom Attributes (for some reason in vROps marked as Custom Tag).

vSphere Tags are consoliadted into one Field. I've created Tag "Purpose" and Tag "OS" for the vCenter Appliance. On the other hand Custom Attributes are always separated so doing the same would create two Custom Tags with just a value in it. In case you would need for example filtering or any other logic behind the Tags, Custom Attributes seems to be a better choice.

Sunday, May 14, 2017

VM Snapshots Deep-Dive

A while ago I received interesting question regarding snapshot consolidation from one of my customers and as I was not 100% sure about the particular details (file naming, consolidation, pointers, etc.) I went to do some testing in a lab. The scenario was pretty simple; create a virtual machine with non-linear snapshot tree and start removing the snapshots.

Lessons learned: When doing such tests, it is always good to add some files or something a bit more sizable into the each snapshot. My initial work started with just creating the folders named snap[1-7] which during consolidation was really not helpful identifying where the data from snapshot actually went.

The non-linear snapshot tree I mentioned earlier looks like this:

First confusion which was sort of most important and took me a while to turn my brain around was the file naming convention. More or less file SnapTest-flat.vmdk is a main data file of the Server, in this case C: drive of the Microsoft Windows server with size around 26GB. This file is not visible in Web Client as only the descriptor <VM name>.vmdk (in our case SnapTest.vmdk) is directly visible. When you will create a first snapshot this is a file which is being used by it as you can see in the following image:

Command grep -E 'displayName|fileName' SnapTest.vmsd is listing all lines containing displayName and/or fileName from the file SnapTest.vmsd. Going through the vSphere documentation you will find:
A .vmsd file that contains the virtual machine's snapshot information and is the primary source of information for the Snapshot Manager. This file contains line entries, which define the relationships between snapshots and between child disks for each snapshot.

With that being said above output of the command is listing our predefined snapshot names (I used the number of the snapshot and the size of the file I've added) and its respected file. So first created snapshot is named Snap1+342MB and using file SnapTest.vmdk.

Using the 2nd useful command during this test grep parentFileNameHint SnapTest-00000[0-9].vmdk is going through all the snapshot files and listing parentFileNameHint. As you probably guessed it, it is a snapshot it is depending on (parent file).

List of tests I performed:
1) Remove Snapshot 5 (Snap5+366MB)
2) Remove Snapshot 4 (Snap4+356MB)
3) Remove Snapshot 3 (Snap3+337MB)
4) Remove Snapshot 2 (Snap2+348MB)
5) Move Here You Are
6) Remove Snapshot 6 (Snap6+168MB)
7) Remove Snapshot 7 (Snap7+348MB)

Now In more details per every case.

1) Remove Snapshot 5 (Snap5+366MB)
Result can be seen in this visualisation. After removing the Snapshot 5 within the Web Client, Snapshot 6 and Snapshot 5 vmdk files were consolidated, size updated accordingly same as the snapshot's vmdk file.

As for the fist example I will add also the command exports here for illustration. Following scenarios should be understandable even without such.

2) Remove Snapshot 4 (Snap4+356MB)
I did this test just to proof myself the proper functionality, so it is very similar to the previous part.

3) Remove Snapshot 3 (Snap3+337MB)
Now with removing Snapshot 3, things are becoming a bit more challenging. On snapshot 3 are currently depending 3 more snapshots (Snap6, Snap7 and You Are Here). As the consolidation in this case would need to be performed with each of them it would be very "costly" operation. The result was that the Snapshot was removed from GUI but the files remained on the disk and all the dependencies were preserved.

4) Remove Snapshot 2 (Snap2+348MB)
Although it might seem complicated on the "paper" the remove process for Snapshot 2 was very similar with every other snapshot removal only in this case Snapshot 2 was consolidated with temporary file preserved from the previous step.

5) Move "Here You Are"
Moving active state over virtual machine named as "Here You Are" is also quite simple operation. I was performing this test more or less to validate, how many snapshots can be dependent on the parent snapshot until the snapshots are consolidated. To spoil the surprise it has to be just one file as in this case on the temporary file are depending only Snapshot 6 and Snapshot 7.

6) Remove Snapshot 6 (Snap6+168MB)
As mentioned in the previous step if there is only one child snapshot to the parent snapshot and the parent snapshot is being removed, data are being consolidated. Otherwise there would be preserved temporary file for child snapshots to work with.

7) Remove Snapshot 7 (Snap7+348MB)The final step was to remove the last Snapshot 7 and be left with just one snapshot Snap1+342MB and the main file. If this file would be removed all the data would be consolidated into the main VMDK and there would be no delta file for "You Are Here" state and therefore no point to get back to.

Overall the work with the snapshots is not a rocket science but my test today showed me a in a bit more detail what is happening in the background with the file names, snapshots IDs in the vmdk files, data consolidation. It also showed that there are temporary parent files left behind if there is more than one direct child snapshot depending on it. It also forced me to refresh the knowledge about the Space Efficient Sparse Virtual Disks (or SE Sparse Disks for short) which was well explained by my colleague Cormac Hogan in late 2012.