In certain cases where we need to restart the VMware vCenter Services, the best practices are to restart the services using the command line provided by VMware instead of doing it from the services console specifically on Windows vCenter Server. This is important as there are multiple Vmware Services which are dependant on other services and the command lines make sure they are managed in a specific order as recommended by Vmware

Below are the command lines to restart All VMware Services, Specific VMware Service or to check the status of Services on Windows hosted vCenter Server or an Appliance hosting vCenter Server

A) On Windows vCenter Server

1. Login\RDP to vCenter Server
2. Open Command Prompt as Admin
3. Go to the following location

C:\Program Files\VMware\vCenter Server\bin

Note: If your vCenter Suite is installed on any other drive other than C:\, please replace the Drive Letter

4. Run the below commands based on your requirement

service-control --start --all > To Start All Vmware Services

service-control --start servicename > To Start Specific Vmware Service

service-control --stop --all > To Stop All Vmware Services

service-control --stop <servicename> > To Stop Specific Vmware Service

service-control --status > To check Status of all VMware Services

B) On Appliance vCenter Server

1. SSH Putty to vCenter
2. Go to the Path /usr/bin/
3. Run the command

service-control --start --all > To Start All Vmware Services

service-control --start servicename > To Start Specific Vmware Service

service-control --stop --all > To Stop All Vmware Services

service-control --stop servicename > To Stop Specific Vmware Service

service-control -status > To check Status of all VMware Services

In this post, I am providing steps in brief to restart the Management Agents on ESXi Hosts using 2 methods : Either by connecting to the Host via a Secure SSH Putty Session OR by taking a remote session of the ESXi Host.

Please note that by restarting the Management Agents, there is no impact to running VMs on this Host, It restart the Management Services which help in communicating with VCenter this Host is connected to.

A) By SSH Putty Session
Run the following commands to restart the Management Agents

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

To check the status of these agents, run the following commands

/etc/init.d/hostd status

/etc/init.d/vpxa status

B)  From Remote Console

1. Connect to the console and Press F2 to Login
2. Enter Root Username and Password
3. Navigate to Troubleshooting Mode Options -> Restart Management Agent and Press Enter
4. Press F11 to confirm the Management Agent Status

Pre-requisites

• The user account upgrading Domain Functional Level should need to be a part of Domain Admins

• The user account upgrading Forest Functional Level should need to be a part of Enterprise Admins

• All the Domain Controllers should have the Operating System same or later than the Domain Functional Level you are trying to upgrade to. For E.g: If you want to upgrade your Domain Functional Level to 2012 R2, All Domain Controllers should be either Windows Server 2012 R2 or higher.

• Once the Forest and Domain Functional Level is upgraded, there is no way to revert it back to previous version unless you have options to run Forest Recovery. Please note that this is a very tedious steps to revert back the Domain and Functional Functional Level, so plan out this activity thoroughly. I will write about this in more details in future posts

• There is no dependancy on Workstation or Domain Member Server Operating Systems on Domain or Forest Functional Level. Only Domain Controllers OS matters

High Level Steps

• Upgrade all your Domain Controllers Operating Systems to support the Domain Functional Level

• Upgrade the Domain Functional Level

• Upgrade the Forest Functional Level

Notes

• There are various new features enabled in each of the Domain and Forest Functional Level upgrade. Please refer to http://technet.microsoft.com/en-us/library/understanding-active-directory-functional-levels(WS.10).aspx for more details on it

Challenges

• Even though you have made a list of all Active Domain Controllers and they meet the requirement of minimum OS Level, there could be few Domain Controllers which may have been demoted\decommissioned in the past and may not have proper metadata cleanup done which poses an obstacle while upgrading the Functional Level. You may receive below errors while trying to upgrade. Make sure all the stale Domain Controller entries are washed out before upgrading the Functional Level.

"You Cannot raise the domain functional level because this domain includes Active Directory Domain Controllers that are not running the appropriate version of Windows"

• Save the output file to note the Domain Controllers still below the required version of Windows Operating System. You may see a past demoted\decommissioned DC listed in the output and you may see this error listed against the Domain Controller

Unknown. The computer object may be damaged or missing

• Try clearing the Metadata of the Domain Controller to proceed with upgrading Domain & Forest Functional Level

Steps to upgrade Domain Functional Level

There are 2 method to upgrade DFL

Method 1: Using Active Directory Users and Computers

1. Logon to Domain Controller using an account with Domain Admin privileges
2. Go to Start -> Windows Administrative Tools -> Open Active Directory Users and Computers
3. Right Click on your domain name on the left
4. Click on Raise Domain Functional Level

Method 2: Using Active Directory Domains and Trusts

1. Logon to Domain Controller using an account with Domain Admin privileges
2. Go to Start -> Windows Administrative Tools -> Open Active Directory Domains and Trusts
3. Right Click on your domain name on the left
4. Click on Raise Domain Functional Level

Upgrade Forest Functional Level

1. Logon to Domain Controller using an account with Enterprise Admin privileges
2. Go to Start -> Windows Administrative Tools -> Open Active Directory Domains and Trusts
3. Right Click on "Active Directory Domains and Trusts" on the left
4. Click on Raise Forest Functional Level

If you have any questions or you need to add anything, please leave a comment.

Like my facebook page https://facebook.com/wintelspot

Key Management Service
This is a Microsoft Service hosted on a Windows Server. This service is used to activate Microsoft products like Operating Systems, MS Office Suites, Sharepoint suites etc.
This service can be hosted on your Datacenter Infrastructure without the need to contact Microsoft over Internet. Clients can connect to KMS using DNS dynamic or static entries and communicates over RPC port 135

KMS Activation Thresholds
There is a minimum threshold of clients requesting for activation only after which the KMS server starts activating the clients. For Windows Server Operating Systems, the threshold is 5 systems and for Windows Desktop Operating, the threshold is 25 systems. Until this threshold is met, the systems are not activated however the client machines contact KMS server every 2 hours to see if they have exceeded the threshold count.
There are no restrictions whether it is Physical or Virtual Server or Desktops. 
The KMS Server provides each host with Client Machine Identification [CMID] and saves it in a CMID Table. 

KMS Activation Renewal
KMS Activations are valid for 180 days. By default, KMS Clients attempt to renew their activation every 7 days. If activation fails, the client will retry every 2 hours. Once the computer is activated, the validity begins again.

KMS Service Publication
KMS Service uses DNS SRV records to store and communicate the location of KMS hosts. The KMS Service can also be published manually to the client computers. Steps to manually publish KMS host is given later in this article. 
If there are more than 1 KMS Server in the environment, the Clients randomly selects a KMS server unless the DnsDomainPublishList has specific Priority and Weight configured in the registry to prioritize a KMS server for client requests.

Important Port and Communication
KMS uses 1688 TCP ports [unidirectional - from client as source and KMS server as destination] to communicate with the client machines. The client send a packet of 250 bytes for activation requests to which KMS Host responds with the Activation Count. If the Count is equal or above threshold value, the client is activated and the session is closed.

Note: KMS Host does not require a dedicated Host.

Activating KMS Host
KMS Keys are provided by Microsoft to activate the KMS host. Once the Key is configured on KMS Host, It needs to be activated with Microsoft either via Telephone or Online. No further communication with Microsoft is required. The same KMS Key can be used in activating a maximum of 6 KMS Hosts in your environment.
Now the 6 KMS hosts can be re-activated with the same key for 9 more times. If you need more than 6 keys, Contact Microsoft Activation Call Center

As far as the KMS Server is operational with valid KMS keys and dynamic DNS update is completed to create a KMS SRV record in the current domain DNS, the client machines should be able to discover the KMS server and activate itself.

Important Commands to Publish KMS Hosts manually and activate Client Machines
slmgr -ckms -> To clear current KMS Host configured on client machine
slmgr -skms <kmshostname>:1688 -> To set KMS Host on client machine
slmgr -ipk <product key of OS> -> To set the Product key of OS. Refer to this for current Product Keys presented by Microsoft [https://docs.microsoft.com/en-us/windows-server/get-started/kmsclientkeys]
slmgr -ato -> Activate the client machine against the set KMS server
slmgr TargetComputer [username] [password] /parameter [options] -> To perform slmgr commands remotely on the Target Computer
nslookup -type=all _vlmcs._tcp -> This command should show you all the KMS Hosts in your environment having valid SRV records

Related Posts

• The other way to activate Microsoft Operating Systems are through MAK [Multiple Activation Keys] Activation. I will write a different post for MAK activations.- I will share the links shortly

• How to use KMS Service in a Domain to activate Workgroup clients and other domain clients - I will share the links shortly

Reference Documents
https://docs.microsoft.com/en-us/previous-versions/tn-archive/dd979804(v%3dtechnet.10)

There are multiple ways to find the serial number of the ESXi Host. Check the Serial Number physically inscribed on the server. It may or may not be present in the Configuration Tab of the ESXi Host when checked from vCenter Console.

It may not be always available in the configuration tab and in order to find the serial number of a Host, we can fetch the details like Make\Model and Serial No from SSH.

1. Open Putty and connect to the ESXi SSH console using root account credentials
2. Run the following command to get the serial no
                            esxcli hardware platform get
3. We get the required details as shown below

Windows Management Instrumentation [WMI] is a Core Infrastructural set developed by Microsoft for management of Windows devices and data. It helps in scripting and fetching Windows component related to the devices and settings.

WMI caters to the requirements through the WBEM repository where it stores the vital information and it may get corrupt denying proper data as required.

Below are few steps to diagnose if WMI repository is corrupt and steps to troubleshoot OR rebuild the WMI repository

To Verify if WMI repository is fine

winmgmt /verifyrepository

If the Results shows Inconsistent, we can run the following command to repair the repository

winmgmt /salvagerepository

Once the repair is completed, try running /verifyrepository to check if it has fixed the WMI corruption issue. If not, follow the below steps to rebuild the repository

1.       Disable and stop the winmgmt service

2.       Remove or Rename C:\Windows\System32\wbem\repository

3.       Enable and start the winmgmt [Windows Management Instrumentation] service

4.       Open a CMD prompt as Administrator

5.       In the CMD prompt Navigate to C:\Windows\System32\wbem\

6.       Run the following command: ‘for /f %s in ('dir /b *.mof') do mofcomp %s’

Note: This will take a minute or so to complete.

7.       Now run the command: ‘for /f %s in ('dir /b en-us\*.mfl') do mofcomp en-us\%s’

8.       Restart the Data Collector service

These steps should fix the issue with WMI repository

I am writing after a long break and starting with the different components of Cisco UCS. This is going to be a very short post where I am only mentioning the various components of Cisco UCS Hardware. I will post more knowledge on the components and also how to create service profiles, Create Pools for MAC, UUID and WWNs using Cisco UCS Manager Console in the future posts.

Fabric Interconnects	Gen1	6120XP - with 20 base ports, can be extended by 1 expansion module 6140XP - with 40 base ports, can be extended by 2 expansion module
	Gen2	6248UP - with 48 base ports, can be extended by 1 expansion module 6296UP - with 96 base ports, can be extended by 3 expansion module
	Gen3	6324 - with 5 ports 6332 - 16UP - with 40 base ports 6332 - with 32 base ports
Fabric Extender	Gen1	2104XP - with 4 ports
	Gen2	2204XP - with 4 ports 2208XP - with 8 ports
	Gen3	2304XP - with 4 ports
Chassis	Gens	5108 - with 8 slots
Blades	Gens	B and C Series Blades

In this post I am mostly listing down the generations of Components and their model numbers which would help you in knowing the product and its features well. 

Imagine we come across a very critical Virtual Machine hosted on VMware platform which is hung at a particular stage. We see that either the machine has freezed at a screen or has been hung at Blue screen of death. What options do we have rather than hard rebooting the machine to bring back the primary functionality of the Virtual Machine online but we often being asked the question as to why the machine got to where it was and how can we avoid it from happening again.

Yes, we all know if we have configured Crash Dump or Minidump settings on the guest OS, we would definitely be able to analyze the dump to understand the state of the Virtual Machine at that stage. However if we do not have Crash Dump Collection Enabled on any machine or if you feel the Pagefile is not configured enough to capture a crash dump or if you realise that the space where the Dump has to be created is not sufficient then we would not get the desired Crash dump for analysis. Well in that case, before rebooting the server, we can take a Virtual Machine snapshot.

Yes, This snapshot can be converted into a Memory Dump which can be then analyzed using various debugger tools like Windebug etc.

1. Download the vmss2core.exe tool
2. Copy it to one of your Windows Server having sufficient free space 
3. Copy the Snapshot File [.vmss] from the datastore, where the VM is located, to the same location where vmss2core.exe is residing
4. Run the utility to convert the snapshot to dump as shown below

vmss2core -W VM_Snapshot_Filename.vmss

5. This will convert the Snapshot File to Memory Dump that we can use to analyse the cause of Server Hung 

The best thing about Vmware platform is the centralized management of all the resources using vCenter Server. Using vSphere or Web Client we all connect to vCenter Server to administer the virtual datacenter. Imagine if there was no vCenter Server and we had to connect to each ESXi host manually and manage the VMs. Yes, it sounds like a tedious task.

vCenter Server also provides multiple features like DRS, sRDS, vMotion, HA, FT etc. Today we are talking about a scenario where the vCenter Server or Appliance goes down and what are its impact on each of these functionalitiess. Let's check the impact of each of these functionalities below

Management :

Managing the environment won't have much impact as we can still connect to each ESXi host via SSH or vSphere client and manage the servers. It is not easy but there is no impact to the environment

Virtual Machines & ESXi Hosts:

There is no dependency of vCenter server on functionality or uptime of any other Virtual Server or ESXi Host. The Hosts can still be connected via SSH or vSphere Client and all the Virtual machines are still working

Distributed Resource Scheduling:

DRS works with vCenter Server to balance the resources and Virtual Machines across ESXi Hosts using DRS Clusters, the DRS functionality will fail if the vCenter server is down

vMotion\svMotion:

vMotion and svMotion features are spanned across hosts and since it is a feature is based on DRS, Both vMotion and svMotion will fail if vCenter server is down

High Availability:

HA will have medium impact as the Hosts\Clusters configured with HA enabled will have the HA running even if vCenter Server is down. However we would not be able to change any settings like Admission Control Policies while the vCenter Server is down.

Fault Tolerance:

FT will also work in case of all Virtual Machines which are configured before vCenter Server went down. No Changes can be done once the vCenter server is down.

Distributed Switch:

Distributed Switch would still continue to work even after vCenter Service is down. It will still connect to the Network it is configured on.

VM Snapshots:

There would be no issues in taking the snapshots of a Virtual Machine. We need to connect to ESXi host and take the VM snapshot.

Virtual Update Manager:

Since Virtual Update Manager is a vCenter plugin, the functionality of VUM will fail while the vCenter is down.

Note:

We have tested these features and impacts mentioned above only on vCenter Server 5.x version only.