Vovando Dev

In the nearest future I am planing to update vSAN to 6.0 version on Production. The procedure is not trivial, so I want to try update on test lab. We need to build an entire embedded vsan environment.

Some facts about vSAN 5.5.

Virtual SAN 5.5 requirements:

– a minimum of 3 hosts contributing local disks, at least one SSD and one HDD per host (not all hosts need to contribute)

– ESXi 5.5 or later on each host

– hosts must be managed by vCenter Server 5.5

– 6GB of RAM on each host (10GB for 7 or more disks per host)

– compatbile storage devices listed on the HCL

– SSDs must be in use by Flash Read Cache or formatted with any filesystem

– a 1:10 ratio of SSD capacity to HDD in each host

– an SAS or SATA HBA, or RAID controller that is set up in RAID0 mode or passthrough

– ESXi Dump Collector and Syslog Collector must be configured to redirect memory dumps and logs onto a network server

– all hosts in a VSAN cluster must be part of a VSAN network and must be on the same subnet

– a private 1GbE IPv4 network, preferably 10GbE with at least one physical NIC dedicated to VSAN on each host

– VMkernel port groups must have the VSAN port property activated

DRS and HA supported.

Fault Tolerance, storage DRS, deduplication and Metro Clustering are NOT supported.

Virtual SAN 5.5 maximums:

– 32 hosts in a cluster

– 1 VSAN datastore per cluster

– 5 disk groups per host

– 35 HDDs per host, 7 HDDs per disk group

– 1 SSD per disk group

– 3000 components per host

– 3200 VMs per cluster, 100 VMs per host

– 2TB VMDK size

– 2 dedicated physical networks

– 3 host failures the cluster can tolerate

NOTE: on a cluster with HA, the maximum number of VMs per host is 64 and per cluster 2048.

 

So let’s start to build my test environment.

1. First, I took one of my physical esxi servers. This server  simply had 12 cpus, a few network cards (only need one here) and 96GB of memory.

Typical install, then I used the dcui and changed it's ip to 192.168.95.120/24 and called it esxi.vovando.com.

I used:

Product: VMware ESXi 5.5

Release date: February 5, 2015

Patch: ESXi550-201502001

Build: 2456374

 

2. Then, I connected to my esxi server using the vsphere client from my Windows 8.1. I proceeded to create two new vswitches, called vSwitch2 and vSwitch3. These are internal vswitches, attached to no uplinks. These vswitches eventually were used for the vmotion network and vsan network for the embedded esxi servers.

 

 

It is CRITICAL to remember to change Promiscous Mode to ACCEPT to allow for internal communication. These switches had VM networks: vMotion-test and vSan-test.

 

 

3. Now, it’s time to create the three embedded esxi servers.  Using the vsphere client, I created three identical esxi servers (esxi1 – 192.168.95.131), (esxi2 – 192.168.95.132) and (esxi3 – 192.168.95.133). I gave each embedded esxi server 2 vcpus, 5 gbs of ram and 3 disks. The 3 disks were 4gbs for the boot disk, 10gbs for the ssd disk   and 100gbs for the HDD to be used by vsan.

 

4. After the installation of each of the embedded esxi servers, I changed the disk type of each of the future ssd disks using the esxcli command. Here are the commands:

 

$ esxcli storage nmp satp rule add –satp=VMW_SATP_LOCAL –device mpx.vmhba1:C0:T1:L0 –option "enable_local enable_ssd"

$ esxcli storage core claiming unclaim –type=device –device mpx.vmhba1:C0:T1:L0

You need to reboot host after it.

 

5. Now, it was time to create the vcenter server. I used the windows-based vcenter server. Ip address that was 192.168.95.130.

 

7. Connected to the vcenter server with the vsphere client, logged in and I created a datacenter and added the three esxi servers.

8. Using the vsphere client, it was time to create the vmotion network and the vsan network, which can't be created with the vsphere client only.

vmotion

esxi01 – 10.1.0.131

esxi02 – 10.1.0.132

esxi03 – 10.1.0.133

vsan

esxi01 – 10.2.0.131

esxi02 – 10.2.0.132

esxi03 – 10.2.0.133

9.  Using the web client (port 9443), create a cluster for vsan and create the disk groups necessary. If you select manual configuration, you need to create three disk groups (one per host) using the fake ssd disk and the fake HDD disk. By the time you are done, you should have a datastore that is roughly 300gb. While creating the cluster, I also enabled DRS (fully automatic) and HA (which uses the vsan network for heartbeats).

After that this Cluster can be used for work.

 

10. vSan related commands.

#esxcli vsan

Usage: esxcli vsan {cmd} [cmd options]

Available Namespaces:

  datastore             Commands for VSAN datastore configuration

  network               Commands for VSAN host network configuration

  storage               Commands for VSAN physical storage configuration

  cluster               Commands for VSAN host cluster configuration

  faultdomain           Commands for VSAN fault domain configuration

  maintenancemode       Commands for VSAN maintenance mode operation

  policy                Commands for VSAN storage policy configuration

  trace                 Commands for VSAN trace configuration

 

#esxcli vsan datastore name get

   Name: vsanDatastore

#esxcli vsan network ipv4 add -i vmk2

This VMKernel network interface is already in use. Can't add again. If you want to reconfigure, use 'set' command.

#esxcli vsan network list

Interface

   VmkNic Name: vmk2

   IP Protocol: IPv4

   Interface UUID: 26e84855-f8e9-f3ad-5ff8-005056873a81

   Agent Group Multicast Address: 224.2.3.4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group Multicast Port: 12345

   Multicast TTL: 5

 

# esxcli vsan network list

Interface

   VmkNic Name: vmk2

   IP Protocol: IPv4

   Interface UUID: 26e84855-f8e9-f3ad-5ff8-005056873a81

   Agent Group Multicast Address: 224.2.3.4

   Agent Group Multicast Port: 23451

   Master Group Multicast Address: 224.1.2.3

   Master Group Multicast Port: 12345

   Multicast TTL: 5

# esxcli vsan cluster get

Cluster Information

   Enabled: true

   Current Local Time: 2015-05-07T16:14:52Z

   Local Node UUID: 5548e218-ae75-c834-0331-005056873a81

   Local Node State: BACKUP

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 5548e429-d572-3814-2780-005056872eec

   Sub-Cluster Backup UUID: 5548e218-ae75-c834-0331-005056873a81

   Sub-Cluster UUID: 523a032b-4d44-8fbd-2954-74154216d6b2

   Sub-Cluster Membership Entry Revision: 11

   Sub-Cluster Member UUIDs: 5548e429-d572-3814-2780-005056872eec, 5548e218-ae75-c834-0331-005056873a81, 5548de32-1dc9-573c-6be5-005056877536

   Sub-Cluster Membership UUID: f68c4b55-ec10-02a5-120c-005056872eec

# esxcli vsan storage list

mpx.vmhba1:C0:T2:L0

   Device: mpx.vmhba1:C0:T2:L0

   Display Name: mpx.vmhba1:C0:T2:L0

   Is SSD: false

   VSAN UUID: 52711c36-d9a2-c17b-7b4f-fe6ca685d0e8

   VSAN Disk Group UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa

   VSAN Disk Group Name: mpx.vmhba1:C0:T1:L0

   Used by this host: true

   In CMMDS: true

   Checksum: 5186061164308225788

   Checksum OK: true

   Emulated DIX/DIF Enabled: false

mpx.vmhba1:C0:T1:L0

   Device: mpx.vmhba1:C0:T1:L0

   Display Name: mpx.vmhba1:C0:T1:L0

   Is SSD: true

   VSAN UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa

   VSAN Disk Group UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa

   VSAN Disk Group Name: mpx.vmhba1:C0:T1:L0

   Used by this host: true

   In CMMDS: true

   Checksum: 10790749511316735198

   Checksum OK: true

   Emulated DIX/DIF Enabled: false

#esxcli vsan policy getdefault

Policy Class  Policy Value

————  ——————————————————–

cluster       (("hostFailuresToTolerate" i1))

vdisk         (("hostFailuresToTolerate" i1))

vmnamespace   (("hostFailuresToTolerate" i1))

vmswap        (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))

vmem          (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))

#grep -i vsan /etc/vmware/esx.conf

/net/vmkernelnic/child[0002]/portgroup = "vSAN"

/net/vswitch/child[0002]/portgroup/child[0000]/name = "vSAN"

/firewall/services/vsanvp/enabled = "true"

/firewall/services/vsanvp/allowedall = "true"

/vsan/faultDomainVersion = "2"

/vsan/hostDecommissionVersion = "0"

/vsan/faultDomainName = ""

/vsan/subClusterUuid = "523a032b-4d44-8fbd-2954-74154216d6b2"

/vsan/autoClaimStorage = "true"

/vsan/enabled = "true"

/vsan/hostDecommissionMode = "decom-mode-none"

/vsan/hostDecommissionState = "decom-state-none"

/vsan/datastoreName = "vsanDatastore"

/vsan/network/child[0000]/ttl = "5"

/vsan/network/child[0000]/vmknic = "vmk2"

/vsan/network/child[0000]/masterPort = "12345"

/vsan/network/child[0000]/masterGroup = "224.1.2.3"

/vsan/network/child[0000]/ifaceUuid = "26e84855-f8e9-f3ad-5ff8-005056873a81"

/vsan/network/child[0000]/agentGroup = "224.2.3.4"

/vsan/network/child[0000]/agentPort = "23451"

/adv/Misc/HostName = "esxi02-vsan6"

#esxcfg-advcfg -l | grep -i vsan

/VSAN/ClomRepairDelay [Integer] : Minutes to wait for absent components to come back before starting repair (REQUIRES clomd RESTART!)

/VSAN/ClomMaxComponentSizeGB [Integer] : Maximum component size used for new placements (REQUIRES clomd RESTART!))

/VSAN/DomLongOpTraceMS [Integer] : Trace ops that take more than the specified value in milliseconds

/VSAN/DomLongOpUrgentTraceMS [Integer] : Urgent trace ops that take more than the specified value in milliseconds

/VSAN/DomBriefIoTraces [Integer] : Enables a brief set of per-IO DOM traces for debugging

/VSAN/DomFullIoTraces [Integer] : Enables full set of per-IO DOM traces; if disabled, IO op traces go to the per-op trace table

/VSAN/TraceEnableDom [Integer] : DOM tracing enabled

/VSAN/TraceEnableDomIo [Integer] : DOMIO tracing enabled

/VSAN/TraceEnableLsom [Integer] : LSOM tracing enabled

/VSAN/TraceEnableCmmds [Integer] : CMMDS/CMMDSResolver tracing enabled

/VSAN/TraceEnableRdt [Integer] : RDT tracing enabled

/VSAN/TraceEnablePlog [Integer] : PLOG tracing enabled

/VSAN/TraceEnableSsdLog [Integer] : SSDLOG tracing enabled

/VSAN/TraceEnableVirsto [Integer] : Virsto tracing enabled

/VSAN/VsanSparseEnabled [Integer] : Enable auto-creation of vsanSparse instead of vmfsSparse redologs, for VSAN 2.0 datastore only

/VSAN/VsanSparseCacheThreshold [Integer] : Maximum number of entries in single VsanSparse cache

/VSAN/VsanSparseCacheOverEvict [Integer] : Percentage of VsanSparseCacheThreshold to add to eviction

/VSAN/VsanSparseSpeculativePrefetch [Integer] : Number of bytes to add onto each extent interrogation request

/VSAN/VsanSparseMaxExtentsPrefetch [Integer] : Maximum number of extents to fetch during interrogation

/VSAN/VsanSparseParallelLookup [Integer] : Request written extent data from each layer in parallel

/VSAN/TraceEnableVsanSparse [Integer] : VsanSparse tracing enabled

/VSAN/TraceEnableVsanSparseIO [Integer] : VsanSparse per-IO tracing enabled

/VSAN/TraceEnableVsanSparseVerbose [Integer] : VsanSparse very verbose tracing enabled

/VSAN/VsanSparseHeapSize [Integer] : Maximum heap size for VsanSparse snapshot consolidation buffers(in KiB)

 

Next time I will tell you how to upgrade your vSan Cluster to vSAN 6.0.

Stay tuned!