In the nearest future I am planing to update vSAN to 6.0 version on Production. The procedure is not trivial, so I want to try update on test lab. We need to build an entire embedded vsan environment.
Some facts about vSAN 5.5.
Virtual SAN 5.5 requirements:
– a minimum of 3 hosts contributing local disks, at least one SSD and one HDD per host (not all hosts need to contribute)
– ESXi 5.5 or later on each host
– hosts must be managed by vCenter Server 5.5
– 6GB of RAM on each host (10GB for 7 or more disks per host)
– compatbile storage devices listed on the HCL
– SSDs must be in use by Flash Read Cache or formatted with any filesystem
– a 1:10 ratio of SSD capacity to HDD in each host
– an SAS or SATA HBA, or RAID controller that is set up in RAID0 mode or passthrough
– ESXi Dump Collector and Syslog Collector must be configured to redirect memory dumps and logs onto a network server
– all hosts in a VSAN cluster must be part of a VSAN network and must be on the same subnet
– a private 1GbE IPv4 network, preferably 10GbE with at least one physical NIC dedicated to VSAN on each host
– VMkernel port groups must have the VSAN port property activated
DRS and HA supported.
Fault Tolerance, storage DRS, deduplication and Metro Clustering are NOT supported.
Virtual SAN 5.5 maximums:
– 32 hosts in a cluster
– 1 VSAN datastore per cluster
– 5 disk groups per host
– 35 HDDs per host, 7 HDDs per disk group
– 1 SSD per disk group
– 3000 components per host
– 3200 VMs per cluster, 100 VMs per host
– 2TB VMDK size
– 2 dedicated physical networks
– 3 host failures the cluster can tolerate
NOTE: on a cluster with HA, the maximum number of VMs per host is 64 and per cluster 2048.
So let’s start to build my test environment.
1. First, I took one of my physical esxi servers. This server simply had 12 cpus, a few network cards (only need one here) and 96GB of memory. 
Typical install, then I used the dcui and changed it's ip to 192.168.95.120/24 and called it esxi.vovando.com.
I used:
Product: VMware ESXi 5.5
Release date: February 5, 2015
Patch: ESXi550-201502001
Build: 2456374
2. Then, I connected to my esxi server using the vsphere client from my Windows 8.1. I proceeded to create two new vswitches, called vSwitch2 and vSwitch3. These are internal vswitches, attached to no uplinks. These vswitches eventually were used for the vmotion network and vsan network for the embedded esxi servers.
It is CRITICAL to remember to change Promiscous Mode to ACCEPT to allow for internal communication. These switches had VM networks: vMotion-test and vSan-test.
3. Now, it’s time to create the three embedded esxi servers. Using the vsphere client, I created three identical esxi servers (esxi1 – 192.168.95.131), (esxi2 – 192.168.95.132) and (esxi3 – 192.168.95.133). I gave each embedded esxi server 2 vcpus, 5 gbs of ram and 3 disks. The 3 disks were 4gbs for the boot disk, 10gbs for the ssd disk and 100gbs for the HDD to be used by vsan.
4. After the installation of each of the embedded esxi servers, I changed the disk type of each of the future ssd disks using the esxcli command. Here are the commands:
$ esxcli storage nmp satp rule add –satp=VMW_SATP_LOCAL –device mpx.vmhba1:C0:T1:L0 –option "enable_local enable_ssd"
$ esxcli storage core claiming unclaim –type=device –device mpx.vmhba1:C0:T1:L0
You need to reboot host after it.
5. Now, it was time to create the vcenter server. I used the windows-based vcenter server. Ip address that was 192.168.95.130.
7. Connected to the vcenter server with the vsphere client, logged in and I created a datacenter and added the three esxi servers.
8. Using the vsphere client, it was time to create the vmotion network and the vsan network, which can't be created with the vsphere client only.
vmotion
esxi01 – 10.1.0.131
esxi02 – 10.1.0.132
esxi03 – 10.1.0.133
vsan
esxi01 – 10.2.0.131
esxi02 – 10.2.0.132
esxi03 – 10.2.0.133
9. Using the web client (port 9443), create a cluster for vsan and create the disk groups necessary. If you select manual configuration, you need to create three disk groups (one per host) using the fake ssd disk and the fake HDD disk. By the time you are done, you should have a datastore that is roughly 300gb. While creating the cluster, I also enabled DRS (fully automatic) and HA (which uses the vsan network for heartbeats).
After that this Cluster can be used for work.
10. vSan related commands.
#esxcli vsan
Usage: esxcli vsan {cmd} [cmd options]
Available Namespaces:
datastore Commands for VSAN datastore configuration
network Commands for VSAN host network configuration
storage Commands for VSAN physical storage configuration
cluster Commands for VSAN host cluster configuration
faultdomain Commands for VSAN fault domain configuration
maintenancemode Commands for VSAN maintenance mode operation
policy Commands for VSAN storage policy configuration
trace Commands for VSAN trace configuration
#esxcli vsan datastore name get
Name: vsanDatastore
#esxcli vsan network ipv4 add -i vmk2
This VMKernel network interface is already in use. Can't add again. If you want to reconfigure, use 'set' command.
#esxcli vsan network list
Interface
VmkNic Name: vmk2
IP Protocol: IPv4
Interface UUID: 26e84855-f8e9-f3ad-5ff8-005056873a81
Agent Group Multicast Address: 224.2.3.4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group Multicast Port: 12345
Multicast TTL: 5
# esxcli vsan network list
Interface
VmkNic Name: vmk2
IP Protocol: IPv4
Interface UUID: 26e84855-f8e9-f3ad-5ff8-005056873a81
Agent Group Multicast Address: 224.2.3.4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group Multicast Port: 12345
Multicast TTL: 5
# esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-05-07T16:14:52Z
Local Node UUID: 5548e218-ae75-c834-0331-005056873a81
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5548e429-d572-3814-2780-005056872eec
Sub-Cluster Backup UUID: 5548e218-ae75-c834-0331-005056873a81
Sub-Cluster UUID: 523a032b-4d44-8fbd-2954-74154216d6b2
Sub-Cluster Membership Entry Revision: 11
Sub-Cluster Member UUIDs: 5548e429-d572-3814-2780-005056872eec, 5548e218-ae75-c834-0331-005056873a81, 5548de32-1dc9-573c-6be5-005056877536
Sub-Cluster Membership UUID: f68c4b55-ec10-02a5-120c-005056872eec
# esxcli vsan storage list
mpx.vmhba1:C0:T2:L0
Device: mpx.vmhba1:C0:T2:L0
Display Name: mpx.vmhba1:C0:T2:L0
Is SSD: false
VSAN UUID: 52711c36-d9a2-c17b-7b4f-fe6ca685d0e8
VSAN Disk Group UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa
VSAN Disk Group Name: mpx.vmhba1:C0:T1:L0
Used by this host: true
In CMMDS: true
Checksum: 5186061164308225788
Checksum OK: true
Emulated DIX/DIF Enabled: false
mpx.vmhba1:C0:T1:L0
Device: mpx.vmhba1:C0:T1:L0
Display Name: mpx.vmhba1:C0:T1:L0
Is SSD: true
VSAN UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa
VSAN Disk Group UUID: 52887c5a-12a2-384d-679b-fb9ed6775bfa
VSAN Disk Group Name: mpx.vmhba1:C0:T1:L0
Used by this host: true
In CMMDS: true
Checksum: 10790749511316735198
Checksum OK: true
Emulated DIX/DIF Enabled: false
#esxcli vsan policy getdefault
Policy Class Policy Value
———— ——————————————————–
cluster (("hostFailuresToTolerate" i1))
vdisk (("hostFailuresToTolerate" i1))
vmnamespace (("hostFailuresToTolerate" i1))
vmswap (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmem (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
#grep -i vsan /etc/vmware/esx.conf
/net/vmkernelnic/child[0002]/portgroup = "vSAN"
/net/vswitch/child[0002]/portgroup/child[0000]/name = "vSAN"
/firewall/services/vsanvp/enabled = "true"
/firewall/services/vsanvp/allowedall = "true"
/vsan/faultDomainVersion = "2"
/vsan/hostDecommissionVersion = "0"
/vsan/faultDomainName = ""
/vsan/subClusterUuid = "523a032b-4d44-8fbd-2954-74154216d6b2"
/vsan/autoClaimStorage = "true"
/vsan/enabled = "true"
/vsan/hostDecommissionMode = "decom-mode-none"
/vsan/hostDecommissionState = "decom-state-none"
/vsan/datastoreName = "vsanDatastore"
/vsan/network/child[0000]/ttl = "5"
/vsan/network/child[0000]/vmknic = "vmk2"
/vsan/network/child[0000]/masterPort = "12345"
/vsan/network/child[0000]/masterGroup = "224.1.2.3"
/vsan/network/child[0000]/ifaceUuid = "26e84855-f8e9-f3ad-5ff8-005056873a81"
/vsan/network/child[0000]/agentGroup = "224.2.3.4"
/vsan/network/child[0000]/agentPort = "23451"
/adv/Misc/HostName = "esxi02-vsan6"
#esxcfg-advcfg -l | grep -i vsan
/VSAN/ClomRepairDelay [Integer] : Minutes to wait for absent components to come back before starting repair (REQUIRES clomd RESTART!)
/VSAN/ClomMaxComponentSizeGB [Integer] : Maximum component size used for new placements (REQUIRES clomd RESTART!))
/VSAN/DomLongOpTraceMS [Integer] : Trace ops that take more than the specified value in milliseconds
/VSAN/DomLongOpUrgentTraceMS [Integer] : Urgent trace ops that take more than the specified value in milliseconds
/VSAN/DomBriefIoTraces [Integer] : Enables a brief set of per-IO DOM traces for debugging
/VSAN/DomFullIoTraces [Integer] : Enables full set of per-IO DOM traces; if disabled, IO op traces go to the per-op trace table
/VSAN/TraceEnableDom [Integer] : DOM tracing enabled
/VSAN/TraceEnableDomIo [Integer] : DOMIO tracing enabled
/VSAN/TraceEnableLsom [Integer] : LSOM tracing enabled
/VSAN/TraceEnableCmmds [Integer] : CMMDS/CMMDSResolver tracing enabled
/VSAN/TraceEnableRdt [Integer] : RDT tracing enabled
/VSAN/TraceEnablePlog [Integer] : PLOG tracing enabled
/VSAN/TraceEnableSsdLog [Integer] : SSDLOG tracing enabled
/VSAN/TraceEnableVirsto [Integer] : Virsto tracing enabled
/VSAN/VsanSparseEnabled [Integer] : Enable auto-creation of vsanSparse instead of vmfsSparse redologs, for VSAN 2.0 datastore only
/VSAN/VsanSparseCacheThreshold [Integer] : Maximum number of entries in single VsanSparse cache
/VSAN/VsanSparseCacheOverEvict [Integer] : Percentage of VsanSparseCacheThreshold to add to eviction
/VSAN/VsanSparseSpeculativePrefetch [Integer] : Number of bytes to add onto each extent interrogation request
/VSAN/VsanSparseMaxExtentsPrefetch [Integer] : Maximum number of extents to fetch during interrogation
/VSAN/VsanSparseParallelLookup [Integer] : Request written extent data from each layer in parallel
/VSAN/TraceEnableVsanSparse [Integer] : VsanSparse tracing enabled
/VSAN/TraceEnableVsanSparseIO [Integer] : VsanSparse per-IO tracing enabled
/VSAN/TraceEnableVsanSparseVerbose [Integer] : VsanSparse very verbose tracing enabled
/VSAN/VsanSparseHeapSize [Integer] : Maximum heap size for VsanSparse snapshot consolidation buffers(in KiB)
Next time I will tell you how to upgrade your vSan Cluster to vSAN 6.0.
Stay tuned!