vSAN Health Service - Component Limits

 

vSAN의 Object를 구성하는 Component는 최대 9000개의 Limit이 존재합니다.

만약, Component 갯수가 너무 많이 늘어나는 경우에는 다음과 같이 vSAN Skyline에서 경고가 발생합니다.

아래 캡쳐 화면을 보시면, 1대를 제외하고 3대의 Component 사용률이 모두 80%를 넘은 상태입니다.

 

 

Component 갯수가 Limit 값인 9000개까지 도달하는 경우에, 정상적인 상황이라면 Host를 추가하거나 불필요한 vSAN Object를 제거해야 합니다.

 

본 케이스의 경우에는 Disk 용량도 많이 남아 있는 상황이었는데 Component만 매우 많이 차 있었던 특이한 상황이었습니다.

 

수집된 자료에서 vSAN Object 현황을 보면, 다음과 같이 매우 많은 Missing VMDK 들이 있는 것을 확인할 수 있었습니다. 정상적이라면, Missing 대신 Exist로 기재되었습니다.

grep -i "Missing" localcli_vsan-debug-object-list.txt   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/01339562-0c73-7cae-319f-e4434b879020/vm-a97f76f3-fb9b-4243-aec7-2c7e0ba5cc9c.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/b84e9562-946b-0a94-c3b9-e4434b879020/vm-de3b4aff-377f-4581-9b8d-d9a0c491391e.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/04489762-1078-43b1-33a7-e4434b879020/vm-0e14031e-9344-467b-b1c2-98fbcb37dc04.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/4c4e9462-388e-9a9a-521d-e4434b879020/vm-735ea28f-4e7b-4bd4-a6aa-7cb8a721314b.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/44289562-b28b-dab6-afbd-e4434b879020/vm-b7a3eca6-2409-4385-b306-0781344002f1.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/f0629762-eab9-2099-4883-e4434b879020/vm-66ba414b-42d0-4f13-9925-894b55bfadff.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/47c09362-2878-d4be-66a3-e4434b879020/vm-7b550227-5a74-4aeb-b2e4-cee545a124ed.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/fdaa2562-e6a8-7830-2cbd-e4434b8791b0/vm-74b72288-b6a5-4aa6-84f1-b3961cb7c874.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/85609562-9810-b192-b0e5-e4434b879020/vm-ac7d73f0-a20e-4f31-acc2-130b13129670.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/bec39262-fe10-99a9-c5e1-e4434b8796f0/vm-85157cc6-5041-47ee-b176-7446cbbaa4d5.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/1d629762-dc86-8fc0-49fb-e4434b879020/vm-f165a574-e4e9-4bd4-85bd-71bbec4cee32.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/ffb09262-bc75-3c7d-3446-e4434b879020/vm-5d03192a-c866-4a8c-83e9-175724dca6b2.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/48229362-12b7-28cd-335b-e4434b8796f0/vm-5958d8b7-9cf7-4570-8a02-552e9b464336.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/43279562-ee9e-99be-75fa-e4434b879020/vm-b7a3eca6-2409-4385-b306-0781344002f1.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/ac029761-b673-576b-74a0-e4434b8791b0/vm-390d932c-2ac1-4979-986c-0bfe4d8c0123.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/306c9862-b8e4-e1bb-3e8a-e4434b879020/vm-76bdb4af-bfa5-4068-a3f3-c64cee3ea8d9.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/2b201561-907a-b149-5ebe-e4434b8791b0/vm-1a9a00bb-0d89-4e56-aee3-07beba044aad.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/98c89762-e4a6-5e99-d7f1-e4434b879020/vm-550a05ab-6714-47c1-9e91-68e54691ebfc.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/4b419262-7202-23bc-6ee0-e4434b879020/vm-abcffe04-45f2-4046-99cb-5056ca824df3.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/eb059662-b04e-2bb8-0104-e4434b879020/vm-7edfb0ae-747c-42b0-8f88-797d1c0c5c12.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/12d49762-b4d8-41ba-5674-e4434b879020/vm-f693e68b-16c6-4ed8-9deb-76206f6394c9.vmdk (Missing) … … …   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/e78a9262-96cd-7fcd-2f65-e4434b8796f0/vm-de504acb-4e64-4974-81b9-b02311d64eb5.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/81129462-1cf9-5399-1b0a-e4434b879020/vm-71cf3b1f-9818-40ee-ac22-6c43342f7a3c.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/546c9362-ae4e-08b4-e861-e4434b879020/vm-5c185eae-03ff-44bc-96e3-c6b8fbf28a53.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/1ebf9462-eccc-379b-ee85-e4434b879020/vm-ffec6ef6-4f2c-4b2b-8ff8-582eb9ba6b4a.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/a2129662-f43e-09be-cb26-e4434b879020/vm-424e55a7-c856-49ca-9938-857be754ec7b.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/8e822562-6417-b74c-5863-e4434b8791b0/vm-1fdcaa3b-fe64-40ec-9ea3-5605a289bdd0.vmdk (Missing)   /vmfs/volumes/vsan:524b2c1b7900b25e-16643ddb9fa35d1d/92592562-562b-cf3c-be24-e4434b8791b0/vm-b7860bba-bc3a-4459-b8c1-f05e3efb1b77.vmdk (Missing)

 

예를 들어, Missing으로 표시된 VMDK의 vSAN Object를 하나 살펴보겠습니다.

아래 예를 보면, 01339562-1af7-0000-3e25-e4434b879020 Object는 총 3개의 Component로 이루어져 있습니다.

01339562-1af7-0000-3e25-e4434b879020 : Object

01339562-8ac4-8d01-7851-e4434b879020 : Component

94ca8463-a4cb-1208-cfb1-e4434b8796f0 : Component

94ca8463-c272-1608-eeb3-e4434b8796f0 : Component

structtype:   ObjectInfo Health:   healthy Object UUID:   01339562-1af7-0000-3e25-e4434b879020 Version:   15 Owner:   xxx.abc.com Policy:   stripeWidth: 1       cacheReservation: 0       proportionalCapacity: [0, 100]       hostFailuresToTolerate: 1       forceProvisioning: 0       spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad       spbmProfileGenerationNumber: 1       replicaPreference: Performance       CSN: 4178       SCSN: 3685       spbmProfileName: vSAN Default Storage Policy   Used:   12582912 Size:   68719476736 Type:   vdisk Path:   /vmfs/volumes/vsan:xxx/xxx/xxx.vmdk (Missing) Group UUID:   01339562-0c73-7cae-319f-e4434b879020 Directory Name:   N/A Configuration:         RAID_1          Component: 01339562-8ac4-8d01-7851-e4434b879020            Component State: ACTIVE,  Address Space(B): 68719476736 (64.00GB),  Disk UUID: 52656941-ba03-439c-85e5-fbd3f1dd79e3,  Disk Name: naa.58ce38ee20a28fe9:2            Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: xxx.abc.com          Component: 94ca8463-a4cb-1208-cfb1-e4434b8796f0            Component State: ACTIVE,  Address Space(B): 68719476736 (64.00GB),  Disk UUID: 52447b23-32de-c0c2-dbfe-4fc3434f4ea5,  Disk Name: naa.58ce38ee20973b9d:2            Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: xxx.abc.com       Witness: 94ca8463-c272-1608-eeb3-e4434b8796f0         Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 525e1ee7-a144-063d-8483-8fc254434cc6,  Disk Name: naa.58ce38ee20973bd9:2         Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: xxx.abc.com

 

위 결과는 localcli_vsan-debug-object-list.txt 파일의 결과이고, 실제로 vSAN Database인 CMMDS에도 동일하게 Object가 조회되는지 검증합니다.

마찬가지로 Object 하나에 3개의 Component가 조회됩니다.

# cat cmmds-tool_-f-json-readdump--p--d-scratchlogcmmdsdump.txt | grep 01339562-0c73-7cae-319f-e4434b879020    "content": { "type": "Configuration", "attributes": {"CSN": 4178, "SCSN": 3685, "addressSpace": 68719476736, "scrubStartTime": 1671697719097614, "objectVersion": 15, "highestDiskVersion": 15, "muxGroup": 364774624145916900, "groupUuid": "01339562-0c73-7cae-319f-e4434b879020", "raidFact": 200, "subClusterUUID": "524b2c1b-7900-b25e-1664-3ddb9fa35d1d", "compositeUuid": "01339562-1af7-0000-3e25-e4434b879020", "objCapabilities": 1},   "child-1": { "type": "RAID_1", "attributes": {"scope": 3},   "child-1": { "type": "Component", "attributes": { "capacity": [0, 68719476736], "addressSpace": 68719476736, "componentState": 5, "componentStateTS": 1669640185, "faultDomainId": "5db32b0c-8a47-dbd8-595e-e4434b8796f0", "flags": 128, "nVotes": 1, "subFaultDomainId": "5db32b0c-8a47-dbd8-595e-e4434b8796f0" }, "componentUuid": "01339562-8ac4-8d01-7851-e4434b879020", "diskUuid": "52656941-ba03-439c-85e5-fbd3f1dd79e3" },   "child-2": { "type": "Component", "attributes": { "capacity": [0, 68719476736], "addressSpace": 68719476736, "componentState": 5, "componentStateTS": 1669646997, "faultDomainId": "5db6feb4-a38d-dfce-01ee-e4434b8791b0", "nVotes": 1, "subFaultDomainId": "5db6feb4-a38d-dfce-01ee-e4434b8791b0"}, "componentUuid": "94ca8463-a4cb-1208-cfb1-e4434b8796f0", "diskUuid": "52447b23-32de-c0c2-dbfe-4fc3434f4ea5"}},   "child-2": { "type": "Witness", "attributes": {"componentState": 5, "componentStateTS": 1669646996, "isWitness": 1, "faultDomainId": "5db32ed6-d8ff-a626-7318-e4434b879020", "nVotes": 1, "subFaultDomainId": "5db32ed6-d8ff-a626-7318-e4434b879020"}, "componentUuid": "94ca8463-c272-1608-eeb3-e4434b8796f0", "diskUuid": "525e1ee7-a144-063d-8483-8fc254434cc6"}},

 

VMDK는 Missing인데, 해당 VMDK와 관련된 Object와 Component는 모두 정상적으로 조회되는 상태입니다.

또한, Component 들의 상태도 componentState 값이 5로 Active를 의미합니다.

즉, VMDK 파일은 없는데 vSAN Object와 Component는 모두 정상적으로 남아 있는 상황이었습니다.

 

따라서, VMDK 파일의 실제 존재 여부를 확인하고 실제로 존재하지 않는 VMDK인 경우 연관된 vSAN Object와 Component를 모두 제거하는 가이드를 제안드렸습니다.

1) object attribute 확인 # /usr/lib/vmware/osfs/bin/objtool getAttr -u 01339562-0c73-7cae-319f-e4434b879020 2) object 제거 # /usr/lib/vmware/osfs/bin/objtool delete -u 01339562-0c73-7cae-319f-e4434b879020 3) 제거된 object 확인 # /usr/lib/vmware/osfs/bin/objtool getAttr -u 01339562-0c73-7cae-319f-e4434b879020

 

위 작업을 진행하고 나서, 다음과 같이 Component 사용량이 줄어들었음을 확인할 수 있었습니다.

 

문제는 이런 경우 왜 VM이 제거되면서 VMDK 파일은 없어졌는데 vSAN 관련 Object는 제거되지 않았는지를 파악하는 것이 향후 문제를 방지하는데 그 다음 단계가 됩니다.