오늘은 vSAN에서 실행 중인 VM이 제거될 때 해당 VM과 연관되어 있던 vSAN Object 제거가 어떻게 이뤄지는지를 살펴보겠습니다.
테스트 용도의 VM을 제거하기 전 해당 VM이 사용하고 있던 VMDK와 연관된 vSAN Object를 알아보고,
VM이 지워지는 과정에서 hostd.log와 vsantrace 로그에 어떠한 내용들이 기록되는지를 확인할 수 있습니다.
삭제 전
VM ID 확인
"test" 이름의 VM의 VM ID는 4
[root@compesxi01:~] vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 1 vCLS-9767ef40-ae51-4905-b9de-5a21ffd81b43 [vsanDatastore] 27512e64-4070-dde4-4dfc-000c292e88b6/vCLS-9767ef40-ae51-4905-b9de-5a21ffd81b43.vmx other3xLinux64Guest vmx-11 vSphere Cluster Service VM is deployed from an OVA with a minimal installed profile of PhotonOS. vSphere Cluster Service manages the resources, power state and availability of these VMs. vSphere Cluster Service VMs are required for maintaining the health and availability of vSphere Cluster Service. Any impact on the power state or resources of these VMs might degrade the health of the vSphere Cluster Service and cause vSphere DRS to cease operation for the cluster. 4 test [vsanDatastore] 06932e64-8205-e28c-db1a-000c292e88b6/test.vmx centos7_64Guest vmx-19 |
vSAN Object 확인
"test" VM에 할당된 전체 VMDK의 Object 목록
Object UUID: 07932e64-2a83-931e-8fd6-000c292e88b6
/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmdk
Object UUID: 07932e64-c8e7-6330-e079-000c292e88b6
/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_1.vmdk
Object UUID: 07932e64-e6c2-b940-6de0-000c292e88b6
/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_2.vmdk
Object UUID: 07932e64-2a83-931e-8fd6-000c292e88b6: Version: 15 Health: healthy Owner: compesxi01.contoso.com Size: 16.00 GB ### <-- Size : 16 * 1024 * 1024 * 1024 = 17,179,869,184 Used: 0.01 GB Policy: stripeWidth: 1 cacheReservation: 0 proportionalCapacity: 0 hostFailuresToTolerate: 1 forceProvisioning: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad ### <-- UUID of storage policy spbmProfileGenerationNumber: 0 CSN: 2 spbmProfileName: vSAN Default Storage Policy ### <-- Name of storage policy Configuration: RAID_1 Component: 07932e64-7c63-0e1f-917b-000c292e88b6 Component State: ACTIVE, Address Space(B): 17179869184 (16.00GB), Disk UUID: 529e3823-5604-3d96-5eb8-ade1c3213729, Disk Name: mpx.vmhba3:C0:T1:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi01.contoso.com Component: 07932e64-4288-0f1f-869c-000c292e88b6 Component State: ACTIVE, Address Space(B): 17179869184 (16.00GB), Disk UUID: 527aa61f-c351-c6ca-0eaf-f862ce39c820, Disk Name: mpx.vmhba3:C0:T1:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi02.contoso.com Witness: 07932e64-c4b3-111f-ba26-000c292e88b6 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 527735d3-9124-fc9a-4874-72f01d89923e, Disk Name: mpx.vmhba3:C0:T1:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi03.contoso.com Type: vdisk Path: /vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmdk (Exists) Group UUID: 06932e64-8205-e28c-db1a-000c292e88b6 ### <-- virtual machine folder UUID Directory Name: N/A Object UUID: 07932e64-c8e7-6330-e079-000c292e88b6: Version: 15 Health: healthy Owner: compesxi01.contoso.com Size: 20.00 GB Used: 0.01 GB Policy: stripeWidth: 1 cacheReservation: 0 proportionalCapacity: 0 hostFailuresToTolerate: 1 forceProvisioning: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad spbmProfileGenerationNumber: 0 CSN: 2 spbmProfileName: vSAN Default Storage Policy Configuration: RAID_1 Component: 07932e64-e00f-e130-d48c-000c292e88b6 Component State: ACTIVE, Address Space(B): 21474836480 (20.00GB), Disk UUID: 52a5696d-7b06-9482-a202-0d36469e6655, Disk Name: mpx.vmhba3:C0:T2:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi02.contoso.com Component: 07932e64-5656-e230-c0c7-000c292e88b6 Component State: ACTIVE, Address Space(B): 21474836480 (20.00GB), Disk UUID: 52f7193e-a297-7cfc-e1c9-ddc9fade9117, Disk Name: mpx.vmhba3:C0:T2:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi01.contoso.com Witness: 07932e64-3000-e430-cd2f-000c292e88b6 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 52628be6-072b-7f33-de77-b8d790c67d96, Disk Name: mpx.vmhba3:C0:T2:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi03.contoso.com Type: vdisk Path: /vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_1.vmdk (Exists) Group UUID: 06932e64-8205-e28c-db1a-000c292e88b6 Directory Name: N/A Object UUID: 07932e64-e6c2-b940-6de0-000c292e88b6: Version: 15 Health: healthy Owner: compesxi01.contoso.com Size: 30.00 GB Used: 0.01 GB Policy: stripeWidth: 1 cacheReservation: 0 proportionalCapacity: 0 hostFailuresToTolerate: 1 forceProvisioning: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad spbmProfileGenerationNumber: 0 CSN: 2 spbmProfileName: vSAN Default Storage Policy Configuration: RAID_1 Component: 07932e64-daab-2c41-2f3e-000c292e88b6 Component State: ACTIVE, Address Space(B): 32212254720 (30.00GB), Disk UUID: 52f7193e-a297-7cfc-e1c9-ddc9fade9117, Disk Name: mpx.vmhba3:C0:T2:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi01.contoso.com Component: 07932e64-c8c6-2e41-a1a4-000c292e88b6 Component State: ACTIVE, Address Space(B): 32212254720 (30.00GB), Disk UUID: 52a5696d-7b06-9482-a202-0d36469e6655, Disk Name: mpx.vmhba3:C0:T2:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi02.contoso.com Witness: 07932e64-9867-3041-0d5b-000c292e88b6 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 527735d3-9124-fc9a-4874-72f01d89923e, Disk Name: mpx.vmhba3:C0:T1:L0:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: compesxi03.contoso.com Type: vdisk Path: /vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_2.vmdk (Exists) Group UUID: 06932e64-8205-e28c-db1a-000c292e88b6 Directory Name: N/A |
hostd 로그에서 VMDK 정보 확인
Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmdk'
Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_1.vmdk'
Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_2.vmdk'
[root@compesxi01:/vmfs/volumes/642cec28-94575026-9a03-000c292e88b6/log] grep -i test hostd.log | grep -i vmdk 2023-04-06T09:38:15.250Z info hostd[1050544] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmdk' specified: '<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>proportionalCapacity</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>proportionalCapacity</id><value xmlns:s19="http://www.w3.org/2001/XMLSchema" xsi:type="s19:int">0</value></propert 8b6/test.vmdk' 2023-04-06T09:38:15.255Z info hostd[1050544] [Originator@6876 sub=DiskLib opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CREATE : CREATE: "/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmdk" -- vmfs capacity=33554432 (16 GB) adapter=lsilogic info=cowGran=0 allocType=1 objType=vsan policy='<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>proportionalCapacity</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>proportionalCapacity</id><value xmlns:s19="http://www.w3.org/2001/XMLSchema" xsi:type="s19:int">0</value></propertyInsta 2023-04-06T09:38:15.393Z info hostd[1050544] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_1.vmdk' specified: '<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>stripeWidth</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>stripeWidth</id><value xmlns:s18="http://www.w3.org/2001/XMLSchema" xsi:type="s18:int">1</value></propertyInstance></cons 8b6/test_1.vmdk' 2023-04-06T09:38:15.398Z info hostd[1050544] [Originator@6876 sub=DiskLib opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CREATE : CREATE: "/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_1.vmdk" -- vmfs capacity=41943040 (20 GB) adapter=lsilogic info=cowGran=0 allocType=1 objType=vsan policy='<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>stripeWidth</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>stripeWidth</id><value xmlns:s18="http://www.w3.org/2001/XMLSchema" xsi:type="s18:int">1</value></propertyInstance></constraint 2023-04-06T09:38:15.524Z info hostd[1050544] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] Storage policy for disk '/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_2.vmdk' specified: '<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>stripeWidth</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>stripeWidth</id><value xmlns:s18="http://www.w3.org/2001/XMLSchema" xsi:type="s18:int">1</value></propertyInstance></cons 8b6/test_2.vmdk' 2023-04-06T09:38:15.529Z info hostd[1050544] [Originator@6876 sub=DiskLib opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CREATE : CREATE: "/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test_2.vmdk" -- vmfs capacity=62914560 (30 GB) adapter=lsilogic info=cowGran=0 allocType=1 objType=vsan policy='<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>stripeWidth</id><namespace>VSAN</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>stripeWidth</id><value xmlns:s18="http://www.w3.org/2001/XMLSchema" xsi:type="s18:int">1</value></propertyInstance></constraint |
hostd 로그에서 vSAN Object 확인
WatchVMObject, VM: '4', OBJ: '06932e64-8205-e28c-db1a-000c292e88b6'
WatchVMObject, VM: '4', OBJ: '07932e64-2a83-931e-8fd6-000c292e88b6'
WatchVMObject, VM: '4', OBJ: '07932e64-c8e7-6330-e079-000c292e88b6'
WatchVMObject, VM: '4', OBJ: '07932e64-e6c2-b940-6de0-000c292e88b6'
[root@compesxi01:/vmfs/volumes/642cec28-94575026-9a03-000c292e88b6/log] grep "WatchVMObject.*VM: '4'" hostd.log 2023-04-06T09:38:16.021Z info hostd[1050544] [Originator@6876 sub=VmObjectStorageMonitor opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] WatchVMObject, VM: '4', OBJ: '06932e64-8205-e28c-db1a-000c292e88b6' 2023-04-06T09:38:16.021Z info hostd[1050544] [Originator@6876 sub=VmObjectStorageMonitor opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] WatchVMObject, VM: '4', OBJ: '07932e64-2a83-931e-8fd6-000c292e88b6' 2023-04-06T09:38:16.021Z info hostd[1050544] [Originator@6876 sub=VmObjectStorageMonitor opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] WatchVMObject, VM: '4', OBJ: '07932e64-c8e7-6330-e079-000c292e88b6' 2023-04-06T09:38:16.021Z info hostd[1050544] [Originator@6876 sub=VmObjectStorageMonitor opID=lg4mgci9-5163-auto-3zg-h5:70002882-d8-01-9d-06f3 user=vpxuser:VSPHERE.LOCAL\Administrator] WatchVMObject, VM: '4', OBJ: '07932e64-e6c2-b940-6de0-000c292e88b6' |
clomd.log에서 VMDK 관련 Entry 확인
17179869184 = 16GiB
21474836480 = 20GiB
32212254720 = 30GiB
[root@compesxi01:/vmfs/volumes/642cec28-94575026-9a03-000c292e88b6/log] grep 17179869184 clomd.log |grep groupUuid |awk -F 'groupUuid' '{print $2}'|awk '{print $2}'|sort |uniq -c|grep 06932e64-8205-e28c-db1a-000c292e88b6 1 06932e64-8205-e28c-db1a-000c292e88b6) [root@compesxi01:/vmfs/volumes/642cec28-94575026-9a03-000c292e88b6/log] grep 21474836480 clomd.log |grep groupUuid |awk -F 'groupUuid' '{print $2}'|awk '{print $2}'|sort |uniq -c|grep 06932e64-8205-e28c-db1a-000c292e88b6 1 06932e64-8205-e28c-db1a-000c292e88b6) [root@compesxi01:/vmfs/volumes/642cec28-94575026-9a03-000c292e88b6/log] grep 32212254720 clomd.log |grep groupUuid |awk -F 'groupUuid' '{print $2}'|awk '{print $2}'|sort |uniq -c|grep 06932e64-8205-e28c-db1a-000c292e88b6 1 06932e64-8205-e28c-db1a-000c292e88b6) |
삭제 후
hostd 로그
2023-04-06T14:09:51.585Z info hostd[1050545] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-6595-auto-539-h5:70003376-dc-ec-34f5 user=vpxuser:VSPHERE.LOCAL\Administrator] State Transition (VM_STATE_ON -> VM_STATE_POWERING_OFF) 2023-04-06T14:09:51.959Z verbose hostd[1051533] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Vigor client state callback OFFLINE reason Failed to establish transport connection. 2023-04-06T14:09:51.960Z info hostd[1057691] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Bootstrap online state is not available (expected if VM just started or exited): Fault cause: vim.fault.GenericVmConfigFault 2023-04-06T14:09:51.961Z info hostd[1057691] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Deferring power op completion until VM is at a stable state 2023-04-06T14:09:51.961Z verbose hostd[1057691] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Fault Tolerance state callback received 2023-04-06T14:09:51.961Z verbose hostd[1050913] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] sleepState value is 0 2023-04-06T14:09:51.970Z warning hostd[1050913] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Query VMX about hlstate failed Fault cause: vim.fault.InvalidPowerState 2023-04-06T14:09:52.030Z info hostd[1052212] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] State Transition (VM_STATE_POWERING_OFF -> VM_STATE_OFF) 2023-04-06T14:09:52.055Z info hostd[1052212] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-5243-auto-41o-h5:70002927-d3-01-01-01-77-0751] Checking for all objects accessibility (VM's current state: VM_STATE_OFF, stable? true, under state update? true, fromCache: false) 2023-04-06T14:10:07.934Z info hostd[1057691] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-6642-auto-54j-h5:70003403-75-40-351e user=vpxuser:VSPHERE.LOCAL\Administrator] State Transition (VM_STATE_OFF -> VM_STATE_DELETING) 2023-04-06T14:10:08.600Z verbose hostd[1057691] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-6642-auto-54j-h5:70003403-75-40-351e user=vpxuser:VSPHERE.LOCAL\Administrator] Unregistered state change callback. 2023-04-06T14:10:08.600Z info hostd[1057691] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52a217702b1225fd-db713a5bdee9ad5e/06932e64-8205-e28c-db1a-000c292e88b6/test.vmx opID=lg4mgci9-6642-auto-54j-h5:70003403-75-40-351e user=vpxuser:VSPHERE.LOCAL\Administrator] State Transition (VM_STATE_DELETING -> VM_STATE_GONE) |
vsantraces 로그
[root@compesxi01:/vsantraces] /usr/lib/vmware/vsan/bin/vsanTraceReader.py vsantraces--2023-04-06T11h31m49s923.gz > vsantraces--2023-04-06T11h31m49s923.txt
Keyword : DOM_OBJ_CLEANUP_DELETE, preCleanupState, VMK_OBJECT_DESTROYED
2023-04-06T14:10:08.574025 [2026991] [cpu5] [c2db81ef CLIENT delete] DOMTraceOperationSendRequestToServerCompleted:5773: {'completionTime': '0:00:00.010036', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'op': '0x45791dcd4a40', 'pendingOps': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.574029 [2026992] [cpu5] [c2db81ef CLIENT delete] DOMTraceFlowctlRecordData:1039: {'completion time': '0:00:00.010039', 'congestion': 0, 'obj': '0x4579299c1d40', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'op': '0x45791dcd4a40'} 2023-04-06T14:10:08.574051 [2026993] [cpu13] [c2db81ef CLIENT delete] DOMTraceClientRPC:7534: {'auxUuid': '00000000-0000-0000-0000-000000000000', 'status': 'VMK_OK', 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.594278 [2027054] [cpu5] [OWNER] DOMTraceObjectCleanupObjectTree:5143: {'cleanupMode': 'DOM_OBJ_CLEANUP_DELETE', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'releaseCleanupStarted': 0} 2023-04-06T14:10:08.594291 [2027065] [cpu5] [OWNER] DOMTraceLivenessTransition:7350: {'barrierOp': '0x0', 'captureStateChanges': False, 'hasDelayedBarrier': False, 'isStale': False, 'liveness': 'strictly live', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'objectIsInitialized': True, 'oldLiveness': 'strictly live'} 2023-04-06T14:10:08.594292 [2027066] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4551: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CMMDS_DELETE', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.594327 [2027089] [cpu5] [26c1c8ac OWNER scrubObject] DOMTraceScrubObjectOpTask:2770: {'isStartTask': True, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'op': '0x4579ab7bb680', 'status': 'VMK_OBJECT_DESTROYED', 'taskIdx': 'HANDLE_OOB_ERRORS'} 2023-04-06T14:10:08.594327 [2027090] [cpu5] [26c1c8ac OWNER scrubObject] DOMTraceDispatcherTrace:7192: {'handlerCode': 'VMK_OBJECT_DESTROYED', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'op': '0x4579ab7bb680', 'opActAfter': 'NONE', 'opActBefore': 'TIMER', 'opActProcessed': 'AUTO_ADVANCE', 'opStateAfter': 'DOM_OP_STATE_START_TASK', 'opStateBefore': 'DOM_OP_STATE_START_TASK', 'opStateProcessed': 'DOM_OP_STATE_PROCESS_RESPONSE', 'resultCode': 'VMK_OBJECT_DESTROYED'} 2023-04-06T14:10:08.595031 [2027118] [cpu12] [26c611c4 COMP] DOMTraceCandidateOwnerCb:8083: {'errorCode': 'VMK_NOT_FOUND', 'obj-32': '0x731e2f00', 'objUuid': '06932e64-1a85-738d-0852-000c292e88b6', 'ownerUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595031 [2027120] [cpu12] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 1, 'isDisabled': False, 'isRetry': False, 'isShared': True, 'numFiringSubscrs-24': 0, 'numRetrySubscrs-24': 0, 'obj': '0x457a731e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 13, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OBJECT_DESTROYED', 'subscrEntry-32': '0x900001c0','subscrOp-32': '0x6c7fd7c0', 'unmarshalsRun': 0} 2023-04-06T14:10:08.595040 [2027126] [cpu3] [] CMMDSTraceActivateCallbacks:964: {'callbackCount': 3, 'duration': '0:00:00.000007', 'entryType': 21, 'entryUUID': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595059 [2027135] [cpu3] [] CMMDSTraceActivateCallbacks:964: {'callbackCount': 3, 'duration': '0:00:00.000002', 'entryType': 23, 'entryUUID': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595064 [2027137] [cpu3] [] CMMDSTraceActivateCallbacks:964: {'callbackCount': 1, 'duration': '0:00:00.000004', 'entryType': 23, 'entryUUID': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595070 [2027141] [cpu12] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 1, 'isDisabled': False, 'isRetry': False, 'isShared': False, 'numFiringSubscrs-24': 0, 'numRetrySubscrs-24': 0, 'obj': '0x457a731e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 23, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OK', 'subscrEntry-32': '0x90000100', 'subscrOp-32': '0x6c7fd7c0', 'unmarshalsRun': 0} 2023-04-06T14:10:08.595072 [2027145] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4562: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_PURGE_COMPONENTS', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595072 [2027146] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4452: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_SHUTDOWN_ASSOC', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595073 [2027147] [cpu5] [26c1c89c OWNER 2PC_POLL] DOMTraceObjectQuiesceOpIterBaseOp:3758: {'baseOp': '0x4579ab7dbe40', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595073 [2027148] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectQuiesceOpIterBaseOp:3758: {'baseOp': '0x4579ab43b340', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595073 [2027149] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4609: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CMMDS_DELETE2', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595077 [2027150] [cpu5] [] DOMTraceLookupOsfsName:5614: {'foundMatch': False, 'object UUID': '06932e64-8205-e28c-db1a-000c292e88b6', 'status': 'VMK_NOT_FOUND'} 2023-04-06T14:10:08.595077 [2027151] [cpu5] [OWNER] DOMTraceOwnerDeleteCmmdsNameEntryFailure:12096: {'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'status': 'VMK_NOT_FOUND'} 2023-04-06T14:10:08.595083 [2027152] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4452: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_INFLIGHT_OPS2', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595083 [2027153] [cpu5] [26c1c89c OWNER 2PC_POLL] DOMTraceObjectQuiesceOpIterBaseOp:3758: {'baseOp': '0x4579ab7dbe40', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595083 [2027154] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectQuiesceOpIterBaseOp:3758: {'baseOp': '0x4579ab43b340', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.595083 [2027155] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4628: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CLOSE_HANDLES', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595083 [2027156] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4682: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CHILD_START', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.595091 [2027164] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectPreCleanupNextState:4759: {'baseOp': '0x4579ab43b340', 'nextState': 'PRECLEANUP_CHILD_DONE', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'status': 'VMK_OK'} 2023-04-06T14:10:08.599182 [2027349] [cpu13] [c2db81ef] DOMTraceIoctlControlNodeCommon:3101: {'ioctlType': 'VSAN_DELETE_OBJECT', 'object UUID': '06932e64-8205-e28c-db1a-000c292e88b6', 'realWorldID': 1099586, 'status': 'VMK_OK', 'vobCtxHandle': '0xffffffff', 'vsanFDSHandle': '0x430b1fa01600'} 2023-04-06T14:10:08.599183 [2027350] [cpu13] [] DOMTraceVsanIoctlDone:4444: {'duration': '0:00:00.035457', 'ioctlCmd': 'FDS_IOCTL_VSAN', 'result': 'VMK_OK', 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'vsanFDSHandle':'0x430b1fa01600', 'vsanIoctlSubtype': 'VSAN_DELETE_OBJECT'} 2023-04-06T14:10:08.615224 [2027352] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4452: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CHILD_DONE', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.615224 [2027353] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectPreCleanupNextState:4759: {'baseOp': '0x4579ab43b340', 'nextState': 'PRECLEANUP_CHILD_DONE', 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'status': 'VMK_OK'} 2023-04-06T14:10:08.635372 [2027360] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4452: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_CHILD_DONE', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.635372 [2027361] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4712: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_INVOKE_CALLBACK', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.635378 [2027362] [cpu5] [] DOMTrace2PCEndSession:6452: {'ownerUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'pendingQueries': 0, 'pendingUpdates': 0, 'totalOpsSeen': 16076} 2023-04-06T14:10:08.635391 [2027365] [cpu5] [OWNER] DOMTraceObjectDestroyObject:4126: {'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:08.635392 [2027366] [cpu5] [26c611bd OWNER OBJ_PRECLEANUP] DOMTraceObjectDispatchPreCleanupOp:4745: {'baseOp': '0x4579ab43b340', 'forMovingObject': False, 'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'preCleanupState': 'PRECLEANUP_DONE', 'retryCount': 0, 'status': 'VMK_OK'} 2023-04-06T14:10:08.635474 [2027427] [cpu5] [OWNER] DOMTraceObject_ObjectDestructor:3940: {'obj': '0x457a087a2bc0', 'objUuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:13.575272 [2029387] [cpu5] [] DOMTraceCmmdsUnregisterSharedCb:2873: {'counter': 1, 'obj-32': '0x731e2f00', 'queryType': 13, 'sharedSubscrEntry-32': '0x4dc3e150', 'sharedSubscrHeader': '0x43144dc3e3a0', 'subUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'subscrEntry-32': '0x900001c0', 'waitRefCount': 4} 2023-04-06T14:10:13.575275 [2029390] [cpu5] [43198b6f7f20] CMMDSTraceUnsubscribe1:1876: {'owner': '00000000-0000-0000-0000-000000000000', 'type': 13, 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:13.575276 [2029392] [cpu5] [COMP] DOMTraceCmmdsUnregisterCb:2378: {'queryType': 23, 'subUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'subscrEntry': '0x457a90000100', 'waitRefCount': 4} 2023-04-06T14:10:13.575278 [2029393] [cpu5] [43198b6f7fc0] CMMDSTraceUnsubscribe1:1876: {'owner': '00000000-0000-0000-0000-000000000000', 'type': 23, 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:13.575278 [2029395] [cpu5] [COMP] DOMTraceCmmdsUnregisterCb:2378: {'queryType': 33, 'subUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'subscrEntry': '0x457a90000040', 'waitRefCount': 4} 2023-04-06T14:10:13.575280 [2029396] [cpu5] [43198b6f80e0] CMMDSTraceUnsubscribe1:1876: {'owner': '00000000-0000-0000-0000-000000000000', 'type': 33, 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:13.575280 [2029398] [cpu5] [COMP] DOMTraceCmmdsUnregisterCb:2378: {'queryType': 35, 'subUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'subscrEntry': '0x457a8fe00f40', 'waitRefCount': 4} 2023-04-06T14:10:13.575281 [2029399] [cpu5] [43198b6f8200] CMMDSTraceUnsubscribe1:1876: {'owner': '00000000-0000-0000-0000-000000000000', 'type': 35, 'uuid': '06932e64-8205-e28c-db1a-000c292e88b6'} 2023-04-06T14:10:13.575310 [2029417] [cpu5] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 0, 'isDisabled': True, 'isRetry': False, 'isShared': True, 'numFiringSubscrs-24': 3, 'numRetrySubscrs-24': 0, 'obj': '0x457a731e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 13, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OK', 'subscrEntry-32': '0x900001c0', 'subscrOp-32': '0x6c7fd7c0', 'unmarshalsRun': 0} 2023-04-06T14:10:13.575312 [2029419] [cpu5] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 0, 'isDisabled': True, 'isRetry': False, 'isShared': False, 'numFiringSubscrs-24': 2, 'numRetrySubscrs-24': 0, 'obj': '0x457a731e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 23, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OK', 'subscrEntry-32': '0x90000100', 'subscrOp-32':'0x6c7fd7c0', 'unmarshalsRun': 0} 2023-04-06T14:10:13.575312 [2029420] [cpu5] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 0, 'isDisabled': True, 'isRetry': False, 'isShared': False, 'numFiringSubscrs-24': 1, 'numRetrySubscrs-24': 0, 'obj': '0x457a731e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 33, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OK', 'subscrEntry-32': '0x90000040', 'subscrOp-32':'0x6c7fd7c0', 'unmarshalsRun': 0} 2023-04-06T14:10:13.575313 [2029421] [cpu5] [] DOMTraceProcessSubscrEntry:1927: {'fetchesRun': 0, 'isDisabled': True, 'isRetry': False, 'isShared': False, 'numFiringSubscrs-24': 0, 'numRetrySubscrs-24': 0, 'obj': '0x457a73 1e2f00', 'objType': 'COMP', 'processTimeMs': 0, 'queryType': 35, 'queryUuid': '06932e64-8205-e28c-db1a-000c292e 88b6', 'role': 'DOM_ROLE_COMPONENT_SERVER', 'status': 'VMK_OK', 'subscrEntry-32': '0x8fe00f40', 'subscrOp-32': '0x6c7fd7c0', 'unmarshalsRun': 0} |
'Storage' 카테고리의 다른 글
Locked Files (0) | 2023.04.25 |
---|---|
SCSI Protocol(+SCSI Sense Code, Errors) (0) | 2023.04.24 |
Driver/Firmware Check - HBA (0) | 2023.03.18 |
vSAN Health Service - Component Limits (2) | 2023.02.28 |
vsan-health service was not able to start (0) | 2023.01.23 |