안녕하세요. 오늘은 ESXi OS에서 Storage 문제 발생 시 Handling 하는 Mechanism 중 하나인 APD에 대해서 다뤄보겠습니다.
APD는 Storage Device로의 Path가 모두 손실되거나, SAN 환경에 갑작스러운 단절이 생긴 경우 발생하는 Event
PDL(Permanent Device Loss)은 APD의 특별한 Case로 Storage Array에서 Device(LUN)이 제거된 경우
APD는 단순히 Storage Array의 Communication이 정상적으로 되지 않는 경우에 발생하는 것이지만, PDL은 명시적으로 Device의 Loss를 Storage Array로부터 상태 전달을 받아 Host가 판단할 수 있는 상황을 의미
APD 도입 배경
- 기존 ESXi Management Application 내에 Storage Connectivity에 문제가 생기는 경우 Handling 하는 방법이 구현되어 있지 않음
- Storage Connectivity에 문제가 생기면서 Data Consistency Issue가 생길 수 있는 가능성
- Application이나 System이 Crash 되는 경우 Data Corruption 발생 가능성
- Application은 Data Consistency를 위해서 fsync + flush 조합과 같은 방식으로 구현이 되어 있어야 함
- ESXi Management Application들이 문제가 발생한 Storage와의 Communication을 시도하면서, Slow Response 발생 가능성 ## hostd 프로세스의 Thread Starvation가능성
APD는 Storage Array와 연결 가능한 모든 Path에 대한 연결이 끊어졌을 때 발생하는 Event로, ESXi의 Storage Stack이 해당 현상이 얼마나 지속될 지 알 수 없는 것이 중요한 점
APD로 영향 받을 수 있는 유형
- Guest OS impact
- Hostd stuck
- ESXi Host Hang
- Resource Consumption ## VMFS Heartbeat Reclaim, PSA Device Probe, PSA issuing IOs to the device
- Commands taking a long time to complete
APD 발생 시 UI 결과와 로그
Storage Adapter 하위의 Device State가 "Dead or Error"로 표시
Device로 연결되는 모든 경로가 "Dead"로 표시
/var/run/log/vmkernel.log에는 다음과 같이 기록
| ## Case1. cpu1:2049)WARNING: NMP: nmp_IssueCommandToDevice:2954:I/O could not be issued to device "naa.60a98000572d54724a34642d71325763" due to Not found cpu1:2049)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60a98000572d54724a34642d71325763": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device. cpu1:2049)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60a98000572d54724a34642d71325763" is blocked. Not starting I/O from device. cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60a98000572d54724a34642d71325763" - issuing command 0x4124007ba7c0 cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world failover device "naa.60a98000572d54724a34642d71325763" - failed to issue command due to Not found (APD), try again... cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:708:Logical device "naa.60a98000572d54724a34642d71325763": awaiting fast path state update... ## Case2. ## APD Timer Start 2023-06-01T12:17:38.499Z cpu62:2198391)LVM: 6021: Received APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 1) 2023-06-01T12:17:38.499Z cpu62:2198391)LVM: 5609: Handling APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 1, cur apd state UNKNOWN, cur dev state 1) 2023-06-01T12:17:38.499Z cpu62:2198391)HBX: 6735: APD EventType: APD_START (3) for vol 'test' 2023-06-01T12:17:38.499Z cpu62:2198391)ScsiDevice: 7002: Device state of naa.60003ff44dc75adc9298fcd222e38251 set to APD_START; token num:1 2023-06-01T12:17:38.499Z cpu62:2198391)StorageApdHandler: 1191: APD start for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:17:38.499Z cpu2:2097578)StorageApdHandler: 408: APD start event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:17:38.499Z cpu2:2097578)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down state. 2023-06-01T12:17:38.819Z cpu63:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... 2023-06-01T12:17:39.818Z cpu63:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... ## APD Timer expired after default timeout value(140 seconds) 2023-06-01T12:19:58.496Z cpu2:2097578)StorageApdHandler: 606: APD timeout event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-01T12:19:58.496Z cpu2:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will n$ 2023-06-01T12:19:58.500Z cpu2:2097578)LVM: 6021: Received APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 2) 2023-06-01T12:19:58.500Z cpu2:2097578)LVM: 5609: Handling APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 2, cur apd state APD_START, cur dev state 1) 2023-06-01T12:19:58.500Z cpu2:2097578)HBX: 6735: APD EventType: APD_TIMEOUT (5) for vol 'test' 2023-06-01T12:19:58.500Z cpu2:2097578)StorageApdHandler: 606: APD timeout event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:19:58.500Z cpu2:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will n$ ## APD Recovered 2023-06-01T15:20:02.768Z cpu1:2097384)ScsiDevice: 7057: Device naa.60003ff44dc75adc8116d1595e544eb9 is Out of APD; token num:1 2023-06-01T15:20:02.768Z cpu8:2097578)StorageApdHandler: 501: APD exit event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9, 0] 2023-06-01T15:20:02.768Z cpu8:2097578)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has exited the All Paths Down state. 2023-06-01T15:20:02.770Z cpu8:2097384)LVM: 6021: Received APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 3) 2023-06-01T15:20:02.770Z cpu8:2097384)LVM: 5609: Handling APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 3, cur apd state APD_TIMEOUT, cur dev state 1) 2023-06-01T15:20:02.770Z cpu8:2097384)ScsiDevice: 7057: Device naa.60003ff44dc75adc9298fcd222e38251 is Out of APD; token num:1 2023-06-01T15:20:02.770Z cpu23:2097375)LVM: 5609: Handling APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (locked, gen 3, cur apd state APD_TIMEOUT, cur dev state 1) 2023-06-01T15:20:02.798Z cpu23:2097375)HBX: 6735: APD EventType: APD_EXIT (4) for vol 'test' 2023-06-01T15:20:02.799Z cpu33:2098245)HBX: 5561: Completed APD/PDL event processing on vol 'test'. 2023-06-01T15:20:02.799Z cpu8:2097578)StorageApdHandler: 501: APD exit event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251, 0] 2023-06-01T15:20:02.799Z cpu8:2097578)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has exited the All Paths Down state. |
APD 동작 방식
Storage Device가 ESXi Host 입장에서 가용하지 않게 되었을 때, Timeout period(140 seconds)가 시작되고 APD state로 진입
Timeout이 만료되기 전까지, Host는 Storage Device와 다시 연결하기 위해서 재연결을 계속 시도
## 이 때 발생하는 I/O는 PSA Layer에서 Queueing 되고 하위 Layer로 전달되지 않음
Timeout이 만료되고, Device가 Recover되지 않으면 Host는 IO를 Retry를 중지
## 예를 들어, 장애가 발생한 시점 직후에, df 명령어를 실행하면 프롬프트가 떨어지지 않고 Timeout period인 140초가 지나고 나면 정상적으로 df 명령어의 결과가 보임
APD Timeout 기간 동안 df 결과
| # df ## 140초 동안 결과가 출력되지 않음 VmFileSystem: Slow refresh failed: Cannot open volume: /vmfs/volumes/6465bcd0-1cbb0916-fea3-005056a130fd Filesystem Bytes Used Available Use% Mounted on NFS 6333186977792 4401786589184 1931400388608 70% /vmfs/volumes/ISOs VMFS-6 341986770944 203662819328 138323951616 60% /vmfs/volumes/w2-tse-d12_1 VMFS-6 536602476544 443750023168 92852453376 83% /vmfs/volumes/645679753b6cd94539b8c824_jhaewon_iscsi_0 VMFS-6 536602476544 87910514688 448691961856 16% /vmfs/volumes/6460af633b6cd94539b8c8fa_jhaewon_iscsi_0 VMFS-6 0 0 0 0% /vmfs/volumes/test VMFS-L 128580583424 5780799488 122799783936 4% /vmfs/volumes/OSDATA-644175c5-99ca7a24-d576-78ac44128f94 vfat 4293591040 211484672 4082106368 5% /vmfs/volumes/BOOTBANK1 vfat 4293591040 65536 4293525504 0% /vmfs/volumes/BOOTBANK2 |
VMCP(VM Component Protection) with APD : "Response for Datastore with All Paths Down"
APD가 발생하면, Timer가 시작되고 140초의 Default Timer가 만료되면 해당 Device는 APD Timeout으로 표시
이 후 HA가 Couting을 시작하고, 3분의 HA Timeout이 만료되면 HA는 VM을 재시작 할 수도 있음 ## 이는 설정에 따라 달라짐
"Conservative"인 경우, 만약 다른 Host에서 VM을 Restart 할 수 있다고 판단되면 VM을 재시작
"Aggressive"인 경우, 다른 Host의 상태를 알지 못해도 일단 VM을 재시작 시도 ## 이는 VM 재시작 실패로 이어질 수 있음(다른 Host가 정상이 아니어서)
APD Advanced Parameter 테스트
[구성 환경]
[테스트 1. APD Event Trigger]
iSCSI Target 서버의 Network Adapter를 Disconnect
Device 및 Path 상태
Device는 "Dead or Error", Path는 "Dead"
/var/run/log/vmkernel.log 확인
| 2023-06-01T12:14:34.387Z cpu3:2097914)StorageApdHandler: 966: APD Handle Created with lock[StorageApd-0x4306042f0f10] 2023-06-01T12:14:34.394Z cpu3:2097914)StorageApdHandler: 966: APD Handle Created with lock[StorageApd-0x430604415660] 2023-06-01T12:17:38.495Z cpu6:2097384)ScsiDevice: 7002: Device state of naa.60003ff44dc75adc8116d1595e544eb9 set to APD_START; token num:1 2023-06-01T12:17:38.495Z cpu6:2097384)StorageApdHandler: 1191: APD start for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-01T12:17:38.495Z cpu0:2097578)StorageApdHandler: 408: APD start event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-01T12:17:38.495Z cpu0:2097578)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has entered the All Paths Down state. 2023-06-01T12:17:38.499Z cpu62:2198391)LVM: 6021: Received APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 1) 2023-06-01T12:17:38.499Z cpu62:2198391)LVM: 5609: Handling APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 1, cur apd state UNKNOWN, cur dev state 1) 2023-06-01T12:17:38.499Z cpu62:2198391)HBX: 6735: APD EventType: APD_START (3) for vol 'test' 2023-06-01T12:17:38.499Z cpu62:2198391)ScsiDevice: 7002: Device state of naa.60003ff44dc75adc9298fcd222e38251 set to APD_START; token num:1 2023-06-01T12:17:38.499Z cpu62:2198391)StorageApdHandler: 1191: APD start for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:17:38.499Z cpu2:2097578)StorageApdHandler: 408: APD start event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:17:38.499Z cpu2:2097578)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down state. 2023-06-01T12:17:38.819Z cpu63:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... 2023-06-01T12:17:39.818Z cpu63:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... 2023-06-01T12:19:58.496Z cpu2:2097578)StorageApdHandler: 606: APD timeout event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-01T12:19:58.496Z cpu2:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will n$ 2023-06-01T12:19:58.500Z cpu2:2097578)LVM: 6021: Received APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 2) 2023-06-01T12:19:58.500Z cpu2:2097578)LVM: 5609: Handling APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 2, cur apd state APD_START, cur dev state 1) 2023-06-01T12:19:58.500Z cpu2:2097578)HBX: 6735: APD EventType: APD_TIMEOUT (5) for vol 'test' 2023-06-01T12:19:58.500Z cpu2:2097578)StorageApdHandler: 606: APD timeout event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-01T12:19:58.500Z cpu2:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will n$ 2023-06-01T15:20:02.768Z cpu1:2097384)ScsiDevice: 7057: Device naa.60003ff44dc75adc8116d1595e544eb9 is Out of APD; token num:1 2023-06-01T15:20:02.768Z cpu8:2097578)StorageApdHandler: 501: APD exit event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9, 0] 2023-06-01T15:20:02.768Z cpu8:2097578)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has exited the All Paths Down state. 2023-06-01T15:20:02.770Z cpu8:2097384)LVM: 6021: Received APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 3) 2023-06-01T15:20:02.770Z cpu8:2097384)LVM: 5609: Handling APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 3, cur apd state APD_TIMEOUT, cur dev state 1) 2023-06-01T15:20:02.770Z cpu8:2097384)ScsiDevice: 7057: Device naa.60003ff44dc75adc9298fcd222e38251 is Out of APD; token num:1 2023-06-01T15:20:02.770Z cpu23:2097375)LVM: 5609: Handling APD EventType: APD_EXIT (4) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (locked, gen 3, cur apd state APD_TIMEOUT, cur dev state 1) 2023-06-01T15:20:02.798Z cpu23:2097375)HBX: 6735: APD EventType: APD_EXIT (4) for vol 'test' 2023-06-01T15:20:02.799Z cpu33:2098245)HBX: 5561: Completed APD/PDL event processing on vol 'test'. 2023-06-01T15:20:02.799Z cpu8:2097578)StorageApdHandler: 501: APD exit event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251, 0] 2023-06-01T15:20:02.799Z cpu8:2097578)StorageApdHandlerEv: 117: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has exited the All Paths Down state. |
[테스트 2. APD Timeout Increase]
1번과 동일한 시나리오에서 APD Timeout 값을 Default인 140 초에서 300초로 변경하고 테스트
ESXi Host > Configure > Misc.APDTimeout
Change timeout limits for storage APD
| The timeout parameter controls how many seconds the ESXi host will retry non-virtual machine I/O commands to a storage device in an all paths down (APD) state. If needed, you can change the default timeout value. The timer starts immediately after the device enters the APD state. When the timeout expires, the host marks the APD device as unreachable and fails any pending or new non-virtual machine I/O. |
/var/run/log/vmkernel.log 확인
| 2023-06-02T06:24:24.238Z cpu29:2097385)ScsiDevice: 7002: Device state of naa.60003ff44dc75adc8116d1595e544eb9 set to APD_START; token num:1 2023-06-02T06:24:24.238Z cpu29:2097385)StorageApdHandler: 1191: APD start for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-02T06:24:24.238Z cpu0:2097578)StorageApdHandler: 408: APD start event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-02T06:24:24.238Z cpu0:2097578)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has entered the All Paths Down state. 2023-06-02T06:24:24.239Z cpu43:2198391)LVM: 6021: Received APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 10) 2023-06-02T06:24:24.239Z cpu43:2198391)LVM: 5609: Handling APD EventType: APD_START (3) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 10, cur apd state UNKNOWN, cur dev state 1) 2023-06-02T06:24:24.239Z cpu43:2198391)HBX: 6735: APD EventType: APD_START (3) for vol 'test' 2023-06-02T06:24:24.239Z cpu43:2198391)ScsiDevice: 7002: Device state of naa.60003ff44dc75adc9298fcd222e38251 set to APD_START; token num:1 2023-06-02T06:24:24.239Z cpu43:2198391)StorageApdHandler: 1191: APD start for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-02T06:24:24.239Z cpu0:2097578)StorageApdHandler: 408: APD start event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-02T06:24:24.239Z cpu0:2097578)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down state. 2023-06-02T06:24:24.986Z cpu53:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... 2023-06-02T06:24:25.987Z cpu48:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... ## APD_TIMEOUT after 300 seconds 2023-06-02T06:29:24.238Z cpu0:2097578)StorageApdHandler: 606: APD timeout event for 0x430604415660 [naa.60003ff44dc75adc8116d1595e544eb9] 2023-06-02T06:29:24.239Z cpu0:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc8116d1595e544eb9] has entered the All Paths Down Timeout state after being in the All Paths Down state for 300 seconds. I/Os will n$ 2023-06-02T06:29:24.240Z cpu1:2097578)LVM: 6021: Received APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (gen 11) 2023-06-02T06:29:24.240Z cpu1:2097578)LVM: 5609: Handling APD EventType: APD_TIMEOUT (5) for device <naa.60003ff44dc75adc9298fcd222e38251:1> (unlocked, gen 11, cur apd state APD_START, cur dev state 1) 2023-06-02T06:29:24.240Z cpu1:2097578)HBX: 6735: APD EventType: APD_TIMEOUT (5) for vol 'test' 2023-06-02T06:29:24.241Z cpu1:2097578)StorageApdHandler: 606: APD timeout event for 0x4306042f0f10 [naa.60003ff44dc75adc9298fcd222e38251] 2023-06-02T06:29:24.241Z cpu1:2097578)StorageApdHandlerEv: 126: Device or filesystem with identifier [naa.60003ff44dc75adc9298fcd222e38251] has entered the All Paths Down Timeout state after being in the All Paths Down state for 300 seconds. I/Os will n$ |
[테스트 2. APD Handling Disable]
2번과 동일한 시나리오에서 APD Handling 값을 Disable 변경하고 테스트
Disable storage APD handling
| The storage all paths down (APD) handling on your ESXi host is enabled by default. When it is enabled, the host continues to retry I/O commands to a storage device in the APD state for a limited time period. When the time period expires, the host stops its retry attempts and terminates any I/O. You can disable the APD handling feature on your host. If you disable the APD handling, the host will indefinitely continue to retry issued commands in an attempt to reconnect to the APD device. Continuing to retry is the same behavior as in ESXi version 5.0. This behavior might cause virtual machines on the host to exceed their internal I/O timeout and become unresponsive or fail. The host might become disconnected from vCenter Server. |
연결이 끊긴 Datastore에 위치한 VMDK를 사용하는 Guest OS 내의 볼륨은 응답 없음으로 빠짐
하지만, Storage Adapter 하위의 Device와 Path 상태는 변경되지 않음
/var/run/log/vmkernel.log 확인
APD 관련 Event는 기록되지 않고, 계속해서 "Retry world failover device" 메시지만 기록
| 2023-06-02T06:36:43.804Z cpu63:2100456 opID=f7b745e)StorageApdHandler: 334: APD handling enabled flag has changed, from 1 to 0 2023-06-02T06:36:43.804Z cpu63:2100456 opID=f7b745e)Config: 716: "APDHandlingEnable" = 0, Old Value: 1, (Status: 0x0) 2023-06-02T06:38:17.759Z cpu36:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... 2023-06-02T06:38:17.988Z cpu36:2097913)WARNING: NMP: nmpDeviceAttemptFailover:716: Retry world failover device "naa.60003ff44dc75adc9298fcd222e38251" - failed to issue command due to Not found (APD), try again... |
추가로 이와 같은 상태에서 ESXi Host에 SSH로 접근하여, 스토리지 관련 명령어(df, ls 등) 시도 시 완료되지 않음
'Storage' 카테고리의 다른 글
| How to use Hexdump (0) | 2023.07.26 |
|---|---|
| How Thin Provisioning Work - Space Reclamation (0) | 2023.06.21 |
| vSAN Objects(vDisk, Home Namespace and etc) (0) | 2023.06.01 |
| ATS(Atomic Test & Set) (0) | 2023.05.15 |
| Locked Files (0) | 2023.04.25 |
