Networking
[NSX] Generating support bundles fail after replacing certificates.
haewon83
2025. 2. 10. 08:08
NSX Manager의 인증서 중 APH-TN 인증서를 갱신하고 나서 Support Bundle 수집에 문제가 생기는 경우가 발생할 수 있습니다.
이러한 경우에는 NSX Manager 각 Node의 APH-TN 인증서를 각 Node의 UID를 이용하여 신규 생성 후 다시 갱신이 필요합니다.
[Symptom]
인증서 교체 후, NSX Manager UI에서 Support Bundle 수집 시 실패 현상 발생
[Troubleshooting Notes]
1. NSX 버전
| $ cat ./etc/nsx_issue version: 4.1.1.0.0.22224317 node-type: nsx-manager nsx-policy-manager nsx-controller build-type: release export-type: unrestricted |
2. Bundle 수집 요청 이력 확인
※ "New bundle request received", "Collecting support bundle content" 키워드 활용
| $ grep "New bundle request received" syslog* syslog:2025-01-24T00:20:48.388Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received syslog.6:2025-01-23T00:07:38.398Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received syslog.6:2025-01-23T00:08:35.876Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received while another request already in progress syslog.6:2025-01-23T00:27:09.872Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received syslog.6:2025-01-23T01:34:17.558Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received syslog.6:2025-01-23T01:53:00.741Z <HOSTNAME> NSX 1518 - [nsx@6876 comp="nsx-manager" subcomp="opsagent" s2comp="sbundle" tid="1551" level="INFO"] New bundle request received $ grep "Collecting support bundle content" syslog* syslog.7:2025-01-23T01:05:38.602Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for local node: c5cd1242-9409-f80d-4d7a-98c725a494a6 syslog.7:2025-01-23T01:53:00.665Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for remote node: d8381242-37d2-faf6-39e2-01a198652f06 (49868bd3b249de22df85) syslog.7:2025-01-23T01:53:00.672Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for remote node: 25cb1242-0cf5-8b9b-c967-c6fa21078dfc (6fd042058e4880f0152d) syslog.7:2025-01-23T02:00:19.555Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for local node: c5cd1242-9409-f80d-4d7a-98c725a494a6 |
3. NSX Manager Node ID 확인
| 25cb1242-0cf5-8b9b-c967-c6fa21078dfc >>>>>>>>>>>>>>>>>>>>>>>>> Node01 d8381242-37d2-faf6-39e2-01a198652f06 >>>>>>>>>>>>>>>>>>>>>>>>> Node02 c5cd1242-9409-f80d-4d7a-98c725a494a6 >>>>>>>>>>>>>>>>>>>>>>>>> Node03 |
4. 2번과 3번 NSX Manager Node에서 다음과 같이 RPC 호출 실패 확인
| ./var/log/syslog 2025-01-23T01:52:59.532Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Support bundle requested for 3 nodes 2025-01-23T01:53:00.665Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for remote node: d8381242-37d2-faf6-39e2-01a198652f06 (49868bd3b249de22df85) 2025-01-23T01:53:00.672Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="INFO"] Collecting support bundle content for remote node: 25cb1242-0cf5-8b9b-c967-c6fa21078dfc (6fd042058e4880f0152d) 2025-01-23T01:53:00.735Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="ERROR" errorCode="NOD101"] Error invoking method RpcMsgHandler on remote node c24eacc3-70e0-4f50-ae78-0c7138a2ec0a code 13 status: CallStatus(code=UNAVAILABLE, message="Requested service vmware.nsx.support_bundle.BundleHostService is not registered with forwarder. Check with service provider") 2025-01-23T01:53:00.736Z <HOSTNAME> NSX 1751 - [nsx@6876 comp="nsx-manager" subcomp="node-mgmt" username="root" level="ERROR" errorCode="NOD101"] Invoking support bundle status request on d8381242-37d2-faf6-39e2-01a198652f06 failed |
6. 관련 KB 내용 토대로 확인
After replacing APH-TN or APH-AR certificates, connections between Manager nodes or between GM and LM are disconnected
https://knowledge.broadcom.com/external/article/373270/after-replacing-aphtn-or-aphar-certifica.html
7. ./var/log/vmware/appl-proxy-rpc.log 로그 확인
| ./var/log/vmware/appl-proxy-rpc.log 2025-01-24T00:25:02.123Z <HOSTNAME> NSX 1831 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="1852" level="ERROR" errorCode="NET1111"] Certificate validation failed: 18-self signed certificate #012Certificate: #012 Data: #012 Version: 3 (0x2) #012 Serial Number: 1736836228533 (0x1946381a5b5) #012 Signature Algorithm: sha256WithRSAEncryption #012 Issuer: CN=VMware-NSX-ApplProxyHub; C=KR #012 Validity #012 Not Before: Jan 14 06:30:28 2025 GMT #012 Not After : Apr 19 06:30:28 2027 GMT #012 Subject: CN=VMware-NSX-ApplProxyHub; C=KR #012 Subject Public Key Info: #012 Public Key Algorithm: rsaEncryption #012 Public-Key: (2048 bit) ... |
8. NSX Manager 각 Node의 UID 확인 후, 각 Node별 CN(Common Name) 설정
| $ cat ./etc/vmware/nsx-appl-proxy/appl-proxy-public-cfg.json { "uuid" : "f7b7fdf0-82fd-4439-b523-7a1f879e19f8" } $ cat ./etc/vmware/nsx-appl-proxy/appl-proxy-public-cfg.json { "uuid" : "c24eacc3-70e0-4f50-ae78-0c7138a2ec0a" } $ cat ./etc/vmware/nsx-appl-proxy/appl-proxy-public-cfg.json { "uuid" : "2e607d21-708c-4548-b8f4-62fe2a06623b" } VMware-NSX-ApplProxyHub/UID=f7b7fdf0-82fd-4439-b523-7a1f879e19f8 VMware-NSX-ApplProxyHub/UID=c24eacc3-70e0-4f50-ae78-0c7138a2ec0a VMware-NSX-ApplProxyHub/UID=2e607d21-708c-4548-b8f4-62fe2a06623b |
9. 위 CN을 이용하여, Self-Signed Certificate 생성하고 갱신 요청