vCenter에는 여러 종류의 인증서가 존재합니다.
인증서를 발급할 때는 UI나 command line에서 발급이 가능한데, 발급 시 필요한 common name이나 SAN(Subject Alternate Name)이 올바르지 않은 경우 발급한 인증서로 갱신했을 때 서비스가 시작되지 않을 수 있습니다.
이러한 현상을 겪고 있는 vCenter에서 Troubleshooting 한 내용을 알아보겠습니다.
[구성 환경]
vCenter Ver/Build: 6.5 / 19261680
Hostname으로 설치하지 않음
DNS 사용하지 않음
[문제 증상]
1. certifiacate-manager 를 이용하여서 8번 전체인증서 갱신이후
vmware 503 service unavailable (failed to connect to endpoint n7vmacore4http16localservicespece = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
2. fixsts.sh 수행후, 서비스 종료 하고 서비스 재기동시
Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=failed to start vapi-endpoint, vpxd-svcs services. Error: Operation time out
[Troubleshooting Notes]
1. 인증서 만료 기간 확인
1-1. STS 인증서
## 만료된 STS LEAF 인증서는 fixsts.sh 로 갱신
| root@photon-machine [ /tmp ]# ./checksts.py 1 VALID CERTS ================ LEAF CERTS: None ROOT CERTS: [] Certificate ### will expire in 2899 days (7.0 years). 1 EXPIRED CERTS ================ LEAF CERTS: [] Certificate: ### expired on 2023-10-08 05:58:40 GMT! ROOT CERTS: None WARNING! You have expired STS certificates. Please follow the KB corresponding to your OS: VCSA: https://kb.vmware.com/s/article/76719 Windows: https://kb.vmware.com/s/article/79263 |
1-2. Machine/Solution User 인증서
## 만료된 인증서 없음
| root@photon-machine [ ~ ]# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not AFter" ;done; [*] Store : MACHINE_SSL_CERT Alias : __MACHINE_CERT Not After : Sep 6 01:48:22 2025 GMT [*] Store : TRUSTED_ROOTS Alias : 2cd1f8cdc86b41e91acc28b25d1a18e3054cd089 Not After : Oct 3 06:07:35 2031 GMT [*] Store : machine Alias : machine Not After : Sep 6 01:48:46 2025 GMT [*] Store : vsphere-webclient Alias : vsphere-webclient Not After : Sep 6 01:48:57 2025 GMT [*] Store : vpxd Alias : vpxd Not After : Sep 6 01:49:08 2025 GMT [*] Store : vpxd-extension Alias : vpxd-extension Not After : Sep 6 01:49:19 2025 GMT [*] Store : SMS Alias : sms_self_signed Not After : Oct 8 06:14:46 2031 GMT [*] Store : BACKUP_STORE_H5C Alias : bkp__MACHINE_CERT Not After : Oct 8 18:08:05 2023 GMT Alias : bkpmachine Not After : Oct 8 05:59:38 2023 GMT Alias : bkpvsphere-webclient Not After : Oct 8 05:59:48 2023 GMT Alias : bkpvpxd Not After : Oct 8 05:59:59 2023 GMT Alias : bkpvpxd-extension Not After : Oct 8 06:00:09 2023 GMT |
1-3. VMCA 인증서(Root CA)
| root@photon-machine [ ~ ]# /usr/lib/vmware-vmca/bin/certool --getrootca --cert=vmca.crt Certificate: Data: Version: 3 (0x2) Serial Number: c9:0e:a1:a8:42:9d:1b:2a Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=photon-machine, OU=VMware Engineering Validity Not Before: Oct 5 06:07:35 2021 GMT Not After : Oct 3 06:07:35 2031 GMT Subject: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=photon-machine, OU=VMware Engineering Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: <snip> Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: <snip> X509v3 Subject Alternative Name: email:email@acme.com, IP Address:127.0.0.1 X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption <snip> Certificate written to file : vmca.crt Status : Success |
2. 구성 환경 확인
## hostname이 photon-machine으로 되어 있고,
## DCName과 PNID가 IP address로 설정
| ***************************************** *** Overview of Customers Environment *** ***************************************** - hostname: photon-machine uptime or last reboot: 1:04 reboot history, based on 'wtmp' file, filtered by 'reboot': # hint: if the 'uptime' command shows a different time than the 1st line of the 'wtmp' file, see https://via.vmw.com/EZvX reboot system boot 4.4.276-1.ph1 Wed Nov 1 00:46:19 2023 - Wed Nov 1 08:49:53 2023 (08:03) reboot system boot 4.4.276-1.ph1 Fri Sep 8 03:08:51 2023 - Wed Nov 1 00:44:44 2023 (53+22:35) reboot system boot 4.4.276-1.ph1 Thu Aug 18 03:07:14 2022 - Fri Sep 8 03:08:40 2023 (386+00:01) reboot system boot 4.4.243-1.ph1 Fri Oct 8 06:51:07 2021 - Thu Aug 18 03:07:03 2022 (313+20:15) ¦I¦ my uptime is '0' days, the GSS average is '74.5' days, based on '6414' samples - the Primary Network Identifyer (PNID) from the VMware Authentication Framework (vmafd) + "DCName" REG_SZ "192.168.1.5" + "DomainName" REG_SZ "vsphere.local" + "PNID" REG_SZ "192.168.1.5" + "SiteName" REG_SZ "contoso" Hint: if the hostname is not identical to the PNID then KB2130599 may be useful |
3. vCenter 6.5 설치 테스트 결과, 설치 시 FQDN을 이용하지 않고 IP Address를 입력하면 다음과 같이 hostname이 자동으로 photon-machine으로 설정
## 고객사도 동일하게 설치했을 것으로 추정
| root@photon-machine [ ~ ]# hostname photon-machine root@photon-machine [ ~ ]# cat /etc/systemd/resolved.conf [Resolve] LLMNR=false DNS=127.0.0.1 192.168.1.5 root@photon-machine [ ~ ]# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" + "CAPath" REG_SZ "/etc/ssl/certs" + "DCName" REG_SZ "192.168.1.5" + "DCPort" REG_DWORD 0x000001bb (443) + "DomainName" REG_SZ "vsphere.local" + "DomainState" REG_DWORD 0x00000001 (1) + "LDU" REG_SZ "e769bfa2-ec01-4770-87ac-7c18e1b6bb44" + "PNID" REG_SZ "192.168.1.5" + "RHTTPProxyPort" REG_DWORD 0x000001bb (443) + "SiteName" REG_SZ "samsung" "CertificateSyncInterval" REG_DWORD 0x0000003c (60) "DcCacheHeartBeat" REG_DWORD 0x0000001e (30) "DcCacheSyncInterval" REG_DWORD 0x0000003c (60) "EnableDCERPC" REG_DWORD 0x00000001 (1) "KeytabPath" REG_SZ "/usr/lib/vmware-vmafd/share/config/krb5.keytab" "Krb5Conf" REG_SZ "/etc/krb5.lotus.conf" "LegacyModeHA" REG_DWORD 0x00000000 (0) "LogFile" REG_SZ "/var/log/vmware/vmafd/vmafdd.log" |
4. /var/log/vmware/vapi/endpoint/endpoint.log 분석
## 문제 증상에서 Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=failed to start vapi-endpoint, vpxd-svcs services. Error: Operation time out 메시지를 확인했기 때문에 우선 vapi-endpoint 로그 분석
## STS 인증서를 갱신한 이후 시점 임에도 No subject alternative names matching IP address 192.168.1.5 found 와 같이 인증서 관련 오류 발생
| 2023-11-01T00:47:30.926Z | INFO | main | ApiEndpointServer | Using default start timeout value: 3,600,000 milliseconds 2023-11-01T00:47:30.944Z | INFO | main | NodeTypeUtil | Node type from configuration file is EMBEDDED 2023-11-01T00:47:30.944Z | INFO | main | ApiEndpointServer | Starting in management node mode. 2023-11-01T00:57:04.973Z | INFO | state-manager1 | BaseServerBuilder | Starting endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346 2023-11-01T00:57:05.144Z | INFO | state-manager1 | DefaultJettyServer | Starting jetty server. 2023-11-01T00:57:05.633Z | INFO | state-manager1 | BaseServerBuilder | Started endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346 . 2023-11-01T00:57:05.634Z | INFO | state-manager1 | DefaultStateManager | Invoking cis-sso-settings-builder 2023-11-01T00:57:05.945Z | INFO | state-manager1 | CertificateUtil | Creating anonymous SSO Admin Client for URI https://10.224.13.10/sso-adminserver/sdk/vsphere.local 2023-11-01T00:57:06.919Z | INFO | state-manager1 | DefaultStateManager | Invoking sts-builder 2023-11-01T00:57:08.196Z | ERROR | state-manager1 | SoapBindingImpl | Error communicating to the remote server https://10.224.13.10/sts/STSService/vsphere.local cohttp://m.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.1.5 found at cohttp://m.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117) at cohttp://m.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208) at cohttp://m.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130) at cohttp://m.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124) at cohttp://m.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121) at cohttp://m.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035) at cohttp://m.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004) at cohttp://m.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862) at cohttp://m.sun.xml.internal.ws.client.Stub.process(Stub.java:448) at cohttp://m.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:250) at cohttp://m.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289) at cohttp://m.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161) at cohttp://m.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114) at cohttp://m.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:784) at cohttp://m.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:714) at cohttp://m.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:473) at cohttp://m.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179) at cohttp://m.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77) at cohttp://m.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54) at cohttp://m.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354) at cohttp://m.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168) at cohttp://m.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.1.5 found at sun.security.ssl.Alert.createSSLException(Alert.java:131) at sun.security.ssl.TransportContext.fatal(TransportContext.java:370) at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:652) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:471) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:367) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:479) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:457) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1290) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1199) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:401) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:373) at sun.net.http://www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:587) at sun.net.http://www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.http://www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340) at sun.net.http://www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315) at sun.net.http://www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264) at cohttp://m.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104) ... 28 more Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.1.5 found at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:173) at sun.security.util.HostnameChecker.match(HostnameChecker.java:99) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:441) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:422) at sun.security.ssl.AbstractTrustManagerWrapper.checkAdditionalTrust(SSLContextImpl.java:1289) at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1256) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:636) ... 45 more |
※ 인증서에서 SAN(Subjet Alternate Name)이란?
5. STS 인증서는 정상적으로 갱신된 것으로 확인
## /var/log/vmware/sso/ssoAdminServer.log 분석
## 2023-11-01-T00:56분경 이전에 STS 인증서가 갱신된 것으로 확인
| [2023-10-31T01:26:12.837Z expiryCheckScheduler-1 opId= ERROR cohttp://m.vmware.identity.admin.server.impl.STSCertExpiryChecker] STS Signing Certificates have expired! [2023-10-31T01:26:12.837Z expiryCheckScheduler-1 opId= DEBUG cohttp://m.vmware.identity.admin.server.impl.STSCertExpiryChecker] logging STS signing certificate expiry event [2023-10-31T01:26:12.854Z expiryCheckScheduler-1 opId= ERROR opslogger] {"user":"n/a","client":"n/a","timestamp":"10/31/2023 01:26:12 UTC","description":"STS Signing Certificates h ave expired","eventSeverity":"WARNING","type":"cohttp://m.vmware.sso.STSCertExpiry"} [2023-11-01T00:44:42.456Z localhost-startStop-2 opId= INFO cohttp://m.vmware.identity.admin.server.impl.AdminApplicationListener] Heartbeat stopped [2023-11-01T00:46:45.171Z localhost-startStop-1 opId= DEBUG cohttp://m.vmware.identity.idp.IdpManagementImpl] catalina.base environment variable is /usr/lib/vmware-sso/vmware-sts [2023-11-01T00:46:45.266Z localhost-startStop-1 opId= DEBUG cohttp://m.vmware.identity.idp.IdpManagementImpl] IDP- store from server file:- MACHINE_SSL_CERT [2023-11-01T00:46:45.279Z localhost-startStop-1 opId= INFO cohttp://m.vmware.identity.admin.server.impl.AdminApplicationListener] Heartbeat started [2023-11-01T00:46:45.309Z expiryCheckScheduler-1 opId= ERROR cohttp://m.vmware.identity.admin.server.impl.STSCertExpiryChecker] STS Signing Certificates have expired! [2023-11-01T00:46:45.313Z expiryCheckScheduler-1 opId= DEBUG cohttp://m.vmware.identity.admin.server.impl.STSCertExpiryChecker] logging STS signing certificate expiry event [2023-11-01T00:46:45.553Z expiryCheckScheduler-1 opId= ERROR opslogger] {"user":"n/a","client":"n/a","timestamp":"11/01/2023 00:46:45 UTC","description":"STS Signing Certificates h ave expired","eventSeverity":"WARNING","type":"cohttp://m.vmware.sso.STSCertExpiry"} [2023-11-01T00:56:06.015Z localhost-startStop-1 opId= DEBUG cohttp://m.vmware.identity.idp.IdpManagementImpl] catalina.base environment variable is /usr/lib/vmware-sso/vmware-sts [2023-11-01T00:56:06.105Z localhost-startStop-1 opId= DEBUG cohttp://m.vmware.identity.idp.IdpManagementImpl] IDP- store from server file:- MACHINE_SSL_CERT [2023-11-01T00:56:06.122Z localhost-startStop-1 opId= INFO cohttp://m.vmware.identity.admin.server.impl.AdminApplicationListener] Heartbeat started [2023-11-01T00:56:06.142Z expiryCheckScheduler-1 opId= INFO cohttp://m.vmware.identity.admin.server.impl.STSCertExpiryChecker] STS Signing Certificates NOT within expiry warning threshold, expires in 729 days ### <-- !! |
6. 문제가 되는 로그가 "No subject alternative names" 이기 때문에 인증서의 SAN(Subject Alternative Name) 확인
6-1. STS 인증서
| # certificate [1] from './storage/vmware-vmon/signingcert.crt' Version: 3 Serial Number: UNKNOWN X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment Issuer: CN=VMware vCenter Server Appliance (1), DC=vsphere, DC=local, C=US, ST=California, O=photon-machine, OU=VMware Engineering Not Before: Nov 1 00:44:37 2023 GMT | Not After : Oct 31 00:44:37 2025 GMT | expires in '729' days Subject: CN=STS, C=DS, ST=VMware, L=VMware, O=VMware, OU=VMware Public-Key: (2048 bit) X509v3 Subject Alternative Name: email:email@acme.com, IP Address:192.168.1.5, DNS:192.168.1.5 ### <-- !! Signature Algorithm: sha256WithRSAEncryption <snip> |
6-2. MACHINE_SSL 인증서
| # openssl x509 -in ./etc/vmware-vpx/ssl/rui.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: f7:2f:b5:1b:63:19:a8:9e Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=photon-machine, OU=VMware Engineering Validity Not Before: Sep 6 01:48:22 2023 GMT Not After : Sep 6 01:48:22 2025 GMT Subject: CN=192.168.1.5, C=US <snip> Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: ### <-- !! IP Address:192.168.1.5 <snip> |
7. STS 인증서는 KB에서 제공하는 fixsts.sh 로 갱신하였고, 나머지 인증서는 사용자가 certificate-manager를 이용하여 갱신했기 때문에 STS 인증서는 용의 선상에서 제외
8. LAB 환경에서 MACHINE_SSL 인증서를 수동으로 갱신할 때, 필요한 정보 확인
## MACHINE_SSL_CERT.cfg 파일에서 IPAddress, Email, Hostname이 SAN 항목에 추가
## 해당 인증서를 가지고 교체 시도하고 서비스 재시작 시 정상적으로 완료
| root@photon-machine [ /certs ]# cp /usr/lib/vmware-vmca/share/config/certool.cfg MACHINE_SSL_CERT.cfg root@photon-machine [ /certs ]# cat MACHINE_SSL_CERT.cfg # # Template file for a CSR request # # Country is needed and has to be 2 characters Country = US Name = CA Organization = VMware OrgUnit = VMware Engineering State = California Locality = Palo Alto IPAddress = 192.168.1.5 Email = email@acme.com Hostname = 192.168.1.5 2023-11-02T17:18:05.508Z INFO certificate-manager Output : Certificate: Data: Version: 3 (0x2) Serial Number: e7:55:1c:86:e3:85:15:61 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=photon-machine, OU=VMware Engineering Validity Not Before: Nov 2 16:45:36 2023 GMT Not After : Nov 1 16:45:36 2025 GMT Subject: CN=192.168.1.5, C=US, ST=California, L=Palo Alto, O=VMware, OU=VMware Engineering Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c6:0f:1f:ee:aa:13:88:7d:03:db:50:2e:ab:01: d6:92:a5:87:3c:67:22:9f:e9:92:eb:a6:18:4e:eb: 7e:d7:ad:c7:f1:94:08:43:f5:85:8c:b9:da:f1:9d: cb:55:1a:ca:8a:06:59:7e:b2:79:e8:45:55:56:3e: d6:91:7d:2a:c5:3c:15:b7:79:f3:76:e9:5f:67:5f: 3e:f2:37:10:9a:0e:84:1e:df:ad:3c:ff:72:cd:e6: be:8e:2d:b3:3e:78:d4:a3:78:e3:51:01:5f:2f:81: 75:5c:fe:8f:13:1d:2a:91:74:67:8d:9a:46:ab:22: 63:d1:d6:bf:1a:aa:31:aa:73:1f:06:13:92:94:23: df:0f:ec:8f:07:00:35:22:0d:31:1f:51:f4:5c:1d: 5e:3d:a2:b0:e6:10:aa:90:53:79:da:ca:a7:5f:07: 03:6c:c6:32:77:30:c7:60:4e:14:be:cd:fb:47:0f: b0:f2:d7:fc:60:29:d3:53:85:3e:70:1a:30:e4:4a: c3:7d:ad:40:6f:a6:ce:78:58:d2:6b:4b:43:4f:78: d7:b3:da:74:3d:31:9f:3b:28:e7:96:6f:33:f9:63: 0c:12:d0:a4:46:77:5f:26:93:21:2e:24:09:76:90: f0:32:c3:03:89:1f:81:42:4f:8b:e8:fa:00:da:c0: 04:ab Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment X509v3 Subject Alternative Name: email:email@acme.com, IP Address:192.168.1.5, DNS:192.168.1.5 ### <-- !! <snip> # /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT # /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert MACHINE_SSL_CERT.crt --key MACHINE_SSL_CERT.priv # service-control --stop --all && service-control --start --all |
9. 문제 환경을 재현하기 위해서 고객과 동일하게 인증서에 SAN 항목을 구성
## IP Address에 실제 IP Address를 입력하고 hostname은 제거
| root@photon-machine [ /certs ]# cp /usr/lib/vmware-vmca/share/config/certool.cfg MACHINE_SSL_CERT.cfg root@photon-machine [ /certs ]# cat MACHINE_SSL_CERT.cfg # # Template file for a CSR request # # Country is needed and has to be 2 characters Country = US Name = CA Organization = VMware OrgUnit = VMware Engineering State = California Locality = Palo Alto IPAddress = 192.168.1.5 Email = email@acme.com root@photon-machine [ /certs ]# cat MACHINE_SSL_CERT.crt -----BEGIN CERTIFICATE----- MIIEAzCCAuugAwIBAgIJAN9jdKYw539MMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYD VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFzAVBgNV BAoMDnBob3Rvbi1tYWNoaW5lMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJpbmcw HhcNMjMxMTAzMDY1ODQ5WhcNMjUxMTAyMDY1ODQ5WjB6MRQwEgYDVQQDDAsxOTIu MTY4LjEuNTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExEjAQBgNV BAcMCVBhbG8gQWx0bzEPMA0GA1UECgwGVk13YXJlMRswGQYDVQQLDBJWTXdhcmUg RW5naW5lZXJpbmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDRePHT AxyeMCY8NMe5aznibhZ7K+db+h1SHPiz39/DIwSuFThLEa2upnzlhM5J+VTZmxss ZIO55k2YZqqL7lJvgTVM/SSap/V1QlJOXLm8+0g43BwhbyUci1WLhiR0rrWWwuIM 2j86FvE7u6Tqcal23hgpGfjB5FMNTvTltvLT9pxem3x0DyjpIvt0om4qDoruT7qf kdSIKEDRRtZOdEQh1YRmlBxHrDcn6o3GaHP5HiFT26lt1+2328O2gETwskD2jH3l 58WJmV2TRHqPmH14JWWHgTmrVmLhATtgTon4gueBomTrEuPmT7J8Iws6AuLPw0YN k7av3C0QbMk1WdXHAgMBAAGjcDBuMAsGA1UdDwQEAwIF4DAfBgNVHREEGDAWgQ5l bWFpbEBhY21lLmNvbYcEwKgBBTAdBgNVHQ4EFgQUupcmsRUu31zPfnZOjhhDeB5A FEswHwYDVR0jBBgwFoAUFqHavWXhrIU8TR7S/W+JiPilq8kwDQYJKoZIhvcNAQEL BQADggEBAKY0xmafGEETcWbEKj2FqSJMJo1VU73KavoZXMX3YtazAytwQpjYtrgF AA781crrrC5NnoXK4t+C9I40tnAQ+t7SLIpVXsNG3XUrEoqVDRZsgSn5ncgWEqsp y6IkNHKHWiD+n6mOlQ5FXYGK7SR8W1YvEPszQRhmm1+Mp2XHrs9unUrZJBVVrGUD GJMLJPjJfRP31qjGaO0pFmBTKc0ugcnFyEJPjxS93/3VqVTAMCxxkGZxJqmX1x2n myjdmLmjntFYyMzSBgg2FV0yjOsQhbJ5EzLpDL1KQl0pHQ3LhBlZxmJ13X8MizU3 gOsVr9nWRO9JFvC43/8YAIVaKjBhU1M= -----END CERTIFICATE----- |
10. 해당 인증서를 이용하여, VECS에 있는 기존 MACHINE_SSL 인증서와 교체하고 서비스 재시작
| ## 기존 MACHINE_SSL 인증서 백업 후 제거 # /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > backup_MACHINE_CERT.crt # /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > backup_MACHINE_CERT.ke # /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT ## 신규 MACHINE_SSL 인증서 적용 # /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert MACHINE_SSL_CERT.crt --key MACHINE_SSL_CERT.priv ## 서비스 재시작 # service-control --stop --all && service-control --start --all |
11. 서비스 재시작 시 고객과 동일한 오류 발생
| 2023-11-03T04:46:25.802Z Running command: ['/usr/bin/systemctl', 'set-environment', 'VMON_PROFILE=NONE'] 2023-11-03T04:46:25.806Z Done running command 2023-11-03T04:46:25.815Z Running command: ['/usr/bin/systemctl', 'daemon-reload'] 2023-11-03T04:46:25.911Z Done running command 2023-11-03T04:46:25.911Z Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-vmon.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true'] 2023-11-03T04:46:25.916Z Done running command 2023-11-03T04:46:27.183Z Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE'] 2023-11-03T04:46:27.187Z Done running command Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start vapi-endpoint, vpxd-svcs services. Error: Operation timed out ### <-- !! |
12. /var/log/vmware/vapi/endpoint/endpoint.log 로그 확인 시 동일한 메시지 확인
| Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.1.5 found at sun.security.ssl.Alert.createSSLException(Alert.java:131) at sun.security.ssl.TransportContext.fatal(TransportContext.java:370) at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:652) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:471) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:367) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:479) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:457) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:155) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1320) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1233) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:417) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:389) at sun.net.http://www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:558) at sun.net.http://www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201) at sun.net.http://www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1354) at sun.net.http://www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1329) at sun.net.http://www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264) at cohttp://m.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:104) ... 28 more Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.1.5 found at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:173) at sun.security.util.HostnameChecker.match(HostnameChecker.java:99) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:441) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:422) at sun.security.ssl.AbstractTrustManagerWrapper.checkAdditionalTrust(SSLContextImpl.java:1291) at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1258) at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:636) |
[Action Plan]
결과적으로 MACHINE_SSL 인증서 갱신 시, SAN 항목에 들어가는 정보를 모두 입력 필요
1. 수동으로 MACHINE_SSL 인증서를 갱신할 때는 다음과 같이 IP Address, Hostname에 모두 IP Address를 올바르게 입력하거나,
| root@photon-machine [ /certs ]# cat MACHINE_SSL_CERT.cfg # # Template file for a CSR request # # Country is needed and has to be 2 characters Country = US Name = CA Organization = VMware OrgUnit = VMware Engineering State = California Locality = Palo Alto IPAddress = 192.168.1.5 Email = email@acme.com Hostname = 192.168.1.5 |
2. /usr/lib/vmware-vmca/bin/certificate-manager 도구를 통해 MACHINE_SSL 인증서를 갱신할 때는
동일하게 configuration 파일에 들어갈 내용 중 IP Address와 Hostname을 모두 IP Address로 입력 필요
| root@photon-machine [ /usr/lib/vmware-vmca/bin ]# /usr/lib/vmware-vmca/bin/certificate-manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | | | *** Welcome to the vSphere 6.5 Certificate Manager *** | | | | -- Select Operation -- | | | | 1. Replace Machine SSL certificate with Custom Certificate | | | | 2. Replace VMCA Root certificate with Custom Signing | | Certificate and replace all Certificates | | | | 3. Replace Machine SSL certificate with VMCA Certificate | | | | 4. Regenerate a new VMCA Root Certificate and | | replace all certificates | | | | 5. Replace Solution user certificates with | | Custom Certificate | | | | 6. Replace Solution user certificates with VMCA certificates | | | | 7. Revert last performed operation by re-publishing old | | certificates | | | | 8. Reset all Certificates | |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _| Note : Use Ctrl-D to exit. Option[1 to 8]: 3 Please provide valid SSO and VC priviledged user credential to perform certificate operations. Enter username [Administrator@vsphere.local]: Enter password: certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : y Press Enter key to skip optional parameters or use Previous value. Enter proper value for 'Country' [Previous value : US] : Enter proper value for 'Name' [Previous value : CA] : Enter proper value for 'Organization' [Previous value : VMware] : Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] : Enter proper value for 'State' [Previous value : California] : Enter proper value for 'Locality' [Previous value : Palo Alto] : Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 192.168.1.5 Enter proper value for 'Email' [Previous value : email@acme.com] : Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : 192.168.1.5 |
'Compute' 카테고리의 다른 글
| RDM disks are seen as "File" type in vCenter UI. (0) | 2025.01.02 |
|---|---|
| How to access vCenter Database (0) | 2023.11.24 |
| ESXi host is disconnected due to firewall policy (0) | 2023.09.14 |
| Failure to pass pre-check for upgrade from 6.7 to 7.0 due to insufficient permission (0) | 2023.08.31 |
| ESXi host was unresponsive due to missing DNS records (0) | 2023.08.29 |
