오늘은 Multi-homed 환경에서 Static Routing Entry의 부재로 인하여 TCP 3-way handshake가 실패하는 현상에 대해서 살펴보겠습니다.
[문제 증상]
Backup Master 서버(A) 에서 ESXi Host의 Backup Interface(B)으로 5989 포트(WBEM, CIM) 연결 실패
| telnet B 5989 Connecting To B... |
[구성 환경]
다음과 같은 환경으로 인해 TCP 3-way handshake 실패
Source인 A와 Destination B는 서로 다른 Subnet
[진행 내역]
1. 5989 포트 Listening 상태 확인 : 정상
# localcli network ip connection list | grep 5989
2. ESXi Firewall 설정 확인 : 5989 포트에 대해서 별도 설정 없음
3. ESXi Firewall을 모두 내리고 테스트 : 동일하게 실패
# localcli network firewall unload
| telnet B 5989 Connecting To B... |
4. Backup Master 서버 Routing Table 확인 : ESXi Host의 Backup Network 대역인 B 에 대해서 Static Routing 설정
5. 문제 증상 재현 시, 네트워크 Packet 수집 : syn packet은 vmnic2(Backup Network)으로 들어오고, syn/ack packet은 vmnic6(Management Network)으로 나가는 것을 확인
# pktcap-uw --uplink vmnic2 --dir 2 -o - | tcpdump-uw -enr -
# pktcap-uw --uplink vmnic6 --dir 2 -o - | tcpdump-uw -enr -
6. ESXi Host Routing Table 확인 : syn/ack flag packet이 돌아가야 하는 목적지가 A 인데,
해당 목적지는 routing table에 따르면 default network으로 vmk0를 타가도록 되어 있음
# esxcli network ip route ipv4 list
7. 아래 명령어를 이용하여 ESXi Host의 Routing Table에 목적지 Network로 A 를 추가한 후 정상적으로 통신 가능
# esxcli network ip route ipv4 add --gateway <B IP 주소가 할당된 대역의 gateway 주소> --network <A 주소의 CIDR>
예제) A가 192.168.1.10/24 이고, B가 192.168.2.10/24이며 192.168.2.0/24의 gateway 주소가 192.168.2.1인 경우
# esxcli network ip route ipv4 add --gateway 192.168.2.1 --network 192.168.1.0/24
[테스트 환경]
테스트를 위해서 아래와 같이 환경을 구성
Guest OS Routing Table
ESXi Host의 VMKernel Interface인 10.10.10.10에 도달하기 위해서는 172.16.1.2 IP Address가 할당된 Interface를 통해서 172.16.1.1 Gateway로 전달
| C:\Users\Administrator>route print =========================================================================== Interface List 7...00 0c 29 ed 61 82 ......Intel(R) 82574L Gigabit Network Connection 5...00 0c 29 ed 61 8c ......Intel(R) 82574L Gigabit Network Connection #2 16...00 50 56 a1 bc c1 ......Intel(R) 82574L Gigabit Network Connection #3 1...........................Software Loopback Interface 1 =========================================================================== IPv4 Route Table =========================================================================== Active Routes: Network Destination Netmask Gateway Interface Metric 0.0.0.0 0.0.0.0 xxx.xxx.239.254 xxx.xxx.231.50 25 0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.2 281 10.10.10.0 255.255.255.0 172.16.1.1 172.16.1.2 28 ### <-- xxx.xxx.224.0 255.255.240.0 On-link 10.131.231.50 281 xxx.xxx.231.50 255.255.255.255 On-link 10.131.231.50 281 xxx.xxx.239.255 255.255.255.255 On-link 10.131.231.50 281 127.0.0.0 255.0.0.0 On-link 127.0.0.1 331 127.0.0.1 255.255.255.255 On-link 127.0.0.1 331 127.255.255.255 255.255.255.255 On-link 127.0.0.1 331 172.16.1.0 255.255.255.0 On-link 172.16.1.2 281 172.16.1.0 255.255.255.0 172.16.1.1 172.16.1.2 27 172.16.1.2 255.255.255.255 On-link 172.16.1.2 281 172.16.1.255 255.255.255.255 On-link 172.16.1.2 281 192.168.1.0 255.255.255.0 On-link 192.168.1.2 281 192.168.1.2 255.255.255.255 On-link 192.168.1.2 281 192.168.1.255 255.255.255.255 On-link 192.168.1.2 281 224.0.0.0 240.0.0.0 On-link 127.0.0.1 331 224.0.0.0 240.0.0.0 On-link xxx.xxx.231.50 281 224.0.0.0 240.0.0.0 On-link 192.168.1.2 281 224.0.0.0 240.0.0.0 On-link 172.16.1.2 281 255.255.255.255 255.255.255.255 On-link 127.0.0.1 331 255.255.255.255 255.255.255.255 On-link xxx.xxx.231.50 281 255.255.255.255 255.255.255.255 On-link 192.168.1.2 281 255.255.255.255 255.255.255.255 On-link 172.16.1.2 281 =========================================================================== Persistent Routes: Network Address Netmask Gateway Address Metric 0.0.0.0 0.0.0.0 192.168.1.1 Default =========================================================================== IPv6 Route Table =========================================================================== Active Routes: If Metric Network Destination Gateway 1 331 ::1/128 On-link 7 281 fe80::/64 On-link 5 281 fe80::/64 On-link 16 281 fe80::/64 On-link 5 281 fe80::99a:66e6:80c9:7b4c/128 On-link 16 281 fe80::106e:107c:8567:feda/128 On-link 7 281 fe80::25ad:1eee:c20e:3151/128 On-link 1 331 ff00::/8 On-link 7 281 ff00::/8 On-link 5 281 ff00::/8 On-link 16 281 ff00::/8 On-link =========================================================================== Persistent Routes: None |
ESXi Host의 Routing Table
현 상황에서는 ESXi Host에서 Guest OS의 172.16.1.2는 Default Network에 속하며, 이는 vmk0 Interface를 통해서 전달
| # esxcli network ip route ipv4 list Network Netmask Gateway Interface Source ------------ ------------- -------------- --------- ------ default 0.0.0.0 xxx.xxx.207.254 vmk0 DHCP ### <-- 10.10.10.0 255.255.255.0 0.0.0.0 vmk3 MANUAL xxx.xxx.192.0 255.255.240.0 0.0.0.0 vmk0 MANUAL 192.168.1.0 255.255.255.0 0.0.0.0 vmk1 MANUAL |
Guest OS에서 ESXi Host의 10.10.10.2 IP Address로 ping 테스트
| C:\> ping 10.10.10.10 Pinging 10.10.10.10 with 32 bytes of data: Reply from 10.10.10.10: bytes=32 time<1ms TTL=63 Reply from 10.10.10.10: bytes=32 time<1ms TTL=63 Reply from 10.10.10.10: bytes=32 time<1ms TTL=63 Reply from 10.10.10.10: bytes=32 time<1ms TTL=63 Ping statistics for 10.10.10.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms |
Guest OS에서 ESXi Host의 5989 포트로 xxx.xxx.192.12 IP Address와 10.10.10.2 IP Address를 이용하여 curl로 연결 테스트
| C:\Users\Administrator\Downloads\curl-8.0.1_8-win64-mingw\curl-8.0.1_8-win64-mingw\bin>curl.exe -sv https://xxx.xxx.192.12:5989 * Trying xxx.xxx.192.12:5989... * Connected to xxx.xxx.192.12 (xxx.xxx.192.12) port 5989 (#0) * ALPN: offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: C:\Users\Administrator\Downloads\curl-8.0.1_8-win64-mingw\curl-8.0.1_8-win64-mingw\bin\curl-ca-bundle.crt * CApath: none * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: self-signed certificate in certificate chain * Closing connection 0 C:\Users\Administrator\Downloads\curl-8.0.1_8-win64-mingw\curl-8.0.1_8-win64-mingw\bin>curl.exe -sv https://10.10.10.10:5989 * Trying 10.10.10.10:5989... * Connected to 10.10.10.10 (10.10.10.10) port 5989 (#0) * ALPN: offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: C:\Users\Administrator\Downloads\curl-8.0.1_8-win64-mingw\curl-8.0.1_8-win64-mingw\bin\curl-ca-bundle.crt * CApath: none * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, unknown CA (560): * SSL certificate problem: self-signed certificate in certificate chain * Closing connection 0 |
Guest OS에서 SSH 연결 테스트 시에, ESXi Host에서 Packet Capture
테스트 환경에서는 syn flag packet이 vmk3로 들어와서 syn/ack flag packet이 vmk0를 통해 Guest OS로 전달
| # pktcap-uw --vmk vmk3 --ip 10.10.10.10 --dir 2 -o - | tcpdump-uw -enr - The name of the vmk is vmk3. The session filter IP(src or dst) address is 10.10.10.10. pktcap: The output file is -. pktcap: No server port specifed, select 46227 as the port. pktcap: Local CID 2. pktcap: Listen on port 46227. pktcap: Main thread: 826863979328. pktcap: Dump Thread: 826864514816. pktcap: Recv Thread: 826865043200. pktcap: Accept... pktcap: Vsock connection from port 1043 cid 2. reading from file -, link-type EN10MB (Ethernet) 04:05:51.472216 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 66: 172.16.1.2.50114 > 10.10.10.10.22: Flags [SEW], seq 3095603784, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 04:05:51.472585 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50114 > 10.10.10.10.22: Flags [.], ack 1414428242, win 8212, length 0 04:05:51.486367 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 82: 172.16.1.2.50114 > 10.10.10.10.22: Flags [P.], seq 0:28, ack 1, win 8212, length 28 04:05:51.503551 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 1514: 172.16.1.2.50114 > 10.10.10.10.22: Flags [.], seq 28:1488, ack 22, win 8212, length 1460 04:05:51.503552 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 90: 172.16.1.2.50114 > 10.10.10.10.22: Flags [P.], seq 1488:1524, ack 22, win 8212, length 36 04:05:51.507823 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 134: 172.16.1.2.50114 > 10.10.10.10.22: Flags [P.], seq 1524:1604, ack 542, win 8210, length 80 04:05:51.522796 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50114 > 10.10.10.10.22: Flags [.], ack 1110, win 8208, length 0 04:05:51.525241 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 134: 172.16.1.2.50114 > 10.10.10.10.22: Flags [P.], seq 1604:1684, ack 1110, win 8208, length 80 04:05:51.535799 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50114 > 10.10.10.10.22: Flags [.], ack 1174, win 8207, length 0 # pktcap-uw --vmk vmk0 --ip 10.10.10.10 --dir 2 -o - | tcpdump-uw -enr - The name of the vmk is vmk0. The session filter IP(src or dst) address is 10.10.10.10. pktcap: The output file is -. pktcap: No server port specifed, select 46222 as the port. pktcap: Local CID 2. pktcap: Listen on port 46222. pktcap: Main thread: 82114788160. pktcap: Dump Thread: 82115323648. reading from file -, link-type EN10MB (Ethernet) pktcap: Recv Thread: 82115852032. pktcap: Accept... pktcap: Vsock connection from port 1042 cid 2. 04:05:51.472267 78:ac:44:12:8f:94 > 00:1c:b0:21:f4:80, ethertype IPv4 (0x0800), length 66: 10.10.10.10.22 > 172.16.1.2.50114: Flags [S.E], seq 1414428241, ack 3095603785, win 65535, options [mss 1460,nop,wscale 9,sackOK,eol], length 0 04:05:51.499924 78:ac:44:12:8f:94 > 00:1c:b0:21:f4:80, ethertype IPv4 (0x0800), length 75: 10.10.10.10.22 > 172.16.1.2.50114: Flags [P.], seq 1:22, ack 29, win 128, length 21 04:05:51.503585 78:ac:44:12:8f:94 > 00:1c:b0:21:f4:80, ethertype IPv4 (0x0800), length 574: 10.10.10.10.22 > 172.16.1.2.50114: Flags [P.], seq 22:542, ack 1525, win 125, length 520 04:05:51.511728 78:ac:44:12:8f:94 > 00:1c:b0:21:f4:80, ethertype IPv4 (0x0800), length 622: 10.10.10.10.22 > 172.16.1.2.50114: Flags [P.], seq 542:1110, ack 1605, win 128, length 568 04:05:51.525280 78:ac:44:12:8f:94 > 00:1c:b0:21:f4:80, ethertype IPv4 (0x0800), length 118: 10.10.10.10.22 > 172.16.1.2.50114: Flags [P.], seq 1110:1174, ack 1685, win 128, length 64 |
ESXi Host의 Routing Table 변경
| # esxcli network ip route ipv4 add --gateway 10.10.10.1 --network 172.16.1.0/24 # esxcli network ip route ipv4 list Network Netmask Gateway Interface Source ------------ ------------- -------------- --------- ------ default 0.0.0.0 10.131.207.254 vmk0 DHCP 10.10.10.0 255.255.255.0 0.0.0.0 vmk3 MANUAL 10.131.192.0 255.255.240.0 0.0.0.0 vmk0 MANUAL 172.16.1.0 255.255.255.0 10.10.10.1 vmk3 MANUAL 192.168.1.0 255.255.255.0 0.0.0.0 vmk1 MANUAL |
ESXi Host의 Routing Table 변경 후, Guest OS에서 SSH 연결 테스트 시에, ESXi Host에서 Packet Capture
Routing Table 변경 후에는 syn flag packet이 vmk3로 들어와서 동일하게 syn/ack flag packet도 vmk3를 통해 Guest OS에게 전달
| # pktcap-uw --vmk vmk0 --ip 10.10.10.10 --dir 2 -o - | tcpdump-uw -enr - The name of the vmk is vmk0. The session filter IP(src or dst) address is 10.10.10.10. pktcap: The output file is -. pktcap: No server port specifed, select 46283 as the port. pktcap: Local CID 2. pktcap: Listen on port 46283. pktcap: Main thread: 385514863424. pktcap: Recv Thread: 385515927296. pktcap: Accept... pktcap: Vsock connection from port 1045 cid 2. pktcap: Dump Thread: 385515398912. reading from file -, link-type EN10MB (Ethernet) # pktcap-uw --vmk vmk3 --ip 10.10.10.10 --dir 2 -o - | tcpdump-uw -enr - The name of the vmk is vmk3. The session filter IP(src or dst) address is 10.10.10.10. pktcap: The output file is -. pktcap: No server port specifed, select 46288 as the port. pktcap: Local CID 2. pktcap: Listen on port 46288. pktcap: Main thread: 826075159360. pktcap: Dump Thread: 826075694848. reading from file -, link-type EN10MB (Ethernet) pktcap: Recv Thread: 826076223232. pktcap: Accept... pktcap: Vsock connection from port 1046 cid 2. 04:11:30.682429 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 66: 172.16.1.2.50148 > 10.10.10.10.22: Flags [SEW], seq 2716946564, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 04:11:30.682500 00:50:56:64:f9:96 > 00:50:56:a1:35:5a, ethertype IPv4 (0x0800), length 66: 10.10.10.10.22 > 172.16.1.2.50148: Flags [S.E], seq 3415070781, ack 2716946565, win 65535, options [mss 1460,nop,wscale 9,sackOK,eol], length 0 04:11:30.682715 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50148 > 10.10.10.10.22: Flags [.], ack 1, win 8212, length 0 04:11:30.696012 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 82: 172.16.1.2.50148 > 10.10.10.10.22: Flags [P.], seq 1:29, ack 1, win 8212, length 28 04:11:30.702834 00:50:56:64:f9:96 > 00:50:56:a1:35:5a, ethertype IPv4 (0x0800), length 75: 10.10.10.10.22 > 172.16.1.2.50148: Flags [P.], seq 1:22, ack 29, win 128, length 21 04:11:30.705980 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 1514: 172.16.1.2.50148 > 10.10.10.10.22: Flags [.], seq 29:1489, ack 22, win 8212, length 1460 04:11:30.705981 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 90: 172.16.1.2.50148 > 10.10.10.10.22: Flags [P.], seq 1489:1525, ack 22, win 8212, length 36 04:11:30.706005 00:50:56:64:f9:96 > 00:50:56:a1:35:5a, ethertype IPv4 (0x0800), length 574: 10.10.10.10.22 > 172.16.1.2.50148: Flags [P.], seq 22:542, ack 1525, win 125, length 520 04:11:30.710455 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 134: 172.16.1.2.50148 > 10.10.10.10.22: Flags [P.], seq 1525:1605, ack 542, win 8210, length 80 04:11:30.714379 00:50:56:64:f9:96 > 00:50:56:a1:35:5a, ethertype IPv4 (0x0800), length 622: 10.10.10.10.22 > 172.16.1.2.50148: Flags [P.], seq 542:1110, ack 1605, win 128, length 568 04:11:30.724556 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50148 > 10.10.10.10.22: Flags [.], ack 1110, win 8208, length 0 04:11:30.726030 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 134: 172.16.1.2.50148 > 10.10.10.10.22: Flags [P.], seq 1605:1685, ack 1110, win 8208, length 80 04:11:30.726061 00:50:56:64:f9:96 > 00:50:56:a1:35:5a, ethertype IPv4 (0x0800), length 118: 10.10.10.10.22 > 172.16.1.2.50148: Flags [P.], seq 1110:1174, ack 1685, win 128, length 64 04:11:30.736557 00:50:56:a1:35:5a > 00:50:56:64:f9:96, ethertype IPv4 (0x0800), length 60: 172.16.1.2.50148 > 10.10.10.10.22: Flags [.], ack 1174, win 8207, length 0 |
'Networking' 카테고리의 다른 글
| [Socket Programming #2] CLOSE_WAIT (0) | 2023.05.19 |
|---|---|
| [Socket Programming #1] Server/Client based on TCP (0) | 2023.05.18 |
| Driver/Firmware Check - Network Adapter (0) | 2023.03.18 |
| VMKernel Network Threads (0) | 2023.02.23 |
| Address Resolution Protocol(ARP) (0) | 2022.08.11 |
