9. ESXi Host에서 vmknic과 vmnic MTU 확인
| # esxcfg-vmknic -l Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack vmk0 External Mgmt IPv4 10.131.192.25 255.255.240.0 10.131.207.255 78:ac:44:4c:9f:d0 1500 65535 true DHCP defaultTcpipStack vmk1 Internal Mgmt IPv4 192.168.1.12 255.255.255.0 192.168.1.255 00:50:56:6e:c1:e4 1500 65535 true STATIC defaultTcpipStack # esxcli network nic list Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description ------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- ----------- vmnic0 0000:19:00.0 ixgben Up Up 10000 Full 78:ac:44:4c:9f:d0 9000 Intel(R) Ethernet Controller X550 vmnic1 0000:19:00.1 ixgben Up Up 10000 Full 78:ac:44:4c:9f:d1 9000 Intel(R) Ethernet Controller X550 vmnic2 0000:1a:00.0 ixgben Up Up 10000 Full 78:ac:44:4c:9f:d2 1500 Intel(R) Ethernet Controller X550 vmnic3 0000:1a:00.1 ixgben Up Up 10000 Full 78:ac:44:4c:9f:d3 1500 Intel(R) Ethernet Controller X550 |
10. esxtop에서 Edge PPS 확인
esxtop 실행 후 'n' 키 입력
vmnic 별 packet rate 확인
| Esxtop version 7.0.2 Secure mode Off Esxtop: top for ESX These single-character commands are available: ^L - redraw screen space - update display h or ? - help; show this text q - quit Interactive commands are: fF Add or remove fields oO Change the order of displayed fields s Set the delay in seconds between updates # Set the number of instances to display W Write configuration file ~/.esxtop60rc k Kill a world e Expand/Rollup Cpu Statistics V View only VM instances L Change the length of the NAME field l Limit display to a single group Sort by: U:%USED R:%RDY N:GID Switch display: c:cpu i:interrupt m:memory n:network ### <-- !! d:disk adapter u:disk device v:disk VM p:power mgmt r:rdma device x:vsan 4:04:39pm up 7 days 55 min, 1282 worlds, 4 VMs, 36 vCPUs; CPU load average: 0.08, 0.09, 0.09 PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX % 67108870 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108872 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108874 Shadow of vmnic1 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108880 vmk0 vmnic0 vSwitch0 11.44 0.01 152.00 7.63 0.01 172.00 0.00 67108881 vmk1 vmnic0 vSwitch0 68.66 0.17 320.00 11.44 0.01 79.00 0.00 67108882 2101272:edge-node-01 vmnic0 vSwitch0 11.44 0.01 82.00 144.96 0.22 201.00 0.00 67108883 2101272:edge-node-01 vmnic0 vSwitch0 3.81 0.00 66.00 152.59 0.23 195.00 0.00 67108884 2101272:edge-node-01 vmnic0 vSwitch0 3.81 0.00 66.00 0.00 0.00 0.00 0.00 67108885 2101777:edge-node-02 vmnic0 vSwitch0 11.44 0.01 82.00 144.96 0.22 201.00 0.00 67108886 2101777:edge-node-02 vmnic0 vSwitch0 11.44 0.01 82.00 144.96 0.22 201.00 0.00 67108887 2101777:edge-node-02 vmnic0 vSwitch0 0.00 0.00 0.00 3.81 0.00 66.00 0.00 67108888 2106815:nsx-mgr01 vmnic0 vSwitch0 0.00 0.00 0.00 3.81 0.00 60.00 0.00 67108889 2106853:vyos01 vmnic0 vSwitch0 7.63 0.00 66.00 7.63 0.00 66.00 0.00 67108890 2106853:vyos01 vmnic0 vSwitch0 3.81 0.00 66.00 7.63 0.00 66.00 0.00 67108891 2106853:vyos01 vmnic0 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108892 2106853:vyos01 vmnic0 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108893 2106853:vyos01 vmnic0 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100663307 Management n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100663309 Shadow of vmnic2 n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100663311 Shadow of vmnic3 n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100663326 2106853:vyos01 vmnic2 vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2214592519 vmnic0 - vSwitch0 80.11 0.18 296.00 22.89 0.02 97.00 0.00 2214592521 vmnic1 - vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2248146956 vmnic2 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2248146958 vmnic3 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
11. vmnic 별 PollWorld Thread 사용률 확인
만약, Core 하나만 사용률이 높은 경우에는 vmnic에 multi RSS 활성화가 필요
multi RSS 활성화는 NIC 제조사별로 다르기 때문에 확인 필요
예를 들어, broadcom의 경우에는 "esxcli system module parameters set -m bnxtnet -p enable_multi_rss=1" 형식으로 multi RSS 활성화 가능
| # net-stats -A -t WwQqih |grep used |grep pollWorld "2098121": {"id": 2098121, "used": 0.07, "ready": 0.01, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[00]-0x430093c2e8c0", "2098122": {"id": 2098122, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[01]-0x430093c2f3c0", "2098123": {"id": 2098123, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[02]-0x430093c2fec0", "2098124": {"id": 2098124, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[03]-0x430093c309c0", "2098125": {"id": 2098125, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[04]-0x430093c314c0", "2098126": {"id": 2098126, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[05]-0x430093c31fc0", "2098127": {"id": 2098127, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[06]-0x430093c32ac0", "2098128": {"id": 2098128, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[07]-0x430093c335c0", "2098129": {"id": 2098129, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[08]-0x430093c340c0", "2098130": {"id": 2098130, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[09]-0x430093c34bc0", "2098131": {"id": 2098131, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[10]-0x430093c356c0", "2098132": {"id": 2098132, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic0-pollWorldRxTx[11]-0x430093c361c0", "2098135": {"id": 2098135, "used": 0.03, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[00]-0x430093c36cc0", "2098136": {"id": 2098136, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[01]-0x430093c377c0", "2098137": {"id": 2098137, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[02]-0x430093c382c0", "2098138": {"id": 2098138, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[03]-0x430093c38dc0", "2098139": {"id": 2098139, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[04]-0x430093c398c0", "2098140": {"id": 2098140, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[05]-0x430093c3a3c0", "2098141": {"id": 2098141, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[06]-0x430093c3aec0", "2098142": {"id": 2098142, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[07]-0x430093c3b9c0", "2098143": {"id": 2098143, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[08]-0x430093c3c4c0", "2098144": {"id": 2098144, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[09]-0x430093c3cfc0", "2098145": {"id": 2098145, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[10]-0x430093c3dac0", "2098146": {"id": 2098146, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic1-pollWorldRxTx[11]-0x430093c3e5c0", "2098150": {"id": 2098150, "used": 0.01, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[00]-0x430093c3f0c0", "2098151": {"id": 2098151, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[01]-0x430093c3fbc0", "2098152": {"id": 2098152, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[02]-0x430093c406c0", "2098153": {"id": 2098153, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[03]-0x430093c411c0", "2098154": {"id": 2098154, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[04]-0x430093c41cc0", "2098155": {"id": 2098155, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[05]-0x430093c427c0", "2098156": {"id": 2098156, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[06]-0x430093c432c0", "2098157": {"id": 2098157, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[07]-0x430093c43dc0", "2098158": {"id": 2098158, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[08]-0x430093c448c0", "2098159": {"id": 2098159, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[09]-0x430093c453c0", "2098160": {"id": 2098160, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[10]-0x430093c45ec0", "2098161": {"id": 2098161, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic2-pollWorldRxTx[11]-0x430093c469c0", "2098165": {"id": 2098165, "used": 0.01, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[00]-0x430093c474c0", "2098166": {"id": 2098166, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[01]-0x430093c47fc0", "2098167": {"id": 2098167, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[02]-0x430093c48ac0", "2098168": {"id": 2098168, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[03]-0x430093c495c0", "2098169": {"id": 2098169, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[04]-0x430093c4a0c0", "2098170": {"id": 2098170, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[05]-0x430093c4abc0", "2098171": {"id": 2098171, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[06]-0x430093c4b6c0", "2098172": {"id": 2098172, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[07]-0x430093c4c1c0", "2098173": {"id": 2098173, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[08]-0x430093c4ccc0", "2098174": {"id": 2098174, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[09]-0x430093c4d7c0", "2098175": {"id": 2098175, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[10]-0x430093c4e2c0", "2098176": {"id": 2098176, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name": "vmnic3-pollWorldRxTx[11]-0x430093c4edc0", |
12. Edge Ring Buffer와 통계 확인
Edge VM인 경우 Edge VM이 위치한 ESXi에서 Edge VM의 Port ID 확인
| # net-stats -l |grep edge 67108882 5 9 vSwitch0 00:50:56:a1:86:fa edge-node-01 67108883 5 9 vSwitch0 00:50:56:a1:4f:a5 edge-node-01 67108884 5 9 vSwitch0 00:50:56:a1:e5:27 edge-node-01 67108885 5 9 vSwitch0 00:50:56:a1:76:8e edge-node-02 67108886 5 9 vSwitch0 00:50:56:a1:16:3f edge-node-02 67108887 5 9 vSwitch0 00:50:56:a1:fe:d9 edge-node-02 |
VSISH 도구로 각 Port의 RX Summary와 Ring Buffer 설정 값 확인
| # vsish -e get /net/portsets/vSwitch0/ports/67108882/vmxnet3/rxSummary stats of a vmxnet3 vNIC rx queue { LRO pkts rx ok:0 LRO bytes rx ok:0 pkts rx ok:85819619 bytes rx ok:26270239653 unicast pkts rx ok:68149246 unicast bytes rx ok:25048132239 multicast pkts rx ok:3610018 multicast bytes rx ok:255476580 broadcast pkts rx ok:14060355 broadcast bytes rx ok:966630834 running out of buffers:0 pkts receive error:0 1st ring size:2048 2nd ring size:2048 # of times the 1st ring is full:0 # of times the 2nd ring is full:0 fail to map a rx buffer:0 request to page in a buffer:0 # of times rx queue is stopped:0 failed when copying into the guest buffer:0 # of pkts dropped due to large hdrs:0 # of pkts dropped due to max number of SG limits:0 pkts rx via data ring ok:0 bytes rx via data ring ok:0 Whether rx burst queuing is enabled:0 current backend burst queue length:0 maximum backend burst queue length so far:0 aggregate number of times packets are requeued:0 aggregate number of times packets are dropped by PktAgingList:0 # of pkts dropped due to large inner (encap) hdrs:0 number of times packets are dropped by burst queue:0 number of times packets are dropped by rx try lock queueing:0 number of packets delivered by burst queue:0 number of packets dropped by packet steering:0 number of memory region lookup pass in Rx.:0 number of packets dropped due to pkt length exceeds vNic mtu:0 number of packets dropped due to pkt truncation:0 } |
13. pnicFeatures 설정 확인
vNIC은 Packet을 수신하기 위한 queue를 기본으로 1개만 사용
이 queue가 1개인 경우 때문에, bottleneck이 발생할 수 있어 더 많은 queue 할당이 필요
queue를 증가시키면 Throughput 향상으로 이어짐
pnicFeatures 기능을 사용하려면 물리 ESXi Host에 multi RSS 기능이 활성화 되어 있어야 함
| # find / -iname edge-node-01.vmx /vmfs/volumes/64fd05ed-f98a694a-a460-78ac444c9fd0/edge-node-01/edge-node-01.vmx # grep pnicFeatures /vmfs/volumes/64fd05ed-f98a694a-a460-78ac444c9fd0/edge-node-01/edge-node-01.vmx ethernet0.pnicFeatures = "4" ethernet1.pnicFeatures = "4" ethernet2.pnicFeatures = "4" ethernet3.pnicFeatures = "4" ethernet4.pnicFeatures = "4" |
14. ctxPerDev 설정 확인
Edge VM의 Tx queue를 3개까지 증설 가능
| # grep ctxPerDev /vmfs/volumes/64fd05ed-f98a694a-a460-78ac444c9fd0/edge-node-01/edge-node-01.vmx ethernet0.ctxPerDev = "3" ethernet1.ctxPerDev = "3" ethernet2.ctxPerDev = "3" ethernet3.ctxPerDev = "3" ethernet4.ctxPerDev = "3" |
15. Edge VM이 DFW Rule에서 제외되었는지 확인
아래 명령어 결과 값이 없어야 정상
| # summarize-dvfilter | grep -i edge -A5 |
16. pNic multi RSS 확인
수신되는 Network Packet을 여러 Processor에서 처리되도록 하기 위해서 RSS 기능 활성화 필요
Uplink 수준에서 RSS 기능을 활성화 하기 위해서는 NIC 제조사 가이드 필요
아래 예시는 bnxtnet 드라이버의 경우에만 해당
| # esxcli system module parameters list -m bnxtnet | grep enable_multi_rss enable_multi_rss bool Enable Multi-RSS support. 0: disable, 1: enable. [Default: 0] num_rss_pools uint Max Number of RSS pools. [Default: 4]. This will take effect when enable_multi_rss is set # esxcli system module parameters set -m bnxtnet --parameter-string "enable_multi_rss=1" |
17. pNic의 Multi Tx Queue 확인
기본적으로 pNic에는 하나의 Tx queue만 활성화
pNic에 Multi Tx Queue를 설정하기 위해서는 아래 명령어 실행 필요
| # esxcli system settings advanced set -i 4 -o /Net/NetSchedHClkMaxHwQueue |
vNic 까지 Multi Tx Queue를 사용하도록 하기 위해서는 아래 설정 추가로 필요
| # esxcli system settings advanced set -i 1 -o /Net/NetSchedHClkMQ # esxcli system settings advanced set -i 1 -o /Net/NetSchedHClkVnicMQ |
'Networking' 카테고리의 다른 글
| [NSX] BGP Basic Check (0) | 2024.01.20 |
|---|---|
| [NSX] Logical Routing (1) | 2024.01.15 |
| [NSX] Basic Check #2 (0) | 2023.12.11 |
| [NSX] Basic Check #1 (1) | 2023.12.11 |
| Load Balancer Packet 수집 (0) | 2023.11.05 |
