Aria Operations에서 특정 Tier-1 Gateway의 "Received Packets dropped" Count의 누적 값이 비정상적으로 증가하는 문제가 있어, 이에 대한 확인해 본 과정을 공유 합니다.
[Symptom]
아래 캡쳐 화면을 보면, 전체 통계치가 1억을 넘는데, 개별 인터페이스 통계치의 합은 3600개 정도
[Troubleshooting Notes]
먼저 각 개별 Metric에 대한 내용을 확인하기 위해서 내부 Lab 환경을 이용하여 Metric별 API 확인
1. 내부 LAB 환경에서 Aria Operation UI 확인
Router Interface Statistics|Down Link|Received Packets dropped|Cumulative
Router Interface Statistics|Router Link|Received Packets dropped|Cumulative
Router Interface Statistics|Up Link|Received Packets dropped|Cumulative
Statistics|Received Packets dropped|Cumulative
2. 각 Metric 별 NSX API 확인
2-1. 전체 Logical Port 정보 조회
| GET https://nsx-mgr01.contoso.com/api/v1/logical-router-ports { "results": [ { "subnets": [ { "ip_addresses": [ "192.168.12.11" ], "prefix_length": 24 } ], "edge_cluster_member_index": [ 0 ], "linked_logical_switch_port_id": { "target_id": "3f0de8d1-d706-4eab-bef7-af8dd836c10c", "target_display_name": "Tier0-01-edge01-uplink01-ulp", "target_type": "LogicalPort", "is_valid": true }, "urpf_mode": "STRICT", "mac_address": "00:50:56:a6:8c:14", "mode": "UNTAGGED", "pim_config": { "enabled": false, "hello_interval": 30, "hold_interval": 0 }, "resource_type": "LogicalRouterUpLinkPort", "id": "9afaf8e0-8f00-4204-bce5-f87a7d9f2946", "display_name": "edge01-uplink01", "description": "Logical router port for interface /infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge01-uplink01", "tags": [ { "scope": "policyPath", "tag": "/infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge01-uplink01" } ], "logical_router_id": "4f22c0b3-4a2f-4840-a8d2-cf8c797c087b", "_create_time": 1705559429240, "_create_user": "nsx_policy", "_last_modified_time": 1705559429240, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "192.168.13.11" ], "prefix_length": 24 } ], "edge_cluster_member_index": [ 0 ], "linked_logical_switch_port_id": { "target_id": "4e65dc04-96bd-4925-97a3-b3d84deaae02", "target_display_name": "Tier0-01-edge01-uplink02-ulp", "target_type": "LogicalPort", "is_valid": true }, "urpf_mode": "STRICT", "mac_address": "00:50:56:a6:ab:70", "mode": "UNTAGGED", "pim_config": { "enabled": false, "hello_interval": 30, "hold_interval": 0 }, "resource_type": "LogicalRouterUpLinkPort", "id": "c9755577-209f-4850-8510-65c63d8d388c", "display_name": "edge01-uplink02", "description": "Logical router port for interface /infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge01-uplink02", "tags": [ { "scope": "policyPath", "tag": "/infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge01-uplink02" } ], "logical_router_id": "4f22c0b3-4a2f-4840-a8d2-cf8c797c087b", "_create_time": 1705559446303, "_create_user": "nsx_policy", "_last_modified_time": 1705559446303, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "192.168.12.12" ], "prefix_length": 24 } ], "edge_cluster_member_index": [ 1 ], "linked_logical_switch_port_id": { "target_id": "c7a1a6bc-34d1-4a93-ac9f-9d87c3653b22", "target_display_name": "Tier0-01-edge02-uplink01-ulp", "target_type": "LogicalPort", "is_valid": true }, "urpf_mode": "STRICT", "mac_address": "00:50:56:a6:49:60", "mode": "UNTAGGED", "pim_config": { "enabled": false, "hello_interval": 30, "hold_interval": 0 }, "resource_type": "LogicalRouterUpLinkPort", "id": "c63a7610-1ed0-4183-b72a-6a3bf281b2e5", "display_name": "edge02-uplink01", "description": "Logical router port for interface /infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge02-uplink01", "tags": [ { "scope": "policyPath", "tag": "/infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge02-uplink01" } ], "logical_router_id": "4f22c0b3-4a2f-4840-a8d2-cf8c797c087b", "_create_time": 1705559467335, "_create_user": "nsx_policy", "_last_modified_time": 1705559467335, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "192.168.13.12" ], "prefix_length": 24 } ], "edge_cluster_member_index": [ 1 ], "linked_logical_switch_port_id": { "target_id": "124d231b-dec9-49e1-b650-beb9439f99b0", "target_display_name": "Tier0-01-edge02-uplink02-ulp", "target_type": "LogicalPort", "is_valid": true }, "urpf_mode": "STRICT", "mac_address": "00:50:56:a6:19:45", "mode": "UNTAGGED", "pim_config": { "enabled": false, "hello_interval": 30, "hold_interval": 0 }, "resource_type": "LogicalRouterUpLinkPort", "id": "80ef0473-06e0-4110-8e41-6879f4d6d219", "display_name": "edge02-uplink02", "description": "Logical router port for interface /infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge02-uplink02", "tags": [ { "scope": "policyPath", "tag": "/infra/tier-0s/Tier0-01/locale-services/default/interfaces/edge02-uplink02" } ], "logical_router_id": "4f22c0b3-4a2f-4840-a8d2-cf8c797c087b", "_create_time": 1705559488294, "_create_user": "nsx_policy", "_last_modified_time": 1705559488294, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "100.64.120.0" ], "prefix_length": 31 }, { "ip_addresses": [ "fc66:613:6598:8000::1" ], "prefix_length": 64 }, { "ip_addresses": [ "fe80::050:56ff:fe56:4452" ], "prefix_length": 64 } ], "linked_logical_router_port_id": "c8f68ee2-6b45-4017-ba3c-1d676b05f4a6", "mac_address": "02:50:56:56:44:52", "resource_type": "LogicalRouterLinkPortOnTIER0", "id": "8866ff61-9a15-5e60-953b-d458fd12a80a", "display_name": "Tier0-01-tier1-01-t0_lrp", "description": "LogicalRouterLinkPortOnTIER0 on provider logical router Tier0-01-t0 to connect to network logical router tier1-01-t1", "logical_router_id": "4f22c0b3-4a2f-4840-a8d2-cf8c797c087b", "_create_time": 1705559911537, "_create_user": "nsx_policy", "_last_modified_time": 1705559911537, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "100.64.120.1" ], "prefix_length": 31 }, { "ip_addresses": [ "fc66:613:6598:8000::2" ], "prefix_length": 64 }, { "ip_addresses": [ "fe80::050:56ff:fe56:4455" ], "prefix_length": 64 } ], "edge_cluster_member_index": [ 1, 0 ], "linked_logical_router_port_id": { "target_id": "8866ff61-9a15-5e60-953b-d458fd12a80a", "target_display_name": "Tier0-01-tier1-01-t0_lrp", "target_type": "LogicalRouterLinkPortOnTIER0", "is_valid": true }, "mac_address": "02:50:56:56:44:55", "resource_type": "LogicalRouterLinkPortOnTIER1", "id": "c8f68ee2-6b45-4017-ba3c-1d676b05f4a6", "display_name": "Tier0-01-tier1-01-t1_lrp", "description": "LogicalRouterLinkPortOnTIER1 on network logical router tier1-01-t1 to connect to provider logical router Tier0-01-t0", "logical_router_id": "5bc895a7-8cc3-4332-9b1a-abe3bd64fa22", "_create_time": 1705559911853, "_create_user": "nsx_policy", "_last_modified_time": 1705559911853, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 }, { "subnets": [ { "ip_addresses": [ "172.31.1.1" ], "prefix_length": 24 } ], "linked_logical_switch_port_id": { "target_id": "44c4d076-1e45-4632-bbf6-c458e1b3c6dc", "target_display_name": "infra-overlay-seg-3101-lp", "target_type": "LogicalPort", "is_valid": true }, "urpf_mode": "STRICT", "mac_address": "02:50:56:56:44:52", "enable_multicast": true, "resource_type": "LogicalRouterDownLinkPort", "id": "a62f9b69-c532-44a8-89a0-3e42c6292d94", "display_name": "infra-overlay-seg-3101-dlrp", "description": "Logical port on logical router /infra/realized-state/enforcement-points/default/tier-1-logical-routers/tier1-01-t1 to connect to segment logical switch infra-overlay-seg-3101-ls", "tags": [ { "scope": "policyPath", "tag": "/infra/segments/overlay-seg-3101" } ], "logical_router_id": "5bc895a7-8cc3-4332-9b1a-abe3bd64fa22", "_create_time": 1705560119045, "_create_user": "nsx_policy", "_last_modified_time": 1710225814240, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 3 }, { "linked_logical_switch_port_id": { "target_id": "965e3e8f-1cf0-4fcb-ab8d-0f810044884e" }, "subnets": [ { "ip_addresses": [ "172.31.1.254" ], "prefix_length": 24 } ], "urpf_mode": "STRICT", "enable_netx": false, "resource_type": "LogicalRouterCentralizedServicePort", "id": "1d3aab19-850a-44df-9be6-ffc8f8030edc", "display_name": "t1-one-arm-default-16c5b577-ca62-4256-8d39-ed0761f2d2b4-svclrp", "description": "Logical router port for interface /infra/tier-1s/one-arm/locale-services/default/interfaces/16c5b577-ca62-4256-8d39-ed0761f2d2b4", "tags": [ { "scope": "policyPath", "tag": "/infra/tier-1s/one-arm/locale-services/default/interfaces/16c5b577-ca62-4256-8d39-ed0761f2d2b4" } ], "logical_router_id": "22f35c65-1975-42a0-af39-ce963d60553c", "_create_time": 1705571684329, "_create_user": "nsx_policy", "_last_modified_time": 1705571684329, "_last_modified_user": "nsx_policy", "_system_owned": false, "_protection": "REQUIRE_OVERRIDE", "_revision": 0 } ], "result_count": 8 } "id": "9afaf8e0-8f00-4204-bce5-f87a7d9f2946", "display_name": "edge01-uplink01", "id": "c9755577-209f-4850-8510-65c63d8d388c", "display_name": "edge01-uplink02", "id": "c63a7610-1ed0-4183-b72a-6a3bf281b2e5", "display_name": "edge02-uplink01", "id": "80ef0473-06e0-4110-8e41-6879f4d6d219", "display_name": "edge02-uplink02", "id": "8866ff61-9a15-5e60-953b-d458fd12a80a", "display_name": "Tier0-01-tier1-01-t0_lrp", "id": "c8f68ee2-6b45-4017-ba3c-1d676b05f4a6", "display_name": "Tier0-01-tier1-01-t1_lrp", "id": "a62f9b69-c532-44a8-89a0-3e42c6292d94", "display_name": "infra-overlay-seg-3101-dlrp", "id": "1d3aab19-850a-44df-9be6-ffc8f8030edc", "display_name": "t1-one-arm-default-16c5b577-ca62-4256-8d39-ed0761f2d2b4-svclrp", |
2-2. 2-1에서 확인한 전체 Logical Router Ports 중 확인이 필요한 Tier-1 Logical Router Port만 API를 이용하여 누적 통계치 확인
Router Interface Statistics|Down Link|Received Packets dropped|Cumulative
| GET https://nsx-mgr01.contoso.com/api/v1/logical-router-ports/c8f68ee2-6b45-4017-ba3c-1d676b05f4a6/statistics/summary { "logical_router_port_id": "c8f68ee2-6b45-4017-ba3c-1d676b05f4a6", "last_update_timestamp": 1711436178815, "rx": { "total_bytes": 9671724, "total_packets": 124770, "dropped_packets": 1431, >>> "blocked_packets": 1, "destination_unsupported_dropped_packets": 0, "firewall_dropped_packets": 1361, "ipsec_dropped_packets": 0, "ipsec_no_sa_dropped_packets": 0, "ipsec_no_vti_dropped_packets": 0, "ipv6_dropped_packets": 0, "kni_dropped_packets": 0, "l4port_unsupported_dropped_packets": 9, "malformed_dropped_packets": 0, "no_receiver_dropped_packets": 0, "no_route_dropped_packets": 0, "proto_unsupported_dropped_packets": 0, "redirect_dropped_packets": 0, "rpf_check_dropped_packets": 0, "ttl_exceeded_dropped_packets": 60 }, "tx": { "total_bytes": 20683348, "total_packets": 174568, "dropped_packets": 1408, "blocked_packets": 0, "firewall_dropped_packets": 1408, "ipsec_dropped_packets": 0, "ipsec_no_sa_dropped_packets": 0, "ipsec_no_vti_dropped_packets": 0, "dad_dropped_packets": 0, "frag_needed_dropped_packets": 0, "ipsec_pol_block_dropped_packets": 0, "ipsec_pol_err_dropped_packets": 0, "no_arp_dropped_packets": 0, "no_linked_dropped_packets": 0, "no_mem_dropped_packets": 0, "non_ip_dropped_packets": 0, "service_insert_dropped_packets": 0 } } |
Router Interface Statistics|Router Link|Received Packets dropped|Cumulative
| GET https://nsx-mgr01.contoso.com/api/v1/logical-router-ports/a62f9b69-c532-44a8-89a0-3e42c6292d94/statistics/summary { "logical_router_port_id": "a62f9b69-c532-44a8-89a0-3e42c6292d94", "last_update_timestamp": 1711435883721, "rx": { "total_bytes": 571464466, "total_packets": 3205271, "dropped_packets": 1046310 >>> }, "tx": { "total_bytes": 231291722, "total_packets": 3295086, "dropped_packets": 27125 } } |
Router Interface Statistics|Up Link|Received Packets dropped|Cumulative
※ Tier-1 Gateway라서 조회할 Logical Port가 없음
3. API 조회 결과 전체 통계치는 다음과 같은 Metric 합으로 계산 가능
Statistics|Received Packets dropped|Cumulative =
Router Interface Statistics|Down Link|Received Packets dropped|Cumulative +
Router Interface Statistics|Router Link|Received Packets dropped|Cumulative +
Router Interface Statistics|Up Link|Received Packets dropped|Cumulative
4. 고객사에서도 동일하게 전체 Logical Router Ports 조회 후, 각 Logical Router Port 별 누적 통계치를 API를 통해 확인
Tier-1 Logical Router Port들만 "dropped_packets" 확인 결과 개별 인터페이스의 "dropped_packets" 값 합산 시, 문제 증상에서 기술했던 전체 누적 통계치 만큼의 수치 확인 가능
3,616 + 3,618 + 68,759,717 + 68,759,769 = 137,526,720
| { "logical_router_port_id" : "e5a1afc3-8985-4487-9d42-8c6d9165ab58", "last_update_timestamp" : 1704440895349, "rx" : { "total_bytes" : 1080106253, "total_packets" : 11078139, "dropped_packets" : 3616, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "blocked_packets" : 3606, "destination_unsupported_dropped_packets" : 0, "firewall_dropped_packets" : 0, "ipsec_dropped_packets" : 0, "ipsec_no_sa_dropped_packets" : 0, "ipsec_no_vti_dropped_packets" : 0, "ipv6_dropped_packets" : 0, "kni_dropped_packets" : 0, "l4port_unsupported_dropped_packets" : 0, "malformed_dropped_packets" : 0, "no_receiver_dropped_packets" : 0, "no_route_dropped_packets" : 0, "proto_unsupported_dropped_packets" : 1, "redirect_dropped_packets" : 0, "rpf_check_dropped_packets" : 0, "ttl_exceeded_dropped_packets" : 9 }, { "logical_router_port_id" : "95dc35ea-2014-49ff-9235-353929aa7b9c", "last_update_timestamp" : 1694658676827, "rx" : { "total_bytes" : 1070352364, "total_packets" : 10933502, "dropped_packets" : 3618, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "blocked_packets" : 3618, "destination_unsupported_dropped_packets" : 0, "firewall_dropped_packets" : 0, "ipsec_dropped_packets" : 0, "ipsec_no_sa_dropped_packets" : 0, "ipsec_no_vti_dropped_packets" : 0, "ipv6_dropped_packets" : 0, "kni_dropped_packets" : 0, "l4port_unsupported_dropped_packets" : 0, "malformed_dropped_packets" : 0, "no_receiver_dropped_packets" : 0, "no_route_dropped_packets" : 0, "proto_unsupported_dropped_packets" : 0, "redirect_dropped_packets" : 0, "rpf_check_dropped_packets" : 0, "ttl_exceeded_dropped_packets" : 0 }, "id" : "493f7734-1666-4a23-a44d-ec45a207f614", "display_name" : "SI_ServiceLinkPort1_6fa67a56-c3a1-43bc-bebd-16adb20f85c6", { "logical_router_port_id" : "493f7734-1666-4a23-a44d-ec45a207f614", "last_update_timestamp" : 1712132863559, "rx" : { "total_bytes" : 7872385704, "total_packets" : 114017292, "dropped_packets" : 68759717, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "blocked_packets" : 68759613, "destination_unsupported_dropped_packets" : 0, "firewall_dropped_packets" : 0, "ipsec_dropped_packets" : 0, "ipsec_no_sa_dropped_packets" : 0, "ipsec_no_vti_dropped_packets" : 0, "ipv6_dropped_packets" : 0, "kni_dropped_packets" : 0, "l4port_unsupported_dropped_packets" : 0, "malformed_dropped_packets" : 0, "no_receiver_dropped_packets" : 0, "no_route_dropped_packets" : 0, "proto_unsupported_dropped_packets" : 93, "redirect_dropped_packets" : 0, "rpf_check_dropped_packets" : 0, "ttl_exceeded_dropped_packets" : 11 }, "id" : "c6edc138-3a3f-4c08-8c03-2415105d9df1", "display_name" : "SI_ServiceLinkPort2_6fa67a56-c3a1-43bc-bebd-16adb20f85c6", { "logical_router_port_id" : "c6edc138-3a3f-4c08-8c03-2415105d9df1", "last_update_timestamp" : 1712132863559, "rx" : { "total_bytes" : 7872035921, "total_packets" : 114010759, "dropped_packets" : 68759769, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "blocked_packets" : 68759672, "destination_unsupported_dropped_packets" : 0, "firewall_dropped_packets" : 0, "ipsec_dropped_packets" : 0, "ipsec_no_sa_dropped_packets" : 0, "ipsec_no_vti_dropped_packets" : 0, "ipv6_dropped_packets" : 0, "kni_dropped_packets" : 0, "l4port_unsupported_dropped_packets" : 0, "malformed_dropped_packets" : 0, "no_receiver_dropped_packets" : 0, "no_route_dropped_packets" : 0, "proto_unsupported_dropped_packets" : 97, "redirect_dropped_packets" : 0, "rpf_check_dropped_packets" : 0, "ttl_exceeded_dropped_packets" : 0 }, |
5. "dropped_packets"의 수치가 높은 Logical Router Port는 Service Insertion을 이용하는 3rd Party Firewall로 해당 Packet 수치는 추가로 3rd Party Firewall 에서 확인 필요
[Conclusion]
1. Aria Operations에서 보여주는 Tier-1 Gateway에서 보여주는 Interface는 Downlink|RouterLink|Uplink만 있기 때문에 Service Insertion 용도로 사용되는 Interface의 통계치를 보기 위해서는 NSX API를 통해 확인 필요
