[NSX] How to remove a stale ingress policy created by NCP
Tanzu, Openshift 등 Kubernetes 환경에서 NCP(NSX Container Plugin)를 활용하면 Kubernetes에 만들어진 Object들을 NSX에서 확인할 수 있습니다.
이번 케이스는 고객사에서 VM 내에 TKGI를 사용하다 운영 중인 VM을 제거하고, 신규로 TKGm을 구축하였는데, TGKI에서 사용하던 Object들이 NSX에 남아 있게 된 사례입니다.
문제는 고객이 특정 Object를 제거하려고 했지만, DELETE Method API를 제공하지 않는터라 원하는 Object를 정리할 수 없는 상황이었습니다.
이 Object를 어떻게 정리했는지 살펴보겠습니다.
1. 먼저 유사한 환경 구성을 위해서 TKGS(vSphere with Tanzu)를 설치
2. 이 후 별도의 Namespace를 생성하고, 테스트 용도로 Deployment, Service, Ingress를 배포
| ## deployment.yaml kind: Deployment apiVersion: apps/v1 metadata: name: hackazon-shop labels: app: hackazon-shop spec: replicas: 2 selector: matchLabels: app: hackazon-shop template: metadata: labels: app: hackazon-shop spec: containers: - name: hackazon-shop image: projects.registry.vmware.com/tanzu_ese_poc/hackazon:1.0 ports: - containerPort: 80 protocol: TCP ## service.yaml kind: Service apiVersion: v1 metadata: name: hackazon-l4 labels: app: hackazon-shop svc: hackazon-l4 spec: ports: - port: 80 selector: app: hackazon-shop type: ClusterIP ## ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hackazon-ingress labels: app: hackazon-shop annotations: kubernetes.io/ingress.class: contour spec: rules: - host: hackazon.contoso.com http: paths: - pathType: Prefix path: / backend: service: name: hackazon-l4 port: number: 80 C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app apply -f deployment.yaml deployment.apps/hackazon-shop created C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app apply -f service.yaml service/hackazon-l4 created C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app apply -f ingress.yaml ingress.networking.k8s.io/hackazon-ingress created C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app get ingress NAME CLASS HOSTS ADDRESS PORTS AGE hackazon-ingress <none> hackazon.contoso.com 80 14h C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app get pods NAME READY STATUS RESTARTS AGE hackazon-shop-6b56688977-47b5d 0/1 ErrImagePull 0 19h hackazon-shop-6b56688977-4jwz2 0/1 ErrImagePull 0 15h hackazon-shop-6b56688977-d8pcs 0/1 ErrImagePull 0 11h hackazon-shop-6b56688977-dz9kj 0/1 ErrImagePull 0 17h hackazon-shop-6b56688977-jsdm5 0/1 ErrImagePull 0 13h hackazon-shop-6b56688977-jwt9g 0/1 ErrImagePull 0 15h hackazon-shop-6b56688977-lr5fp 0/1 ErrImagePull 0 9h hackazon-shop-6b56688977-nd9zs 0/1 ErrImagePull 0 17h hackazon-shop-6b56688977-p8jtm 0/1 ErrImagePull 0 19h hackazon-shop-6b56688977-phz9r 0/1 ErrImagePull 0 3h32m hackazon-shop-6b56688977-qhst8 0/1 ErrImagePull 0 5h22m hackazon-shop-6b56688977-s4pjx 0/1 ErrImagePull 0 80m hackazon-shop-6b56688977-twh6n 0/1 ErrImagePull 0 3h21m hackazon-shop-6b56688977-x6gnr 0/1 ErrImagePull 0 11h hackazon-shop-6b56688977-x75n4 0/1 ErrImagePull 0 13h hackazon-shop-6b56688977-zb8qw 0/1 ErrImagePull 0 89m hackazon-shop-6b56688977-zlbqj 0/1 ErrImagePull 0 7h37m hackazon-shop-6b56688977-zpmp9 0/1 ErrImagePull 0 9h C:\Users\Administrator.AD\Downloads\vsphere-plugin\bin>kubectl -n ns-hackazon-app get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hackazon-l4 ClusterIP 10.96.0.245 <none> 80/TCP 14h |
3. 위와 같이 Resource를 생성하고 나면, NSX UI에서 다음 Object 검색 가능
4. 고객은 위 Object를 DELETE Method를 이용하여 제거 실패했기 때문에, 내부에서도 재현 테스트
"Method is not allowed" 메시지와 함께, 실패 확인
| curl -X 'https://nsx-mgr01.contoso.com/api/v1/fabric/container-ingress-policies/c0af3f11-d076-47ea-bc2d-07fd956cf80d' \ -H 'Accept: application/json' \ { "external_id": "c0af3f11-d076-47ea-bc2d-07fd956cf80d", "container_cluster_id": "4bd9b4bf-50c8-5c70-ba40-7e2f940d24d5", "container_project_id": "da080956-dbe9-4f2b-b716-c227f50ed83c", "spec": "rules:\n- host: hackazon.contoso.com\n http:\n paths:\n - backend:\n service:\n name: hackazon-l4\n port:\n number: 80\n path: /\n pathType: Prefix\n", "origin_properties": [], "network_status": "HEALTHY", "container_application_ids": [ "0f763abb-2d04-488c-bf79-7766ea28947b" ], "resource_type": "ContainerIngressPolicy", "display_name": "hackazon-ingress", "tags": [ { "scope": "dis:k8s:app", "tag": "hackazon-shop" } ], "scope": [ { "scope_id": "4bd9b4bf-50c8-5c70-ba40-7e2f940d24d5", "scope_type": "CONTAINER_CLUSTER" } ], "_last_sync_time": 1719138625916 } curl -X DELETE 'https://nsx-mgr01.contoso.com/api/v1/fabric/container-ingress-policies/c0af3f11-d076-47ea-bc2d-07fd956cf80d' \ -H 'X-Allow-Overwrite: true' { "module_name": "common-services", "error_message": "Method is not allowed", "error_code": 282 } |
5. 개발팀에 확인 결과, 해당 Object를 NSX에서 제거하는 방법은 별도로 없기 때문에 Object가 포함되어 있는 Cluster ID를 이용하여 Cluster 제거 가이드
| curl -X GET 'https://nsx-mgr01.contoso.com/api/v1/fabric/container-ingress-policies/c0af3f11-d076-47ea-bc2d->07fd956cf80d' -H 'Accept: application/json' { "external_id": "c0af3f11-d076-47ea-bc2d-07fd956cf80d", "container_cluster_id": "4bd9b4bf-50c8-5c70-ba40-7e2f940d24d5", >>>>>>>>>>>>>>>>>>>>>>> curl -k -u 'admin:XXXX' -X DELETE "https://nsx-mgr01.contoso.com/api/v1/fabric/container-clusters/<container_cluster_id>" |
테스트 결과 다음과 같이 vCenter에는 남아 있지만,
NSX UI에서는 제거된 것을 확인
6. 위 API와 별개로 NCP Clean Script를 사용하여 특정 Kubernetes Cluster를 제거하는 방안도 가능
For Kubernetes : https://docs.vmware.com/en/VMware-NSX-Container-Plugin/3.2/ncp-kubernetes/GUID-0AC013DC-AF04-4FEB-97EC-B693237AD6CF.html
For OpenShift : https://docs.vmware.com/en/VMware-NSX-Container-Plugin/3.2/ncp-openshift/GUID-0AC013DC-AF04-4FEB-97EC-B693237AD6CF.html
이와 관련하여 Public KB를 별도로 Publish 했으니 참고하시기 바랍니다.
How to remove a stale ingress policy created by NCP
https://knowledge.broadcom.com/external/article/370823