AEWS 2주차 정리

🐧 AWS EKS 환경 설치 준비 (Arch Linux)

1. 시스템 업데이트

yay -Syu

2. AWS CLI 설치

yay -S aws-cli-v2

# 설치확인
aws --version

# 결과
aws-cli/2.24.0 Python/3.13.1 Linux/6.13.1-arch1-1 source/x86_64.arch

3. eksctl 설치

최신 eksctl 설치 필요 (version: 1.31 지원)

ARCH=$(uname -m)
if [[ "$ARCH" == "x86_64" ]]; then ARCH="amd64"; fi
if [[ "$ARCH" == "aarch64" ]]; then ARCH="arm64"; fi

curl --silent --location "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_${ARCH}.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin/
sudo chmod +x /usr/local/bin/eksctl

# 설치확인
eksctl version

# 결과
0.203.0

4. kubectl 설치

yay -S kubectl

# 설치확인
kubectl version --client=true

# 결과
Client Version: v1.32.1
Kustomize Version: v5.5.0

5. Helm 설치

yay -S helm

# 설치확인
helm version

# 결과
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}

6. krew 설치 및 플러그인 추가

yay -S krew

# 설치확인
kubectl krew version

# 결과
OPTION            VALUE
GitTag            v0.4.4
GitCommit         343e657
IndexURI          https://github.com/kubernetes-sigs/krew-index.git
BasePath          /home/devshin/.krew
IndexPath         /home/devshin/.krew/index/default
InstallPath       /home/devshin/.krew/store
BinPath           /home/devshin/.krew/bin
DetectedPlatform  linux/amd64

echo 'export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"' >> ~/.zshrc
echo 'export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"' >> ~/.bashrc

source ~/.zshrc  # zsh 사용 시
source ~/.bashrc # bash 사용 시

kubectl krew install neat get-all df-pv stern
kubectl krew list

# 결과
PLUGIN   VERSION
df-pv    v0.3.0
get-all  v1.3.8
krew     v0.4.4
neat     v2.0.4
stern    v1.32.0

7. kube-ps1 설치

yay -S kube-ps1
echo "source /opt/kube-ps1/kube-ps1.sh" >> ~/.bashrc
echo 'PS1="[\u@\h \W $(kube_ps1)]$ "' >> ~/.bashrc
source ~/.bashrc

echo "source /opt/kube-ps1/kube-ps1.sh" >> ~/.zshrc
echo 'PROMPT="$(kube_ps1)$PROMPT"' >> ~/.zshrc
source ~/.zshrc

8. kubectx 설치

yay -S kubectx

9. kubecolor 설치 및 alias 설정

yay -S kubecolor

echo "alias k=kubectl" >> ~/.bashrc
echo "alias kubectl=kubecolor" >> ~/.bashrc
echo 'complete -F __start_kubectl kubecolor' >> ~/.bashrc
source ~/.bashrc

echo "alias k=kubectl" >> ~/.zshrc
echo "alias kubectl=kubecolor" >> ~/.zshrc
echo 'autoload -U compinit && compinit' >> ~/.zshrc
echo "compdef kubecolor=kubectl" >> ~/.zshrc
source ~/.zshrc

10. (옵션) AWS 세션 매니저 설치

yay -S aws-session-manager-plugin

# 설치확인
session-manager-plugin --version

# 결과
1.2.707.0

11. (옵션) sshpass 설치

yay -S sshpass

# 설치확인
sshpass -V

# 결과
sshpass 1.10
(C) 2006-2011 Lingnu Open Source Consulting Ltd.
(C) 2015-2016, 2021-2022 Shachar Shemesh
This program is free software, and can be distributed under the terms of the GPL
See the COPYING file for more information.

Using "assword" as the default password prompt indicator.

12. (옵션) Wireshark 설치

패킷 캡쳐 및 캡쳐된 파일에서 패킷 내용 확인

yay -S wireshark-qt

# Wireshark는 일반 사용자 권한으로 실행할 수 없을 수 있으므로, wireshark 그룹에 사용자를 추가해야 함
sudo usermod -aG wireshark $USER
newgrp wireshark  # 변경 사항 즉시 적용

# 설치 확인
wireshark --version

# 결과
Wireshark 4.4.3.

---

🔐 AWS Configure 자격 증명 설정

aws configure
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXX
Default region name [None]: ap-northeast-2
Default output format [None]: json

• AWS Access Key ID: (발급받은 Access Key ID 입력)
• AWS Secret Access Key: (Secret Access Key 입력)
• Default region name: ap-northeast-2 (서울 리전, 원하는 리전 선택 가능)
• Default output format: json 또는 yaml (기본값: json)

---

🚀 AWS CloudFormation을 통한 기본 실습 환경 배포

1. CloudFormation 템플릿 다운로드

cd Downloads
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-2week.yaml

2. CloudFormation 스택 배포

aws cloudformation deploy --template-file ~/Downloads/myeks-2week.yaml \
    --stack-name myeks --parameter-overrides KeyName=kp-aews SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 --region ap-northeast-2

# 결과
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - myeks

operator-host라는 이름의 t2.small EC2 인스턴스가 하나 배포됨

3. 배포된 EC2 인스턴스에 접속

(1) CloudFormation 스택 배포 완료 후 운영서버 EC2 IP 출력

aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[*].OutputValue' --output text

✅ 출력

15.165.15.90

(2) 운영서버 EC2에 SSH 접속

# ssh -i kp-aews.pem ec2-user@15.165.15.90
ssh -i kp-aews.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)

4. ssh-agent를 이용한 키 관리

(1) ssh-agent 시작 및 키 추가

eval "$(ssh-agent -s)"
# 결과
ssh-add ~/Downloads/kp-aews.pem

(2) SSH 접속 (비밀번호 없이)

ssh ec2-user@15.165.15.90

5. 운영 서버 EC2(operator-host)에서 EKS 접근 설정

(1) AWS IAM 자격증명 설정

[root@operator-host ~]# aws configure
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXX
Default region name [None]: ap-northeast-2
Default output format [None]: json

(2) IAM 사용자 확인 (get-caller-identity)

[root@operator-host ~]# aws sts get-caller-identity --query Arn
"arn:aws:iam::378102432899:user/eks-user"

6. Peering connections 확인

myeks-VPC와 operator-VPC가 피어링으로 연결된 상태

myeks-VPC 라우팅 경로

• myeks-VPC 퍼블릭 서브넷에선 172.20.0.0/16 대역과 통신 시 인터넷 경유 없이 피어링(Peering)을 통해 Operator VPC와 직접 통신

operator-VPC 라우팅 경로

• operator-VPC 퍼블릭 서브넷에선 192.168.0.0/16 대역과 통신 시 인터넷 경유 없이 피어링(Peering)을 통해 myeks-VPC와 직접 통신

---

🚢 eksctl을 통해 EKS 배포

1. 클러스터명 변수 지정

export CLUSTER_NAME=myeks

2. myeks-VPC 변수 지정

export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
echo $VPCID

✅ 출력

vpc-02725f328b257230c

AWS 콘솔 VPC > Your VPCs에서 조회한 myeks-VPC ID : vpc-02725f328b257230c

3. myeks 퍼블릭 서브넷 변수 지정

AZ1, AZ2, AZ3에 해당하는 퍼블릭 서브넷의 ID를 변수로 지정

export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)

echo $PubSubnet1 $PubSubnet2 $PubSubnet3

✅ 출력

subnet-0b53307d4d544b3bf subnet-019936bd535b68960 subnet-0d19099a6b73555f8

서브넷 대역(CIDR) 범위

• AZ1: 192.168.1.0/24, AZ2: 192.168.2.0/24, AZ3: 192.168.3.0/24

4. myeks.yaml 파일 작성

변경할 항목

VPC ID, Public Subnet1 ID, Public Subnet2 ID, Public Subnet3 ID, SSH Public Key Name

echo $VPCID
vpc-02725f328b257230c

echo $PubSubnet1 $PubSubnet2 $PubSubnet3
subnet-0b53307d4d544b3bf subnet-019936bd535b68960 subnet-0d19099a6b73555f8

myeks.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: myeks
  region: ap-northeast-2
  version: "1.31"

kubernetesNetworkConfig:
  ipFamily: IPv4

iam:
  vpcResourceControllerPolicy: true
  withOIDC: true

accessConfig:
  authenticationMode: API_AND_CONFIG_MAP

vpc:
  autoAllocateIPv6: false
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: true # if you only want to allow private access to the cluster
    publicAccess: true # if you want to allow public access to the cluster
  id: vpc-02725f328b257230c  # 각자 환경 정보로 수정
  manageSharedNodeSecurityGroupRules: true # if you want to manage the rules of the shared node security group
  nat:
    gateway: Disable
  subnets:
    public:
      ap-northeast-2a:
        az: ap-northeast-2a
        cidr: 192.168.1.0/24
        id: subnet-0b53307d4d544b3bf  # 각자 환경 정보로 수정
      ap-northeast-2b:
        az: ap-northeast-2b
        cidr: 192.168.2.0/24
        id: subnet-019936bd535b68960  # 각자 환경 정보로 수정
      ap-northeast-2c:
        az: ap-northeast-2c
        cidr: 192.168.3.0/24
        id: subnet-0d19099a6b73555f8  # 각자 환경 정보로 수정

addons:
  - name: vpc-cni # no version is specified so it deploys the default version
    version: latest # auto discovers the latest available
    attachPolicyARNs: # attach IAM policies to the add-on's service account
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    configurationValues: |-
      enableNetworkPolicy: "true"

  - name: kube-proxy
    version: latest

  - name: coredns
    version: latest

  - name: metrics-server
    version: latest

privateCluster:
  enabled: false
  skipEndpointCreation: false

managedNodeGroups:
- amiFamily: AmazonLinux2023
  desiredCapacity: 3
  disableIMDSv1: true
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false # Disable ALB Ingress Controller
      appMesh: false
      appMeshPreview: false
      autoScaler: false
      awsLoadBalancerController: true # Enable AWS Load Balancer Controller
      certManager: true # Enable cert-manager
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: true # Enable ExternalDNS
      fsx: false
      imageBuilder: true
      xRay: false
  instanceSelector: {}
  instanceType: t3.medium
  preBootstrapCommands:
    # install additional packages
    - "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y"
    # disable hyperthreading
    - "for n in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -s -d, -f2- | tr ',' '\n' | sort -un); do echo 0 > /sys/devices/system/cpu/cpu${n}/online; done"
  labels:
    alpha.eksctl.io/cluster-name: myeks
    alpha.eksctl.io/nodegroup-name: ng1
  maxSize: 3
  minSize: 3
  name: ng1
  privateNetworking: false
  releaseVersion: ""
  securityGroups:
    withLocal: null
    withShared: null
  ssh:
    allow: true
    publicKeyName: kp-aews  # 각자 환경 정보로 수정
  tags:
    alpha.eksctl.io/nodegroup-name: ng1
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 30
  volumeThroughput: 125
  volumeType: gp3

5. 최종 yaml로 eks 배포

(1) kubeconfig 파일 경로 위치 지정

export KUBECONFIG=~/Downloads/kubeconfig

(2) 배포

eksctl create cluster -f myeks.yaml --verbose 4

배포완료

6. 배포 후 기본 정보 확인

(1) API 서버 엔드포인트 및 OIDC 정보

API Server Endpoint와 OpenID Connect Provider URL(oidc) 포함

(2) Compute

Node groups에서 AMI(AL2023) 확인 가능

(3) Networking

Access는 Public and Private으로 설정됨

(4) Add-ons

VPC CNI에서 Edit 후 IRSA 권한 설정 확인 → 해당 IAM Role 확인

(5) Access

IAM Access Entries에서 설치 시 사용한 자격증명 사용자 확인

AmazonEKSClusterAdminPolicy 관리자 정책이 포함되어 있어 EKS 관리 콘솔 접근 및 관리 가능

7. EKS 정보 확인

kubectl cluster-info

✅ 출력

Kubernetes control plane is running at https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

8. 네임스페이스 default 변경 적용

kubens default

# 결과
✔ Active namespace is "default"

9. 현재 Kubernetes 컨텍스트 확인

cat $KUBECONFIG | grep current-context

✅ 출력

current-context: eks-user@myeks.ap-northeast-2.eksctl.io

10. Kubernetes 컨텍스트 이름 변경

kubectl config rename-context "eks-user@myeks.ap-northeast-2.eksctl.io" "eksworkshop"

# 결과
Context "eks-user@myeks.ap-northeast-2.eksctl.io" renamed to "eksworkshop".

cat $KUBECONFIG | grep current-context

✅ 출력

current-context: eksworkshop

11. 노드 정보 확인

kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone

✅ 출력

NAME                                               STATUS   ROLES    AGE    VERSION               INSTANCE-TYPE   CAPACITYTYPE   ZONE
ip-192-168-1-193.ap-northeast-2.compute.internal   Ready    <none>   169m   v1.31.4-eks-aeac579   t3.medium       ON_DEMAND      ap-northeast-2a
ip-192-168-2-52.ap-northeast-2.compute.internal    Ready    <none>   169m   v1.31.4-eks-aeac579   t3.medium       ON_DEMAND      ap-northeast-2b
ip-192-168-3-72.ap-northeast-2.compute.internal    Ready    <none>   169m   v1.31.4-eks-aeac579   t3.medium       ON_DEMAND      ap-northeast-2

kubectl get node -v=6

✅ 출력

I0212 15:08:46.728964  137397 loader.go:402] Config loaded from file:  /home/devshin/Downloads/kubeconfig
I0212 15:08:46.729339  137397 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0212 15:08:46.729350  137397 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0212 15:08:46.729355  137397 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0212 15:08:46.729359  137397 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0212 15:08:47.170830  137397 round_trippers.go:560] GET https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 436 milliseconds
NAME                                               STATUS   ROLES    AGE    VERSION
ip-192-168-1-193.ap-northeast-2.compute.internal   Ready    <none>   170m   v1.31.4-eks-aeac579
ip-192-168-2-52.ap-northeast-2.compute.internal    Ready    <none>   170m   v1.31.4-eks-aeac579
ip-192-168-3-72.ap-northeast-2.compute.internal    Ready    <none>   170m   v1.31.4-eks-aeac579

12. pod 정보 확인

kubectl get pod -A

✅ 출력

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   aws-node-6mctb                    2/2     Running   0          173m
kube-system   aws-node-b66dj                    2/2     Running   0          173m
kube-system   aws-node-rf79g                    2/2     Running   0          173m
kube-system   coredns-86f5954566-gqf97          1/1     Running   0          177m
kube-system   coredns-86f5954566-nntgz          1/1     Running   0          177m
kube-system   kube-proxy-jg5qj                  1/1     Running   0          173m
kube-system   kube-proxy-t2sqh                  1/1     Running   0          173m
kube-system   kube-proxy-w96mt                  1/1     Running   0          173m
kube-system   metrics-server-86bbfd75bb-j72mf   1/1     Running   0          177m
kube-system   metrics-server-86bbfd75bb-pbpkd   1/1     Running   0          177m

kubectl get pdb -n kube-system

✅ 출력

NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
coredns          N/A             1                 1                     178m
metrics-server   N/A             1                 1                     178m

13. 관리형 노드 그룹 확인

eksctl get nodegroup --cluster $CLUSTER_NAME

✅ 출력

CLUSTER	NODEGROUP	STATUS	CREATED			MIN SIZE	MAX SIZE	DESIRED CAPACITY	INSTANCE TYPE	IMAGE ID		ASG NAME					TYPE
myeks	ng1		ACTIVE	2025-02-12T03:17:03Z	3		3		3			t3.medium	AL2023_x86_64_STANDARD	eks-ng1-84ca7c14-790d-ef45-8256-776960f87794	managed

aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name ng1 | jq

✅ 출력

{
  "nodegroup": {
    "nodegroupName": "ng1",
    "nodegroupArn": "arn:aws:eks:ap-northeast-2:378102432899:nodegroup/myeks/ng1/84ca7c14-790d-ef45-8256-776960f87794",
    "clusterName": "myeks",
    "version": "1.31",
    "releaseVersion": "1.31.4-20250203",
    "createdAt": "2025-02-12T12:17:03.928000+09:00",
    "modifiedAt": "2025-02-12T15:08:14.235000+09:00",
    "status": "ACTIVE",
    "capacityType": "ON_DEMAND",
    "scalingConfig": {
      "minSize": 3,
      "maxSize": 3,
      "desiredSize": 3
    },
    "instanceTypes": [
      "t3.medium"
    ],
    "subnets": [
      "subnet-0b53307d4d544b3bf",
      "subnet-019936bd535b68960",
      "subnet-0d19099a6b73555f8"
    ],
    "amiType": "AL2023_x86_64_STANDARD",
    "nodeRole": "arn:aws:iam::378102432899:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-aJPw1zdjbXYF",
    "labels": {
      "alpha.eksctl.io/cluster-name": "myeks",
      "alpha.eksctl.io/nodegroup-name": "ng1"
    },
    "resources": {
      "autoScalingGroups": [
        {
          "name": "eks-ng1-84ca7c14-790d-ef45-8256-776960f87794"
        }
      ]
    },
    "health": {
      "issues": []
    },
    "updateConfig": {
      "maxUnavailable": 1
    },
    "launchTemplate": {
      "name": "eksctl-myeks-nodegroup-ng1",
      "version": "1",
      "id": "lt-0ec600e4f000289da"
    },
    "tags": {
      "aws:cloudformation:stack-name": "eksctl-myeks-nodegroup-ng1",
      "alpha.eksctl.io/cluster-name": "myeks",
      "alpha.eksctl.io/nodegroup-name": "ng1",
      "aws:cloudformation:stack-id": "arn:aws:cloudformation:ap-northeast-2:378102432899:stack/eksctl-myeks-nodegroup-ng1/c502af50-e8ef-11ef-89e7-022a714bdf75",
      "eksctl.cluster.k8s.io/v1alpha1/cluster-name": "myeks",
      "aws:cloudformation:logical-id": "ManagedNodeGroup",
      "alpha.eksctl.io/nodegroup-type": "managed",
      "alpha.eksctl.io/eksctl-version": "0.203.0"
    }
  }
}

14. eks addon 확인

eksctl get addon --cluster $CLUSTER_NAME

✅ 출력

2025-02-12 15:29:21 [ℹ]  Kubernetes version "1.31" in use by cluster "myeks"
2025-02-12 15:29:21 [ℹ]  getting all addons
2025-02-12 15:29:23 [ℹ]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME		VERSION			STATUS	ISSUES	IAMROLE										UPDATE AVAILABLE	CONFIGURATION VALUES		POD IDENTITY ASSOCIATION ROLES
coredns		v1.11.4-eksbuild.2	ACTIVE	0													
kube-proxy	v1.31.3-eksbuild.2	ACTIVE	0													
metrics-server	v0.7.2-eksbuild.1	ACTIVE	0													
vpc-cni		v1.19.2-eksbuild.5	ACTIVE	0	arn:aws:iam::378102432899:role/eksctl-myeks-addon-vpc-cni-Role1-fGF6qGwGjFyL			enableNetworkPolicy: "true"	

15. 관리형 노드그룹(EC2) 접속 및 노드 정보 확인

(1) EC2 인스턴스의 IAM Role 확인

EC2 인스턴스 보안 설정에서 IAM Role이 매핑되어 있으며, EC2 인스턴스 프로파일에 연결됨

managedNodeGroups의 IAM 설정과 매칭되어 awsLoadBalancerController 실행 시 필요한 권한을 제공

awsLoadBalancerController, certManager, externalDNS 권한 활성화

managedNodeGroups:
- amiFamily: AmazonLinux2023
  desiredCapacity: 3
  disableIMDSv1: true
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false # Disable ALB Ingress Controller
      appMesh: false
      appMeshPreview: false
      autoScaler: false
      awsLoadBalancerController: true # Enable AWS Load Balancer Controller
      certManager: true # Enable cert-manager
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: true # Enable ExternalDNS
      fsx: false
      imageBuilder: true
      xRay: false

(2) 인스턴스 공인 IP 확인

[root@operator-host ~]# aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

✅ 출력

-----------------------------------------------------------------------------------------
|                                   DescribeInstances                                   |
+----------------------+-----------------+----------------+-----------------+-----------+
|      InstanceID      |  InstanceName   | PrivateIPAdd   |   PublicIPAdd   |  Status   |
+----------------------+-----------------+----------------+-----------------+-----------+
|  i-0d14501e2c8353ae6 |  myeks-ng1-Node |  192.168.3.72  |  43.201.115.81  |  running  |
|  i-00db22b426cc7efb5 |  operator-host  |  172.20.1.100  |  15.165.15.90   |  running  |
|  i-090451779dfb774e9 |  myeks-ng1-Node |  192.168.1.193 |  43.202.57.204  |  running  |
|  i-0e8bab88cd0a40ae8 |  myeks-ng1-Node |  192.168.2.52  |  15.164.179.214 |  running  |
+----------------------+-----------------+----------------+-----------------+-----------+

(3) 인스턴스 공인 IP 변수 지정

export N1=43.202.57.204
export N2=15.164.179.214
export N3=43.201.115.81
echo $N1, $N2, $N3

✅ 출력

43.202.57.204, 15.164.179.214, 43.201.115.81

(4) ping 테스트

ping -c 2 $N1

✅ 출력

# 출력: ping 실패 (100% packet loss)
PING 43.202.57.204 (43.202.57.204) 56(84) bytes of data.

--- 43.202.57.204 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1037ms

(5) 관리형 워커노드 보안그룹 확인

• 노드그룹 Remote Access 보안그룹과 EKS 클러스터 보안그룹 존재
• Remote Access 보안그룹: sg-08385fad7996593f1
• EKS 클러스터 보안그룹: sg-073b338fccd7776e7

(6) Remote Access 보안그룹 설정

• 집 공인 IP와 Operator 서버 내부 IP(172.20.1.100)를 소스 IP로 추가

• Remote Access 보안그룹 ID 변수 지정

export MNSGID=sg-08385fad7996593f1

• 집 공인 IP 인바운드 규칙 추가

aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr $(curl -s ipinfo.io/ip)/32

# 결과
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0e2dcc79069298593",
            "GroupId": "sg-08385fad7996593f1",
            "GroupOwnerId": "378102432899",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "182.230.60.93/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-0e2dcc79069298593"
        }
    ]
}

• Operator 서버 내부 IP 인바운드 규칙 추가

aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr 172.20.1.100/32

# 결과
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-062028d793058f3d1",
            "GroupId": "sg-08385fad7996593f1",
            "GroupOwnerId": "378102432899",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "172.20.1.100/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-062028d793058f3d1"
        }
    ]
}

(7) ping 테스트

ping -c 2 $N1

✅ 출력

# ping 성공
PING 43.202.57.204 (43.202.57.204) 56(84) bytes of data.
64 bytes from 43.202.57.204: icmp_seq=1 ttl=115 time=6.78 ms
64 bytes from 43.202.57.204: icmp_seq=2 ttl=115 time=20.1 ms

--- 43.202.57.204 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 6.783/13.453/20.123/6.670 ms

# Operator-host에서 ping 테스트
[root@operator-host ~]# ping -c 2 192.168.1.193

✅ 출력

# ping 성공
PING 192.168.1.193 (192.168.1.193) 56(84) bytes of data.
64 bytes from 192.168.1.193: icmp_seq=1 ttl=127 time=0.227 ms
64 bytes from 192.168.1.193: icmp_seq=2 ttl=127 time=0.498 ms

--- 192.168.1.193 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 0.227/0.362/0.498/0.136 ms

• Operator-Host에서 워커노드 서브넷으로 ping 성공
• myeks-VPC 보안그룹에 Operator-VPC(172.20.1.100/32) 인바운드 규칙 추가로 접근 가능

(8) 워커 노드 SSH 접속 확인

ssh ec2-user@$N1

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Mon Feb  3 23:59:19 2025 from 52.94.123.246
[ec2-user@ip-192-168-1-193 ~]$

ssh ec2-user@$N2

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Mon Feb  3 23:59:19 2025 from 52.94.123.246
[ec2-user@ip-192-168-2-52 ~]$

ssh ec2-user@$N3

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Mon Feb  3 23:59:19 2025 from 52.94.123.246
[ec2-user@ip-192-168-3-72 ~]$

(9) 운영서버 EC2에서 워커노드 EC2 접속 및 정보 확인

인스턴스 공인 IP 변수 지정

[root@operator-host ~]# export N1=192.168.1.193
[root@operator-host ~]# export N2=192.168.2.52
[root@operator-host ~]# export N3=192.168.3.72
[root@operator-host ~]# echo $N1, $N2, $N3
192.168.1.193, 192.168.2.52, 192.168.3.72

각 워커노드에 ping 테스트

[root@operator-host ~]# ping -c 2 $N1

✅ 출력

PING 192.168.1.193 (192.168.1.193) 56(84) bytes of data.
64 bytes from 192.168.1.193: icmp_seq=1 ttl=127 time=0.232 ms
64 bytes from 192.168.1.193: icmp_seq=2 ttl=127 time=0.242 ms

--- 192.168.1.193 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.232/0.237/0.242/0.005 ms

16. 노드 정보 확인

(1) 각 노드의 호스트 정보 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i hostnamectl; echo; done

✅ 출력

>> node 43.202.57.204 <<
 Static hostname: ip-192-168-1-193.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: ec2e13998d891ffbacc5d853155112da
         Boot ID: 3b6c5280edb84574b64c4edc1f352d1e
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250128
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.124-134.200.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

>> node 15.164.179.214 <<
 Static hostname: ip-192-168-2-52.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: ec2e325020be14105530b18b0e81710a
         Boot ID: 1fdda72d00cd465d8074e7c6238882a9
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250128
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.124-134.200.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

>> node 43.201.115.81 <<
 Static hostname: ip-192-168-3-72.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: ec2951238a61149817871c92befba51d
         Boot ID: 9a1cd0af9840421a9234e72a44777415
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250128
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.124-134.200.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

(2) 각 노드의 네트워크 인터페이스 정보 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done

✅ 출력

>> node 43.202.57.204 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:93:a6:bc:61 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.1.193/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5
       valid_lft 3137sec preferred_lft 3137sec
    inet6 fe80::23:93ff:fea6:bc61/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni01a4864c88a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 56:be:6a:bd:2d:2a brd ff:ff:ff:ff:ff:ff link-netns cni-a59ccd2b-5db2-0159-b78e-e0797b300a23
    inet6 fe80::54be:6aff:febd:2d2a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:49:8e:e5:f1:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.1.237/24 brd 192.168.1.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::49:8eff:fee5:f105/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: eni61c5a949744@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 52:41:42:c3:03:3f brd ff:ff:ff:ff:ff:ff link-netns cni-74caca86-36e2-a922-2920-c2c8c00e7b43
    inet6 fe80::5041:42ff:fec3:33f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 15.164.179.214 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:c5:5b:d1:77:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.2.52/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5
       valid_lft 3138sec preferred_lft 3138sec
    inet6 fe80::4c5:5bff:fed1:7757/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: enibce2df30e87@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 4a:c8:23:ab:85:39 brd ff:ff:ff:ff:ff:ff link-netns cni-5408bb28-ad79-a3f3-3a60-9442968852b1
    inet6 fe80::48c8:23ff:feab:8539/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:91:4b:4c:9c:e9 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.2.42/24 brd 192.168.2.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::491:4bff:fe4c:9ce9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: enic99196c7a64@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 8a:00:6e:37:b1:71 brd ff:ff:ff:ff:ff:ff link-netns cni-9343fb30-bdc8-5fab-b46d-3a5db58f8007
    inet6 fe80::8800:6eff:fe37:b171/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 43.201.115.81 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:d5:3d:4a:54:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.72/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 3144sec preferred_lft 3144sec
    inet6 fe80::8d5:3dff:fe4a:54bd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(3) 각 노드의 라우팅 테이블 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done

✅ 출력

>> node 43.202.57.204 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.193 metric 1024 
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.193 metric 1024 
192.168.1.59 dev eni61c5a949744 scope link 
192.168.1.90 dev eni01a4864c88a scope link 

>> node 15.164.179.214 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.52 metric 1024 
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.52 metric 1024 
192.168.2.72 dev enibce2df30e87 scope link 
192.168.2.248 dev enic99196c7a64 scope link 

>> node 43.201.115.81 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.72 metric 1024 
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.72 metric 1024 

(4) 각 노드의 cgroup 버전 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i stat -fc %T /sys/fs/cgroup/; echo; done

✅ 출력

>> node 43.202.57.204 <<
cgroup2fs

>> node 15.164.179.214 <<
cgroup2fs

>> node 43.201.115.81 <<
cgroup2fs

(5) 각 노드의 Kubelet 설정 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json.d/00-nodeadm.conf | jq; echo; done

✅ 출력

>> node 43.202.57.204 <<
{
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "clusterDNS": [
    "10.100.0.10"
  ],
  "kind": "KubeletConfiguration",
  "maxPods": 17
}

>> node 15.164.179.214 <<
{
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "clusterDNS": [
    "10.100.0.10"
  ],
  "kind": "KubeletConfiguration",
  "maxPods": 17
}

>> node 43.201.115.81 <<
{
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "clusterDNS": [
    "10.100.0.10"
  ],
  "kind": "KubeletConfiguration",
  "maxPods": 17
}

17. 운영서버 EC2에서 EKS 사용 설정

(1) IAM 자격증명 설정

[root@operator-host ~]# aws configure
AWS Access Key ID : XXXXXXXXXXXXXXXXXX
AWS Secret Access Key : XXXXXXXXXXXXXXXXXX
Default region name [ap-northeast-2]: ap-northeast-2
Default output format [json]: json

(2) IAM 자격 확인

[root@operator-host ~]# aws sts get-caller-identity --query Arn

✅ 출력

"arn:aws:iam::378102432899:user/eks-user"

(3) kubeconfig 생성

[root@operator-host ~]# cat ~/.kube/config
cat: /root/.kube/config: No such file or directory

[root@operator-host ~]# aws eks update-kubeconfig --name myeks --user-alias eks-user
Added new context eks-user to /root/.kube/config

(4) kubeconfig 정보 확인

(eks-user:N/A) [root@operator-host ~]# cat ~/.kube/config

✅ 출력

EKS 클러스터 정보, 인증서, 사용자 정보 포함

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYnRmWjRVZ2Fwdkl3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBeU1USXdNekF6TURSYUZ3MHpOVEF5TVRBd016QTRNRFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUM5UVd3L3U5UGhSOVpueEtYU3oxemZmR1p3OHlyNzJXS0lZVHozc1c4c2hXUlhpSzh6UHo4RmVaMHQKU2Q3Ym13RzdTK1JvbXltbTVreVQ3K2F1ZWt5SGZHOExhZzNKRUlMWnBQL2lOaFNGRUsvWWlSYkN0VGY3ZWp0NAovOFlwWG9Ic0pDbXpjZ3ZvQ0dwT0dnZ2dHUFh3dlNBM0lLWW5iRmVYNENTbjVUcmpVWXpURjZYVm0wSVZMT05PCjFIY1lJK0ZxQit2dDVGd3FoOVVFYUsrRHBVa3krWmswdi9FOVJ1ZDU1QzFuWUdsRFJBNnNVS1NoamZORzZ6bS8KcjZ0QUFyUmsvTDN1cGkvajMyVHE5amF2dW1ScHlHNTM2YlhIeUpULzFuU2Y5dTF4ZzNnbThYWWtLZUdXeko3TApQbzc2aHJoUnF1eEVnd0hHaysvNGV1ZzIzQi8xQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRMGpTUFNVN1pqcWFOTTJBa0NJN1FqK2RQV2xqQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQUtoSXd2cmNOTQpuNS9WS2RsdFBpTjdrcTBIZ0tHdFVMbS8zQnlSZ2daSHY3MkFKNVk3bUdsZUFabUUzOFEyNVV6M2NjT25DOEVsCjNOYnlSdzFGMFdCYnR6TU1iNmxnSDZMWW5TcHZlckROWUZ5SGNQQmZKQmRLd0hyTFhBTXNGUlNCczRUL0Nhdm0KdE4reEZaVkh5azRBbk81VlN6QjJRbmJ5dENqbmRxc3ZjajJOWS9WK3ZQbzJMQzZBckh6TXhodmF2eVRyVEtDLwpCK242c2tQM0FTNy9hT3JEeEhyaHlXQUFLUGpqSXhBeUhsZS9Db1FBQjVoNTFQQ0d3NlFZRlR0N3hyaHVMWjdSCjFMc3FIazlobGRXUDRSdjZ5Skx0a0ErOVRCdEM1Y251Z1d1VHFFUXBQZVExTWRLcTgyY2M1aFlaZXFHdXFTRjgKVU9tcHRXTFVsbVpHCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com
  name: arn:aws:eks:ap-northeast-2:378102432899:cluster/myeks
contexts:
- context:
    cluster: arn:aws:eks:ap-northeast-2:378102432899:cluster/myeks
    user: eks-user
  name: eks-user
current-context: eks-user
kind: Config
preferences: {}
users:
- name: eks-user
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - ap-northeast-2
      - eks
      - get-token
      - --cluster-name
      - myeks
      - --output
      - json
      command: aws

(5) EKS API DNS 조회

(eks-user:N/A) [root@operator-host ~]# APIDNS=$(aws eks describe-cluster --name myeks | jq -r .cluster.endpoint | cut -d '/' -f 3)

(eks-user:N/A) [root@operator-host ~]# dig +short $APIDNS

✅ 출력

43.201.119.98
15.165.73.21

(6) 클러스터 정보 확인

(eks-user:N/A) [root@operator-host ~]# kubectl cluster-info

✅ 출력

Kubernetes control plane is running at https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

(7) 노드 정보 확인

(eks-user:N/A) [root@operator-host ~]# kubectl get node -v6

✅ 출력

I0212 17:07:06.500164    3826 loader.go:395] Config loaded from file:  /root/.kube/config
I0212 17:07:07.311634    3826 round_trippers.go:553] GET https://D25BC1FD83873C599EC920D5193DC864.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 805 milliseconds
NAME                                               STATUS   ROLES    AGE     VERSION
ip-192-168-1-193.ap-northeast-2.compute.internal   Ready    <none>   4h48m   v1.31.4-eks-aeac579
ip-192-168-2-52.ap-northeast-2.compute.internal    Ready    <none>   4h48m   v1.31.4-eks-aeac579
ip-192-168-3-72.ap-northeast-2.compute.internal    Ready    <none>   4h48m   v1.31.4-eks-aeac579

---

🌐 네트워크 기본 정보 확인

1. CNI 정보 확인

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

✅ 출력

amazon-k8s-cni-init:v1.19.2-eksbuild.5
amazon-k8s-cni:v1.19.2-eksbuild.5
amazon

2. 노드 IP 확인

aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

✅ 출력

------------------------------------------------------------------
|                        DescribeInstances                       |
+----------------+-----------------+------------------+----------+
|  InstanceName  |  PrivateIPAdd   |   PublicIPAdd    | Status   |
+----------------+-----------------+------------------+----------+
|  myeks-ng1-Node|  192.168.3.72   |  43.201.115.81   |  running |
|  operator-host |  172.20.1.100   |  15.165.15.90    |  running |
|  myeks-ng1-Node|  192.168.1.193  |  43.202.57.204   |  running |
|  myeks-ng1-Node|  192.168.2.52   |  15.164.179.214  |  running |
+----------------+-----------------+------------------+----------+

3. Pod 정보 확인

k get pod -A

✅ 출력

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   aws-node-6mctb                    2/2     Running   0          5h8m
kube-system   aws-node-b66dj                    2/2     Running   0          5h8m
kube-system   aws-node-rf79g                    2/2     Running   0          5h8m
kube-system   coredns-86f5954566-gqf97          1/1     Running   0          5h12m
kube-system   coredns-86f5954566-nntgz          1/1     Running   0          5h12m
kube-system   kube-proxy-jg5qj                  1/1     Running   0          5h8m
kube-system   kube-proxy-t2sqh                  1/1     Running   0          5h8m
kube-system   kube-proxy-w96mt                  1/1     Running   0          5h8m
kube-system   metrics-server-86bbfd75bb-j72mf   1/1     Running   0          5h12m
kube-system   metrics-server-86bbfd75bb-pbpkd   1/1     Running   0          5h12m

4. 파드 IP 확인

kubectl get pod -n kube-system -o=custom-columns=NAME:.metadata.name,IP:.status.podIP,STATUS:.status.phase

✅ 출력

aws-node-6mctb(192.168.1.193), kube-proxy-w96mt(192.168.1.193)가 서버의 IP와 같은 이유?

호스트 서버의 네트워크 네임스페이스를 공유하기 때문

NAME                              IP              STATUS
aws-node-6mctb                    192.168.1.193   Running
aws-node-b66dj                    192.168.2.52    Running
aws-node-rf79g                    192.168.3.72    Running
coredns-86f5954566-gqf97          192.168.1.59    Running
coredns-86f5954566-nntgz          192.168.2.248   Running
kube-proxy-jg5qj                  192.168.3.72    Running
kube-proxy-t2sqh                  192.168.2.52    Running
kube-proxy-w96mt                  192.168.1.193   Running
metrics-server-86bbfd75bb-j72mf   192.168.1.90    Running
metrics-server-86bbfd75bb-pbpkd   192.168.2.72    Running

5. 노드 네트워크 정보 확인

(1) CNI 로그 파일 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i tree /var/log/aws-routed-eni; echo; done

✅ 출력

>> node 43.202.57.204 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── egress-v6-plugin.log
├── ipamd.log
├── network-policy-agent.log
└── plugin.log

0 directories, 5 files

>> node 15.164.179.214 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── egress-v6-plugin.log
├── ipamd.log
├── network-policy-agent.log
└── plugin.log

0 directories, 5 files

>> node 43.201.115.81 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── ipamd.log
└── network-policy-agent.log

0 directories, 3 files

(2) 노드별 IP 주소 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -br -c addr; echo; done

✅ 출력

>> node 43.202.57.204 <<
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.1.193/24 metric 1024 fe80::23:93ff:fea6:bc61/64 
eni01a4864c88a@if3 UP             fe80::54be:6aff:febd:2d2a/64 
ens6             UP             192.168.1.237/24 fe80::49:8eff:fee5:f105/64 
eni61c5a949744@if3 UP             fe80::5041:42ff:fec3:33f/64 

>> node 15.164.179.214 <<
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.2.52/24 metric 1024 fe80::4c5:5bff:fed1:7757/64 
enibce2df30e87@if3 UP             fe80::48c8:23ff:feab:8539/64 
ens6             UP             192.168.2.42/24 fe80::491:4bff:fe4c:9ce9/64 
enic99196c7a64@if3 UP             fe80::8800:6eff:fe37:b171/64 

>> node 43.201.115.81 <<
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.3.72/24 metric 1024 fe80::8d5:3dff:fe4a:54bd/64 

(3) 네트워크 인터페이스 상세 정보 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done

✅ 출력

>> node 43.202.57.204 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:93:a6:bc:61 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.1.193/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5
       valid_lft 2264sec preferred_lft 2264sec
    inet6 fe80::23:93ff:fea6:bc61/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni01a4864c88a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 56:be:6a:bd:2d:2a brd ff:ff:ff:ff:ff:ff link-netns cni-a59ccd2b-5db2-0159-b78e-e0797b300a23
    inet6 fe80::54be:6aff:febd:2d2a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:49:8e:e5:f1:05 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.1.237/24 brd 192.168.1.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::49:8eff:fee5:f105/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: eni61c5a949744@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 52:41:42:c3:03:3f brd ff:ff:ff:ff:ff:ff link-netns cni-74caca86-36e2-a922-2920-c2c8c00e7b43
    inet6 fe80::5041:42ff:fec3:33f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 15.164.179.214 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:c5:5b:d1:77:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.2.52/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5
       valid_lft 2265sec preferred_lft 2265sec
    inet6 fe80::4c5:5bff:fed1:7757/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: enibce2df30e87@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 4a:c8:23:ab:85:39 brd ff:ff:ff:ff:ff:ff link-netns cni-5408bb28-ad79-a3f3-3a60-9442968852b1
    inet6 fe80::48c8:23ff:feab:8539/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:91:4b:4c:9c:e9 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.2.42/24 brd 192.168.2.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::491:4bff:fe4c:9ce9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: enic99196c7a64@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 8a:00:6e:37:b1:71 brd ff:ff:ff:ff:ff:ff link-netns cni-9343fb30-bdc8-5fab-b46d-3a5db58f8007
    inet6 fe80::8800:6eff:fe37:b171/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 43.201.115.81 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:d5:3d:4a:54:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.72/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 2270sec preferred_lft 2270sec
    inet6 fe80::8d5:3dff:fe4a:54bd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

---

📡 보조 IPv4 주소를 파드가 사용하는지 확인

1. CoreDNS 파드 IP 확인

kubectl get pod -n kube-system -l k8s-app=kube-dns -owide

✅ 출력

CoreDNS 파드가 보조 IPv4 주소(192.168.2.248)를 사용하고 있음

NAME                       READY   STATUS    RESTARTS   AGE     IP              NODE                                               NOMINATED NODE   READINESS GATES
coredns-86f5954566-gqf97   1/1     Running   0          5h39m   192.168.1.59    ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
coredns-86f5954566-nntgz   1/1     Running   0          5h39m   192.168.2.248   ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>

2. 라우팅 테이블 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done

✅ 출력

>> node 43.202.57.204 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.193 metric 1024 
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.193 metric 1024 
192.168.1.59 dev eni61c5a949744 scope link 
192.168.1.90 dev eni01a4864c88a scope link 

>> node 15.164.179.214 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.52 metric 1024 
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.52 metric 1024 
192.168.2.72 dev enibce2df30e87 scope link 
192.168.2.248 dev enic99196c7a64 scope link 

>> node 43.201.115.81 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.72 metric 1024 
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.72 metric 1024 

3. ENI 증가와 Pod IP 할당 확인 실습

(1) 각 노드에서 ENI와 라우팅 테이블을 모니터링

# 터미널 1
ssh ec2-user@$N1
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"

# 터미널 2
ssh ec2-user@$N2
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"

# 터미널 3
ssh ec2-user@$N3
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"

(2) netshoot-pod 디플로이먼트 생성

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netshoot-pod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: netshoot-pod
  template:
    metadata:
      labels:
        app: netshoot-pod
    spec:
      containers:
      - name: netshoot-pod
        image: nicolaka/netshoot
        command: ["tail"]
        args: ["-f", "/dev/null"]
        terminationGracePeriodSeconds: 0
EOF

netshot pod 생성 직후

netshot pod 생성 완료

(3) 파드 생성 후 IP 확인

k get pod -owide

✅ 출력

NAME                            READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
netshoot-pod-744bd84b46-4cfhr   1/1     Running   0          69s   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
netshoot-pod-744bd84b46-hkbd6   1/1     Running   0          69s   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
netshoot-pod-744bd84b46-pzv6d   1/1     Running   0          69s   192.168.2.226   ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>

4. 파드 이름 변수 지정

PODNAME1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].metadata.name}')
PODNAME2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].metadata.name}')
PODNAME3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].metadata.name}')

5. 노드 라우팅 정보 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done

✅ 출력

NAME                            READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
netshoot-pod-744bd84b46-4cfhr   1/1     Running   0          95m   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
netshoot-pod-744bd84b46-hkbd6   1/1     Running   0          95m   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
netshoot-pod-744bd84b46-pzv6d   1/1     Running   0          95m   192.168.2.226   ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>
NAME                            IP
netshoot-pod-744bd84b46-4cfhr   192.168.3.146
netshoot-pod-744bd84b46-hkbd6   192.168.1.176
netshoot-pod-744bd84b46-pzv6d   192.168.2.226

>> node 43.202.57.204 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.193 metric 1024 
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.193 metric 1024 
192.168.1.59 dev eni61c5a949744 scope link 
192.168.1.90 dev eni01a4864c88a scope link 
192.168.1.176 dev eni1d5278cfd98 scope link 

>> node 15.164.179.214 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.52 metric 1024 
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.52 metric 1024 
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.52 metric 1024 
192.168.2.72 dev enibce2df30e87 scope link 
192.168.2.226 dev enia1aaba2012e scope link 
192.168.2.248 dev enic99196c7a64 scope link 

>> node 43.201.115.81 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.72 metric 1024 
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.72 metric 1024 
192.168.3.146 dev enia32f5632a44 scope link 

6. Pod IP 확인과 네트워크 네임스페이스 진입

(1) Pod IP 및 ENI 연결 확인

[ec2-user@ip-192-168-3-72 ~]$ ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:d5:3d:4a:54:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.72/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 3318sec preferred_lft 3318sec
    inet6 fe80::8d5:3dff:fe4a:54bd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: enia32f5632a44@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 76:e0:80:d5:57:23 brd ff:ff:ff:ff:ff:ff link-netns cni-48fae262-f8c0-ab50-61f1-b90d6344980d
    inet6 fe80::74e0:80ff:fed5:5723/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:da:c7:72:d0:e1 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.3.120/24 brd 192.168.3.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::8da:c7ff:fe72:d0e1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

ENI(enia32f5632a44@if3)가 추가되어 있으며, Pod 네트워크 네임스페이스와 연결됨

3: enia32f5632a44@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 76:e0:80:d5:57:23 brd ff:ff:ff:ff:ff:ff link-netns cni-48fae262-f8c0-ab50-61f1-b90d6344980d
    inet6 fe80::74e0:80ff:fed5:5723/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(2) lsns 명령어로 네트워크 네임스페이스 확인

pause 컨테이너의 PID(110946)를 확인

[ec2-user@ip-192-168-3-72 ~]$ sudo lsns -t net
        NS TYPE NPROCS    PID USER     NETNSID NSFS                                                COMMAND
4026531840 net     115      1 root  unassigned                                                     /usr/li
4026532216 net       2 110946 65535          0 /run/netns/cni-48fae262-f8c0-ab50-61f1-b90d6344980d /pause

(3) nsenter 명령어로 네트워크 네임스페이스 진입

[ec2-user@ip-192-168-3-72 ~]$ PID=110946
[ec2-user@ip-192-168-3-72 ~]$ sudo nsenter -t $PID -n ip -c addr

✅ 출력

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
3: eth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether e2:ab:55:84:93:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.3.146/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0ab:55ff:fe84:93dc/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(4) Pod IP 확인 결과

• Pod IP(192.168.3.146)가 출력되며, 이는 서버 IP가 아닌 Pod IP임
• Pod의 인터페이스 eth0@if3와 ENI enia32f5632a44@if3가 연결되어 있음

k get pod -owide

✅ 출력

NAME                            READY   STATUS    RESTARTS   AGE    IP              NODE                                               NOMINATED NODE   READINESS GATES
netshoot-pod-744bd84b46-4cfhr   1/1     Running   0          114m   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
netshoot-pod-744bd84b46-hkbd6   1/1     Running   0          114m   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
netshoot-pod-744bd84b46-pzv6d   1/1     Running   0          114m   192.168.2.226   ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>

7. Pod에 Zsh로 진입하여 네트워크 정보 확인

(1) Pod 목록 확인

k get pod

✅ 출력

NAME                            READY   STATUS    RESTARTS   AGE
netshoot-pod-744bd84b46-4cfhr   1/1     Running   0          117m
netshoot-pod-744bd84b46-hkbd6   1/1     Running   0          117m
netshoot-pod-744bd84b46-pzv6d   1/1     Running   0          117m

(2) Zsh로 Pod 내부 진입 및 네트워크 확인

kubectl exec -it $PODNAME1 -- zsh

# 결과
                    dP            dP                           dP   
                    88            88                           88   
88d888b. .d8888b. d8888P .d8888b. 88d888b. .d8888b. .d8888b. d8888P 
88'  `88 88ooood8   88   Y8ooooo. 88'  `88 88'  `88 88'  `88   88   
88    88 88.  ...   88         88 88    88 88.  .88 88.  .88   88   
dP    dP `88888P'   dP   `88888P' dP    dP `88888P' `88888P'   dP   
                                                                    
Welcome to Netshoot! (github.com/nicolaka/netshoot)
Version: 0.13

Netshoot 컨테이너에 진입 후, ip -c addr로 네트워크 정보 확인

netshoot-pod-744bd84b46-4cfhr [ ~ ] ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
3: eth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether e2:ab:55:84:93:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.3.146/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e0ab:55ff:fe84:93dc/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(3) Pod 내부 라우팅 정보 확인

netshoot-pod-744bd84b46-4cfhr [ ~ ] cat /etc/resolv.conf

✅ 출력

search default.svc.cluster.local svc.cluster.local cluster.local ap-northeast-2.compute.internal
nameserver 10.100.0.10
options ndots:5

8. Pod 정보 확인

k get pod -owide

✅ 출력

NAME                            READY   STATUS    RESTARTS   AGE    IP              NODE                                               NOMINATED NODE   READINESS GATES
netshoot-pod-744bd84b46-4cfhr   1/1     Running   0          3h3m   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
netshoot-pod-744bd84b46-hkbd6   1/1     Running   0          3h3m   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
netshoot-pod-744bd84b46-pzv6d   1/1     Running   0          3h3m   192.168.2.226   ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>

9. 파드간 통신 테스트

(1) 파드 IP 변수 지정

PODIP1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].status.podIP}')
PODIP2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].status.podIP}')
PODIP3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].status.podIP}')

(2) 파드1에서 파드2로 ping 테스트

kubectl exec -it $PODNAME1 -- ping -c 2 $PODIP2

✅ 출력

PING 192.168.1.176 (192.168.1.176) 56(84) bytes of data.
64 bytes from 192.168.1.176: icmp_seq=1 ttl=125 time=1.38 ms
64 bytes from 192.168.1.176: icmp_seq=2 ttl=125 time=1.23 ms

--- 192.168.1.176 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.225/1.303/1.381/0.078 ms

(3) 통신 확인을 위한 패킷 캡처

ICMP 패킷만 캡처하는 명령 사용

[ec2-user@ip-192-168-1-193 ~]$ sudo tcpdump -i any -nn icmp
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

[ec2-user@ip-192-168-2-52 ~]$ sudo tcpdump -i any -nn icmp
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

[ec2-user@ip-192-168-3-72 ~]$ sudo tcpdump -i any -nn icmp
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

(4) Pod 간 Ping 테스트

kubectl exec -it $PODNAME1 -- ping -c 2 $PODIP2

13:00:13.795567 ens5  In  IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 1, length 64
13:00:13.795632 eni1d5278cfd98 Out IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 1, length 64
13:00:13.795646 eni1d5278cfd98 In  IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 1, length 64
13:00:13.795654 ens5  Out IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 1, length 64
13:00:14.796938 ens5  In  IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 2, length 64
13:00:14.796970 eni1d5278cfd98 Out IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 2, length 64
13:00:14.796989 eni1d5278cfd98 In  IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 2, length 64
13:00:14.796996 ens5  Out IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 2, length 64

13:00:13.794944 enia32f5632a44 In  IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 1, length 64
13:00:13.794990 ens5  Out IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 1, length 64
13:00:13.796201 ens5  In  IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 1, length 64
13:00:13.796236 enia32f5632a44 Out IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 1, length 64
13:00:14.796330 enia32f5632a44 In  IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 2, length 64
13:00:14.796363 ens5  Out IP 192.168.3.146 > 192.168.1.176: ICMP echo request, id 110, seq 2, length 64
13:00:14.797526 ens5  In  IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 2, length 64
13:00:14.797542 enia32f5632a44 Out IP 192.168.1.176 > 192.168.3.146: ICMP echo reply, id 110, seq 2, length 64

tcpdump 결과 분석

• ens5와 eni1d5278cfd98 인터페이스를 통해 ICMP 요청과 응답이 오고감
• NAT 없이 원본 IP로 통신됨

10. 네트워크 경로 확인

(1) ip rule 확인

[ec2-user@ip-192-168-1-193 ~]$ ip rule

✅ 출력

0:	from all lookup local
512:	from all to 192.168.1.90 lookup main
512:	from all to 192.168.1.59 lookup main
512:	from all to 192.168.1.176 lookup main
1024:	from all fwmark 0x80/0x80 lookup main
32766:	from all lookup main
32767:	from all lookup default

(2) ip route 테이블 확인

각 인터페이스의 로컬 IP 정보 확인

[ec2-user@ip-192-168-1-193 ~]$ ip route show table local

✅ 출력

local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 
local 192.168.1.193 dev ens5 proto kernel scope host src 192.168.1.193 
local 192.168.1.237 dev ens6 proto kernel scope host src 192.168.1.237 
broadcast 192.168.1.255 dev ens5 proto kernel scope link src 192.168.1.193 
broadcast 192.168.1.255 dev ens6 proto kernel scope link src 192.168.1.237

기본 게이트웨이 및 ENI별 연결 정보 확인

[ec2-user@ip-192-168-1-193 ~]$ ip route show table main

✅ 출력

default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.193 metric 1024 
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.193 metric 1024 
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.193 metric 1024 
192.168.1.59 dev eni61c5a949744 scope link 
192.168.1.90 dev eni01a4864c88a scope link 
192.168.1.176 dev eni1d5278cfd98 scope link

---

📶 파드에서 외부 통신 테스트 및 확인

• Pod에서 외부로 ping 테스트하며 NAT 과정을 확인
• Pod는 내부 IP로 통신하지만, 외부로 나갈 때는 서버의 유동 공인 IP로 NAT되어 통신됨

1. Pod에서 외부로 ping 테스트

kubectl exec -it $PODNAME1 -- ping -c 1 www.google.com

✅ 출력

13:35:28.230157 enia32f5632a44 In  IP 192.168.3.146 > 172.217.161.228: ICMP echo request, id 116, seq 1, length 64
13:35:28.230179 ens5  Out IP 192.168.3.72 > 172.217.161.228: ICMP echo request, id 35777, seq 1, length 64
13:35:28.256496 ens5  In  IP 172.217.161.228 > 192.168.3.72: ICMP echo reply, id 35777, seq 1, length 64
13:35:28.256529 enia32f5632a44 Out IP 172.217.161.228 > 192.168.3.146: ICMP echo reply, id 116, seq 1, length 64

2. 외부로 지속적 ping 테스트

kubectl exec -it $PODNAME1 -- ping -i 0.1 www.google.com

64 bytes from sin01s16-in-f4.1e100.net (172.217.25.164): icmp_seq=174 ttl=47 time=35.0 ms

veth(enia32f5632a44)에서는 Pod IP가 보이지만, ens5에서는 서버 IP가 보이며 NAT가 적용됨

13:37:01.153999 ens5  In  IP 172.217.25.164 > 192.168.3.72: ICMP echo reply, id 48046, seq 174, length 64
13:37:01.154037 enia32f5632a44 Out IP 172.217.25.164 > 192.168.3.146: ICMP echo reply, id 122, seq 174, length 64

3. 서버 공인 IP 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i curl -s ipinfo.io/ip; echo; echo; done

✅ 출력

출력된 IP는 서버의 유동 공인 IP

>> node 43.202.57.204 <<
43.202.57.204

>> node 15.164.179.214 <<
15.164.179.214

>> node 43.201.115.81 <<
43.201.115.81

Pod가 외부 통신할 때 서버의 공인 IP로 NAT되어 나감

4. Pod 외부 인터넷 통신 시 NAT 확인

for i in $PODNAME1 $PODNAME2 $PODNAME3; do echo ">> Pod : $i <<"; kubectl exec -it $i -- curl -s ipinfo.io/ip; echo; echo; done

✅ 출력

>> Pod : netshoot-pod-744bd84b46-4cfhr <<
43.201.115.81

>> Pod : netshoot-pod-744bd84b46-hkbd6 <<
43.202.57.204

>> Pod : netshoot-pod-744bd84b46-pzv6d <<
15.164.179.214

5. 라우팅 테이블 확인

서버의 네트워크 인터페이스 및 라우팅 테이블 확인

ssh ec2-user@$N3

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 11:05:36 2025 from 182.230.60.93
[ec2-user@ip-192-168-3-72 ~]$ 

(1) 네트워크 인터페이스 정보 확인

[ec2-user@ip-192-168-3-72 ~]$ ip -c addr

✅ 출력

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:d5:3d:4a:54:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.72/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 2513sec preferred_lft 2513sec
    inet6 fe80::8d5:3dff:fe4a:54bd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: enia32f5632a44@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 76:e0:80:d5:57:23 brd ff:ff:ff:ff:ff:ff link-netns cni-48fae262-f8c0-ab50-61f1-b90d6344980d
    inet6 fe80::74e0:80ff:fed5:5723/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:da:c7:72:d0:e1 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.3.120/24 brd 192.168.3.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::8da:c7ff:fe72:d0e1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(2) 라우팅 테이블 확인

[ec2-user@ip-192-168-3-72 ~]$ ip route show table main

✅ 출력

default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.72 metric 1024 
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.72 metric 1024 
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.72 metric 1024 
192.168.3.146 dev enia32f5632a44 scope link

• 기본 경로(ens5)로 나갈 때 iptables의 SNAT 규칙이 적용되어 서버 IP(192.168.3.72)로 변환됨

(3) iptables 규칙 확인

[ec2-user@ip-192-168-3-72 ~]$ sudo iptables -t nat -S | grep 'A AWS-SNAT-CHAIN'
-A AWS-SNAT-CHAIN-0 -d 192.168.0.0/16 -m comment --comment "AWS SNAT CHAIN" -j RETURN
-A AWS-SNAT-CHAIN-0 ! -o vlan+ -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.3.72 --random-fully

• 192.168.0.0/16 대역(VPC 내부)은 SNAT 없이 통신하고, 외부 통신 시에는 SNAT 규칙 적용

(4) Pod 간 통신 시 NAT 미적용 이유

• VPC 내부 대역은 NAT 없이 통신하며, 외부 인터넷 대역은 SNAT 적용
• 같은 VPC에서 NAT을 적용하면 클라이언트 IP가 감춰지고 오버헤드가 발생하기 때문에 내부망에서는 NAT 없이 바로 통신
• 외부 인터넷 통신은 SNAT --to-source 192.168.3.72로 변환되어 나가도록 설정

6. AWS-SNAT-CHAIN-0의 SNAT 규칙과 Pod 통신 확인

(1) iptables 초기화

sudo iptables -t filter --zero; sudo iptables -t nat --zero; sudo iptables -t mangle --zero; sudo iptables -t raw --zero

(2) watch 명령어로 AWS-SNAT-CHAIN-0, KUBE-POSTROUTING, POSTROUTING 체인 모니터링

watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'

(3) AWS-SNAT-CHAIN-0 규칙 분석

• 192.168.0.0/16 대역은 SNAT 없이 통신(RETURN)
• 외부 대역은 SNAT되어 EC2 노드1 IP 192.168.3.72로 변경

7. conntrack을 이용한 연결 추적

conntrack 명령어로 NAT된 연결 상태 확인

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo conntrack -L -n |grep -v '169.254.169'; echo; done

✅ 출력

conntrack v1.4.5 (conntrack-tools): 
icmp     1 28 src=172.30.66.58 dst=8.8.8.8 type=8 code=0 id=34392 src=8.8.8.8 dst=172.30.85.242 type=0 code=0 id=50705 mark=128 use=1
tcp      6 23 TIME_WAIT src=172.30.66.58 dst=34.117.59.81 sport=58144 dport=80 src=34.117.59.81 dst=172.30.85.242 sport=80 dport=44768 [ASSURED] mark=128 use=1

8. Pod 통신 테스트

operator-host에서 Pod로 ping 테스트

(eks-user:N/A) [root@operator-host ~]# POD1IP=192.168.1.176
(eks-user:N/A) [root@operator-host ~]# ping -c 1 $POD1IP
# 결과
PING 192.168.1.176 (192.168.1.176) 56(84) bytes of data.
64 bytes from 192.168.1.176: icmp_seq=1 ttl=126 time=0.674 ms

--- 192.168.1.176 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.674/0.674/0.674/0.000 ms

VPC 내부에서는 NAT 없이 바로 통신됨

15:07:35.091537 ens5  In  IP 172.20.1.100 > 192.168.1.176: ICMP echo request, id 5445, seq 1, length 64
15:07:35.091600 eni1d5278cfd98 Out IP 172.20.1.100 > 192.168.1.176: ICMP echo request, id 5445, seq 1, length 64
15:07:35.091614 eni1d5278cfd98 In  IP 192.168.1.176 > 172.20.1.100: ICMP echo reply, id 5445, seq 1, length 64
15:07:35.091621 ens5  Out IP 192.168.1.176 > 172.20.1.100: ICMP echo reply, id 5445, seq 1, length 64

VPC 내부에서는 NAT 없이 통신하며 외부 통신 시 AWS-SNAT-CHAIN-0 규칙에 의해 EC2 노드 IP로 SNAT되는 과정을 확인할 수 있음

---

🛜 파드1 → 운영서버 EC2 통신

1. vpc cni env 정보 확인

(eks-user:N/A) [root@operator-host ~]# kubectl get ds aws-node -n kube-system -o json | jq '.spec.template.spec.containers[0].env'

2. 운영서버 EC2 SSH 접속

(eks-user:N/A) [root@operator-host ~]# kubectl exec -it $PODNAME1 -- ping 172.20.1.100

3. 파드1 배치 워커노드에서 tcpdump 확인

tcpdump로 NAT 적용 확인

(eks-user:N/A) [root@operator-host ~]# sudo tcpdump -i any -nn icmp

Pod IP → 서버 IP로 SNAT되어 외부 통신

15:35:03.714838 enia32f5632a44 In  IP 192.168.3.146 > 172.20.1.100: ICMP echo request, id 140, seq 77, length 64
15:35:03.714869 ens5  Out IP 192.168.3.72 > 172.20.1.100: ICMP echo request, id 56175, seq 77, length 64

4. 사내 내부에 연결 확장된 네트워크 대역과 SNAT 없이 통신 가능하게 설정 해보기

(1) 워커노드(192.168.1.193) 접속

ssh ec2-user@$N1

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 09:05:57 2025 from 182.230.60.93
[ec2-user@ip-192-168-1-193 ~]$ 

(2) NAT 정책 확인

[ec2-user@ip-192-168-1-193 ~]$ sudo iptables -t filter --zero; sudo iptables -t nat --zero; sudo iptables -t mangle --zero; sudo iptables -t raw --zero
[ec2-user@ip-192-168-1-193 ~]$ watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'

(3) SNAT 제외 네트워크 대역 설정

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS=172.20.0.0/16
# 결과
daemonset.apps/aws-node env updated

(4) NAT 정책 변경 확인

(5) Ping 테스트로 SNAT 제외 확인

(eks-user:N/A) [root@operator-host ~]# kubectl exec -it $PODNAME1 -- ping 172.20.1.100

15:53:57.159915 ens5  In  IP 172.20.1.100 > 192.168.3.146: ICMP echo reply, id 146, seq 23, length 64
15:53:57.159929 enia32f5632a44 Out IP 172.20.1.100 > 192.168.3.146: ICMP echo reply, id 146, seq 23, length 64

아래의 내용이 추가됨

kubectl get ds aws-node -n kube-system -o json | jq '.spec.template.spec.containers[0].env'
[
	...
  {
    "name": "AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS",
    "value": "172.20.0.0/16"
  }
  ...
]

ens5로 NAT 되던 이전 상태에서 이제는 172.20.0.0/16 대역으로는 NAT 없이 바로 통신됨

(6) iptables 모니터링 명령어 실행 후 AWS-SNAT-CHAIN-0 체인에서 카운트 증가 확인

ssh ec2-user@$N3

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250203:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 14:21:22 2025 from 182.230.60.93
[ec2-user@ip-192-168-3-72 ~]$ watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'

5. 디플로이먼트 삭제로 실습 마무리

kubectl delete deploy netshoot-pod

---

📏 노드에 파드 생성 갯수 제한

1. kube-ops-view 설치

• kube-ops-view는 노드와 파드 정보를 시각화하는 도구

helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=LoadBalancer --set env.TZ="Asia/Seoul" --namespace kube-system

# 설치 완료 시 출력
"geek-cookbook" already exists with the same configuration, skipping
NAME: kube-ops-view
LAST DEPLOYED: Thu Feb 13 01:05:07 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get svc -w kube-ops-view'
  export SERVICE_IP=$(kubectl get svc --namespace kube-system kube-ops-view -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo http://$SERVICE_IP:8080

2. kube-ops-view 접속 URL 확인 (1.5 배율)

kubectl get svc -n kube-system kube-ops-view -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "KUBE-OPS-VIEW URL = http://"$1":8080/#scale=1.5"}'

✅ 출력

KUBE-OPS-VIEW URL = http://a87c9a7b2db234e36a4a71827a7fbe54-363200146.ap-northeast-2.elb.amazonaws.com:8080/#scale=1.5

3. kube-ops-view 서비스 확인

kubectl get svc -n kube-system kube-ops-view

✅ 출력

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP                                                                   PORT(S)          AGE
kube-ops-view   LoadBalancer   10.100.146.32   a87c9a7b2db234e36a4a71827a7fbe54-363200146.ap-northeast-2.elb.amazonaws.com   8080:32597/TCP   19h

• 설치 시 LoadBalancer 서비스가 함께 배포되며, 컨트롤러 매니저가 클라우드 제공자의 로드밸런서를 자동으로 생성
• External IP로 할당된 로드밸런서 도메인을 통해 접속 가능
• 포트는 8080으로 설정

4. 워커노드의 인스턴스 정보 확인

t3 타입의 정보 확인

aws ec2 describe-instance-types --filters Name=instance-type,Values=t3.\* \
 --query "InstanceTypes[].{Type: InstanceType, MaxENI: NetworkInfo.MaximumNetworkInterfaces, IPv4addr: NetworkInfo.Ipv4AddressesPerInterface}" \ 
 --output table

✅ 출력

--------------------------------------
|        DescribeInstanceTypes       |
+----------+----------+--------------+
| IPv4addr | MaxENI   |    Type      |
+----------+----------+--------------+
|  12      |  3       |  t3.large    |
|  6       |  3       |  t3.medium   |
|  15      |  4       |  t3.xlarge   |
|  15      |  4       |  t3.2xlarge  |
|  2       |  2       |  t3.nano     |
|  2       |  2       |  t3.micro    |
|  4       |  3       |  t3.small    |
+----------+----------+--------------+

• EC2 인스턴스 타입에 따라 최대 Pod 배포 개수가 제한됨
• 장착할 수 있는 네트워크 인터페이스(ENI) 개수와 각 ENI에 할당할 수 있는 보조 IP 개수가 이미 정해져 있기 때문

5. t3.medium 인스턴스에서 Pod 최대 개수 계산

• Max ENI(네트워크 인터페이스): 3개
• ENI당 IP 할당 개수: 6개
• 계산 방법: (6 - 1) * 3 = 15
  • 각 ENI에서 1개의 IP는 이미 사용 중이므로 제외
  • 총 15개의 IP를 Pod에 할당 가능
• 추가 고려사항: CNI 데몬셋과 Kube-Proxy가 2개의 IP를 사용 (IP 공유)
• 최종 결과:t3.medium 인스턴스에서는 최대 17개의 Pod 배포 가능

6. 워커노드 상세 정보 확인

각 워커노드에서 최대 17개의 Pod이 배포 가능함을 확인할 수 있음

kubectl describe node | grep Allocatable: -A6

✅ 출력

Allocatable:
  cpu:                1930m
  ephemeral-storage:  27845546346
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3364544Ki
  pods:               17
--
Allocatable:
  cpu:                1930m
  ephemeral-storage:  27845546346
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3364536Ki
  pods:               17
--
Allocatable:
  cpu:                1930m
  ephemeral-storage:  27845546346
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             3364544Ki
  pods:               17

---

📊 최대 파드 생성 및 확인

1. 워커 노드 3대 EC2 모니터링

[ec2-user@ip-192-168-1-193 ~]$ while true; do ip -br -c addr show && echo "--------------" ; date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done

[ec2-user@ip-192-168-2-52 ~]$ while true; do ip -br -c addr show && echo "--------------" ; date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done

[ec2-user@ip-192-168-3-72 ~]$ while true; do ip -br -c addr show && echo "--------------" ; date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done

2. 디플로이먼트 생성 전 상태 확인

--------------
2025-02-13 12:20:21
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.2.52/24 metric 1024 fe80::4c5:5bff:fed1:7757/64 
enibce2df30e87@if3 UP             fe80::48c8:23ff:feab:8539/64 
enic99196c7a64@if3 UP             fe80::8800:6eff:fe37:b171/64 
ens7             UP             192.168.2.138/24 fe80::487:86ff:fe90:b33d/64 
--------------

--------------
2025-02-13 12:21:10
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.3.72/24 metric 1024 fe80::8d5:3dff:fe4a:54bd/64 
enif2c43957bf8@if3 UP             fe80::a0e3:15ff:fe62:4731/64 
ens7             UP             192.168.3.218/24 fe80::80c:3cff:fe65:9c27/64 
--------------

Every 2.0s: kubectl get pods -o wide                 gram88: 09:22:04 PM
                                                           in 0.538s (0)
No resources found in default namespace.

3. 디플로이먼트 생성

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
EOF

4. 디플로이먼트 생성 후 상태 확인

ENI 추가

• eni189e26c0cf7@if3 UP 상태로 네트워크 인터페이스 추가 확인
• eni422a7c2b629@if3 UP 상태로 네트워크 인터페이스 추가 확인

--------------
2025-02-13 12:24:00
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.1.193/24 metric 1024 fe80::23:93ff:fea6:bc61/64 
eni01a4864c88a@if3 UP             fe80::54be:6aff:febd:2d2a/64 
eni61c5a949744@if3 UP             fe80::5041:42ff:fec3:33f/64 
ens7             UP             192.168.1.232/24 fe80::d8:5dff:fe80:2cc9/64 
eni189e26c0cf7@if3 UP             fe80::b463:11ff:fec2:9e12/64 
--------------

--------------
2025-02-13 12:24:36
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens5             UP             192.168.3.72/24 metric 1024 fe80::8d5:3dff:fe4a:54bd/64 
enif2c43957bf8@if3 UP             fe80::a0e3:15ff:fe62:4731/64 
ens7             UP             192.168.3.218/24 fe80::80c:3cff:fe65:9c27/64 
eni422a7c2b629@if3 UP             fe80::b455:d1ff:fed1:d8f6/64 
--------------

Pod 스케줄링 추가

Every 2.0s: kubectl get pods -o wide                 gram88: 09:25:54 PM                                                                               in 0.487s (0)
NAME                                READY   STATUS    RESTARTS   AGE    IP              NODE                                               NOMINATED NODE   READINESS GATES
nginx-deployment-6c8cb99bb9-77rg2   1/1     Running   0          3m5s   192.168.3.170   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
nginx-deployment-6c8cb99bb9-lcfx9   1/1     Running   0          3m5s   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

5. Pod 개수 8개로 증가

kubectl scale deployment nginx-deployment --replicas=8
# 결과
deployment.apps/nginx-deployment scaled

6. Pod 개수 30개로 증가

• Pod 개수 증가로 물리 랜카드(ens6) 추가 → 총 3장 사용
• 랜카드 3장으로 확장되며 각 랜카드의 Private IPv4 할당도 증가

kubectl scale deployment nginx-deployment --replicas=30
# 결과
deployment.apps/nginx-deployment scaled

7. Pod 개수 50개로 증가

kubectl scale deployment nginx-deployment --replicas=50
# 결과
deployment.apps/nginx-deployment scaled

일부 Pod가 Pending 상태로 남아있음

k get pod

✅ 출력

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6c8cb99bb9-4mz8l   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-5lbxg   1/1     Running   0          4m10s
nginx-deployment-6c8cb99bb9-5m868   1/1     Running   0          4m11s
nginx-deployment-6c8cb99bb9-77rg2   1/1     Running   0          8m19s
nginx-deployment-6c8cb99bb9-7f5tw   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-7jvvw   1/1     Running   0          4m10s
nginx-deployment-6c8cb99bb9-7kzcw   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-7nzn6   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-7qchj   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-8s2z8   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-98tfm   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-9pvbx   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-b2qjf   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-bgxsj   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-g54wb   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-gkpcw   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-h8jtz   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-h9ltb   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-hdbx8   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-hh659   1/1     Running   0          4m10s
nginx-deployment-6c8cb99bb9-hmfz8   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-jnm66   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-kbmgh   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-kjkxd   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-kzfk2   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-lcfx9   1/1     Running   0          8m19s
nginx-deployment-6c8cb99bb9-lh6l6   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-lvq6b   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-lxqcr   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-mgqcd   1/1     Running   0          28s
nginx-deployment-6c8cb99bb9-pvhps   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-qbpmp   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-qwh72   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-r58vr   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-s7bsw   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-sdxw7   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-t2h2t   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-tb47g   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-tklxp   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-ttbzh   1/1     Running   0          4m10s
nginx-deployment-6c8cb99bb9-v5dfl   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-v888x   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-vstpp   1/1     Running   0          2m38s
nginx-deployment-6c8cb99bb9-w2dr8   1/1     Running   0          4m10s
nginx-deployment-6c8cb99bb9-ww4c9   0/1     Pending   0          27s
nginx-deployment-6c8cb99bb9-xk74m   1/1     Running   0          2m39s
nginx-deployment-6c8cb99bb9-xn6k7   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-xp5tt   1/1     Running   0          28s
nginx-deployment-6c8cb99bb9-z2jwk   1/1     Running   0          27s
nginx-deployment-6c8cb99bb9-z4649   0/1     Pending   0          27s

IP 미할당으로 인해 네트워크 네임스페이스 준비 불가 상태

k describe pod nginx-deployment-6c8cb99bb9-7nzn6

✅ 출력

Name:             nginx-deployment-6c8cb99bb9-7nzn6
Namespace:        default
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=nginx
                  pod-template-hash=6c8cb99bb9
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/nginx-deployment-6c8cb99bb9
Containers:
  nginx:
    Image:        nginx:alpine
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cnct2 (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  kube-api-access-cnct2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  61s   default-scheduler  0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

8. 디플로이먼트 삭제

kubectl delete deploy nginx-deployment
# 결과
deployment.apps "nginx-deployment" deleted

---

⚖️ AWS LoadBalancer Controller 배포

1. 설치 전 crd 확인

kubectl get crd

✅ 출력

NAME                                         CREATED AT
cninodes.vpcresources.k8s.aws                2025-02-12T03:09:25Z
eniconfigs.crd.k8s.amazonaws.com             2025-02-12T03:13:55Z
policyendpoints.networking.k8s.aws           2025-02-12T03:09:25Z
securitygrouppolicies.vpcresources.k8s.aws   2025-02-12T03:09:25Z

2. Helm Chart 설치

helm repo add eks https://aws.github.io/eks-charts

# 결과
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME
"eks" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "geek-cookbook" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: aws-load-balancer-controller
LAST DEPLOYED: Thu Feb 13 21:53:34 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

3. 설치 확인

kubectl get crd

✅ 출력

ingressclassparams.elbv2.k8s.aws, targetgroupbindings.elbv2.k8s.aws 추가

NAME                                         CREATED AT
cninodes.vpcresources.k8s.aws                2025-02-12T03:09:25Z
eniconfigs.crd.k8s.amazonaws.com             2025-02-12T03:13:55Z
ingressclassparams.elbv2.k8s.aws             2025-02-13T12:53:32Z
policyendpoints.networking.k8s.aws           2025-02-12T03:09:25Z
securitygrouppolicies.vpcresources.k8s.aws   2025-02-12T03:09:25Z
targetgroupbindings.elbv2.k8s.aws            2025-02-13T12:53:32Z

kubectl explain ingressclassparams.elbv2.k8s.aws

✅ 출력

GROUP:      elbv2.k8s.aws
KIND:       IngressClassParams
VERSION:    v1beta1

DESCRIPTION:
    IngressClassParams is the Schema for the IngressClassParams API
    
FIELDS:
  apiVersion	<string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind	<string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata	<ObjectMeta>
    Standard object's metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec	<Object>
    IngressClassParamsSpec defines the desired state of IngressClassParams

GROUP:      elbv2.k8s.aws
KIND:       TargetGroupBinding
VERSION:    v1beta1

DESCRIPTION:
    TargetGroupBinding is the Schema for the TargetGroupBinding API
    
FIELDS:
  apiVersion	<string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

  kind	<string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

  metadata	<ObjectMeta>
    Standard object's metadata. More info:
    https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

  spec	<Object>
    TargetGroupBindingSpec defines the desired state of TargetGroupBinding

  status	<Object>
    TargetGroupBindingStatus defines the observed state of TargetGroupBinding

---

🕸️ 서비스/파드 배포 테스트 with NLB

1. 모니터링

watch -d kubectl get pod,svc,ep,endpointslices

2. 디플로이먼트 & 서비스 생성

cat << EOF > echo-service-nlb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: deploy-websrv
  template:
    metadata:
      labels:
        app: deploy-websrv
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: aews-websrv
        image: k8s.gcr.io/echoserver:1.5
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc-nlb-ip-type
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing 
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8080"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb
kubectl apply -f echo-service-nlb.yaml
deployment.apps/deploy-echo created
service/svc-nlb-ip-type created

3. 로드밸런서 상태 확인

네트워크 로드밸런서가 provisioning 되고 있음

aws elbv2 describe-load-balancers --query 'LoadBalancers[*].State.Code' --output text

✅ 출력

provisioning

4. deploy, pod 정보 확인

kubectl get deploy,pod

✅ 출력

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deploy-echo   2/2     2            2           3m21s

NAME                              READY   STATUS    RESTARTS   AGE
pod/deploy-echo-bf9bdb8bc-qcq6k   1/1     Running   0          3m21s
pod/deploy-echo-bf9bdb8bc-xlpz4   1/1     Running   0          3m21s

5. 서비스, 엔드포인트, 인그레스 클래스 파라미터, 타겟 그룹 바인딩 확인

kubectl get svc,ep,ingressclassparams,targetgroupbindings

✅ 출력

NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP                                                                         PORT(S)        AGE
service/kubernetes        ClusterIP      10.100.0.1       <none>                                                                              443/TCP        33h
service/svc-nlb-ip-type   LoadBalancer   10.100.253.160   k8s-default-svcnlbip-b30242b032-f4b2a1499955f080.elb.ap-northeast-2.amazonaws.com   80:30138/TCP   3m47s

NAME                        ENDPOINTS                              AGE
endpoints/kubernetes        192.168.2.87:443,192.168.3.197:443     33h
endpoints/svc-nlb-ip-type   192.168.1.176:8080,192.168.3.77:8080   3m47s

NAME                                   GROUP-NAME   SCHEME   IP-ADDRESS-TYPE   AGE
ingressclassparams.elbv2.k8s.aws/alb                                           9m48s

NAME                                                               SERVICE-NAME      SERVICE-PORT   TARGET-TYPE   AGE
targetgroupbinding.elbv2.k8s.aws/k8s-default-svcnlbip-f4a394e732   svc-nlb-ip-type   80             ip            3m43s

targetgroupbinding.elbv2.k8s.aws/k8s-default-svcnlbip-f4a394e732는 아래 정보와 매칭됨

Target Group Binding Attribute에서 Deregistration delay는 graceful shutdown과 유사한 개념

• 이 설정은 대상 타겟이 종료되기 전에 연결을 해제하는 시간을 의미

6. Target Group Binding Attribute 수정

echo-service-nlb.yaml 파일을 열어 다음 줄 추가함

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=60

빠른 실습을 위해서 등록 취소 지연(드레이닝 간격) 60초로 수정

7. 배포

kubectl apply -f echo-service-nlb.yaml

# 결과
deployment.apps/deploy-echo unchanged
service/svc-nlb-ip-type configured

Deregistration delay가 60초로 변경됨

네트워크 로드밸런서가 Active 상태로 전환됨

Target Groups에서 Targets가 Healthy 상태로 표시됨

8. 웹 접속 주소 확인

kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "Pod Web URL = http://"$1 }'

✅ 출력

Pod Web URL = http://k8s-default-svcnlbip-b30242b032-f4b2a1499955f080.elb.ap-northeast-2.amazonaws.com

서버 접근 확인됨 (http://k8s-default-svcnlbip-b30242b032-f4b2a1499955f080.elb.ap-northeast-2.amazonaws.com)

9. 파드 로깅 모니터링

Stern으로 파드 로그 모니터링 진행

kubectl stern -l  app=deploy-websrv

분산 접속 테스트

(eks-user:N/A) [root@operator-host ~]# NLB=$(kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
(eks-user:N/A) [root@operator-host ~]# curl -s $NLB

✅ 출력

Hostname: deploy-echo-bf9bdb8bc-xlpz4

Pod Information:
	-no pod information available-

Server values:
	server_version=nginx: 1.13.0 - lua: 10008

Request Information:
	client_address=192.168.2.29
	method=GET
	real path=/
	query=
	request_version=1.1
	request_uri=http://k8s-default-svcnlbip-b30242b032-f4b2a1499955f080.elb.ap-northeast-2.amazonaws.com:8080/

Request Headers:
	accept=*/*
	host=k8s-default-svcnlbip-b30242b032-f4b2a1499955f080.elb.ap-northeast-2.amazonaws.com
	user-agent=curl/8.3.0

Request Body:
	-no body in request-

100번 요청을 보내 각 파드로 분산된 접속 수 확인

(eks-user:N/A) [root@operator-host ~]# for i in {1..100}; do curl -s $NLB | grep Hostname ; done | sort | uniq -c | sort -nr

✅ 출력

     56 Hostname: deploy-echo-bf9bdb8bc-qcq6k
     44 Hostname: deploy-echo-bf9bdb8bc-xlpz4

100번의 요청 중 56번은 deploy-echo-bf9bdb8bc-qcq6k 파드로, 44번은 deploy-echo-bf9bdb8bc-xlpz4 파드로 분산되어 접속됨

10. AWS Console에서 NLB 리스너 및 대상 확인

NLB 도메인으로 접근하면 80번 포트에서 Listener가 요청을 받고, Resource map을 통해 192.168.1.176과 192.168.3.77의 8080번 포트로 전달됨

192.168.1.176과 192.168.3.77은 아래 출력된 파드들의 IP임

k get pod -owide
NAME                          READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
deploy-echo-bf9bdb8bc-qcq6k   1/1     Running   0          27m   192.168.3.77    ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
deploy-echo-bf9bdb8bc-xlpz4   1/1     Running   0          27m   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

11. replicas 확장으로 타겟 개수 증가

replicas를 2개에서 3개로 확장하여 타겟을 3개로 증가시킴

kubectl scale deployment deploy-echo --replicas=3
# 결과
deployment.apps/deploy-echo scaled

12. 실습 리소스 삭제

kubectl delete deploy deploy-echo; kubectl delete svc svc-nlb-ip-type
# 결과
deployment.apps "deploy-echo" deleted
service "svc-nlb-ip-type" deleted

---

🛤️ Ingress 실습

Ingress는 클러스터 내부 서비스(ClusterIP, NodePort, LoadBalancer)를 외부에 HTTP/HTTPS로 노출하는 웹 프록시 역할을 함

1. 게임 파드와 Service, Ingress 배포

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: public.ecr.aws/l6m2t8p7/docker-2048:latest
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
EOF             number: 80e-2048rget-type: ipt-facing

# 결과
namespace/game-2048 created
deployment.apps/deployment-2048 created
service/service-2048 created

2. 모니터링

watch -d kubectl get pod,ingress,svc,ep,endpointslices -n game-2048

✅ 출력

Every 2.0s: kubectl get pod,ingress,svc,ep,endpointslices -n game-2048                                                         gram88: 10:37:51 PM
                                                                                                                                     in 0.718s (0)
NAME                                   READY   STATUS    RESTARTS   AGE
pod/deployment-2048-7df5f9886b-lxtmt   1/1     Running   0          95s
pod/deployment-2048-7df5f9886b-zw8n9   1/1     Running   0          95s

NAME                                     CLASS   HOSTS   ADDRESS                                                                        PORTS   AGE
ingress.networking.k8s.io/ingress-2048   alb     *       k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com   80      95s

NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/service-2048   NodePort   10.100.135.188   <none>        80:31872/TCP   95s

NAME                     ENDPOINTS                          AGE
endpoints/service-2048   192.168.1.176:80,192.168.3.19:80   95s

NAME                                                ADDRESSTYPE   PORTS   ENDPOINTS                    AGE
endpointslice.discovery.k8s.io/service-2048-2h29k   IPv4          80      192.168.3.19,192.168.1.176   95s

3. 생성 확인

game-2048 네임스페이스의 인그레스, 서비스, 엔드포인트, 파드 확인

kubectl get ingress,svc,ep,pod -n game-2048

✅ 출력

NAME                                     CLASS   HOSTS   ADDRESS                                                                        PORTS   AGE
ingress.networking.k8s.io/ingress-2048   alb     *       k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com   80      2m36s

NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/service-2048   NodePort   10.100.135.188   <none>        80:31872/TCP   2m36s

NAME                     ENDPOINTS                          AGE
endpoints/service-2048   192.168.1.176:80,192.168.3.19:80   2m36s

NAME                                   READY   STATUS    RESTARTS   AGE
pod/deployment-2048-7df5f9886b-lxtmt   1/1     Running   0          2m36s
pod/deployment-2048-7df5f9886b-zw8n9   1/1     Running   0          2m36s

game-2048 네임스페이스의 타겟 그룹 바인딩 확인

kubectl get targetgroupbindings -n game-2048

✅ 출력

NAME                               SERVICE-NAME   SERVICE-PORT   TARGET-TYPE   AGE
k8s-game2048-service2-168626cccc   service-2048   80             ip            3m20s

Listener가 80번 포트로 요청을 받아 rules를 통해 대상 그룹으로 전달함

4. Ingress 설정 확인

kubectl describe ingress -n game-2048 ingress-2048
kubectl get ingress -n game-2048 ingress-2048 -o jsonpath="{.status.loadBalancer.ingress[*].hostname}{'\n'}"

✅ 출력

Name:             ingress-2048
Labels:           <none>
Namespace:        game-2048
Address:          k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com
Ingress Class:    alb
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   service-2048:80 (192.168.3.19:80,192.168.1.176:80)
Annotations:  alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/target-type: ip
Events:
  Type    Reason                  Age    From     Message
  ----    ------                  ----   ----     -------
  Normal  SuccessfullyReconciled  8m18s  ingress  Successfully reconciled
k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com

• 모든 요청(*)은 service-2048:80으로 전달됨

Every 2.0s: kubectl get pod,ingress,svc,ep,endpointslices -n game-2048                                                         gram88: 10:46:22 PM
                                                                                                                                     in 0.724s (0)
NAME                                   READY   STATUS    RESTARTS   AGE
pod/deployment-2048-7df5f9886b-lxtmt   1/1     Running   0          10m  
pod/deployment-2048-7df5f9886b-zw8n9   1/1     Running   0          10m  

NAME                                     CLASS   HOSTS   ADDRESS                                                                        PORTS   AGE
ingress.networking.k8s.io/ingress-2048   alb     *       k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com   80      10m  

NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/service-2048   NodePort   10.100.135.188   <none>        80:31872/TCP   10m  

NAME                     ENDPOINTS                          AGE
endpoints/service-2048   192.168.1.176:80,192.168.3.19:80   10m  

NAME                                                ADDRESSTYPE   PORTS   ENDPOINTS                    AGE
endpointslice.discovery.k8s.io/service-2048-2h29k   IPv4          80      192.168.3.19,192.168.1.176   10m  

• service-2048은 NodePort 타입이며 10.100.135.188 할당
• Pod Endpoint IP: 192.168.1.176:80, 192.168.3.19:80
• ALB는 이 두 Pod로 트래픽 전달

5. 게임 접속 URL 확인

kubectl get ingress -n game-2048 ingress-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "Game URL = http://"$1 }'

✅ 출력

Game URL = http://k8s-game2048-ingress2-70d50ce3fd-1110214105.ap-northeast-2.elb.amazonaws.com

6. Pod 대상 타겟 확인

kubectl get pod -n game-2048 -owide

✅ 출력

NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
deployment-2048-7df5f9886b-lxtmt   1/1     Running   0          13m   192.168.3.19    ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
deployment-2048-7df5f9886b-zw8n9   1/1     Running   0          13m   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

• ALB의 대상으로 직접 Pod IP가 사용됨 (NodePort 아님)
• Pod IP: 192.168.3.19 (노드: ip-192-168-3-72), 192.168.1.176 (노드: ip-192-168-1-193)

7. Pod 개수 증가에 따른 타겟 그룹 확장

pod를 3개로 증가시키면 타겟그룹에 하나가 더 추가됨

kubectl scale deployment -n game-2048 deployment-2048 --replicas 3
# 결과
deployment.apps/deployment-2048 scaled

8. 실습 리소스 삭제

kubectl delete ingress ingress-2048 -n game-2048
# 결과
ingress.networking.k8s.io "ingress-2048" deleted

kubectl delete svc service-2048 -n game-2048 && kubectl delete deploy deployment-2048 -n game-2048 && kubectl delete ns game-2048
# 결과
service "service-2048" deleted
deployment.apps "deployment-2048" deleted
namespace "game-2048" deleted

---

🌍 external DNS 실습

1. 도메인 생성과 Zone ID 확인

2. Route 53 도메인 ID 조회 및 변수 지정

MyDomain=gagajin.com

MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text)

변수 확인

echo $MyDomain, $MyDnzHostedZoneId

✅ 출력

gagajin.com /hostedzone/EXAMPLEID123456789

3. 도메인의 A 레코드 값 반복 조회

while true; do aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" | jq ; date ; echo ; sleep 1; done

4. ExternalDNS 배포

curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml

MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -
# 결과
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created

5. 연결 및 모니터링

kubectl get pod -l app.kubernetes.io/name=external-dns -n kube-system

✅ 출력

NAME                           READY   STATUS    RESTARTS   AGE
external-dns-dc4878f5f-98vcp   1/1     Running   0          25s

kubectl logs deploy/external-dns -n kube-system -f

6. 테트리스 게임 배포

테트리스 Deployment 및 서비스 배포

# 테트리스 디플로이먼트 배포
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tetris
  labels:
    app: tetris
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tetris
  template:
    metadata:
      labels:
        app: tetris
    spec:
      containers:
      - name: tetris
        image: bsord/tetris
---
apiVersion: v1
kind: Service
metadata:
  name: tetris
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    #service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"
spec:
  selector:
    app: tetris
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb
EOF

deployment.apps/tetris created
service/tetris created

배포 상태 확인

kubectl get deploy,svc,ep tetris

✅ 출력

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tetris   1/1     1            1           88s

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP                                                                       PORT(S)        AGE
service/tetris   LoadBalancer   10.100.6.207   k8s-default-tetris-b179cd251d-f36283420d6a53f5.elb.ap-northeast-2.amazonaws.com   80:31999/TCP   88s

NAME               ENDPOINTS          AGE
endpoints/tetris   192.168.1.176:80   88s

7. NLB에 도메인 연결

kubectl annotate service tetris "external-dns.alpha.kubernetes.io/hostname=tetris.$MyDomain"
# 결과
service/tetris annotated

8. Route 53에서 도메인 연결 확인

Route 53 > Hosted Zone > gagajin.com에서 tetris.gagajin.com 도메인이 NLB와 매칭된 상태 확인

도메인 연결 확인

• 도메인 값: tetris.gagajin.com
• 매칭된 서비스: k8s-default-tetris-b179cd251d-f36283420d6a53f5.elb.ap-northeast-2.amazonaws.com (NLB)

9. 도메인 조회

dig +short tetris.$MyDomain @8.8.8.8

✅ 출력

43.200.102.104
3.39.61.119
43.202.39.97

10. 도메인 상태 체크

echo -e "My Domain Checker Site1 = https://www.whatsmydns.net/#A/tetris.$MyDomain"
echo -e "My Domain Checker Site2 = https://dnschecker.org/#A/tetris.$MyDomain"

tetris.gagajin.com 접속

11. 리소스 삭제 및 A 레코드 제거 확인

external-dns에 의해 A레코드도 자동 삭제

kubectl delete deploy,svc tetris

---

🗺️ Topology Aware Routing 실습

1. 현재 노드 AZ 배포 확인

kubectl get node --label-columns=topology.kubernetes.io/zone

✅ 출력

NAME                                               STATUS   ROLES    AGE   VERSION               ZONE
ip-192-168-1-193.ap-northeast-2.compute.internal   Ready    <none>   35h   v1.31.4-eks-aeac579   ap-northeast-2a
ip-192-168-2-52.ap-northeast-2.compute.internal    Ready    <none>   35h   v1.31.4-eks-aeac579   ap-northeast-2b
ip-192-168-3-72.ap-northeast-2.compute.internal    Ready    <none>   35h   v1.31.4-eks-aeac579   ap-northeast-2c

2. 테스트를 위한 디플로이먼트와 서비스 배포

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-echo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: deploy-websrv
  template:
    metadata:
      labels:
        app: deploy-websrv
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: websrv
        image: registry.k8s.io/echoserver:1.5
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc-clusterip
spec:
  ports:
    - name: svc-webport
      port: 80
      targetPort: 8080
  selector:
    app: deploy-websrv
  type: ClusterIP
EOF

# 결과
deployment.apps/deploy-echo created
service/svc-clusterip created

확인

kubectl get deploy,svc,ep,endpointslices
kubectl get pod -owide
kubectl get svc,ep svc-clusterip
kubectl get endpointslices -l kubernetes.io/service-name=svc-clusterip
kubectl get endpointslices -l kubernetes.io/service-name=svc-clusterip -o yaml

✅ 출력

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deploy-echo   3/3     3            3           34s

NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes      ClusterIP   10.100.0.1       <none>        443/TCP   35h
service/svc-clusterip   ClusterIP   10.100.101.191   <none>        80/TCP    34s

NAME                      ENDPOINTS                                                 AGE
endpoints/kubernetes      192.168.2.87:443,192.168.3.197:443                        35h
endpoints/svc-clusterip   192.168.1.176:8080,192.168.2.91:8080,192.168.3.146:8080   34s

NAME                                                 ADDRESSTYPE   PORTS   ENDPOINTS                                  AGE
endpointslice.discovery.k8s.io/kubernetes            IPv4          443     192.168.2.87,192.168.3.197                 35h
endpointslice.discovery.k8s.io/svc-clusterip-8gc6m   IPv4          8080    192.168.1.176,192.168.2.91,192.168.3.146   34s
NAME                           READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
deploy-echo-75b7b9558c-8kxhk   1/1     Running   0          34s   192.168.2.91    ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>
deploy-echo-75b7b9558c-dwhmg   1/1     Running   0          34s   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
deploy-echo-75b7b9558c-frsbd   1/1     Running   0          34s   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/svc-clusterip   ClusterIP   10.100.101.191   <none>        80/TCP    35s

NAME                      ENDPOINTS                                                 AGE
endpoints/svc-clusterip   192.168.1.176:8080,192.168.2.91:8080,192.168.3.146:8080   35s
NAME                  ADDRESSTYPE   PORTS   ENDPOINTS                                  AGE
svc-clusterip-8gc6m   IPv4          8080    192.168.1.176,192.168.2.91,192.168.3.146   36s
apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 192.168.1.176
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-192-168-1-193.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-frsbd
      namespace: default
      uid: c79e9a50-6fe7-4862-b235-f48410dfd4e7
    zone: ap-northeast-2a
  - addresses:
    - 192.168.2.91
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-192-168-2-52.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-8kxhk
      namespace: default
      uid: fc2d6025-50e4-488d-844d-b74e64656869
    zone: ap-northeast-2b
  - addresses:
    - 192.168.3.146
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-192-168-3-72.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-dwhmg
      namespace: default
      uid: 013991cc-6c6a-465a-be3c-2a7128ed4869
    zone: ap-northeast-2c
  kind: EndpointSlice
  metadata:
    annotations:
      endpoints.kubernetes.io/last-change-trigger-time: "2025-02-13T15:02:52Z"
    creationTimestamp: "2025-02-13T15:02:50Z"
    generateName: svc-clusterip-
    generation: 4
    labels:
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: svc-clusterip
    name: svc-clusterip-8gc6m
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: svc-clusterip
      uid: f97f9347-cd50-40d7-881e-253b2efe5825
    resourceVersion: "437737"
    uid: 2f964436-d4df-49bd-af2d-b11ec22702b2
  ports:
  - name: svc-webport
    port: 8080
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""

4. 접속 테스트를 수행할 클라이언트 파드 배포

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: netshoot-pod
spec:
  containers:
  - name: netshoot-pod
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF
pod/netshoot-pod created

확인

kubectl get pod -owide

✅ 출력

NAME                           READY   STATUS    RESTARTS   AGE    IP              NODE                                               NOMINATED NODE   READINESS GATES
deploy-echo-75b7b9558c-8kxhk   1/1     Running   0          101s   192.168.2.91    ip-192-168-2-52.ap-northeast-2.compute.internal    <none>           <none>
deploy-echo-75b7b9558c-dwhmg   1/1     Running   0          101s   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
deploy-echo-75b7b9558c-frsbd   1/1     Running   0          101s   192.168.1.176   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>
netshoot-pod                   1/1     Running   0          16s    192.168.1.98    ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

• Netshoot Pod가 AZ 1번에 배포됨
• AZ 1번에서 배포된 deploy-echo Pod는 같은 AZ에 있는 netshoot-pod에 연결되면 좋지만, 부하분산 시 다른 AZ의 Pod로도 연결됨

5. 테스트 파드(netshoot-pod)에서 ClusterIP 접속 시 부하분산 확인

kubectl exec -it netshoot-pod -- curl svc-clusterip | grep Hostname

✅ 출력

Hostname: deploy-echo-75b7b9558c-8kxhk
# 또는
Hostname: deploy-echo-75b7b9558c-dwhmg
# 또는
Hostname: deploy-echo-75b7b9558c-frsbd

• netshoot-pod에서 curl svc-clusterip 명령어로 서비스 도메인명을 통해 Pod에 접근
• 서비스 생성 시 ClusterIP라는 가상 IP(VIP)가 생성되어 고정 진입점 역할 수행

100번 반복 접속 : 3개의 파드로 AZ(zone) 상관없이 랜덤 확률 부하분산 동작

kubectl exec -it netshoot-pod -- zsh -c "for i in {1..100}; do curl -s svc-clusterip | grep Hostname; done | sort | uniq -c | sort -nr"

✅ 출력

   43 Hostname: deploy-echo-75b7b9558c-frsbd
   36 Hostname: deploy-echo-75b7b9558c-dwhmg
   21 Hostname: deploy-echo-75b7b9558c-8kxhk

6. iptables 기반 서비스 라우팅 분석

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SERVICES

✅ 출력

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SVC-UAGC4PYEYZJJEW6D  tcp  --  *      *       0.0.0.0/0            10.100.145.143       /* kube-system/aws-load-balancer-webhook-service:webhook-server cluster IP */ tcp dpt:443
  110  6600 KUBE-SVC-KBDEBIL6IU6WL7RF  tcp  --  *      *       0.0.0.0/0            10.100.101.191       /* default/svc-clusterip:svc-webport cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-I7SKRZYQ7PWYV5X7  tcp  --  *      *       0.0.0.0/0            10.100.83.10         /* kube-system/eks-extension-metrics-api:metrics-api cluster IP */ tcp dpt:443
  110 10560 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
    0     0 KUBE-SVC-Z4ANX4WAEWEBLCTM  tcp  --  *      *       0.0.0.0/0            10.100.101.216       /* kube-system/metrics-server:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.100.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-7EJNTS7AENER2WX5  tcp  --  *      *       0.0.0.0/0            10.100.146.32        /* kube-system/kube-ops-view:http cluster IP */ tcp dpt:8080
  302 18120 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

7. KUBE-SERVICES 체인 분석

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF

✅ 출력

Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   45  2700 KUBE-SEP-RSD5LYH4WSEXMEXJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.1.176:8080 */ statistic mode random probability 0.33333333349
   23  1380 KUBE-SEP-NLHUM4JSBL2UEFAX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.2.91:8080 */ statistic mode random probability 0.50000000000
   42  2520 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

• 랜덤 확률 기반으로 트래픽을 각 Pod로 분산
• 각 Pod로 33%, 50% 등의 확률로 트래픽 전달

모든 워커 노드의 iptables 규칙 일관성 확인

ssh ec2-user@$N2 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-RSD5LYH4WSEXMEXJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.1.176:8080 */ statistic mode random probability 0.33333333349
    0     0 KUBE-SEP-NLHUM4JSBL2UEFAX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.2.91:8080 */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

ssh ec2-user@$N3 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-RSD5LYH4WSEXMEXJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.1.176:8080 */ statistic mode random probability 0.33333333349
    0     0 KUBE-SEP-NLHUM4JSBL2UEFAX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.2.91:8080 */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

3개의 SEP는 각각 개별 파드 접속 정보

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SEP-RSD5LYH4WSEXMEXJ
Chain KUBE-SEP-RSD5LYH4WSEXMEXJ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.1.176        0.0.0.0/0            /* default/svc-clusterip:svc-webport */
   45  2700 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport */ tcp to:192.168.1.176:8080

ssh ec2-user@$N2 sudo iptables -v --numeric --table nat --list KUBE-SEP-RSD5LYH4WSEXMEXJ
Chain KUBE-SEP-RSD5LYH4WSEXMEXJ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.1.176        0.0.0.0/0            /* default/svc-clusterip:svc-webport */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport */ tcp to:192.168.1.176:8080

ssh ec2-user@$N3 sudo iptables -v --numeric --table nat --list KUBE-SEP-RSD5LYH4WSEXMEXJ
Chain KUBE-SEP-RSD5LYH4WSEXMEXJ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.1.176        0.0.0.0/0            /* default/svc-clusterip:svc-webport */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport */ tcp to:192.168.1.176:8080

8. Topology Aware Routing 설정

kubectl annotate service svc-clusterip "service.kubernetes.io/topology-mode=auto"
# 결과
service/svc-clusterip annotated

endpointslices 확인

각 Endpoint에 hints로 AZ(zone) 정보 포함됨

kubectl get endpointslices -l kubernetes.io/service-name=svc-clusterip -o yaml
apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 192.168.1.176
    conditions:
      ready: true
      serving: true
      terminating: false
    hints:
      forZones:
      - name: ap-northeast-2a
    nodeName: ip-192-168-1-193.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-frsbd
      namespace: default
      uid: c79e9a50-6fe7-4862-b235-f48410dfd4e7
    zone: ap-northeast-2a
  - addresses:
    - 192.168.2.91
    conditions:
      ready: true
      serving: true
      terminating: false
    hints:
      forZones:
      - name: ap-northeast-2b
    nodeName: ip-192-168-2-52.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-8kxhk
      namespace: default
      uid: fc2d6025-50e4-488d-844d-b74e64656869
    zone: ap-northeast-2b
  - addresses:
    - 192.168.3.146
    conditions:
      ready: true
      serving: true
      terminating: false
    hints:
      forZones:
      - name: ap-northeast-2c
    nodeName: ip-192-168-3-72.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-dwhmg
      namespace: default
      uid: 013991cc-6c6a-465a-be3c-2a7128ed4869
    zone: ap-northeast-2c
  kind: EndpointSlice
  metadata:
    creationTimestamp: "2025-02-13T15:02:50Z"
    generateName: svc-clusterip-
    generation: 5
    labels:
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: svc-clusterip
    name: svc-clusterip-8gc6m
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: svc-clusterip
      uid: f97f9347-cd50-40d7-881e-253b2efe5825
    resourceVersion: "444502"
    uid: 2f964436-d4df-49bd-af2d-b11ec22702b2
  ports:
  - name: svc-webport
    port: 8080
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""

9. 부하 분산 테스트

kubectl exec -it netshoot-pod -- zsh -c "for i in {1..100}; do curl -s svc-clusterip | grep Hostname; done | sort | uniq -c | sort -nr"

✅ 출력

  100 Hostname: deploy-echo-75b7b9558c-frsbd

• 100번 접속 시 동일 AZ의 파드로만 연결
• 크로스 네트워크 비용이 발생하지 않음

10. IPTables 정책 확인

ssh 명령어로 각 노드에서 iptables 확인

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SERVICES

✅ 출력

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
    0     0 KUBE-SVC-UAGC4PYEYZJJEW6D  tcp  --  *      *       0.0.0.0/0            10.100.145.143       /* kube-system/aws-load-balancer-webhook-service:webhook-server cluster IP */ tcp dpt:443
  100  6000 KUBE-SVC-KBDEBIL6IU6WL7RF  tcp  --  *      *       0.0.0.0/0            10.100.101.191       /* default/svc-clusterip:svc-webport cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-I7SKRZYQ7PWYV5X7  tcp  --  *      *       0.0.0.0/0            10.100.83.10         /* kube-system/eks-extension-metrics-api:metrics-api cluster IP */ tcp dpt:443
  100  9600 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-Z4ANX4WAEWEBLCTM  tcp  --  *      *       0.0.0.0/0            10.100.101.216       /* kube-system/metrics-server:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.100.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-7EJNTS7AENER2WX5  tcp  --  *      *       0.0.0.0/0            10.100.146.32        /* kube-system/kube-ops-view:http cluster IP */ tcp dpt:8080
  108  6480 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

KUBE-SVC-KBDEBIL6IU6WL7RF 체인에서 동일 AZ의 파드로만 트래픽 전달 확인

Topology Mode의 hints 사용 시 kube-proxy가 해당 AZ에 있는 Pod로만 트래픽을 분산하도록 규칙이 변경됨

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
# 출력
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  100  6000 KUBE-SEP-RSD5LYH4WSEXMEXJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.1.176:8080 */

ssh ec2-user@$N2 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
# 출력
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-NLHUM4JSBL2UEFAX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.2.91:8080 */

ssh ec2-user@$N3 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
# 출력
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

11. 파드 개수를 1개로 줄여 같은 AZ에 목적지 파드가 없을 경우

(1) 파드 개수 1개로 축소

kubectl scale deployment deploy-echo --replicas 1
# 결과
deployment.apps/deploy-echo scaled

(2) 파드 AZ 확인

kubectl get pod -owide

✅ 출력

NAME                           READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
deploy-echo-75b7b9558c-dwhmg   1/1     Running   0          42m   192.168.3.146   ip-192-168-3-72.ap-northeast-2.compute.internal    <none>           <none>
netshoot-pod                   1/1     Running   0          41m   192.168.1.98    ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

(3) 100번 반복 테스트

kubectl exec -it netshoot-pod -- zsh -c "for i in {1..100}; do curl -s svc-clusterip | grep Hostname; done | sort | uniq -c | sort -nr"

✅ 출력

100번 모두 deploy-echo-75b7b9558c-dwhmg 파드로 접속됨

  100 Hostname: deploy-echo-75b7b9558c-dwhmg

(4) IPTables 정책 확인

각 노드에서 KUBE-SVC-KBDEBIL6IU6WL7RF 체인 확인 시 모두 같은 AZ의 파드로만 연결됨

ssh ec2-user@$N1 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  100  6000 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

ssh ec2-user@$N2 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

ssh ec2-user@$N3 sudo iptables -v --numeric --table nat --list KUBE-SVC-KBDEBIL6IU6WL7RF
Chain KUBE-SVC-KBDEBIL6IU6WL7RF (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-6CT57P6L6QUJZOMH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/svc-clusterip:svc-webport -> 192.168.3.146:8080 */

(5) EndpointSlices 확인

hint 정보 없음

kubectl get endpointslices -l kubernetes.io/service-name=svc-clusterip -o yaml

✅ 출력

apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 192.168.3.146
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: ip-192-168-3-72.ap-northeast-2.compute.internal
    targetRef:
      kind: Pod
      name: deploy-echo-75b7b9558c-dwhmg
      namespace: default
      uid: 013991cc-6c6a-465a-be3c-2a7128ed4869
    zone: ap-northeast-2c
  kind: EndpointSlice
  metadata:
    creationTimestamp: "2025-02-13T15:02:50Z"
    generateName: svc-clusterip-
    generation: 7
    labels:
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: svc-clusterip
    name: svc-clusterip-8gc6m
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: svc-clusterip
      uid: f97f9347-cd50-40d7-881e-253b2efe5825
    resourceVersion: "447576"
    uid: 2f964436-d4df-49bd-af2d-b11ec22702b2
  ports:
  - name: svc-webport
    port: 8080
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""

12. 실습 리소스 삭제

kubectl delete deploy deploy-echo; kubectl delete svc svc-clusterip
# 결과
deployment.apps "deploy-echo" deleted
service "svc-clusterip" deleted

---

🧪 AWS Load Balancer Controller로 블루/그린 배포, 카나리 배포, A/B 테스트 실습

1. 샘플 애플리케이션 클론

(eks-user:N/A) [root@operator-host ~]# git clone https://github.com/paulbouwer/hello-kubernetes.git
# 결과
Cloning into 'hello-kubernetes'...
remote: Enumerating objects: 294, done.
remote: Total 294 (delta 0), reused 0 (delta 0), pack-reused 294 (from 1)
Receiving objects: 100% (294/294), 168.42 KiB | 7.66 MiB/s, done.
Resolving deltas: 100% (120/120), done.

2. 샘플 애플리케이션 v1 설치

(eks-user:N/A) [root@operator-host ~]# helm install --create-namespace --namespace hello-kubernetes v1 \
>   ./hello-kubernetes/deploy/helm/hello-kubernetes \
>   --set message="You are reaching hello-kubernetes version 1" \
>   --set ingress.configured=true \
>   --set service.type="ClusterIP"
# 결과
NAME: v1
LAST DEPLOYED: Fri Feb 14 00:57:28 2025
NAMESPACE: hello-kubernetes
STATUS: deployed
REVISION: 1
TEST SUITE: None

3. 샘플 애플리케이션 v2 설치

(eks-user:N/A) [root@operator-host ~]# helm install --create-namespace --namespace hello-kubernetes v2 \
>   ./hello-kubernetes/deploy/helm/hello-kubernetes \
>   --set message="You are reaching hello-kubernetes version 2" \
>   --set ingress.configured=true \
>   --set service.type="ClusterIP"
# 결과
NAME: v2
LAST DEPLOYED: Fri Feb 14 00:57:59 2025
NAMESPACE: hello-kubernetes
STATUS: deployed
REVISION: 1
TEST SUITE: None

4. 배포 확인

(eks-user:N/A) [root@operator-host ~]# kubectl get pod,svc,ep -n hello-kubernetes

✅ 출력

NAME                                       READY   STATUS    RESTARTS   AGE
pod/hello-kubernetes-v1-7b546f6687-67w5x   1/1     Running   0          92s
pod/hello-kubernetes-v1-7b546f6687-nc877   1/1     Running   0          92s
pod/hello-kubernetes-v2-7b9df8f6c5-4qtd7   1/1     Running   0          60s
pod/hello-kubernetes-v2-7b9df8f6c5-g2jp9   1/1     Running   0          60s

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/hello-kubernetes-v1   ClusterIP   10.100.175.94    <none>        80/TCP    92s
service/hello-kubernetes-v2   ClusterIP   10.100.160.118   <none>        80/TCP    60s

NAME                            ENDPOINTS                               AGE
endpoints/hello-kubernetes-v1   192.168.1.195:8080,192.168.2.237:8080   92s
endpoints/hello-kubernetes-v2   192.168.1.176:8080,192.168.3.146:8080   60s

5. 인그레스 생성

(eks-user:N/A) [root@operator-host ~]# cat <<EOF | kubectl apply -f -
> apiVersion: networking.k8s.io/v1
> kind: Ingress
> metadata:
>   name: "hello-kubernetes"
>   namespace: "hello-kubernetes"
>   annotations:
>     alb.ingress.kubernetes.io/scheme: internet-facing
>     alb.ingress.kubernetes.io/target-type: ip
>     alb.ingress.kubernetes.io/actions.blue-green: |
>       {
>         "type":"forward",
>         "forwardConfig":{
>           "targetGroups":[
>             {
>               "serviceName":"hello-kubernetes-v1",
>               "servicePort":"80",
>               "weight":100
>             },
>             {
>               "serviceName":"hello-kubernetes-v2",
>               "servicePort":"80",
>               "weight":0
>             }
>           ]
>         }
>       }
>   labels:
>     app: hello-kubernetes
> spec:
>   ingressClassName: alb
>   rules:
>     - http:
>         paths:
>           - path: /
>             pathType: Prefix
>             backend:
>               service:
>                 name: blue-green
>                 port:
>                   name: use-annotation
> EOF
ingress.networking.k8s.io/hello-kubernetes created

6. 인그레스 및 포워딩 확인

인그레스 확인

(eks-user:N/A) [root@operator-host ~]# kubectl describe ingress -n hello-kubernetes

✅ 출력

Name:             hello-kubernetes
Labels:           app=hello-kubernetes
Namespace:        hello-kubernetes
Address:          k8s-hellokub-hellokub-7e40b1a1ff-555575423.ap-northeast-2.elb.amazonaws.com
Ingress Class:    alb
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   blue-green:use-annotation (<error: services "blue-green" not found>)
Annotations:  alb.ingress.kubernetes.io/actions.blue-green:
                {
                  "type":"forward",
                  "forwardConfig":{
                    "targetGroups":[
                      {
                        "serviceName":"hello-kubernetes-v1",
                        "servicePort":"80",
                        "weight":100
                      },
                      {
                        "serviceName":"hello-kubernetes-v2",
                        "servicePort":"80",
                        "weight":0
                      }
                    ]
                  }
                }
              alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/target-type: ip
Events:
  Type    Reason                  Age   From     Message
  ----    ------                  ----  ----     -------
  Normal  SuccessfullyReconciled  3m4s  ingress  Successfully reconciled

포워딩 확인

현재 버전 1만 응답

(eks-user:N/A) [root@operator-host ~]# ELB_URL=$(kubectl get ingress -n hello-kubernetes -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
(eks-user:N/A) [root@operator-host ~]# while true; do curl -s $ELB_URL | grep version; sleep 1; done
  You are reaching hello-kubernetes version 1
  You are reaching hello-kubernetes version 1
  You are reaching hello-kubernetes version 1
  You are reaching hello-kubernetes version 1
  You are reaching hello-kubernetes version 1
  You are reaching hello-kubernetes version 1
  ....

ALB의 포워딩 대상 타겟 그룹은 2가지가 있음

• 기존 버전: k8s-hellokub-hellokub-bb26fff580으로 100% 트래픽 전달
• 신규 버전: k8s-hellokub-hellokub-f58a344fd6으로 0% 트래픽 전달

7. 블루/그린 배포

블루/그린 배포 설정

(eks-user:N/A) [root@operator-host ~]# cat <<EOF | kubectl apply -f -
> apiVersion: networking.k8s.io/v1
> kind: Ingress
> metadata:
>   name: "hello-kubernetes"
>   namespace: "hello-kubernetes"
>   annotations:
>     alb.ingress.kubernetes.io/scheme: internet-facing
>     alb.ingress.kubernetes.io/target-type: ip
>     alb.ingress.kubernetes.io/actions.blue-green: |
>       {
>         "type":"forward",
>         "forwardConfig":{
>           "targetGroups":[
>             {
>               "serviceName":"hello-kubernetes-v1",
>               "servicePort":"80",
>               "weight":0
>             },
>             {
>               "serviceName":"hello-kubernetes-v2",
>               "servicePort":"80",
>               "weight":100
>             }
>           ]
>         }
>       }
>   labels:
>     app: hello-kubernetes
> spec:
>   ingressClassName: alb
>   rules:
>     - http:
>         paths:
>           - path: /
>             pathType: Prefix
>             backend:
>               service:
>                 name: blue-green
>                 port:
>                   name: use-annotation
> EOF

# 결과
ingress.networking.k8s.io/hello-kubernetes configured

타겟 그룹이 version 2로 100% 포워딩됨

(eks-user:N/A) [root@operator-host ~]# while true; do curl -s $ELB_URL | grep version; sleep 1; done
  You are reaching hello-kubernetes version 2
  You are reaching hello-kubernetes version 2
  You are reaching hello-kubernetes version 2
  You are reaching hello-kubernetes version 2
  You are reaching hello-kubernetes version 2
  You are reaching hello-kubernetes version 2
  .....

ALB 타겟 그룹 변경 확인

k8s-hellokub-hellokub-f58a344fd6 100으로 바뀌었다.

8. 카나리 배포

카나리 배포 설정

(eks-user:N/A) [root@operator-host ~]# cat <<EOF | kubectl apply -f -
> apiVersion: networking.k8s.io/v1
> kind: Ingress
> metadata:
>   name: "hello-kubernetes"
>   namespace: "hello-kubernetes"
>   annotations:
>     alb.ingress.kubernetes.io/scheme: internet-facing
>     alb.ingress.kubernetes.io/target-type: ip
>     alb.ingress.kubernetes.io/actions.blue-green: |
>       {
>         "type":"forward",
>         "forwardConfig":{
>           "targetGroups":[
>             {
>               "serviceName":"hello-kubernetes-v1",
>               "servicePort":"80",
>               "weight":90
>             },
>             {
>               "serviceName":"hello-kubernetes-v2",
>               "servicePort":"80",
>               "weight":10
>             }
>           ]
>         }
>       }
>   labels:
>     app: hello-kubernetes
> spec:
>   ingressClassName: alb
>   rules:
>     - http:
>         paths:
>           - path: /
>             pathType: Prefix
>             backend:
>               service:
>                 name: blue-green
>                 port:
>                   name: use-annotation
> EOF
# 결과
ingress.networking.k8s.io/hello-kubernetes configured

접속 결과 확인

(eks-user:N/A) [root@operator-host ~]# for i in {1..100};  do curl -s $ELB_URL | grep version ; done | sort | uniq -c | sort -nr
     86   You are reaching hello-kubernetes version 1
     14   You are reaching hello-kubernetes version 2

9. a/b 테스트

A/B 테스트 설정

(eks-user:N/A) [root@operator-host ~]# cat <<EOF | kubectl apply -f -
> apiVersion: networking.k8s.io/v1
> kind: Ingress
> metadata:
>   name: "hello-kubernetes"
>   namespace: "hello-kubernetes"
>   annotations:
>     alb.ingress.kubernetes.io/scheme: internet-facing
>     alb.ingress.kubernetes.io/target-type: ip
>     alb.ingress.kubernetes.io/conditions.ab-testing: >
>       [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "HeaderName", "values":["aews-study"]}}]
>     alb.ingress.kubernetes.io/actions.ab-testing: >
>       {"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"hello-kubernetes-v2","servicePort":80}]}}
>   labels:
>     app: hello-kubernetes
> spec:
>   ingressClassName: alb
>   rules:
>     - http:
>         paths:
>           - path: /
>             pathType: Prefix
>             backend:
>               service:
>                 name: ab-testing
>                 port:
>                   name: use-annotation
>           - path: /
>             pathType: Prefix
>             backend:
>               service:
>                 name: hello-kubernetes-v1
>                 port:
>                   name: http
> EOF
# 결과
ingress.networking.k8s.io/hello-kubernetes configured

• HTTP Header HeaderName: aews-study 조건으로 트래픽 분할

A/B 테스트 확인

HeaderName: aews-study 포함 시 version 2로 100% 포워딩

(eks-user:N/A) [root@operator-host ~]# for i in {1..100};  do curl -s -H "HeaderName: aews-study" $ELB_URL | grep version ; done | sort | uniq -c | sort -nr
    100   You are reaching hello-kubernetes version 2

포함되지 않을 시 version 1로 100% 포워딩

(eks-user:N/A) [root@operator-host ~]# for i in {1..100};  do curl -s $ELB_URL | grep version ; done | sort | uniq -c | sort -nr
    100   You are reaching hello-kubernetes version 1

10. 실습 리소스 삭제

(eks-user:N/A) [root@operator-host ~]# kubectl delete ingress -n hello-kubernetes hello-kubernetes && kubectl delete ns hello-kubernetes
# 결과
ingress.networking.k8s.io "hello-kubernetes" deleted
namespace "hello-kubernetes" deleted

---

🕵️‍♂️ 네트워크 분석 툴

1. Kubeskoop 설치

(eks-user:N/A) [root@operator-host ~]# kubectl apply -f https://raw.githubusercontent.com/alibaba/kubeskoop/main/deploy/skoopbundle.yaml
# 결과
namespace/kubeskoop created
daemonset.apps/kubeskoop-exporter created
configmap/kubeskoop-config created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prometheus-server-conf created
deployment.apps/prometheus-deployment created
service/prometheus-service created
service/loki-service created
configmap/grafana-datasources created
deployment.apps/grafana created
service/grafana created
deployment.apps/grafana-loki created
configmap/grafana-loki-config created
clusterrole.rbac.authorization.k8s.io/kubeskoop-controller created
clusterrolebinding.rbac.authorization.k8s.io/kubeskoop-controller created
role.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/controller created
configmap/kubeskoop-controller-config created
deployment.apps/controller created
service/controller created
deployment.apps/webconsole created
service/webconsole created

(eks-user:N/A) [root@operator-host ~]# kubectl patch service webconsole -n kubeskoop -p '{"spec": {"type": "LoadBalancer"}}'
service/webconsole patched
(eks-user:N/A) [root@operator-host ~]# kubectl patch service prometheus-service -n kubeskoop -p '{"spec": {"type": "LoadBalancer"}}'
service/prometheus-service patched
(eks-user:N/A) [root@operator-host ~]# kubectl patch service grafana -n kubeskoop -p '{"spec": {"type": "LoadBalancer"}}'
service/grafana patched

2. kubeskoop 웹 접속

admin / kubeskoop

(eks-user:N/A) [root@operator-host ~]# kubectl get svc -n kubeskoop webconsole
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP                                                                   PORT(S)        AGE
webconsole   LoadBalancer   10.100.14.77   ac2514347b0c84f0b84e1aa71fe7e0dc-996135232.ap-northeast-2.elb.amazonaws.com   80:32223/TCP   84s

(eks-user:N/A) [root@operator-host ~]# kubectl get svc -n kubeskoop webconsole -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "KubeSkoop URL = http://"$1""}'
KubeSkoop URL = http://ac2514347b0c84f0b84e1aa71fe7e0dc-996135232.ap-northeast-2.elb.amazonaws.com

3. 패킷 캡처링 테스트

netshoot-pod 정보 확인

(eks-user:N/A) [root@operator-host ~]# k get pod -owide
NAME           READY   STATUS    RESTARTS   AGE    IP             NODE                                               NOMINATED NODE   READINESS GATES
netshoot-pod   1/1     Running   0          109m   192.168.1.98   ip-192-168-1-193.ap-northeast-2.compute.internal   <none>           <none>

netshoot-pod ping 테스트

(eks-user:N/A) [root@operator-host ~]# ping 192.168.1.98
PING 192.168.1.98 (192.168.1.98) 56(84) bytes of data.
64 bytes from 192.168.1.98: icmp_seq=1 ttl=126 time=1.08 ms
64 bytes from 192.168.1.98: icmp_seq=2 ttl=126 time=0.644 ms
64 bytes from 192.168.1.98: icmp_seq=3 ttl=126 time=0.649 ms
64 bytes from 192.168.1.98: icmp_seq=4 ttl=126 time=0.680 ms
64 bytes from 192.168.1.98: icmp_seq=5 ttl=126 time=0.767 ms

4. 프로메테우스 웹 접속

(eks-user:N/A) [root@operator-host ~]# kubectl get svc -n kubeskoop prometheus-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "prometheus URL = http://"$1""}'
prometheus URL = http://af6d238bde2c24a829f56fd1abdc6989-1339249886.ap-northeast-2.elb.amazonaws.com

5. 그라파나 웹 접속

admin / kubeskoop

(eks-user:N/A) [root@operator-host ~]# kubectl get svc -n kubeskoop grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "grafana URL = http://"$1""}'
grafana URL = http://a87ee7a3d119b4d0a9a742a9d992375d-114222539.ap-northeast-2.elb.amazonaws.com

🔗 AWS EKS IPVS 모드 설정

IPVS 모드

• IPVS 는 리눅스 커널에서 동작하는 소프트웨어 로드밸런서이다. 백엔드(플랫폼)으로 Netfilter 를 사용하며, TCP/UDP 요청을 처리 할 수 있다.
• iptables 의 rule 기반 처리의 성능 한계와 분산 알고리즘이 없어서, 최근에는 대체로 IPVS 를 사용한다.

1. 서버별 IPVS 모듈 설정

서버 1, 2, 3 각각 IPVS 모듈 설정

[ec2-user@ip-192-168-1-193 ~]$ sudo sh -c 'cat << EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_lc
ip_vs_wlc
ip_vs_lblc
ip_vs_lblcr
ip_vs_sh
ip_vs_dh
ip_vs_sed
ip_vs_nq
nf_conntrack
EOF'

[ec2-user@ip-192-168-1-193 ~]$ sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_lc
sudo modprobe ip_vs_wlc
sudo modprobe ip_vs_lblc
sudo modprobe ip_vs_lblcr
sudo modprobe ip_vs_sh
sudo modprobe ip_vs_dh
sudo modprobe ip_vs_sed
sudo modprobe ip_vs_nq
sudo modprobe nf_conntrack

[ec2-user@ip-192-168-2-52 ~]$ sudo sh -c 'cat << EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_lc
ip_vs_wlc
ip_vs_lblc
ip_vs_lblcr
ip_vs_sh
ip_vs_dh
ip_vs_sed
ip_vs_nq
nf_conntrack
EOF'

[ec2-user@ip-192-168-2-52 ~]$ sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_lc
sudo modprobe ip_vs_wlc
sudo modprobe ip_vs_lblc
sudo modprobe ip_vs_lblcr
sudo modprobe ip_vs_sh
sudo modprobe ip_vs_dh
sudo modprobe ip_vs_sed
sudo modprobe ip_vs_nq
sudo modprobe nf_conntrack

[ec2-user@ip-192-168-3-72 ~]$ sudo sh -c 'cat << EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_lc
ip_vs_wlc
ip_vs_lblc
ip_vs_lblcr
ip_vs_sh
ip_vs_dh
ip_vs_sed
ip_vs_nq
nf_conntrack
EOF'

[ec2-user@ip-192-168-3-72 ~]$ sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_lc
sudo modprobe ip_vs_wlc
sudo modprobe ip_vs_lblc
sudo modprobe ip_vs_lblcr
sudo modprobe ip_vs_sh
sudo modprobe ip_vs_dh
sudo modprobe ip_vs_sed
sudo modprobe ip_vs_nq
sudo modprobe nf_conntrack

2. IPVS 모듈 확인

각 서버에서 lsmod | grep ^ip_vs로 IPVS 모듈이 로드되었는지 확인

[ec2-user@ip-192-168-1-193 ~]$ sudo lsmod | grep ^ip_vs
# 출력
ip_vs_nq               16384  0
ip_vs_sed              16384  0
ip_vs_dh               16384  0
ip_vs_sh               16384  0
ip_vs_lblcr            16384  0
ip_vs_lblc             16384  0
ip_vs_wlc              16384  0
ip_vs_lc               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 192512  20 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed

[ec2-user@ip-192-168-2-52 ~]$ sudo lsmod | grep ^ip_vs
# 출력
ip_vs_nq               16384  0
ip_vs_sed              16384  0
ip_vs_dh               16384  0
ip_vs_sh               16384  0
ip_vs_lblcr            16384  0
ip_vs_lblc             16384  0
ip_vs_wlc              16384  0
ip_vs_lc               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 192512  20 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed

[ec2-user@ip-192-168-3-72 ~]$ sudo lsmod | grep ^ip_vs
# 출력
ip_vs_nq               16384  0
ip_vs_sed              16384  0
ip_vs_dh               16384  0
ip_vs_sh               16384  0
ip_vs_lblcr            16384  0
ip_vs_lblc             16384  0
ip_vs_wlc              16384  0
ip_vs_lc               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 192512  20 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed

3. AWS EKS에 IPVS 모드 반영

aws eks update-addon --cluster-name $CLUSTER_NAME --addon-name kube-proxy \
  --configuration-values '{"ipvs": {"scheduler": "rr"}, "mode": "ipvs"}' \
  --resolve-conflicts OVERWRITE

# 결과
{
    "update": {
        "id": "3e57ed90-40b8-3ebb-98ac-73e497864986",
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            {
                "type": "ResolveConflicts",
                "value": "OVERWRITE"
            },
            {
                "type": "ConfigurationValues",
                "value": "{\"ipvs\": {\"scheduler\": \"rr\"}, \"mode\": \"ipvs\"}"
            }
        ],
        "createdAt": "2025-02-14T02:06:42.661000+09:00",
        "errors": []
    }
}

4. kube-proxy 데몬셋 재시작

kubectl -n kube-system rollout restart ds kube-proxy
# 결과
daemonset.apps/kube-proxy restarted

5. kube-proxy 설정 확인

kubectl get cm -n kube-system kube-proxy-config -o yaml

✅ 출력

mode: "ipvs", scheduler: "rr" 설정 확인

apiVersion: v1
data:
  config: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig
      qps: 5
    clusterCIDR: ""
    configSyncPeriod: 15m0s
    conntrack:
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: "rr"
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 0.0.0.0:10249
    mode: "ipvs"
    nodePortAddresses: null
    oomScoreAdj: -998
    portRange: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2025-02-12T03:13:56Z"
  labels:
    eks.amazonaws.com/component: kube-proxy
    k8s-app: kube-proxy
  name: kube-proxy-config
  namespace: kube-system
  resourceVersion: "466743"
  uid: 83ba2b60-5afb-4ea8-b037-84a02d9036ef

6. kube-ipvs0 인터페이스 생성 확인

IPVS 모드로 변경 시 kube-ipvs0 인터페이스가 생성되어 서비스 virtual IP 할당됨

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done

✅ 출력

>> node 43.202.57.204 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:23:93:a6:bc:61 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.1.193/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5
       valid_lft 2300sec preferred_lft 2300sec
    inet6 fe80::23:93ff:fea6:bc61/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni01a4864c88a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 56:be:6a:bd:2d:2a brd ff:ff:ff:ff:ff:ff link-netns cni-a59ccd2b-5db2-0159-b78e-e0797b300a23
    inet6 fe80::54be:6aff:febd:2d2a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: eni61c5a949744@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 52:41:42:c3:03:3f brd ff:ff:ff:ff:ff:ff link-netns cni-74caca86-36e2-a922-2920-c2c8c00e7b43
    inet6 fe80::5041:42ff:fec3:33f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
14: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:d8:5d:80:2c:c9 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 192.168.1.232/24 brd 192.168.1.255 scope global ens7
       valid_lft forever preferred_lft forever
    inet6 fe80::d8:5dff:fe80:2cc9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
38: enica32516c01a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 92:2a:72:57:7d:03 brd ff:ff:ff:ff:ff:ff link-netns cni-f0cb87e6-1f26-e2c9-b622-9a99215f4cca
    inet6 fe80::902a:72ff:fe57:7d03/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
41: enifc4d699b169@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether b6:46:07:f2:a2:65 brd ff:ff:ff:ff:ff:ff link-netns cni-2c0732df-1963-6937-d7ad-2ea9fe9f3481
    inet6 fe80::b446:7ff:fef2:a265/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
42: enid72dce04775@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 5e:54:5b:e7:37:87 brd ff:ff:ff:ff:ff:ff link-netns cni-50b39a65-fb46-3e73-0831-6f9b159d1c56
    inet6 fe80::5c54:5bff:fee7:3787/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
43: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether ce:47:f1:9e:78:ad brd ff:ff:ff:ff:ff:ff
    inet 10.100.110.88/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.214.208/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.73.70/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.17.159/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.14.77/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.145.143/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.83.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.146.32/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.101.216/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

>> node 15.164.179.214 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:c5:5b:d1:77:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.2.52/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5
       valid_lft 2295sec preferred_lft 2295sec
    inet6 fe80::4c5:5bff:fed1:7757/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: enibce2df30e87@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 4a:c8:23:ab:85:39 brd ff:ff:ff:ff:ff:ff link-netns cni-5408bb28-ad79-a3f3-3a60-9442968852b1
    inet6 fe80::48c8:23ff:feab:8539/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: enic99196c7a64@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 8a:00:6e:37:b1:71 brd ff:ff:ff:ff:ff:ff link-netns cni-9343fb30-bdc8-5fab-b46d-3a5db58f8007
    inet6 fe80::8800:6eff:fe37:b171/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
28: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:0e:23:61:2c:f9 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.2.136/24 brd 192.168.2.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::40e:23ff:fe61:2cf9/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
34: eni79cb46fcdac@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether c6:d6:b9:a4:cf:99 brd ff:ff:ff:ff:ff:ff link-netns cni-188308ca-6087-e1e0-f4f3-7e6fd2da16e3
    inet6 fe80::c4d6:b9ff:fea4:cf99/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
39: enib1a09dbf60e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 76:23:31:29:2f:9f brd ff:ff:ff:ff:ff:ff link-netns cni-cb0b81af-c51f-85de-47fa-5036f76d4745
    inet6 fe80::7423:31ff:fe29:2f9f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
40: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 82:2e:0b:7a:92:81 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.17.159/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.73.70/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.110.88/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.146.32/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.83.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.145.143/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.101.216/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.214.208/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.14.77/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

>> node 43.201.115.81 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:d5:3d:4a:54:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.72/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 2304sec preferred_lft 2304sec
    inet6 fe80::8d5:3dff:fe4a:54bd/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: enif2c43957bf8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether a2:e3:15:62:47:31 brd ff:ff:ff:ff:ff:ff link-netns cni-8acd7723-2d0b-690f-3d4b-d3e902287dd1
    inet6 fe80::a0e3:15ff:fe62:4731/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
14: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:0c:3c:65:9c:27 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 192.168.3.218/24 brd 192.168.3.255 scope global ens7
       valid_lft forever preferred_lft forever
    inet6 fe80::80c:3cff:fe65:9c27/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
35: eniaad872f8f96@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 5e:71:1d:0d:58:76 brd ff:ff:ff:ff:ff:ff link-netns cni-83401432-a5a6-e355-338b-5b0b2be54e2f
    inet6 fe80::5c71:1dff:fe0d:5876/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
39: eni1a06225ba03@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 6e:99:bf:09:eb:4a brd ff:ff:ff:ff:ff:ff link-netns cni-76c37d54-4b57-554c-2033-1bd0903ea10a
    inet6 fe80::6c99:bfff:fe09:eb4a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
42: eni00c25e070e3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 5a:70:f9:2d:fb:d6 brd ff:ff:ff:ff:ff:ff link-netns cni-ea99b434-526f-c3dc-8f3c-312ea67461e7
    inet6 fe80::5870:f9ff:fe2d:fbd6/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
43: enif2b59b1644b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 7e:a5:ca:a5:92:ae brd ff:ff:ff:ff:ff:ff link-netns cni-d32736a6-d35d-b2bf-009d-446df94aa7b8
    inet6 fe80::7ca5:caff:fea5:92ae/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
44: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 2a:28:b6:08:eb:09 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.73.70/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.101.216/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.14.77/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.145.143/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.214.208/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.110.88/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.83.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.146.32/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.100.17.159/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

---

🗑️ (실습 완료 후) 자원 삭제

Amazon EKS 클러스터 삭제(10분 정도 소요)

eksctl delete cluster --name $CLUSTER_NAME
# 결과
2025-02-14 02:20:39 [ℹ]  deleting EKS cluster "myeks"
2025-02-14 02:20:39 [ℹ]  will drain 0 unmanaged nodegroup(s) in cluster "myeks"
2025-02-14 02:20:39 [ℹ]  starting parallel draining, max in-flight of 1
2025-02-14 02:20:40 [ℹ]  deleted 0 Fargate profile(s)
2025-02-14 02:20:40 [✔]  kubeconfig has been updated
2025-02-14 02:20:40 [ℹ]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2025-02-14 02:22:55 [ℹ]  4 sequential tasks: { delete nodegroup "ng1", delete IAM OIDC provider, delete addon IAM "eksctl-myeks-addon-vpc-cni", delete cluster control plane "myeks" [async] }
2025-02-14 02:22:55 [ℹ]  will delete stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:22:55 [ℹ]  waiting for stack "eksctl-myeks-nodegroup-ng1" to get deleted
2025-02-14 02:22:55 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:23:26 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:23:58 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:25:36 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:27:34 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:29:08 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:30:35 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:31:32 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:33:04 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-02-14 02:33:05 [ℹ]  will delete stack "eksctl-myeks-addon-vpc-cni"
2025-02-14 02:33:05 [ℹ]  will delete stack "eksctl-myeks-cluster"
2025-02-14 02:33:06 [✔]  all cluster resources were deleted

aws cloudformation delete-stack --stack-name myeks