Post

AEWS 4์ฃผ์ฐจ ์ •๋ฆฌ

Posted
By GAGA
119 min read

๐Ÿš€ ์‹ค์Šต ํ™˜๊ฒฝ ๋ฐฐํฌ

Image

๐Ÿ—๏ธ AWS CloudFormation์„ ํ†ตํ•ด ๊ธฐ๋ณธ ์‹ค์Šต ํ™˜๊ฒฝ ๋ฐฐํฌ

1. yaml ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ

1
2
3
4
5

curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-4week.yaml
# ๊ฒฐ๊ณผ
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--100 21983  100 21983    0     0   201k      0 --:--:-- --:--:-- --:--:--  202k

2. ๋ณ€์ˆ˜ ์ง€์ •

1
2
3
4
5

CLUSTER_NAME=myeks
SSHKEYNAME=kp-aews # SSH ํ‚คํŽ˜์–ด ์ด๋ฆ„
MYACCESSKEY=XXXXXXXXXXXXXXXXXX # IAM User ์•ก์„ธ์Šค ํ‚ค
MYSECRETKEY=XXXXXXXXXXXXXXXXXX # IAM User ์‹œํฌ๋ฆฟ ํ‚ค
WorkerNodeInstanceType=t3.medium # ์›Œ์ปค๋…ธ๋“œ ์ธ์Šคํ„ด์Šค ํƒ€์ž…

3. CloudFormation ์Šคํƒ ๋ฐฐํฌ

1
2
3
4
5
6

aws cloudformation deploy --template-file myeks-4week.yaml --stack-name $CLUSTER_NAME --parameter-overrides KeyName=$SSHKEYNAME SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32  MyIamUserAccessKeyID=$MYACCESSKEY MyIamUserSecretAccessKey=$MYSECRETKEY ClusterBaseName=$CLUSTER_NAME WorkerNodeInstanceType=$WorkerNodeInstanceType --region ap-northeast-2

# ๊ฒฐ๊ณผ
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - myeks

4. CloudFormation ์Šคํƒ ๋ฐฐํฌ ์™„๋ฃŒ ํ›„ ์ž‘์—…์šฉ EC2 IP ์ถœ๋ ฅ

1

aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text

โœ…ย ์ถœ๋ ฅ

1

13.124.11.68

5. ๋ฐฐํฌ ๊ณผ์ • ์‚ดํŽด๋ณด๊ธฐ

(1) ์šด์˜์„œ๋ฒ„ EC2 SSH ์ ‘์†

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

ssh -i kp-aews.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)

The authenticity of host '13.124.11.68 (13.124.11.68)' can't be established.
ED25519 key fingerprint is SHA256:GaT1nuQgtyirycvJg2yQG/bVRT87T7sukVFzeOtkySk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '13.124.11.68' (ED25519) to the list of known hosts.
   ,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2026-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

[root@operator-host ~]#

(2) AWS CLI ์ž๊ฒฉ์ฆ๋ช… ์„ค์ •

1
2
3
4
5

[root@operator-host ~]# aws configure
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXX
Default region name [None]: ap-northeast-2
Default output format [None]: json

(3) ๋ฃจํŠธ ์‚ฌ์šฉ์ž ๋ฐ ํ™ˆ ๋””๋ ‰ํ† ๋ฆฌ ํ™•์ธ ์ž‘์—…

1
2

[root@operator-host ~]# whoami
root

1
2

[root@operator-host ~]# pwd
/root

(4) cloud-init ์‹คํ–‰ ๊ณผ์ • ๋กœ๊ทธ ํ™•์ธ

1

[root@operator-host ~]# tail -f /var/log/cloud-init-output.log

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

LICENSE
README.md
kubecolor
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--100    97  100    97    0     0    260      0 --:--:-- --:--:-- --:--:--   260
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0  9.9M    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--100  9.9M  100  9.9M    0     0  5993k      0  0:00:01  0:00:01 --:--:--  136M
Userdata End!
Cloud-init v. 19.3-46.amzn2.0.4 finished at Tue, 25 Feb 2025 12:58:35 +0000. Datasource DataSourceEc2.  Up 86.51 seconds

(5) eks ์„ค์ • ํŒŒ์ผ ํ™•์ธ

1

[root@operator-host ~]# cat myeks.yaml

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: myeks
  region: ap-northeast-2
  version: "1.31"

iam:
  withOIDC: true 

  serviceAccounts:
  - metadata:
      name: aws-load-balancer-controller
      namespace: kube-system
    wellKnownPolicies:
      awsLoadBalancerController: true

vpc:
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: true 
    publicAccess: true 
  id: vpc-017a9a38a294509ea
  subnets:
    public:
      ap-northeast-2a:
        az: ap-northeast-2a
        cidr: 192.168.1.0/24
        id: subnet-011d8d6df3bab1c31
      ap-northeast-2b:
        az: ap-northeast-2b
        cidr: 192.168.2.0/24
        id: subnet-004ed4a345eecd440
      ap-northeast-2c:
        az: ap-northeast-2c
        cidr: 192.168.3.0/24
        id: subnet-068e9402c8bb97c66

addons:
  - name: vpc-cni # no version is specified so it deploys the default version
    version: latest # auto discovers the latest available
    attachPolicyARNs: # attach IAM policies to the add-on's service account
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    configurationValues: |-
      enableNetworkPolicy: "true"

  - name: kube-proxy
    version: latest

  - name: coredns
    version: latest

  - name: metrics-server
    version: latest

  - name: aws-ebs-csi-driver
    version: latest
    wellKnownPolicies:
      ebsCSIController: true

managedNodeGroups:
- amiFamily: AmazonLinux2023
  desiredCapacity: 3
  iam:
    withAddonPolicies:
      certManager: true 
      externalDNS: true 
  instanceType: t3.medium
  preBootstrapCommands:
    # install additional packages
    - "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y"
  labels:
    alpha.eksctl.io/cluster-name: myeks
    alpha.eksctl.io/nodegroup-name: ng1
  maxPodsPerNode: 60
  maxSize: 3
  minSize: 3
  name: ng1
  ssh:
    allow: true
    publicKeyName: kp-aews
  tags:
    alpha.eksctl.io/nodegroup-name: ng1
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 60
  volumeThroughput: 125
  volumeType: gp3

(6) cloud-init ์ •์ƒ ์™„๋ฃŒ ํ›„ eksctl ์‹คํ–‰ ๊ณผ์ • ๋กœ๊ทธ ํ™•์ธ

1

[root@operator-host ~]# more create-eks.log

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

2025-02-25 21:57:51 [โ–ถ]  Setting credentials expiry window to 30 minutes
2025-02-25 21:57:51 [โ–ถ]  role ARN for the current session is "arn:aws:iam::378102432899:user/eks-user"
2025-02-25 21:57:52 [โ„น]  eksctl version 0.204.0
2025-02-25 21:57:52 [โ„น]  using region ap-northeast-2
2025-02-25 21:57:52 [โœ”]  using existing VPC (vpc-017a9a38a294509ea) and subnets (private:map[] public:map[ap-northeast-2a:{subnet-011d8d6df3bab1c31 ap-northeast-2a 192.168.1.0/24 0 } ap-northeast-2b:{subnet-004ed4a345eecd440 ap-northeast-2b 192.168.2.0/24 0 } ap-northeast-2c:{subnet-068e9402c8bb97c66 ap-northeast-2c 192.168.3.0/24 0 }])
2025-02-25 21:57:52 [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
2025-02-25 21:57:52 [โ„น]  nodegroup "ng1" will use "" [AmazonLinux2023/1.31]
2025-02-25 21:57:52 [โ„น]  using EC2 key pair "kp-aews"
2025-02-25 21:57:52 [โ„น]  using Kubernetes version 1.31
2025-02-25 21:57:52 [โ„น]  creating EKS cluster "myeks" in "ap-northeast-2" region with managed nodes
2025-02-25 21:57:52 [โ–ถ]  cfg.json = \
{
    "kind": "ClusterConfig",
    "apiVersion": "eksctl.io/v1alpha5",
    "metadata": {
        "name": "myeks",
        "region": "ap-northeast-2",
        "version": "1.31"
    },      
    "iam": {
        "withOIDC": true,
        "serviceAccounts": [
            {
                "metadata": {
                    "name": "aws-load-balancer-controller",
                    "namespace": "kube-system"
                },
                "wellKnownPolicies": {
                    "imageBuilder": false,
                    "autoScaler": false,
                    "awsLoadBalancerController": true,
                    "externalDNS": false,
                    "certManager": false,
                    "ebsCSIController": false,
                    "efsCSIController": false
                }
            }
        ],  
        "vpcResourceControllerPolicy": true
    },      
    "accessConfig": {
        "authenticationMode": "API_AND_CONFIG_MAP"
    },      
    "vpc": {
        "id": "vpc-017a9a38a294509ea",
        "cidr": "192.168.0.0/16",
        "subnets": {
            "public": {
                "ap-northeast-2a": {
                    "id": "subnet-011d8d6df3bab1c31",
                    "az": "ap-northeast-2a",
                    "cidr": "192.168.1.0/24"
                },
                "ap-northeast-2b": {
                    "id": "subnet-004ed4a345eecd440",
                    "az": "ap-northeast-2b",
                    "cidr": "192.168.2.0/24"
                },
                "ap-northeast-2c": {
                    "id": "subnet-068e9402c8bb97c66",
                    "az": "ap-northeast-2c",
                    "cidr": "192.168.3.0/24"
--More--(10%)

(7) ์…ธ ์ข…๋ฃŒ ๋ช…๋ น ์‹คํ–‰

1

[root@operator-host ~]# exit

๐Ÿš€ AWS EKS ์„ค์น˜ ํ™•์ธ (์Šคํƒ ์ƒ์„ฑ ์‹œ์ž‘ ํ›„ ์•ฝ 20๋ถ„ ๊ฒฝ๊ณผ)

1. eksctl ํด๋Ÿฌ์Šคํ„ฐ ์กฐํšŒ

1

eksctl get cluster

โœ…ย ์ถœ๋ ฅ

1
2

NAME	REGION		EKSCTL CREATED
myeks	ap-northeast-2	True

2. ํด๋Ÿฌ์Šคํ„ฐ ๋…ธ๋“œ ๊ทธ๋ฃน ์กฐํšŒ

1

eksctl get nodegroup --cluster $CLUSTER_NAME

โœ…ย ์ถœ๋ ฅ

1
2

CLUSTER	NODEGROUP	STATUS		CREATED			MIN SIZEMAX SIZE	DESIRED CAPACITY	INSTANCE TYPE	IMAGE ID	ASG NAME	TYPE
myeks	ng1		CREATING	2025-02-25T13:12:08Z	3	3	3			t3.medium	AL2023_x86_64_STANDARD	managed

3. ํด๋Ÿฌ์Šคํ„ฐ ์• ๋“œ์˜จ ์กฐํšŒ

1

eksctl get addon --cluster $CLUSTER_NAME

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

2025-02-25 22:13:56 [โ„น]  Kubernetes version "1.31" in use by cluster "myeks"
2025-02-25 22:13:56 [โ„น]  getting all addons
2025-02-25 22:13:57 [โ„น]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME		VERSION			STATUS		ISSUES	IAMROLEUPDATE AVAILABLE	CONFIGURATION VALUES		POD IDENTITY ASSOCIATION ROLES
coredns		v1.11.4-eksbuild.2	DEGRADED	1		
kube-proxy	v1.31.3-eksbuild.2	ACTIVE		0		
metrics-server	v0.7.2-eksbuild.2	DEGRADED	1		
vpc-cni		v1.19.2-eksbuild.5	ACTIVE		0	arn:aws:iam::378102432899:role/eksctl-myeks-addon-vpc-cni-Role1-Q7K66W6aHXAn	enableNetworkPolicy: "true"

4. ํด๋Ÿฌ์Šคํ„ฐ IAM ์„œ๋น„์Šค ๊ณ„์ • ์กฐํšŒ

1

eksctl get iamserviceaccount --cluster $CLUSTER_NAME

โœ…ย ์ถœ๋ ฅ

1
2

NAMESPACE	NAME				ROLE ARN
kube-system	aws-load-balancer-controller	arn:aws:iam::378102432899:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-RDjfak64nvXd

5. kubeconfig ์ƒ์„ฑ

(1) ์ž๊ฒฉ์ฆ๋ช… ์‚ฌ์šฉ์ž ํ™•์ธ

1

aws sts get-caller-identity --query Arn

โœ…ย ์ถœ๋ ฅ

1

"arn:aws:iam::378102432899:user/eks-user"

(2) kubeconfig ์—…๋ฐ์ดํŠธ ๋ช…๋ น ์‹คํ–‰

1
2
3
4

aws eks update-kubeconfig --name myeks --user-alias eks-user # ์ถœ๋ ฅ๋œ ์ž๊ฒฉ์ฆ๋ช… ์‚ฌ์šฉ์ž

# ๊ฒฐ๊ณผ
Added new context eks-user to /home/devshin/.kube/config

6. Kubernetes ํด๋Ÿฌ์Šคํ„ฐ ๋ฐ ๋ฆฌ์†Œ์Šค ์ƒํƒœ ํ™•์ธ

(1) ํด๋Ÿฌ์Šคํ„ฐ ์ •๋ณด ์กฐํšŒ

1

kubectl cluster-info

โœ…ย ์ถœ๋ ฅ

1
2
3
4

Kubernetes control plane is running at https://79E4B0C88ABCA6E051CDC256189CC3B2.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://79E4B0C88ABCA6E051CDC256189CC3B2.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

(2) ๊ธฐ๋ณธ ๋„ค์ž„์ŠคํŽ˜์ด์Šค ์„ค์ •

1

kubectl ns default

(3) ๋…ธ๋“œ ์ •๋ณด ์กฐํšŒ

  • ๋””๋ฒ„๊ทธ ๋ ˆ๋ฒจ 6์œผ๋กœ ๋…ธ๋“œ ์ •๋ณด ์กฐํšŒ
1

kubectl get node -v6

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

I0225 22:24:07.471314   48192 loader.go:402] Config loaded from file:  /home/devshin/.kube/config
I0225 22:24:07.471812   48192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0225 22:24:07.471828   48192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0225 22:24:07.471837   48192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0225 22:24:07.471844   48192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0225 22:24:07.905428   48192 round_trippers.go:560] GET https://79E4B0C88ABCA6E051CDC256189CC3B2.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 427 milliseconds
NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-1-51.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.31.5-eks-5d632ec
ip-192-168-2-42.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.31.5-eks-5d632ec
ip-192-168-3-30.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.31.5-eks-5d632ec
  • ์ธ์Šคํ„ด์Šค ์œ ํ˜•, ์šฉ๋Ÿ‰ ์œ ํ˜•, ๊ฐ€์šฉ ์˜์—ญ ๋ผ๋ฒจ ์ •๋ณด ์ƒ์„ธ ์กฐํšŒ
1

kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone

โœ…ย ์ถœ๋ ฅ

1
2
3
4

NAME                                              STATUS   ROLES    AGE   VERSION               INSTANCE-TYPE   CAPACITYTYPE   ZONE
ip-192-168-1-51.ap-northeast-2.compute.internal   Ready    <none>   12m   v1.31.5-eks-5d632ec   t3.medium       ON_DEMAND      ap-northeast-2a
ip-192-168-2-42.ap-northeast-2.compute.internal   Ready    <none>   12m   v1.31.5-eks-5d632ec   t3.medium       ON_DEMAND      ap-northeast-2b
ip-192-168-3-30.ap-northeast-2.compute.internal   Ready    <none>   12m   v1.31.5-eks-5d632ec   t3.medium       ON_DEMAND      ap-northeast-2c

(4) ํŒŒ๋“œ ์ •๋ณด ์กฐํšŒ

1

kubectl get pod -A

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   aws-node-p4v96                       2/2     Running   0          12m
kube-system   aws-node-qgc5t                       2/2     Running   0          12m
kube-system   aws-node-r5nbp                       2/2     Running   0          12m
kube-system   coredns-86f5954566-c8wl2             1/1     Running   0          18m
kube-system   coredns-86f5954566-d6vwh             1/1     Running   0          18m
kube-system   ebs-csi-controller-7f8f8cb84-p57xw   6/6     Running   0          10m
kube-system   ebs-csi-controller-7f8f8cb84-z4t4z   6/6     Running   0          10m
kube-system   ebs-csi-node-gdh58                   3/3     Running   0          10m
kube-system   ebs-csi-node-hx5jb                   3/3     Running   0          10m
kube-system   ebs-csi-node-j46zg                   3/3     Running   0          10m
kube-system   kube-proxy-s6tdr                     1/1     Running   0          12m
kube-system   kube-proxy-v8nh9                     1/1     Running   0          12m
kube-system   kube-proxy-z9l58                     1/1     Running   0          12m
kube-system   metrics-server-6bf5998d9c-c8tbf      1/1     Running   0          18m
kube-system   metrics-server-6bf5998d9c-tftq9      1/1     Running   0          18m

(5) ํŒŒ๋“œ ์ค‘๋‹จ ํ—ˆ์šฉ(PDB) ์กฐํšŒ

1

kubectl get pdb -n kube-system

โœ…ย ์ถœ๋ ฅ

1
2
3
4

NAME                 MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
coredns              N/A             1                 1                     18m
ebs-csi-controller   N/A             1                 1                     10m
metrics-server       N/A             1                 1                     18m

7. krew ํ”Œ๋Ÿฌ๊ทธ์ธ ํ™•์ธ

1

kubectl krew list

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7

PLUGIN   VERSION
ctx      v0.9.5
df-pv    v0.3.0
get-all  v1.3.8
krew     v0.4.4
neat     v2.0.4
stern    v1.32.0

๐Ÿ”Œ ๋…ธ๋“œ IP ์ •๋ณด ํ™•์ธ ๋ฐ SSH ์ ‘์†

1. EC2 ๊ณต์ธ IP ๋ณ€์ˆ˜ ์ง€์ •

1
2
3
4

export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
echo $N1, $N2, $N3

โœ…ย ์ถœ๋ ฅ

1

15.164.227.37, 3.38.205.159, 43.200.163.0

2. EC2 ๋ณด์•ˆ ๊ทธ๋ฃน ์กฐํšŒ (remoteAccess ํ•„ํ„ฐ ์ ์šฉ)

1

aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" | jq

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

{
  "SecurityGroups": [
    {
      "GroupId": "sg-0387b57d9b1586fe5",
      "IpPermissionsEgress": [
        {
          "IpProtocol": "-1",
          "UserIdGroupPairs": [],
          "IpRanges": [
            {
              "CidrIp": "0.0.0.0/0"
            }
          ],
          "Ipv6Ranges": [],
          "PrefixListIds": []
        }
      ],
      "Tags": [
        {
          "Key": "Name",
          "Value": "eksctl-myeks-nodegroup-ng1/SSH"
        },
        {
          "Key": "alpha.eksctl.io/cluster-name",
          "Value": "myeks"
        },
        {
          "Key": "alpha.eksctl.io/eksctl-version",
          "Value": "0.204.0"
        },
        {
          "Key": "alpha.eksctl.io/nodegroup-name",
          "Value": "ng1"
        },
        {
          "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
          "Value": "myeks"
        },
        {
          "Key": "aws:cloudformation:stack-id",
          "Value": "arn:aws:cloudformation:ap-northeast-2:378102432899:stack/eksctl-myeks-nodegroup-ng1/0e0c9500-f37a-11ef-b856-0237bb259921"
        },
        {
          "Key": "alpha.eksctl.io/nodegroup-type",
          "Value": "managed"
        },
        {
          "Key": "aws:cloudformation:stack-name",
          "Value": "eksctl-myeks-nodegroup-ng1"
        },
        {
          "Key": "aws:cloudformation:logical-id",
          "Value": "SSH"
        }
      ],
      "VpcId": "vpc-017a9a38a294509ea",
      "SecurityGroupArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group/sg-0387b57d9b1586fe5",
      "OwnerId": "378102432899",
      "GroupName": "eksctl-myeks-nodegroup-ng1-remoteAccess",
      "Description": "Allow SSH access",
      "IpPermissions": [
        {
          "IpProtocol": "tcp",
          "FromPort": 22,
          "ToPort": 22,
          "UserIdGroupPairs": [],
          "IpRanges": [
            {
              "Description": "Allow SSH access to managed worker nodes in group ng1",
              "CidrIp": "0.0.0.0/0"
            }
          ],
          "Ipv6Ranges": [
            {
              "Description": "Allow SSH access to managed worker nodes in group ng1",
              "CidrIpv6": "::/0"
            }
          ],
          "PrefixListIds": []
        }
      ]
    }
  ]
}

3. ๋ณด์•ˆ ๊ทธ๋ฃน ID ํ™˜๊ฒฝ ๋ณ€์ˆ˜ ์„ค์ •

1

export MNSGID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" --query 'SecurityGroups[*].GroupId' --output text)

4. ํ•ด๋‹น ๋ณด์•ˆ๊ทธ๋ฃน ์ธ๋ฐ”์šด๋“œ ๊ทœ์น™์— ๋ณธ์ธ์˜ ์ง‘ ๊ณต์ธ IP ์ถ”๊ฐ€

1

aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr $(curl -s ipinfo.io/ip)/32

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0f7c4b1164d618a4c",
            "GroupId": "sg-0387b57d9b1586fe5",
            "GroupOwnerId": "378102432899",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "182.230.60.93/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-0f7c4b1164d618a4c"
        }
    ]
}

5. ํ•ด๋‹น ๋ณด์•ˆ ๊ทธ๋ฃน์˜ ์ธ๋ฐ”์šด๋“œ ๊ทœ์น™์— ์šด์˜ ์„œ๋ฒ„ ๋‚ด๋ถ€ IP ์ถ”๊ฐ€

1

aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr 172.20.1.100/32

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0831563999950b76b",
            "GroupId": "sg-0387b57d9b1586fe5",
            "GroupOwnerId": "378102432899",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "172.20.1.100/32",
            "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-0831563999950b76b"
        }
    ]
}

6. ์›Œ์ปค ๋…ธ๋“œ SSH ์ ‘์†

(1) ๊ฐ ๋…ธ๋“œ์— SSH ์›๊ฒฉ ์ ‘์† ํ›„ ํ˜ธ์ŠคํŠธ๋ช… ์ถœ๋ ฅ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -o StrictHostKeyChecking=no ec2-user@$i hostname; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11

>> node 15.164.227.37 <<
Warning: Permanently added '15.164.227.37' (ED25519) to the list of known hosts.
ec2-user@15.164.227.37: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

>> node 3.38.205.159 <<
Warning: Permanently added '3.38.205.159' (ED25519) to the list of known hosts.
ec2-user@3.38.205.159: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

>> node 43.200.163.0 <<
Warning: Permanently added '43.200.163.0' (ED25519) to the list of known hosts.
ec2-user@43.200.163.0: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

(2) N1 ๋…ธ๋“œ ์ ‘์†

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

ssh ec2-user@$N1

# ๊ฒฐ๊ณผ
A newer release of "Amazon Linux" is available.
  Version 2023.6.20250211:
  Version 2023.6.20250218:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 05:52:48 2025 from 52.94.123.236
[ec2-user@ip-192-168-1-51 ~]$ exit
logout
Connection to 15.164.227.37 closed.

(3) N2 ๋…ธ๋“œ ์ ‘์†

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

ssh ec2-user@$N2

# ๊ฒฐ๊ณผ
A newer release of "Amazon Linux" is available.
  Version 2023.6.20250211:
  Version 2023.6.20250218:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 05:52:48 2025 from 52.94.123.236
[ec2-user@ip-192-168-2-42 ~]$ exit
logout
Connection to 3.38.205.159 closed.

(4) N3 ๋…ธ๋“œ ์ ‘์†

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

ssh ec2-user@$N3

# ๊ฒฐ๊ณผ
A newer release of "Amazon Linux" is available.
  Version 2023.6.20250211:
  Version 2023.6.20250218:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Wed Feb 12 05:52:48 2025 from 52.94.123.236
[ec2-user@ip-192-168-3-30 ~]$ exit
logout
Connection to 43.200.163.0 closed.

7. ๋…ธ๋“œ ๊ธฐ๋ณธ ์ •๋ณด ํ™•์ธ

(1) ๋…ธ๋“œ๋ณ„ ์‹œ์Šคํ…œ ์ •๋ณด ์กฐํšŒ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i hostnamectl; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

>> node 15.164.227.37 <<
 Static hostname: ip-192-168-1-51.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm ๐Ÿ–ด
      Machine ID: ec290d14e0f34366b2d3f2ea33b06253
         Boot ID: 89441b0cd579455ca36a97ae72436762
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250203
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.127-135.201.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

>> node 3.38.205.159 <<
 Static hostname: ip-192-168-2-42.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm ๐Ÿ–ด
      Machine ID: ec23b4795af458cac1beebec40e88e9b
         Boot ID: 2e16f7e351a642deb439e4360f0e4f5c
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250203
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.127-135.201.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

>> node 43.200.163.0 <<
 Static hostname: ip-192-168-3-30.ap-northeast-2.compute.internal
       Icon name: computer-vm
         Chassis: vm ๐Ÿ–ด
      Machine ID: ec203a98663eebbcd25282168fa4a01d
         Boot ID: 39dfaedc08cf445ca387e7ec0b9c7823
  Virtualization: amazon
Operating System: Amazon Linux 2023.6.20250203
     CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023
          Kernel: Linux 6.1.127-135.201.amzn2023.x86_64
    Architecture: x86-64
 Hardware Vendor: Amazon EC2
  Hardware Model: t3.medium
Firmware Version: 1.0

(2) ๋…ธ๋“œ๋ณ„ ๋„คํŠธ์›Œํฌ ์ธํ„ฐํŽ˜์ด์Šค ์ •๋ณด ์กฐํšŒ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

>> node 15.164.227.37 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:79:1b:57:05:df brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.1.51/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5
       valid_lft 2091sec preferred_lft 2091sec
    inet6 fe80::79:1bff:fe57:5df/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni97f4361e4c2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 32:96:47:9a:d4:2f brd ff:ff:ff:ff:ff:ff link-netns cni-d5c2a5e6-1a50-b510-b895-2a949906fb4d
    inet6 fe80::3096:47ff:fe9a:d42f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: enif5e3248355d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether c2:2e:98:dd:be:8d brd ff:ff:ff:ff:ff:ff link-netns cni-914d38b7-bff1-33e3-c759-762f2221a32a
    inet6 fe80::c02e:98ff:fedd:be8d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ee:52:94:22:3d brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.1.137/24 brd 192.168.1.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::ee:52ff:fe94:223d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 3.38.205.159 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:8d:29:e9:31:07 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.2.42/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5
       valid_lft 2091sec preferred_lft 2091sec
    inet6 fe80::48d:29ff:fee9:3107/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni98403b04a75@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 02:d7:7d:25:42:c3 brd ff:ff:ff:ff:ff:ff link-netns cni-1c292e75-bd4d-12d0-5ce3-a6a0d152a92f
    inet6 fe80::d7:7dff:fe25:42c3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: enib3cc1ab608a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 7e:c2:a3:56:c6:38 brd ff:ff:ff:ff:ff:ff link-netns cni-18f2ec3b-04f9-6d85-85c0-38893a1630e2
    inet6 fe80::7cc2:a3ff:fe56:c638/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:31:11:9e:26:ad brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.2.136/24 brd 192.168.2.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::431:11ff:fe9e:26ad/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

>> node 43.200.163.0 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:67:d0:6f:e6:9b brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 192.168.3.30/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
       valid_lft 2087sec preferred_lft 2087sec
    inet6 fe80::867:d0ff:fe6f:e69b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eni356985de846@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether d6:cd:95:65:d8:36 brd ff:ff:ff:ff:ff:ff link-netns cni-d3e410d2-9acb-0b2a-15eb-577b3914e495
    inet6 fe80::d4cd:95ff:fe65:d836/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: eni7432c2a8810@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether ee:79:9f:e1:ae:27 brd ff:ff:ff:ff:ff:ff link-netns cni-c80bddf0-cde4-c490-17c1-38c73da57ae0
    inet6 fe80::ec79:9fff:fee1:ae27/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: eni8a456b324b3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 8e:46:59:f7:d9:26 brd ff:ff:ff:ff:ff:ff link-netns cni-23fd0956-85bd-af34-5a2c-fb5d936419d7
    inet6 fe80::8c46:59ff:fef7:d926/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
6: enid3abba3d96f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether f6:99:0c:7c:00:1d brd ff:ff:ff:ff:ff:ff link-netns cni-db4e1b60-fa45-df03-34e4-7db99d67f7a2
    inet6 fe80::f499:cff:fe7c:1d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
7: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:4a:30:b7:cd:cb brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 192.168.3.77/24 brd 192.168.3.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 fe80::84a:30ff:feb7:cdcb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
8: eni0910e13e62f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 92:95:95:ba:c4:b5 brd ff:ff:ff:ff:ff:ff link-netns cni-3cc85dca-5044-6749-7925-406cfe916181
    inet6 fe80::9095:95ff:feba:c4b5/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

(3) ๋…ธ๋“œ๋ณ„ ๋ธ”๋ก ๋””๋ฐ”์ด์Šค ์กฐํšŒ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i lsblk; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

>> node 15.164.227.37 <<
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1       259:0    0  60G  0 disk 
โ”œโ”€nvme0n1p1   259:1    0  60G  0 part /
โ”œโ”€nvme0n1p127 259:2    0   1M  0 part 
โ””โ”€nvme0n1p128 259:3    0  10M  0 part /boot/efi

>> node 3.38.205.159 <<
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1       259:0    0  60G  0 disk 
โ”œโ”€nvme0n1p1   259:1    0  60G  0 part /
โ”œโ”€nvme0n1p127 259:2    0   1M  0 part 
โ””โ”€nvme0n1p128 259:3    0  10M  0 part /boot/efi

>> node 43.200.163.0 <<
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1       259:0    0  60G  0 disk 
โ”œโ”€nvme0n1p1   259:1    0  60G  0 part /
โ”œโ”€nvme0n1p127 259:2    0   1M  0 part 
โ””โ”€nvme0n1p128 259:3    0  10M  0 part /boot/efi

(4) ๋…ธ๋“œ๋ณ„ ๋ฃจํŠธ ํŒŒ์ผ์‹œ์Šคํ…œ ์šฉ๋Ÿ‰ ์กฐํšŒ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i df -hT /; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11

>> node 15.164.227.37 <<
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs    60G  3.3G   57G   6% /

>> node 3.38.205.159 <<
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs    60G  3.3G   57G   6% /

>> node 43.200.163.0 <<
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs    60G  3.3G   57G   6% /

8. ์Šคํ† ๋ฆฌ์ง€ํด๋ž˜์Šค ์กฐํšŒ

1

kubectl get sc

โœ…ย ์ถœ๋ ฅ

1
2

NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  36m

9. CSI ๋…ธ๋“œ ์กฐํšŒ

1

kubectl get csinodes

โœ…ย ์ถœ๋ ฅใ…‡

1
2
3
4

NAME                                              DRIVERS   AGE
ip-192-168-1-51.ap-northeast-2.compute.internal   1         26m
ip-192-168-2-42.ap-northeast-2.compute.internal   1         26m
ip-192-168-3-30.ap-northeast-2.compute.internal   1         26m

10. maxPods ์„ค์ • ํ™•์ธ

(1) ๋…ธ๋“œ ์ตœ๋Œ€ ํŒŒ๋“œ ์ˆ˜ ์กฐํšŒ

1

kubectl get nodes -o custom-columns="NAME:.metadata.name,MAXPODS:.status.capacity.pods"

โœ…ย ์ถœ๋ ฅ

1
2
3
4

NAME                                              MAXPODS
ip-192-168-1-51.ap-northeast-2.compute.internal   60
ip-192-168-2-42.ap-northeast-2.compute.internal   60
ip-192-168-3-30.ap-northeast-2.compute.internal   60

(2) ๋…ธ๋“œ๋ณ„ kubelet ๊ธฐ๋ณธ ์„ค์ •์—์„œ maxPods ๊ฐ’ ํ™•์ธ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json | grep maxPods; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

>> node 15.164.227.37 <<
    "maxPods": 17,

>> node 3.38.205.159 <<
    "maxPods": 17,

>> node 43.200.163.0 <<
    "maxPods": 17,

(3) ๋…ธ๋“œ๋ณ„ kubelet ์ถ”๊ฐ€ ์„ค์ • ํŒŒ์ผ์—์„œ maxPods ๊ฐ’ ํ™•์ธ

1

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json.d/00-nodeadm.conf | grep maxPods; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

>> node 15.164.227.37 <<
    "maxPods": 60

>> node 3.38.205.159 <<
    "maxPods": 60

>> node 43.200.163.0 <<
    "maxPods": 60

11. ์šด์˜ ์„œ๋ฒ„ EC2 SSH ์›๊ฒฉ ์ ‘์† ํ›„ ๊ธฐ๋ณธ ์ •๋ณด ํ™•์ธ

(1) ์šด์˜์„œ๋ฒ„ SSH ์ ‘์†

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

ssh -i kp-aews.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)

Warning: Identity file kp-aews.pem not accessible: No such file or directory.
Last login: Tue Feb 25 22:17:38 2025 from 182.230.60.93
   ,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2026-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

Last login: Tue Feb 25 22:17:38 KST 2025 on pts/0
(eks-user@myeks:N/A) [root@operator-host ~]#

(2) default ๋„ค์ž„์ŠคํŽ˜์ด์Šค ์ ์šฉ

1
2
3
4

(eks-user@myeks:N/A) [root@operator-host ~]# k ns default
# ๊ฒฐ๊ณผ
Context "eks-user@myeks.ap-northeast-2.eksctl.io" modified.
Active namespace is "default".

(3) ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์ •๋ณด ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host ~]# export | egrep 'ACCOUNT|AWS_|CLUSTER|KUBERNETES|VPC|Subnet' | egrep -v 'KEY'

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9

declare -x ACCOUNT_ID="xxxxxxxxxxxx"
declare -x AWS_DEFAULT_REGION="ap-northeast-2"
declare -x AWS_PAGER=""
declare -x CLUSTER_NAME="myeks"
declare -x KUBERNETES_VERSION="1.31"
declare -x PubSubnet1="subnet-011d8d6df3bab1c31"
declare -x PubSubnet2="subnet-004ed4a345eecd440"
declare -x PubSubnet3="subnet-068e9402c8bb97c66"
declare -x VPCID="vpc-017a9a38a294509ea"

(4) krew ํ”Œ๋Ÿฌ๊ทธ์ธ ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host ~]# kubectl krew list

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

PLUGIN       VERSION
ctx          v0.9.5
df-pv        v0.3.0
get-all      v1.3.8
krew         v0.4.4
neat         v2.0.4
ns           v0.9.5
oomd         v0.0.7
stern        v1.32.0
view-secret  v0.13.0

12. ์ธ์Šคํ„ด์Šค ์ •๋ณด ์กฐํšŒ

1

(eks-user@myeks:default) [root@operator-host ~]# aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

---------------------------------------------------------------------------------------
|                                  DescribeInstances                                  |
+----------------------+-----------------+---------------+----------------+-----------+
|      InstanceID      |  InstanceName   | PrivateIPAdd  |  PublicIPAdd   |  Status   |
+----------------------+-----------------+---------------+----------------+-----------+
|  i-0c8dd7ee129df9f70 |  myeks-ng1-Node |  192.168.3.30 |  43.200.163.0  |  running  |
|  i-0cf5292e9106cff08 |  operator-host  |  172.20.1.100 |  13.124.11.68  |  running  |
|  i-013587fe8ee35bdf3 |  myeks-ng1-Node |  192.168.1.51 |  15.164.227.37 |  running  |
|  i-05e7d5c353553c7ff |  myeks-ng1-Node |  192.168.2.42 |  3.38.205.159  |  running  |
+----------------------+-----------------+---------------+----------------+-----------+

13. PrivateIP ๋ณ€์ˆ˜ ์ง€์ •

1
2
3
4
5
6
7

(eks-user@myeks:default) [root@operator-host ~]# N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# echo "export N1=$N1" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo "export N2=$N2" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo "export N3=$N3" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo $N1, $N2, $N3

โœ…ย ์ถœ๋ ฅ

1

192.168.1.51, 192.168.2.42, 192.168.3.30

14. ๋…ธ๋“œ IP ๋กœ ping ํ…Œ์ŠคํŠธ

1

(eks-user@myeks:default) [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ping -c 1 $i ; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

>> node 192.168.1.51 <<
PING 192.168.1.51 (192.168.1.51) 56(84) bytes of data.
64 bytes from 192.168.1.51: icmp_seq=1 ttl=127 time=0.830 ms

--- 192.168.1.51 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.830/0.830/0.830/0.000 ms

>> node 192.168.2.42 <<
PING 192.168.2.42 (192.168.2.42) 56(84) bytes of data.
64 bytes from 192.168.2.42: icmp_seq=1 ttl=127 time=1.04 ms

--- 192.168.2.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.047/1.047/1.047/0.000 ms

>> node 192.168.3.30 <<
PING 192.168.3.30 (192.168.3.30) 56(84) bytes of data.
64 bytes from 192.168.3.30: icmp_seq=1 ttl=127 time=1.39 ms

--- 192.168.3.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.392/1.392/1.392/0.000 ms

15. kube-ops-view ์„ค์น˜

1
2
3
4

helm repo add geek-cookbook https://geek-cookbook.github.io/charts/

# ๊ฒฐ๊ณผ
"geek-cookbook" already exists with the same configuration, skipping

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=ClusterIP  --set env.TZ="Asia/Seoul" --namespace kube-system

# ๊ฒฐ๊ณผ
NAME: kube-ops-view
LAST DEPLOYED: Tue Feb 25 23:43:00 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:8080

16. gp3 ์Šคํ† ๋ฆฌ์ง€ ํด๋ž˜์Šค ์ƒ์„ฑ ๋ฐ ์กฐํšŒ

(1) gp3 ์Šคํ† ๋ฆฌ์ง€ ํด๋ž˜์Šค ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  allowAutoIOPSPerGBIncrease: 'true'
  encrypted: 'true'
  fsType: xfs # ๊ธฐ๋ณธ๊ฐ’์ด ext4
EOF

# ๊ฒฐ๊ณผ
storageclass.storage.k8s.io/gp3 created

(2) ์กฐํšŒ

1

kubectl get sc

โœ…ย ์ถœ๋ ฅ

1
2
3

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2             kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  101m
gp3 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   31s

17. ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์„ค์ •

(1) ๋ณธ์ธ PC

1
2
3
4
5
6
7
8
9
10
11

export CLUSTER_NAME=myeks
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text) #์‚ฌ์šฉ ๋ฆฌ์ „์˜ ์ธ์ฆ์„œ ARN ํ™•์ธ
MyDomain=gagajin.com # ๊ฐ์ž ์ž์‹ ์˜ ๋„๋ฉ”์ธ ์ด๋ฆ„ ์ž…๋ ฅ
MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)

(2) ์šด์˜ ์„œ๋ฒ„

1
2
3
4
5
6
7
8
9
10
11

(eks-user@myeks:default) [root@operator-host ~]# export CLUSTER_NAME=myeks
(eks-user@myeks:default) [root@operator-host ~]# export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
(eks-user@myeks:default) [root@operator-host ~]# export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
(eks-user@myeks:default) [root@operator-host ~]# export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
(eks-user@myeks:default) [root@operator-host ~]# export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
(eks-user@myeks:default) [root@operator-host ~]# export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
(eks-user@myeks:default) [root@operator-host ~]# export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
(eks-user@myeks:default) [root@operator-host ~]# export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
(eks-user@myeks:default) [root@operator-host ~]# export CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text) #์‚ฌ์šฉ ๋ฆฌ์ „์˜ ์ธ์ฆ์„œ ARN ํ™•์ธ
(eks-user@myeks:default) [root@operator-host ~]# MyDomain=gagajin.com # ๊ฐ์ž ์ž์‹ ์˜ ๋„๋ฉ”์ธ ์ด๋ฆ„ ์ž…๋ ฅ
(eks-user@myeks:default) [root@operator-host ~]# MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)

18. ExternalDNS ์ƒ์„ฑ

1
2
3
4
5
6
7

curl -s https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml | MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst | kubectl apply -f -

# ๊ฒฐ๊ณผ
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created

19. AWS LoadBalancerController ์ƒ์„ฑ

1
2
3
4

helm repo add eks https://aws.github.io/eks-charts

# ๊ฒฐ๊ณผ
"eks" already exists with the same configuration, skipping

1
2
3
4
5
6
7
8
9
10
11
12

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \
  --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

# ๊ฒฐ๊ณผ
NAME: aws-load-balancer-controller
LAST DEPLOYED: Tue Feb 25 23:46:40 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

20. kubeopsview Ingress ๊ตฌ์„ฑ

๊ทธ๋ฃน ์„ค์ •์„ ํ†ตํ•ด ๋‹จ์ผ ALB๋ฅผ ์—ฌ๋Ÿฌ Ingress๊ฐ€ ๊ณต์šฉ์œผ๋กœ ์‚ฌ์šฉํ•˜๋„๋ก ์„ค์ •

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/load-balancer-name: $CLUSTER_NAME-ingress-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/target-type: ip
  labels:
    app.kubernetes.io/name: kubeopsview
  name: kubeopsview
  namespace: kube-system
spec:
  ingressClassName: alb
  rules:
  - host: kubeopsview.$MyDomain
    http:
      paths:
      - backend:
          service:
            name: kube-ops-view
            port:
              number: 8080
        path: /
        pathType: Prefix
EOF
# ๊ฒฐ๊ณผ
ingress.networking.k8s.io/kubeopsview created

21. ์„ค์น˜๋œ ํŒŒ๋“œ ์ •๋ณด ์กฐํšŒ

1

kubectl get pods -n kube-system

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

NAME                                          READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-554fbd9d-kbctn   1/1     Running   0          8m7s
aws-load-balancer-controller-554fbd9d-mtlsg   1/1     Running   0          8m7s
aws-node-p4v96                                2/2     Running   0          101m
aws-node-qgc5t                                2/2     Running   0          101m
aws-node-r5nbp                                2/2     Running   0          101m
coredns-86f5954566-c8wl2                      1/1     Running   0          106m
coredns-86f5954566-d6vwh                      1/1     Running   0          106m
ebs-csi-controller-7f8f8cb84-p57xw            6/6     Running   0          98m
ebs-csi-controller-7f8f8cb84-z4t4z            6/6     Running   0          98m
ebs-csi-node-gdh58                            3/3     Running   0          98m
ebs-csi-node-hx5jb                            3/3     Running   0          98m
ebs-csi-node-j46zg                            3/3     Running   0          98m
external-dns-dc4878f5f-fskxk                  1/1     Running   0          111s
kube-ops-view-657dbc6cd8-pxkvr                1/1     Running   0          11m
kube-proxy-s6tdr                              1/1     Running   0          101m
kube-proxy-v8nh9                              1/1     Running   0          101m
kube-proxy-z9l58                              1/1     Running   0          101m
metrics-server-6bf5998d9c-c8tbf               1/1     Running   0          106m
metrics-server-6bf5998d9c-tftq9               1/1     Running   0          106m

22. Ingress, ์„œ๋น„์Šค ๋ฐ ์—”๋“œํฌ์ธํŠธ ์กฐํšŒ

1

kubectl get ingress,svc,ep -n kube-system

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

NAME                                    CLASS   HOSTS                     ADDRESS                                                       PORTS   AGE
ingress.networking.k8s.io/kubeopsview   alb     kubeopsview.gagajin.com   myeks-ingress-alb-77245841.ap-northeast-2.elb.amazonaws.com   80      2m2s

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/aws-load-balancer-webhook-service   ClusterIP   10.100.246.75   <none>        443/TCP                  8m54s
service/eks-extension-metrics-api           ClusterIP   10.100.95.88    <none>        443/TCP                  112m
service/kube-dns                            ClusterIP   10.100.0.10     <none>        53/UDP,53/TCP,9153/TCP   107m
service/kube-ops-view                       ClusterIP   10.100.19.254   <none>        8080/TCP                 12m
service/metrics-server                      ClusterIP   10.100.8.241    <none>        443/TCP                  107m

NAME                                          ENDPOINTS                                                        AGE
endpoints/aws-load-balancer-webhook-service   192.168.1.114:9443,192.168.2.127:9443                            8m54s
endpoints/eks-extension-metrics-api           172.0.32.0:10443                                                 112m
endpoints/kube-dns                            192.168.3.140:53,192.168.3.184:53,192.168.3.140:53 + 3 more...   107m
endpoints/kube-ops-view                       192.168.2.249:8080                                               12m
endpoints/metrics-server                      192.168.3.110:10251,192.168.3.8:10251                            107m

23. Kube Ops View ์ ‘์† ์ •๋ณด ํ™•์ธ

1

echo -e "Kube Ops View URL = https://kubeopsview.$MyDomain/#scale=1.5"

โœ…ย ์ถœ๋ ฅ

1

Kube Ops View URL = https://kubeopsview.gagajin.com/#scale=1.5

์ ‘์† ํ™”๋ฉด

Image

๐Ÿ“ฆ Bookinfo ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋ฐฐํฌ

1. Bookinfo ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋ฐฐํฌ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

kubectl apply -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/bookinfo/platform/kube/bookinfo.yaml

# ๊ฒฐ๊ณผ
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

2. ALB Ingress ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/load-balancer-name: $CLUSTER_NAME-ingress-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/target-type: ip
  labels:
    app.kubernetes.io/name: bookinfo
  name: bookinfo
spec:
  ingressClassName: alb
  rules:
  - host: bookinfo.$MyDomain
    http:
      paths:
      - backend:
          service:
            name: productpage
            port:
              number: 9080
        path: /
        pathType: Prefix
EOF

# ๊ฒฐ๊ณผ
ingress.networking.k8s.io/bookinfo created

3. ALB Ingress ์กฐํšŒ

1

kubectl get ingress

โœ…ย ์ถœ๋ ฅ

1
2

NAME       CLASS   HOSTS                  ADDRESS   PORTS   AGE
bookinfo   alb     bookinfo.gagajin.com             80      0s

HTTPS ๋ฆฌ์Šค๋„ˆ ๊ทœ์น™

  • ๋„๋ฉ”์ธ์ด bookinfo.gagajin.com์ด๊ณ  ๊ฒฝ๋กœ๊ฐ€ /*์ด๋ฉด, ํŠธ๋ž˜ํ”ฝ์€ k8s-default-productp-d3c7ff7881๋กœ ์ „๋‹ฌ
  • ๋„๋ฉ”์ธ์ด kubeopsview.gagajin.com์ด๊ณ  ๊ฒฝ๋กœ๊ฐ€ /*์ด๋ฉด, ํŠธ๋ž˜ํ”ฝ์€ k8s-kubesyst-kubeopsv-d848006cb0๋กœ ์ „๋‹ฌ

Image

4. ๋ฐ˜๋ณต ์ ‘์† ์‹คํ–‰

1

while true; do curl -s -k https://bookinfo.$MyDomain/productpage | grep -o "<title>.*</title>" ; echo "--------------" ; sleep 1; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

<title>Simple Bookstore App</title>
--------------
<title>Simple Bookstore App</title>
--------------
<title>Simple Bookstore App</title>
--------------
<title>Simple Bookstore App</title>
--------------
<title>Simple Bookstore App</title>
...

5. productpage ์ ‘์†

  • ์ดˆ๊ธฐ ์ง„์ž…

Image

  • ์ƒˆ๋กœ ๊ณ ์นจ ํ›„ Reviews์™€ Ratings ๋ณ€๊ฒฝ ์‚ฌํ•ญ ํ™•์ธ

Image

๐Ÿ“Š prometheus-stack ์„ค์น˜

1. helm ์ €์žฅ์†Œ ์ถ”๊ฐ€

1
2
3

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

"prometheus-community" already exists with the same configuration, skipping

2. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

cat monitor-values.yaml
prometheus:
  prometheusSpec:
    scrapeInterval: "15s"
    evaluationInterval: "15s"
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    retention: 5d
    retentionSize: "10GiB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 30Gi

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - prometheus.gagajin.com
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - grafana.gagajin.com
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

  persistence:
    enabled: true
    type: sts
    storageClassName: "gp3"
    accessModes:
      - ReadWriteOnce
    size: 20Gi

alertmanager:
  enabled: false
defaultRules:
  create: false
kubeControllerManager:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false

3. helm ๋ฐฐํฌ

1
2

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 69.3.1 \
-f monitor-values.yaml --create-namespace --namespace monitoring

โœ…ย ์ถœ๋ ฅ

  • ๋ฐฐํฌ ํ›„, ํ•ด๋‹น Ingress์™€ ์™ธ๋ถ€ ์ ‘๊ทผ ์„œ๋น„์Šค(ex. ๊ทธ๋ผํŒŒ๋‚˜)๋Š” ๋™์ผํ•œ ALB๋ฅผ ๊ณต์œ ํ•จ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

NAME: kube-prometheus-stack
LAST DEPLOYED: Wed Feb 26 00:42:35 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"

Get Grafana 'admin' user password by running:

  kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Access Grafana local instance:

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prometheus-stack" -oname)
  kubectl --namespace monitoring port-forward $POD_NAME 3000

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
  • 4๊ฐœ์˜ Ingress๊ฐ€ ํ•˜๋‚˜์˜ ALB๋ฅผ ๊ณต์œ ํ•˜๊ณ  ์žˆ์Œ

Image

๐Ÿ“ Logging in EKS

1. ๋ชจ๋“  ๋กœ๊น… ํ™œ์„ฑํ™”

1
2

aws eks update-cluster-config --region ap-northeast-2 --name $CLUSTER_NAME \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
    "update": {
        "id": "ea2559f4-c4d8-3661-8735-63667c8ff514",
        "status": "InProgress",
        "type": "LoggingUpdate",
        "params": [
            {
                "type": "ClusterLogging",
                "value": "{\"clusterLogging\":[{\"types\":[\"api\",\"audit\",\"authenticator\",\"controllerManager\",\"scheduler\"],\"enabled\":true}]}"
            }
        ],
        "createdAt": "2025-02-26T00:53:57.522000+09:00",
        "errors": []
    }
}

2. ๋กœ๊ทธ ๊ทธ๋ฃน ํ™•์ธ

1

aws logs describe-log-groups | jq

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "logGroups": [
    {
      "logGroupName": "/aws/eks/myeks/cluster",
      "creationTime": 1740498852305,
      "metricFilterCount": 0,
      "arn": "arn:aws:logs:ap-northeast-2:378102432899:log-group:/aws/eks/myeks/cluster:*",
      "storedBytes": 0,
      "logGroupClass": "STANDARD",
      "logGroupArn": "arn:aws:logs:ap-northeast-2:378102432899:log-group:/aws/eks/myeks/cluster"
    }
  ]
}
  • ๋กœ๊ทธ ๊ทธ๋ฃน์€ AWS CloudWatch์˜ Log Groups > Log Streams์—์„œ ํ™•์ธ ๊ฐ€๋Šฅ

Image

  • Log Streams ์ค‘ ํ•˜๋‚˜๋ฅผ ์„ ํƒํ•œ ํ™”๋ฉด

Image

  • ํด๋ฆญ ์‹œ ๊ฐ ๋กœ๊ทธ์˜ ์ƒ์„ธ ์ •๋ณด ํ™•์ธ ๊ฐ€๋Šฅ

Image

3. ์‹ ๊ทœ ๋กœ๊ทธ ์‹ค์‹œ๊ฐ„ ์ถœ๋ ฅ

1

aws logs tail /aws/eks/$CLUSTER_NAME/cluster --follow

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11

2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2b3c4273-a038-4a6f-abd5-edc834383b56","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s","verb":"update","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"kube-scheduler/v1.31.5 (linux/amd64) kubernetes/226ac90/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"kube-scheduler","uid":"01d6ca7e-dd37-473d-aa26-01494c12e266","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"1724"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:03.969196Z","stageTimestamp":"2025-02-25T13:11:03.979371Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4df656f9-d98e-4235-82df-765e8fbff0f6","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"kube-controller-manager/v1.31.5 (linux/amd64) kubernetes/226ac90/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"kube-controller-manager","uid":"c29f94f4-c32a-4257-bb39-a55ef1a5d344","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"1725"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.040056Z","stageTimestamp":"2025-02-25T13:11:04.047041Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7561cbc1-669c-483d-beb4-e2d5f476db84","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cp-vpc-resource-controller","verb":"update","user":{"username":"eks:vpc-resource-controller","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cp-vpc-resource-controller","uid":"f59db5c0-b9d4-44a6-ae4a-5ac89057d314","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"1715"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.141659Z","stageTimestamp":"2025-02-25T13:11:04.149050Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks-vpc-resource-controller-rolebinding/kube-system\" of Role \"eks-vpc-resource-controller-role\" to User \"eks:vpc-resource-controller\""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"1161d386-d9d5-4fb4-b502-badd4b8183f2","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/roles?allowWatchBookmarks=true\u0026resourceVersion=1\u0026timeout=7m40s\u0026timeoutSeconds=460\u0026watch=true","verb":"watch","user":{"username":"system:apiserver","uid":"9ca9c2e9-c750-4409-8614-48aa8a617154","groups":["system:authenticated","system:masters"]},"sourceIPs":["::1"],"userAgent":"kube-apiserver/v1.31.5 (linux/amd64) kubernetes/226ac90","objectRef":{"resource":"roles","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:03:24.390790Z","stageTimestamp":"2025-02-25T13:11:04.391153Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"1c3da5ba-2fbc-4d35-b44b-ffac911ef1a2","stage":"ResponseComplete","requestURI":"/api/v1/resourcequotas?allowWatchBookmarks=true\u0026resourceVersion=1\u0026timeout=7m40s\u0026timeoutSeconds=460\u0026watch=true","verb":"watch","user":{"username":"system:apiserver","uid":"9ca9c2e9-c750-4409-8614-48aa8a617154","groups":["system:authenticated","system:masters"]},"sourceIPs":["::1"],"userAgent":"kube-apiserver/v1.31.5 (linux/amd64) kubernetes/226ac90","objectRef":{"resource":"resourcequotas","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:03:24.390287Z","stageTimestamp":"2025-02-25T13:11:04.391156Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"16bbd516-cfef-4d53-b89c-c4f1748d4298","stage":"ResponseStarted","requestURI":"/api/v1/resourcequotas?allowWatchBookmarks=true\u0026resourceVersion=1719\u0026timeout=7m28s\u0026timeoutSeconds=448\u0026watch=true","verb":"watch","user":{"username":"system:apiserver","uid":"9ca9c2e9-c750-4409-8614-48aa8a617154","groups":["system:authenticated","system:masters"]},"sourceIPs":["::1"],"userAgent":"kube-apiserver/v1.31.5 (linux/amd64) kubernetes/226ac90","objectRef":{"resource":"resourcequotas","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.391651Z","stageTimestamp":"2025-02-25T13:11:04.392143Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"24eccec6-3653-47a8-aeda-a673e79def24","stage":"ResponseStarted","requestURI":"/apis/rbac.authorization.k8s.io/v1/roles?allowWatchBookmarks=true\u0026resourceVersion=1709\u0026timeout=5m5s\u0026timeoutSeconds=305\u0026watch=true","verb":"watch","user":{"username":"system:apiserver","uid":"9ca9c2e9-c750-4409-8614-48aa8a617154","groups":["system:authenticated","system:masters"]},"sourceIPs":["::1"],"userAgent":"kube-apiserver/v1.31.5 (linux/amd64) kubernetes/226ac90","objectRef":{"resource":"roles","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.391661Z","stageTimestamp":"2025-02-25T13:11:04.392143Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"bf6b9d21-54b8-41a6-805f-2876e4e548a7","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-coredns-autoscaler","verb":"get","user":{"username":"eks:coredns-autoscaler","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"eks-coredns-autoscaler","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.726606Z","stageTimestamp":"2025-02-25T13:11:04.736303Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:coredns-autoscaler/kube-system\" of Role \"eks:coredns-autoscaler\" to User \"eks:coredns-autoscaler\""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"e923c96c-37ad-46e1-b8cf-cfdd70a223b4","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-coredns-autoscaler","verb":"update","user":{"username":"eks:coredns-autoscaler","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"eks-coredns-autoscaler","uid":"932dade7-3029-44d3-97fe-d3e2ce464d77","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"1717"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:04.737429Z","stageTimestamp":"2025-02-25T13:11:04.744137Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:coredns-autoscaler/kube-system\" of Role \"eks:coredns-autoscaler\" to User \"eks:coredns-autoscaler\""}}
2025-02-25T15:54:13.360000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"6ae84f9d-e668-48d2-ad5b-b2ce59de7016","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s","verb":"update","user":{"username":"eks:cloud-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"aws-cloud-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cloud-controller-manager","uid":"1c063b11-ea0f-4687-957a-64abd9ba643f","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"1726"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:11:05.063753Z","stageTimestamp":"2025-02-25T13:11:05.069911Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"eks:cloud-controller-manager\" of ClusterRole \"eks:cloud-controller-manager\" to User \"eks:cloud-controller-manager\""}}
...

4. ํ•„ํ„ฐ ์ ์šฉ ์‹ ๊ทœ ๋กœ๊ทธ ์‹ค์‹œ๊ฐ„ ์ถœ๋ ฅ

  • kube-proxy
1

aws logs tail /aws/eks/$CLUSTER_NAME/cluster --filter-pattern kube-proxy

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12

2025-02-25T15:54:16.406000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a82382c7-8df1-404c-bf42-2ef6298fe4b1","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-ebs-csi-aws-com","verb":"update","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=865e7205-fc52-4566-9fce-c2cff49b97e0"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-snapshotter-leader-ebs-csi-aws-com","uid":"92ac90d6-7010-493c-b66b-e2ae3b219877","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"14090"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:19.049041Z","stageTimestamp":"2025-02-25T14:00:19.058387Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:54:16.406000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"35842df4-2154-4f96-85ee-fda63be025df","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s","verb":"get","user":{"username":"eks:cloud-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.166.208"],"userAgent":"aws-cloud-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cloud-controller-manager","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:19.276815Z","stageTimestamp":"2025-02-25T14:00:19.281496Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"eks:cloud-controller-manager\" of ClusterRole \"eks:cloud-controller-manager\" to User \"eks:cloud-controller-manager\""}}
2025-02-25T15:54:16.406000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"3af25ed2-f450-411c-be26-fb4b8c4705c6","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-certificates-controller","verb":"get","user":{"username":"eks:certificate-controller","groups":["system:authenticated"]},"sourceIPs":["10.0.166.208"],"userAgent":"eks-certificates-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"eks-certificates-controller","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:19.597957Z","stageTimestamp":"2025-02-25T14:00:19.602796Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:certificate-controller/kube-system\" of Role \"eks:certificate-controller\" to User \"eks:certificate-controller\""}}
2025-02-25T15:54:16.406000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"c1f85bd8-e8a2-401f-933c-7b0719df8801","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-192-168-2-42.ap-northeast-2.compute.internal?timeout=10s","verb":"update","user":{"username":"system:node:ip-192-168-2-42.ap-northeast-2.compute.internal","uid":"aws-iam-authenticator:378102432899:AROAVQCFJISBU4MUNGJ6F","groups":["system:nodes","system:authenticated"],"extra":{"accessKeyId":["ASIAVQCFJISBXQFDTLPR"],"arn":["arn:aws:sts::378102432899:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-L6JI06tBfF9M/i-05e7d5c353553c7ff"],"canonicalArn":["arn:aws:iam::378102432899:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-L6JI06tBfF9M"],"principalId":["AROAVQCFJISBU4MUNGJ6F"],"sessionName":["i-05e7d5c353553c7ff"],"sigs.k8s.io/aws-iam-authenticator/principalId":["AROAVQCFJISBU4MUNGJ6F"]}},"sourceIPs":["192.168.2.42"],"userAgent":"kubelet/v1.31.5 (linux/amd64) kubernetes/5fcf7ca","objectRef":{"resource":"leases","namespace":"kube-node-lease","name":"ip-192-168-2-42.ap-northeast-2.compute.internal","uid":"8aede9d1-e662-4a6e-8485-9111f8361d31","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"14075"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:20.402559Z","stageTimestamp":"2025-02-25T14:00:20.411611Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:54:16.406000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"39c6e8c0-3efc-4487-bbf9-c91b02f80ab3","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-ebs-csi-aws-com","verb":"update","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=865e7205-fc52-4566-9fce-c2cff49b97e0"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-resizer-ebs-csi-aws-com","uid":"3f6b69bd-79a5-4eec-afbb-2e4dde349fd2","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"14098"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:20.535694Z","stageTimestamp":"2025-02-25T14:00:20.549846Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"1e99e72b-20d7-40ac-8c19-7d0993709028","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s","verb":"get","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.166.208"],"userAgent":"kube-controller-manager/v1.31.5 (linux/amd64) kubernetes/226ac90/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"kube-controller-manager","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:21.019326Z","stageTimestamp":"2025-02-25T14:00:21.023031Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"00642e57-9ceb-42e9-970b-d2a37f34246b","stage":"ResponseComplete","requestURI":"/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true\u0026resourceVersion=12312\u0026timeout=7m14s\u0026timeoutSeconds=434\u0026watch=true","verb":"watch","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=865e7205-fc52-4566-9fce-c2cff49b97e0"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"storageclasses","apiGroup":"storage.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T13:53:07.025573Z","stageTimestamp":"2025-02-25T14:00:21.027134Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"ebs-csi-provisioner-binding\" of ClusterRole \"ebs-external-provisioner-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"3145c12d-9b32-407d-8a52-4197e4992f8e","stage":"ResponseStarted","requestURI":"/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true\u0026resourceVersion=14112\u0026timeout=9m27s\u0026timeoutSeconds=567\u0026watch=true","verb":"watch","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=865e7205-fc52-4566-9fce-c2cff49b97e0"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"storageclasses","apiGroup":"storage.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:21.028988Z","stageTimestamp":"2025-02-25T14:00:21.029785Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"ebs-csi-provisioner-binding\" of ClusterRole \"ebs-external-provisioner-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"d34b1bc7-7757-408a-a570-0b14475f7903","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/ebs-csi-aws-com","verb":"update","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=865e7205-fc52-4566-9fce-c2cff49b97e0"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"ebs-csi-aws-com","uid":"1f757aa3-b7b7-4f26-a1b4-f45e6cd16f2e","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"14105"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:21.830745Z","stageTimestamp":"2025-02-25T14:00:21.839082Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5dee7581-096d-4e9c-9881-44c3e323292c","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s","verb":"get","user":{"username":"eks:cloud-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.166.208"],"userAgent":"aws-cloud-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cloud-controller-manager","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:22.000008Z","stageTimestamp":"2025-02-25T14:00:22.004881Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"eks:cloud-controller-manager\" of ClusterRole \"eks:cloud-controller-manager\" to User \"eks:cloud-controller-manager\""}}
2025-02-25T15:54:16.407000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5702a190-62e1-4c05-8a38-47eaa94d7374","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-ebs-csi-aws-com","verb":"get","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=31351c1c-35cf-4748-a3f2-851d2e72fa8b"],"authentication.kubernetes.io/node-name":["ip-192-168-2-42.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["65d9b899-3d7d-4a54-a9b7-7fd4c0be0af4"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-p57xw"],"authentication.kubernetes.io/pod-uid":["4cf5ca4a-fae6-490f-9757-6fb2f93248bf"]}},"sourceIPs":["192.168.2.47"],"userAgent":"csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-resizer-ebs-csi-aws-com","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T14:00:22.178659Z","stageTimestamp":"2025-02-25T14:00:22.182656Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
...

5. ๋กœ๊ทธ ์ŠคํŠธ๋ฆผ ์‹ค์‹œ๊ฐ„ ์ถœ๋ ฅ

  • kube-apiserver ๋กœ๊ทธ ํ™•์ธ
1

aws logs tail /aws/eks/$CLUSTER_NAME/cluster --log-stream-name-prefix kube-apiserver --follow

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13

2025-02-25T15:59:06.241000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"b06fe424-afeb-4e3f-910b-e05548a7b5b2","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-192-168-2-42.ap-northeast-2.compute.internal?timeout=10s","verb":"update","user":{"username":"system:node:ip-192-168-2-42.ap-northeast-2.compute.internal","uid":"aws-iam-authenticator:378102432899:AROAVQCFJISBU4MUNGJ6F","groups":["system:nodes","system:authenticated"],"extra":{"accessKeyId":["ASIAVQCFJISBVL7MICKQ"],"arn":["arn:aws:sts::378102432899:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-L6JI06tBfF9M/i-05e7d5c353553c7ff"],"canonicalArn":["arn:aws:iam::378102432899:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-L6JI06tBfF9M"],"principalId":["AROAVQCFJISBU4MUNGJ6F"],"sessionName":["i-05e7d5c353553c7ff"],"sigs.k8s.io/aws-iam-authenticator/principalId":["AROAVQCFJISBU4MUNGJ6F"]}},"sourceIPs":["192.168.2.42"],"userAgent":"kubelet/v1.31.5 (linux/amd64) kubernetes/5fcf7ca","objectRef":{"resource":"leases","namespace":"kube-node-lease","name":"ip-192-168-2-42.ap-northeast-2.compute.internal","uid":"8aede9d1-e662-4a6e-8485-9111f8361d31","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46557"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.088440Z","stageTimestamp":"2025-02-25T15:59:06.097720Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
2025-02-25T15:59:06.323000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5bd76ccc-9d05-4ebc-9125-32f27207c823","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"kube-controller-manager/v1.31.5 (linux/amd64) kubernetes/226ac90/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"kube-controller-manager","uid":"c29f94f4-c32a-4257-bb39-a55ef1a5d344","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46592"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:05.978575Z","stageTimestamp":"2025-02-25T15:59:05.989628Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
2025-02-25T15:59:06.323000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"87f7d01e-cee5-49e6-bdd1-7b8196c41235","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-coredns-autoscaler","verb":"get","user":{"username":"eks:coredns-autoscaler","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"eks-coredns-autoscaler","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.137901Z","stageTimestamp":"2025-02-25T15:59:06.143032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:coredns-autoscaler/kube-system\" of Role \"eks:coredns-autoscaler\" to User \"eks:coredns-autoscaler\""}}
2025-02-25T15:59:06.323000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7478ba71-efe6-4415-85eb-4c2836c1e572","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-coredns-autoscaler","verb":"update","user":{"username":"eks:coredns-autoscaler","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"eks-coredns-autoscaler","uid":"932dade7-3029-44d3-97fe-d3e2ce464d77","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46580"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.143926Z","stageTimestamp":"2025-02-25T15:59:06.156049Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:coredns-autoscaler/kube-system\" of Role \"eks:coredns-autoscaler\" to User \"eks:coredns-autoscaler\""}}
2025-02-25T15:59:06.573000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ef9046f2-1f70-41d3-a7a2-f4d64f57e4a4","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/aws-load-balancer-controller-leader","verb":"update","user":{"username":"system:serviceaccount:kube-system:aws-load-balancer-controller","uid":"995573f3-a0e4-4424-9868-81cbf2295751","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=e5b23c6c-a2ce-448f-9aa8-0422d01ebae3"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["aws-load-balancer-controller-554fbd9d-kbctn"],"authentication.kubernetes.io/pod-uid":["dc66f5b2-ef3f-4ea7-908f-0db4fbd2f275"]}},"sourceIPs":["192.168.1.114"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"aws-load-balancer-controller-leader","uid":"29062207-ba15-45fd-a7f3-52b4580d16f3","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46593"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.198663Z","stageTimestamp":"2025-02-25T15:59:06.221411Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"aws-load-balancer-controller-leader-election-rolebinding/kube-system\" of Role \"aws-load-balancer-controller-leader-election-role\" to ServiceAccount \"aws-load-balancer-controller/kube-system\""}}
2025-02-25T15:59:06.828000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"abab7321-e086-4097-b246-924c9f1769b6","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-attacher-leader-ebs-csi-aws-com","verb":"get","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=ebc1f5d6-37da-45f3-b291-82cf0fcc6367"],"authentication.kubernetes.io/node-name":["ip-192-168-2-42.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["65d9b899-3d7d-4a54-a9b7-7fd4c0be0af4"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-p57xw"],"authentication.kubernetes.io/pod-uid":["4cf5ca4a-fae6-490f-9757-6fb2f93248bf"]}},"sourceIPs":["192.168.2.47"],"userAgent":"csi-attacher/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-attacher-leader-ebs-csi-aws-com","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.454289Z","stageTimestamp":"2025-02-25T15:59:06.457958Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:59:07.075000+00:00 kube-apiserver-audit-4e3006095bad62c5b3575ab7dbd2cfbb {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"47e16d35-f40b-4fdf-a301-4f1bd38e5879","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s","verb":"update","user":{"username":"eks:cloud-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.116.214"],"userAgent":"aws-cloud-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cloud-controller-manager","uid":"1c063b11-ea0f-4687-957a-64abd9ba643f","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46596"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.593881Z","stageTimestamp":"2025-02-25T15:59:06.601897Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"eks:cloud-controller-manager\" of ClusterRole \"eks:cloud-controller-manager\" to User \"eks:cloud-controller-manager\""}}
2025-02-25T15:59:07.244000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7715eca9-a8e1-4752-a460-0515516362fd","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-attacher-leader-ebs-csi-aws-com","verb":"update","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=09a67c38-ccca-4b60-aed4-964aac7bfc6b"],"authentication.kubernetes.io/node-name":["ip-192-168-1-51.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["58a47c72-0c03-4bac-b427-6b6b69b61305"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-z4t4z"],"authentication.kubernetes.io/pod-uid":["5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee"]}},"sourceIPs":["192.168.1.226"],"userAgent":"csi-attacher/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-attacher-leader-ebs-csi-aws-com","uid":"a567568d-f785-4767-b0a2-204ca3a0fb95","apiGroup":"coordination.k8s.io","apiVersion":"v1","resourceVersion":"46581"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:06.183577Z","stageTimestamp":"2025-02-25T15:59:06.190737Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:59:07.244000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"76427af1-8ce6-467b-8c5e-1d54f3ba12b5","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-ebs-csi-aws-com","verb":"get","user":{"username":"system:serviceaccount:kube-system:ebs-csi-controller-sa","uid":"3f266af7-bcc9-4b75-9cd1-ef47f6fc1abb","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=ebc1f5d6-37da-45f3-b291-82cf0fcc6367"],"authentication.kubernetes.io/node-name":["ip-192-168-2-42.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["65d9b899-3d7d-4a54-a9b7-7fd4c0be0af4"],"authentication.kubernetes.io/pod-name":["ebs-csi-controller-7f8f8cb84-p57xw"],"authentication.kubernetes.io/pod-uid":["4cf5ca4a-fae6-490f-9757-6fb2f93248bf"]}},"sourceIPs":["192.168.2.47"],"userAgent":"csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"leases","namespace":"kube-system","name":"external-resizer-ebs-csi-aws-com","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:07.058273Z","stageTimestamp":"2025-02-25T15:59:07.063437Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ebs-csi-leases-rolebinding/kube-system\" of Role \"ebs-csi-leases-role\" to ServiceAccount \"ebs-csi-controller-sa/kube-system\""}}
2025-02-25T15:59:07.244000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7edc0aeb-e68d-40e2-96a5-5a795501f0e8","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/amazon-network-policy-controller-k8s","verb":"get","user":{"username":"eks:network-policy-controller","groups":["system:authenticated"]},"sourceIPs":["10.0.166.208"],"userAgent":"controller/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"amazon-network-policy-controller-k8s","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:07.064113Z","stageTimestamp":"2025-02-25T15:59:07.070093Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"eks:network-policy-controller/kube-system\" of Role \"eks:network-policy-controller\" to User \"eks:network-policy-controller\""}}
2025-02-25T15:59:07.244000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"a2b02ed1-1be2-463c-8397-ff0bdbb72a96","stage":"ResponseComplete","requestURI":"/api/v1/nodes","verb":"list","user":{"username":"system:serviceaccount:kube-system:kube-ops-view","uid":"36439bb7-a7d4-464c-95fe-cfb4ff623176","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=220d0150-9f1a-4707-b10c-070055e30a53"],"authentication.kubernetes.io/node-name":["ip-192-168-2-42.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["65d9b899-3d7d-4a54-a9b7-7fd4c0be0af4"],"authentication.kubernetes.io/pod-name":["kube-ops-view-657dbc6cd8-pxkvr"],"authentication.kubernetes.io/pod-uid":["37bf2637-622f-4ee2-9804-a73d74e16f0e"]}},"sourceIPs":["192.168.2.249"],"userAgent":"pykube-ng/20.4.1","objectRef":{"resource":"nodes","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:07.188115Z","stageTimestamp":"2025-02-25T15:59:07.196179Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:kube-system:kube-ops-view, seconds after warning threshold: 959","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kube-ops-view\" of ClusterRole \"kube-ops-view\" to ServiceAccount \"kube-ops-view/kube-system\""}}
2025-02-25T15:59:07.244000+00:00 kube-apiserver-audit-4ff1d80de4851cc512375bb0568780f7 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"6265477b-e567-4f35-89e4-bdbaf878bc61","stage":"ResponseComplete","requestURI":"/api/v1/pods","verb":"list","user":{"username":"system:serviceaccount:kube-system:kube-ops-view","uid":"36439bb7-a7d4-464c-95fe-cfb4ff623176","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=220d0150-9f1a-4707-b10c-070055e30a53"],"authentication.kubernetes.io/node-name":["ip-192-168-2-42.ap-northeast-2.compute.internal"],"authentication.kubernetes.io/node-uid":["65d9b899-3d7d-4a54-a9b7-7fd4c0be0af4"],"authentication.kubernetes.io/pod-name":["kube-ops-view-657dbc6cd8-pxkvr"],"authentication.kubernetes.io/pod-uid":["37bf2637-622f-4ee2-9804-a73d74e16f0e"]}},"sourceIPs":["192.168.2.249"],"userAgent":"pykube-ng/20.4.1","objectRef":{"resource":"pods","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-02-25T15:59:07.202326Z","stageTimestamp":"2025-02-25T15:59:07.226347Z","annotations":{"authentication.k8s.io/stale-token":"subject: system:serviceaccount:kube-system:kube-ops-view, seconds after warning threshold: 959","authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kube-ops-view\" of ClusterRole \"kube-ops-view\" to ServiceAccount \"kube-ops-view/kube-system\""}}
...
  • CoreDNS scale ์ถ•์†Œ (2๊ฐœ โ†’ 1๊ฐœ)
1
2
3

(eks-user@myeks:default) [root@operator-host ~]# kubectl scale deployment -n kube-system coredns --replicas=1
# ๊ฒฐ๊ณผ
deployment.apps/coredns scaled

1

aws logs tail /aws/eks/$CLUSTER_NAME/cluster --log-stream-name-prefix kube-controller-manager --follow

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9

2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.919710       9 replica_set.go:624] "Too many replicas" logger="replicaset-controller" replicaSet="kube-system/coredns-86f5954566" need=1 deleting=1
2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.919757       9 replica_set.go:251] "Found related ReplicaSets" logger="replicaset-controller" replicaSet="kube-system/coredns-86f5954566" relatedReplicaSets=["kube-system/coredns-86f5954566"]
2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.919823       9 controller_utils.go:608] "Deleting pod" logger="replicaset-controller" controller="coredns-86f5954566" pod="kube-system/coredns-86f5954566-c8wl2"
2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.980081       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="60.447223ms"
2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.993874       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="13.736734ms"
2025-02-25T16:11:02.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:02.996455       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="79.629ยตs"
2025-02-25T16:11:08.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:08.243402       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="102.085ยตs"
2025-02-25T16:11:09.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:09.191159       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="101.657ยตs"
2025-02-25T16:11:09.000000+00:00 kube-controller-manager-4e3006095bad62c5b3575ab7dbd2cfbb I0225 16:11:09.200150       9 replica_set.go:679] "Finished syncing" logger="replicaset-controller" kind="ReplicaSet" key="kube-system/coredns-86f5954566" duration="121.163ยตs"

6. CloudWatch Log Insights

(1) /aws/eks/myeks/cluster ๋กœ๊ทธ ๊ทธ๋ฃน์„ ์„ ํƒ ํ›„ ์ฟผ๋ฆฌ ์‹คํ–‰

Image

(2) EC2 Instance๊ฐ€ NodeNotReady ์ƒํƒœ์ธ ๋กœ๊ทธ ๊ฒ€์ƒ‰

1
2
3

fields @timestamp, @message
| filter @message like /NodeNotReady/
| sort @timestamp desc

ํ˜„์žฌ ๋…ธ๋“œ์— ์ด์ƒ์ด ์—†์–ด ๊ฒฐ๊ณผ๊ฐ€ ์—†์Œ

Image

(3) kube-apiserver-audit ๋กœ๊ทธ์—์„œ userAgent ์ •๋ ฌํ•ด์„œ ์•„๋ž˜ 4๊ฐœ ํ•„๋“œ ์ •๋ณด ๊ฒ€์ƒ‰

1
2
3
4

fields userAgent, requestURI, @timestamp, @message
| filter @logStream ~= "kube-apiserver-audit"
| stats count(userAgent) as count by userAgent
| sort count desc

Image

7. ๋กœ๊น… ๋„๊ธฐ

(1) EKS Control Plane ๋กœ๊น…(CloudWatch Logs) ๋น„ํ™œ์„ฑํ™”

1

eksctl utils update-cluster-logging --cluster $CLUSTER_NAME --region ap-northeast-2 --disable-types all --approve

โœ…ย ์ถœ๋ ฅ

1
2

2025-02-26 01:21:15 [โ„น]  will update CloudWatch logging for cluster "myeks" in "ap-northeast-2" (no types to enable & disable types: api, audit, authenticator, controllerManager, scheduler)
2025-02-26 01:21:46 [โœ”]  configured CloudWatch logging for cluster "myeks" in "ap-northeast-2" (no types enabled & disabled types: api, audit, authenticator, controllerManager, scheduler)

(2) ๋กœ๊ทธ ๊ทธ๋ฃน ์‚ญ์ œ

1

aws logs delete-log-group --log-group-name /aws/eks/$CLUSTER_NAME/cluster

๐Ÿณ ์ปจํ…Œ์ด๋„ˆ ํŒŒ๋“œ ๋กœ๊น…

1. NGINX ์›น์„œ๋ฒ„ ๋ฐฐํฌ with Ingress(ALB)

1
2
3
4
5
6
7
8
9
10
11

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# ๊ฒฐ๊ณผ
"bitnami" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "geek-cookbook" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. โŽˆHappy Helming!โŽˆ

2. ๋„๋ฉ”์ธ, ์ธ์ฆ์„œ ํ™•์ธ

1

echo $MyDomain $CERT_ARN

โœ…ย ์ถœ๋ ฅ

1

gagajin.com arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b

3. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

cat <<EOT > nginx-values.yaml
service:
  type: NodePort
  
networkPolicy:
  enabled: false
  
resourcesPreset: "nano"

ingress:
  enabled: true
  ingressClassName: alb
  hostname: nginx.$MyDomain
  pathType: Prefix
  path: /
  annotations: 
    alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/load-balancer-name: $CLUSTER_NAME-ingress-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/target-type: ip
EOT

4. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์กฐํšŒ

1

cat nginx-values.yaml

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

service:
  type: NodePort
  
networkPolicy:
  enabled: false
  
resourcesPreset: "nano"

ingress:
  enabled: true
  ingressClassName: alb
  hostname: nginx.gagajin.com
  pathType: Prefix
  path: /
  annotations: 
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/target-type: ip

5. ๋ฐฐํฌ

1

helm install nginx bitnami/nginx --version 19.0.0 -f nginx-values.yaml

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

NAME: nginx
LAST DEPLOYED: Wed Feb 26 21:22:14 2025
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: nginx
CHART VERSION: 19.0.0
APP VERSION: 1.27.4

Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami for more information.

** Please be patient while the chart is being deployed **
NGINX can be accessed through the following DNS name from within your cluster:

    nginx.default.svc.cluster.local (port 80)

To access NGINX from outside the cluster, follow the steps below:

1. Get the NGINX URL and associate its hostname to your cluster external IP:

   export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
   echo "NGINX URL: http://nginx.gagajin.com"
   echo "$CLUSTER_IP  nginx.gagajin.com" | sudo tee -a /etc/hosts

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - cloneStaticSiteFromGit.gitSync.resources
  - resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

โš  SECURITY WARNING: Original containers have been substituted. This Helm chart was designed, tested, and validated on multiple platforms using a specific set of Bitnami and Tanzu Application Catalog containers. Substituting other containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Substituted images detected:
  - docker.io/bitnami/nginx:1.27.4-debian-12-r1

โš  WARNING: Original containers have been retagged. Please note this Helm chart was tested, and validated on multiple platforms using a specific set of Tanzu Application Catalog containers. Substituting original image tags could cause unexpected behavior.

Retagged images:
  - docker.io/bitnami/nginx:1.27.4-debian-12-r1

6. Ingress, Deployment, ์„œ๋น„์Šค, ์—”๋“œํฌ์ธํŠธ ์กฐํšŒ - Nginx

1

kubectl get ingress,deploy,svc,ep nginx

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11

NAME                              CLASS   HOSTS               ADDRESS                                                       PORTS   AGE
ingress.networking.k8s.io/nginx   alb     nginx.gagajin.com   myeks-ingress-alb-77245841.ap-northeast-2.elb.amazonaws.com   80      64s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           64s

NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/nginx   NodePort   10.100.47.81   <none>        80:32507/TCP,443:32693/TCP   64s

NAME              ENDPOINTS                             AGE
endpoints/nginx   192.168.2.63:8443,192.168.2.63:8080   64s

7. ๋ฐฐํฌ ์ƒ์„ธ ์ •๋ณด ํ™•์ธ - Nginx

1

kubectl describe deploy nginx

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

Name:                   nginx
Namespace:              default
CreationTimestamp:      Wed, 26 Feb 2025 21:22:15 +0900
Labels:                 app.kubernetes.io/instance=nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=nginx
                        app.kubernetes.io/version=1.27.4
                        helm.sh/chart=nginx-19.0.0
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: nginx
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/instance=nginx,app.kubernetes.io/name=nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=nginx
                    app.kubernetes.io/version=1.27.4
                    helm.sh/chart=nginx-19.0.0
  Service Account:  nginx
  Init Containers:
   preserve-logs-symlinks:
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -ec
      #!/bin/bash
      . /opt/bitnami/scripts/libfs.sh
      # We copy the logs folder because it has symlinks to stdout and stderr
      if ! is_dir_empty /opt/bitnami/nginx/logs; then
        cp -r /opt/bitnami/nginx/logs /emptydir/app-logs-dir
      fi
      
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Environment:          <none>
    Mounts:
      /emptydir from empty-dir (rw)
  Containers:
   nginx:
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Ports:           8080/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             tcp-socket :http delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:            http-get http://:http/ delay=5s timeout=3s period=5s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:            false
      NGINX_HTTP_PORT_NUMBER:   8080
      NGINX_HTTPS_PORT_NUMBER:  8443
    Mounts:
      /certs from certificate (rw)
      /opt/bitnami/nginx/conf from empty-dir (rw,path="app-conf-dir")
      /opt/bitnami/nginx/logs from empty-dir (rw,path="app-logs-dir")
      /opt/bitnami/nginx/tmp from empty-dir (rw,path="app-tmp-dir")
      /tmp from empty-dir (rw,path="tmp-dir")
  Volumes:
   empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
   certificate:
    Type:          Secret (a volume populated by a Secret)
    SecretName:    nginx-tls
    Optional:      false
  Node-Selectors:  <none>
  Tolerations:     <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-7c94c9bdcb (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  102s  deployment-controller  Scaled up replica set nginx-7c94c9bdcb to 1

8. ALB TargetGroupBindings ํ™•์ธ

1

kubectl get targetgroupbindings

โœ…ย ์ถœ๋ ฅ

1
2
3

NAME                              SERVICE-NAME   SERVICE-PORT   TARGET-TYPE   AGE
k8s-default-nginx-342d095714      nginx          http           ip            2m8s
k8s-default-productp-d3c7ff7881   productpage    9080           ip            21h

9. ์ ‘์† ์ฃผ์†Œ ํ™•์ธ ๋ฐ ์ ‘์†

(1) ํ™•์ธ

1

echo -e "Nginx WebServer URL = https://nginx.$MyDomain"

โœ…ย ์ถœ๋ ฅ

1

Nginx WebServer URL = https://nginx.gagajin.com

(2) ์ ‘์†

1

curl -s https://nginx.$MyDomain

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

10. ๋ฐ˜๋ณต ์ ‘์†

nginx ๋กœ๊ทธ ๋ชจ๋‹ˆํ„ฐ๋ง์„ ์œ„ํ•ด ๋ฐ˜๋ณต ์ ‘์† ์„ค์ •

1

while true; do curl -s https://nginx.$MyDomain -I | head -n 1; date; sleep 1; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7

HTTP/2 200 
Wed Feb 26 09:27:31 PM KST 2025
HTTP/2 200 
Wed Feb 26 09:27:33 PM KST 2025
HTTP/2 200 
Wed Feb 26 09:27:34 PM KST 2025
...

โ˜๏ธ CloudWatch Container observability ์„ค์น˜

1. IRSA ์„ค์ •

1
2
3
4
5
6
7

eksctl create iamserviceaccount \
  --name cloudwatch-agent \
  --namespace amazon-cloudwatch --cluster $CLUSTER_NAME \
  --role-name $CLUSTER_NAME-cloudwatch-agent-role \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  --role-only \
  --approve

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

2025-02-26 21:46:00 [โ„น]  1 existing iamserviceaccount(s) (kube-system/aws-load-balancer-controller) will be excluded
2025-02-26 21:46:00 [โ„น]  1 iamserviceaccount (amazon-cloudwatch/cloudwatch-agent) was included (based on the include/exclude rules)
2025-02-26 21:46:00 [!]  serviceaccounts in Kubernetes will not be created or modified, since the option --role-only is used
2025-02-26 21:46:00 [โ„น]  1 task: { create IAM role for serviceaccount "amazon-cloudwatch/cloudwatch-agent" }
2025-02-26 21:46:00 [โ„น]  building iamserviceaccount stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-02-26 21:46:00 [โ„น]  deploying stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-02-26 21:46:00 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-02-26 21:46:30 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"

2. addon ๋ฐฐํฌ

(1) AWS ๊ณ„์ • ID ์กฐํšŒ ๋ฐ ํ™˜๊ฒฝ ๋ณ€์ˆ˜ ์„ค์ •

1

export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)

(2) EKS ํด๋Ÿฌ์Šคํ„ฐ์— amazon-cloudwatch-observability ์• ๋“œ์˜จ ๋ฐฐํฌ

1

aws eks create-addon --addon-name amazon-cloudwatch-observability --cluster-name $CLUSTER_NAME --service-account-role-arn arn:aws:iam::$ACCOUNT_ID:role/$CLUSTER_NAME-cloudwatch-agent-role

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "myeks",
        "status": "CREATING",
        "addonVersion": "v3.3.1-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:ap-northeast-2:378102432899:addon/myeks/amazon-cloudwatch-observability/b8caa12a-714e-a4b9-05b5-a368820767bf",
        "createdAt": "2025-02-26T21:56:54.393000+09:00",
        "modifiedAt": "2025-02-26T21:56:54.411000+09:00",
        "serviceAccountRoleArn": "arn:aws:iam::378102432899:role/myeks-cloudwatch-agent-role",
        "tags": {}
    }
}

3. addon ํ™•์ธ

1

aws eks list-addons --cluster-name myeks --output table

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12

---------------------------------------
|             ListAddons              |
+-------------------------------------+
||              addons               ||
|+-----------------------------------+|
||  amazon-cloudwatch-observability  ||
||  aws-ebs-csi-driver               ||
||  coredns                          ||
||  kube-proxy                       ||
||  metrics-server                   ||
||  vpc-cni                          ||
|+-----------------------------------+|
  • EKS ํด๋Ÿฌ์Šคํ„ฐ์— amazon-cloudwatch-observability ์• ๋“œ์˜จ์ด ์ถ”๊ฐ€๋จ

4. CRD ๋ฐฐํฌ ํ™•์ธ

1

kubectl get crd | grep -i cloudwatch

โœ…ย ์ถœ๋ ฅ

1
2
3
4

amazoncloudwatchagents.cloudwatch.aws.amazon.com   2025-02-26T12:57:16Z
dcgmexporters.cloudwatch.aws.amazon.com            2025-02-26T12:57:17Z
instrumentations.cloudwatch.aws.amazon.com         2025-02-26T12:57:17Z
neuronmonitors.cloudwatch.aws.amazon.com           2025-02-26T12:57:17Z
  • CloudWatch ๊ด€๋ จ 4๊ฐœ์˜ CRD๊ฐ€ ๋ฐฐํฌ๋จ

5. amazon-cloudwatch ๋„ค์ž„์ŠคํŽ˜์ด์Šค ๋ฆฌ์†Œ์Šค ์กฐํšŒ

1

kubectl get ds,pod,cm,sa,amazoncloudwatchagent -n amazon-cloudwatch

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

NAME                                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
daemonset.apps/cloudwatch-agent                              3         3         3       3            3           kubernetes.io/os=linux     14m
daemonset.apps/cloudwatch-agent-windows                      0         0         0       0            0           kubernetes.io/os=windows   14m
daemonset.apps/cloudwatch-agent-windows-container-insights   0         0         0       0            0           kubernetes.io/os=windows   14m
daemonset.apps/dcgm-exporter                                 0         0         0       0            0           kubernetes.io/os=linux     14m
daemonset.apps/fluent-bit                                    3         3         3       3            3           kubernetes.io/os=linux     14m
daemonset.apps/fluent-bit-windows                            0         0         0       0            0           kubernetes.io/os=windows   14m
daemonset.apps/neuron-monitor                                0         0         0       0            0           <none>                     14m

NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/amazon-cloudwatch-observability-controller-manager-6f76854spzlp   1/1     Running   0          14m
pod/cloudwatch-agent-87hnx                                            1/1     Running   0          14m
pod/cloudwatch-agent-h6rpg                                            1/1     Running   0          14m
pod/cloudwatch-agent-kfwzk                                            1/1     Running   0          14m
pod/fluent-bit-8264s                                                  1/1     Running   0          14m
pod/fluent-bit-9l69f                                                  1/1     Running   0          14m
pod/fluent-bit-tfjbr                                                  1/1     Running   0          14m

NAME                                                    DATA   AGE
configmap/cloudwatch-agent                              1      14m
configmap/cloudwatch-agent-windows                      1      14m
configmap/cloudwatch-agent-windows-container-insights   1      14m
configmap/cwagent-clusterleader                         0      14m
configmap/dcgm-exporter-config-map                      2      14m
configmap/fluent-bit-config                             5      14m
configmap/fluent-bit-windows-config                     5      14m
configmap/kube-root-ca.crt                              1      14m
configmap/neuron-monitor-config-map                     1      14m

NAME                                                                SECRETS   AGE
serviceaccount/amazon-cloudwatch-observability-controller-manager   0         14m
serviceaccount/cloudwatch-agent                                     0         14m
serviceaccount/dcgm-exporter-service-acct                           0         14m
serviceaccount/default                                              0         14m
serviceaccount/neuron-monitor-service-acct                          0         14m

NAME                                                                                          MODE        VERSION   READY   AGE   IMAGE   MANAGEMENT
amazoncloudwatchagent.cloudwatch.aws.amazon.com/cloudwatch-agent                              daemonset   0.0.0             14m           managed
amazoncloudwatchagent.cloudwatch.aws.amazon.com/cloudwatch-agent-windows                      daemonset   0.0.0             14m           managed
amazoncloudwatchagent.cloudwatch.aws.amazon.com/cloudwatch-agent-windows-container-insights   daemonset   0.0.0             14m           managed
  • ๋ชจ๋“  ๋…ธ๋“œ์—์„œ ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•˜๊ธฐ ์œ„ํ•ด ๋ฆฌ๋ˆ…์Šค์šฉ cloudwatch-agent ๋ฐ๋ชฌ์…‹์ด ๋…ธ๋“œ ์ˆ˜์— ๋งž๊ฒŒ ๋ฐฐํฌ๋จ
  • Windows ๋…ธ๋“œ๋Š” ์กด์žฌํ•˜์ง€ ์•Š์•„ cloudwatch-agent-windows ๋ฐ๋ชฌ์…‹์€ 0์ž„

6. ๋…ธ๋“œ์˜ ๋กœ๊ทธ ํ™•์ธ

(1) application ๋กœ๊ทธ ์†Œ์Šค(All log files inย /var/log/containers โ†’ ์‹ฌ๋ณผ๋ฆญ ๋งํฌ /var/log/pods/<์ปจํ…Œ์ด๋„ˆ>, ๊ฐ ์ปจํ…Œ์ด๋„ˆ/ํŒŒ๋“œ ๋กœ๊ทธ

1

for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/containers; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

>>>>> 15.164.227.37 <<<<<
/var/log/containers
โ”œโ”€โ”€ aws-load-balancer-controller-554fbd9d-kbctn_kube-system_aws-load-balancer-controller-ae03ebdd899a3b4960dc6eaf290d27f46ceff51545b68f2dad3b013ed05cf9a4.log -> /var/log/pods/kube-system_aws-load-balancer-controller-554fbd9d-kbctn_dc66f5b2-ef3f-4ea7-908f-0db4fbd2f275/aws-load-balancer-controller/0.log
โ”œโ”€โ”€ aws-node-r5nbp_kube-system_aws-eks-nodeagent-97deaf50ee032a4728028115ecadc4df9800f763105986e526a375367aaff758.log -> /var/log/pods/kube-system_aws-node-r5nbp_d1476562-ea81-46d8-bc36-4da33468aaa5/aws-eks-nodeagent/0.log
โ”œโ”€โ”€ aws-node-r5nbp_kube-system_aws-node-b011407ecb43724bd80edbfe11b5a306901b691b00b96400e4ce7c84666a967a.log -> /var/log/pods/kube-system_aws-node-r5nbp_d1476562-ea81-46d8-bc36-4da33468aaa5/aws-node/0.log
โ”œโ”€โ”€ aws-node-r5nbp_kube-system_aws-vpc-cni-init-4c907842128039d3ff9c6ecaf89063504f5d80438084589f6a44445e6c54c591.log -> /var/log/pods/kube-system_aws-node-r5nbp_d1476562-ea81-46d8-bc36-4da33468aaa5/aws-vpc-cni-init/0.log
โ”œโ”€โ”€ cloudwatch-agent-kfwzk_amazon-cloudwatch_otc-container-b64522fc2d70ee5ad6d8de6d49d866d636764c86f9231f18c9eb004c8587b20b.log -> /var/log/pods/amazon-cloudwatch_cloudwatch-agent-kfwzk_a5b6a6d2-d0c3-45de-89ac-d9c865c553f4/otc-container/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_csi-attacher-e6c94cfbc24f90637b32b7f596daec2007e13a35c61794bd5429fc573d2de07a.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/csi-attacher/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_csi-provisioner-903745a8a728c5de43eca34f2012a52a6ef0fe103722048a135d82fab33c9226.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/csi-provisioner/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_csi-resizer-37d345a943121b4e5d348d3147b7928fbf94783ea87baf7e1a29bee600e44c6b.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/csi-resizer/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_csi-snapshotter-796eb70b19fb6e9a72c8cae8d8f52762c0d0da4431f45eeb10dfeed6de63172d.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/csi-snapshotter/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_ebs-plugin-d50cc0f88f910d81a23c77774d8fe84f451bfa825496e60ececc442552eb1639.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/ebs-plugin/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-z4t4z_kube-system_liveness-probe-8bb874394b3681e04338edd503c4a560c7ebabd2a6608261c88fa7e70529695b.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-z4t4z_5b150c0c-9d2b-4c5f-ba59-bbcff29bb0ee/liveness-probe/0.log
โ”œโ”€โ”€ ebs-csi-node-j46zg_kube-system_ebs-plugin-0824d0fbca762ba2532e33435a5b448a400f52b5cd7516fc7671a5845605d51e.log -> /var/log/pods/kube-system_ebs-csi-node-j46zg_d160163c-dd04-4259-ab96-f6850b5a407a/ebs-plugin/0.log
โ”œโ”€โ”€ ebs-csi-node-j46zg_kube-system_liveness-probe-3ca5cf0a203bb1eb886f05c02c308e79fd6e79777bbc98635d28e59fee8c88f8.log -> /var/log/pods/kube-system_ebs-csi-node-j46zg_d160163c-dd04-4259-ab96-f6850b5a407a/liveness-probe/0.log
โ”œโ”€โ”€ ebs-csi-node-j46zg_kube-system_node-driver-registrar-4bfca921825f106ecd08694b261b361499bc612b6e1b3b34537b30edaea8d251.log -> /var/log/pods/kube-system_ebs-csi-node-j46zg_d160163c-dd04-4259-ab96-f6850b5a407a/node-driver-registrar/0.log
โ”œโ”€โ”€ external-dns-dc4878f5f-fskxk_kube-system_external-dns-4c9f28bd84caed6a9f6e345991516527d2bcb271b1b161c5a58fd3c7710c7d5d.log -> /var/log/pods/kube-system_external-dns-dc4878f5f-fskxk_2f69ba60-e611-4718-8a4d-ea14f438867e/external-dns/0.log
โ”œโ”€โ”€ fluent-bit-tfjbr_amazon-cloudwatch_fluent-bit-948fb449154984e2ceff0ed1ca6cb86fdad141ca144e1995353f35647e7e0207.log -> /var/log/pods/amazon-cloudwatch_fluent-bit-tfjbr_6280ec3f-666d-485a-be7a-9756737ee8cd/fluent-bit/0.log
โ”œโ”€โ”€ kube-prometheus-stack-grafana-0_monitoring_grafana-07ff1d680b4d33ce368a00dcf109cb4278046c9d6b98b63c998de65312f4ac73.log -> /var/log/pods/monitoring_kube-prometheus-stack-grafana-0_b098dea3-a8ec-4fb1-a7b0-ff701a65f0ba/grafana/0.log
โ”œโ”€โ”€ kube-prometheus-stack-grafana-0_monitoring_grafana-sc-dashboard-d15b03ba0da8001c49f9ae652817fdd8b4b650dbd864b9e9fb772fcefb1645d4.log -> /var/log/pods/monitoring_kube-prometheus-stack-grafana-0_b098dea3-a8ec-4fb1-a7b0-ff701a65f0ba/grafana-sc-dashboard/0.log
โ”œโ”€โ”€ kube-prometheus-stack-grafana-0_monitoring_grafana-sc-datasources-31143a8a39f55140e75cbff4d3eb9d0e011582c8a15c2400287486b0f6f17014.log -> /var/log/pods/monitoring_kube-prometheus-stack-grafana-0_b098dea3-a8ec-4fb1-a7b0-ff701a65f0ba/grafana-sc-datasources/0.log
โ”œโ”€โ”€ kube-prometheus-stack-grafana-0_monitoring_init-chown-data-be60a46537cc11fbbb453426899a8b9918ac0a922e59505d0b0ebf9cbb6846a2.log -> /var/log/pods/monitoring_kube-prometheus-stack-grafana-0_b098dea3-a8ec-4fb1-a7b0-ff701a65f0ba/init-chown-data/0.log
โ”œโ”€โ”€ kube-prometheus-stack-prometheus-node-exporter-lt42c_monitoring_node-exporter-01861c25bd8c9d23b5535cf83af5dc3fea762d1cf1b35475dcbd351158c88cd1.log -> /var/log/pods/monitoring_kube-prometheus-stack-prometheus-node-exporter-lt42c_ec3fd0cb-6046-44ea-a6be-30e58ac0cbd2/node-exporter/0.log
โ”œโ”€โ”€ kube-proxy-v8nh9_kube-system_kube-proxy-a99b17ae17e49653b07611fd64abd6cb31026644be61bbc40bc4829cb9ad4fbc.log -> /var/log/pods/kube-system_kube-proxy-v8nh9_3a48dd47-bf53-4ec0-8060-2db03b661f8c/kube-proxy/0.log
โ”œโ”€โ”€ ratings-v1-65f797b499-fs82w_default_ratings-0bf6628633bfcca12bbdbbe1dfab7eb5c7ab97421a3fcc08ab3799287fc94dd5.log -> /var/log/pods/default_ratings-v1-65f797b499-fs82w_5ed60936-90e6-43bb-8ff9-cc48e8513767/ratings/0.log
โ””โ”€โ”€ reviews-v2-65cb66b45c-8rvgv_default_reviews-e1bf161e3331be5881dbc3c17a7ab70450454eb4242e8e4b0cd8d41ba5d3f74d.log -> /var/log/pods/default_reviews-v2-65cb66b45c-8rvgv_e577fc82-eff2-4eaf-9534-7b2a0e6d4851/reviews/0.log

0 directories, 24 files

>>>>> 3.38.205.159 <<<<<
/var/log/containers
โ”œโ”€โ”€ amazon-cloudwatch-observability-controller-manager-6f76854spzlp_amazon-cloudwatch_manager-21d672489b323c5ddde198ca793164cf1d83ada8a0aaa6ca1f271b17631174e3.log -> /var/log/pods/amazon-cloudwatch_amazon-cloudwatch-observability-controller-manager-6f76854spzlp_8c35db22-377e-4b33-819b-fa7324622171/manager/0.log
โ”œโ”€โ”€ aws-load-balancer-controller-554fbd9d-mtlsg_kube-system_aws-load-balancer-controller-f3c273a6d3458a8641e154652ae4ddac77f9dac11a9989c19a2d8b1e649f152a.log -> /var/log/pods/kube-system_aws-load-balancer-controller-554fbd9d-mtlsg_5819b483-9f6b-4aee-96ac-2a6308632fc3/aws-load-balancer-controller/0.log
โ”œโ”€โ”€ aws-node-qgc5t_kube-system_aws-eks-nodeagent-329a39ed4aa628e60b6de898061099e81899cf47eaea11b4de183bc047b0c273.log -> /var/log/pods/kube-system_aws-node-qgc5t_310d7c55-bc9e-474a-928f-b19218586bb2/aws-eks-nodeagent/0.log
โ”œโ”€โ”€ aws-node-qgc5t_kube-system_aws-node-131feacde9ef43a840464fab4dfdcca474f22bf4c729cb72eb06a6b0e52775aa.log -> /var/log/pods/kube-system_aws-node-qgc5t_310d7c55-bc9e-474a-928f-b19218586bb2/aws-node/0.log
โ”œโ”€โ”€ aws-node-qgc5t_kube-system_aws-vpc-cni-init-0955e671e6632f35365190de7baf154098e1874e15da88593d58c4c8b60b0ebe.log -> /var/log/pods/kube-system_aws-node-qgc5t_310d7c55-bc9e-474a-928f-b19218586bb2/aws-vpc-cni-init/0.log
โ”œโ”€โ”€ cloudwatch-agent-h6rpg_amazon-cloudwatch_otc-container-60691c257f2dc3f6c88c750c4b62a063881ebd20191d22c0931c3e39edc530f0.log -> /var/log/pods/amazon-cloudwatch_cloudwatch-agent-h6rpg_acdaf801-9bdc-442a-aa75-382c955a6877/otc-container/0.log
โ”œโ”€โ”€ details-v1-79dfbd6fff-584wb_default_details-b3894830ed1617fe5450afd0a995fd7b73b29cba7926429c5d097bde263655fc.log -> /var/log/pods/default_details-v1-79dfbd6fff-584wb_b48b1f96-e092-46d5-8e67-cd82ac2e6c37/details/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_csi-attacher-1e5760913f98d5f0bbe1a2b361ad912c868db722f01b60a9a7ce90a109e70b5d.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/csi-attacher/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_csi-provisioner-be3e325f052fe16b3092a07e91e114dc9dc432e5d22f243e6588827c212bc6a7.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/csi-provisioner/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_csi-resizer-bc3fcf25d070d8aba39417b1366d13792ca13effda7ccee5eac9c350aacf335f.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/csi-resizer/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_csi-snapshotter-b5c546a2c80d6b12dc57145efadd7096af9b26a617c96348e96c76dabfc6e9aa.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/csi-snapshotter/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_ebs-plugin-1f144c5d362b9aa9defc70faceceeca9aaf5bf2432ff29094ca7f14aff0d7286.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/ebs-plugin/0.log
โ”œโ”€โ”€ ebs-csi-controller-7f8f8cb84-p57xw_kube-system_liveness-probe-5df88285ad500c9f746b9e2a33c300935449361510cf5257f5927bc244a00be2.log -> /var/log/pods/kube-system_ebs-csi-controller-7f8f8cb84-p57xw_4cf5ca4a-fae6-490f-9757-6fb2f93248bf/liveness-probe/0.log
โ”œโ”€โ”€ ebs-csi-node-hx5jb_kube-system_ebs-plugin-3752c118d606b7aef00dc996750ad7a5ceb8cdd6c1fe52f6be372bbf7ac7c75d.log -> /var/log/pods/kube-system_ebs-csi-node-hx5jb_7d00ede6-b633-4562-ade4-5c61317bf843/ebs-plugin/0.log
โ”œโ”€โ”€ ebs-csi-node-hx5jb_kube-system_liveness-probe-b4704bbfc6c455d7d5783c46f5d7edeaab33a1ff639fb3e705ae7339e7cc2fee.log -> /var/log/pods/kube-system_ebs-csi-node-hx5jb_7d00ede6-b633-4562-ade4-5c61317bf843/liveness-probe/0.log
โ”œโ”€โ”€ ebs-csi-node-hx5jb_kube-system_node-driver-registrar-075052709e559f266c4781c82fd440ea8a242b22ef144b992315505de11821eb.log -> /var/log/pods/kube-system_ebs-csi-node-hx5jb_7d00ede6-b633-4562-ade4-5c61317bf843/node-driver-registrar/0.log
โ”œโ”€โ”€ fluent-bit-9l69f_amazon-cloudwatch_fluent-bit-36dd0ff870718ead8bab9b0e527b10ce8ba557fc4fa0ca5739463cc3cbf06fbc.log -> /var/log/pods/amazon-cloudwatch_fluent-bit-9l69f_e07292b4-1788-44fb-8e6e-4d6da7d4bd24/fluent-bit/0.log
โ”œโ”€โ”€ kube-ops-view-657dbc6cd8-pxkvr_kube-system_kube-ops-view-03cea4cf3d718da69b11be38f26cea545a73de5e067e258a2defdfe2717a1b5a.log -> /var/log/pods/kube-system_kube-ops-view-657dbc6cd8-pxkvr_37bf2637-622f-4ee2-9804-a73d74e16f0e/kube-ops-view/0.log
โ”œโ”€โ”€ kube-prometheus-stack-prometheus-node-exporter-h442l_monitoring_node-exporter-990676e672a4e571e148c5fafb4a1a443ec0882f51c071a1ba4045888eedb35d.log -> /var/log/pods/monitoring_kube-prometheus-stack-prometheus-node-exporter-h442l_7627e5da-0717-4488-8ce1-c0e3cb456fc1/node-exporter/0.log
โ”œโ”€โ”€ kube-proxy-z9l58_kube-system_kube-proxy-c8a65c0d7d4f876e3eec50f7adcd3a349af990918e6affddc365e25ecfc58265.log -> /var/log/pods/kube-system_kube-proxy-z9l58_143e79b7-03d1-4bc9-af70-4e316835bebb/kube-proxy/0.log
โ”œโ”€โ”€ nginx-7c94c9bdcb-9g66b_default_nginx-caae567c481a991fbfc88920be6f7ad4b747d7157917de3f84861f2e4f9214d1.log -> /var/log/pods/default_nginx-7c94c9bdcb-9g66b_cfa0d8d5-8e43-4b49-8cac-2467feac7430/nginx/0.log
โ”œโ”€โ”€ nginx-7c94c9bdcb-9g66b_default_preserve-logs-symlinks-05070e719f8e24ce045ecb8acafe47cd184eefaa68e42b7d929232b8f91c9093.log -> /var/log/pods/default_nginx-7c94c9bdcb-9g66b_cfa0d8d5-8e43-4b49-8cac-2467feac7430/preserve-logs-symlinks/0.log
โ”œโ”€โ”€ productpage-v1-dffc47f64-t8bs9_default_productpage-a045f8233a3098e7bb02b6487751f7a2780ed628233317d1c45fa22103ecfebe.log -> /var/log/pods/default_productpage-v1-dffc47f64-t8bs9_82e8306c-9a93-42ad-a5c8-6b7960291bed/productpage/0.log
โ””โ”€โ”€ reviews-v1-5c4d6d447c-7b69s_default_reviews-a0c4cb095ac797334d9f33a2d5cfac40849e1aae7099f760d8fde06aa8cf29e3.log -> /var/log/pods/default_reviews-v1-5c4d6d447c-7b69s_440483a1-6ed2-4f1b-825c-73b5ba45393c/reviews/0.log

0 directories, 24 files

>>>>> 43.200.163.0 <<<<<
/var/log/containers
โ”œโ”€โ”€ aws-node-p4v96_kube-system_aws-eks-nodeagent-29c619954bdcd1b8e8fe69944adc0a11efea521f7af573163f02e141836fdff8.log -> /var/log/pods/kube-system_aws-node-p4v96_c7f74999-0576-407f-8bb6-7363e257d07c/aws-eks-nodeagent/0.log
โ”œโ”€โ”€ aws-node-p4v96_kube-system_aws-node-055f423d886edae30761d2c3cb3e0e6180947dcffa411525d1aadaaa1adff9a5.log -> /var/log/pods/kube-system_aws-node-p4v96_c7f74999-0576-407f-8bb6-7363e257d07c/aws-node/0.log
โ”œโ”€โ”€ aws-node-p4v96_kube-system_aws-vpc-cni-init-4ad0fce252c9c30b0572d945e8f7bf9fd3e25ecf52a7ea7e92ddc4c24c081c4a.log -> /var/log/pods/kube-system_aws-node-p4v96_c7f74999-0576-407f-8bb6-7363e257d07c/aws-vpc-cni-init/0.log
โ”œโ”€โ”€ cloudwatch-agent-87hnx_amazon-cloudwatch_otc-container-3970ce7ea4baccdfc1509e8f49e0a3bc647fe21be59ea60d25115d78df641de4.log -> /var/log/pods/amazon-cloudwatch_cloudwatch-agent-87hnx_97bb652a-8c32-42fb-a36b-f4c82301ca39/otc-container/0.log
โ”œโ”€โ”€ coredns-86f5954566-d6vwh_kube-system_coredns-47afd8adff91e102d81a8465187fa4142b5974a191adb3a66bee6a571c6ae410.log -> /var/log/pods/kube-system_coredns-86f5954566-d6vwh_44a7e298-9a71-45c3-b013-78fd714dc910/coredns/0.log
โ”œโ”€โ”€ ebs-csi-node-gdh58_kube-system_ebs-plugin-aadf0d58897fb5be64696bcea86e2069121224357deb2945b7d2209a712a6b92.log -> /var/log/pods/kube-system_ebs-csi-node-gdh58_125250cb-6169-4d7f-a67e-48f7a1dad6ef/ebs-plugin/0.log
โ”œโ”€โ”€ ebs-csi-node-gdh58_kube-system_liveness-probe-bd1fe3ce30be5008c7d12090f240b8bb11b2688c6b1837344df1a8f5bd894c42.log -> /var/log/pods/kube-system_ebs-csi-node-gdh58_125250cb-6169-4d7f-a67e-48f7a1dad6ef/liveness-probe/0.log
โ”œโ”€โ”€ ebs-csi-node-gdh58_kube-system_node-driver-registrar-0d2ae3e69caf9b791f1792fafa95b508d879e6e9bf1f667e6bd789db90c48076.log -> /var/log/pods/kube-system_ebs-csi-node-gdh58_125250cb-6169-4d7f-a67e-48f7a1dad6ef/node-driver-registrar/0.log
โ”œโ”€โ”€ fluent-bit-8264s_amazon-cloudwatch_fluent-bit-acfa337b0c7ed38f64db47b027408a27f82f5857c23a2a518ba476b9052c6479.log -> /var/log/pods/amazon-cloudwatch_fluent-bit-8264s_956023fe-06a8-43b7-87f9-eaa5b3ebf6ed/fluent-bit/0.log
โ”œโ”€โ”€ kube-prometheus-stack-kube-state-metrics-5dbfbd4b9-jgnqz_monitoring_kube-state-metrics-d936a02626be02bd153b9335309bdc58635eff941ad0a44024f62b227ae8a4d3.log -> /var/log/pods/monitoring_kube-prometheus-stack-kube-state-metrics-5dbfbd4b9-jgnqz_c3c88c9a-e333-45cc-94f3-17d463c1f6b5/kube-state-metrics/0.log
โ”œโ”€โ”€ kube-prometheus-stack-operator-76bdd654bf-st47m_monitoring_kube-prometheus-stack-920d8607cf65582926adda2de88a16c8bfd2497bf9a6e95b36a420ce070d23ba.log -> /var/log/pods/monitoring_kube-prometheus-stack-operator-76bdd654bf-st47m_4ddc004c-c404-4afd-87e0-5b20c6658f4a/kube-prometheus-stack/0.log
โ”œโ”€โ”€ kube-prometheus-stack-prometheus-node-exporter-kcpsg_monitoring_node-exporter-c55ed57dc100621c8862329c5ef6706661be5911e70543edec5d1e8f23440d10.log -> /var/log/pods/monitoring_kube-prometheus-stack-prometheus-node-exporter-kcpsg_8a6a5dd7-a90b-4b52-a9c0-0be029e6a60e/node-exporter/0.log
โ”œโ”€โ”€ kube-proxy-s6tdr_kube-system_kube-proxy-e9f14023b1fa22b7635908d1f4f0a281ef488fdd505596a1b0c6fc6197f50bd0.log -> /var/log/pods/kube-system_kube-proxy-s6tdr_73fab99f-ca37-4889-bde3-41edd1bdb6e9/kube-proxy/0.log
โ”œโ”€โ”€ metrics-server-6bf5998d9c-c8tbf_kube-system_metrics-server-11c9d62da3624469d83d05266914f32cb6eac70257e95fc25d7bd475155fb996.log -> /var/log/pods/kube-system_metrics-server-6bf5998d9c-c8tbf_438a7a99-9cdc-4162-a997-8079928f224e/metrics-server/0.log
โ”œโ”€โ”€ metrics-server-6bf5998d9c-tftq9_kube-system_metrics-server-0ceb1cf3e114275f884451369ee24c7e6c0cd2a28cd32bd4eeece8a12a5feaf0.log -> /var/log/pods/kube-system_metrics-server-6bf5998d9c-tftq9_8079e24b-42c0-4f1d-9fe2-7ca81b85cef1/metrics-server/0.log
โ”œโ”€โ”€ prometheus-kube-prometheus-stack-prometheus-0_monitoring_config-reloader-8d13b28d34f56a4929f1fb83781191d2a49f3a8b27e994e2a9d69ce2af86536e.log -> /var/log/pods/monitoring_prometheus-kube-prometheus-stack-prometheus-0_8ea36b37-436c-44d4-abee-8b9a013dc06e/config-reloader/0.log
โ”œโ”€โ”€ prometheus-kube-prometheus-stack-prometheus-0_monitoring_init-config-reloader-f3b7bc42e009d5ea5c0c765da46c9a3176dea606e3f019403d4483a5a916bc90.log -> /var/log/pods/monitoring_prometheus-kube-prometheus-stack-prometheus-0_8ea36b37-436c-44d4-abee-8b9a013dc06e/init-config-reloader/0.log
โ”œโ”€โ”€ prometheus-kube-prometheus-stack-prometheus-0_monitoring_prometheus-088bda24f30d1af1650405f1a411b15bb947f59868da68f56ac55f555fd8b3ea.log -> /var/log/pods/monitoring_prometheus-kube-prometheus-stack-prometheus-0_8ea36b37-436c-44d4-abee-8b9a013dc06e/prometheus/0.log
โ””โ”€โ”€ reviews-v3-f68f94645-nsbl8_default_reviews-783c91ace3e5873f33437ffa59ce1f210580e599d287b3cf725eced6386ca915.log -> /var/log/pods/default_reviews-v3-f68f94645-nsbl8_8cebd5e6-30ac-428f-b7d8-b694a2dd1e3a/reviews/0.log

0 directories, 19 files

(2) host ๋กœ๊ทธ ์†Œ์Šค(Logs fromย /var/log/dmesg,ย /var/log/secure, andย /var/log/messages), ๋…ธ๋“œ(ํ˜ธ์ŠคํŠธ) ๋กœ๊ทธ

1

for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/ -L 1; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

>>>>> 15.164.227.37 <<<<<
/var/log/
โ”œโ”€โ”€ README -> ../../usr/share/doc/systemd/README.logs
โ”œโ”€โ”€ amazon
โ”œโ”€โ”€ audit
โ”œโ”€โ”€ aws-routed-eni
โ”œโ”€โ”€ btmp
โ”œโ”€โ”€ chrony
โ”œโ”€โ”€ cloud-init-output.log
โ”œโ”€โ”€ cloud-init.log
โ”œโ”€โ”€ containers
โ”œโ”€โ”€ dmesg
โ”œโ”€โ”€ dnf.librepo.log
โ”œโ”€โ”€ dnf.log
โ”œโ”€โ”€ dnf.rpm.log
โ”œโ”€โ”€ hawkey.log
โ”œโ”€โ”€ journal
โ”œโ”€โ”€ lastlog
โ”œโ”€โ”€ pods
โ”œโ”€โ”€ private
โ”œโ”€โ”€ sa
โ”œโ”€โ”€ tallylog
โ””โ”€โ”€ wtmp

10 directories, 11 files

>>>>> 3.38.205.159 <<<<<
/var/log/
โ”œโ”€โ”€ README -> ../../usr/share/doc/systemd/README.logs
โ”œโ”€โ”€ amazon
โ”œโ”€โ”€ audit
โ”œโ”€โ”€ aws-routed-eni
โ”œโ”€โ”€ btmp
โ”œโ”€โ”€ chrony
โ”œโ”€โ”€ cloud-init-output.log
โ”œโ”€โ”€ cloud-init.log
โ”œโ”€โ”€ containers
โ”œโ”€โ”€ dmesg
โ”œโ”€โ”€ dnf.librepo.log
โ”œโ”€โ”€ dnf.log
โ”œโ”€โ”€ dnf.rpm.log
โ”œโ”€โ”€ hawkey.log
โ”œโ”€โ”€ journal
โ”œโ”€โ”€ lastlog
โ”œโ”€โ”€ pods
โ”œโ”€โ”€ private
โ”œโ”€โ”€ sa
โ”œโ”€โ”€ tallylog
โ””โ”€โ”€ wtmp

10 directories, 11 files

>>>>> 43.200.163.0 <<<<<
/var/log/
โ”œโ”€โ”€ README -> ../../usr/share/doc/systemd/README.logs
โ”œโ”€โ”€ amazon
โ”œโ”€โ”€ audit
โ”œโ”€โ”€ aws-routed-eni
โ”œโ”€โ”€ btmp
โ”œโ”€โ”€ chrony
โ”œโ”€โ”€ cloud-init-output.log
โ”œโ”€โ”€ cloud-init.log
โ”œโ”€โ”€ containers
โ”œโ”€โ”€ dmesg
โ”œโ”€โ”€ dnf.librepo.log
โ”œโ”€โ”€ dnf.log
โ”œโ”€โ”€ dnf.rpm.log
โ”œโ”€โ”€ hawkey.log
โ”œโ”€โ”€ journal
โ”œโ”€โ”€ lastlog
โ”œโ”€โ”€ pods
โ”œโ”€โ”€ private
โ”œโ”€โ”€ sa
โ”œโ”€โ”€ tallylog
โ””โ”€โ”€ wtmp

10 directories, 11 files

(3) dataplane ๋กœ๊ทธ ์†Œ์Šค(/var/log/journalย forย kubelet.service,ย kubeproxy.service, andย docker.service), ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ๋ฐ์ดํ„ฐํ”Œ๋ ˆ์ธ ๋กœ๊ทธ

1

for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/journal -L 1; echo; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

>>>>> 15.164.227.37 <<<<<
/var/log/journal
โ”œโ”€โ”€ ec22753c501541d270d53e1cc6b319a7
โ””โ”€โ”€ ec290d14e0f34366b2d3f2ea33b06253

2 directories, 0 files

>>>>> 3.38.205.159 <<<<<
/var/log/journal
โ”œโ”€โ”€ ec22753c501541d270d53e1cc6b319a7
โ””โ”€โ”€ ec23b4795af458cac1beebec40e88e9b

2 directories, 0 files

>>>>> 43.200.163.0 <<<<<
/var/log/journal
โ”œโ”€โ”€ ec203a98663eebbcd25282168fa4a01d
โ””โ”€โ”€ ec22753c501541d270d53e1cc6b319a7

2 directories, 0 files

7. cloudwatch-agent ์„ค์ • ํ™•์ธ

1

kubectl describe cm cloudwatch-agent -n amazon-cloudwatch

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Name:         cloudwatch-agent
Namespace:    amazon-cloudwatch
Labels:       app.kubernetes.io/component=amazon-cloudwatch-agent
              app.kubernetes.io/instance=amazon-cloudwatch.cloudwatch-agent
              app.kubernetes.io/managed-by=amazon-cloudwatch-agent-operator
              app.kubernetes.io/name=cloudwatch-agent
              app.kubernetes.io/part-of=amazon-cloudwatch-agent
              app.kubernetes.io/version=1.300052.0b1024
Annotations:  <none>

Data
====
cwagentconfig.json:
----
{"agent":{"region":"ap-northeast-2"},"logs":{"metrics_collected":{"application_signals":{"hosted_in":"myeks"},"kubernetes":{"cluster_name":"myeks","enhanced_container_insights":true}}},"traces":{"traces_collected":{"application_signals":{}}}}

BinaryData
====

Events:  <none>

8. cloudwatch-agent ๋ฐ๋ชฌ์…‹ ์ƒ์„ธ ์กฐํšŒ

1

kubectl describe -n amazon-cloudwatch ds cloudwatch-agent

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

Name:           cloudwatch-agent
Selector:       app.kubernetes.io/component=amazon-cloudwatch-agent,app.kubernetes.io/instance=amazon-cloudwatch.cloudwatch-agent,app.kubernetes.io/managed-by=amazon-cloudwatch-agent-operator,app.kubernetes.io/part-of=amazon-cloudwatch-agent
Node-Selector:  kubernetes.io/os=linux
Labels:         app.kubernetes.io/component=amazon-cloudwatch-agent
                app.kubernetes.io/instance=amazon-cloudwatch.cloudwatch-agent
                app.kubernetes.io/managed-by=amazon-cloudwatch-agent-operator
                app.kubernetes.io/name=cloudwatch-agent
                app.kubernetes.io/part-of=amazon-cloudwatch-agent
                app.kubernetes.io/version=1.300052.0b1024
Annotations:    amazon-cloudwatch-agent-operator-config/sha256: f76363aaebaaaa494aa0e7edfc9a2329b93ab3e4ad302f54e8e3e2a284dbef1c
                deprecated.daemonset.template.generation: 1
                prometheus.io/path: /metrics
                prometheus.io/port: 8888
                prometheus.io/scrape: true
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=amazon-cloudwatch-agent
                    app.kubernetes.io/instance=amazon-cloudwatch.cloudwatch-agent
                    app.kubernetes.io/managed-by=amazon-cloudwatch-agent-operator
                    app.kubernetes.io/name=cloudwatch-agent
                    app.kubernetes.io/part-of=amazon-cloudwatch-agent
                    app.kubernetes.io/version=1.300052.0b1024
  Annotations:      amazon-cloudwatch-agent-operator-config/sha256: f76363aaebaaaa494aa0e7edfc9a2329b93ab3e4ad302f54e8e3e2a284dbef1c
                    prometheus.io/path: /metrics
                    prometheus.io/port: 8888
                    prometheus.io/scrape: true
  Service Account:  cloudwatch-agent
  Containers:
   otc-container:
    Image:       602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/observability/cloudwatch-agent:1.300052.0b1024
    Ports:       4315/TCP, 4316/TCP, 2000/TCP, 4311/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  128Mi
    Environment:
      K8S_NODE_NAME:   (v1:spec.nodeName)
      HOST_IP:         (v1:status.hostIP)
      HOST_NAME:       (v1:spec.nodeName)
      K8S_NAMESPACE:   (v1:metadata.namespace)
      POD_NAME:        (v1:metadata.name)
    Mounts:
      /dev/disk from devdisk (ro)
      /etc/amazon-cloudwatch-observability-agent-cert from agenttls (ro)
      /etc/amazon-cloudwatch-observability-agent-client-cert from agentclienttls (ro)
      /etc/amazon-cloudwatch-observability-agent-server-cert from agentservertls (ro)
      /etc/cwagentconfig from otc-internal (rw)
      /rootfs from rootfs (ro)
      /run/containerd/containerd.sock from containerdsock (rw)
      /sys from sys (ro)
      /var/lib/docker from varlibdocker (ro)
      /var/lib/kubelet/pod-resources from kubelet-podresources (rw)
      /var/run/docker.sock from dockersock (ro)
  Volumes:
   otc-internal:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cloudwatch-agent
    Optional:  false
   kubelet-podresources:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pod-resources
    HostPathType:  Directory
   rootfs:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
   dockersock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
   varlibdocker:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker
    HostPathType:  
   containerdsock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/containerd/containerd.sock
    HostPathType:  
   sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
   devdisk:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/disk/
    HostPathType:  
   agenttls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  amazon-cloudwatch-observability-agent-cert
    Optional:    false
   agentclienttls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  amazon-cloudwatch-observability-agent-client-cert
    Optional:    false
   agentservertls:
    Type:               Secret (a volume populated by a Secret)
    SecretName:         amazon-cloudwatch-observability-agent-server-cert
    Optional:           false
  Priority Class Name:  system-node-critical
  Node-Selectors:       kubernetes.io/os=linux
  Tolerations:          op=Exists
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  50m   daemonset-controller  Created pod: cloudwatch-agent-87hnx
  Normal  SuccessfulCreate  50m   daemonset-controller  Created pod: cloudwatch-agent-h6rpg
  Normal  SuccessfulCreate  50m   daemonset-controller  Created pod: cloudwatch-agent-kfwzk

9. Fluent Bit ๋กœ๊ทธ INPUT/FILTER/OUTPUT ์„ค์ • ํ™•์ธ

1

kubectl describe cm fluent-bit-config -n amazon-cloudwatch

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218

Name:         fluent-bit-config
Namespace:    amazon-cloudwatch
Labels:       k8s-app=fluent-bit
Annotations:  <none>

Data
====
application-log.conf:
----
[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                aws
  Match               application.*
  az                  false
  ec2_instance_id     false
  Enable_Entity       true

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0
  Use_Pod_Association On

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights
  add_entity          true

dataplane-log.conf:
----
[INPUT]
  Name                systemd
  Tag                 dataplane.systemd.*
  Systemd_Filter      _SYSTEMD_UNIT=docker.service
  Systemd_Filter      _SYSTEMD_UNIT=containerd.service
  Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
  DB                  /var/fluent-bit/state/systemd.db
  Path                /var/log/journal
  Read_From_Tail      ${READ_FROM_TAIL}

[INPUT]
  Name                tail
  Tag                 dataplane.tail.*
  Path                /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_dataplane_tail.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                modify
  Match               dataplane.systemd.*
  Rename              _HOSTNAME                   hostname
  Rename              _SYSTEMD_UNIT               systemd_unit
  Rename              MESSAGE                     message
  Remove_regex        ^((?!hostname|systemd_unit|message).)*$

[FILTER]
  Name                aws
  Match               dataplane.*
  imds_version        v2

[OUTPUT]
  Name                cloudwatch_logs
  Match               dataplane.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/dataplane
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights

fluent-bit.conf:
----
[SERVICE]
  Flush                     5
  Grace                     30
  Log_Level                 error
  Daemon                    off
  Parsers_File              parsers.conf
  storage.path              /var/fluent-bit/state/flb-storage/
  storage.sync              normal
  storage.checksum          off
  storage.backlog.mem_limit 5M

@INCLUDE application-log.conf
@INCLUDE dataplane-log.conf
@INCLUDE host-log.conf

host-log.conf:
----
[INPUT]
  Name                tail
  Tag                 host.dmesg
  Path                /var/log/dmesg
  Key                 message
  DB                  /var/fluent-bit/state/flb_dmesg.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 host.messages
  Path                /var/log/messages
  Parser              syslog
  DB                  /var/fluent-bit/state/flb_messages.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 host.secure
  Path                /var/log/secure
  Parser              syslog
  DB                  /var/fluent-bit/state/flb_secure.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                aws
  Match               host.*
  imds_version        v2

[OUTPUT]
  Name                cloudwatch_logs
  Match               host.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/host
  log_stream_prefix   ${HOST_NAME}.
  auto_create_group   true
  extra_user_agent    container-insights

parsers.conf:
----
[PARSER]
  Name                syslog
  Format              regex
  Regex               ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
  Time_Key            time
  Time_Format         %b %d %H:%M:%S

[PARSER]
  Name                container_firstline
  Format              regex
  Regex               (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
  Time_Key            time
  Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
  Name                cwagent_firstline
  Format              regex
  Regex               (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
  Time_Key            time
  Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

BinaryData
====

Events:  <none>
  • Path์— ์ง€์ •๋œ ๊ฒฝ๋กœ๊ฐ€ ๋กœ๊ทธ ์ˆ˜์ง‘ ๋Œ€์ƒ์ž„

10. Fluent Bit ํŒŒ๋“œ ๋กœ๊ทธ ์ˆ˜์ง‘: HostPath ๋ณผ๋ฅจ ํ™•์ธ

1

kubectl describe -n amazon-cloudwatch ds fluent-bit

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

Name:           fluent-bit
Selector:       k8s-app=fluent-bit
Node-Selector:  kubernetes.io/os=linux
Labels:         k8s-app=fluent-bit
                kubernetes.io/cluster-service=true
                version=v1
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=fluent-bit
                    kubernetes.io/cluster-service=true
                    version=v1
  Annotations:      checksum/config: 343bfdc4f6cc519ad04fcf86e8ed75b5adb1dbbf7352093d37c3acbdc05bdd69
  Service Account:  cloudwatch-agent
  Containers:
   fluent-bit:
    Image:      602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/observability/aws-for-fluent-bit:2.32.5
    Port:       <none>
    Host Port:  <none>
    Limits:
      cpu:     500m
      memory:  250Mi
    Requests:
      cpu:     50m
      memory:  25Mi
    Environment:
      AWS_REGION:      ap-northeast-2
      CLUSTER_NAME:    myeks
      READ_FROM_HEAD:  Off
      READ_FROM_TAIL:  On
      HOST_NAME:        (v1:spec.nodeName)
      HOSTNAME:         (v1:metadata.name)
      CI_VERSION:      k8s/1.3.17
    Mounts:
      /etc/amazon-cloudwatch-observability-agent-client-cert from agentclienttls (ro)
      /etc/amazon-cloudwatch-observability-agent-server-cert from agentservertls (ro)
      /fluent-bit/etc/ from fluent-bit-config (rw)
      /run/log/journal from runlogjournal (ro)
      /var/fluent-bit/state from fluentbitstate (rw)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (ro)
      /var/log/dmesg from dmesg (ro)
  Volumes:
   fluentbitstate:
    Type:          HostPath (bare host directory volume)
    Path:          /var/fluent-bit/state
    HostPathType:  
   varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
   varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:  
   fluent-bit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      fluent-bit-config
    Optional:  false
   runlogjournal:
    Type:          HostPath (bare host directory volume)
    Path:          /run/log/journal
    HostPathType:  
   dmesg:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/dmesg
    HostPathType:  
   agentclienttls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  amazon-cloudwatch-observability-agent-client-cert
    Optional:    false
   agentservertls:
    Type:               Secret (a volume populated by a Secret)
    SecretName:         amazon-cloudwatch-observability-agent-server-cert
    Optional:           false
  Priority Class Name:  system-node-critical
  Node-Selectors:       kubernetes.io/os=linux
  Tolerations:          op=Exists
Events:                 <none>

11. CloudWatch ๋กœ๊ทธ ๊ทธ๋ฃน ๊ตฌ์„ฑ

  • CloudWatch์— 3๊ฐ€์ง€ ๋กœ๊ทธ ๊ทธ๋ฃน์ด ์ƒ์„ฑ๋จ: application, dataplane, performance

Image

  • application ๋กœ๊ทธ ๊ทธ๋ฃน

Image

  • dataplane ๋กœ๊ทธ ๊ทธ๋ฃน
  • ๋…ธ๋“œ๊ฐ€ 3๊ฐœ์ด๋ฏ€๋กœ, 3๊ฐœ์˜ ๊ฐœ๋ณ„ ๋กœ๊ทธ ์ •๋ณด๊ฐ€ ํ™•์ธ๋จ

Image

  • Syncing iptables Rules
  • iptables ๊ทœ์น™ ๋™๊ธฐํ™” ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Œ

Image

  • performance ๋กœ๊ทธ ๊ทธ๋ฃน

Image

12. ์šด์˜ ์„œ๋ฒ„ EC2 ๋กœ๊ทธ ํ™•์ธ

(1) httpd ์„ค์น˜

1

(eks-user@myeks:default) [root@operator-host ~]# yum install -y httpd

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                       | 3.6 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.4.62-1.amzn2.0.2 will be installed
--> Processing Dependency: httpd-filesystem = 2.4.62-1.amzn2.0.2 for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: httpd-tools = 2.4.62-1.amzn2.0.2 for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: /etc/mime.types for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: httpd-filesystem for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: mod_http2 for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: system-logos-httpd for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.62-1.amzn2.0.2.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.7.2-1.amzn2.0.1 will be installed
---> Package apr-util.x86_64 0:1.6.3-1.amzn2.0.1 will be installed
--> Processing Dependency: apr-util-bdb(x86-64) = 1.6.3-1.amzn2.0.1 for package: apr-util-1.6.3-1.amzn2.0.1.x86_64
---> Package generic-logos-httpd.noarch 0:18.0.0-4.amzn2 will be installed
---> Package httpd-filesystem.noarch 0:2.4.62-1.amzn2.0.2 will be installed
---> Package httpd-tools.x86_64 0:2.4.62-1.amzn2.0.2 will be installed
---> Package mailcap.noarch 0:2.1.41-2.amzn2 will be installed
---> Package mod_http2.x86_64 0:1.15.19-1.amzn2.0.2 will be installed
--> Running transaction check
---> Package apr-util-bdb.x86_64 0:1.6.3-1.amzn2.0.1 will be installed
--> Finished Dependency Resolution
...                          

Complete!

(2) Apache Bench ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ

  • Apache Bench(ab) ๋„๊ตฌ๋ฅผ ์‚ฌ์šฉํ•ด 30,000๋ฒˆ์˜ ์š”์ฒญ ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ๋ฅผ ์‹คํ–‰
1

(eks-user@myeks:default) [root@operator-host ~]# ab -c 500 -n 30000 https://nginx.$MyDomain/

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

This is ApacheBench, Version 2.3 <$Revision: 1913912 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking nginx.gagajin.com (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
Completed 30000 requests
Finished 30000 requests

Server Software:        nginx
Server Hostname:        nginx.gagajin.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        nginx.gagajin.com

Document Path:          /
Document Length:        615 bytes

Concurrency Level:      500
Time taken for tests:   36.291 seconds
Complete requests:      30000
Failed requests:        0
Total transferred:      26100000 bytes
HTML transferred:       18450000 bytes
Requests per second:    826.66 [#/sec] (mean)
Time per request:       604.845 [ms] (mean)
Time per request:       1.210 [ms] (mean, across all concurrent requests)
Transfer rate:          702.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        8  569 172.0    559    1022
Processing:     5   30  37.5     14     223
Waiting:        3   22  24.7     12     218
Total:         60  599 177.1    606    1115

Percentage of the requests served within a certain time (ms)
  50%    606
  66%    699
  75%    729
  80%    751
  90%    816
  95%    878
  98%    974
  99%   1013
 100%   1115 (longest request)

(3) Nginx ๋กœ๊ทธ ์‹ค์‹œ๊ฐ„ ๋ชจ๋‹ˆํ„ฐ๋ง

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

(eks-user@myeks:default) [root@operator-host ~]# kubectl stern deploy/nginx
....
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
nginx-7c94c9bdcb-9g66b nginx 192.168.1.98 - - [26/Feb/2025:14:19:59 +0000] "GET / HTTP/1.1" 200 615 "-" "ApacheBench/2.3" "13.124.11.68"
...

(4) cloudWatch ๋กœ๊ทธ๊ทธ๋ฃน ํ™•์ธ

  • application์—์„œ nginx ๊ฒ€์ƒ‰
  • nginx ๊ด€๋ จ๋กœ๊ทธ์˜ user-agent๊ฐ€ ApacheBench์ธ ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์—ˆ์Œ

Image

13. Logs Insights ํ™•์ธ

(1) application ๋กœ๊ทธ ๊ทธ๋ฃน

  • ๋กœ๊ทธ ๊ทธ๋ฃน ์„ ํƒ : /aws/containerinsights/<CLUSTER_NAME>/application
  • ๋ถ„์„ ์˜ˆ์‹œ - ์ปจํ…Œ์ด๋„ˆ๋ณ„ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์˜ค๋ฅ˜
  • ์ปจํ…Œ์ด๋„ˆ ์ด๋ฆ„๋ณ„๋กœ, stderr ์ŠคํŠธ๋ฆผ์— ๊ธฐ๋ก๋œ ์˜ค๋ฅ˜ ์ˆ˜๋ฅผ ์ง‘๊ณ„ํ•˜์—ฌ ๋‚ด๋ฆผ์ฐจ์ˆœ ์ •๋ ฌํ•จ
1
2
3

stats count() as error_count by kubernetes.container_name
| filter stream="stderr"
| sort error_count desc

Image

(2) performance ๋กœ๊ทธ ๊ทธ๋ฃน

  • ๋กœ๊ทธ ๊ทธ๋ฃน ์„ ํƒ : /aws/containerinsights/<CLUSTER_NAME>/performance
  • ๋ถ„์„ ์˜ˆ์‹œ - ๋…ธ๋“œ๋ณ„ ํ‰๊ท  CPU ์‚ฌ์šฉ๋ฅ 
  • ๊ฐ ๋…ธ๋“œ์˜ ํ‰๊ท  CPU ์‚ฌ์šฉ๋ฅ ์„ ๊ณ„์‚ฐํ•˜์—ฌ ๋‚ด๋ฆผ์ฐจ์ˆœ ์ •๋ ฌํ•จ
1
2

STATS avg(node_cpu_utilization) as avg_node_cpu_utilization by NodeName
| SORT avg_node_cpu_utilization DESC

Image

  • ์ถ”๊ฐ€ ๋ถ„์„ ์˜ˆ์‹œ - ์š”์ฒญ๋œ Pod์™€ ์‹คํ–‰ ์ค‘์ธ Pod ๊ฐ„ ๋น„๊ต
  • ๊ฐ Pod์—์„œ ์š”์ฒญ๋œ ์ปจํ…Œ์ด๋„ˆ ์ˆ˜์™€ ์‹ค์ œ ์‹คํ–‰ ์ค‘์ธ ์ปจํ…Œ์ด๋„ˆ ์ˆ˜๋ฅผ ๋น„๊ตํ•˜์—ฌ ๋ถ€์กฑํ•œ Pod ์ˆ˜๋ฅผ ๊ณ„์‚ฐํ•จ
1
2
3
4
5

fields @timestamp, @message 
| sort @timestamp desc 
| filter Type="Pod" 
| stats min(pod_number_of_containers) as requested, min(pod_number_of_running_containers) as running, ceil(avg(pod_number_of_containers-pod_number_of_running_containers)) as pods_missing by kubernetes.pod_name 
| sort pods_missing desc

Image

(3) Container Insights ๋Œ€์‹œ๋ณด๋“œ

  • ์ ‘๊ทผ ๋ฐฉ๋ฒ•: CloudWatch ์ฝ˜์†” โ†’ Insights โ†’ Container Insights
  • ๋ถ„์„ ๋‚ด์šฉ: ํด๋Ÿฌ์Šคํ„ฐ๋ณ„, Pod๋ณ„ CPU ์‚ฌ์šฉ๋ฅ  ๋“ฑ ๋‹ค์–‘ํ•œ ๋ฆฌ์†Œ์Šค ์ •๋ณด๋ฅผ ์‹œ๊ฐํ™”ํ•จ

Image

(4) Container map

Image

(5) Container Insights

Image

14. CloudWatch Container Observability ์‚ญ์ œ

1

aws eks delete-addon --cluster-name $CLUSTER_NAME --addon-name amazon-cloudwatch-observability

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "myeks",
        "status": "DELETING",
        "addonVersion": "v3.3.1-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:ap-northeast-2:378102432899:addon/myeks/amazon-cloudwatch-observability/b8caa12a-714e-a4b9-05b5-a368820767bf",
        "createdAt": "2025-02-26T21:56:54.393000+09:00",
        "modifiedAt": "2025-02-26T23:44:23.726000+09:00",
        "serviceAccountRoleArn": "arn:aws:iam::378102432899:role/myeks-cloudwatch-agent-role",
        "tags": {}
    }
}

15. CloudWatch ๋กœ๊ทธ ๊ทธ๋ฃน ์‚ญ์ œ

Image

๐Ÿ“Š ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„ค์น˜

1. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ๋‹ค์šด๋กœ๋“œ

1

(eks-user@myeks:default) [root@operator-host ~]# wget https://github.com/prometheus/prometheus/releases/download/v3.2.0/prometheus-3.2.0.linux-amd64.tar.gz

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

--2025-03-01 09:28:27--  https://github.com/prometheus/prometheus/releases/download/v3.2.0/prometheus-3.2.0.linux-amd64.tar.gz
Resolving github.com (github.com)... 20.200.245.247
Connecting to github.com (github.com)|20.200.245.247|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/6838921/ea79bd2f-216d-454e-8d68-b9093ea1bb3a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20250301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250301T002827Z&X-Amz-Expires=300&X-Amz-Signature=21dff9d3f7513db6d5054b11f86136d020903db8aadccee79da5407c67fa8583&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dprometheus-3.2.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2025-03-01 09:28:27--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/6838921/ea79bd2f-216d-454e-8d68-b9093ea1bb3a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20250301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250301T002827Z&X-Amz-Expires=300&X-Amz-Signature=21dff9d3f7513db6d5054b11f86136d020903db8aadccee79da5407c67fa8583&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dprometheus-3.2.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 114163078 (109M) [application/octet-stream]
Saving to: โ€˜prometheus-3.2.0.linux-amd64.tar.gzโ€™

100%[==============================>] 114,163,078 51.0MB/s   in 2.1s   

2025-03-01 09:28:30 (51.0 MB/s) - โ€˜prometheus-3.2.0.linux-amd64.tar.gzโ€™ saved [114163078/114163078]

2. ์••์ถ• ํ•ด์ œ

1
2
3
4
5
6
7
8

(eks-user@myeks:default) [root@operator-host ~]# tar -xvf prometheus-3.2.0.linux-amd64.tar.gz
# ๊ฒฐ๊ณผ
prometheus-3.2.0.linux-amd64/
prometheus-3.2.0.linux-amd64/prometheus.yml
prometheus-3.2.0.linux-amd64/LICENSE
prometheus-3.2.0.linux-amd64/NOTICE
prometheus-3.2.0.linux-amd64/prometheus
prometheus-3.2.0.linux-amd64/promtool

1
2
3
4
5
6
7
8

(eks-user@myeks:default) [root@operator-host ~]# cd prometheus-3.2.0.linux-amd64
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# ls -l
total 285504
-rw-r--r-- 1 1001 docker     11357 Feb 18 04:00 LICENSE
-rw-r--r-- 1 1001 docker      3773 Feb 18 04:00 NOTICE
-rwxr-xr-x 1 1001 docker 150306663 Feb 18 03:44 prometheus
-rw-r--r-- 1 1001 docker       934 Feb 18 04:00 prometheus.yml
-rwxr-xr-x 1 1001 docker 142027433 Feb 18 03:45 promtool

3. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„ค์น˜ ๋ฐ ์ดˆ๊ธฐ ๊ตฌ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# mv prometheus /usr/local/bin/
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# mv promtool /usr/local/bin/
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# mkdir -p /etc/prometheus /var/lib/prometheus
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# mv prometheus.yml /etc/prometheus/
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# cat /etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
  • 9090 ํฌํŠธ๊ฐ€ ์—ด๋ ค ์žˆ์–ด ํ•ด๋‹น ํฌํŠธ์˜ ๋ฉ”ํŠธ๋ฆญ ๋ฐ์ดํ„ฐ๋ฅผ ์Šคํฌ๋ž˜ํ•‘ํ•จ
  • ์Šคํฌ๋ž˜ํ•‘ ๋Œ€์ƒ(target)์€ ๋กœ์ปฌํ˜ธ์ŠคํŠธ์˜ 9090 ํฌํŠธ์ด๋ฉฐ, ์ด๋Š” Prometheus ์ž์ฒด ๋ฉ”ํŠธ๋ฆญ์„ ์˜๋ฏธํ•จ
  • ์ด ์Šคํฌ๋ž˜ํ•‘ ์ž‘์—…์€ โ€œprometheusโ€๋ผ๋Š” job์œผ๋กœ ๊ตฌ๋ถ„๋˜์–ด ์žˆ์Œ
  • ๊ตฌ์„ฑ ํŒŒ์ผ(prometheus.yml)์—์„œ๋Š” ์ „์—ญ ์„ค์ •(์Šคํฌ๋žฉ ๋ฐ ํ‰๊ฐ€ ๊ฐ„๊ฒฉ 15์ดˆ)๊ณผ Alertmanager, ๊ทœ์น™ ํŒŒ์ผ, ๊ทธ๋ฆฌ๊ณ  ์Šคํฌ๋ž˜ํ•‘ ๋Œ€์ƒ์ด ์ •์˜๋˜์–ด ์žˆ์Œ

4. User ๋ฐ ๊ถŒํ•œ ์„ค์ •

1
2
3

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# useradd --no-create-home --shell /sbin/nologin prometheus
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

5. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„œ๋น„์Šค ํŒŒ์ผ ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# tee /etc/systemd/system/prometheus.service > /dev/null <<EOF
> [Unit]
> Description=Prometheus
> Wants=network-online.target
> After=network-online.target
> 
> [Service]
> User=prometheus
> Group=prometheus
> Type=simple
> ExecStart=/usr/local/bin/prometheus \
>   --config.file=/etc/prometheus/prometheus.yml \
>   --storage.tsdb.path=/var/lib/prometheus \
>   --web.listen-address=0.0.0.0:9090
> 
> [Install]
> WantedBy=multi-user.target
> EOF

6. ์‹œ์Šคํ…œ ๋ฐ๋ชฌ ๋ฆฌ๋กœ๋“œ ๋ฐ ์„œ๋น„์Šค ํ™œ์„ฑํ™”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# systemctl daemon-reload
(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# systemctl enable --now prometheus
Created symlink from /etc/systemd/system/multi-user.target.wants/prometheus.service to /etc/systemd/system/prometheus.service.

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# systemctl status prometheus
โ— prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2025-03-01 09:48:32 KST; 184ms ago
 Main PID: 21312 (prometheus)
    Tasks: 8
   Memory: 15.4M
   CGroup: /system.slice/prometheus.service
           โ””โ”€21312 /usr/local/bin/prometheus --config.file=/etc/prome...

Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4โ€ฆms
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4โ€ฆms
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Mar 01 09:48:32 operator-host prometheus[21312]: time=2025-03-01T00:4...
Hint: Some lines were ellipsized, use -l to show in full.

7. ๋„คํŠธ์›Œํฌ ํฌํŠธ ์ƒํƒœ ํ™•์ธ

1
2
3
4
5
6
7
8
9

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# ss -tnlp
State    Recv-Q   Send-Q     Local Address:Port      Peer Address:Port  Process                                                                 
LISTEN   0        128              0.0.0.0:111            0.0.0.0:*      users:(("rpcbind",pid=1718,fd=8))                                      
LISTEN   0        128              0.0.0.0:22             0.0.0.0:*      users:(("sshd",pid=2363,fd=3))                                         
LISTEN   0        100            127.0.0.1:25             0.0.0.0:*      users:(("master",pid=2160,fd=13))                                      
LISTEN   0        128            127.0.0.1:46109          0.0.0.0:*      users:(("containerd",pid=2806,fd=9))                                   
LISTEN   0        128                 [::]:111               [::]:*      users:(("rpcbind",pid=1718,fd=11))                                     
LISTEN   0        128                 [::]:22                [::]:*      users:(("sshd",pid=2363,fd=4))                                         
LISTEN   0        128                    *:9090                 *:*      users:(("prometheus",pid=21312,fd=6))
  • Prometheus ํ”„๋กœ์„ธ์Šค๊ฐ€ 0.0.0.0:9090์—์„œ ์ •์ƒ์ ์œผ๋กœ LISTEN ์ค‘์ž„์„ ํ™•์ธํ•จ

8. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ๋ฉ”ํŠธ๋ฆญ ์กฐํšŒ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# curl localhost:9090/metrics
...
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 0
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 0
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 0
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 0
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="queue_time"} 0
prometheus_engine_query_duration_seconds_count{slice="queue_time"} 0
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="result_sort"} 0
prometheus_engine_query_duration_seconds_count{slice="result_sort"} 0
...
promhttp_metric_handler_requests_total{code="200"} 8
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

9. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น์ ‘์†

1

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# echo -e "http://$(curl -s ipinfo.io/ip):9090"

โœ…ย ์ถœ๋ ฅ

1

http://13.124.11.68:9090

Image

  • Status > Target health์—์„œ "job=prometheus"๋กœ ๋“ฑ๋ก๋œ ์—”๋“œํฌ์ธํŠธ๊ฐ€ ์ •์ƒ์ž„์„ ํ™•์ธํ•จ

Image

  • Query ํƒญ์—์„œ ๋‹ค์–‘ํ•œ ๋ฉ”ํŠธ๋ฆญ(ex. prometheus_build_info)์„ ํƒ์ƒ‰ํ•˜์—ฌ ์‹œ์Šคํ…œ ์ƒํƒœ ๋ฐ ์„ฑ๋Šฅ ์ •๋ณด๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Œ

Image

๐Ÿ–ฅ๏ธ Node Exporter ์„ค์น˜

1. Node Exporter ์ตœ์‹ ๋ฒ„์ „ ๋‹ค์šด๋กœ๋“œ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(eks-user@myeks:default) [root@operator-host prometheus-3.2.0.linux-amd64]# cd ~
(eks-user@myeks:default) [root@operator-host ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
# ๊ฒฐ๊ณผ
--2025-03-01 10:12:16--  https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
Resolving github.com (github.com)... 20.200.245.247
Connecting to github.com (github.com)|20.200.245.247|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/9524057/c181ae2d-a1b3-4bac-883f-2a071c7ba341?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20250301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250301T011216Z&X-Amz-Expires=300&X-Amz-Signature=4dfa054b7bea1d228763e4611ccb54632f7a3d23a5ae551ddc93f7ba3fac4cd1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-1.9.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2025-03-01 10:12:16--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/9524057/c181ae2d-a1b3-4bac-883f-2a071c7ba341?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20250301%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250301T011216Z&X-Amz-Expires=300&X-Amz-Signature=4dfa054b7bea1d228763e4611ccb54632f7a3d23a5ae551ddc93f7ba3fac4cd1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-1.9.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11569068 (11M) [application/octet-stream]
Saving to: โ€˜node_exporter-1.9.0.linux-amd64.tar.gzโ€™

100%[==============================>] 11,569,068  42.1MB/s   in 0.3s   

2025-03-01 10:12:18 (42.1 MB/s) - โ€˜node_exporter-1.9.0.linux-amd64.tar.gzโ€™ saved [11569068/11569068]

2. ์••์ถ• ํ•ด์ œ

1
2
3
4
5
6

(eks-user@myeks:default) [root@operator-host ~]# tar xvfz node_exporter-1.9.0.linux-amd64.tar.gz
# ๊ฒฐ๊ณผ
node_exporter-1.9.0.linux-amd64/
node_exporter-1.9.0.linux-amd64/LICENSE
node_exporter-1.9.0.linux-amd64/NOTICE
node_exporter-1.9.0.linux-amd64/node_exporter

3. ๋ฐ”์ด๋„ˆ๋ฆฌ ๋ณต์‚ฌ

1
2

(eks-user@myeks:default) [root@operator-host ~]# cd node_exporter-1.9.0.linux-amd64
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# cp node_exporter /usr/local/bin/

4. ์‚ฌ์šฉ์ž ๋ฐ ๊ถŒํ•œ ์„ค์ •

1
2
3

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# groupadd -f node_exporter
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# useradd -g node_exporter --no-create-home --shell /sbin/nologin node_exporter
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# chown node_exporter:node_exporter /usr/local/bin/node_exporter

5. systemd ์„œ๋น„์Šค ํŒŒ์ผ ์ƒ์„ฑ ๋ฐ ์„œ๋น„์Šค ์‹คํ–‰

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
> [Unit]
> Description=Node Exporter
> Documentation=https://prometheus.io/docs/guides/node-exporter/
> Wants=network-online.target
> After=network-online.target
> 
> [Service]
> User=node_exporter
> Group=node_exporter
> Type=simple
> Restart=on-failure
> ExecStart=/usr/local/bin/node_exporter \
>   --web.listen-address=:9200
> 
> [Install]
> WantedBy=multi-user.target
> EOF

6. ์„œ๋น„์Šค ํ™œ์„ฑํ™” ๋ฐ ์ƒํƒœ ํ™•์ธ

(1) ์‹œ์Šคํ…œ ๋ฐ๋ชฌ์„ ๋ฆฌ๋กœ๋“œํ•œ ํ›„, Node Exporter ์„œ๋น„์Šค ํ™œ์„ฑํ™”

1
2
3
4

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# systemctl daemon-reload
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# systemctl enable --now node_exporter
# ๊ฒฐ๊ณผ
Created symlink from /etc/systemd/system/multi-user.target.wants/node_exporter.service to /etc/systemd/system/node_exporter.service.

(2) Node Exporter ์ƒํƒœ ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# systemctl status node_exporter

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

โ— node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2025-03-01 10:17:08 KST; 192ms ago
     Docs: https://prometheus.io/docs/guides/node-exporter/
 Main PID: 22042 (node_exporter)
    Tasks: 4
   Memory: 2.2M
   CGroup: /system.slice/node_exporter.service
           โ””โ”€22042 /usr/local/bin/node_exporter --web.listen-address=...

Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Mar 01 10:17:08 operator-host node_exporter[22042]: time=2025-03-01T0...
Hint: Some lines were ellipsized, use -l to show in full.

(3) ํฌํŠธ ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# ss -tnlp

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9

State    Recv-Q   Send-Q     Local Address:Port      Peer Address:Port  Process                                                                 
LISTEN   0        128              0.0.0.0:111            0.0.0.0:*      users:(("rpcbind",pid=1718,fd=8))                                      
LISTEN   0        128              0.0.0.0:22             0.0.0.0:*      users:(("sshd",pid=2363,fd=3))                                         
LISTEN   0        100            127.0.0.1:25             0.0.0.0:*      users:(("master",pid=2160,fd=13))                                      
LISTEN   0        128            127.0.0.1:46109          0.0.0.0:*      users:(("containerd",pid=2806,fd=9))                                   
LISTEN   0        128                 [::]:111               [::]:*      users:(("rpcbind",pid=1718,fd=11))                                     
LISTEN   0        128                    *:9200                 *:*      users:(("node_exporter",pid=22042,fd=3))                               
LISTEN   0        128                 [::]:22                [::]:*      users:(("sshd",pid=2363,fd=4))                                         
LISTEN   0        128                    *:9090                 *:*      users:(("prometheus",pid=21312,fd=6))
  • 9200 ํฌํŠธ๊ฐ€ Node Exporter ํ”„๋กœ์„ธ์Šค์— ์˜ํ•ด LISTEN ์ค‘์ž„์„ ํ™•์ธํ•จ

7. Node Exporter ๋ฉ”ํŠธ๋ฆญ ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# curl localhost:9200/metrics

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
...
node_scrape_collector_duration_seconds{collector="hwmon"} 7.93e-06
node_scrape_collector_duration_seconds{collector="infiniband"} 2.0005e-05
node_scrape_collector_duration_seconds{collector="ipvs"} 2.2186e-05
node_scrape_collector_duration_seconds{collector="loadavg"} 2.8009e-05
node_scrape_collector_duration_seconds{collector="mdadm"} 4.2583e-05
node_scrape_collector_duration_seconds{collector="meminfo"} 0.000220746
node_scrape_collector_duration_seconds{collector="netclass"} 0.003341259
node_scrape_collector_duration_seconds{collector="netdev"} 0.000211741
node_scrape_collector_duration_seconds{collector="netstat"} 0.001199888
node_scrape_collector_duration_seconds{collector="nfs"} 1.4731e-05
node_scrape_collector_duration_seconds{collector="nfsd"} 2.0009e-05
...
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
  • ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น(http://13.124.11.68:9090/query)์—์„œ โ€œnodeโ€๋ฅผ ๊ฒ€์ƒ‰ํ•ด๋„ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜ํƒ€๋‚˜์ง€ ์•Š์Œ

Image

  • ์ด๋Š” Node Exporter๊ฐ€ ์„ค์น˜๋˜์–ด ์žˆ์œผ๋‚˜, ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค์˜ ์ˆ˜์ง‘ ๋Œ€์ƒ(target)์œผ๋กœ ๋“ฑ๋ก๋˜์ง€ ์•Š์•˜๊ธฐ ๋•Œ๋ฌธ์ž„

Image

8. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„ค์ •์— ์ˆ˜์ง‘ ๋Œ€์ƒ node_exporter ์ถ”๊ฐ€

1
2
3
4
5
6
7
8

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# cat << EOF >> /etc/prometheus/prometheus.yml
> 
>   - job_name: 'node_exporter'
>     static_configs:
>       - targets: ["127.0.0.1:9200"]
>         labels:
>           alias: 'myec2'
> EOF
  • ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„ค์ • ํŒŒ์ผ์— โ€˜node_exporterโ€™ job์„ ์ถ”๊ฐ€ํ•˜๊ณ , static_config์—์„œ ํƒ€๊ฒŸ์œผ๋กœ [โ€œ127.0.0.1:9200โ€]์„ ์ง€์ •ํ•จ

9. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ๋ฐ๋ชฌ ์žฌ๊ธฐ๋™

1
2

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# systemctl restart prometheus.service
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# systemctl status prometheus

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

โ— prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2025-03-01 10:29:29 KST; 17s ago
 Main PID: 22319 (prometheus)
    Tasks: 7
   Memory: 24.0M
   CGroup: /system.slice/prometheus.service
           โ””โ”€22319 /usr/local/bin/prometheus --config.file=/etc/prome...

Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2โ€ฆms
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2โ€ฆms
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Mar 01 10:29:29 operator-host prometheus[22319]: time=2025-03-01T01:2...
Hint: Some lines were ellipsized, use -l to show in full.

10. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น ์ƒˆ๋กœ๊ณ ์นจ

  • node_exporter๊ฐ€ UP ์ƒํƒœ๋กœ ํ‘œ์‹œ๋จ
  • static ํƒ€๊ฒŸ ๊ด€๋ฆฌ ๋ฐฉ์‹์€ ๋งค๋ฒˆ ์ˆ˜๋™์œผ๋กœ ์ˆ˜์ •ํ•ด์•ผ ํ•˜๋Š” ๋ถˆํŽธํ•จ์ด ์žˆ์Œ

Image

  • node ๊ด€๋ จ ๋ฉ”ํŠธ๋ฆญ(ex. node_disk_read_bytes_total ๋“ฑ) ์กฐํšŒ ๊ฐ€๋Šฅํ•จ

Image

๐Ÿ“Š ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค-์Šคํƒ ์„ค์น˜

1. Helm ์ €์žฅ์†Œ ๋“ฑ๋ก

1
2
3

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# ๊ฒฐ๊ณผ
"prometheus-community" already exists with the same configuration, skipping

2. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

cat <<EOT > monitor-values.yaml
prometheus:
  prometheusSpec:
    scrapeInterval: "15s"
    evaluationInterval: "15s"
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    retention: 5d
    retentionSize: "10GiB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 30Gi

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - prometheus.$MyDomain
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - grafana.$MyDomain
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

  persistence:
    enabled: true
    type: sts
    storageClassName: "gp3"
    accessModes:
      - ReadWriteOnce
    size: 20Gi

alertmanager:
  enabled: false
defaultRules:
  create: false
kubeControllerManager:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false
EOT

3. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์กฐํšŒ

1

cat monitor-values.yaml

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

prometheus:
  prometheusSpec:
    scrapeInterval: "15s"
    evaluationInterval: "15s"
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    retention: 5d
    retentionSize: "10GiB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 30Gi

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - prometheus.gagajin.com
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator

  ingress:
    enabled: true
    ingressClassName: alb
    hosts: 
      - grafana.gagajin.com
    paths: 
      - /*
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
      alb.ingress.kubernetes.io/success-codes: 200-399
      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
      alb.ingress.kubernetes.io/group.name: study
      alb.ingress.kubernetes.io/ssl-redirect: '443'

  persistence:
    enabled: true
    type: sts
    storageClassName: "gp3"
    accessModes:
      - ReadWriteOnce
    size: 20Gi

alertmanager:
  enabled: false
defaultRules:
  create: false
kubeControllerManager:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false

4. ๋ฐฐํฌ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 69.3.1 -f monitor-values.yaml --create-namespace --namespace monitoring
# ๊ฒฐ๊ณผ
NAME: kube-prometheus-stack
LAST DEPLOYED: Sat Mar  1 10:55:37 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"

Get Grafana 'admin' user password by running:

  kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Access Grafana local instance:

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prometheus-stack" -oname)
  kubectl --namespace monitoring port-forward $POD_NAME 3000

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

5. Helm ๋ฆด๋ฆฌ์Šค ์ƒํƒœ ํ™•์ธ

1

helm list -n monitoring

โœ…ย ์ถœ๋ ฅ

1
2

NAME                 	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART                       	APP VERSION
kube-prometheus-stack	monitoring	1       	2025-03-01 10:55:37.941454098 +0900 KST	deployed	kube-prometheus-stack-69.3.1	v0.80.0

6. ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ๋ฆฌ์†Œ์Šค ํ™•์ธ

1

kubectl get sts,ds,deploy,pod,svc,ep,ingress,pvc,pv -n monitoring

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

NAME                                                           READY   AGE
statefulset.apps/kube-prometheus-stack-grafana                 1/1     73s
statefulset.apps/prometheus-kube-prometheus-stack-prometheus   1/1     71s

NAME                                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter   3         3         3       3            3           kubernetes.io/os=linux   73s

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-prometheus-stack-kube-state-metrics   1/1     1            1           73s
deployment.apps/kube-prometheus-stack-operator             1/1     1            1           73s

NAME                                                           READY   STATUS    RESTARTS   AGE
pod/kube-prometheus-stack-grafana-0                            3/3     Running   0          73s
pod/kube-prometheus-stack-kube-state-metrics-5dbfbd4b9-b2bfq   1/1     Running   0          73s
pod/kube-prometheus-stack-operator-76bdd654bf-t5gh7            1/1     Running   0          73s
pod/kube-prometheus-stack-prometheus-node-exporter-44zpf       1/1     Running   0          73s
pod/kube-prometheus-stack-prometheus-node-exporter-b2zk8       1/1     Running   0          73s
pod/kube-prometheus-stack-prometheus-node-exporter-x79kf       1/1     Running   0          73s
pod/prometheus-kube-prometheus-stack-prometheus-0              2/2     Running   0          71s

NAME                                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/kube-prometheus-stack-grafana                    ClusterIP   10.100.33.238    <none>        80/TCP              73s
service/kube-prometheus-stack-grafana-headless           ClusterIP   None             <none>        9094/TCP            73s
service/kube-prometheus-stack-kube-state-metrics         ClusterIP   10.100.131.25    <none>        8080/TCP            73s
service/kube-prometheus-stack-operator                   ClusterIP   10.100.186.182   <none>        443/TCP             73s
service/kube-prometheus-stack-prometheus                 ClusterIP   10.100.140.111   <none>        9090/TCP,8080/TCP   73s
service/kube-prometheus-stack-prometheus-node-exporter   ClusterIP   10.100.122.125   <none>        9100/TCP            73s
service/prometheus-operated                              ClusterIP   None             <none>        9090/TCP            71s

NAME                                                       ENDPOINTS                                               AGE
endpoints/kube-prometheus-stack-grafana                    192.168.1.82:3000                                       73s
endpoints/kube-prometheus-stack-grafana-headless           192.168.1.82:9094                                       73s
endpoints/kube-prometheus-stack-kube-state-metrics         192.168.1.238:8080                                      73s
endpoints/kube-prometheus-stack-operator                   192.168.3.147:10250                                     73s
endpoints/kube-prometheus-stack-prometheus                 192.168.3.89:9090,192.168.3.89:8080                     73s
endpoints/kube-prometheus-stack-prometheus-node-exporter   192.168.1.51:9100,192.168.2.42:9100,192.168.3.30:9100   73s
endpoints/prometheus-operated                              192.168.3.89:9090                                       71s

NAME                                                         CLASS   HOSTS                    ADDRESS                                                       PORTS   AGE
ingress.networking.k8s.io/kube-prometheus-stack-grafana      alb     grafana.gagajin.com      myeks-ingress-alb-77245841.ap-northeast-2.elb.amazonaws.com   80      73s
ingress.networking.k8s.io/kube-prometheus-stack-prometheus   alb     prometheus.gagajin.com   myeks-ingress-alb-77245841.ap-northeast-2.elb.amazonaws.com   80      73s

NAME                                                                                                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0   Bound    pvc-afacff05-8dfd-4a4d-a597-1d76f1d05be5   30Gi       RWO            gp3            <unset>                 3d10h
persistentvolumeclaim/storage-kube-prometheus-stack-grafana-0                                                        Bound    pvc-86ba8583-882e-4d2b-837b-dac7c7ac231d   20Gi       RWO            gp3            <unset>                 3d10h

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                     STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-86ba8583-882e-4d2b-837b-dac7c7ac231d   20Gi       RWO            Delete           Bound    monitoring/storage-kube-prometheus-stack-grafana-0                                                        gp3            <unset>                          3d10h
persistentvolume/pvc-afacff05-8dfd-4a4d-a597-1d76f1d05be5   30Gi       RWO            Delete           Bound    monitoring/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0   gp3            <unset>                          3d10h

7. ์ „์ฒด ๋ฆฌ์†Œ์Šค ํ™•์ธ

1

kubectl get-all -n monitoring

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

NAME                                                                                                                NAMESPACE   AGE
configmap/kube-prometheus-stack-apiserver                                                                           monitoring  3m13s  
configmap/kube-prometheus-stack-cluster-total                                                                       monitoring  3m13s  
configmap/kube-prometheus-stack-grafana                                                                             monitoring  3m13s  
configmap/kube-prometheus-stack-grafana-config-dashboards                                                           monitoring  3m13s  
configmap/kube-prometheus-stack-grafana-datasource                                                                  monitoring  3m13s  
configmap/kube-prometheus-stack-grafana-overview                                                                    monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-coredns                                                                         monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-cluster                                                               monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-multicluster                                                          monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-namespace                                                             monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-node                                                                  monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-pod                                                                   monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-workload                                                              monitoring  3m13s  
configmap/kube-prometheus-stack-k8s-resources-workloads-namespace                                                   monitoring  3m13s  
configmap/kube-prometheus-stack-kubelet                                                                             monitoring  3m13s  
configmap/kube-prometheus-stack-namespace-by-pod                                                                    monitoring  3m13s  
configmap/kube-prometheus-stack-namespace-by-workload                                                               monitoring  3m13s  
configmap/kube-prometheus-stack-node-cluster-rsrc-use                                                               monitoring  3m13s  
configmap/kube-prometheus-stack-node-rsrc-use                                                                       monitoring  3m13s  
configmap/kube-prometheus-stack-nodes                                                                               monitoring  3m13s  
configmap/kube-prometheus-stack-nodes-aix                                                                           monitoring  3m13s  
configmap/kube-prometheus-stack-nodes-darwin                                                                        monitoring  3m13s  
configmap/kube-prometheus-stack-persistentvolumesusage                                                              monitoring  3m13s  
configmap/kube-prometheus-stack-pod-total                                                                           monitoring  3m13s  
configmap/kube-prometheus-stack-prometheus                                                                          monitoring  3m13s  
configmap/kube-prometheus-stack-proxy                                                                               monitoring  3m13s  
configmap/kube-prometheus-stack-workload-total                                                                      monitoring  3m13s  
configmap/kube-root-ca.crt                                                                                          monitoring  3d10h  
configmap/prometheus-kube-prometheus-stack-prometheus-rulefiles-0                                                   monitoring  3m11s  
endpoints/kube-prometheus-stack-grafana                                                                             monitoring  3m13s  
endpoints/kube-prometheus-stack-grafana-headless                                                                    monitoring  3m13s  
endpoints/kube-prometheus-stack-kube-state-metrics                                                                  monitoring  3m13s  
endpoints/kube-prometheus-stack-operator                                                                            monitoring  3m13s  
endpoints/kube-prometheus-stack-prometheus                                                                          monitoring  3m13s  
endpoints/kube-prometheus-stack-prometheus-node-exporter                                                            monitoring  3m13s  
endpoints/prometheus-operated                                                                                       monitoring  3m11s  
persistentvolumeclaim/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0  monitoring  3d10h  
persistentvolumeclaim/storage-kube-prometheus-stack-grafana-0                                                       monitoring  3d10h  
pod/kube-prometheus-stack-grafana-0                                                                                 monitoring  3m13s  
pod/kube-prometheus-stack-kube-state-metrics-5dbfbd4b9-b2bfq                                                        monitoring  3m13s  
pod/kube-prometheus-stack-operator-76bdd654bf-t5gh7                                                                 monitoring  3m13s  
pod/kube-prometheus-stack-prometheus-node-exporter-44zpf                                                            monitoring  3m13s  
pod/kube-prometheus-stack-prometheus-node-exporter-b2zk8                                                            monitoring  3m13s  
pod/kube-prometheus-stack-prometheus-node-exporter-x79kf                                                            monitoring  3m13s  
pod/prometheus-kube-prometheus-stack-prometheus-0                                                                   monitoring  3m11s  
secret/kube-prometheus-stack-admission                                                                              monitoring  3d10h  
secret/kube-prometheus-stack-grafana                                                                                monitoring  3m13s  
secret/prometheus-kube-prometheus-stack-prometheus                                                                  monitoring  3m11s  
secret/prometheus-kube-prometheus-stack-prometheus-thanos-prometheus-http-client-file                               monitoring  3m11s  
secret/prometheus-kube-prometheus-stack-prometheus-tls-assets-0                                                     monitoring  3m11s  
secret/prometheus-kube-prometheus-stack-prometheus-web-config                                                       monitoring  3m11s  
secret/sh.helm.release.v1.kube-prometheus-stack.v1                                                                  monitoring  3m23s  
serviceaccount/default                                                                                              monitoring  3d10h  
serviceaccount/kube-prometheus-stack-grafana                                                                        monitoring  3m13s  
serviceaccount/kube-prometheus-stack-kube-state-metrics                                                             monitoring  3m13s  
serviceaccount/kube-prometheus-stack-operator                                                                       monitoring  3m13s  
serviceaccount/kube-prometheus-stack-prometheus                                                                     monitoring  3m13s  
serviceaccount/kube-prometheus-stack-prometheus-node-exporter                                                       monitoring  3m13s  
service/kube-prometheus-stack-grafana                                                                               monitoring  3m13s  
service/kube-prometheus-stack-grafana-headless                                                                      monitoring  3m13s  
service/kube-prometheus-stack-kube-state-metrics                                                                    monitoring  3m13s  
service/kube-prometheus-stack-operator                                                                              monitoring  3m13s  
service/kube-prometheus-stack-prometheus                                                                            monitoring  3m13s  
service/kube-prometheus-stack-prometheus-node-exporter                                                              monitoring  3m13s  
service/prometheus-operated                                                                                         monitoring  3m11s  
controllerrevision.apps/kube-prometheus-stack-grafana-56f654f5d6                                                    monitoring  3m13s  
controllerrevision.apps/kube-prometheus-stack-prometheus-node-exporter-89f764479                                    monitoring  3m13s  
controllerrevision.apps/prometheus-kube-prometheus-stack-prometheus-88cb76465                                       monitoring  3m11s  
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter                                                       monitoring  3m13s  
deployment.apps/kube-prometheus-stack-kube-state-metrics                                                            monitoring  3m13s  
deployment.apps/kube-prometheus-stack-operator                                                                      monitoring  3m13s  
replicaset.apps/kube-prometheus-stack-kube-state-metrics-5dbfbd4b9                                                  monitoring  3m13s  
replicaset.apps/kube-prometheus-stack-operator-76bdd654bf                                                           monitoring  3m13s  
statefulset.apps/kube-prometheus-stack-grafana                                                                      monitoring  3m13s  
statefulset.apps/prometheus-kube-prometheus-stack-prometheus                                                        monitoring  3m11s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-grafana-cs68n                                                  monitoring  3m13s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-grafana-headless-g5tcd                                         monitoring  3m13s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-kube-state-metrics-8kqsr                                       monitoring  3m13s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-operator-ffmfx                                                 monitoring  3m13s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-prometheus-bplw2                                               monitoring  3m13s  
endpointslice.discovery.k8s.io/kube-prometheus-stack-prometheus-node-exporter-tpjll                                 monitoring  3m13s  
endpointslice.discovery.k8s.io/prometheus-operated-srd8x                                                            monitoring  3m11s  
targetgroupbinding.elbv2.k8s.aws/k8s-monitori-kubeprom-1c4114721c                                                   monitoring  3m11s  
targetgroupbinding.elbv2.k8s.aws/k8s-monitori-kubeprom-8a6213e683                                                   monitoring  3m9s   
prometheus.monitoring.coreos.com/kube-prometheus-stack-prometheus                                                   monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-apiserver                                                monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-coredns                                                  monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-grafana                                                  monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kube-proxy                                               monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kube-state-metrics                                       monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kubelet                                                  monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-operator                                                 monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-prometheus                                               monitoring  3m13s  
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-prometheus-node-exporter                                 monitoring  3m13s  
ingress.networking.k8s.io/kube-prometheus-stack-grafana                                                             monitoring  3m13s  
ingress.networking.k8s.io/kube-prometheus-stack-prometheus                                                          monitoring  3m13s  
rolebinding.rbac.authorization.k8s.io/kube-prometheus-stack-grafana                                                 monitoring  3m13s  
role.rbac.authorization.k8s.io/kube-prometheus-stack-grafana                                                        monitoring  3m13s

8. ๋ชจ๋‹ˆํ„ฐ๋ง CRD ๋ฐ ServiceMonitor ํ™•์ธ

1

kubectl get prometheus,servicemonitors -n monitoring

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13

NAME                                                                VERSION   DESIRED   READY   RECONCILED   AVAILABLE   AGE
prometheus.monitoring.coreos.com/kube-prometheus-stack-prometheus   v3.1.0    1         1       True         True        3m52s

NAME                                                                                  AGE
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-apiserver                  3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-coredns                    3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-grafana                    3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kube-proxy                 3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kube-state-metrics         3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-kubelet                    3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-operator                   3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-prometheus                 3m52s
servicemonitor.monitoring.coreos.com/kube-prometheus-stack-prometheus-node-exporter   3m52s
  • Alertmanager, ServiceMonitor ๋“ฑ ๋ชจ๋‹ˆํ„ฐ๋ง ๊ด€๋ จ CRD๋“ค์ด ์กด์žฌํ•จ์„ ํ™•์ธ
1

kubectl get crd | grep monitoring

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10

alertmanagerconfigs.monitoring.coreos.com    2025-02-25T15:42:28Z
alertmanagers.monitoring.coreos.com          2025-02-25T15:42:29Z
podmonitors.monitoring.coreos.com            2025-02-25T15:42:30Z
probes.monitoring.coreos.com                 2025-02-25T15:42:30Z
prometheusagents.monitoring.coreos.com       2025-02-25T15:42:31Z
prometheuses.monitoring.coreos.com           2025-02-25T15:42:31Z
prometheusrules.monitoring.coreos.com        2025-02-25T15:42:32Z
scrapeconfigs.monitoring.coreos.com          2025-02-25T15:42:32Z
servicemonitors.monitoring.coreos.com        2025-02-25T15:42:33Z
thanosrulers.monitoring.coreos.com           2025-02-25T15:42:33Z

9. Persistent Volume ๋ฐ PVC ์ƒํƒœ ์ ๊ฒ€

1

kubectl df-pv

โœ…ย ์ถœ๋ ฅ

1
2
3
4

 PV NAME                                   PVC NAME                                                                                      NAMESPACE   NODE NAME                                        POD NAME                                       VOLUME MOUNT NAME                               SIZE  USED    AVAILABLE  %USED  IUSED  IFREE     %IUSED 
 pvc-86ba8583-882e-4d2b-837b-dac7c7ac231d  storage-kube-prometheus-stack-grafana-0                                                       monitoring  ip-192-168-1-51.ap-northeast-2.compute.internal  kube-prometheus-stack-grafana-0                storage                                         19Gi  196Mi   19Gi       0.96   85     10485675  0.00   
 pvc-afacff05-8dfd-4a4d-a597-1d76f1d05be5  prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0  monitoring  ip-192-168-3-30.ap-northeast-2.compute.internal  prometheus-kube-prometheus-stack-prometheus-0  prometheus-kube-prometheus-stack-prometheus-db  29Gi  1956Mi  28Gi       6.38   113    15728527  0.00

10. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ๋ฒ„์ „ ๋ฐ ๋นŒ๋“œ ์ •๋ณด ํ™•์ธ

1

echo -e "https://prometheus.$MyDomain/api/v1/status/buildinfo"

โœ…ย ์ถœ๋ ฅ

1

https://prometheus.gagajin.com/api/v1/status/buildinfo

Image

11. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น ๋Œ€์‹œ๋ณด๋“œ ๋ฐ ๊ทธ๋ผํŒŒ๋‚˜ ์ ‘์†

  • ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น ์ ‘์† ํ™”๋ฉด

Image

  • ๊ทธ๋ผํŒŒ๋‚˜ ์›น ์ ‘์† ํ™”๋ฉด
  • Grafana๋Š” Helm ํŒŒ๋ผ๋ฏธํ„ฐ์— ์„ค์ •๋œ ๋กœ๊ทธ์ธ ์ •๋ณด(admin/prom-operator)๋ฅผ ์‚ฌ์šฉํ•ด ์ ‘์†ํ•˜๋ฉฐ, ๋‹ค์–‘ํ•œ ๋Œ€์‹œ๋ณด๋“œ๋ฅผ ํ†ตํ•ด ๋ชจ๋‹ˆํ„ฐ๋ง ๋ฐ์ดํ„ฐ๋ฅผ ์‹œ๊ฐํ™”ํ•  ์ˆ˜ ์žˆ์Œ

Image

12. ALB ํ™•์ธ

  • Ingress ๋ฆฌ์†Œ์Šค๋ฅผ ํ†ตํ•ด ์™ธ๋ถ€ ์ ‘๊ทผ์„ ์œ„ํ•œ ALB๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ๊ตฌ์„ฑ๋˜์–ด ์žˆ์Œ์„ ํ™•์ธํ•จ

Image

AWS CNI Metrics ์ˆ˜์ง‘์„ ์œ„ํ•œ ์‚ฌ์ „ ์„ค์ •

  • ์ถ”๊ฐ€ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜์˜ ๋ฉ”ํŠธ๋ฆญ ์ˆ˜์ง‘์„ ์œ„ํ•ด, ๋Œ€์ƒ์— ์†์‰ฝ๊ฒŒ ๋ฉ”ํŠธ๋ฆญ์„ ์˜ฌ๋ฆด ์ˆ˜ ์žˆ๋Š” ๋ฐฉ๋ฒ•์œผ๋กœ ServiceMonitor์™€ PodMonitor๋ฅผ ํ™œ์šฉํ•จ
  • Prometheus Operator๋Š” ๊ธฐ์กด Config ์ˆ˜์ • ๋Œ€์‹ , PodMonitor CRD๋ฅผ ์ƒ์„ฑํ•˜์—ฌ ์ˆ˜์ง‘ ๋Œ€์ƒ์„ ๋™์ ์œผ๋กœ ๋“ฑ๋กํ•  ์ˆ˜ ์žˆ๋„๋ก ์ง€์›ํ•จ

1. PodMonitor CRD ์ƒ์„ฑ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

cat <<EOF | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: aws-cni-metrics
  namespace: kube-system
spec:
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  podMetricsEndpoints:
  - interval: 30s
    path: /metrics
    port: metrics
  selector:
    matchLabels:
      k8s-app: aws-node
EOF

# ๊ฒฐ๊ณผ
podmonitor.monitoring.coreos.com/aws-cni-metrics created

2. PodMonitor ์ƒ์„ฑ ํ™•์ธ

1

kubectl get podmonitor -n kube-system

โœ…ย ์ถœ๋ ฅ

1
2

NAME              AGE
aws-cni-metrics   40s

3. AWS CNI Metrics ํ™•์ธ

  • ์›Œ์ปค ๋…ธ๋“œ์—์„œ 61678 ํฌํŠธ๋กœ ์ ‘๊ทผํ•˜์—ฌ, AWS CNI ๊ด€๋ จ ๋ฉ”ํŠธ๋ฆญ(ex. awscni_add_ip_req_count ๋“ฑ)์ด ์ถœ๋ ฅ๋˜๋Š”์ง€ ํ™•์ธํ•จ
1

curl -s $N1:61678/metrics | grep '^awscni'

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

awscni_add_ip_req_count 11
awscni_assigned_ip_addresses 8
awscni_assigned_ip_per_cidr{cidr="192.168.1.114/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.127/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.134/32"} 0
awscni_assigned_ip_per_cidr{cidr="192.168.1.194/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.226/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.238/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.62/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.71/32"} 0
awscni_assigned_ip_per_cidr{cidr="192.168.1.82/32"} 1
awscni_assigned_ip_per_cidr{cidr="192.168.1.87/32"} 1
awscni_assigned_ip_per_eni{eni="eni-022faaaa5981f056b"} 0
awscni_assigned_ip_per_eni{eni="eni-0417b2799144172b1"} 5
awscni_assigned_ip_per_eni{eni="eni-0940759276408f3f7"} 3
awscni_aws_api_latency_ms_sum{api="AssignPrivateIpAddresses",error="false",status="200"} 363
awscni_aws_api_latency_ms_count{api="AssignPrivateIpAddresses",error="false",status="200"} 1
awscni_aws_api_latency_ms_sum{api="AttachNetworkInterface",error="false",status="200"} 1494
awscni_aws_api_latency_ms_count{api="AttachNetworkInterface",error="false",status="200"} 2
awscni_aws_api_latency_ms_sum{api="CreateNetworkInterface",error="false",status="200"} 898
awscni_aws_api_latency_ms_count{api="CreateNetworkInterface",error="false",status="200"} 2
awscni_aws_api_latency_ms_sum{api="CreateTags",error="false",status="200"} 141
awscni_aws_api_latency_ms_count{api="CreateTags",error="false",status="200"} 1
awscni_aws_api_latency_ms_sum{api="DescribeInstances",error="false",status="200"} 276
awscni_aws_api_latency_ms_count{api="DescribeInstances",error="false",status="200"} 2
awscni_aws_api_latency_ms_sum{api="DescribeNetworkInterfaces",error="false",status="200"} 235
awscni_aws_api_latency_ms_count{api="DescribeNetworkInterfaces",error="false",status="200"} 1
awscni_aws_api_latency_ms_sum{api="DescribeSubnets",error="false",status="200"} 304
awscni_aws_api_latency_ms_count{api="DescribeSubnets",error="false",status="200"} 2
awscni_aws_api_latency_ms_sum{api="GetMetadata",error="false",status="200"} 79921
awscni_aws_api_latency_ms_count{api="GetMetadata",error="false",status="200"} 132346
awscni_aws_api_latency_ms_sum{api="GetMetadata",error="true",status="404"} 8654
awscni_aws_api_latency_ms_count{api="GetMetadata",error="true",status="404"} 15247
awscni_aws_api_latency_ms_sum{api="ModifyNetworkInterfaceAttribute",error="false",status="200"} 993
awscni_aws_api_latency_ms_count{api="ModifyNetworkInterfaceAttribute",error="false",status="200"} 3
awscni_aws_api_latency_ms_sum{api="waitForENIAndIPsAttached",error="false",status="200"} 1703
awscni_aws_api_latency_ms_count{api="waitForENIAndIPsAttached",error="false",status="200"} 2
awscni_build_info{goversion="go1.22.10",version=""} 1
awscni_del_ip_req_count{reason="PodDeleted"} 9
awscni_ec2api_req_count{fn="AssignPrivateIpAddresses"} 1
awscni_ec2api_req_count{fn="AttachNetworkInterface"} 2
awscni_ec2api_req_count{fn="CreateNetworkInterface"} 2
awscni_ec2api_req_count{fn="CreateTags"} 1
awscni_ec2api_req_count{fn="DescribeInstances"} 2
awscni_ec2api_req_count{fn="DescribeNetworkInterfaces"} 83
awscni_ec2api_req_count{fn="DescribeSubnets"} 2
awscni_ec2api_req_count{fn="ModifyNetworkInterfaceAttribute"} 3
awscni_eni_allocated 3
awscni_eni_max 3
awscni_force_removed_enis 0
awscni_force_removed_ips 0
awscni_ip_max 15
awscni_ipamd_action_inprogress{fn="increaseDatastorePool"} 0
awscni_ipamd_action_inprogress{fn="nodeIPPoolReconcile"} 0
awscni_ipamd_action_inprogress{fn="nodeInit"} 0
awscni_no_available_ip_addresses 0
awscni_reconcile_count{fn="eniDataStorePoolReconcileAdd"} 76160
awscni_total_ip_addresses 15
awscni_total_ipv4_prefixes 0
  • PodMonitor์˜ aws-cni-metrics๊ฐ€ ๋Œ€์ƒ ํƒ€๊ฒŸ์— ์˜ฌ๋ผ๊ฐ

Image

4. ์›Œ์ปค๋…ธ๋“œ ํฌํŠธ ์ƒํƒœ ํ™•์ธ

  • ์›Œ์ปค ๋…ธ๋“œ์— SSH ์ ‘์† ํ›„, ss -tnlp ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ํฌํŠธ 61678์ด ์—ด๋ ค ์žˆ์œผ๋ฉฐ, aws-k8s-agent๊ฐ€ ํ•ด๋‹น ํฌํŠธ๋ฅผ ์‚ฌ์šฉ ์ค‘์ž„์„ ํ™•์ธํ•จ
  • PodMonitor๋ฅผ ์ƒ์„ฑํ•˜๋ฉด Prometheus ํƒ€๊ฒŸ์— aws-cni-metrics๊ฐ€ ์ž๋™์œผ๋กœ ๋“ฑ๋ก๋˜๊ณ , ์ดํ›„ ServiceMonitor ์„ค์ •๋„ ์ž๋™์œผ๋กœ ๋ฐ˜์˜๋จ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

ssh ec2-user@$N1

A newer release of "Amazon Linux" is available.
  Version 2023.6.20250211:
  Version 2023.6.20250218:
Run "/usr/bin/dnf check-release-update" for full release and version update info
   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Sat Mar  1 02:39:19 2025 from 182.230.60.93
[ec2-user@ip-192-168-1-51 ~]$ sudo ss -tnlp
State    Recv-Q   Send-Q     Local Address:Port      Peer Address:Port  Process                                                                 
LISTEN   0        4096           127.0.0.1:50052          0.0.0.0:*      users:(("controller",pid=3098,fd=13))                                  
LISTEN   0        4096           127.0.0.1:50051          0.0.0.0:*      users:(("aws-k8s-agent",pid=2975,fd=10))                               
LISTEN   0        4096           127.0.0.1:61679          0.0.0.0:*      users:(("aws-k8s-agent",pid=2975,fd=12))                               
LISTEN   0        4096           127.0.0.1:10248          0.0.0.0:*      users:(("kubelet",pid=2493,fd=22))                                     
LISTEN   0        128              0.0.0.0:22             0.0.0.0:*      users:(("sshd",pid=1566,fd=3))                                         
LISTEN   0        4096           127.0.0.1:46111          0.0.0.0:*      users:(("containerd",pid=2478,fd=11))                                  
LISTEN   0        4096                   *:8163                 *:*      users:(("controller",pid=3098,fd=7))                                   
LISTEN   0        4096                   *:8162                 *:*      users:(("controller",pid=3098,fd=8))                                   
LISTEN   0        4096                   *:10256                *:*      users:(("kube-proxy",pid=2715,fd=16))                                  
LISTEN   0        128                 [::]:22                [::]:*      users:(("sshd",pid=1566,fd=4))                                         
LISTEN   0        4096                   *:10249                *:*      users:(("kube-proxy",pid=2715,fd=12))                                  
LISTEN   0        4096                   *:10250                *:*      users:(("kubelet",pid=2493,fd=12))                                     
LISTEN   0        4096                   *:61680                *:*      users:(("controller",pid=3098,fd=12))                                  
LISTEN   0        4096                   *:61678                *:*      users:(("aws-k8s-agent",pid=2975,fd=11))                               
LISTEN   0        4096                   *:9100                 *:*      users:(("node_exporter",pid=1591439,fd=3))

๐ŸŒ nginx ์›น ์„œ๋ฒ„(with helm)์— metrics ์ˆ˜์ง‘ ์„ค์ • ์ถ”๊ฐ€

1. ๋ฉ”ํŠธ๋ฆญ ์ˆ˜์ง‘ ๋Œ€์ƒ ํ™•์ธ

  • ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„œ๋ฒ„๋Š” ๋…ธ๋“œ์˜ OS ๋ฐ ์ƒํƒœ ์ •๋ณด๋ฅผ 9100 ํฌํŠธ์˜ /metrics ์—”๋“œํฌ์ธํŠธ์—์„œ ์ˆ˜์ง‘ํ•จ
  • ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค๊ฐ€ ์ด ์—”๋“œํฌ์ธํŠธ๋กœ ํ†ต์‹ ํ•  ์ˆ˜ ์žˆ์–ด์•ผ ํ•˜๋ฉฐ, ๋‚ด๋ถ€ IP(์˜ˆ: http://192.168.1.51:9100/metrics)๋Š” ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์„œ๋ฒ„์™€์˜ ๋„คํŠธ์›Œํฌ ์—ฐ๊ฒฐ์ด ๊ฐ€๋Šฅํ•ด์•ผ ํ•จ

Image

2. ์›น ์ธํ„ฐํŽ˜์ด์Šค ์˜ต์…˜ ๋ฐ ์„ค์ • ํ™•์ธ

  • Status โ†’ Configuration ๋ฉ”๋‰ด์—์„œ Runtime & Build Information์„ ํ™•์ธํ•˜์—ฌ ๋ฒ„์ „, ์ž‘์—… ๋””๋ ‰ํ† ๋ฆฌ, ์Šคํ† ๋ฆฌ์ง€ ๋ณด์กด ๊ธฐ๊ฐ„ ๋“ฑ์˜ ์ •๋ณด๋ฅผ ์ ๊ฒ€ํ•จ

Image

  • TSDB์— ๋Œ€ํ•œ ์ƒํƒœ

Image

  • Command-line flags

Image

  • Configuration

Image

3. Service Discovery ๋ฐ ๋Œ€์ƒ ํƒ€๊ฒŸ ๊ด€๋ฆฌ

  • Kubernetes์˜ ServiceMonitor์™€ PodMonitor๋ฅผ ์‚ฌ์šฉํ•ด ๋Œ€์ƒ ์—”๋“œํฌ์ธํŠธ๋ฅผ ์ž๋™์œผ๋กœ ๊ฒ€์ƒ‰ํ•˜๊ณ , ์„ค์ •์ด ์˜ฌ๋ฐ”๋ฅธ ๋Œ€์ƒ๋งŒ Prometheus ํƒ€๊ฒŸ์— ๋“ฑ๋ก๋จ
  • ์˜ˆ๋ฅผ ๋“ค์–ด, kube-system ๋„ค์ž„์ŠคํŽ˜์ด์Šค์—์„œ PodMonitor๋ฅผ ์ƒ์„ฑํ–ˆ์„ ๋•Œ 44๊ฐœ์˜ ์—”๋“œํฌ์ธํŠธ ์ค‘ 3๊ฐœ๋งŒ ์˜ฌ๋ฐ”๋ฅธ pod ๋ผ๋ฒจ ๋งค์นญ์œผ๋กœ Target Health์— ํ‘œ์‹œ๋จ
  • ์˜ฌ๋ฐ”๋ฅธ ๋ผ๋ฒจ์ด ์ง€์ •๋œ ๋Œ€์ƒ๋งŒ ์ˆ˜์ง‘๋˜์–ด Prometheus์—์„œ ๋ชจ๋‹ˆํ„ฐ๋ง๋˜๋ฉฐ, ์ด๋Š” ์„œ๋น„์Šค์˜ ์‹ค์ œ active Pod ์ •๋ณด๋ฅผ ๋ฐ˜์˜ํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋จ Image

Image

4. kube-apiserver ํƒ€๊ฒŸ ์ƒํƒœ ํ™•์ธ

  • API ์„œ๋ฒ„์˜ ์—”๋“œํฌ์ธํŠธ์ธ 192.168.2.241๊ณผ 192.168.3.124๋Š” owned ENI์— ์†ํ•˜๋ฉฐ, Prometheus๊ฐ€ ํ•ด๋‹น ์—”๋“œํฌ์ธํŠธ๋ฅผ ์ •์ƒ์ ์œผ๋กœ ํƒ€๊ฒŸ์— ๋“ฑ๋กํ•˜์—ฌ ๋ฉ”ํŠธ๋ฆญ ์ •๋ณด๋ฅผ ์ˆ˜์ง‘ํ•˜๊ณ  ์žˆ์Œ

Image

5. ๋ฉ”ํŠธ๋ฆญ ์กฐํšŒ

  • node_cpu_seconds_total: ๊ฐ ๋…ธ๋“œ๊ฐ€ CPU๋ฅผ ์‚ฌ์šฉํ•œ ์ด ์‹œ๊ฐ„์„ ์ธก์ •ํ•˜์—ฌ, CPU ์‚ฌ์šฉ๋Ÿ‰์„ ๋‚˜ํƒ€๋ƒ„

Image Image

6. PromQL ์ฟผ๋ฆฌ ํ™•์ธ

(1) mode="idle" ์ƒํƒœ์ธ CPU ์‚ฌ์šฉ๋Ÿ‰๋งŒ ์กฐํšŒ

1

node_cpu_seconds_total{mode="idle"}

Image Image

(2) 1๋ถ„ ๊ฐ„๊ฒฉ์˜ ๋ฐ์ดํ„ฐ ํ™•์ธ

  • ๊ธฐ๋ณธ ์ฟผ๋ฆฌ๋Š” 15์ดˆ ๊ฐ„๊ฒฉ์˜ ์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด์—ฌ์คŒ
  • 1๋ถ„ ๋™์•ˆ ์ˆ˜์ง‘๋œ ๋ฐ์ดํ„ฐ๋ฅผ ํ™•์ธํ•˜๋ ค๋ฉด ์•„๋ž˜ ์ฟผ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•จ
1

(node_cpu_seconds_total{mode="idle"}[1m])
  • ์ด ์ฟผ๋ฆฌ๋Š” 1๋ถ„ ๋™์•ˆ ์ˆ˜์ง‘๋œ 4๊ฐœ์˜ ์‹œ๊ณ„์—ด ๋ฐ์ดํ„ฐ๋ฅผ ์ถœ๋ ฅํ•จ Image โ€”

๐Ÿ” nginx ํŒŒ๋“œ ํ™•์ธ

1. nginx ํŒŒ๋“œ๋‚ด์— ์ปจํ…Œ์ด๋„ˆ ๊ฐฏ์ˆ˜ ํ™•์ธ

1

kubectl describe pod -l app.kubernetes.io/instance=nginx

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108

Name:             nginx-7c94c9bdcb-9g66b
Namespace:        default
Priority:         0
Service Account:  nginx
Node:             ip-192-168-2-42.ap-northeast-2.compute.internal/192.168.2.42
Start Time:       Wed, 26 Feb 2025 21:22:15 +0900
Labels:           app.kubernetes.io/instance=nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=nginx
                  app.kubernetes.io/version=1.27.4
                  helm.sh/chart=nginx-19.0.0
                  pod-template-hash=7c94c9bdcb
Annotations:      <none>
Status:           Running
IP:               192.168.2.63
IPs:
  IP:           192.168.2.63
Controlled By:  ReplicaSet/nginx-7c94c9bdcb
Init Containers:
  preserve-logs-symlinks:
    Container ID:    containerd://05070e719f8e24ce045ecb8acafe47cd184eefaa68e42b7d929232b8f91c9093
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Image ID:        docker.io/bitnami/nginx@sha256:a633d685dae033ffde21c0b87445f5e7c20c3895126443f6d7c9d9d325e071be
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -ec
      #!/bin/bash
      . /opt/bitnami/scripts/libfs.sh
      # We copy the logs folder because it has symlinks to stdout and stderr
      if ! is_dir_empty /opt/bitnami/nginx/logs; then
        cp -r /opt/bitnami/nginx/logs /emptydir/app-logs-dir
      fi
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 26 Feb 2025 21:22:24 +0900
      Finished:     Wed, 26 Feb 2025 21:22:24 +0900
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Environment:          <none>
    Mounts:
      /emptydir from empty-dir (rw)
Containers:
  nginx:
    Container ID:    containerd://caae567c481a991fbfc88920be6f7ad4b747d7157917de3f84861f2e4f9214d1
    Image:           docker.io/bitnami/nginx:1.27.4-debian-12-r1
    Image ID:        docker.io/bitnami/nginx@sha256:a633d685dae033ffde21c0b87445f5e7c20c3895126443f6d7c9d9d325e071be
    Ports:           8080/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Wed, 26 Feb 2025 21:22:26 +0900
    Ready:           True
    Restart Count:   0
    Limits:
      cpu:                150m
      ephemeral-storage:  2Gi
      memory:             192Mi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             128Mi
    Liveness:             tcp-socket :http delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:            http-get http://:http/ delay=5s timeout=3s period=5s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:            false
      NGINX_HTTP_PORT_NUMBER:   8080
      NGINX_HTTPS_PORT_NUMBER:  8443
    Mounts:
      /certs from certificate (rw)
      /opt/bitnami/nginx/conf from empty-dir (rw,path="app-conf-dir")
      /opt/bitnami/nginx/logs from empty-dir (rw,path="app-logs-dir")
      /opt/bitnami/nginx/tmp from empty-dir (rw,path="app-tmp-dir")
      /tmp from empty-dir (rw,path="tmp-dir")
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  certificate:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-tls
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

2. ํŒŒ๋ผ๋ฏธํ„ฐ ํŒŒ์ผ ์ƒ์„ฑ

  • nginx-values.yaml ํŒŒ์ผ์„ ์ƒ์„ฑํ•ด Metrics ๊ธฐ๋Šฅ์„ ํ™œ์„ฑํ™”ํ•˜๊ณ , ์„œ๋น„์Šค ํฌํŠธ๋ฅผ 9113์œผ๋กœ ๋ณ€๊ฒฝํ•จ
  • ๋˜ํ•œ, ServiceMonitor CRD ์„ค์ •์„ ํ†ตํ•ด nginx๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋ง ๋Œ€์ƒ์— ์ถ”๊ฐ€ํ•˜๋„๋ก ๊ตฌ์„ฑํ•จ
1
2
3
4
5
6
7
8
9
10
11
12

cat <<EOT > nginx-values.yaml
metrics:
  enabled: true

  service:
    port: 9113

  serviceMonitor:
    enabled: true
    namespace: monitoring
    interval: 10s
EOT

3. ๋ฐฐํฌ

1

helm upgrade nginx bitnami/nginx --reuse-values -f nginx-values.yaml

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

Release "nginx" has been upgraded. Happy Helming!
NAME: nginx
LAST DEPLOYED: Sat Mar  1 15:03:00 2025
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: nginx
CHART VERSION: 19.0.0
APP VERSION: 1.27.4

Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami for more information.

** Please be patient while the chart is being deployed **
NGINX can be accessed through the following DNS name from within your cluster:

    nginx.default.svc.cluster.local (port 80)

To access NGINX from outside the cluster, follow the steps below:

1. Get the NGINX URL and associate its hostname to your cluster external IP:

   export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
   echo "NGINX URL: http://nginx.gagajin.com"
   echo "$CLUSTER_IP  nginx.gagajin.com" | sudo tee -a /etc/hosts

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - cloneStaticSiteFromGit.gitSync.resources
  - metrics.resources
  - resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

โš  SECURITY WARNING: Original containers have been substituted. This Helm chart was designed, tested, and validated on multiple platforms using a specific set of Bitnami and Tanzu Application Catalog containers. Substituting other containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Substituted images detected:
  - docker.io/bitnami/nginx:1.27.4-debian-12-r1

โš  WARNING: Original containers have been retagged. Please note this Helm chart was tested, and validated on multiple platforms using a specific set of Tanzu Application Catalog containers. Substituting original image tags could cause unexpected behavior.

Retagged images:
  - docker.io/bitnami/nginx:1.27.4-debian-12-r1

4. ๋ชจ๋‹ˆํ„ฐ๋ง

1

watch -d "kubectl get pod; echo; kubectl get servicemonitors -n monitoring"

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Every 2.0s: kubectl get pod; echo; kubectl get servicemonitors -n monitoring    gram88: 03:04:35 PM

NAME                             READY   STATUS    RESTARTS   AGE
details-v1-79dfbd6fff-584wb      1/1     Running   0          3d15h
nginx-85df7754bf-55bj4           2/2     Running   0          94s
productpage-v1-dffc47f64-t8bs9   1/1     Running   0          3d15h
ratings-v1-65f797b499-fs82w      1/1     Running   0          3d15h
reviews-v1-5c4d6d447c-7b69s      1/1     Running   0          3d15h
reviews-v2-65cb66b45c-8rvgv      1/1     Running   0          3d15h
reviews-v3-f68f94645-nsbl8       1/1     Running   0          3d15h

NAME                                             AGE
kube-prometheus-stack-apiserver                  4h8m
kube-prometheus-stack-coredns                    4h8m
kube-prometheus-stack-grafana                    4h8m
kube-prometheus-stack-kube-proxy                 4h8m
kube-prometheus-stack-kube-state-metrics         4h8m
kube-prometheus-stack-kubelet                    4h8m
kube-prometheus-stack-operator                   4h8m
kube-prometheus-stack-prometheus                 4h8m
kube-prometheus-stack-prometheus-node-exporter   4h8m
nginx                                            95s

5. nginx ๋ชจ๋‹ˆํ„ฐ๋ง ๋Œ€์ƒ ๋ฐ ServiceMonitor ๋ฆฌ์†Œ์Šค ์กฐํšŒ ์ž‘์—…

(1) Pod, Service, Endpoint ์ •๋ณด ์กฐํšŒ

1

kubectl get pod,svc,ep

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

NAME                                 READY   STATUS    RESTARTS   AGE
pod/details-v1-79dfbd6fff-584wb      1/1     Running   0          3d15h
pod/nginx-85df7754bf-55bj4           2/2     Running   0          5m42s
pod/productpage-v1-dffc47f64-t8bs9   1/1     Running   0          3d15h
pod/ratings-v1-65f797b499-fs82w      1/1     Running   0          3d15h
pod/reviews-v1-5c4d6d447c-7b69s      1/1     Running   0          3d15h
pod/reviews-v2-65cb66b45c-8rvgv      1/1     Running   0          3d15h
pod/reviews-v3-f68f94645-nsbl8       1/1     Running   0          3d15h

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
service/details       ClusterIP   10.100.244.166   <none>        9080/TCP                                    3d15h
service/kubernetes    ClusterIP   10.100.0.1       <none>        443/TCP                                     3d17h
service/nginx         NodePort    10.100.47.81     <none>        80:32507/TCP,443:32693/TCP,9113:31463/TCP   2d17h
service/productpage   ClusterIP   10.100.98.77     <none>        9080/TCP                                    3d15h
service/ratings       ClusterIP   10.100.232.111   <none>        9080/TCP                                    3d15h
service/reviews       ClusterIP   10.100.139.243   <none>        9080/TCP                                    3d15h

NAME                    ENDPOINTS                                                  AGE
endpoints/details       192.168.2.48:9080                                          3d15h
endpoints/kubernetes    192.168.2.241:443,192.168.3.124:443                        3d17h
endpoints/nginx         192.168.3.184:9113,192.168.3.184:8443,192.168.3.184:8080   2d17h
endpoints/productpage   192.168.2.34:9080                                          3d15h
endpoints/ratings       192.168.1.194:9080                                         3d15h
endpoints/reviews       192.168.1.127:9080,192.168.2.176:9080,192.168.3.36:9080    3d15h

(2) nginx ServiceMonitor ์กฐํšŒ

1

kubectl get servicemonitor -n monitoring nginx

โœ…ย ์ถœ๋ ฅ

1
2

NAME    AGE
nginx   6m11s

(3) nginx ServiceMonitor YAML

1

kubectl get servicemonitor -n monitoring nginx -o yaml | kubectl neat

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    app.kubernetes.io/version: 1.27.4
    helm.sh/chart: nginx-19.0.0
  name: nginx
  namespace: monitoring
spec:
  endpoints:
  - interval: 10s
    path: /metrics
    port: metrics
  jobLabel: ""
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/instance: nginx
      app.kubernetes.io/name: nginx

6. view-secret ํ”Œ๋Ÿฌ๊ทธ์ธ ์„ค์น˜

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

kubectl krew install view-secret

# ๊ฒฐ๊ณผ
Updated the local copy of plugin index.
  New plugins available:
    * apidocs
    * cnp-viz
    * pickdeep
    * slowdrain
    * vault-login
Installing plugin: view-secret
Installed plugin: view-secret
\
 | Use this plugin:
 | 	kubectl view-secret
 | Documentation:
 | 	https://github.com/elsesiy/kubectl-view-secret
/
WARNING: You installed plugin "view-secret" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

7. ํ”„๋กœ๋ฉ”ํ…Œ์šฐ์Šค ์›น UI ํ™•์ธ

  • nginx๊ฐ€ ServiceMonitor CRD๋ฅผ ํ†ตํ•ด ์„œ๋น„์Šค ๋””์Šค์ปค๋ฒ„๋ฆฌ ๋Œ€์ƒ์— ์ž๋™ ๋“ฑ๋ก๋จ

Image

  • ๋˜ํ•œ, ๋Œ€์ƒ ํƒ€๊ฒŸ์— ์ž๋™ ๋“ฑ๋ก๋จ

Image

8. ๋™์  ๋Œ€์ƒ ํƒ€๊ฒŸ ๊ตฌ์„ฑ ๋ฐ Config-Reload ์ ์šฉ ํ™•์ธ

  • ๋ณ€๊ฒฝ ์‚ฌํ•ญ ํ™•์ธ : monitoring ๋„ค์ž„์ŠคํŽ˜์ด์Šค์˜ secret ์ค‘ โ€œstack prometheusโ€ ํ•ญ๋ชฉ์—์„œ ๋ณ€๊ฒฝ ์‚ฌํ•ญ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Œ
1

kubectl get secret  -n monitoring

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

NAME                                                                             TYPE                 DATA   AGE
kube-prometheus-stack-admission                                                  Opaque               3      3d14h
kube-prometheus-stack-grafana                                                    Opaque               3      4h27m
prometheus-kube-prometheus-stack-prometheus                                      Opaque               1      4h26m
prometheus-kube-prometheus-stack-prometheus-thanos-prometheus-http-client-file   Opaque               1      4h26m
prometheus-kube-prometheus-stack-prometheus-tls-assets-0                         Opaque               1      4h26m
prometheus-kube-prometheus-stack-prometheus-web-config                           Opaque               1      4h26m
sh.helm.release.v1.kube-prometheus-stack.v1                                      helm.sh/release.v1   1      4h27m
  • nginx ๊ด€๋ จ ๊ตฌ์„ฑ ํ™•์ธ : kubectl view-secret ํ”Œ๋Ÿฌ๊ทธ์ธ์œผ๋กœ secret์˜ ๋‚ด์šฉ์„ zcatํ•˜์—ฌ nginx ๊ด€๋ จ ๋ถ€๋ถ„์„ ํ™•์ธํ•˜๋ฉด, job_name์ด ์ž๋™์œผ๋กœ ์ถ”๊ฐ€๋œ ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์Œ
  • ์ ์šฉ ๋งค์ปค๋‹ˆ์ฆ˜ : config-reloader๊ฐ€ ๋™์ ์œผ๋กœ ๋Œ€์ƒ์„ ๋ฐœ๊ฒฌํ•˜๊ณ , ํ•ด๋‹น ๋Œ€์ƒ์ด ๊ตฌ์„ฑ ํŒŒ์ผ์— ๋ฐ˜์˜๋˜๋ฉฐ, ๋ณ€๊ฒฝ๋œ ์„ค์ •์ด Prometheus ์„œ๋ฒ„์— ์ž๋™ ์ ์šฉ๋˜๋„๋ก ์ฒ˜๋ฆฌ๋จ
1

kubectl view-secret -n monitoring prometheus-kube-prometheus-stack-prometheus | zcat | grep nginx -A 20

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

Viewing only available key: prometheus.yaml.gz

gzip: stdin: unexpected end of file
- job_name: serviceMonitor/monitoring/nginx/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  scrape_interval: 10s
  metrics_path: /metrics
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_instance
    - __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance
    regex: (nginx);true
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_name
    - __meta_kubernetes_service_labelpresent_app_kubernetes_io_name
    regex: (nginx);true
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace

9. ๋งคํŠธ๋ฆญ ํ™•์ธ

1
2
3

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# NGINXIP=$(kubectl get pod -l app.kubernetes.io/instance=nginx -o jsonpath="{.items[0].status.podIP}")
(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# curl -s http://$NGINXIP:9113/metrics

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13

# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.8562e-05
go_gc_duration_seconds{quantile="0.25"} 8.4429e-05
go_gc_duration_seconds{quantile="0.5"} 0.000116249
go_gc_duration_seconds{quantile="0.75"} 0.000185463
go_gc_duration_seconds{quantile="1"} 0.003941106
go_gc_duration_seconds_sum 0.014826624
go_gc_duration_seconds_count 62
...
promhttp_metric_handler_requests_total{code="200"} 484
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

10. ์‹ค์ œ ์ปค๋„ฅ์…˜, ์›น ์ ‘์† ๋ฐ ์Šค์ผ€์ผ๋ง ๊ฒ€์ฆ

(1) ์‹ค์ œ ์ปค๋„ฅ์…˜ Active ์ •๋ณด ํ™•์ธ

  • Nginx-exporter ์—”๋“œํฌ์ธํŠธ์—์„œ ํ™œ์„ฑ ์ปค๋„ฅ์…˜ ์ˆ˜๋ฅผ ์กฐํšŒํ•จ
1

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# curl -s http://$NGINXIP:9113/metrics | grep ^nginx_connections_active

โœ…ย ์ถœ๋ ฅ

1

nginx_connections_active 1

(2) ๋ฐ˜๋ณต ์ ‘์†์„ ํ†ตํ•œ ์‘๋‹ต ํ™•์ธ

1

(eks-user@myeks:default) [root@operator-host node_exporter-1.9.0.linux-amd64]# while true; do curl -s https://nginx.$MyDomain -I | head -n 1; date; sleep 1; done

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8

HTTP/2 200 
Sat Mar  1 15:56:46 KST 2025
HTTP/2 200 
Sat Mar  1 15:56:47 KST 2025
HTTP/2 200 
Sat Mar  1 15:56:48 KST 2025
HTTP/2 200 
...

(3) ์›น ์ ‘์† ์ฃผ์†Œ ํ™•์ธ

1

echo -e "Nginx WebServer URL = https://nginx.$MyDomain"

โœ…ย ์ถœ๋ ฅ

1

Nginx WebServer URL = https://nginx.gagajin.com

(4) ์›น ํŽ˜์ด์ง€ ์ ‘์† ํ™•์ธ

1

curl -s https://nginx.$MyDomain

โœ…ย ์ถœ๋ ฅ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

(5) nginx ์Šค์ผ€์ผ ์•„์›ƒ ๊ฒ€์ฆ

1
2
3

kubectl scale deployment nginx --replicas 2
# ๊ฒฐ๊ณผ
deployment.apps/nginx scaled
  • ํ™•์žฅ ํ›„ Prometheus์—์„œ nginx_up ์ฟผ๋ฆฌ๋ฅผ ์‹คํ–‰ํ•˜์—ฌ 2๊ฐœ์˜ ๋Œ€์ƒ์ด ์ถœ๋ ฅ๋จ

Image

๐Ÿ“Š ๊ทธ๋ผํŒŒ๋‚˜

1. ๋Œ€์‹œ๋ณด๋“œ ์ž„ํฌํŠธ ๋ฐ CPU ์ ์œ ์œจ ๋Œ€์‹œ๋ณด๋“œ ํŽธ์ง‘

  • Dashboard์—์„œ New โ†’ Import๋กœ ์ด๋™ํ•œ ํ›„, 17900์„ ์ž…๋ ฅํ•˜๊ณ  Load๋ฅผ ํด๋ฆญํ•จ
  • 3๊ฐ€์ง€๋ฅผ ์…‹ํŒ…ํ•ด๋ณด์ž. (CPU ์ ์œ ์œจ, ๋ฉ”๋ชจ๋ฆฌ ์ ์œ ์œจ, ๋””์Šคํฌ ์‚ฌ์šฉ๋ฅ )

Image

2. CPU ์ ์œ ์œจ ๋Œ€์‹œ๋ณด๋“œ ํŽธ์ง‘

  • ๊ธฐ์กด ์ฟผ๋ฆฌ์—์„œ๋Š” ๋ ˆ์ด๋ธ” ํ•„ํ„ฐ๊ฐ€ node="$node"๋กœ ๋˜์–ด ์žˆ์–ด โ€œNo dataโ€๊ฐ€ ์ถœ๋ ฅ๋จ
1

sum by (instance) (irate(node_cpu_seconds_total{mode!~"guest.*|idle|iowait", node="$node"}[5m]))

Image

  • ์ฟผ๋ฆฌ์˜ ํ•„ํ„ฐ๋ฅผ instance="$instance"๋กœ ์ˆ˜์ •ํ•˜๋‹ˆ CPU ์‚ฌ์šฉ๋Ÿ‰ ๋ฐ์ดํ„ฐ๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ํ‘œ์‹œ๋จ
1

sum by (instance) (irate(node_cpu_seconds_total{mode!~"guest.*|idle|iowait", instance="$instance"}[5m]))

Image

  • ๋ณ€๊ฒฝ ํ›„ ๋Œ€์‹œ๋ณด๋“œ๋ฅผ ์ €์žฅํ•จ

3. ๋ฉ”๋ชจ๋ฆฌ ์ ์œ ์œจ ๋Œ€์‹œ๋ณด๋“œ ํŽธ์ง‘

  • ๊ธฐ์กด ์ฟผ๋ฆฌ
1

(node_memory_MemTotal_bytes{node="$node"}-node_memory_MemAvailable_bytes{node="$node"})/node_memory_MemTotal_bytes{node="$node"}
  • ์ˆ˜์ •๋œ ์ฟผ๋ฆฌ
1

(node_memory_MemTotal_bytes{instance="$instance"}-node_memory_MemAvailable_bytes{instance="$instance"})/node_memory_MemTotal_bytes{instance="$instance"}

Image Image

  • ๋ณ€๊ฒฝ ํ›„ ๋Œ€์‹œ๋ณด๋“œ๋ฅผ ์ €์žฅํ•จ

4. ๋””์Šคํฌ ์‚ฌ์šฉ๋ฅ  ๋Œ€์‹œ๋ณด๋“œ ํŽธ์ง‘

  • ๊ธฐ์กด ์ฟผ๋ฆฌ
1

sum(node_filesystem_size_bytes{node="$node"} - node_filesystem_avail_bytes{node="$node"}) by (node) / sum(node_filesystem_size_bytes{node="$node"}) by (node)
  • ์ˆ˜์ •๋œ ์ฟผ๋ฆฌ
1

sum(node_filesystem_size_bytes{instance="$instance"} - node_filesystem_avail_bytes{instance="$instance"}) by (instance) / sum(node_filesystem_size_bytes{instance="$instance"}) by (instance)

Image

  • ๋ณ€๊ฒฝ ํ›„ ๋Œ€์‹œ๋ณด๋“œ๋ฅผ ์ €์žฅํ•จ

5. ์ˆ˜์ • ๊ฒฐ๊ณผ ํ™•์ธ

Image

  • ๋ณ€๊ฒฝ๋œ ์ฟผ๋ฆฌ๋“ค์„ ์ ์šฉํ•œ ํ›„, ๊ฐ ๋Œ€์‹œ๋ณด๋“œ์—์„œ ์ •์ƒ์ ์œผ๋กœ ๋ฉ”ํŠธ๋ฆญ ๋ฐ์ดํ„ฐ๊ฐ€ ํ‘œ์‹œ๋จ

๐ŸŽ›๏ธ ์ค‘๊ธ‰ ํŒจ๋„

1. ๋…ธ๋“œ๋ณ„ 5๋ถ„๊ฐ„ CPU ์‚ฌ์šฉ ๋ณ€ํ™”์œจ

1

sum(rate(node_cpu_seconds_total[5m])) **by (instance)**

Image

2. ๋„ค์ž„์ŠคํŽ˜์ด์Šค ๋ณ„ ๋””ํ”Œ๋กœ์ด๋จผํŠธ ๊ฐฏ์ˆ˜

1

count(kube_deployment_status_replicas_available) by (namespace)

Image

3. nginx ํŒŒ๋“œ ์ˆ˜

1

kube_deployment_spec_replicas{deployment="nginx"}

Image

4. ๋…ธ๋“œ ๋ณ„ 1๋ถ„๊ฐ„ CPU ์‚ฌ์šฉ๋ฅ 

1

1 - (avg(rate(node_cpu_seconds_total{mode="idle"}[1m])) by (instance))

Image

5. ๋…ธ๋“œ OS ์ •๋ณด

1

node_os_info

Image

  • Transform data โ†’ Organize fields by name : id_like, instance, name, pretty_name ์„ ํƒ

Image

6. ์ตœ์ข… ๊ฒฐ๊ณผ

Image

๐Ÿ—‘๏ธ (์‹ค์Šต ์™„๋ฃŒ ํ›„) ์ž์› ์‚ญ์ œ

1
2

(eks-user@myeks:default) [root@operator-host ~]# nohup sh -c "eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME" > /root/delete.log 2>&1 &
[1] 16984

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

(eks-user@myeks:default) [root@operator-host ~]# tail -f delete.log

# ๊ฒฐ๊ณผ
nohup: ignoring input
2025-03-01 19:14:50 [โ„น]  deleting EKS cluster "myeks"
2025-03-01 19:14:50 [โ„น]  will drain 0 unmanaged nodegroup(s) in cluster "myeks"
2025-03-01 19:14:50 [โ„น]  starting parallel draining, max in-flight of 1
2025-03-01 19:14:51 [โ„น]  deleted 0 Fargate profile(s)
2025-03-01 19:14:51 [โœ”]  kubeconfig has been updated
2025-03-01 19:14:51 [โ„น]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2025-03-01 19:15:40 [โ„น]  
4 sequential tasks: { delete nodegroup "ng1", 
    2 sequential sub-tasks: { 
        2 parallel sub-tasks: { 
            2 sequential sub-tasks: { 
                delete IAM role for serviceaccount "amazon-cloudwatch/cloudwatch-agent",
                delete serviceaccount "amazon-cloudwatch/cloudwatch-agent",
            },
            2 sequential sub-tasks: { 
                delete IAM role for serviceaccount "kube-system/aws-load-balancer-controller",
                delete serviceaccount "kube-system/aws-load-balancer-controller",
            },
        },
        delete IAM OIDC provider,
    }, 
    2 parallel sub-tasks: { 
        delete addon IAM "eksctl-myeks-addon-aws-ebs-csi-driver",
        delete addon IAM "eksctl-myeks-addon-vpc-cni",
    }, delete cluster control plane "myeks" [async] 
}
2025-03-01 19:15:40 [โ„น]  will delete stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:15:40 [โ„น]  waiting for stack "eksctl-myeks-nodegroup-ng1" to get deleted
2025-03-01 19:15:41 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:16:11 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:17:09 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:18:09 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:19:28 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:20:53 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:21:29 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:23:01 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:23:39 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:25:17 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1"
2025-03-01 19:25:18 [โ„น]  will delete stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2025-03-01 19:25:18 [โ„น]  waiting for stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" to get deleted
2025-03-01 19:25:18 [โ„น]  will delete stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-03-01 19:25:18 [โ„น]  waiting for stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent" to get deleted
2025-03-01 19:25:18 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2025-03-01 19:25:18 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-03-01 19:25:48 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2025-03-01 19:25:48 [โ„น]  waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2025-03-01 19:25:48 [โ„น]  serviceaccount "amazon-cloudwatch/cloudwatch-agent" was already deleted
2025-03-01 19:25:48 [โ„น]  deleted serviceaccount "kube-system/aws-load-balancer-controller"
2025-03-01 19:25:49 [โ„น]  will delete stack "eksctl-myeks-addon-aws-ebs-csi-driver"
2025-03-01 19:25:49 [โ„น]  will delete stack "eksctl-myeks-addon-vpc-cni"
2025-03-01 19:25:49 [โ„น]  will delete stack "eksctl-myeks-cluster"
2025-03-01 19:25:49 [โœ”]  all cluster resources were deleted
EKS
AEWS
This post is licensed under CC BY 4.0 by the author.