AEWS 5์ฃผ์ฐจ ์ ๋ฆฌ
๐ ์ค์ต ํ๊ฒฝ ๋ฐฐํฌ
1. YAML ํ์ผ ๋ค์ด๋ก๋
1
2
3
4
5
6
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-5week.yaml
# ๊ฒฐ๊ณผ
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22269 100 22269 0 0 119k 0 --:--:-- --:--:-- --:--:-- 120k
2. ๋ณ์ ์ง์
1
2
3
4
CLUSTER_NAME=myeks
SSHKEYNAME=kp-aews # SSH ํคํ์ด ์ด๋ฆ
MYACCESSKEY=XXXXXXXXXXXXXXXXXX # IAM User ์ก์ธ์ค ํค
MYSECRETKEY=XXXXXXXXXXXXXXXXXX # IAM User ์ํฌ๋ฆฟ ํค
3. CloudFormation ์คํ ๋ฐฐํฌ
1
2
3
4
5
6
aws cloudformation deploy --template-file myeks-5week.yaml --stack-name $CLUSTER_NAME --parameter-overrides KeyName=$SSHKEYNAME SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 MyIamUserAccessKeyID=$MYACCESSKEY MyIamUserSecretAccessKey=$MYSECRETKEY ClusterBaseName=$CLUSTER_NAME --region ap-northeast-2
# ๊ฒฐ๊ณผ
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - myeks
4. CloudFormation ์คํ ๋ฐฐํฌ ์๋ฃ ํ ์์ ์ฉ EC2 IP ์ถ๋ ฅ
1
aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text
โ ย ์ถ๋ ฅ
1
43.203.230.202
โณ AWS EKS ์ค์น ํ์ธ (์คํ ์์ฑ ์์ ํ ์ฝ 20๋ถ ๊ฒฝ๊ณผ)
1. eksctl ํด๋ฌ์คํฐ ์กฐํ
1
eksctl get cluster
โ ย ์ถ๋ ฅ
1
2
NAME REGION EKSCTL CREATED
myeks ap-northeast-2 True
2. kubeconfig ์์ฑ
(1) ์๊ฒฉ์ฆ๋ช ์ฌ์ฉ์ ํ์ธ
1
aws sts get-caller-identity --query Arn
โ ย ์ถ๋ ฅ
1
"arn:aws:iam::378102432899:user/eks-user"
(2) kubeconfig ์ ๋ฐ์ดํธ ๋ช ๋ น ์คํ
1
2
3
4
aws eks update-kubeconfig --name myeks --user-alias eks-user
# ๊ฒฐ๊ณผ
Updated context eks-user in /home/devshin/.kube/config
3. Kubernetes ํด๋ฌ์คํฐ ๋ฐ ๋ฆฌ์์ค ์ํ ํ์ธ
(1) ์ธ์คํด์ค ์ ํ, ์ฉ๋ ์ ํ, ๊ฐ์ฉ ์์ญ ๋ผ๋ฒจ ์ ๋ณด ์์ธ ์กฐํ
1
kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone
โ ย ์ถ๋ ฅ
1
2
3
4
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE CAPACITYTYPE ZONE
ip-192-168-1-57.ap-northeast-2.compute.internal Ready <none> 11m v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2a
ip-192-168-2-122.ap-northeast-2.compute.internal Ready <none> 11m v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2b
ip-192-168-3-77.ap-northeast-2.compute.internal Ready <none> 11m v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2c
(2) ํ๋ ์ ๋ณด ์กฐํ
1
kubectl get pod -A
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-kb2hq 2/2 Running 0 12m
kube-system aws-node-sp6sb 2/2 Running 0 12m
kube-system aws-node-v5wpl 2/2 Running 0 12m
kube-system coredns-86f5954566-k7vk8 1/1 Running 0 18m
kube-system coredns-86f5954566-wxc2z 1/1 Running 0 18m
kube-system ebs-csi-controller-549bf6879f-2j7fb 6/6 Running 0 8m48s
kube-system ebs-csi-controller-549bf6879f-c8l6b 6/6 Running 0 8m48s
kube-system ebs-csi-node-7l559 3/3 Running 0 8m48s
kube-system ebs-csi-node-kjjww 3/3 Running 0 8m48s
kube-system ebs-csi-node-rjlkh 3/3 Running 0 8m48s
kube-system kube-proxy-6kwh5 1/1 Running 0 12m
kube-system kube-proxy-6nd4k 1/1 Running 0 12m
kube-system kube-proxy-hwdzn 1/1 Running 0 12m
kube-system metrics-server-6bf5998d9c-vb69r 1/1 Running 0 18m
kube-system metrics-server-6bf5998d9c-wgzvt 1/1 Running 0 18m
(3) ํ๋ ์ค๋จ ํ์ฉ(PDB) ์กฐํ
1
kubectl get pdb -n kube-system
โ ย ์ถ๋ ฅ
1
2
3
4
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
coredns N/A 1 1 18m
ebs-csi-controller N/A 1 1 9m21s
metrics-server N/A 1 1 18m
๐ป ๋ ธ๋ IP ์ ๋ณด ํ์ธ ๋ฐ SSH ์ ์
1. EC2 ๊ณต์ธ IP ๋ณ์ ์ง์
1
2
3
4
export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
echo $N1, $N2, $N3
โ ย ์ถ๋ ฅ
1
43.202.58.26, 43.203.112.26, 3.35.234.18
2. EC2 ๋ณด์ ๊ทธ๋ฃน ์กฐํ (remoteAccess ํํฐ ์ ์ฉ)
1
aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
{
"SecurityGroups": [
{
"GroupId": "sg-019d4308377b3778b",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"UserIdGroupPairs": [],
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": []
}
],
"Tags": [
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "aws:cloudformation:stack-name",
"Value": "eksctl-myeks-nodegroup-ng1"
},
{
"Key": "aws:cloudformation:logical-id",
"Value": "SSH"
},
{
"Key": "aws:cloudformation:stack-id",
"Value": "arn:aws:cloudformation:ap-northeast-2:378102432899:stack/eksctl-myeks-nodegroup-ng1/8611cce0-fb00-11ef-9407-0aed35b0017b"
},
{
"Key": "Name",
"Value": "eksctl-myeks-nodegroup-ng1/SSH"
},
{
"Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
"Value": "myeks"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
},
{
"Key": "alpha.eksctl.io/eksctl-version",
"Value": "0.205.0"
},
{
"Key": "alpha.eksctl.io/cluster-name",
"Value": "myeks"
}
],
"VpcId": "vpc-0f112f45f56e43be6",
"SecurityGroupArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group/sg-019d4308377b3778b",
"OwnerId": "378102432899",
"GroupName": "eksctl-myeks-nodegroup-ng1-remoteAccess",
"Description": "Allow SSH access",
"IpPermissions": [
{
"IpProtocol": "tcp",
"FromPort": 22,
"ToPort": 22,
"UserIdGroupPairs": [],
"IpRanges": [
{
"Description": "Allow SSH access to managed worker nodes in group ng1",
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [
{
"Description": "Allow SSH access to managed worker nodes in group ng1",
"CidrIpv6": "::/0"
}
],
"PrefixListIds": []
}
]
}
]
}
3. ๋ณด์ ๊ทธ๋ฃน ID ํ๊ฒฝ ๋ณ์ ์ค์
1
export MNSGID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" --query 'SecurityGroups[*].GroupId' --output text)
4. ํด๋น ๋ณด์๊ทธ๋ฃน ์ธ๋ฐ์ด๋ ๊ท์น์ ๋ณธ์ธ์ ์ง ๊ณต์ธ IP ์ถ๊ฐ
1
aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr $(curl -s ipinfo.io/ip)/32
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-0d3a8d286d364fb14",
"GroupId": "sg-019d4308377b3778b",
"GroupOwnerId": "378102432899",
"IsEgress": false,
"IpProtocol": "-1",
"FromPort": -1,
"ToPort": -1,
"CidrIpv4": "182.230.60.93/32",
"SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-0d3a8d286d364fb14"
}
]
}
5. ํด๋น ๋ณด์ ๊ทธ๋ฃน์ ์ธ๋ฐ์ด๋ ๊ท์น์ ์ด์ ์๋ฒ ๋ด๋ถ IP ์ถ๊ฐ
1
aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr 172.20.1.100/32
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-0801c124e84bdde93",
"GroupId": "sg-019d4308377b3778b",
"GroupOwnerId": "378102432899",
"IsEgress": false,
"IpProtocol": "-1",
"FromPort": -1,
"ToPort": -1,
"CidrIpv4": "172.20.1.100/32",
"SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:378102432899:security-group-rule/sgr-0801c124e84bdde93"
}
]
}
6. ์์ปค ๋ ธ๋ SSH ์ ์
1
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -o StrictHostKeyChecking=no ec2-user@$i hostname; echo; done
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
>> node 43.202.58.26 <<
Warning: Permanently added '43.202.58.26' (ED25519) to the list of known hosts.
ec2-user@43.202.58.26: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>> node 43.203.112.26 <<
Warning: Permanently added '43.203.112.26' (ED25519) to the list of known hosts.
ec2-user@43.203.112.26: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>> node 3.35.234.18 <<
Warning: Permanently added '3.35.234.18' (ED25519) to the list of known hosts.
ec2-user@3.35.234.18: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
7. ์ด์ ์๋ฒ EC2 SSH ์๊ฒฉ ์ ์ ํ ๊ธฐ๋ณธ ์ ๋ณด ํ์ธ
(1) ์ด์์๋ฒ SSH ์ ์
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
ssh -i kp-aews.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)
Last login: Fri Mar 7 12:25:06 2025 from 182.230.60.93
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###| AL2 End of Life is 2026-06-30.
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
8 package(s) needed for security, out of 9 available
Run "sudo yum update" to apply all updates.
Last login: Fri Mar 7 12:25:06 KST 2025 on pts/0
(eks-user@myeks:N/A) [root@operator-host ~]#
(2) default ๋ค์์คํ์ด์ค ์ ์ฉ
1
2
3
4
(eks-user@myeks:N/A) [root@operator-host ~]# kubectl ns default
# ๊ฒฐ๊ณผ
Context "eks-user@myeks.ap-northeast-2.eksctl.io" modified.
Active namespace is "default".
(3) ํ๊ฒฝ๋ณ์ ์ ๋ณด ํ์ธ
1
(eks-user@myeks:default) [root@operator-host ~]# export | egrep 'ACCOUNT|AWS_|CLUSTER|KUBERNETES|VPC|Subnet' | egrep -v 'KEY'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
declare -x ACCOUNT_ID="xxxxxxxxxxxx"
declare -x AWS_DEFAULT_REGION="ap-northeast-2"
declare -x AWS_PAGER=""
declare -x CLUSTER_NAME="myeks"
declare -x KUBERNETES_VERSION="1.31"
declare -x PubSubnet1="subnet-0332437e7ce3e6cfd"
declare -x PubSubnet2="subnet-02c90216149fc6da1"
declare -x PubSubnet3="subnet-0e81647e7ff562c7f"
declare -x VPCID="vpc-0f112f45f56e43be6"
(4) krew ํ๋ฌ๊ทธ์ธ ํ์ธ
1
(eks-user@myeks:default) [root@operator-host ~]# kubectl krew list
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
PLUGIN VERSION
ctx v0.9.5
df-pv v0.3.0
get-all v1.3.8
krew v0.4.4
neat v2.0.4
ns v0.9.5
oomd v0.0.7
stern v1.32.0
view-secret v0.13.0
8. ์ธ์คํด์ค ์ ๋ณด ํ์ธ
1
(eks-user@myeks:default) [root@operator-host ~]# aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
-----------------------------------------------------------------------------------------
| DescribeInstances |
+----------------------+-----------------+----------------+-----------------+-----------+
| InstanceID | InstanceName | PrivateIPAdd | PublicIPAdd | Status |
+----------------------+-----------------+----------------+-----------------+-----------+
| i-03d393483b56aa18a | myeks-ng1-Node | 192.168.3.77 | 3.35.234.18 | running |
| i-07a48f3f0fff2d64c | operator-host | 172.20.1.100 | 43.203.230.202 | running |
| i-08261e8c315e879e8 | myeks-ng1-Node | 192.168.1.57 | 43.202.58.26 | running |
| i-01861ab2663f5d732 | myeks-ng1-Node | 192.168.2.122 | 43.203.112.26 | running |
+----------------------+-----------------+----------------+-----------------+-----------+
9. PrivateIP ๋ณ์ ์ง์
1
2
3
4
5
6
7
(eks-user@myeks:default) [root@operator-host ~]# N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
(eks-user@myeks:default) [root@operator-host ~]# echo "export N1=$N1" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo "export N2=$N2" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo "export N3=$N3" >> /etc/profile
(eks-user@myeks:default) [root@operator-host ~]# echo $N1, $N2, $N3
โ ย ์ถ๋ ฅ
1
192.168.1.57, 192.168.2.122, 192.168.3.77
10. ๋ ธ๋ IP ๋ก ping ํ ์คํธ
1
(eks-user@myeks:default) [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ping -c 1 $i ; echo; done
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
>> node 192.168.1.57 <<
PING 192.168.1.57 (192.168.1.57) 56(84) bytes of data.
64 bytes from 192.168.1.57: icmp_seq=1 ttl=127 time=0.455 ms
--- 192.168.1.57 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.455/0.455/0.455/0.000 ms
>> node 192.168.2.122 <<
PING 192.168.2.122 (192.168.2.122) 56(84) bytes of data.
64 bytes from 192.168.2.122: icmp_seq=1 ttl=127 time=4.17 ms
--- 192.168.2.122 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.178/4.178/4.178/0.000 ms
>> node 192.168.3.77 <<
PING 192.168.3.77 (192.168.3.77) 56(84) bytes of data.
64 bytes from 192.168.3.77: icmp_seq=1 ttl=127 time=6.52 ms
--- 192.168.3.77 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 6.525/6.525/6.525/0.000 ms
11. EKS ๋ฐฐํฌ ํ ์ค์ต ํธ์๋ฅผ ์ํ ์ค์
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cat << EOF >> ~/.bashrc
# eksworkshop
export CLUSTER_NAME=myeks
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text)
MyDomain=gagajin.com # ๊ฐ์ ์์ ์ ๋๋ฉ์ธ ์ด๋ฆ ์
๋ ฅ
MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)
EOF
12. AWS LoadBalancerController ์ค์น
1
2
3
4
5
6
7
8
9
10
11
12
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \
--set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
# ๊ฒฐ๊ณผ
NAME: aws-load-balancer-controller
LAST DEPLOYED: Fri Mar 7 14:33:22 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!
13. ExternalDNS ์ค์น
1
curl -s https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml | MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst | kubectl apply -f -
โ ย ์ถ๋ ฅ
1
2
3
4
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created
14. gp3 ์คํ ๋ฆฌ์ง ํด๋์ค ์ค์น
(1) ์คํ ๋ฆฌ์ง ํด๋์ค ๋ฐฐํฌ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
type: gp3
allowAutoIOPSPerGBIncrease: 'true'
encrypted: 'true'
fsType: xfs # ๊ธฐ๋ณธ๊ฐ์ด ext4
EOF
# ๊ฒฐ๊ณผ
storageclass.storage.k8s.io/gp3 created
(2) ํ์ธ
1
kubectl get sc
โ ย ์ถ๋ ฅ
1
2
3
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 163m
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 29s
15. kube-ops-view ์ค์น
(1) helm ๋ฐฐํฌ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=ClusterIP --set env.TZ="Asia/Seoul" --namespace kube-system
# ๊ฒฐ๊ณผ
NAME: kube-ops-view
LAST DEPLOYED: Fri Mar 7 14:37:42 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:8080
(2) Ingress ์ค์
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/load-balancer-name: $CLUSTER_NAME-ingress-alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/target-type: ip
labels:
app.kubernetes.io/name: kubeopsview
name: kubeopsview
namespace: kube-system
spec:
ingressClassName: alb
rules:
- host: kubeopsview.$MyDomain
http:
paths:
- backend:
service:
name: kube-ops-view
port:
number: 8080 # name: http
path: /
pathType: Prefix
EOF
# ๊ฒฐ๊ณผ
ingress.networking.k8s.io/kubeopsview created
- ๊ทธ๋ฃน ์ค์ ์ ํตํด ๋จ์ผ ALB๋ฅผ ์ฌ๋ฌ Ingress๊ฐ ๊ณต์ฉ์ผ๋ก ์ฌ์ฉํ๋๋ก ์ค์
16. prometheus-stack ์ค์น
(1) helm ์ ์ฅ์ ์ถ๊ฐ
1
2
3
4
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# ๊ฒฐ๊ณผ
"prometheus-community" already exists with the same configuration, skipping
(2) ํ๋ผ๋ฏธํฐ ํ์ผ ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
cat <<EOT > monitor-values.yaml
prometheus:
prometheusSpec:
scrapeInterval: "15s"
evaluationInterval: "15s"
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
retention: 5d
retentionSize: "10GiB"
# Enable vertical pod autoscaler support for prometheus-operator
verticalPodAutoscaler:
enabled: true
ingress:
enabled: true
ingressClassName: alb
hosts:
- prometheus.$MyDomain
paths:
- /*
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/ssl-redirect: '443'
grafana:
defaultDashboardsTimezone: Asia/Seoul
adminPassword: prom-operator
defaultDashboardsEnabled: false
ingress:
enabled: true
ingressClassName: alb
hosts:
- grafana.$MyDomain
paths:
- /*
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/ssl-redirect: '443'
kube-state-metrics:
rbac:
extraRules:
- apiGroups: ["autoscaling.k8s.io"]
resources: ["verticalpodautoscalers"]
verbs: ["list", "watch"]
customResourceState:
enabled: true
config:
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: autoscaling.k8s.io
kind: "VerticalPodAutoscaler"
version: "v1"
labelsFromPath:
verticalpodautoscaler: [metadata, name]
namespace: [metadata, namespace]
target_api_version: [apiVersion]
target_kind: [spec, targetRef, kind]
target_name: [spec, targetRef, name]
metrics:
- name: "vpa_containerrecommendations_target"
help: "VPA container recommendations for memory."
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
valueFrom: [target, memory]
labelsFromPath:
container: [containerName]
commonLabels:
resource: "memory"
unit: "byte"
- name: "vpa_containerrecommendations_target"
help: "VPA container recommendations for cpu."
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
valueFrom: [target, cpu]
labelsFromPath:
container: [containerName]
commonLabels:
resource: "cpu"
unit: "core"
selfMonitor:
enabled: true
alertmanager:
enabled: false
defaultRules:
create: false
kubeControllerManager:
enabled: false
kubeEtcd:
enabled: false
kubeScheduler:
enabled: false
prometheus-windows-exporter:
prometheus:
monitor:
enabled: false
EOT
(3) helm ๋ฐฐํฌ
1
2
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 69.3.1 \
-f monitor-values.yaml --create-namespace --namespace monitoring
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
NAME: kube-prometheus-stack
LAST DEPLOYED: Fri Mar 7 14:47:53 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace monitoring get pods -l "release=kube-prometheus-stack"
Get Grafana 'admin' user password by running:
kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
Access Grafana local instance:
export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prometheus-stack" -oname)
kubectl --namespace monitoring port-forward $POD_NAME 3000
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
(4) kube-prometheus-stack ๋ชจ๋ํฐ๋ง ์ค์ ๊ฐ ์กฐํ
1
helm get values -n monitoring kube-prometheus-stack
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
USER-SUPPLIED VALUES:
alertmanager:
enabled: false
defaultRules:
create: false
grafana:
adminPassword: prom-operator
defaultDashboardsEnabled: false
defaultDashboardsTimezone: Asia/Seoul
ingress:
annotations:
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/target-type: ip
enabled: true
hosts:
- grafana.gagajin.com
ingressClassName: alb
paths:
- /*
kube-state-metrics:
customResourceState:
config:
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: autoscaling.k8s.io
kind: VerticalPodAutoscaler
version: v1
labelsFromPath:
namespace:
- metadata
- namespace
target_api_version:
- apiVersion
target_kind:
- spec
- targetRef
- kind
target_name:
- spec
- targetRef
- name
verticalpodautoscaler:
- metadata
- name
metrics:
- commonLabels:
resource: memory
unit: byte
each:
gauge:
labelsFromPath:
container:
- containerName
path:
- status
- recommendation
- containerRecommendations
valueFrom:
- target
- memory
type: Gauge
help: VPA container recommendations for memory.
name: vpa_containerrecommendations_target
- commonLabels:
resource: cpu
unit: core
each:
gauge:
labelsFromPath:
container:
- containerName
path:
- status
- recommendation
- containerRecommendations
valueFrom:
- target
- cpu
type: Gauge
help: VPA container recommendations for cpu.
name: vpa_containerrecommendations_target
enabled: true
rbac:
extraRules:
- apiGroups:
- autoscaling.k8s.io
resources:
- verticalpodautoscalers
verbs:
- list
- watch
selfMonitor:
enabled: true
kubeControllerManager:
enabled: false
kubeEtcd:
enabled: false
kubeScheduler:
enabled: false
prometheus:
ingress:
annotations:
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:378102432899:certificate/f967e8ca-f0b5-471d-bbe4-bee231aeb32b
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/target-type: ip
enabled: true
hosts:
- prometheus.gagajin.com
ingressClassName: alb
paths:
- /*
prometheusSpec:
evaluationInterval: 15s
podMonitorSelectorNilUsesHelmValues: false
retention: 5d
retentionSize: 10GiB
scrapeInterval: 15s
serviceMonitorSelectorNilUsesHelmValues: false
verticalPodAutoscaler:
enabled: true
prometheus-windows-exporter:
prometheus:
monitor:
enabled: false
(5) ํ๋ก๋ฉํ ์ฐ์ค ์น ์ ์
1
echo -e "https://prometheus.$MyDomain"
โ ย ์ถ๋ ฅ
1
https://prometheus.gagajin.com
(6) ๊ทธ๋ผํ๋ ์น ์ ์ : admin / prom-operator
1
echo -e "https://grafana.$MyDomain"
โ ย ์ถ๋ ฅ
1
https://grafana.gagajin.com
(7) ALB TargetGroupBindings ํ์ธ
1
kubectl get targetgroupbindings.elbv2.k8s.aws -A
โ ย ์ถ๋ ฅ
1
2
3
4
NAMESPACE NAME SERVICE-NAME SERVICE-PORT TARGET-TYPE AGE
kube-system k8s-kubesyst-kubeopsv-1a26c69536 kube-ops-view 8080 ip 18m
monitoring k8s-monitori-kubeprom-72c21479ad kube-prometheus-stack-prometheus 9090 ip 8m48s
monitoring k8s-monitori-kubeprom-b4f67c8fac kube-prometheus-stack-grafana 80 ip 8m50s
(8) kube-state-metrics Pod ์กฐํ
1
kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics
โ ย ์ถ๋ ฅ
1
2
NAME READY STATUS RESTARTS AGE
kube-prometheus-stack-kube-state-metrics-5674c7ddd8-njk9x 1/1 Running 0 9m24s
(9) kube-state-metrics Pod ์์ธ ์ ๋ณด ํ์ธ
1
kubectl describe pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
Name: kube-prometheus-stack-kube-state-metrics-5674c7ddd8-njk9x
Namespace: monitoring
Priority: 0
Service Account: kube-prometheus-stack-kube-state-metrics
Node: ip-192-168-1-57.ap-northeast-2.compute.internal/192.168.1.57
Start Time: Fri, 07 Mar 2025 14:48:06 +0900
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=kube-prometheus-stack
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-state-metrics
app.kubernetes.io/part-of=kube-state-metrics
app.kubernetes.io/version=2.14.0
helm.sh/chart=kube-state-metrics-5.29.0
pod-template-hash=5674c7ddd8
release=kube-prometheus-stack
Annotations: <none>
Status: Running
SeccompProfile: RuntimeDefault
IP: 192.168.1.236
IPs:
IP: 192.168.1.236
Controlled By: ReplicaSet/kube-prometheus-stack-kube-state-metrics-5674c7ddd8
Containers:
kube-state-metrics:
Container ID: containerd://622a2c62b6fa0ee99a1ef52ad38dd7719646ce04da8a98366c552dc87e24727b
Image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.14.0
Image ID: registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:37d841299325c23b56e5951176ce8ef317d537447c0f1b2d2437dddbb1f51165
Ports: 8080/TCP, 8081/TCP
Host Ports: 0/TCP, 0/TCP
Args:
--port=8080
--resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
--custom-resource-state-config-file=/etc/customresourcestate/config.yaml
State: Running
Started: Fri, 07 Mar 2025 14:48:11 +0900
Ready: True
Restart Count: 0
Liveness: http-get http://:8080/livez delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:8081/readyz delay=5s timeout=5s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/customresourcestate from customresourcestate-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cwrtt (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
customresourcestate-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-prometheus-stack-kube-state-metrics-customresourcestate-config
Optional: false
kube-api-access-cwrtt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned monitoring/kube-prometheus-stack-kube-state-metrics-5674c7ddd8-njk9x to ip-192-168-1-57.ap-northeast-2.compute.internal
Normal Pulling 10m kubelet Pulling image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.14.0"
Normal Pulled 10m kubelet Successfully pulled image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.14.0" in 4.192s (4.192s including waiting). Image size: 15427061 bytes.
Normal Created 10m kubelet Created container kube-state-metrics
Normal Started 10m kubelet Started container kube-state-metrics
- Service Account ๋งค์นญ๋จ
- ํด๋น ๊ถํ์ผ๋ก ํ๋ก๋ฉํ ์ฐ์ค๊ฐ ๋ฆฌ์์ค ์ ๋ณด ์์งํจ
(10) CustomResourceStateMetrics ConfigMap ์กฐํ
1
kubectl describe cm -n monitoring kube-prometheus-stack-kube-state-metrics-customresourcestate-config
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
Name: kube-prometheus-stack-kube-state-metrics-customresourcestate-config
Namespace: monitoring
Labels: app.kubernetes.io/component=metrics
app.kubernetes.io/instance=kube-prometheus-stack
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-state-metrics
app.kubernetes.io/part-of=kube-state-metrics
app.kubernetes.io/version=2.14.0
helm.sh/chart=kube-state-metrics-5.29.0
release=kube-prometheus-stack
Annotations: meta.helm.sh/release-name: kube-prometheus-stack
meta.helm.sh/release-namespace: monitoring
Data
====
config.yaml:
----
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: autoscaling.k8s.io
kind: VerticalPodAutoscaler
version: v1
labelsFromPath:
namespace:
- metadata
- namespace
target_api_version:
- apiVersion
target_kind:
- spec
- targetRef
- kind
target_name:
- spec
- targetRef
- name
verticalpodautoscaler:
- metadata
- name
metrics:
- commonLabels:
resource: memory
unit: byte
each:
gauge:
labelsFromPath:
container:
- containerName
path:
- status
- recommendation
- containerRecommendations
valueFrom:
- target
- memory
type: Gauge
help: VPA container recommendations for memory.
name: vpa_containerrecommendations_target
- commonLabels:
resource: cpu
unit: core
each:
gauge:
labelsFromPath:
container:
- containerName
path:
- status
- recommendation
- containerRecommendations
valueFrom:
- target
- cpu
type: Gauge
help: VPA container recommendations for cpu.
name: vpa_containerrecommendations_target
BinaryData
====
Events: <none>
- CustomResourceStateMetrics ์ค์ ํ์ผ ์ฌ์ฉ๋จ
- ํด๋น ํ์ผ์ Volume Mountํ์ฌ ์ ์ฉํจ
- Mount ๊ฒฐ๊ณผ๋ฅผ ํ๊ฒฝ๋ณ์ argument๋ก ์ ๋ฌํจ
(11) ClusterRole์์ verticalpodautoscalers ๊ถํ ํ์ธ
1
kubectl describe clusterrole kube-prometheus-stack-kube-state-metrics | grep verticalpodautoscalers
โ ย ์ถ๋ ฅ
1
verticalpodautoscalers.autoscaling.k8s.io [] [] [list watch]
16. EKS Node Viewer ์ค์น
(1) ๋ก์ปฌ PC์ ์ค์น - Arch Linux
- Go ์ค์น
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
sudo pacman -S go
# ๊ฒฐ๊ณผ
warning: go-2:1.24.0-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Packages (1) go-2:1.24.0-1
Total Installed Size: 237.31 MiB
Net Upgrade Size: 0.00 MiB
:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring [##############] 100%
(1/1) checking package integrity [##############] 100%
(1/1) loading package files [##############] 100%
(1/1) checking for file conflicts [##############] 100%
(1/1) checking available disk space [##############] 100%
:: Processing package changes...
(1/1) reinstalling go [##############] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
- eks-node-viewer ์ค์น
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest
# ๊ฒฐ๊ณผ
go: downloading github.com/awslabs/eks-node-viewer v0.7.1
go: downloading k8s.io/client-go v0.31.3
go: downloading github.com/charmbracelet/bubbletea v1.2.4
go: downloading github.com/charmbracelet/bubbles v0.20.0
go: downloading github.com/charmbracelet/lipgloss v1.0.0
go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb
go: downloading golang.org/x/text v0.19.0
go: downloading k8s.io/apimachinery v0.31.3
go: downloading k8s.io/api v0.31.3
go: downloading github.com/aws/aws-sdk-go v1.55.5
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.194.0
go: downloading sigs.k8s.io/karpenter v1.0.4
...
- PATH ์ค์
1
2
echo 'export PATH="$PATH:$HOME/go/bin"' >> ~/.profile
source ~/.profile
(2) ์ด์์๋ฒ EC2์ ์ค์น
- GO ์ค์น
1
2
3
4
5
6
7
8
# userdata ํตํด ์ด๋ฏธ ์ค์น ๋์ด ์์
(eks-user@myeks:default) [root@operator-host ~]# yum install golang -y
# ๊ฒฐ๊ณผ
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00
Package golang-1.22.7-1.amzn2.0.1.x86_64 already installed and latest version
Nothing to do
- eks-node-viewer ์ค์น
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(eks-user@myeks:default) [root@operator-host ~]# go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest
# ๊ฒฐ๊ณผ
go: downloading github.com/awslabs/eks-node-viewer v0.7.1
go: downloading github.com/aws/aws-sdk-go v1.55.5
go: downloading github.com/charmbracelet/bubbletea v1.2.4
go: downloading k8s.io/apimachinery v0.31.3
go: downloading k8s.io/client-go v0.31.3
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.194.0
go: downloading go.uber.org/multierr v1.11.0
go: downloading k8s.io/api v0.31.3
go: downloading sigs.k8s.io/karpenter v1.0.4
go: downloading github.com/charmbracelet/bubbles v0.20.0
go: downloading github.com/charmbracelet/lipgloss v1.0.0
go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb
...
(3) ์ฌ์ฉ
- ๊ธฐ๋ณธ ์ฌ์ฉ
1
eks-node-viewer
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
3 nodes ( 1060m/5790m) 18.3% cpu โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
26 pods (0 pending 26 running 26 bound)
ip-192-168-3-77.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโโโโ
ip-192-168-2-122.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโโโโ
ip-192-168-1-57.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโโโโ
CPU ๋ฐ ๋ฉ๋ชจ๋ฆฌ ์ฌ์ฉ๋ ํ์
1
eks-node-viewer --resources cpu,memory
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
3 nodes ( 1060m/5790m) 18.3% cpu โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1380Mi/10093572Ki 14.0% memory โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
26 pods (0 pending 26 running 26 bound)
ip-192-168-3-77.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-2-122.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-1-57.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
โ๏ธ HPA - Horizontal Pod Autoscaler
1. ๊ทธ๋ผํ๋ ๋์๋ณด๋ Import ์ค์
- 22251 - https://grafana.com/grafana/dashboards/22251-kubernetes-autoscaling-horizontal-pod-autoscaler/
2. ์ํ ์ ํ๋ฆฌ์ผ์ด์ ๋ฐฐํฌ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
cat << EOF > php-apache.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: registry.k8s.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache
EOF
kubectl apply -f php-apache.yaml
# ๊ฒฐ๊ณผ
deployment.apps/php-apache created
service/php-apache created
- php-apache Deployment์ Service๋ฅผ YAML ํ์ผ๋ก ์์ฑํ์ฌ ๋ฐฐํฌํจ
3. index.php ํ์ธ
1
kubectl exec -it deploy/php-apache -- cat /var/www/html/index.php
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
<?php
$x = 0.0001;
for ($i = 0; $i <= 1000000; $i++) {
$x += sqrt($x);
}
echo "OK!";
?>
kubectl exec๋ช ๋ น์ด๋ก ์ปจํ ์ด๋ ๋ด index.php ํ์ผ ๋ด์ฉ ์กฐํ- PHP ์ฝ๋๊ฐ ์ ์์ ์ผ๋ก ์ถ๋ ฅ๋จ
4. ๋ฆฌ์์ค ๋ชจ๋ํฐ๋ง
1
watch -d 'kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
Every 2.0s: kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node gram88: 05:51:09 PM
NAME READY STATUS RESTARTS AGE
pod/php-apache-d87b7ff46-zngp4 1/1 Running 0 2m32s
NAME CPU(cores) MEMORY(bytes)
php-apache-d87b7ff46-zngp4 1m 8Mi
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
ip-192-168-1-57.ap-northeast-2.compute.internal 138m 7% 975Mi 29%
ip-192-168-2-122.ap-northeast-2.compute.internal 56m 2% 713Mi 21%
ip-192-168-3-77.ap-northeast-2.compute.internal 109m 5% 877Mi 26%
watch๋ช ๋ น์ด๋ก HPA, Pod, ๋ ธ๋์ CPU/๋ฉ๋ชจ๋ฆฌ ์ฌ์ฉ๋ ๋ชจ๋ํฐ๋ง- Pod ๋ด
top๋ช ๋ น์ด๋ฅผ ์ฌ์ฉํด ์์ธ ๋ฆฌ์์ค ์ํ ํ์ธ
5. Pod ๋ด ๋ฆฌ์์ค ๋ชจ๋ํฐ๋ง
1
kubectl exec -it deploy/php-apache -- top
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
top - 09:06:07 up 6:03, 0 users, load average: 0.02, 0.03, 0.06
Tasks: 7 total, 1 running, 6 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.7 us, 0.8 sy, 0.0 ni, 95.5 id, 0.0 wa, 0.0 hi, 0.0 si,
KiB Mem: 3919532 total, 3228076 used, 691456 free, 3172 buffers
KiB Swap: 0 total, 0 used, 0 free. 2375336 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
1 root 20 0 166268 19244 13968 S 0.0 0.5 0:00.09
8 www-data 20 0 166292 7252 1968 S 0.0 0.2 0:00.00
9 www-data 20 0 166292 7252 1968 S 0.0 0.2 0:00.00
10 www-data 20 0 166292 7252 1968 S 0.0 0.2 0:00.00
11 www-data 20 0 166292 7252 1968 S 0.0 0.2 0:00.00
12 www-data 20 0 166292 7252 1968 S 0.0 0.2 0:00.00
20 root 20 0 21920 2404 2080 R 0.0 0.1 0:00.01
kubectl exec -it deploy/php-apache -- top๋ช ๋ น์ผ๋ก Pod ๋ด์์ ์ค์๊ฐ ๋ฆฌ์์ค ์ฌ์ฉ ์ํ๋ฅผ ํ์ธํจ
6. CPU ๋ถํ ํ ์คํธ
1
2
(eks-user@myeks:default) [root@operator-host ~]# PODIP=$(kubectl get pod -l run=php-apache -o jsonpath="{.items[0].status.podIP}")
(eks-user@myeks:default) [root@operator-host ~]# curl -s $PODIP; echo
- ์ด์์๋ฒ์์ Pod์ IP๋ฅผ ๊ฐ์ ธ์ ๋ฐ๋ณต์ ์ผ๋ก
curl์์ฒญ์ ๋ณด๋ด๋ฉด, ์ํ์น ์น ๋ฐ๋ชฌ์์ CPU ์ฌ์ฉ๋์ด ์ฆ๊ฐํจ - ๊ฐ ์์ฒญ๋ง๋ค CPU ๋ฐฑ๋ง ๋ฒ ์ฐ์ฐ์ ์ํํ์ฌ ๋ถํ๊ฐ ๋ฐ์ํจ
7. HPA ์ ์ฑ ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
averageUtilization: 50
type: Utilization
EOF
# ๊ฒฐ๊ณผ
horizontalpodautoscaler.autoscaling/php-apache created
- CPU ํ๊ท ํ์ฉ๋ฅ ์ด 50%๋ฅผ ์ด๊ณผํ๋ฉด ์๋์ผ๋ก Pod ์๋ฅผ ๋๋ฆฌ๋๋ก HPA๋ฅผ ์์ฑํจ
8. HPA ์ ์ฑ ์์ธ ์กฐํ
1
kubectl get hpa php-apache -o yaml | kubectl neat
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
maxReplicas: 10 # [3] ํฌ๋๋ฅผ ์ต๋ 10๊ฐ๊น์ง ๋๋ฆฝ๋๋ค.
metrics:
- resource:
name: cpu
target:
averageUtilization: 50 # [2] CPU ํ์ฉ๋ฅ ์ด 50% ์ด์์ธ ๊ฒฝ์ฐ
type: Utilization
type: Resource
minReplicas: 1 # [4] ์ต์ 1๊ฐ๊น์ง ์ค์ด๋ค ์๋ ์์ต๋๋ค.
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache # [1] php-apache ์ ์์ ์ฌ์ฉ๋์์
HPA ์ ์ฑ ์ค์ ํ ๊ทธ๋ผํ๋๋ฅผ ์๋ก ๊ณ ์นจํ๋ฉด ๋ฐ์๋จ
9. ๋ถํ ๋ฐ์ ํ ์คํธ
1
(eks-user@myeks:default) [root@operator-host ~]# kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
kubectl run -i --tty load-generator ...๋ช ๋ น์ด๋ก ๋ฐ๋ณต์ ์ธ HTTP ์์ฒญ์ ๋ณด๋ด ๋ถํ๋ฅผ ๋ฐ์์ํด- ๋ถํ ๋ฐ์ ์ PHP Apache ์๋น์ค์ CPU ์ฌ์ฉ๋์ด ์ฆ๊ฐํจ
10. ํด๋ฌ์คํฐ ์ํ ๋ชจ๋ํฐ๋ง
1
watch -d 'kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node'
โ ย ์ถ๋ ฅ
- ๋ถํ ๋ฐ์ ์ด๊ธฐ ์ํ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Every 2.0s: kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node gram88: 06:15:15 PM
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache Deployment/php-apache cpu: 0%/50% 1 10 1 19m
NAME READY STATUS RESTARTS AGE
pod/php-apache-d87b7ff46-zngp4 1/1 Running 0 26m
NAME CPU(cores) MEMORY(bytes)
php-apache-d87b7ff46-zngp4 1m 11Mi
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
ip-192-168-1-57.ap-northeast-2.compute.internal 152m 7% 981Mi 29%
ip-192-168-2-122.ap-northeast-2.compute.internal 85m 4% 686Mi 20%
ip-192-168-3-77.ap-northeast-2.compute.internal 118m 6% 835Mi 25%
- ๋ถํ ์ง์ ํ ์ํ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Every 2.0s: kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node gram88: 06:21:11 PM
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache Deployment/php-apache cpu: 39%/50% 1 10 8 25m
NAME READY STATUS RESTARTS AGE
pod/load-generator 1/1 Running 0 4m20s
pod/php-apache-d87b7ff46-6mbp9 1/1 Running 0 3m47s
pod/php-apache-d87b7ff46-fzjts 1/1 Running 0 3m17s
pod/php-apache-d87b7ff46-hkr6k 1/1 Running 0 3m32s
pod/php-apache-d87b7ff46-lbpmv 1/1 Running 0 3m17s
pod/php-apache-d87b7ff46-n6qjb 1/1 Running 0 3m17s
pod/php-apache-d87b7ff46-nk6kf 1/1 Running 0 3m47s
pod/php-apache-d87b7ff46-vp2nx 1/1 Running 0 3m47s
pod/php-apache-d87b7ff46-zngp4 1/1 Running 0 32m
NAME CPU(cores) MEMORY(bytes)
load-generator 10m 0Mi
php-apache-d87b7ff46-6mbp9 95m 12Mi
php-apache-d87b7ff46-fzjts 54m 11Mi
php-apache-d87b7ff46-hkr6k 140m 11Mi
php-apache-d87b7ff46-lbpmv 59m 11Mi
php-apache-d87b7ff46-n6qjb 76m 11Mi
php-apache-d87b7ff46-nk6kf 60m 11Mi
php-apache-d87b7ff46-vp2nx 49m 12Mi
php-apache-d87b7ff46-zngp4 99m 11Mi
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
ip-192-168-1-57.ap-northeast-2.compute.internal 397m 20% 1347Mi 41%
ip-192-168-2-122.ap-northeast-2.compute.internal 354m 18% 724Mi 22%
ip-192-168-3-77.ap-northeast-2.compute.internal 416m 21% 942Mi 28%
- HPA, Pod, ๋ ธ๋์ CPU/๋ฉ๋ชจ๋ฆฌ ์ฌ์ฉ๋์ ์ค์๊ฐ์ผ๋ก ํ์ธํจ
11. HPA ํ๋ก๋ฉํ ์ฐ์ค ๋ฉํธ๋ฆญ ํ์ธ
1
kube_horizontalpodautoscaler_status_desired_replicas
12. ์ด์์๋ฒ EC2 ์ํ ํ์ธ
(1) kube-state-metrics Pod ์์ธ ์ ๋ณด ํ์ธ
1
(eks-user@myeks:default) [root@operator-host ~]# kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -owide
โ ย ์ถ๋ ฅ
1
2
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-prometheus-stack-kube-state-metrics-5674c7ddd8-njk9x 1/1 Running 0 3h39m 192.168.1.236 ip-192-168-1-57.ap-northeast-2.compute.internal <none> <none>
(2) kube-state-metrics Pod IP ์กฐํ
1
(eks-user@myeks:default) [root@operator-host ~]# kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o jsonpath="{.items[*].status.podIP}"
โ ย ์ถ๋ ฅ
1
192.168.1.236
- Target health๋ฅผ ํ์ธํ๋ฉด, http://192.168.1.236:8080/metrics ์ฃผ์์์ kube-state-metrics์ ๋ฉํธ๋ฆญ ๋ฐ์ดํฐ๋ฅผ ์กฐํํ ์ ์์
(3) ์ค์ ๋ฉํธ๋ฆญ ์กฐํ ํ ์คํธ
- POD IP๋ฅผ ์ถ์ถํ์ฌ kube-state-metrics์ ๋ฉํธ๋ฆญ ์๋ํฌ์ธํธ๋ฅผ ํ์ธํจ
- ์ถ์ถํ POD IP๋ฅผ ์ฌ์ฉํด, curl ๋ช ๋ น์ด๋ก HorizontalPodAutoscaler ๊ด๋ จ HELP ๋ฉ์์ง ๋ฐ ์ธ๋ถ ์ ๋ณด๋ฅผ ์กฐํํจ
1
(eks-user@myeks:default) [root@operator-host ~]# PODIP=$(kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o jsonpath="{.items[*].status.podIP}")
1
(eks-user@myeks:default) [root@operator-host ~]# curl -s http://$PODIP:8080/metrics | grep -i horizontalpodautoscaler | grep HELP
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
# HELP kube_horizontalpodautoscaler_info Information about this autoscaler.
# HELP kube_horizontalpodautoscaler_metadata_generation [STABLE] The generation observed by the HorizontalPodAutoscaler controller.
# HELP kube_horizontalpodautoscaler_spec_max_replicas [STABLE] Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than MinReplicas.
# HELP kube_horizontalpodautoscaler_spec_min_replicas [STABLE] Lower limit for the number of pods that can be set by the autoscaler, default 1.
# HELP kube_horizontalpodautoscaler_spec_target_metric The metric specifications used by this autoscaler when calculating the desired replica count.
# HELP kube_horizontalpodautoscaler_status_target_metric The current metric status used by this autoscaler when calculating the desired replica count.
# HELP kube_horizontalpodautoscaler_status_current_replicas [STABLE] Current number of replicas of pods managed by this autoscaler.
# HELP kube_horizontalpodautoscaler_status_desired_replicas [STABLE] Desired number of replicas of pods managed by this autoscaler.
# HELP kube_horizontalpodautoscaler_annotations Kubernetes annotations converted to Prometheus labels.
# HELP kube_horizontalpodautoscaler_labels [STABLE] Kubernetes labels converted to Prometheus labels.
# HELP kube_horizontalpodautoscaler_status_condition [STABLE] The condition of this autoscaler.
- HPA์ ์ค์ (์ต์/์ต๋ Replica, CPU ๋ชฉํ ํ์ฉ๋ฅ ๋ฑ)๊ณผ ์ํ(ํ์ฌ/์ํ๋ Replica ์, ์กฐ๊ฑด ๋ฑ)๋ฅผ ํ์ธํจ
1
(eks-user@myeks:default) [root@operator-host ~]# curl -s http://$PODIP:8080/metrics | grep -i horizontalpodautoscaler
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# HELP kube_horizontalpodautoscaler_info Information about this autoscaler.
# TYPE kube_horizontalpodautoscaler_info gauge
kube_horizontalpodautoscaler_info{namespace="default",horizontalpodautoscaler="php-apache",scaletargetref_api_version="apps/v1",scaletargetref_kind="Deployment",scaletargetref_name="php-apache"} 1
# HELP kube_horizontalpodautoscaler_metadata_generation [STABLE] The generation observed by the HorizontalPodAutoscaler controller.
# TYPE kube_horizontalpodautoscaler_metadata_generation gauge
kube_horizontalpodautoscaler_metadata_generation{namespace="default",horizontalpodautoscaler="php-apache"} 0
# HELP kube_horizontalpodautoscaler_spec_max_replicas [STABLE] Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than MinReplicas.
# TYPE kube_horizontalpodautoscaler_spec_max_replicas gauge
kube_horizontalpodautoscaler_spec_max_replicas{namespace="default",horizontalpodautoscaler="php-apache"} 10
# HELP kube_horizontalpodautoscaler_spec_min_replicas [STABLE] Lower limit for the number of pods that can be set by the autoscaler, default 1.
# TYPE kube_horizontalpodautoscaler_spec_min_replicas gauge
kube_horizontalpodautoscaler_spec_min_replicas{namespace="default",horizontalpodautoscaler="php-apache"} 1
# HELP kube_horizontalpodautoscaler_spec_target_metric The metric specifications used by this autoscaler when calculating the desired replica count.
# TYPE kube_horizontalpodautoscaler_spec_target_metric gauge
kube_horizontalpodautoscaler_spec_target_metric{namespace="default",horizontalpodautoscaler="php-apache",metric_name="cpu",metric_target_type="utilization"} 50
# HELP kube_horizontalpodautoscaler_status_target_metric The current metric status used by this autoscaler when calculating the desired replica count.
# TYPE kube_horizontalpodautoscaler_status_target_metric gauge
kube_horizontalpodautoscaler_status_target_metric{namespace="default",horizontalpodautoscaler="php-apache",metric_name="cpu",metric_target_type="utilization"} 0
kube_horizontalpodautoscaler_status_target_metric{namespace="default",horizontalpodautoscaler="php-apache",metric_name="cpu",metric_target_type="average"} 0.001
# HELP kube_horizontalpodautoscaler_status_current_replicas [STABLE] Current number of replicas of pods managed by this autoscaler.
# TYPE kube_horizontalpodautoscaler_status_current_replicas gauge
kube_horizontalpodautoscaler_status_current_replicas{namespace="default",horizontalpodautoscaler="php-apache"} 1
# HELP kube_horizontalpodautoscaler_status_desired_replicas [STABLE] Desired number of replicas of pods managed by this autoscaler.
# TYPE kube_horizontalpodautoscaler_status_desired_replicas gauge
kube_horizontalpodautoscaler_status_desired_replicas{namespace="default",horizontalpodautoscaler="php-apache"} 1
# HELP kube_horizontalpodautoscaler_annotations Kubernetes annotations converted to Prometheus labels.
# TYPE kube_horizontalpodautoscaler_annotations gauge
# HELP kube_horizontalpodautoscaler_labels [STABLE] Kubernetes labels converted to Prometheus labels.
# TYPE kube_horizontalpodautoscaler_labels gauge
# HELP kube_horizontalpodautoscaler_status_condition [STABLE] The condition of this autoscaler.
# TYPE kube_horizontalpodautoscaler_status_condition gauge
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="AbleToScale",status="true"} 1
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="AbleToScale",status="false"} 0
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="AbleToScale",status="unknown"} 0
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingActive",status="true"} 1
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingActive",status="false"} 0
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingActive",status="unknown"} 0
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingLimited",status="true"} 1
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingLimited",status="false"} 0
kube_horizontalpodautoscaler_status_condition{namespace="default",horizontalpodautoscaler="php-apache",condition="ScalingLimited",status="unknown"} 0
- ์ต๋ 8๊ฐ๊น์ง ์ค์ผ์ผ๋ง๋จ
- ๋ถํ๋ฅผ ์ค๋ Pod๊ฐ 8๊ฐ๋ก ๋ถ์ฐ๋์ด CPU 50% ๋ชฉํ์น๋ฅผ ๋์ง ์์
13. ๊ด๋ จ ์ค๋ธ์ ํธ ์ญ์
1
2
3
4
5
6
7
8
kubectl delete deploy,svc,hpa,pod --all
# ๊ฒฐ๊ณผ
deployment.apps "php-apache" deleted
service "kubernetes" deleted
service "php-apache" deleted
horizontalpodautoscaler.autoscaling "php-apache" deleted
pod "php-apache-d87b7ff46-zngp4" deleted
โก KEDA - Kubernetes based Event Driven Autoscaler
1. KEDA ๋์๋ณด๋ ์ฌ์ฉ
- KEDA ๋์๋ณด๋ JSON ํ์ผ์ ๋ณต์ฌํ์ฌ, ๊ทธ๋ผํ๋์ โImport via dashboard JSON modelโ์ ๋ถ์ฌ๋ฃ์
- KEDA ๋์๋ณด๋ ๋ชจ๋ํฐ๋ง
2. Metrics API ํ์ธ
1
kubectl get --raw "/apis/metrics.k8s.io" -v=6 | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
I0307 21:59:22.832615 118946 loader.go:402] Config loaded from file: /home/devshin/.kube/config
I0307 21:59:22.832899 118946 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0307 21:59:22.832911 118946 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0307 21:59:22.832916 118946 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0307 21:59:22.832921 118946 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0307 21:59:23.349279 118946 round_trippers.go:560] GET https://C7C93340F5C4C4F9E2C362E8D72C06EF.gr7.ap-northeast-2.eks.amazonaws.com/apis/metrics.k8s.io 200 OK in 516 milliseconds
{
"kind": "APIGroup",
"apiVersion": "v1",
"name": "metrics.k8s.io",
"versions": [
{
"groupVersion": "metrics.k8s.io/v1beta1",
"version": "v1beta1"
}
],
"preferredVersion": {
"groupVersion": "metrics.k8s.io/v1beta1",
"version": "v1beta1"
}
}
3. KEDA ์ค์น
(1) ํ๋ผ๋ฏธํฐ ํ์ผ ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cat <<EOT > keda-values.yaml
metricsServer:
useHostNetwork: true
prometheus:
metricServer:
enabled: true
port: 9022
portName: metrics
path: /metrics
serviceMonitor:
# Enables ServiceMonitor creation for the Prometheus Operator
enabled: true
podMonitor:
# Enables PodMonitor creation for the Prometheus Operator
enabled: true
operator:
enabled: true
port: 8080
serviceMonitor:
# Enables ServiceMonitor creation for the Prometheus Operator
enabled: true
podMonitor:
# Enables PodMonitor creation for the Prometheus Operator
enabled: true
webhooks:
enabled: true
port: 8020
serviceMonitor:
# Enables ServiceMonitor creation for the Prometheus webhooks
enabled: true
EOT
(2) Helm ๋ฆฌํฌ์งํ ๋ฆฌ ์ถ๊ฐ
1
2
3
helm repo add kedacore https://kedacore.github.io/charts
# ๊ฒฐ๊ณผ
"kedacore" has been added to your repositories
(3) Helm ๋ฆฌํฌ์งํ ๋ฆฌ ์ ๋ฐ์ดํธ
1
2
3
4
5
6
7
8
9
helm repo update
# ๊ฒฐ๊ณผ
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kedacore" chart repository
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "geek-cookbook" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. โHappy Helming!โ
(4) KEDA Helm ์ค์น
1
helm install keda kedacore/keda --version 2.16.0 --namespace keda --create-namespace -f keda-values.yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
NAME: keda
LAST DEPLOYED: Fri Mar 7 22:03:19 2025
NAMESPACE: keda
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
:::^. .::::^: ::::::::::::::: .:::::::::. .^.
7???~ .^7????~. 7??????????????. :?????????77!^. .7?7.
7???~ ^7???7~. ~!!!!!!!!!!!!!!. :????!!!!7????7~. .7???7.
7???~^7????~. :????: :~7???7. :7?????7.
7???7????!. ::::::::::::. :????: .7???! :7??77???7.
7????????7: 7???????????~ :????: :????: :???7?5????7.
7????!~????^ !77777777777^ :????: :????: ^???7?#P7????7.
7???~ ^????~ :????: :7???! ^???7J#@J7?????7.
7???~ :7???!. :????: .:~7???!. ~???7Y&@#7777????7.
7???~ .7???7: !!!!!!!!!!!!!!! :????7!!77????7^ ~??775@@@GJJYJ?????7.
7???~ .!????^ 7?????????????7. :?????????7!~: !????G@@@@@@@@5??????7:
::::. ::::: ::::::::::::::: .::::::::.. .::::JGGGB@@@&7:::::::::
?@@#~
P@B^
:&G:
!5.
.Kubernetes Event-driven Autoscaling (KEDA) - Application autoscaling made simple.
Get started by deploying Scaled Objects to your cluster:
- Information about Scaled Objects : https://keda.sh/docs/latest/concepts/
- Samples: https://github.com/kedacore/samples
Get information about the deployed ScaledObjects:
kubectl get scaledobject [--namespace <namespace>]
Get details about a deployed ScaledObject:
kubectl describe scaledobject <scaled-object-name> [--namespace <namespace>]
Get information about the deployed ScaledObjects:
kubectl get triggerauthentication [--namespace <namespace>]
Get details about a deployed ScaledObject:
kubectl describe triggerauthentication <trigger-authentication-name> [--namespace <namespace>]
Get an overview of the Horizontal Pod Autoscalers (HPA) that KEDA is using behind the scenes:
kubectl get hpa [--all-namespaces] [--namespace <namespace>]
Learn more about KEDA:
- Documentation: https://keda.sh/
- Support: https://keda.sh/support/
- File an issue: https://github.com/kedacore/keda/issues/new/choose
4. KEDA ์ค์น ํ์ธ
(1) KEDA ๊ทธ๋ผํ๋ ํ์ธ
(2) KEDA CRD ์กฐํ
1
kubectl get crd | grep keda
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
cloudeventsources.eventing.keda.sh 2025-03-07T13:03:21Z
clustercloudeventsources.eventing.keda.sh 2025-03-07T13:03:21Z
clustertriggerauthentications.keda.sh 2025-03-07T13:03:21Z
scaledjobs.keda.sh 2025-03-07T13:03:21Z
scaledobjects.keda.sh 2025-03-07T13:03:21Z
triggerauthentications.keda.sh 2025-03-07T13:03:21Z
(3) KEDA ๋ค์์คํ์ด์ค ๋ด ๋ชจ๋ ๋ฆฌ์์ค ์กฐํ
1
kubectl get all -n keda
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
NAME READY STATUS RESTARTS AGE
pod/keda-admission-webhooks-86cffccbf5-sms4j 1/1 Running 0 101s
pod/keda-operator-6bdffdc78-x2qsj 1/1 Running 1 (89s ago) 101s
pod/keda-operator-metrics-apiserver-74d844d769-s5b6j 1/1 Running 0 101s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/keda-admission-webhooks ClusterIP 10.100.157.132 <none> 443/TCP,8020/TCP 101s
service/keda-operator ClusterIP 10.100.75.203 <none> 9666/TCP,8080/TCP 101s
service/keda-operator-metrics-apiserver ClusterIP 10.100.94.172 <none> 443/TCP,9022/TCP 101s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/keda-admission-webhooks 1/1 1 1 101s
deployment.apps/keda-operator 1/1 1 1 101s
deployment.apps/keda-operator-metrics-apiserver 1/1 1 1 101s
NAME DESIRED CURRENT READY AGE
replicaset.apps/keda-admission-webhooks-86cffccbf5 1 1 1 101s
replicaset.apps/keda-operator-6bdffdc78 1 1 1 101s
replicaset.apps/keda-operator-metrics-apiserver-74d844d769 1 1 1 101s
(4) KEDA Admission Webhook ๊ตฌ์ฑ ์กฐํ
1
kubectl get validatingwebhookconfigurations keda-admission -o yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
meta.helm.sh/release-name: keda
meta.helm.sh/release-namespace: keda
creationTimestamp: "2025-03-07T13:03:22Z"
generation: 2
labels:
app.kubernetes.io/component: operator
app.kubernetes.io/instance: keda
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keda-admission-webhooks
app.kubernetes.io/part-of: keda-operator
app.kubernetes.io/version: 2.16.0
helm.sh/chart: keda-2.16.0
name: keda-admission
resourceVersion: "165834"
uid: 30d6be12-b069-4922-8081-91a2a9b13e56
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFKSE1RMHdDd1lEVlFRREV3UkxSVVJCTUI0WERUSTFNRE13TnpFeU1ETXpNMW9YRFRNMU1ETXdOVEV6TURNegpNMW93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjA2V2N0SUx6U1lWMFovZ1IvSkhMQ29kM0VKQW4wSzlJS0QKeXpDZ3Z2bFhyRnJyc1d5ajRVazFQZCs2eVdQSTlFME1TdjFGSXNtRitYZkEyaE0rcmlHVmMyNjZjWFExakVXVApNdXpuVXExU2t5RHZNd1cwS2E4NDFuWklxWGZrOW95NkVOSnhRWHdjVkNTb2JMZTVObk5XUktFWHZNR3VwUm84CkZ4SVcxV1d2UkhSZStLYTVGb2hTU1l0Y1lORXMwV2ZSaVA0NEZGVGtydkE5UG1uQnovcmY5SHBCRis2TTZDczIKQVlWT1ZkTitBZldyTXJ1UTNxUE84OXowY3l1L2lSaHIxNVpBcWtNLy8vaDRWVmtZWUNublVFOWRZaEdSVXJZZQp4MW95VFJ2R2x0OVo3MnJkV0F3a3ZaSVJFWStERFlHbUY4Nk8ybEhsUTd3M3VwTzZBVU1DQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkhhd05EK1cKWkw4djYrMi8zc3J2QXdkSFNNUC9NQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUZDRHFEUVZJNElRQUFJQmZNYlBqK1pOMm5NbkprQVNCU2VQeEdwVmZQcjU4MWZtbW9yTFl2dkpsa2lkClM3WnRoSGd3aHJsMWY5ZVc4ZVBHMHo4RVEvclBGWTFFdXE0dDRHaHBFRjNqUXBvdGNMMDRrVlQ5eUhTcWIvV2UKelVFSlVlVnJJYWlud3hzZTJ5bjREcHQvWVNVMG9WM1QrUE5Qc1pZbS9PNENOb1pMRTNaU1lxNHU0U0lKZFArYgpCdm1iMkU2T0lYS1Q5N0xvTytYRDY0WHRkVnZNa25BVGhLV2VwZENjd2tCcEJwajczdFMweDR2dXNNRDVDL09nClF4eEJTbWRtR3FvTVVZT0VuMGhWUHFGTi96YWZjekNzNG51Y1EvSlZFU1VXaDVFZ0xoNXFtcXlvSHp2a2JlRWQKKzlHeDgzRTV4M0dnUWdFTnhZUGFQcWhMT0xZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
service:
name: keda-admission-webhooks
namespace: keda
path: /validate-keda-sh-v1alpha1-scaledobject
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: vscaledobject.kb.io
namespaceSelector: {}
objectSelector: {}
rules:
- apiGroups:
- keda.sh
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- scaledobjects
scope: '*'
sideEffects: None
timeoutSeconds: 10
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFKSE1RMHdDd1lEVlFRREV3UkxSVVJCTUI0WERUSTFNRE13TnpFeU1ETXpNMW9YRFRNMU1ETXdOVEV6TURNegpNMW93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjA2V2N0SUx6U1lWMFovZ1IvSkhMQ29kM0VKQW4wSzlJS0QKeXpDZ3Z2bFhyRnJyc1d5ajRVazFQZCs2eVdQSTlFME1TdjFGSXNtRitYZkEyaE0rcmlHVmMyNjZjWFExakVXVApNdXpuVXExU2t5RHZNd1cwS2E4NDFuWklxWGZrOW95NkVOSnhRWHdjVkNTb2JMZTVObk5XUktFWHZNR3VwUm84CkZ4SVcxV1d2UkhSZStLYTVGb2hTU1l0Y1lORXMwV2ZSaVA0NEZGVGtydkE5UG1uQnovcmY5SHBCRis2TTZDczIKQVlWT1ZkTitBZldyTXJ1UTNxUE84OXowY3l1L2lSaHIxNVpBcWtNLy8vaDRWVmtZWUNublVFOWRZaEdSVXJZZQp4MW95VFJ2R2x0OVo3MnJkV0F3a3ZaSVJFWStERFlHbUY4Nk8ybEhsUTd3M3VwTzZBVU1DQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkhhd05EK1cKWkw4djYrMi8zc3J2QXdkSFNNUC9NQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUZDRHFEUVZJNElRQUFJQmZNYlBqK1pOMm5NbkprQVNCU2VQeEdwVmZQcjU4MWZtbW9yTFl2dkpsa2lkClM3WnRoSGd3aHJsMWY5ZVc4ZVBHMHo4RVEvclBGWTFFdXE0dDRHaHBFRjNqUXBvdGNMMDRrVlQ5eUhTcWIvV2UKelVFSlVlVnJJYWlud3hzZTJ5bjREcHQvWVNVMG9WM1QrUE5Qc1pZbS9PNENOb1pMRTNaU1lxNHU0U0lKZFArYgpCdm1iMkU2T0lYS1Q5N0xvTytYRDY0WHRkVnZNa25BVGhLV2VwZENjd2tCcEJwajczdFMweDR2dXNNRDVDL09nClF4eEJTbWRtR3FvTVVZT0VuMGhWUHFGTi96YWZjekNzNG51Y1EvSlZFU1VXaDVFZ0xoNXFtcXlvSHp2a2JlRWQKKzlHeDgzRTV4M0dnUWdFTnhZUGFQcWhMT0xZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
service:
name: keda-admission-webhooks
namespace: keda
path: /validate-keda-sh-v1alpha1-triggerauthentication
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: vstriggerauthentication.kb.io
namespaceSelector: {}
objectSelector: {}
rules:
- apiGroups:
- keda.sh
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- triggerauthentications
scope: '*'
sideEffects: None
timeoutSeconds: 10
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFKSE1RMHdDd1lEVlFRREV3UkxSVVJCTUI0WERUSTFNRE13TnpFeU1ETXpNMW9YRFRNMU1ETXdOVEV6TURNegpNMW93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjA2V2N0SUx6U1lWMFovZ1IvSkhMQ29kM0VKQW4wSzlJS0QKeXpDZ3Z2bFhyRnJyc1d5ajRVazFQZCs2eVdQSTlFME1TdjFGSXNtRitYZkEyaE0rcmlHVmMyNjZjWFExakVXVApNdXpuVXExU2t5RHZNd1cwS2E4NDFuWklxWGZrOW95NkVOSnhRWHdjVkNTb2JMZTVObk5XUktFWHZNR3VwUm84CkZ4SVcxV1d2UkhSZStLYTVGb2hTU1l0Y1lORXMwV2ZSaVA0NEZGVGtydkE5UG1uQnovcmY5SHBCRis2TTZDczIKQVlWT1ZkTitBZldyTXJ1UTNxUE84OXowY3l1L2lSaHIxNVpBcWtNLy8vaDRWVmtZWUNublVFOWRZaEdSVXJZZQp4MW95VFJ2R2x0OVo3MnJkV0F3a3ZaSVJFWStERFlHbUY4Nk8ybEhsUTd3M3VwTzZBVU1DQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkhhd05EK1cKWkw4djYrMi8zc3J2QXdkSFNNUC9NQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUZDRHFEUVZJNElRQUFJQmZNYlBqK1pOMm5NbkprQVNCU2VQeEdwVmZQcjU4MWZtbW9yTFl2dkpsa2lkClM3WnRoSGd3aHJsMWY5ZVc4ZVBHMHo4RVEvclBGWTFFdXE0dDRHaHBFRjNqUXBvdGNMMDRrVlQ5eUhTcWIvV2UKelVFSlVlVnJJYWlud3hzZTJ5bjREcHQvWVNVMG9WM1QrUE5Qc1pZbS9PNENOb1pMRTNaU1lxNHU0U0lKZFArYgpCdm1iMkU2T0lYS1Q5N0xvTytYRDY0WHRkVnZNa25BVGhLV2VwZENjd2tCcEJwajczdFMweDR2dXNNRDVDL09nClF4eEJTbWRtR3FvTVVZT0VuMGhWUHFGTi96YWZjekNzNG51Y1EvSlZFU1VXaDVFZ0xoNXFtcXlvSHp2a2JlRWQKKzlHeDgzRTV4M0dnUWdFTnhZUGFQcWhMT0xZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
service:
name: keda-admission-webhooks
namespace: keda
path: /validate-keda-sh-v1alpha1-clustertriggerauthentication
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: vsclustertriggerauthentication.kb.io
namespaceSelector: {}
objectSelector: {}
rules:
- apiGroups:
- keda.sh
apiVersions:
- v1alpha1
operations:
- CREATE
- UPDATE
resources:
- clustertriggerauthentications
scope: '*'
sideEffects: None
timeoutSeconds: 10
(5) KEDA ๋ชจ๋ํฐ๋ง ๋ฆฌ์์ค ์กฐํ (PodMonitor, ServiceMonitor)
1
kubectl get podmonitor,servicemonitors -n keda
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
NAME AGE
podmonitor.monitoring.coreos.com/keda-operator 3m18s
podmonitor.monitoring.coreos.com/keda-operator-metrics-apiserver 3m18s
NAME AGE
servicemonitor.monitoring.coreos.com/keda-admission-webhooks 3m18s
servicemonitor.monitoring.coreos.com/keda-operator 3m18s
servicemonitor.monitoring.coreos.com/keda-operator-metrics-apiserver 3m18s
(6) External Metrics API ์๋น์ค ๊ตฌ์ฑ ์กฐํ
1
kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
meta.helm.sh/release-name: keda
meta.helm.sh/release-namespace: keda
creationTimestamp: "2025-03-07T13:03:22Z"
labels:
app.kubernetes.io/component: operator
app.kubernetes.io/instance: keda
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
app.kubernetes.io/part-of: keda-operator
app.kubernetes.io/version: 2.16.0
helm.sh/chart: keda-2.16.0
name: v1beta1.external.metrics.k8s.io
resourceVersion: "165835"
uid: e54891d8-04a1-4dc8-b29b-7b81c620f60c
spec:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFKSE1RMHdDd1lEVlFRREV3UkxSVVJCTUI0WERUSTFNRE13TnpFeU1ETXpNMW9YRFRNMU1ETXdOVEV6TURNegpNMW93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjA2V2N0SUx6U1lWMFovZ1IvSkhMQ29kM0VKQW4wSzlJS0QKeXpDZ3Z2bFhyRnJyc1d5ajRVazFQZCs2eVdQSTlFME1TdjFGSXNtRitYZkEyaE0rcmlHVmMyNjZjWFExakVXVApNdXpuVXExU2t5RHZNd1cwS2E4NDFuWklxWGZrOW95NkVOSnhRWHdjVkNTb2JMZTVObk5XUktFWHZNR3VwUm84CkZ4SVcxV1d2UkhSZStLYTVGb2hTU1l0Y1lORXMwV2ZSaVA0NEZGVGtydkE5UG1uQnovcmY5SHBCRis2TTZDczIKQVlWT1ZkTitBZldyTXJ1UTNxUE84OXowY3l1L2lSaHIxNVpBcWtNLy8vaDRWVmtZWUNublVFOWRZaEdSVXJZZQp4MW95VFJ2R2x0OVo3MnJkV0F3a3ZaSVJFWStERFlHbUY4Nk8ybEhsUTd3M3VwTzZBVU1DQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkhhd05EK1cKWkw4djYrMi8zc3J2QXdkSFNNUC9NQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUZDRHFEUVZJNElRQUFJQmZNYlBqK1pOMm5NbkprQVNCU2VQeEdwVmZQcjU4MWZtbW9yTFl2dkpsa2lkClM3WnRoSGd3aHJsMWY5ZVc4ZVBHMHo4RVEvclBGWTFFdXE0dDRHaHBFRjNqUXBvdGNMMDRrVlQ5eUhTcWIvV2UKelVFSlVlVnJJYWlud3hzZTJ5bjREcHQvWVNVMG9WM1QrUE5Qc1pZbS9PNENOb1pMRTNaU1lxNHU0U0lKZFArYgpCdm1iMkU2T0lYS1Q5N0xvTytYRDY0WHRkVnZNa25BVGhLV2VwZENjd2tCcEJwajczdFMweDR2dXNNRDVDL09nClF4eEJTbWRtR3FvTVVZT0VuMGhWUHFGTi96YWZjekNzNG51Y1EvSlZFU1VXaDVFZ0xoNXFtcXlvSHp2a2JlRWQKKzlHeDgzRTV4M0dnUWdFTnhZUGFQcWhMT0xZPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
group: external.metrics.k8s.io
groupPriorityMinimum: 100
service:
name: keda-operator-metrics-apiserver
namespace: keda
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2025-03-07T13:03:51Z"
message: all checks passed
reason: Passed
status: "True"
type: Available
(7) External Metrics API ์์ธ ์ ๋ณด ์กฐํ
1
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "externalmetrics",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
5. php-apache ๋ฐฐํฌ (KEDA ๋ค์์คํ์ด์ค)
1
2
3
4
kubectl apply -f php-apache.yaml -n keda
# ๊ฒฐ๊ณผ
deployment.apps/php-apache created
service/php-apache created
- ์์ ์ค์ตํ๋ php-apache ์ ํ๋ฆฌ์ผ์ด์ ์ KEDA ๋ค์์คํ์ด์ค๋ก ๋ฐฐํฌํจ
6. ๋ฐฐํฌ ์ํ ํ์ธ
1
kubectl get pod -n keda
โ ย ์ถ๋ ฅ
1
2
3
4
5
NAME READY STATUS RESTARTS AGE
keda-admission-webhooks-86cffccbf5-sms4j 1/1 Running 0 8m14s
keda-operator-6bdffdc78-x2qsj 1/1 Running 1 (8m2s ago) 8m14s
keda-operator-metrics-apiserver-74d844d769-s5b6j 1/1 Running 0 8m14s
php-apache-d87b7ff46-qk6p7 0/1 ContainerCreating 0 1s
- KEDA ๋ค์์คํ์ด์ค ๋ด์์ Pod ์ํ๋ฅผ ์กฐํํจ
7. KEDA Cron Scaler ์ค์
- php-apache ์ ํ๋ฆฌ์ผ์ด์
์ ์ค์ผ์ผ๋ง์ ์ํด cron ํธ๋ฆฌ๊ฑฐ ๊ธฐ๋ฐ์
ScaledObject๋ฅผ ์์ฑํจ - ์๋ YAML ํ์ผ์ ์์ฑํ์ฌ KEDA Cron Scaler ์ค์ ์ ์ ์ฉํจ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat <<EOT > keda-cron.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: php-apache-cron-scaled
spec:
minReplicaCount: 0
maxReplicaCount: 2 # Specifies the maximum number of replicas to scale up to (defaults to 100).
pollingInterval: 30 # Specifies how often KEDA should check for scaling events
cooldownPeriod: 300 # Specifies the cool-down period in seconds after a scaling event
scaleTargetRef: # Identifies the Kubernetes deployment or other resource that should be scaled.
apiVersion: apps/v1
kind: Deployment
name: php-apache
triggers: # Defines the specific configuration for your chosen scaler, including any required parameters or settings
- type: cron
metadata:
timezone: Asia/Seoul
start: 00,15,30,45 * * * *
end: 05,20,35,50 * * * *
desiredReplicas: "1"
EOT
kubectl apply -f keda-cron.yaml -n keda
# ๊ฒฐ๊ณผ
scaledobject.keda.sh/php-apache-cron-scaled created
8. ์ค์ผ์ผ๋ง ๋ฐ ์ํ ๋ชจ๋ํฐ๋ง
1
watch -d 'kubectl get ScaledObject,hpa,pod -n keda'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Every 2.0s: kubectl get ScaledObject,hpa,pod -n keda gram88: 10:15:50 PM in 0.780s (0)
NAME SCALETARGETKIND SCALE TARGETNAME MIN MAX READY ACTIVE FALLBACK PAUSED TRIGGERS AUTHENTICATIONS AGE
scaledobject.keda.sh/php-apache-cron-scaled apps/v1.Deployment php-a pache 0 2 True True False Unknown
2m38s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/keda-hpa-php-apache-cron-scaled Deployment/php-apache <unknown>/1 (avg) 1 2 0 2m38s
NAME READY STATUS RESTARTS AGE
pod/keda-admission-webhooks-86cffccbf5-sms4j 1/1 Running 0 12m
pod/keda-operator-6bdffdc78-x2qsj 1/1 Running 1 (12m ago) 12m
pod/keda-operator-metrics-apiserver-74d844d769-s5b6j 1/1 Running 0 12m
pod/php-apache-d87b7ff46-z7lhl 1/1 Running 0 39s
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE FALLBACK PAUSED TRIGGERS AUTHENTICATIONS AGE
scaledobject.keda.sh/php-apache-cron-scaled apps/v1.Deployment php-apache 0 2 True True False Unknown 5m54s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/keda-hpa-php-apache-cron-scaled Deployment/php-apache <unknown>/1 (avg) 1 2 1 5m54s
NAME READY STATUS RESTARTS AGE
pod/keda-admission-webhooks-86cffccbf5-sms4j 1/1 Running 0 15m
pod/keda-operator-6bdffdc78-x2qsj 1/1 Running 1 (15m ago) 15m
pod/keda-operator-metrics-apiserver-74d844d769-s5b6j 1/1 Running 0 15m
pod/php-apache-d87b7ff46-z7lhl 1/1 Running 0 3m54s
9. HPA ์์ธ ์ค์ ํ์ธ
1
kubectl get hpa -o jsonpath="{.items[0].spec}" -n keda | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
{
"maxReplicas": 2,
"metrics": [
{
"external": {
"metric": {
"name": "s0-cron-Asia-Seoul-00,15,30,45xxxx-05,20,35,50xxxx",
"selector": {
"matchLabels": {
"scaledobject.keda.sh/name": "php-apache-cron-scaled"
}
}
},
"target": {
"averageValue": "1",
"type": "AverageValue"
}
},
"type": "External"
}
],
"minReplicas": 1,
"scaleTargetRef": {
"apiVersion": "apps/v1",
"kind": "Deployment",
"name": "php-apache"
}
}
10. KEDA ๋ฐ deployment ๋ฑ ์ญ์
(1) ํ ์คํธ ๋ฆฌ์์ค ์ญ์
1
2
3
4
5
6
kubectl delete ScaledObject -n keda php-apache-cron-scaled && kubectl delete deploy php-apache -n keda && helm uninstall keda -n keda
# ๊ฒฐ๊ณผ
scaledobject.keda.sh "php-apache-cron-scaled" deleted
deployment.apps "php-apache" deleted
release "keda" uninstalled
(2) ๋ค์์คํ์ด์ค ์ญ์
1
2
3
4
kubectl delete namespace keda
# ๊ฒฐ๊ณผ
namespace "keda" deleted
โฌ๏ธ VPA - Vertical Pod Autoscaler
1. ๊ทธ๋ผํ๋ ๋์๋ณด๋ Import ์ค์
2. ์ด์์๋ฒ EC2์์ VPA ํด๋ก ๋ฐ ๋๋ ํ ๋ฆฌ ๊ตฌ์กฐ ํ์ธ
1
2
3
(eks-user@myeks:default) [root@operator-host ~]# git clone https://github.com/kubernetes/autoscaler.git
(eks-user@myeks:default) [root@operator-host ~]# cd ~/autoscaler/vertical-pod-autoscaler/
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# tree hack
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
hack
โโโ api-docs
โย ย โโโ config.yaml
โโโ boilerplate.go.txt
โโโ convert-alpha-objects.sh
โโโ deploy-for-e2e-locally.sh
โโโ deploy-for-e2e.sh
โโโ dev-deploy-locally.sh
โโโ e2e
โย ย โโโ Dockerfile.externalmetrics-writer
โย ย โโโ k8s-metrics-server.yaml
โย ย โโโ metrics-pump.yaml
โย ย โโโ prometheus-adapter.yaml
โย ย โโโ prometheus.yaml
โย ย โโโ recommender-externalmetrics-deployment.yaml
โย ย โโโ vpa-rbac.diff
โโโ emit-metrics.py
โโโ generate-api-docs.sh
โโโ generate-crd-yaml.sh
โโโ generate-flags.sh
โโโ lib
โย ย โโโ util.sh
โโโ local-cluster.md
โโโ run-e2e-locally.sh
โโโ run-e2e.sh
โโโ run-e2e-tests.sh
โโโ tools.go
โโโ update-codegen.sh
โโโ update-kubernetes-deps-in-e2e.sh
โโโ update-kubernetes-deps.sh
โโโ verify-codegen.sh
โโโ verify-vpa-flags.sh
โโโ vpa-apply-upgrade.sh
โโโ vpa-down.sh
โโโ vpa-process-yaml.sh
โโโ vpa-process-yamls.sh
โโโ vpa-up.sh
โโโ warn-obsolete-vpa-objects.sh
3 directories, 34 files
3. OpenSSL ๋ฒ์ ํ์ธ ๋ฐ ์ ๊ทธ๋ ์ด๋ ์์
(1) ํ์ฌ OpenSSL ๋ฒ์ ํ์ธ
1
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# openssl version
โ ย ์ถ๋ ฅ
1
OpenSSL 1.0.2k-fips 26 Jan 2017
(2) ๊ธฐ์กด OpenSSL ์ ๊ฑฐ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# yum remove openssl -y
# ๊ฒฐ๊ณผ
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package openssl.x86_64 1:1.0.2k-24.amzn2.0.14 will be erased
--> Processing Dependency: openssl for package: python2-cryptography-1.7.2-2.amzn2.x86_64
--> Processing Dependency: openssl for package: ec2-instance-connect-1.1-19.amzn2.noarch
--> Processing Dependency: openssl for package: rng-tools-6.8-3.amzn2.0.5.x86_64
--> Running transaction check
---> Package ec2-instance-connect.noarch 0:1.1-19.amzn2 will be erased
---> Package python2-cryptography.x86_64 0:1.7.2-2.amzn2 will be erased
--> Processing Dependency: python-cryptography >= 1.5 for package: python-jwcrypto-0.4.2-1.amzn2.0.2.noarch
--> Processing Dependency: python2-cryptography >= 0.8.1 for package: python2-oauthlib-2.0.1-8.amzn2.0.1.noarch
---> Package rng-tools.x86_64 0:6.8-3.amzn2.0.5 will be erased
--> Running transaction check
---> Package python-jwcrypto.noarch 0:0.4.2-1.amzn2.0.2 will be erased
---> Package python2-oauthlib.noarch 0:2.0.1-8.amzn2.0.1 will be erased
--> Processing Dependency: python-oauthlib for package: cloud-init-19.3-46.amzn2.0.5.noarch
--> Running transaction check
---> Package cloud-init.noarch 0:19.3-46.amzn2.0.5 will be erased
--> Processing Dependency: /usr/bin/openssl for package: authconfig-6.2.8-30.amzn2.0.2.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package authconfig.x86_64 0:6.2.8-30.amzn2.0.2 will be erased
--> Finished Dependency Resolution
amzn2-core/2/x86_64 | 3.6 kB 00:00
amzn2extra-docker/2/x86_64 | 2.9 kB 00:00
Dependencies Resolved
========================================================================
Package Arch Version Repository Size
========================================================================
Removing:
openssl x86_64 1:1.0.2k-24.amzn2.0.14 installed 830 k
Removing for dependencies:
authconfig x86_64 6.2.8-30.amzn2.0.2 installed 2.2 M
cloud-init noarch 19.3-46.amzn2.0.5 installed 3.2 M
ec2-instance-connect noarch 1.1-19.amzn2 installed 23 k
python-jwcrypto noarch 0.4.2-1.amzn2.0.2 installed 250 k
python2-cryptography x86_64 1.7.2-2.amzn2 installed 2.6 M
python2-oauthlib noarch 2.0.1-8.amzn2.0.1 installed 780 k
rng-tools x86_64 6.8-3.amzn2.0.5 installed 102 k
Transaction Summary
========================================================================
Remove 1 Package (+7 Dependent packages)
Installed size: 10 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : cloud-init-19.3-46.amzn2.0.5.noarch 1/8
Erasing : python2-oauthlib-2.0.1-8.amzn2.0.1.noarch 2/8
Erasing : python-jwcrypto-0.4.2-1.amzn2.0.2.noarch 3/8
Erasing : ec2-instance-connect-1.1-19.amzn2.noarch 4/8
Erasing : python2-cryptography-1.7.2-2.amzn2.x86_64 5/8
Erasing : rng-tools-6.8-3.amzn2.0.5.x86_64 6/8
Erasing : authconfig-6.2.8-30.amzn2.0.2.x86_64 7/8
Erasing : 1:openssl-1.0.2k-24.amzn2.0.14.x86_64 8/8
Verifying : authconfig-6.2.8-30.amzn2.0.2.x86_64 1/8
Verifying : 1:openssl-1.0.2k-24.amzn2.0.14.x86_64 2/8
Verifying : python2-cryptography-1.7.2-2.amzn2.x86_64 3/8
Verifying : ec2-instance-connect-1.1-19.amzn2.noarch 4/8
Verifying : python-jwcrypto-0.4.2-1.amzn2.0.2.noarch 5/8
Verifying : rng-tools-6.8-3.amzn2.0.5.x86_64 6/8
Verifying : python2-oauthlib-2.0.1-8.amzn2.0.1.noarch 7/8
Verifying : cloud-init-19.3-46.amzn2.0.5.noarch 8/8
Removed:
openssl.x86_64 1:1.0.2k-24.amzn2.0.14
Dependency Removed:
authconfig.x86_64 0:6.2.8-30.amzn2.0.2
cloud-init.noarch 0:19.3-46.amzn2.0.5
ec2-instance-connect.noarch 0:1.1-19.amzn2
python-jwcrypto.noarch 0:0.4.2-1.amzn2.0.2
python2-cryptography.x86_64 0:1.7.2-2.amzn2
python2-oauthlib.noarch 0:2.0.1-8.amzn2.0.1
rng-tools.x86_64 0:6.8-3.amzn2.0.5
Complete!
(3) OpenSSL 1.1.1 ์ค์น
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# yum install openssl11 -y
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package openssl11.x86_64 1:1.1.1zb-1.amzn2.0.1 will be installed
--> Processing Dependency: openssl11-libs(x86-64) = 1:1.1.1zb-1.amzn2.0.1 for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libcrypto.so.1.1()(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Processing Dependency: libssl.so.1.1()(64bit) for package: 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64
--> Running transaction check
---> Package openssl11-libs.x86_64 1:1.1.1zb-1.amzn2.0.1 will be installed
--> Processing Dependency: openssl11-pkcs11 for package: 1:openssl11-libs-1.1.1zb-1.amzn2.0.1.x86_64
--> Running transaction check
---> Package openssl11-pkcs11.x86_64 0:0.4.10-6.amzn2.0.1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================
Package Arch Version Repository Size
========================================================================
Installing:
openssl11 x86_64 1:1.1.1zb-1.amzn2.0.1 amzn2-core 321 k
Installing for dependencies:
openssl11-libs x86_64 1:1.1.1zb-1.amzn2.0.1 amzn2-core 1.4 M
openssl11-pkcs11 x86_64 0.4.10-6.amzn2.0.1 amzn2-core 61 k
Transaction Summary
========================================================================
Install 1 Package (+2 Dependent packages)
Total download size: 1.8 M
Installed size: 4.3 M
Downloading packages:
(1/3): openssl11-libs-1.1.1zb-1.amzn2.0.1.x86_64.r | 1.4 MB 00:00
(2/3): openssl11-pkcs11-0.4.10-6.amzn2.0.1.x86_64. | 61 kB 00:00
(3/3): openssl11-1.1.1zb-1.amzn2.0.1.x86_64.rpm | 321 kB 00:00
------------------------------------------------------------------------
Total 8.0 MB/s | 1.8 MB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:openssl11-libs-1.1.1zb-1.amzn2.0.1.x86_64 1/3
Installing : openssl11-pkcs11-0.4.10-6.amzn2.0.1.x86_64 2/3
Installing : 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64 3/3
Verifying : openssl11-pkcs11-0.4.10-6.amzn2.0.1.x86_64 1/3
Verifying : 1:openssl11-1.1.1zb-1.amzn2.0.1.x86_64 2/3
Verifying : 1:openssl11-libs-1.1.1zb-1.amzn2.0.1.x86_64 3/3
Installed:
openssl11.x86_64 1:1.1.1zb-1.amzn2.0.1
Dependency Installed:
openssl11-libs.x86_64 1:1.1.1zb-1.amzn2.0.1
openssl11-pkcs11.x86_64 0:0.4.10-6.amzn2.0.1
Complete!
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# openssl11 version
# โ
์ถ๋ ฅ
OpenSSL 1.1.1zb 11 Feb 2025
(4) ์คํฌ๋ฆฝํธ ๋ด OpenSSL ๋ช ๋ น์ด ์์ ๋ฐ ์ปค๋ฐ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# sed -i 's/openssl/openssl11/g' ~/autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# git status
Refresh index: 100% (7921/7921), done.
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: pkg/admission-controller/gencerts.sh
no changes added to commit (use "git add" and/or "git commit -a")
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# git config --global user.email "you@example.com"
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# git config --global user.name "Your Name"
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# git add .
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# git commit -m "openssl version modify"
[master 93371700c] openssl version modify
1 file changed, 11 insertions(+), 11 deletions(-)
4. VPA ์ ์คํฌ๋ฆฝํธ ์คํ ํ ์คํธ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# ./hack/vpa-up.sh
# ๊ฒฐ๊ณผ
HEAD is now at 200a292f8 Merge pull request #7774 from jm-franc/vpa-finalize-branch
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding created
serviceaccount/vpa-admission-controller created
serviceaccount/vpa-recommender created
serviceaccount/vpa-updater created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
deployment.apps/vpa-updater created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
./hack/../hack/../pkg/admission-controller/gencerts.sh: line 42: openssl: command not found
deployment.apps/vpa-admission-controller created
service/vpa-webhook created
- ์คํฌ๋ฆฝํธ ์์ ํ ์ฌ์คํ
- ์ด๊ธฐ ์คํ ์
openssl: command not found์ค๋ฅ๊ฐ ๋ฐ์ํ์ผ๋, ์คํฌ๋ฆฝํธ ์์ ํ ์ฌ์คํํ์ฌ ์ธ์ฆ์ ์์ฑ ๋ฐ ๊ด๋ จ ๋ฆฌ์์ค ์์ฑ ์ฑ๊ณต
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# sed -i 's/openssl/openssl11/g' ~/autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# ./hack/vpa-up.sh
# ๊ฒฐ๊ณผ
M vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh
HEAD is now at 200a292f8 Merge pull request #7774 from jm-franc/vpa-finalize-branch
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:evictioner unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding unchanged
serviceaccount/vpa-admission-controller unchanged
serviceaccount/vpa-recommender unchanged
serviceaccount/vpa-updater unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding unchanged
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater unchanged
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater unchanged
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender unchanged
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender unchanged
deployment.apps/vpa-updater unchanged
deployment.apps/vpa-recommender unchanged
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
..................+++++
........................................................+++++
e is 65537 (0x010001)
Can't load /root/.rnd into RNG
140227333396288:error:2406F079:random number generator:RAND_load_file:Cannot open file:crypto/rand/randfile.c:98:Filename=/root/.rnd
Generating RSA private key, 2048 bit long modulus (2 primes)
.....+++++
..+++++
e is 65537 (0x010001)
Signature ok
subject=CN = vpa-webhook.kube-system.svc
Getting CA Private Key
Uploading certs to the cluster.
secret/vpa-tls-certs created
Deleting /tmp/vpa-certs.
deployment.apps/vpa-admission-controller unchanged
service/vpa-webhook unchanged
5. VPA ๊ด๋ จ CRD ์กฐํ
1
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# kubectl get crd | grep autoscaling
โ ย ์ถ๋ ฅ
1
2
verticalpodautoscalercheckpoints.autoscaling.k8s.io 2025-03-07T15:40:06Z
verticalpodautoscalers.autoscaling.k8s.io 2025-03-07T15:40:06Z
6. VPA ์นํํฌ ๊ตฌ์ฑ ์กฐํ
1
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# kubectl get mutatingwebhookconfigurations vpa-webhook-config
โ ย ์ถ๋ ฅ
1
2
NAME WEBHOOKS AGE
vpa-webhook-config 1 30s
1
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# kubectl get mutatingwebhookconfigurations vpa-webhook-config -o json | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
{
"apiVersion": "admissionregistration.k8s.io/v1",
"kind": "MutatingWebhookConfiguration",
"metadata": {
"creationTimestamp": "2025-03-07T15:41:33Z",
"generation": 1,
"name": "vpa-webhook-config",
"resourceVersion": "210727",
"uid": "07860446-72c4-4019-bcdb-83d8022f42d8"
},
"webhooks": [
{
"admissionReviewVersions": [
"v1"
],
"clientConfig": {
"caBundle": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMVENDQWhXZ0F3SUJBZ0lVTzk2L0tmeTNpSERLNFUxVTlsS2ZOWExmUk9Zd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09kbkJoWDNkbFltaHZiMnRmWTJFd0lCY05NalV3TXpBM01UVTBNVEExV2hnUApNakk1T0RFeU1qRXhOVFF4TURWYU1Ca3hGekFWQmdOVkJBTU1Eblp3WVY5M1pXSm9iMjlyWDJOaE1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQStwbHJKQ2d2eFRVQk95L2gyd0lEZVZKMnhaYnoKZzEyWTMxZGdpMTZ1cWIxVnFYMWpxazMydXc4cDlzYzZPa1RzcE5QamZmYmltdkdlOHByUWRtRHNkYVMxSEthUgpkSUJnWTBsVjJaemZ2a2FDUXN1VCtCMGRtWG1rYkdmSzlYOFpkVERlWmVCc3M0UmpFWThmVHdDYngzWHF2RkprCkVIeUpYZ3lodHlpZTVKZ0FxVnpQVXVReHoxdHpwemViMURpY3FuanpVSlRtR2tVdEcwSTBQVjlYcEtZeHgvdUEKYzcxOWg4ZlRIUjhqQUlQeG5yRWIzcG0ramg3SXJLWUU0VlJtenJyOXFFa3B3U3lzTng0L29WSDFVZ0FsSjU0bwpHcDV3a2dXdG4zanFLcU9rcm8wVVY5UTVFaGxMK2NIMmtqODN3NFlWVjFEOXJNZldaWW1VSndvWTN3SURBUUFCCm8yc3dhVEFkQmdOVkhRNEVGZ1FVZmowdzZnbjBSUUlXcmFNTTRFU1MrdGVKcUhJd0h3WURWUjBqQkJnd0ZvQVUKZmowdzZnbjBSUUlXcmFNTTRFU1MrdGVKcUhJd0RBWURWUjBUQkFVd0F3RUIvekFaQmdOVkhSRUVFakFRZ2c1MgpjR0ZmZDJWaWFHOXZhMTlqWVRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUFrYUpuV1NFT1BCNDhPZUErbVI4CkZyZThWeHhrK2pHZUV4cUt3Q0lNREtLSldLTmlKYWhnUUVzTmo2Zmhvc01uY205Mys4NDRSWURHZk9DNUoyYnAKSlpQdnlhUVpVZll0QVRGYUwzVE82UHNDeVV5V1BVN3BLdzV0K1hVbkVUTHhESUVYcllIYUxJVkxPcUpwT0hRSAovS0RYbUxyazNEYzhicTZWdCtqbXYyOC95SGJZdE5UbDNJTTZVQWpxcTVvWXJtQVFEZ1dQRUx1aWZ3QTEzV3BMCjJwalFxU1NuRktCV3FGVnQ1b2ZXUHJhNkpDcVB1TnMvd21IRGVnK1dZNjhQU0xkNDhYeHpVNkNWNlRLWUpjcjMKUkwxWUU0Yks4MmgzWDYweWJub0pzSHM2VVYreDNzRXEwcjdPRnQwcDNYeFFrUmF1VnNTM21CalIvbnBPd3IzWQo3QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
"service": {
"name": "vpa-webhook",
"namespace": "kube-system",
"port": 443
}
},
"failurePolicy": "Ignore",
"matchPolicy": "Equivalent",
"name": "vpa.k8s.io",
"namespaceSelector": {
"matchExpressions": [
{
"key": "kubernetes.io/metadata.name",
"operator": "NotIn",
"values": [
""
]
}
]
},
"objectSelector": {},
"reinvocationPolicy": "Never",
"rules": [
{
"apiGroups": [
""
],
"apiVersions": [
"v1"
],
"operations": [
"CREATE"
],
"resources": [
"pods"
],
"scope": "*"
},
{
"apiGroups": [
"autoscaling.k8s.io"
],
"apiVersions": [
"*"
],
"operations": [
"CREATE",
"UPDATE"
],
"resources": [
"verticalpodautoscalers"
],
"scope": "*"
}
],
"sideEffects": "None",
"timeoutSeconds": 30
}
]
}
7. ๋ชจ๋ํฐ๋ง ๋ช ๋ น์ด ์คํ
1
watch -d "kubectl top pod;echo "----------------------";kubectl describe pod | grep Requests: -A2"
โ ย ์ถ๋ ฅ
1
2
3
4
Every 2.0s: kubectl top pod;echo ------------------โฆ gram88: 12:45:35 AM in 1.434s (1)
No resources found in default namespace.
----------------------
No resources found in default namespace.
8. ๊ณต์ ์์ ๋ฐฐํฌ โ Hamster ์ ํ๋ฆฌ์ผ์ด์
(1) hamster.yaml ํ์ผ ํ์ธ
1
2
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# cd ~/autoscaler/vertical-pod-autoscaler/
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# cat examples/hamster.yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# This config creates a deployment with two pods, each requesting 100 millicores
# and trying to utilize slightly above 500 millicores (repeatedly using CPU for
# 0.5s and sleeping 0.5s).
# It also creates a corresponding Vertical Pod Autoscaler that adjusts the
# requests.
# Note that the update mode is left unset, so it defaults to "Auto" mode.
---
apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
name: hamster-vpa
spec:
# recommenders field can be unset when using the default recommender.
# When using an alternative recommender, the alternative recommender's name
# can be specified as the following in a list.
# recommenders:
# - name: 'alternative'
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hamster
resourcePolicy:
containerPolicies:
- containerName: '*'
minAllowed:
cpu: 100m
memory: 50Mi
maxAllowed:
cpu: 1
memory: 500Mi
controlledResources: ["cpu", "memory"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hamster
spec:
selector:
matchLabels:
app: hamster
replicas: 2
template:
metadata:
labels:
app: hamster
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534 # nobody
containers:
- name: hamster
image: registry.k8s.io/ubuntu-slim:0.14
resources:
requests:
cpu: 100m
memory: 50Mi
command: ["/bin/sh"]
args:
- "-c"
- "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"
- ํด๋น ํ์ผ์ ๋ ๊ฐ์ Pod๋ฅผ ์์ฑํ๋ฉฐ, ๊ฐ Pod์ CPU ์์ฒญ ๋ฐ ์ค์ ์ฌ์ฉ๋์ ๋ฐ๋ผ VPA๊ฐ ์๋ ์กฐ์ ๋๋๋ก ๊ตฌ์ฑ๋จ
(2) ์์ ๋ฐฐํฌ ๋ฐ VPA ์์ฑ
- VPA๋ โAutoโ ๋ชจ๋๋ก ์ค์ ๋์ด ์์ด, Pod์ ๋ฆฌ์์ค ์์ฒญ์ ๋ฐ๋ผ ์๋์ผ๋ก ์กฐ์ ๋จ
1
2
3
4
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# kubectl apply -f examples/hamster.yaml && kubectl get vpa -w
# ๊ฒฐ๊ณผ
verticalpodautoscaler.autoscaling.k8s.io/hamster-vpa created
deployment.apps/hamster created
1
2
3
4
5
6
NAME MODE CPU MEM PROVIDED AGE
hamster-vpa Auto 1s
hamster-vpa Auto 410m 262144k True 45s
hamster-vpa Auto 442m 262144k True 105s
hamster-vpa Auto 442m 262144k True 2m45s
...
- ๋ชจ๋ํฐ๋ง ๊ฒฐ๊ณผ, Pod์ CPU ์ฌ์ฉ๋์ด 410m ์ ๋๋ก ์ฆ๊ฐํ๋ฉด์ ์ถ๊ฐ Pod๊ฐ ์์ฑ๋จ
- ๋ํ, ๋ฉ๋ชจ๋ฆฌ ์ฌ์ฉ๋ ์์น์ ๋ฐ๋ผ ์ ๊ท Pod๊ฐ ์์๋๊ณ ๊ธฐ์กด Pod๊ฐ ์ข ๋ฃ๋๋ ๋ฑ VPA์ ์๋ ์ค์ผ์ผ๋ง ํจ๊ณผ๊ฐ ํ์ธ๋จ
9. ๊ทธ๋ผํ๋ ๋์๋ณด๋ ์กฐํ
10. ํ๋ก๋ฉํ ์ฐ์ค ๋ฉํธ๋ฆญ ์กฐํ
1
kube_customresource_vpa_containerrecommendations_target{resource="cpu"}
1
kube_customresource_vpa_containerrecommendations_target{resource="memory"}
11. ํ ์คํธ ์ข ๋ฃ ๋ฐ ๋ฆฌ์์ค ์ ๋ฆฌ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
(eks-user@myeks:default) [root@operator-host vertical-pod-autoscaler]# kubectl delete -f examples/hamster.yaml && cd ~/autoscaler/vertical-pod-autoscaler/ && ./hack/vpa-down.sh
# ๊ฒฐ๊ณผ
verticalpodautoscaler.autoscaling.k8s.io "hamster-vpa" deleted
deployment.apps "hamster" deleted
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalercheckpoints.autoscaling.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalers.autoscaling.k8s.io" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-status-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:evictioner" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-actor" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-status-actor" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-target-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-target-reader-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-evictioner-binding" deleted
serviceaccount "vpa-admission-controller" deleted
serviceaccount "vpa-recommender" deleted
serviceaccount "vpa-updater" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-admission-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-admission-controller" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-status-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-status-reader-binding" deleted
role.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" deleted
rolebinding.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" deleted
role.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" deleted
rolebinding.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" deleted
deployment.apps "vpa-updater" deleted
deployment.apps "vpa-recommender" deleted
Deleting VPA Admission Controller certs.
secret "vpa-tls-certs" deleted
Unregistering VPA admission controller webhook
Warning: deleting cluster-scoped resources, not scoped to the provided namespace
mutatingwebhookconfiguration.admissionregistration.k8s.io "vpa-webhook-config" deleted
deployment.apps "vpa-admission-controller" deleted
service "vpa-webhook" deleted
resource mapping not found for name: "verticalpodautoscalers.autoscaling.k8s.io" namespace: "" from "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "verticalpodautoscalercheckpoints.autoscaling.k8s.io" namespace: "" from "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
๐ CAS - Cluster Autoscaler
1. ์ค์ ์ ํ์ธ
- AWS CLI๋ฅผ ์ฌ์ฉํ์ฌ ํด๋ฌ์คํฐ ๋ ธ๋์ ํ๊ทธ ์ ๋ณด๋ฅผ ์กฐํํจ
1
aws ec2 describe-instances --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Reservations[*].Instances[*].Tags[*]" --output json | jq
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
[
[
[
{
"Key": "aws:ec2launchtemplate:version",
"Value": "1"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "aws:ec2launchtemplate:id",
"Value": "lt-0053100b6356270f0"
},
{
"Key": "k8s.io/cluster-autoscaler/enabled",
"Value": "true"
},
{
"Key": "kubernetes.io/cluster/myeks",
"Value": "owned"
},
{
"Key": "aws:ec2:fleet-id",
"Value": "fleet-cd95ab35-728e-ee0d-049a-a1a26e56a0e2"
},
{
"Key": "aws:eks:cluster-name",
"Value": "myeks"
},
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "aws:autoscaling:groupName",
"Value": "eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
},
{
"Key": "k8s.io/cluster-autoscaler/myeks",
"Value": "owned"
}
]
],
[
[
{
"Key": "aws:ec2:fleet-id",
"Value": "fleet-59b7350e-c706-c496-2eba-25a2335385de"
},
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
},
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "k8s.io/cluster-autoscaler/enabled",
"Value": "true"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "aws:ec2launchtemplate:id",
"Value": "lt-0053100b6356270f0"
},
{
"Key": "k8s.io/cluster-autoscaler/myeks",
"Value": "owned"
},
{
"Key": "aws:eks:cluster-name",
"Value": "myeks"
},
{
"Key": "aws:ec2launchtemplate:version",
"Value": "1"
},
{
"Key": "kubernetes.io/cluster/myeks",
"Value": "owned"
},
{
"Key": "aws:autoscaling:groupName",
"Value": "eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea"
}
]
],
[
[
{
"Key": "k8s.io/cluster-autoscaler/myeks",
"Value": "owned"
},
{
"Key": "kubernetes.io/cluster/myeks",
"Value": "owned"
},
{
"Key": "aws:ec2launchtemplate:id",
"Value": "lt-0053100b6356270f0"
},
{
"Key": "aws:ec2launchtemplate:version",
"Value": "1"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "aws:eks:cluster-name",
"Value": "myeks"
},
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "aws:ec2:fleet-id",
"Value": "fleet-e715233f-520e-eea7-0c9a-010a399d2bc7"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "k8s.io/cluster-autoscaler/enabled",
"Value": "true"
},
{
"Key": "aws:autoscaling:groupName",
"Value": "eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
}
]
]
]
- ์ถ๋ ฅ ๊ฒฐ๊ณผ์์ autoscaler ๊ด๋ จ ํ๊ทธ(
k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/myeks๋ฑ)๋ฅผ ํ์ธํ ์ ์์
2. ํ์ฌ auto ์ค์ผ์ผ ๊ทธ๋ฃน ์ ๋ณด ํ์ธ
1
2
3
aws autoscaling describe-auto-scaling-groups \
--query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='myeks']].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" \
--output table
โ ย ์ถ๋ ฅ
1
2
3
4
5
-----------------------------------------------------------------
| DescribeAutoScalingGroups |
+------------------------------------------------+----+----+----+
| eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea | 3 | 3 | 3 |
+------------------------------------------------+----+----+----+
3. Auto Scaling ๊ทธ๋ฃน ํฌ๊ธฐ ์์ (MaxSize 6๊ฐ๋ก ๋ณ๊ฒฝ)
1
2
export ASG_NAME=$(aws autoscaling describe-auto-scaling-groups --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='myeks']].AutoScalingGroupName" --output text)
aws autoscaling update-auto-scaling-group --auto-scaling-group-name ${ASG_NAME} --min-size 3 --desired-capacity 3 --max-size 6
- ๋ณ๊ฒฝ ํ, ASG ์ ๋ณด๊ฐ ์ ๋ฐ์ดํธ ๋์์์ ์ฌํ์ธํจ
1
aws autoscaling describe-auto-scaling-groups --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='myeks']].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" --output table
โ ย ์ถ๋ ฅ
1
2
3
4
5
-----------------------------------------------------------------
| DescribeAutoScalingGroups |
+------------------------------------------------+----+----+----+
| eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea | 3 | 6 | 3 |
+------------------------------------------------+----+----+----+
4. CAS ์ค์น
- ๊ณต์ Cluster Autoscaler YAML ํ์ผ์ ๋ค์ด๋ก๋๋ฐ์
1
curl -s -O https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml
- ํ์ผ ๋ด
<YOUR CLUSTER NAME>์ ํ๊ฒฝ ๋ณ์$CLUSTER_NAME๋ก ๋์ฒดํ ํ, EKS์ ์ ์ฉํจ
1
2
3
4
5
6
7
8
9
10
sed -i -e "s|<YOUR CLUSTER NAME>|$CLUSTER_NAME|g" cluster-autoscaler-autodiscover.yaml
kubectl apply -f cluster-autoscaler-autodiscover.yaml
# ๊ฒฐ๊ณผ
serviceaccount/cluster-autoscaler created
clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
role.rbac.authorization.k8s.io/cluster-autoscaler created
clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
deployment.apps/cluster-autoscaler created
5. CAS ๋ฐฐํฌ ํ ์ํ ํ์ธ
(1) Cluster Autoscaler Pod ์กฐํ
1
kubectl get pod -n kube-system | grep cluster-autoscaler
โ ย ์ถ๋ ฅ
1
cluster-autoscaler-6df6d76b9f-jbpfg 1/1 Running 0 3m7s
(2) Cluster Autoscaler Deployment ์์ธ ์ ๋ณด ์กฐํ
1
kubectl describe deployments.apps -n kube-system cluster-autoscaler
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Name: cluster-autoscaler
Namespace: kube-system
CreationTimestamp: Sat, 08 Mar 2025 01:16:16 +0900
Labels: app=cluster-autoscaler
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=cluster-autoscaler
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=cluster-autoscaler
Annotations: prometheus.io/port: 8085
prometheus.io/scrape: true
Service Account: cluster-autoscaler
Containers:
cluster-autoscaler:
Image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.26.2
Port: <none>
Host Port: <none>
Command:
./cluster-autoscaler
--v=4
--stderrthreshold=info
--cloud-provider=aws
--skip-nodes-with-local-storage=false
--expander=least-waste
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/myeks
Limits:
cpu: 100m
memory: 600Mi
Requests:
cpu: 100m
memory: 600Mi
Environment: <none>
Mounts:
/etc/ssl/certs/ca-certificates.crt from ssl-certs (ro)
Volumes:
ssl-certs:
Type: HostPath (bare host directory volume)
Path: /etc/ssl/certs/ca-bundle.crt
HostPathType:
Priority Class Name: system-cluster-critical
Node-Selectors: <none>
Tolerations: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: cluster-autoscaler-6df6d76b9f (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m8s deployment-controller Scaled up replica set cluster-autoscaler-6df6d76b9f to 1
- command ๋ถ๋ถ์ ํด๋ฌ์คํฐ ์ด๋ฆ์ด ํฌํจ๋ ๊ฒ์ ํ์ธํจ
6. nginx ์ ํ๋ฆฌ์ผ์ด์ ๋ฐฐํฌ (์ค์ผ์ผ ์์ ํ ์คํธ)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
(eks-user@myeks:default) [root@operator-host ~]# cat << EOF > nginx.yaml> apiVersion: apps/v1
> kind: Deployment
> metadata:
> name: nginx-to-scaleout
> spec:
> replicas: 1
> selector:
> matchLabels:
> app: nginx
> template:
> metadata:
> labels:
> service: nginx
> app: nginx
> spec:
> containers:
> - image: nginx
> name: nginx-to-scaleout
> resources:
> limits:
> cpu: 500m
> memory: 512Mi
> requests:
> cpu: 500m
> memory: 512Mi
> EOF
(eks-user@myeks:default) [root@operator-host ~]# kubectl apply -f nginx.yaml
deployment.apps/nginx-to-scaleout created
- nginx ๋ฐฐํฌ YAML(
nginx.yaml)์ ์์ฑ ๋ฐ ์ ์ฉํ์ฌ ๋ฐฐํฌํจ
7. ๋ชจ๋ํฐ๋ง ์ค์
(1) ์ด์์ฉ EC2
1
(eks-user@myeks:default) [root@operator-host ~]# while true; do kubectl get node; echo "------------------------------" ; date ; sleep 1; done
(2) ๋ก์ปฌ PC
1
while true; do aws ec2 describe-instances --query "Reservations[*].Instances[*].{PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output text ; echo "------------------------------"; date; sleep 1; done
8. ์ค์ผ์ผ ์์
1
2
3
4
kubectl scale --replicas=15 deployment/nginx-to-scaleout && date
# ๊ฒฐ๊ณผ
deployment.apps/nginx-to-scaleout scaled
Sat Mar 8 01:28:59 AM KST 2025
- ๊ธฐ์กด 1๊ฐ์ Pod์์ 15๊ฐ๋ก ํ์ฅ
9. Pod ์ํ ํ์ธ
1
k get pod
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
NAME READY STATUS RESTARTS AGE
nginx-to-scaleout-7cfb655fb5-27rvm 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-2jx4x 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-4kqxc 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-5kxw4 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-68zsm 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-9js9r 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-f8lhn 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-fbh8w 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-g5txl 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-kkrll 1/1 Running 0 7m14s
nginx-to-scaleout-7cfb655fb5-sctn6 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-sgf5p 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-sxzqd 1/1 Running 0 58s
nginx-to-scaleout-7cfb655fb5-x72m6 0/1 Pending 0 58s
nginx-to-scaleout-7cfb655fb5-ztnc5 0/1 Pending 0 58s
- ์ผ๋ถ Pod๊ฐ
Pending์ํ๋ก ๋จ์ ์์ โ ๋ ธ๋ ๋ถ์กฑ ๊ฐ๋ฅ์ฑ ์์
10. Pending ์ํ์ Pod ์์ธ ์ ๋ณด ์กฐํ
1
k describe pod nginx-to-scaleout-7cfb655fb5-27rvm
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
Name: nginx-to-scaleout-7cfb655fb5-27rvm
Namespace: default
Priority: 0
Service Account: default
Node: ip-192-168-3-75.ap-northeast-2.compute.internal/192.168.3.75
Start Time: Sat, 08 Mar 2025 01:30:24 +0900
Labels: app=nginx
pod-template-hash=7cfb655fb5
service=nginx
Annotations: <none>
Status: Running
IP: 192.168.3.86
IPs:
IP: 192.168.3.86
Controlled By: ReplicaSet/nginx-to-scaleout-7cfb655fb5
Containers:
nginx-to-scaleout:
Container ID: containerd://e9729263b5d8d29c75a718c529458a3b8adeab7f7594a19da712374703d94ff0
Image: nginx
Image ID: docker.io/library/nginx@sha256:9d6b58feebd2dbd3c56ab5853333d627cc6e281011cfd6050fa4bcf2072c9496
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 08 Mar 2025 01:30:34 +0900
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 500m
memory: 512Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9br8f (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-9br8f:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m45s default-scheduler 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.
Normal TriggeredScaleUp 2m35s cluster-autoscaler pod triggered scale-up: [{eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea 3->6 (max: 6)}]
Warning FailedScheduling 85s default-scheduler 0/6 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 4 Insufficient cpu. preemption: 0/6 nodes are available: 2 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.
Normal Scheduled 80s default-scheduler Successfully assigned default/nginx-to-scaleout-7cfb655fb5-27rvm to ip-192-168-3-75.ap-northeast-2.compute.internal
Normal Pulling 79s kubelet Pulling image "nginx"
Normal Pulled 70s kubelet Successfully pulled image "nginx" in 8.569s (8.569s including waiting). Image size: 72195292 bytes.
Normal Created 70s kubelet Created container nginx-to-scaleout
Normal Started 70s kubelet Started container nginx-to-scaleout
Pending์ํ์ Pod๊ฐ ๋ ธ๋ ๋ฆฌ์์ค ๋ถ์กฑ์ผ๋ก ์ค์ผ์ค๋ง๋์ง ์๊ณ ์์
11. ์๋ ํ์ฅ๋ ๋ ธ๋ ํ์ธ
1
kubectl get nodes
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
NAME STATUS ROLES AGE VERSION
ip-192-168-1-223.ap-northeast-2.compute.internal Ready <none> 3m29s v1.31.5-eks-5d632ec
ip-192-168-1-57.ap-northeast-2.compute.internal Ready <none> 13h v1.31.5-eks-5d632ec
ip-192-168-2-122.ap-northeast-2.compute.internal Ready <none> 13h v1.31.5-eks-5d632ec
ip-192-168-2-175.ap-northeast-2.compute.internal Ready <none> 3m38s v1.31.5-eks-5d632ec
ip-192-168-3-75.ap-northeast-2.compute.internal Ready <none> 3m34s v1.31.5-eks-5d632ec
ip-192-168-3-77.ap-northeast-2.compute.internal Ready <none> 13h v1.31.5-eks-5d632ec
- Cluster Autoscaler(CAS)๊ฐ ๋ถ์กฑํ ๋ฆฌ์์ค๋ฅผ ๊ฐ์งํ๊ณ , ๋ ธ๋๋ฅผ ์ถ๊ฐ ์์ฑ
- ๊ธฐ์กด 3๊ฐ์ ๋ ธ๋์์ 6๊ฐ๋ก ์ฆ๊ฐ
12. Auto Scaling Group ์ํ ํ์ธ
1
2
3
aws autoscaling describe-auto-scaling-groups \
--query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='myeks']].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" \
--output table
โ ย ์ถ๋ ฅ
MinSize: 3 โ 6์ผ๋ก ์ฆ๊ฐ
1
2
3
4
5
6
-----------------------------------------------------------------
| DescribeAutoScalingGroups |
+------------------------------------------------+----+----+----+
| eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea | 3 | 6 | 6 |
+------------------------------------------------+----+----+----+
13. ๋ ธ๋ ๋ฆฌ์์ค ์ฌ์ฉ๋ ๋ชจ๋ํฐ๋ง
1
eks-node-viewer --resources cpu,memory
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
6 nodes ( 9200m/11580m) 79.4% cpu โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
10020Mi/20187152Ki 50.8% memory โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
54 pods (0 pending 54 running 54 bound)
ip-192-168-3-77.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-2-122.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-1-57.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-2-175.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-3-75.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
ip-192-168-1-223.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโโ
14. Auto Scaling ์ฆ์ค ์ด๋ฒคํธ ํ์ธ
- CloudTrail์์
CreateFleet์ด๋ฒคํธ ํ์ธ
15. ์ค์ผ์ผ ์ธ ๋ฐ ํด๋ฌ์คํฐ ์ ๋ฆฌ
- Auto Scaling์ผ๋ก ์ฆ๊ฐํ ๋
ธ๋๊ฐ ๋ค์ ๊ฐ์ํ๋๋ก
Deployment์ญ์
1
2
3
(eks-user@myeks:default) [root@operator-host ~]# kubectl delete -f nginx.yaml && date
deployment.apps "nginx-to-scaleout" deleted
Sat Mar 8 01:44:36 KST 2025
16. Auto Scaling Group ํฌ๊ธฐ ์๋ณต
MaxSize๋ฅผ 3์ผ๋ก ์กฐ์ ํ์ฌ ๋ ธ๋ ๊ฐ์๋ฅผ ์๋ ์ํ๋ก ๋ณต์
1
aws autoscaling update-auto-scaling-group --auto-scaling-group-name ${ASG_NAME} --min-size 3 --desired-capacity 3 --max-size 3
17. ํ์ฌ Auto Scaling Group ์ํ ํ์ธ
1
aws autoscaling describe-auto-scaling-groups --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='myeks']].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" --output table
โ ย ์ถ๋ ฅ
1
2
3
4
5
-----------------------------------------------------------------
| DescribeAutoScalingGroups |
+------------------------------------------------+----+----+----+
| eks-ng1-52cab746-dd2b-385e-8ad0-075de5ebbbea | 3 | 3 | 3 |
+------------------------------------------------+----+----+----+
18. CAS ๋ฐฐํฌ ์ญ์
1
2
3
4
5
6
7
8
9
kubectl delete -f cluster-autoscaler-autodiscover.yaml
# ๊ฒฐ๊ณผ
serviceaccount "cluster-autoscaler" deleted
clusterrole.rbac.authorization.k8s.io "cluster-autoscaler" deleted
role.rbac.authorization.k8s.io "cluster-autoscaler" deleted
clusterrolebinding.rbac.authorization.k8s.io "cluster-autoscaler" deleted
rolebinding.rbac.authorization.k8s.io "cluster-autoscaler" deleted
deployment.apps "cluster-autoscaler" deleted
19. ์ ์ฒด ํด๋ฌ์คํฐ ์ญ์
- Karpenter ์ค์ต ํ๊ฒฝ ์ค๋น๋ฅผ ์ํด EKS ํด๋ฌ์คํฐ ์ ์ฒด ์ญ์ ์งํ
nohup์ ์ฌ์ฉํ์ฌ ๋ฐฑ๊ทธ๋ผ์ด๋์์ ์คํ
1
2
(eks-user@myeks:default) [root@operator-host ~]# nohup sh -c "eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME" > /root/delete.log 2>&1 &
[1] 23126
๐จ Karpenter
- ๊ณ ์ฑ๋ฅ์ ์ง๋ฅํ Kubernetes ๋ ธ๋ ํ๋ก๋น์ ๋ ๋ฐ ๊ด๋ฆฌ ์๋ฃจ์
- ๋์ ์ผ๋ก ์ธ์คํด์ค ์ ํ ์ ํ (์: Spot, AWS Graviton ๋ฑ)
- ์๋ ์ํฌ๋ก๋ Consolidation ๊ธฐ๋ฅ
- ๋น ๋ฅธ ๋ ธ๋ ๊ตฌ๋ ์๊ฐ์ผ๋ก ์๊ฐ๊ณผ ๋น์ฉ ๋ญ๋น ์ต์ํ
1. ๋ณ์์ค์
1
2
3
4
5
6
7
8
9
10
11
12
export KARPENTER_NAMESPACE="kube-system"
export KARPENTER_VERSION="1.2.1"
export K8S_VERSION="1.32"
export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov
export CLUSTER_NAME="gagajin-karpenter-demo" # ${USER}-karpenter-demo
export AWS_DEFAULT_REGION="ap-northeast-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export TEMPOUT="$(mktemp)"
export ALIAS_VERSION="$(aws ssm get-parameter --name "/aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2023/x86_64/standard/recommended/image_id" --query Parameter.Value | xargs aws ec2 describe-images --query 'Images[0].Name' --image-ids | sed -r 's/^.*(v[[:digit:]]+).*$/\1/')"
echo "${KARPENTER_NAMESPACE}" "${KARPENTER_VERSION}" "${K8S_VERSION}" "${CLUSTER_NAME}" "${AWS_DEFAULT_REGION}" "${AWS_ACCOUNT_ID}" "${TEMPOUT}" "${ALIAS_VERSION}"
โ ย ์ถ๋ ฅ
1
kube-system 1.2.1 1.32 gagajin-karpenter-demo ap-northeast-2 378102432899 /tmp/tmp.WyTYetLBs2 v20250228
2. Karpenter ๋ฆฌ์์ค ๋ฐฐํฌ (CloudFormation)
1
2
3
4
5
6
7
8
9
10
11
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \
&& aws cloudformation deploy \
--stack-name "Karpenter-${CLUSTER_NAME}" \
--template-file "${TEMPOUT}" \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "ClusterName=${CLUSTER_NAME}"
# ๊ฒฐ๊ณผ
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - Karpenter-gagajin-karpenter-demo
- ๋ฐฐํฌ๋ resources ํ์ธ
- ํด๋ฌ์คํฐ ์์ฑ ์ ๋ฉํ๋ฐ์ดํฐ ํ์ฉ
- ํด๋ฌ์คํฐ ์์ฑ ์ ๋ฉํ๋ฐ์ดํฐ์ ํ๊ทธ๋ฅผ ์ถ๊ฐํ๋ฉด, EKS ๋ฐฐํฌ ์ ํ์ํ ๋ชจ๋ ๋ฆฌ์์ค์ ํด๋น ํ๊ทธ๊ฐ ์ ์ฉ๋จ
- ์ด๋ฅผ ํตํด CloudFormation์ผ๋ก Role ๋ฐ Policy๊ฐ ์๋์ผ๋ก ์์ฑ๋์ด ๊ด๋ฆฌ๊ฐ ํธ๋ฆฌํด์ง
3. ํด๋ฌ์คํฐ ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
eksctl create cluster -f - <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ${AWS_DEFAULT_REGION}
version: "${K8S_VERSION}"
tags:
karpenter.sh/discovery: ${CLUSTER_NAME}
iam:
withOIDC: true
podIdentityAssociations:
- namespace: "${KARPENTER_NAMESPACE}"
serviceAccountName: karpenter
roleName: ${CLUSTER_NAME}-karpenter
permissionPolicyARNs:
- arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}
iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
username: system:node:
groups:
- system:bootstrappers
- system:nodes
## If you intend to run Windows workloads, the kube-proxy group should be specified.
# For more information, see https://github.com/aws/karpenter/issues/5099.
# - eks:kube-proxy-windows
managedNodeGroups:
- instanceType: m5.large
amiFamily: AmazonLinux2023
name: ${CLUSTER_NAME}-ng
desiredCapacity: 2
minSize: 1
maxSize: 10
iam:
withAddonPolicies:
externalDNS: true
addons:
- name: eks-pod-identity-agent
EOF
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
2025-03-08 15:02:27 [โน] eksctl version 0.203.0
2025-03-08 15:02:27 [โน] using region ap-northeast-2
2025-03-08 15:02:27 [โน] setting availability zones to [ap-northeast-2a ap-northeast-2c ap-northeast-2d]
2025-03-08 15:02:27 [โน] subnets for ap-northeast-2a - public:192.168.0.0/19 private:192.168.96.0/19
2025-03-08 15:02:27 [โน] subnets for ap-northeast-2c - public:192.168.32.0/19 private:192.168.128.0/19
2025-03-08 15:02:27 [โน] subnets for ap-northeast-2d - public:192.168.64.0/19 private:192.168.160.0/19
2025-03-08 15:02:27 [โน] nodegroup "gagajin-karpenter-demo-ng" will use "" [AmazonLinux2023/1.32]
2025-03-08 15:02:27 [โน] using Kubernetes version 1.32
2025-03-08 15:02:27 [โน] creating EKS cluster "gagajin-karpenter-demo" in "ap-northeast-2" region with managed nodes
2025-03-08 15:02:27 [โน] 1 nodegroup (gagajin-karpenter-demo-ng) was included (based on the include/exclude rules)
2025-03-08 15:02:27 [โน] will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2025-03-08 15:02:27 [โน] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-2 --cluster=gagajin-karpenter-demo'
2025-03-08 15:02:27 [โน] Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "gagajin-karpenter-demo" in "ap-northeast-2"
2025-03-08 15:02:27 [โน] CloudWatch logging will not be enabled for cluster "gagajin-karpenter-demo" in "ap-northeast-2"
2025-03-08 15:02:27 [โน] you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-2 --cluster=gagajin-karpenter-demo'
2025-03-08 15:02:27 [โน] default addons metrics-server, vpc-cni, kube-proxy, coredns were not specified, will install them as EKS addons
2025-03-08 15:02:27 [โน]
2 sequential tasks: { create cluster control plane "gagajin-karpenter-demo",
2 sequential sub-tasks: {
6 sequential sub-tasks: {
1 task: { create addons },
wait for control plane to become ready,
associate IAM OIDC provider,
no tasks,
update VPC CNI to use IRSA if required,
create IAM identity mappings,
},
create managed nodegroup "gagajin-karpenter-demo-ng",
}
}
2025-03-08 15:02:27 [โน] building cluster stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:02:28 [โน] deploying stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:02:58 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:03:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:04:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:05:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:06:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:07:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:08:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:09:28 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:10:29 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 15:10:31 [โน] creating addon
2025-03-08 15:10:32 [โน] successfully created addon
2025-03-08 15:10:32 [โน] creating addon
2025-03-08 15:10:33 [โน] successfully created addon
2025-03-08 15:10:33 [!] recommended policies were found for "vpc-cni" addon, but since OIDC is disabled on the cluster, eksctl cannot configure the requested permissions; the recommended way to provide IAM permissions for "vpc-cni" addon is via pod identity associations; after addon creation is completed, add all recommended policies to the config file, under `addon.PodIdentityAssociations`, and run `eksctl update addon`
2025-03-08 15:10:33 [โน] creating addon
2025-03-08 15:10:34 [โน] successfully created addon
2025-03-08 15:10:34 [โน] creating addon
2025-03-08 15:10:35 [โน] successfully created addon
2025-03-08 15:10:35 [โน] creating addon
2025-03-08 15:10:36 [โน] successfully created addon
2025-03-08 15:12:38 [โน] addon "vpc-cni" active
2025-03-08 15:12:39 [โน] deploying stack "eksctl-gagajin-karpenter-demo-addon-vpc-cni"
2025-03-08 15:12:39 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-addon-vpc-cni"
2025-03-08 15:13:09 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-addon-vpc-cni"
2025-03-08 15:13:09 [โน] updating addon
2025-03-08 15:13:20 [โน] addon "vpc-cni" active
2025-03-08 15:13:20 [โน] checking arn arn:aws:iam::378102432899:role/KarpenterNodeRole-gagajin-karpenter-demo against entries in the auth ConfigMap
2025-03-08 15:13:20 [โน] adding identity "arn:aws:iam::378102432899:role/KarpenterNodeRole-gagajin-karpenter-demo" to auth ConfigMap
2025-03-08 15:13:21 [โน] building managed nodegroup stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:13:21 [โน] deploying stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:13:21 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:13:51 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:14:48 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:16:25 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 15:16:25 [โน] waiting for the control plane to become ready
2025-03-08 15:16:25 [โ] saved kubeconfig as "/home/devshin/.kube/config"
2025-03-08 15:16:25 [โน] no tasks
2025-03-08 15:16:25 [โ] all EKS cluster resources for "gagajin-karpenter-demo" have been created
2025-03-08 15:16:25 [โน] nodegroup "gagajin-karpenter-demo-ng" has 2 node(s)
2025-03-08 15:16:25 [โน] node "ip-192-168-18-49.ap-northeast-2.compute.internal" is ready
2025-03-08 15:16:25 [โน] node "ip-192-168-88-167.ap-northeast-2.compute.internal" is ready
2025-03-08 15:16:25 [โน] waiting for at least 1 node(s) to become ready in "gagajin-karpenter-demo-ng"
2025-03-08 15:16:25 [โน] nodegroup "gagajin-karpenter-demo-ng" has 2 node(s)
2025-03-08 15:16:25 [โน] node "ip-192-168-18-49.ap-northeast-2.compute.internal" is ready
2025-03-08 15:16:25 [โน] node "ip-192-168-88-167.ap-northeast-2.compute.internal" is ready
2025-03-08 15:16:25 [โ] created 1 managed nodegroup(s) in cluster "gagajin-karpenter-demo"
2025-03-08 15:16:25 [โน] 1 task: {
2 sequential sub-tasks: {
create IAM role for pod identity association for service account "kube-system/karpenter",
create pod identity association for service account "kube-system/karpenter",
} }2025-03-08 15:16:26 [โน] deploying stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 15:16:26 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 15:16:56 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 15:16:57 [โน] created pod identity association for service account "karpenter" in namespace "kube-system"
2025-03-08 15:16:57 [โน] all tasks were completed successfully
2025-03-08 15:16:58 [โน] kubectl command should work with "/home/devshin/.kube/config", try 'kubectl get nodes'
2025-03-08 15:16:58 [โ] EKS cluster "gagajin-karpenter-demo" in "ap-northeast-2" region is ready
4. EKS ๋ฐฐํฌ ํ์ธ
(1) ํด๋ฌ์คํฐ ์กฐํ
1
eksctl get cluster
โ ย ์ถ๋ ฅ
1
2
NAME REGION EKSCTL CREATED
gagajin-karpenter-demo ap-northeast-2 True
(2) Nodegroup ์ ๋ณด ์กฐํ
1
eksctl get nodegroup --cluster $CLUSTER_NAME
โ ย ์ถ๋ ฅ
1
2
CLUSTER NODEGROUP STATUS CREATEDMIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE
gagajin-karpenter-demo gagajin-karpenter-demo-ng ACTIVE 2025-03-08T06:13:47Z 1 10 2 m5.large AL2023_x86_64_STANDARD eks-gagajin-karpenter-demo-ng-78caba31-af01-b10d-8106-c33d85fc7eb7 managed
(3) IAM Identity Mapping ์กฐํ
1
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
โ ย ์ถ๋ ฅ
1
2
3
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::378102432899:role/KarpenterNodeRole-gagajin-karpenter-demosystem:node: system:bootstrappers,system:nodes
arn:aws:iam::378102432899:role/eksctl-gagajin-karpenter-demo-node-NodeInstanceRole-vOdC2X7NmA3o system:node: system:bootstrappers,system:nodes
(4) IAM ServiceAccount ์กฐํ
1
eksctl get iamserviceaccount --cluster $CLUSTER_NAME
โ ย ์ถ๋ ฅ
1
No iamserviceaccounts found
(5) Add-on ์ ๋ณด ์กฐํ
1
eksctl get addon --cluster $CLUSTER_NAME
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
2025-03-08 15:35:40 [โน] Kubernetes version "1.32" in use by cluster "gagajin-karpenter-demo"
2025-03-08 15:35:40 [โน] getting all addons
2025-03-08 15:35:42 [โน] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME VERSION STATUS ISSUES IAMROLEUPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES
coredns v1.11.4-eksbuild.2 ACTIVE 0
eks-pod-identity-agent v1.3.4-eksbuild.1 ACTIVE 0 v1.3.5-eksbuild.2
kube-proxy v1.32.0-eksbuild.2 ACTIVE 0
metrics-server v0.7.2-eksbuild.2 ACTIVE 0
vpc-cni v1.19.2-eksbuild.1 ACTIVE 0 arn:aws:iam::378102432899:role/eksctl-gagajin-karpenter-demo-addon-vpc-cni-Role1-rgV8iaCEeQSl v1.19.3-eksbuild.1,v1.19.2-eksbuild.5
5. kubectl Context ์ค์
1
2
kubectl ctx
kubectl config rename-context "<๊ฐ์ ์์ ์ IAM User>@<์์ ์ Nickname>-karpenter-demo.ap-northeast-2.eksctl.io" "karpenter-demo"
6. EKS ํ์ธ
(1) ํด๋ฌ์คํฐ ์ ๋ณด ํ์ธ
1
kubectl cluster-info
โ ย ์ถ๋ ฅ
1
2
3
4
Kubernetes control plane is running at https://20ECC1FD640C207538AA1DED1C19CCC7.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://20ECC1FD640C207538AA1DED1C19CCC7.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
(2) ๋ ธ๋ ์์ธ ์ ๋ณด ์กฐํ
1
kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone
โ ย ์ถ๋ ฅ
1
2
3
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE CAPACITYTYPE ZONE
ip-192-168-18-49.ap-northeast-2.compute.internal Ready <none> 25m v1.32.1-eks-5d632ec m5.large ON_DEMAND ap-northeast-2a
ip-192-168-88-167.ap-northeast-2.compute.internal Ready <none> 25m v1.32.1-eks-5d632ec m5.large ON_DEMAND ap-northeast-2d
(3) kube-system ๋ค์์คํ์ด์ค์ Pod ์กฐํ
1
kubectl get pod -n kube-system -owide
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aws-node-h94d7 2/2 Running 0 26m 192.168.18.49 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
aws-node-vfq4h 2/2 Running 0 26m 192.168.88.167 ip-192-168-88-167.ap-northeast-2.compute.internal <none> <none>
coredns-844d8f59bb-kbt92 1/1 Running 0 30m 192.168.0.156 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
coredns-844d8f59bb-qk5zr 1/1 Running 0 30m 192.168.22.61 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
eks-pod-identity-agent-k94qf 1/1 Running 0 26m 192.168.88.167 ip-192-168-88-167.ap-northeast-2.compute.internal <none> <none>
eks-pod-identity-agent-qgwxx 1/1 Running 0 26m 192.168.18.49 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
kube-proxy-qsnv7 1/1 Running 0 26m 192.168.18.49 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
kube-proxy-vrwsr 1/1 Running 0 26m 192.168.88.167 ip-192-168-88-167.ap-northeast-2.compute.internal <none> <none>
metrics-server-74b6cb4f8f-647bp 1/1 Running 0 30m 192.168.81.147 ip-192-168-88-167.ap-northeast-2.compute.internal <none> <none>
metrics-server-74b6cb4f8f-spvsb 1/1 Running 0 30m 192.168.30.193 ip-192-168-18-49.ap-northeast-2.compute.internal <none> <none>
(4) Pod Disruption Budget(PDB) ์กฐํ
1
kubectl get pdb -A
โ ย ์ถ๋ ฅ
1
2
3
NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
kube-system coredns N/A 1 1 30m
kube-system metrics-server N/A 1 1 30m
(5) aws-auth ConfigMap ํ์ธ
1
kubectl describe cm -n kube-system aws-auth
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Name: aws-auth
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
mapUsers:
----
[]
mapRoles:
----
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::378102432899:role/KarpenterNodeRole-gagajin-karpenter-demo
username: system:node:
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::378102432899:role/eksctl-gagajin-karpenter-demo-node-NodeInstanceRole-vOdC2X7NmA3o
username: system:node:
BinaryData
====
Events: <none>
(6) AWS ์น ๊ด๋ฆฌ ์ฝ์ ํ์ธ
- Access
- Add-ons
7. ์ค์ต ๋๊ตฌ ๋ฐฐํฌ
(1) Helm์ผ๋ก kube-ops-view ์ค์น
1
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=LoadBalancer --set env.TZ="Asia/Seoul" --namespace kube-system
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
NAME: kube-ops-view
LAST DEPLOYED: Sat Mar 8 15:50:08 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w kube-ops-view'
export SERVICE_IP=$(kubectl get svc --namespace kube-system kube-ops-view -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo http://$SERVICE_IP:8080
(2) ๋ก๋๋ฐธ๋ฐ์ ๋๋ฉ์ธ ํ์ธ
1
echo -e "http://$(kubectl get svc -n kube-system kube-ops-view -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"):8080/#scale=1.5"
โ ย ์ถ๋ ฅ
1
http://ab3d2771bfd1f4594a3f2caf543ffc5a-1334305580.ap-northeast-2.elb.amazonaws.com:8080/#scale=1.5
8. Karpenter ์ค์น ์ค๋น ๋ฐ ๋ณ์ ์ค์
(1) ECR ๋ก๊ทธ์์
1
helm registry logout public.ecr.aws
(2) Karpenter ์ค์น๋ฅผ ์ํ ๋ณ์ ์ค์ ๋ฐ ํ์ธ
1
2
3
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" --query "cluster.endpoint" --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
echo "${CLUSTER_ENDPOINT} ${KARPENTER_IAM_ROLE_ARN}"
โ ย ์ถ๋ ฅ
1
https://20ECC1FD640C207538AA1DED1C19CCC7.gr7.ap-northeast-2.eks.amazonaws.com arn:aws:iam::378102432899:role/gagajin-karpenter-demo-karpenter
9. Karpenter ์ค์น
1
2
3
4
5
6
7
8
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
--set "settings.clusterName=${CLUSTER_NAME}" \
--set "settings.interruptionQueue=${CLUSTER_NAME}" \
--set controller.resources.requests.cpu=1 \
--set controller.resources.requests.memory=1Gi \
--set controller.resources.limits.cpu=1 \
--set controller.resources.limits.memory=1Gi \
--wait
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
Release "karpenter" does not exist. Installing it now.
Pulled: public.ecr.aws/karpenter/karpenter:1.2.1
Digest: sha256:e35d51812e8fae0193cd7a008d570b1ed9013c71f38700b1d8654d0a8ff24c40
NAME: karpenter
LAST DEPLOYED: Sat Mar 8 16:36:21 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
10. ์ค์น ํ์ธ
(1) Helm ๋ฆด๋ฆฌ์ค ํ์ธ
1
helm list -n kube-system
โ ย ์ถ๋ ฅ
1
2
3
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
karpenter kube-system 1 2025-03-08 16:36:21.505226432 +0900 KST deployed karpenter-1.2.1 1.2.1
kube-ops-view kube-system 1 2025-03-08 15:50:08.883312674 +0900 KST deployed kube-ops-view-1.2.2 20.4.0
(2) Karpenter ๋ค์์คํ์ด์ค ๋ฆฌ์์ค ํ์ธ
1
2
kubectl get-all -n $KARPENTER_NAMESPACE
kubectl get all -n $KARPENTER_NAMESPACE
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
NAME NAMESPACE AGE
configmap/amazon-vpc-cni kube-system 91m
configmap/aws-auth kube-system 88m
configmap/coredns kube-system 91m
configmap/extension-apiserver-authentication kube-system 94m
configmap/kube-apiserver-legacy-service-account-token-tracking kube-system 94m
configmap/kube-proxy kube-system 91m
configmap/kube-proxy-config kube-system 91m
configmap/kube-root-ca.crt kube-system 94m
endpoints/eks-extension-metrics-api kube-system 94m
endpoints/karpenter kube-system 5m53s
endpoints/kube-dns kube-system 91m
endpoints/kube-ops-view kube-system 52m
endpoints/metrics-server kube-system 91m
pod/aws-node-h94d7 kube-system 87m
pod/aws-node-vfq4h kube-system 87m
pod/coredns-844d8f59bb-kbt92 kube-system 91m
pod/coredns-844d8f59bb-qk5zr kube-system 91m
pod/eks-pod-identity-agent-k94qf kube-system 87m
pod/eks-pod-identity-agent-qgwxx kube-system 87m
pod/karpenter-584f97f49d-47vjg kube-system 5m53s
pod/karpenter-584f97f49d-4gld6 kube-system 5m53s
pod/kube-ops-view-6658c477d4-d8bsn kube-system 52m
pod/kube-proxy-qsnv7 kube-system 87m
pod/kube-proxy-vrwsr kube-system 87m
pod/metrics-server-74b6cb4f8f-647bp kube-system 91m
pod/metrics-server-74b6cb4f8f-spvsb kube-system 91m
secret/sh.helm.release.v1.karpenter.v1 kube-system 5m54s
secret/sh.helm.release.v1.kube-ops-view.v1 kube-system 52m
serviceaccount/attachdetach-controller kube-system 94m
serviceaccount/aws-cloud-provider kube-system 94m
serviceaccount/aws-node kube-system 91m
serviceaccount/certificate-controller kube-system 94m
serviceaccount/clusterrole-aggregation-controller kube-system 94m
serviceaccount/coredns kube-system 91m
serviceaccount/cronjob-controller kube-system 94m
serviceaccount/daemon-set-controller kube-system 94m
serviceaccount/default kube-system 94m
serviceaccount/deployment-controller kube-system 94m
serviceaccount/disruption-controller kube-system 94m
serviceaccount/endpoint-controller kube-system 94m
serviceaccount/endpointslice-controller kube-system 94m
serviceaccount/endpointslicemirroring-controller kube-system 94m
serviceaccount/ephemeral-volume-controller kube-system 94m
serviceaccount/expand-controller kube-system 94m
serviceaccount/generic-garbage-collector kube-system 94m
serviceaccount/horizontal-pod-autoscaler kube-system 94m
serviceaccount/job-controller kube-system 94m
serviceaccount/karpenter kube-system 5m53s
serviceaccount/kube-ops-view kube-system 52m
serviceaccount/kube-proxy kube-system 91m
serviceaccount/legacy-service-account-token-cleaner kube-system 94m
serviceaccount/metrics-server kube-system 91m
serviceaccount/namespace-controller kube-system 94m
serviceaccount/node-controller kube-system 94m
serviceaccount/persistent-volume-binder kube-system 94m
serviceaccount/pod-garbage-collector kube-system 94m
serviceaccount/pv-protection-controller kube-system 94m
serviceaccount/pvc-protection-controller kube-system 94m
serviceaccount/replicaset-controller kube-system 94m
serviceaccount/replication-controller kube-system 94m
serviceaccount/resourcequota-controller kube-system 94m
serviceaccount/root-ca-cert-publisher kube-system 94m
serviceaccount/service-account-controller kube-system 94m
serviceaccount/service-controller kube-system 94m
serviceaccount/statefulset-controller kube-system 94m
serviceaccount/tagging-controller kube-system 94m
serviceaccount/ttl-after-finished-controller kube-system 94m
serviceaccount/ttl-controller kube-system 94m
serviceaccount/validatingadmissionpolicy-status-controller kube-system 94m
serviceaccount/volumeattributesclass-protection-controller kube-system 94m
service/eks-extension-metrics-api kube-system 94m
service/karpenter kube-system 5m53s
service/kube-dns kube-system 91m
service/kube-ops-view kube-system 52m
service/metrics-server kube-system 91m
controllerrevision.apps/aws-node-84c84d44d kube-system 89m
controllerrevision.apps/aws-node-85cf4d8b48 kube-system 91m
controllerrevision.apps/eks-pod-identity-agent-657664cdbb kube-system 91m
controllerrevision.apps/kube-proxy-786f8bf4cd kube-system 91m
daemonset.apps/aws-node kube-system 91m
daemonset.apps/eks-pod-identity-agent kube-system 91m
daemonset.apps/kube-proxy kube-system 91m
deployment.apps/coredns kube-system 91m
deployment.apps/karpenter kube-system 5m53s
deployment.apps/kube-ops-view kube-system 52m
deployment.apps/metrics-server kube-system 91m
replicaset.apps/coredns-844d8f59bb kube-system 91m
replicaset.apps/karpenter-584f97f49d kube-system 5m53s
replicaset.apps/kube-ops-view-6658c477d4 kube-system 52m
replicaset.apps/metrics-server-74b6cb4f8f kube-system 91m
lease.coordination.k8s.io/apiserver-awyw7cmmyp2p2veemcjxpeqz7a kube-system 90m
lease.coordination.k8s.io/apiserver-zyak7sreoc7mvosk7yxanwlowm kube-system 94m
lease.coordination.k8s.io/cloud-controller-manager kube-system 94m
lease.coordination.k8s.io/cp-vpc-resource-controller kube-system 94m
lease.coordination.k8s.io/eks-certificates-controller kube-system 94m
lease.coordination.k8s.io/eks-coredns-autoscaler kube-system 94m
lease.coordination.k8s.io/karpenter-leader-election kube-system 5m47s
lease.coordination.k8s.io/kms-storage-migrator-lease kube-system 79m
lease.coordination.k8s.io/kube-controller-manager kube-system 94m
lease.coordination.k8s.io/kube-scheduler kube-system 94m
endpointslice.discovery.k8s.io/eks-extension-metrics-api-726cg kube-system 94m
endpointslice.discovery.k8s.io/karpenter-22c5b kube-system 5m53s
endpointslice.discovery.k8s.io/kube-dns-qqht7 kube-system 91m
endpointslice.discovery.k8s.io/kube-ops-view-9kvqc kube-system 52m
endpointslice.discovery.k8s.io/metrics-server-pj9ln kube-system 91m
poddisruptionbudget.policy/coredns kube-system 91m
poddisruptionbudget.policy/karpenter kube-system 5m53s
poddisruptionbudget.policy/metrics-server kube-system 91m
rolebinding.rbac.authorization.k8s.io/eks-vpc-resource-controller-rolebinding kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:addon-manager kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:authenticator kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:az-poller kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:certificate-controller kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:cloud-controller-manager:apiserver-authentication-reader kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:coredns-autoscaler kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:fargate-manager kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:k8s-metrics kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:network-policy-controller kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:node-manager kube-system 94m
rolebinding.rbac.authorization.k8s.io/eks:service-operations kube-system 94m
rolebinding.rbac.authorization.k8s.io/karpenter kube-system 5m53s
rolebinding.rbac.authorization.k8s.io/karpenter-dns kube-system 5m53s
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader kube-system 91m
rolebinding.rbac.authorization.k8s.io/system::extension-apiserver-authentication-reader kube-system 94m
rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager kube-system 94m
rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler kube-system 94m
rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer kube-system 94m
rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider kube-system 94m
rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner kube-system 94m
role.rbac.authorization.k8s.io/eks-vpc-resource-controller-role kube-system 94m
role.rbac.authorization.k8s.io/eks:addon-manager kube-system 94m
role.rbac.authorization.k8s.io/eks:authenticator kube-system 94m
role.rbac.authorization.k8s.io/eks:az-poller kube-system 94m
role.rbac.authorization.k8s.io/eks:certificate-controller kube-system 94m
role.rbac.authorization.k8s.io/eks:coredns-autoscaler kube-system 94m
role.rbac.authorization.k8s.io/eks:fargate-manager kube-system 94m
role.rbac.authorization.k8s.io/eks:k8s-metrics kube-system 94m
role.rbac.authorization.k8s.io/eks:network-policy-controller kube-system 94m
role.rbac.authorization.k8s.io/eks:node-manager kube-system 94m
role.rbac.authorization.k8s.io/eks:service-operations-configmaps kube-system 94m
role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader kube-system 94m
role.rbac.authorization.k8s.io/karpenter kube-system 5m53s
role.rbac.authorization.k8s.io/karpenter-dns kube-system 5m53s
role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager kube-system 94m
role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler kube-system 94m
role.rbac.authorization.k8s.io/system:controller:bootstrap-signer kube-system 94m
role.rbac.authorization.k8s.io/system:controller:cloud-provider kube-system 94m
role.rbac.authorization.k8s.io/system:controller:token-cleaner kube-system 94m
NAME READY STATUS RESTARTS AGE
pod/aws-node-h94d7 2/2 Running 0 87m
pod/aws-node-vfq4h 2/2 Running 0 87m
pod/coredns-844d8f59bb-kbt92 1/1 Running 0 91m
pod/coredns-844d8f59bb-qk5zr 1/1 Running 0 91m
pod/eks-pod-identity-agent-k94qf 1/1 Running 0 87m
pod/eks-pod-identity-agent-qgwxx 1/1 Running 0 87m
pod/karpenter-584f97f49d-47vjg 1/1 Running 0 5m54s
pod/karpenter-584f97f49d-4gld6 1/1 Running 0 5m54s
pod/kube-ops-view-6658c477d4-d8bsn 1/1 Running 0 52m
pod/kube-proxy-qsnv7 1/1 Running 0 87m
pod/kube-proxy-vrwsr 1/1 Running 0 87m
pod/metrics-server-74b6cb4f8f-647bp 1/1 Running 0 91m
pod/metrics-server-74b6cb4f8f-spvsb 1/1 Running 0 91m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/eks-extension-metrics-api ClusterIP 10.100.22.96 <none> 443/TCP 94m
service/karpenter ClusterIP 10.100.217.16 <none> 8080/TCP 5m54s
service/kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 91m
service/kube-ops-view LoadBalancer 10.100.74.88 ab3d2771bfd1f4594a3f2caf543ffc5a-1334305580.ap-northeast-2.elb.amazonaws.com 8080:32265/TCP 52m
service/metrics-server ClusterIP 10.100.120.139 <none> 443/TCP 91m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/aws-node 2 2 2 2 2 <none> 91m
daemonset.apps/eks-pod-identity-agent 2 2 2 2 2 <none> 91m
daemonset.apps/kube-proxy 2 2 2 2 2 <none> 91m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 91m
deployment.apps/karpenter 2/2 2 2 5m54s
deployment.apps/kube-ops-view 1/1 1 1 52m
deployment.apps/metrics-server 2/2 2 2 91m
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-844d8f59bb 2 2 2 91m
replicaset.apps/karpenter-584f97f49d 2 2 2 5m54s
replicaset.apps/kube-ops-view-6658c477d4 1 1 1 52m
replicaset.apps/metrics-server-74b6cb4f8f 2 2 2 91m
(3) Karpenter CRD ํ์ธ
1
kubectl get crd | grep karpenter
โ ย ์ถ๋ ฅ
1
2
3
ec2nodeclasses.karpenter.k8s.aws 2025-03-08T07:36:21Z
nodeclaims.karpenter.sh 2025-03-08T07:36:21Z
nodepools.karpenter.sh 2025-03-08T07:36:21Z
- Karpenter๋ ๋
ธ๋ ์ฉ๋ ์ถ์ ์ ์ํด ๋ค์ ํ๊ทธ ํค๋ฅผ ํ์ฉํจ
karpenter.sh/managed-bykarpenter.sh/nodepoolkubernetes.io/cluster/${CLUSTER_NAME}
11. ํ๋ก๋ฉํ ์ฐ์ค ์ค์น ๋ฐ ํ์ธ
(1) Helm ์ ์ฅ์ ์ถ๊ฐ ๋ฐ ๋ค์์คํ์ด์ค ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
helm repo add grafana-charts https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
# ๊ฒฐ๊ณผ
"grafana-charts" has been added to your repositories
"prometheus-community" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kedacore" chart repository
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "grafana-charts" chart repository
...Successfully got an update from the "geek-cookbook" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. โHappy Helming!โ
namespace/monitoring created
(2) prometheus-values.yaml ํ์ผ ๋ค์ด๋ก๋
1
2
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/prometheus-values.yaml | envsubst | tee prometheus-values.yaml
helm install --namespace monitoring prometheus prometheus-community/prometheus --values prometheus-values.yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
alertmanager:
persistentVolume:
enabled: false
server:
fullnameOverride: prometheus-server
persistentVolume:
enabled: false
extraScrapeConfigs: |
- job_name: karpenter
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- kube-system
relabel_configs:
- source_labels:
- __meta_kubernetes_endpoints_name
- __meta_kubernetes_endpoint_port_name
action: keep
regex: karpenter;http-metrics
NAME: prometheus
LAST DEPLOYED: Sat Mar 8 16:45:55 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local
Get the Prometheus server URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9090
#################################################################################
###### WARNING: Persistence is disabled!!! You will lose your data when #####
###### the Server pod is terminated. #####
#################################################################################
The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local
Get the Alertmanager URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=alertmanager,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9093
#################################################################################
###### WARNING: Pod Security Policy has been disabled by default since #####
###### it deprecated after k8s 1.25+. use #####
###### (index .Values "prometheus-node-exporter" "rbac" #####
###### . "pspEnabled") with (index .Values #####
###### "prometheus-node-exporter" "rbac" "pspAnnotations") #####
###### in case you still need it. #####
#################################################################################
The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.monitoring.svc.cluster.local
Get the PushGateway URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9091
For more information on running Prometheus, visit:
https://prometheus.io/
(3) Prometheus Alertmanager ์ ๊ฑฐ
1
2
3
kubectl delete sts -n monitoring prometheus-alertmanager
# ๊ฒฐ๊ณผ
statefulset.apps "prometheus-alertmanager" deleted
(4) Prometheus ํฌํธํฌ์๋ฉ ์คํ
๋ก์ปฌ ์ ๊ทผ์ ์ํด Prometheus ์๋ฒ Pod์ ํฌํธํฌ์๋ฉ์ ์ค์ ํจ
1
2
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9090 &
โ ย ์ถ๋ ฅ
1
2
3
4
[1] 198317
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Target health์์ Karpenter ํ์ธ
12. ๊ทธ๋ผํ๋ ์ค์น ๋ฐ ํ์ธ
(1) ๊ทธ๋ผํ๋ ์ค์น
1
2
curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/grafana-values.yaml | tee grafana-values.yaml
helm install --namespace monitoring grafana grafana-charts/grafana --values grafana-values.yaml
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
version: 1
url: http://prometheus-server:80
access: proxy
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/default
dashboards:
default:
capacity-dashboard:
url: https://karpenter.sh/preview/getting-started/getting-started-with-karpenter/karpenter-capacity-dashboard.json
performance-dashboard:
url: https://karpenter.sh/preview/getting-started/getting-started-with-karpenter/karpenter-performance-dashboard.json
NAME: grafana
LAST DEPLOYED: Sat Mar 8 16:53:05 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
grafana.monitoring.svc.cluster.local
Get the Grafana URL to visit by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 3000
3. Login with the password from step 1 and the username: admin
#################################################################################
###### WARNING: Persistence is disabled!!! You will lose your data when #####
###### the Grafana pod is terminated. #####
#################################################################################
(2) ๊ทธ๋ผํ๋ admin ์ํธ ํ์ธ
1
2
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
โ ย ์ถ๋ ฅ
1
tELFbo0QSkJiEMFe3HerjWhLiCaDARc1toF3vf3n
(3) ํฌํธํฌ์๋ฉ ๋ฐ ์ ์
1
kubectl port-forward --namespace monitoring svc/grafana 3000:80 &
- ์ด๋ฏธ ๊ธฐ๋ณธ ๋์๋ณด๋์ โKarpenter Capacity v1โ์ โKarpenter Performance v1โ์ด ํฌํจ๋์ด ์์
13. Alias Version ํ์ธ
1
echo $ALIAS_VERSION
โ ย ์ถ๋ ฅ
1
v20250228
14. ๋ ธ๋ํ ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
cat <<EOF | envsubst | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
requirements:
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: kubernetes.io/os
operator: In
values: ["linux"]
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["2"]
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
expireAfter: 720h # 30 * 24h = 720h
limits:
cpu: 1000
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: default
spec:
role: "KarpenterNodeRole-${CLUSTER_NAME}" # replace with your cluster name
amiSelectorTerms:
- alias: "al2023@${ALIAS_VERSION}" # ex) al2023@latest
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: "${CLUSTER_NAME}" # replace with your cluster name
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: "${CLUSTER_NAME}" # replace with your cluster name
EOF
โ ย ์ถ๋ ฅ
1
2
nodepool.karpenter.sh/default created
ec2nodeclass.karpenter.k8s.aws/default created
15. ๋ ธ๋ํ ๋ฆฌ์์ค ํ์ธ
1
kubectl get nodepool,ec2nodeclass,nodeclaims
โ ย ์ถ๋ ฅ
1
2
3
4
5
NAME NODECLASS NODES READY AGE
nodepool.karpenter.sh/default default 0 True 101s
NAME READY AGE
ec2nodeclass.karpenter.k8s.aws/default True 101s
16. Deployment ์์ฑ ๋ฐ ์ด๊ธฐ ๋ฐฐํฌ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 0
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: 1
securityContext:
allowPrivilegeEscalation: false
EOF
# ๊ฒฐ๊ณผ
deployment.apps/inflate created
17. ์์คํ ๋ฆฌ์์ค ๋ชจ๋ํฐ๋ง
1
eks-node-viewer --resources cpu,memory
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
2 nodes ( 2700m/3860m) 69.9% cpu โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
2588Mi/14531936Ki 18.2% memory โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
19 pods (0 pending 19 running 19 bound)
ip-192-168-18-49.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโ
ip-192-168-88-167.ap-northeast-2.compute.internal cpu โโโโโโโโโโโโโโโ
memory โโโโโโโโโโโโโโโ
18. Deployment ์ํ ํ์ธ
1
kubectl get pod
โ ย ์ถ๋ ฅ
1
No resources found in default namespace.
19. Deployment ์ค์ผ์ผ ์์
(1) ์ค์ผ์ผ ๋ช ๋ น ์คํ
1
kubectl scale deployment inflate --replicas 5
โ ย ์ถ๋ ฅ
1
deployment.apps/inflate scaled
(2) Pod ์ํ ํ์ธ
1
k get pod
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
NAME READY STATUS RESTARTS AGE
inflate-5c5f75666d-5lcq7 0/1 Pending 0 30s
inflate-5c5f75666d-7sbpl 0/1 Pending 0 30s
inflate-5c5f75666d-99b26 0/1 Pending 0 30s
inflate-5c5f75666d-qbkmb 0/1 Pending 0 30s
inflate-5c5f75666d-xdxbt 0/1 Pending 0 30s
- Pending ์ํ์ Pod๋ ๋ ธ๋ ๋ฆฌ์์ค ๋ถ์กฑ์ผ๋ก ์ค์ผ์ค๋ง๋์ง ์์, Karpenter๊ฐ ์๋์ผ๋ก ๋ ธ๋๋ฅผ ์ฆ์คํ๋๋ก ํธ๋ฆฌ๊ฑฐ๋จ
20. NodeClaim ์์ฑ ๋ฐ ์ต์ ๋ ธ๋ ์ ํ ํ์ธ
(1) NodeClaim ์์ฑ ํ์ธ
1
kubectl logs -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller | grep 'launched nodeclaim' | jq '.'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
"level": "INFO",
"time": "2025-03-08T08:14:37.515Z",
"logger": "controller",
"message": "launched nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-sbszr"
},
"namespace": "",
"name": "default-sbszr",
"reconcileID": "bd896395-de19-4f5a-b4f1-315a6e83163b",
"provider-id": "aws:///ap-northeast-2a/i-0166a9530235c41bd",
"instance-type": "c5a.2xlarge",
"zone": "ap-northeast-2a",
"capacity-type": "on-demand",
"allocatable": {
"cpu": "7910m",
"ephemeral-storage": "17Gi",
"memory": "14162Mi",
"pods": "58",
"vpc.amazonaws.com/pod-eni": "38"
}
}
(2) ์ต์ ๋ ธ๋ ์ ์ ํ์ธ
1
kubectl describe nodeclaims
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
Name: default-sbszr
Namespace:
Labels: karpenter.k8s.aws/ec2nodeclass=default
karpenter.k8s.aws/instance-category=c
karpenter.k8s.aws/instance-cpu=8
karpenter.k8s.aws/instance-cpu-manufacturer=amd
karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz=3300
karpenter.k8s.aws/instance-ebs-bandwidth=3170
karpenter.k8s.aws/instance-encryption-in-transit-supported=true
karpenter.k8s.aws/instance-family=c5a
karpenter.k8s.aws/instance-generation=5
karpenter.k8s.aws/instance-hypervisor=nitro
karpenter.k8s.aws/instance-memory=16384
karpenter.k8s.aws/instance-network-bandwidth=2500
karpenter.k8s.aws/instance-size=2xlarge
karpenter.sh/capacity-type=on-demand
karpenter.sh/nodepool=default
kubernetes.io/arch=amd64
kubernetes.io/os=linux
node.kubernetes.io/instance-type=c5a.2xlarge
topology.k8s.aws/zone-id=apne2-az1
topology.kubernetes.io/region=ap-northeast-2
topology.kubernetes.io/zone=ap-northeast-2a
Annotations: compatibility.karpenter.k8s.aws/cluster-name-tagged: true
karpenter.k8s.aws/ec2nodeclass-hash: 7257177872106214015
karpenter.k8s.aws/ec2nodeclass-hash-version: v4
karpenter.k8s.aws/tagged: true
karpenter.sh/nodepool-hash: 6821555240594823858
karpenter.sh/nodepool-hash-version: v3
API Version: karpenter.sh/v1
Kind: NodeClaim
Metadata:
Creation Timestamp: 2025-03-08T08:14:35Z
Finalizers:
karpenter.sh/termination
Generate Name: default-
Generation: 1
Owner References:
API Version: karpenter.sh/v1
Block Owner Deletion: true
Kind: NodePool
Name: default
UID: fed3ee12-cde3-45c3-ac2e-fd640a513a3a
Resource Version: 23535
UID: ece9fcc1-4677-47e8-bd23-1f49dcc0eed5
Spec:
Expire After: 720h
Node Class Ref:
Group: karpenter.k8s.aws
Kind: EC2NodeClass
Name: default
Requirements:
Key: node.kubernetes.io/instance-type
Operator: In
Values:
c4.2xlarge
c4.4xlarge
c5.2xlarge
c5.4xlarge
c5a.2xlarge
c5a.4xlarge
c5a.8xlarge
c5d.2xlarge
c5d.4xlarge
c5n.2xlarge
c5n.4xlarge
c6i.2xlarge
c6i.4xlarge
c6id.2xlarge
c6id.4xlarge
c6in.2xlarge
c6in.4xlarge
c7i-flex.2xlarge
c7i-flex.4xlarge
c7i.2xlarge
c7i.4xlarge
m4.2xlarge
m4.4xlarge
m5.2xlarge
m5.4xlarge
m5a.2xlarge
m5a.4xlarge
m5ad.2xlarge
m5ad.4xlarge
m5d.2xlarge
m5d.4xlarge
m5zn.2xlarge
m5zn.3xlarge
m6i.2xlarge
m6i.4xlarge
m6id.2xlarge
m6id.4xlarge
m7i-flex.2xlarge
m7i-flex.4xlarge
m7i.2xlarge
m7i.4xlarge
r3.2xlarge
r4.2xlarge
r4.4xlarge
r5.2xlarge
r5.4xlarge
r5a.2xlarge
r5a.4xlarge
r5ad.2xlarge
r5ad.4xlarge
r5b.2xlarge
r5d.2xlarge
r5d.4xlarge
r5dn.2xlarge
r5n.2xlarge
r6i.2xlarge
r6i.4xlarge
r6id.2xlarge
r7i.2xlarge
r7i.4xlarge
Key: kubernetes.io/os
Operator: In
Values:
linux
Key: karpenter.sh/capacity-type
Operator: In
Values:
on-demand
Key: karpenter.k8s.aws/instance-category
Operator: In
Values:
c
m
r
Key: karpenter.k8s.aws/instance-generation
Operator: Gt
Values:
2
Key: kubernetes.io/arch
Operator: In
Values:
amd64
Key: karpenter.sh/nodepool
Operator: In
Values:
default
Key: karpenter.k8s.aws/ec2nodeclass
Operator: In
Values:
default
Resources:
Requests:
Cpu: 5150m
Pods: 9
Status:
Allocatable:
Cpu: 7910m
Ephemeral - Storage: 17Gi
Memory: 14162Mi
Pods: 58
vpc.amazonaws.com/pod-eni: 38
Capacity:
Cpu: 8
Ephemeral - Storage: 20Gi
Memory: 15155Mi
Pods: 58
vpc.amazonaws.com/pod-eni: 38
Conditions:
Last Transition Time: 2025-03-08T08:14:37Z
Message:
Observed Generation: 1
Reason: Launched
Status: True
Type: Launched
Last Transition Time: 2025-03-08T08:14:54Z
Message:
Observed Generation: 1
Reason: Registered
Status: True
Type: Registered
Last Transition Time: 2025-03-08T08:15:04Z
Message:
Observed Generation: 1
Reason: Initialized
Status: True
Type: Initialized
Last Transition Time: 2025-03-08T08:16:08Z
Message:
Observed Generation: 1
Reason: Consolidatable
Status: True
Type: Consolidatable
Last Transition Time: 2025-03-08T08:15:04Z
Message:
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Image ID: ami-089f1bf55c5291efd
Last Pod Event Time: 2025-03-08T08:15:08Z
Node Name: ip-192-168-104-216.ap-northeast-2.compute.internal
Provider ID: aws:///ap-northeast-2a/i-0166a9530235c41bd
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Launched 8m36s karpenter Status condition transitioned, Type: Launched, Status: Unknown -> True, Reason: Launched
Normal DisruptionBlocked 8m33s karpenter Nodeclaim does not have an associated node
Normal Registered 8m19s karpenter Status condition transitioned, Type: Registered, Status: Unknown -> True, Reason: Registered
Normal Initialized 8m9s karpenter Status condition transitioned, Type: Initialized, Status: Unknown -> True, Reason: Initialized
Normal Ready 8m9s karpenter Status condition transitioned, Type: Ready, Status: Unknown -> True, Reason: Ready
Normal Unconsolidatable 7m3s karpenter Can't replace with a cheaper node
21. ๋ฐฐํฌ๋ ๋ ธ๋ ํ๊ทธ ํ์ธ
1
kubectl get node -l karpenter.sh/registered=true -o jsonpath="{.items[0].metadata.labels}" | jq '.'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "c5a.2xlarge",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "ap-northeast-2",
"failure-domain.beta.kubernetes.io/zone": "ap-northeast-2a",
"k8s.io/cloud-provider-aws": "0e7edd29c7c0d27647d5edc662e2959d",
"karpenter.k8s.aws/ec2nodeclass": "default",
"karpenter.k8s.aws/instance-category": "c",
"karpenter.k8s.aws/instance-cpu": "8",
"karpenter.k8s.aws/instance-cpu-manufacturer": "amd",
"karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz": "3300",
"karpenter.k8s.aws/instance-ebs-bandwidth": "3170",
"karpenter.k8s.aws/instance-encryption-in-transit-supported": "true",
"karpenter.k8s.aws/instance-family": "c5a",
"karpenter.k8s.aws/instance-generation": "5",
"karpenter.k8s.aws/instance-hypervisor": "nitro",
"karpenter.k8s.aws/instance-memory": "16384",
"karpenter.k8s.aws/instance-network-bandwidth": "2500",
"karpenter.k8s.aws/instance-size": "2xlarge",
"karpenter.sh/capacity-type": "on-demand",
"karpenter.sh/initialized": "true",
"karpenter.sh/nodepool": "default",
"karpenter.sh/registered": "true",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-192-168-104-216.ap-northeast-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "c5a.2xlarge",
"topology.k8s.aws/zone-id": "apne2-az1",
"topology.kubernetes.io/region": "ap-northeast-2",
"topology.kubernetes.io/zone": "ap-northeast-2a"
}
- ์ ๊ท ๋ฐฐํฌ๋ ์ธ์คํด์ค์ ํ๊ทธ ์ ๋ณด๋ฅผ ํ์ธํจ
22. ์ค์ผ์ผ ์์ ์งํ
- inflate Deployment์ Replica ์๋ฅผ 30์ผ๋ก ํ์ฅํจ
1
2
3
kubectl scale deployment inflate --replicas 30
# ๊ฒฐ๊ณผ
deployment.apps/inflate scaled
23. CreateFleet ์ด๋ฒคํธ ํ์ธ ๋ฐ ์ต์ ์ธ์คํด์ค ์ ์
- ์์ฒญ ํ๋ผ๋ฏธํฐ๋ ์จ๋๋งจ๋ ์ธ์คํด์ค์ ๋ํด โlowest-priceโ ํ ๋น ์ ๋ต๊ณผ ๋ชฉํ ์ฉ๋ ์ง์ (TargetCapacitySpecification)์ ํฌํจํจ
1
2
3
4
5
6
7
8
9
10
11
"requestParameters": {
"CreateFleetRequest": {
"TargetCapacitySpecification": {
"DefaultTargetCapacityType": "on-demand",
"TotalTargetCapacity": 1
},
"Type": "instant",
"OnDemandOptions": {
"AllocationStrategy": "lowest-price"
},
...
- ์ธ์คํด์ค ์ ํ์ด โc5a.2xlargeโ๋ก, ap-northeast-2a ์์ญ์์ ์์ฑ๋ ๊ฒ์ ํ์ธํ ์ ์์
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
"responseElements": {
"CreateFleetResponse": {
"fleetInstanceSet": {
"item": {
"lifecycle": "on-demand",
"instanceIds": {
"item": "i-0166a9530235c41bd"
},
"instanceType": "c5a.2xlarge",
"launchTemplateAndOverrides": {
"overrides": {
"subnetId": "subnet-034bd815740cf5569",
"imageId": "ami-089f1bf55c5291efd",
"instanceType": "c5a.2xlarge",
"availabilityZone": "ap-northeast-2a"
},
"launchTemplateSpecification": {
"launchTemplateId": "lt-05d9b8217f57246a8",
"version": 1
}
}
}
},
"xmlns": "http://ec2.amazonaws.com/doc/2016-11-15/",
"requestId": "deedb0a3-693c-446d-86c7-f124525c998e",
"fleetId": "fleet-f1063486-3db7-c91e-84b8-06009123cc8a",
"errorSet": ""
}
},
24. ์ค์ผ์ผ ๋ค์ด ๋ฐ ๋ ธ๋ ์ ๋ฆฌ
(1) Deployment ์ญ์ (์ค์ผ์ผ ๋ค์ด)
1
2
3
4
kubectl delete deployment inflate && date
# ๊ฒฐ๊ณผ
deployment.apps "inflate" deleted
Sat Mar 8 05:43:59 PM KST 2025
(2) NodeClaim ํ์ธ
1
kubectl get nodeclaims
โ ย ์ถ๋ ฅ
1
2
NAME TYPE CAPACITY ZONE NODE READY AGE
default-sbszr c5a.2xlarge on-demand ap-northeast-2a ip-192-168-104-216.ap-northeast-2.compute.internal True 31m
(3) CloudTrail ๋ก๊ทธ ํ์ธ
- CloudTrail์์ Karpenter์ ์ํด ๋ ธ๋๊ฐ ์ข ๋ฃ๋ ์ด๋ฒคํธ๋ฅผ ํ์ธํ ์ ์์
25. Spot-to-Spot Consolidation
(1) ๊ธฐ์กด nodepool ์ญ์
1
2
3
4
kubectl delete nodepool,ec2nodeclass default
# ๊ฒฐ๊ณผ
nodepool.karpenter.sh "default" deleted
ec2nodeclass.karpenter.k8s.aws "default" deleted
(2) ์ NodePool ์์ฑ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
cat <<EOF | envsubst | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
- key: kubernetes.io/os
operator: In
values: ["linux"]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-size
operator: NotIn
values: ["nano","micro","small","medium"]
- key: karpenter.k8s.aws/instance-hypervisor
operator: In
values: ["nitro"]
expireAfter: 1h # nodes are terminated automatically after 1 hour
limits:
cpu: "1000"
memory: 1000Gi
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized # policy enables Karpenter to replace nodes when they are either empty or underutilized
consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: default
spec:
role: "KarpenterNodeRole-${CLUSTER_NAME}" # replace with your cluster name
amiSelectorTerms:
- alias: "bottlerocket@latest"
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: "${CLUSTER_NAME}" # replace with your cluster name
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: "${CLUSTER_NAME}" # replace with your cluster name
EOF
# ๊ฒฐ๊ณผ
nodepool.karpenter.sh/default created
ec2nodeclass.karpenter.k8s.aws/default created
(3) ์์ฑ๋ NodePool ํ์ธ
1
kubectl get nodepool,ec2nodeclass
โ ย ์ถ๋ ฅ
1
2
3
4
5
NAME NODECLASS NODES READY AGE
nodepool.karpenter.sh/default default 0 True 103s
NAME READY AGE
ec2nodeclass.karpenter.k8s.aws/default True 103s
(4) ์ํ Deployment ์์ฑ (inflate)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 5
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: 1
memory: 1.5Gi
securityContext:
allowPrivilegeEscalation: false
EOF
# ๊ฒฐ๊ณผ
deployment.apps/inflate created
(5) Deployment ์ค์ผ์ผ ์์ (Replica 12๋ก ์ฆ๊ฐ)
1
2
3
4
kubectl scale deployment/inflate --replicas 12
# ๊ฒฐ๊ณผ
deployment.apps/inflate scaled
(6) NodeClaim ์ํ ํ์ธ
1
kubectl get nodeclaims
โ ย ์ถ๋ ฅ
1
2
3
NAME TYPE CAPACITY ZONE NODE READY AGE
default-25frs c7i-flex.2xlarge spot ap-northeast-2d ip-192-168-173-102.ap-northeast-2.compute.internal True 3m46s
default-62j8w c7i-flex.2xlarge spot ap-northeast-2d ip-192-168-75-113.ap-northeast-2.compute.internal True 90s
(7) NodeClaim ๋ก๊ทธ ํ์ธ
1
kubectl logs -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller | grep 'launched nodeclaim' | jq '.'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
{
"level": "INFO",
"time": "2025-03-08T09:12:33.578Z",
"logger": "controller",
"message": "launched nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-25frs"
},
"namespace": "",
"name": "default-25frs",
"reconcileID": "192608af-43da-481c-a4a1-37c2b68a04f7",
"provider-id": "aws:///ap-northeast-2d/i-06d0fd600455c0855",
"instance-type": "c7i-flex.2xlarge",
"zone": "ap-northeast-2d",
"capacity-type": "spot",
"allocatable": {
"cpu": "7910m",
"ephemeral-storage": "17Gi",
"memory": "14162Mi",
"pods": "58",
"vpc.amazonaws.com/pod-eni": "18"
}
}
{
"level": "INFO",
"time": "2025-03-08T09:14:49.820Z",
"logger": "controller",
"message": "launched nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-62j8w"
},
"namespace": "",
"name": "default-62j8w",
"reconcileID": "b9b2fc42-f72c-42b1-9e19-d12aec4f9997",
"provider-id": "aws:///ap-northeast-2d/i-06b608eaa760c3e36",
"instance-type": "c7i-flex.2xlarge",
"zone": "ap-northeast-2d",
"capacity-type": "spot",
"allocatable": {
"cpu": "7910m",
"ephemeral-storage": "17Gi",
"memory": "14162Mi",
"pods": "58",
"vpc.amazonaws.com/pod-eni": "18"
}
}
(8) Deployment ์ค์ผ์ผ ๋ค์ด (Replica 5๋ก ๊ฐ์)
1
2
3
kubectl scale deployment/inflate --replicas 5
# ๊ฒฐ๊ณผ
deployment.apps/inflate scaled
(9) Karpenter ์ปจํธ๋กค๋ฌ ๋ก๊ทธ ์ค์๊ฐ ํ์ธ
1
kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller | jq '.'
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
{
"level": "INFO",
"time": "2025-03-08T07:36:28.990Z",
"logger": "controller",
"message": "starting server",
"commit": "058c665",
"name": "health probe",
"addr": "[::]:8081"
}
{
"level": "INFO",
"time": "2025-03-08T07:36:28.990Z",
"logger": "controller.controller-runtime.metrics",
"message": "Starting metrics server",
"commit": "058c665"
}
{
"level": "INFO",
"time": "2025-03-08T07:36:28.990Z",
"logger": "controller.controller-runtime.metrics",
"message": "Serving metrics server",
"commit": "058c665",
"bindAddress": ":8080",
"secure": false
}
{
"level": "INFO",
"time": "2025-03-08T07:36:29.091Z",
"logger": "controller",
"message": "attempting to acquire leader lease kube-system/karpenter-leader-election...",
"commit": "058c665"
}
{
"level": "INFO",
"time": "2025-03-08T09:14:46.889Z",
"logger": "controller",
"message": "found provisionable pod(s)",
"commit": "058c665",
"controller": "provisioner",
"namespace": "",
"name": "",
"reconcileID": "be8e6d0a-c3ca-4f5a-8e85-c32d96409c6d",
"Pods": "default/inflate-75b9664d55-nz2vs, default/inflate-75b9664d55-qf72x, default/inflate-75b9664d55-k5q7x, default/inflate-75b9664d55-jf9fp, default/inflate-75b9664d55-4tqz4",
"duration": "78.131326ms"
}
{
"level": "INFO",
"time": "2025-03-08T09:14:46.890Z",
"logger": "controller",
"message": "computed new nodeclaim(s) to fit pod(s)",
"commit": "058c665",
"controller": "provisioner",
"namespace": "",
"name": "",
"reconcileID": "be8e6d0a-c3ca-4f5a-8e85-c32d96409c6d",
"nodeclaims": 1,
"pods": 5
}
{
"level": "INFO",
"time": "2025-03-08T09:14:46.904Z",
"logger": "controller",
"message": "created nodeclaim",
"commit": "058c665",
"controller": "provisioner",
"namespace": "",
"name": "",
"reconcileID": "be8e6d0a-c3ca-4f5a-8e85-c32d96409c6d",
"NodePool": {
"name": "default"
},
"NodeClaim": {
"name": "default-62j8w"
},
"requests": {
"cpu": "5150m",
"memory": "7680Mi",
"pods": "9"
},
"instance-types": "c5.2xlarge, c5.4xlarge, c5a.2xlarge, c5a.4xlarge, c5d.4xlarge and 55 other(s)"
}
{
"level": "INFO",
"time": "2025-03-08T09:14:49.820Z",
"logger": "controller",
"message": "launched nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-62j8w"
},
"namespace": "",
"name": "default-62j8w",
"reconcileID": "b9b2fc42-f72c-42b1-9e19-d12aec4f9997",
"provider-id": "aws:///ap-northeast-2d/i-06b608eaa760c3e36",
"instance-type": "c7i-flex.2xlarge",
"zone": "ap-northeast-2d",
"capacity-type": "spot",
"allocatable": {
"cpu": "7910m",
"ephemeral-storage": "17Gi",
"memory": "14162Mi",
"pods": "58",
"vpc.amazonaws.com/pod-eni": "18"
}
}
{
"level": "INFO",
"time": "2025-03-08T09:15:02.806Z",
"logger": "controller",
"message": "registered nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-62j8w"
},
"namespace": "",
"name": "default-62j8w",
"reconcileID": "98a8f8be-fa9c-4b9e-9326-fc1f0fa48805",
"provider-id": "aws:///ap-northeast-2d/i-06b608eaa760c3e36",
"Node": {
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal"
}
}
{
"level": "INFO",
"time": "2025-03-08T09:15:13.815Z",
"logger": "controller",
"message": "initialized nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-62j8w"
},
"namespace": "",
"name": "default-62j8w",
"reconcileID": "d5b9c913-61ad-408b-b58c-d9d502fc0128",
"provider-id": "aws:///ap-northeast-2d/i-06b608eaa760c3e36",
"Node": {
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal"
},
"allocatable": {
"cpu": "7910m",
"ephemeral-storage": "18191325562",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "15038356Ki",
"pods": "58"
}
}
{
"level": "INFO",
"time": "2025-03-08T09:17:15.795Z",
"logger": "controller",
"message": "disrupting nodeclaim(s) via delete, terminating 1 nodes (5 pods) ip-192-168-75-113.ap-northeast-2.compute.internal/c7i-flex.2xlarge/spot",
"commit": "058c665",
"controller": "disruption",
"namespace": "",
"name": "",
"reconcileID": "9384814e-9766-4e8d-8674-3f9ab20fe294",
"command-id": "0586cd24-7ecf-49f1-90dd-f8f7b99ab85a",
"reason": "underutilized"
}
{
"level": "INFO",
"time": "2025-03-08T09:17:16.699Z",
"logger": "controller",
"message": "tainted node",
"commit": "058c665",
"controller": "node.termination",
"controllerGroup": "",
"controllerKind": "Node",
"Node": {
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal"
},
"namespace": "",
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal",
"reconcileID": "653758ba-6f7d-4f3b-a684-e6127056300e",
"taint.Key": "karpenter.sh/disrupted",
"taint.Value": "",
"taint.Effect": "NoSchedule"
}
{
"level": "INFO",
"time": "2025-03-08T09:18:26.567Z",
"logger": "controller",
"message": "deleted node",
"commit": "058c665",
"controller": "node.termination",
"controllerGroup": "",
"controllerKind": "Node",
"Node": {
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal"
},
"namespace": "",
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal",
"reconcileID": "9969d8c3-171d-4ab2-9146-59c38d0328c4"
}
{
"level": "INFO",
"time": "2025-03-08T09:18:26.805Z",
"logger": "controller",
"message": "deleted nodeclaim",
"commit": "058c665",
"controller": "nodeclaim.lifecycle",
"controllerGroup": "karpenter.sh",
"controllerKind": "NodeClaim",
"NodeClaim": {
"name": "default-62j8w"
},
"namespace": "",
"name": "default-62j8w",
"reconcileID": "41e08b5c-b338-46e0-a810-5084d2543888",
"Node": {
"name": "ip-192-168-75-113.ap-northeast-2.compute.internal"
},
"provider-id": "aws:///ap-northeast-2d/i-06b608eaa760c3e36"
}
(10) Deployment ์ค์ผ์ผ ๋ค์ด (Replica 1๋ก ๊ฐ์)
1
2
3
4
kubectl scale deployment/inflate --replicas 1
# ๊ฒฐ๊ณผ
deployment.apps/inflate scaled
(11) ์ต์ข NodeClaim ํ์ธ
1
kubectl get nodeclaims
โ ย ์ถ๋ ฅ
1
2
NAME TYPE CAPACITY ZONE NODE READY AGE
default-25frs c7i-flex.2xlarge spot ap-northeast-2d ip-192-168-173-102.ap-northeast-2.compute.internal True 8m19s
๐๏ธ (์ค์ต ์๋ฃ ํ) ์์ ์ญ์
1. Karpenter helm ์ญ์
1
2
3
helm uninstall karpenter --namespace "${KARPENTER_NAMESPACE}"
# ๊ฒฐ๊ณผ
release "karpenter" uninstalled
2. Karpenter IAM Role ๋ฑ ์์ฑํ CloudFormation ์ญ์
1
aws cloudformation delete-stack --stack-name "Karpenter-${CLUSTER_NAME}"
3. EC2 Launch Template ์ญ์
1
2
3
aws ec2 describe-launch-templates --filters "Name=tag:karpenter.k8s.aws/cluster,Values=${CLUSTER_NAME}" |
jq -r ".LaunchTemplates[].LaunchTemplateName" |
xargs -I{} aws ec2 delete-launch-template --launch-template-name {}
4. ํด๋ฌ์คํฐ ์ญ์
- ํด๋ฌ์คํฐ ์ญ์ ์ดํ์๋, Karpenter IAM Role ์์ฑํ CloudFormation ์ญ์ ๊ฐ ์ ์๋ ๊ฒฝ์ฐ AWS CloudFormation ๊ด๋ฆฌ ์ฝ์์์ ์ง์ ์ญ์
1
eksctl delete cluster --name "${CLUSTER_NAME}"
โ ย ์ถ๋ ฅ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
2025-03-08 18:24:05 [โน] deleting EKS cluster "gagajin-karpenter-demo"
2025-03-08 18:24:05 [โน] will drain 0 unmanaged nodegroup(s) in cluster "gagajin-karpenter-demo"
2025-03-08 18:24:05 [โน] starting parallel draining, max in-flight of 1
2025-03-08 18:24:06 [โน] deleted 0 Fargate profile(s)
2025-03-08 18:24:06 [โ] kubeconfig has been updated
2025-03-08 18:24:06 [โน] cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2025-03-08 18:24:33 [โน]
5 sequential tasks: { delete nodegroup "gagajin-karpenter-demo-ng", delete IAM OIDC provider, delete addon IAM "eksctl-gagajin-karpenter-demo-addon-vpc-cni", deleting IAM resources stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter", delete cluster control plane "gagajin-karpenter-demo" [async]
}
2025-03-08 18:24:33 [โน] will delete stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:24:33 [โน] waiting for stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng" to get deleted
2025-03-08 18:24:33 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
Handling connection for 3000
E0308 18:24:47.985519 202336 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod 1c5446ac40bdda929ba9e41299e84b9ba76306a71d67f1f5d5200312a0e32a66, uid : network namespace for sandbox \"1c5446ac40bdda929ba9e41299e84b9ba76306a71d67f1f5d5200312a0e32a66\" is closed"
error: lost connection to pod
2025-03-08 18:25:04 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:26:02 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
Handling connection for 9090
E0308 18:27:07.509260 198328 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod a7d027394b53b0faa7cf416b83a796e180602f4ad2815db4bb2cc2bd38d2e999, uid : failed to find sandbox \"a7d027394b53b0faa7cf416b83a796e180602f4ad2815db4bb2cc2bd38d2e999\" in store: not found"
error: lost connection to pod
2025-03-08 18:27:49 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:29:02 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:30:35 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:31:09 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:32:32 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:33:57 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-nodegroup-gagajin-karpenter-demo-ng"
2025-03-08 18:33:58 [โน] will delete stack "eksctl-gagajin-karpenter-demo-addon-vpc-cni"
2025-03-08 18:33:58 [โน] will delete stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 18:33:58 [โน] waiting for stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter" to get deleted
2025-03-08 18:33:58 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 18:34:29 [โน] waiting for CloudFormation stack "eksctl-gagajin-karpenter-demo-podidentityrole-kube-system-karpenter"
2025-03-08 18:34:29 [โน] will delete stack "eksctl-gagajin-karpenter-demo-cluster"
2025-03-08 18:34:29 [โ] all cluster resources were deleted
[1]- Exit 1 kubecolor --namespace monitoring port-forward $POD_NAME 9090
[2]+ Exit 1 kubecolor port-forward --namespace monitoring svc/grafana 3000:80
This post is licensed under CC BY 4.0 by the author.