本文记录基于 KEDA 和 WorkloadSpread 的弹性 HPA 实验过程
准备集群
本文通过 kind 创建一个四节点集群环境:
Linux amd64 机器安装 kind 命令:
# curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.12.0/kind-linux-amd64
# chmod +x ./kind
# mv ./kind /usr/bin/kind
初始化集群:
# cat << EOF > cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
# kind create cluster --config cluster.yaml
安装 KEDA
# helm repo add kedacore https://kedacore.github.io/charts
# helm repo update
# kubectl create namespace keda
# helm install keda kedacore/keda --namespace keda
安装 OpenKruise
# helm repo add openkruise https://openkruise.github.io/charts/
# helm repo update
# helm install kruise openkruise/kruise --version 1.0.1 --set featureGates="WorkloadSpread=true"
安装 Ingress-Nginx-Controller
# kubectl create ns ingress-nginx
# cat << EOF | kubectl apply -f -
apiVersion: v1
data:
allow-snippet-annotations: "true"
http-snippet: |
server {
listen 8080;
server_name _ ;
location /stub_status {
stub_status on;
}
location / {
return 404;
}
}
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 1.1.0
helm.sh/chart: ingress-nginx-4.0.13
name: ingress-nginx-controller
namespace: ingress-nginx
EOF
# cat << EOF > values.yaml
controller:
containerPort:
http: 80
https: 443
status: 8080
EOF
# helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --values values.yaml
# cat << EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx-controller-8080
namespace: ingress-nginx
spec:
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP
ports:
- name: myapp
port: 8080
targetPort: status
EOF
安装 Nginx-Prometheus-Exporter
# cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-exporter
namespace: ingress-nginx
labels:
app: ingress-nginx-exporter
spec:
selector:
matchLabels:
app: ingress-nginx-exporter
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: ingress-nginx-exporter
spec:
containers:
- image: nginx/nginx-prometheus-exporter:0.10
imagePullPolicy: IfNotPresent
args:
- -nginx.scrape-uri=http://ingress-nginx-controller-8080.ingress-nginx.svc.cluster.local:8080/stub_status
name: main
ports:
- name: http
containerPort: 9113
protocol: TCP
resources:
limits:
cpu: "200m"
memory: "256Mi"
EOF
安装 Prometheus-Operator
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# helm repo update
# helm install kube-prometheus-stack-1640678515 prometheus-community/kube-prometheus-stack --namespace prometheus --create-namespace
# cat << EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
labels:
release: kube-prometheus-stack-1640678515
name: ingress-nginx-monitor
namespace: ingress-nginx
spec:
selector:
matchLabels:
app: ingress-nginx-exporter
podMetricsEndpoints:
- port: http
EOF
测试环境是否正确
# kubectl run --rm -i --tty busybox --image=odise/busybox-curl --restart=Never -- sh
/ # curl -L http://ingress-nginx-controller-8080.ingress-nginx.svc.cluster.local:8080/stub_status
Active connections: 1
server accepts handled requests
182 182 181
Reading: 0 Writing: 1 Waiting: 0
如执行上述 curl 后输出类似内容,则表示接口正常
查看 grafana 面板
# kubectl port-forward service/kube-prometheus-stack-1640678515-grafana -n prometheus 30080:80
Forwarding from 127.0.0.1:30080 -> 3000
Forwarding from [::1]:30080 -> 3000
浏览器打开 http://localhost:30080
,用户名为 admin,密码为 prom-operator
部署测试服务
# cat << EOF | kubectl apply -f -
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
name: hello-web
namespace: ingress-nginx
labels:
app: hello-web
spec:
replicas: 1
selector:
matchLabels:
app: hello-web
template:
metadata:
labels:
app: hello-web
spec:
containers:
- name: hello-web
image: zhangsean/hello-web
ports:
- containerPort: 80
resources:
requests:
cpu: "1"
memory: "256Mi"
limits:
cpu: "2"
memory: "512Mi"
---
kind: Service
apiVersion: v1
metadata:
name: hello-web
namespace: ingress-nginx
spec:
type: ClusterIP
selector:
app: hello-web
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-web
namespace: ingress-nginx
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hello-web
port:
number: 80
ingressClassName: nginx
EOF
给节点打标:
# kubectl label node kind-worker2 resource-pool=fixed
# kubectl label node kind-worker3 resource-pool=elastic
部署 WorkloadSpread
扫描二维码关注公众号,回复:
17089213 查看本文章
# cat << EOF | kubectl apply -f -
apiVersion: apps.kruise.io/v1alpha1
kind: WorkloadSpread
metadata:
name: workloadspread-demo
namespace: ingress-nginx
spec:
targetRef:
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
name: hello-web
subsets:
- name: subset-a
requiredNodeSelectorTerm:
matchExpressions:
- key: resource-pool
operator: In
values:
- fixed
maxReplicas: 3
- name: subset-b
requiredNodeSelectorTerm:
matchExpressions:
- key: resource-pool
operator: In
values:
- elastic
scheduleStrategy:
type: Adaptive
EOF
优先向 subset-a 部署,超过最大值,则向 subset-b 部署
部署 ScaledObject
# cat << EOF | kubectl apply -f -
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: ingress-nginx-scaledobject
namespace: ingress-nginx
spec:
maxReplicaCount: 10
minReplicaCount: 1
pollingInterval: 10
cooldownPeriod: 2
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 10
scaleTargetRef:
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
name: hello-web
triggers:
- type: prometheus
metadata:
serverAddress: http://kube-prometheus-stack-1640-prometheus.prometheus:9090/
metricName: nginx_http_requests_total
query: sum(rate(nginx_http_requests_total[3m]))
threshold: '50'
EOF
压测
使用 go-stress-testing
(https://github.com/link1st/go-stress-testing/releases) 工具进行压测:
# kubectl run --rm -i --tty busybox --image=health/go-stress-testing --restart=Never -- sh
sh-4.2# go-stress-testing -c 100 -n 10000 -u http://ingress-nginx-controller.ingress-nginx
停止压测,看到 Pod 数量在减少:
Ref: https://openkruise.io/zh/docs/best-practices/elastic-deployment