我们在部署Airflow的时候,用的外部数据源,本文我们实践下在K8S上部署Redis单节点,后续的部署我们均会优先考虑K8S部署学习实践的方式
1. Windows10下安装Kubectl
我的电脑是Windows10,首先访问https://kubernetes.io/zh-cn/docs/tasks/tools/install-kubectl-windows/下载最新版,目前的最新版是https://dl.k8s.io/release/v1.26.0/bin/windows/amd64/kubectl.exe
2. 在Docker DeskTop中启用K8S
重启后执行如下命令查看可用节点
C:\Users\Administrator>kubectl get nodes
NAME STATUS ROLES AGE VERSION
docker-desktop Ready control-plane 77s v1.25.2
3. 创建命名空间middleware中间件
K8S命名空间类似Nacos命名空间,都是为了便于分类管理
kubectl create namespace middleware
4. 编写redis.yaml文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: middleware
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
project: redis
app: redis
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
revisionHistoryLimit: 5
template:
metadata:
labels:
project: redis
app: redis
spec:
containers:
- name: redis
image: redis:7.0.5-alpine3.16
imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: middleware
labels:
project: redis
app: redis
spec:
type: NodePort
selector:
project: redis
app: redis
ports:
- port: 6379
targetPort: 6379
nodePort: 30000
5. 运行并访问
kubectl apply -f redis.yaml
查看命名空间middleware下运行的pod,OK
kubectl get pod --namespace middleware
NAME READY STATUS RESTARTS AGE
redis-5d7f465bc8-kkxhc 1/1 Running 0 106s
现在Airflow就可以在配置该redis的链接了
# 内部容器访问
--set data.brokerUrl=redis://redis-pod.middleware:6379/0
# 或者直接IP:Port方式访问
--set data.brokerUrl=redis://127.0.0.1:30000/0
欢迎关注公众号算法小生与我沟通交流