目录
3、Grafana监控配置--安装Dashboard相应的插件
一、Prometheus部署、配置、数据展示
Prometheus 是一个开源监控系统,它本身已经成为了云原生中指标监控的事实标准 。
1、Prometheus架构
-
Prometheus Server ,监控、告警平台核心,抓取目标端监控数据,生成聚合数据,存储时间序列数据
-
exporter,由被监控的对象提供,提供API暴漏监控对象的指标,供prometheus 抓取
-
node-exporter
-
blackbox-exporter
-
redis-exporter
-
mysql-exporter
-
custom-exporter
-
...
-
-
pushgateway,提供一个网关地址,外部数据可以推送到该网关,prometheus也会从该网关拉取数据
-
Alertmanager,接收Prometheus发送的告警并对于告警进行一系列的处理后发送给指定的目标
-
Grafana:配置数据源,图标方式展示数据
2、 Prometheus配置文件
部署文件如下:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitor
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# prometheus自身的指标
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# apiserver组件相关指标
# apiserver默认禁用http协议访问,https需要配置相关证书和token
- job_name: 'kubernetes-apiserver'
static_configs:
- targets: ['10.1.0.1']
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# kubelet组件相关指标
# 利用node服务发现和ssl证书认证
- job_name: 'kubernetes-sd-kubelet'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# node节点相关指标
# 利用node-exporter采集,插件默认采集kubelet的10250端口,需要replace为9100
- job_name: 'kubernetes-sd-node-exporter'
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:9100'
target_label: __address__
action: replace
# server服务的监控指标
# 利用endpoints服务发现,keep类似label标签,默认发现全部的endpoint,这里规定server必须包含
# annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
# 8080为server暴露出的监控端口
- job_name: 'kubernetes-sd-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
# contains服务的监控指标
# 具体所有容器的资源监控
- job_name: 'kubernetes-sd-cadvisor'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- target_label: __metrics_path__
replacement: /metrics/cadvisor
3、Prometheus部署
# 1、提前部署merics、dashboard、nginx-ingress,如果存在可忽略
# merics
$ kubectl create kube-merics.yaml
# dashboard
$ kubectl create -f kube-dashboard.yaml
# nginx-ingress
$ kubectl create -f nginx-ingress-clusterrole.yaml
# 2、必要组件,必须部署
# 命名空间
$ kubectl create prometheus-namespace.yaml
# 给node打上label
$ kubectl label node k8s-slave1 app=prometheus
#部署configmap
$ kubectl create -f prometheus-configmap.yaml
# rbac
$ kubectl create -f prometheus-rbac.yaml
# deployment
$ kubectl create -f prometheus-deployment.yaml
# service
$ kubectl create -f prometheus-svc.yaml
# ingress
$ kubectl create -f prometheus-ingress.yaml
# 访问测试
$ kubectl -n monitor get ingress
4、Prometheus数据展示
k8s集群状态查看:
[root@k8s-master ~]# kubectl get po -n monitor
NAME READY STATUS RESTARTS AGE
grafana-d7c4c4bf7-w4scz 1/1 Running 1 6d20h
kube-state-metrics-59f9c568fc-99gb2 1/1 Running 2 8d
node-exporter-78ffd 1/1 Running 2 8d
node-exporter-gtccl 1/1 Running 2 8d
node-exporter-gw4mv 1/1 Running 2 8d
prometheus-64987d8b68-bfqq2 1/1 Running 1 6d20h
Prometheus界面展示
注意需要本机hosts文件配置域名解析
二、Grafana部署与配置
可视化面板,功能齐全的度量仪表盘和图形编辑器,支持 Graphite、zabbix、InfluxDB、Prometheus、OpenTSDB、Elasticsearch 等作为数据源,比 Prometheus 自带的图表展示功能强大太多,更加灵活,有丰富的插件,功能更加强大。
1、Grafana安装
注意点:
-
使用最新版本的镜像: https://github.com/grafana/grafana
-
通过环境变量设置管理员账户密码
-
GF_SECURITY_ADMIN_USER
-
GF_SECURITY_ADMIN_PASSWORD
-
-
通过设置securityContext的方式让grafana进程使用root启动
-
数据挂载到本地
-
配置ingress暴露访问入口
[root@k8s-master k8s-primetheus]# cat prometheus-grafana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitor
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
volumes:
- name: storage
hostPath:
path: /data/grafana/
nodeSelector:
app: prometheus
securityContext:
runAsUser: 0
containers:
- name: grafana
image: grafana/grafana:7.1.1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
name: grafana
env:
- name: GF_SECURITY_ADMIN_USER
value: admin
- name: GF_SECURITY_ADMIN_PASSWORD
value: admin
readinessProbe:
failureThreshold: 10
httpGet:
path: /api/health
port: 3000
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/health
port: 3000
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 150m
memory: 512Mi
requests:
cpu: 150m
memory: 512Mi
volumeMounts:
- mountPath: /var/lib/grafana
name: storage
---
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: monitor
spec:
type: ClusterIP
ports:
- port: 3000
selector:
app: grafana
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: grafana
namespace: monitor
spec:
rules:
- host: gra.haha.com
http:
paths:
- path: /
backend:
serviceName: grafana
servicePort: 3000
注意本机hosts配置域名解析
[root@k8s-master k8s-primetheus]# kubectl get ing -n monitor
NAME CLASS HOSTS ADDRESS PORTS AGE
grafana <none> gra.haha.com 80 8d
prometheus <none> pro.haha.com 80 10d
2、Grafana监控配置--导入Dashboard的配置
grafana默认是空的,需要配置dashboared才能把prometheus采集的数据展示出来
配置数据源:URL:http://prometheus:9090
如何丰富Grafana监控面板:1、导入dashboard;2、安装相应的插件;3、自定义监控面板
我们开始利用导入现有的dashboard模版方式进行配置
导入Dashboard的配置
dashboard模板官方地址: Dashboards | Grafana Labshttps://grafana.com/grafana/dashboards
- 可以选择合适的模板导入,推荐以下两个
-
Node Exporter :https://grafana.com/grafana/dashboards/8919
-
Prometheus:https://grafana.com/grafana/dashboards/13105
- 导入方法:填写对应的UID
3、Grafana监控配置--安装Dashboard相应的插件
DevOpsProdigy KubeGraf 是一个非常优秀的 Grafana Kubernetes 插件,是 Grafana 官方的 Kubernetes 插件的升级版本,该插件可以用来可视化和分析 Kubernetes 集群的性能,通过各种图形直观的展示了 Kubernetes 集群的主要服务的指标和特征,还可以用于检查应用程序的生命周期和错误日志。
1、插件下载
# 进入grafana容器内部执行安装
$ kubectl -n monitor exec -ti grafana-594f447d6c-jmjsw bash
bash-5.0# grafana-cli plugins install devopsprodigy-kubegraf-app 1.4.1
installing devopsprodigy-kubegraf-app @ 1.4.1
from: https://grafana.com/api/plugins/devopsprodigy-kubegraf-app/versions/1.4.1/download
into: /var/lib/grafana/plugins
✔ Installed devopsprodigy-kubegraf-app successfully
Restart grafana after installing plugins . <service grafana-server restart>
bash-5.0# grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.5.0
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.5.0/download
into: /var/lib/grafana/plugins
✔ Installed grafana-piechart-panel successfully
Restart grafana after installing plugins . <service grafana-server restart>
# 也可以下载离线包进行安装
# 重建pod生效
$ kubectl -n monitor delete po grafana-594f447d6c-jmjsw
2、插件配置
登录grafana界面,Configuration -> Plugins 中找到安装的插件,点击插件进入插件详情页面,点击 [Enable]按钮启用插件,点击 `Set up your first k8s-cluster` 创建一个新的 Kubernetes 集群:
- Name:haha-k8s
- URL:https://kubernetes.default:443
- Access:使用默认的Server(default)
- Skip TLS Verify:勾选,跳过证书合法性校验
- Auth:勾选TLS Client Auth以及With CA Cert,勾选后会下面有三块证书内容需要填写,内容均来自`~/.kube/config`文件,需要对文件中的内容做一次base64 解码
- CA Cert:使用config文件中的`certificate-authority-data`对应的内容
- Client Cert:使用config文件中的`client-certificate-data`对应的内容
- Client Key:使用config文件中的`client-key-data`对应的内容
3、插件界面展示