k8s部署kube-prometheus
听说kube-prometheus开箱即用？
使用kind创建集群
安装好kind后直接kind create cluster --config config.yaml --name cluster启动集群，config.yaml如下，是一个有一个master两个node的测试集群。测试完成后kind delete cluster --name cluster即可删除集群。
# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
在尝试过minikube和kind后还是选择了kind。
部署kube-prometheus
一般在集群中部署Prometheus有三种方法：Prometheus Operator、kube-prometheus 和 community helm chart。
Prometheus Operator
The Prometheus Operator uses Kubernetes custom resources to simplify the deployment and configuration of Prometheus, Alertmanager, and related monitoring components.
kube-prometheus
kube-prometheus provides example configurations for a complete cluster monitoring stack based on Prometheus and the Prometheus Operator. This includes deployment of multiple Prometheus and Alertmanager instances, metrics exporters such as the node_exporter for gathering node metrics, scrape target configuration linking Prometheus to various metrics endpoints, and example alerting rules for notification of potential issues in the cluster.
helm chart
The prometheus-community/kube-prometheus-stack helm chart provides a similar feature set to kube-prometheus. This chart is maintained by the Prometheus community. For more information, please see the chart’s readme
kube-prometheus提供了许多开箱即用的配置，例如Alertmanager、node-exporter以及Grafana dashboards。
将kube-prometheus项目clone至本地，然后执行:
# Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/
由于创建镜像需要拉取的镜像托管在quay.io，会出现ImagePullBackOff错误：
k describe pods -n monitoring prometheus-operator-75d9b475d9-f29g8检查一下，是
- quay.io/brancz/kube-rbac-proxy:v0.11.0
- quay.io/prometheus-operator/prometheus-operator:v0.49.0
两个镜像拉取失败，这时借助阿里云容器镜像服务进行拉取。
- 创建GitHub repo，路径格式为：image-name/tag/Dockerfile
- 在阿里云容器镜像服务中创建仓库image-name
- 选择代码源为GitHub repo，选择海外构建
- 进入仓库，添加构建规则，立即构建
- 构建完成后登录到阿里云容器镜像服务：
docker login --username=xxx registry.cn-hangzhou.aliyuncs.com - 拉取构建好的镜像：
docker pull registry.cn-hangzhou.aliyuncs.com/chengleqi/prometheus-operator:v0.49.0 - 进行retag：
docker tag registry.cn-hangzhou.aliyuncs.com/chengleqi/prometheus-operator:v0.49.0 quay.io/prometheus-operator/prometheus-operator:v0.49.0 - 删除之前的镜像：
docker rmi registry.cn-hangzhou.aliyuncs.com/chengleqi/prometheus-operator:v0.49.0
brancz/kube-rbac-proxy:v0.11.0也重复上述步骤，此时就有了kube-operator所需的镜像。
使用kind load docker-image quay.io/brancz/kube-rbac-proxy:v0.11.0 quay.io/prometheus-operator/prometheus-operator:v0.49.0 --name cluster将镜像加载进kind集群，然后prometheus-operator-75d9b475d9-f29g8这个pod就正常运行了。
若还有镜像报错，重复上述步骤，直到所有pod都处于Running状态，到此kube-prometheus部署成功。
测试kube-prometheus
Prometheus
这里设置service的端口转发，接着就能进入Prometheus的UI尝试PromQL了。
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
grafana
同样设置端口转发
kubectl --namespace monitoring port-forward svc/grafana 3000
之后就可以尝试自带的面板了(例如node-exporter)：
Grafana不仅可以对接Prometheus，还可以对接zabbix等很多数据源，是一个非常漂亮和好用的可视化工具。