{"id":679,"date":"2020-12-02T10:21:30","date_gmt":"2020-12-02T09:21:30","guid":{"rendered":"https:\/\/www.ume.li\/blog\/?p=679"},"modified":"2020-12-02T15:57:35","modified_gmt":"2020-12-02T14:57:35","slug":"monitoring-component-pack-kubernetes-with-prometheus","status":"publish","type":"post","link":"https:\/\/www.ume.li\/blog\/2020\/12\/02\/monitoring-component-pack-kubernetes-with-prometheus\/","title":{"rendered":"Monitoring Component Pack \/ Kubernetes with Prometheus – Grafana"},"content":{"rendered":"

You have successfully installed HCL ComponentPack for HCL Connections. Now you want to know what is happening in that black box ?
\nThe goal is to see something like this
\n\"\"<\/a><\/p>\n

Steps to reproduce:<\/strong>
\n– Update Kubernetes to 1.18.12<\/a>
\n– Install Helm3<\/a>
\n– Install nfs-client<\/a>
\n– install metrics
\n– Install Traefik<\/a>
\n– Install Prometheus <\/p>\n

Disclaimer: use the following on your own risk.<\/p>\n

Update Kubernetes to 1.18 because it’s now supported for Component Pack. If you are still on 1.11 you will be surprised after the update to 1.16. The coredns will no longer start. Coredns removed the proxy plugin, you need to switch to the forward plugin:
\nkubectl -n kube-system edit cm coredns<\/code>
\nchange ‘proxy . \/etc\/resolv.conf’ to ‘forward . \/etc\/resolv.conf’.
\nGoing to 1.19 has not been an option. Some of the helm charts provide incompatible ingresss or services due to the api changes between 1.18 and 1.19<\/p>\n

Installing Helm 3<\/strong>
\nAt the moment the Component Pack still requires Helm 2. But both versions can be used in parallel.
\nBefore installing Helm 3:
\nmv \/usr\/local\/bin\/helm \/usr\/local\/bin\/helm2<\/code>
\nand after installing Helm 3:
\nmv \/usr\/local\/bin\/helm \/usr\/local\/bin\/helm3<\/code>
\nNow my environment knows helm2 and helm3. It’s possible to migrate from helm 2 to helm 3 but I wanted an emergency option to be able to re-install Component Pack if needed. helm2 list<\/code> shows the Component Pack installs. helm3 list<\/code> will show all the new stuff.<\/p>\n

nfs-client: <\/strong>
\nI’m lazy. I already have a working nfs server in my environment. I don’t want to handle every single pv\/pvc manually. That’s why I use the nfs client provisioner. I cloned the repo and applied the 3 yaml files. I updated the deploy\\deployment.yaml with the values for my nfs server.
\nkubectl apply -f deploy\\class.yaml
\nkubectl apply -f deploy\\deployment.yaml
\nkubectl apply -f deploy\\rbac.yaml
\nkubectl patch storageclass managed-nfs-storage -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io\/is-default-class\":\"true\"}}}'
\n<\/code>
\nThe last line defines the new storage class as default.<\/p>\n

metrics: <\/strong>
\nget the yaml and add the –kubelet-insecure-tls to the args.
\nI was not able to get the metrics pod up without this. Probably proper certificates with all the right SAN’s would help. But as this is only in my lab, I don’t care.<\/p>\n


\nwget https:\/\/github.com\/kubernetes-sigs\/metrics-server\/releases\/download\/v0.4.1\/components.yaml
\n<\/code>
\n
\n - args:
\n - --cert-dir=\/tmp
\n - --secure-port=4443
\n - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
\n - --kubelet-use-node-status-port
\n - --kubelet-insecure-tls
\nkubectl apply -f components.yaml
\n<\/code>
\nNow you should get the metrics pod in the kube-system namespace.<\/p>\n

Traefik: <\/strong>
\nhelm3 show values traefik\/traefik > values.yml<\/code> to get the variables from the helm chart.<\/p>\n

In order to prevent conflicts between the different ingress controllers I added this ingressClass.
\nSave it in ic.yaml and kubectl apply -f ic.yaml<\/code><\/p>\n

apiVersion: networking.k8s.io\/v1beta1
\nkind: IngressClass
\nmetadata:
\n name: traefik-lb
\nspec:
\n controller: traefik.io\/ingress-controller
\n<\/code><\/p>\n

There are 2 ingress controllers on my system now Traefik and cnx-ingress.<\/p>\n

I defined the listening ports for web and websecure as 31080 and 31443 with
\nkubectl edit svc traefik<\/code>.<\/p>\n

There’s also a dashboard available for traefik.
\n\"\"<\/a><\/p>\n

Prometheus\/Grafana<\/strong>
\nThere are a lot of tutorials out there on how to install and configure prometheus.<\/p>\n

So the compressed version:<\/p>\n


\nhelm3 repo add stable https:\/\/charts.helm.sh\/stable
\nhelm3 repo add prometheus-community https:\/\/prometheus-community.github.io\/helm-charts
\nhelm3 repo update
\nhelm3 show values prometheus-community\/kube-prometheus-stack > values.yaml
\n<\/code>
\nedit values.yaml file:
\n – enable the ingress creation, add the ingressClassName: traefik-lb and define the hostname
\n – assign the volumeclaimtemplates the storage class, managed-nfs-storage in my case
\n
\n storage:
\n volumeClaimTemplate:
\n spec:
\n storageClassName: managed-nfs-storage
\n accessModes: [\"ReadWriteOnce\"]
\n resources:
\n requests:
\n storage: 20Gi
\n # selector: {}
\n<\/code><\/p>\n


\nkubectl create namespace monitoring
\nhelm3 install prometheus prometheus-community\/kube-prometheus-stack -n monitoring -f values.yaml
\nhelm3 install metrics-adapter prometheus-community\/prometheus-adapter -n monitoring<\/p>\n

<\/code>
\nupdate the grafana ingress, add the ingressClassName
\nkubectl -n monitoring edit ingress prometheus-grafana<\/code>
\n
\nspec:
\n ingressClassName: traefik-lb
\n rules:
\n - host: grafana.ume.li
\n<\/code><\/p>\n

If you are using more then one node, make sure that the appropriate network ports (9000\/9100 TCP) are open between the nodes.<\/p>\n

Next step is to create the routes on your front Proxy\/LoadBalancer so that the site is available under a nice url.
\nThe embedded grafana has already the prometheus configured. The only thing I added was this dashboard<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

You have successfully installed HCL ComponentPack for HCL Connections. Now you want to know what is happening in that black<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[54,23],"tags":[],"class_list":["post-679","post","type-post","status-publish","format-standard","hentry","category-component-pack","category-connections"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/posts\/679","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/comments?post=679"}],"version-history":[{"count":23,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/posts\/679\/revisions"}],"predecessor-version":[{"id":706,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/posts\/679\/revisions\/706"}],"wp:attachment":[{"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/media?parent=679"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/categories?post=679"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ume.li\/blog\/wp-json\/wp\/v2\/tags?post=679"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}