Category: component pack

I had some issues with my componet pack pods and dns resolution.
The following command runs a ping to my internal ihs host (interservice url) on every pod.


kubectl get pods -n connections -o name | xargs -I{} kubectl -n connections exec {} -- sh -c "echo {} && ping ihs.interservice.url -c 1" | tee ping.log


Most of the connections pods display the ip. Some do actually ping, but on most pods, ping itself is prohibited.
Another option is to use nslookup


kubectl get pods -n connections -o name | xargs -I{} kubectl -n connections exec {} -- sh -c "echo {} && nslookup ihs.interservice.url " | tee nslookup.log


You have successfully installed HCL ComponentPack for HCL Connections. Now you want to know what is happening in that black box ?
The goal is to see something like this

Steps to reproduce:
– Update Kubernetes to 1.18.12
– Install Helm3
– Install nfs-client
– install metrics
– Install Traefik
– Install Prometheus

Disclaimer: use the following on your own risk.

Update Kubernetes to 1.18 because it’s now supported for Component Pack. If you are still on 1.11 you will be surprised after the update to 1.16. The coredns will no longer start. Coredns removed the proxy plugin, you need to switch to the forward plugin:
kubectl -n kube-system edit cm coredns
change ‘proxy . /etc/resolv.conf’ to ‘forward . /etc/resolv.conf’.
Going to 1.19 has not been an option. Some of the helm charts provide incompatible ingresss or services due to the api changes between 1.18 and 1.19

Installing Helm 3
At the moment the Component Pack still requires Helm 2. But both versions can be used in parallel.
Before installing Helm 3:
mv /usr/local/bin/helm /usr/local/bin/helm2
and after installing Helm 3:
mv /usr/local/bin/helm /usr/local/bin/helm3
Now my environment knows helm2 and helm3. It’s possible to migrate from helm 2 to helm 3 but I wanted an emergency option to be able to re-install Component Pack if needed. helm2 list shows the Component Pack installs. helm3 list will show all the new stuff.

nfs-client:
I’m lazy. I already have a working nfs server in my environment. I don’t want to handle every single pv/pvc manually. That’s why I use the nfs client provisioner. I cloned the repo and applied the 3 yaml files. I updated the deploy\deployment.yaml with the values for my nfs server.
kubectl apply -f deploy\class.yaml
kubectl apply -f deploy\deployment.yaml
kubectl apply -f deploy\rbac.yaml
kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

The last line defines the new storage class as default.

metrics:
get the yaml and add the –kubelet-insecure-tls to the args.
I was not able to get the metrics pod up without this. Probably proper certificates with all the right SAN’s would help. But as this is only in my lab, I don’t care.


wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml


- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --kubelet-insecure-tls
kubectl apply -f components.yaml

Now you should get the metrics pod in the kube-system namespace.

Traefik:
helm3 show values traefik/traefik > values.yml to get the variables from the helm chart.

In order to prevent conflicts between the different ingress controllers I added this ingressClass.
Save it in ic.yaml and kubectl apply -f ic.yaml

apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: traefik-lb
spec:
controller: traefik.io/ingress-controller


There are 2 ingress controllers on my system now Traefik and cnx-ingress.

I defined the listening ports for web and websecure as 31080 and 31443 with
kubectl edit svc traefik.

There’s also a dashboard available for traefik.

Prometheus/Grafana
There are a lot of tutorials out there on how to install and configure prometheus.

So the compressed version:


helm3 repo add stable https://charts.helm.sh/stable
helm3 repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm3 repo update
helm3 show values prometheus-community/kube-prometheus-stack > values.yaml

edit values.yaml file:
– enable the ingress creation, add the ingressClassName: traefik-lb and define the hostname
– assign the volumeclaimtemplates the storage class, managed-nfs-storage in my case

storage:
volumeClaimTemplate:
spec:
storageClassName: managed-nfs-storage
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
# selector: {}



kubectl create namespace monitoring
helm3 install prometheus prometheus-community/kube-prometheus-stack -n monitoring -f values.yaml
helm3 install metrics-adapter prometheus-community/prometheus-adapter -n monitoring

update the grafana ingress, add the ingressClassName
kubectl -n monitoring edit ingress prometheus-grafana

spec:
ingressClassName: traefik-lb
rules:
- host: grafana.ume.li


If you are using more then one node, make sure that the appropriate network ports (9000/9100 TCP) are open between the nodes.

Next step is to create the routes on your front Proxy/LoadBalancer so that the site is available under a nice url.
The embedded grafana has already the prometheus configured. The only thing I added was this dashboard

Ran into a small issue with the activities plus. If you only want to install the activities plus in your test kubernetes cluster and you dont want to upload all images, then you will fail.

the support/setupImages.sh script has 2 errors in it.

1. it does not recognize the parameter kudos-boards. ./setupImages.sh -st kudos-boards …. will fail. You need to change the line ‘starter_stack_options=’ and add ‘kudos-boards’ to the list.
2. if you only run setupImages.sh -st kudos-boards it will not push all the required stuff. you need at least to run it with -st customizer,kudos-boards
   or change the #infra block in setupImages.sh to

           # Infra
           if [[ ${STARTER_STACKS} =~ "customizer" ]] || [[ ${STARTER_STACKS} =~ "orientme" ]] || [[ ${STARTER_STACKS} =~ "kudos-boards" ]]; then
                   arr=(haproxy.tar mongodb-rs-setup.tar mongodb.tar redis-sentinel.tar redis.tar appregistry-client.tar appregistry-service.tar)
                   for f in "${arr[@]}"; do
                           setup_image $f
                   done
           fi
   

I don’t like the default persistentVolume paths /pv-connections. I use /data for my test environment.
This needs a small tweak in the boards-cp.yaml file:

minio:
    persistentVolumePath: data
    nfs:
       server: 192.168.1.2

Rule #1: Drain the worker node before a reboot. 
Consequences if you do not follow the Rule #1: the mongo-db gets corrupt and you need to repair it.
Steps to repair the customizer DB: delete the contents in mongo-node-0, mongo-node-1 and mongo-node-2’s subdirectory /data/db/ and register the customizer apps in the appregistry again.

I’m sure that there’s a way to repair the mongo db directories…

According to this cve There’s a potential issue with kubernetes 1.11.1 which is used in the component pack 6.0.0.6.
So I was wondering if it is possible to upgrade the kubernetes version to the patched point release 1.11.5.
The long version can be found in the official documentation.

The short version:

#on Master Node

yum-config-manager --enable kubernetes

yum install kubeadm-1.11.5-0 --disableexcludes=kubernetes

kubeadm upgrade plan
kubeadm upgrade apply v1.11.5

#Masterupdate
kubectl drain $MASTER --ignore-daemonsets
yum install kubectl-1.11.5-0 kubelet-1.11.5-0 --disableexcludes=kubernetes
kubectl uncordon $MASTER

yum-config-manager --disable kubernetes

#repeat for each master goto Masterupdate

#for each node
kubectl drain $NODE

#Nodeupdate
#on node $NODE

yum-config-manager --enable kubernetes
yum install kubectl-1.11.5-0 kubelet-1.11.5-0 kubeadm-1.11.5-0 --disableexcludes=kubernetes

yum-config-manager --disable kubernetes
kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
systemctl daemon-reload
systemctl restart kubelet

#back on the master
kubectl uncordon $NODE

#repeat for each node: restart from Nodeupdate

The upgrade on my little lab environment (1 Master + 3 Worker) went smooth.

Now the Customizer is up and running. Now I tried to put the elasticsearch stack. Following the IBM documentation, I was able to add a new worker node, dedicated solely for elasticsearch.
Putting the additional images in my dockerregistry, installing the helm charts. So far so good. But my es-master, es-data and es-client would not start. The log revealed the missing elasticsearch-secret. After some hints from Nico directing my search to the connections-env part. There’s parameter “createSecret=false” which drew my attention.

A short helm delete connections-env --purge and reinstall of the connections-env with createSecret=true did the trick.