Fun time with kubernetes and dns

Posted by

We had installed the component pack for HCL Connections 8 in a brand new kubernetes environment. All the containers started and looked fine. Once we routed the traffic through the mw-proxy, Connections stopped working.

After a long investigation period we found the problem in the containers:

nslookup ihs-internal.test.com worked
ping ihs-internal.test.com returned a “Name or service not known” error
getent hosts ihs-internal.test.com returned nothing

Of course we did the first tests with a busybox conatiner. And of course nsloookup, ping and getent worked in the busybox….

After some extensive researching, the source of the problem had been identified in the /etc/resolv.conf. The file contained the line:
options ndots:5

For one deployment, that could have been fixed with a simple dnsConfig entry:

dnsConfig:
options:
- name: ndots
value: "2"

But the Componenpack has 20+ Deployments..

Digging deeper into that issue showed that each worker node had a /etc/resolv.conf like this:

nameserver 10.100.50.10
nameserver 10.100.50.11
search servers.cnx.test.com

The next step had been to add the doamin test.com as a suffix

nameserver 10.100.50.10
nameserver 10.100.50.11
search servers.cnx.test.com test.com

Which did not help at all.

So we removed the search line completely. As we always use the FQDN and don’t rely on the shortnames.

nameserver 10.100.50.10
nameserver 10.100.50.11

So after all kubernetes nodes had been patched with this new resolv.conf, nslookup, ping and getent returned the expected results in all the Componentpack containers.