How to Debug DNS Resolution issue in Kubernetes?

How to Debug DNS Resolution issue in Kubernetes?

Create a file named dnsutils.yaml with the following contents:

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

Then create a pod using this file and verify its status:

[root@kubernetesmaster ~]# kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
pod/dnsutils created

[root@kubernetesmaster ~]# kubectl get pods dnsutils
NAME       READY   STATUS    RESTARTS   AGE
dnsutils   1/1     Running   0          53m
[root@kubernetesmaster ~]#

Once that pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

[root@kubernetesmaster ~]# kubectl exec -ti dnsutils -- nslookup kubernetes.default
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

[root@kubernetesmaster ~]#

Check the local DNS configuration 

Take a look inside the resolv.conf file. It should look like following.

[root@kubernetesmaster ~]# kubectl exec dnsutils cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[root@kubernetesmaster ~]#

Errors such as the following indicate a problem with the coredns/kube-dns add-on or associated Services:

kubectl exec -ti dnsutils -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10#53

nslookup: can't resolve 'kubernetes.default'

Check if the DNS pod is running

Use the kubectl get pods command to verify that the DNS pod is running.

For CoreDNS:

[root@kubernetesmaster ~]# kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-576cbf47c7-jhwgd   1/1     Running   0          96m
coredns-576cbf47c7-t97b4   1/1     Running   0          95m
[root@kubernetesmaster ~]#

For kube-dns:

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                    READY     STATUS    RESTARTS   AGE
...
kube-dns-v19-ezo1y      3/3       Running   0           1h
...
If you see that no pod is running or that the pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually.

Check for Errors in the DNS pod

Use kubectl logs command to see logs for the DNS containers.

For CoreDNS: Expected to see these kind of logs for a healthy pod

[root@kubernetesmaster ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
.:53
2020/02/14 17:48:13 [INFO] CoreDNS-1.2.2
2020/02/14 17:48:13 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2020/02/14 17:48:13 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
.:5

Is DNS service up?

Verify that the DNS service is up by using the kubectl get service command.

[root@kubernetesmaster ~]# kubectl get svc --namespace=kube-system
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
kube-dns             ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP       53d

You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.

[root@kubernetesmaster ~]# kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                                                   AGE
kube-dns   xxx.xxx.xxx.xxx:53,yyy.yyy.yyy.yyy:53                       53d
[root@kubernetesmaster ~]#

How to delete dnsutils pod?

[root@kubernetesmaster ~]# kubectl delete -f https://k8s.io/examples/admin/dns/dnsutils.yaml
pod "dnsutils" deleted
[root@kubernetesmaster ~]#