Disable then Stop firewall if enabled

$ sudo ufw status
$ sudo ufw disable 

Disable and then Stop AppArmor

$ sudo systemctl stop apparmor
$ sudo systemctl disable apparmor

systemd-resolved

systemd-resolved is a system service in Linux that serves as a network name resolution manager. Its primary function is to provide network name resolution services to local applications running on the system. It offers several features, including caching and validating DNS/DNSSEC (Domain Name System/Domain Name System Security Extensions) stub resolution, as well as LLMNR (Link-Local Multicast Name Resolution) and MulticastDNS (mDNS) resolution and response.

When enabled in k3s installations systemd-resolved will cause issues with communication between the pods, therefore it needs to be disabled.

Code below will stop and disable systemd-resolved and create a single DNS entry in /etc/resolv.conf, this can be adjusted per customer basis depending what DNS server customer is using:

sudo su -
systemctl disable systemd-resolved.service
systemctl stop systemd-resolved
unlink /etc/resolv.conf
echo "nameserver 8.8.8.8" >> /etc/resolv.conf

Note, that using sudo will not work here, you need to switch user to root to complete.

Fix for EC2 instances (mostly affects just support labs)

If you’re deploying a test instance in AWS EC2, you will get errors like sudo: unable to resolve host adam-enterprise-2-sib: Temporary failure in name resolution on every "su -" :

You can fix this by setting your hostname in hostnamectl if it isn’t already:
hostnamectl set-hostname adam-enterprise-2-sib
Then adding a line with hostname pointing to your localhost IP, into your /etc/hosts file:
127.0.0.1 adam-enterprise-2-sib

(The original line with localhost doesn’t need to be removed.)

Changes to DNS while k3s is running

If you make any changes to /etc/resolv.conf while k3s is running, your changes will not propagate to coredns, so containers inside k3s will still try to use the old IP to resolve hostnames.

If the old DNS server is not running anymore, this will mean the following issues persisting even AFTER the DNS IP has been corrected in resolv.conf:
- disconnection from rancher
- not getting updates
- ErrImagePull
- logs pods such as cattle-cluster-agent-xxxx referring to “lookup” errors and failing to connect to some pod on port 53.

Solution 1: you can use the /usr/local/bin/k3s-killall.sh script, preferably as root to not just gracefully stop the k3s service, but also stop all pods (including coredns). See https://docs.k3s.io/upgrades/killall.

Once this script has finished, you can restart k3s with systemctl start k3s.service.

Solution 2: reboot also works.