[k8s] suppress connection error warnings when disconnected from k8s

[asaiacai]

suppresses urllib3 warnings for sky show-gpus if k8s cluster gets disconnected. Closes #3591

before changes

(base) gcpuser@k3s-ebd1-head-82q3gnw3-compute:~/skypilot$ sky show-gpus --cloud kubernetes
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56eeb60>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56ef3a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56ef6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
No GPUs found in Kubernetes cluster. If your cluster contains GPUs, make sure nvidia.com/gpu resource is available on the nodes and the node labels for identifying GPUs (e.g., skypilot.co/accelerator) are setup correctly. To further debug, run: sky check

after changes

(base) gcpuser@k3s-ebd1-head-82q3gnw3-compute:~/skypilot$ sky show-gpus --cloud kubernetes
No GPUs found in Kubernetes cluster. If your cluster contains GPUs, make sure nvidia.com/gpu resource is available on the nodes and the node labels for identifying GPUs (e.g., skypilot.co/accelerator) are setup correctly. To further debug, run: sky check 

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)

To create a disconnected cluster I (un)installed via k3s on a GCE VM

sky launch -c k3s --cloud gcp
ssh k3s
curl -sfL https://get.k3s.io | sh -
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown -R $USER ~/.kube
sky check
sky show-gpus --cloud kubernetes
bash /usr/local/bin/k3s-uninstall.sh
sky show-gpus --cloud kubernetes

[Michaelvll]

Thanks @asaiacai! This looks good to me. I will let @romilbhardwaj to do a final check. : )

[romilbhardwaj]

Thanks @asaiacai!

[asaiacai]

@romilbhardwaj edited so the top level kubernetes APIs are all wrapped in a logging context. I've attached some example commands and outputs after disconnecting here including show-gpus launch exec down jobs launch and serve up. I guess one thing to note is the trace dumps for some of these are quite verbose, not sure if the intention is to catch these and replace them with something shorter but let me know.