Dockerhub creds for image pulls

[Christop1964]

What happened?

We always reach the docker rate limit (100 / 6) when we deploy Kubernetes with kubespray.
Not sure which default id get used when pull images from docker.
We are looking for a possibility to use a private dockerhub account to pull Images from docker.

We are using the kubspray ansible playbook

The solution from issue #6883 is not working anymore because containered_config is depricated.

What did you expect to happen?

That the deployment of kubernetes with a private dockerhub account does not fail when images get pulled from docker.

How can we reproduce it (as minimally and precisely as possible)?

We looking for a possibility to use a private dockerhub account to pull Images from docker

OS

printf "$(uname -srm)\n$(cat /etc/os-release)"

Linux 4.18.0-552.el8_10.x86_64 x86_64

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"[root@kubetest-ocp-host ~]#

Version of Ansible

ansible --version

ansible [core 2.16.7]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.12.1 (main, Feb 21 2024, 10:25:11) [GCC 8.5.0 20210514 (Red Hat 8.5.0-21)] (/usr/bin/python3)
jinja version = 3.1.4
libyaml = True

Version of Python

python3 --version Python 3.12.1

Version of Kubespray (commit)

README.md: - kubernetes v1.29.5

Network plugin used

calico

Full inventory with variables

cat inventory/mycluster/inventory.yaml

all:
hosts:
kube-32:
ansible_host: 172.16.3.92
ip: 172.16.3.92
access_ip: 172.16.3.92
kube-33:
ansible_host: 172.16.3.93
ip: 172.16.3.93
access_ip: 172.16.3.93
kube-34:
ansible_host: 172.16.3.94
ip: 172.16.3.94
access_ip: 172.16.3.94
kube-35:
ansible_host: 172.16.3.95
ip: 172.16.3.95
access_ip: 172.16.3.95
kube-36:
ansible_host: 172.16.3.96
ip: 172.16.3.96
access_ip: 172.16.3.96
children:
kube_control_plane:
hosts:
kube-32:
kube-33:
kube_node:
hosts:
kube-32:
kube-33:
kube-34:
kube-35:
kube-36:
etcd:
hosts:
kube-32:
kube-33:
kube-34:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

Output of ansible run

2024-06-26 14:08:17,320: [I] TASK [download : debug] ********************************************************
2024-06-26 14:08:17,320: [I] ok: [kube-32] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-33] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-34] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-35] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:17,320: [I] ok: [kube-36] => {
2024-06-26 14:08:17,320: [I] "msg": "Pull docker.io/library/nginx:1.25.2-alpine required is: True"
2024-06-26 14:08:17,320: [I] }
2024-06-26 14:08:52,456: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (4 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (4 retries left).
2024-06-26 14:08:52,457: [I]
2024-06-26 14:08:52,457: [I] TASK [download : Download_container | Download image if required] **************
2024-06-26 14:08:52,457: [I] changed: [kube-33]
2024-06-26 14:08:52,457: [I] changed: [kube-32]
2024-06-26 14:08:52,457: [I] changed: [kube-36]
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (3 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (2 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (3 retries left).
2024-06-26 14:08:52,457: [I] FAILED - RETRYING: [kube-35]: Download_container | Download image if required (1 retries left).
2024-06-26 14:08:52,457: [I] fatal: [kube-35]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.803006", "end": "2024-06-26 14:08:36.020879", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:35.217873", "stderr": "time="2024-06-26T14:08:35+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time="2024-06-26T14:08:35+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) "]}
2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (2 retries left).
2024-06-26 14:08:52,458: [I] FAILED - RETRYING: [kube-34]: Download_container | Download image if required (1 retries left).
2024-06-26 14:08:52,459: [I] fatal: [kube-34]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "docker.io/library/nginx:1.25.2-alpine"], "delta": "0:00:00.867968", "end": "2024-06-26 14:08:52.421105", "msg": "non-zero return code", "rc": 1, "start": "2024-06-26 14:08:51.553137", "stderr": "time="2024-06-26T14:08:51+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."\nctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit", "stderr_lines": ["time="2024-06-26T14:08:51+02:00" level=warning msg="DEPRECATION: The mirrors property of [plugins.\\\"io.containerd.grpc.v1.cri\\\".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.0. Use config_path instead."", "ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.4 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.5 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.6 s total: 0.0 B (0.0 B/s) \ndocker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| \nindex-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB \nelapsed: 0.7 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.4 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.5 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.6 s total: 0.0 B (0.0 B/s) ", "docker.io/library/nginx:1.25.2-alpine: resolved |\u001b[32m++++++++++++++++++++++++++++++++++++++\u001b[0m| ", "index-sha256:7272a6e0f728e95c8641d219676605f3b9e4759abbdb6b39e5bbd194ce55ebaf: downloading |\u001b[32m\u001b[0m--------------------------------------| 0.0 B/1.6 KiB ", "elapsed: 0.7 s total: 0.0 B (0.0 B/s) "]}
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:52,460: [I] NO MORE HOSTS LEFT *************************************************************
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:52,460: [I] PLAY RECAP *********************************************************************
2024-06-26 14:08:52,460: [I] kube-32 : ok=354 changed=58 unreachable=0 failed=0 skipped=417 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-33 : ok=324 changed=58 unreachable=0 failed=0 skipped=361 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-34 : ok=308 changed=55 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-35 : ok=297 changed=53 unreachable=0 failed=1 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I] kube-36 : ok=298 changed=54 unreachable=0 failed=0 skipped=358 rescued=0 ignored=0
2024-06-26 14:08:52,460: [I]
2024-06-26 14:08:53,599: [E] Command finished with rc = 2

Anything else we need to know

No response

[tico88612]

Maybe you need to set containerd_registry_auth in inventory/<YOUR_INVENTORY>/group_vars/all/containerd.yml?

kubespray/inventory/sample/group_vars/all/containerd.yml

Lines 43 to 46 in 9a69221

	# containerd_registry_auth:
	# - registry: 10.0.0.2:5000
	# username: user
	# password: pass