Docker keeps cached manifests and indicies forever somewhere.

[sedyh]

Description

Hello. I can't find a reliable way to force docker to invalidate the manifest and index cache. Whatever happens, docker keeps them in cache until another digest comes from the registry on the head request. But I can't find a command that always cleans them on the client side. Can it be a bug?

Reproduce

1. Pull the image.
2. Wait for docker to load the index and select the manifest with the desired platform.
3. Cut the connection from client-side or registry-side when docker start downloading layers.
4. Try to reset the cache.
5. Pull the image.
6. Docker will make a head request and will find both index and manifest from somewhere, but it shouldn't.

Expected behavior

Here's what I tried:

• docker rmi <image> - works only after docker gets the layers, but if the layers are not downloaded - manifest and index will be kept in cache forever.
• docker system prune -a - doesn't affect manifest and indicies.
• docker stop; sudo rm -rf /var/lib/docker/image/overlay2/; docker start - doesn't affect manifest and indicies.
• docker manifest rm <manifest> - does not see cached manifest.

docker version

Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:29 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:29 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-76060800daily20240311-generic
 Operating System: Linux Mint 21.3
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 31.19GiB
 Name: sedyh-pc
 ID: d93f83cd-e4c5-4838-8692-caef30ab5abe
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: https://xxxxx:[email protected]:9991
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[thaJeztah]

Sounds related to the caching added in this PR;

• cache manifests on pull #41607

Curious; was there a specific problem you ran into, for which you can't have the cached ones?

[sedyh]

Curious; was there a specific problem you ran into, for which you can't have the cached ones?

Hello. I'm making a mitm proxy that prevents image downloading when the critical vulnerability was found.

A fairly good way to cancel the download was to replace the layer body with a denied error from the registry after issuing the manifest and index to the client (since only client knows the desired platform). I could specifically change the hash to the random one in the response to the head request, and upon a subsequent unsuccessful attempt to access my random hash, it would invalidate the hash as I want, but also show the warning since its a deprecated fallback. I would like to avoid it and not remake all the hashes in the image with others (its probably the right way but too hard at the moment cause then I would have to insert some garbage into the contents of the layers).

The warning looks like this:
https://docs.docker.com/engine/deprecated/#pulling-images-from-non-compliant-image-registries

[thaJeztah]

Thanks for the extra info.

Hm wondering (but not at my computer right now); wouldn't it still do a HEAD request, and would that request be interceptible for your proxy to return an error?

[sedyh]

Wouldn't it still do a HEAD request, and would that request be interceptible for your proxy to return an error?

All I want ideally is to always force any client to invalidate the cache when it contacts me (the proxy side). In the future, caching will be handled by the proxy itself, if necessary.

Currently, I have implemented behavior that does not replace invalidation, but adds some atomicity. It makes it so that until layers are received, manifests will always be requested, and when layers are received, docker rmi <image> can be done.

It is a bit strange, it looks like this:

--> HEAD /v2/dockerhub/maven/manifests/latest
<-- Docker-Content-Digest: sha256:7c03439155 <- random hash
<-- 200 OK: 
{}

--> GET /v2/dockerhub/maven/manifests/sha256:7c03439155 
<-- 403 Forbidden: 
  "errors": [
    {
      "code": "DENIED",
      "message": "errorcode404:{
        \"errors\":[
          {
            \"code\":\"NOT_FOUND\",
            \"message\":\"artifact dockerhub/library/maven@sha256:7c03439155 not found\"
          }
        ]
      }"
    }
  ]
}

I guess, after the digests from local storage and head request do not match and docker gets error response on the new digest request, it sometimes (maybe I'm getting confused), tries to make a normal get request with the old digest, which turned out to be quite convenient for me as a proxy. Then I work as usual: I just wait for the manifest, download all the remaining parts and decide whether to return an error based on the analysis results.

Index:

GET /v2/dockerhub/maven/manifests/latest
<-- 200 OK: 
{"manifests": [ {"digest": "sha256:c1a446", "type": "v2", "platform": "linux/amd64"} ]}

Manifest:

GET /v2/dockerhub/maven/manifests/sha256:c1a446
<-- 200 OK: 
{"config": {"digest": "sha256:d5beea"}, "layers": [ {"digest": "sha256:6a299a"} ]}

Config:

GET /v2/dockerhub/maven/blobs/sha256:d5beea
<-- 200 OK: 
{"architecture": "amd64", "os": "linux"}

Layers:

GET /v2/dockerhub/maven/blobs/sha256:6a299a
<-- 200 OK: 
8f cd f9 0b a2 39 6b 45 9c ... 50

[sedyh]

All I want ideally is to always force any client to invalidate the cache when it contacts me (the proxy side). In the future, caching will be handled by the proxy itself, if necessary.

Is there any way to force docker to completely invalidate the image cache without warning and without adding garbage to the contents of all parts of the image to change the hashes?

[vvoland]

The manifests are cached in the containerd content store in the moby namespace. You can access it with ctr:

ctr  -n moby content ls

You can purge all manifest cache with:

ctr -n moby content rm $(ctr -n moby content ls -q)

[vvoland]

There are no plans from our side to expose the manifest cache for user manipulation - I'll close the issue, but feel free to reach out if you have any further questions!

[sedyh]

There are no plans from our side to expose the manifest cache for user manipulation

Is there some reason for this? It would be logical to be able to clear the entire cache via rmi / prune. There is no such situation in podman - everything is cleaned there entirely.

It also seems to me that this situation is not specific to this proxy, it could just be a network error on downloading layer after manifest is pulled and there is no way for the user to solve this problem using docker cli.