--keep-container leads to no space left on device

[peterschmidt85]

Steps to reproduce

1. Add an instance with dstack pool add-ssh
2. Run multiple runs

Actual:

Cannot create container. Error: Failed to run docker pull or docker create: Cannot create dir for runner: mkdir
/root/.dstack/runners/20240518-152340: no space left on device Check CLI and server logs for more details.

Expected:

Old containers should stay on the instance.

Question: Why do we use --keep-container?

[peterschmidt85]

@TheBits Any particular reason why we keep container?

[r4victor]

@peterschmidt85, the last container should be kept for debugging purposes. There is no good reason to keep all the containers – the previous container should be deleted when a new container is run.

[peterschmidt85]

This issue is stale because it has been open for 30 days with no activity.

[jvstme]

This bug is also relevant for VM-based cloud backends, not only add-ssh