Docker compose argument to replace env_file directive, or argument to enable host environment passthrough

[nullbio]

Description

Right now, the env file passed into --env-file using docker compose CLI is not the same as the env_file: directive in the compose.yml (a common source of confusion I see on stackoverflow).

The CLI argument is to supply Docker compose with an environment that can be referenced inside the compose.yml file like so:

environment:
  - FROM_HOST=$HOST_ENV_VAR

If we want to inject environment variables into the container itself, we need to specify the env_file: .env.dev directive, or limit ourselves to -e flags and the docker compose run command.

But there are a lot of tools that are coming out now that allow developers to manage their environment variables in different ways, which call for new feature(s) to support their flows well. For example, https://github.com/dotenvx/dotenvx and https://github.com/envkey/envkey.

The common trend with these tools is that they inject the decrypted environment variables into the process you wish to run, for example:

dotenvx run -- sh -c 'echo $DECRYPTED_VAR'

When it comes to Docker, the commonly recommended way to deal with loading the encrypted env files into the container is to install the tool into the containers image, as well as copy the encrypted env file into the image, or alternatively bind mount it, and do something like this with your command: directives in your compose.yml:

    command: sh -c "DOTENV_PRIVATE_KEY=$KEY exec dotenvx run -o -f/app/.env.dev -- postgres"

Which isn't great for a few reasons:

• Having to modify every build stage to add the binary to the image is a lot of work, really clutters your Dockerfile, and can be tricky in scenarios where you'd normally just use a preimage, for example this issue.
• You now have a bind mount using the volumes directive to an env file when normally you'd have just declared it in the env_file directive (if you were using a regular .env file that wasn't encrypted). Or you can build it into the image, but this is annoying for development, requiring rebuilds on env file change.
• You need to call your docker compose command using the same injection technique, so that docker compose can see your decrypted env vars (like in the above case, to get the $KEY value from the host), and also pass in your secret key manually like so, so that the containers process can use its own dotenvx to decrypt your encrypted file:

dotenvx -f.env.dev run -- env KEY=secret_key_here docker compose up

So now you have two layers of injection happening, one on the host that is calling docker compose, so that we can access decrypted env var values in compose.yml, and one layer using the tool built into the image, so that the container can decrypt env vars at runtime, using the passed along secret key.

To cut a long story short, the existing ways to deal with this are pretty terrible, as you can see. But a simple way to clean all of this up would be to allow the injected env into docker compose up to be able to be passed along to the container, instead of relying on the env_file: directive.

There are a few ways to approach this.

One would be a docker compose argument that simply passes along the env that is available to docker compose itself, so we could do something like:

dotenvx -f .env.dev run -- docker compose up --passthrough-env

Keys are already decrypted by this point, so this solves everything.

Another way would be to add a --container-env-file override, that overrides the value of the env_file: directive. That way you could do something like:

dotenvx -f .env.dev run -- docker compose up --container-env-file $(./prepare-env.sh)

Where the prepare-env.sh script could take in the injected, unencrypted env vars, store them in a tmp file, and return a path to this file.

Or using a bash process substitution with the env command, as mentioned in this stackoverflow:

dotenvx -f .env.dev run -- docker compose up --container-env-file <(env)

This would allow us to:

• remove the added dependency inside the image (no longer would we need to install the tool in the image itself)
• resolve the issue of having to bind mount the encrypted env file or install it on the image
• resolve the issue of having to pass along the secret key (and prevent it from accidently getting logged in shell history)
• restore the command: directives back to their original form, and allow people to use preimages directly in the compose.yml again without needing build stages for every service