[engine] Support remapping envvars for providers

[pgavlin]

These changes add support for remapping environment variables when launching providers. This allows users to work around problems with dynamic provider configuration that is stored in statefiles causing problems during refresh and destroy operations.

Background

pulumi up is distinctly different from pulumi destroy and pulumi refresh in that it involves running the Pulumi program associated with the stack's project. As it runs, the Pulumi program defines the desired state for resources--including provider resources--using values computed by the program in coordination with the Pulumi engine. When the program creates a provider resource, the inputs for the provider are either sourced from the program itself (i.e. from values provided by the program) or are read out-of-band by the provider plugin. The exact set of configuration that may be sourced from the environment is particular to each provider--for example, the Kubernetes provider uses the ambient kubeconfig by default, the AWS provider reads various AWS-specific environment variables, etc. Any explicitly-provided inputs are written into the stack's statefile.

For example, consider the following program:

import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", { region: "us-east-1" });
const defaultRegion = new aws.Provider("default-region");

The usEast1 provider's region is explicitly specified by the program, but the defaultRegion provider's region will be read from the environment (e.g. from the AWS_REGION environment variable). In the resulting statefile, the usEast1 provider's state will include the region input, but the defaultRegion provider's state will not.

Because pulumi refresh and pulumi destroy do not run the Pulumi program associated with the stack's project, they are unable to recompute configuration values that were explicitly provided by the program, and must use the values stored in the statefile. Unfortunately, this may include credential information, which is what causes the errors described here. The current workaround--which is certainly not sufficient for explicitly-instantiated providers--is to use environment variables to provide credentials out-of-band.

The clearest/most complete solution here is to run the Pulumi program associated with a stack's project as part of pulumi refresh and pulumi destroy. Unfortunately, this is a major behavioral change, and the exact semantics of the run are not clear.

#4981 tracks this issue in general.

Changes in this PR

These changes allow explicitly-instantiated providers to make use of the same workaround that is available to default providers: pass dynamic, environmentally-sourced provider configuration in environment variables rather than as provider inputs. The environment variable remapping allows users to replace the value for a provider environment variable with the value of a different environment variable before the provider is loaded. This allows users to place configuration in environment variables that the provider would not normally read and remap them to provider-supported envvars, which allows multiple distinct sets of environment variables for providers.

For the example above, this might look like so:

import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", {
    pluginEnvVars: { "AWS_REGION": { from: "US_EAST_1_REGION" } },
});
const defaultRegion = new aws.Provider("default-region");

Or, if the providers needed different credentials (much more common):

import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", {
    pluginEnvVars: {
	"AWS_ACCESS_KEY_ID": { from: "US_EAST_1_AWS_ACCESS_KEY_ID" },
	"AWS_SECRET_ACCESS_KEY": { from: "US_EAST_1_AWS_SECRET_ACCESS_KEY" },
	"AWS_SESSION_TOKEN": { from: "US_EAST_1_AWS_SESSION_TOKEN" },
    },
});
const defaultRegion = new aws.Provider("default-region");

[pgavlin]

Some downsides:

1. Ergonomics aren't great?
2. Requires changes to the generated SDKs unless we move the remapping into provider resource options

We might be able to improve the ergonomics by simplifying this a bit and just supporting an envvar prefix. Something like this:

const usEast1 = new aws.Provider("us-east-1", {}, { envVarPrefix: "US_EAST_1" });

The engine would then do something like this:

const envvars = Object.entries(process.env).filter(([k]) => k.startsWith(envVarPrefix + "_")).map(([k, v]) => [k.slice(envVarPrefix.length + 1), v]);