Allow AWS Secrets to be used within IncludeConfig context

[markpanganiban]

Bug report

As discussed with Ben, I tested this PR on Platform but the AWS Secrets are not being detected.

Using slurm and local nextflow CLI, the secrets/pipeline secrets are working.

Expected behavior and actual behavior

Nextflow should be able to parse the config coming from AWS SecretsManager when used in IncludeConfig context.

Steps to reproduce the problem

On your repository's GitHub nextflow.config, add the following.

includeConfig "https://${secrets.PAT}@raw.githubusercontent.com/markpanganiban/nf-test/master/ubuntu.config"

Make sure that you create the PAT secret with your GitHub token. Select the secrets on Pipeline Secrets menu.

Program output

May-30 20:56:00.760 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /.nextflow/secrets/store.json
May-30 20:56:00.762 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@664e848c] - activable => nextflow.secret.LocalSecretsProvider@664e848c
May-30 20:56:00.771 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
May-30 20:56:00.837 [main] TRACE nextflow.file.FileSystemPathFactory - File system path factories: []
May-30 20:56:00.838 [main] TRACE nextflow.file.FileSystemPathFactory - File system path factories: []
May-30 20:56:00.839 [main] TRACE nextflow.config.ConfigBase - Include config file: https://[email protected]/markpanganiban/nf-test/master/ubuntu.config [parent: /.nextflow/assets/markpanganiban/nf-test/nextflow.config]
May-30 20:56:00.841 [main] TRACE n.file.http.XFileSystemProvider - File remote URL: https://[email protected]/markpanganiban/nf-test/master/ubuntu.config
May-30 20:56:00.892 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 250ms (attempt=1)
May-30 20:56:01.158 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 750ms (attempt=2)
May-30 20:56:01.922 [main] DEBUG n.file.http.XFileSystemProvider - Got HTTP error=404 waiting for 2250ms (attempt=3)
May-30 20:56:04.188 [main] TRACE nextflow.file.FileSystemPathFactory - File system path factories: []
May-30 20:56:04.194 [main] ERROR nextflow.cli.Launcher - Unable to parse config file: '/.nextflow/assets/markpanganiban/nf-test/nextflow.config'

  Config file does not exist: https://[email protected]/markpanganiban/nf-test/master/ubuntu.config

java.nio.file.NoSuchFileException: Config file does not exist: https://[email protected]/markpanganiban/nf-test/master/ubuntu.config
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.config.ConfigBase.memoizedMethodPriv$readConfigFilePath(ConfigBase.groovy:145)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

Environment

• Nextflow version: 24.04.2
• Operating system: Amazon Linux

[pditommaso]

The log reports the local secret provider was used, not the amazon one. It may be caused by a configuration error