kafka-connect-s3 - Trusted Account Credentials Documentation #366

Tamir-Schwarz · 2020-10-01T08:52:59Z

Hi,

We are using kafka-connect-s3 connector version 5.5.1.
We tried to use the Trusted Account Credentials feature.

According to the documentation,
we need to add the following properties:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role sts.role.session.name=session-name sts.role.external.id=external-id

It didn't work for us.

After digging the code,
we found that in order to make it work we need to add a prefix 's3.credentials.provider' to 'sts.role.arn', 'sts.role.session.name' and 'sts.role.external.id'

Following is the configuration that worked for us:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider s3.credentials.provider.sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role s3.credentials.provider.sts.role.session.name=session-name s3.credentials.provider.sts.role.external.id=external-id

In addition,
It worth mentioning that 'external.id' is mandatory, while it is not mandatory on AWS

The text was updated successfully, but these errors were encountered:

hafizmujadidKhalid · 2022-01-06T19:41:33Z

@Tamir-Schwarz How it worked for you? I am getting following error:

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=eu-central-1
tasks.max=2
topics=kafka-demo
s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/-demo-owner-readwrite-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=data-key

[Worker-0db15ad1ff259517f] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

timvryan · 2022-05-17T16:20:30Z

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

hafizmujadidKhalid · 2022-05-17T16:58:22Z

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

No, it did not work for me.

akhon · 2022-10-13T20:54:56Z

same problem

mjallday · 2022-10-13T21:07:54Z

kafka-connect-storage-cloud/kafka-connect-s3/src/main/java/io/confluent/connect/s3/S3SinkConnectorConfig.java

Lines 806 to 837 in c633f08

    
           public AWSCredentialsProvider getCredentialsProvider() { 
        
             try { 
        
               AWSCredentialsProvider provider = ((Class<? extends AWSCredentialsProvider>) 
        
                   getClass(S3SinkConnectorConfig.CREDENTIALS_PROVIDER_CLASS_CONFIG)).newInstance(); 
        
               if (provider instanceof Configurable) { 
        
                 Map<String, Object> configs = originalsWithPrefix(CREDENTIALS_PROVIDER_CONFIG_PREFIX); 
        
                 configs.remove(CREDENTIALS_PROVIDER_CLASS_CONFIG.substring( 
        
                     CREDENTIALS_PROVIDER_CONFIG_PREFIX.length() 
        
                 )); 
        
                 configs.put(AWS_ACCESS_KEY_ID_CONFIG, awsAccessKeyId()); 
        
                 configs.put(AWS_SECRET_ACCESS_KEY_CONFIG, awsSecretKeyId().value()); 
        
                 ((Configurable) provider).configure(configs); 
        
               } else { 
        
                 final String accessKeyId = awsAccessKeyId(); 
        
                 final String secretKey = awsSecretKeyId().value(); 
        
                 if (StringUtils.isNotBlank(accessKeyId) && StringUtils.isNotBlank(secretKey)) { 
        
                   BasicAWSCredentials basicCredentials = new BasicAWSCredentials(accessKeyId, secretKey); 
        
                   provider = new AWSStaticCredentialsProvider(basicCredentials); 
        
                 } 
        
               } 
        
               return provider; 
        
             } catch (IllegalAccessException | InstantiationException e) { 
        
               throw new ConnectException( 
        
                   "Invalid class for: " + S3SinkConnectorConfig.CREDENTIALS_PROVIDER_CLASS_CONFIG, 
        
                   e 
        
               ); 
        
             } 
        
           }

kafka-connect-storage-cloud/kafka-connect-s3/src/main/java/io/confluent/connect/s3/storage/S3Storage.java

Lines 172 to 176 in c633f08

    
           protected AWSCredentialsProvider newCredentialsProvider(S3SinkConnectorConfig config) { 
        
             log.info("Returning new credentials provider based on the configured " 
        
                    + "credentials provider class"); 
        
             return config.getCredentialsProvider(); 
        
           }

kafka-connect-storage-cloud/kafka-connect-s3/src/main/java/io/confluent/connect/s3/auth/AwsAssumeRoleCredentialsProvider.java

Lines 86 to 101 in c633f08

    
           public AWSCredentials getCredentials() { 
        
             if (basicCredentials != null) { 
        
               return new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName) 
        
                       .withStsClient(AWSSecurityTokenServiceClientBuilder.standard() 
        
                          .withCredentials(new AWSStaticCredentialsProvider(basicCredentials)).build()) 
        
                       .withExternalId(roleExternalId) 
        
                       .build() 
        
                       .getCredentials(); 
        
             } else { 
        
               return new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName) 
        
                     .withStsClient(AWSSecurityTokenServiceClientBuilder.defaultClient()) 
        
                     .withExternalId(roleExternalId) 
        
                     .build() 
        
                     .getCredentials(); 
        
             } 
        
           }

first glance it looks like the credential provider is being invoked correctly. we don't see similar issues when using a regular role so it's something related to the assume role configuration.

gotpist1 · 2023-02-15T11:26:11Z

Im also getting the same error using localstack and root role. Any updates on this? Also, specifying AWS_REGION ENV variable gives another error:

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@16971f4e: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@7cecef68: Failed to connect to service endpoint: ]
  at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:138)```

karlamaria100 · 2023-02-26T21:02:09Z

Region issue still happening

This passes the `s3.region` configuration value into credential providers. It also updates the AwsAssumeRoleCredentialsProvider to specify a region when building a AWSSecurityTokenServiceClient. Fixes confluentinc#366

This adds a `region` configuration item for the AwsAssumeRoleCredentialsProvider. It is used when building a AWSSecurityTokenServiceClient. It not specified, uses the default region selector. Fixes confluentinc#366

chriline · 2024-04-22T11:59:02Z

https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html#credentials-providers is still using s3.credentials.provider.sts.role.arn, s3.credentials.provider.sts.role.session.name and s3.credentials.provider.sts.role.external.id while https://docs.confluent.io/kafka-connectors/s3-sink/current/configuration_options.html is using sts.role.arn, sts.role.session.name, and sts.role.external.id.

I believe we're also running into @hafizmujadidKhalid's issue:
We've set up io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider like so

s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/my-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=external-id

and get the following error

[Worker-0f3f4a9ad02a8f900] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

AyoubOukh · 2024-06-02T14:59:41Z

getting the same issue

mt-rpranata mentioned this issue Apr 27, 2021

Guidance on how to use AWSAssumeRoleCredentialsProvider #409

Closed

munkyboy mentioned this issue Oct 28, 2023

Pass region into credentials provider #695

Closed

munkyboy linked a pull request Oct 29, 2023 that will close this issue

add region option to AwsAssumeRoleCredentialsProvider #697

Open

AyoubOukh mentioned this issue Jun 2, 2024

AwsAssumeRoleCredentialsProvider fails with region missing #754

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kafka-connect-s3 - Trusted Account Credentials Documentation #366

kafka-connect-s3 - Trusted Account Credentials Documentation #366

Tamir-Schwarz commented Oct 1, 2020

hafizmujadidKhalid commented Jan 6, 2022

timvryan commented May 17, 2022

hafizmujadidKhalid commented May 17, 2022

akhon commented Oct 13, 2022

mjallday commented Oct 13, 2022

gotpist1 commented Feb 15, 2023

karlamaria100 commented Feb 26, 2023

chriline commented Apr 22, 2024

AyoubOukh commented Jun 2, 2024

kafka-connect-s3 - Trusted Account Credentials Documentation #366

kafka-connect-s3 - Trusted Account Credentials Documentation #366

Comments

Tamir-Schwarz commented Oct 1, 2020

hafizmujadidKhalid commented Jan 6, 2022

timvryan commented May 17, 2022

hafizmujadidKhalid commented May 17, 2022

akhon commented Oct 13, 2022

mjallday commented Oct 13, 2022

gotpist1 commented Feb 15, 2023

karlamaria100 commented Feb 26, 2023

chriline commented Apr 22, 2024

AyoubOukh commented Jun 2, 2024