[RFC] SAM CLI remote invoke command for interacting with the cloud resources

[hnnasit]

#2550 - Related GitHub issue

Summary

SAM CLI supports invoking Lambda functions locally using the sam local invoke command during development. But it does not support invoking and testing the functions deployed to the cloud. The SAM CLI sync command can be used to accelerate the development process by quickly deploying the local changes to the cloud as the changes are made, but users have to rely on either AWS console or AWS CLI to test the new changes. This proposal outlines the design of a new SAM CLI command to interact with the cloud resources and verify any new changes deployed.

Proposed Design

We are working on adding a new SAM CLI command which will help interact with the AWS resources on cloud and print the response on the console. The general idea is the command will use the provided resource logical-id and stack-name to look up the physical-id from the deployed stack and call the boto3 API. The command will support a payload option which can be used to send data to the resource. Payload could be a different parameter in the API definition for each resource. For instance, Lambda Function payload is the JSON provided as input. For SQS, payload is the Entries parameter in the API definition. This payload can be passed in as a string, an existing file path, or be read from stdin.

The proposed AWS resources and the boto3 APIs this new command will call are as follows :

• Lambda Function: Invokes the function by calling either Lambda.Client.invoke or Lambda.Client.invoke_with_response_stream boto3 API based on the function configuration in the template. The Payload parameter in the API definition can be passed as payload to the command.
• Step Functions: Starts the step function execution by calling the SFN.Client.start_execution boto3 API. The input parameter in the API definition can be passed as payload to the command.
• SQS: Sends messages to the Queue by calling the SQS.Client.send_message_batch boto3 API. The Entries parameter in the API definition can be passed as payload to the command.
• Kinesis: Writes data records in the Kinesis data stream by calling the Kinesis.Client.put_records boto3 API. The Records parameter in the API definition can be passed as payload to the command.

This will help developers validate their changes are working as expected on the cloud using SAM CLI and also help improve the developer experience while using sam sync. The command will allow developers to invoke or send an event to the resource using the logical-id defined in the SAM template instead of finding the physical-id from the cloud themselves. If the stack-name option is provided, the command will use the resource-id as the logical-id to search for the resource.

sam remote invoke --stack-name MyStack --resource-id MyFunction --payload '{ "message": "HelloWorld" }'

If no stack-name option is provided and the resource physical-id is provided as resource-id, the command will find the resource information from the CloudFormation template and call the resource boto3 API.

sam remote invoke --resource-id MyFunctionPhysicalId --payload '{ "message": "HelloWorld" }'

The command will also support a tail option for tailing CloudWatch logs and XRay traces which will run sam logs --tail --include-traces after the invocation is completed.

sam remote invoke --stack-name MyStack --resource-id MyFunction --payload foo/bar/event.json --tail

Developers will also be able to save their invoke configuration in the samconfig.toml file.

# this config can be invoked by running "sam remote invoke --config-env my-test-config"
[my-test-config.remote_invoke.parameters]
resource_id = "MyFunction"
payload = '{"message": "Hello World"}'

The sam local generate-event command can also be used to create payloads for different resource types and these payloads can be used to invoke the Lambda Function on cloud using this new command.

Example for starting a Step Function execution using the remote invoke command:

sam remote invoke --stack-name MyStack --resource-id MySFN --payload '{"first_name" : "test"}'

Example for sending a message to a SQS Queue using the remote invoke command:

sam remote invoke --stack-name MyStack --resource-id MySQSQueue --payload "hello world"

Example for sending data to a Kinesis stream using the remote invoke command:

sam remote invoke --stack-name MyStack --resource-id MyKinesis --payload '{"first_name" : "test"}'

We are looking for general comments on the following topics as well as thoughts regarding the new SAM CLI command:

• Which resources besides Lambda Function would you find useful to invoke or interact with using sam remote invoke command while developing Serverless applications using SAM CLI?
• Additional parameters available for the resource API call can be used with the parameter option. For instance, Lambda Function parameters Qualifier with value MyQualifier and InvocationType value Event can be passed as --parameter Qualifier:MyQualifier --parameter InvocationType:Event.
• The output format of the command response printed on the console. The options are either to print the entire response json string on the console which can be useful to pipe or print the relevant information in a readable format.
• Thoughts about including the XRay traces for the --tail option.

We encourage the community to provide feedback and suggestions for the proposed design and how SAM CLI can help improve the developer experience.

[ljacobsson]

This is great. A few points:

• Pick --stack-name from samconfig.toml it it's present (with the option to override). The developer will typically have the sam stack open in a code editor when they want to invoke something.
• The semantics of 'invoke' gets a bit odd with SQS/Kinesis. Do you "invoke" a queue/stream? Is the 'invoke' subcommand even needed? What about sam remote --resource-id ...?
• Lambda has the feature of shareable test events. Would be great if this command could leverage them (and the default test events you get in the console) and be as simple as sam remote invoke --resource-id MyFunction --test-event MyTestEvent

[hnnasit]

Thanks for the feedback @ljacobsson , this is really helpful 🙏. As for the naming, we will review this with our UX team and see how we can improve this. As for your third point, currently Lambda doesn’t have a boto3 API for getting the console events, but in general seems like a good use case.

[ljacobsson]

You're right, the default console events, like these can't be fetched via any API

Most of them are available with sam local generate-event anyway.

However, users can create shared events:

These example payloads get stored in the EventBridge Schema Registry and can be accessed via this code:

import sys
import boto3
import json

function_name = sys.argv[1]
event_name = sys.argv[2]

client = boto3.client("schemas")
response = client.describe_schema(
    RegistryName="lambda-testevent-schemas",
    SchemaName="_" + function_name + "-schema",
)
test_payload = json.loads(response["Content"])["components"]["examples"][event_name]["value"]
print(json.dumps(test_payload, indent=2))

Would also be great to be able to save test events to this registry to enable team collaboration and reuse of the command:

sam remote invoke --resource-id MyFunctionPhysicalId --payload '{ "message": "HelloWorld" }' --save-as HelloWorld

[hnnasit]

This is great, thanks for providing the example and the code snippet. I will dive deeper on this use case. 👍

[hnnasit]

Hello @ljacobsson, the proposed feature has been released in SAM CLI 1.98.0. You can find the documentation with further details here and here. Feel free to try it out and provide any feedback.

[kadrach]

Love it! It would be great if SAM can print the response to stdout, and relevant CloudWatch logs (perhaps traces) for the function execution to stderr.

[hnnasit]

Thank you for your feedback @kadrach 🙏.