Native support for Terraform in SAM CLI (now in Beta)

[super132]

Terraform Support
Recently we added native support for CDK applications in the SAM CLI - you can see our launch announcement blog post and download the beta version of the CLI.

We would like to collect the community’s feedback on whether we should add native support for Terraform in the SAM CLI. Terraform users would be able to use the SAM CLI’s build, package, local invoke, and start-api commands. The build and package commands make it easy to create deployment packages or OCI images that are ready to be used in AWS Lambda. The local invoke and start-api commands makes it easy to emulate AWS Lambda and API Gateway in local Docker container for testing purposes.

With this change, the SAM CLI would parse HCL definitions to understand functions and APIs that can be locally tested and built. The SAM CLI would not write HCL, translate to, or force the use of CloudFormation. Deployments should still be performed using the Terraform toolkit.

What feedback are we looking for?

We are moving the beta feedback to discussion for better gathering feedbacks. You can refer to the original issue for the past discussions and feedbacks. Thanks.

Is this something the community is interested in? Vote with 👍🏽 .
The sam build, package, and local invoke commands rely on the CodeUri property of a function resource pointing to a local path where the source code for the Lambda function is stored. The SAM CLI automatically pulls dependencies, compiles if necessary, and prepares a deployment package for AWS Lambda. The package command today produces an output template that contains the S3 URL for the deployment package the SAM CLI prepared. How would you like this work with Terraform?

[ceptoplex]

What I'm currently missing is a way to support having the Terraform configuration files in a subdirectory, e.g.:

/project
  /infrastructure
    main.tf
    providers.tf
  /src
    ... (lambda code) ...

When using e.g. sam build --beta-features --hook-name terraform everything from the working directory seems to be copied to a temporary scratch folder, but it is also expected to run this command with your Terraform configuration root (here infrastructure) being the working directory. This way, in the above example, I'm unable to write Terraform configuration that depends on files from src as it is not copied to this scratch folder, e.g., the actual build process.

I saw this parameter but not sure if it's configurable with CLI parameters?

__

Btw: Link to launch announcement should rather be this one, right?

[moelasmar]

Thanks @ceptoplex for raising this comment. The current Beta version only supports that the Terraform configuration files, and src directory are in the same level (which is the current directory). This can be fixed by accepting a new parameter to let SAM CLI knows where the project root directory, so SAM CLI can copy the whole project.

I will raise this with the team to see how can we prioritize it.

[jordancparker]

Is there any update to this ?

[praneetap]

We will have something on this soon! Thanks for your patience :)

[moelasmar]

Thanks @ceptoplex for your patience. You can now use sam build for terraform projects with nested directories structure. Please refer to this reply to get more info on how to use it.

[rberger]

I've had a hard time wrapping my head around what it means for "Native Support for Terraform in SAM CLI" I realize what I think of / what we need is very different than what this change does.

I now think that "Native Support for Terraform in SAM CLI" means that SAM uses Terraform to at least update the Lambda Function code. Not clear if it also means it would deploy the Lambda infrastructure by triggering Terraform.

But in ether case, that's not really what we need / want.

We use Terraform as our IaC but we do not use Terraform to normally deploy the Lambda image (code) to the Lambda function from our dev environment.

Instead we run Terraform only to build out or update infrastructure and all the terraform code is in a directory tree distinct from our Lambda function code.

We currently deploy our Lambda function code as zip images or docker images via our CI/CD pipeline or via some internal tooling. Both deal with the build process (Python, Node or Ruby, but mostly Python), packaging into zip or docker images, and then uploading to the lambda function using AWS CLI/SDK.

So I am wondering if SAM in general or Sam for Terraform can be of use for us? We would love to use SAM in our environment in lieu of our internally built tools for doing the dev cycles of development and testing. Would also love to be able to take advantage of the ecosystem of SAM tooling primarily for the development and testing side of SAM, not the deployment so much. Unless we had a way to connect SAM's intentions of deployment to our mechanisms of deployment.

[moelasmar]

Thank you @rberger for your insightful feedback. I am sorry for the previous confusion caused by the unclear title and description of the discussion. I will make sure to revise it soon to provide better clarity.

From my understanding of your use case, you use Terraform to manage your infrastructure, which is not typically a repeating process. Your development team updates the code for Lambda functions and pushes it to a repository that triggers a CI/CD pipeline. The pipeline builds the code, packages it into either a zip file or Docker image, and uploads it to S3/ECR. Finally, it updates the Lambda function using AWS CLI/SDK.

We are currently exploring the possibility of offering our aws-lambda-builders to Terraform customers. This way, customers can define their Lambda functions using Terraform and simply run the terraform apply command to handle the building, packaging, and updating of the functions. This eliminates the need to maintain the building, packaging, and updating logic within your CI/CD pipeline, as you would only need to call terraform apply in response to changes in the code or the Terraform project. Additionally, this solution would allow you to use the AWS SAM CLI locally to build the Lambda functions with sam build and test them locally with sam local commands.

Please let us know if this solution meets your needs and requirements.

[rberger]

Actually we do not want to use Terraform to regularly deploy the lambda function images.

We want to do what SAM CLI build and accelerate does now during development cycles on the developers computers, and have CI do the build and deploy of lambda function images after PR merge to main and to do promotions.

Code deploys happen often. Infrastructure deploys happen less frequently.

We have tooling to do this now, but it feels like we are reinventing the wheel and would be nice to use something like SAM that only worried about code like assets and not infrastructure deployment. We're doing all this in a polyglot monorepo that supports lambdas written in python, ruby and nodejs (soon rust!?) as well as the terraform code. Our terraform code is in a different tree (but same monorepo) from our lambda code.

There doesn't have to be much explicit sharing between the tooling for the lambda code and infrastructure, but there is implicit sharing via naming conventions. Primarily for the lambda function names and all their assets.

If we do have to share things between terraform and the tooling for build/deploy/run of the lambdas, we use SSM Parameters tied together via naming conventions.

All configuration of the lambdas are done in the terraform.

In the ideal world (if I could dream) SAM would use some intermediary mechanism that could be fed from CDK, Terraform and other tooling that handles provisioning/deployment of infrastructure to understand how things are "wired up", and SAM focused only on code like asset build and deployments (Lambda code, Stepfunction asl, etc). I do understand that would be non-trivial though.

[bosatsu]

I love that AWS SAM is heading in this direction. My company requires that we use Terraform for our IaC as they are using Hashicorp's Sentinel for governance. I just found this page after doing a comparison between AWS CDK / SAM and Terraform / SAM, hoping that we could use Terraform / SAM, but it just doesn't feel like its there yet.

Couple of issues that I have run into.

1. There aren't a lot of great examples out there
   I wanted to create an echo endpoint using API Gateway and Lambda that would return the path that a user calls. This requires using the AWS_PROXY integration, which is difficult to navigate by itself, much less in terraform with SAM. These are the resources I had to cobble together to get it working:

• Using AWS SAM with Terraform
  Better together: AWS SAM CLI and HashiCorp Terraform
  Using the AWS SAM CLI with Terraform for local debugging and testing

2. SAM for terraform doesn't seem to support sam local start-api, which seems like a huge miss. Half the reason I used SAM in the past is because of this feature to test my api end to end.

❯ sam local start-api
Error: Template does not have any APIs connected to Lambda functions

When I look at .aws-sam/build/template.yaml, only the CF for the Lambda is there, nothing for the API Gateway.

[moelasmar]

Thank you for your feedback @bosatsu. You are correct, the current beta version of SAM CLI does not support testing API Gateway locally. At this time, it only supports testing Lambda functions. However, we value the feedback and input from our community and are actively collecting information to determine the most important use cases to implement next. Stay tuned for future updates.

[praneetap]

sam local start-api is now supported for API GW REST APIs. Please let us know if you have feedback!

[RisingPhorce]

I really like the idea that Terraform could the common Infrastructure as Code layer with CDK as an option for cloud-native style applications. To that end it would be really nice if CDK supported CDK for Terraform (CDKTF). Perhaps activated with a flag like --beta-features --hook-name cdktf. This would move nicely in the direction of an ideal solution for us. Developers can write simpler modern, cloud-native apps (event-driven architectures, Lambda, API Gateway, Event Bridge, et. al.) in their language of choice and use CDKTF to "synthesize" the CDK to Terraform. Build/CI/CD process would use existing institutional Terraform knowledge, output, security/IAM, etc. i.e. CDK to those that want it, Terraform for all.

[moelasmar]

Thanks @RisingPhorce for sharing your feedback, I will discuss it with our PM to add it to our priorities.

[moelasmar]

We would greatly appreciate your input regarding a workaround that we've identified for addressing the known limitations of the Terraform support feature in the AWS SAM CLI repository, as documented in issues #4395 and #4396.

This workaround requires customers to run terraform apply on Terraform project whenever a new Lambda Layer resource is added (it is not required to do so when making changes to Lambda Function or Lambda Layer attributes or source code). By utilizing this approach, we believe that the limitations can be effectively resolved using AWS SAM CLI.

We are interested to hear if this workaround is a visible workaround that you can do, and we are also seeking your feedback on the possibility of AWS SAM CLI running terraform apply on your behalf, with a confirmation prompt if needed.

[pkondrat]

I think that the direction is very right. There are probably a few companies that are using TF + AWS in order to be a bit less vendor locked-in.
I did some tests back in April. Looks very promising and it has a potential. One thing is still a blocker for us -> lack of support for Terraform Enterprise and their remote executions and variables.

Even if we switch the execution mode temporarily to Local, then we are missing all the variables’ values that we maintain on TFE.
I tried to export those values and failed to inject them on the execution.
TFE has a good API, so I don’t think it should be a big challenge to integrate with it and start supporting TFE as such. Happy to help if needed, just let me know.

[moelasmar]

Hello @pkondrat, thanks for your feedback. We are looking for the options to support the TF Enterprise.

[moelasmar]

We release the Terraform support in sam local start-api for V1 API Gateway resources in release V1.90.0. Now you can test your V1 API Gateway services that integrate to lambda functions locally. Please let us know if you have any feedback.

[moelasmar]

Please refer to this Docs to get more info regarding SAM CLI and Terraform integrations.

[moelasmar]

Terraform support in SAM CLI tool is GA now (announcement) in release 1.97.0.
Please refer to this blogpost to know what is supported in the new SAM CLI release, and also refer to this docs to get more info regarding the SAM CLI and Terraform integrations.

[pkondrat]

Hi @moelasmar,
congrats on reaching GA with TF support.
I tried to validate if you managed to tame the Remote executions (Terraform Enterprise support) but unfortunately sam build command still fails with "Error: Saving a generated plan in currently not supported" .
That is a TF restriction done to keep the potential sensitive details of terraform plan output on the server, not local machine.
https://discuss.hashicorp.com/t/saving-a-generated-plan-is-currently-not-supported/2116/4

Getting the same (as suggested above) via API was also not possible as it requires admin access on the workspace, which is not the common case in bigger organisations.
https://developer.hashicorp.com/terraform/enterprise/v202309-1/api-docs/plans#retrieve-the-json-execution-plan

I have also tried the -skip-prepare-infra command but it does not seem to have any effect.

One thing also struck me - terraform plan is just the delta between code and infra state. What in the case when there are no changes and you just want to attach SAM to an existing TF project. Plan would be empty then. Wouldn't it make more sense to base everything on a state version?
Just ask user for: TF URL, Token, Workspace name/id and simply fetch the content of current-state-version via API? You can even download it without much hassle.

[moelasmar]

Thanks @pkondrat for your feedback. When we did the investigation on supporting TF Enterprise, we found that it is easier to use the Terraform plan to stay aligned with our approach on supporting the normal Terraform applications. Not sure if using the state will be applicable or not, and what are the problems that we may face. I will sync with The team to investigate this and see what we can do.

Regarding the following, I have a question, is the delta thing here only applied for Terraform Enterprise projects, because when I use Terraform and run terraform plan I got a complete plan file even for applied projects.

One thing also struck me - terraform plan is just the delta between code and infra state. What in the case when there are no changes and you just want to attach SAM to an existing TF project. Plan would be empty then. Wouldn't it make more sense to base everything on a state version?

[pkondrat]

That would be great. Thanks.
For the plan question - plan is just some sort of change set / delta between what you have locally in the code and the infrastructure state. It will printout different resources along with value changes and symbols (~ - or +), but only the ones that you are touching. The ones that stays exactly the same, will not be there.

[mildaniel]

Hey @pkondrat, can you try updating Terraform to version 1.6 and trying the integration again? Version 1.6 of Terraform allows saving local plans when using Terraform Cloud. Let us know if this works for you and if you see and discrepancies in behaviour!