Build a cache layer for secrets stored in AWS Secrets manager using AWS Lambda extension

What is the AWS Lambda extension?

A month back AWS announced a preview of Lambda Extensions, a new way to easily integrate Lambda with your favorite monitoring, observability, security, and governance tools. Extensions can be published as Lambda layers, there are two types are extension:

• Internal extensions → Run as part of the runtime process, in-process with your code. Internal extensions enable use cases such as automatically instrumenting code.
• External extensions → Allow you to run separate processes from the runtime but still within the same execution environment as the Lambda function. External extensions can start before the runtime process and can continue after the runtime shuts down. These extensions run as companion processes to Lambda functions.

Usecase

The whole idea of extensions was fascinating to me. So I thought, instead of just using the extensions for just implementing observability patterns, why can’t we build a cache layer for secrets using extensions.

As always, I googled my idea to see anyone has already implemented this. A famous company named “Square” has many smarter engineers than me, and they have already published a Lambda extension with a similar idea in GitHub 😟 . To be clear, they did a fantastic job, but like always, there is room for improvement, isn’t it right?

The extension they published was storing the secret in “/tmp” directory of the Lambda, a BIG NO NO from a security perspective, and no cache refresh logic. So I decided to let me build an extension that can take care of all the below use cases:

• Prefetch secrets from AWS Secret manager and save it into an in-memory cache.
• The extension will read “config.yaml” file in the lambda function to get the list of secrets that needs to be prefetched and cached. This way, the Lambda function controls what needs to be cached, and the extension code can be generic and reused across multiple Lambdas for a similar use case.
• Cache validity is specified through an environment variable in Lambda, instead of extension defining it.

#aws-lambda #secrets #cache #aws