Implement ephemeral resources

[mitchnielsen]

Summary

Terraform 1.10 adds support for ephemeral resources, which were added to help keep sensitive information out of plan and state files.

Let's look to see if there are any opportunities to make use of this in our provider.

Acceptance criteria

• Opportunities to implement ephemeral resources are identified (if any)
• Issue(s) created for implementation

[mitchnielsen]

Research

• https://developer.hashicorp.com/terraform/plugin/framework/ephemeral-resources covers how to implement this using the plugin framework. It looks like this is a new object that sits alongside resources and datasources.
• https://www.hashicorp.com/en/blog/terraform-1-10-improves-handling-secrets-in-state-with-ephemeral-values is a blog post that gives good context of the core issue and an example.
• ephemeral/password: Introduce a new ephemeral password resource hashicorp/terraform-provider-random#625 gives us an example of how ephemeral resource support was added for their random provider.
• https://support.hashicorp.com/hc/en-us/articles/36370466952979-List-of-Ephemeral-Resources-released-by-Top-Terraform-Providers is a list of providers who have added support for ephemeral resources.
• https://www.youtube.com/watch?v=JnXEiWA1TE0 is a helpful YouTube video that explains the context for why ephemeral resources are helpful, example implementations, and limitations. They link to a repo with examples and notes.
• https://log.martinatkins.me/2024/05/22/terraform-ephemeral-values/ - this is a great, longer writeup about the implementation concepts.

[mitchnielsen]

From what I'm seeing so far, these are primarily helpful for configuring providers and defining locals and outputs. That's because at the moment, you can't reference ephemeral resources in regular resources and datasources (because that would persist their data to state).

• Here is an example from the google provider where an ephemeral resource is used to retrieve a secret value used to configure the provider. Looking at the source schema for that attribute, I don't see that anything was required to support this.

[mitchnielsen]

I'm also seeing discussions that a slightly alternative approach might be available in Terraform 1.11 (now looking like 1.14 based on the linked milestone), where instead of creating a separate ephemeral resource, you'll be able to mark attributes as write-only. From the PR description:

Write-only attributes are managed resource attributes whose values are not saved to the Terraform plan or state artifacts and can accept ephemeral values.

I could see us using this for attributes marked as Sensitive, such as service account api_key and block data.

[mitchnielsen]

After chatting with @jamiezieziula, we're going to try this out for working with the api_key attribute for service accounts. This resource/datasource currently stores that attribute in state (as expected), so we could potentially remove that attribute and shift it to an ephemeral resource. We already have logic to get the API key and rotate it if the expiration date has passed (or been changed manually in the configuration), so we should be able to use that for the implementation.

Will give that a try.

[mitchnielsen]

See #375 (comment). Because API key values are not stored in Prefect, we don't have a consistent way to access them, which "breaks" the implementation for us here.