Driftctl : advanced demo

How to use driftctl : an advanced demo with multiple Terraform states and output filtering.

In this advanced demo for driftctl, you will learn how to use the tool in a more realistic real-life environment, with multiple Terraform states and output filtering. We will demonstrate how manual changes can impact drift detection and how driftctl complements Terraform plan!

Whether it’s a script gone wild, a bad API call from a trusted Lambda, or just the daily SNAFU, you want to know about the situation.
Driftctl will do just that.

Requirements

We recommend using an AWS account dedicated to testing.

An AWS account with IAM keys (AWS Homepage). See this AWS CLI documentation for a quick start.
Terraform 0.14.x (download link)
Driftctl (github)
Example Terraform code for this lab (here)

Create a test AWS environment

To start this driftctl advanced demo, clone the example Terraform code and execute it with Terraform. This Terraform configuration will simply create a VPC and a basically locked down security group.

Disclaimer: we used simple Terraform resources using the AWS provider: we did not try to create the most advanced, useful or complete Terraform configuration.

Export your AWS variables (or AWS_ACCESS_KEY_ID / AWS_SECRET_KEY pair):

Initialize both Terraform environments : app_env_1 & app_env_2

Run Terraform:

Do the same for the second environment:

Congrats, your AWS account now includes:

A VPC with a random name (from app_env_1)
A security group with a single rule (from app_env_2)

An overview of driftctl scanning options

As of this writing, driftctl can scan for AWS resources and complement Terraform in drift detection. More providers are on their way!

Using driftctl, you can:

Select one or multiple Terraform states, locally or on S3 (with --from <statefiles>)
Ignore some resources or fields by design (filling the .driftignore)
Filter the output to choose only some resources or tags (with --filter <expression>)
Format the output for readability or further processing (with --output <format>)

Driftctl with multiple local Terraform states

The small lab we created above is simulating 2 different “environments” (in real life it can be different applications, environments, or teams), with distinct Terraform states.

If we run driftctl with a single state, resources from the other state will be detected as drifts, or more precisely, unmanaged resources, which is not true and not what we want.

Here’s how to use driftctl with multiple states:

Alternatively, if you want to use pattern matching to load a bunch of Terraform state files all at once, you can use glob pattern (more examples here in the docs)

In this exact scenario, with only 2 directories containing 1 state each, we can load them all at once using the following:

Create a Security Group drift and catch it

Let’s manually add a rule to the security group we created, so we can detect it using driftctl:

Go to the VPC Security Group in the AWS console.
Select the security group named “Super Secure Security Group”
Click on “Edit inbound rules” in the “Inbound rules” tab
Click on “Add Rule”
- Select “All Traffic”, from “Anywhere”, add a random description
- Click on “Save Rules”

Run Terraform from the folder where the security group is managed and confirm it doesn’t catch the manual change:

Now run driftctl again:

Holy cow, driftctl caught the drift!

Ignore resources with .driftignore

In many cases, you’ll want driftctl to ignore some resources forever, to stop being notified about their existence, and also stop lowering the coverage score. Those reasons can include the very IAM key you use to run this lab (by definition it can’t be in this Terraform repo), resources that can’t or won’t be managed by Terraform, resources managed from teams sharing the same AWS account but not using Terraform, some legacy resources, or simply manual tests running a bit longer than expected. This is exactly the same approach as the usual .gitignore : you simply add to this file all the resources you want to be ignored.

Create an unmanaged S3 bucket, catch it, ignore it.

To proceed with this step:

Go to the S3 dashboard
Manually create an S3 bucket that you’ll name however you like

Obviously, as this is completely done outside of its control, this bucket can’t be detected as a drift by Terraform, so we’re in the dark, as expected.

Confirm driftctl detects the manually created bucket:

Open the .driftignore file at the root of the repository, where we execute driftctl and add a line like aws_s3_bucket

(in my case: aws_s3_bucket.randomBucket514)

Now run driftctl again:

Now the manually created S3 bucket is ignored forever!

Output filters for better accuracy

It’s often helpful to be able to filter the output dynamically, to see results only for one type of resource (like only IAM users) or a specific tag (like only a specific environment).

Filtering in driftctl is implemented as in the AWS CLI: you won’t be lost! (hint: it’s JMESPath).

Let’s filter only VPC resources:

Let’s now filter only for anything matching the “app_env_1” “Environment” tag on EC2:

You now can collect data only for the exact setup you want! Perfect for those reports 😉

JSON output manipulation

We’ve covered driftctl human-readable output, but it’s also often useful to process it further. That’s why driftctl can output to JSON!

Here’s how to generate a JSON file directly:

Take a look at the generated JSON file:

Now you can process this file using a processor like jq, for example, to catch only the coverage percentage and maybe send it to a database, from which a graph can be generated:

The possibilities are now endless!

Shutting down the lab

Don’t forget to destroy the resources we created for this lab:

Delete the S3 bucket you randomly created and your AWS account is now as clean as before.

Recap

We covered a lot in this driftctl advanced demo:

We used driftctl in combination with multiple Terraform state files.
We detected manual drifts from Terraform-managed resources as well as entirely unmanaged resources.
We manipulated driftctl output to ignore and filter resources based on type or tags.
We used and processed driftctl JSON output.

Stay in touch

Get product updates and occasional news.