GET STARTED ON DRIFTCTL

Configure

Advanced usage

Getting started on driftctl: start tracking drifts from a clean state

How to start tracking drifts from a clean baseline even if you are moving to Terraform.

This post is part of the walkthrough that will get you started on driftctl in a couple of steps. If you haven’t already done so, you should consider setting up your AWS authentication to use driftctl in a secure way and configuring your Terraform provider version as well.

TL;DR

Many of us have reasons to not target 100% IaC coverage of our AWS accounts. Some of us are moving to Terraform. In this case, driftctl continuously delivers drift notifications that we don’t care about. Behold! There’s a solution.

In this post, we’ll show how to start using driftctl with a clean state, by ignoring all the current resources that are not yet under control. This will be easily done in a couple of seconds by generating a .driftignore file from a driftctl JSON output.

We’ll start with a heavily drifted AWS account, and end up with a clean state so you can actually start tracking resources. We’ll do all the steps with both the CLI and Docker container version.

What kind of messages can you expect to find in the CLI?

Found 11 resource(s) – 73% coverage
– 8 managed by terraform
– 2 not managed by terraform
– 1 found in terraform state but missing on the cloud provider

At this stage and before digging into the specifics of how to create a clean baseline, it is probably useful to list the type of messages you can expect to find in the CLI upon running a scan :

“x resource(s) managed by Terraform”: this one is pretty clear. It displays the number of resources in your infra that are both listed in your terraform state and on your cloud provider.
“x resource(s) not managed by Terraform”: represents resources that are actually present on your cloud provider but not on your terraform state. Those resources are not under the control of your Infrastructure as Code and you might want to consider moving them to Terraform.
“x resource(s) found in Terraform state but missing on the cloud provider”: this is a list of resources that are defined as code in your terraform codebase, but do not appear on your cloud provider. Those resources were probably deleted manually after your last terraform apply directly on your cloud console and will be provisioned again at your next apply unless you modify your terraform code.
“Congratulations, your infrastructure is fully in sync”: when you see this message, this means that you have a 100% match between what is listed in your Infrastructure as Code and what is running on your cloud provider. Congratulations! Make sure to run driftctl on a regular basis, to spot any new changes and bring your control back to its maximum in case of discrepancies.

Day 1, too many unmanaged resources

Back to our tutorial : what are unmanaged resources? Those are simply resources (EC2 instances, Security Groups, S3 buckets…) that are not managed by Terraform. When we run driftctl against a bunch of Terraform states, unmanaged resources will show up.

But in many cases, it’s perfectly understandable and acceptable for the time being to have unmanaged resources in the AWS account.

Many of us:

do not have the luxury to have AWS accounts fully managed by Terraform,
use a mix of CloudFormation (CFN) and Terraform, or some legacy scripts running here and there,
work with different teams on shared AWS accounts,
are currently moving existing projects to Terraform ,
create new projects in existing AWS accounts,
will fix those drifts soon, but not now (“it’s in the backlog!”)

So, the driftctl experience is quite frustrating for now in this mixed situation: while we have very good reasons to want to know what goes wrong from now on, we are still getting a ton of noise in the output that renders the tool almost useless.

Sure, there’s the .driftignore file, that works similarly to the .gitignore file, and we could still fill the file manually. But it works with a handful of resources, definitely not with a lot of resources.

The good news is that driftctl v0.9.0 ships with a generator of .driftignore, so we can get started in no time and ignore all the drifts!

How to automatically generate a .driftignore file

If you’re like me, always trying stuff here and there, or if you are moving to Terraform, you’ll end up with a messy driftctl output like that:

				
					$ driftctl scan --from tfstate://**/*.tfstate
Scanned states (4) 
Scanned resources (62)      
Found resources not managed ty terraform:
  aws_iam_access_key:
    - AKIASBXWQ3AYR4KWTMHD (User: driftctl_ro)
    - AKIASBXWQ3AYT7WXIBNE (User: labs)
    - AKIASBXWQ3AYUFW6ZCWO (User: endgame-s3-demo)
    - AKIASBXWQ3AYWVVI6DEW (User: terraform)
  aws_iam_policy_attachment:
    - AWSServiceRoleForAccessAnalyzer-arn:aws:iam::aws:policy/aws-service-role/AccessAnalyzerServiceRolePolicy
    - AWSServiceRoleForConfig-arn:aws:iam::aws:policy/aws-service-role/AWSConfigServiceRolePolicy
    - AWSServiceRoleForRDS-arn:aws:iam::aws:policy/aws-service-role/AmazonRDSServiceRolePolicy
    - AWSServiceRoleForSSO-arn:aws:iam::aws:policy/aws-service-role/AWSSSOServiceRolePolicy
    - driftctl_ro-arn:aws:iam::141177182257:policy/driftctl_ro
    - endgame-s3-demo-arn:aws:iam::aws:policy/AmazonS3FullAccess
    - labs-arn:aws:iam::aws:policy/AdministratorAccess
    - terraform-arn:aws:iam::aws:policy/AdministratorAccess
  aws_iam_role:
    - driftctl_assume_role
  aws_lambda_event_source_mapping:
    - Source: arn:aws:sqs:us-east-1:141177182257:queue-6qaidu.fifo Dest: arn:aws:lambda:us-east-1:141177182257:function:lk6n03-lambda-demo
  aws_iam_policy:
    - arn:aws:iam::141177182257:policy/cloudskiff_qa_policy
    - arn:aws:iam::141177182257:policy/driftctl_ro
    - arn:aws:iam::141177182257:policy/test123
  aws_iam_role_policy:
    - driftctl_assume_role:driftctl_policy
  aws_iam_user:
    - driftctl_ro
    - endgame-s3-demo
    - labs
    - terraform
  aws_s3_bucket:
    - a-bucket-filled-with-files
    - blob-9082437
    - dctl-1
    - driftctl-tfstates
    - s3-bucket-1-6qaidu
    - test-resource-exposure-caxyxllsajfrzbwe
  aws_s3_bucket_policy:
    - test-resource-exposure-caxyxllsajfrzbwe
  aws_sqs_queue:
    - https://sqs.us-east-1.amazonaws.com/141177182257/queue-6qaidu.fifo
Found 32 resource(s)
 - 3% coverage
 - 1 managed by terraform
 - 31 not managed by terraform
 - 0 found in terraform state but missing on the cloud provider

Hint: use gen-driftignore command to generate a .driftignore file based on your drifts

It’s cool the first time we launch the tool, so we have a clear overview of drifted and unmanaged resources, based on the cumulative layers of Terraform states – but it’s pretty useless afterwards, as we won’t fix all the drifts right away.

Step 1 - Generate a JSON output

So let’s start by creating a workable JSON output of this scan:

				
					$ driftctl scan --from tfstate://**/*.tfstate --output json://drifts.json
Scanned states (4) 
Scanned resources (62)      

Hint: use gen-driftignore command to generate a .driftignore file based on your drifts

Take a look at this drifts.json file. It contains the exact same output, but as a structured JSON:

				
					$ jq '.unmanaged' < drifts.json
[
  {
    "id": "AKIASBXWQ3AYR4KWTMHD",
    "type": "aws_iam_access_key"
  },
  {
    "id": "AKIASBXWQ3AYT7WXIBNE",
    "type": "aws_iam_access_key"
  },
[...]

Step 2 - Generate a .driftignore from JSON

Now we will simply use the new driftctl gen-drifignore command to generate a .driftignore file:

				
					$ driftctl gen-driftignore -i drifts.json > .driftignore

Take a look at the newly generated file, all the resources are now included with the proper format!

				
					$ head .driftignore
# Resources not covered by IaC
aws_iam_access_key.AKIASBXWQ3AYR4KWTMHD
aws_iam_access_key.AKIASBXWQ3AYT7WXIBNE
[...]

Profit!

Finally, we can enjoy a driftctl scan with all the drifts ignored, while keeping a complete list of them in the .driftignore file, so we don’t forget about them.

				
					$ driftctl scan --from tfstate://**/*.tfstate
Scanned states (4) 
Scanned resources (62)      
Found 1 resource(s)
 - 100% coverage
Congrats! Your infrastructure is fully in sync.

Wait! I'm using Docker.

Here’s a quick primer on how to achieve all the above steps very quickly using the official Docker image.

The pre-requisites for this example are:

~/.aws contains the AWS profiles, so we can share one named AWS_PROFILE
current directory contains a few Terraform states (available under /app inside the container)
~/.driftctl to reuse the local provider cache

Confirm we can actually run a successful driftctl scan using Docker:

				
					$ docker run -it --rm \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd):/app \
  -v ~/.driftctl:/root/.driftctl \
  -e AWS_PROFILE=cloudskiff-demo \
  -e AWS_REGION=us-east-1 \
  cloudskiff/driftctl scan --from tfstate://**/*.tfstate 
[...]

Now we can easily generate the JSON file:

				
					$ docker run -it --rm \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd):/app \
  -v ~/.driftctl:/root/.driftctl \
  -e AWS_PROFILE=cloudskiff-demo \
  -e AWS_REGION=us-east-1 \
  cloudskiff/driftctl scan --from tfstate://**/*.tfstate --output json://drifts.json
[...]

Finally generate the .driftignore file from the JSON:

				
					$ docker run -it --rm \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd):/app \
  -v ~/.driftctl:/root/.driftctl \
  -e AWS_PROFILE=cloudskiff-demo \
  -e AWS_REGION=us-east-1 \
  cloudskiff/driftctl gen-driftignore -i drifts.json > .driftignore
[...]

We now have our .driftignore file fully generated in seconds, using driftctl and Docker!

Next Steps

Now we’re starting from a clean state, the next step could be to generate your own HTML reports and quickly spot any relevant information easy to read and to share.

Stay in touch

Get product updates and occasional news.