Max’s notebook

Some Thoughts on Terraform CI for Monorepos

2022-01-08T00:00:00+00:00

Continuous integration and deployment for terraform monorepos is not a solved problem. I’m not proposing to solve it, but this is a record of my thoughts and experiments.

As an aside, CI for terraform stand-alone repos is, in fact, a very solved problem. See “Automate Terraform with GitHub Actions” in the References section for a very approachable example.

The Problem Space

We have a repo storing multiple terraform configurations or stacks (databases, users, kubernetes clusters, etc), and we want to organize it in a way that supports continuous integration and deployment.

Constraints

We are using terraform open source
We are using Github Actions as our CI tool
We need to be able to deploy changes to stacks independently

Using Terraform

To do this using terraform, we had two options:

create one workflow for each stack, as each stack will have it’s own state
- pro: fully isolated components and life-cycle management
- con: workflows scale linearly with stacks
pick a level of abstraction and have all resources share a workflow, state
- pro: workflows scale linearly with the level of abstraction (for example, environment)
- con: large blast radius as all resources share state
- con: harder to review changes as resource quantity grows
- con: every resource requires a count attribute for conditional creation (for example, create resource $FOO in staging but not prod)

Using Terragrunt

Terragrunt allows full access to all of the features of terraform while helping to address some of these concerns: maintain one workflow per level-of-abstraction, using terragrunt run-all to plan and apply resources while dynamic backend generation ensures separate state for each module. This is the route we chose in the end.

Final repo structure

├── prod
│   ├── app1
│   │   ├── main.tf
│   │   └── terragrunt.hcl
│   ├── app2
│   │   ├── main.tf
│   │   └── terragrunt.hcl
│   ├── cache
│   │   ├── main.tf
│   │   └── terragrunt.hcl
│   ├── database
│   │   ├── main.tf
│   │   └── terragrunt.hcl
│   └── terragrunt.hcl
└── staging
    ├── app1
    │   ├── main.tf
    │   └── terragrunt.hcl
    ├── app2
    │   ├── main.tf
    │   └── terragrunt.hcl
    ├── cache
    │   ├── main.tf
    │   └── terragrunt.hcl
    ├── database
    │   ├── main.tf
    │   └── terragrunt.hcl
    └── terragrunt.hcl

What Worked Well

Having a single point of entry makes it very easy to understand the changes getting rolled out to each environment, and we can (and do) include an environment-level plan for drift-detection during code review. Also, not needing to manage backend or provider configurations for each stack by hand is really nice.

What Didn’t Work so Well

There’s no consistent path for a stack to get from staging to production. I really like Kief Morris’ pipeline-per-stack model (read more about it in the “Using Pipelines to Manage Environments with Infrastructure as Code” article, linked in the References section), but I found a few drawbacks for our use-case:

more complex workflows are needed to manage multiple environments for each stack (and maybe the logic should live there?)
as mentioned above, the code in the stack itself becomes harder to reason about: counts or other conditional attributes are needed for each resource and/or module as it moves from dev to prod

What’s Next?

There are many other features that terragrunt has, like dependency blocks, which provide more in-depth configuration options (and remove the need for many data blocks) that I’m excited to explore. On the CI front, I’m eagerly awaiting updates on Hashicorp’s testing experiment, and whatever usability improvements we come across as we put more and more pressure on the current CI pattern.

References

Boto Over Time

2020-07-05T00:00:00+00:00

If you’ve worked with AWS using python, then you’ve come across the AWS SDK. The current generation is boto3, the previous version is boto, and you can use both side-by-side in the same code-base, and after a few incidents due to this, I will never do this thing.

How many ways can you grant an app running on an EC2 instance access to AWS resources?

Here are a few:

dedicated IAM credentials in an app-specific config file
dedicated IAM credentials in the .boto file or in the .aws/ directory
instance profiles or roles

Guess what I inherited? (hint: it was all of them.)

Bonus round: Configuration management

This application uses ansible for config management, which suffers from the exact same issue since it uses the same libraries, sometimes in parallel too (for an example, see the s3 module), so debugging and deploying reliable fixes was harder still.

…WAT

The application originally got it’s access by reading dedicated creds from the config file. While this isn’t ideal (roles with short-lived credentials ftw), I’ve seen it a lot.
Some crons and app functionality needed access to $UNIQUE_SERVICE_SET_1 and didn’t read from the config file, so it read from that user’s .boto or .aws files
When a new instance was provisioned, scripts run by cloud_init needed access to $UNIQUE_SERVICE_SET_2, so it read from the environment and got access through the instance profile and role
ansible… well, ansible DGAF #YOLOSWAG From the aws_s3 module docs:

Ansible uses the boto configuration file (typically ~/.boto) if no credentials are provided. See https://boto.readthedocs.io/en/latest/boto_config_tut.html

And then? We find it and kill it

Spelunk the App the first: find all the code that loads the IAM creds, and identify the services and calls made
Spelunk the App again: compare these calls against the IAM policy, and patch to match the code when needed
Remove the creds from the config file and cross your fingers
Test: did it work?
Ship it if so, fix it if not
Remove the user creds
Spelunk the config management: find the calls and services, remove unused, patch the policy where required
Remove the non-instance profile creds
Test it again: how about now?
How about the crons? You did check the crons, didn’t you? (Narrator: they did check the crons)
Ship it if so, fix it if not
Find surprise edge cases and cross-service library usage by watching breakage in prod
Cry while fixing and testing and shipping
Express anger at vague error messages
Express gratitude for fast deployments
Go to bed. It was a very long week

Ansible Notes

2020-05-03T00:00:00+00:00

A quick and dirty reference for ansible things I find useful.

define inventory

➜  ~ cat server_inventory
host1.example.com
host2.example.com

[server_group]
10.10.10.101
10.10.10.102    ansible_user=myuser ansible_ssh_private_key_file=my_key
10.10.10.103    ansible_user=not-me ansible_ssh_private_key_file=yo_key
10.10.10.104

remote ssh ftw

# run `systemctl restart yo-service` as root across all the servers in server_group, 
defined in server_inventory, and do it in groups of 10 at a time
ansible server_group -i server_inventory -b -m shell -a \
    "systemctl restart yo-service" -f10

remote script execution

# run a local script across a bunch of servers
ansible server_group -i server_inventory -b -m script -a my_script.sh

run an ad-hoc command across specific servers

# run `systemctl stop yo-other-service` on these specific hosts
ansible -b -m shell -a "systemctl stop yo-other-service" \
    10.10.10.10:\
    10.10.10.11:\
    10.10.10.12

playbook structure

➜  ~ cat playbook.yml
---
- hosts: "{{ hosts }}" # read hosts at runtime
  become: yes # with sudo
  gather_facts: false # don't need server metadata, just go

  tasks:
    - name: install nginx
      apt: name=nginx state=latest
    - name: restart nginx
      service: name=nginx state=restarted
      register: nginx_status
    - name: debug nginx status
      debug:
        var: nginx_status
    - name: debug message
      debug:
        msg: "output of nginx restart: "

run the playbook

# run the playbook `playbook.yml` against localhost
ansible-playbook playbook.yml -e hosts=localhost

role structure

➜  ansible-role tree
.
├── files # static files
│   └── pub_key
├── meta # dependencies on other roles
│   └── main.yml
├── tasks
│   ├── main.yml # top level tasks
│   └── nginx.yml # other tasks, needs to be included in `main.yml`
├── templates # templates ala jinja2
│   └── nginx.conf.j2 
└── vars
    ├── main.yml
    └── secrets.yml # needs to be included in `tasks/main.yml`

5 directories, 6 files

Bash Notes

2019-07-23T00:00:00+00:00

A quick and dirty reference for bash things I find useful.

Loops

one-liner:

for i in $(ls); do cat $i; done

in a shell script:

#!/bin/bash
for i in $(ls)
do
    cat $i
done

Read a csv as input

one-liner:

while IFS="," read col1 col2 col3; do echo $col3 | md5sum; done < my.csv

in a shell script:

#!/bin/bash
## usage: ./script.sh my.csv

CSV_FILE=$1

while IFS=, read col1 col2 col3
do 
    echo $col3 | md5sum
done < $CSV_FILE

kill all the pids

for pid in $(ps aux | grep [m]yprocess | awk '{ print $2} '); do kill -9 $pid; done
# explanation of the bracket syntax: https://askubuntu.com/a/153430
# The square bracket expression is part of ... grep's character class pattern matching.
# omission mine

Compression

ignore dir tree, just compress the files

# one file from the dir, ignore the tree
tar czfv archive.tgz -C /path/to/dir/ file_in_dir_to_archive

# the whole dir, ignore the tree
tar czfv archive.tgz -C /path/to/dir .

extract the files to dir

# tar Xtract Zip File
tar xzfv archive.tgz -C files/should/live/here

unzip to dir

unzip archive.zip -d files/should/live/here

misc

listen on $PORT with netcat

nc -lvnp 9000

serve `pwd` with python

python -m SimpleHTTPServer 9000

move stuff when you only have a network connection

cat my_bin | base64 | nc -v my.otherhost.com:9000

When Scaling Out Isn’t an Option

2019-06-09T00:00:00+00:00

This is a follow up of sorts to this post, where I wrote about load balancing, but I’m going to focus on configuring the application itself.

First as Transport, Then as Logic

Everything I discussed previously made the assumption that the application was horizontally scalable: we could successfully run many instances of the application/business logic component, and all would share the same data store. It turns out this was not true.

The application wasn’t actually capable of scaling horizontally, in spite of our tests and the assurances of the vendor/shepherd of the open source project. We learned when our three-node group became a one-node group, in Production. Like you do. Since running multiple nodes in parallel was no longer an option, we needed to figure out a way to deploy and manage what I’ll call a stateful pair: an application server using a unique configuration to access a dedicated database.

Chef to the Rescue(!)

As mentioned before, my employer is a big user (and I am a huge fan of) Chef. When building out the cookbook for the application, our targets were functionally identical: all of the nodes would be sharing the same configuration, so changes should be rolled out to all of the nodes. One cookbook reading the Chef environment for its attributes deployed to all the nodes–pretty clean and understandable, but this pattern only works when there’s one configuration per environment. Our use case had changed, and our cookbooks must as well.

I ended up re-writing the cookbook as a custom resource, which was thankfully straightforward since I’d already written the install and configuration logic. The biggest additions to the custom resource were adding a version specification for installation, and getting configuration values from variables instead of attributes.

Once completed, I wrote one cookbook using the new resource, provisioned a database, and updated the environment attributes for each node.

# using custom resource looks like this:

maxs_app 'node one' do
  my_database node['node_one']['db_name'].to_s
  my_user node['node_one']['db_user'].to_s
  my_password node['node_one']['db_password'].to_s
  my_heap node['node_one']['heap'].to_s
end

In The End

We have a unified configuration and installation for this application, both in a (sadly non-functional) clustered configuration, and in a stateful-pair, my broken three-node cluster in Prod is now a functional five un-clustered node group, and my configurations can still be versioned, and deployments can even be canary’ed! I’m still not thrilled that the clustering was broken, but it’s a much lower priority now that we’re running with more capacity and safe(er).

CIS Benchmarks and Packer

2018-08-27T00:00:00+00:00

We wanted to start using the security benchmarks provided by the Center for Internet Security around the same timeas needing to build an updated version of our Golden Image, so I thought it’d be a good time to kill two birds with one stone and put them together as one project.

The golden image had one particularly challenging and time consuming requirement: an encrypted root volume.

At the time, it was all done manually:

create temp_instance using a base image
stop temp_instance
create temp_ami from temp_instance
copy temp_ami and tell aws to encrypt the copy, creating encrypted_ami
terminate temp_instance and associated resources
deregister temp_ami
here is encrypted_ami

I wrote a script in python to accomplish this and have a record of the process. But wait a sec, isn’t this what Packer does?

At the time, the answer was “no.” When I started this project, I was working with Packer 1.2.X, possibly 1.1.X (unfortunately, I don’t know the specific version), which did not produce an encrypted AMI even when explicitly defining it in the template. Creating encrypted AMIs (not ec2 instances) out of the box also wasn’t suported by AWS at the time, though this may have changed.

Once we had the encrypted AMI, we started working on the CIS benchmarks using Packer and Chef. Thankfully, Packer supports Chef, so we could leverage all of our existing work and focus on the items found by scanning our instances using Chef Compliance, and add them to the Chef cookbooks used by Packer.

One issue we ran into was that occasionally cookbook dependencies weren’t found, breaking the Packer build, even after a successful berks install. My then-coworker (Thanks again, Andrew!) found a workaround for this: remove the version number from the cookbook in your berks directory, changing the missing directory from cookbook_0.1.0 to cookbook, which let Packer find it at run time.

For CentOS, the process was pretty straight forward. Find the open item from Chef Compliance, research the fix, add a recipe introducing the fix, or document the exception and associated reasoning, rinse, repeat. For Ubuntu, not so much. We were working of a 16.04 AMI that ran apt update during boot (which I believe is standard practice for all AMIs from Canonical). This presents an issue for installing the chef client: apt is already running as part of apt-daily.service, so we get Could not get lock /var/lib/apt/lists/lock , stopping chef from installing, breaking the build. This issue appears to be well-documented. The solution? Many sleeps.

We dealt with it using additional provisioners before running Chef:

  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
      ]
    },
    {
      "type": "shell",
      "script": "stop_updates.sh"
    },
    {
        "type": "chef-solo",
    # etc, etc, etc

stop_updates.sh itself:

#!/bin/bash

date
echo "stopping apt-daily.service"
sudo systemctl stop apt-daily.service
echo "stopped service"
sudo systemctl kill --kill-who=all apt-daily.service
echo "services should be killed"

echo "sleeping for 60 seconds"
sleep 60

echo "wait until apt run has completed..."
while [ `ps aux | grep -E '[a]pt|[d]pkg|[l]ock_is_held' | wc -l` -ne 0 ]; do
    echo "apt, dpkg, or lock process still found"
    sleep 5;
done
echo "there should be no more apt processes by now"
date

Kind of ugly all in all, but that’s how we did it.

Eventually, Packer updated and supported encrypted AMIs (hooray!), which allowed us to retire the python script, and move forward using Packer and Chef.

Useful Snippits

For the packer file itself:

  "provisioners": [
    {
        "type": "chef-solo",
        "cookbook_paths": [
            "{{user `cookbook_path`}}",
            "{{user `berks_path`}}"
        ],
        "run_list" : ["recipe[your_hardening_cookbook]"],
    }]
}

And the config file used by packer, since we had to support different accounts:

{
    "berks_path" : "",
    "cookbook_path": "",
    "profile": "",
    "region": "",
    "source_ami": "",
    "target_ami_regions": "",
    "subnet_id" : "",
    "vpc_id" : ""
}

Other Resources

TIL you cannot AWS-ELB All the Things

2018-08-21T00:00:00+00:00

08.14.2019 Update: After many conversations with AWS Support, the limit of 50 listeners is now a soft limit!

A recent project of mine was to take one of the core applications at my job from hand-rolled snowflake servers to documented and repeatable infrastructure. This is meant to be a record of how I built it, what problems I ran into, and how I did, or did not, solve them, in hopes of helping someone in the future–or possibly myself the next time around.

So here we go.

WHAT IS IT?

The application in question processes and routes messages sent over via TCP streams, as VPN tunneling is the preferred method of data transport (the application can receive messages via HTTP, but only a few customers deliver their data this way).

WHY WAS IT HARD?

AWS Load Balancer limitations
Replacing the load balancer
Fail over

HOW’D YOU DO IT?

Step One: Get a clean and reliable installation of the application

Nothing crazy, CloudFormation for infrastructure provisioning, Chef for configuration management, and Artifactory for artifact storage. Small highlight: the vendor doesn’t support running this application as a non-privileged user, and getting to the point where that was made clear took a very, very long time.

Step Two: reliably provision the database

Manual schema and user creation for the win. As far as I know, there isn’t a way to include database provisioning as a part of CloudFormation–once the RDS instance was created, we had to connect to the db and create the relevant schema and users. There is a possibility of addressing this by having the database master user/password used by the application, but I chose against this in the name of separation of concern.

Step Two-and-a-half: Provide new db details to Chef

The urls and passwords are created at runtime, so they can’t be passed to Chef in advance to allow for a more ephemeral approach.

Step Three: Load balancers

Shouldn’t that be part of step two? Aren’t you using AWS, so can’t you just use one of the Elastic Load Balancers? No. No I cannot. Even though that was the design provided by the vendor, still, I cannot. This is where I hit the first major issue: AWS load balancer limitations. The application requires port-driven routing: client one on port 9001, client two on port 9002, client two sending data type two on port 9003, etc, etc, etc. The AWS load balancers max out at 50 port-bound listeners, effectively supporting less than 50 clients. I have defeated the purpose of a load balancer when I go down the path of having multiple load balancers for the same application, each with their own configurations. As a replacement, I chose HAProxy, which supports load balancing TCP streams and is simple to configure (!).

After much research and fighting with embedded ruby and help from a very, very patient former coworker (Thanks Andrew!) I was able to use Chef to build a config for HAProxy that let me store client-specific configurations and application nodes in JSON:

node['maxs_haproxy']['clients'] = [
  {
    "client_name": "client_one",
    "client_port": 9001
  } 
]
node['maxs_haproxy']['app_nodes'] = [
  {
  "app_node_name": "node_one",
  "app_node_ip": "192.168.0.1"
  }
]

Hooray! I have a load balancer! It’s easy to manage the config! I know what’s what! This is great! I’m going home! I did not go home.

Step Three-and-a-half: Provide new load balancer and application instance details to Chef

Ips are created at runtime as well, can’t be passed to Chef in advance.

Step Four: How do I Fail Over?

I’m using a software-defined load balancer instead of one provided by my IaaS provider, and I don’t want a single point of failure. After some reasearch, I found that using keepalived for fail over with HAProxy is a popular pattern, but there were two issues I kept coming across while trying to set this up:

managing master/backup state (the nodes have to be self-aware) and IP addresses (the nodes have to be other-aware) for configuration at infrastructure provisioning time
understanding how fail over and recovery work in keepalived

Managing state – IPs change when you bring up new nodes (I didn’t want to hard-code the IPs in CloudFormation), so how do you manage dynamic ip addresses that directly impact application configuration? How do I know what the IP of my master AND backup node is at provisioning time, so this info can be passed to each node and keepalived can be configured correctly?

I don’t think I solved it the best way, but here’s what I did: during infrastructure provisioning, I assigned master/backup state to the nodes via Chef roles, and added the IPs as attributes to the Chef server, allowing the nodes to properly configure keepalived.

Failing over – keepalived can use VRRP as a heartbeat between nodes, checking to make sure that the master node is up and running. If the master isn’t available, or declares a failed status check, the backup will designate itself the master node, and run a script containing whatever application logic is required.

There are lots of tutorials and examples of keepalived configuration, but one thing that I kept missing (or that wasn’t in the guides I found), was the link between the health check and priority.

The initial state of a node is master or backup, and each is given a priority value, commonly 150 for the backup, and 151 for the master.

I didn’t realize that the health check should be weighted as well, allowing for a failed check to change the priority value of a node, along with changing the master-backup state.

This kept me stuck in a state where I could not recover to my master node, even when it became healthy again.

How it looks at the end

Room for improvement

Service discovery to handle master-backup state assignment and transfer

Max’s notebook

Some Thoughts on Terraform CI for Monorepos

The Problem Space

Constraints

Using Terraform

Using Terragrunt

Final repo structure

What Worked Well

What Didn’t Work so Well

What’s Next?

References

Boto Over Time

How many ways can you grant an app running on an EC2 instance access to AWS resources?

Bonus round: Configuration management

…WAT

And then? We find it and kill it

Ansible Notes

define inventory

remote ssh ftw

remote script execution

run an ad-hoc command across specific servers

playbook structure

run the playbook

role structure

Bash Notes

Loops

one-liner:

in a shell script:

Read a csv as input

one-liner:

in a shell script:

kill all the pids

Compression

ignore dir tree, just compress the files

extract the files to dir

unzip to dir

misc

listen on $PORT with netcat

serve pwd with python

move stuff when you only have a network connection

When Scaling Out Isn’t an Option

First as Transport, Then as Logic

Chef to the Rescue(!)

In The End

CIS Benchmarks and Packer

Useful Snippits

Other Resources

TIL you cannot AWS-ELB All the Things

WHAT IS IT?

WHY WAS IT HARD?

HOW’D YOU DO IT?

Step One: Get a clean and reliable installation of the application

Step Two: reliably provision the database

Step Two-and-a-half: Provide new db details to Chef

Step Three: Load balancers

Step Three-and-a-half: Provide new load balancer and application instance details to Chef

Step Four: How do I Fail Over?

How it looks at the end

Room for improvement

Resources

serve `pwd` with python