A collection of sorts
27 Aug 2018
We wanted to start using the security benchmarks provided by the Center for Internet Security around the same timeas needing to build an updated version of our Golden Image, so I thought it’d be a good time to kill two birds with one stone and put them together as one project.
The golden image had one particularly challenging and time consuming requirement: an encrypted root volume.
At the time, it was all done manually:
I wrote a script in python to accomplish this and have a record of the process. But wait a sec, isn’t this what Packer does?
At the time, the answer was “no.” When I started this project, I was working with Packer 1.2.X, possibly 1.1.X (unfortunately, I don’t know the specific version), which did not produce an encrypted AMI even when explicitly defining it in the template. Creating encrypted AMIs (not ec2 instances) out of the box also wasn’t suported by AWS at the time, though this may have changed.
Once we had the encrypted AMI, we started working on the CIS benchmarks using Packer and Chef. Thankfully, Packer supports Chef, so we could leverage all of our existing work and focus on the items found by scanning our instances using Chef Compliance, and add them to the Chef cookbooks used by Packer.
One issue we ran into was that occasionally cookbook dependencies weren’t found, breaking the Packer build, even after a successful berks install. My then-coworker (Thanks again, Andrew!) found a workaround for this: remove the version number from the cookbook in your berks directory, changing the missing directory from cookbook_0.1.0 to cookbook, which let Packer find it at run time.
For CentOS, the process was pretty straight forward. Find the open item from Chef Compliance, research the fix, add a recipe introducing the fix, or document the exception and associated reasoning, rinse, repeat. For Ubuntu, not so much. We were working of a 16.04 AMI that ran apt update during boot (which I believe is standard practice for all AMIs from Canonical). This presents an issue for installing the chef client: apt is already running as part of apt-daily.service, so we get Could not get lock /var/lib/apt/lists/lock , stopping chef from installing, breaking the build. This issue appears to be well-documented. The solution? Many sleeps.
We dealt with it using additional provisioners before running Chef:
"provisioners": [
{
"type": "shell",
"inline": [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
]
},
{
"type": "shell",
"script": "stop_updates.sh"
},
{
"type": "chef-solo",
# etc, etc, etc
stop_updates.sh itself:
#!/bin/bash
date
echo "stopping apt-daily.service"
sudo systemctl stop apt-daily.service
echo "stopped service"
sudo systemctl kill --kill-who=all apt-daily.service
echo "services should be killed"
echo "sleeping for 60 seconds"
sleep 60
echo "wait until apt run has completed..."
while [ `ps aux | grep -E '[a]pt|[d]pkg|[l]ock_is_held' | wc -l` -ne 0 ]; do
echo "apt, dpkg, or lock process still found"
sleep 5;
done
echo "there should be no more apt processes by now"
date
Kind of ugly all in all, but that’s how we did it.
Eventually, Packer updated and supported encrypted AMIs (hooray!), which allowed us to retire the python script, and move forward using Packer and Chef.
For the packer file itself:
"provisioners": [
{
"type": "chef-solo",
"cookbook_paths": [
"{{user `cookbook_path`}}",
"{{user `berks_path`}}"
],
"run_list" : ["recipe[your_hardening_cookbook]"],
}]
}
And the config file used by packer, since we had to support different accounts:
{
"berks_path" : "",
"cookbook_path": "",
"profile": "",
"region": "",
"source_ami": "",
"target_ami_regions": "",
"subnet_id" : "",
"vpc_id" : ""
}