Gruntwork Newsletter, November 2017

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we made a bunch of changes to make it easier to keep your code DRY, work with multiple AWS accounts, and work with multiple Terraform modules. We also created a crowdfunding spreadsheet to fund new infrastructure and fixed a number of bugs.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

Make your code more DRY by using Terragrunt with shared variable files

Terragrunt

Motivation: The Terraform code in the infrastructure-live repo we set up for most customers is not as DRY as it could be. In particular, many variables, such as account ID and region, are repeated multiple times, making it harder to maintain the code.

Solution: Our first step in improving this situation is to add a small tweak to Terragrunt that extends the find_in_parent_files helper so you can specify the filename to search for (e.g.: find_in_parent_files(“account.tfvars”)) and even a fallback value in case that file isn’t found (e.g.: find_in_parent_files("account.tfvars", “fallback.tfvars")). You can combine this with the the extra_arguments helper to pull in common variable files for all of your modules. For example, you could create account.tfvars, region.tfvars, and environment.tfvars files that contain oft-repeated variables such as account ID and AWS region, and pull the variables in those .tfvars files automatically into every module by adding the following to the root terraform.tfvars:

Copy
terraform {
extra_arguments "common_vars" {
commands = ["${get_terraform_commands_that_need_vars()}"]

Copy
optional_var_files = [
"${find_in_parent_folder("account.tfvars", "ignore")}",
"${find_in_parent_folder("region.tfvars", "ignore")}",
"${find_in_parent_folder("environment.tfvars", "ignore")}",
]
}
}

The optional_var_files helper adds-var-file arguments to your Terraform commands if the file you specify exists.

What to do about it: Update to Terragrunt v0.13.9 and try out this new approach to make your infrastructure-live repo more DRY! Check out this PR and this PR in the Acme Reference Architecture for an example of the types of changes you’ll need to make and browse the Acme infrastructure-live repo to see the result (note: only customers who purchased the Gruntwork Reference Architecture have access to the Acme repos).

Make it easier to work with multiple AWS accounts on the CLI

Terragrunt

Motivation: If your IAM users are defined in one AWS account (e.g., the security account) and your infrastructure is defined in several other AWS accounts (e.g., the dev, stage, and prod accounts), to switch between accounts, you’d have to assume an IAM role in those other accounts, which is a tedious process on the CLI. You’d have to run a bunch of commands (albeit, aws-auth made that a little easier) and the credentials you get back are only good for 1 hour!

Solution:

• Terragrunt v0.13.10: Terragrunt will assume any IAM role specified in the role_arn parameter of the S3 backend config before creating the S3 bucket or DynamoDB table.
• Terragrunt v0.13.11: You can set the --terragrunt-iam-role parameter (or the corresponding TERRAGRUNT_IAM_ROLE environment variable) to tell Terragrunt to assume an IAM role before running any Terraform commands. Terragrunt will run all the STS assume-role commands for you every time you execute it, so you no longer have to jump through hoops, and you get new credentials for every single command you execute, so the 1-hour expiration should no longer be a problem.
• gruntkms v0.0.6: You can set the --role-arn parameter (or corresponding ROLE_ARN environment variable) to tell Terragrunt to assume an IAM role before running any KMS commands. This makes it easier to encrypt/decrypt secrets in other AWS accounts.

What to do: Install Terragrunt v0.13.11 and gruntkms v0.0.6 and start using the --terragrunt-iam-role and --iam-role parameters to easily switch between accounts!

Multi-account support for OpenVPN

OpenVPN Package

Motivation: Several customers wanted to use OpenVPN with a multi-account setup where all the IAM users are defined in one AWS account (e.g., the security account) and each environment is defined in other AWS accounts (e.g., the dev, stage, and prod accounts). Our OpenVPN implementation uses IAM groups to manage who has access to the OpenVPN server, but you can only add IAM users from the same account to an IAM group, whereas with multi-account setups, those IAM users may be in a totally separate account.

Solution: We updated package-openvpn to work better with multiple accounts. You can now assume an IAM role in the AWS accounts where your OpenVPN servers are deployed to request/revoke certs.

What to do: Check out package-openvpn, v0.4.0 for upgrade instructions.

Multi-account reference architecture examples (Acme)

Reference Architecture

Motivation: Several customers asked us for a repo that shows examples of how to use the Gruntwork modules in a multi-account configuration, where each environment (e.g., stage, prod, etc) is deployed in a separate AWS account. This gives you much more fine-grained control over security, and complete isolation between accounts.

Solution: We created example repos for a fictional company called Acme that shows how you can use our modules and Terraform with multiple AWS accounts: Acme multi-account reference architecture. This is in addition to the Acme single-account reference architecture examples we added a few months ago.

What to do: Check out the new example code. Note that you’ll only be able to access the Acme repos if you purchased the Reference Architecture!

Important logging fix

AWS Monitoring Package

Motivation: A few of our customers were trying to send multiple log files to CloudWatch Logs using the run-cloudwatch-logs-agent.sh script, but only syslog would ever actually show up.

Solution: We fixed a bug in the run-cloudwatch-logs-agent.sh script so it sets a unique name log stream name for each log file, so all log files now show up in the CloudWatch Logs UI!

What to do: We highly recommend updating all your Packer templates to use v0.7.0 of module-aws-monitoring, rebuilding your AMIs, and redeploying.

Make it easier to work with multiple modules

Terragrunt

Motivation: Terragrunt has had support for xxx-all commands (e.g., apply-all, plan-all, etc) for a while, but they were error prone to use.

Solution: We are starting to improve the xxx-all commands so you can use them on a regular basis to spin up or tear down an entire environment:

• Terragrunt v0.13.12: added a validate-all command that makes it easy to check if all of your dependencies are defined correctly and your Terraform code is free of basic errors.
• Terragrunt v0.13.13: updated the xxx-all commands so that they work with the —-terragrunt-source parameter, which makes testing easier by allowing you to deploy multiple modules from a local checkout rather than published code.
• Terragrunt v0.13.14: fixed a bug where Terragrunt would silently swallow parsing errors during the xxx-all commands.
• Terragrunt v0.13.15: fixed a bug with recursive dependencies and with multiple modules trying to create the same S3 bucket at the same time.
• Terragrunt v0.13.16: Terragrunt will now cache downloaded code in your HOME directory instead of tmp. This should hopefully finally fix caching issues!

What to do: Grab Terragrunt v0.13.15, starting using the xxx-all commands, and let us know what other improvements could make this better!

Fix ENI issue with Stateful Server, AMI Cluster, Kafka, and ZooKeeper

Stateful Server, AMI Cluster, Kafka, ZooKeeper

Motivation: We created an attach-eni script that could be used to attach an ENI on Amazon Linux, where Amazon handles all the local ethernet config details automatically, and Ubuntu, where the script handles all the ethernet config details. Unfortunately, it looks like the way we handle those details works on t2 instances, but not other instance types, such as m4 and c4, as it turns out different instance types mount ethernet devices differently.

Solution: We fixed this bug in module-server, v0.2.1. Since this script is used under the hood by our AMI cluster, Kafka, and ZooKeeper modules, we also updated all of those, releasing module-asg, v0.6.3, package-zookeeper, v0.0.10, and package-kafka, v0.0.9.

What to do: Update to the versions above if you’re using instance types other than t2.

Crowdfunding new infrastructure; now with deadlines

New infrastructure

Motivation: We created a simple spreadsheet as a poor-man’s crowdfunding solution for new infrastructure projects. The goal was (a) to be more transparent about our funding process and (b) provide access to better infrastructure at the lowest cost anywhere by splitting the cost amongst many companies. Although customers keep asking us for new infrastructure, it seems like few of you have added your names to that spreadsheet. This makes it very difficult for us to plan our roadmap and commit to deadlines.

Solution: We have added “drop dead dates” for each project. If not enough companies have signed up for a project by the time that date passes, the project is canceled, and we will focus on something else instead. Projects such as API Gateway, ELK, and Kubernetes are all going to expire soon without your help!

What to do: If you need new infrastructure, simply waiting and hoping someone else will pay for it won’t work! Please put your name down in the Crowdfunding Spreadsheet ASAP, or the project won’t happen at all. If you have questions or feedback, please email us at support@gruntwork.io.

Other updates:

• Terragrunt v0.13.6: Terragrunt should now do a better job of checking if S3 backends need initialization, including creating the DynamoDB table automatically, even if the S3 bucket already exists.
• Terragrunt v0.13.7: Terragrunt now properly handles the dynamodb_table param, auto-creating the DynamoDB table as necessary.
• module-aws-monitoring, v0.6.0: The asg-xxx-alarms modules now allow you to create alarms for a list of ASGs, rather than just one.
• package-zookeeper, v0.0.9: Add execute permissions on run-exhibitor script, as they seem to get lost after a Packer build.
• terraform-aws-vault, v0.0.4: You can now specify 0 or more security group IDs that allow SSH access to the Vault cluster.
• terraform-aws-vault, v0.0.5: You can now specify 0 or more CIDR blocks that allow inbound access to the Vault cluster.
• terraform-aws-consul, v0.0.4: You can now control access to the cluster using security groups.
• package-lambda, v0.1.1: Fixed the function_name output in the lambda module.
• module-vpc, v0.3.1: Added VPC endpoints for DynamoDB, so all of your requests to DynamoDB now stay within the VPC rather than going over the public Internet.
• module-vpc, v0.3.2: Add output for private persistence subnet with proper naming convention.
• module-data-storage, v0.3.1: The aurora module now exposes engine and engine_version parameters so you have more control over what type of Aurora engine you’re running (e.g., you can use the Postgres-compatible one).

DevOps News

Amazon Elasticsearch Service now supports VPCs

What happened: AWS has announced that its managed Elasticsearch Service now works within your VPCs.

Why it matters: In the past, to talk to your Elasticsearch cluster, you had to send all requests over the public Internet. With VPC support, all traffic between your apps and the Elasticsearch cluster stay within the AWS network, which makes things more secure by default.

What to do about it: Version 1.2.0 of the Terraform AWS provider has support for this new functionality (see the CHANGELOG for details). Use the vpc options in the aws_elasticsearch_domain resource.

Amazon ElastiCache for Redis now supports encryption

What happened: Amazon ElastiCache for Redis now supports encryption for all data at rest and all data in transit.

Why it matters: You now can have end-to-end encryption with Redis, which improves your security posture, and is required for many types of compliance (e.g., HIPAA, PCI).

What to do about it: The Terraform AWS provider does not yet support this functionality (follow this issue for details). If you need to enable this now, you’ll have to do it manually in the console.

Docker is making Kubernetes a first-class citizen

What happened: Docker has announced that Kubernetes will become a “first-class citizen” of Docker Community Edition and Docker Enterprise Edition.

Why it matters: It’s not entirely clear what “first-class citizen” really means, but this is a strong sign that Kubernetes is winning the Docker orchestration wars.

What to do about it: If you’re interested in funding a Kubernetes module (i.e., instead of ECS or Nomad), add your name to the Crowdfunding Spreadsheet! This project will expire if we don’t get enough backers by December 15th.

ECR lifecycle policies

What happened: Amazon ECR now supports lifecycle policies, which allow you to define rules for cleaning up your ECR repos.

Why it matters: Without these rules, the repos gradually fill up with Docker images, and it’s harder to figure out what versions you should use. You can now add a rule, for example, to expire all Docker images that are more than 14 days old.

What to do about it. Version 1.2.0 of the Terraform AWS provider has support for this new functionality (see the CHANGELOG for details). Use the aws_ecr_lifecycle_policy resource.

Amazon Aurora now has PostgreSQL compatibility

What happened: Amazon Aurora announced compatibility with PostgreSQL a while back and now this functionality is generally available.

Why this matters: Amazon claims that Aurora offers up to 3x better performance than PostgeSQL, while exposing a compatible API. It’s a fully managed service, so it has automatic failover, backups, and replication functionality built-in.

What to do about it: Try it out by launching an Aurora RDS instance! We recently released module-data-storage, v0.3.1, which allows you to specify the new engine and engine_version parameters for this purpose.

The ALB now supports multiple TLS certs

What happened: Amazon’s Application Load Balancer (ALB) now allows you to attach multiple TLS certificates to each listener.

Why this matters: You can now using a single ALB for all of your load balancing, even across multiple domain names, and it will pick the proper TLS cert for that domain name automatically. This was possible with TLS certs that specified multiple domain names before, but now it’s easy to do with any TLS certs.

What to do about it: Terraform does not yet have support for this new functionality (follow this issue for details), so if you want to use it, you’ll have to enable it manually in the AWS console.

Talk: Reusable, composable, battle-tested Terraform Modules

What happened: The video and slides are now available for Jim’s HashiConf talk, Reusable, composable, battle-tested Terraform Modules.

Why it matters: The talk is an overview of why modules will change how you build and manage infrastructure, as well as a tutorial on how to build modules that are highly reusable and tested.

What to do about it: Go watch and share the talk!

Security Updates

Below is a list of security updates that may impact your services. It is up to you to scan this list and decide which of these apply and what to do about them. Note that for critical security updates, we send out emails immediately to the Gruntwork Security Alerts list.

Jenkins

• Security Advisory 2017–10–11, SA 2017–10–23: A number of security vulnerabilities were found in Jenkins and its plugins, including several that allow arbitrary code execution for logged-in users.

WordPress

• WordPress 4.8.2: Several vulnerabilities were discovered in Wordpress, a web blogging tool. They would allow remote attackers to exploit path-traversal issues, perform SQL injections and various cross-site scripting attacks. These are fairly serious vulnerabilities, so we strongly recommend updating immediately.

Git

• USN-3438: Multiple vulnerabilities were discovered in Git that could allow an attacker to possibly run arbitrary code.

Ruby

• USN-3439: Multiple vulnerabilities were discovered in Ruby.

NodeJS

• CVE-2017–14919: A vulnerability was discovered with zlib that allowed a potential Denial of Service attack. Upgrade to NodeJS v8.8.0 or higher to resolve the issue.

Play Framework

• 20171005-CorsVaryHeader: Play’s CORS filter will overwrite the Vary header in certain configurations, which in some circumstances can cause cache poisoning. This could expose sensitive information to unauthorized users.

PostgreSQL

• ALAS-2017–908: A couple security vulnerabilities were found in Postgres 9.6 that may allow remote attackers to gain access to certain accounts or passwords.

Linux

• USN-3441–1: Multiple vulnerabilities were discovered in curl, including ones that allow for remote execution of arbitrary code if an attacker compromised an endpoint called by curl.
• USN-3454–1: The libffi library, which is used by Python, Haskell, JRuby, and Java (among others) to call functions compiled in “foreign” languages was found to have a vulnerability that could facilitate execution of arbitrary code.
• USN-3456–1: A number of vulnerabilities were found in the Linux kernel, including some that could allow remote attackers to perform DoS attacks or possibly execute arbitrary code.
• USN-3458–1: A vulnerability was found with the “International Components for Unicode” (ICU) library that allows an attacker to execute arbitrary code given a carefully crafted string. Note that many Java, C- and C++-based libraries rely on ICU.
• USN-3461–1: NVIDIA display drivers were found to have a number of vulnerabilities. This may affect CUDA users that rely on these drivers.
• USN-3464–1: Several vulnerability were found with wget, including ones that allow possible remote execution of arbitrary code if the attacker exploited an endpoint called by wget.

Amazon Linux

Amazon Linux is subject to all the vulnerabilities listed in the “Linux” section, but Amazon Linux users generally upgrade when Amazon issues new releases. Note that in some cases, the severity is “critical” because Amazon Linux packages install old, outdated version of tools you would normally install on your own, such as Java or NodeJS.

• ALAS-2017–913: Fix a vulnerability in Tomcat.
• ALAS-2017–916: Fix an important vulnerability in wget.
• ALAS-2017–917: A critical update for Amazon Linux.

OpenSSL

• OpenSSL 1.02m / 1.10g: This release fixes CVE-2017–3736 and CVE-2017–3735. It is of moderate importance. Note that some languages like NodeJS that rely on OpenSSL may issue separate releases to incorporate these changes, depending on the severity (in this case “moderate”) of the vulnerability.

MySQL Server

• DSA-4002–1: Several issues have been discovered in the MySQL database server. The vulnerabilities are addressed by upgrading MySQL to the new upstream version 5.5.58, which includes additional changes, such as performance improvements, bug fixes, new features, and possibly incompatible changes. Please see the MySQL 5.5 Release Notes and Oracle’s Critical Patch Update advisory for further details. Note that, as of November 6, 2017, Amazon RDS does not yet support these versions.

X-Server

• DSA-4000–1: Several vulnerabilities have been discovered in the X.Org X server. An attacker who’s able to connect to an X server could cause a denial of service or potentially the execution of arbitrary code.

WPA Protocol

• DSA-3999: Mathy Vanhoef of the imec-DistriNet research group of KU Leuven discovered multiple vulnerabilities in the WPA protocol, used for authentication in wireless networks. Those vulnerabilities applies to both the access point (implemented in hostapd) and the station (implemented in wpa_supplicant).

Got questions or feedback on the newsletter? Email us at support@gruntwork.io. Want to learn more about Gruntwork? Check out https://www.gruntwork.io/.