aws

Gruntwork Newsletter, March 2019

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the…
Gruntwork Newsletter, March 2019
YB
Yevgeniy Brikman
Co-Founder
Published February 7, 2019

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, Terraform 0.12 beta1 and terraform-aws-provider 2.0.0 came out, both of which have great new features, but also significant backwards incompatibilities; details on our upgrade plans below. Also, we started expanding our terraform-aws-influx module with support for the entire TICK stack (Telegraf, InfluxDB, Chronograf, and Kapacitor), added OAuth support to Gruntwork Houston, updated Terratest with support for testing Helm charts, updated Terragrunt’s security and access logging, and made many other updates and fixes.

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates

Terraform 0.12 and terraform-aws-provider 2.0.0 plans

Motivation: In the last week or so, there were two major releases:

Both of these contain some great new features, but they also include significant backwards incompatibilities and they will NOT work with some of the repos in the Infrastructure as Code Library, as well as the code in your repos!

Solution: Here are our plans for these two new releases:

  1. We will test Terraform 0.12 beta1 over the next few weeks to see how stable it is. If it seems reasonably stable, we will start the process of updating all 300,000+ lines of code in our Infrastructure as Code Library to support it and take advantage of the many new features.
  2. We started this week to update our repos to support AWS Provider 2.0.0. It works fine with most of our code, but there are known backwards compatibilities with at least several repos, including module-ecs and package-static-assets. Until the upgrade is done, the best workaround is for you to pin the AWS provider version in all your modules (e.g., in your infrastructure-modules repo) as follows:
provider "aws" {
# Provider version 2.X series is the latest, but has
# breaking changes with 1.X series, so we pin to 1.X.
version = "~> 1.60.0"
region = "us-east-1"

# ... the rest of your code
}

See this commit for the upgrade we did to the Acme Reference Architecture examples to pin all the modules.

What to do about it: For now, your best bet is to pin your AWS provider to 1.X. We’ll let you know as soon as our AWS Provider 2.0.0 upgrade is complete, as well as the outcome of our Terraform 0.12 investigations.

TICK Package

Motivation: In December, we open sourced our modules for running InfluxDB Enterprise on AWS. We’ve now started working with the InfluxData team to expand these modules with support for their entire time-series platform, known as the TICK stack (Telegraf, InfluxDB, Chronograf, and Kapacitor).

Solution: We’ve updated the terraform-aws-influx module to include a set a reusable modules to deploy and run a TICK cluster in AWS. These modules can be combined configured in a variety of ways to meet your specific needs.

We’ve added modules to do the following:

  • Run Telegraf on an application server to forward logs and machine metrics information to InfluxDB.
  • Run a Chronograf server to provide a web-interface for the rest of the TICK stack.
  • Support for Kapacitor is coming soon!

What to do about it: All of this new code is in the terraform-aws-influx repo and fully open source! We also have some examples to help get you started.

Gruntwork Houston now support OAuth!

Motivation: Authentication is a tricky beast on the best of days. Managing credentials for AWS console access, CLI access, SSHing to your servers, and requesting VPN certificates is far more difficult than it should be. We created Gruntwork Houston to allow you to manage all of this with your existing Identity Provider (e.g., Google, Active Directory, Okta, etc) via SAML, but it turns out that configuring SAML is far more difficult than it should be too!

Solution: We’ve updated Gruntwork Houston to be able to use OAuth authentication via GitHub. With this approach you and your entire team will now be able to get secure access to all your AWS accounts via web and CLI, ssh to your EC2 instances, and request VPN certificates from your VPN servers by just authenticating via your GitHub account:

We’ll also be adding OAuth via Google in the near future!

What to do about it: If you’re interested in trying out Gruntwork Houston with GitHub or Google OAuth, contact us at support@gruntwork.io!

Terratest Now Supports Helm Chart Testing

Motivation: As we started developing our own helm charts, we realized that there wasn’t a clear standard on how you might test the charts. In the past two months, we have expanded our Kubernetes offerings in Terratest, Gruntwork’s swiss army knife for infrastructure testing. This month we added our initial support for helm charts.

Solution: We identified two tiers of testing in Helm: Template tests and Integration tests.

  • Helm Template tests are tests designed to test the logic of the templates. These tests should run helm template with various input values and parse the yaml to validate any logic embedded in the templates (e.g by reading them in using client-go). Since templates are not statically typed, the goal of these tests is to promote fast cycle time
  • Helm Integration tests are tests that are designed to deploy the infrastructure and validate that it actually works as expected. If you consider the templates to be syntactic tests, these are semantic tests that validate the behavior of the deployed resources.

This month we added support in Terratest to provide a framework for executing both kinds of tests against your charts.

What to do about it: Check out v0.14.2 and give it a spin. You can view examples/helm-basic-example for a helm chart and the corresponding tests (helm_basic_example_template_test.go and helm_basic_example_integration_test.go) for an example of how to use Terratest to run Helm Tests. Also checkout our blog post demonstrating the new capabilities.

Other Terratest updates:

  • v0.13.26: This release updates the Google Cloud (cloud.google.com/go) dependency to v0.35.1.
  • v0.13.27: terraform.Options now exposes a VarFiles field that you can use to pass variable files to your Terraform commands at test time (via the -var-file flag) and a Targets field that you can use to specify targets for your Terraform commands at test time (via the -target argument).
  • v0.13.28: Introduce integration testing capabilities for helm .
  • v0.13.29: Add test-structure helper for save/load kubectl options (test_structure.SaveKubectlOptions and test_structure.LoadKubectlOptions).
  • v0.13.30: Add Get helpers for the ClusterRole (GetClusterRole) and Role (GetRole) resources.
  • v0.14.0: Standardize all k8s functions to take in KubectlOptions so that the kubeconfig path and context is configurable. This is a breaking change in the k8s module. See the release notes for more details.
  • v0.14.1: We now publish checksums for our binaries as part of our releases.
  • v0.14.2: helm.Install now supports installing remote charts for testing purposes.

Encryption and access logging for Terragrunt

Motivation: Some Terragrunt users wanted Terragrunt to have more secure settings when using Terragrunt to configure S3 buckets and DynamoDB tables for Terraform state storage.

Solution: Terragrunt now does the following:

  • Server-side encryption for S3 buckets is enabled by default. Terraform state may contain secrets, so this is a good security measure to have in place.
  • Access-logging for S3 buckets is enabled by default. This gives you a record of who has been accessing your Terraform state.
  • Server-side encryption for DynamoDB is NOT enabled by default (Terraform doesn’t store any secrets in DynamoDB and enabling encryption takes 1–2 minutes), but can easily be enabled by setting the enable_lock_table_ssencryption setting to true.

What to do about it: If you’d like to take advantage of these new security settings, grab Terragrunt v0.18.0.

Open Source Updates

  • terraform-aws-couchbase, v0.1.4: You can now configure custom tags for Couchbase’s Security Group using the new, optional security_group_tags input parameter.
  • terraform-aws-vault, v0.11.4: You can now enable S3 bucket versioning when using S3 as storage backend.
  • fetch, v0.3.3: Release assets can now be identified using regex. If multiple assets match the regex, fetch will automatically download all of them. You can also now pass in multiple checksums, only allowing the download if the computed checksum matches any of the provided ones.
  • cloud-nuke, v0.1.5: Cloud Nuke now supports deleting EKS cluster resources.
  • pre-commit, v0.0.8: Updated the shellcheck pre-commit hook to only match *sh shebangs (e.g., sh, bash, zsh, etc). It used to (inadvertently) match a much broader range of files, such as bats tests.
  • bash-commons, v0.1.1: Extracted lots of great helper functions from package-kafka and package-zookeeper and moved them to bash-commons. These include array_split, array_prepend, assert_exactly_one_of, file_replace_text_in_files, os_user_exists, os_create_user, os_change_dir_owner, and a number of AWS EC2 and ENI functions.

Other Updates

  • module-aws-monitoring, v0.12.0: Fix errors in the new connection count and low request count alarms to remove the “client-tls-negotiation-error” portion that was accidentally copy/pasted into them.
  • module-security, v0.15.8: iam-groups module now creates an additional IAM group that has the iam-user-self-mgmt IAM policy already attached to make it easier to associate the rules of that policy to an IAM user via the group.
  • module-security, v0.16.0: Update the fail2ban module so it works properly on Amazon Linux 2. Update the ami-builder in os-hardening to support a new parallel_build param that lets you control whether the builds run in parallel and call udevadm settle in the partition-volume script to ensure all symlinks are in place before going on to subsequent steps (e.g., formatting).
  • module-ecs, v0.11.5: Fixes bug in the ecs-cluster module where it wouldn’t do the proper rollout in clusters with more than 10 instances.
  • kubergrunt, v0.3.1: You can now specify the Tiller container image to use with helm deploy, using the --tiller-image and --tiller-version flags. Additionally, helm configure now initializes the helm home directory with the stable repository.
  • kubergrunt, v0.3.2: kubergrunt can now authenticate with GKE.
  • terraform-kubernetes-helm, v0.2.2: This release breaks out the role creation pieces of k8s-namespace into its own submodule, k8s-namespace-roles. This allows you to create the same roles on a preexisting namespace (e.g default or kube-system).
  • terraform-kubernetes-helm, v0.2.3: This release adds another set of permissions to the rbac_tiller_resource_access role that allows Tiller to manage PodDisruptionBudgets.
  • package-static-assets, v0.4.1: You can now customize the response codes CloudFormation sends for 404 and 500 errors using the new input parameters error_404_response_code and error_500_response_code, respectively.
  • package-static-assets, v0.4.2: Fix compatibility issues with AWS provider 2.0.0
  • package-elk, v0.2.8: This release adds ability for the user to pass through custom JVM options for logstash. Previously we were relying on default JVM options. Now — our installation will pass through a templated jvm.options file. The user can then pass parameters through terraform/user-data script to the run-logstash script via a new parameter --auto-fill-jvm (eg:--auto-fill-jvm '<__XMS__>=4g')
  • gruntkms, v0.0.8: gruntkms will now write errors to stderr instead of stdout.
  • module-asg, v0.6.25: Fixes issue #63, where ALB/ELB health checks were being skipped in the rolling deployment script for the server-group module when using python3.
  • package-zookeeper, v0.5.3: Update module-asg dependency to v0.6.25 to address skipped ALB/ELB health checks when using python3.
  • package-kafka, v0.5.1: Update module-asg dependency to v0.6.25 and package-zookeeper to v0.5.3 to address skipped ALB/ELB health checks when using python3.
  • module-load-balancer, v0.13.1: You can now attach additional security groups to your load balancer using the additional_security_group_ids input parameter.
  • module-ci, v0.13.9: The update-terraform-variable script now uses pipes (|) instead of slashes (/) in a sed call so that you don't get errors if the --value parameter contains a slash.
  • package-messaging, v0.1.4: The kinesis -module now supports server-side encryption.

DevOps News

New features in Amazon DocumentDB

What happened: Amazon DocumentDB has added a number of new features around auditing/logging and aggregations, arrays, and indexing.

Why it matters: Amazon recently launched DocumentDB as a hosted version of MongoDB. As usual with AWS launches, the initial version was fairly minimal, but several updates have been released that make the service more compelling:

  • Auditing of database events: DocumentDB can now send Data Definition Language (DDL), authentication, authorization, and user management events to Amazon CloudWatch Logs.
  • Aggregations, arrays, and indexing:**** DocumentDB now supports new aggregation string operators ($concat, $substr, $substrBytes, $substrCP, $strcasecmp), an array aggregation operator ($size), an aggregation group accumulator operator ($push), and aggregation stages ($redact and $indexStats) that allow you to compose powerful aggregations over your documents. Additionally, it also supports positional array operators ($[] and $[<identifier>]) for updating elements in an array and hint() for selecting an index.

What to do about it: Check out the event auditing docs and this blog post for more info.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Docker

  • CVE-2019–5736: Two researchers recently demonstrated a way for a Docker container to overwrite the runc engine that runs Docker containers. This means that a malicious container could run arbitrary code on the host computer. All it takes to exploit this vulnerability is for users to run the wrong Docker container. Therefore, we recommend immediately updating all version of Docker to at least v18.09.02. We notified the Security Alerts mailing list about this vulnerability on February 12th.

Chrome

  • CVE-2019–5786: A new zero-day vulnerability in Google Chrome has been announced. This a high severity flaw that can be exploited by a remote attacker to execute arbitrary code and take full control of the target computer. The vulnerability, tracked as CVE-2019–5786, impacts all major operating systems, is so bad that few details are being released until most users have upgraded, and is actively being exploited in attacks in the wild (so this is not some theoretical issue!). Update your Chrome browser immediately! We notified the Security Alerts mailing list about this vulnerability on March 8, 2019.