Read our newest book, Fundamentals of DevOps and Software Delivery, for free!

Introducing: The Gruntwork Module, Service, and Architecture Catalogs

Headshot of Yevgeniy Brikman

Yevgeniy Brikman

AUG 26, 2020 | 11 min read
Featured Image of Introducing: The Gruntwork Module, Service, and Architecture Catalogs
Today, I’m excited to reveal the new design of the Gruntwork Infrastructure as Code Library (IaC Library):
  • The Gruntwork Module CatalogBuild your infrastructure by mixing & matching hundreds of reusable, battle-tested modules.
  • The Gruntwork Service Catalog [NEW]Deploy off-the-shelf services, without writing any code. Each service combines multiple modules into a highly configurable package that’s designed to be deployed directly to production.
  • The Gruntwork Architecture Catalog [NEW]Deploy proven, end-to-end architectures that contain all the services you need to go to prod, already wired together and fully automated.
Read on to learn how you can use these three types of catalogs to go to production on AWS reliably and securely, in days rather than months.

Let’s say you just finished working on your new project — maybe it’s a Node.js app at a new startup or maybe it’s a set of Java microservices at a major enterprise — and after months of coding, it’s time to ship it to production. You sign up for an AWS account, log in, and you’re greeted with a page that looks something like this:
Phew. Well, I guess the good news is that you have access to a nearly endless list of tools and services. However, the bad news is that you have access to a nearly endless list of tools and services.You figure that you probably need to deploy some servers, so you start learning about EC2 instances, Auto Scaling Groups, Amazon Machine Images, and Elastic Load Balancers. But as you’re trying to make progress with that, all you hear about is Kubernetes, Kubernetes, Kubernetes. So then you start reading about Docker Orchestration, EKS, control planes, worker nodes, auto scalers, RBAC, Istio, and Helm charts. And you’ve barely made any progress there before your teammates start yelling about the fact that Serverless is the future! So now you’re knee deep in docs about AWS Lambda, API Gateway, ECS Fargate, Serverless Framework, and Aurora.But wait, first you have to get all the networking details in place. So you scrap everything you’ve done so far and start learning about VPCs, subnets, route tables, NAT Gateways, Network ACLs, and Transit Gateways. But before you can finish any of that, you discover something called Landing Zone. So then you’re digging through documentation about AWS Config, CloudTrail, IAM Roles, GuardDuty, Account Vending Machines… And after months of work, it dawns on you that you’ve still got to sort out a CI / CD pipeline, data storage, data backup, monitoring, log aggregation, alerting, secrets management, SSH access, VPN access, DNS, TLS, CDN, and so much more.And here’s the crazy thing: there are a thousand other developers at a thousand other companies doing the exact same thing. Every single company that wants to go to production on the public cloud needs to solve the same set of problems: networking, orchestration, data storage, secrets management, CI / CD, monitoring, and so on. So why are you doing it all from scratch?Perhaps you tried to use some of the open source solutions that are out there, but many of them don’t quite seem to do what you want, many don’t work at all, and virtually none of them have automated tests or docs. You start thinking to yourself, has anyone actually used this code before? In production? Is anyone maintaining these repos? Do I really want to bet my company on this code?The reality is that, under the hood, most companies want more or less the same basic infrastructure to run their web services. You can think of it like this:
Your app is unique. The infrastructure under the hood can be almost completely generic, just so long as it is reliable, secure, tested, and maintained.This is precisely what we do at Gruntwork. Over the last 5 years, we’ve built up a library of over 300,000 lines of reusable, battle-tested, commercially supported and maintained infrastructure code that is used in production by hundreds of companies.The big change we’re announcing today is that the Gruntwork IaC Library now consists of three layers: modules, services, and architectures. Let’s go through these layers one at a time, starting by looking at modules.

The first step in going to prod is to create modules, which are the basic building blocks of your infrastructure. Each module is a single piece of reusable infrastructure defined as code: e.g., a Terraform module to deploy a server, or a load balancer, or a database. You can think of them like the furniture in a house:
For the last few years, the core of the Gruntwork Infrastructure as Code Library has been the Module Catalog, which contains hundreds of reusable, battle-tested modules that you can use to assemble your infrastructure. Here are just a few examples of the modules you’ll find in the Module Catalog:
  • single-server: A module written in Terraform that you can use to deploy a single server (EC2 instance) and configure its permissions (IAM role), firewall settings (security group), IP settings (EIPs), and DNS records (Route 53).
  • asg-rolling-deploy: A module written in Terraform that you can use to deploy multiple EC2 instances in an Auto Scaling Group (ASG) that can scale up and down in response to load, automatically replace failed instances, and deploy updates via a zero-downtime, rolling deployment.
  • eks-cluster-control-plane: A module written in Terraform that deploys the control plane of an Elastic Kubernetes Service (EKS) cluster. This allows you to run your apps in AWS as Docker containers.
  • cloudwatch-log-aggregation-scripts: A module written in Bash that can configure EC2 instances to send their logs to CloudWatch.
  • ssh-grunt: A module written in Go that can be installed on EC2 instances to manage SSH access using IAM groups.
And here’s an example of how you might use all these modules together to configure an EKS cluster. First, you create a Packer template that will build an Amazon Machine Image (AMI) for EKS worker nodes with cloudwatch-log-aggregation-scripts and ssh-grunt installed:
{ "builders": [{ "ami_name": "eks-worker-node-{{isotime | clean_resource_name}}", "ami_description": "An EKS-optimized AMI for worker nodes.", "instance_type": "t3.micro", "region": "us-east-1", "type": "amazon-ebs", "source_ami": "ami-abcd1234", "ssh_username": "ec2-user", }], "provisioners": [{ "type": "shell", "inline": [ "gruntwork-install --module-name cloudwatch-log-aggregation-scripts --repo https://github.com/gruntwork-io/module-aws-monitoring --tag v0.22.1", "gruntwork-install --module-name ssh-grunt --repo https://github.com/gruntwork-io/module-security --tag v0.34.5" ] }] }
Next, you write Terraform code to deploy the Kubernetes control plane using the eks-cluster-control-plane module and deploy Kubernetes worker nodes using the asg-rolling-deploy module, passing in the ID of the AMI you built with the Packer template via the ami input variable:
module "control_plane" { source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-cluster-control-plane?ref=v0.22.0" cluster_name = "example-cluster" kubernetes_version = "1.17" enabled_cluster_log_types = ["api", "audit", "authenticator"] vpc_id = var.vpc_id vpc_master_subnet_ids = var.control_plane_subnet_ids # ... (other params omitted) ... }module "worker_nodes" { source = "git::git@github.com:gruntwork-io/module-asg.git//modules/asg-rolling-deploy?ref=v0.10.0" ami = "<YOUR AMI ID>" vpc_subnet_ids = var.worker_node_subnet_ids min_size = 2 max_size = 10 # ... (other params omitted) ... }
Run terraform apply and voila, you now have a fully working Kubernetes cluster and can start deploying Docker containers to it using kubectl apply or helm install!

As you just saw with the EKS cluster example, once you have modules, the next step is to combine multiple modules together to create services, which are designed to be deployed directly to production. A service includes everything you need in production, including the Terraform code for provisioning, Packer templates or Dockerfiles for configuration management, built-in monitoring (metrics, logging, alerting), built-in security features (SSH access, secrets management, server hardening, encryption in transit and at rest), and so on. You can think of services as the rooms in a house, which combine various sets of furniture (various sets of modules) for specific use cases:
Today, I’m excited to announce the Gruntwork Service Catalog, a major new addition to the IaC Library. It contains dozens of highly configurable, battle-tested, production-grade services that you can deploy off-the-shelf without writing any code. For example, instead of writing your own EKS cluster code as in the previous section, you can use the off-the-shelf eks-cluster service from the Gruntwork Service Catalog, which includes:
  • Terraform code to provision the cluster, using modules such as eks-cluster-control-plane to deploy the control plane, asg-rolling-deploy to deploy the worker nodes, and a number of other modules for configuring auto scaling, auto healing, namespaces, load balancing (ingress), permissions (IAM and RBAC), secrets management, etc.
  • A Packer template to build an AMI for the worker nodes, using modules such as cloudwatch-log-aggregation-scripts to configure log aggregation, ssh-grunt to manage SSH access, and a number of other modules for configuring sever hardening, bootstrapping, metrics, etc.
  • Scripts, documentation, automated tests, and much more.
Here’s an example of how you can use this eks-cluster service to deploy a production-grade EKS cluster with a single Terragrunt configuration file:
# Use off-the-shelf code from the Gruntwork Service Catalog terraform { source = "git::git@github.com:gruntwork-io/aws-service-catalog.git//modules/services/eks-cluster?ref=v0.0.5" }# Pull in outputs from your VPC dependency "vpc" { config_path = "../../networking/vpc" }# Configuration for the eks-cluster service inputs = { cluster_name = "example-eks-cluster" cluster_instance_type = "m4.large" cluster_instance_ami = "<YOUR AMI ID>" # Deploy a a separate ASG per subnet / availability zone. autoscaling_group_configurations = { for subnet_id in dependency.vpc.outputs.private_app_subnet_ids : subnet_id => { min_size = 3 max_size = 5 subnet_ids = [subnet_id] tags = [] } } # ... (other params omitted) ... }
Plug the code above into a file called terragrunt.hcl, run terragrunt apply, and you have a production-grade EKS cluster in minutes! Note: Terragrunt is not required; all the code above works with pure Terraform and Terraform Cloud / Terraform Enterprise too!The Gruntwork Service Catalog includes dozens of other off-the-shelf services you can deploy in minutes, such as services for deploying and configuring other orchestration tools (e.g., ECS, ASGs), networking (e.g., VPCs, VPNs), data storage (e.g., RDS, ElastiCache, Elasticsearch, S3), AWS account structure (Landing Zone), CI / CD pipelines (both for apps and infrastructure), and much more. We also have a Service Catalog where all the services are configured to be compliant with the CIS AWS Foundations Benchmarks out-of-the-box. In the future, we’ll be adding out-of-the-box support for other major standards, such as PCI, HIPAA, and NIST 800–53.

The final step is to combine multiple services together to create your production architecture. The architecture is your entire end-to-end tech stack, that includes everything you use in production, including networking, orchestration, data storage, secrets management, CI / CD, monitoring, and so on, all wired together as a single, working system. You can think of the architecture as an entire house, which combines various sets of rooms (various sets of services) for specific use cases:
Today, I’m also excited to announce the Gruntwork Architecture Catalog, another major new addition to the IaC Library. For the last few years, we’ve offered the Gruntwork Reference Architecture as a product you can buy that made direct use of the Gruntwork Module Catalog under the hood. We are now turning the Reference Architecture into an Architecture Catalog, a collection of battle-tested, off-the-shelf architectures for AWS that make use of the Gruntwork Service Catalog under the hood.For example, we have ECS-based, ASG-based, and EKS-based architectures in the Architecture Catalog. Here’s a glimpse of what the EKS-based architecture includes:
  • Landing Zone: multi-account setup, with a security baseline applied to each account that sets up IAM users, IAM roles, IAM groups, CloudTrail, AWS Config, GuardDuty, etc.
  • Networking: VPCs, multiple tiers of subnets (public, private app, private persistence), route tables, Internet Gateway, NAT Gateways, NACLs, etc.
  • Orchestration: Kubernetes running on EKS, fully-managed control plane, choice of worker nodes (self-managed, AWS-managed, or Fargate), auto scaling, auto healing, RBAC mapping, ALB ingress controller, etc.
  • Data storage: RDS, ElastiCache, Amazon Elasticsearch, S3, automated failover, automated backup, automated replication, etc.
  • Data privacy: encryption in transit using TLS (with ACM for public endpoints, self-signed certs for private endpoints), encryption at rest using KMS, secrets management using AWS Secrets Manager.
  • CI / CD: secure, automated, GitOps-based pipelines for apps and infrastructure code that integrates with your CI server of choice (e.g. Jenkins, CircleCi, etc.).
  • Monitoring: custom metrics and dashboards in CloudWatch, log aggregation in CloudWatch logs, alerts via SNS, etc.
  • Compliance: We can configure any of the architectures in the Architecture Catalog to be compliant with the CIS AWS Foundations Benchmark out-of-the-box. In the future, we’ll be adding out-of-the-box support for other major standards, such as PCI, HIPAA, and NIST 800–53.
  • And much more: SSH, VPN, DNS, CDN, sample apps, server hardening, high availability design, vertical and horizontal scalability, etc.
All of this is wired together for you, fully automated, and we can deploy it into your AWS accounts in about 1 day, giving you all the code to manage it.

As a Gruntwork customer, you get access to all our Module Catalog and our Service Catalog. You could, for example, use the hundreds of modules from the Module Catalog directly to assemble your own services:
Or you could use the dozens of services from the Service Catalog and start deploying infrastructure right away:
Or you can have us deploy an architecture for you from our internal Architecture Catalog (in the future, we’ll grant customers direct access to this, too) and start building on top of a battle-tested, end-to-end tech stack, in about 1 day:
Whichever choice you pick, at the end of the day, you’ll be able to focus far more on your app, and get to production in days, rather than months:
And since it’s all commercially supported and maintained, you’ll be able to keep your infrastructure working smoothly and securely over the long-term.Here’s how you get started:
  • The Module Catalog for AWS, with compliance for the CIS AWS Foundations Benchmarks, is available today for all subscribers. You can sign up on our website today!
  • The Service Catalog and Architecture Catalog for AWS, with compliance for the CIS AWS Foundations Benchmarks, are available today as part of a private, invite-only alpha. If you’re interested, contact us to learn how to get access!
  • Catalogs for other major compliance standards (PCI, HIPAA, NIST 800–53) and other clouds (GCP, Azure) are coming in the future. Contact us to learn more.
Share
Grunty
Resources

Explore our latest blog

Get the most up-to-date information and trends from our DevOps community.
TerraformResouces Image

Promotion Workflows with Terraform

How to configure GitOps-driven, immutable infrastructure workflows for Terraform using Gruntwork Patcher.
avatar

Jason Griffin

October 3, 2023 7 min read
TerraformResouces Image

The Impact of the HashiCorp License Change on Gruntwork Customers

How to configure GitOps-driven, immutable infrastructure workflows for Terraform using Gruntwork Patcher.
avatar

Josh Padnick

October 3, 2023 7 min read