05-dev-environment.md

Browse the Repo

_docs

_images

01-architecture-overview.md

02-whats-deployed.md

03-security-compliance-compatibility.md

04-how-code-is-organized.md

05-dev-environment.md

06-ci-cd.md

07-monitoring-alerting-logging.md

08-ssh-vpn.md

09-accounts-and-auth.md

10-gruntwork-tools.md

11-deploying-a-docker-service.md

12-migration.md

13-deploying-the-reference-architecture-fr...

14-undeploying-the-reference-architecture.md

15-adding-new-environments-regions-and-acc...

README.md

dev

master

prod

security

shared-services

stage

.gitignore

CODEOWNERS

README.md

Browse the Repo

_docs

_images

01-architecture-overview.md

02-whats-deployed.md

03-security-compliance-compatibility.md

04-how-code-is-organized.md

05-dev-environment.md

06-ci-cd.md

07-monitoring-alerting-logging.md

08-ssh-vpn.md

09-accounts-and-auth.md

10-gruntwork-tools.md

11-deploying-a-docker-service.md

12-migration.md

13-deploying-the-reference-architecture-fr...

14-undeploying-the-reference-architecture.md

15-adding-new-environments-regions-and-acc...

README.md

dev

master

prod

security

shared-services

stage

.gitignore

CODEOWNERS

README.md

Multi-account Reference Architecture

End-to-end tech stack designed to deploy into multiple AWS accounts. Includes VPCs, EKS, ALBs, CI / CD, monitoring, alerting, VPN, DNS, and more.

Running An App in the Dev Environment

Now that you have an idea of what the architecture looks like, what's deployed, security compliance, and how the code is organized, it's time to start running some of that code on your own computer!

Here's what we'll cover:

Prerequisites
Checkout the code
Run the app
Make a service call
Change the code
Update the application configuration
Encrypt a secret
Apply schema migrations
Commit your changes

Prerequisites

You will need to install the following software on your computer:

Git: Used for version control.
Docker: Used to package apps so they run the same way in all environments, including your dev environment.
Terraform: Used to provision and manage infrastructure as code.
Terragrunt: A thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules.
Packer: Used to package apps as Amazon Machine Images.
GruntKMS: Used to encrypt and decrypt secrets.

Checkout the code

You will need to checkout the code for your app(s) using Git. For this tutorial, we are going to checkout sample-app-frontend-multi-account-acme, which is a sample app that's handy for demonstrating all the key concepts in this tutorial.

To check this app out onto your computer, run:

git clone git@github.com:gruntwork-io/sample-app-frontend-multi-account-acme.git

Note that sample-app-frontend-multi-account-acme depends on another sample app called sample-app-backend-multi-account-acme, so you may want to checkout that app too:

git clone git@github.com:gruntwork-io/sample-app-backend-multi-account-acme.git

Run the app

sample-app-frontend-multi-account-acme is packaged as a Docker container so that it runs the same way in all environments, including on your computer. Moreover, we've used Docker Compose to configure all the Docker containers you need to run your entire tech stack (i.e., all apps, databases, etc) in the docker-compose.yml file, so you can fire everything up with a single command:

cd sample-app-frontend-multi-account-acme
docker-compose up

Your entire stack will boot up in a few seconds and you should be able to test it by going to http://localhost:3000/sample-app-frontend-multi-account-acme.

The app that's running is a simple Node.js app, but the ideas demonstrated work more or less the same way in any language. Take a look at app/index.js to see the code. You should also see other endpoints in that file that you can try.

Make a service call

One of the other endpoints you can try is http://localhost:3000/sample-app-frontend-multi-account-acme/service. This demonstrates how sample-app-frontend-multi-account-acme can make a service call to sample-app-backend-multi-account-acme. How does this work?

If you dig through the code in app/server.js, you'll see that sample-app-frontend-multi-account-acme is getting the IP address of sample-app-backend-multi-account-acme from an environment variable called BACKEND_URL and falling back to the value backend if that environment variable is not set. Here's how this works:

In dev, Docker Compose Networking sets up a single network where each service xxx is accessible at the hostname xxx. Since our service is called backend, it is accessible at http(s)://backend.
In other environments (e.g. stage, prod) we configure Terraform to set the BACKEND_URL environment variable for your Docker containers, except we point the variable to an internal Application Load Balancer (see Architecture overview), which is configured to route traffic to sample-app-backend-multi-account-acme.

We use a similar approach to allow the app to talk to all of its other dependencies too (e.g., the database, cache, etc), with slightly different service names and environment variables for each one.

Change the code

Make a change to the app code, such as modifying app/index.js or app/index.html. Refresh the page in your browser and you should see your changes immediately! Here's how that works:

In docker-compose.yml, we configure Docker to mount the code from your host OS into the Docker container (look for the volumes settings in docker-compose.yml). That way, every time you make a change on your host OS, it's reflected immediately in the Docker container.
The Docker container uses nodemon in dev mode, so the Node app restarts automatically with every change.

This setup allows you to iterate rapidly!

Update the application configuration

sample-app-frontend-multi-account-acme stores most of its configuration settings in files. The advantage of this is that the files are checked into version control (so you have a history of config changes) and are versioned and deployed with the app code itself (so you don't get a mismatch between app and config).

The files live in the config directory.
The names use the format example-config-<environment>.json, where environment is an environment name, such as development, stage, or prod. Note that the same basic config approach shown here works with any file format and naming convention; we only use JSON because it's easy to use with the sample Node.js app.
In app/server.js, the app loads the proper config file for the current environment based on the environment variable VPC_NAME.
In dev, the VPC_NAME environment variable is configured in docker-compose.yml.
In other environments (e.g., stage, prod) the VPC_NAME environment variable is set by Terraform.

Encrypt a secret

Occasionally, you may need to encrypt some secret data. For example, you may want to store the password to your database in your app's configuration. Storing secrets in plain text anywhere, including version control, is a major security risk and should be avoided (see Gruntwork Security Best Practices for more info). A better way to handle this is to encrypt the secrets before putting them in version control and to have your app decrypt those secrets before it boots.

Managing encryption keys securely is very tricky, so we strongly recommend using Amazon's Key Management Service (KMS):

Create "master key" in KMS using the kms-master-key module. Note that we've already created one master key in each environment, so for most use cases, you should use those and not create any new ones. You can see all available master keys in the Encryption Keys page in IAM.
Use the master key, along with gruntkms, to encrypt and decrypt secrets with a single CLI command.

For example, you could run gruntkms on your computer to encrypt a database password:

# A developer encrypts a plaintext secret
$ echo "super secret database password" | gruntkms encrypt --key-id "alias/cmk-stage" --aws-region "us-east-1"
kmscrypt::AQICAHhQYFj4xrlpRdnui/MrOlrIt+gSSrFxZay4ZMDMofceSwEXSzGkmBBWbG6==

You can now safely check the kmscrypt::... ciphertext into version control.

You can then give your apps access to the same KMS key via IAM permissions and configure your apps to decrypt the ciphertext using gruntkms just before booting:

# An app decrypts the ciphertext before booting
echo "kmscrypt::AQICAHhQYFj4xrlpRdnui/MrOlrIt+gSSrFxZay4ZMDMofceSwEXSzGkmBBWbG6==" | gruntkms decrypt --aws-region "us-east-1"
"super secret database password"

Make sure to read through the gruntkms docs to learn how how the kmscrypt:: prefix works and how to decrypt all secrets in a config file in a single command (see bin/run-app.sh in the sample apps for an example).

Apply schema migrations

sample-app-backend-multi-account-acme contains example code of how to talk to a relational database. It also contains an example of how to apply schema migrations to your database before the app boots. The idea is to version and package the schema migration code with the app code so that whenever you deploy a new version of the app, it always ensures the schema it depends on is in place before booting.

Under the hood, the example code manages the schema migrations using simple .sql files (see the sql folder) and uses Flyway to apply those migrations (see bin/run-app.sh). The same basic approach should work with any other schema migration tool (e.g., Luiqibase, ActiveRecord) as long as that tool obtains a lock before applying schema changes (to ensure you only apply the schema changes once even if multiple copies of the app boot up at the same time).

Commit your changes

If you've made any changes to the code during this tutorial, it's time to commit them back to Git!

git add <files_you_modified>
git commit -m "<commit message>"
git push origin master

Next steps

Now that you've committed some code, it's time to learn how the automated Build, test, and deployment (CI/CD) process works.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?

Buy Now

"https://cdn.gruntwork.io/gruntwork-website/"

{"index":{"js":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.c7884255553b53fbca3a.js","map":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.1b14c1b7d19f1f5eb35d6e118e838255.map"},"styles":{"css":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.f22938926651ddec7c49.css","js":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.e782420e74a20dcb8691.js","map":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.d5e2af49807c6ca33f8367d621ece507.map"},"vendors":{"css":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.29f7d0366a0978763f96.css","js":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.fa8174a130797d75d12c.js","map":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.57243d94deeeb29d5061288a338b4eb6.map"}}

Multi-account Reference Architecture

Code Preview

Preview the Code

Running An App in the Dev Environment

Prerequisites

Checkout the code

Run the app

Make a service call

Change the code

Update the application configuration

Encrypt a secret

Apply schema migrations

Commit your changes

Next steps

Questions? Ask away.

Ready to hand off the Gruntwork?