AWS re:Invent 2016, The Short Version

Amazon’s annual conference on AWS, re:Invent, just ended last Friday, and the sheer number of new services and features they announced is a little overwhelming. I think the total tally comes to over 40, and that’s just too much to keep in your head.

So for myself and my colleagues who want to know “what did they announce and should we change anything?”, here’s my as-briefly-as-possible summary of the new hotness and why they matter. It’s roughly ordered starting with the most personally exciting to me.

Data Analytics

**Amazon Athena: Query Data Stored in S3 Using SQL **This was one of the most exciting new service announcements. Say you have data sitting in S3, such as JSON, CSV, a log file with custom delimiters, or common columnar storage formats used in Hadoop (e.g. Apache ORC). Athena lets you turn an S3 object into a “table” and query it using standard SQL! You can use the AWS Console or even an off-the-shelf SQL client to run your query. You can visualize your data using Amazon QuickSight.

Data Storage

AWS Aurora is Now Postgres-CompatibleAWS Aurora now has a preview mode of Postgres compatibility and implements all Postgres 9.6.1 features with Stored Procedure support coming soon. Because it’s “cloud-native” it can use S3 for backup automatically, and it’s twice as performant as non-Aurora Postgres (in terms of transactions per second it can handle).

**AWS SnowMobile & AWS SnowBall Updates** Use a physical device to efficiently migrate data from your data center to AWS’s. Or if you’re really overloaded with data (at the exabyte scale), load it into a big rig truck that physically drives to the AWS data center. Crazy.

AI-as-a-Service

**Amazon Lex: Build Your Own ChatBot or VoiceBot **Anyone who’s familiar with the Amazon Echo has come to love the easy voice interactions with Alexa. Well, Amazon’s taken the underlying technology that converts voice to text, extracts semantic concepts from a statement, and infers intent, and is offering it all as an AWS Service. Practically, this is perfect for building a ChatBot for VoiceBot, and you can plug in Lambda functions to take action at any point!

Amazon Rekognition: Object and Scene Detection in ImagesGive Rekognition an image and it returns probability estimates of objects and scenery that are likely depicted. It’s really slick and really impressive. You can even detect faces, gender, sentiment, and other human traits.

Amazon Polly: Convert Text to SpeechMore Amazon Echo tech. Convert text into spoken speech which you can download as an MP3 file.

Convenience

Amazon LightSail: Point-and-Click Setup of an app server on AWSLaunch an app server such as WordPress or Joomla in just a few clicks and without the usual overhead of Security Groups, EBS Volumes, Elastic IPs or other concepts potentially confusing for AWS beginners. Your app server will cost just a few dollars per month, and you can use one click to install a popular open source application. This is basically AWS copying CloudWith.me and to some extent Digital Ocean.

Interestingly, these instances launch in a “shadow VPC” and you won’t see them in the AWS Web Console. If your LightSail instances need access to your other VPC resources, the Shadow VPC can be VPC Peered to your existing VPC.

This service isn’t for power users, but it highlights an emerging trend in the cloud: convenience.

DevOps

AWS X-Ray: Request TracingAWS X-Ray is a request tracing service, which allows you to track a request from the moment it enters your stack, all the way through multiple microservices, and to the database (think of it as a managed version of OpenTracing.io). Built into the X-Ray SDK are tracing library calls you’ll need to add to your code. AWS automatically adds request tracing ID headers for you at the ELB level and provides a way to visualize the flow of that request through your entire infrastructure.

**AWS CodeBuild: Continuous Integration on AWS **This is basically CircleCI, TravisCI or any other hosted build service, but operated by AWS. I’m in love with the concept and love the idea that I can pay to scale as much as I want, but it’s a young product. As this Hacker News thread helpfully points out, there’s no caching between builds, no obvious GitHub integration (e.g. triggering a build from a GitHub Pull Request), notifications are through SNS, and there’s no obvious way to SSH to your build environment. So stay tuned on this one, as it doesn’t appear ready for prime time just yet.

AWS Personal Health DashboardA dashboard that shows you alerts and issues across your entire AWS account, and what steps you should take to remediate them. An alert from this dashboard can emit a CloudWatch Event which can be handled by a Lambda, or send an email or, trigger any other type of automation. That means we’re not far off from having a library of common remediations as Lambda’s that you can just plug in to create true self-healing infrastructure.

EC2 Container Service (ECS)

Blox: An Open Source Set of Tools for ECSThe primary alternative to ECS is usually Kubernetes. ECS is easier to set up, but Kubernetes has more power and flexibility behind it (except in the case of security, where IAM Roles for ECS Tasks give you fine-grained permissions per container). With Blox, Amazon is trying to replicate the concept of Kubernetes Addon’s. They’re open source in the hopes that other community members also contribute more addon’s, er “blox.”

Blox comes with just two tools (“blocks”?) right now. One allows you to query an API to get the state of your cluster, and the other allows you to run a daemon on every node in your cluster (like a log aggregation service). Kubernetes already has this as a built-in concept called Daemon Sets.

This is an exciting update, but the real question will be if Blox can reach the breadth and community involvement level of Kubernetes addon’s.

Lambda

AWS GreenGrass: For IoT Devices in Remote Parts of the WorldInstead of using one paradigm for your server-side code (Lambda) and one for your devices (AWS IoT), GreenGrass is a new IoT Device SDK that gives you a Lambda environment for the IoT device, so you have the same programming paradigm everywhere. AWS has built-in periodic cloud syncing, which does a good job of handling intermittent Internet connectivity, and you can manage which Lambdas get deployed to the device from an AWS API.

Lambda@EdgeEmbed logic in your CloudFront distribution using Lambdas. This is useful if you want to take some simple steps to avoid going back to the origin server, and it’s a cool idea!

Security

**AWS Shield: DDoS protection for AWS **AWS Shield is a set of automatic network mechanisms that protect you from DDoS attacks that target Layer 3 or Layer 4 of the OSI Network Model. If you want Layer 7 protection (so-called “application layer attacks”), then you’ll have to be a paying support customer and upgrade to their AWS Shield Advanced. This is a simplified take on CloudFlare, and it’s really nice for those customers that just need the basic DDoS protection for free.

Data Processing

AWS Batch: Run Batch Jobs in a Managed WayIf you want to run a batch job across large numbers of instances, possibly using Spot Instances and a variety of instance types, in the past you had to manage this framework yourself. Now AWS manages it for you. You specify a shell script, Linux executable, or container image, give some parameters on what type of instances and on how many it should run, and AWS takes care of the rest.

AWS Step Functions: Build Complex Combinations of Lambda’sAWS noticed that users don’t just run one Lambda when an event happens. Sometimes they run a chain of Lambdas with complex flow logic, including possibly parallel processing. Step Functions gives you a way to visually design a “finite state machine” and will track the execution of a series of Lambda functions.

EC2 Updates

**Lots of New EC2 Instance Types **Amazon added higher-end T2 instances for people who want CPU bursting but still lots of CPU, and announced new generations of many new instance-type families.

**The New F1 Instance Type with Programmable Hardware **Some teams have aggressive CPU needs and could benefit from custom hardware. If those teams are willing to program an FPGA (in contrast to relying on a purpose-built CPU), they can essentially deploy their own custom hardware and get, according to AWS, up to a 30x boost in performance.

Attach GPUs Like You Attach EBS VolumesSometimes you want 1 or 2 GPUs to do some fancy programming. In the past, to get access to these GPUs in the cloud, you needed an expensive instance type. Now you can just attach GPUs like EBS Volumes.

IPv6 Support for EC2 and VPCThis one was little discussed but still significant. You can now opt in your VPC to IPv6. This actually simplifies your network structure by removing the need for NAT Gateways and introducing the concept of an “egress-only Internet Gateway.”

Mobile

AWS PinPoint: Manage Mobile Push Notification CampaignsTitle says it all.

Summary

So there it is! Whew, just writing this out was exhausting; it’s hard to fathom humans actually implementing all of this. Thanks for reading.

Get your DevOps superpowers at Gruntwork.io.