01 September 2020
This is the first of a 5-part series on AWS exploits and similar findings discovered over the course of 2020. All findings discussed in this series have been disclosed to the AWS security team and had patches rolled out to all affected regions, where necessary. A big thanks to my friend and fellow Australian Aidan Steele for co-authoring this series with me. Check out parts 2 and 4 for his work!
This first part covers an issue which could lead to a denial-of-service on some CloudFormation actions within a single AWS account using the
Fn::Cidr CloudFormation intrinsic function.
Fn::Cidr (also written as
!Cidr) is an intrinsic function within CloudFormation that allows template authors to have an array of CIDR address block returned, usually used during programmatic creation of VPC structures where the base CIDR range is dynamic. The function handles both IPv4 and IPv6 range calculations.
After generating this for a VPC CIDR, developers would typically use
Fn::Select to choose an appropriate subset of the base CIDR range for the creation of VPC subnets.
During a time when I was debugging a VPC creation template, I used CloudFormation outputs to determine the expected values from an IPv6 range. I noticed that the length of the output was very long and was curious to see what the limits of this field was. I checked the published limits and didn’t find what I was looking for, so decided to find out directly. I slowly incremented the field until finally I ended up consuming the full 450kb available to me within the template, which looked like this:
AWSTemplateFormatVersion: "2010-09-09" Description: A very long CFN template Resources: VPC: Type: AWS::EC2::VPC Properties: CidrBlock: 10.0.0.0/24 VPCCidrBlockIpv6: Type: AWS::EC2::VPCCidrBlock Properties: VpcId: !Ref VPC AmazonProvidedIpv6CidrBlock: true Outputs: VeryLongOutput: Value: !Join ["##", [ !Join ["::", !Cidr [!Select [0, !GetAtt VPC.Ipv6CidrBlocks], 256, 64] ], !Join ["::", !Cidr [!Select [0, !GetAtt VPC.Ipv6CidrBlocks], 256, 64] ], ... !Join ["::", !Cidr [!Select [0, !GetAtt VPC.Ipv6CidrBlocks], 256, 64] ] ]]
In this template, I am creating a simple VPC and generating an output using
Fn::Join that is as long as possible. At the time, my rough calculations was that this would generate an output string roughly 30MB long.
As I watched the stack creation in the management console, I suddenly stopped receiving updates. I refreshed the page and saw that my stack refused to show any updates or the output I had constructed.
I moved over to the CLI and entered:
aws cloudformation describe-stacks --stack-name mystack
and waited. The CLI didn’t return any results and simply hanged. I tried the following command:
aws cloudformation describe-stacks
and had the same issue. The same thing happened when I used a different user. I realized I had found a way to deny the ability to describe stacks to other users so decided to raise this with the AWS security team.
As this was effectively a denial-of-service to some stack actions within the containing account, this issue was raised with the AWS security team via email. Additionally, as I am
a threat to friends with some leaders of the CloudFormation team, I was able to connect with them directly to work through the issue.
Once patched, the following error occurred when attempting to replicate the issue:
Essentially, they’ve placed a limit on the size any generated outputs and will rollback the stack if this limit is exceeded.
This issue wasn’t always present within CloudFormation. I believe this issue comes as a result of an assumption made (which was correct for a certain time) that a users raw template had limits on it that meant the size of the outputs data structures and/or API responses would not exceed a certain value. As new features came along, like the introduction of
Fn::Cidr, these assumptions no longer held true. It’s a reminder to us developers that new features sometimes have unintended consequences and we need to consider all of the previous assumptions made to see if they still hold.
Thanks to the AWS security and CloudFormation team members who worked with me to help remediate this issue.