26 June 2019
This week Control Tower was released publicly, a managed service that replicates and iterates on the AWS Landing Zone solution previously released. It has an in-built capability of an “Account Vending Machine” which is really useful for quickly provisioning accounts to a secured state within an organization, though due to a lack of ability to programmatically delete accounts its effectiveness for the creation of ephemeral / sandbox accounts is somewhat limited.
AWS teased the idea of ephemeral accounts with an amazing talk at re:Invent 2018 and pairing blog post, though I’m yet to see any discussion on the full lifecycle of these ephemeral accounts.
An AWS account created with AWS Organizations has a surprisingly complex sequence of steps to actually be deleted, which is as follows:
Most of the above steps are (likely deliberately) not available via an API action. This makes automating the process of removing ephemeral sandbox accounts challenging, but I wanted to see if it was possible at all.
I wanted to create a solution, deployable via CloudFormation that makes as much use of AWS services as possible to assist with automating the above. Here’s how the solution accomplishes each of the above requirements:
First step is an easy one. The ListAccounts API call gives us the root account e-mail so we can call this programmatically.
This step is first where I have had to break out Puppeteer to automate the tasks. Puppeteer is an browser automation tool which can be programmatically run, such as in my case using a Lambda function. The browser is mocked and will visit the AWS forgot password page and request a reset.
Unfortunately this step will always require a CAPTCHA solve to combat against exactly what I’m trying to do. There are services available that will solve these for you in a reasonable amount of time. My favourite is 2captcha which will solve a simple CAPTCHA image at a rate of $0.50-$1.00 per thousand solves which is very reasonable.
So I have Puppeteer download the CAPTCHA image presented and upload that to the 2captcha servers for a result. I’ll typically receive a response in 10-20 seconds, which I then use to continue with the password reset.
An e-mail is sent to the account owner with a reset link which I need to visit in order to set the root account password. When setting up the accounts, I use a dedicated domain / subdomain for all root account which I set up in Route 53. I set the MX (mail) record for this domain / subdomain to point to the SES inbound address and use an SES Receipt Rule to process the mail, which is stored in S3 and triggers a Lambda to process the contents of this e-mail.
The Lambda will parse the e-mail body and look for links that looks like the reset link format. A lot of the time these e-mails are just some of the promotional / “helpful” e-mails they sent to the account holder, so many of these e-mails are simply skipped over.
Once a correctly formatted link is found, we again use Puppeteer to automate a browser to set a root user password and then login. In the automation, I use AWS Secrets Manager to generate a random root user password for this process.
I’ve never had to do this as I believe it’s inherited from the organization. Check this is set up in your master before attempting the automation.
I’ve also never had to perform this. It seems to be automatically performed when creating the account in AWS Organizations.
It is unfortunate that we can’t tokenize a payment method and use it for child accounts. We again have to use Puppeteer to enter in the payment details which I retrieve from an AWS Secrets Manager secret I suggest you make before deploying the automation. I’d recommend using a reloadable debit card for this to reduce the risk.
It’s not well documented, but one of the most common missed steps when making an account independant is the phone verification. To do this, you must visit this special link which throws you half-way into the standard onboarding flow you would go through with a standalone account. The step requests a phone number capable of receiving calls which only when entered will give you a 4 digit code for you to enter via your phone keypad which verifies you.
Amazon Connect is targetted as a virtual call center solution, but I wanted to determine whether this could be used to automate this phone verification. I had previously created the CloudFormation Provider for Amazon Connect solution which again uses Puppeteer to automate the creation and management of Connect instances as there are no APIs available for this yet.
One thing that Amazon Connect cannot do is execute DTMF tones (that’s the beeping tones you hear when using your keypad) whilst in an IVR flow. What I discovered however is that you could play arbitrary sound prompts by having it invoke a Lambda and determining which predefined prompts to play. The DTMF tones are indeed simply sounds that are interpreted over the line, so this solution is actually viable. The DTMF tone .wav files are publicly available and can be used as prompts in Amazon Connect if uploaded prior.
I use Puppeteer to first create the Amazon Connect instance, register a contact flow, register a phone number and use that phone number in the verification step. The contact flow has a 10 second wait at the start on order to both give the Amazon verification call time to say some generic instructions and also to have Puppeteer read the digits presented on screen and update the Lambda environment to ensure the correct prompt IDs that correlate to the digits are given when the wait time is complete.
As an aside, you don’t have visibility into the call as audio before an agent picks up is not recorded by the Amazon Connect call recording feature. Luckily, when testing this it worked the very first time.
At this point, the hard parts are done and all the prerequisites for removing the account should be met (unless the account was created in the last week). The call to remove the account from the organization is one that has an API call, RemoveAccountFromOrganization, so this is easily automated.
One last time, we need to use the session that is logged in via the root user account and delete the account with Puppeteer. This requires 4 checkboxes and 2 confirmations. Once this is done, the process is complete and the account is fully removed.
The caveats with this type of automation are important to note:
I’m still working on releasing an easy-to-use template to consume all of the automation above but when I release it, it will be available here. If you are interested in testing or consuming these templates, please reach out.
It’s unfortunate that AWS hasn’t released functionality to programmatically remove the accounts it allows us to create, so it is my hope that we see some change in this area soon. With some creativity, a lot of things can be automated however automating account deletions in its current state is right at the edge of what is practical.
One Cloud Please