Following an introduction to AWS Landing Zone, I’ll dive deeper into Control Tower as an orchestration tool in this post.
More on Landing Zone
In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility.
Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:
- Multi-account structure
- Identity and Access Management
- Governance (controls and guardrails)
- Networking
- Additional Security Services
The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization’s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely.
This post is about Control Tower.
Intro to Control Tower
Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account.
The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:
The detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:
- Mandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced.
- Optional controls including strongly recommended controls and Elective controls.
There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as “Operational Best Practices for “NIST 800 53 rev4” and “Security Best Practices for EKS”. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment.
The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.
Mandatory Accounts
The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that’s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I’ll elaborate in the next section.
The management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself.
The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account.
Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.
Each enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.
Account Enrolment
Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.
Going forward, the client should create new accounts using Control Tower’s Account Factory to save a separate enrolment step. For existing accounts and OUs we’ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.
To enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.
Account Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.
Delegated Administrator
Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator.
There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.
Whether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup.
If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:
- From the old administrator account, remove all the members
- From the management account, revoke delegation
- From the management account, delegate new administrator account (audit account)
- From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward
Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.
Control Tower Customization
Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That’s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:
| AFC Account Factory Customization | AFT Account Factory Terraform | CfCT Customizations for Control Tower | |
|---|---|---|---|
| Supported IaC language | Anything that Service Catalog Support (e.g. CloudFormation and Terraform HCL) | Terraform HCL | CloudFormation |
| Readiness | AFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console. | Users need to first bootstrap the solution, which creates Terraform pipelines and other components. | Users need to first bootstrap the solution which creates a pipeline and other components. |
| How it works | Control Tower launches the specified Service Catalog product (aka blueprint) during account creation. | Quite complex. Explained in the section below | Control Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions. |
| Prebuilt customizations | No. However, for certain products, there are some partner-maintained blueprints | Yes | No |
AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.
Customizations for Control Tower
In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select “AWS CodeCommit” as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.
The solution is based on Control Tower’s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket.
Account Factory Terraform
Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.
Once installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):
- Account requests – handles placing or updating account requests. See example here.
- AFT account provisioning customizations – manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available here
- Global customizations – Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT. Examples available
- Account customizations – Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT. Examples available
With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:
When I first looked into how this work I’m very concerned about the maintainability of this “solution”. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.
Summary
Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:
- you have general regulatory requirement and need to enable governance capabilities
- your networking stack isn’t centrally managed or isn’t large enough to warrant infrastructure automation
- If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform)
Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.
On the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.