If you want to deploy Cloudformation Stacksets with CDK, you’ve come to the right place.
Code repo here: https://github.com/mbennettcanada/stacksets-cdk
Assumptions and Warning
I’ll cut start out directly to the assumptions I’m making while writing this:
- You are using aws organizations to handle multiple aws accounts
- You have delegated stackset operations to a non-management account.
- You are using typescript CDK and want to use that to deploy and manage stacksets for your organization
- You use github and want to use that as your source for the cdk
If the above are true I’ll issue one warning about the pattern to be presented here: It solves a couple problems with stacksets but does not solve the most glaring: That you cannot ‘diff’ or ‘plan’ a stackset to see what will happen in the stackset instances. This makes for all-or-nothing deploys that can be painful if done poorly (what if the cloudformation wants to replace your load balancer because of a property you did not anticipate?).
For this reason I would only suggest using stacksets for deploying simple, stateless infrastructure that will be applied widely.
I see the largest benefit from using this for things like IAM policies to reference in you permission sets in IAM identity center, or creating ssm parameters in accounts for use by application stacks, etc. Pain will come swiftly if you use this for dynamic, or stateful things. Just don’t do it. In this example I’ll be deploying the github OIDC identity provider to accounts in an OU. This is something that will be set ONCE for all accounts in an ou, and is a pain to bootstrap manually across a whole org. It is an excellent fit for stackset automation.
I’ll break this post up into x main sections:
- AWS Organization Setup
- Environment Setup (CDK Bootstrapping)
- CDK Application and StacksetWaiter custom resource
- CDK Pipeline Setup and Deploy
AWS Organization Setup
For reference, the org account structure I’m using looks like this:
The “Operations account is my stackset delegated admin, and I’ll be deploying my stackset to all the accounts in the ‘Sandbox’ OU.
Next in your operations account, go to codebuild console, and go to settings along the left and click ‘connections’. Add a github connection to allow codebuild to connect to your github account. Write down the arn of the connection for later. If you need help here there is decent documentation HERE.
CDK Environment Setup
First, you’ll need to set up the CDK bootstrap with a custom assets bucket that will be shared across your organization. The reason the bucket and key must be shared with the org is because this method utilizes that bucket to deploy templates for cdk stacksets to be deployed org-wide. The easiest way to do this is to get the cloudformation template from the cdk bootstrap command and customize it a bit. We’re going to use a qualifier for this environment called ‘orgshared’ to keep this stack bootstrapping separate from the rest of our stacks. This ensures only things that are meant to be shared org wide are.
npx cdk bootstrap --qualifier orgshared --profile <CLI_PROFILE_HERE> --show-template > cdk-bootstrap.yaml
Additions to cdk-bootstrap.yaml
1. To that cdk-template.yaml, find the parameters section and add the following parameter:
OrgID:
Description: >-
Your AWS organization ID. Used to share your kms key and s3 bucket for deploying stacksets across your organization
Type: String
2. Find the FileAssetBucketEncryptionKey resource (at time of writing, first resource in the list) and add a statement to the key policy:
- Action:
- kms:Decrypt
- kms:DescribeKey
Effect: Allow
Principal:
AWS: "*"
Resource: "*"
Condition:
StringEquals:
kms:ViaService:
- Fn::Sub: s3.${AWS::Region}.amazonaws.com
ForAnyValue:StringLike:
aws:PrincipalOrgID:
- !Ref OrgID
3. Find the StagingBucketPolicy resource and add the following two statements to that policy:
- Sid: 'AllowOrgGet'
Effect: Allow
Principal: '*'
Action:
- s3:Get*
Resource: !Sub '${StagingBucket.Arn}/*'
Condition:
ForAnyValue:StringLike:
aws:PrincipalOrgID:
- !Ref OrgID
- Sid: 'AllowOrgList'
Effect: Allow
Principal: '*'
Action: s3:ListBucket
Resource: !Sub '${StagingBucket.Arn}'
Condition:
ForAnyValue:StringLike:
aws:PrincipalOrgID:
- !Ref OrgID
Deploy the cdk-bootstrap.yaml
You can do this via console or cli in your org’s stackset admin account, the only real trick is to add the qualifier parameter to differentiate this bootstrap environment from others. You’ll want to do this mostly for separation of concerns reasons. Every stack you deploy to this qualified environment will have it’s cloudformation template deployed to the asset bucket, and all accounts in the organization will be able to access it. For this reason its a REALLY good idea to not deploy secrets in plaintext via cdk, now more than ever.
CLI to deploy that boostrap with ‘stacksets’ qualifier (be sure to replace the org id and cli profile name):
aws cloudformation create-stack \
--stack-name CDKToolkit-orgshared \
--parameters ParameterKey=Qualifier,ParameterValue=orgshared ParameterKey=OrgID,ParameterValue=<YOUR_ORG_ID_HERE> \
--template-body file://cdk-bootstrap.yaml \
--profile <CLI_PROFILE_NAME>
--capabilities CAPABILITY_NAMED_IAMCDK Application and StacksetWaiter custom resource
We’ll start off from a blank cdk typescript app, but I’ll only show the component and use of it here, so if you don’t have one, find an empty dir you like the name of and run the following:
cdk init app --language typescriptFirst things first, set your qualifier in cdk.json by adding the following line to the “context” object:
"@aws-cdk/core:bootstrapQualifier": "orgshared",You’ll now want to add some directories to your lib/ directory to keep ’em separated. Create ‘stackset-templates’, ‘constructs’ and ‘lambdas’ directories like this:
lib/
constructs/
lambdas/
stackset-templates/
lib/stackset-templates/github-oidc-provider.ts
In the ‘stackset-templates’ dir create a file named ‘github-oidc-provider.ts’ and put the following contents in it:
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as serviceCatalog from "aws-cdk-lib/aws-servicecatalog";
import path = require("path");
import { IBucket } from "aws-cdk-lib/aws-s3/lib/bucket";
interface GithubOIDCProviderStacksetTemplateProps extends serviceCatalog.ProductStackProps {
assetBucket: IBucket; //this would be used for deploying any assets for lambdas to be deployed by this stackset. CDK would deploy to that bucket via the delegated admin account. It should be available to any account that the stackset is deployed to.
}
export class GithubOIDCProviderStacksetTemplate extends serviceCatalog.ProductStack {
constructor(scope: Construct, id: string, props: GithubOIDCProviderStacksetTemplateProps) {
super(scope, id, props);
const githubDomain = 'token.actions.githubusercontent.com';
const ghProvider = new cdk.aws_iam.OpenIdConnectProvider(this, 'githubProvider', {
url: `https://${githubDomain}`,
clientIds: ['sts.amazonaws.com'],
});
}
}
You’ll notice we are using the serviceCatalog.ProductStack object to wrap our CDK in. This construct will allow us to output the cloudformation of any cdk constructs we wrap it with, and deploy it to the cdk assets bucket. We can then go on to pass that specific bucket (with region) into the stackset construct to pull the cfn from. Since we specify the region in the bucket name this can be deployed across regions, and since the bucket is shared with our whole org, all accounts can read the cloudformation templates there and deploy them. Since these assets are shared widely, be sure to keep secrets out of the templates you deploy.
If you need to pass in arguments to your cloudformation template be sure you are passing in literals and not references.
lib/constructs/github-oidc-stackset.ts
Next we’ll create a construct to wrap the above template in and create the stackset. In the /lib/constructs directory create a github-oidc-stackset.ts file and put the following:
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from "aws-cdk-lib/aws-s3";
import * as serviceCatalog from 'aws-cdk-lib/aws-servicecatalog';
import { GithubOIDCProviderStacksetTemplate } from '../stackset-templates/github-oidc-provider';
import path = require('path');
import { StacksetInstancesWaiter } from './stackset-instances-waiter';
export interface GithubOIDCProviderStacksetProps {
stacksetName: string;
stacksetRegions: string[];
targetAccounts: string[];
targetOrgUnits: string[];
assetBucketName: string;
}
export class GithubOIDCProviderStackset extends Construct {
constructor(scope: Construct, id: string, props: GithubOIDCProviderStacksetProps) {
super(scope, id);
if (props.targetOrgUnits.length == 0) {
throw new Error(`targetOrgUnits must be specified to bootstrap cdk for ${props.stacksetName}`);
}
const stackTemplate = new GithubOIDCProviderStacksetTemplate(this, `${props.stacksetName}-template`, {
assetBucket: s3.Bucket.fromBucketName(this, "stagingBucket", props.assetBucketName),
});
if (props.targetAccounts.length > 0) { // if accounts are provided, don't deploy to the whole ou.
const stackset = new cdk.CfnStackSet(this, props.stacksetName, {
permissionModel: "SERVICE_MANAGED", //SERVICE_MANAGED is the only type of deployment supported by delegated admin stackset accounts
stackSetName: props.stacksetName,
autoDeployment: {
enabled: true,
retainStacksOnAccountRemoval: false,
},
callAs: "DELEGATED_ADMIN", //required for delegated admin account
description:
`${props.stacksetName} StackSet managed with CDK`,
capabilities: [
'CAPABILITY_IAM',
'CAPABILITY_NAMED_IAM'],
templateUrl: serviceCatalog.CloudFormationTemplate.fromProductStack(stackTemplate).bind(this).httpUrl,
stackInstancesGroup: [
{
regions: props.stacksetRegions,
deploymentTargets: {
organizationalUnitIds: props.targetOrgUnits,
accounts: props.targetAccounts,
accountFilterType: "INTERSECTION" //This limits to the specific accounts in the 'accounts' property.
},
}],
parameters: [],
operationPreferences: {
failureToleranceCount: 1,
maxConcurrentCount: 30,
}
});
const stacksetWaiter = new StacksetInstancesWaiter(this, `${props.stacksetName}-waiter`, { stackSetName: stackset.stackSetName })
stacksetWaiter.node.addDependency(stackset);
}
else {
const stackset = new cdk.CfnStackSet(this, props.stacksetName, {
permissionModel: "SERVICE_MANAGED", //SERVICE_MANAGED is the only type of deployment supported by delegated admin stackset accounts
stackSetName: props.stacksetName,
autoDeployment: {
enabled: true,
retainStacksOnAccountRemoval: false,
},
callAs: "DELEGATED_ADMIN", //required for delegated admin account
description:
`${props.stacksetName} StackSet managed with CDK`,
capabilities: [
'CAPABILITY_IAM',
'CAPABILITY_NAMED_IAM'],
templateUrl: serviceCatalog.CloudFormationTemplate.fromProductStack(stackTemplate).bind(this).httpUrl,
stackInstancesGroup: [
{
regions: props.stacksetRegions,
deploymentTargets: {
organizationalUnitIds: props.targetOrgUnits,
accountFilterType: "NONE"
},
}],
parameters: [],
operationPreferences: {
failureToleranceCount: 1,
maxConcurrentCount: 30,
}
});
const stacksetWaiter = new StacksetInstancesWaiter(this, `${props.stacksetName}-waiter`, { stackSetName: stackset.stackSetName })
stacksetWaiter.node.addDependency(stackset);
}
}
}Several notes on this block:
- This construct is designed so you pass in either org units and possibly account ID’s. If you pass in account ID’s it will only deploy to those accounts within the org unit you pass in. You can change this around by switching up the
deploymentTargetsaccountFilterTypeproperty. For more info on that, check out documentation HERE operationPreferencesshows a failure tolerance of 1, meaning if one fails, the stackset is considered failed.- stacksetWaiter is a state machine (yet to be defined) that will take the stackset name as a parameter and continually check all stackset instances for successful completion. If any stacksets fail it will return the error and fail the cdk deployment.
lib/constructs/stackset-instances-waiter
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as cr from 'aws-cdk-lib/custom-resources';
import path = require('path');
export interface StacksetInstancesWaiterProps {
// The name of the stackset to inspect for instances and wait for completion
readonly stackSetName: string;
}
export class StacksetInstancesWaiter extends Construct {
constructor(scope: Construct, id: string, props: StacksetInstancesWaiterProps) {
super(scope, id);
//Stackset instance waiter custom resource
const waiterRole = new iam.Role(this, 'WaiterRole', {
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')],
inlinePolicies: {
'WaiterPolicy': new iam.PolicyDocument({
statements: [
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: [
'cloudformation:ListStackInstances',
'organizations:ListDelegatedAdministrators'
],
resources: ["*"]
})
]
})
}
})
const handlerCommonConfig = {
runtime: lambda.Runtime.PYTHON_3_11,
code: lambda.Code.fromAsset(path.join(__dirname, '../lambdas/stackset-waiter')),
role: waiterRole,
timeout: cdk.Duration.minutes(15),
}
const onEventHandler = new lambda.Function(this, 'OnEvent', {
...handlerCommonConfig,
handler: 'index.on_event',
})
const isCompleteHandler = new lambda.Function(this, 'IsComplete', {
...handlerCommonConfig,
handler: 'index.is_complete',
})
const provider = new cr.Provider(this, 'Provider', {
onEventHandler,
isCompleteHandler,
queryInterval: cdk.Duration.minutes(15),
totalTimeout: cdk.Duration.hours(1)
})
const StacksetWaiter = new cdk.CustomResource(this, 'StacksetWaiter', {
serviceToken: provider.serviceToken,
properties: {
stackSetName: props.stackSetName,
datetime: new Date().toISOString()
}
})
}
}You’ll notice we’re using the aws-cdk-lib/custom-resources construct as it abstracts out many of the boring parts of the state machine and event handling.
We create a custom role for the lambda to use to scope it’s permissions down to the minimal set, you shouldn’t need to adjust this. We then set up the new event handlers to pass to the CustomResource provider, and set the timeout to one hour. If your stackset instances take more than an hour to deploy, this is the place to adjust the timeout for the waiter to allow for that
The actual custom resource is created last and has the stackset name passed in along with the provider service token.
lib/lambdas/stackset-waiter/index.py
Now to the actual waiter lambda. Lets look at the code there:
import time
import datetime
import boto3
import botocore.exceptions
def on_event(event, context):
print(event)
request_type = event['RequestType']
if request_type == 'Create': return on_create(event)
if request_type == 'Update': return on_update(event)
if request_type == 'Delete': return on_delete(event)
if request_type == 'IsComplete': return is_complete(event, context)
raise Exception("Invalid request type: %s" % request_type)
def on_create(event):
props = event["ResourceProperties"]
print("create new resource with props %s" % props)
return {}
def on_update(event):
physical_id = event["PhysicalResourceId"]
props = event["ResourceProperties"]
print("update resource %s with props %s" % (physical_id, props))
return {}
def on_delete(event):
physical_id = event["PhysicalResourceId"]
print("delete resource %s" % physical_id)
return {}
def is_complete(event, context):
if event['RequestType'] == "Delete":
return { 'IsComplete': True }
resource_props = event["ResourceProperties"]
stackset_name = resource_props["stackSetName"]
print(f"Checking for {resource_props['stackSetName']} stackset instance success")
client = boto3.client('cloudformation')
is_ready = False
for i in range(17): #4 min and 15 seconds
try:
stack_instances = client.list_stack_instances(StackSetName=stackset_name,CallAs="DELEGATED_ADMIN")
all_complete = True
for i in stack_instances["Summaries"]:
if i["Status"] == "CURRENT":
if i["StackInstanceStatus"]["DetailedStatus"] == "SUCCEEDED":
print(f"Stack Success: {i['StackId']}")
pass
elif i["StackInstanceStatus"]["DetailedStatus"] == "FAILED":
all_complete = False
print(f"Stack Fail: {i['StackId']}")
raise Exception(f"A stack instance failed to deploy: {i}")
else:
all_complete = False
break
elif i["Status"] == "OUTDATED":
if i["StackInstanceStatus"]["DetailedStatus"] == "FAILED":
all_complete = False
print(f"Stack Fail: {i['StackId']}")
raise Exception(f"A stack instance failed to deploy: {i}")
else:
all_complete = False
break
elif i["Status"] == "INOPERABLE":
all_complete = False
raise Exception(f"A stack instance has become inoperable: {i}")
else:
print(f"Unknown stack status:{i['Status']} in stack ID: {i['StackId']}")
all_complete = False
break
if all_complete:
is_ready = True
print("Stackset is ready")
break
except client.exceptions.StackSetNotFoundException:
# this means the stack set is not yet propagated, hurry up and wait.
pass
except Exception as e:
raise e
time.sleep(15)
return { 'IsComplete': is_ready }
This lambda is pretty simple, poll the stackset and look through all the instances for failures. Raise them if they exist, wait for complete otherwise.
CDK Pipeline Setup and deploy.
Now we’ll set up the stack, the self-mutating pipeline, and initialize the stackset construct we set up above. We use cdk-pipelines to deploy straight from the main branch of our github repo. This won’t stop us from deploying locally if we need but it will give us CI/CD out of the box.
lib/stacksets-cdk-stack.ts
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as codecommit from 'aws-cdk-lib/aws-codecommit';
import { CodeBuildStep, CodePipeline, CodePipelineSource } from 'aws-cdk-lib/pipelines';
import { DeployGithubProviderStage } from './pipeline-stage';
// import * as sqs from 'aws-cdk-lib/aws-sqs';
export class StacksetsCdkStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const pipeline = new CodePipeline(this, 'Pipeline', {
pipelineName: 'cdk-stacksets-pipeline',
selfMutation: true,
synth: new CodeBuildStep('SynthStep', {
input: CodePipelineSource.connection('mbennettcanada/stacksets-cdk', 'main',{connectionArn: "arn:aws:codestar-connections:us-east-2:761018853327:connection/27a113c9-a53d-41f9-ade6-7f4e88c9a492"}),
installCommands: [
'npm install -g aws-cdk'
],
commands: [
'npm ci',
'npm run build',
'npx cdk synth'
]
}
)
});
const deployWave = pipeline.addWave('DeployChanges');
const iamIdentityCenterStage = new DeployGithubProviderStage(this, 'Deploy');
deployWave.addStage(iamIdentityCenterStage);
pipeline.buildPipeline()
}
}Line 16 is what you’ll really need to edit here. You’ll need to put your own repo in there as well as your own connection ARN from Codepipeline connections (see first part of this blog).
lib/pipeline-stage.ts
Now we’ll set up the stage that we referenced above.
import { DeployGithubProviderStack } from './stackset';
import { Stage, StageProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
export class DeployGithubProviderStage extends Stage {
constructor(scope: Construct, id: string, props?: StageProps) {
super(scope, id, props);
new DeployGithubProviderStack(this, 'DeployGithubProvider');
}
}Honestly nothing to see here, just creating a new stage with a new stack that will initialize the Stackset construct.
lib/stackset.ts
Meat and potatoes Marie. Baby steps to the best part that pulls it all together:
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { GithubOIDCProviderStackset } from './constructs/github-oidc-stackset';
export class DeployGithubProviderStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const githubStackset = new GithubOIDCProviderStackset(this, "GithubOIDCProviderStackset", {
stacksetName: "GithubOIDCStackset",
stacksetRegions: ["us-east-2"],
targetAccounts: [], // If narrowing down to specific accounts, put those ID's here.
targetOrgUnits: ["ou-wivp-spzyzet9"],
assetBucketName: "cdk-orgshared-assets-761018853327-us-east-2" // This is setup in your cdk bootstrap. Get bucket name there.
})
}
}Line 10: Set to whatever you want for the name.
Line 11: Set to the regions you want this stackset deployed to.
Line 12: Use if narrowing down the accounts within the OU you list in line 13
Line 13: OU id to deploy the stackset to.
Line 14: The s3 bucket that we deployed in the second phase of this post. Likely this will be cdk-orgshared-assets-<DELEGATED_STACKSET_ADMIN_ID>-<REGION> but pull up s3 and give it a gander. If you use the one I give you here it’s gonna fail.
DEPLOY IT.
Alright you’ve got it all configured, time to watch the magic happen:
cdk deploy --all --profile <WHATEVER>After that deploys you’ll have a pipeline that will trigger and run. Check out codepipeline for funsies.
If you want to manually deploy the cdk stackset stage, run:
cdk ls --profile <WHATEVER>
cdk deploy StacksetsCdkStack/Deploy/DeployGithubProvider --profile <WHATEVERConclusion
Nice work eh. You’ve now got an IAAC self mutating pipeline to deploy IAAC templated stacks to accounts across an organization using CDK.
The real benefit of stacksets is that when an OU changes (accounts added to it) the stackset will deploy to that new account too. This helps with scaling out your org in an automated way.
If you need to look at a complete example, check out the code here: https://github.com/mbennettcanada/stacksets-cdk
If you’ve got questions or want help with your AWS environment feel free to reach out: About Me
One thought on “Deploy Stacksets via CDK”