What's the right way to rollback deployments using cdk-pipelines?

[samlaf]

There aren't a lot of examples out there regarding the "right" way to deploy/test/rollback changes using [cdk-pipelines]
(https://docs.aws.amazon.com/cdk/api/latest/docs/pipelines-readme.html).

cdk-pipelines don't seem to use CodeDeploy, but I haven't found any documentation as to why not. Instead, from what I understand, they apply cloudformation changesets directly and call that the "deploy" stage:

What happens when my Postman tests fail? The pipeline is stopped, but does it rollback my changes? Or do I have to do it manually? What if I want to use blue-green or canary deployments?

Here is the code that produced the above image:

export class OaMediaServerRoutesPipelineStack extends Stack {
	constructor(scope: Construct, id: string, props: MultistackProps) {
		super(scope, id, props);

		const yarnCommand = 'workspaces focus @oa/cdk-stacks @oa/lib @oa/http-api';

		// CodePipeline is the modern api, which replaced CdkPipeline:
		// https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/pipelines/ORIGINAL_API.md
		const pipeline = new CodePipeline(
			this,
			`Pipeline`,
			pipelineConfig(this, props, 'ServerRoutes', yarnCommand)
		);

		const stage = pipeline.addStage(
			new OaMediaHttpApiServerRoutesStage(this, `${props.fullAppName}ServerRoutes`, props)
		);
		stage.addPost(
			new ShellStep('Postman Tests', {
				env: {
					ENVIRONMENT: props.environment
				},
				installCommands: [
					'npm i -g newman',
					// codebuild runs as root user, so don't need sudo
					// https://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html#troubleshooting-root-build-commands
					'apt-get update',
					'apt-get install -y jq'
				],
				commands: ['cd http-api/tests', 'bash ./run $ENVIRONMENT']
			})
		);
	}
}

[s1mrankaur]

Looking for a solution too. Did you find any?

[samlaf]

I don't think there's a "proper" way to do it, unfortunately.
I ended up playing with canary deployments, but it turns out aws does not recommend their use with code pipelines.
So we just added a messaging bot that messages us on discord when the pipeline fails, and we have to manually correct the bug.

[jabalsad]

There is no CDK solution from what I can tell currently. There is an open issue for CDK to integrate with cloudformation's rollback feature, but this solution also leaves some to be desired.

For the moment, I've opted for just running cdk synth, uploading the assets manually to S3, and invoking a cloudformation change set change with the generated templates. By doing this, I at least have access to CFN's rollback feature.

One of the downsides of CFN's rollback feature is that the stack is effectively undeployable during the monitoring period.

If you need more flexibility than that, cobbling together actions using Step Functions is probably the solution that will allow you to do exactly what you need.