localstack-samples/sample-dms-postgresql-kinesis

This scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host the DMS replication instance, a database, a Kinesis stream, and a replication task.

• LocalStack Auth Token
• Python 3.10 & pip
• Docker Compose
• CDK with the cdklocal wrapper.

Start LocalStack Pro with the LOCALSTACK_AUTH_TOKEN pre-configured:

export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
docker-compose up

The Docker Compose file will start LocalStack Pro container and a Postgres container. The Postgres container will be used to showcase how to reach a database external to LocalStack.

Install all the dependencies by running the following command:

make install

To deploy the infrastructure, you can run the following command:

make deploy

NOTE: By default we create the kinesis target endpoint with default settings. For non-default settings we enable capturing events related to DDL operations and include NULL and empty column values from the events. In order to create the target with non-default values set the environment to KINESIS_TARGET=non-default. To know more about these settings, checkout the official AWS documentation.

After successful deployment, you will see the following output:

Outputs:
DMsSampleSetupStack.cdcTask = arn:aws:dms:us-east-1:000000000000:task:F6V3I917K2919C2HGVXCKE8O8AY19SX7M4TZH2U
DMsSampleSetupStack.dbSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-postgressecret-cb6c3bd1-vgGron
DMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-1d69ef19
Stack ARN:
arn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/8f4fb494

✨  Total time: 45.06s

You can run the tasks by executing the following command:

make run

A replication task gets deployed with the stack:

A CDC replication task runs against the RDS database:

• Creates three tables: authors, accounts, books
• Starts CDC replication task
• Captures and logs 4 Kinesis events: 1 for awsdms_apply_exceptions table, 3 for our tables
• Makes 3 inserts
• Captures and logs 3 Kinesis events
• Makes 3 table alterations, 1 per table
• Captures and logs 3 Kinesis events for non-default settings else 0
• Logs table_statistics for the task

You can deploy and run the stack on AWS by running the following commands:

make deploy-aws
make run-aws

This project is licensed under the Apache 2.0 License.