This scenario demonstrates how to use Database Migration Service (DMS) to create change data capture (CDC) tasks using the Cloud Development Kit in Python. It is a self-contained setup that will create a VPC to host the DMS replication instance, a database, a Kinesis stream, and a replication task.
- LocalStack Auth Token
- Python 3.10 &
pip
- Docker Compose
- CDK with the
cdklocal
wrapper.
Start LocalStack Pro with the LOCALSTACK_AUTH_TOKEN
pre-configured:
export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
docker-compose up
The Docker Compose file will start LocalStack Pro container and a Postgres container. The Postgres container will be used to showcase how to reach a database external to LocalStack.
Install all the dependencies by running the following command:
make install
To deploy the infrastructure, you can run the following command:
make deploy
NOTE: By default we create the kinesis target endpoint with default settings. For non-default settings we enable capturing events related to DDL operations and include
NULL
and empty column values from the events. In order to create the target with non-default values set the environment toKINESIS_TARGET=non-default
. To know more about these settings, checkout the official AWS documentation.
After successful deployment, you will see the following output:
Outputs:
DMsSampleSetupStack.cdcTask = arn:aws:dms:us-east-1:000000000000:task:F6V3I917K2919C2HGVXCKE8O8AY19SX7M4TZH2U
DMsSampleSetupStack.dbSecret = arn:aws:secretsmanager:us-east-1:000000000000:secret:DMsSampleSetupStack-postgressecret-cb6c3bd1-vgGron
DMsSampleSetupStack.kinesisStream = arn:aws:kinesis:us-east-1:000000000000:stream/DMsSampleSetupStack-TargetStream3B4B2880-1d69ef19
Stack ARN:
arn:aws:cloudformation:us-east-1:000000000000:stack/DMsSampleSetupStack/8f4fb494
✨ Total time: 45.06s
You can run the tasks by executing the following command:
make run
A replication task gets deployed with the stack:
A CDC replication task runs against the RDS database:
- Creates three tables:
authors
,accounts
,books
- Starts CDC replication task
- Captures and logs 4 Kinesis events: 1 for
awsdms_apply_exceptions
table, 3 for our tables - Makes 3 inserts
- Captures and logs 3 Kinesis events
- Makes 3 table alterations, 1 per table
- Captures and logs 3 Kinesis events for non-default settings else 0
- Logs
table_statistics
for the task
You can deploy and run the stack on AWS by running the following commands:
make deploy-aws
make run-aws
This project is licensed under the Apache 2.0 License.