Managing Kinesis streams for Replicate control tables
In a Replicate task writing data to an Amazon Kinesis endpoint, if you want to enable the control tables for the task and have the control table data separated from the replicated table data (where all the replicated table data is sent to the same stream) then the below settings and configuration will need to be put into place.
- In the Amazon Kinesis endpoint, under the "Data Message Publishing", change the setting "Publish the data to" to the 'Separate stream for each table' option. Enabling this allows for each control tables data to be written to their own stream in Kinesis. E.g. Replication status data will be written to a stream called attrep_status (see below)
- For each table add a column called $stream with the type STRING and set it's value to the name of the stream you'd like the data to be written to (see below).
You can also do this using a global transformation to add the column to all tables
- Ensure the streams exist on Kinesis prior to starting the tasks
You will also need to ensure that the AWS IAM user that Replicate uses to write to the Kinesis stream has all the required stream names in the privilege/allow list.
DataImage25.png
DataImage84.png
| Files | ||
|---|---|---|
|
|
DataImage25.png
37 KB
|
|
|
|
DataImage84.png
5 KB
|