How to add missing columns to the change tables

Instructions

Step 1.  Stop the task. Mark the time at which the task was stopped, and the latency on the task when the task was stopped. You will use these two values  to calculate the Task Resume Timestamp.

Task Resume Timestamp  = (Time the task was stopped) - (latency in minutes) - (10 minutes)

You can check latency by looking at the graph in the Monitor tab.

This is one of the methods you can use to resume a Qlik Replicate task in order to not miss any changes. However please be aware that this will replay changes and cause duplicates.

Step 2. In Qlik Replicate add the required columns to your tables in the “Transform” tab. Then save the task.

Step 2. Create the columns on the Synapse change tables as is defined in Qlik Replicate. In order to find out the correct datatype for the new Synapse columns, refer to the datatype mapping section of the Qlik Replicate documentation (click here).

For example, you can look at the Qlik Replicate datatype which is visible in the table settings (image below).

Then compare the Azure Synapse datatype mapping to match what the STRING datatype is converted to in Synapse. The image below shows that the Qlik Replicate STRING datatype needs to be the VARCHAR datatype in Synapse. So, this means you would create a column on the Synapse change table of the type VARCHAR and the length 500.

Step 3. Once your columns have been added correctly on the change tables in Synapse, refresh the task's metadata by selecting Run -> Advanced Options -> Select 'Recreate all tables and then stop' -> press OK.

This will reload metadata then stop the task. Please ensure that full load is not enabled and that the “Change Table Creation” settings, in the “Store Changes Settings” tab are NOT set to “DROP and CREATE” or else this will delete all data in the change tables.

Step 4. Once the metadata is refreshed (this may take some time), resume the task by going to Run → Advanced Run Options → Select "Tables are already loaded. Start processing changes from" → Enter the timestamp acquired in step 1 (Task Resume Timestamp). 

Once you've resumed the task from timestamp, the task will reprocess some changes which will inevitably send duplicate records to the target table. This is so that missing changes can be avoided. Ensure that your downstream processes are capable of identifying and handling the duplicates. If you'd like help with that, please contact the IBT Helpdesk.