Qlik Replicate how to find the correct timestamp to resume a task

How to restart logstream task from timestamp

How do you find which timestamp to use to restart your tasks without missing data? This article will give insight into how to find the timestamp at which Replicate was last reading from at the source database.

Starting task from correct timestamp

When a Qlik Replicate task is stopped, an entry is created in the task logs. For example:

2023-10-19T14:32:51 [SORTER          ]I:  Final saved task state. Stream position 00000037.6ad8b1ed.00000001.0000.00.0000:244870.867634.16, Source id 684257314, next Target id 2193264361, confirmed Target id 2193264357, last source timestamp 1697686214000000 

The “last source timestamp” number “1697686214000000” can be converted into a readable format by going to https://www.epochconverter.com/

This timestamp represents the time at which this Qlik Replicate task was processing/reading from the source database’s transaction/redo logs when it was stopped. Also notice that the timestamp on the left in the logs “2023-10-19T14:32:51” differs from the source timestamp “2:30 PM”. This log timestamp should not be used as it does not indicate the source timestamp. Using it can result in data loss. Please use the source timestamp.

Once you have converted the source timestamp into a readable format, you can start the task by timestamp. You can choose to go back some time from this point to ensure that there are no missing changes, then clean up the duplicates at the target. In this case we go back one hour.

When the task is started from timestamp, the start of task log file should show the timestamp at which the task was started from in UTC time. This is because Qlik Replicate automatically converts server time to UTC in the logs. Example below:

2023-10-19T14:36:01 [TASK_MANAGER    ]I:  Task 'LogStreamTask' running CDC only in fresh start mode, starting from log position: 'timestamp:2023-10-19T02:30:14' (UTC).

Relevant recommendations and best practices

1.     Qlik recommends that when the logstream staging/parent task is started by timestamp then you must start all the associated child/downstream tasks using the same timestamp.

Refer to this article for more information: https://community.qlik.com/t5/Official-Support-Articles/Qlik-Replicate-Starting-Log-Stream-Tasks-with-Timestamp/ta-p/1879839

2.     When the logstream parent/staging task is reloaded or started by timestamp, the child/downstream tasks will stop capturing changes and the following error messages will appear in the task logs:

[SOURCE_CAPTURE ]W: End of timeline, no more records to read (ar_cdc_channel.c:1210)

[SORTER ]E: End of time-line reached and all records were applied, task will stop [1020101] (sorter_transaction.c:3486)

          There are 2 different solutions to this problem:

              (option 1) Reload the child/downstream task.

              (option 2) Start the child/downstream task at a timestamp before the parent/staging task was reloaded/restarted.

Reference to the article for more information: https://community.qlik.com/t5/Official-Support-Articles/Qlik-Replicate-Log-Stream-task-error-End-of-time-line-reached/ta-p/1904185