Enable and manage asset updates

Owned by Vytas Glambinskas
Nov 12, 2023
3 min read

Any asset created from an external connector can be turned into an updating asset.

Enabling asset updates

Asset updates can be toggled on from the Updates tab found on the asset overview page. By default, updates are toggled off. Once turned on, a range of information will be supplied to ensure you complete the configuration of an updating asset and are informed on how to initiate and monitor an update.

Set up an updating asset

  1. From the updates tab, toggle updates on.

  2. Select whether the update action is Append or Replace. This is important to let us know how the new data is to be processed relative to the existing data. Click save and enable updates to proceed

Note: This selection is not required for file assets as these can only be processed with a replace update action

  1. A range of processes take place

    1. A distinct data location on the asset connector is created for future data loads

    2. A distinct folder is created on the asset connector for .tnf files to be added. These are the trigger files that initiate a data ingest job

    3. A distinct folder is created into which update job status files will be loaded following a successful or failed data ingest

    4. A polling mechanism is initiated which polls the .tnf location for the arrival of any trigger files

    5. A modal containing all of the above information and instructions on how to perform updates is presented

  2. You can view the updates history table at any time. It will always contain at least one line item for the initial data load that took place when the asset was first created. Any subsequent update jobs will be listed in the updates history table. This table contains

    1. Trigger file name - the name of the .tnf file that was discovered by the polling mechanism. This is what signals that data has been loaded and is available for ingest.

    2. Time started

    3. Time completed

    4. Duration

    5. Status

      1. Completed

      2. Failed

      3. In progress

Initiating asset updates

Prepare your data

Upload the refreshed data content to a new subdirectory underneath the base location of the product in the object store source (e.g. AWS S3, GCP Cloud Storage, Azure Blob storage).

Note: This folder can have any name other than ‘singleload' which is a protected convention for the first data load when creating a product in the UI.

Warning: The data loaded into the new sub-folder must follow the same structure as the original data load used to create the product. This includes the same set of folders representing the tables in the product, the same data format (e.g. Parquet) and the same schema for each table.

Trigger the ingest

Once refreshed data content has been loaded to the endpoint location in step 1, a product update is initiated by copying a Transfer Notification File (TNF) into the /tnfs/ subdirectory of the product connector location. 

A TNF is a JSON text file containing the name of the newly created unique subdirectory containing the updated content - an example is shown below:

{"DataFolder":"newly_created_folder"}

where newly_created_folder is replaced by the newly created folder in the endpoint location (e.g. second_load).

The TNF file:

  • Must have the extension .tnf

  • Can be named using whichever convention you wish.

  • Must have the structure shown above.

The Data Exchange monitors the /tnfs/ subdirectories of repeating data products on a regular basis (parameterisable frequency with a default of 5 minutes), and when detected, initiates a load of the refreshed data from the sub-directory specified in the “DataFolder” JSON field.

When the data load process is complete the provided tnf will be moved to the /tnfs_complete/ subdirectory of the data product endpoint location with an additional field containing the status of the data ingest job, e.g.:

{"DataFolder":"newly_created_folder" "status":"success"}

You can view the status of your data product updates at any time.