For several years, it has been a limiting factor that Azure Data Factory (ADF) and Azure Synapse Analytics, and more recently Fabric, did not allow dynamic content to be passed into the invoked pipeline name within the Execute Pipeline activity.

I’ve blogged about a workaround for this in Azure Synapse Analytics previously here:

https://www.purplefrogsystems.com/2022/04/how-to-parameterise-the-execute-pipeline-activity-in-azure-synapse-analytics/

Fortunately, with the introduction of the “Invoke Pipeline (Preview)” activity, you can now dynamically execute pipelines.

The only minor drawback is that you need to provide the guid ID of the pipeline and the guid ID of the workspace it belongs to, rather than just the name. However, this is not a significant setback, as these IDs can be retrieved programmatically using the “List Data Pipelines” Fabric API or by hardcoding the values into a metadata-driven ETL control table (perhaps stored in a table in the new Fabric Azure SQL DB).

Note that for this to work, you need to set up a new cloud connection. Specifically, you will need a “Fabric Data Pipelines” connection, which currently only supports OAuth 2.0 connectivity.

Hopefully this helps simplify your metadata driven ETLs / ELTs.

Tags:

« Previous Post
Next Post »