Data Factory

  • ADF Data flow string split

    This is a quick blog showing how to do a string split to get particular items in ADF data flows. Consider the following data where names and colours are combined into the FullName and Colours columns respectively. Note the delimiter for FullName is a space and the delimiter for Colours is a comma. To get each individual item and create new columns for this data use the split function in a Derived column transformation. The syntax for this…

    » Read more
  • ADF breaking out of a ForEach activity

    Currently, the ForEach activity in ADF (or Synapse) does not break out when an inner activity fails. There are several ways you can force the ForEach to break out, the most common is to cancel the pipeline run when an inner activity fails and there are already many blogs out there that cover this. My requirement is slightly different and that is what I will show here. Consider a scenario where you have a ForEach activity which has multiple inner…

    » Read more
  • ADF Dataflow CTE workaround

    At the time of writing, it is not possible to write a query using a CTE in the source of a dataflow. However, there are a few options to deal with this limitation: re-write the query using subqueries instead of CTEs use a stored procedure that contains the query and reference the stored proc in the source of the dataflow write the query as a view and reference the view in the source of the dataflow (this is my preferred method and the one I will…

    » Read more
  • How to parameterise the Execute Pipeline activity in Azure Synapse Analytics.

    Unfortunately, as of April 2022 there is not an option to parameterise or add dynamic content to an “Execute Pipeline” activity to invoke a pipeline run. However, with the use of a Microsoft API there is method which we can use to overcome this. In this blog post we’ll use the “Pipeline – Create Pipeline Run” API documented below: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-pipeline-run…

    » Read more
  • Execute SQL Task in ADF

    If you’re new to ADF or Synapse pipelines and looking for a way to execute SQL commands in pipelines, unfortunately there is no equivalent to the Execute SQL Task in SSIS, but I have found some alternatives which work well for me. For something as simple as a SELECT statement where you want to return results to be used later, you can use a Lookup activity: You can then reference the output of this activity in other activities using dynamic…

    » Read more
  • ADF Switch Activity – A neat solution for nested IFs.

    Whilst carrying out some work for a client using Azure Data Factory I was presented with the challenge of triggering different activities depending on the result of a stored procedure. In fact the challenge posed was to… Execute ‘Copy A’ activity if the result of a stored procedure returned (A), Execute ‘Copy B’ activity if the result of a stored procedure returned (B), Execute ‘Copy C’ activity if the result of a stored…

    » Read more
  • ADF: Where have my connections gone?

    If you’ve been doing any development work in ADF this week you might have noticed that “Connections” has moved. But where has it gone? When you click onto “Connections” now you’ll receive the following message: Clicking this button takes you to the new Management Hub, the new fourth icon which goes alongside the existing three. Within this area you now have access to your Linked Services, Integration Runtimes…

    » Read more
  • Azure Data Factory (ADF) Parent Activities don’t trigger ON ERROR OUTPUT when Child Activities Error… What happens?

    So that title is a big question that a client of ours came across that required some additional investigation… What does happen and how do we get around it? In the below example we’ll use a simple IF Activity as our Parent Activity. To explain briefly how it functions, you provide it with a query that evaluates to either True of False. This then passes down to a set of Activities If True and a different set of Activities If False. We…

    » Read more
  • Lookups and If Conditions in Azure Data Factory v2 (ADFv2)

    Azure Data Factory v2 (ADFv2) has some significant improvements over v1, and we now consider ADF as a viable platform for most of our cloud based projects. But things aren’t always as straightforward as they could be. I’m sure this will improve over time, but don’t let that stop you from getting started now. This post provides a walk through of using the ‘Lookup’ and ‘If Condition’ activities to do some…

    » Read more