Tag Archive: Azure Data Factory

  • Execute SQL Task in ADF

    If you’re new to ADF or Synapse pipelines and looking for a way to execute SQL commands in pipelines, unfortunately there is no equivalent to the Execute SQL Task in SSIS, but I have found some alternatives which work well for me. For something as simple as a SELECT statement where you want to return results to be used later, you can use a Lookup activity: You can then reference the output of this activity in other activities using dynamic…

    » Read more
  • ADF Switch Activity – A neat solution for nested IFs.

    Whilst carrying out some work for a client using Azure Data Factory I was presented with the challenge of triggering different activities depending on the result of a stored procedure. In fact the challenge posed was to… Execute ‘Copy A’ activity if the result of a stored procedure returned (A), Execute ‘Copy B’ activity if the result of a stored procedure returned (B), Execute ‘Copy C’ activity if the result of a stored…

    » Read more
  • ADF: Where have my connections gone?

    If you’ve been doing any development work in ADF this week you might have noticed that “Connections” has moved. But where has it gone? When you click onto “Connections” now you’ll receive the following message: Clicking this button takes you to the new Management Hub, the new fourth icon which goes alongside the existing three. Within this area you now have access to your Linked Services, Integration Runtimes…

    » Read more
  • Azure Data Factory (ADF) Parent Activities don’t trigger ON ERROR OUTPUT when Child Activities Error… What happens?

    So that title is a big question that a client of ours came across that required some additional investigation… What does happen and how do we get around it? In the below example we’ll use a simple IF Activity as our Parent Activity. To explain briefly how it functions, you provide it with a query that evaluates to either True of False. This then passes down to a set of Activities If True and a different set of Activities If False. We…

    » Read more
  • Azure SSIS – How to Setup, Deploy, Execute & Schedule Packages

    Welcome back to work in 2018! 🙂 Let’s get stuck in with a hot topic. How do we actually use our beloved SQL Server Integration Services (SSIS) packages in Azure with all this new platform as a service (PaaS) stuff? Well, in this post I’m going to go through it end to end. Post Contents Azure Services Setup Azure Data Factory v2 Azure SQL Instance Azure SSIS IR Creating & Deploying the SSIS Package SSIS Azure Feature Pack…

    » Read more
  • Business Intelligence in Azure – SQLBits 2018 Precon

    What can you expect from my SQLBits pre conference training day in February 2018 at the London Olympia? Well my friends, in short, we are going to take whirlwind tour of the entire business intelligence stack of services in Azure. No stone will be left unturned. No service will be left without scalability. We’ll cover them all and we certainly aren’t going to check with the Azure bill payer before turning up the compute on our data…

    » Read more
  • RDP to Azure Batch Service Compute Nodes

    Did you know it’s now possible to RDP to your Azure Batch Service compute nodes? I’ve used the batch service to handle the compute for my Azure Data Factory custom activities for a while now. Which I’ve basically been doing blindly because the code execution and logging is provided to ADF, with no visibility to the underlying pool of VM’s doing the work. Well, no more is this the case! In the Azure portal go to your Batch…

    » Read more
  • What’s New in Azure Data Factory Version 2 (ADFv2)

    I’m sure for most cloud data wranglers the release of Azure Data Factory Version 2 has been long overdue. Well good news friends. It’s here! So, what new features does the service now offer for handling our Azure data solutions?… In short, loads! In this post, I’ll try and give you an overview of what’s new and what to expect from ADFv2. However, I’m sure more questions than answers will be raised here. As developers we…

    » Read more
  • Chaining Azure Data Factory Activities and Datasets

    As I work with Azure Data Factory (ADF) and help others in the community more and more I encounter some confusion that seems to exist surrounding how to construct a complete dependency driven ADF solution. One that chains multiple executions and handles all of your requirements. In this post I hope to address some of that confusion and will allude to some emerging best practices for Azure Data Factory usage. First a few simple questions: Why is…

    » Read more
  • Passing Parameters to U-SQL from Azure Data Factory

    Let’s try and keep this post short and sweet. Diving right in imagine a scenario where we have an Azure Data Factory (ADF) pipeline that includes activities to perform U-SQL jobs in Azure Data Lake (ADL) Analytics. We want to control the U-SQL by passing the ADF time slice value to the script, hopefully a fairly common use case. This isn’t yet that intuitive when constructing the ADF JSON activity so I hope this post will save…

    » Read more