Call today 0845 643 6463

Tag Archive: Azure Data Factory

  • Synapse Copy Activity Fails Over Certain File Size – ADF

    July 21st, 2023 in Azure, Blog, CSV, Data Factory, Data Lake, Synapse

    Copy Activity Issue in ADF / Synapse Analytics Recently, when trying to copy a .csv file from an FTP source to a Azure Data Lake using a Copy Activity in Azure Synapse, I had an issue where files > 16MB in size would fail. To overcome this, I took the first 13k rows and created another file from this, which resulted in a 4MB file. I tested this extra small file and it worked in the copy activity no issues. I multiplied these same 13k rows out…

    » Read more

  • Azure Data Factory Pricing – How much is my pipeline actually costing me?

    December 1st, 2022 in ADFv2, Azure, Blog, Data Factory

    Has a client ever asked you how much it actually costs to run a single pipeline in Azure Data Factory? Have you ever thought ADF pricing is just a black box? Well, hopefully my latest blog post will give you an indication on how you can start calculating the cost of a pipeline run! I will base my analysis on a sample pipeline containing the following activities as shown below: 1 x Lookup Activity (Pipeline Activity) 1 x Copy Data Activity (Data…

    » Read more

  • Execute SQL Task in ADF

    February 25th, 2022 in Azure, Blog, Data Factory, SQL Server, SSIS, Synapse

    If you’re new to ADF or Synapse pipelines and looking for a way to execute SQL commands in pipelines, unfortunately there is no equivalent to the Execute SQL Task in SSIS, but I have found some alternatives which work well for me. For something as simple as a SELECT statement where you want to return results to be used later, you can use a Lookup activity: You can then reference the output of this activity in other activities using dynamic…

    » Read more

  • ADF Switch Activity – A neat solution for nested IFs.

    December 17th, 2020 in Azure, Blog, Business Intelligence, Data Factory, SQL Server

    Whilst carrying out some work for a client using Azure Data Factory I was presented with the challenge of triggering different activities depending on the result of a stored procedure. In fact the challenge posed was to… Execute ‘Copy A’ activity if the result of a stored procedure returned (A), Execute ‘Copy B’ activity if the result of a stored procedure returned (B), Execute ‘Copy C’ activity if the result of a stored…

    » Read more

  • ADF: Where have my connections gone?

    June 12th, 2020 in Azure, Blog, Business Intelligence, Data Factory

    If you’ve been doing any development work in ADF this week you might have noticed that “Connections” has moved. But where has it gone? When you click onto “Connections” now you’ll receive the following message: Clicking this button takes you to the new Management Hub, the new fourth icon which goes alongside the existing three. Within this area you now have access to your Linked Services, Integration Runtimes…

    » Read more

  • Azure Data Factory (ADF) Parent Activities don’t trigger ON ERROR OUTPUT when Child Activities Error… What happens?

    May 4th, 2020 in Azure, Blog, Data Factory

    So that title is a big question that a client of ours came across that required some additional investigation… What does happen and how do we get around it? In the below example we’ll use a simple IF Activity as our Parent Activity. To explain briefly how it functions, you provide it with a query that evaluates to either True of False. This then passes down to a set of Activities If True and a different set of Activities If False. We…

    » Read more

  • Azure SSIS – How to Setup, Deploy, Execute & Schedule Packages

    January 8th, 2018 in ADFv2, Azure, Blog, PowerShell

    Welcome back to work in 2018! 🙂 Let’s get stuck in with a hot topic. How do we actually use our beloved SQL Server Integration Services (SSIS) packages in Azure with all this new platform as a service (PaaS) stuff? Well, in this post I’m going to go through it end to end. Post Contents Azure Services Setup Azure Data Factory v2 Azure SQL Instance Azure SSIS IR Creating & Deploying the SSIS Package SSIS Azure Feature Pack…

    » Read more

  • Business Intelligence in Azure – SQLBits 2018 Precon

    November 17th, 2017 in Azure, Blog, Community

    What can you expect from my SQLBits pre conference training day in February 2018 at the London Olympia? Well my friends, in short, we are going to take whirlwind tour of the entire business intelligence stack of services in Azure. No stone will be left unturned. No service will be left without scalability. We’ll cover them all and we certainly aren’t going to check with the Azure bill payer before turning up the compute on our data…

    » Read more

  • RDP to Azure Batch Service Compute Nodes

    November 6th, 2017 in Azure, Blog

    Did you know it’s now possible to RDP to your Azure Batch Service compute nodes? I’ve used the batch service to handle the compute for my Azure Data Factory custom activities for a while now. Which I’ve basically been doing blindly because the code execution and logging is provided to ADF, with no visibility to the underlying pool of VM’s doing the work. Well, no more is this the case! In the Azure portal go to your Batch…

    » Read more

  • What’s New in Azure Data Factory Version 2 (ADFv2)

    September 24th, 2017 in ADFv2, Azure, Blog, C#, JSON, PowerShell

    I’m sure for most cloud data wranglers the release of Azure Data Factory Version 2 has been long overdue. Well good news friends. It’s here! So, what new features does the service now offer for handling our Azure data solutions?… In short, loads! In this post, I’ll try and give you an overview of what’s new and what to expect from ADFv2. However, I’m sure more questions than answers will be raised here. As developers we…

    » Read more