Synapse

  • Synapse Script Activity Error “Argument {0} is null or empty.\r\nParameter name: paraKey”

    Came across this issue when trying to get a range of values that are calculated dynamically at runtime. Using the script activity in ADF, I wanted a result set that I could reference as parameters for a notebook later in the pipeline but was experiencing an error I had not seen before referencing null arguments. The Problem The script is simple – get a few key dates from the last few months and set these as variables. Finally, return these…

    » Read more
  • Automate Power BI single table refresh using Azure Synapse Analytics and Power BI APIs

    In a blog I posted back in June 2022, I explained how you could trigger a Power BI dataset refresh via Azure Synapse Analytics. https://www.purplefrogsystems.com/2022/06/how-to-trigger-a-power-bi-dataset-refresh-via-azure-synapse-analytics-in-3-simple-steps/ However, what if you just wanted to refresh a select number of tables or objects rather than a full dataset? This might be the case if you are performing intraday loads in your ETL. There…

    » Read more
  • Synapse Copy Activity Fails Over Certain File Size – ADF

    Copy Activity Issue in ADF / Synapse Analytics Recently, when trying to copy a .csv file from an FTP source to a Azure Data Lake using a Copy Activity in Azure Synapse, I had an issue where files > 16MB in size would fail. To overcome this, I took the first 13k rows and created another file from this, which resulted in a 4MB file. I tested this extra small file and it worked in the copy activity no issues. I multiplied these same 13k rows out…

    » Read more
  • @@ROWCOUNT in Synapse Dedicated SQL Pool

    If you’ve been working with dedicated SQL pools in Synapse, you may have noticed that the SQL Server system variable @@RowCount is not supported. I will show you a workaround in synapse to get the same information. First, use the query label option to give your query an identifier that you can use to track it: SELECT * FROM Dim.Date OPTION (LABEL = ‘MyQuery’) You can then use DMVs to find your query using the specified label…

    » Read more
  • Linked Server with Unique Identifier Issue

    There is currently an issue with Linked Servers when querying and filtering by a guid (uniqueidentifier). If you query a Synapse SQL pool from your on-prem SQL server, for example, and attempt to filter the results using a uniqueidentifier column, you will get the following error message: Msg 103010, Level 16, State 1, Line 1 Parse error at line: 1, column: 408: Incorrect syntax near ‘guid’. Msg 110811, Level 16, State 1, Line 1…

    » Read more
  • Part 3: Natural Language Processing – Sentiment Analysis and Opinion Mining

    If you remember in part 2 we discussed what Key Word Analysis is and how this can be implemented to gain deeper insight from textual data. But we can go one step deeper and extract feelings and opinions from the same data. We can do this through Sentiment Analysis and Opinion mining! In this blog I will talk you through what they are and how we can implement them using Microsoft’s Cognitive Services. What is Sentiment Analysis? We should…

    » Read more
  • ADF Data flow string split

    This is a quick blog showing how to do a string split to get particular items in ADF data flows. Consider the following data where names and colours are combined into the FullName and Colours columns respectively. Note the delimiter for FullName is a space and the delimiter for Colours is a comma. To get each individual item and create new columns for this data use the split function in a Derived column transformation. The syntax for this…

    » Read more
  • ADF breaking out of a ForEach activity

    Currently, the ForEach activity in ADF (or Synapse) does not break out when an inner activity fails. There are several ways you can force the ForEach to break out, the most common is to cancel the pipeline run when an inner activity fails and there are already many blogs out there that cover this. My requirement is slightly different and that is what I will show here. Consider a scenario where you have a ForEach activity which has multiple inner…

    » Read more
  • How to Trigger a Power BI Dataset Refresh via Azure Synapse Analytics in 3 Simple Steps.

    In this blog post we’ll explore how to trigger a Power BI Dataset via Azure Synapse Analytics in 3 simple steps: Step 1: Create a new security group called “AzureSynapsePowerBIIntegration” in your Azure Active Directory. Next add the name of your Azure Synapse Workspace as a member of your new security group. This will be your service principal. Step 2: Login to powerbi.com and in the top right hand corner locate “Settings” and…

    » Read more
  • ADF Dataflow CTE workaround

    At the time of writing, it is not possible to write a query using a CTE in the source of a dataflow. However, there are a few options to deal with this limitation: re-write the query using subqueries instead of CTEs use a stored procedure that contains the query and reference the stored proc in the source of the dataflow write the query as a view and reference the view in the source of the dataflow (this is my preferred method and the one I will…

    » Read more