Azure

  • Fabric Trials: The Countdown Is On

    We are drawing near to the end of Microsoft Fabric trials! ⌛ With the countdown ending mid-June for many trial capacities, it is important that you take measures to protect any Microsoft Fabric resources you have. Microsoft have informed users that once the trial ends, the capacity will be converted to a Power BI Premium capacity where Fabric items can no longer be used or stored. From here, you will have a 7 day retention period before all…

    » Read more
  • Synapse Data Flow bug

    I recently came across some unexpected output in a dedicated SQL Pool in Azure Synapse Analytics, upon investigation I realised this is as a result of an issue in Data Flows. This blog shows the issue using dummy data and a temporary workaround. At the time of writing (01/03/2024), this is still an issue and has been raised with Microsoft. I will provide a further update once Microsoft have resolved the issue or provided a suitable fix. Below is…

    » Read more
  • Upload files to a LakeHouse in OneLake using Powershell

    In this blog post, I will be sharing a PowerShell script that uploads files to a OneLake Lakehouse. This is the script: The script is downloadable at the bottom of the page as a txt file. To use the script there are few steps: Install the Powershell module Az.Storage. This code exists in the script and just needs to be uncommented for the first run, if the module is not already installed. Fill in the variables at the top of the script: tenantID…

    » Read more
  • Our Microsoft AI and ML Predictions for 2024

    As we step into 2024, Microsoft’s commitment to advancing Artificial Intelligence (AI) and Machine Learning (ML) is evident through its diverse range of offerings. The tech giant continues to push the boundaries of innovation, providing businesses with powerful tools to transform data into intelligent insights. In this blog, we’ll explore Microsoft’s current AI and ML landscape and peek into the possibilities that will…

    » Read more
  • Microsoft Fabric enters General Availability!

    Microsoft Fabric enters General Availability!

    After 6 months of public preview, Microsoft Fabric is now in production and generally available! 🎉 The announcement was made today at Microsoft Ignite, in Seattle. A round up of the news can be found at https://news.microsoft.com/ignite-2023-book-of-news/.   Here is a summary of some of the Fabric changes that we are looking forward to: Mirroring in Fabric Announced – expected in Q4 2023 Mirroring is a real-time clone of an Azure…

    » Read more
  • Synapse Script Activity Error “Argument {0} is null or empty.\r\nParameter name: paraKey”

    Came across this issue when trying to get a range of values that are calculated dynamically at runtime. Using the script activity in ADF, I wanted a result set that I could reference as parameters for a notebook later in the pipeline but was experiencing an error I had not seen before referencing null arguments. The Problem The script is simple – get a few key dates from the last few months and set these as variables. Finally, return these…

    » Read more
  • Automate Power BI single table refresh using Azure Synapse Analytics and Power BI APIs

    In a blog I posted back in June 2022, I explained how you could trigger a Power BI dataset refresh via Azure Synapse Analytics. https://www.purplefrogsystems.com/2022/06/how-to-trigger-a-power-bi-dataset-refresh-via-azure-synapse-analytics-in-3-simple-steps/ However, what if you just wanted to refresh a select number of tables or objects rather than a full dataset? This might be the case if you are performing intraday loads in your ETL. There…

    » Read more
  • Synapse Copy Activity Fails Over Certain File Size – ADF

    Copy Activity Issue in ADF / Synapse Analytics Recently, when trying to copy a .csv file from an FTP source to a Azure Data Lake using a Copy Activity in Azure Synapse, I had an issue where files > 16MB in size would fail. To overcome this, I took the first 13k rows and created another file from this, which resulted in a 4MB file. I tested this extra small file and it worked in the copy activity no issues. I multiplied these same 13k rows out…

    » Read more
  • PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘builtin_function_or_method’ and ‘float’ “

    This error was something I saw at the same time as the error I discussed in my previous blogpost (here), where we are seeing conflicting data types when trying to divide each value of a count of values by the number of days in 3 months (approximately) to get a frequency value over 3 months. I did show the code to fix the error we will discuss in the previous blog post, but I will go into more detail here. The code (without the line that fixes…

    » Read more
  • PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “

    Here’s the first in what will be an adhoc series of short blog posts where I will write a paragraph on the solution to problems I come across when I’m using PySpark. In this post I will discuss using the Map() function to apply a function to every value in an RDD and then getting the error message: “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “ We see this error because we are…

    » Read more