Azure Archives - Purple Frog Systems

Automate Power BI single table refresh using Azure Synapse Analytics and Power BI APIs
August 23^rd, 2023 in Azure, Blog, Power BI, Synapse, Tabular
In a blog I posted back in June 2022, I explained how you could trigger a Power BI dataset refresh via Azure Synapse Analytics. https://www.purplefrogsystems.com/2022/06/how-to-trigger-a-power-bi-dataset-refresh-via-azure-synapse-analytics-in-3-simple-steps/ However, what if you just wanted to refresh a select number of tables or objects rather than a full dataset? This might be the case if you are performing intraday loads in your ETL. There…
» Read more
Synapse Copy Activity Fails Over Certain File Size – ADF
July 21^st, 2023 in Azure, Blog, CSV, Data Factory, Data Lake, Synapse
Copy Activity Issue in ADF / Synapse Analytics Recently, when trying to copy a .csv file from an FTP source to a Azure Data Lake using a Copy Activity in Azure Synapse, I had an issue where files > 16MB in size would fail. To overcome this, I took the first 13k rows and created another file from this, which resulted in a 4MB file. I tested this extra small file and it worked in the copy activity no issues. I multiplied these same 13k rows out…
» Read more
PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘builtin_function_or_method’ and ‘float’ “
March 31^st, 2023 in Azure, Blog, Python, Spark
This error was something I saw at the same time as the error I discussed in my previous blogpost (here), where we are seeing conflicting data types when trying to divide each value of a count of values by the number of days in 3 months (approximately) to get a frequency value over 3 months. I did show the code to fix the error we will discuss in the previous blog post, but I will go into more detail here. The code (without the line that fixes…
» Read more
PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “
March 31^st, 2023 in Azure, Big Data, Blog, Python, Spark
Here’s the first in what will be an adhoc series of short blog posts where I will write a paragraph on the solution to problems I come across when I’m using PySpark. In this post I will discuss using the Map() function to apply a function to every value in an RDD and then getting the error message: “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “ We see this error because we are…
» Read more
Part 4: Natural Language Processing – Bringing it all together!
January 23^rd, 2023 in AI, Azure, Big Data, Blog, Business Intelligence, DAX, Power BI, Python
Here’s the final post in this blog series on natural language processing where we are going to bring everything together and web scrape Trust Pilot for review data, which we will then perform Natural Language Processing on and then display in a Power Bi dashboard. I’ll be talking exclusively practically in this demo, so for a refresher on the theory please refer back to my earlier blogposts (Part 1, Part 2, Part 3). To re-iterate the goal…
» Read more
Azure Data Factory Pricing – How much is my pipeline actually costing me?
December 1^st, 2022 in ADFv2, Azure, Blog, Data Factory
Has a client ever asked you how much it actually costs to run a single pipeline in Azure Data Factory? Have you ever thought ADF pricing is just a black box? Well, hopefully my latest blog post will give you an indication on how you can start calculating the cost of a pipeline run! I will base my analysis on a sample pipeline containing the following activities as shown below: 1 x Lookup Activity (Pipeline Activity) 1 x Copy Data Activity (Data…
» Read more
Part 3: Natural Language Processing – Sentiment Analysis and Opinion Mining
November 24^th, 2022 in AI, Azure, Big Data, Blog, Business Intelligence, Python, Synapse
If you remember in part 2 we discussed what Key Word Analysis is and how this can be implemented to gain deeper insight from textual data. But we can go one step deeper and extract feelings and opinions from the same data. We can do this through Sentiment Analysis and Opinion mining! In this blog I will talk you through what they are and how we can implement them using Microsoft’s Cognitive Services. What is Sentiment Analysis? We should…
» Read more
What are SQL Server Index Fragmentation and Index Fill Factor?
November 9^th, 2022 in Azure, Blog, SQL Server, Uncategorised
What is index fragmentation and how does it occur? It is important to know that SQL Server data is stored in data pages, with each page holding 8KB of data. There are two types of fragmentation, both are a result of these pages not being used as efficiently as possible. When you UPDATE or INSERT data on a page that is already full, SQL Server creates a new page. The information from the original page will be split 50/50 with half being added to…
» Read more
Part 2 : Natural Language Processing- Key Word Analysis
August 10^th, 2022 in AI, Azure, Big Data, Blog, Machine Learning, Python, Uncategorized
Here we are with part 2 of this blog series on web scraping and natural language processing (NLP). In the first part I discussed what web scraping was, why it’s done and how it can be done. In this part I will give you details on what NLP is at a high level, and then go into detail of an application of NLP called key word analysis (KWA). What is NLP? NLP is a form of artificial intelligence which deals with the interactions between humans…
» Read more
Query Store Forced Plan Failures
July 28^th, 2022 in Azure, Blog, SQL Server, Uncategorized
Query Store is a fantastic feature of both SQL Server and Azure SQL DB. It allows you to monitor how queries execute against the database which is invaluable for troubleshooting performance issues. More than that though, it gives you the option to force an erratic query to use a particular execution plan, this helps avoid queries from running with inefficient plans and provides predictability and stability on the server. Here we can see an…
» Read more

Alex Whittles (MVP)
Jeet Kainth
Jon Fletcher
Nick Edwards
Joe Billingham
Lewis Prince
Tom Owen
Callum Bryant
James Snow
Laura Adamson
Annabel Hemingbrough
Hollie Whittles