Spark Archives - Purple Frog Systems

Microsoft Fabric: Should I Use a Data Warehouse or Lakehouse?
September 1^st, 2023 in Blog, Data Warehouse, Fabric, Power BI, Spark, SQL
Fabric is the newest data analytics platform from Microsoft. It brings together the entire analytics process onto the Power BI service, meaning that you no longer need to be moving your data around from different services. The introduction of OneLake means that all data that you import whether from Cloud servers, On-Premises servers or more is available in one central location. With Fabric, you need to choose how you will store the data that you…
» Read more
How do I make a multi-line string in Python (so I can ctrl+C my SQL code)
August 29^th, 2023 in Blog, Jupyter, Machine Learning, Python, Spark, SQL
This should be a quick one, but it is something that took me a little while to work out, but now I know I use it pretty much every day in my projects to save me a lot of time. This is the ability to copy and paste my SQL code into my python code as a multi-line string without having to reduce it to be all on one line or using escape characters. Both of which can take a bit of time depending on how big your SQL is. You usually need to have SQL…
» Read more
What is Microsoft Fabric? (Power BI + Synapse + DW + DataLake + ML)
May 23^rd, 2023 in AI, Big Data, Blog, Business Intelligence, Data Factory, Data Lake, Fabric, Power BI, PowerBI, Spark
At today’s Build conference, Microsoft announced Fabric. What is this? In simple terms, think of taking Synapse Analytics, Data Warehousing, Data Lakes, Data Factory, Spark Notebooks and Machine Learning, and bring them all together into Power BI. This is underpinned by Microsoft OneLake, a high performance scalable data lake storage layer, supporting all of the above. OneLake is, as the name implies, one data lake that can be used across…
» Read more
PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘builtin_function_or_method’ and ‘float’ “
March 31^st, 2023 in Azure, Blog, Python, Spark
This error was something I saw at the same time as the error I discussed in my previous blogpost (here), where we are seeing conflicting data types when trying to divide each value of a count of values by the number of days in 3 months (approximately) to get a frequency value over 3 months. I did show the code to fix the error we will discuss in the previous blog post, but I will go into more detail here. The code (without the line that fixes…
» Read more
PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “
March 31^st, 2023 in Azure, Big Data, Blog, Python, Spark
Here’s the first in what will be an adhoc series of short blog posts where I will write a paragraph on the solution to problems I come across when I’m using PySpark. In this post I will discuss using the Map() function to apply a function to every value in an RDD and then getting the error message: “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “ We see this error because we are…
» Read more
Power BI Databricks Spark connection error
May 19^th, 2020 in Big Data, Blog, Cloud, Power BI, Spark
When querying data from Azure Databricks (spark) into Power BI you may encounter an error: “ODBC:ERROR [HY000] [Microsoft][Hardy] (100) The host and port specified for the connection do not seem to belong to a Spark server. Please check your configuration.“ This is usually caused by trying to connect to a ‘Standard’ Databricks instance, but Power BI (and ODBC in general) can only connect to Databricks using a…
» Read more

Alex Whittles (MVP)
Jeet Kainth
Jon Fletcher
Nick Edwards
Joe Billingham
Lewis Prince
Tom Owen
Callum Bryant
James Snow
Laura Adamson
Annabel Hemingbrough
Hollie Whittles

Microsoft Fabric: Should I Use a Data Warehouse or Lakehouse?

How do I make a multi-line string in Python (so I can ctrl+C my SQL code)

What is Microsoft Fabric? (Power BI + Synapse + DW + DataLake + ML)

PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘builtin_function_or_method’ and ‘float’ “

PySpark Problems: Using Map() gives the error “TypeError: unsupported operand type(s) for /: ‘Row’ and ‘float’ “

Power BI Databricks Spark connection error