• How to Handle Deleted Rows in an Incremental Load

    Most data warehouse issues are apparent: a pipeline fails, an alert fires, you get a phone call at an unsociable hour. Deleted source data doesn’t do this. A row is removed from the source, and no one notices. It sits in the warehouse silently skewing measures and KPIs, until the finance director asks why a cancelled order is still in the revenue report. What’s causing this? Incremental loads use watermarks to avoid re-extracting…

    » Read more
  • Python Dictionaries – What do I need to know?

    Python Dictionaries – What do I need to know?

    When you’re learning Python, dictionaries are one of the most important data structures you’ll encounter. Unlike lists (see blog post) which store items by position, dictionaries store data as key-value pairs, making them the natural choice whenever you need to look something up by name rather than by index. In this post, we’ll cover everything you need to know about Python dictionaries: what they are, how they work, and how to…

    » Read more
  • Back Up and Restore a Fabric Lakehouse

    Jon wearing a black and purple Purple Frog Systems polo shirt, standing in front of a graphic background featuring hexagonal patterns and an image of a lakeside house surrounded by trees. Text on the image reads: “Back Up and Restore a Fabric Lakehouse.”

    Fabric Lakehouses have been around for a while, but they still offer no native way to back them up or restore them. Fortunately, the way Lakehouses are structured makes it possible to build your own backup process. A Lakehouse is split into two areas: Files and Tables. The Files area is straightforward, you can download and back up these items directly. The Tables are powered by Delta tables, which are made up of two key components: the…

    » Read more
  • Python Lists – What do I need to know?

    Python Lists – What Do I Need to Know?

    When you’re learning Python, lists are one of the first data structures you’ll encounter. They’re simple, powerful, and flexible. But what should you really know to use them effectively? In this post, we’ll cover everything you need to know about Python lists: what they are, how they work, and how to use them efficiently. What is a Python List? A list is a mutable, ordered, heterogeneous collection of items. Think of it as a…

    » Read more
  • How to Effectively Restore a Table in a Fabric Warehouse

    How to Effectively Restore a Table in a Fabric Warehouse

    If you’ve ever used a database, you’ve probably experienced that heart-jolting moment when you accidentally drop a table or run an update statement without a WHERE clause. In that instant, the table is either gone or potentially corrupted beyond repair. Fortunately, there are ways to recover data, depending on the environment you’re working in. In traditional databases like on-prem SQL Server, or other systems that rely on physical…

    » Read more
  • My Experience at Fabric Community Conference Europe 2024

    Stockholm: A city of charm and innovation Let’s start with the location, Stockholm. As a seasoned traveller, Stockholm is one of the few major European cities that I have not yet visited. So, when given the chance to go to FabCon Europe, I jumped at the chance. Stockholm is a great city, with beautiful historic architecture mixed with modern innovation and the people there were friendly and warm. The conference centre, the Stockholmsmässan,…

    » Read more
  • Upload files to a LakeHouse in OneLake using Powershell

    In this blog post, I will be sharing a PowerShell script that uploads files to a OneLake Lakehouse. This is the script: The script is downloadable at the bottom of the page as a txt file. To use the script there are few steps: Install the Powershell module Az.Storage. This code exists in the script and just needs to be uncommented for the first run, if the module is not already installed. Fill in the variables at the top of the script: tenantID…

    » Read more
  • How to combine images using Python

    How to combine images using Python

    In this blog post, I will be showing how to combine images using Python. I used this code in a for loop to combine hundreds of different images for one of our clients. These are the two images that we will combine. Pillow is a PIL fork, install Pillow using pip and import PIL into the Python script. Next, set the two variables. These images are saved in the same folder as the script, therefore only the filenames are required to reference them.…

    » Read more
  • How to add text to images using Python

    In this blog post, I will be showing how to add text to an image in Python. I used this code to generate hundreds of images with different text for one of our clients. The full script is downloadable at the bottom with the required extras. Firstly, install the Python library Pillow. Pillow is a PIL fork, install Pillow using pip and import PIL into the Python script.           In Python, select the image that you would…

    » Read more
  • SQL Server Agent Gantt Chart in Power BI

    In this blog post, I walk-through how to use a Power BI template file I have created that displays a Gantt chart which shows which SQL Server Agent Jobs have run and when. The purpose of the chart is to identify when jobs are overlapping and taking server resource off each other. If two or more resource intensive jobs overlap, it’s a good idea to schedule them at different times if possible. To use the template file, input the server name.…

    » Read more