Recently, I presented a talk at the Birmingham Data User Group, titled “Stop Hand-Whisking Your Data: Faster Fabric Development with the fabric-toolbox”. On the back of that, I wanted to summarise the talk into a blog and share how you can use fabric-toolbox to enhance your Fabric development.

Data engineering can feel abstract. Layers upon layers of transformation, movement, and logic that are hard to visualise at a glance. So, when talking about Microsoft Fabric and how to get the most out of it, sometimes the best way to explain it is through something we all understand: baking a cake.

Microsoft Fabric — Your Data Kitchen

Think of Microsoft Fabric as a professional kitchen. You’ve got one central space, OneLake, acting as the underlying store for everything. Within it, you have different workspaces, each like a dedicated area of the kitchen: a hot zone for active work, a prep bench for staging, and so on. The beauty of Fabric is that, just like renting a kitchen rather than building your own, you’re paying for the capacity you use rather than maintaining all the infrastructure yourself. One platform. One bill. Everything connected.

Within that kitchen, Fabric breaks down into key workloads: Data Engineering (pipelines and ingestion), Data Warehouses and Lakehouses (where data lives), Power BI (reporting and dashboards), and Real-Time Intelligence (live streaming data you can act on immediately). It opens up so many possibilities for organisations to harness meaningful insights into their data.

The Ingredients: Your Data Layers

At Purple Frog, we work with an ETL (Extract, Transform, Load) approach, others use the medallion architecture. Think of it in three stages:

Bronze (Extract): Raw data straight from source systems à Point of Sales (POS), HR, finance, Excel spreadsheets. It’s all the individual ingredients before they’ve been combined. A bag of flour on its own isn’t a cake, but you need it to make one.

Silver (Transform): This is where the baking begins. We clean and validate the data, remove duplication, and build accuracy. Just like weighing your ingredients…get it wrong, and your cake won’t rise, or worse, it’ll collapse entirely.

Gold (Load): The finished sponge cakes. Data warehouse and lakehouse tables that are structured, reliable, and ready to be queried and reported on. This is the point where you have something digestible, that resembles a cake, but there is still more we can do to enhance our final product!

A question that often comes up: how long should you store data in your Bronze layer? It depends on your business needs. Bronze is your full historical record, and it can save you enormously; if a source system has an issue or overrides data, you haven’t lost anything. That said, it’s worth defining a sensible retention period with stakeholders. Storing five years of data you’ll never report on will bloat your environment unnecessarily.

The Recipe: Pipelines and Notebooks

Pipelines are perfect for core ETL patterns. They’re no-code, drag-and-drop, and built for data movement. You can set schedules, configure dynamic file mapping, and build in native API calls without writing complex scripts. Built-in error handling and alerting means that if something fails overnight, you and your client know about it. At Purple Frog, we use the Outlook integration to fire emails to both our team and key stakeholders on pipeline success or failure.

On schema changes when using pipelines: our approach is to hardcode the specific columns we need, and cast the datatypes, rather than using SELECT *. If a source schema changes unexpectedly, this prevents an immediate failure and gives us time to review and adapt properly.

Spark Notebooks come into their own for complex transformations or tens of millions of rows. Spark’s distributed processing handles large datasets in a way pipelines can’t match. You don’t need to be a Spark or Python expert to get started either! Fabric provides helpful code snippets to get you started on loading and writing data. Notebooks can also be parameterised and reused across different pipelines; One skilled developer can build a reusable function that the whole team benefits from.

Kitchen Tools: Enter the fabric-toolbox

Up until recently, I was mixing cake batter entirely by hand. It worked eventually, but it was slow, tiring, and the results were inconsistent. Then I got an electric whisk, and suddenly the whole process was faster, smoother, and considerably less lumpy.

That’s essentially what the Microsoft fabric-toolbox is!

It’s an open-source GitHub repository built and maintained by the Fabric Customer Advisory Team, with community contributions that are reviewed before publishing. It’s a collection of tools, snippets, and solutions for things Fabric doesn’t natively handle (or could handle better). If you use it and find it useful, please star it on GitHub. Microsoft uses that signal to prioritise contributions and keep the toolbox evolving.

Fabric Assessment Tool

Before any migration from Synapse to Fabric, point this at your existing Synapse environment and it scans all your Spark resources: pipelines, notebooks, dataflows. It produces the scripts for a detailed HTML report, ranks resources by migration complexity, and flags things that haven’t been touched in years (a perfect opportunity to clean house before you migrate). It also exports JSON for deeper analysis. If you’re sitting on a large Synapse environment and wondering where to begin, start here.

Migration Assistant / ADF to Fabric Wizard

A community-built online wizard that maps existing ADF and Synapse resources to their Fabric equivalents, no installation required. Register with your directory, upload your template, select your target workspace, and the wizard does the heavy lifting. It generates a dependency graph for review before anything is published and flags mapping issues. One important note: data stays where it is. This handles pipelines and connections only, so your GDPR, ISO 27001, and SOC compliance remain intact. Works best for simpler ADF/Synapse environments. If you’ve got hundreds of complex activities, tackle it in sections or perhaps an opportunity to clean up and optimise your solution.

Pipeline Scheduling Snippets

Fabric’s native scheduling covers daily, weekly, and monthly runs, but if you need something more specific (say, a different ETL process on the first day of each month), it won’t handle that natively. There’s a JSON snippet in the Toolbox that covers this cleanly. Copy it into your pipeline’s JSON editor, delete the conditions you don’t need, wire up your event pipelines, and you’re done. I wish I’d found it before building workaround logic from scratch. There’s also a file arrival trigger snippet, so when a new file lands in your Lakehouse it can automatically kick off downstream processing.

Lineage Extractor

A visual map of how data flows through your entire environment, Bronze through to Gold and out to reports. Run it from a notebook directly in Fabric. When you’re considering a change to a dataset or transformation and need to know what else might break downstream, this is what you reach for.

Fabric Cost Analysis Tool

A monitoring solution that breaks down capacity usage and costs by workspace, individual item, and date range. If your billing has spiked, this is how you find out why. It also has a built-in data agent: ask it in plain English what a resource cost between two dates, and it answers. A good example of making Fabric accessible to stakeholders who aren’t going to open a notebook.

Emergency Toolkit Scripts

When baking, things can go wrong and with some quick fixes, you can be back on track to your final product. The same goes for Fabric, you may have unexpected spikes in CU usage or slow-running queries suddenly. Three scripts worth keeping handy:

The Icing on the Cake: Semantic Models and Reporting

Once your Gold layer is ready, the semantic model brings it to life for reporting: DAX measures, relationships, and formatting that make data actually useful. A poorly built semantic model produces inaccurate results, and inaccurate data destroys trust faster than almost anything else. If stakeholders can’t trust the numbers, they’ll stop using the reports or start building their own. I’m not sure about you, but I would much prefer to consume one of these cakes over the other!

Side-by-side comparison of two strawberry cakes. On the left, a mishapen cake covered in lumpy white icing that has melted and drooped, with strawberry slices used as eyes giving it an accidental skull-like appearance. On the right, a neatly presented two-layer cake with smooth pink buttercream, piped filling, and fresh strawberries arranged on top, displayed on a blue-rimmed plate with a red gingham tablecloth

A well-built, visually clear report does the opposite. It builds confidence, becomes something people actually want to use, and they’ll come back asking for more. That’s the relationship you want with your stakeholders.

The fabric-toolbox, used alongside Fabric’s native capabilities, gives you a clear migration path from Synapse, scheduling snippets for the gaps, lineage visibility to manage change with confidence, cost monitoring you can query in plain English, and a set of scripts for when things need fixing quickly. Microsoft Fabric is already a strong platform for modern data analytics. The Toolbox is the electric whisk that makes everything faster, smoother, and considerably less lumpy.

Keep an eye on the toolbox also! With it being a community repository, it is often contributed to with new features that may be exactly what you need or even something you didn’t know was capable. Who know’s, if you come up with a work around or tool, you could contribute too and help other users who are in need of help!

Tags: , , , , ,

« Previous Post
Next Post »