As an AI engineer immersed in the ever-evolving world of data and machine learning, conferences offer more than just networking opportunities; they’re essential refuelling stations for inspiration, learning and innovation.

To be frank, SQLBits usually doesn’t fully fit the bill for me as an AI engineer, but as the lines between AI and data engineering have become increasingly blurred, the content of the talks has evolved to reflect this shift. For me, SQLBits this year was a place where the AI community could start to feel at home.

In this blog, I’m going to discuss my experience at SQLBits 2025 and reflect on how that experience mirrors the ongoing fusion of AI into the broader data community.

The fusion of data and intelligence…

This year, SQLBits was an undeniable testament to how deeply artificial intelligence has been embedded into the fabric of data engineering, analytics and even business intelligence. In previous years, AI and machine learning talks were limited in number and largely introductory in nature (With me doing one of said talks!). But this year’s conference made one thing crystal clear: AI is no longer an emerging technology within the data world. It’s infrastructure. Crucial.

From the main stage keynotes to breakout sessions, the convergence of machine learning, large language models (LLMs) and traditional data platforms was not only discussed, it was demonstrated in production-ready form. This is something I’ve only typically seen at AI-specific conferences.

Microsoft and a host of community experts showcased tangible examples of AI embedded into ETL pipelines, real-time dashboards and other analytic use cases.

What stood out most to me was the practical, down-to-earth tone. The sessions weren’t about exploring theoretical frontiers or showcasing moonshot prototypes. They focused instead on the everyday tooling that AI engineers use—Python notebooks, large language models, sentiment analysis and Spark clusters, to name a few.

I saw live demonstrations of:

  • Embedding OpenAI models into SQL queries to enable natural language querying of databases.
  • Integrating Hugging Face transformers into dataflows for contextual enrichment and classification.
  • Using vector embeddings and semantic search to redefine how knowledge is surfaced across massive, structured datasets.

The lines between data engineering and AI engineering have always been a little blurry, but SQLBits 2025 fully embraced that ambiguity and turned it into a strength. The sessions encouraged AI engineers and data engineers alike to think beyond batch vs real-time, or model vs pipeline, and instead focus on how to unify data and intelligence into one continuous, value-driven loop.

It was a refreshing shift from asking, “Can we do this?” to answering, “Here’s exactly how we are doing it, and how you can too.”

Stand-out sessions for me

I would just like to highlight some sessions which I enjoyed and got the most value from:

Christian Henrik Reich delivered a compelling session titled “Outperform Spark with Python Notebooks in Fabric”, where he challenged the default use of Spark in Microsoft Fabric for all data workloads. He demonstrated that while Spark is powerful for large-scale distributed processing, it often introduces unnecessary overhead for smaller or more iterative tasks. In such cases, Python notebooks can offer significantly better performance, faster startup times, and more efficient resource usage. Christian emphasised the practicality of using Python notebooks, especially when combined with lightweight, high-performance libraries like DuckDB and Polars, to handle many data engineering workflows more effectively.

Carlos Contreras delivered a focused session on reducing hallucinations in natural language to T-SQL translations using large language models (LLMs) within SQL Server on AWS. He emphasised grounding LLMs with up-to-date database schemas and leveraging techniques like prompt engineering and Retrieval-Augmented Generation (RAG) to improve accuracy. By integrating AWS services and low-code approaches, Carlos showcased how to build trustworthy, efficient AI-driven query systems that bridge the gap between natural language and reliable SQL output.

Simon Whiteley presented a session titled “Automating Engineering with AI – LLMs in Metadata-Driven Frameworks.” He explored how large language models (LLMs) can transform data engineering by shifting from traditional code generation to configuration-driven automation. By integrating LLMs with metadata, engineers can automate repetitive tasks such as schema creation and pipeline configuration, enhancing efficiency and reducing errors. Simon emphasised that this approach allows data teams to focus on strategic initiatives rather than manual coding, marking a significant evolution in data engineering practices.

To conclude my time at SQLBits 2025…

SQLBits 2025 marked a turning point where AI truly took centre stage within the data engineering community. With practical, production-focused content and sessions that embraced the convergence of LLMs, data platforms and automation, the conference provided great value for AI engineers like me.

It showcased that AI is no longer peripheral to the data world; it’s a core part of its foundation.

Tags: , , , , , ,

« Previous Post
Next Post »