Call today 0845 643 6463

Blog

  • Calling U-SQL Stored Procedures with C# Code Behind

    April 22nd, 2017 in Azure, Blog, C#, U-SQL, Visual Studio

    So friends, some more lessons learnt when developing with U-SQL and Azure Data Lake. I’ll try and keep this short. Problem You have a U-SQL stored procedure written and working fine within your Azure Data Lake Analytics service. But we need to add some more business logic or something requiring a little C# magic. This is the main thing I love about U-SQL, having that C# code behind file where I can extend my normal SQL behaviour. So,…

    » Read more

  • Wearing Four Hats at SQLBits 2017

    April 13th, 2017 in Blog, Community, Just For Fun, Purple Frog Systems

    I know there are lots of great blog posts already out there from great people sharing their experiences from SQLBits. However, I hope none of them can offer an account quite as unique as mine from this years event. Why? Well, most people go to SQLBits as attendee’s to learn, network and socialise. This certainly applies to me. But ‘attendee’ was just 1 of my 4 hats (roles) at SQLBits 2017. In this post I’d like to…

    » Read more

  • Storing U-SQL Assemblies in Azure Blob Storage

    March 17th, 2017 in Azure, Blog, U-SQL

    I’m hoping the title of this post is fairly self explanatory. Your here because like me you found that the MSDN language reference page for creating U-SQL assemblies states that it’s possible to store the DLL’s in Azure Blob Storage. But it doesn’t actually tell you how. Well please continue my friends and I’ll show you how. The offending article: https://msdn.microsoft.com/en-us/library/azure/mt763293.aspx The…

    » Read more

  • Creating a U-SQL Date Dimension & Numbers Table in Azure Data Lake

    February 2nd, 2017 in Azure, Blog, U-SQL

    Now we all know what a date dimension is and there are plenty of really great examples out there for creating them in various languages. Well, here’s my U-SQL version creating the output from scratch using a numbers table. Remember that U-SQL needs to be handled slightly differently because we don’t have any iterative functionality available. Plus its ability to massively parallelise jobs means we can’t write something that…

    » Read more

  • Passing Parameters to U-SQL from Azure Data Factory

    February 1st, 2017 in Azure, Blog, U-SQL

    Let’s try and keep this post short and sweet. Diving right in imagine a scenario where we have an Azure Data Factory (ADF) pipeline that includes activities to perform U-SQL jobs in Azure Data Lake (ADL) Analytics. We want to control the U-SQL by passing the ADF time slice value to the script, hopefully a fairly common use case. This isn’t yet that intuitive when constructing the ADF JSON activity so I hope this post will save…

    » Read more

  • Sampling data in Data Lake U-SQL for Power BI

    January 26th, 2017 in Big Data, Blog, Business Intelligence, Cloud, Data Lake, Power BI

    Being able to hook Power BI directly into Azure Data Lake Storage (ADLS) is a very powerful tool (and it will be even more so when you can link to ADLS files that are in a different Azure account!! – not yet available as at January 2017). However there is a problem, Data Lake is designed to scale to petabytes of data whereas Power BI has a 10GB limit. Yes this is compressed, so we’d expect around 100GB of raw data, however to load…

    » Read more

  • Using Azure Data Factory Configuration Files

    January 23rd, 2017 in Azure, Blog, JSON, Visual Studio

    Like most things developed its very normal to have multiple environments for the same solution; dev, test, prod etc. Azure Data Factory is no exception to this. However, where it does differ slightly is the way it handles publishing to different environments from the Visual Studio tools provided. In this post we’ll explore exactly how to create Azure Data Factory (ADF) configuration files to support such deployments to different Azure…

    » Read more

  • Writing a U-SQL Merge Statement

    December 13th, 2016 in Azure, Blog, C#, U-SQL, Visual Studio

    Unlike T-SQL, U-SQL does not currently support MERGE statements. Our friend that we have come to know and love since its introduction in SQL Server 2008. Not only that, but U-SQL also doesn’t currently support UPDATE statements either… I know… Open mouth emoji required! This immediately leads to the problem of change detection in our data and how, for example, we should handle the ingestion of a daily rolling 28-day TSV extract,…

    » Read more

  • Showing December to December to see opening values for a year in SSAS/MDX

    December 12th, 2016 in Analysis Services, Blog

    I came across an interesting MDX challenge this week; within a cube’s Date dimension, how to show December twice, once where it should be and again as the opening value for the following year. i.e. for each year I need to show Dec (prev yr), Jan, …, Nov, Dec. Why? Well if you consider the following pivot chart, you can clearly see growth from Jan to Feb, Feb to Mar, etc., but it’s very difficult to see the growth between Dec…

    » Read more

  • Azure Data Lake Authentication from Azure Data Factory

    December 2nd, 2016 in Azure, Blog, C#, PowerShell, Visual Studio

    rereTo set the scene for the title of this blog post lets firstly think about other services within Azure. You’ll probably already know that most services deployed require authentication via some form of connection string and generated key. These keys can be granted various levels of access and also recycled as required, for example an IoT Event Hub seen below (my favourite service to play with). Then we have other services like SQLDB that…

    » Read more