0845 643 64 63

Storing U-SQL Assemblies in Azure Blob Storage

I’m hoping the title of this post is fairly self explanatory. Your here because like me you found that the MSDN language reference page for creating U-SQL assemblies states that it’s possible to store the DLL’s in Azure Blob Storage. But it doesn’t actually tell you how. Well please continue my friends and I’ll show you how.

The offending article: https://msdn.microsoft.com/en-us/library/azure/mt763293.aspx

The offending text snippet:

“Assembly_Source
Specifies the assembly DLL either in form of a binary literal or as a string literal or static string expression/string variable. The binary literal represents the actual .NET assembly DLL, while the string values represent a URI or file path to a .NET assembly DLL file in either an accessible Azure Data Lake Storage or Windows Azure Blob Storage. If the provided source is a valid .NET assembly, the assembly will be copied and registered, otherwise an error is raised.”

Before going any further, this post isn’t a dig at the usual lack of Microsoft documentation. Mainly because when I posted this problem as a question on Stack Overflow the missing information was provided from the horses mouth, Mr Michael Rys (@MikeDoesBigData). Therefore, all is forgiven and I’m more than happy to write this post on Microsoft’s behalf and for my fellow developers. #SQLFamily

http://stackoverflow.com/questions/40842170/create-usql-assembly-from-dll-in-azure-blob-storage

Thanks again Mike. Moving on…

Assumptions

Within your Azure subscription you have the following services already deployed and running.

  • Azure Data Lake Analytics (ADLa)
  • Azure Data Lake Store (ADLs)
  • Azure Storage, with a suitable blob container.

You are also comfortable with referencing assemblies in your U-SQL scripts and so far have done so by in lining the complied assembly in the U-SQL file. Or have stored the DLL in ADLs with a simple file path reference to the ADLs root directory.

Granting Access

The most important thing you’ll need to do to get this working, as Mike mentions in the SO answer, is allow your ADLa service to access the blob storage account. This only requires a few clicks in the Azure portal.

From the ADLa blade choose Data Sources and click Add Data Source.

Populate the preceding blade drop down menus with your preferred choices and click Add. You should then have the storage account listed as a ADLa data source. As below

Note; The Azure Storage account doesn’t need to be in the same data centre as the ADLa service, unlike ADLa and ADLs.

Create Assembly

Next the U-SQL.

To reference a DLL in the blob storage account container we need to create the assembly using the wasb URL. Like this:

wasb://YourBlobContainerName@YourBlobStorageAccountName.blob.core.windows.net/YourAssembly.dll

Complete CREATE ASSEMBLY syntax.

1
2
3
CREATE ASSEMBLY IF NOT EXISTS [YourSchema].[PurpleFrog.Pauls.DataLakeHelperFunctions]
FROM "wasb://AllSupportingFiles@MiscBlobsAccount.blob.core.windows.net/
      PurpleFrog.Pauls.DataLakeHelperFunctions.dll";

Why Do This

Hopefully pre-empting some comments on this post. Given that we can inline the assembly and store it in ADLs. Why would you want to put the DLL’s in a separate storage account?

Well, this is really just for operational convenience. In a recent project I was working on we had created a lot of custom code. Not just for Azure Data Lake, but also Azure Data Factory. We therefore used a blob storage account as a support bucket for all compiled code and parent object files. This gave us a centralised place to deploy to regardless of what service was consuming the libraries. Again, just for convenience. All DLL’s in one place for all services.

I hope you found this post helpful.

Many thanks for reading.


My SQL Relay 2016

In 2015 I attended a SQL Relay day in Birmingham, as a humble attendee with a colleague. I sat on the edge of my seat in every session soaking up all I could from the […] Continue Reading…

Paul’s Frog Blog

Hi, I'm Paul Andrew and yes I know it's not the best picture! Anyway, a little bit about me; I have many years experience working with SQL Server and Microsoft technologies. I'm the newest member of the Purple Frog Systems team and fairly new to the SQL Server community. This is my blog of all things in my professional world with no specific theme or subject, but it should be pure gold. Ha! Regardless I hope to inspire and help others with my knowledge sharing as I travel through the modern data driven world.
Thanks for visiting.
@mrpaulandrew