Tag Archive: Azure

  • Azure Business Intelligence – The Icon Game!

    As Azure becomes the new normal for many organisations our architecture diagrams become ever more complicated. Articulating our designs/data flows to management or technical audiences therefore requires a new group of cloud service icons in our pretty pictures. Especially for hybrid solutions. Sadly those icons aren’t yet that familiar for most. So, here’s a very simple blog post to help you recognise what’s in the Azure stack from a…

    » Read more
  • Calling U-SQL Stored Procedures with C# Code Behind

    So friends, some more lessons learnt when developing with U-SQL and Azure Data Lake. I’ll try and keep this short. Problem You have a U-SQL stored procedure written and working fine within your Azure Data Lake Analytics service. But we need to add some more business logic or something requiring a little C# magic. This is the main thing I love about U-SQL, having that C# code behind file where I can extend my normal SQL behaviour. So,…

    » Read more
  • Creating a U-SQL Date Dimension & Numbers Table in Azure Data Lake

    Now we all know what a date dimension is and there are plenty of really great examples out there for creating them in various languages. Well, here’s my U-SQL version creating the output from scratch using a numbers table. Remember that U-SQL needs to be handled slightly differently because we don’t have any iterative functionality available. Plus its ability to massively parallelise jobs means we can’t write something that…

    » Read more
  • Using Azure Data Factory Configuration Files

    Like most things developed its very normal to have multiple environments for the same solution; dev, test, prod etc. Azure Data Factory is no exception to this. However, where it does differ slightly is the way it handles publishing to different environments from the Visual Studio tools provided. In this post we’ll explore exactly how to create Azure Data Factory (ADF) configuration files to support such deployments to different Azure…

    » Read more
  • Writing a U-SQL Merge Statement

    Unlike T-SQL, U-SQL does not currently support MERGE statements. Our friend that we have come to know and love since its introduction in SQL Server 2008. Not only that, but U-SQL also doesn’t currently support UPDATE statements either… I know… Open mouth emoji required! This immediately leads to the problem of change detection in our data and how, for example, we should handle the ingestion of a daily rolling 28-day TSV extract,…

    » Read more
  • Azure Data Lake Authentication from Azure Data Factory

    rereTo set the scene for the title of this blog post lets firstly think about other services within Azure. You’ll probably already know that most services deployed require authentication via some form of connection string and generated key. These keys can be granted various levels of access and also recycled as required, for example an IoT Event Hub seen below (my favourite service to play with). Then we have other services like SQLDB that…

    » Read more
  • Creating Azure Data Factory Custom Activities

    When creating an Azure Data Factory (ADF) solution you’ll quickly find that currently it’s connectors are pretty limited to just other Azure services and the T within ETL (Extract, Transform, Load) is completely missing altogether. In these situations where other functionality is required we need to rely on the extensibility of Custom Activities. A Custom Activity allows the use of .Net programming within your ADF pipeline.…

    » Read more
  • Azure Stream Analytics Windowing Queries

    Warning, this is going to be a dry post. Beer or water required before you continue reading. For those that don’t already know Azure Stream Analytics is Microsoft’s cloud based service for the handling and manipulation of data feeds, predominantly in real-time. However the service has many applications with the ability to ingest a wide variety of data sources.  One such use case for the analysis of data using the service comes…

    » Read more
  • Windows IoT UWP Tasks Tickers and Threading

    Upon entering the IoT revolution a few things immediately became apparent; We now had the ability to collect and handle more sensor data than we’d ever before possibly conceived. These sensors and there data will/are going to change the very way we live… Not convinced by this one? Talk to anybody that has a Fitbit attached to their wrist about how many steps they do each day! More relevant to this article. The procedural tools we…

    » Read more
  • Azure Virual Machine CPU Cores Quota

    Did you know that by default your Azure subscription has a limit on the number of CPU cores you can consume for virtual machines? Until recently I didn’t. Maybe like you I’d only ever created one of two small virtual machines (VMs) within my MSDN subscription for testing and always ended up deleting them before coming close to the limit. Per Azure subscription the default quota for virtual machine CPU cores is 20. To clarify because…

    » Read more