Azure

Creating Azure Data Factory Custom Activities

When creating an Azure Data Factory (ADF) solution you’ll quickly find that currently it’s connectors are pretty limited to just other Azure services and the T within ETL (Extract, Transform, Load) is completely missing altogether. In these situations where other functionality is required we need to rely on the extensibility of Custom Activities. A Custom Activity allows the use of .Net programming within your ADF pipeline. However, getting such an activity setup can be tricky and requires a fair bit of messing about. In this post a hope to get you started with all the basic plumbing needed to use the ADF Custom Activity component.

Visual Studio

Firstly, we need to get the Azure Data Factory tools for Visual Studio, available via the below link. This makes the process of developing custom activities and ADF pipelines a little bit easier. Compared to doing all the development work in the Azure portal. But be warned, because this stuff is still fairly new there are some pain points/quirks to overcome which I’ll point out.

https://visualstudiogallery.msdn.microsoft.com/371a4cf9-0093-40fa-b7dd-be3c74f49005

Once you have this extension available in Visual Studio create yourself a new solution with 2x projects. Data Factory and a C# Class Library. You can of course use VB if you prefer.
vsdatafactoryproject

Azure Services

Next, like the Visual Studio section above this is really a set of prerequisites for making the ADF custom activity work. Assuming you already have an ADF service running in your Azure subscription you’ll also need:

  • Azure Batch Service (ABS) – this acts as the compute for your C# called by the ADF custom activity. absThe ABS is a strange service which you’ll find when you spin one up. Under the hood it’s basically a virtual machine requiring CPU, RAM and an Operating System. Which you have to choose when deploying it (Windows or Linux available). But none of the graphical interface is available to use in a typical way, no RDP access to the Windows server below. Instead you give the service a compute Pool, where you need to assign CPU cores. The pool in turn has Tasks created in it by the calling services. Sadly because ADF is just for orchestration we need this virtual machine style glue and compute layer to handle our compiled C#.
  • Azure Storage Account (ASC) – this is required to house your compiled C# in it’s binary .DLL form. Aascs you’ll see further down this actually gets zipped up as well with all it’s supporting packages. It would be nice if the ABS allowed access to the OS storage for this, but no such luck I’m afraid.

At this point, if your doing this for the first time you’ll probably be thinking the same as me… Why on earth do I need all this extra engineering? What are these additional services going to cost? And, why can I not simply inline my C# in the ADF JSON pipeline and get it to handle the execution?

Well, I have voiced these very questions to the Microsoft Azure Research team and the Microsoft Tiger Team. The only rational answer is to keep ADF as a dum orchestrator that simply runs other services. Which is fine if it didn’t need this extensibility to do such simple things. This then leads into the argument about ADF being designed for data transformation. Should it just be for E and L, not T?

Let’s bottle up these frustrations for another day before this blog post turns into a rant!

C# Class Library

Moving on. Now for those of you that have ever read my posts before you’ll know that I don’t claim to be a C# expert. Well today is no exception! Expect fluffy descriptions in the next bit 🙂

First in your class project lets add the NuGet packages and references you’ll need for the library project to work with ADF. Using the Package Manager Console (Visual Studio > Tools > NuGet Package Manager > Package Manager Console) run the following installation lines to add all your required references.

Install-Package Microsoft.Azure.Management.DataFactories
Install-Package Azure.Storage

Next the fun bit. Whatever class name you decide to use it will need to inherit from IDotNetActivity which is the interface used at runtime by ADF. Then within the your new class you need to create an IDictionary method called Execute. It is this method that will be ran by the ABS when called from ADF.

Within the IDictionary method. Extended properties and details about the datasets and services on each side of the custom activity pipeline can be accessed. Here is the minimum of what you’ll need to connect the dots between ADF and your C#.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
using System;
using System.Collections.Generic;
using System.Linq;
 
using Microsoft.Azure;
using Microsoft.Azure.Management.DataFactories.Models;
using Microsoft.Azure.Management.DataFactories.Runtime;
 
namespace ClassLibrary1
{
    public class Class1 : IDotNetActivity
    {
        public IDictionary<string, string> Execute(
                IEnumerable linkedServices,
                IEnumerable datasets,
                Activity activity,
                IActivityLogger logger)
        {
            logger.Write("Start");
 
            //Get extended properties
            DotNetActivity dotNetActivityPipeline = (DotNetActivity)activity.TypeProperties;
 
            string sliceStartString = dotNetActivityPipeline.ExtendedProperties["SliceStart"];
 
            //Get linked service details
            Dataset inputDataset = datasets.Single(dataset => dataset.Name == activity.Inputs.Single().Name);
            Dataset outputDataset = datasets.Single(dataset => dataset.Name == activity.Outputs.Single().Name);
 
            /*
                DO STUFF
            */
 
            logger.Write("End");
 
            return new Dictionary<string, string>();
        }
    }
}

How you use the declared datasets will greatly depend on the linked services you have in and out of the pipeline. You’ll notice that I’ve also called the IActivityLogger using the write method to make user log entries. I’ll show you where this gets written to later from the Azure portal.

adfreferencetoclassesI appreciate that the above code block isn’t actually doing anything and that it’s probably just raised another load of questions. Patience, more blog posts are coming! Depending on what other Azure services you want your C# class to use next we’ll have to think about registering it as an Azure app so the compiled program can authenticate against other components. Sorry, but that’s for another time.

The last and most important thing to do here is add a reference to the C# class library in your ADF project. This is critical for a smooth deployment of the solution and complied C#.

Data Factory

Within your new or existing ADF project you’ll need to add a couple of things, specifically for the custom activity. I’m going to assume you have some datasets/data tables defined for the pipeline input and output.

Linked services first, corresponding to the above and what you should now have deployed in the Azure portal;

  • Azure Batch Linked Service – I would like to say that when presented with the JSON template for the ABS that filling in the gaps is pretty intuitive for even the most none technical peeps amongst us. However the names and descriptions are wrong within the typeProperties component! Here’s my version below with the corrections and elaborations on the standard Visual Studio template. Please extend your sympathies for the pain it took me to figure out where the values don’t match the attribute tags!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
  "$schema": "http://datafactories.schema.management.azure.com/schemas/2015-09-01/
              Microsoft.DataFactory.LinkedService.json",
    "name": "AzureBatchLinkedService1",
    "properties": {
        "type": "AzureBatch",
      "typeProperties": {
        "accountName": "<Azure Batch account name>",
        //Fine - get it from the portal, under service properties.
 
        "accessKey": "<Azure Batch account key>",
        //Fine -  get it from the portal, under service properties.
 
        "poolName": "<Azure Batch pool name>",
        //WRONG - this actually needs to be the pool ID
        //that you defined when you deployed the service.
        //Using the Pool Name will error during deployment.
 
        "batchUri": "<Azure Batch uri>",
        //PARTLY WRONG - this does need to be the full URI that you
        //get from the portal. You need to exclude the batch
        //account name. So just something like https://northeurope.batch.azure.com
        //depending on your region.
        //With the full URI you'll get a message that the service can't be found!
 
        "linkedServiceName": "<Specify associated storage linked service reference here>"
        //Fine - as defined in your Data Factory. Not the storage
        //account name from the portal.
      }
    }
}
  • Azure Storage Linked Service – the JSON template here is ok to trust. It only requires the connection string for your blob store which can be retrieved from the Azure Portal and inserted in full. Nice simple authentication.

Once we have the linked services in place lets add the pipeline. Its worth noting that by pipeline I mean the ADF component that houses our activities. A pipeline is not the entire ADF end to end solution in this context. Many people do use it as a broad term for all ADF things incorrectly.

  • Dot Net Activity – here we need to give ADF all the bits it needs to go away and execute our C#. Which is again defined in the typeProperties. Below is a JSON snippet of just the typeProperties block that I’ve commented on to go into more detail about each attribute.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
"typeProperties": {
  "assemblyName": "<Name of the output DLL to be used by the activity. e.g: MyDotNetActivity.dll>",
  //Once your C# class library has been built the DLL name will come from the name as the
  //project in Visual Studio by default. You can also change this in the project properties
  //if you wish.
 
  "entryPoint": "<Namespace and name of the class that implements the IDotNetActivity interface e.g: MyDotNetActivityNS.MyDotNetActivity>",
  //This needs to include the namespace as well as the class. Which is what the default is
  //alluding to where the dot separation is used. Typically your namespace will be inheritated
  //from the project default. You might override this to be the CS filename though so be careful.
 
  "packageLinkedService": "<Name of the linked service that refers to the blob that contains the zip file>",
  //Just to the clear. Your storage account linked service name.
 
  "packageFile": "<Location and name of the zip file that was uploaded to the Azure blob storage e.g: customactivitycontainer/MyDotNetActivity.zip>"
  //Here's the ZIP file. If you haven't already you'll need to create a container in your
  //storage account under blobs. Reference that here. The ZIP filename will be the same
  //as the DDL file name. Don't worry about where the ZIP files gets created just yet.
}

adfsolutionwithcsharpBy now you should have a solution that looks something like the solution explorer panel on the right. In mine I’ve kept all the default naming conventions for ease of understanding.

Deployment Time

If you have all the glue in place you can now right click on your ADF project and select Publish. This launches a wizard which takes you through the deployment process. Again I’ve made an assumption here that you are logged into Visual Studio with the correct credentials for your Azure subscription. The wizard will guide you through where the ADF project is going to be deployed, it will also validate the JSON content before sending it up and it will also detect if files in the target ADF service can be deleted.

With the reference in place to the C# class library the deployment wizard will detect the project dependency and zip up the compiled DLLs from your bin folder and upload them into the blob storage linked service referenced in the activity pipeline.

Sadly there is no local testing available for this lot and we just have to develop by trial/deploy/run and error.

 

 

 

 

Runtime

adfmonandmanageTo help with debugging from the portal if you go to the ADF Monitor & Manage area you should have your pipeline displayed. Clicking on the custom activity block will reveal the log files in the right hand panel. The first is the default system stack trace and the other is anything written out by the C# logger.Write call(s). These will become your new best friend when trying to figure out what isn’t working.

Of course you don’t need to perform a full publish of the ADF project every time if your only developing the C# code. Simply build the solution and upload a new ZIP file to your blob storage account using something like Microsoft Azure Storage Explorer. Then rerun the time slice for the output dataset.
adfmonitoring
If nothing appears to be happening you may also want to check on your ABS to ensure tasks are being created from ADF. If you haven’t assigned the compute pool any CPU cores it will just sit there and your ADF pipeline activity will time out with no errors and no clues as to what might have gone wrong. Trust me, I’ve been there too.
azurebatchmon
I hope this post was helpful and gave you a steer as to the requirements for extending your existing ADF solutions with .Net activity.

Many thanks for reading.

Azure Stream Analytics Windowing Queries

stream-analyticsWarning, this is going to be a dry post. Beer or water required before you continue reading.

For those that don’t already know Azure Stream Analytics is Microsoft’s cloud based service for the handling and manipulation of data feeds, predominantly in real-time. However the service has many applications with the ability to ingest a wide variety of data sources.  One such use case for the analysis of data using the service comes when performing aggregations on the stream received by the job as an input. Like conventional querying the aggregation would be done against some grouping of the data set. Stream Analytics is no exception, however we need to understand that where we would previously alter our query to aggregate things by different groups of attributes (the query changes). We now fix our query when starting the job and stream data past it (the data changes). It is this shift in mind-set from a fixed dataset to a data stream that means we need to consider different grouping conditions for the attributes presented.

In Stream Analytics because, as the name suggests, data is streamed to/through the service time becomes the key for the grouping we impose for our aggregations. It is this time period for the grouping that is referred to as the window. To better understand what these windows look like lets compare some query syntax that we might use to produce a count on a dataset by day vs how we would aggregate this in a data stream by a window of day.


Dataset Data Stream
dataset-query data-stream-query2

If only everything could be explained by comparing a SQL like syntax for two different technologies 🙂

Assuming SQL is part of your brain parser we can predict from the above queries what results we are expecting to see. However for the data stream if nothing is received by our input within the defined window the count will be zero. I repeat the data changes, not the query as it’s streamed through the service. Thinking about the opposite of that for the above dataset query the results will only grow to include many day values as more data is inserted into the table. Where as the results in data stream will always return only one value as the data streams through the service and the window moves with the defined time period in the grouping.

Hopefully we now understand what a window is in our Stream Analytics job. Both in English and SQL!

If you still aren’t sure try this 1 liner from Microsoft’s MSDN page.

A window contains event data along a timeline and enables you to perform various operations against the events within that window.

Reference: https://msdn.microsoft.com/en-us/library/azure/dn835019.aspx

Next then, window types. This window of time for our stream of data can then take 3x different forms depending on what we need to report on; sliding, tumbling and hoping. Lets explore each in more detail with some potential real world examples to give there purpose some context. For the examples I’m going to use the health services as a basis for my scenarios.

Sliding

As a name suggests this first type of Stream Analytics windows slides with time. It has a defined size or duration and once set will move forwards aggregating any values in its scope.

Query example:

1
2
3
4
5
6
7
8
9
10
11
12
SELECT 
	DateAdd(hour,-1,System.TimeStamp) AS 'WinStartTime',
	System.TimeStamp AS 'WinEndTime',
	AVG([WaitTime]) AS 'SenorValue'
INTO
	[TargetAlerts]
FROM 
	[SystemInput]
TIMESTAMP BY 
	[CreatedDateTime]
GROUP BY
	SlidingWindow(Duration(hour, 1)) --window criteria

Scenario:

We are running an A&E department and we want to know the average wait time for all patients currently in the waiting room with a window duration of 1 hour. We may then want alerts if the current window value exceeds a set tolerance. What else happened in that window to clause that? Or we could just provide this information to waiting patients on a TV screen with a note that the sample is from a fixed size window. Call centres often provide there average wait time to answer the phone without any context of the window in which the aggregation was taken resulting in mislead expectations.

Tumbling

A tumbling window is a little harder to explain because how can time tumble. Well it doesn’t and nothing about tumbling down something should be implied. Instead think of the day as 24 separate windows all 1 hour in size or duration. Values stream into our job in the current 1 hour window. Then as we tick into the next hour the aggregate tumbles over and resets for the new window.

Query example:

1
2
3
4
5
6
7
8
9
10
11
12
SELECT 
	DateAdd(day,-1,System.TimeStamp) AS 'WinStartTime',
	System.TimeStamp AS 'WinEndTime',
	COUNT([ReferralId]) AS 'SenorValue'
INTO
	[DashboardOutput]
FROM 
	[SystemInput]
TIMESTAMP BY 
	[CreatedDateTime]
GROUP BY
	TumblingWindow(Duration(day, 1), Offset(millisecond, -1)) --window criteria

Scenario:

We are monitoring referrals being entered onto our hospitals computer system with a tumbling window of 1 day. Each day we count the number of referrals entered and display the information on a real-time dashboard. Triggers could then be put in place if the referrals entered fall below a given tolerance. Or if they exceed expected amounts because data isn’t being entered uniformly throughout the week.

Hoping

Hoping windows get a little easier to understand assuming we are comfortable with tumbling windows. The reason being is that a hop is the same as a tumble, but with an overlap in time. Sticking with our 24 hour day we could still have 24 hoping windows, but they now have a size or duration of 1.5hours meaning a 30 minute overlap.

Query example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
SELECT 
	DateAdd(hour,-1,System.TimeStamp) AS 'WinStartTime',
	System.TimeStamp AS 'WinEndTime',
	[WardName],
	COUNT([PatientId]) AS 'SenorValue'
INTO
	[DashboardOutput]
FROM 
	[SystemInput]
TIMESTAMP BY 
	[CreatedDateTime]
GROUP BY
	[WardName],
	HoppingWindow(Duration(hour, 1), Hop(minute, 30), Offset(millisecond, -1)) --window criteria

Scenario:

We are monitoring the number of patients on our wards with a simple count. A patient is then moved between wards but the move involves escorting the patient by wheel chair to the other side of the hospital. We want to allow enough time for the physical move to occur in our window aggregation so we allow a 30 minute overlap in the tumble. This means the patient may be double counted for a brief time. But that might be considered a better position than them not appearing on any ward. Eg. They virtual disappeared.

 

I hope you found this post useful. I did say it was going to be dry. There are a few pictures on the MSDN pages which might help with understanding but for me they were a little static and needed some animation or colour to be properly representative of each window type.

Many thanks for reading.

Windows IoT UWP Tasks Tickers and Threading

Upon entering the IoT revolution a few things immediately became apparent;

  • We now had the ability to collect and handle more sensor data than we’d ever before possibly conceived.
  • These sensors and there data will/are going to change the very way we live… Not convinced by this one? Talk to anybody that has a Fitbit attached to their wrist about how many steps they do each day!
  • More relevant to this article. The procedural tools we know and love like T-SQL and PowerShell are no longer going to be enough to deliver these new world real-time data requirements.

Moving on from the philosophy lets focus on my third point and the acceptance that we need to enter the realm of object orientated programming (OOP), specifically in the article C# .Net. Now I will state from the outset that I still consider myself to be a C# novice, but through IoT development the learning continues thick and fast. Although if you ask me directly to explain polymorphism I’ll still be running for Google 🙂

For procedural people already working with Microsoft products this learning and development can be fairly nice and sometimes even natural. However I expect people already involved in hard-core OOP software development and not very procedural this might seem a little backwards or just very obvious. Just a speculation at this point. At the moment I’m the former and if your reading this I hope you are too.

So why do we need OOP  for our data? What’s your point Paul?

Well being a Microsoft aligned person more and more I find myself working on Windows 10 IoT Core with C# based on the Universal Windows Platform (UWP) framework to develop apps and drive my sensors collecting that very precious data. For those of you that haven’t encountered the UWP concept yet I recommend visiting these Microsoft pages: https://msdn.microsoft.com/en-gb/windows/uwp/get-started/whats-a-uwp

Assuming you are familiar with a little UWP dev lets continue and dive straight into the first problem you’ll encounter, or may have already encountered.

Threading

In reverse order to my article title I know, but threading is basically the issue that we first need to work around when developing an IoT application. The UWP framework is great and very flexible however it only offers a cut down version of the full fat .Net library (at present). Or to be more accurate when working with a UWP solution the number of SDK’s available in your references will be very limited compared to what you might normally see.

UWPRefManager

This limit includes the well known System.Threading and classes like the following example from MSDN.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
using System;
using System.Threading;
 
public class MonitorSample
{
   public static void Main(String[] args)
   {
      int result = 0;
      Cell cell = new Cell( );
 
      CellProd prod = new CellProd(cell, 20);
      CellCons cons = new CellCons(cell, 20);
 
      Thread producer = new Thread(new ThreadStart(prod.ThreadRun));
      Thread consumer = new Thread(new ThreadStart(cons.ThreadRun));
 
         producer.Start( );
         consumer.Start( );
 
         producer.Join( );
         consumer.Join( );  
 
      Environment.ExitCode = result;
   }
}

Threading is simply not available on the Universal Windows Platform.

Tasks

Enter our new friends async and await tasks or asynchronous programming.

1
2
using System;
using System.Threading.Tasks;

Now I’m not even going to try and give you a lesson on C# as I’d probably just embarrass myself, so instead I will again direct your attention to following MSDN pages:

https://msdn.microsoft.com/en-us/library/mt674882.aspx

However what I will do is try and to give you some context for using this new “threading” none blocking concept within your UWP IoT application. The example I like to call on is very simple. You have an IoT sensor device that needs to do two things:

  1. Send JSON messages containing your data to an Azure IoT Event Hub (Device to Cloud)
  2. Receive messages containing device management instructions (Cloud to Device)

These two fundamental bits of functionality have to happen asynchronously. We can’t be waiting around to send messages because we are working on what has just been received. To handle this we need something like the following example at the core of our UWP app.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
namespace IoTSensor
{
    public sealed partial class MainPage : Page
    {
        private MainViewModel doTo;
        public MainPage()
        {
            this.InitializeComponent();
            doTo = this.DataContext as MainViewModel;
 
            Loaded += async (sender, args) =&gt;
            {
                await doTo.SendDeviceToCloudMessagesAsync();
                await doTo.ReceiveCloudToDeviceMessageAsync();
            };
 
        }
    }
}

Now both send and receive can occur without any blocking behaviour.

Tickers

Lastly lets think about tickers created using something like DispatcherTimer(). The good old fashioned clock cycle if you prefer.

We might need a ticker to cycle/iterate over a block of code that is doing something with our IoT sensors. For example if you wanted to collect a temperature reading every 10 seconds. Using an async task with a ticker would be the way to achieve that. For example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
namespace IoTSensor
{
    using System;
    using System.Threading.Tasks;
    using Windows.UI.Xaml;
    using Windows.UI.Xaml.Controls;
    using GHIElectronics.UWP.Shields;
 
    public sealed partial class MainPage : Page
    {
        FEZHAT hat; //THANKS https://www.ghielectronics.com/
        DispatcherTimer sensorCollection;
 
        public MainPage()
        {
            this.InitializeComponent();
        }
 
        private async Task SetupDevice()
        {
            this.hat = await FEZHAT.CreateAsync();
 
            this.sensorCollection = new DispatcherTimer();
            this.sensorCollection.Interval = TimeSpan.FromSeconds(10);
            this.sensorCollection.Tick += this.sensorCollection_Tick;
            this.sensorCollection.Start();
        }
 
        private void sensorCollection_Tick(object sender, object e)
        {
            //Get values and send to cloud etc...
        }
 
        private async void Page_Loaded(object sender, RoutedEventArgs e)
        {
            await SetupDevice();
        }
    }
}

I do hope this high level article has been of some use. I will attempt to follow up with a more deep dive look at the above once I’ve slept on the concepts and forced myself to leave the beloved SQL behind for another couple of weeks while we voyage every further into the Internet of Things!

Many thanks for reading


Azure Virual Machine CPU Cores Quota

Did you know that by default your Azure subscription has a limit on the number of CPU cores you can consume for virtual machines? Until recently I didn’t. Maybe like you I’d only ever created one of two small virtual machines (VMs) within my MSDN subscription for testing and always ended up deleting them before coming close to the limit.

Per Azure subscription the default quota for virtual machine CPU cores is 20.

To clarify because the relationship between virtual and physical CPU cores can get a little confusing. This initial limit is directly related to what you see in the Azure portal when sizing your VM during creation.

AzureBasicVMsFor example, using the range of basic virtual machines to the right you would hit your default limit with;

  • 10x A2 Basic’s
  • 5x A3 Basic’s
  • 2.5x A4 Basic’s (if it were possible to have 0.5 of a VM)

Fortunately this initial quota is only a soft limit and easily lifted for your production Azure subscriptions.

Before we look at how to solve this problem it’s worth learning how to recognise when you’ve hit the VM CPU core limit, as its not always obvious.

Limit Symptoms Lesson 1

When creating a new VM you might find the some of the larger size options are greyed out without any obvious reason why. This will be because consumed cores + new machine size cores will be greater than your current subscription limit. Therefore ‘Not Available’. Example below.
AzureVMSizesGrey

The other important quirk here is that the size blade will only dynamically alter its colourations and availability state if the CPU cores are currently being used in VMs that are running. If you have already reached your limit, but the VMs are stopped and appearing as de-allocated resources, you will still be able to proceed and deploy another VM exceeding your limit. This becomes clearer in lesson 2 below where the failure occurs.

Limit Symptoms Lesson 2

AzureVMCreateNormErrorIf you are unlucky enough to have exceeded your limit with de-allocated resource and you were able to get past the VM size selection without issue your new VM will now be deploying on your Azure dashboard if pinned. All good right?… Wrong! You’ll then hit this deployment failure alert from the lovely notifications bell. Example on the right.

Note; I took this screen shot from a different Azure subscription where I’d already increased my quota to 40x cores.

This could occur in the following scenario.

  • You have 2x A4 Basic virtual machines already created. 1x is running and 1x is stopped.
  • The stopped VM meant you were able to proceed in creating a third A4 Basic VM.
  • During deployment the Azure portal has now done its sums of all resources covering both stopped and running VM’s.
  • 8x cores on running VM + 8x cores on stopped VM + 8x cores on newly deployed VM. Total 24.
  • This has exceeded your limit by 4x cores.
  • Deployment failed.

In short; this is the difference in behaviour between validation of your VM at creation time vs validation of your VM at deployment time. AKA a feature!

Limit Symptoms Lesson 3

AzureVMCreateWithJSONError2If deploying VMs using a JSON template you will probably be abstracted away from the size and cores consumed by the new VM because in default template this is just hardcodes into the relevant JSON attribute.

Upon clicking ‘Create’ on the portal blade you will be presented with an error similar to the example on the right. This is of course a little more obvious compared to lesson 1 and more helpful than lesson 2 in the sense that the deployment hasn’t been started yet. But still this doesn’t really give you much in terms of a solution, unless you are already aware of the default quota.

Apparently my storage account name wasn’t correct when I took this screen shot either. Maybe another blob post required here covered where the Azure portal is case sensitive and where it isn’t! Moving on.

 

 The Solution

As promised, the solution to all the frustration you’ve encountered above.

AzureHelpAndSupportTo increase the number of CPU cores available to your Azure subscription you will need to raise a support ticket with the Azure help desk… Don’t sigh!… I assure this is not as troublesome as you might think.

Within the Help and Support section of your Azure portal there are a series of predefined menus to do exactly this. Help and Support will be on your dashboard by default, or available via the far left hand root menu.

Within the Help and Support blade click to add a New Support Request. Then follow the prompts selecting the Issue Type as ‘Quota’, your affected Subscription and the Quota Type as ‘Cores per Subscription’.

AzureIncreaseCPUQuota

Once submitted a friendly Microsft human will review and approve the request to make sure it’s reasonable…. Requesting 1000 extra CPU cores might get rejected! For me requesting an additional 30 cores took only hours to get approved and made available. Not 2 – 4 business days as the expectation managing auto reply would have you believe.

Of course I can’t promise this will always happen as quickly so my advise would be; know your current limits and allow for time to change them if you need to scale your production systems.

I hope this post saved you the time I lost when creating 17x VMs for a community training session.

Many thanks for reading.


My Introduction to the SQL Server Community

Once I was blind, but now I see!

In the early part of 2015 (I think) a work colleague and friend suggested that on Thursday evening I attend the SQL Server Midlands User Group. The event was completely new to me, I’d neverSQLBitsXVPic1 before ventured outside my professional comfort zone, or even had any appreciation that there was such a gathering about SQL Server locally. Anyway, I registered and got some more details. The event was currently ran by some guy called Alex Whittles.

Thursday evening came around and I made the journey from Stafford to Birmingham and the venue for the user group, the Midlands Art Centre (MAC). Little did I know that this event was to become the catalyst for my career and involvement in the SQL Server community. Needless to say the event was a massive eye opener to how close the SQL Server user community was, not just socially, but in terms of knowledge sharing and skills transfer. Family is the best way to describe it, a massive SQL Family.

Time passed and I proceeded to attend the next few Midlands User Groups, meeting new people and listening to the rich variety of speakers that gave up there time to attend each session. In short I was hooked. I might compare myself to the robot Johnny 5 from the 1986 film Short Circuit needing “more input”. Well the webinars started to flow and I quickly realised that I needed to shed my current skin and public sector employer and engage more with the SQL Server Community with a view to expanding my horizons.

Next I attended the Azure User Group, SQL Relay, SQL Saturday Exeter and then the big one SQL Bits XV (which was also the technical launch for SQL Server 2016). I was honoured to be an official helper at my first ever SQL Bits which felt incredible to be part of this huge event. Meeting people that are as passionate about what they do as I am was truly inspiring.

During this time my ambitions were rewarded and through the SQL Server Midlands User Group that guy I mentioned above, Alex Whittles, became my boss! I joined Purple Frog Systems as a Business Intelligence Developer in March 2016.

SQLBitsXVPic2

So what next? Well my friend, keep an eye on this blog and lets find out.

#SQLFamily

Thanks Rob for the pictures.

 


 

 

Paul’s Frog Blog

This blog is now closed to new comments or posts.

Please see new posts here