Tag Archive: JSON

  • Python Dictionaries – What do I need to know?

    Python Dictionaries – What do I need to know?

    When you’re learning Python, dictionaries are one of the most important data structures you’ll encounter. Unlike lists (see blog post) which store items by position, dictionaries store data as key-value pairs, making them the natural choice whenever you need to look something up by name rather than by index. In this post, we’ll cover everything you need to know about Python dictionaries: what they are, how they work, and how to…

    » Read more
  • Chaining Azure Data Factory Activities and Datasets

    As I work with Azure Data Factory (ADF) and help others in the community more and more I encounter some confusion that seems to exist surrounding how to construct a complete dependency driven ADF solution. One that chains multiple executions and handles all of your requirements. In this post I hope to address some of that confusion and will allude to some emerging best practices for Azure Data Factory usage. First a few simple questions: Why is…

    » Read more
  • Passing Parameters to U-SQL from Azure Data Factory

    Let’s try and keep this post short and sweet. Diving right in imagine a scenario where we have an Azure Data Factory (ADF) pipeline that includes activities to perform U-SQL jobs in Azure Data Lake (ADL) Analytics. We want to control the U-SQL by passing the ADF time slice value to the script, hopefully a fairly common use case. This isn’t yet that intuitive when constructing the ADF JSON activity so I hope this post will save…

    » Read more
  • Using Azure Data Factory Configuration Files

    Like most things developed its very normal to have multiple environments for the same solution; dev, test, prod etc. Azure Data Factory is no exception to this. However, where it does differ slightly is the way it handles publishing to different environments from the Visual Studio tools provided. In this post we’ll explore exactly how to create Azure Data Factory (ADF) configuration files to support such deployments to different Azure…

    » Read more
  • Creating Azure Data Factory Custom Activities

    When creating an Azure Data Factory (ADF) solution you’ll quickly find that currently it’s connectors are pretty limited to just other Azure services and the T within ETL (Extract, Transform, Load) is completely missing altogether. In these situations where other functionality is required we need to rely on the extensibility of Custom Activities. A Custom Activity allows the use of .Net programming within your ADF pipeline.…

    » Read more
  • Azure Virual Machine CPU Cores Quota

    Did you know that by default your Azure subscription has a limit on the number of CPU cores you can consume for virtual machines? Until recently I didn’t. Maybe like you I’d only ever created one of two small virtual machines (VMs) within my MSDN subscription for testing and always ended up deleting them before coming close to the limit. Per Azure subscription the default quota for virtual machine CPU cores is 20. To clarify because…

    » Read more