BI In the cloud
Traditionally Business Intelligence and data analytics systems were built on premise, installed on your internal servers.
With the rise and rise of cloud based services and storage, we're now entering a world where you can build your BI infrastructure without any capital investment in hardware. Cloud services, such as Microsoft's Azure, offer more scalability, performance and reliability than is usually available with on-premise solutions.
Is It Expensive?
With Cloud compute costs starting from under 50p (GBP) per hour, and global geo-redundant storage from under 3p (GBP) per GB, cloud data is not only here, it's very vost effective.
You only pay for what you use, so it's almost negligable to start off, then the costs only grow when you use it more. And you can turn it off at any time.
Is This 'Big Data'?
'Big Data' is a very popular phrase these days, and it means many differnet things to many different people. You could think of it as any of:
- 'All Data' - Storing everything, for ever. This gives you phenominal flexability to re-use historical data that you didn't think was relevant at the time.
- 'Unstructured Data' - Traditional data warehouses have focused on a specific data model, confirming data to a predefined structure so that it can be analysed effectively. Cloud services such as Hadoop and Data Lake allow you to take unstructured data (text, tweets, images, etc.) and perform complex analysis over it
- 'Lots of Data' - With the scalability of cloud services, you can scale your systems up to cope with petabytes of data, not the gigabytes or terrabytes that we're used to on prem
Does 'Cloud' mean 'Big Data'? No it doesn't. You can host a traditional 'On Prem' BI solution in the cloud, and gain benefit from the scalability, reduced infrastructure and sliding scale pricing model, without using any 'Big Data' solutions at all - if that suits your business.
What the cloud brings is the opportunity to expand and to make use of new technology to enhance what you already have. It's up to you which components you choose to implement.
Big Data / BI Hybrid
Microsoft Azure offers a huge range of functionality, each of which can be selected and enabled when needed throughout the life of a project. We can start off by building a more traditional data analytics and reporting solution, and can design it in such a way as to enable your data to be accessed by any number of 'big data' tools in the future. You may not need a Big Data solution now, but it's highly likely that you will in the future. Preparing the ground work for it now will significantly simplify any future adoption and reduce future costs.
Machine Learning and Artificial Intelligence
Azure Machine Learning offers a very powerful mechanism to use artificial intelligence and automated machine learning algorithms to understand your data and determine patterns of behaviour, which can then be used to predict future behaviour.
Purple Frog can help design your BI solution to enable this functionality over your data, right now.
Scenario 1 - IAAS (Infrastructure as a Service)
- Free up space in your server room
- Scale up the size and performance of the new server on demand
- Zero capital infrastructure costs, all costs are now OpEx and allocated to a specific project
- Geo-redundant infrastructure for increased resilience.
Scenario 2 - PAAS (Platform as a Service)
Your company already has a legacy data warehouse, but it's 10 years old and no longer satisfies the reporting flexibility or power that your business now requires.
Using the existing solution design as a starting point, we would redesign and enhance the data model to expand the reporting scope and functionality.
We would take a parallel extract of the source data, alongside your exsting data warehouse, and import it into Azure Data Lake.
From here, we can populate a new data warehouse using Azure's SQL DW, a cloud hosted masively parallel procesing datawarehouse engine.
The SQL DW can be queried in the same way as your existing on-prem solution, and can also be used to drive web based, mobile/tablet friendly reporting and dashboards for your remote workforce.
- Revised data model with more power
- Runs in parallel with yoru existing solution for ease of transition
- Mobile/Tablet friendly reporting for remote workforce
- Opens up the power of Azure for the next phase of your business
Scenario 3 - Real Time
A retail organisation has traditionally used an overnight load for their data warehouse, and now wants more real-time data as well as more insight into customer behaviour.
If required, the existing Data Warehouse can remain as it is, without any risk or changes to it. Retail transactions can be sent up to the cloud services, in real time. These are stored in Azure Data Lake, with a full history.
As they're received by Azure, they can be monitored in real time to detect any anomolies or spikes in the data patterns, raising alerts to be investigated and populating real-time performance and exception dashboards.
Data Lake Analytics can then be used to perform complex ad-hoc 'big data' analysis of trending or behaviour that falls outside of the scope of the legacy data warehouse reporting.
Machine Learning can also be brought in, to perform complex basket analysis of the data, to provide the 'Customers who bought this, also bought this' functionality that you see on many retail websites.
- 'All Data' stored forever
- Real-time monitoring
- Ad-hoc complex analysis
- Machine Learning
Is this for You?
There are of course an unlimited set of scenarios, and your organisation will need its own unique set of functionality that suits your requirements.
Purple Frog will help guide you through this process, and bring the power of the cloud to your data.