Big Data: Finding the incremental value
I recently spoke at a conference on the topic of Big Data. In preparation I asked my contacts for their opinions on the topic. I also read a mountain of academic papers, business articles and a new book on the topic titled “Creating Value with Big Data Analytics”. It’s clear that Big Data divides opinions!
Business leaders appear to have bought in to the narrative that if you have a Big Data solution you’ll be able to find new pearls of insight that transform your business. Pearls that otherwise would not have been available.
Those of a more sceptical disposition contend that the data is often already available. What is required is the business ability to access the traditional (aka boring) relational data that’s already in warehouses and turn it into insight that guides business decisions.
It ultimately boils down to value. Actually, it’s really about the <strong>incremental value</strong> achieved from Big Data vs traditional data analysis and insight.
The question is not, will we get value from Big Data? It is instead how much extra value will we get from using Big Data?
There’s more to the value equation than just technology too. What are the processes to turn Big Data into insight that can be applied? Who will run it and maintain it? Where is the data coming from? Do we have permission to use the data in this way? How will we store all of this data?
Big Data Value Creation Model
Creating Value with Big Data Analytics included the following value chain which I think should be considered in any business that wants to get more out of data:
In closing out here are my pros and cons of Big Data.
List of Big Data pros
- Speed of data processing – configured correctly, Big Data applications can process huge volumes of data in real-time. Applications of this include identifying fraud, monitoring threats, or personalising real-time customer experiences
- Handling unstructured data – photos, documents and audio recordings can all be processed by Big Data applications to understand what is happening
- Cost efficiency – compared with expensive Oracle data warehouses, open source Big Data technologies can offer a significant cost improvement on traditional RDBMS solutions
- Salary boost – certainly as an individual, Big Data skills mean big salary increases
- Marketing – similar to the salary point; claiming your service/product uses Big Data will give it extra credit with prospective buyers
- Flexibility – sudden change in data fields or you need to ingest data from another source? No problem Big Data has your back
List of Big Data cons
- Costs – I’ve heard accounts of costs spiraling when organisations are storing huge amounts of information in cloud based solutions. Remember Amazon’s pricing scales just as well as its services
- Technical debt – building a solution that uses new cutting edge technology will come with issues that you don’t forsee and that haven’t been documented fully
- People – similar to the above, finding capable data scientists and big data engineers is hard and expensive
- Training – former skills may become redundant and there will be a wholesale need to train staff on the use of the new platform
- Value generation – focus on technology doesn’t allow business and insight processes to catch up and fully leverage Big Data
- Unnecessary – I hear this argument against Big Data frequently; why should we build something with Hadoop when we can just use Oracle
- Lack of business case – a second frequent challenge is that the reasons for building a Big Data solution don’t stack up against commercial business drivers
- Lack of user friendly front-end – this is improving, but there are still very few tools that allow an intermediate level user to analyse a Big Data solution. An exmaple being that there are no Multichannel Campaign Management tools that can run directly off traditional Big Data stacks
- Still requires an RDBMS – Often the Big Data solution is acting as a data warehouse and is then injecting data into a normal RDBMS for reporting. You can’t just abandon your traditional tools
Bonus Big Data model
The following model from an HBR article on the Internet of Things gives another great value creation flow for Big Data: