Big Data: What You Need to Know

This article is written for those interested in big data – either simply knowing what it is and what it refers to, or for those considering how it can improve their business.

We often get questions relating to big data and whether we are capable of delivering solutions in that realm. The short answer: of course. That said, it is usually the company using the analytics themselves that needs to be ready to use big data. We’ve been delivering reporting solutions since 1979. The technologies change, and new concepts like big data are established, but our job of delivering accurate reporting and intelligence solutions will never change. The question, however, is whether or not our client has a use case that requires big data.

The definition:

Big data is understood in many different ways, but the most commonly agreed upon definition is a large amount of data, some of it structured and some of it in unstructured forms, that is analyzed in real or near real-time for the purposes of highlighting previously unidentified trends or facts.

Of course, it is impossible for any one person or entity to be all-knowing in any one realm. Studying vast amounts of data to encounter otherwise invisible or granular realities is the closest that one can get to become omniscient and for that reason it is something that every company with available resources is looking into.

A recent Forbes article estimated that about 50% of Fortune 500 companies are doing big data projects, but from my experiences hearing about company leaders asking their employees to study things like Hadoop, I would estimate that many more are in experimental phases of big data analysis.

Some examples of problems that leaders are looking at big-data to solve:

• Toronto Mayor John Tory spoke in April about the city’s plans to hire big data experts to help solve the city’s traffic woes. Anyone living or working in the Greater Toronto Area can agree that the traffic is incredibly bad and that there are no simple solutions or immediate fixes. Mayor Tory hopes that studying big data can give the city staff some ideas.
• Police forces across North America are using big data to determine when, where and in which circumstances crime occurs so that they can create better preventative measures.
• Banks use big data to verify purchases and to try and find anomalies within your bank account that may indicate fraud, allowing them to have quicker and more accurate detection protocols.
• Amazon uses big data to show visitors products that they may be interested in, based on a combination of which products one has viewed, how long they stayed on that page, their general search history and their previous purchases.

Questions to consider:

“Is your traditional BI environment very mature or basic?”

The low-hanging fruit is a much easier goal for organizations not yet deep into the use of analytics. Simpler data analyses would likely have a more immediate ROI and be effective at initiating the business users to the idea of using data to strengthen their decision-making abilities. The idea of collecting every bit of data and analyzing it is an attractive one, though it is better to focus on standardized, easily understood metrics before moving into something more complex.

“Why BI?”

As mentioned above, there still needs to be a business case for a big data initiative. A “me too” attitude to any business process is likely to end in failure.

Consider the example of a Canada-wide retailer. This retailer has BI to report on location-based sales, time-based sales, and which products are more likely to be bought together. Operationally, this company is fairly advanced in BI and has grown rather efficient in their structure. Now, though, management wants to figure out a way to bring more people into the store and make each visitor more likely to spend money. This company can set up sensors in their store that measure how many people come in and where they spend the most time while they are inside. Any changes from this point on have measurable outcomes to be examined. That is, the effects of moving products to different parts of the store and laying out the store differently can be empirically tested. At the same time, the company can register clicks, page view length, and likelihood to buy products on the website and understand which are more popular than others, perhaps leading to more dynamic pricing that can maximize profits.

The key.

As you can see, the possible insights that big data can provide are of great potential in many aspects of the modern world – for social benefit, such as the lessening of Toronto’s traffic problem, and for economic benefit, such as a company becoming more knowledgeable about what their customer base desires. The key is to know, before creating a massive “data lake” filled with all sorts of different information, what the end goal of the project is. What sort of insight is looked for, and what data might be necessary to attain such an insight?

Final note.

It seems that the promise of big data’s potential has businesses climbing over themselves to gather all the data that is available to them. This is not necessarily wrong, but it alone will not help in establishing a big data initiative that truly provides value to the company’s bottom line. Just because much of the big data collected is unstructured does not mean that the company’s efforts to make sense of the data should be unstructured as well.