At this year’s Gartner IT Expo, several Gartner analysts described the newest challenge of information management as “Bigger Data.” Big Data is yesterday’s problem. Now we’re dealing with Bigger Data. Bigger Data isn’t a problem driven by the amount of data. We already knew that data was going to grow. Bigger Data refers to the growth of the scope of problem that Big Data represents.
Sure, at first Big Data was a big deal. But it is a well-defined problem. We knew we were going to have a lot of data to store and to move. And the answer was pretty straightforward — buy more storage and build bigger and better networks.
But that was really just the first order of magnitude of the problem. You see, now that we’ve started to figure out how to address dealing with petabytes of data, there are some people out there that have started applying analytics to all that data, with the goal of turning it into “information.”
This is an important distinction. Data by itself is just a bunch of facts that individually might be interesting but really don’t provide a lot of business value. Take for example something as simple as the number of visitors to a web site. Perhaps that number suddenly drops by 50%. And while that is an alarming statistic, it’s not actionable. That is, you can’t determine what action you should take just by knowing this single data element. Now, if you combine that data with some other data such as a simultaneous drop in search engine ranking from number 1 to number 12, you suddenly have information. Placing data in context creates information. Information is actionable.
The issue with creating information from Big Data is multi-faceted. First, there’s the basic issue of the volume of the data. But what’s even more challenging is the fact that those data elements are probably stored in ways that don’t facilitate easy correlation. Let’s go back to our fictional web site. Perhaps we stored our visitor counts in a simple database with two fields: day and total visits. But perhaps we stored our search engine ranking in another database with two fields: month and ranking. We don’t have a common index between the data sets to form a comparison. We’ll have to manipulate the data in some way to come up with a correlation. In this case, the manipulation isn’t too complex; perhaps we’ll just summarize the visit counts by month in order to come up with a monthly total. However, when we start thinking about Big Data and the amount of potential data elements that we might want to correlate, it’s easy to understand how much of a challenge developing meaningful information analytics could be.
And while this may seem like enough of a problem, the real challenge is going to come from your competitors. They are going to solve these problems. They are going to derive information that you won’t have. And they are going to use that information to create competitive advantages that you won’t be able to overcome. That’s the problem of Bigger Data. The video clip from Gartner in the tweet below is just a piece of the what Peter Sondergaard discussed around Big Data.
Peter Sondergaard, Gartner, Says Big Data Creates Big Jobsat #GartnerSYM See a brief video here: gtnr.it/RSQmZX
— Gartner (@Gartner_inc) October 23, 2012
The Bigger Data problem stems from the need for you to not only store and move Big Data, but the need to be able to use that data to compete. Your competitors are already working on it. Welcome to Bigger Data.
About Michael Anderson
Michael, very insightful article. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. There are open source big data processing platforms to help companies with these challenges by deriving insights from massive data sets quick and simple. One example is http://hpccsystems.com