As I think more and more about big data analytics, I wonder if we will really find the answers we are looking for in that mass of data. This whole thing is reminding me of the search for the Higgs boson particle – we all have this sense that the answers must be somewhere in that mass of big data that we are collecting – and if we collect the right information and analyze it the right way at the right time then maybe we will have all the answers.
The answers out of big data are turning it into some “God Data” – in the same way; the Higgs particle was called the God Particle – it could contain all the answers to the question of life, the universe, and everything. If we ask the right question, we’ll get the answer to everything.
The question is – will we end up with the answer or just more questions? Maybe, as Douglas Adams put it – we may get the right answer, but we never asked the right question.
Maybe it’s a human nature thing – when confronted by a giant repository of information, our initial instinct (no matter how incorrect) is to believe that everything we are looking for is there; we just need to figure out how to find it. Just like those who use Google for all of their research have this impression that Google is attempting to present the most relevant information to you based on your search query. Having worked for a similar company in the past, I can tell you that relevance may not be the major factor in the list of results you get – yes – even barring the ads. Don’t you sometimes wonder why it takes a lot of manual mining to find what you are looking for, despite the engine supposedly refining its search based on your search history – you nearly never get the most relevant stuff upfront?
But I digress. I sense that the reality is that big data will give us some of the answers but not all of them, and possibly not the ones we want. Why is this? Simply because no matter how much data we currently collect, it’s still not enough for us to read the user’s tea leaves and truly understand what they are looking for.
Not to say that that day isn’t coming. Once every device is studded with sensors and begins to report our every action, keystroke, location, movement, heart rate, etc., we may then see the patterns. There is currently, and from what I can see, no service which can provide the most relevant info (yes, even Google Now) when you need it – and it’s really only because it still doesn’t know enough. With its ability to record everything you see and your location, Glass might help get us there. However, privacy concerns might kill this source of data.
I sense that answers will come – but only when we hit some tipping point with the data – that we finally have enough to answer the questions – and when we know how to ask the right ones. That day may be closer than we think.
