Big Data – Looking Beyond Horizons
Big data is a field that deals with data sets that are too large or complex to be dealt with by traditional data-processing application software. Systematic extraction of information from these large data sets can be difficult when using legacy data-processing applications. There are 4 main categories that typically fall in this domain: Descriptive Analytics, Diagnostic Analytics, Predictive Analytics and Prescriptive Analytics.
Big Data – Limited Perceptions
When a layman is told by a company, ‘We analyze big data’, what they typically think of is the vast amounts of data points collected by marketing and advertising companies to target consumers with better ads and apply more subliminal conditioning to create desire for targeted products. And for most companies, that is the extent of big data usage – clean out outliers and use clustered data to identify goals and how to achieve them.
Enterprises have awakened to the reality that their big data stores represent a largely untapped gold mine that could help them lower costs, increase revenue and become more competitive. Within today’s competitive marketplace the ability to make the right tactical decisions, and to implement those decisions at the right time, can be the difference between success and failure. The intense focus on using big data to identify goals and make tactical decisions sometimes blurs the opportunity to look beyond those limited horizons and to fully capitalize on the intrinsic value of that data.
Expanding the Horizons
If you want to figure out trends of coronavirus spread and mutation, the analysis involved requires big data collection, warehousing, cleaning and then extracting meaningful analysis and information from the data. Raw data is seldom helpful in reaching a meaningful conclusion.
Want to make a self-driving car? You need the capability to process millions of visual data points per second. The AI systems used to create sophisticated systems cannot be trained by a lone developer sitting in a dark room somewhere. Light Detection and Ranging AI usually requires billions of data points to create the required neural networks, accurate geographical data in a 3D map and then make split second decisions in real time.
Even security systems run on big data now – by doing the exact opposite of conventional analytical systems. The outliers are detected, but not removed. Differentiating between inconsequential and threatening anomalies – a skill at which even humans fail occasionally- is a core functionality of these security systems. Such a system falls under the category of a diagnostic application.
So, as an IT or marketing or product-based company, ask yourself – are you as much a driver in the field of big data analytics as you aim to be? Even if we do decide to take up the challenge and drive innovation, there are a lot of hurdles in our path such as unstructured data, siloed data, and data growth. How can we, as an industry, make sure we are contributing to the field in some way and not simply relying on industry behemoths to lead us towards better analytics systems?