Tuesday, Mar 03 2015

The Big Data Technology Forecast for 2015

Written by

03 The Big Data Technology Forecast for 2015

It hasn’t taken long for businesses to jump on big data in a big way. The hype surrounding big data’s potential has only been circulating for a few years, but the technologies of big data are already disrupting the digital age — in 2014 alone, big data initiatives within companies were rapidly moving from the test phase to actual production.

The coming year will see big data evolving more into real-time usage, as new technologies enable enterprise functionality that can actually impact business. Here are some of the major developments poised to take place in big data for 2015.

More focus on data agility

The enormous potential behind big data lies in the ability to actually use it — which requires making sense out of massive streams of warehoused data and finding actionable connections. Data agility is crucial for any organization looking to move forward from capturing and managing data, to actively using the information.

However, traditional data warehouses and legacy databases aren’t fast or flexible enough to structure data into a usable format. In 2015, more organizations will transition away from legacy data resources and implement more agile platforms, with the ability to measure results in response times and operational impact rather than data volume.

Data lakes continue to evolve

The data lake, an object-based storage repository that stores structured, unstructured, and semi-structured raw data in native format, became highly popular in 2014 — mainly due to their scalability, agility, and cost effectiveness. This year, the data lake will evolve and gain more capabilities that will enable this platform to process data internally.

With an evolved data lake, large-scale enterprise processing platforms will be able to migrate from batch processing to real-time, with file-based, Hadoop, and database engine integration. The ability to store massive amounts of data will be less important than having access to tools that can process that data quickly, and pinpoint actionable trends and results.

More organizations embrace self-service big data

Traditionally, companies using big data have tasked IT with establishing centralized structures before allowing users to access the data. This process creates a bottleneck that slows down business, discovery, and exploration. But more organizations are moving toward self-service structures as the comfort levels with structure-on-read increase.

With direct access to big data, developers, data scientists, and data analysts can explore data directly, without waiting for IT to establish structures. This will allow companies to reduce costs and time-to-implementation, leverage new sources of data, and respond quickly to both opportunities and threats.

New business models evolve through Hadoop vendor consolidation

Hadoop is a big name in big data, and initially large enterprises like Intel were investing in the development of their own Hadoop distribution. But even with the unusually rapid pace of global Hadoop adoption, the big data platform is still in the innovation stage — and more companies are turning to established Hadoop vendors for support in the stack. Intel, for example, stayed with its own distribution for just one year before switching to distribution vendor Cloudera.

This year, the evolution of open source software (OSS) platforms like Hadoop will continue, with new models blending deeper innovations and greater community involvement.

Enterprise architects take the lead

The hype surrounding big data is already giving way to real-world usage, and enterprise architects are leading the way. As the understanding of the Hadoop technology stack improves, professionals who have been working most closely within this platform are creating more sophisticated and well-defined requirements for big data applications that are highly usable and scalable.

Last year, the Hadoop ecosystem introduced a landslide of tools, applications, and components. This year, those tools will be put to use as enterprise architects work to integrate Hadoop into the data center, and deliver the type of big data business results that have until now existed only as possibilities.