It is no secret that data has been exploding in virtually every facet of every industry, from financial services, manufacturing, retail, and biotechnology to telecom, healthcare, transportation, and energy. In fact, data created in the previous two years was greater than the previous 5,000 years of humanity and 2017 will create even more data in one year alone.
There are lots of technologies being used to process this data. In the early days of big data, the primary technologies used were Hadoop and MapReduce, in the batch analysis of large datasets. Today, while Hadoop is still used in a variety of big data workloads, the more prominently used processing framework is Apache Spark, which is significantly faster than Hadoop. Moreover, the shift to the use of real-time analytics is accelerating, which has spawned further growth in emerging areas such as machine learning, natural language processing, and data visualization. Supporting these additional data use cases inevitably means all roads point to the cloud.
More Data, More Access Problems
The multitude of data-related use cases gives rise to the formation of new roles within every organization. Data Engineers, Data Analysts, and Data Scientists are some of the fastest growing careers in the marketplace today and must co-exist with traditional IT roles such as ETL Developers, Data Stewards, and MDM Specialists, as well as other line-of-business roles such as Business Analysts. But managing the access rights for all these roles on a daily basis is not easy. The upcoming deadline for enforcement of the European Union's General Data Protection Regulation (GDPR) also furthers the urgency for data privacy, security and governance processes.
One way to ensure that all data professionals are able to access data in a secure and governed manner is through the use of an integration platform-as-a-service (iPaaS) solution. iPaaS usage has been increasing steadily over the past five years, and today consist of a host of enterprise-level capabilities such as monitoring, logging, APIs, user management, and agile DevOps processes. IT is responsible for ensuring that the iPaaS functions as an effective point of control in not only facilitating the flow of data through an enterprise but also in ensuring that every motion of this data is tracked.
Empowering the Data-Driven Organization
But IT's responsibilities do not end there. They are also responsible for empowering all data professionals with the right user-interface to access and manipulate data for their daily needs — all through an iPaaS to ensure that security, privacy, and governance procedures are being followed. The way to fulfill these needs is through purpose-built "data apps" that enable the respective "data personas" to access and utilize the data they need without having to rely on IT for assistance with manipulating, transforming, processing, or analyzing that data.
Long-term, the future of these data apps is very much dependent on the robustness of the underlying iPaaS, as well as continued investment in iterating — and expanding upon the variety of these data apps in proportion with any data-related professions that crop up. What this also means is that the role of central IT will change from one that generally "keeps the lights on" — to one that has a DevOps and product-focused mentality for their main internal customers — the data professionals. Advanced API functionality built into the iPaaS will also enable a whole new breed of data apps that are focused on monetizing data that a company creates — and sharing it with other customers, and partners in the supply chain.