Businesses today need to quickly deliver data to support more analytic applications, Data Science and AI, and microservices, used by their line of business personnel as well as their customers. 

 

Modern Data Architectures need to be developed to accommodate these new analytic requirements, larger data volumes and take advantage of newer technologies such as cloud, data lakes and data streaming.

 

Additionally, the concurrency of the data is becoming imperative, e.g. compliance & regulatory data, web data, social media, etc.

 

Putting DataOps in place along with procedures for sharing and using data across enterprise (by way of Business & Enterprise Architecture and IT Strategy & Planning) will enable every business user to deliver and provision data with ease and speed – to provide the competitive edge many companies seek.

 

But what is DataOps?

 

DataOps is a collaborative Data Management practice introduced in an official manner in mid-2018 on the Gartner 2018 Hype Cycle. It is not a technology solution, but a process for managing data, people and technology in a manner that improves efficiency and ways in which data is used across a company.

 

DataOps is the application of DevOps practices to data management and integration combined with AL/ML to reduce the cycle time of data analytics, with a focus on automation, collaboration and monitoring.

 

It offers a blueprint to solve architectural complexities and data challenges posed by data drift, hybrid multi-cloud architectures and real-time analytics (+ AI and ML).

 

Key Concerns & Points for Reflection

– What is DataOps?

– How it can help Data Scientist send more time modeling and less time working with data.

– It is important to have a view of the entire lifecycle of the data from source through the pipelines to operation.

– Some companies are automating data access to provide self-service to analysts.

– Data sources are growing (GSK has 1,800 sources), data is constantly changing, requiring rework or faulty results.

– The concurrency of the data is becoming imperative, e.g. wellhead data, web data, social media, etc.

 

– Putting DataOps in place can be the competitive edge many companies seek.

– The technology and product space is rapidly changing – which means everyone (especially IT Leaders, CTOs, CIOs, Business & Enterprise Architects) needs to keep up by getting their ‘hands dirty’ so that they can create scalable architectures and business centric solutions that provide real tangible benefits and ROI.

Parting Thoughts

The key point to remember is that DataOps is a process – not a solution that can be purchased and installed on the network or in the cloud – and a core component of (future) enterprise Data Architecture.