What devops needs to know about data governance

Data governance is an umbrella term encompassing several different disciplines and practices, and the priorities often depend on who is driving the effort. Chief data officers, privacy officers, security officers, and leaders in risk management usually focus on the privacy, security, and regulations that drive data governance programs. Data scientists, marketers, devops leaders, and business analysts are more likely to focus on proactive data governance, including data catalogs, data integration, data quality, data lineage, customer data profiles, and master data management.

There’s much to unpack in all the terms, practices, and technologies, and some capabilities and objectives overlap. Shams Chauthani, CTO and senior vice president of engineering at Zilliant, agrees that because there are multiple objectives, collaboration between business stakeholders, IT, and data teams is key to successful programs.

“Data governance is often treated in a compartmentalized way, mostly as a compliance requirement managed by IT,” he says. “In today’s digital age, data is the biggest asset, and to treat data governance as a siloed effort run exclusively by IT is a disservice to the entire organization. For organizations to fully deliver the promise of smarter decision-making with data, the data governance process must be continuously improved with engagements from all stakeholders.”

For this article, I consulted with industry experts to identify what devops leaders and teams should know about data governance and how they can contribute to its goals.

Data governance is a major organizational change

Grant Fritchey, devops advocate at Redgate Software, suggests that engaging stakeholders is just the start. “Data governance is everyone’s responsibility,” he says. “You can’t simply assign a person or a department the job of data governance and expect success. Data governance has to become a part of what everyone in IT does as a part of their jobs.”

One way to achieve the required collaboration and define responsibilities is to connect data governance directly to employee workflows. John Milburn, CEO of Clear Skye, says, “Data governance is not so much a technical function as it is a people and processes one. That’s why it’s critical that your governance solution aligns with employees’ existing workflow, or else it’s bound for failure.”

Devops teams should look for opportunities to improve dataops, including automations to support data integration, cataloging, and quality. John Wills, field CTO at Alation, says, “The new generation of data governance is active, catalog-led governance–meaning it’s vital for all knowledge workers including devops and dataops teams and is part of their daily work.”

Privacy and data protection regulations

Immuta CEO Matthew Carroll shares one key reason for top-down buy-in for data governance programs. “We’re seeing a major shift in the data management and governance landscape as more consumers become aware of their privacy rights due in large part to new regulations. As a result, organizations are grappling with how best to protect data assets and adhere to privacy regulations while attempting to scale and derive value from their data faster, ultimately driving a need for more digitized data policies and automated cloud data access and security.”

Barr Moses, CEO and cofounder of Monte Carlo, agrees. “Data governance is more critical than ever before as companies ingest more data and greater data regulation, like GDPR and CCPA, rolls out. We can meet these compliance headwinds before they slow us down by making data more accessible, meaningful, compliant, and reliable.”

Leaders must trust the data for decision-making

Looking one step beyond compliance considerations, the next level of importance that drives data governance efforts is trust that data is accurate, timely, and meets other data quality requirements.

Moses has several recommendations for tech teams. She says, “Teams must have visibility into critical tables and reports and treat data integrity like a first-class citizen. True data governance needs to go beyond defining and mapping the data to truly comprehending its use. An approach that prioritizes observability into the data can provide collective significance around specific analytics use cases and allow teams to prioritize what data matters most to the business.”

Kirk Haslbeck, vice president of data quality at Collibra, shares several best practices that improve overall trust in the data. He says, “Trusted data starts with data observability, using metadata for context and proactively monitoring data quality issues. While data quality and observability establish that your data is fit to use, data governance ensures its use is streamlined, secure, and compliant. Both data governance and data quality need to work together to create value from data.”

Data as competitive differentiation

Once there’s a baseline trust in the data, business leaders want to use data, analytics, and machine learning to transform the business. Haslbeck continues, “Every business is looking for data for a competitive edge, and data governance and data quality should be a priority.”

How does data governance create a competitive edge? John Wheeler, senior advisor for risk and technology at AuditBoard, explains, “Data governance is now a strategic priority for organizations looking for new digital products and services for growth. As such, data governance requires strong leadership from chief data officers or chief digital officers who understand the need for data consistency, quality, transparency, and accuracy.”

One area of differentiation is happening in open data models for managing access in business-to-business partnership models. For example, open banking and the financial-grade API aims to secure the exchange of data and banking systems to an ecosystem of developers, fintech vendors, and partners.

Brook Lovatt, chief product officer at Cloudentity, shared specifics of open data models with me. He says, “There are more opportunities to evolve and innovate in open data governance models than in closed ones, but open models also require a new set of security and compliance considerations. These open data specifications provide patterns and protocols that determine how systems communicate with each other, allowing data to flow between apps, services, platforms, and providers.”

Devops practices that support data governance

Data privacy, security, quality, and reliability are all reasons why data governance is important to data-driven organizations. Here are several recommendations for devops leaders and teams.

• Wills says devops teams should take an active role in configuring and updating data catalogs. “The data catalog is an enterprise system of reference containing a fabric of related knowledge, including technical assets such as tables, columns, queries, and models, and nontechnical ones such as glossaries and metrics. The result is a rich inventory of high-quality assets and contextual knowledge that can be trusted and that drives productivity through powerful search, reuse, collaboration, and crowdsourcing.”
• Steve Jones, devops advocate at Redgate Software, recommends instituting data governance as “part of regular work to ensure that as changes are made to schemas, new data is collected, and the data is classified and protected appropriately.”
• Milburn recommends “integrating your governance solution with your IT service management platform as a smart way for enterprises to protect their data while supporting workers with a familiar user experience.”
• Eldad Chai, CEO and cofounder of Satori Cyber, says that data security operations should be part of the data governance plan. “As big data evolves and multicloud environments gain traction, high-growth companies need a nondisruptive, compliant way to manage their tech stacks and the sensitive data stored within them. A modern data governance strategy such as data security operations (datasecops) empowers companies to reclaim control, reduce risk, maintain compliance, and provide secure access to data to drive better business decisions—automatically and easily.”

Data governance, along with continuous testing and shift-left security practices, are key disciplines devops teams must build into the fabric of their application architectures and development processes. Bolting on these practices as a secondary concern can lead to business risks, technical debt, and missed innovation opportunities.