TLDR: A shift left approach to data management moves data quality and governance responsibilities closer to enterprise data sources. It ensures consistency and reliability in the data quality available for analytics, introducing cost efficiency and enhancing the quality of insights. AI can be used to accelerate shift-left adoption. Read on to understand how shift left benefits data management with a real-world use case.
Poor data quality drives up costs and slows down decision-making. When checks happen too late, often right before data is used, issues are harder to catch and more expensive to fix.
A shift-left approach brings data quality checks earlier, embedding them where data is created and transformed. This ensures data is reliable from the start and fit for use across the organisation.
This article examines how shifting left brings data storage and cost efficiencies, while improving both the speed and accuracy of analytics. We also examine the role AI agents can play in making shift-left more feasible.
What Is Shift Left in Data Management?
The shift-left paradigm represents a transformative approach to data management, bringing data quality and governance operations closer to their point of origin. It has its roots in software development principles, where it signifies a move towards testing earlier in the software development cycle. Bringing the paradigm to data means shifting data quality and controls to the earliest stages of the data lifecycle.
Any data management value chain contains three key components:
Producers or software components generating the data
Platforms or data storage and processing infrastructure
Consumers or the tools and teams utilising the data to accomplish tasks
A common data management pattern in enterprise environments is to have a data warehouse that sources data directly from producers, providing some level of basic curation on the data (such as deduplication and conformation), and then presents the data to the consumer in a format of their choice. The data consumer raises issues when a data inconsistency is detected, and the data warehouse team identifies the location of the issue and takes appropriate action to resolve it. This approach is reactive and delays data decision-making. In the worst case, decision-making is based on erroneous data.
In the shift-left paradigm, producers are responsible for governance and data controls, allowing consumers to prioritise data utilisation. Data contracts are used to set expectations around data quality, types, and formats that all consumers can consistently expect from producers.
The Business Benefits of Shifting Left with Data
V2 supported a large retail customer in adopting shift-left ways of working. Our team sat down with key business stakeholders to understand the data lineage and source systems of the data they were ingesting and reporting on. Based on this analysis, we established relevant shift left practices, described in detail later in the article.
After completing the shift left process, the reporting data quality increased by 30%. Our client was also able to decrease their time to insight by leveraging intra-day updates to the data. Time savings of 15 hrs each week were reported due to automation.
Overall, we are observing the following business benefits.
Early Detection of Data Anomalies
Identify and Resolve at Speed
Shift left enables the identification and resolution of data quality issues early in the processing stage. Automation can be used to notify producers when issues occur, and they can have a relevant fix in place within a timely manner to minimise further impact. This means organisations do not have to worry about inconsistent or erroneous data affecting key decision-making processes further downstream.
Increased Data Processing Accuracy
Handle More Data with Confidence
Increased data visibility, as it arrives in its raw form, allows for enhanced unit testing as well as data testing. You can quickly ensure that the expected output from each transformation and pipeline is met.
Enhanced Decision Making
Access Improved Insights Sooner
Data is sourced, tested, and processed in the earliest stages. This enables a trusted single source of truth to be readily available, allowing for near real-time decision-making. Shift left also enables the ability to retrieve and process data as it is produced. Organisations can leverage near real-time data streaming capabilities and decrease time to insights.
Heightened Governance
Control Data Usage With Confidence
Being close to the source of the data allows organisations to govern and maintain their data in the earliest stages, setting the rest of the organisation up for success.
How Organisations Shift Left in Data
V2 helped our client shift left by adopting key paradigms, such as data contracts, data quality testing, and autonomous alerts, as close to the source as possible. Here are some key aspects to consider if you want to start shifting left.
Understanding Data Lineage
It is paramount to understand how your organisation is collating all its data and where it is being produced. This can be from internal or external APIs, applications, IoT, etc. Data lineage tracking is crucial for identifying the origin of each data point used for decision-making. It provides the foundational understanding necessary for an organisation to work in a shift-left manner.
Development of Data Contracts
A data contract is a formal, early-stage agreement between data producers and data consumers. It serves as a binding specification for:
Management guidelines
Data format, structure, semantics, and usage
Quality standards
Quality tests
Update frequency
Service expectations
The data contract guarantees that data quality receives attention from the point of generation. It holds producers accountable for the reliability, observability, and integrity of the data they create, allowing consumers to confidently ensure the data they receive is accurate. The contract also supports non-breaking changes, inspiring trust between the data source and the downstream dependencies.
When it comes to implementing a data contract, there is usually a mutual conversation and agreement between what the producer can provide and how this aligns with solving the consumer's requirements. Several stakeholders meet and discuss their data expectations, service level agreements, schema, and management. Once an agreement is met, both parties sign off and adhere to the contract.
It is important to note that, although the terminology of a contract is used, it is not a legally binding document. If a producer needs to make any changes or updates to the contract, a certain amount of time is required to notify all consumers of the changes so they can plan accordingly.
Data quality testing
The next component is testing that the data quality produced and consumed adheres to the data contract. Producers should conduct tests to catch anomalies early and prevent erroneous records from influencing key business decisions. It is the producer’s responsibility to ensure that the data adheres to the schema defined within the data contract and that the correct values are populated.
However, the consumer should also check that the values provided within each column are of the expected types. This can lead to various types of quality verification tests to run on top of the raw data. If a test fails, a subsequent action should be taken at the consumer level to notify the producers of the issue for further investigation and resolution.
Data monitoring and observability
Data monitoring is proactive and centralised rather than ad hoc by different consumers. Automated systems validate data against contracts and raise proactive alerts. However, excessive alerts can cause fatigue and lead to the main challenge being overlooked. That is why, in most cases, data monitoring and observability can become quite reactive and require a high level of manual analysis if not done correctly.
Within a shift-left ideation, monitoring and observability are achieved by leveraging automated processes to identify issues as close to their source as possible. Whether this be a data contract violation or a problem with source data ingestion, alerts are automatically sent out to the relevant parties or communication channels for visibility. This removes the reactive approach of responding to issues as they arise.
Using AI for shift left acceleration
AI is a key shift left enabler and can automate data quality checks against the data contract throughout the data pipeline. Autonomous AI agents can analyse massive datasets, identify complex patterns and anomalies, and predict potential issues before they propagate. They can also work together to be proactive in resolving or supporting the retrospective resolution of issues. AI can provide recommendations, understand root causes, and simplify data management tasks for data custodians and stewards.
Shift left, powered by AI and grounded in data contracts, transforms data quality from reactive firefighting to proactive assurance.
Stage | Traditional Workflow | Shift Left Workflow with AI |
Detection | Reactive, often by downstream consumers | Proactive monitoring and automated alerts |
Investigation | Manual context building and analysis | Automated context building and AI-assisted investigation |
Root cause analysis | Manual analysis and guesswork | AI-powered root cause analysis |
Resolution | Manual implementation of fixes | Automated fixes with human oversight |
Resolution time | Low velocity and potentially delayed resolution | Faster and more efficient incident resolution |
Cost of Resolution | Higher due to delays and manual effort | Lower due to automation and early detection |
Shifting left with AI agents — what it can look like
Data management includes data classification, data quality, and incident management. AI Agents can manage many tasks within this space, with human intervention only required for output validation. For example, an organisation may have the following AI agents:
Incident management agent
This agent is responsible for reviewing and responding to data-related incidents reported through internal processes. This could be bugs or issues. It accesses internal communication channels and project management tools to raise alerts and support triage and response.
Data quality agent
This agent reviews and enforces the data quality rules described in data contracts. It works closely with the incident management agent to provide relevant context when an issue is raised.
Data classification agent
This agent is responsible for ensuring that all data adheres to regulations and complies with the organisation's standards. Its main functions include:
Reviewing the sample data provided
Classifying data points according to privacy and compliance categories.
It can access the organisation's data classification rules, data compliance, and regulations, as well as internal communication channels. Again, this agent works closely with the data quality agent and the incident management agent to provide relevant details for internal communication and triage.
Final words
The implementation of intelligent shift-left practices forms the essential foundation for establishing data trust in your organisation. Organisations can transition from a reactive to a proactive data management approach by combining well-defined data contracts with intelligent AI agents. Shift left establishes trust during the early stages of the data lifecycle, enabling improved data quality and confident decision-making.
Contact V2 to learn more about how we can help your organisation shift left your data management.
