The data problem that blocks every AI project

The conversation tends to go the same way. You ask whether the data is available. The answer is yes. You ask whether it is clean enough to build on. The answer is also yes. Six weeks later, an engineer has spent three of those weeks trying to reconcile customer records that were entered differently in two different systems by two different people over four years, and the AI project has not started yet.

Data optimism is almost universal and almost always wrong.

The people who know the business best are the people who most confidently believe the data is fine. This is not because they are careless. It is because they use the data operationally every day, they know how to navigate its quirks, they know which fields to ignore and which to trust, and they have developed workarounds for its gaps without consciously registering them as gaps. When you ask them whether the data is usable, they answer from their operational experience, not from the perspective of a system that will attempt to use that data programmatically, at scale, without any of the contextual judgment they bring to each manual interaction.

The engineer who looks at the data for the first time sees it differently. The date field that is sometimes formatted as DD/MM/YYYY and sometimes as MM-DD-YY and occasionally as "January 3rd" is not a minor annoyance — it is a blocker. The customer name field where some records have company names and some have contact names and some have both concatenated together is not a known quirk to be navigated — it is six hours of cleaning per thousand records. The product category field where the same category is spelled three different ways across four years of entries because nobody enforced a controlled vocabulary is not a historical artifact — it is a classification problem that has to be solved before the AI can use the field for anything.

Data optimism compounds when the data lives in multiple systems. "We have all the customer data in the CRM" is true. What is also true is that the invoicing system has a different customer table that was never linked to the CRM, the support ticket system uses email as the customer identifier rather than the CRM ID, and there is a spreadsheet that the ops team maintains manually because the CRM cannot handle the specific workflow they need. Joining these systems to produce a coherent view of the customer — the thing the AI project actually needs — is not a data engineering task. It is an archaeological project.

What data archaeology looks like in practice.

Data archaeology is the work of understanding what data you actually have as opposed to what you believe you have. It requires someone with technical ability to pull the data, inspect it systematically, and report honestly on what they find. It is not glamorous. It is frequently demoralising. And it is the work that almost no AI project budgets for upfront, which means it happens as an unplanned interruption to the build rather than as a planned prerequisite to it.

The archaeology typically reveals four categories of problem. The first is structural inconsistency: the same information stored in different formats or in different fields depending on when it was entered and by whom. The second is referential integrity failures: records that reference other records that no longer exist, or that reference the same entity by different identifiers in different systems. The third is missing data: fields that are required for the AI use case but were never collected, or that were collected inconsistently and are blank for a significant portion of records. The fourth is accuracy degradation: data that was accurate when it was entered and has since become false because the underlying reality changed and nobody updated the record.

Each of these categories requires a different remediation strategy. Structural inconsistency can often be resolved programmatically, though it takes time to map the variants and write the normalisation rules. Referential integrity failures require decisions about which system's version of the record to trust, which is a business decision, not an engineering one. Missing data either limits the scope of the AI feature or requires a collection effort to fill the gaps — a process that can take months. Accuracy degradation is the hardest: you often cannot distinguish a stale record from an accurate one without contacting the underlying entity, which at scale is not practical.

The assessment to run before you commit to a project.

The solution is to front-load the archaeology rather than discover it mid-build. Before committing to an AI project, run a data readiness assessment. This is a structured review of the data the project will depend on, designed to surface the categories of problem described above and produce an honest estimate of the remediation effort required.

The assessment has three components. The first is a schema review: what fields exist, what types they are, what the documented constraints are, and whether the actual data matches the documented schema. The second is a sampling review: pull a random sample of records — 200 is typically sufficient to reveal systemic patterns — and manually inspect them for completeness, consistency, and plausible accuracy. The third is a provenance review: where does each piece of data come from, who is responsible for its quality, when was it last audited, and what happens to it when the underlying reality changes.

The output of the assessment is not a clean-or-not verdict. It is an inventory of specific problems with specific remediation estimates. This inventory becomes the basis for an honest project scope that includes the data preparation work, not just the AI build work. Teams that run this assessment before committing save weeks of blocked engineering and the morale damage that comes from discovering mid-build that the foundation is not what was promised.

Data readiness is a continuous problem, not a one-time fix.

Passing the data readiness assessment before launch is necessary but not sufficient. Data quality degrades over time. New records are entered by new people who do not follow the same conventions as the people who set them up. Systems are updated and the data model changes without a migration plan for existing records. Third-party data sources change their schema and the integration breaks silently. A knowledge base document is accurate on launch day and outdated three weeks later when the policy it describes changes.

Production AI systems need a data quality monitoring process that runs continuously. For structured data, this means automated checks: missing value rates by field, distribution shifts that suggest a schema change, referential integrity failures, and records that fail validation rules. For knowledge bases and document stores, this means freshness tracking: when was each document last reviewed, who owns it, and what is the trigger that initiates a review when the underlying information might have changed.

This process does not run itself. It requires a named owner, a defined cadence, and organisational permission to pause an AI feature when its data quality drops below the threshold where its outputs are reliable. Most organisations have none of these things when they launch an AI project, and they discover the need for them when a user catches an error that was caused by stale or corrupt data and asks why the system was trusted to operate with data in that state.

• 01
  Run a data readiness assessment before you commit to build. Pull 200 random records from every data source the project will use. Inspect them manually for completeness, consistency, and accuracy. If more than 10% of records have systemic problems, you have a data preparation project before you have an AI project. Do this before the engineering estimate, not after.
• 02
  Include data preparation time in the project scope explicitly. Data cleaning and normalisation should appear as named tasks in the project plan with assigned owners and estimates. "Data is ready" is not an assumption that belongs in an engineering estimate — it is a dependency that needs to be verified and tracked.
• 03
  Assign a named data owner before you start. Someone needs to be accountable for the quality of every data source the AI system depends on. This person is involved from the assessment phase, owns the remediation plan, and is the escalation point when data quality issues are detected in production.
• 04
  Document data provenance for every source. For each piece of data the system uses: where does it come from, who is responsible for its accuracy, how often does it change, and what triggers a review when the underlying reality changes? Without this documentation, data quality problems are discovered when users report errors, not proactively.
• 05
  Build data quality monitoring before you launch, not after. Automated checks for missing values, schema violations, and distribution shifts should be running from day one. For knowledge bases, freshness tracking and review triggers should be in place before the first document goes stale.
• 06
  Give yourself permission to shrink the scope based on data reality. If the data is only good enough to support 60% of the originally planned feature, build the 60% that is well-supported rather than the 100% that looks impressive in a demo and fails in production. Scope is negotiable. Data quality is not.

Frequently asked questions.