AI Projects Live in Data Preparation.

If you have built an AI or machine learning project before, you have probably heard some version of this stat: roughly 80% of the total effort goes into collecting, cleaning, merging, labelling, and otherwise wrangling data. Only the remaining 20% goes to the part everyone gets excited about — modelling.

It sounds like an exaggeration until you have lived through it. Then it sounds optimistic!

Phase 2 was about understanding what data you have and whether it can support your goals. We are now moving into Phase 3 where you actually roll up your sleeves and get that data into shape. 

Scope boundaries are reconfirmed and properly governed going forward, and cost and timeline forecasts are rebuilt from the ground up—based on evidence, not optimism

Two Pipelines, Not One.

A mistake teams often make is treating data preparation as a single, one-time task — something you do before training and then forget about. In practice, you need to design and build two pipelines:

Training Data Pipeline.

Gathers, cleans, formats, and labels historical data to train (and retrain) your model.

Inference Data Pipeline.

Handles the ongoing, live flow of new and production data the model will actually see once deployed.

These two pipelines need to treat data consistently. If your training data was cleaned and formatted one way, but production data flowing into the model at inference time is handled differently, you get training/serving skew — a quiet, hard-to-diagnose source of degraded model performance. We will dig into pipeline design in our following posts.

The Questions You Have to Ask About Every Data Source.

Before you can clean or merge anything, you need a clear picture of where your data is actually coming from:

> Is it sitting in a database, coming through an API, or arriving as a live stream?

> Is it internal to your organization, or sourced from a third party?

> Who owns it — and who's accountable if it changes, breaks, or disappears?

Each of these answers changes how you build your pipeline. A nightly database extract behaves very differently from a streaming event feed. Internal data you control is a very different risk profile from a third-party dataset you're licensing and hoping stays consistent.

What Preparation Involves.

Once you know your sources, the real work starts.

Data merging.

Combining sources that were never designed to talk to each other.

Data cleaning.

Removing duplicates, fixing malformed records, handling missing values.

Standardising entries.

Making sure “NY,” “New York,” and “N.Y.” are treated as the same thing.

Data enhancement.

Enriching records with additional context or derived features.

Handling corrupt data, incorrect data, and outliers.

Deciding what to fix, what to flag, and what to throw out.

None of this is glamorous.

All of it is foundational.

Structured vs. Unstructured Data: Different Rules Apply.

One distinction is worth calling out early, because it shapes almost every decision later in this series: not all data is prepared the same way, and the split between structured and unstructured data is the biggest reason why.

Structured data.

Lives in rows and columns with a defined schema — database tables, CRM records, transaction logs, spreadsheets. It's relatively easy to validate (a field either matches its expected type and range or it doesn't) and relatively easy to store and query at scale.

Unstructured data.

Free text, images, audio, video, PDFs, scanned documents — has no predefined schema. It typically makes up the majority of the data most organizations actually hold, and it requires fundamentally different tooling: NLP pipelines for text, computer vision preprocessing for images, speech-to-text for audio, before it can even be assessed for quality.

Most real projects deal with both, often mixed — a structured customer record with an unstructured support-ticket transcript attached to it. Treating both with the same cleaning checklist, the same pipeline architecture, or the same labelling approach is one of the more common planning mistakes in Phase 3. We'll flag where the two diverge throughout the rest of this series.

It's Not a Solo Job.

Good data preparation is rarely the work of one role. It typically draws on:

Data Engineers.

Who build and maintain the pipelines.

Data Analysts.

Who understand the shape and meaning of the data.

Domain Experts.

Who who catch the errors and edge cases that only make sense in context.

Skip any of these, and you risk building a technically clean pipeline that's still feeding your model data that doesn't reflect reality.

Before Moving to the Next Phase.

CPMAI frames Phase 3 around six questions you should be able to answer confidently before advancing to modelling. Below is a starting answer for each, plus concrete actions you can turn into an SOP for your team. 

1. How should data be cleaned and prepared to meet project requirements?

Clean to the standard your model actually needs — defined in Phase 2 — not to an abstract standard of “perfectly clean.”

Actions to take:

> Before cleaning begins, document acceptable thresholds per field — max missingness %, duplication tolerance, valid ranges — tied back to project requirements.

> Build a data quality checklist the team signs off on before data moves to training.

> Assign a named owner for each cleaning rule, so decisions are not anonymous or undocumented.

> Log every transformation applied to the data so cleaning is auditable and reversible.

2. How can we create repeatable steps for data engineering?

Treat your pipeline as code — version-controlled, tested, and orchestrated — not as a sequence of manual steps someone remembers.

Actions to take:

> Move all cleaning and transformation logic into version-controlled scripts; retire one-off notebook edits.

> Adopt an orchestration tool (Airflow, Dagster, Prefect, or a cloud-native equivalent) to schedule and sequence pipeline steps.

> Write automated tests for each pipeline stage: schema checks, row-count checks, null-rate checks.

> Maintain a runbook documenting what each pipeline step does and why, so it is not knowledge and IP in someone’s head.

> Establish a change-management process — code review and a staging environment — for any update to pipeline logic.

3. How can we continuously monitor and evaluate data quality?

Build monitoring into production from day one; do not treat quality checks as something that only happens before launch.

Actions to take:

> Stand up a data quality dashboard tracking completeness, uniqueness, and freshness over time.

> Set automated alert thresholds for schema violations and statistical drift (tools like Great Expectations, Deequ, Monte Carlo, or Soda can help).

> Schedule recurring manual audits by domain experts — monthly or quarterly, depending on how fast your data changes.

> Write an escalation protocol: who gets notified when a quality metric drops, and what they check first.

> Feed production model performance metrics back into a regular data-quality review, so a drop in accuracy automatically triggers a data investigation. 

4. How can we effectively use or modify third-party data?

Treat it as a hybrid of vendor management and data engineering — verify it legally and technically before it enters your pipeline.

Actions to take:

> Require legal review of licensing and usage rights before ingestion begins, including commercial-use terms.

> Run an independent quality validation pass on third-party data — do not rely solely on the vendor's claims.

> Document provenance and collection methodology for every third-party source, so bias or skew can be traced later.

> Build a normalisation layer that maps external schemas into your internal schema, rather than special-casing the source. 

> Maintain a contingency plan or alternate source for any third-party data your pipeline critically depends on. 

5. When and how should humans be involved with data labelling?

Reserve human effort for ambiguous or expert-judgment cases, and formalise the process so it is consistent and repeatable.

Actions to take:

> Write labelling guidelines with worked examples of edge cases before labelling starts.

> Route the most uncertain or ambiguous cases to human (ideally domain-expert) reviewers using active learning, rather than labelling everything manually.

> Track inter-annotator agreement and revisit guidelines whenever agreement is low.

> Set a recurring cadence for reviewing and re-labelling new edge cases that appear after launch.

> Keep a labelling changelog so guideline updates — and their effect on the dataset — stay traceable.

6. What additional steps can we take to augment data?

Use augmentation deliberately, aimed at specific gaps — not as a blanket substitute for real data.

Actions to take:

> Identify specific underrepresented classes or scenarios before choosing an augmentation technique.

> Match the technique to the data type — back-translation or paraphrasing for text, SMOTE or synthetic generation for tabular data, transformations or GANs for images.

> Have domain experts spot-check augmented data for unrealistic artifacts before it enters training.

> Track the ratio of augmented to real data in your training set, so the model does not overfit to synthetic patterns.

Upcoming blogs will go deeper on each of these. 

> Building pipelines that are actually repeatable — not just scripts that worked once. 

> Cleaning, standardization, and how to keep monitoring data quality after launch.

> Tackle the human side: labelling decisions, working with third-party data, and augmentation strategies for when you simply do not have enough data yet.

Do not treat data preparation as a chore to rush through on the way to the “real” work. For most AI projects, it is the real work — and the questions above are your checklist for doing it properly and successfully.

This is the first post in a 4-part series walking through Phase 3 of CPMAI (Cognitive Project Management for AI): Data Preparation.