Structured vs. Unstructured Data

Here is a number that surprises a lot of people: only about 20% of enterprise data is structured. The other 80% — emails, PDFs, images, videos, call transcripts, social media posts, scanned contracts — is unstructured.

Yet when most people picture "data," they picture rows and columns in a spreadsheet or a database table.

That gap matters.

If you are only thinking about the structured 20%, you are missing most of the opportunity — and most of the challenge — that comes with building AI systems.

What Structured Data Looks Like.

Structured data is the data most of us grew up working with professionally: customer records in a database, transaction logs, spreadsheets, survey responses with predefined fields. It is organized into rows and columns, it is searchable with straightforward queries, and it has historically been the easiest kind of data to build models on.

Structured data is not going anywhere, and it is still foundational for plenty of AI use cases — predictive analytics, forecasting, recommendation systems built on clean transactional histories.

But it is also the smaller slice of what most organizations actually have sitting in their systems.

What Unstructured Data Looks Like.

Unstructured data is everything that does not fit neatly into a table. Think images and video, PDFs and scanned documents, emails, chat logs, audio recordings, and free-text fields in otherwise structured systems. It is messy - inconsistent in format, and historically has been much harder to work with — which is exactly why so much of it has sat untouched in archives, inboxes, and file shares for years.

That has changed. 

Advances in natural language processing (NLP) and computer vision mean unstructured data is no longer a dead end. It is now one of the richest sources of value in an organization, precisely because almost nobody has fully mined it yet.

Where Machine Learning Shines.

This is the part worth sitting with: the biggest opportunities for machine learning today are often in unstructured data.

> Pattern recognition models can read through years of customer support tickets and surface recurring issues. 

> NLP models can extract meaning from contracts, emails, or reviews at a scale no team of humans could match. 

> Computer vision models can process images or video for quality control, safety monitoring, or content moderation.

This is a different kind of value creation than traditional business intelligence, which was largely built on structured data. Unstructured data lets AI answer questions that were previously impossible to ask at scale — not because the data didn't exist, but because there was no practical way to process it.

Where The Real Gems Are Hidden.

There is a deeper reason unstructured data is so valuable, and it goes beyond "there is more of it." Structured data tends to capture the summary of what happened — a transaction total, a status field, a closed date. Unstructured data captures the substance — the actual conversation, negotiation, or interaction between the people and systems involved.

Think about what actually lives in an organization's unstructured data:

> Emails and negotiation threads — where the real back-and-forth of a deal happened: what was offered, what was pushed back on, what compromise was actually reached, and why. A structured CRM field might just say "deal closed," while the email chain holds the entire story of how and why it closed on those specific terms.

> Support tickets and call transcripts — where customers describe problems in their own words, often revealing issues, workarounds, or unmet needs that never make it into a structured ticket category or resolution code.

> Meeting notes and internal chat logs — where the actual reasoning behind a decision lives, as opposed to just the decision itself.

> Contracts and scanned correspondence — where the specific terms, exceptions, and side agreements between entities are recorded in full, rather than reduced to a handful of database fields.

This is the layer where the real interactions between people, teams, and organizations actually took place — the how and why, not just the what. Structured data is good at telling you the outcome. Unstructured data is where you find the context, the nuance, and often the exceptions that explain why the outcome looks the way it does.

This is precisely why NLP and pattern recognition have become so valuable: they let organizations surface these buried details at scale — recurring objections across thousands of sales negotiations, root causes hidden across years of support interactions, or informal agreements between entities that were never captured in a structured system at all. Data that used to require a human to read every document, one at a time, can now be mined for patterns and specifics that were previously invisible simply because nobody had the time to look.

What This Means For Your Pipeline.

Recognising the shift toward unstructured data is not just a conceptual exercise — it has real, practical implications for how you build:

> Storage and pipelines need to accommodate different formats, not just tabular exports.

> Labeling becomes a bigger part of the job, since unstructured data often needs to be annotated before a model can learn from it.

> Transformation steps — like converting text into embeddings, or extracting frames from video — become a core part of your data preparation work, not an afterthought.

> Skills and tooling shift too. Teams that are experts in SQL and dashboards may need new tools and new expertise to work with this kind of data effectively.

The Underestimated Element.

Transformation Is A Project, Not A Line Item.

Here is something that rarely gets enough airtime: turning unstructured (or legacy structured) data into something usable is often a project in its own right — and one that's consistently underestimated in scope and timeline.

It is easy to plan for "data transformation" as a line item. It is much harder to plan for what that actually involves once you are in it:

Understanding What Fields Are Supposed to Mean.

Inconsistent field names, repurposed columns, and undocumented conventions are the norm rather than the exception, especially in systems that have been extended and patched over many years. A field might technically still exist, but what it meant when it was created, and what it's actually been used for since, can be two very different things.

Legacy Systems and Disappearing Expertise.

Data pulled from mainframes — often as YAML or other legacy file formats — comes with a compounding problem: the people who understood what those fields actually meant, and how they map to a modern platform, are often close to retirement or already gone. Their institutional knowledge doesn't live in the data itself; it lives in their heads. Losing access to that expertise can turn a technical mapping exercise into an archaeological one.

Access & Permissions.

Who actually has access to the mainframe, and under what restrictions? This is frequently a separate hurdle from the technical mapping work — involving IT security, legacy system owners, and sometimes vendors who manage systems the organization no longer maintains in-house.

Business Relevance, Not Just Technical Mapping.

Not everything in a legacy system is worth carrying forward. Part of this work is a genuine business assessment: what data is still meaningful today, what is obsolete, and what needs to be reshaped to fit current definitions and use cases — rather than simply replicated as-is.

It Is Rarely Purely A Technical Exercise.

Getting this right typically requires input from business users who understand what the data is used for, support teams who know where the quirks and workarounds live, and sometimes the original vendors who built or maintain the legacy system. Treating this as an engineering-only task is one of the fastest ways to underestimate both the timeline and the risk.

If you have been through a transformation project, none of this is surprising. If you have not, it is worth building this reality into your project timeline from day one — because "data transformation" almost always takes longer, and requires more people than it looks like on paper.

Reframing “Unstructured”.

The word "unstructured" carries a bit of a stigma — it sounds like a problem to be solved rather than an asset to be used. A better way to think about it: unstructured does not mean unusable. It means underexploited.

If your organisation has years of documents, images, or communications that have never been systematically analysed, that is not clutter. That is a data asset sitting quietly, waiting for the right project to unlock it. 

As you move through your AI project's Data Understanding phase, it is worth taking real inventory of what unstructured data you have — not just what is easy to query, but everything that is actually available to you.

That inventory is often where the most interesting AI opportunities are hiding.