Cataloging 70,000 datasets across two data platforms

The situation

A game publisher with several studios ran its data on two platforms, Snowflake and Databricks. Each studio had built its own pipelines, naming conventions, and habits over years. Nobody could say how many datasets existed, which ones were current, or who to ask about any of them.

An estate like that has predictable costs. Analysts rebuild tables that already exist because they cannot find the original. Engineers are wary of retiring anything, so storage keeps growing. Privacy requests under GDPR and CCPA mean finding every copy of a player's data, which is hard without a list of where data lives. And any plan to put AI on top of the data needs metadata the AI can read.

What I did

Ran a proof of concept instead of a vendor beauty contest. We compared three metadata platforms side by side, on our own estate, and I made the recommendation from what each one did with our metadata rather than from its demo.

Chose for openness. The recommendation went to the platform with an open metadata model and a strong API, so the catalog would not tie the company to either warehouse vendor and could feed lineage, access policy, and AI tooling later.

Covered both platforms from the start. The catalog ingested from Snowflake and Databricks alike, so a dataset's owner, description, and lineage lived in one place whichever platform held it.

The result

More than 70,000 datasets were cataloged across both platforms, and the rollout finished ahead of schedule. For the first time, an analyst in one studio could find and trust a table built by another, and privacy work could start from a list.

What carries over to your team

• Write the evaluation criteria before the first demo. It turns a subjective choice into a scored one, and it gives leadership something to sign off on.
• Test on your own metadata. A catalog that looks clean on a vendor's sample can struggle with your naming conventions and volume.
• Automate ingestion first and ask people to curate second. Documentation projects that start with a blank page stall.
• Pick a catalog that outlives your warehouse choice. The metadata is the asset, and you will want it when you add lineage, access policy, or AI.

This work comes from an in-house leadership role, not a Millwright Data client engagement. The company is not named. The figures are ones I can walk you through on a call.