Data platforms for semiconductor manufacturing
Fabs produce more data than almost any other industry, most of it sensor telemetry that only matters if it reaches the right model or engineer in time. I spent six years running data platforms for a semiconductor manufacturer.
What is different about fab data
Volume is the obvious difference, but it is not the hard part. The hard part is that volume keeps growing while the questions stay urgent. A query that ran in seconds on one fab's telemetry takes minutes once the data from several fabs lands in the same tables, and a pipeline that was cheap last year becomes the biggest line in the cloud bill this year.
Manufacturing data also crosses boundaries that other industries keep apart. One analysis might join tool telemetry, yield, test results, and finance data from an ERP system, each owned by a different team with its own idea of what a lot or a wafer means.
Where I help
- Pipelines that stay predictable. Spark partitioning and tuning so query performance and pipeline cost stay predictable as volumes grow, instead of being rediscovered every quarter.
- Data models over telemetry. Models over high-volume manufacturing and sensor data that analysts and engineers can query without knowing how it was partitioned.
- ML that reaches production. Airflow and PySpark workflows feeding high-volume training jobs, with CI/CD so a model change ships like any other code.
- Finance and ERP data. Financial systems migrations, including ETL between SAP and Snowflake, run across onshore and offshore teams.
Related
- Case study
99.8%Keeping ML training pipelines at 99.8% uptime
A semiconductor manufacturer's SaaS MLOps venture needed pipelines that customers could count on. They ran at 99.8% uptime while volumes grew across fabs.
Read it - Service
Build
You know what you need and want it built properly. Designed, built with your engineers, documented, and running in your accounts.
Read it - Service
Diagnostic
A number moved and nobody can say why, or a big vendor contract is on the table. Two weeks, a written memo, and a fix-first order.
Read it
Working in semiconductor?
Bring the number that moved or the bill that grew. You will leave the call with a first read and a straight answer on whether I can help.