Keeping ML training pipelines at 99.8% uptime

The situation

Semiconductor fabs produce enormous volumes of manufacturing and sensor data, and small shifts in that data can mean yield problems worth a great deal of money. The manufacturer had turned its machine learning work into a SaaS MLOps venture, which raised the bar. Internal users tolerate a late pipeline. Paying customers do not.

The pipelines had to transform data in near real time, feed high-volume training jobs, and do it across two clouds, while data volumes kept growing as more fabs came online.

What I did

Built pipelines as production software. Workflows ran on Airflow and PySpark, with CI/CD in GitLab and work tracked in Jira. Every change went through review and automated tests before it reached production.

Managed infrastructure as code. Terraform defined the infrastructure across AWS and GCP, so environments matched and a rebuild was a command rather than a project.

Tuned for growth, not for today. Data models over manufacturing and sensor telemetry used Spark partitioning and tuning so that query performance and pipeline cost stayed predictable as volumes grew across fabs.

Changed how the team worked. I led a team of data engineers and analysts through roadmap planning, sprint delivery, and a move to a DevOps model, where the people who build a pipeline also run it. We worked with ML, finance, IT, and security to enforce data governance.

The result

The pipelines feeding ML training ran at 99.8% uptime. During those six years, I also ran a financial systems migration on SAP and Snowflake as Scrum Master across onshore and offshore teams, using a data vault architecture.

What carries over to your team

• If customers depend on a pipeline, give it the same engineering discipline as the product: review, tests, and CI/CD.
• Partition and tune for the volume you expect next year. Retrofitting a partition scheme on a large table is far more expensive than designing it.
• Infrastructure as code pays off most when you run more than one environment or cloud.
• Teams that run what they build fix root causes faster, because they are the ones paged.

This work comes from an in-house leadership role, not a Millwright Data client engagement. The company is not named. The figures are ones I can walk you through on a call.