Resilient Data Pipeline

For a source that changes without notice

Completed
Dec 2023

You depend on a system you do not control and it changes without telling you. The question is what happens to your integration the morning after.

A .NET background worker drives a real browser, parses semi-structured pages defensively, normalises them into a typed model, writes idempotently to MongoDB and rolls the results up into aggregate figures.

Evidence

What it looks like running

A real run against the live source: the worker drives Chrome through the league page, rejects the fixtures that have not been played, and writes the finished matches to MongoDB
Startup: sixteen leagues loaded from Data/leagues.json, then the league and season are chosen at the prompt rather than hardcoded
Thirty matches found on the league page, and the unplayed fixtures rejected from their header alone — four elements where a played match publishes six
The end of the run: eleven finished MLS matches written, none skipped as already present, and the worker exiting cleanly
The Express and Pug reader listing the matches stored in the us_mls_2026 collection, with the league-season selector above them
The Leagues page rendering the sixteen leagues the worker seeded into MongoDB from Data/leagues.json