Jobs Index

Challenge: every ATS and job platform has its own API, format, rate limits and failure modes.

Solution: one small, pluggable collector per source. Each one runs in parallel with its own timeout and worker limit and maps the result to one shared schema.

Challenge: companies embed their job boards on career pages in many different ways.

Solution: discovery tools that inspect career pages and detect the board behind them. A board is only added after its answers are verified, and conflicts go to manual review instead of overwriting data.

Challenge: one posting shows up on several sources, gets reposted, or comes back from a new source as if it were new.

Solution: a stable identity and a content fingerprint for every job. That separates a really new job from a new source, and a real closing from a gap in one day's run.

Challenge: a list of today's jobs says nothing about which are new, which changed or how long they stay open.

Solution: immutable daily snapshots turned into a full history (SCD Type 2). That gives first-seen and closing dates, plus a Kaplan-Meier estimate of how long jobs stay open that still counts the ones that are open.

Challenge: a source can fail quietly and a run can come back half empty.

Solution: each run is compared with the previous ones: failed or empty sources, volume drops, gaps, slow runs. A run that collapses never replaces the published site, and errors fail the workflow.

Challenge: develop cheaply on a laptop and still run on a real cloud platform.

Solution: one set of dbt models that builds the lakehouse on DuckDB locally and on Databricks (Delta Lake, Auto Loader, incremental MERGE). Both were checked to give identical results.

Challenge: the daily build runs on a free CI machine with about 7 GB of memory.

Solution: streaming and batching instead of loading everything at once cut peak memory from 5.05 GB to 2.81 GB.

Challenge: salaries in many currencies and periods, locations like “Remote (LATAM)”, one skill written ten ways.

Solution: parsers and a taxonomy normalize each field and record how confident they are. Values that can't be read stay unknown instead of being guessed.