Watching NHS Jobs So My Sister Doesn't Have To

My sister was job-hunting on the NHS Scotland jobs board, specifically Medical & Dental vacancies. New postings can close fast, so the only reliable way to not miss one was to keep refreshing the page — over and over, all day, hoping to catch something the moment it appeared.

There's no API, but there's something better

First stop was checking whether NHS Scotland exposed anything I could just call. They don't — no public jobs API. The obvious fallback was a headless browser: point Playwright at the page, let it render, and scrape the DOM.

Before reaching for that, I opened the network tab to see how the page actually loaded its results. Turned out the site's own internal endpoint — the one its Load more button calls — returns the job list as fully-rendered HTML, details and all, no client-side rendering to fake. That meant I didn't need a browser at all: a plain HTTP request plus Cheerio to parse the HTML was enough to get the same structured data Playwright would have, for a fraction of the memory and cold-start cost.

V1: a Next.js site that crawled itself

The first version was a single Next.js app, hosted on Vercel's free Hobby plan. It crawled the NHS page on an interval, parsed the jobs, and rendered them with the filters she actually asked for — split into old jobs and new jobs, plus a quick overview of each posting — rather than a generic dashboard I guessed she'd want.

To tell old from new without a database, the browser did the remembering: previously-seen jobs were saved to localStorage, and a button in the UI let her mark the current batch as "seen" so the next crawl could diff against it and flag anything new.

The catch with V1

It worked, but it only worked while something was actively looking at it:

• A device had to stay on the page for the crawl loop and the localStorage diff to run at all.
• Browser notifications only fire while the tab is capable of running — lock the phone or switch apps and NHS could post ten jobs and nothing would happen.
• Even with notification permission granted, there was no getting around it: some device, somewhere, had to be sitting on that tab, awake.

V2: give it a real backend

So I split the crawler out of the website entirely. It became its own Express.js API — also hosted free on Vercel — whose only job is to crawl NHS, parse jobs with Cheerio, and store them in MongoDB: the current jobs, plus a log of every crawl attempt (counts, timing, failures).

Something still had to trigger the crawl every minute. Vercel's free plan won't run a background loop for you, so I pointed cron-job.org's free scheduler at the crawl endpoint instead — it calls the API on the minute, forever, without needing a server I'd have to pay to keep alive.

For notifications, I skipped Firebase Cloud Messaging — which still would have needed a device registered and reachable — and wired up a Telegram bot instead. The moment the crawl finds jobs that aren't in Mongo yet, it messages her directly. No app to keep open, no permission prompts, no device that has to be awake — just a message that shows up.

Where it landed

She gets a Telegram message within a minute of a new Medical & Dental vacancy going live on NHS Scotland. No tab, no phone screen, no manual refreshing. And the whole thing — crawler API, database, cron trigger, bot — still runs on free tiers: Vercel Hobby, MongoDB's free tier, cron-job.org, and Telegram.

What I'd take from this

• Before reaching for a headless browser, check what the page's own "load more" actually calls — a lot of sites without a public API still serve structured HTML under the hood.
• If "always watching" is a requirement, get the clock off the client on day one. A tab is not a scheduler.
• For a single recipient, a webhook-style bot (Telegram) beats a push-notification service — no registration flow, no permissions, no client SDK to keep updated.