Checking interactive course pages

The problem, stated precisely

Rize courses link out to externally-hosted HTML: Khan-Academy-style articles with embedded comprehension checks, simulations, practice exams, interactive graphs. They are pedagogically worth it, and they rot in ways that nobody notices until a student is stuck.

The failure mode that matters is not the 404. It is the page that returns HTTP 200 and is quietly empty.

So the check has to open the page in a real browser and look at what rendered. That is the whole idea. Everything below is mechanics.

What to check, and how hard to complain

The severity split matters more than the check list. A board that goes red every week gets ignored, and then it is worth nothing. Third-party links rot constantly and are usually not the course's fault, so they warn. Only things that break the page for a student fail.

The idea that keeps this from becoming a system

Greg's condition was explicit: no bespoke solution that the team then has to maintain. The move that satisfies it is pushing the domain knowledge into the pages instead of the tool.

Each page exposes one function:

window.__selfCheck()  // → { ok: boolean, checks: [{ name, ok, detail? }] }

The page author writes it while building the page, because they are the only one who knows what "working" means for it. The checker calls it and fails the run on any ok: false. The checker itself never learns anything about calculus, or about any particular widget, so it never has to change. A new page inherits the check for free by starting from the template.

What a good self-check looks like

SelfCheck.register('slider changes the slope', () => {
  gap.value = '2';    update();
  const wide  = readSlope();
  gap.value = '0.05'; update();
  const tight = readSlope();
  const ok = Math.abs(wide - 4) < 0.01 && Math.abs(tight - 2.05) < 0.01;
  return ok || { ok: false, detail: `expected 4 and 2.05, got ${wide} and ${tight}` };
});

Check the things that fail silently, roughly one per interactive element: a visual mounted, an input changes the output, every comprehension check grades correctly, and data-driven content is present. Do not check prose or styling; a human notices those immediately. The detail string lands verbatim on the status board that instructors and managers read, so make it a diagnosis.

The procedure

1. One checker repo, not one per page

   Pages live in many small repos. Put the checker in a single repo that checks all of them. Adding a workflow to each page repo is fifty copies of one file to maintain, which is the thing you were asked not to build.

   gh repo create <org>/rize-interactive-health --public
   # copy in manifest.json, package.json, scripts/, template/, and
   # workflow/check.yml → .github/workflows/check.yml

   Then in repo settings: Pages source = main branch /docs folder, and Actions → Workflow permissions → Read and write, so the run can commit the status board back. Optionally add a SLACK_WEBHOOK secret and a HELPDESK_URL variable.

2. Build the manifest from Sanity, not from memory

   The list of what to check comes from what courses actually link to, so it cannot drift. You run this query through the Sanity MCP; CI never holds Sanity credentials.

   *[defined(richTextContent)]{
     _type, "c": course->code,
     "h": richTextContent[].markDefs[].href
   }{
     "k": _type + "~" + coalesce(c,"?") + "~" +
          array::join(h[string::startsWith(@, $prefix)], ";")
   }[k match "*github*"]{k}

   Use string::startsWith, never match, on these hrefs. GROQ's match returns true against the null hrefs inside markDefs and will silently select every document in the dataset. It looks like it worked. It did not.

   Save the rows as COURSE|URL lines and run python3 scripts/build_manifest.py --rows rows.txt. URLs that drop out of Sanity are marked retired rather than deleted, because an unwiring is sometimes a mistake and a silent deletion hides it.

   Keep the exact URLs courses link to, including deep links and query strings (/macro-interactives/u1.html, /formula-trainer/?unit=3). A deep link can 404 while the repo's home page is perfectly fine.

3. Schedule it around terms, not just weekly

   Weekly year-round, plus daily through December, January, July and August, so a break surfaces weeks before students arrive rather than during week one. That daily-in-the- run-up cron is the part that answers "ahead of upcoming semesters"; a plain weekly cron does not.

4. Break something on purpose before you believe it

   Point one manifest entry at a URL that does not exist, run the checker, confirm it goes red and names the page, then put it back. A check nobody has seen fail is not yet known to work. Do the same for a self-check: comment out the line that draws the graph and confirm the run fails with a useful message.

5. Hand over three things

   The status board URL for the instructor briefing folder, with one line: if a page here is red, or a page looks wrong while you are teaching, file a helpdesk ticket and say which page. The page template, to start every new page from. And one sentence for their manager: it runs weekly and before each term, it costs nothing, and nobody has to remember to run it.

   Instructors are the backstop for what the robot cannot see. They are not the primary detector, and a status indicator that is green because nobody has looked is worse than no indicator.

Standing advice when you build these pages

• Write the self-checks in the same session as the page. Part of building it, like alt text. Not an optional extra offered afterwards.
• Vendor every library; no CDN. A page with no external runtime dependency can only break when somebody edits it, and that is caught on the next push. Pinned is better than floating; vendored is better than pinned.
• No API keys. A key that expires on a schedule nobody tracks is the single most reliable way to have a dead page in week three.
• Put the pages repos in the Rize org, not a personal account. The eventual handoff to a product-team repo becomes a transfer rather than a rebuild.
• Do not tighten the warnings into failures. Tempting, and it is how the board becomes noise.

Where everything lives

The local source of truth on Samuel's machine is projects/coursedev/interactive-health/.