Topics: web compliance, testing, public infrastructure, evaluation, semantic web
Technologies: python (headless browsers, e.g. Playwright), javascript, RDF / linked data (optional)
Tldr; take a single civic standard, build it into a rigorous automated test suite that scans real public-sector websites, and measure how often it gets the answer right.
Motivation: Public-serving organisations will be expected to meet standards for privacy, accessibility, provenance, and transparency when they publish online, yet little exists that checks automatically and at scale whether a given website actually does. An existing prototype (the dotPublic compliance checker) runs a catalogue of checks across nine categories of civic standard, combining deterministic pattern matching, behavioural observation of what a page does when visited, and AI-assisted judgement for questions pattern matching cannot answer. What it lacks is a rigorously validated single-standard suite with known error rates. This project takes one standard, for example Privacy or Transparency, and builds it out properly: a full set of checks, run against a corpus of real public-sector sites, with results compared against a hand-labelled ground truth so that false positives and false negatives can be measured rather than assumed.
Implementation: You will select one civic standard and decompose it into individual, testable checks, distinguishing those answerable by deterministic pattern matching (e.g. whether a privacy policy exists at a known location) from those requiring behavioural observation (e.g. whether third-party requests fire before any consent is given) from those needing AI-assisted judgement (e.g. whether a policy is written in plain language). You will assemble a corpus of public-sector websites, hand-label a sample against your checks to create a ground truth, run your suite across the corpus, and report precision and recall for each check. You will document where the checker is reliable, where it is not, and why. The deliverable is a working, validated single-standard suite and an evaluation of its accuracy..
What you will learn: (i) how civic and legal standards translate into concrete, testable checks; (ii) how to observe website behaviour programmatically using headless browsers; (iii) how to design a falsifiable evaluation with a labelled ground truth; (iv) how to reason about false positives and false negatives in a compliance setting; (v) where automated checking reaches its limits and human oversight is required.
References:
[1] dotPublic compliance checker prototype: https://dotpublic.org/checker
[2] Web Content Accessibility Guidelines (WCAG): https://www.w3.org/WAI/standards-guidelines/wcag/
[3] Data Privacy Vocabulary (DPV): https://w3id.org/dpv