Add Ohio State University mirror site (port 40016) + deterministic task verifiers#63
Open
StephenQSstarThomas wants to merge 2 commits into
Open
Add Ohio State University mirror site (port 40016) + deterministic task verifiers#63StephenQSstarThomas wants to merge 2 commits into
StephenQSstarThomas wants to merge 2 commits into
Conversation
Integrate the osu mirror from PR #12 onto the current main as an additive 17th site instead of the PR's stale base (which replaced merriam_webster and reused port 40015). Registered in all three places (websyn_start.sh, control_server.py, Dockerfile EXPOSE 40000-40016); the readiness gate in websyn_start.sh is now derived from ${#SITES[@]} so it no longer hardcodes the site count. osu ships no frozen DB/images in git or HF: its data is generated from seed_data.py, so a Dockerfile build step creates instance_seed/osu.db so the boot reset and /reset/osu have a seed to restore from. Verified: image builds, osu serves on 40016, and /reset byte-identity holds.
Reviewer role for PR #12 (previously unreviewed). One verify_N.py per task over a shared verify_lib.py, mirroring sites/merriam_webster/verify: deterministic-first (navigation anti-shortcut + hardcoded on-site ground truth) with an anchored llm_screenshot_shows confirmation that is skipped under --no_llm. tasks.jsonl gains verifier_path + judge_rubric per row (ground truth stays out of the agent-facing file). selfcheck.py proves all 20 verifiers PASS a correct trajectory and FAIL a wrong-answer / no-navigation one (20/20). TASK_REVIEW.md records feasibility: all 20 tasks are on-site answerable; 6 are flagged as answerable-from-prior-knowledge (mitigated by the nav check) and task 1 has a 36-vs-27 varsity-sports inconsistency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Ohio State University mirror site (port 40016) + deterministic task verifiers
Picks up the unreviewed #12
(Ohio State mirror by @richard-peng-xia), rebases it cleanly onto current
main,and adds the missing grading contract (the reviewer-role deliverable): a
deterministic verifier per task +
judge_rubric, plus a task-reasonableness review.Why not just merge #12
#12 was cut from a stale base: its
websyn_start.shwould have droppedbookingand replaced
merriam_webster, and it reused port 40015, whichmainnow assignsto
merriam_webster. This branch instead adds osu as an additive 17th site on port40016, so it merges onto the current
mainwith no collision or regression.What's in it
1. The osu site, integrated (commit 1)
sites/osu/— Flask app,seed_data.py, 24 templates (16 colleges, 16 depts,20 programs, 15 research centers, 15 faculty, 27 athletic teams, 24 news, 16 events).
websyn_start.sh(SITES +${#SITES[@]}-derivedreadiness gate, no more hardcoded count),
control_server.py(SITES),Dockerfile(
EXPOSE 40000-40016).seed_data.py, so aDockerfilebuild step createsinstance_seed/osu.dbso theboot reset and
/reset/osuhave a seed to restore from.2. The grading contract (commit 2) — mirrors
sites/merriam_webster/verify/sites/osu/verify/verify_lib.py+verify_0.py … verify_19.py— one per task,deterministic-first (navigation anti-shortcut + hardcoded on-site ground truth),
with an anchored
llm_screenshot_showsconfirmation that is skipped under--no_llm.sites/osu/tasks.jsonl— each row gainsverifier_path+judge_rubric(groundtruth stays out of this agent-facing file).
sites/osu/verify/selfcheck.py— deterministic self-test.sites/osu/verify/{README.md,TASK_REVIEW.md}.Verification performed
selfcheck.py: 20/20 — every verifier PASSes a correct trajectory, FAILs awrong answer, and FAILs a correct answer with no page visit (nav anti-shortcut).
docker buildsucceeds; the osu build-time seed step runs; osu serves on 40016inside the image;
/resetbyte-identity holds (md5 match).py_compileclean;websyn_start.shbash -nclean; the three registration pointsare consistent (osu @ index 16 → 40016, EXPOSE covers it).
Task review (see
sites/osu/verify/TASK_REVIEW.md)All 20 tasks are on-site answerable, stable, and non-HITL. Flagged (kept as-is per the
reviewer/contributor split, mitigated by the nav check): 6 answerable-from-prior-
knowledge tasks (Big Ten, 1870, Ohio Stadium, Ryan Day, "name 3 colleges", JD), and
task 1's internal inconsistency (site states 36 varsity sports but only 27
teams are seeded — verifier anchors to the stated 36). Minor site notes: unused
_health.py;news_articlecommits on GET.Closes #12 (supersedes it against current
main).