SeeHere: audio descriptions for lecture videos, grounded in visual evidence

When a lecturer says “look at this chart”, blind and low-vision students hear nothing. SeeHere watches a recorded lecture, finds what is on screen but never said aloud, and writes short audio descriptions that a CP-SAT solver fits into natural pauses. Every line is linked to its evidence (keyframe, OCR, transcript) and checked, and uncertain claims are held for an instructor. Live on Vultr Cloud Compute with Vultr Serverless Inference.

Live demoGitHub0

Overview

The problem

Lecture videos rely on slides. Charts, equations, labelled diagrams and tables carry the content, while the lecturer says “look at this” instead of reading them aloud. For blind and low-vision students, that part of the lecture is missing. Captions don’t help, because the problem is the visuals that were never spoken.

What SeeHere does

SeeHere turns a recorded lecture into an audio-described lecture in about a minute:

  • Finds the gap. It detects slide changes, reads on-screen text with OCR and uses a vision model for charts and diagrams. It then compares everything with the transcript, skips visuals the lecturer already read aloud and prioritises the unspoken ones.
  • Resolves pointers. Phrases like “this line” or “over here” are linked to the region of the slide they refer to.
  • Never talks over the lecturer. An OR-Tools CP-SAT solver places short descriptions in natural pauses. Each description is voiced, its real clip length is measured, and the schedule is re-checked.
  • Shows its evidence. Every description is linked to its keyframe, OCR quotes and transcript, and gets a verification state. Claims the evidence can’t support are held for an instructor instead of being guessed.
  • Exports what institutions need: a WebVTT description track, a described MP4, an accessible transcript and a coverage and verification report.

Real result (live on Vultr)

On a 1:36 biology lecture, SeeHere found 14 visuals on screen. 3 had already been read aloud, so they were skipped. 9 were described in the pauses without overlapping the lecturer. 1 was held for review: an equation the OCR misread, which SeeHere flagged instead of voicing. Upload to finished took about 55 seconds.

Try it

Live demo: https://65-20-87-43.sslip.io.

Demo video (4:16): https://drive.google.com/file/d/1JVgxgrURHVeb9HwqYSpLBWoVyXv5K5MS/view?usp=sharing

Upload a lecture, or open the demo project from the README. The live system status is at https://65-20-87-43.sslip.io/status.

Honest limits

Verification checks quotes and numbers against what is on screen. It isn’t full fact-checking or WCAG certification, and an instructor reviews the descriptions before they are published.

How it uses Vultr

SeeHere runs in production on Vultr. Everything in the demo runs there; no step depends on a local machine. 1) Vultr Cloud Compute (4 vCPU / 8 GB, Mumbai) runs the whole stack with Docker Compose: a) a Caddy reverse proxy with automatic HTTPS b) the Next.js web app and FastAPI backend c) PostgreSQL 16, which is both the database and the job queue (SKIP LOCKED) d) the processing worker that runs the 14-stage pipeline: ffmpeg, faster-whisper speech-to-text, Tesseract OCR, OR-Tools scheduling and Piper text-to-speech. It’s provisioned with cloud-init and updated with deploy.sh, and has firewall rules, health checks, database backups and a rollback script. 2) Vultr Serverless Inference runs the vision model that reads charts, tables and diagrams on each keyframe. Every call is measured: count, failures and latency appear on the live status page and at /api/metrics. If inference is unavailable, the pipeline falls back to OCR-only. 3) Cost-aware design. Heavy processing runs locally on the instance, so it has no per-request cost. Only the vision step uses metered inference. The status page counts every hosted-model call, including its failures and latency, so the inference usage shown is measured, not estimated.

No comments yet

Be the first to comment on this project.