Lecture Lense

Ninety minutes of lecture, and every idea in it findable in seconds.

Problem

A recorded lecture is the worst possible container for what it contains. The material is all there, in order, and completely unsearchable — finding the four minutes where the lecturer defined a term means scrubbing, guessing, and scrubbing again. Auto-captions help a little and hurt a little: they're searchable, but they mis-hear exactly the words that matter most, the domain terms nobody outside the room says.

The thing that was missing wasn't transcription. It was structure — and a way to trust it, because a summary you can't check against the source is just a confident stranger's notes.

Constraints

• No team. One person across five stacks — a Next.js app, a Node worker, a Python inference service, the database, and the ops holding them together. Anything I couldn't debug alone at eleven at night was out before it was considered, which is most of what kept the architecture at three services instead of eight.
• Recordings are long and processing is slow. An hour of audio can't be handled inside a request. Everything after upload had to be a queue, with failure treated as normal rather than exceptional.
• Free tiers only. No budget, so no managed vector database, no per-minute transcription API, and no always-on managed anything. Whatever I couldn't get free, I had to be able to run myself.
• Generated text is guilty until proven otherwise. A model will produce plausible notes about a lecture it misheard. Anything it wrote had to point back at the audio it came from.

Approach

Three services and one database. A Next.js app for everything a person touches, a Node worker running the pipeline, and a Python service that owns the models. Postgres holds the users, the transcript segments, their embeddings, and every generated artifact — one store, so a question can be answered with a single join instead of a fan-out across three systems.

• apps
  • web# Next.js — workspace, chat, library
  • worker# BullMQ pipeline stages
• services
  • inference# FastAPI — Whisper, embeddings
• packages
  • ai# prompts, providers, retrieval
  • db# Drizzle schema + pgvector
• evals# golden sets, run in CI

The pipeline is ten stages — ingest, transcribe, correct, chapter, embed, notes, glossary, study, highlights, finalize — chained as parent and child jobs. Each one writes its own row before it starts and checks for that row before it runs, so a stage that dies halfway through resumes instead of redoing the hour of transcription in front of it.

The stage worth describing is correction. Whisper mishears domain vocabulary in a specific, predictable way: it substitutes a common word that sounds like the uncommon one. My first attempt was a dictionary of known substitutions, which fixed the errors I'd already seen and none of the ones I hadn't. The replacement passes a window of segments to a model along with what the lecture is about, and asks for edits with reasons — so "code" becomes "dam" in a hydraulics lecture because the surrounding sentences are about spillways, not because anyone wrote that rule down.

Retrieval is deliberately unclever: vector search and Postgres full-text run separately and their rankings are fused, because the questions people ask are half conceptual and half a half-remembered exact phrase.

// Every answer cites the segments it read, and a citation is a timestamp.
const hits = await retrieve({ query, scope: { mediaId }, k: 5 });

return {
  answer,
  citations: hits.map((h) => ({
    mediaId: h.mediaId,
    startMs: h.startMs,   // clicking this seeks the player
    text: h.textCorrected,
  })),
};

Tradeoffs

• Self-hosted Whisper over a transcription API. A hosted endpoint is four cents an hour and instant to integrate; running faster-whisper on a CPU box is free and slower than real time. I took the slow one because the pipeline is asynchronous anyway — nobody is waiting on a spinner — and because the cost of the whole product then lives in one place I control.
• One Postgres over a purpose-built vector database. A dedicated store would search faster at scale I don't have. Keeping embeddings beside the segments they came from meant chapters, glossary terms, and citations all resolve in the same query, and there was one fewer service to keep alive on a free tier.
• Correcting transcripts with a model, knowing it can invent. A model that rewrites text can rewrite it wrongly. The guard is a cap on how much of any window it may change, plus every edit stored with its original and a reason, revertible in the UI. Confident silence would have been the worse failure.
• Batch processing over anything live. No real-time transcription, no live meeting bot. Both are a different architecture wearing the same product's clothes.

Outcome

An hour of audio costs about five cents to process and comes back as chapters, three lengths of notes, a glossary, flashcards, and an index that answers questions with the timestamp it read the answer from. Retrieval lands at 0.82 precision@5 on a hand-labelled question set, and the correction pass fixes the domain terms the dictionary never knew about.

The whole thing runs on free tiers with one small box doing the work, and one person on call for it. Those two constraints shaped most of the decisions above.

Decisions

What was chosen against, and why:

• Postgres with pgvector over MongoDB and Chroma. The earlier version of this used both, plus a separate relational store for users — three databases, three failure modes, and no way to ask a question that crossed them. Consolidating cost me document flexibility I wasn't using and bought back the only thing in short supply, which was attention. Against it: pgvector's index needs tuning that a dedicated store handles for you.
• Timestamps as the primary key of the product. Every artifact — chapter, note heading, flashcard, citation — carries a time and seeks the player. It constrained the UI heavily, since everything has to belong somewhere on a timeline. It also made the product checkable, which was the point.
• Evaluating prompts in CI, before building more features. Golden sets for correction, retrieval, and chaptering, run on every change to the prompts, failing the build on a regression. It's unglamorous work that shipped nothing users can see, and building it alone meant a week spent labelling data instead of adding features. Without it, though, there is no second reader — "I improved the prompt" stays a feeling rather than a claim.
• Model tiering over one good model everywhere. A cheap model does correction, chaptering, and flashcards; a stronger one writes notes and answers questions. It cut the per-recording cost by most of it, at the price of a provider abstraction and a second set of prompt behaviours to keep an eye on.