Acervo — Bernard Odon

Modelling a YouTube archive so a video is canonical and its appearances are not — and why that made everything else simple.

Python CLI, PostgreSQL · 2025

Acervo is an incremental crawler that turns YouTube channels and playlists into a local, searchable knowledge base — transcripts, summaries, and classification, built up over repeated runs rather than one big import. The whole system rests on a single modelling decision made early: a video is not the same kind of thing as its appearance in a collection.

Problem

The same video can show up in a channel's uploads and in an unrelated playlist, at different positions, discovered on different runs. If "this video is in this collection" and "this video exists, with this transcript and this summary" are treated as one record, importing a new playlist that happens to contain an already-known video either duplicates the video's expensive data (a wasted re-fetch and re-summarize) or silently merges two collections' organization into one.

Constraints

Runs are incremental, not a single batch import — the system has to be correct across many runs spread over time, not just on a clean first pass.
LLM calls cost money, so summarization is gated behind explicit approval rather than running automatically on every discovered video.
A collection export must not leak items from another collection, even when both contain the same underlying video.

Mechanism

Video is the canonical record, identified by video_id — it owns the transcript, the summary, and processing state, and that data is reused wherever the same video appears. CollectionItem records the separate fact that a video showed up in a specific channel or playlist, at a specific position, with whatever payload that source provided. Each crawl run also keeps the raw provider observation it produced, so a playlist import doesn't overwrite or blend with what a channel import already recorded for the same video.

Decisions

Separate the video from its appearance in a collection

Chose: `Video` (canonical, reusable) and `CollectionItem` (this appearance, this position, this source) as two distinct records.
Rejected: One record per video per collection, duplicating transcript and summary whenever a video appears more than once.
Cost: Every query that wants 'this collection's videos' has to join across two tables instead of reading one.

Approval gates before spending on summarization

Chose: Discovery and transcript extraction happen automatically; LLM summarization and classification wait for explicit approval.
Rejected: Summarizing every discovered video immediately.
Cost: A discovered video can sit unsummarized for a while — the archive is intentionally not 'complete' the moment a crawl finishes.

Gemini video as a transcript fallback, not the default path

Chose: Try transcript extraction first; fall back to Gemini's video understanding only when a transcript fails or isn't enough.
Rejected: Using Gemini video processing as the primary extraction method.
Cost: Two different code paths to maintain for getting from video to text, instead of one.

What broke

Preserving each run's raw provider observation, instead of just merging new data into the existing record, was added specifically because an early version conflated "the video's data" with "what this particular API response said" — which meant a playlist re-crawl could quietly overwrite context that a channel crawl had already established. Keeping the raw observation per run, scoped to the collection that produced it, was the fix.

What I'd change

The organize-by-theme step still needs approval before it runs, same as summarization, and it's the least developed part of the system — I'd rather finish making that step observable (what it changed, why) before extending it further.