The problem
Long interviews, lectures and meetings contain useful facts that are difficult to find again. Replaying two hours to locate a single statement is slow. A summary saves time, but without a link to the supporting passage, checking it becomes another search task.
CORTEX turns that into a question-and-answer workflow with the source close at hand. The demo uses a public two-hour interview with the founder of amoCRM, a CRM company; the underlying problem applies to any long spoken recording.
The product principle
The useful outcome is an answer the listener can check quickly. Finding a passage, explaining it and opening the original recording belong in one flow. The timestamp is simply how the interface gets there.
The knowledge source is the transcript of the spoken audio. Video frames provide visual previews for playback; the system does not analyse video imagery to answer questions.
A focused first version
- Included
- One recording with a prepared transcript, natural-language questions, suggested starting questions, semantic retrieval and streamed answers. Supporting source cards show the opening words of a passage and a video frame; clicking opens the recording at that moment. Daily limits keep the public demo bounded.
- Deliberately deferred
- User uploads, automatic transcription as a product feature, accounts and search across multiple recordings. A fixed corpus let me work through the complete answer-to-source experience before adding ingestion and library management.
From a question to the original recording
Preparation happens offline: a machine-generated transcript was proofread by hand, split into overlapping text fragments and embedded. The fragments retain the recording’s timestamps. At query time, the API embeds the question and searches the existing vectors.
- AskThe visitor asks in their own words. They do not need to remember the wording used in the interview.
- RetrieveCosine similarity ranks transcript fragments against the question. The closest passages become context for generation.
- AnswerThe LLM receives the question, retrieved passages and explicit instructions to ground its answer in that context and acknowledge missing information.
- Show the sourceThe model cites positions from the supplied transcript. Source cards pair the opening words of the original utterance with a frame from the recording.
- VerifyA card or an inline reference opens the player at the relevant moment. The listener can hear the source without manually scrubbing through the recording.
My role
- Product and system design
- I defined the concept, first-version scope and answer-to-source workflow. I designed the transcript preparation, chunking, vector index, retrieval and answer rules, as well as the source verification experience.
- Implementation and delivery
- I built the frontend and backend and used AI coding agents as implementation tools. I retained responsibility for product and architecture decisions, code review, testing, deployment and release acceptance.
- What I used
- The interview is third-party public demo material. Embedding and language models are existing services; my work is the product, retrieval system and verification experience around them.
Engineering decisions
- Semantic retrieval
- Comparing embedding vectors lets a question find relevant passages even when the wording differs. Retrieval supplies context to the language model: this is retrieval-augmented generation, or RAG.
- A file-backed vector index
- The fixed corpus has 88 fragments. An exhaustive scan in API memory is sufficient, so the vectors and text ship as files in the image, with integrity checks at startup. A dedicated vector database would add an operational dependency without being necessary for this scope.
- An explicit grounding contract
- The prompt requires answers and supporting references to come from the retrieved context, with a fallback when it lacks an answer. This guides generation; it is not a guarantee that every claim is correct. Source verification remains part of the product.
- Source text independent of generated prose
- A card’s excerpt comes from a prepared transcript lookup keyed by the utterance’s exact start time. An unmatched reference gets no guessed neighbouring excerpt. The frame helps identify the moment; listening to the recording is how the visitor checks the claim.
What grounding does and does not guarantee
Retrieval selects a limited context, and the model can still choose a weak supporting passage. A valid timestamp proves that a moment exists, not that it supports every statement in the answer. The system does not have a separate factual validator.
The interface exposes the source for human review. There is no formal accuracy benchmark yet, and a response saying a topic was not discussed should not be treated as an exhaustive search of the entire recording.
Verifying the source link
Existing checks connect the transcript, vector corpus and source lookup. Interface tests exercise reference mapping and deduplication; browser checks verify that the corresponding card opens playback at the expected second. Recorded production acceptance includes a real question through to source playback.
These checks establish that the verification path works. They do not turn the language model into a source of guaranteed facts or measure quality across an evaluation set.
Live today
The public demo supports questions about one prepared interview, semantic retrieval, streamed answers and playback from source cards. The interface, recording and answers are in Russian; this English case explains the workflow and engineering choices.
The product demonstrates a complete path from question to source verification. It does not yet accept arbitrary recordings or operate as a commercial recording library.
The next useful step
A small evaluation set with expected supporting passages would make retrieval and citation changes measurable. An English source recording would also let international visitors judge answer quality directly, beyond understanding the interface. Neither is implemented in the current demo.
Technology
- Frontend
- Nuxt 3, Vue 3, TypeScript; static output
- API
- Python, FastAPI, server-sent events; SQLite for limits
- Retrieval and generation
- Offline LlamaIndex SentenceSplitter; bge-m3 embeddings, local file-backed vectors and cosine similarity; LLM via OpenRouter
- Delivery
- Docker Compose and nginx; separate build and production hosts