Update, 14 Aug: ten days after publishing, OpenAI shipped a soft version of this. Computer History turns Mac activity into ChatGPT memory: it keeps typing, drops audio and pixels, and is off by default.
Agents are no longer limited by intelligence. They use tools well, work for hours unsupervised, and recover from their own mistakes. What stops them is that they know almost nothing about the company they work in.
Even for the most agent-first and software-centric companies, perhaps twenty percent of context is captured and retrievable for agents. Full capture through APIs and connectors, from communications like meetings and email to product engineering in Linear and GitHub, still does not encapsulate what is shown on a screen, spoken in a room, or drawn on a whiteboard. The setup is twenty people in one software company, everyone in the office, three months. Everyone opts in, including the founders. Everyone can read everything that is captured.
This is a thought experiment for what it would take to capture everything, and get significantly closer to perfect information for agents.
Capture has three surfaces: what is on a screen, what is said, and who is in a room with whom. The plan below shows where each one sits, where capture deliberately stops, and what flows into the database while people work.
Three things about the plan are deliberate. The bathroom is outside capture. The entrance carries a notice, and any visitor who declines turns capture off in the rooms they are in, with the fact of the stop written to the log. And capture ends at the door, which means the record is of the office, not of the people, so phones, homes, commutes, the walk to lunch and the conversation after work are all outside it, and a meaningful share of real decisions happen there.
Nothing is summarised or thrown away on the way in, because the summary you would write today is worse than the one a better model writes next year from the same source. Structure is derived on top and re-derived whenever the models improve.
The raw layer is an append-only event stream. Each event carries a timestamp, a person, a source, a modality and its payload. On top sit reconstructed conversations, screen sessions segmented into task episodes, and rolling summaries at one minute, one hour and three hours. That is the whole schema. There is no company object model, no project graph, no decision log.
The bet behind keeping raw audio rather than only transcripts, and raw video rather than only frames, is that today's models throw away more than they should. A transcript loses who hesitated, who interrupted, who was in the room and said nothing. None of that is usable now. All of it is recoverable later if the source survives.
Screen capture is close to solved and mostly already open source. Wearable audio is not, and the gap is specific enough to name.
| Surface | Approach | Battery | Data path | Status |
|---|---|---|---|---|
| Screen | ScreenCaptureKit, event driven. Frames paired with the macOS accessibility tree, Apple Vision OCR as fallback. Screenpipe is a working open-source precedent. | mains | Local SQLite, then sync | works |
| Screen, private windows | Excluded at the OS level rather than filtered after the fact, so blocklisted apps and private browsing never enter the pipeline. Plus a classifier for what the blocklist misses. | — | Never written | works |
| Audio, open | Omi pendant. Two mics, open developer kit, phone relays raw audio to your own database and your own keys. | 10–14 h | Yours | too short |
| Audio, all day | Plaud NotePin S, or Note Pro with a four-mic array. Clip-on, comfortable, genuinely all day. | 20–30 h | Closed cloud | closed |
| Audio, both | All-day battery and an open pipe in the same device. | — | — | does not exist |
| Transcription | Speaker separation comes for free from one mic per person. Diarization still runs for anyone unmiked in the room. | — | API | works |
| Camera | Shared rooms only, for room composition: who was present, who was at the whiteboard, what was on it. Not private offices, not the bathroom. | mains | Raw, kept | heavy |
| Keystrokes | Out. Screen capture already sees the result, and capturing keys directly is a different object both legally and in how it feels. | — | — | excluded |
| Personal devices | Out. A known and growing hole, since a real amount of work now happens on a phone. | — | — | excluded |
The install target is a desktop app that a person never opens and never configures. It records, it excludes what it is told to exclude, and it sends. Everything else is server side.
The unsolved half. Capture is easy. Ten thousand hours of transcript is a haystack, not context. Retrieval at this scale is the part that is genuinely not solved.
There is no separate interface for agents. Humans and agents hit the same database with the same tools, and everyone can read all of it, founders included. The bet behind raw storage is that agents will get increasingly better at querying the data for themselves.
In Germany and across the EU most of this is not permissible: continuous monitoring of individual workstations, recording conversations without consent, and deploying monitoring technology without works council co-determination are all out. Opt-in also does less work than it looks like it does, because consent from an employee is weak ground when the power imbalance makes it questionably free.
The observer effect. The claim is that people are self-conscious at first and then stop noticing, because capture is omnipresent, the culture accepts it, and everybody already knows the honest version is messier than the presented one. The counter-claim is that this only holds while capture is harmless. Attach it to evaluation and people manage their captured self permanently, and what you collect is a performance rather than work.
Everyone sees everything. A company that captures everything and then restricts who may query it has rebuilt information asymmetry with better tooling. So access is flat by design. The cost of flat access is that there is no private room left inside the company, which is a real loss and not only a leak of leverage.
Retention is forever. The database persists. A person's captured context stays after they leave, and they can take a copy.
Breach. This is the highest value target a twenty-person company could possibly hold, it sits in the cloud, and it is fully discoverable in litigation. The position taken here is to accept that risk and behave like a company that expects to be read, with separate handling for the small number of genuinely high-stakes screens.
Nothing here is running, and none of it works outside a company that already has high trust. Everyone opted in, the founders included, and everyone can read everything, because capture without those two conditions is not a context system, it is surveillance with better retrieval.
The reason to want it anyway is that every company describes its work far better than it does its work, and agents that only ever see the description cannot do the work. Close that gap and a company could produce a great deal more of what people actually need, with the record belonging to everyone inside it rather than to whoever runs it. The other half of the case is cultural. A company where nobody has to reconstruct what happened, where nobody holds context to stay useful, and where the honest messy version of the work is simply the version everyone already sees, should be a better place to spend a day, not a worse one.
There is a high probability that radical AI-first cultures will adopt capture patterns close to this one. A company that holds its full context and makes it accessible to humans and agents can produce more at significantly lower cost. Still, capture-everything setups need high scrutiny at first, so they do not turn into anxiety-inducing, less productive companies. If some companies get this right and we avoid the low-trust downsides, it is a future worth looking forward to.