Claude can't watch videos. One stdlib Python script fixes that for a few cents per recording, and turns screen recordings from your cofounder into bug fixes.
Frontier reasoning models like Claude have no video input at all, and every image you show them costs real tokens — then costs them again on every following turn, because images sit in the conversation history. The fix is division of labor: your expensive model does the reasoning, a cheap multimodal model does the looking and hands back text. A 300-token description carries the useful signal of a 5,000-token image, on every turn that follows.
Our setup: Gemini 3.5 Flash (Google's fast multimodal model, which reads video natively) is the witness. Claude Code is the judge. The bridge is one Python script and one Claude skill. We use it daily: one cofounder records himself hitting a UI (User Interface) problem with Cap, sends the link, and Claude on the other end watches the recording, maps what it saw onto the codebase, and fixes it.
Measured with the API's countTokens endpoint: Gemini samples video at 1 frame per second and bills about 100–110 input tokens per second of footage. The witness doesn't need to be smart. It describes, Claude decides.
Stdlib only, no dependencies. It takes a file, a URL, or a previously uploaded file id plus a question, sends it to the Gemini API (Application Programming Interface, Google's HTTP endpoint for the model), and prints the model's text answer. Files under 19MB go inline as base64; bigger ones go through Gemini's Files API with upload-and-poll, so hour-long recordings work too.
Four details do most of the work:
file_data.file_uri accepts any public URL, so a share link goes straight to the model and nothing gets downloaded. When a host refuses Google's fetcher the API answers 400 Cannot fetch content from the provided URL, and the script prints the curl command to fall back to.thinkingConfig.thinkingBudget: 0 drops the model's internal reasoning tokens, which were 93% of the output bill on these describe-the-screen prompts and changed no answers. Set LOOK_THINK=<n> to put it back.uploaded files/<id> on stderr. Pass that id back for follow-up questions instead of re-uploading 90MB. Uploads expire after about 48 hours.video/mp4-style content type) or a directory where a video was expected.#!/usr/bin/env python3
"""look.py <file|url|files/id> [question...] — cheap eyes: describe a video/image via the Gemini API."""
import base64, functools, json, mimetypes, os, socket, subprocess, sys, time
import urllib.error, urllib.parse, urllib.request
BASE = "https://generativelanguage.googleapis.com"
MODEL = os.environ.get("LOOK_MODEL", "gemini-3.5-flash")
_T = os.environ.get("LOOK_THINK", "0")
THINK = int(_T) if _T.lstrip("-").isdigit() else 0 # 0 = no thinking tokens billed
DEFAULT_Q = ("Describe this recording shot by shot: timestamps, every screen shown, "
"all on-screen text, every click/scroll/interaction, and any moments of "
"hesitation, confusion, or UI glitches.")
INLINE_LIMIT = 19_000_000 # measured: 18.9MB raw (~25MB base64) returns 200 — do not lower this
MIMES = { # what the Gemini API accepts, plus the aliases Python's mimetypes emits
"video/mp4", "video/mpeg", "video/mov", "video/quicktime", "video/avi", "video/x-msvideo",
"video/x-flv", "video/mpg", "video/webm", "video/wmv", "video/x-ms-wmv", "video/3gpp",
"image/png", "image/jpeg", "image/webp", "image/heic", "image/heif",
"audio/wav", "audio/x-wav", "audio/mp3", "audio/mpeg", "audio/aac", "audio/x-aac",
"audio/ogg", "audio/flac", "audio/x-flac", "audio/aiff", "audio/x-aiff", "audio/mp4",
}
HINT = "" # extra stderr advice printed alongside a 400
def die(msg):
sys.exit("look.py: " + msg)
@functools.cache
def key():
k = os.environ.get("GEMINI_API_KEY") or ""
if not k:
try:
k = open(os.path.expanduser("~/.config/gemini/key")).read().strip()
except OSError:
die("no GEMINI_API_KEY set and no readable ~/.config/gemini/key")
return k
def api(path, data, headers, timeout=600, soft400=False):
req = urllib.request.Request(BASE + path, data=data,
headers={"x-goog-api-key": key(), **headers})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.load(r)
except urllib.error.HTTPError as e:
if soft400 and e.code == 400:
return None # caller gets one retry before the error is final
body = e.read().decode("utf-8", "replace").strip().replace("\n", " ")[:2000]
print(f"ERROR {e.code}: {body or '(empty body)'}", file=sys.stderr)
if e.code == 400 and HINT:
print(HINT, file=sys.stderr)
if e.code == 404 and "/" in MODEL:
print(f"LOOK_MODEL={MODEL} has a provider prefix; use a bare id "
f"like gemini-3.5-flash", file=sys.stderr)
sys.exit(1)
except (socket.timeout, TimeoutError):
die(f"timed out after {timeout}s on {path}")
except urllib.error.URLError as e:
die(f"network error on {path}: {e.reason}")
def mime_of(path):
m = mimetypes.guess_type(path)[0] or subprocess.run(
["file", "-b", "--mime-type", path], capture_output=True, text=True).stdout.strip()
if m not in MIMES:
die(f"unsupported media type {m or '(unknown)'} for {path}")
return m
def upload(path, mime):
try:
blob = open(path, "rb").read()
except OSError as e:
die(f"cannot read {path}: {e.strerror}")
if len(blob) < INLINE_LIMIT:
return {"inline_data": {"mime_type": mime, "data": base64.b64encode(blob).decode()}}
up = api("/upload/v1beta/files", blob,
{"X-Goog-Upload-Protocol": "raw", "Content-Type": mime})
name, uri = up["file"]["name"], up["file"]["uri"]
for _ in range(90): # videos need server-side processing before they can be used
state = api(f"/v1beta/{name}", None, {}, timeout=30)["state"]
if state == "ACTIVE":
print(f"uploaded {name} (expires ~48h) — pass this id instead of the file "
f"for follow-up questions", file=sys.stderr)
return {"file_data": {"file_uri": uri, "mime_type": mime}}
if state == "FAILED":
die(f"Gemini failed to process {name}")
time.sleep(2)
die(f"still processing after 3min — it may finish anyway; re-run: look.py {name} <question>")
def media_part(inp):
global HINT
if os.path.exists(inp): # a local file always wins over URL/id interpretation
if os.path.isdir(inp):
die(f"{inp} is a directory (a .cap project is a directory, not a video) — run: "
f"cap export {inp} -o /tmp/out.mp4 --json, then pass /tmp/out.mp4")
return upload(inp, mime_of(inp))
if urllib.parse.urlparse(inp).scheme in ("http", "https"):
HINT = (f"if Google cannot fetch that host, download it first: "
f"curl -L -o /tmp/video.mp4 '{inp}' — then pass /tmp/video.mp4")
return {"file_data": {"file_uri": inp}}
if inp.startswith("files/"): # reuse of an earlier upload
return {"file_data": {"file_uri": f"{BASE}/v1beta/{inp}"}}
die(f"no such file: {inp}")
def main():
if len(sys.argv) < 2:
sys.exit(__doc__)
inp, q = sys.argv[1], " ".join(sys.argv[2:]) or DEFAULT_Q
path, hdrs = f"/v1beta/models/{MODEL}:generateContent", {"Content-Type": "application/json"}
body = {"contents": [{"parts": [{"text": q}, media_part(inp)]}],
"generationConfig": {"thinkingConfig": {"thinkingBudget": THINK}}}
resp = api(path, json.dumps(body).encode(), hdrs, soft400=(THINK == 0))
if resp is None: # gemini-3.5-flash-lite rejects thinkingBudget=0; retry with its default
del body["generationConfig"]
resp = api(path, json.dumps(body).encode(), hdrs)
cand = (resp.get("candidates") or [{}])[0]
if cand.get("finishReason", "STOP") != "STOP":
print(f"WARNING: finishReason={cand['finishReason']}", file=sys.stderr)
parts = cand.get("content", {}).get("parts") or []
text = "".join(p["text"] for p in parts if "text" in p) # 3.x replies are multi-part
if not text.strip():
die("no text in response: " + json.dumps(resp)[:2000])
print(text)
if __name__ == "__main__":
main()
mkdir -p ~/.config/gemini && echo 'YOUR_KEY' > ~/.config/gemini/key && chmod 600 ~/.config/gemini/key (or export GEMINI_API_KEY).~/.claude/scripts/look.py and chmod +x it.~/.claude/scripts/look.py photo.png "what color is this?"Model swap is one env var: LOOK_MODEL=gemini-3.5-flash-lite for bulk work. Use bare model ids; a google/ prefix is OpenRouter syntax and 404s here. The question you send matters more than the model, so ask for timestamps, on-screen text, clicks, and hesitation rather than "describe this video".
Cap is an open-source screen recorder with a CLI (Command-Line Interface) built to be driven by agents: every command takes --json for machine-readable output.
# install
curl -fsSL https://cap.so/install-cli.sh | sh
# the workflow
cap auth status --json # uploads need a Cap login or CAP_API_KEY
cap record # start a recording (or use the desktop app)
cap recordings list --json # find recordings in the desktop library
cap export <project.cap> -o out.mp4 # a .cap project is a directory, not a video
cap upload out.mp4 # get a shareable cap.so linkYour cofounder records the bug, uploads, and pastes the link in chat. One catch when consuming a share link: the cap.so/s/<id> page is HTML, not the video. The page's og:video meta tag carries the videoId, and this URL is the video itself:
https://cap.so/api/playlist?videoId=<id>&videoType=mp4Hand that URL to look.py directly. Gemini fetches it server-side, so the video never touches your disk. Drop &videoType=mp4 and the endpoint returns a 547-byte error blob instead of a video. If the API answers 400 Cannot fetch content from the provided URL, that host refused Google's fetcher: curl -L -o video.mp4 "<url>" and pass the file.
A skill is a markdown file that teaches Claude Code when and how to use a tool. Drop this at ~/.claude/skills/cap-review/SKILL.md and any session can "watch" a recording on request:
---
name: cap-review
description: Watch a cofounder's cap.so screen recording (or any video/
screenshot) with cheap Gemini "eyes" and turn it into concrete UI /
onboarding fixes. Use when the user says "cap review", "watch this video",
"my cofounder sent a recording", pastes a cap.so link, or points at an
.mp4/.mov/.webm showing a UI problem. Pass the whole video to look.py in
one call; never scrub frames with ffmpeg unless one specific frame decides
something. Claude never ingests the video itself, the script returns a
text description to reason over.
---
# cap-review
`~/.claude/scripts/look.py <file|url|files/id> [question]` sends media to the
Gemini API and prints text. Model `gemini-3.5-flash`;
`LOOK_MODEL=gemini-3.5-flash-lite` for bulk runs. Use bare model ids, a
`google/` prefix 404s.
## Input resolution
0. Screenshot pasted into the conversation: look at it. No look.py round trip.
1. Local file: pass the path.
2. cap.so or cap.link share page: take the videoId from the `og:video` meta,
pass `https://cap.so/api/playlist?videoId=<id>&videoType=mp4` straight to
look.py. Gemini fetches that URL itself, so nothing downloads.
3. `400 Cannot fetch content from the provided URL`: the host refused
Google's fetcher. `curl -L -o /tmp/video.mp4 '<url>'` and pass the file.
4. `.cap` project (a directory, not a video): `cap export <project.cap>
-o /tmp/out.mp4 --json` first.
5. Follow-up question about a big video: look.py prints `uploaded files/<id>`
on stderr. Pass that id back instead of uploading again.
## Workflow
1. Ask the eyes to describe, never to judge. They have no product context, so
"is anything broken or stale or wrong here" gets a confident no. They are
the witness; you are the judge.
2. One call, whole video. The built-in default question already asks for
timestamps, on-screen text, clicks and hesitation. Write your own only
when the review has a specific focus, e.g. "transcribe everything said
aloud, with timestamps" when the sender narrated.
3. Map the description onto the codebase to find the screens and components.
4. Report: timestamp, what the video shows, the code location, the proposed
fix. Then fix what the user asked for.
5. The one exception to the no-frames rule: when a single frame decides
something subtle (pixel alignment, exact copy), run `ffmpeg -ss <t>
-i video.mp4 -frames:v 1 /tmp/frame.png` and read that frame directly.
## Mechanics
Files under 19MB go inline; larger ones upload and wait up to 3 minutes for
processing. Thinking is off by default, set `LOOK_THINK=<n>` to turn it back
on. Cost is about 100-110 input tokens per second of footage: a 3-minute
recording runs ~3 cents on flash and under a cent on flash-lite.Add a few lines to your global ~/.claude/CLAUDE.md so the loop runs without prompting:
## Cap video workflow (core)
Screen recordings are a core feedback loop with my cofounder, both directions:
- Incoming — a recording (cap.so link or file) showing a UI/onboarding
problem: use the `cap-review` skill. Cheap eyes via look.py, map findings
to code, fix.
- Outgoing — whenever I demo something and want my cofounder to see it:
use the `cap` CLI to record → export → upload and give me the shareable
link to send. Preflight `cap auth status --json`; if it reports
unauthenticated, tell me to sign into Cap Desktop or set `CAP_API_KEY`
instead of continuing.
Zero ceremony: a pasted cap link or video file IS the request. Watch it.
Don't ask permission, don't make me name the skill.
Never hand-transcribe a video or paste frames to the main model — the
look.py eyes exist for that. One exception: a single decisive frame pulled
with `ffmpeg -ss <t> -i video.mp4 -frames:v 1`.My cofounder hit a broken OAuth (Open Authorization — the sign-in-via-browser handoff) flow in our product's embedded terminal, recorded four minutes of it with Cap, and sent the link. Claude handed the share URL to the eyes, got the recording described shot by shot, and came back with: the paste handler in the embedded terminal double-fires, the code lands mangled, the login fails at 2:41 with "Invalid code", and here's the file to fix. Watching cost about four cents. No one transcribed anything.