Video Understanding API

Turn any YouTube, TikTok, Instagram Reels, Reddit, or Pinterest video URL into typed JSON — chapters, entities, products shown, claims, and key moments — in one API call. No file upload, no indexing job to poll: send the same bare URL you'd send for a transcript, and a vision LLM reads the video itself.

Read the docs Pricing

Most video-understanding APIs — TwelveLabs among them — start with a file upload: you fetch the video yourself, push the bytes to their storage, then poll an indexing job before you can ask anything about it. FrameFetch skips all of that. You already send FrameFetch a bare social-video URL for a transcript or metadata call — add "structured" to fields on that exact same call and you get the video, understood: FrameFetch downloads it, samples keyframes, and runs the vision analysis server-side. One URL in, typed JSON out — no storage bucket to manage, no upload step, no job status to poll.

Add "structured" to fields on /v1/extract (or /v1/batch) and FrameFetch auto-includes "transcript", downloads the source video for its own keyframe pass, and runs one gpt-4o-mini vision call (temperature 0, so results are deterministic) over the transcript plus up to 12 sampled keyframes. The response carries a structured object with five typed arrays.

curl -X POST https://framefetch.net/v1/extract \
  -H "Authorization: Bearer <your-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=jNQXAC9IVRw",
    "fields": ["structured"]
  }'

"structured": {
  "chapters": [ { "start_sec": 0, "end_sec": 19, "title": "At the zoo", "summary": "A first-person walk past the elephant enclosure." } ],
  "entities": [ { "name": "San Diego Zoo", "type": "place" }, { "name": "elephant", "type": "other" } ],
  "products_shown": [],
  "claims": [ { "text": "Elephants have really, really long trunks.", "timestamp_sec": 14 } ],
  "key_moments": [ { "timestamp_sec": 3, "description": "The narrator points the camera at the elephants." } ],
  "keyframe_mode": "uniform"
}

Grounded in evidence, not guesswork: products_shown and claims both distinguish what was seen from what was said, so a caller can tell an on-screen product placement from a spoken mention.

The vision call runs at temperature: 0, so the same video produces the same structured reading every time. Unlike audio_digest, structured carries no expiring signed URL — it takes part in FrameFetch's normal content-hash cache. Ask for it once, and every subsequent call for that same URL (with structured in fields) is served from cache at the price floor instead of re-running the model.

The keyframe count is fixed by the engine at up to 12 — it isn't a caller knob — so every structured call is a bounded, roughly constant-cost analysis. That's why it's priced flat rather than per-frame or per-token: $0.03 per video, charged only when the analysis is actually produced. See the full rate card.

Structured analysis is treated exactly like digest: best-effort, never a hard failure of the whole call. If no vision provider is configured, the provider returns an error, or the model's output can't be parsed, structured is simply omitted, a plain-English string is appended to the top-level warnings array, and the call is not billed for it — the rest of the response (transcript, metadata, frames) stays intact. These degrade cases never surface as an HTTP error code, only as a warning string.

FrameFetch ships an MCP server at POST https://framefetch.net/mcp. The framefetch_extract tool takes the same fields: ["structured"] argument — identical behavior to the HTTP API, so an agent can ask for a typed reading of a video the same way it asks for a transcript or a digest.