How to give an AI agent video data
An AI agent can't watch a video. To reason about one it needs the video turned into text and images first. This guide shows the four things an agent needs — transcript, metadata, frames, and on-screen text — and how to fetch all of them from a single URL in one call.
The short answer
Send a social-video URL to an extraction API and get back: a transcript (what is said), metadata + insights (what the video is, who posted it, how it performed), sampled frames (what it looks like at chosen moments), and on-screen text (what's written on screen, via OCR). With FrameFetch that is one POST /v1/extract across YouTube, YouTube Shorts, TikTok, Instagram Reels, Pinterest, and Reddit — or one MCP tool call if your agent speaks Model Context Protocol.
Why agents need this
Large language models are text-and-image reasoners. A raw .mp4 is neither. Four derived signals close the gap:
| Signal | Answers | Built from |
|---|---|---|
| Transcript | What is being said? | Platform captions, else Whisper speech-to-text |
| Metadata & insights | What is this and how did it do? | Title, author, duration, date, views, likes, comments |
| Frames | What does it look like? | Parametric sampling — every Nth, 1-per-second, or a time range, at any width |
| Text overlay | What's written on screen? | OCR on each frame — captions, price tags, signage, with confidence + position |
Feed any subset into your model's context and it can summarise, classify, fact-check, caption, or search the video without ever downloading it.
Step 1 — Get a transcript
The cheapest, highest-value signal. Captions when the platform has them; Whisper when it doesn't — so you still get text for caption-less Reddit and TikTok clips.
curl -X POST https://framefetch.net/v1/transcript \
-H "Authorization: Bearer <your-key>" \
-H "Content-Type: application/json" \
-d '{ "url": "https://www.tiktok.com/@user/video/123" }'Step 2 — Add metadata and engagement
Title, author, duration, upload date, plus views/likes/comments — useful for ranking, dedup, and "is this worth processing" gates before you spend on a transcript or frames.
curl -X POST https://framefetch.net/v1/metadata \
-H "Authorization: Bearer <your-key>" \
-H "Content-Type: application/json" \
-d '{ "url": "https://www.youtube.com/watch?v=jNQXAC9IVRw" }'Step 3 — Sample frames (when vision matters)
For visual questions — products shown, scene changes, layout — pull frames. Sampling is parametric so you don't pay for 30 fps you don't need: one per second, every Nth frame, or a specific [start,end] range, at the width you want.
curl -X POST https://framefetch.net/v1/frames \
-H "Authorization: Bearer <your-key>" \
-H "Content-Type: application/json" \
-d '{ "url": "https://www.youtube.com/watch?v=jNQXAC9IVRw",
"frames": { "mode": "fps", "fps": 1, "width": 512 } }'Frames come back as time-limited signed image URLs you can hand straight to a vision model.
Step 4 — Read on-screen text (OCR) new
A transcript only captures spoken audio — it misses burned-in captions, price tags, and signage. Add text_overlay alongside frames to run OCR on each extracted frame:
curl -X POST https://framefetch.net/v1/extract \
-H "Authorization: Bearer <your-key>" \
-H "Content-Type: application/json" \
-d '{ "url": "https://www.youtube.com/watch?v=jNQXAC9IVRw",
"fields": ["frames", "text_overlay"],
"frames": { "mode": "fps", "fps": 1, "width": 512 } }'Each frame gets a matching textOverlay entry with the detected text, per-line confidence, and a bounding box — a frame with no text simply returns an empty result, not an error.
POST /v1/extract and "fields": ["metadata","transcript","frames","text_overlay"]. You're billed only for what you request.
Wiring it into an agent (MCP)
If you build on Claude, Cursor, or any MCP client, add FrameFetch as a server and the model can call a video URL directly — no glue code:
claude mcp add --transport http framefetch \ https://framefetch.net/mcp \ --header "Authorization: <your-key>"
It exposes framefetch_extract, framefetch_search, framefetch_account and framefetch_platform_capabilities.
Letting the agent pay for itself (x402)
An autonomous agent shouldn't need a human to provision an API key. With x402 the agent calls the endpoint, gets an HTTP 402 with payment requirements, pays in USDC from its own wallet, and retries — no signup, no human. FrameFetch settles x402 on Base mainnet and declares the x402 Bazaar discovery extension, so it becomes indexed for Bazaar-based discovery automatically once a real payment settles on it. Humans can still use a free tier, prepaid credits, or a Stripe card.
What it costs
Pay per call: metadata is sub-cent, transcripts are metered per minute of audio, frames and on-screen text are metered per frame. A free credit is included on signup, and identical requests are cached at the price floor. See the pricing page for exact numbers.
FAQ
How does an AI agent read a video?
It doesn't watch it — it reads a transcript, metadata, sampled frames, and on-screen text derived from the video. FrameFetch returns all four from one URL.
What's the easiest way to get a video transcript via API?
POST the URL to /v1/transcript. Captions are used when present, Whisper otherwise, so caption-less videos still return text.
Can an agent read text that appears on screen?
Yes — add text_overlay alongside frames to run OCR on each extracted frame and get back the on-screen text with confidence and position.
Can an agent pay without an account?
Yes — via x402 (USDC) the agent pays per call with no signup or human in the loop.