How to Scrape YouTube Data in 2026: Channels, Videos, Transcripts

YouTube is the largest video library on the web and, for many topics, the best primary source there is: product reviews, lectures, interviews, earnings calls. For research and AI work, the transcripts are often worth more than the videos. This guide shows how to pull public YouTube data (channels, videos, Shorts, comments, playlists, community posts and transcripts) with plain HTTPS calls, with working code in cURL, JavaScript and Python.

Why is YouTube data harder to get than it looks?

YouTube does have an official API, which makes it easier than most platforms. The friction shows up once you build on it:

• Quota, not just keys. The YouTube Data API needs a Google Cloud project and meters usage in daily quota units. Some calls, search in particular, cost far more units than others, so a search-heavy product hits the ceiling quickly.
• Transcripts are not really on offer. The captions endpoints are built for videos you manage. Getting the spoken text of someone else’s video is not a simple API call.
• Some surfaces are missing or awkward. Shorts, community posts and live streams are either absent from the official API or mixed in with everything else.

Scraping YouTube yourself is possible, but the page data is a large, deeply nested blob that changes regularly, and at volume you will run into consent pages and bot checks.

What YouTube data can you get?

Every call below is live on crawlfeed today. The table is generated from the API’s own catalogue, so the endpoints and prices are exactly what you will be charged:

Everything comes back in crawlfeed’s unified types (Profile, Post, Comment and friends), which are the same across all 26 platforms. A channel is a Profile, and a video, a Short or a community post is a Post.

Get a YouTube channel

You need an API key (new accounts get 100 free credits) and the channel handle, without the @:

curl -G https://api.crawlfeed.dev/v1/youtube/profile \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "handle=MrBeast"

The response is a Profile wrapped in the standard envelope. followers is the subscriber count and posts_count the number of videos. Trimmed, and with illustrative numbers, it looks like this:

{
  "data": {
    "platform": "youtube",
    "handle": "MrBeast",
    "display_name": "MrBeast",
    "verified": true,
    "followers": 1234567,
    "posts_count": 1234,
    "bio": "…",
    "country": "…",
    "links": ["…"]
  },
  "meta": { "credits_used": 1, "credits_remaining": 99, "cache": "MISS", "data_age_s": 0 }
}

A channel costs 1 credit. A field YouTube does not expose, such as following, comes back as null rather than disappearing.

List a channel’s videos

/v1/youtube/posts returns a channel’s uploads, newest first, one page per call:

curl -G https://api.crawlfeed.dev/v1/youtube/posts \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "handle=MrBeast"

Each item is a Post with the title and description in text, created_at, media (the video url, thumbnail_url and duration_s), engagement (views, likes, comments), hashtags and mentions.

To get the next page, pass meta.next_cursor back as cursor. When next_cursor is missing, you have reached the end of the channel. Every page is billed, so stop when you have enough:

import os, requests

videos, cursor = [], None
while True:
    params = {"handle": "MrBeast", **({"cursor": cursor} if cursor else {})}
    res = requests.get(
        "https://api.crawlfeed.dev/v1/youtube/posts",
        params=params,
        headers={"Authorization": f"Bearer {os.environ['CRAWLFEED_KEY']}"},
    )
    body = res.json()
    videos.extend(body["data"])
    cursor = body["meta"].get("next_cursor")
    if not cursor or len(videos) >= 150:
        break

For one video you already know, pass its id (the v= value in the URL) to /v1/youtube/post.

Get comments on a video

Pass the video id as post_id:

curl -G https://api.crawlfeed.dev/v1/youtube/comments \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "post_id=<post_id>"

Each Comment has the text, the author, likes, created_at and reply_to_id. Comments page with cursor, like videos. For audience research, comment threads are usually where the useful signal is.

Get a YouTube transcript

This is the call most people come for. Pass a video id or URL:

curl -G https://api.crawlfeed.dev/v1/youtube/transcript \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "id=<id>"

A transcript costs 1 credit and includes timings when they are available, so you can link a quote back to the exact moment in the video. It is the natural input for summaries, search indexes and retrieval pipelines that feed an LLM.

Shorts, live streams, playlists and community posts

A YouTube channel is more than its uploads list, and each surface has its own call:

• Shorts: /v1/youtube/reels?handle=… lists a channel’s Shorts on their own.
• Live streams: /v1/youtube/live/videos?handle=… lists a channel’s live and past live streams.
• Playlists: /v1/youtube/collections?handle=… lists a channel’s playlists, /v1/youtube/collection?id=… describes one, and /v1/youtube/collection/posts?id=… returns the videos in it.
• Community posts: /v1/youtube/community/posts?id=… takes a channel handle and returns its community tab; /v1/youtube/community/post?id=… fetches one post by URL.

curl -G https://api.crawlfeed.dev/v1/youtube/collection/posts \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "id=<id>"

Search, hashtags and trending

To discover videos rather than read a known channel:

• Search: /v1/youtube/search?q=… runs YouTube’s own search, and /v1/youtube/search/posts?q=… returns matching videos as posts.
• Hashtags: /v1/youtube/hashtag?tag=… returns videos using a hashtag, without the #.
• Trending: /v1/youtube/trending returns what is trending now, with an optional country.

curl -G https://api.crawlfeed.dev/v1/youtube/search/posts \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "q=coffee"

None of these touch a Google Cloud quota: a search costs the same 1 credit as any other call.

Handle errors and rate limits

Errors are RFC 9457 problem details with a stable code, so branch on the code, not the message:

A call that fails is refunded automatically, so retrying never charges you twice.

Add Arabic sentiment and dialect

If your audience comments in Arabic, add enrich=arabic to a channel, video or comments call. Each item gains an enrichment field with the dialect, sentiment, entities and topics, for 3 extra credits per call.

Use YouTube data from Claude or Cursor

If you would rather ask than code, connect the crawlfeed MCP server and your assistant gets the same calls as tools:

claude mcp add --transport http crawlfeed https://mcp.crawlfeed.dev/mcp \
  --header "Authorization: Bearer $CRAWLFEED_KEY"

Then ask something like “summarise the transcript of MrBeast’s latest video and list the main themes in its comments”. The setup for every client is in our guide to MCP.

Wrapping up

The YouTube Data API is fine within its quota, but a scraping API gives you transcripts, Shorts, playlists and community posts without a Cloud project or quota planning. With crawlfeed, every YouTube call costs one credit, and the same code works for TikTok, Instagram, X and LinkedIn. The full reference is at /docs/platforms/youtube.