How to Scrape X (Twitter) Data in 2026: Profiles, Posts, Replies

X (formerly Twitter) is still where news breaks, companies announce things and arguments happen in public. That makes its data valuable for monitoring, research and AI agents, and awkward to collect since the platform closed most of its free access. This guide shows how to pull public X data (profiles, posts, replies, Communities and video transcripts) with plain HTTPS calls, with working code in cURL, JavaScript and Python.

Why is X data hard to get?

X used to be the easiest platform to research. Since 2023 it has become one of the harder ones:

• The official API is paid. Reading data through the X API requires a paid tier, and the lower tiers cap how many posts you can read. Costs grow with every account you track.
• Logged-out viewing is restricted. Most timelines, replies and search results are only visible when signed in, and X actively limits automated sessions, so a DIY scraper needs accounts, proxies and constant repair.
• The web app’s data is deeply nested. Posts, quoted posts, media and counts arrive in a GraphQL format that changes shape between endpoints and over time.

Here is how the usual options compare:

The rest of this guide takes the last route.

What X data can you get?

Every call below is live on crawlfeed today. The table is generated from the API’s own catalogue, so the endpoints and prices are exactly what you will be charged:

Everything comes back in crawlfeed’s unified types (Profile, Post, Comment and friends), which are the same across all 26 platforms. The platform id is simply x.

Get an X profile

You need an API key (new accounts get 100 free credits) and a handle, without the @:

curl -G https://api.crawlfeed.dev/v1/x/profile \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "handle=nasa"

The response is a Profile wrapped in the standard envelope. Trimmed, and with illustrative numbers, it looks like this:

{
  "data": {
    "platform": "x",
    "handle": "nasa",
    "display_name": "NASA",
    "verified": true,
    "followers": 1234567,
    "following": 123,
    "posts_count": 12345,
    "bio": "…",
    "links": ["…"]
  },
  "meta": { "credits_used": 1, "credits_remaining": 99, "cache": "MISS", "data_age_s": 0 }
}

A profile costs 1 credit. Counts are always numbers, and a field X does not expose comes back as null rather than disappearing.

Get an account’s posts

/v1/x/posts returns an account’s recent posts, newest first, one page per call:

curl -G https://api.crawlfeed.dev/v1/x/posts \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "handle=nasa"

Each item is a Post with text, created_at, lang, media (images and videos with url, thumbnail_url and duration_s), engagement (likes, comments, shares, views, saves), hashtags, mentions and is_ad. On X, shares counts reposts plus quotes, and saves counts bookmarks.

To get the next page, pass meta.next_cursor back as cursor. When next_cursor is missing, you have reached the end. Every page is billed, so stop as soon as you have what you need:

let cursor;
const posts = [];
do {
  const params = new URLSearchParams({ handle: 'nasa', ...(cursor ? { cursor } : {}) });
  const res = await fetch(`https://api.crawlfeed.dev/v1/x/posts?${params}`, {
    headers: { Authorization: `Bearer ${process.env.CRAWLFEED_KEY}` },
  });
  const { data, meta } = await res.json();
  posts.push(...data);
  cursor = meta.next_cursor;
} while (cursor && posts.length < 200);

To read one post you already know, pass its id to /v1/x/post. The id is the long number at the end of a post’s URL.

Get the replies to a post

On X, the replies to a post are its comments. Pass the post id as post_id:

curl -G https://api.crawlfeed.dev/v1/x/comments \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "post_id=<post_id>"

Each Comment has the text, the author, likes, created_at, lang and reply_to_id, so you can rebuild a thread or measure how a conversation turned. Replies page with cursor, exactly like posts.

Read X Communities

X Communities are topic groups with their own feed. Two calls cover them:

• The Community itself: /v1/x/community?id=… returns the Community as a Profile, so it has the same fields as an account.
• Its posts: /v1/x/community/posts?id=… returns the posts shared in it, paged with cursor.

curl -G https://api.crawlfeed.dev/v1/x/community/posts \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "id=<id>"

Pass the Community id or its URL as id. Communities are useful when a niche talks among itself rather than on public timelines.

Get a transcript of an X video

For posts with video, /v1/x/transcript returns the spoken text, with timings when available. Pass the post id or URL:

curl -G https://api.crawlfeed.dev/v1/x/transcript \
  -H "Authorization: Bearer $CRAWLFEED_KEY" \
  --data-urlencode "id=<id>"

A transcript costs 1 credit, the same as every other X call.

Handle errors and rate limits

Errors are RFC 9457 problem details with a stable code, so branch on the code, not the message:

A call that fails is refunded automatically, so retrying never charges you twice.

Add Arabic sentiment and dialect

A large share of X conversation is in Arabic. Add enrich=arabic to a profile, posts or replies call and each item gains an enrichment field with the dialect, sentiment, entities and topics, for 3 extra credits per call. That is enough to chart how sentiment in a reply thread shifts without training your own classifier.

Use X data from Claude or Cursor

If you would rather ask than code, connect the crawlfeed MCP server and your assistant gets the same calls as tools:

claude mcp add --transport http crawlfeed https://mcp.crawlfeed.dev/mcp \
  --header "Authorization: Bearer $CRAWLFEED_KEY"

Then ask something like “what has @nasa posted this week, and what are the most-liked replies saying?”. The setup for every client is in our guide to MCP.

Wrapping up

For public X data, a scraping API saves you from paying for an API tier sized for someone else’s use case, or from maintaining logged-in scrapers. With crawlfeed, every X call costs one credit, and the same code works for TikTok, Instagram, YouTube and LinkedIn. The full reference is at /docs/platforms/x.