[Feat] TikTok public data scraper (scrape, comments, user search, trending)#1589
Merged
Conversation
# Conflicts: # surfsense_mcp/mcp_server/core/auth/__init__.py # surfsense_mcp/mcp_server/core/auth/headers.py # surfsense_mcp/mcp_server/core/auth/identity.py # surfsense_mcp/mcp_server/core/auth/middleware.py # surfsense_mcp/mcp_server/core/transport/__init__.py # surfsense_mcp/mcp_server/core/transport/http.py # surfsense_mcp/mcp_server/core/workspace_matching.py # surfsense_mcp/mcp_server/features/knowledge_base/annotations.py # surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py # surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py # surfsense_mcp/mcp_server/features/scrapers/annotations.py # surfsense_mcp/mcp_server/features/scrapers/platforms/__init__.py # surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py # surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py # surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py # surfsense_mcp/mcp_server/features/scrapers/platforms/web.py # surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py # surfsense_mcp/mcp_server/features/scrapers/run_history.py # surfsense_mcp/src/surfsense_mcp/__main__.py # surfsense_mcp/src/surfsense_mcp/features/knowledge_base/__init__.py # surfsense_mcp/src/surfsense_mcp/features/scrapers/__init__.py # surfsense_mcp/tests/test_auth_headers.py # surfsense_mcp/tests/test_request_auth.py # surfsense_mcp/tests/test_workspace_context.py # surfsense_web/content/docs/how-to/mcp-server.mdx # surfsense_web/lib/mcp/clients.ts
A profile's account data (name, followers, bio, verification) lives in the page's rehydration blob and loads over plain HTTP without a signed request, so emit it first and always. The video listing needs a signed item_list XHR that TikTok withholds from anonymous sessions, so it stays best-effort and degrades to an ErrorItem. A blocked profile now yields its metadata instead of only an ErrorItem.
Video/general search is login-walled for anonymous sessions, but the Users tab (/api/search/user) returns public account records without a redirect, so this exposes the one reliably-unblocked search path. A keyword yields TikTokProfileItems (name, followers, bio, verification), deduped per query, capped, and degraded to an ErrorItem when a query is empty/withheld. Reuses the browser capture (generalized over XHR markers + extractor) and the shared profile item shape. Billed per account on a new TIKTOK_USER meter (TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
Comments load over a signed /api/comment/list XHR that TikTok serves to anonymous sessions once the comments panel is opened (unlike profile-video and general-search feeds), so this is a reliable verb. Given video URLs it returns CommentItems (text, author, likes, reply counts; replies carry repliesToId), deduped per video, capped, and degraded to an ErrorItem for empty/withheld videos or a bad_url ErrorItem for non-video inputs. Generalizes the browser capture over a pluggable interaction step so the comments flow (open panel, scroll the panel to paginate) reuses the same warm+capture scaffolding as listing/user-search. Billed per comment on a new TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
The Explore feed (/api/explore/item_list) is a global trending-video feed served to anonymous sessions, and it returns the same itemStruct shape as the other listings — so the verb reuses parse_video, the listing flow, the TikTokVideoItem output, and the per-video billing meter wholesale. Adds a browser-capture marker + fetch_trending, a synthetic-target orchestrator entry, and the tiktok.trending capability, surfaced on the chat subagent.
…uct surfaces Brings the three newer verbs to parity with tiktok.scrape everywhere it was wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description, the playground catalog, the native docs page, and the SEO marketing page. Also adds live e2e stages for comments, user search, and trending. Docs/FAQ now state the real contract: profile metadata is reliable while its video list can be withheld, and keyword video search is walled (use user search for accounts).
CREDO23
changed the title
The run-reader set gained export_run (CSV export of a stored run), but this manifest assertion still expected only read_run/search_run. Add export_run so the agent-door test reflects the shipped readers.
Sync with the advanced dev tip so the PR compares against current dev (drops already-merged commits from the diff). No history rewritten.
The profile feed (/api/post/item_list) returns an empty 200 to headless sessions but serves data headful on the same proxy IP. Run fetch_item_list headful and dismiss the login modal that blocks mid-scroll.
Profile videos now return via the headful item_list capture; reword the stage 2 comments to expect real videos, ErrorItem only as fallback.
Single switch promising an Xvfb display so the stealth browser can run headful (TikTok's profile feed is empty to headless Chromium). Defaults FALSE.
For every browser-running role (api/worker/all, not migrate) the entrypoint brings up a virtual display and exports DISPLAY, registering Xvfb for graceful shutdown. Off keeps browsers headless.
fetch_item_list goes headful only when the flag promises a display; otherwise it stays headless so the browser launch never fails and the empty feed degrades to an ErrorItem. Verified under xvfb-run: real videos returned on a clean proxy IP, graceful degrade on a flagged one.
The xvfb runtime path is now implemented, so drop the stale 'deferred to Slice B' note.
Bounds the browser-listing retry-on-empty; default 3, set 1 for a static IP.
The profile feed is withheld from flagged IPs and the proxy rotates per request, so re-fetching an empty capture (up to TIKTOK_LISTING_MAX_ATTEMPTS) turns a bad first draw into a hit instead of an ErrorItem.
…ideos The playbook listed search_queries as a video source in three places, then contradicted itself. Replace with action-oriented guidance: videos via hashtags/URLs, accounts via user_search. Drop 'login-walled' (the agent has no login path).
The hero, intro, use case, API-reference fields, and two FAQs all listed search as a video path, contradicting the FAQ that admits keyword video search is login-walled. Reconcile every claim to one truth: videos via hashtag/profile/URL; keyword search returns no videos; keyword -> user_search for accounts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native TikTok connector for public data — no login, no official Research API. Four verbs, each a capability (MCP tool + chat subagent + playground), billed per returned item (meters default off):
tiktok.scrape— videos by URL / profile / hashtag / searchtiktok.comments— a video's comment threadtiktok.user_search— accounts by keywordtiktok.trending— the Explore trending feedKnown limits (by design →
ErrorItem, never billed): profile video lists and keyword video search are gated for anonymous sessions. Profile metadata still returns reliably; useuser_searchfor account discovery.