feat(instagram): anonymous Instagram scraper across REST, agent, and MCP surfaces#1593
Conversation
…for enhanced multi-agent chat capabilities
Added detection for Instagram's soft login wall, which returns a 200 status with login HTML. Implemented a new function to identify login redirects and adjusted the fetch logic to treat these cases similarly to 401/403 responses.
Added tests to handle scenarios where a 200 status response is redirected to a login page, ensuring proper rotation of IPs and raising errors when persistent login redirects occur. Updated the _FakeSession class to support login wall detection.
- Added support for extracting likes, comments, username, timestamp, and caption from Open Graph meta tags. - Implemented fallback mechanisms to ensure graceful degradation when expected data is missing. - Updated unit tests to validate new parsing logic and ensure robustness against unrecognized formats.
- Enhanced the regular expression for mentions to prevent trailing punctuation from being included in handles. - Added support for extracting media IDs from deep-link meta tags in anonymous posts. - Updated unit tests to validate the new media ID extraction and ensure proper handling of mentions.
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
AnishSarkar22
changed the title
…ng TikTok scraping capabilities, including comments, user search, and trending features. Updated README and documentation to reflect the addition of 24 native tools.
2 participants
Description
app.utils.proxy+scrapling), no browser and no login/sessionidever; its input/output mirrors the public Instagram actor spec (resultsType/directUrls/camelCase, additiveextra="allow"parity) so existing callers work unchanged.csrftoken+mid+x-ig-app-id) on a sticky-IP proxy, then rotates the IP and re-warms on login walls (401/403), backs off on 429, returnsNoneon 404, and raisesInstagramAccessBlockedErroron an/accounts/login/redirect instead of a silent empty.web_profile_infoJSON, profile feed posts/reels viaparse_media, and single post/reel extraction that reads the embedded mobile-v1PolarisMediaJSON with Open Graph meta as a lossy fallback (parse_post); numeric-ID post URLs are skipped since the page keys on the shortcode.site:instagram.com <query>via the existinggoogle_searchplatform) because Instagram's native keyword search is login-walled; keeps only profile hits, de-dupes, and caps atsearchLimit.302to login for logged-out callers; comment content stays login-walled and only the comment count (commentsCount) is exposed.images/childPosts), tagged users, coauthor producers, location, product type, pin state, and related profiles.app/capabilities/instagram/) withscrapeanddetailsverbs (definitions, executors, schemas) exposing REST endpoints, plus per-returned-item billing.id, and appliesresultsLimit/onlyPostsNewerThanas request-time policy.auth-utils.scripts/stress/stress_instagram_scraper.py) with discovery, single-post probe, mentions-wall probe, and field-coverage modes.likesCountis best-effort (often withheld anonymously), deep feed pagination is out of scope for v1, and the$3.50/1k itemsmeter should be re-measured before high-volume use.Motivation and Context
FIX #
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR introduces a complete anonymous Instagram scraper platform integration across the entire SurfSense stack. The implementation adds the ability to scrape public Instagram posts, reels, and profile details without authentication, using a proxy-based anonymous approach similar to the existing Reddit scraper. The feature includes a proprietary scraping layer (
app/proprietary/platforms/instagram), two capability verbs (instagram.scrapeandinstagram.details), an AI agent subagent for multi-agent chat, MCP server tools, comprehensive unit tests, an e2e testing script, billing integration with per-item metering, and full marketing/documentation pages on the web frontend.⏱️ Estimated Review Time: 3+ hours
💡 Review Order Suggestion
surfsense_backend/app/proprietary/platforms/instagram/README.mdsurfsense_backend/app/proprietary/platforms/instagram/__init__.pysurfsense_backend/app/proprietary/platforms/instagram/schemas.pysurfsense_backend/app/proprietary/platforms/instagram/url_resolver.pysurfsense_backend/app/proprietary/platforms/instagram/parsers.pysurfsense_backend/app/proprietary/platforms/instagram/fetch.pysurfsense_backend/app/proprietary/platforms/instagram/scraper.pysurfsense_backend/app/capabilities/instagram/__init__.pysurfsense_backend/app/capabilities/instagram/scrape/schemas.pysurfsense_backend/app/capabilities/instagram/scrape/executor.pysurfsense_backend/app/capabilities/instagram/scrape/definition.pysurfsense_backend/app/capabilities/instagram/details/schemas.pysurfsense_backend/app/capabilities/instagram/details/executor.pysurfsense_backend/app/capabilities/instagram/details/definition.pysurfsense_backend/app/capabilities/core/billing.pysurfsense_backend/app/capabilities/core/types.pysurfsense_backend/app/config/__init__.pysurfsense_backend/.env.examplesurfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/agent.pysurfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/description.mdsurfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/system_prompt.mdsurfsense_backend/app/agents/chat/multi_agent_chat/subagents/builtins/instagram/tools/index.pysurfsense_backend/app/agents/chat/multi_agent_chat/subagents/registry.pysurfsense_backend/app/agents/chat/multi_agent_chat/constants.pysurfsense_backend/app/routes/__init__.pysurfsense_backend/tests/unit/platforms/instagram/test_skeleton.pysurfsense_backend/tests/unit/platforms/instagram/test_parsers.pysurfsense_backend/tests/unit/platforms/instagram/test_discovery.pysurfsense_backend/tests/unit/platforms/instagram/test_fetch_resilience.pysurfsense_backend/tests/unit/platforms/instagram/test_budget.pysurfsense_backend/tests/unit/capabilities/instagram/test_executor.pysurfsense_backend/tests/unit/capabilities/instagram/test_registry.pysurfsense_backend/tests/unit/capabilities/instagram/test_schemas.pysurfsense_backend/tests/unit/capabilities/test_billing.pysurfsense_backend/scripts/e2e_instagram_scraper.pysurfsense_mcp/mcp_server/features/scrapers/platforms/instagram.pysurfsense_mcp/mcp_server/features/scrapers/__init__.pysurfsense_mcp/mcp_server/selfcheck.pysurfsense_mcp/mcp_server/server.pysurfsense_web/lib/connectors-marketing/instagram.tsxsurfsense_web/lib/connectors-marketing/index.tssurfsense_web/lib/playground/catalog.tssurfsense_web/lib/playground/platform-icons.tsxsurfsense_web/content/docs/connectors/native/instagram.mdxsurfsense_web/content/docs/connectors/native/meta.jsonsurfsense_web/app/(home)/mcp-server/page.tsxsurfsense_web/components/homepage/hero-section.tsxsurfsense_web/components/homepage/use-cases.tsxsurfsense_web/content/docs/how-to/mcp-server.mdxsurfsense_backend/.gitignoresurfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.pysurfsense_web/lib/auth-utils.tssurfsense_web/lib/connectors-marketing/google-maps.tsxsurfsense_web/lib/connectors-marketing/google-search.tsxsurfsense_web/lib/connectors-marketing/reddit.tsxsurfsense_web/lib/connectors-marketing/web-crawl.tsxsurfsense_web/lib/connectors-marketing/youtube.tsxsurfsense_web/content/docs/connectors/index.mdxsurfsense_web/content/docs/connectors/native/index.mdxsurfsense_web/public/connectors/instagram.svg