MediaCrawler is a Python-based multi-platform crawler for collecting publicly available content and comments from major Chinese social platforms such as Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.
This repository provides a multi-platform social-media data collection tool focused on public information. The README says it supports keyword-based searches, fetching by specific post ID, second-level comments, creator-page collection, login-state caching, proxy-pool support, and comment word-cloud generation across several platforms. It also includes a web UI for configuration, monitoring, and data preview/export.
It addresses the need to collect public social-media content from multiple platforms in one place, instead of building separate collectors for each site. The README positions it as a way to reduce technical difficulty by avoiding complex JS reverse engineering and relying on browser automation with preserved login state.
Conceptually, MediaCrawler uses browser automation to access platform pages in a logged-in browser context, then extracts data from search results, post-detail pages, creator pages, and comments depending on the chosen mode. The README says it can run through an existing Chrome browser via CDP to reuse cookies, login status, and extensions, or use standard Playwright mode. It also describes a web interface that sends crawler settings to a backend API and lets users view logs and results.
Its attention likely comes from broad platform coverage, a clear beginner-friendly setup path, and a feature list that is practical for social-data collection workflows. The repository also presents active documentation for both CLI and WebUI usage, plus a prominent comparison to a separate Pro version, which may help draw interest. The large star count and strong daily growth in the metadata suggest it is widely noticed right now.
The README itself points to a separate MediaCrawlerPro project as a related option with additional features and a different architecture. Beyond that, the repository mentions BrowserAct, a sponsored no-code data extraction product, as a contrasting approach for extracting data from arbitrary websites. No other direct in-repo alternatives are described in the provided material.
AI-explained · grounded in each repo's README