Crawlee for Python v1.0 发布解读统一存储、自适应爬取与反检测能力的全面升级【免费下载链接】crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.项目地址: https://gitcode.com/GitHub_Trending/cr/crawleeCrawlee 是一个用于构建可靠爬虫的 Web 爬取与浏览器自动化库支持 JavaScript/TypeScript 与 Python 双生态。本文以 Crawlee 官方博客发布的Crawlee for Python v1.0公告为主体系统解读 v1.0 版本带来的七大核心能力——统一存储客户端体系、实验性 SQL 存储、自适应 Playwright 爬虫、Impit HTTP 客户端、Sitemap 请求加载器、robots.txt 协议支持、浏览器指纹与 OpenTelemetry 可观测性并对照本仓库Crawlee 的 Node.js/TypeScript 实现中的源码证据帮助你理解这些设计理念在两套语言实现中的落地方式。读完本文你将掌握 v1.0 各特性的配置要点与实战代码并能在自己的爬虫项目中按需组合使用。一、版本背景从 Beta 走向稳定Crawlee for Python 于 2024 年 7 月以 Beta 模式启动。在过去的一年里该项目积累了数千名早期采用者、大量社区功能请求以及多位贡献者。经过数月开发与打磨v1.0 正式脱离 Beta进入生产/稳定状态。v1.0 的核心承诺是严格遵循语义化版本Semantic Versioning从该版本起你可以将其视为爬虫项目的稳定基础破坏性变更只会在主版本升级时发生。快速上手安装或升级到最新版本pip install --upgrade crawlee如果你是从旧版本升级需要参考官方的 Upgrading to v1 迁移指南处理接口变化。v1.0 新特性一览新的存储客户端系统New storage client system自适应 Playwright 爬虫Adaptive Playwright crawlerImpit HTTP 客户端Impit HTTP clientSitemap 请求加载器Sitemap request loaderrobots.txt 排除标准Robots exclusion standard指纹生成FingerprintingOpenTelemetry 可观测性Open telemetry下文将逐一展开。二、新的存储客户端系统一套 API任意后端v1.0 最重大的架构变化是引入统一的存储客户端接口。在此之前数据集Dataset、键值存储Key-Value Store与请求队列Request Queue在不同存储位置下的处理方式存在差异v1.0 将它们统一为一致的接口。这意味着无论数据存储在内存、本地文件系统、数据库、Apify 平台还是自定义后端爬虫代码的 API 保持不变。带来的直接收益是更少的代码重复、更好的可扩展性与更清爽的开发体验也为社区自行实现并共享存储客户端打开了大门。文件系统存储客户端以下示例展示了使用文件系统后端持久化数据的配置方式from crawlee.configuration import Configuration from crawlee.crawlers import ParselCrawler from crawlee.storage_clients import FileSystemStorageClient # Create a new instance of storage client. storage_client FileSystemStorageClient() # Create a configuration with custom settings. configuration Configuration( storage_dir./my_storage, purge_on_startFalse, ) # And pass them to the crawler. crawler ParselCrawler( storage_clientstorage_client, configurationconfiguration, )关键参数说明storage_dir本地存储目录数据将持久化在该路径下purge_on_start是否在启动时清空已有存储数据设为False可保留历史数据。内存存储客户端内存型存储适合测试或短生命周期的爬取任务from crawlee.crawlers import ParselCrawler from crawlee.storage_clients import MemoryStorageClient # Create a new instance of storage client. storage_client MemoryStorageClient() # And pass it to the crawler. crawler ParselCrawler(storage_clientstorage_client)实验性 SQL 存储客户端v1.0 还引入了实验性的SqlStorageClient支持使用 SQL 数据库进行持久化目前支持SQLite 与 PostgreSQL。该后端支持多个爬虫进程并发访问从而支撑分布式爬取场景。SQL 存储客户端基于SQLAlchemy 2构建提供自动建表、连接池与数据库专属优化同时保持与其他存储客户端相同的接口切换后端无需改动爬虫逻辑。客户端通过上下文管理器保证连接的正确处理import asyncio from crawlee.crawlers import ParselCrawler from crawlee.storage_clients import SqlStorageClient async def main() - None: # Create SQL storage client (defaults to SQLite). async with SqlStorageClient() as storage_client: # Pass it to the crawler. crawler ParselCrawler(storage_clientstorage_client) # ... define your handlers and crawling logic await crawler.run([https://crawlee.dev]) if __name__ __main__: asyncio.run(main())使用 PostgreSQL 时只需提供连接字符串import asyncio from crawlee.crawlers import ParselCrawler from crawlee.storage_clients import SqlStorageClient async def main() - None: async with SqlStorageClient( connection_stringpostgresqlasyncpg://user:passlocalhost/crawlee_db ) as storage_client: crawler ParselCrawler(storage_clientstorage_client) # ... define your handlers and crawling logic await crawler.run([https://crawlee.dev]) if __name__ __main__: asyncio.run(main())作为实验性特性其实现可能在后续版本中随社区反馈而演进。仓库对照存储架构的设计同源统一存储接口的设计理念在当前仓库的 Node.js/TypeScript 实现中同样清晰可见。核心存储抽象集中在 packages/core/src/storages 目录数据集、键值存储、请求队列等 17 个文件而具体的持久化实现文件系统、内存等由 packages/fs-storage/src/file-system-storage.ts 与 packages/core/src/memory-storage 等模块提供。测试用例如 packages/fs-storage/test/key-value-store验证了不同后端在统一 API 下的一致行为。这正是文档所述同一套 API、任意后端在仓库中的真实体现。三、自适应 Playwright 爬虫简单页面走 HTTP复杂页面走浏览器有些网站用普通 HTTP 请求即可快速爬取另一些则需要浏览器渲染动态内容。传统上你必须提前决策使用轻量 HTTP 爬虫ParselCrawler或BeautifulSoupCrawler还是基于浏览器的PlaywrightCrawler。v1.0 引入的AdaptivePlaywrightCrawler自动为每个页面选择合适的方式。工作原理检测与置信度机制自适应爬虫使用检测机制对比同一页面的纯 HTTP 请求结果与浏览器渲染结果。如果两者一致则继续使用更快的 HTTP 方式如果出现差异则回退到浏览器爬取。随着时间推移它会针对不同类型的页面建立哪种渲染方式更合适的置信度并定期用浏览器复核确保预测不偏离实际。这使得爬取更快、成本更低同时仍能可靠处理复杂动态网站——在简单页面上获得速度在 JS 重度网站上获得鲁棒性。实战示例以下示例使用静态 Parsel 解析器处理 HTTP 响应仅在必要时回退到 Playwrightimport asyncio from datetime import timedelta from crawlee.crawlers import AdaptivePlaywrightCrawler, AdaptivePlaywrightCrawlingContext async def main() - None: crawler AdaptivePlaywrightCrawler.with_parsel_static_parser() crawler.router.default_handler async def request_handler(context: AdaptivePlaywrightCrawlingContext) - None: # Locate element h2 within 5 seconds h2 await context.query_selector_one(h2, timedelta(milliseconds5000)) # Do stuff with element found by the selector context.log.info(h2) await crawler.run([https://crawlee.dev/]) if __name__ __main__: asyncio.run(main())在该示例中无需 JavaScript 渲染的页面会走快速 HTTP 客户端其余页面自动交给 Playwright 处理。你不需要编写两套爬虫也不必提前猜测使用哪种方式——Crawlee 动态适配。仓库对照自适应机制的 TypeScript 实现同一设计在当前仓库Node.js/TypeScript 版中已有成熟实现。packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts 定义了AdaptivePlaywrightCrawler及其上下文类型其上下文中的page属性在 HTTP-only 渲染模式下访问时会抛出错误并触发浏览器重试见该文件第 93-105 行的接口注释渲染类型预测器位于 packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts负责判断页面属于静态还是动态渲染。文档所述检测—置信度—定期复核的机制与仓库中RenderingTypePredictor及统计字段如httpOnlyRequestHandlerRuns、browserRequestHandlerRuns、renderingTypeMispredictions见 adaptive-playwright-crawler.ts的设计一脉相承。相关测试见 test/core/crawlers/adaptive_playwright_crawler.test.ts。四、Impit HTTP 客户端Rust 内核的默认 HTTP 客户端v1.0 引入了全新的默认 HTTP 客户端ImpitHttpClient基于 Impit 库构建。Impit 使用Rust 编写、通过绑定暴露给 Python带来更好的性能、异步优先设计、HTTP/3 支持以及**浏览器模拟browser impersonation**能力。它开箱即可模拟真实浏览器使爬虫更难被常见反爬系统识别和拦截——更少的误判、更具韧性的爬取、更少复杂的绕行方案。Impit 同样由 Apify 作为开源项目开发你可以深入研究其内部实现甚至贡献改进。默认情况下 Crawlee 在底层自动使用ImpitHttpClient你也可以自行创建实例并按需配置如启用 HTTP/3 或选择特定浏览器配置再传入爬虫。实战示例显式使用ImpitHttpClient搭配ParselCrawlerimport asyncio from crawlee.crawlers import ParselCrawler, ParselCrawlingContext from crawlee.http_clients import ImpitHttpClient async def main() - None: http_client ImpitHttpClient( # Optional additional keyword arguments for impit.AsyncClient. http3True, browserfirefox, verifyTrue, ) crawler ParselCrawler( http_clienthttp_client, # Limit the crawl to max requests. Remove or increase it for crawling all links. max_requests_per_crawl10, ) # Define the default request handler, which will be called for every request. crawler.router.default_handler async def request_handler(context: ParselCrawlingContext) - None: context.log.info(fProcessing {context.request.url} ...) # Enqueue all links from the page. await context.enqueue_links() # Extract data from the page. data { url: context.request.url, title: context.selector.css(title::text).get(), } # Push the extracted data to the default dataset. await context.push_data(data) # Run the crawler with the initial list of URLs. await crawler.run([https://crawlee.dev]) if __name__ __main__: asyncio.run(main())参数说明http3启用 HTTP/3QUIC协议browser指定模拟的浏览器配置如firefoxverify是否校验 TLS 证书。使用ImpitHttpClient你可以在不增加额外依赖或插件的情况下获得隐蔽性stealth。更多进阶配置可参考官方 HTTP clients 指南。仓库对照HTTP 客户端抽象层在本仓库中HTTP 客户端同样被抽象为可替换组件基础接口定义于 packages/http-client/src/base-http-client.ts基于 Fetch 的实现位于 packages/http-client/src/fetch-http-client.ts而 Impit 客户端在 packages/impit-client/src/index.ts 中提供另有 Got-Scraping 客户端packages/got-scraping-client/src/index.ts可供选择。这种可插拔 HTTP 客户端的设计与 Python 版将 Impit 作为默认客户端并允许自定义的思路完全一致。HTTP 客户端相关指南与示例可参见 docs/guides/http-clients.mdx 及 docs/guides/http-clients 目录下的示例文件。五、Sitemap 请求加载器直接从 sitemap 灌入请求队列许多网站通过 sitemap 暴露站点结构。这些 XML 文件提供了全部可用 URL 的清晰清单通常是发现站点内容最高效的方式。在旧版本中你需要手动抓取并解析这些 XML 文件再喂给爬虫v1.0 不再需要这一步。新的SitemapRequestLoader支持直接从 sitemap 将 URL 加载进请求队列并提供过滤与批量处理选项使大规模爬取尤其 sitemap 已覆盖全站时变得非常简单。实战示例加载 sitemap、只过滤出文档页面并用ParselCrawler处理import asyncio import re from crawlee.crawlers import ParselCrawler, ParselCrawlingContext from crawlee.http_clients import ImpitHttpClient from crawlee.request_loaders import SitemapRequestLoader async def main() - None: # Create an HTTP client for fetching the sitemap. http_client ImpitHttpClient() # Create a sitemap request loader with filtering rules. sitemap_loader SitemapRequestLoader( sitemap_urls[https://crawlee.dev/sitemap.xml], http_clienthttp_client, include[re.compile(r.*docs.*)], # Only include URLs containing docs. max_buffer_size500, # Keep up to 500 URLs in memory before processing. ) # Convert the sitemap loader into a request manager linked # to the default request queue. request_manager await sitemap_loader.to_tandem() # Create a crawler and pass the request manager to it. crawler ParselCrawler( request_managerrequest_manager, max_requests_per_crawl10, # Limit the max requests per crawl. ) crawler.router.default_handler async def handler(context: ParselCrawlingContext) - None: context.log.info(fProcessing {context.request.url}) # New links will be enqueued directly to the queue. await context.enqueue_links() # Extract data using Parsels XPath and CSS selectors. data { url: context.request.url, title: context.selector.xpath(//title/text()).get(), } # Push extracted data to the dataset. await context.push_data(data) await crawler.run() if __name__ __main__: asyncio.run(main())要点说明sitemap_urls一个或多个 sitemap 地址include/excludeURL 过滤规则支持正则或 globmax_buffer_size处理前在内存中保留的最大 URL 数用于流式批量控制to_tandem()将 sitemap 加载器转换为与默认请求队列关联的请求管理器Tandem 模式新发现的链接可直接入队。将SitemapRequestLoader直接与爬虫连接后你完全可以跳过解析 XML 的样板代码专注于数据提取。仓库对照SitemapRequestLoader 的完整实现本仓库的 packages/core/src/storages/sitemap_request_loader.ts 提供了对应的 Node.js/TypeScript 实现。从源码可见其完整选项体系第 21-34 行的 schema 定义sitemapUrls必填、proxyUrl、persistStateKey、signal、timeoutMillis、maxBufferSize默认 200、enqueueStrategy默认SameHostname、include/exclude支持 glob 字符串、{ glob }对象、RegExp、{ regexp }对象、persistenceOptions与httpClient。该文件还实现了流式加载基于 Node.jsTransform流缓冲区满时暂停直到排空见第 96-99 行注释与文档中过滤与批量处理的描述相互印证。底层 sitemap 解析逻辑位于 packages/utils/src/internals/sitemap.ts并有 packages/utils/test/sitemap.test.ts 测试覆盖。相关教程见 docs/guides/request_loaders.mdx 与 docs/guides/request_loaders_sitemap_basic.ts。六、尊重 robots.txt负责任爬取只需一个开关尊重robots.txt是负责任 Web 爬取的重要部分。这份简单文件让网站所有者声明哪些页面不应被自动化代理抓取。v1.0 让遵守这些规则变得极其简单只需在爬虫上设置respect_robots_txt_file选项Crawlee 就会在发出请求前自动检查该文件。这不仅能帮助你构建合规爬虫还能通过跳过被禁止或不相关的页面来节省时间与带宽——登录页、搜索结果页或管理后台通常在robots.txt中被排除Crawlee 会自动处理。实战示例让ParselCrawler遵循 robots 排除标准import asyncio from crawlee.crawlers import ParselCrawler, ParselCrawlingContext async def main() - None: # Create a new crawler instance with robots.txt compliance enabled. crawler ParselCrawler( respect_robots_txt_fileTrue, ) # Define the default request handler. crawler.router.default_handler async def request_handler(context: ParselCrawlingContext) - None: context.log.info(fProcessing {context.request.url}) # Extract the data from website. data { url: context.request.url, title: context.selector.xpath(//title/text()).get(), } # Push extracted data to the dataset. await context.push_data(data) # Run the crawler with the list of start URLs. # The crawler will check the robots.txt file before making requests. # In this example, https://news.ycombinator.com/login will be skipped # because its disallowed in the sites robots.txt file. await crawler.run( [https://news.ycombinator.com/, https://news.ycombinator.com/login] ) if __name__ __main__: asyncio.run(main())启用该选项后你无需手动检查哪些 URL 被允许Crawlee 会替你完成让你专注于爬取逻辑与数据提取。仓库对照robots 解析与 enqueue 策略联动本仓库同样内置 robots 支持packages/utils/src/internals/robots.ts 提供robots.txt的解析与匹配逻辑并有 packages/utils/test/robots.test.ts 测试验证在入队环节packages/core/src/enqueue_links/shared.ts 的EnqueueStrategy枚举中定义了robotsTxt策略第 53 行packages/core/src/enqueue_links/enqueue_links.ts 负责在入队时执行该策略实现自动跳过被 robots.txt 禁止的 URL这一行为。仓库中还提供了完整的 robots 文件示例与爬虫配置示例见 docs/guides/avoid_blocking.mdx 目录下的相关文档。七、浏览器指纹生成让每个会话看起来像真实设备现代网站常依靠浏览器指纹区分真实用户与自动化流量。它们不仅检查 User-Agent 头还会组合数十种细微信号——支持的字体、Canvas 渲染、WebGL 特性、媒体设备、屏幕分辨率等共同形成独特的设备指纹轻易暴露无头浏览器或自动化框架。在没有指纹的情况下Playwright 会话往往看起来千篇一律更容易被反爬系统标记。v1.0 集成FingerprintGenerator自动为每个PlaywrightCrawler会话注入真实、随机的指纹修改 HTTP 头、浏览器 API 及其他底层信号使每次爬取看起来都像真实设备上的真实浏览器。实战示例创建指纹生成器并传入爬虫import asyncio from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext from crawlee.fingerprint_suite import ( DefaultFingerprintGenerator, HeaderGeneratorOptions, ScreenOptions, ) async def main() - None: # Use default fingerprint generator with desired fingerprint options. # Generator will generate real looking browser fingerprint based on the options. # Unspecified fingerprint options will be automatically selected by the generator. fingerprint_generator DefaultFingerprintGenerator( header_optionsHeaderGeneratorOptions(browsers[chrome]), screen_optionsScreenOptions(min_width400), ) crawler PlaywrightCrawler( # Limit the crawl to max requests. Remove or increase it for crawling all links. max_requests_per_crawl10, # Headless mode, set to False to see the browser in action. headlessFalse, # Browser types supported by Playwright. browser_typechromium, # Fingerprint generator to be used. By default no fingerprint generation is done. fingerprint_generatorfingerprint_generator, ) # Define the default request handler, which will be called for every request. crawler.router.default_handler async def request_handler(context: PlaywrightCrawlingContext) - None: context.log.info(fProcessing {context.request.url} ...) # Find a link to the next page and enqueue it if it exists. await context.enqueue_links(selector.morelink) # Run the crawler with the initial list of URLs. await crawler.run([https://news.ycombinator.com/]) if __name__ __main__: asyncio.run(main())配置说明DefaultFingerprintGenerator默认指纹生成器未指定的选项由生成器自动挑选header_optionsHTTP 头生成选项例如browsers[chrome]指定模拟 Chrome 浏览器screen_options屏幕选项例如min_width400限制最小屏幕宽度browser_typechromiumPlaywright 支持的浏览器类型。在这个示例中每个 Playwright 实例都从独特的真实指纹启动。从网站视角看爬虫表现得像真实浏览器会话降低被检测或拦截的概率。仓库对照指纹生成器的类型体系本仓库的浏览器池browser-pool内置了完整的指纹生成抽象packages/browser-pool/src/fingerprinting/types.ts 定义了FingerprintGenerator接口getFingerprint方法、BrowserName枚举chrome/firefox/safari/edge第 23-28 行、OperatingSystemsName枚举linux/macos/windows/android/ios第 49-61 行与DeviceCategorymobile/desktop第 63-73 行并基于fingerprint-generator库生成包含 HTTP 头的完整指纹。指纹注入通过 packages/browser-pool/src/fingerprinting/hooks.ts 的浏览器钩子完成相关测试见 test/browser-pool/ 目录。避免被屏蔽的完整指南见 docs/guides/avoid_blocking.mdxPlaywright/Puppeteer 的具体指纹开关示例见 docs/guides/avoid_blocking_playwright.ts 与 docs/guides/avoid_blocking_playwright_fingerprints_off.ts。八、OpenTelemetry 集成把爬虫接入标准可观测体系在生产环境运行爬虫你往往需要的不仅是日志而是对爬虫行为的可见性——它正在做什么、性能如何、瓶颈在哪里。v1.0 通过CrawlerInstrumentor增加了基础 OpenTelemetry 插桩提供收集爬虫 trace 与 metric 的标准方式。启用 OpenTelemetry 后Crawlee 自动记录请求与响应含耗时、重试与错误资源使用事件内存、并发度、系统快照爬虫、路由与处理器handler的生命周期事件。这些信号可导出到任何 OpenTelemetry 兼容后端如 Jaeger、Prometheus 或 Grafana在其中构建实时仪表盘或分析 trace 以理解爬虫性能。实战示例import asyncio from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.trace import set_tracer_provider from crawlee.crawlers import BasicCrawlingContext, ParselCrawler, ParselCrawlingContext from crawlee.otel import CrawlerInstrumentor from crawlee.storages import Dataset, KeyValueStore, RequestQueue def instrument_crawler() - None: resource Resource.create( { service.name: ExampleCrawler, service.version: 1.0.0, environment: development, } ) # Set up the OpenTelemetry tracer provider and exporter provider TracerProvider(resourceresource) otlp_exporter OTLPSpanExporter(endpointlocalhost:4317, insecureTrue) provider.add_span_processor(SimpleSpanProcessor(otlp_exporter)) set_tracer_provider(provider) # Instrument the crawler with OpenTelemetry CrawlerInstrumentor( instrument_classes[RequestQueue, KeyValueStore, Dataset] ).instrument() async def main() - None: instrument_crawler() crawler ParselCrawler(max_requests_per_crawl100) kvs await KeyValueStore.open() crawler.pre_navigation_hook async def pre_nav_hook(_: BasicCrawlingContext) - None: # Simulate some pre-navigation processing await asyncio.sleep(0.01) crawler.router.default_handler async def handler(context: ParselCrawlingContext) - None: await context.push_data({url: context.request.url}) await kvs.set_value(keyurl, valuecontext.request.url) await context.enqueue_links() await crawler.run([https://crawlee.dev/]) if __name__ __main__: asyncio.run(main())配置流程拆解创建带资源属性服务名、版本、环境的TracerProvider配置 OTLP Span 导出器本例指向localhost:4317insecureTrue表示非 TLS挂载SimpleSpanProcessor并设为全局 tracer provider实例化CrawlerInstrumentor指定要插桩的存储类请求队列、键值存储、数据集并调用instrument()正常编写爬虫逻辑含pre_navigation_hook前置导航钩子运行后 trace 自动上报。配置完成后trace 与 metric 即可通过标准 OpenTelemetry 导出器OTLP、console 或自定义后端输出方便将 Crawlee 接入既有监控管线。仓库对照otel 包的完整实现本仓库中对应能力由 packages/otel 包提供packages/otel/src/instrumentation.ts 实现插桩逻辑packages/otel/src/wrapWithSpan.ts 提供函数级 span 包装packages/otel/src/types.ts 与 packages/otel/src/internal-types.ts 定义类型体系packages/otel/src/constants.ts 定义常量。配套测试覆盖了插桩、补丁方法与 hook 投递等场景见 packages/otel/test/ 与 test/otel/ 目录。追踪与监控的完整文档见 docs/guides/trace-and-monitor-crawlers.mdx配套示例包括 docs/guides/trace_and_monitor_basic.ts、docs/guides/trace_and_monitor_custom.ts、docs/guides/trace_and_monitor_register_hook.ts 与 docs/guides/trace_and_monitor_wrap_with_span.ts。九、小结v1.0 带来的组合拳Crawlee for Python v1.0 的七项新特性彼此配合覆盖了爬虫开发的全链路特性解决的问题一句话价值统一存储客户端存储后端 API 不一致一套 API 切换内存/文件/SQL/平台/自定义后端SQL 存储客户端实验性分布式与持久化SQLite/PostgreSQL 多进程并发访问自适应 Playwright 爬虫静态/动态页面选型简单页走 HTTP、复杂页走浏览器自动切换Impit HTTP 客户端HTTP 客户端性能与隐蔽性Rust 内核、HTTP/3、浏览器模拟开箱即用Sitemap 请求加载器大规模 URL 发现从 sitemap 直接灌入队列并过滤robots.txt 支持合规与效率一个开关自动跳过禁止页面指纹生成反检测每个浏览器会话拥有真实随机指纹OpenTelemetry生产可观测性标准 trace/metric 接入任意后端如果你准备将生产爬虫项目升级到 v1.0建议按以下顺序规划迁移先按 Upgrading to v1 指南处理破坏性变更再评估存储后端本地文件系统仍是多数场景的首选SQL 适合分布式随后按站点特征决定是否启用自适应爬虫与 Impit 默认客户端最后为合规与可观测性分别开启 robots 与 OpenTelemetry。本仓库Crawlee 的 Node.js/TypeScript 实现已在对应模块中落地了几乎全部同源设计可作为理解这些机制底层原理的绝佳参考。【免费下载链接】crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.项目地址: https://gitcode.com/GitHub_Trending/cr/crawlee创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
