第 49 课 | Playwright 进阶反爬对抗与浏览器指纹使用 Playwright 替代 Selenium配合 stealth 插件、代理 IP 池、人类行为模拟应对更复杂的网站反爬机制。一、Playwright vs Selenium为什么迁移维度SeleniumPlaywright速度较慢需显式等待快自动等待元素就绪反检测容易被识别内置多种反检测特性API 设计传统 WebDriver 模式现代 async/await简洁直观移动端不支持原生支持移动端模拟iPhone/Android网络拦截需要额外配置内置 route 拦截可修改请求/响应多页面窗口切换复杂Context 隔离天然支持多标签页安装需要 WebDriver 匹配浏览器版本一条命令安装浏览器迁移成本低Playwright 的 API 设计更现代从 Selenium 迁移通常只需 1-2 小时。二、反爬策略全景 目标网站️ 指纹伪装层UA/Viewport/Locale/Timezone️ Stealth 层隐藏 webdriver/自动化特征 代理层IP 池轮换/地域伪装 行为层随机延迟/鼠标轨迹/滚动✅ 成功获取数据三、浏览器指纹伪装3.1 什么是浏览器指纹网站通过以下信息构建你的「指纹」User-Agent浏览器类型和版本屏幕分辨率时区语言偏好字体列表WebGL 渲染器Canvas 指纹如果指纹与正常浏览器不一致就会被识别为爬虫。3.2 完整指纹配置fromplaywright.sync_apiimportsync_playwrightimportrandom# 真实浏览器 User-Agent 池USER_AGENTS[# Chrome WindowsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36,# Chrome MacMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36,# Edge WindowsMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0,# Firefox WindowsMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0,]# 常见屏幕分辨率VIEWPORTS[{width:1920,height:1080},{width:2560,height:1440},{width:1366,height:768},{width:1440,height:900},]# 中文语言与地区LOCALES[zh-CN,zh-TW,zh-HK]TIMEZONES[Asia/Shanghai,Asia/Hong_Kong,Asia/Taipei]defcreate_stealth_context(browser):创建带随机指纹的浏览器上下文contextbrowser.new_context(viewportrandom.choice(VIEWPORTS),user_agentrandom.choice(USER_AGENTS),localerandom.choice(LOCALES),timezone_idrandom.choice(TIMEZONES),# 额外指纹配置geolocation{latitude:31.2304,longitude:121.4737},# 上海permissions[geolocation],color_schemelight,device_scale_factor1,is_mobileFalse,has_touchFalse,)returncontext四、Stealth 插件隐藏自动化特征4.1 默认情况下 Playwright 仍然可被检测// 网站在浏览器中运行以下检测代码navigator.webdriver// Playwright 默认返回 true4.2 注入反检测脚本definject_stealth_scripts(page):注入反检测脚本隐藏自动化特征# 1. 隐藏 navigator.webdriverpage.add_init_script( Object.defineProperty(navigator, webdriver, { get: () undefined, }); )# 2. 伪造 Chrome 运行时对象page.add_init_script( window.chrome { runtime: {}, loadTimes: function() {}, csi: function() {}, app: {}, }; )# 3. 伪造权限查询page.add_init_script( const originalQuery window.navigator.permissions.query; window.navigator.permissions.query (parameters) ( parameters.name notifications ? Promise.resolve({ state: Notification.permission }) : originalQuery(parameters) ); )# 4. 伪造插件列表page.add_init_script( Object.defineProperty(navigator, plugins, { get: () [1, 2, 3, 4, 5], }); )# 5. 伪造语言列表page.add_init_script( Object.defineProperty(navigator, languages, { get: () [zh-CN, zh, en], }); )五、代理 IP 池importrandomimporttimeclassProxyPool:简易代理 IP 池def__init__(self,proxies:list[str]):self.proxiesproxies self.failed_proxies:dict[str,float]{}# 失败的代理及其冷却时间self.cooldown_seconds300# 失败后冷却 5 分钟defget_proxy(self)-str:获取一个可用代理nowtime.time()available[pforpinself.proxiesifpnotinself.failed_proxiesornowself.failed_proxies[p]]ifnotavailable:# 所有代理都失败了重置冷却self.failed_proxies.clear()availableself.proxiesreturnrandom.choice(available)defmark_failed(self,proxy:str):标记代理失败self.failed_proxies[proxy]time.time()self.cooldown_secondsdefcreate_browser_with_proxy(playwright,proxy_config:dict):使用代理创建浏览器# 代理格式: {server: http://proxy_ip:port, username: user, password: pass}returnplaywright.chromium.launch(headlessTrue,proxyproxy_configifproxy_configelseNone,)六、人类行为模拟importrandomimporttimeimportasynciofromplaywright.async_apiimportPageclassHumanBehavior:模拟人类浏览行为staticmethodasyncdefrandom_delay(min_ms500,max_ms2000):随机延迟模拟人类阅读和思考时间delayrandom.uniform(min_ms,max_ms)/1000awaitasyncio.sleep(delay)staticmethodasyncdefhuman_type(page:Page,selector:str,text:str):模拟人类打字逐字输入带随机间隔elementawaitpage.query_selector(selector)awaitelement.click()forcharintext:awaitpage.keyboard.type(char,delayrandom.randint(50,200))# 偶尔停顿模拟思考ifrandom.random()0.1:awaitasyncio.sleep(random.uniform(0.3,1.0))staticmethodasyncdefhuman_scroll(page:Page):模拟人类滚动不规则的滚动距离和速度for_inrange(random.randint(3,8)):scroll_distancerandom.randint(200,800)awaitpage.evaluate(fwindow.scrollBy(0,{scroll_distance}))awaitasyncio.sleep(random.uniform(1.0,3.0))staticmethodasyncdefhuman_mouse_move(page:Page):模拟鼠标移动轨迹# 从当前位置移动到随机目标xrandom.randint(100,800)yrandom.randint(100,600)awaitpage.mouse.move(x,y,stepsrandom.randint(5,15))七、完整爬取示例fromplaywright.sync_apiimportsync_playwrightdefcrawl_36kr_articles():爬取 36 氪文章教学示例withsync_playwright()asp:browserp.chromium.launch(headlessTrue)contextcreate_stealth_context(browser)pagecontext.new_page()# 注入反检测脚本inject_stealth_scripts(page)# 访问目标页面page.goto(https://36kr.com/,wait_untilnetworkidle)# 等待内容加载page.wait_for_selector(.article-item,timeout10000)# 提取文章列表articlespage.evaluate( () { const items document.querySelectorAll(.article-item); return Array.from(items).slice(0, 10).map(item ({ title: item.querySelector(.article-title)?.innerText || , url: item.querySelector(a)?.href || , summary: item.querySelector(.article-description)?.innerText || , })); } )context.close()browser.close()returnarticles八、运行cdcode/lesson-49-playwright uvsyncuv run playwright_advanced.py九、本课小结Playwright 是 Selenium 的现代替代品API 更简洁、反检测更强四层反爬策略指纹伪装 → Stealth 注入 → 代理轮换 → 行为模拟浏览器指纹包含 UA、分辨率、时区、语言等十余个维度人类行为模拟的关键是「随机性」——固定的延迟反而容易被识别反爬是攻防博弈没有一劳永逸的方案需要持续更新策略配套代码code/lesson-49-playwright/playwright_advanced.py
