行业资讯
Python反反爬实战：Playwright+指纹抹除绕过网站检测的6个技巧
Python反反爬实战Playwright指纹抹除绕过网站检测的6个技巧前言做量化数据采集绕不开一个问题反爬。东财、同花顺、新浪财经……主流财经网站的反爬机制一个比一个狠。用requests直接请求分分钟被封IP。用Selenium检测到webdrivertrue直接返回空数据。本文分享我在搭建A股数据采集系统时用Playwright指纹抹除绕过网站检测的6个实战技巧。所有代码均在Python 3.10环境下验证通过。## 一、为什么选Playwright而不是Selenium先说结论Selenium已不适合做数据采集。| 维度 | Selenium | Playwright ||------|----------|------------|| 指纹检测 | 容易被识别 | 原生更干净 || 速度 | 较慢 | 快2-3倍 || 网络拦截 | 不支持 | 原生支持 || 异步支持 | 差 | 原生async || 反检测插件 | 需要额外配置 | 生态更成熟 |关键原因Selenium会注入window.navigator.webdriver true几乎所有主流反爬系统都会检测这个字段。Playwright虽然也会注入但通过playwright-stealth插件可以轻松抹除。## 二、环境搭建bashpip install playwright playwright-stealthplaywright install chromium核心依赖就两个-playwright浏览器自动化框架-playwright-stealth指纹抹除插件一站式解决webdriver、chrome属性、语言属性等检测点## 三、6个实战技巧### 技巧1基础指纹抹除这是最基本的操作抹除Playwright的自动化特征pythonfrom playwright.async_api import async_playwrightfrom playwright_stealth import stealth_asyncasync def create_stealth_browser(): pw await async_playwright().start() browser await pw.chromium.launch(headlessFalse) context await browser.new_context( viewport{width: 1920, height: 1080}, user_agentMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 ) page await context.new_page() # 关键注入stealth脚本 await stealth_async(page) return pagestealth_async(page) 一行代码搞定以下检测点- navigator.webdriver 设为 undefined- 伪造 chrome.runtime 属性- 补全 navigator.plugins 数组- 伪造 navigator.languages- 修复 window.chrome 加载时序### 技巧2随机化User-Agent固定UA是最容易被检测的点。建议维护一个UA池每次请求随机选取pythonimport randomUA_POOL [ “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36”, “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36”, “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36”, “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0”,]context await browser.new_context( user_agentrandom.choice(UA_POOL),)**踩坑记录**UA要和浏览器内核匹配。用Chrome的UA但实际跑Firefox内核会被navigator.userAgentData检测出来。### 技巧3Viewport和屏幕分辨率一致化很多反爬系统会检查window.screen和viewport是否一致pythonimport random# 常见分辨率池VIEWPORTS [ {“width”: 1920, “height”: 1080}, {“width”: 1366, “height”: 768}, {“width”: 1536, “height”: 864}, {“width”: 1440, “height”: 900},]viewport random.choice(VIEWPORTS)context await browser.new_context( viewportviewport, screenviewport, # 关键screen和viewport保持一致)**踩坑记录**如果viewport设为1920x1080但screen是1366x768部分反爬脚本会判定为自动化工具。### 技巧4请求间隔随机化行为模拟固定间隔请求是最典型的爬虫特征。模拟人类行为的关键是**随机化**pythonimport asyncioimport randomasync def human_like_delay(): “”“模拟人类操作间隔”“” await asyncio.sleep(random.uniform(0.5, 2.5))async def random_scroll(page): “”“模拟人类随机滚动”“” for _ in range(random.randint(1, 3)): scroll_y random.randint(100, 500) await page.evaluate(fwindow.scrollBy(0, {scroll_y})“) await asyncio.sleep(random.uniform(0.3, 1.0))async def human_click(page, selector): “”“模拟人类点击带偏移”” element await page.query_selector(selector) box await element.bounding_box() if box: # 点击位置加随机偏移 x box[“x”] box[“width”] * random.uniform(0.2, 0.8) y box[“y”] box[“height”] * random.uniform(0.2, 0.8) await page.mouse.click(x, y) await asyncio.sleep(random.uniform(0.3, 0.8))### 技巧5拦截请求屏蔽无用资源加载图片、CSS、字体等资源不仅拖慢速度还会增加被检测的概率资源加载时序异常pythonasync def block_unnecessary_resources(route): “”“屏蔽非必要资源请求”“” if route.request.resource_type in [“image”, “stylesheet”, “font”, “media”]: await route.abort() else: await route.continue_()await page.route(“/*“, block_unnecessary_resources)这一招能把页面加载时间从5秒降到1秒以内同时减少网络指纹特征。### 技巧6Cookie持久化会话复用频繁创建新会话是爬虫的典型特征。正确做法是复用Cookiepythonimport jsonimport osCOOKIE_FILE “cookies.jsonasync def save_cookies(context): “”“保存cookie到本地””” cookies await context.cookies() with open(COOKIE_FILE, “w”) as f: json.dump(cookies, f)async def load_cookies(context): “”“从本地加载cookie”“” if os.path.exists(COOKIE_FILE): with open(COOKIE_FILE, “r”) as f: cookies json.load(f) await context.add_cookies(cookies) return True return False# 使用示例async def main(): page await create_stealth_browser() context page.context # 尝试复用已有cookie if not await load_cookies(context): # 首次访问正常浏览获取cookie await page.goto(“https://example.com”) await human_like_delay() await save_cookies(context) # 后续请求直接复用 await page.goto(“https://example.com/data”)**踩坑记录**Cookie有过期时间建议每次请求后检查是否还有效。东财的Cookie有效期通常是2小时超过后需要重新获取。## 四、完整示例采集东财个股资金流把上面的技巧组合起来采集东方财富个股资金流数据pythonimport asyncioimport jsonfrom playwright.async_api import async_playwrightfrom playwright_stealth import stealth_asyncasync def fetch_money_flow(stock_code“000001”): “”“采集个股资金流数据”“” async with async_playwright() as pw: browser await pw.chromium.launch(headlessTrue) context await browser.new_context( viewport{“width”: 1920, “height”: 1080}, screen{“width”: 1920, “height”: 1080}, user_agent“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36” ) page await context.new_page() await stealth_async(page) # 屏蔽无用资源 await page.route(/*”, lambda route: ( route.abort() if route.request.resource_type in [“image”, “stylesheet”, “font”] else route.continue_() )) # 访问资金流页面 url fhttps://data.eastmoney.com/zjlx/{stock_code}.html await page.goto(url, wait_until“networkidle”) # 等待数据加载 await page.wait_for_selector(“.flow-data”, timeout10000) # 提取数据 data await page.evaluate(“” () { const rows document.querySelectorAll(‘.flow-data tr’); return Array.from(rows).map(row Array.from(row.cells).map(cell cell.textContent.trim()) ); } “”) await browser.close() return data# 运行result asyncio.run(fetch_money_flow(“000001”))print(json.dumps(result, ensure_asciiFalse, indent2))## 五、注意事项1.合规使用采集频率控制在合理范围建议单IP每分钟不超过10次请求2.IP代理大规模采集时建议配合代理池推荐按次计费的短效代理3.验证码遇到验证码说明已被标记需要更换IP或降低频率4.数据时效网站DOM结构可能更新选择器需要定期维护## 六、总结反爬对抗本质是指纹博弈。核心思路就三条1.抹除自动化特征stealth插件一键搞定2.模拟人类行为随机化一切可随机参数3.减少请求特征屏蔽无用资源、复用Cookie这套方案在我实际运行的数据采集系统中稳定运行了3个月日均请求5000次被封率低于0.1%。—作者简介h29hjA股量化交易者专注Python数据采集与量化分析。欢迎关注获取更多实战分享。
郑州网站建设
网页设计
企业官网
