Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

时间:2021-09-22 03:02:08

背景说明

感觉微信公众号算得是比较难爬的平台之一,不过一番折腾之后还是小有收获的。没有用Scrapy(估计爬太快也有反爬限制),但后面会开始整理写一些实战出来。简单介绍下本次的开发环境:

  • python3
  • requests
  • psycopg2 (操作postgres数据库)

抓包分析

本次实战对抓取的公众号没有限制,但不同公众号每次抓取之前都要进行分析。打开Fiddler,将手机配置好相关代理,为避免干扰过多,这里给Fiddler加个过滤规则,只需要指定微信域名mp.weixin.qq.com就好:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

Fiddler配置Filter规则

平时关注的公众号也比较多,本次实战以“36氪”公众号为例,继续往下看:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

“36氪”公众号

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

公众号右上角 -> 全部消息

在公众号主页,右上角有三个实心圆点,点击进入消息界面,下滑找到并点击“全部消息”,往下请求加载几次历史文章,然后回到Fiddler界面,不出意外的话应该可以看到这几次请求,可以看到返回的数据是json格式的,同时文章数据是以json字符串的形式定义在general_msg_list字段中:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

公众号文章列表抓包请求

分析文章列表接口

把请求URL和Cookie贴上来进行分析:

  1. https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MzI2NDk5NzA0Mw==&f=json&offset=10&count=10&is_ok=1&scene=126&uin=777&key=777&pass_ticket=QhOypNwH5dAr5w6UgMjyBrTSOdMEUT86vWc73GANoziWFl8xJd1hIMbMZ82KgCpN&wxtoken=&appmsg_token=971_LwY7Z%252BFBoaEv5z8k_dFWfJkdySbNkMR4OmFxNw~~&x5=1&f=json
  2. Cookie: pgv_pvid=2027337976; pgv_info=ssid=s3015512850; rewardsn=; wxtokenkey=777; wxuin=2089823341; devicetype=android-26; version=26070237; lang=zh_CN;pass_ticket=NDndxxaZ7p6Z9PYulWpLqMbI0i3ULFeCPIHBFu1sf5pX2IhkGfyxZ6b9JieSYRUy;wap_sid2=CO3YwOQHEogBQnN4VTNhNmxQWmc3UHI2U3kteWhUeVExZHFVMnN0QXlsbzVJRUJKc1pkdVFUU2Y5UzhSVEtOZmt1VVlYTkR4SEllQ2huejlTTThJWndMQzZfYUw2SldLVGVMQUthUjc3QWdVMUdoaGN0Nml2SU05cXR1dTN2RkhRUVd1V2Y3SFJ5d01BQUF+fjCB1pLcBTgNQJVO

下面把重要的参数说明一下,没提到的说明就不那么重要了:

  • __biz:相当于是当前公众号的id(唯一固定标志)
  • offset:文章数据接口请求偏移量标志(从0开始),每次返回的json数据中会有下一次请求的offset,注意这里并不是按某些规则递增的
  • count:每次请求的数据量(亲测最多可以是10)
  • pass_ticket:可以理解是请求票据,而且隔一段时间后(大概几个小时)就会过期,这也是为什么微信公众号比较难按固定规则进行抓取的原因
  • appmsg_token:同样理解为非固定有过期策略的票据
  • Cookie:使用的时候可以把整段贴上去,但最少仅需要wap_sid2这部分

是不是感觉有点麻烦,毕竟不是要搞大规模专业的爬虫,所以单就一个公众号这么分析下来,还是可以往下继续的,贴上截取的一段json数据,用于设计文章数据表:

  1. {
  2. "ret": 0,
  3. "errmsg": "ok",
  4. "msg_count": 10,
  5. "can_msg_continue": 1,
  6. "general_msg_list": "{\"list\":[{\"comm_msg_info\":{\"id\":1000005700,\"type\":49,\"datetime\":1535100943,\"fakeid\":\"3264997043\",\"status\":2,\"content\":\"\"},\"app_msg_ext_info\":{\"title\":\"金融危机又十年:钱荒之下,二手基金迎来高光时刻\",\"digest\":\"退出永远是基金的主旋律。\",\"content\":\"\",\"fileid\":100034824,\"content_url\":\"http:\\/\\/mp.weixin.qq.com\\/s?__biz=MzI2NDk5NzA0Mw==&mid=2247518479&idx=1&sn=124ab52f7478c1069a6b4592cdf3c5f5&chksm=eaa6d8d3ddd151c5bb95a7ae118de6d080023246aa0a419e1d53bfe48a8d9a77e52b752d9b80&scene=27#wechat_redirect\",\"source_url\":\"\",\"cover\":\"http:\\/\\/mmbiz.qpic.cn\\/mmbiz_jpg\\/QicyPhNHD5vYgdpprkibtnWCAN7l4ZaqibKvopNyCWWLQAwX7QpzWicnQSVfcBZmPrR5YuHS45JIUzVjb0dZTiaLPyA\\/0?wx_fmt=jpeg\",\"subtype\":9,\"is_multi\":0,\"multi_app_msg_item_list\":[],\"author\":\"石亚琼\",\"copyright_stat\":11,\"duration\":0,\"del_flag\":1,\"item_show_type\":0,\"audio_fileid\":0,\"play_url\":\"\",\"malicious_title_reason_id\":0,\"malicious_content_type\":0}}]}",
  7. "next_offset": 20,
  8. "video_count": 1,
  9. "use_video_tab": 1,
  10. "real_type": 0
  11. }

可以简单抽取想要的数据,这里将文章表结构定义如下,顺便贴上建表的SQL语句:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

文章数据表

  1. -- ----------------------------
  2. -- Table structure for tb_article
  3. -- ----------------------------
  4. DROP TABLE IF EXISTS "public"."tb_article";
  5. CREATE TABLE "public"."tb_article" (
  6. "id" serial4 PRIMARY KEY,
  7. "msg_id" int8 NOT NULL,
  8. "title" varchar(200) COLLATE "pg_catalog"."default" NOT NULL,
  9. "author" varchar(20) COLLATE "pg_catalog"."default",
  10. "cover" varchar(500) COLLATE "pg_catalog"."default",
  11. "digest" varchar(200) COLLATE "pg_catalog"."default",
  12. "source_url" varchar(800) COLLATE "pg_catalog"."default",
  13. "content_url" varchar(600) COLLATE "pg_catalog"."default" NOT NULL,
  14. "post_time" timestamp(6),
  15. "create_time" timestamp(6) NOT NULL
  16. )
  17. ;
  18. COMMENT ON COLUMN "public"."tb_article"."id" IS '自增主键';
  19. COMMENT ON COLUMN "public"."tb_article"."msg_id" IS '消息id (唯一)';
  20. COMMENT ON COLUMN "public"."tb_article"."title" IS '标题';
  21. COMMENT ON COLUMN "public"."tb_article"."author" IS '作者';
  22. COMMENT ON COLUMN "public"."tb_article"."cover" IS '封面图';
  23. COMMENT ON COLUMN "public"."tb_article"."digest" IS '关键字';
  24. COMMENT ON COLUMN "public"."tb_article"."source_url" IS '原文地址';
  25. COMMENT ON COLUMN "public"."tb_article"."content_url" IS '文章地址';
  26. COMMENT ON COLUMN "public"."tb_article"."post_time" IS '发布时间';
  27. COMMENT ON COLUMN "public"."tb_article"."create_time" IS '入库时间';
  28. COMMENT ON TABLE "public"."tb_article" IS '公众号文章表';
  29. -- ----------------------------
  30. -- Indexes structure for table tb_article
  31. -- ----------------------------
  32. CREATE UNIQUE INDEX "unique_msg_id" ON "public"."tb_article" USING btree (
  33. "msg_id" "pg_catalog"."int8_ops" ASC NULLS LAST
  34. );

附请求文章接口并解析数据保存到数据库的相关代码:

  1. class WxMps(object):
  2. """微信公众号文章、评论抓取爬虫"""
  3.  
  4. def __init__(self, _biz, _pass_ticket, _app_msg_token, _cookie, _offset=0):
  5. self.offset = _offset
  6. self.biz = _biz # 公众号标志
  7. self.msg_token = _app_msg_token # 票据(非固定)
  8. self.pass_ticket = _pass_ticket # 票据(非固定)
  9. self.headers = {
  10. 'Cookie': _cookie, # Cookie(非固定)
  11. 'User-Agent': 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 '
  12. }
  13. wx_mps = 'wxmps' # 这里数据库、用户、密码一致(需替换成实际的)
  14. self.postgres = pgs.Pgs(host='localhost', port='5432', db_name=wx_mps, user=wx_mps, password=wx_mps)
  15.  
  16. def start(self):
  17. """请求获取公众号的文章接口"""
  18.  
  19. offset = self.offset
  20. while True:
  21. api = 'https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz={0}&f=json&offset={1}' \
  22. '&count=10&is_ok=1&scene=124&uin=777&key=777&pass_ticket={2}&wxtoken=&appmsg_token' \
  23. '={3}&x5=1&f=json'.format(self.biz, offset, self.pass_ticket, self.msg_token)
  24.  
  25. resp = requests.get(api, headers=self.headers).json()
  26. ret, status = resp.get('ret'), resp.get('errmsg') # 状态信息
  27. if ret == 0 or status == 'ok':
  28. print('Crawl article: ' + api)
  29. offset = resp['next_offset'] # 下一次请求偏移量
  30. general_msg_list = resp['general_msg_list']
  31. msg_list = json.loads(general_msg_list)['list'] # 获取文章列表
  32. for msg in msg_list:
  33. comm_msg_info = msg['comm_msg_info'] # 该数据是本次推送多篇文章公共的
  34. msg_id = comm_msg_info['id'] # 文章id
  35. post_time = datetime.fromtimestamp(comm_msg_info['datetime']) # 发布时间
  36. # msg_type = comm_msg_info['type'] # 文章类型
  37. # msg_data = json.dumps(comm_msg_info, ensure_ascii=False) # msg原数据
  38. app_msg_ext_info = msg.get('app_msg_ext_info') # article原数据
  39. if app_msg_ext_info:
  40. # 本次推送的首条文章
  41. self._parse_articles(app_msg_ext_info, msg_id, post_time)
  42. # 本次推送的其余文章
  43. multi_app_msg_item_list = app_msg_ext_info.get('multi_app_msg_item_list')
  44. if multi_app_msg_item_list:
  45. for item in multi_app_msg_item_list:
  46. msg_id = item['fileid'] # 文章id
  47. if msg_id == 0:
  48. msg_id = int(time.time() * 1000) # 设置唯一id,解决部分文章id=0出现唯一索引冲突的情况
  49. self._parse_articles(item, msg_id, post_time)
  50. print('next offset is %d' % offset)
  51. else:
  52. print('Before break , Current offset is %d' % offset)
  53. break
  54. def _parse_articles(self, info, msg_id, post_time):
  55. """解析嵌套文章数据并保存入库"""
  56. title = info.get('title') # 标题
  57. cover = info.get('cover') # 封面图
  58. author = info.get('author') # 作者
  59. digest = info.get('digest') # 关键字
  60. source_url = info.get('source_url') # 原文地址
  61. content_url = info.get('content_url') # 微信地址
  62. # ext_data = json.dumps(info, ensure_ascii=False) # 原始数据
  63. self.postgres.handler(self._save_article(), (msg_id, title, author, cover, digest,
  64. source_url, content_url, post_time,
  65. datetime.now()), fetch=True)
  66. @staticmethod
  67. def _save_article():
  68. sql = 'insert into tb_article(msg_id,title,author,cover,digest,source_url,content_url,post_time,create_time) ' \
  69. 'values(%s,%s,%s,%s,%s,%s,%s,%s,%s)'
  70. return sql
  71. if __name__ == '__main__':
  72. biz = 'MzI2NDk5NzA0Mw==' # "36氪"
  73. pass_ticket = 'NDndxxaZ7p6Z9PYulWpLqMbI0i3ULFeCPIHBFu1sf5pX2IhkGfyxZ6b9JieSYRUy'
  74. app_msg_token = '971_Z0lVNQBcGsWColSubRO9H13ZjrPhjuljyxLtiQ~~'
  75. cookie = 'wap_sid2=CO3YwOQHEogBQnN4VTNhNmxQWmc3UHI2U3kteWhUeVExZHFVMnN0QXlsbzVJRUJKc1pkdVFUU2Y5UzhSVEtOZmt1VVlYTkR4SEllQ2huejlTTThJWndMQzZfYUw2SldLVGVMQUthUjc3QWdVMUdoaGN0Nml2SU05cXR1dTN2RkhRUVd1V2Y3SFJ5d01BQUF+fjCB1pLcBTgNQJVO'
  76. # 以上信息不同公众号每次抓取都需要借助抓包工具做修改
  77. wxMps = WxMps(biz, pass_ticket, app_msg_token, cookie)
  78. wxMps.start() # 开始爬取文章

分析文章评论接口

获取评论的思路大致是一样的,只是会更加麻烦一点。首先在手机端点开一篇有评论的文章,然后查看Fiddler抓取的请求:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

公众号文章评论

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

公众号文章评论接口抓包请求

提取其中的URL和Cookie再次分析:

  1. https://mp.weixin.qq.com/mp/appmsg_comment?action=getcomment&scene=0&__biz=MzI2NDk5NzA0Mw==&appmsgid=2247518723&idx=1&comment_id=433253969406607362&offset=0&limit=100&uin=777&key=777&pass_ticket=NDndxxaZ7p6Z9PYulWpLqMbI0i3ULFeCPIHBFu1sf5pX2IhkGfyxZ6b9JieSYRUy&wxtoken=777&devicetype=android-26&clientversion=26070237&appmsg_token=971_dLK7htA1j8LbMUk8pvJKRlC_o218HEgwDbS9uARPOyQ34_vfXv3iDstqYnq2gAyze1dBKm4ZMTlKeyfx&x5=1&f=json
  2. Cookie: pgv_pvid=2027337976; pgv_info=ssid=s3015512850; rewardsn=; wxuin=2089823341; devicetype=android-26; version=26070237; lang=zh_CN; pass_ticket=NDndxxaZ7p6Z9PYulWpLqMbI0i3ULFeCPIHBFu1sf5pX2IhkGfyxZ6b9JieSYRUy; wap_sid2=CO3YwOQHEogBdENPSVdaS3pHOWc1V2QzY1NvZG9PYk1DMndPS3NfbGlHM0Vfal8zLU9kcUdkWTQxdUYwckFBT3RZM1VYUXFaWkFad3NVaWFXZ28zbEFIQ2pTa1lqZktfb01vcGdPLTQ0aGdJQ2xOSXoxTVFvNUg3SVpBMV9GRU1lbnotci1MWWl5d01BQUF+fjCj45PcBTgNQAE=; wxtokenkey=777

接着分析参数:

  • __biz:同上
  • pass_ticket:同上
  • Cookie:同上
  • offset和limit:代表偏移量和请求数量,由于公众号评论最多展示100条,所以这两个参数也不用改它
  • comment_id:获取文章评论数据的标记id,固定但需要从当前文章结构(Html)解析提取
  • appmsgid:票据id,非固定每次需要从当前文章结构(Html)解析提取
  • appmsg_token:票据token,非固定每次需要从当前文章结构(Html)解析提取

可以看到最后三个参数要解析html获取(当初真的找了好久才想到看文章网页结构)。从文章请求接口可以获得文章地址,对应上面的content_url字段,但请求该地址前仍需要对url做相关处理,不然上面三个参数会有缺失,也就获取不到后面评论内容:

  1. def _parse_article_detail(self, content_url, article_id):
  2. """从文章页提取相关参数用于获取评论,article_id是已保存的文章id"""
  3. try:
  4. api = content_url.replace('amp;', '').replace('#wechat_redirect', '').replace('http', 'https')
  5. html = requests.get(api, headers=self.headers).text
  6. except:
  7. print('获取评论失败' + content_url)
  8. else:
  9. # group(0) is current line
  10. str_comment = re.search(r'var comment_id = "(.*)" \|\| "(.*)" \* 1;', html)
  11. str_msg = re.search(r"var appmsgid = '' \|\| '(.*)'\|\|", html)
  12. str_token = re.search(r'window.appmsg_token = "(.*)";', html)
  13. if str_comment and str_msg and str_token:
  14. comment_id = str_comment.group(1) # 评论id(固定)
  15. app_msg_id = str_msg.group(1) # 票据id(非固定)
  16. appmsg_token = str_token.group(1) # 票据token(非固定)

再回来看该接口返回的json数据,分析结构后然后定义数据表(含SQL):

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

文章评论数据表

  1. -- ----------------------------
  2. -- Table structure for tb_article_comment
  3. -- ----------------------------
  4. DROP TABLE IF EXISTS "public"."tb_article_comment";
  5. CREATE TABLE "public"."tb_article_comment" (
  6. "id" serial4 PRIMARY KEY,
  7. "article_id" int4 NOT NULL,
  8. "comment_id" varchar(50) COLLATE "pg_catalog"."default",
  9. "nick_name" varchar(50) COLLATE "pg_catalog"."default" NOT NULL,
  10. "logo_url" varchar(300) COLLATE "pg_catalog"."default",
  11. "content_id" varchar(50) COLLATE "pg_catalog"."default" NOT NULL,
  12. "content" varchar(3000) COLLATE "pg_catalog"."default" NOT NULL,
  13. "like_num" int2,
  14. "comment_time" timestamp(6),
  15. "create_time" timestamp(6) NOT NULL
  16. )
  17. ;
  18. COMMENT ON COLUMN "public"."tb_article_comment"."id" IS '自增主键';
  19. COMMENT ON COLUMN "public"."tb_article_comment"."article_id" IS '文章外键id';
  20. COMMENT ON COLUMN "public"."tb_article_comment"."comment_id" IS '评论接口id';
  21. COMMENT ON COLUMN "public"."tb_article_comment"."nick_name" IS '用户昵称';
  22. COMMENT ON COLUMN "public"."tb_article_comment"."logo_url" IS '头像地址';
  23. COMMENT ON COLUMN "public"."tb_article_comment"."content_id" IS '评论id (唯一)';
  24. COMMENT ON COLUMN "public"."tb_article_comment"."content" IS '评论内容';
  25. COMMENT ON COLUMN "public"."tb_article_comment"."like_num" IS '点赞数';
  26. COMMENT ON COLUMN "public"."tb_article_comment"."comment_time" IS '评论时间';
  27. COMMENT ON COLUMN "public"."tb_article_comment"."create_time" IS '入库时间';
  28. COMMENT ON TABLE "public"."tb_article_comment" IS '公众号文章评论表';
  29. -- ----------------------------
  30. -- Indexes structure for table tb_article_comment
  31. -- ----------------------------
  32. CREATE UNIQUE INDEX "unique_content_id" ON "public"."tb_article_comment" USING btree (
  33. "content_id" COLLATE "pg_catalog"."default" "pg_catalog"."text_ops" ASC NULLS LAST
  34. );

万里长征快到头了,最后贴上这部分代码,由于要先获取文章地址,所以和上面获取文章数据的代码是一起的:

  1. import json
  2. import re
  3. import time
  4. from datetime import datetime
  5.  
  6. import requests
  7.  
  8. from utils import pgs
  9.  
  10. class WxMps(object):
  11. """微信公众号文章、评论抓取爬虫"""
  12.  
  13. def __init__(self, _biz, _pass_ticket, _app_msg_token, _cookie, _offset=0):
  14. self.offset = _offset
  15. self.biz = _biz # 公众号标志
  16. self.msg_token = _app_msg_token # 票据(非固定)
  17. self.pass_ticket = _pass_ticket # 票据(非固定)
  18. self.headers = {
  19. 'Cookie': _cookie, # Cookie(非固定)
  20. 'User-Agent': 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 '
  21. }
  22. wx_mps = 'wxmps' # 这里数据库、用户、密码一致(需替换成实际的)
  23. self.postgres = pgs.Pgs(host='localhost', port='5432', db_name=wx_mps, user=wx_mps, password=wx_mps)
  24.  
  25. def start(self):
  26. """请求获取公众号的文章接口"""
  27.  
  28. offset = self.offset
  29. while True:
  30. api = 'https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz={0}&f=json&offset={1}' \
  31. '&count=10&is_ok=1&scene=124&uin=777&key=777&pass_ticket={2}&wxtoken=&appmsg_token' \
  32. '={3}&x5=1&f=json'.format(self.biz, offset, self.pass_ticket, self.msg_token)
  33.  
  34. resp = requests.get(api, headers=self.headers).json()
  35. ret, status = resp.get('ret'), resp.get('errmsg') # 状态信息
  36. if ret == 0 or status == 'ok':
  37. print('Crawl article: ' + api)
  38. offset = resp['next_offset'] # 下一次请求偏移量
  39. general_msg_list = resp['general_msg_list']
  40. msg_list = json.loads(general_msg_list)['list'] # 获取文章列表
  41. for msg in msg_list:
  42. comm_msg_info = msg['comm_msg_info'] # 该数据是本次推送多篇文章公共的
  43. msg_id = comm_msg_info['id'] # 文章id
  44. post_time = datetime.fromtimestamp(comm_msg_info['datetime']) # 发布时间
  45. # msg_type = comm_msg_info['type'] # 文章类型
  46. # msg_data = json.dumps(comm_msg_info, ensure_ascii=False) # msg原数据
  47.  
  48. app_msg_ext_info = msg.get('app_msg_ext_info') # article原数据
  49. if app_msg_ext_info:
  50. # 本次推送的首条文章
  51. self._parse_articles(app_msg_ext_info, msg_id, post_time)
  52. # 本次推送的其余文章
  53. multi_app_msg_item_list = app_msg_ext_info.get('multi_app_msg_item_list')
  54. if multi_app_msg_item_list:
  55. for item in multi_app_msg_item_list:
  56. msg_id = item['fileid'] # 文章id
  57. if msg_id == 0:
  58. msg_id = int(time.time() * 1000) # 设置唯一id,解决部分文章id=0出现唯一索引冲突的情况
  59. self._parse_articles(item, msg_id, post_time)
  60. print('next offset is %d' % offset)
  61. else:
  62. print('Before break , Current offset is %d' % offset)
  63. break
  64.  
  65. def _parse_articles(self, info, msg_id, post_time):
  66. """解析嵌套文章数据并保存入库"""
  67.  
  68. title = info.get('title') # 标题
  69. cover = info.get('cover') # 封面图
  70. author = info.get('author') # 作者
  71. digest = info.get('digest') # 关键字
  72. source_url = info.get('source_url') # 原文地址
  73. content_url = info.get('content_url') # 微信地址
  74. # ext_data = json.dumps(info, ensure_ascii=False) # 原始数据
  75.  
  76. content_url = content_url.replace('amp;', '').replace('#wechat_redirect', '').replace('http', 'https')
  77. article_id = self.postgres.handler(self._save_article(), (msg_id, title, author, cover, digest,
  78. source_url, content_url, post_time,
  79. datetime.now()), fetch=True)
  80. if article_id:
  81. self._parse_article_detail(content_url, article_id)
  82.  
  83. def _parse_article_detail(self, content_url, article_id):
  84. """从文章页提取相关参数用于获取评论,article_id是已保存的文章id"""
  85.  
  86. try:
  87. html = requests.get(content_url, headers=self.headers).text
  88. except:
  89. print('获取评论失败' + content_url)
  90. else:
  91. # group(0) is current line
  92. str_comment = re.search(r'var comment_id = "(.*)" \|\| "(.*)" \* 1;', html)
  93. str_msg = re.search(r"var appmsgid = '' \|\| '(.*)'\|\|", html)
  94. str_token = re.search(r'window.appmsg_token = "(.*)";', html)
  95.  
  96. if str_comment and str_msg and str_token:
  97. comment_id = str_comment.group(1) # 评论id(固定)
  98. app_msg_id = str_msg.group(1) # 票据id(非固定)
  99. appmsg_token = str_token.group(1) # 票据token(非固定)
  100.  
  101. # 缺一不可
  102. if appmsg_token and app_msg_id and comment_id:
  103. print('Crawl article comments: ' + content_url)
  104. self._crawl_comments(app_msg_id, comment_id, appmsg_token, article_id)
  105.  
  106. def _crawl_comments(self, app_msg_id, comment_id, appmsg_token, article_id):
  107. """抓取文章的评论"""
  108.  
  109. api = 'https://mp.weixin.qq.com/mp/appmsg_comment?action=getcomment&scene=0&__biz={0}' \
  110. '&appmsgid={1}&idx=1&comment_id={2}&offset=0&limit=100&uin=777&key=777' \
  111. '&pass_ticket={3}&wxtoken=777&devicetype=android-26&clientversion=26060739' \
  112. '&appmsg_token={4}&x5=1&f=json'.format(self.biz, app_msg_id, comment_id,
  113. self.pass_ticket, appmsg_token)
  114. resp = requests.get(api, headers=self.headers).json()
  115. ret, status = resp['base_resp']['ret'], resp['base_resp']['errmsg']
  116. if ret == 0 or status == 'ok':
  117. elected_comment = resp['elected_comment']
  118. for comment in elected_comment:
  119. nick_name = comment.get('nick_name') # 昵称
  120. logo_url = comment.get('logo_url') # 头像
  121. comment_time = datetime.fromtimestamp(comment.get('create_time')) # 评论时间
  122. content = comment.get('content') # 评论内容
  123. content_id = comment.get('content_id') # id
  124. like_num = comment.get('like_num') # 点赞数
  125. # reply_list = comment.get('reply')['reply_list'] # 回复数据
  126.  
  127. self.postgres.handler(self._save_article_comment(), (article_id, comment_id, nick_name, logo_url,
  128. content_id, content, like_num, comment_time,
  129. datetime.now()))
  130.  
  131. @staticmethod
  132. def _save_article():
  133. sql = 'insert into tb_article(msg_id,title,author,cover,digest,source_url,content_url,post_time,create_time) ' \
  134. 'values(%s,%s,%s,%s,%s,%s,%s,%s,%s) returning id'
  135. return sql
  136.  
  137. @staticmethod
  138. def _save_article_comment():
  139. sql = 'insert into tb_article_comment(article_id,comment_id,nick_name,logo_url,content_id,content,like_num,' \
  140. 'comment_time,create_time) values(%s,%s,%s,%s,%s,%s,%s,%s,%s)'
  141. return sql
  142.  
  143. if __name__ == '__main__':
  144. biz = 'MzI2NDk5NzA0Mw==' # "36氪"
  145. pass_ticket = 'NDndxxaZ7p6Z9PYulWpLqMbI0i3ULFeCPIHBFu1sf5pX2IhkGfyxZ6b9JieSYRUy'
  146. app_msg_token = '971_Z0lVNQBcGsWColSubRO9H13ZjrPhjuljyxLtiQ~~'
  147. cookie = 'wap_sid2=CO3YwOQHEogBQnN4VTNhNmxQWmc3UHI2U3kteWhUeVExZHFVMnN0QXlsbzVJRUJKc1pkdVFUU2Y5UzhSVEtOZmt1VVlYTkR4SEllQ2huejlTTThJWndMQzZfYUw2SldLVGVMQUthUjc3QWdVMUdoaGN0Nml2SU05cXR1dTN2RkhRUVd1V2Y3SFJ5d01BQUF+fjCB1pLcBTgNQJVO'
  148. # 以上信息不同公众号每次抓取都需要借助抓包工具做修改
  149. wxMps = WxMps(biz, pass_ticket, app_msg_token, cookie)
  150. wxMps.start() # 开始爬取文章及评论

文末小结

最后展示下数据库里的数据,单线程爬的慢而且又没这方面的数据需求,所以也只是随便试了下手:

Python如何爬取微信公众号文章和评论(基于 Fiddler 抓包分析)

抓取的部分数据

有时候写爬虫是个细心活,如果觉得太麻烦的话,推荐了解下WechatSogou这个工具。有问题的欢迎底部留言讨论。

完整代码:GitHub

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持服务器之家。

原文链接:https://www.jianshu.com/p/b5b01ded8f98