Scrapy这个成熟的爬虫框架，用起来之后发现并没有想象中的那么难。即便是在一些小型的项目上，用scrapy甚至比用requests、urllib、urllib2更方便，简单，效率也更高。废话不多说，下面详细介绍下如何用scrapy将妹子图爬下来，存储在你的硬盘之中。关于Python、Scrapy的安装以及scrapy的原理这里就不作介绍，自行google、百度了解学习。
一、开发工具
Pycharm 2017
Python 2.7
Scrapy 1.5.0
requests
二、爬取过程
1、创建mzitu项目
进入"E:\Code\PythonSpider>"目录执行scrapy startproject mzitu命令创建一个爬虫项目：
scrapy startproject mzitu
执行完成后，生产目录文件结果如下：
├── mzitu
│ ├── mzitu
│ │ ├── __init__.py
│ │ ├── items.py
│ │ ├── middlewares.py
│ │ ├── pipelines.py
│ │ ├── settings.py
│ │ └── spiders
│ │ ├── __init__.py
│ │ └── Mymzitu.py
│ └── scrapy.cfg
2、进入mzitu项目，编写修改items.py文件
定义titile，用于存储图片目录的名称
定义img，用于存储图片的url
定义name，用于存储图片的名称
# -*- coding: utf-8 -*- # Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html import scrapy class MzituItem(scrapy.Item):
# define the fields for your item here like:
title = scrapy.Field()
img = scrapy.Field()
name = scrapy.Field()
3、编写修改spiders/Mymzitu.py文件
# -*- coding: utf-8 -*-
import scrapy
from mzitu.items import MzituItem
from lxml import etree
import requests
import sys
reload(sys)
sys.setdefaultencoding('utf8') class MymzituSpider(scrapy.Spider):
def get_urls():
url = 'http://www.mzitu.com'
headers = {}
headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'
r = requests.get(url,headers=headers)
html = etree.HTML(r.text)
urls = html.xpath('//*[@id="pins"]/li/a/@href')
return urls name = 'Mymzitu'
allowed_domains = ['www.mzitu.com']
start_urls = get_urls() def parse(self, response):
item = MzituItem()
#item['title'] = response.xpath('//h2[@class="main-title"]/text()')[0].extract()
item['title'] = response.xpath('//h2[@class="main-title"]/text()')[0].extract().split('（')[0]
item['img'] = response.xpath('//div[@class="main-image"]/p/a/img/@src')[0].extract()
item['name'] = response.xpath('//div[@class="main-image"]/p/a/img/@src')[0].extract().split('/')[-1]
yield item next_url = response.xpath('//div[@class="pagenavi"]/a/@href')[-1].extract()
if next_url:
yield scrapy.Request(next_url, callback=self.parse)
我们要爬取的是妹子图网站“最新”的妹子图片，对应的主url是http://www.mzitu.com，通过查看网页源代码发现每一个图片主题的url在<li>标签中，通过上面代码中get_urls函数可以获取，并且返回一个url列表，这里必须说明一下，用python写爬虫，像re、xpath、Beautiful Soup之类的模块必须掌握一个，否则根本无法下手。这里使用xpath工具来获取url地址，在lxml和scrapy中，都支持使用xpath。
def get_urls():
url = 'http://www.mzitu.com'
headers = {}
headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'
r = requests.get(url,headers=headers)
html = etree.HTML(r.text)
urls = html.xpath('//*[@id="pins"]/li/a/@href')
return urls
name定义爬虫的名称，allowed_domains定义包含了spider允许爬取的域名(domain)列表(list)，start_urls定义了爬取了url列表。
name = 'Mymzitu'
allowed_domains = ['www.mzitu.com']
start_urls = get_urls()
分析图片详情页，获取图片主题、图片url和图片名称，同时获取下一页，循环爬取：
def parse(self, response):
item = MzituItem()
#item['title'] = response.xpath('//h2[@class="main-title"]/text()')[0].extract()
item['title'] = response.xpath('//h2[@class="main-title"]/text()')[0].extract().split('（')[0]
item['img'] = response.xpath('//div[@class="main-image"]/p/a/img/@src')[0].extract()
item['name'] = response.xpath('//div[@class="main-image"]/p/a/img/@src')[0].extract().split('/')[-1]
yield item next_url = response.xpath('//div[@class="pagenavi"]/a/@href')[-1].extract()
if next_url:
yield scrapy.Request(next_url, callback=self.parse)
4、编写修改pipelines.py文件，下载图片
# -*- coding: utf-8 -*- # Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import requests
import os class MzituPipeline(object):
def process_item(self, item, spider):
headers = {
'Referer': 'http://www.mzitu.com/'
}
local_dir = 'E:\\data\\mzitu\\' + item['title']
local_file = local_dir + '\\' + item['name']
if not os.path.exists(local_dir):
os.makedirs(local_dir)
with open(local_file,'wb') as f:
f.write(requests.get(item['img'],headers=headers).content)
return item
5、middlewares.py文件中新增一个RotateUserAgentMiddleware类
class RotateUserAgentMiddleware(UserAgentMiddleware):
def __init__(self, user_agent=''):
self.user_agent = user_agent
def process_request(self, request, spider):
ua = random.choice(self.user_agent_list)
if ua:
request.headers.setdefault('User-Agent', ua)
#the default user_agent_list composes chrome,IE,firefox,Mozilla,opera,netscape
#for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php
user_agent_list = [
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"\
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",\
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",\
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",\
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",\
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",\
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",\
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",\
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",\
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",\
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
6、settings.py设置
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 100
# Disable cookies (enabled by default)
COOKIES_ENABLED = False
DOWNLOADER_MIDDLEWARES = {
'mzitu.middlewares.MzituDownloaderMiddleware': 543,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None,
'mzitu.middlewares.RotateUserAgentMiddleware': 400,
}
7、运行爬虫
进入E:\Code\PythonSpider\mzitu目录，运行scrapy crawl Mymzitu命令启动爬虫：
运行结果及完整代码详见：https://github.com/Eivll0m/PythonSpider/tree/master/mzitu