官方文档:https://docs.scrapy.org/en/latest/topics/item-pipeline.html
激活pipeline,需要在settings里配置,然而这里配置的pipeline会作用于所有的spider。加入项目中有很多spider在运行。item pipeline的处理就会很麻烦,你可以通过process_item(self,item,spider)中的spider参数来判断是来自哪个爬虫,但是这种方法很冗余。更好的做法是配置spider类中的custom_settings属性。为每一个spider配置不同的pipeline。示例如下:
同时,这里你也会看到custom_settings的用法和用处。
class XiaohuaSpider(scrapy.Spider):
name = 'xiaohua'
custom_settings = {
'ITEM_PIPELINES ':{
'TB.pipelines.TBMongoPipeline':300,
}
}
一 method
1 process_item(self,item,spider)
This method is called for every item pipeline component
2 open_spider(self,spider)
This method is called when the spider is opened.
3 close_spider(self,spider)
4 from_crawler(cls,crawler)
It must return a new instance of the pipeline
二 Item Pipeline example
1 write items to mongodb
import pymongo class MongoPipeline(object): collection_name = 'scrapy_items' def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db @classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
) def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db] def close_spider(self, spider):
self.client.close() def process_item(self, item, spider):
self.db[self.collection_name].insert_one(dict(item))
return item
2 duplicates filter
from scrapy.exceptions import DropItem class DuplicatesPipeline(object): def __init__(self):
self.ids_seen = set() def process_item(self, item, spider):
if item['id'] in self.ids_seen:
raise DropItem("Duplicate item found: %s" % item)
else:
self.ids_seen.add(item['id'])
return item