爬取某招聘网站特殊字段的信息

时间:2022-10-30 19:48:40

举例:爬取带有“数据分析”职位的信息

需要材料:Python、

pip install requests
pip install  lxml 
pip install  pandas 

第一步:引用

import time
import requests
from lxml import etree
import pandas as pd
from pandas import DataFrame
from pandas import Series

 

第二步:找到可打开的正确的url

url = 'http://search.51job.com/list/080200,000000,0000,00,9,99,%25E6%2595%25B0%25E6%258D%25AE%25E5%2588%2586%25E6%259E%2590,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=’

第三步:向这个URL发出请求,文本编码修改为中文字符

head = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0'}
s = requests.Session()
res = s.get(url,headers = head)
res.encoding = 'gbk'
root = etree.HTML(res.text)

第四步:在原网址查看所需字段的位置

爬取某招聘网站特殊字段的信息

读取所需数据

position = root.xpath('//div[@class="el"]/p/span/a/@title')
company = root.xpath('//div[@class="el"]/span[@class="t2"]/a/@title')
place = root.xpath('//div[@class="el"]/span[@class="t3"]/text()')
salary = root.xpath('//div[@class="el"]/span[@class="t4"]/text()')
date = root.xpath('//div[@class="el"]/span[@class="t5"]/text()')
job = DataFrame([position,company,place,salary,date]).T
job.columns = ['position','company','place','salary','date']
job['page'] = 1
job.head()

测试运行

爬取某招聘网站特殊字段的信息

封装函数【要找到自增规律】

爬取某招聘网站特殊字段的信息

 

def Crawler_51job(df_origin,n):
    head = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0'}
    s = requests.Session()
    for i in range(1,n+1):
        url = '

'http://search.51job.com/jobsearch/search_result.php?fromJs=1&jobarea=000000%2C00&district=000000&funtype=0000&industrytype=00&issuedate=9&providesalary=99&keyword=%E4%BF%A1%E7%94%A8%E7%AE%A1%E7%90%86&keywordtype=2&curr_page=' + str(i) + '&lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&list_type=0&fromType=14&dibiaoid=0&confirmdate=9'

'
        res = s.get(url,headers = head)
        res.encoding = 'gbk'
        root = etree.HTML(res.text)
        position = root.xpath('//div[@class="el"]/p/span/a/@title')
        company = root.xpath('//div[@class="el"]/span[@class="t2"]/a/@title')
        place = root.xpath('//div[@class="el"]/span[@class="t3"]/text()')
        salary = root.xpath('//div[@class="el"]/span[@class="t4"]/text()')
        date = root.xpath('//div[@class="el"]/span[@class="t5"]/text()')
        df = DataFrame([position,company,place,salary,date]).T
        df.columns = ['position','company','place','salary','date']
        df = DataFrame([position,company,place,salary,date]).T
        df.columns = ['position','company','place','salary','date']
        df['page'] = i
        time.sleep(2)
        df_origin = pd.concat([df_origin,df])
    return(df_origin)

函数调用与数据输出

job = Crawler_51job(job,17)

job = Crawler_51job(job,17)
job.to_csv('51job.csv',index = 0)

结果(excel坏了,用记事本查看一下)

爬取某招聘网站特殊字段的信息

 

 这样就完成了一个爬虫小程序