Python Scrapy练习:爬取一周天气预报
2018-06-12 14:46:14 # 爬虫 # Scrapy

Scrapy

Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架。 其可以应用在数据挖掘，信息处理或存储历史数据等一系列的程序中。其最初是为了页面抓取 (更确切来说, 网络抓取 )所设计的， 也可以应用在获取API所返回的数据(例如 Amazon Associates Web Services ) 或者通用的网络爬虫。Scrapy用途广泛，可以用于数据挖掘、监测和自动化测试。 Scrapy 使用了 Twisted异步网络库来处理网络通讯。整体架构大致如下图片.png Scrapy主要包括了以下组件： ○引擎(Scrapy) 用来处理整个系统的数据流, 触发事务(框架核心) ○调度器(Scheduler) 用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址 ○下载器(Downloader) 用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的) ○爬虫(Spiders) 爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面 ○项目管道(Pipeline) 负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。 ○下载器中间件(Downloader Middlewares) 位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。 ○爬虫中间件(Spider Middlewares) 介于Scrapy引擎和爬虫之间的框架，主要工作是处理蜘蛛的响应输入和请求输出。 ○调度中间件(Scheduler Middewares) 介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。

Scrapy运行流程大概如下：

1、引擎从调度器中取出一个链接(URL)用于接下来的抓取 2、引擎把URL封装成一个请求(Request)传给下载器 3、下载器把资源下载下来，并封装成应答包(Response) 4、爬虫解析Response 5、解析出实体（Item）,则交给实体管道进行进一步的处理 6、解析出的是链接（URL）,则把URL交给调度器等待抓取

爬取(http://www.tianqi.com/)任意个城市一周日期、天气、风向、温度、天气图片，并保存为TXT文档、JSON文档，以及存储到Mysql数据库中。

1
2
3
scrapy startproject weather
cd weather
scrapy genspider SZtianqi suzhou.tianqi.com

目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
.
├── scrapy.cfg
└── weather
    ├── __init__.py
    ├── __pycache__
    │   ├── __init__.cpython-36.pyc
    │   ├── items.cpython-36.pyc
    │   ├── pipelines.cpython-36.pyc
    │   └── settings.cpython-36.pyc
    ├── data    #保存数据的文件夹
    │   └── weather.json
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders
        ├── SZtianqi.py
        ├── __init__.py
        └── __pycache__
            ├── SZtianqi.cpython-36.pyc
            └── __init__.cpython-36.pyc
items.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class WeatherItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    date=scrapy.Field()
    week=scrapy.Field()
    img=scrapy.Field()
    temperature=scrapy.Field()
    weather=scrapy.Field()
    wind=scrapy.Field()
SZtianqi.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# -*- coding: utf-8 -*-
import scrapy
from weather.items import WeatherItem

class SztianqiSpider(scrapy.Spider):
    name = 'SZtianqi'
    allowed_domains = ['tianqi.com']
    start_urls = []
    citys=['luoyang','nanyang']
    for city in citys:
        start_urls.append('http://'+city+'.tianqi.com')
    def parse(self, response):
        '''
        筛选信息的函数：
        date = 今日日期
        week = 星期几
        img = 表示天气的图标
        temperature = 当天的温度
        weather = 当天的天气
        wind = 当天的风向
        '''
        items=[]

        sixday=response.xpath('//div[@class="day7"]')
        for i in range(1,8):
            item=WeatherItem()
            item['date']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/b/text()').extract_first()
            item['week']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/span/text()').extract_first()
            item['img']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/img/@src').extract_first()
            h=sixday.xpath('./div[@class="zxt_shuju"]/ul/li[' + str(i) + ']/span/text()').extract_first()
            g=sixday.xpath('./div[@class="zxt_shuju"]/ul/li[' + str(i) + ']/b/text()').extract_first()
            item['temperature']=g+'~'+h+'℃'
            item['weather']=sixday.xpath('./ul[@class="txt txt2"]/li['+str(i)+']/text()').extract_first()
            item['wind']=sixday.xpath('./ul[@class="txt"]/li['+str(i)+']/text()').extract_first()
            items.append(item)
        return items
pipelines.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import os
import requests
import json
import codecs
import pymysql

class WeatherPipeline(object):
    def process_item(self, item, spider):
        '''
        处理每一个从SZtianqi传过来的
        item
        '''
        base_dir=os.getcwd()
        filename=base_dir+'\data\weather.txt'

        with open(filename,'a') as f:
            f.write(item['date']+'\n')
            f.write(item['week']+'\n')
            f.write(item['temperature']+'\n')
            f.write(item['weather']+'\n')
            f.write(item['wind']+'\n')

        #下载图片
        with open(base_dir+'/data/'+item['date']+'.png','wb') as f:
            f.write(requests.get(item['img']).content)

        return item

class W2json(object):
    def process_item(self,item,spider):
        # 爬取的信息保存到json
        base_dir=os.getcwd()
        filename=base_dir+'\data\weather.json'

        with codecs.open(filename,'a') as f:
            line=json.dumps(dict(item),ensure_ascii=False)+'\n'
            f.write(line)

        return item

class W2mysql(object):
    def process_item(self,item,spider):
        #将爬取的信息保存到mysql

        date = item['date']
        week = item['week']
        temperature = item['temperature']
        weather = item['weather']
        wind = item['wind']
        img = item['img']

        connection=pymysql.connect(
            host='localhost',
            user='root',
            passwd='liuxianglai',
            db='scrapyDB',
            charset='utf8mb4',
            cursorclass = pymysql.cursors.DictCursor)

        try:
            with connection.cursor() as cursor:

                sql="""INSERT INTO WEATHER(date,week,temperature,weather,wind,img) VALUES (%s, %s,%s,%s,%s,%s)"""
                cursor.execute(sql,(date,week,temperature,weather,wind,img))

            connection.commit()

        finally:
            connection.close()

        return item

settings.py
1
2
3
4
5
6
7
8
9
BOT_NAME = 'weather'

SPIDER_MODULES = ['weather.spiders']
NEWSPIDER_MODULE = 'weather.spiders'

ITEM_PIPELINES = {'weather.pipelines.W2mysql': 300,
                  'weather.pipelines.WeatherPipeline': 500,
                  'weather.pipelines.W2json': 400}
ROBOTSTXT_OBEY = False#让scrapy不要遵守robot协议，因为默认scrapy遵守robot协议
Mysql创建scrapyDB数据库
1
2
3
4
5
6
7
8
9
10
11
CREATE DATABASE scrapyDB CHARACTER SET 'utf8';

CREATE TABLE weather(
id INT AUTO_INCREMENT,
date char(24),
week char(24),
img char(128),
temperature char(24),
weather char(24),
wind char(24),
PRIMARY KEY(id) )ENGINE=InnoDB DEFAULT CHARSET='utf8'
1
scrapy crawl Sutianqi#运行

图片.png图片.png图片.png

weather.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{"date": "06月12日", "week": "星期二", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "21~34℃", "weather": "晴", "wind": "南风"}
{"date": "06月13日", "week": "星期三", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "20~33℃", "weather": "多云", "wind": "南风"}
{"date": "06月12日", "week": "星期二", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "22~37℃", "weather": "晴", "wind": "南风"}
{"date": "06月14日", "week": "星期四", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "22~33℃", "weather": "多云", "wind": "东南风"}
{"date": "06月13日", "week": "星期三", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~38℃", "weather": "晴", "wind": "南风"}
{"date": "06月15日", "week": "星期五", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "23~34℃", "weather": "阴", "wind": "东南风"}
{"date": "06月14日", "week": "星期四", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~37℃", "weather": "晴", "wind": "东北风"}
{"date": "06月16日", "week": "星期六", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~31℃", "weather": "多云", "wind": "东南风"}
{"date": "06月15日", "week": "星期五", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~35℃", "weather": "晴", "wind": "东风"}
{"date": "06月17日", "week": "星期日", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~32℃", "weather": "多云", "wind": "东风"}
{"date": "06月16日", "week": "星期六", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "24~32℃", "weather": "阴", "wind": "东南风"}
{"date": "06月18日", "week": "星期一", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~33℃", "weather": "多云", "wind": "东风"}
{"date": "06月17日", "week": "星期日", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "22~34℃", "weather": "阴", "wind": "东风"}
{"date": "06月18日", "week": "星期一", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "22~33℃", "weather": "多云", "wind": "东风"}

weather.txt图片.png
Prev
2018-06-12 14:46:14 # 爬虫 # Scrapy
Next