Python Scrapy练习:爬取一周天气预报
2018-06-12 14:46:14

Scrapy

Scrapy是一个为了爬取网站数据,提取结构性数据而编写的应用框架。 其可以应用在数据挖掘,信息处理或存储历史数据等一系列的程序中。其最初是为了页面抓取 (更确切来说, 网络抓取 )所设计的, 也可以应用在获取API所返回的数据(例如 Amazon Associates Web Services ) 或者通用的网络爬虫。Scrapy用途广泛,可以用于数据挖掘、监测和自动化测试。 Scrapy 使用了 Twisted异步网络库来处理网络通讯。整体架构大致如下图片.png Scrapy主要包括了以下组件: ○引擎(Scrapy) 用来处理整个系统的数据流, 触发事务(框架核心) ○调度器(Scheduler) 用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL(抓取网页的网址或者说是链接)的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址 ○下载器(Downloader) 用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的) ○爬虫(Spiders) 爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面 ○项目管道(Pipeline) 负责处理爬虫从网页中抽取的实体,主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后,将被发送到项目管道,并经过几个特定的次序处理数据。 ○下载器中间件(Downloader Middlewares) 位于Scrapy引擎和下载器之间的框架,主要是处理Scrapy引擎与下载器之间的请求及响应。 ○爬虫中间件(Spider Middlewares) 介于Scrapy引擎和爬虫之间的框架,主要工作是处理蜘蛛的响应输入和请求输出。 ○调度中间件(Scheduler Middewares) 介于Scrapy引擎和调度之间的中间件,从Scrapy引擎发送到调度的请求和响应。

Scrapy运行流程大概如下:

1、引擎从调度器中取出一个链接(URL)用于接下来的抓取 2、引擎把URL封装成一个请求(Request)传给下载器 3、下载器把资源下载下来,并封装成应答包(Response) 4、爬虫解析Response 5、解析出实体(Item),则交给实体管道进行进一步的处理 6、解析出的是链接(URL),则把URL交给调度器等待抓取

爬取(http://www.tianqi.com/)任意个城市一周日期、天气、风向、温度、天气图片,并保存为TXT文档、JSON文档,以及存储到Mysql数据库中。

1
2
3
scrapy startproject weather
cd weather
scrapy genspider SZtianqi suzhou.tianqi.com

目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
.
├── scrapy.cfg
└── weather
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-36.pyc
│ ├── items.cpython-36.pyc
│ ├── pipelines.cpython-36.pyc
│ └── settings.cpython-36.pyc
├── data #保存数据的文件夹
│ └── weather.json
├── items.py
├── middlewares.py
├── pipelines.py
├── settings.py
└── spiders
├── SZtianqi.py
├── __init__.py
└── __pycache__
├── SZtianqi.cpython-36.pyc
└── __init__.cpython-36.pyc
items.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class WeatherItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
date=scrapy.Field()
week=scrapy.Field()
img=scrapy.Field()
temperature=scrapy.Field()
weather=scrapy.Field()
wind=scrapy.Field()
SZtianqi.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# -*- coding: utf-8 -*-
import scrapy
from weather.items import WeatherItem

class SztianqiSpider(scrapy.Spider):
name = 'SZtianqi'
allowed_domains = ['tianqi.com']
start_urls = []
citys=['luoyang','nanyang']
for city in citys:
start_urls.append('http://'+city+'.tianqi.com')
def parse(self, response):
'''
筛选信息的函数:
date = 今日日期
week = 星期几
img = 表示天气的图标
temperature = 当天的温度
weather = 当天的天气
wind = 当天的风向
'''
items=[]

sixday=response.xpath('//div[@class="day7"]')
for i in range(1,8):
item=WeatherItem()
item['date']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/b/text()').extract_first()
item['week']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/span/text()').extract_first()
item['img']=sixday.xpath('./ul[@class="week"]/li['+str(i)+']/img/@src').extract_first()
h=sixday.xpath('./div[@class="zxt_shuju"]/ul/li[' + str(i) + ']/span/text()').extract_first()
g=sixday.xpath('./div[@class="zxt_shuju"]/ul/li[' + str(i) + ']/b/text()').extract_first()
item['temperature']=g+'~'+h+'℃'
item['weather']=sixday.xpath('./ul[@class="txt txt2"]/li['+str(i)+']/text()').extract_first()
item['wind']=sixday.xpath('./ul[@class="txt"]/li['+str(i)+']/text()').extract_first()
items.append(item)
return items
pipelines.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import os
import requests
import json
import codecs
import pymysql

class WeatherPipeline(object):
def process_item(self, item, spider):
'''
处理每一个从SZtianqi传过来的
item
'''
base_dir=os.getcwd()
filename=base_dir+'\data\weather.txt'

with open(filename,'a') as f:
f.write(item['date']+'\n')
f.write(item['week']+'\n')
f.write(item['temperature']+'\n')
f.write(item['weather']+'\n')
f.write(item['wind']+'\n')

#下载图片
with open(base_dir+'/data/'+item['date']+'.png','wb') as f:
f.write(requests.get(item['img']).content)

return item

class W2json(object):
def process_item(self,item,spider):
# 爬取的信息保存到json
base_dir=os.getcwd()
filename=base_dir+'\data\weather.json'

with codecs.open(filename,'a') as f:
line=json.dumps(dict(item),ensure_ascii=False)+'\n'
f.write(line)

return item

class W2mysql(object):
def process_item(self,item,spider):
#将爬取的信息保存到mysql

date = item['date']
week = item['week']
temperature = item['temperature']
weather = item['weather']
wind = item['wind']
img = item['img']

connection=pymysql.connect(
host='localhost',
user='root',
passwd='liuxianglai',
db='scrapyDB',
charset='utf8mb4',
cursorclass = pymysql.cursors.DictCursor)

try:
with connection.cursor() as cursor:

sql="""INSERT INTO WEATHER(date,week,temperature,weather,wind,img) VALUES (%s, %s,%s,%s,%s,%s)"""
cursor.execute(sql,(date,week,temperature,weather,wind,img))

connection.commit()

finally:
connection.close()

return item

settings.py
1
2
3
4
5
6
7
8
9
BOT_NAME = 'weather'

SPIDER_MODULES = ['weather.spiders']
NEWSPIDER_MODULE = 'weather.spiders'

ITEM_PIPELINES = {'weather.pipelines.W2mysql': 300,
'weather.pipelines.WeatherPipeline': 500,
'weather.pipelines.W2json': 400}
ROBOTSTXT_OBEY = False#让scrapy不要遵守robot协议,因为默认scrapy遵守robot协议
Mysql创建scrapyDB数据库
1
2
3
4
5
6
7
8
9
10
11
CREATE DATABASE scrapyDB CHARACTER SET 'utf8';

CREATE TABLE weather(
id INT AUTO_INCREMENT,
date char(24),
week char(24),
img char(128),
temperature char(24),
weather char(24),
wind char(24),
PRIMARY KEY(id) )ENGINE=InnoDB DEFAULT CHARSET='utf8'
1
scrapy crawl Sutianqi#运行

图片.png图片.png图片.png

weather.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{"date": "06月12日", "week": "星期二", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "21~34℃", "weather": "晴", "wind": "南风"}
{"date": "06月13日", "week": "星期三", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "20~33℃", "weather": "多云", "wind": "南风"}
{"date": "06月12日", "week": "星期二", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "22~37℃", "weather": "晴", "wind": "南风"}
{"date": "06月14日", "week": "星期四", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "22~33℃", "weather": "多云", "wind": "东南风"}
{"date": "06月13日", "week": "星期三", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~38℃", "weather": "晴", "wind": "南风"}
{"date": "06月15日", "week": "星期五", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "23~34℃", "weather": "阴", "wind": "东南风"}
{"date": "06月14日", "week": "星期四", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~37℃", "weather": "晴", "wind": "东北风"}
{"date": "06月16日", "week": "星期六", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~31℃", "weather": "多云", "wind": "东南风"}
{"date": "06月15日", "week": "星期五", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b0.png", "temperature": "24~35℃", "weather": "晴", "wind": "东风"}
{"date": "06月17日", "week": "星期日", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~32℃", "weather": "多云", "wind": "东风"}
{"date": "06月16日", "week": "星期六", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "24~32℃", "weather": "阴", "wind": "东南风"}
{"date": "06月18日", "week": "星期一", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "23~33℃", "weather": "多云", "wind": "东风"}
{"date": "06月17日", "week": "星期日", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b2.png", "temperature": "22~34℃", "weather": "阴", "wind": "东风"}
{"date": "06月18日", "week": "星期一", "img": "http://pic9.tianqijun.com/static/wap2018/ico1/b1.png", "temperature": "22~33℃", "weather": "多云", "wind": "东风"}

weather.txt图片.png