我scrapy在python脚本中运行
scrapy
def setup_crawler(domain): dispatcher.connect(stop_reactor, signal=signals.spider_closed) spider = ArgosSpider(domain=domain) settings = get_project_settings() crawler = Crawler(settings) crawler.configure() crawler.crawl(spider) crawler.start() reactor.run()
它成功运行并停止,但结果在哪里?我想要json格式的结果,该怎么办?
result = responseInJSON
就像我们使用命令一样
scrapy crawl argos -o result.json -t json
您需要手动设置FEED_FORMAT和FEED_URI设置:
FEED_FORMAT
FEED_URI
settings.overrides['FEED_FORMAT'] = 'json' settings.overrides['FEED_URI'] = 'result.json'
如果要将结果放入变量中,可以定义一个Pipeline将项目收集到列表中的类。使用spider_closed信号处理程序查看结果:
Pipeline
spider_closed
import json from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy import log, signals from scrapy.utils.project import get_project_settings class MyPipeline(object): def process_item(self, item, spider): results.append(dict(item)) results = [] def spider_closed(spider): print results # set up spider spider = TestSpider(domain='mydomain.org') # set up settings settings = get_project_settings() settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1} # set up crawler crawler = Crawler(settings) crawler.signals.connect(spider_closed, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) # start crawling crawler.start() log.start() reactor.run()
仅供参考,看看Scrapy如何解析命令行参数。