Scrapy notes
Ставим miniconda3
Ставим scrapy:
Create project:
Crawl and save to json:
Crawl spider:
Add delay between requests:
DOWNLOAD_DELAY = 3
scrapy crawl ptichka_stores -o quotes.json --set DOWNLOAD_DELAY=3
Or else we can specify Globaly in the settings.py DOWNLOAD_DELAY = 3
by default scrapy takes 0.25 seconds delay between 2 requests.
Last updated
