Scrapy notes

Ставим miniconda3

Getting started with conda — conda 23.7.4.dev56 documentation

Ставим scrapy:

conda install -c conda-forge scrapy

Create project:

scrapy startproject tutorial

Crawl and save to json:

scrapy crawl quotes -o quotes.json

Crawl spider:

scrapy crawl quotes

Add delay between requests:

DOWNLOAD_DELAY = 3

scrapy crawl ptichka_stores -o quotes.json --set DOWNLOAD_DELAY=3

Or else we can specify Globaly in the settings.py DOWNLOAD_DELAY = 3

by default scrapy takes 0.25 seconds delay between 2 requests.