The source code you can found here. It is available as part of scrapy-feedstreaming project.

You can install scrapy-feedstreaming from PyPi:

python3 -m pip install -U scrapy-feedstreaming

You should also have Scrapy installed.

This is based on scrapy.extensions.feedexport.FeedExporter.

I want to focus on my use-case first.

---

I am scraping some big site, it takes many dozens of hours to finish some logical chunk of scraping. There are also sporadic failures. I have retry mechanism, but sometimes, it takes couple of hours till couple of days until the site is fixed (there are also some SSL-related and other issues that retry does solve).

#libraries #python #framework #scrapy