Bump scrapy from 2.1.0 to 2.5.0

[dependabot-preview]

Bumps scrapy from 2.1.0 to 2.5.0.

Release notes

Sourced from scrapy's releases.

2.5.0

• Official Python 3.9 support
• Experimental HTTP/2 support
• New get_retry_request() function to retry requests from spider callbacks
• New headers_received signal that allows stopping downloads early
• New Response.protocol attribute

See the full changelog

2.4.1

• Fixed feed exports overwrite support

• Fixed the asyncio event loop handling, which could make code hang

• Fixed the IPv6-capable DNS resolver CachingHostnameResolver for download handlers that call reactor.resolve

• Fixed the output of the genspider command showing placeholders instead of the import part of the generated spider module (issue 4874)

2.4.0

Hihglights:

• Python 3.5 support has been dropped.

• The file_path method of media pipelines can now access the source item.

  This allows you to set a download file path based on item data.

• The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes.

• You can now choose whether feed exports overwrite or append to the output file.

  For example, when using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file.

• Zstd-compressed responses are now supported if zstandard is installed.

• In settings, where the import path of a class is required, it is now possible to pass a class object instead.

See the full changelog

2.3.0

Hihglights:

• Feed exports now support Google Cloud Storage as a storage backend

• The new FEED_EXPORT_BATCH_ITEM_COUNT setting allows to deliver output items in batches of up to the specified number of items.

  It also serves as a workaround for delayed file delivery, which causes Scrapy to only start item delivery after the crawl has finished when using certain storage backends (S3, FTP, and now GCS).

• The base implementation of item loaders has been moved into a separate library, itemloaders, allowing usage from outside Scrapy and a separate release schedule

... (truncated)

Changelog

Sourced from scrapy's changelog.

Scrapy 2.5.0 (2021-04-06)

Highlights:

• Official Python 3.9 support
• Experimental HTTP/2 support <http2>
• New ~scrapy.downloadermiddlewares.retry.get_retry_request function to retry requests from spider callbacks
• New ~scrapy.signals.headers_received signal that allows stopping downloads early
• New Response.protocol <scrapy.http.Response.protocol> attribute

Deprecation removals

• Removed all code that was deprecated in 1.7.0 <1.7-deprecations> and had not already been removed in 2.4.0 <2.4-deprecation-removals>. (4901)
• Removed support for the SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE environment variable, deprecated in 1.8.0 <1.8-deprecations>. (4912)

Deprecations

• The scrapy.utils.py36 module is now deprecated in favor of scrapy.utils.asyncgen. (4900)

New features

• Experimental HTTP/2 support <http2> through a new download handler that can be assigned to the https protocol in the DOWNLOAD_HANDLERS setting. (1854, 4769, 5058, 5059, 5066)

• The new scrapy.downloadermiddlewares.retry.get_retry_request function may be used from spider callbacks or middlewares to handle the retrying of a request beyond the scenarios that ~scrapy.downloadermiddlewares.retry.RetryMiddleware supports. (3590, 3685, 4902)

• The new ~scrapy.signals.headers_received signal gives early access to response headers and allows stopping downloads <topics-stop-response-download>. (1772, 4897)

• The new Response.protocol <scrapy.http.Response.protocol> attribute gives access to the string that identifies the protocol used to download a response. (4878)

• Stats <topics-stats> now include the following entries that indicate the number of successes and failures in storing feeds <topics-feed-exports>:

  feedexport/success_count/<storage type>
  feedexport/failed_count/<storage type>
  

  Where <storage type> is the feed storage backend class name, such as ~scrapy.extensions.feedexport.FileFeedStorage or ~scrapy.extensions.feedexport.FTPFeedStorage.

  (3947, 4850)

• The ~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware spider middleware now logs ignored URLs with INFO logging level <levels> instead of DEBUG, and it now includes the following entry into stats <topics-stats> to keep track of the number of ignored URLs:

  urllength/request_ignored_count
  

  (5036)

• The ~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware downloader middleware now logs the number of decompressed responses and the total count of resulting bytes:

  httpcompression/response_bytes
  httpcompression/response_count
  

... (truncated)

Commits

• e63188c Bump version: 2.4.1 → 2.5.0
• a71d6ef 2.5.0 release notes (#5028)
• 099fb6e Merge pull request #5076 from wRAR/qualname
• 9e3b868 Use qualname in middleware handling.
• 8c5a3a5 Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg
• 9e7cbc0 Fix type for urlencode's doseq argument
• 127a858 Merge pull request #5063 from noviluni/update_usageError_message
• 12f2006 Merge pull request #4799 from GeorgeA92/patch-2
• f0c8d31 Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta
• cc095aa Windows pip installation guide (#4736)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
• @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

• Update frequency (including time of day and day of week)
• Pull request limits (per update run and/or open at any time)
• Out-of-range updates (receive only lockfile updates, if desired)
• Security updates (receive only security updates, if desired)