Scrapy Adds Verbatim URL Support and Refines GCS Feed Exports

Scrapy recently introduced support for verbatim URLs to handle non standard web servers and fixed a resource leak in Google Cloud Storage feed exports. These updates improve the robustness of data extraction pipelines when dealing with unconventional URL encoding and cloud storage backends.

A significant update to the Scrapy request engine allows developers to send requests with URLs exactly as they are defined, bypassing the default percent encoding logic. The new verbatim_url meta key, introduced in commit b2d8b06, provides this flexibility for edge cases where a target server expects specific characters that Scrapy or its underlying libraries would usually escape.

Standard behavior in Scrapy involves using the safe_url_string utility from the w3lib library to ensure URLs are RFC compliant. While this is ideal for most modern web infrastructure, some legacy systems or specialized APIs require raw input that does not strictly follow these rules. By setting meta={"verbatim_url": True} in a Request object, the internal _set_url method in scrapy/http/request/__init__.py now skips the usual sanitization step.

This change also has a direct impact on how Scrapy identifies unique requests. Normally, the request fingerprinter canonicalizes URLs to avoid duplicate crawling of the same page reached via different parameter orders or fragments. When a request is marked as verbatim, the fingerprinter skips canonicalization entirely. This ensures that two verbatim requests with different character encodings or parameter orders are treated as distinct entities, preventing the crawler from accidentally filtering out what it perceives as a duplicate.

# Example of using verbatim URLs for a specific request
from scrapy import Request

request = Request(
    url="http://example.com/api?data[0]=raw_value",
    meta={"verbatim_url": True}
)

The feed export system is a core component for data engineers using Scrapy to push scraped items directly into cloud storage. A recent fix in commit 13c1c1f addresses a potential file descriptor leak in the GCSFeedStorage class. Previously, temporary files used during the upload process to Google Cloud Storage were not always closed properly after the transfer was complete.

The update to scrapy/extensions/feedexport.py wraps the upload logic in a try finally block. This ensures that the file handle is closed regardless of whether the upload succeeds or fails. In high volume scraping environments where hundreds of feed exports might trigger concurrently, failing to close these handles could eventually lead to the process hitting the system limit for open files, causing the entire crawler to crash.

This fix is particularly important for long running jobs that use the GCS backend. By ensuring that the file.close() method is called even after an OSError or a network timeout during the blob.upload_from_file call, the project maintains better system stability. Operators should notice more consistent memory and resource usage when running large scale extraction tasks that target Google Cloud Storage buckets.

Debugging complex extraction logic often involves using the open_in_browser() utility to see how Scrapy perceives a specific page. This tool works by taking the response body, injecting a <base> tag to ensure relative links and assets load correctly from the original host, and opening the result in a local web browser.

In commit df2f3d7, the project fixed a logic issue in scrapy/utils/response.py where the utility would fail to inject a base URL if it found a commented out <base> tag earlier in the document. The previous implementation checked for the presence of the string in the raw body before stripping comments. The new approach correctly ignores HTML comments when searching for an existing base tag.

This improvement ensures that developers do not get broken previews when working with sites that have archived or commented out configuration tags. The fix also included an update to ensure that the utility correctly identifies the head section of the document even if a fake head tag exists within a comment. These small but vital refinements to the developer experience reduce friction during the initial phases of spider development and maintenance.

Beyond functional changes, the Scrapy team is actively preparing for future Python releases. Commit 58af57a introduced a workaround for a performance regression in the code coverage tools when running on Python 3.14. By switching the coverage core to ctrace instead of the new sysmon default on that version, the project maintains fast CI build times.

The project also addressed regressions caused by the release of testfixtures 12.0.0. Several tests that relied on specific output formats or behaviors of that library were updated in commit 90deebe. This commitment to testing infrastructure ensures that Scrapy remains a stable foundation for production data pipelines as the broader Python ecosystem evolves.

Operators and developers using Scrapy should consider the following points based on the recent activity:

1. Review any custom middleware that handles URL normalization if you plan to use the new verbatim URL feature.
2. Ensure that your environment has the latest version of the Google Cloud Storage client library to benefit from the improved storage reliability.
3. Verify that your local debugging workflows still function as expected with the improved browser preview logic, especially on sites with complex HTML comment structures.
4. Keep an eye on the transition to newer Python versions as the project continues to refine its compatibility layer for the upcoming release.