Scrapy Adds HTTP/2 Support and TLS Version Refinements

Scrapy continues to evolve as the primary framework for large scale web scraping in the Python ecosystem. Recent updates show a clear focus on modernizing the underlying network stack and adopting contemporary features. By integrating support for HTTP/2 and refining how security protocols are handled, the framework ensures it can interact with modern web servers efficiently. These changes also include improvements to the developer experience through better type hints and the adoption of dataclasses for data modeling. The project maintainers are prioritizing stability and performance as the web moves toward more sophisticated protocols.

One of the most significant recent changes is the addition of support for HTTP/2 and SOCKS proxies within the HttpxDownloadHandler. This update is tracked in commit https://github.com/scrapy/scrapy/commit/4e956bd2de5e319bebad2d603a2f5ee34d9d2ffb. Historically, Scrapy has relied heavily on the Twisted library for its networking needs. While Twisted is robust and has served the project well for over a decade, the rise of the httpx library has provided a more modern alternative that natively supports many newer web standards.

HTTP/2 offers several advantages over the older HTTP/1.1 protocol that are critical for high volume data extraction. The most notable feature is multiplexing, which allows multiple requests and responses to be sent over a single TCP connection simultaneously. In a web scraping context, this reduces the overhead of establishing new connections and helps avoid bottlenecks caused by head of line blocking. By supporting HTTP/2 through the httpx based handler, Scrapy users can now extract data from high performance websites with much greater efficiency and lower latency.

Security and transport layer encryption are areas where Scrapy is making important strides. Commit https://github.com/scrapy/scrapy/commit/d9e2f5fbf7da59ea61595cffea5e3399fb9c3574 introduces new settings for configuring the minimum and maximum versions of TLS. This replaces the older approach of specifying a single TLS method. In the modern web, servers often support a range of TLS versions, and forcing a specific version can lead to connection failures when a server upgrades its security configuration.

By allowing developers to define a version range, Scrapy can negotiate the best possible security protocol with the target server. This is important for maintainers of long term scraping projects who need their crawlers to remain functional as internet security standards evolve. It also allows for stricter security postures where a developer might want to disable older, insecure versions of TLS while still allowing for future versions as they become available. As browsers like Chrome and Firefox deprecate TLS 1.0 and 1.1, it is essential for scrapers to follow suit to maintain connectivity.

For many years, the standard way to define data structures in Scrapy was using the scrapy.Item class. While this provided a dictionary like interface that was familiar to many, it lacked the modern features found in newer Python versions. Commit https://github.com/scrapy/scrapy/commit/5149e2c67993c642a384072ac43157491a3b2052 marks a shift in the documentation, switching many examples from scrapy.Item to Python dataclasses.

Dataclasses, introduced in Python 3.7, provide a more concise and readable way to define data containers. They offer automatic generation of methods for initialization and representation. More importantly, they integrate seamlessly with modern editors and static analysis tools. This means that developers can get better autocompletion and type checking when working with their scraped data. It reduces the likelihood of errors caused by typos in field names and makes the code base easier to maintain over time. The move toward dataclasses also aligns Scrapy with the broader Python ecosystem where these structures have become standard.

Web scraping often involves dealing with websites that do not strictly follow internet standards. Commit https://github.com/scrapy/scrapy/commit/b2d8b06be61df40fd652af2f3fa8238a82b8645d introduces support for sending requests with unsafe URLs. Some servers use characters in their URLs that are technically forbidden by standard specifications but are nonetheless required to access certain pages. By allowing Scrapy to handle these edge cases, the framework becomes even more capable of extracting data from the messy reality of the live web without crashing.

Other recent updates focus on resource management and observability. For example, commit https://github.com/scrapy/scrapy/commit/13c1c1faf81100394cffc6cf088c18c315f09ede ensures that temporary files used during Google Cloud Storage uploads are properly closed. This prevents file descriptor leaks, which can be a critical issue for long running crawlers that process thousands of items in a single session. In another update, commit https://github.com/scrapy/scrapy/commit/d2290c35c23b9fe1973764f75a5f7cb72033108a allows developers to configure the log level of retry give up messages. Finally, commit https://github.com/scrapy/scrapy/commit/d59f9b644af2adcdd9842dc4ffc7505335427a7e improves settings type hints, making the source code more accessible.

The steady pace of updates to Scrapy shows that it remains a vital project for the web scraping community. The move toward httpx and HTTP/2 support suggests that the framework is preparing for a future where high performance networking is the standard. Developers should watch for further integration of asynchronous features and more refinements to the item pipeline as dataclasses become more prevalent. As the web becomes more complex, the ability of Scrapy to handle non standard URLs and sophisticated security configurations will be paramount for any serious crawler.

For those who depend on Scrapy for their data needs, these recent commits provide a clear signal that the framework is being actively maintained and modernized. You can follow the ongoing development on the official Scrapy repository at https://github.com/scrapy/scrapy. As more websites adopt HTTP/3 and newer encryption standards, we can expect the Scrapy team to continue their work in bridging the gap between raw web protocols and high level data extraction tools.