Scrapy 2.17 Tightens Link Extraction and Item Field Order

Scrapy is a Python web scraping framework, and this week the scrapy/scrapy repo mixed a 2.17.0 release pass with two changes that matter to extraction pipelines: stricter link filtering and stable item field order. The seven day window shows 10 commits and 79 changed files, so the useful signal is in the narrow behavior changes rather than raw volume. For teams that run crawlers as part of data engineering jobs, these are the details that keep queues and exports predictable.

The most operator visible change is in LxmlLinkExtractor. The change that added deny_attrs and deny_tags gives crawler authors a more direct way to reject links by source attribute or source tag. The implementation touched scrapy/linkextractors/lxmlhtml.py, which is the right place for this behavior because it keeps the decision near HTML link discovery rather than later spider code.

For data collection jobs, that matters less as a convenience feature and more as crawl hygiene. Large sites place URLs in tracking attributes, hidden navigation, preload tags, and other page furniture. If a spider can block those candidates at extraction time, the downstream queue sees fewer useless requests. That reduces duplicate normalization work and makes frontier rules easier to audit.

There is still a tradeoff. Filters at this layer are blunt if a site mixes useful and useless URLs in the same tag or attribute pattern. Teams should add the new options after looking at crawl samples, not as a default copied between domains.

Another practical change is the commit that kept Item fields in definition order. It touched scrapy/item.py, a small surface with large effects when scraped records leave Scrapy and become CSV, JSON Lines, parquet staging input, or dataframe rows.

Stable field order is not the same as schema validation, but it removes one source of noise. Many ETL jobs compare exported feeds in tests, generate column lists, or rely on predictable human review. If the item class says sku, name, price, and url, seeing that order in output is easier to reason about than an order shaped by internals.

The important part is restraint. This is not a new item model. It is a behavioral cleanup that makes Scrapy better aligned with how engineers already read item classes. That kind of change rarely looks dramatic in a diff, but it lowers friction around review and regression testing.

The selector work is mostly cleanup, but it is still relevant to scraper maintenance. The project clarified Selector.type and adjusted selector documentation examples in docs/topics/selectors.rst. Selector type is one of those details that only becomes visible when a response is parsed as the wrong thing, or when tests do not describe the expected mode clearly.

The UTF 16 test change is similar. A commit fixed the UTF 16 response test on big endian systems. That does not say the average spider changed behavior this week. It says the test suite was made less tied to one machine model. For a parser library used on many platforms, that is worth doing.

Small parser fixes are easy to skip in release scanning. They matter because scraping fails at boundaries: encoding, malformed HTML, mixed XML and HTML assumptions, and selector context. Better tests here mean fewer false failures for contributors and fewer untested assumptions for users.

This activity also looks like a release pass. Scrapy added the 2.17.0 release notes, bumped the version to 2.17.0, and then updated the tooling pins. The changed files include docs/news.rst, tox.ini, and the pre commit config.

The tooling update is routine, but not empty. The excerpt shows Ruff, Black for docs, Pytest, Pylint, build, Pillow, Protego, and boto stubs moving forward. That matters for contributors because local checks and CI drift less when pins are refreshed together. It also matters for downstream packagers who watch Python version support and build tooling closely.

The release note file was touched three times in the window, which fits the shape of a project settling public documentation around code changes. For operators, that is a signal to read the notes before assuming this is only maintenance. The feature set is small, but it is close to crawler behavior and feed shape.

• Review spiders that use LxmlLinkExtractor on noisy HTML. The new tag and attribute filters are worth testing on crawl samples before they enter shared settings.

• Add one export regression test around important Item classes if feed column order matters to your pipeline. The source order change should help, but tests make the contract explicit.

• Read the 2.17.0 notes before upgrading pinned production crawlers. This week was only 10 commits, but it touched extraction, item behavior, selector docs, and the tool chain behind the project.