[jnioche]

Related to #158. As the next release is a major one, the DOM-based filters that have a JSoupFilter equivalent are removed rather than deprecated.

Changes

• Removed org.apache.stormcrawler.parse.filter.LDJsonParseFilter and org.apache.stormcrawler.parse.filter.LinkParseFilter. Their JSoupFilter equivalents, org.apache.stormcrawler.jsoup.LDJsonParseFilter and org.apache.stormcrawler.jsoup.LinkParseFilter, are already used by the archetypes. jsoup.LDJsonParseFilter uses a CSS selector and does not need a DOM at all.
• JSONResourceWrapper (OpenSearch) now delegates needsDOM() to the filter it wraps; it used to always return false, so a wrapped filter requiring the DOM would get a null document. Added JSONResourceWrapperTest.
• Removed RobotsTags.extractMetaTags(DocumentFragment), which was not called anywhere (JSoupParserBolt uses the String variant), along with its static XPath expression.
• Removed the LD-JSON entry from test.parsefilters.json and the corresponding test in XPathFilterTest; the JSoup version is covered by JsoupFilterTest and JSoupFiltersTest.
• Docs: removed the two filters from the list of provided ParseFilters. The JSoupFilters were not documented at all: added a JSoup Filters section to internals.adoc (interface, configuration, order relative to the ParseFilters, list of the provided implementations with their syntax), jsoup.filters.config.file to the configuration reference, and a mention in architecture.adoc.

Migration

Users with either filter in parsefilters.json need to move it to jsoupfilters.json with the org.apache.stormcrawler.jsoup class name. The parameters of LDJsonParseFilter are unchanged. The XPath expressions of LinkParseFilter keep working for attributes (e.g. //IMG/@src); element names are lowercased automatically.

parse.filter.XPathFilter is kept for now: its expression syntax differs from the JSoup version (string() and serialize()) and it is the only XPath extraction available to the Tika ParserBolt.

🤖 Generated with Claude Code

[rzo1]

LGTM, and thanks for documenting the JSoupFilters. One question: JSoupFilters only run in JSoupParserBolt, so anyone using the Tika ParserBolt for HTML with the removed LinkParseFilter has no replacement. You kept parse.filter.XPathFilter for that reason; should the migration note say so, or should LinkParseFilter stay for Tika as well? Could the note also mention that RobotsTags.extractMetaTags(DocumentFragment) is gone?

[jnioche]

LGTM, and thanks for documenting the JSoupFilters. One question: JSoupFilters only run in JSoupParserBolt, so anyone using the Tika ParserBolt for HTML with the removed LinkParseFilter has no replacement. You kept parse.filter.XPathFilter for that reason; should the migration note say so, or should LinkParseFilter stay for Tika as well?

The Tika bolt is getting support for JSOUP filters in #2256

[rzo1]

Did another (human) driven pass. It looks good to me. Thanks @jnioche !