Repository navigation
Conversation
e4bc5ad to
dbccf97
Compare
28c304c to
66fb544
Compare
bhabegger
changed the title
thomasmueller
left a comment
thomasmueller
left a comment
There was a problem hiding this comment.
Just some bikeshedding: I wonder if we should use the term "Lucene 9" at all.
bhabegger
changed the title
f0a7199 to
de83bda
Compare
7cbee48 to
5b36b34
Compare
bhabegger
changed the title
|
5b36b34 to
c335365
Compare
amit-jain
left a comment
amit-jain
left a comment
There was a problem hiding this comment.
I think there are some architectural gaps which we should address, some are even mentioned in the readme as well and could be addressed with these changes.
Mainly we should reuse the FulltextIndexEditor and FulltextIndexTracker from the oak-search module which the elastic module also reuses and extends.
|
Another important missing piece that is needed is the local fs index file support with the IndexCopier etc class which then get called with https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/reader/DefaultIndexReaderFactory.java#L97 |
7c648cd to
a82839a
Compare
Can you remind me what this is for ? Is this a must for this PR or can it be moved to later ? |
https://jackrabbit.apache.org/oak/docs/query/lucene.html#copyonread |
Ok, so this is in fact a performance optimization in the case the NodeStore is remote (many cases I agree). This doesn't block functional testing though (unless performance is so degraded that this becomes just not usable at all even restricted to testing). I would accept this for now, and leave open to solve the performance issue in maybe some way different than how it was managed in the legacy. Ok, with you to differ ? |
402d45c to
605a3a6
Compare
FYI, last commit adds CopyOnRead |
819a148 to
aa4ba72
Compare
|
Tested locally : 
|
Migrate the standalone LuceneNg tests to JUnit 5. Tests that extend JUnit 4 base classes from other modules stay on JUnit 4; both run in the same build. Temporary directories that were created under target keep that location. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Undeclared and String properties index every non-binary value as a string term, including Date, Path, Reference, WeakReference, URI and Decimal values. Boolean, Name, Path, Reference/WeakReference and URI definitions index only matching values and strings. Decimal definitions index and query Double points. A declared Binary type produces no property field. The accepted value types are documented in the README and covered case by case in PropertyTypeIndexingTest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ed type Long, Date, Double and Decimal properties sort numerically; all other declared types sort by the indexed string form. Single- and multi-valued properties now write the same doc-values type (SORTED_NUMERIC or SORTED_SET), so documents mixing both are no longer rejected by Lucene and dropped, and a multi-valued property sorts by its smallest value. Ordered Decimal, Name, Path, Reference, WeakReference and URI properties can now be sorted. Existing indexes with ordered properties need reindexing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merges write segment files through the index definition's NodeBuilder, which is not thread-safe; as in oak-lucene for repository-backed directories, use SerialMergeScheduler.
… not queries Query planning no longer refreshes the tracker with the querying session's root, which serialized all queries and could reopen older index versions. As in oak-lucene, the provider observes repository changes through a BackgroundObserver.
Index files were split into hard-coded 32 KB blobs, 32 times more than oak-lucene's default for repository-backed directories. Files now use the definition's blobSize (about 1 MB by default) and must record it; there is no implicit fallback size.
Query translation moves from LuceneNgIndex to LuceneNgQueryBuilder and requires the planner's result: only planner-validated restrictions become clauses, typed by their declared definition. Exclusive bounds beyond Long.MIN/MAX_VALUE now match nothing. Query behaviour is covered end to end by the shared comparison fixture, translation edge cases by unit tests.
As in oak-lucene, a query that outlives an index refresh restarts on the new index version and the shared FulltextPathCursor skips rows already returned, instead of continuing from a position in the previous version. Read limits, the size estimate, excerpts and facets follow the shared cursor's behaviour.
… in LuceneNg When the shared planner evaluates a relative property such as jcr:content/keyword on descendant documents of an nt:base rule, LuceneNg now queries the mapped field, adjusts path restrictions and returns the transformed ancestor paths, as oak-lucene does, instead of returning no results.
…is requested Fulltext queries highlighted every result row whether or not the query selected an excerpt. As in oak-lucene, excerpts are generated only for requested rep:excerpt columns.
…tor term Documents index their ancestor paths, as in oak-lucene, so a descendant restriction is a single term lookup instead of a prefix query walking every path below it. Existing LuceneNg indexes need reindexing.
…ync lane LuceneNg has no near-real-time index, so writing during user commits put a second writer on the repository-backed index. Definitions with an async lane are now indexed only by the async indexer; Oak reports the missing nrt/sync provider.
Lucene rejects empty facet labels, which made the whole document fail and left the node unindexed or stale. As in oak-lucene, empty facet values are skipped.
…as String Lucene 9 rejects a field whose shape differs between documents, so a property declared with incompatible types in two indexing rules (or explicitly and through a regex definition) dropped the nodes of one rule. Such properties are now indexed and queried as String in all rules, with a warning naming the property and rules.
The writer no longer downloads the whole index before an incremental indexing run. Files already on local disk are read locally; others are read from the repository and copied in the background, as the reader cache does.
… tracking As in oak-lucene, only readers record the current local folder of an index. The writer registering its folder could make the cache delete a folder that readers of another index version still used.
…down Once the copier is closed, or its executor no longer accepts tasks, files are read remotely without scheduling a copy and directories are closed directly, instead of failing with RejectedExecutionException and leaving the local directory open.
LuceneNg now defaults to repository.home/indexNg. Its startup cleanup assumes no folder under its root is in use, which is only true for its own folders; it also no longer deletes nrt-* folders, which only oak-lucene creates.
Creating an existing file now fails with FileAlreadyExistsException instead of silently replacing it, files are listed in sorted order, and temporary files use Lucene's temp file names.
As in oak-search-elastic, the exported package is marked @internal and carries an OSGi package version.
…ecific references
…E queries As in oak-lucene and for property LIKE restrictions, the pattern is converted with QueryUtils.sqlLikeToLuceneWildcardQuery, so \_ and \% stay literal.
…vation step fails Each deactivation step logs its failure instead of aborting, so the open indexes, the local cache copier and the executor are always released.
…roperties by depth Documents index their depth, as in oak-lucene, so a direct-children restriction evaluated on descendant documents matches the ancestor and the depth in Lucene instead of returning the whole subtree for post-filtering. Existing LuceneNg indexes need reindexing.
ff387fb to
2bcbbf2
Compare
4 participants
Summary
Adds
oak-search-lucene-ng, an experimental Lucene 9 index provider. Indexes opt in withtype=luceneNg; existing indexes are unaffected. The goal is an initial version that can betested in real deployments alongside
oak-lucene, replicating its behaviour on the sharedoak-searchframework (planner, tracker, editor, document maker and result cursor).:luceneNgnode, chunked by thedefinition's
blobSize; readers and the async writer can cache segments on local disk(
repository.home/indexNgby default).relative properties, fulltext with excerpts, sorting and facets (insecure, statistical,
secure). Results go through the shared fulltext cursor, so a query that outlives an index
refresh restarts on the new version without duplicates.
nrt/synclanes are not supported.indexed as strings: Lucene 9 requires one field shape across documents.
Known gaps (configurable analyzers, suggestions, spellcheck, similarity, binary text
extraction, NRT, copy-on-write, JMX) are listed in the module README.
The shared comparison fixture (
AbstractIndexComparisonTestin theoak-searchtest-jar)runs the same query scenarios against
oak-luceneand LuceneNg.