Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Download the latest `.jar` file from:

Corese supports pluggable storage backends.
Create your own storage implementation (PostgreSQL, Redis, etc.)
or use built-in backends (`graph`, `memory`).
or use the default native backend (`memory`).

**- [Storage Backend Plugin Guide](docs/STORAGE_PLUGIN_GUIDE.md)**

Expand Down
13 changes: 13 additions & 0 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -344,3 +344,16 @@ tasks.named("generateGrammarSource") {
tasks.named("javaccSparqlCorese") {
dependsOn("createGeneratedDirs")
}

// Standalone storage measurements use the existing test runtime, without timing JUnit.
tasks.register("storageBenchmarkClasspath") {
group = "verification"
description = "Writes the standalone storage benchmark runtime classpath."
dependsOn(tasks.testClasses)
doLast {
layout.buildDirectory.file("benchmarks/classpath.txt").get().asFile.apply {
parentFile.mkdirs()
writeText(sourceSets.test.get().runtimeClasspath.asPath)
}
}
}
11 changes: 4 additions & 7 deletions docs/STORAGE_PLUGIN_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,9 @@ multiple providers support a configuration, Corese selects the highest
priority; equal priorities are ordered by plugin name. Names must be unique and
non-blank.

The built-in providers are:
The built-in provider is:

- `memory`: the default in-memory backend for tests and small datasets;
- `graph`: a compatibility adapter around the legacy Corese `Graph` backend,
requiring `graph` and `valueFactory` configuration properties.
- `memory`: the default native in-memory backend with SPO/POS/OSP indexes,
graph partitions, atomic `addAll` and session-scoped serializable snapshots.

See `GraphStoragePlugin` and `MemoryStoragePlugin` for complete provider
implementations, and `docs/source/next_storage.md` for the public storage
contract.
See `MemoryStoragePlugin` for the complete provider implementation, and `docs/source/next_storage.md` for the public storage contract.
69 changes: 69 additions & 0 deletions docs/memory-storage-final-validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Finition et validation du stockage mémoire — 14 octobre 2026

Les points signalés dans les deux dernières reviews sont corrigés. Aucun commit
ni push n'a été effectué. Les modifications préexistantes du nettoyage sont
conservées.

## Timeout et transactions

`NextSparqlPipelineExecutor` exécute toujours l'évaluation sur le thread qui possède
la transaction. Le watchdog utilise désormais un seul `AtomicInteger` : le premier
CAS départage fin d'évaluation et expiration, sans intervalle entre la revendication
d'un timeout et sa publication. Un résultat ne peut pas être accepté après que le
watchdog a gagné cette transition.

L'échéance utilise un temps monotone mesuré à l'entrée, incluant le délai de
démarrage du watchdog. La fin d'évaluation vérifie aussi le budget, même si le
watchdog n'a pas encore été planifié. Après expiration, l'appelant attend la fin du
watchdog et de sa fermeture des ressources avant de consommer l'interruption du
timeout ; celui-ci ne peut donc plus interrompre une requête suivante.

Deux tests couvrent l'expiration dans une transaction, sa réutilisation et son
commit, ainsi que la conservation d'une interruption externe déjà présente.
Le test existant couvre le succès de SELECT et ASK avec timeout dans une transaction.

Les suppressions Sonar sont localisées et commentées : `S2142` pour consommer le
signal du watchdog après son arrêt, et `S9391` pour conserver la liste mutable du
dataset sans ajouter de pipeline stream à sa préparation.

## Sondes et documentation

Le runner actif `corese-storage-benchmarks/benchmarks/profile-memory.py` compile
les deux sondes depuis leur nouveau dépôt, avec le classpath du checkout Corese
sélectionné. Il ne cherche plus ces sources dans `corese-core/src/test`.
Il enregistre les commandes et hashes des sources des deux dépôts et utilise Java 21,
G1 et un heap de 2 Go pour le protocole de référence. Le backend actif est `memory` ;
les anciennes campagnes et leurs runtimes restent des archives.

La documentation corrige les liens et packages déplacés, le moniteur partagé de
`Database`, le registre `State.graphs`, le seuil de huit entrées de `CompactTermMap`
et les limites de l'isolation des valeurs temporelles. Les mesures et résultats
de qualité historiques sont distingués de la validation du checkout actuel.
Le README W3C indique que les sélections autres que `memory` sont rejetées.

## Résultats

| Vérification | Résultat |
| --- | --- |
| Corese, `test storageBenchmarkClasspath --offline`, Java 21 | **3 155 tests**, zéro échec, erreur ou test ignoré |
| W3C, backend `memory` | **2 915 cas**, aucun changement de statut ni d'outcome |
| JUnit du harness W3C | 3 053 tests, les cinq échecs préexistants et treize tests ignorés attendus, zéro erreur |
| `validateEarlReport` | Parsing de 37 551 statements, quinze contrôles SPARQL et cohérence JSON/EARL validés |
| `enforceW3cRegressions` | Zéro régression contre la baseline officielle |
| Sonar way local, exécuteur et tests modifiés | 597 règles, deux fichiers, zéro anomalie et zéro échec d'analyse |
| Compilation stricte de l'exécuteur | `-Xlint:all -Werror`, zéro avertissement |
| Mémoire accessible inclusive, 600k, unitaire | **90 827 088 octets** |
| Mémoire accessible inclusive, 600k, batch | **90 827 088 octets** |
| Contrôle du mode heap libéré | Exécution validée sur 120 statements et trois snapshots ; ce contrôle n'est pas une mesure de performance |

La valeur mémoire est exactement celle de la review précédente, après renommage
du backend en `memory`. Elle est inférieure de huit octets à l'ancienne référence
`indexed-memory`. Aucun fichier de production du stockage natif n'est modifié par
cette passe ; ses chemins d'ingestion et de `find()` conservent leurs sources.
Il n'a pas été lancé de nouvelle campagne chronométrée d'allocations ou de latence.

Les résultats bruts, commandes, rapports Sonar et audits sont conservés dans
`build/cleanup-final/`. La dernière modification de JavaDoc et d'annotation de
l'exécuteur a été compilée strictement et comparée aux instructions testées :
elles sont identiques. L'analyse Sonar locale ne préjuge pas d'un profil upstream
personnalisé.
57 changes: 53 additions & 4 deletions docs/source/next_storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,56 @@ results manage that stream through their own `close()` contract.
## Transactions

Call `transactions().supportsTransactions()` before beginning a transaction.
The current memory and legacy Graph adapters report no transaction support.
A future native backend may advertise its supported isolation levels and return
an `AutoCloseable` transaction handle. Closing an active transaction rolls it
back.
The `memory` backend advertises `SERIALIZABLE` and returns an `AutoCloseable`
transaction handle. Closing an active transaction rolls it back.

## Native memory backend

The `memory` backend is selected by default by `Storages.create()` or explicitly:

```java
StorageManager storage = Storages.create(
StorageConfig.builder().type("memory").build());
```

This implementation uses SPO, POS and OSP hash indexes in each graph partition.
Bound RDF terms select index prefixes; values are compared by RDF identity, including
literal lexical form, datatype and language. Prefix iteration is reproducible for
an identical insertion history. It supplies neither sorted value ranges nor a
SPARQL result order without `ORDER BY`. Named graphs survive `clear` and the
removal of their last statement until `dropGraph`.

`find` captures partition snapshot references under a shared monitor and returns
a lazy stream over that snapshot without allocating an intermediate collection.
`count` and `contains` traverse the selected index without materializing the matches.
Searches across every graph visit all partitions.

`addAll` consumes its iterable once, deduplicates quads in private staging, builds
all permutations and publishes the touched partitions together. An ordinary source
or finalization exception leaves the batch unpublished. Earlier writes in an
explicit transaction survive a failed batch. Other sessions see the batch only
after publication, or after transaction commit. A session cannot perform another
write while it consumes a batch source. Bulk construction needs temporary memory;
a smaller retained index does not necessarily imply a smaller load peak.

Sessions share committed data and own independent transactions, even on the same
thread. Only `SERIALIZABLE` is supported. Beginning a transaction copies the graph
directory and shares its partitions (time and memory proportional to graph count).
The first write to a shared partition copies that graph's index containers;
other graphs remain shared. Autocommit writers also copy partitions still used by
snapshots. Statements and cached immutable keys remain shared. Commit verifies a
global version and publishes the private state atomically.
Any intervening committed write causes a conservative conflict, including writes
to disjoint graphs. Roll back the still-active transaction before retrying.
Transaction reads and writes must run on the creating thread. Autocommit protects
each operation; use an explicit transaction for a consistent multi-operation view.

Closing a session rolls back its transaction. Closing the owner shuts down the
shared backend. The storage architecture and benchmarking synthesis are documented in
`corese-docs` under `Architecture/corese-storage-manager/memory-storage.md`.

The isolation guarantees cover storage operations on RDF term identities.
Corese's built-in temporal literals expose their mutable calendar reference;
this backend preserves that API behavior. Do not mutate those calendars while
terms are in use: doing so can change temporal evaluation without a storage write.
Index snapshots share terms and do not provide deep isolation of calendar objects.
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
import fr.inria.corese.core.next.data.api.exception.IncorrectFormatException;

import java.io.Serial;
import java.io.IOException;
import java.io.ObjectInputStream;

/**
* Immutable default implementation of an RDF IRI.
Expand All @@ -13,6 +15,10 @@ public final class SimpleIRI extends AbstractIRI {
@Serial
private static final long serialVersionUID = -2444556019914696994L;

// SimpleIRI is immutable. Preserve AbstractIRI's exact hash, but parse it once.
// Transient so older serialized instances rebuild the cache during deserialization.
private transient int cachedHash;

/**
* Creates an IRI from its complete string representation.
*
Expand All @@ -21,6 +27,8 @@ public final class SimpleIRI extends AbstractIRI {
*/
public SimpleIRI(String fullIRI) {
super(fullIRI);
// This constructor has already inferred the canonical split in AbstractIRI.
cachedHash = 31 * (31 * 7 + getNamespace().hashCode()) + getLocalName().hashCode();
}

/**
Expand All @@ -32,6 +40,28 @@ public SimpleIRI(String fullIRI) {
*/
public SimpleIRI(String namespace, String localName) {
super(namespace, localName);
cachedHash = super.hashCode();
}

@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof IRI that)) {
return false;
}
return stringValue().equals(that.stringValue());
}

@Override
public int hashCode() {
return cachedHash;
}

@Serial
private void readObject(ObjectInputStream stream) throws IOException, ClassNotFoundException {
stream.defaultReadObject();
cachedHash = super.hashCode();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -239,12 +239,10 @@ private String hashFirstDegreeQuads(String blankNode, Map<String, List<Statement
return hash("");
}

List<String> nquads = new ArrayList<>();
for (Statement quad : quads) {
nquads.add(serializeQuad(quad, blankNode));
}

Collections.sort(nquads);
List<String> nquads = quads.stream()
.map(quad -> serializeQuad(quad, blankNode))
.sorted()
.toList();
return hash(String.join("\n", nquads) + "\n");
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,7 @@ private void writePropertyElement(Writer writer, IRI predicate, Value object, St
}

writer.write(currentIndent);
writer.write(String.format("<%s", elementName));
writer.write("<" + elementName);

if (object.isIRI()) {
writer.write(String.format(" %s=\"%s\"/>", SerializationConstants.RDF_RESOURCE_ATTRIBUTE, escapeXmlAttribute(object.stringValue())));
Expand Down Expand Up @@ -371,7 +371,7 @@ private void writePropertyElement(Writer writer, IRI predicate, Value object, St

writer.write(escapeXmlContent(literal.stringValue()));

writer.write(String.format("</%s>", elementName));
writer.write("</" + elementName + ">");
if(this.config instanceof LineEndingOptions lineEndingOptions) {
writer.write(lineEndingOptions.getLineEnding());
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,7 @@ protected void writeHeader(Writer writer) throws IOException {
if (option instanceof BaseIRIOptions baseIRIOptions
&& option instanceof LineEndingOptions lineEndingOptions
&& baseIRIOptions.getBaseIRI() != null) {
writer.write(String.format("@base <%s> .%s",
baseIRIOptions.getBaseIRI(),
lineEndingOptions.getLineEnding()));
writer.write("@base <" + baseIRIOptions.getBaseIRI() + "> ." + lineEndingOptions.getLineEnding());
}

Set<String> actuallyUsedNamespaces = Set.of();
Expand Down Expand Up @@ -359,7 +357,7 @@ protected void writeIRI(Writer writer, IRI iri) throws IOException {
if (prefixed != null) {
writer.write(prefixed);
} else {
writer.write(String.format("<%s>", escapeIRIString(iri.stringValue())));
writer.write("<" + escapeIRIString(iri.stringValue()) + ">");
}
}

Expand Down
Loading
Loading