diff --git a/.skills/tinker-dev/SKILL.md b/.skills/tinker-dev/SKILL.md index ce4a03c43a7..05c90ec89fc 100644 --- a/.skills/tinker-dev/SKILL.md +++ b/.skills/tinker-dev/SKILL.md @@ -141,8 +141,12 @@ ones that are easy to miss: use the **tinker-doc** skill. The mechanical minimum: add new content to the right book and update its `index.asciidoc`. - **Changelog**: for user-visible or API changes, add a concise `CHANGELOG.asciidoc` entry in the - correct version section. Do not invent version numbers or release names. See the **tinker-doc** - skill for changelog discipline (one entry per user-facing change, not per commit) and voice. + correct version section. Do not invent version numbers or release names. Never add the + `*(breaking)*` marker to an entry you write: it is applied only at release, to the lines generated + from JIRA issues labelled `breaking` (see `docs/src/dev/developer/release.asciidoc`), so a + hand-written entry without it is correct. Explain a breaking change in the upgrade docs instead. + See the **tinker-doc** skill for changelog discipline (one entry per user-facing change, not per + commit) and voice. Otherwise, match the existing code in neighboring files — explicit imports (no wildcards), `final` where practical, SLF4J logging over `println`, Hamcrest matchers, and closing any diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 5423e359be7..b7be6bb4df8 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -27,8 +27,11 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima * Added a `Char` type to `gremlin-go` to represent the Gremlin `Character` type in `GremlinLang`. * Fixed `gremlin-go` to report a malformed or truncated GraphBinary response as a deserialization error rather than a bare decoder message. -* Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. *(breaking)* -* Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. *(breaking)* +* Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. +* Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. +* Fixed a `NullPointerException` when rolling back a transaction on a `TinkerStorageGraph` that has a vertex index but no edge index. +* Added a pluggable storage layer to `TinkerStorageGraph` that durably persists each committed transaction to disk. +* Removed automatic persistence from `TinkerMemoryGraph`, which is now purely in-memory and ignores `gremlin.tinkergraph.graphLocation`/`graphFormat`. Use `TinkerStorageGraph` for durability or `g.io()` for interchange. [[release-4-0-0-beta-3]] === TinkerPop 4.0.0-beta.3 (July 20, 2026) diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md index a46282bb2c6..4b4389d696c 100644 --- a/THREAT_MODEL.md +++ b/THREAT_MODEL.md @@ -75,7 +75,7 @@ under the License. | **Core traversal machine + structure API** (`gremlin-core`) | `GraphTraversal`, strategies, `Vertex`/`Edge` | filesystem (IO formats) | **In** | | **Serialization** (`gremlin-core`/`gremlin-util`/`gremlin-shaded`) | GraphSON + GraphBinary wire readers. Gryo (Kryo) is IO-format-only, not on the wire | deserializes untrusted wire bytes. Gryo only reads IO/file bytes | **In** (see §9) | | **Gremlin Language Variants** (Java `gremlin-driver`, `gremlin-python`, `-dotnet`, `-go`, `-js`) | deserialize server responses | network (connect) | **In**, response-deserialization robustness + TLS cert validation. Java shares the JVM serializers. The others have own per-language deserializers not covered by hardening the JVM server | -| **Reference graph** (`tinkergraph-gremlin`) | in-memory graph, the **shipped default graph**, with optional file persistence via the `io()` readers | filesystem (persistence) | **In**, TinkerPop reference code on the default reachable path, so a defect in unmodified TinkerGraph is `VALID` (§3). Loading a persisted graph uses the same GraphSON/Gryo/GraphML readers (§6 IO surface). A custom `graphFormat` reader is provider code (§3) | +| **Reference graph** (`tinkergraph-gremlin`) | in-memory graph, the **shipped default graph**. `TinkerStorageGraph` optionally persists committed transactions to a storage directory through a pluggable storage engine (the `graphbinary` reference engine) | filesystem (storage directory) | **In**, TinkerPop reference code on the default reachable path, so a defect in unmodified TinkerGraph is `VALID` (§3). The storage directory is operator-trusted like any data directory, so its files are not an attack surface (§6). A custom storage engine is provider code (§3) | | **OLAP** (`gremlin-core`/`tinkergraph` computer, `hadoop-gremlin`, `spark-gremlin`) | `GraphComputer`, remote-reachable when the operator configures an OLAP-capable graph | network, filesystem, cluster | **In**, reference code (incl. Hadoop/Spark modules). The third-party Hadoop/Spark runtime + cluster config are out (§3) | | **`gremlin-console`** | local Groovy REPL. `:remote console` submits **arbitrary Groovy scripts** to a server and deserializes responses. `:install` loads plugins (credentials/hadoop/spark) via Grape | network (connects) | Local REPL **Out** (operator-trusted). As a remote client, submitting arbitrary Groovy = server-side RCE **by-design** (§9), subject to server auth. Response-deserialization robustness + TLS cert validation **In** (like a GLV). Plugins are operator-installed code in the operator's own JVM, so the install decision, the third-party download channel (Grape), and plugin behavior are **Out** (trusted-input, §3) | | **`gremlint`** | client-side Gremlin formatter (parse-and-reprint, no query execution); published as a library for integration into client applications. | none (runs in the caller's browser/process) | **In**, valid surface for security reports since it's a published, user-integrated library | @@ -267,11 +267,12 @@ Per-surface trust table: | Gremlin Server — script request | Gremlin (or, if `gremlin-groovy` is enabled, Groovy) script text | **yes** (pre-auth if auth off) | auth; script restriction / sandbox; who may script; traversal-step allow-list; resource limits | | Gremlin Server — transaction id | server-generated UUID, echoed back by the client on follow-up requests | **yes** (a client can present any id) | looked up with no owning-user check, so any client presenting a valid id shares that transaction (see §11b) | | Request deserialization (GraphSON / GraphBinary) | serialized bytes | **yes** (pre-auth only when auth is off, see the ordering note below) | robustness of the wire serializers | -| Graph IO — Gryo/GraphSON/GraphML files (`io()` step, persistence, OLAP) | on-disk / cluster bytes | only if the caller loads untrusted files | GraphSON, the hardened Gryo mappers the IO paths build (`registrationRequired=true` plus `javaSerializationAllowed=false`), and GraphML with the default XML factory owe deserializer integrity. Unlocked Gryo, a Gryo mapper that keeps Java serialization, and a caller-supplied unhardened `XMLInputFactory` (XXE) are the caller's responsibility | +| Graph IO — Gryo/GraphSON/GraphML files (`io()` step, OLAP) | on-disk / cluster bytes | only if the caller loads untrusted files | GraphSON, the hardened Gryo mappers the IO paths build (`registrationRequired=true` plus `javaSerializationAllowed=false`), and GraphML with the default XML factory owe deserializer integrity. Unlocked Gryo, a Gryo mapper that keeps Java serialization, and a caller-supplied unhardened `XMLInputFactory` (XXE) are the caller's responsibility | | Gremlin string parser (`gremlin-language` ANTLR) | Gremlin string | **yes** | parser robustness, no crash/hang/OOM on malformed input and no grammar breakout / step injection (distinct from execution cost, §8/Q7) | | Any string the grammar accepts as an argument (e.g. a `regex` pattern) | Gremlin string | **yes** | a grammatically valid string must not enable DoS (e.g. ReDoS via a pathological pattern), the Q7 super-linear-amplification carve-out (§8) | | GLV (client) — server response | serialized bytes from the server | yes if the server is malicious/compromised, or a MITM (TLS off / cert not validated) | response-deserialization robustness; TLS with cert validation | | `gremlin-server.yaml`, host, data dir | local | no — operator-trusted | filesystem permissions | +| `TinkerStorageGraph` storage directory (`snapshot.gbin`, `log.gbin`, `INDEXES`, `VERSION`, `settings.properties`) | on-disk bytes decoded when the graph opens | no — operator-trusted, a data directory | filesystem permissions. The engine's checks against damaged files (checksums, bounded decoding, the `recover` open, described in `tinkergraph-gremlin/DESIGN.md`) are a reliability property against disk damage, not a security property | - **Pre-auth vs post-auth (ordering note):** "pre-auth" marks what an attacker can reach **without valid credentials**, which is what makes a defect there security-critical (§8). Gremlin Server authenticates @@ -442,7 +443,7 @@ Per-surface trust table: (`GryoMapper.Builder.javaSerializationAllowed(boolean)` selects the behavior), so a break there is `VALID`. A directly built `GryoMapper` and `GryoPool` keep them, as do the `spark-gremlin` and Hadoop object pools that additionally run unlocked; those remain the user's responsibility. - Gryo is not on the wire, so this is an IO/file-surface concern (`io()` step, persistence, OLAP). + Gryo is not on the wire, so this is an IO/file-surface concern (`io()` step, OLAP). - **A `TraversalStrategy` is not an access-control boundary on its own.** The Gremlin language lets any request remove or replace strategies on its traversal source (`withoutStrategies()`, @@ -517,6 +518,10 @@ Per-surface trust table: - "Calling application concatenates untrusted input into a Groovy string (Gremlin-injection)" — the calling application's responsibility (§9). A `gremlin-language` parser breakout is the opposite: an in-model TinkerPop bug (§8). Disposition: `BY-DESIGN: property-disclaimed`. +- "A crafted or corrupted `TinkerStorageGraph` store file crashes, hangs, or exhausts memory when the graph opens" — + the storage directory is operator-trusted (§5, §6), so writing a malicious file there already requires the + operator's access. A store that fails to open, or opens wrongly, after ordinary disk damage is an ordinary + reliability bug, not a security report. Disposition: `OUT-OF-MODEL: trusted-input`. - Findings in any test-only module, any module containing examples, or build tooling (e.g. `gremlin-test`, `gremlin-examples`, `gremlin-tools`) — out of scope (§3). Disposition: `OUT-OF-MODEL: unsupported-component`. diff --git a/docs/src/reference/implementations-tinkergraph.asciidoc b/docs/src/reference/implementations-tinkergraph.asciidoc index 4e3c873e3c0..5338974e15c 100644 --- a/docs/src/reference/implementations-tinkergraph.asciidoc +++ b/docs/src/reference/implementations-tinkergraph.asciidoc @@ -39,13 +39,15 @@ under the License. ---- -image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory (with optional -persistence), graph engine that provides both OLTP and OLAP functionality. It is non-transactional by default but does +image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory graph engine that +provides both OLTP and OLAP functionality. It is non-transactional by default but does have a lightweight transactional form that can be instantiated offering simple `ThreadLocal` transactions supporting -`read committed` transaction isolation. As of 4.0.0, `TinkerGraph` is an interface with two implementations: -`TinkerMemoryGraph`, the in-memory, non-transactional implementation that `TinkerGraph.open()` constructs, and -`TinkerStorageGraph`, the transactional implementation formerly named `TinkerTransactionGraph`. TinkerGraph is -deployed with TinkerPop and serves as the reference +`read committed` transaction isolation, with optional durable persistence to disk. As of 4.0.0, `TinkerGraph` is an +interface with two implementations: `TinkerMemoryGraph`, the in-memory, non-transactional implementation that +`TinkerGraph.open()` constructs, and `TinkerStorageGraph`, the transactional implementation formerly named +`TinkerTransactionGraph`. Transactions and durability are separate concerns. `TinkerStorageGraph` writes nothing to +disk unless a storage engine is configured, and without one it is a purely in-memory graph that retains its +transaction support. TinkerGraph is deployed with TinkerPop and serves as the reference implementation for other providers to study in order to understand the semantics of the various methods of the TinkerPop API. Its status as a reference implementation does not however imply that it is not suitable for production. TinkerGraph has many practical use cases in production applications and their development. Some examples of TinkerGraph @@ -172,19 +174,38 @@ TinkerGraph has several settings that can be provided on creation via `Configura |gremlin.tinkergraph.defaultVertexPropertyCardinality |The default `VertexProperty.Cardinality` to use when `Vertex.property(k,v)` is called. |gremlin.tinkergraph.allowNullPropertyValues |A boolean value that determines whether or not `null` property values are allowed and defaults to `false`. |gremlin.tinkergraph.vertexLabelCardinality |The `LabelCardinality` for vertices. Options are `ONE` (default, single immutable label, backward-compatible with 3.x), `ONE_OR_MORE` (at least one label, mutable), or `ZERO_OR_MORE` (zero or more labels). See <>. -|gremlin.tinkergraph.graphLocation |The path and file name for where TinkerGraph should persist the graph data. If a -value is specified here, the `gremlin.tinkergraph.graphFormat` should also be specified. If this value is not -included (default), then the graph will stay in-memory and not be loaded/persisted to disk. -|gremlin.tinkergraph.graphFormat |The format to use to serialize the graph which may be one of the following: -`graphml`, `graphson`, `gryo`, or a fully qualified class name that implements Io.Builder interface (which allows for -external third party graph reader/writer formats to be used for persistence). -If a value is specified here, then the `gremlin.tinkergraph.graphLocation` should -also be specified. If this value is not included (default), then the graph will stay in-memory and not be -loaded/persisted to disk. +|gremlin.tinkergraph.storage |The durable storage engine used by `TinkerStorageGraph` to persist committed transactions +to disk. The value is either a built-in engine name (`graphbinary`) or a fully qualified class name of a +`TinkerStorage` implementation. When not specified (default), the graph holds data only in memory. This setting is +only valid on `TinkerStorageGraph`. The in-memory `TinkerMemoryGraph`, which `TinkerGraph.open()` and +`gremlin.graph=...TinkerGraph` create, ignores it and every other `gremlin.tinkergraph.storage.*` setting and logs a +warning when one is set. +|gremlin.tinkergraph.storage.directory |The directory in which `TinkerStorageGraph` stores its durable data. Required +when `gremlin.tinkergraph.storage` is set, and only valid with it: a `TinkerStorageGraph` given a directory but no +storage engine fails to open rather than keep its data only in memory. A relative path resolves against the working directory +of the JVM, so an absolute path is safer for a graph opened by Gremlin Server or the Gremlin Console. +|gremlin.tinkergraph.storage.sync |The durability applied to each committed transaction by a `TinkerStorageGraph` +storage engine. `commit` (default) forces each commit to disk so an acknowledged commit survives an operating system +crash or power loss. `os` only flushes to the operating system, so a commit survives a crash of the JVM process but +may be lost on an operating system crash or power loss. Only meaningful when `gremlin.tinkergraph.storage` is set. +|gremlin.tinkergraph.storage.compactThreshold |The size in bytes at which a `TinkerStorageGraph` storage engine +automatically compacts its append log, bounding the log growth and restart time of a long-running graph that is never +explicitly closed. Defaults to `67108864` (64 MB). A value of `0` disables automatic compaction, leaving it to +`close()` or an explicit `compact()`. A negative value fails the open. Only meaningful when +`gremlin.tinkergraph.storage` is set. +|gremlin.tinkergraph.storage.preserveVertexPropertyIds |Whether a `TinkerStorageGraph` storage engine persists +vertex-property ids so they are stable across a close and reopen. Defaults to `false`, which regenerates every +vertex-property id on load, whether it was generated or given explicitly, and keeps the store smaller. Vertex and edge +ids are always preserved regardless. Fixed when the store is created. Only meaningful when +`gremlin.tinkergraph.storage` is set. +|gremlin.tinkergraph.storage.recover |Whether a `TinkerStorageGraph` opens a damaged store by recovering what remains +readable rather than failing. Defaults to `false`. A graph opened this way is read-only and leaves the storage +directory unchanged. See <>. Only meaningful when +`gremlin.tinkergraph.storage` is set. |========================================================= -NOTE: To use <>, configure `gremlin.graph` as -`org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. +NOTE: To use <> and <>, configure +`gremlin.graph` as `org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph`. The `IdManager` settings above refer to how TinkerGraph will control identifiers for vertices, edges and vertex properties. There are several options for each of these settings: `ANY`, `LONG`, `INTEGER`, `UUID`, `STRING` or the @@ -197,32 +218,11 @@ type as well as generate new identifiers with that specified type. TIP: Setting the `IdManager` to `ANY` also allows `String` type ID values to be used. -If the TinkerGraph is configured for persistence with `gremlin.tinkergraph.graphLocation` and -`gremlin.tinkergraph.graphFormat`, then the graph will be written to the specified location with the specified -format when `Graph.close()` is called. In addition, if these settings are present, TinkerGraph will attempt to -load the graph from the specified location. - -IMPORTANT: If choosing `graphson` as the `gremlin.tinkergraph.graphFormat` with untyped GraphSON, be sure to also -establish the various `IdManager` settings to ensure that identifiers are properly coerced to the appropriate types, -as untyped GraphSON can lose the identifier's type during serialization (i.e. it will assume `Integer` when the -default for TinkerGraph is `Long`, which could lead to load errors that result in a message like, "Vertex with id -already exists"). Typed GraphSON preserves the identifier's type across the serialization round-trip (for example, a -`Long` identifier is written as `{"@type":"g:Int64","@value":...}`), so the `IdManager` coercion above is not -required in that case. - -These settings can be supplied programmatically by building a `Configuration` and passing it to `TinkerGraph.open()`. -For example, to persist to disk as GraphSON while keeping `Long` identifiers: - -[source,java] ----- -BaseConfiguration conf = new BaseConfiguration(); -conf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/tinkergraph.json"); -conf.setProperty("gremlin.tinkergraph.graphFormat", "graphson"); -conf.setProperty("gremlin.tinkergraph.vertexIdManager", "LONG"); -Graph graph = TinkerGraph.open(conf); -// ... work with the graph ... -graph.close(); // persists the graph to the configured graphLocation ----- +Durable persistence is provided by `TinkerStorageGraph` through its pluggable storage layer and is described in the +<>. It is enabled by configuration rather than by the choice of +implementation. Neither `TinkerMemoryGraph` nor a `TinkerStorageGraph` without a configured storage engine holds data +across JVM restarts. Moving data in and out of any TinkerGraph in an interchange format is handled on demand by the +`io()` step rather than by graph configuration, as shown below. It is important to consider the data being imported to TinkerGraph with respect to `defaultVertexPropertyCardinality` setting. For example, if a `.gryo` file is known to contain multi-property data, be sure to set the default @@ -278,6 +278,11 @@ The default configuration of TinkerGraph remains non-transactional. NOTE: This feature was first made available in TinkerPop 3.7.0 as `TinkerTransactionGraph`. The class was renamed to `TinkerStorageGraph` in 4.0.0. +`TinkerStorageGraph` requires no storage configuration to provide transactions. With no storage engine set it creates +no files and writes nothing to disk, which suits testing and any other case that calls for transaction semantics +without durable state. Configuring a storage engine adds durability to that same graph and is described in the +<>. + ==== Transaction Semantics `TinkerStorageGraph` only has support for `ThreadLocal` transactions, so embedded graph transactions may not be fully @@ -296,9 +301,9 @@ below). [[testing-remote-providers]] ==== Testing Remote Providers -These transaction semantics described above may not fit use cases for some production scenarios that require strict -ACID-like transactions. Therefore, it is recommended that `TinkerStorageGraph` be used as a `Graph` for test -environments where you still require access to a `Graph` that supports transactions. `TinkerStorageGraph` does fully +These transaction semantics described above may not fit production scenarios that require strict ACID-like +transactions or many concurrent writers (see <>). They do make `TinkerStorageGraph` a +good `Graph` for test environments where you still require access to a `Graph` that supports transactions. `TinkerStorageGraph` does fully support TinkerPop's `Transaction` interface which still makes it a useful `Graph` for exploring the <>. @@ -424,6 +429,261 @@ g.V().valueMap() <4> Add a second vertex without committing <5> Rollback the change +[[tinkergraph-gremlin-persistence]] +=== Persistence + +`TinkerStorageGraph` can durably persist to disk through a pluggable storage layer built on its transaction support. +When a storage engine is configured, each committed transaction is written to disk before it is applied in memory, and +the graph is rebuilt from that data when it is opened again. The in-memory `TinkerMemoryGraph` does not retain data +across restarts. + +A storage engine is selected with the `gremlin.tinkergraph.storage` configuration key, and +`gremlin.tinkergraph.storage.directory` names the directory that holds the durable data. The value of the storage key is +either a built-in engine name or the fully qualified class name of a `TinkerStorage` implementation, following the same +enum-name-or-class-name convention as the `IdManager` settings. The built-in `graphbinary` engine records committed +transactions in an append-only log serialized with GraphBinary and periodically folds that log into a compact +snapshot. The design of that engine, including its file formats and how it detects and handles damage, is described in +the link:https://github.com/apache/tinkerpop/blob/x.y.z/tinkergraph-gremlin/DESIGN.md[TinkerStorageGraph storage design] +notes. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph") +conf.setProperty("gremlin.tinkergraph.storage", "graphbinary") +conf.setProperty("gremlin.tinkergraph.storage.directory", "/data/mygraph") + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV("person").property("name","marko").iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.V().count() +==>1 +---- + +The graph remains the authoritative copy in memory and is mirrored to disk, so a persisted graph must still fit in +memory. + +==== Durability + +A commit is acknowledged only once it has been written to the store, so a commit that fails to persist is rolled back +and none of its changes become visible. How far an acknowledged commit is protected is governed by the +`gremlin.tinkergraph.storage.sync` setting. Under the default `commit` mode every commit is forced to disk, so it +survives an operating system crash or power loss. The `os` mode only hands the commit to the operating system, which is +faster but protects it against a crash of the JVM process alone, not an operating system crash or power loss. Files that +are replaced whole, such as a new snapshot, are made durable by forcing the storage directory after they are renamed +into place. Some file systems, including certain network and FUSE mounts, cannot force a directory, and there that step +is skipped with a logged warning. A power loss soon after a compaction on such a file system can then undo the +compaction, and with it the commits acknowledged since, so a store that needs the full guarantee belongs on a local file +system. + +After a crash, opening the store again restores every acknowledged commit. A commit that was in progress when the +process stopped was never acknowledged and is not restored. The open repairs what such a crash leaves behind on its +own and logs a warning saying so. + +The store grows with each commit until it is compacted, which rewrites the committed state into a fresh snapshot and +reclaims the space. Compaction runs when the graph is closed after any commit, can be requested by calling the +graph's `compact()` method, and runs automatically once the log grows past +`gremlin.tinkergraph.storage.compactThreshold` bytes, so a long-running graph that is never closed does not grow without +bound. Setting that threshold to `0` disables automatic compaction. + +==== Concurrent Writes + +Each commit writes to the store while holding a single lock for the whole graph, and under the default `commit` sync +mode forces the write to disk before releasing it. Commits therefore complete one at a time, and write throughput is +bounded by how quickly the disk completes a forced write. The `os` sync mode shortens each commit at the cost of the +weaker guarantee described above. A transaction holds the locks on the elements it changed until its commit completes, +so the forced write also lengthens the window in which another transaction touching those elements fails with a +`TransactionException` (see "Best Practices" below). Automatic compaction runs as part of the commit that crosses the +threshold and rewrites the whole graph while holding that same lock, so every other commit waits until it finishes. +That pause grows with the size of the graph, which can be avoided by setting +`gremlin.tinkergraph.storage.compactThreshold` to `0` and calling the graph's `compact()` at a quiet time. +Reads, including the commit of a transaction that only read, do not take the lock and are not held up by either. A +persisted `TinkerStorageGraph` is best suited to a single writer, or a few writers that mostly change different +elements, with reads free to run alongside them. + +==== What a Store Holds + +A persisted graph can only hold values that its storage engine can write. The `graphbinary` engine stores ids, +property values and meta-property values with the GraphBinary serializers, so each must be one of the types GraphBinary +supports: `String`, `Character`, `Boolean`, `Byte`, `Short`, `Integer`, `Long`, `Float`, `Double`, `BigInteger`, +`BigDecimal`, `UUID`, `OffsetDateTime`, `Duration`, `ByteBuffer` for binary data, and `List`, `Set` and `Map` of those +types. Other types, including `java.util.Date`, `Instant`, `LocalDate`, `LocalDateTime`, `ZonedDateTime`, arrays such +as `byte[]`, and custom classes, are rejected when the transaction commits. The commit fails and is rolled back, and +the error names the property and the type that could not be stored. Convert such values first, for example a date to an +`OffsetDateTime` and a `byte[]` to a `ByteBuffer`. In the Gremlin Console, note that a Groovy string with an +interpolated value, such as `"v${i}"`, is a `GString` rather than a `String` and must be converted with `toString()`. +A graph with no storage engine configured accepts any value. + +The changes of a single transaction cannot add up to more than just under 2 GB once encoded, and neither can any +single element. A transaction that exceeds that fails when it commits and is rolled back, and the store is left as it +was. Very large loads should be committed in smaller transactions. + +Collection values come back with the contents they were written with but not always as the same class. A `List` keeps +its order. A `Map` or `Set` is read back as a `HashMap` or `HashSet`, so a `LinkedHashMap`, `TreeMap` or +`LinkedHashSet` loses both its type and its iteration order across a reopen. A graph that depends on that order should +store a `List` instead, such as a list of keys alongside the map. + +A collection value is stored as it is when its transaction commits. Changing that same collection object in place +afterward, such as adding to a `List` that was set as a property value, changes it in memory without a commit, so the +store does not record the change. A later compaction writes the graph as it is in memory and so picks the change up, +while a crash before then loses it, which leaves what survives a reopen dependent on timing. A collection should be +changed by setting the property again with a new collection inside a transaction. + +Vertex and edge ids are always preserved across a reopen. Vertex-property ids are not, by default, whether they were +generated by the graph or given explicitly with `T.id`, so a vertex property receives a new id after a reopen. Setting +`gremlin.tinkergraph.storage.preserveVertexPropertyIds` to `true` when the store is created persists every +vertex-property id, at the cost of a larger store. + +Rebuilding the graph on reopen is subject to `gremlin.tinkergraph.defaultVertexPropertyCardinality`, exactly as +importing data through the `io()` step is (see <>). TinkerGraph has a single graph-wide +cardinality rather than a per-key or per-property one, and the cardinality a property was written with is not +recorded. A graph that holds `list` or `set` multi-properties must therefore set this default to `list` or `set` when +the store is created. + +WARNING: A store created with the default `single` cardinality keeps only the last value assigned to each property key +when the graph is reloaded, even where the values were added with `list` or `set` cardinality. The other values are +discarded on reopen with only a logged warning, and the next compaction removes them from the store for good. The +default cardinality is one of the settings fixed when the store is created, so a graph that needs multi-properties must +set `gremlin.tinkergraph.defaultVertexPropertyCardinality` to `list` or `set` when its store is first opened. + +The property keys a graph indexes are recorded alongside its data and the indexes are rebuilt when the graph is opened +again, so an index created with `createIndex()` survives a restart and covers the data that was already stored. An +index dropped with `dropIndex()` stays dropped. Index changes are recorded when `createIndex()` or `dropIndex()` is +called, independent of any open transaction, so an index created inside a transaction that is rolled back is still +recorded. `clear()` removes the recorded indexes along with the data, as it does in memory. An index only affects how +fast a lookup runs and never its result, so if the record of indexes cannot be read the graph still opens, with no +indexes. + +`clear()` empties a persisted graph the same way it empties an in-memory one: at once and outside any transaction, so +it needs no commit and cannot be rolled back. The calling thread's uncommitted changes are discarded along with the +rest of the graph, and a transaction still open on another thread fails with a conflict if it commits a change to an +element that was cleared. The store is emptied in the same call, so the graph is still empty when it is opened again. + +==== Settings Fixed at Creation + +The settings that shape the stored data are fixed when a store is created: the vertex, edge and vertex-property id +managers, `gremlin.tinkergraph.defaultVertexPropertyCardinality`, `gremlin.tinkergraph.vertexLabelCardinality`, +`gremlin.tinkergraph.allowNullPropertyValues`, `gremlin.tinkergraph.storage.preserveVertexPropertyIds` and the storage +engine itself. The store records their values, defaults included, in a `settings.properties` file in its storage +directory. When the graph is opened again, any of these settings left out of the configuration is taken from that +record, so a store can be opened with nothing more than `gremlin.tinkergraph.storage` and +`gremlin.tinkergraph.storage.directory`, and `configuration()` on the opened graph reports the recorded values. A +setting that is given a different value fails the open. Changing one is a migration: export the graph with the `io()` +step and import it into a graph opened on a new storage directory with the new settings. Settings that only affect how +the engine runs, such as `gremlin.tinkergraph.storage.sync` and `gremlin.tinkergraph.storage.compactThreshold`, may +change between opens. An unusable value for a fixed setting fails the first open before anything is recorded, so the +store can then be created with the corrected value. The `settings.properties` file is the store's record of these +settings and must not be edited. An edit is not detected, and a store read with settings other than the ones it was +written with can lose data at its next compaction. + +==== One Graph per Directory + +A storage location may be opened by only one graph at a time. `TinkerStorageGraph` takes an exclusive lock on the +storage directory when it opens, so a second attempt to open the same location, whether from the same JVM or another +process, fails rather than corrupting the data. The lock is released when the graph is closed, and from then on any +commit against that graph instance fails, since the location may already belong to another graph. A new store is only +created in an empty directory, so pointing a graph at a directory that holds unrelated files fails rather than mixing +the store into it. Hidden files count, so the root of a mounted volume, which often holds `lost+found`, or a directory +holding a stray `.DS_Store` is refused, and a store on its own volume belongs in a subdirectory of the mount. A +recovery open, described below, takes the same lock, so it too fails while another graph has the +store open. A store in a directory the process cannot write can only be opened for recovery. + +If the storage directory or its files are removed or replaced while the graph is open, the graph stops accepting +commits and writes nothing more to that location, not even when it is closed. Its committed data stays readable in +memory until then, so it can still be exported with the `io()` step. The commit that discovers the change fails and is +rolled back in memory, even though it may already have been written to the store's new location. + +A store records the storage format version it was written with, and opening a store in a format this version cannot +read fails with a clear error rather than misreading the data. There is no in-place format migration. To move a graph +across an incompatible storage format, export it with the `io()` step before upgrading and read it back afterward. + +==== Failures and Recovery + +A failure to write to the store while the graph is running, such as a full or failing disk, stops the graph from +accepting further commits until it is closed and opened again. The commit that hit the failure may or may not be present +after the reopen, because it may have been written in full before the error was reported. A commit whose changes reached +the store but could not then be applied to the graph in memory reports that it was recorded and is present after the +reopen, while the graph refuses further commits in the same way. Until then the graph's committed data stays readable +and can be exported with the `io()` step, while explicit compaction and `clear()` are refused and index changes take +effect in memory only. Reopening rebuilds the graph from what actually reached disk. A failure during compaction is +different: it leaves the existing store intact, does not affect the commit that preceded it, and is attempted again +later. A compaction that fails when the graph is closed is logged, and the graph still closes and releases its storage +directory. + +A store that has been damaged, for example by a failing disk or by one of its files being deleted, edited or copied in +from another store, normally fails to open rather than load data it cannot trust. The error says what is wrong. One +kind of damage cannot be detected: a store whose log was cut back to an earlier commit reads as a store that simply +holds fewer commits, so those commits are lost without a warning. + +Setting `gremlin.tinkergraph.storage.recover` to `true` opens a damaged store with everything that remains readable. +The storage directory must exist, since recovery never creates one, but it need not be writable, so a store can be +recovered from a read-only copy or mount. Recovery keeps every transaction up to the first one that cannot be read and +every element of the snapshot that can be read, and a warning reports what was left out. A missing or damaged +`VERSION` file does not stop recovery, which then reads the store in the format of the running version. Some damage, +such as a snapshot from another store or damage to the start of the snapshot, cannot be recovered and still fails the +open. + +The recovered graph is read-only with respect to its store, so the damaged files stay exactly as they were. Reads work +as usual. Changes can still be made in memory, but they are never recorded: a commit that changed anything is refused +and rolled back, compaction is refused, and `createIndex()`, `dropIndex()` and `clear()` take effect in memory only. An +index created this way can speed up the reads an export does. The intended use is to export the recovered graph with +the `io()` step and read it into a graph opened on a new storage directory: + +[source,groovy] +---- +conf.setProperty("gremlin.tinkergraph.storage.recover", true) +damaged = TinkerStorageGraph.open(conf) +traversal().with(damaged).io("/tmp/recovered.kryo").write().iterate() +damaged.close() + +fresh = new BaseConfiguration() +fresh.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph") +fresh.setProperty("gremlin.tinkergraph.storage", "graphbinary") +fresh.setProperty("gremlin.tinkergraph.storage.directory", "/data/mygraph-recovered") +graph = TinkerStorageGraph.open(fresh) +g = traversal().with(graph) +g.io("/tmp/recovered.kryo").read().iterate() +g.tx().commit() +---- + +The `io()` step carries the graph's data but not its index definitions, so any index the damaged store had must be +created again with `createIndex()` on the new graph. + +==== Gremlin Server + +Gremlin Server persists a graph the same way. A graph in the server's `graphs` configuration is pointed at a +properties file that sets the storage keys, and the server opens that graph at startup, commits to it as clients +request, and closes it on shutdown. A graph configured without a storage engine remains transactional but in-memory, +so its data is lost when the server stops. + +[source,properties] +---- +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph +gremlin.tinkergraph.storage=graphbinary +gremlin.tinkergraph.storage.directory=/data/mygraph +---- + +[source,yaml] +---- +graphs: { + graph: conf/tinkerstoragegraph-persistent.properties } +---- + +Persistence is distinct from interchange. `TinkerMemoryGraph` is purely in-memory and does not persist. To move data in +or out of any TinkerGraph in an interchange format such as GraphML, GraphSON, or Gryo, use the `io()` step directly: + +[source,groovy] +---- +g.io("/tmp/graph.kryo").write().iterate() +g.io("/tmp/graph.kryo").read().iterate() +---- + [[tinkergraph-gql]] === TinkerGQL diff --git a/docs/src/upgrade/release-4.x.x.asciidoc b/docs/src/upgrade/release-4.x.x.asciidoc index 36a01282cca..2222688b14b 100644 --- a/docs/src/upgrade/release-4.x.x.asciidoc +++ b/docs/src/upgrade/release-4.x.x.asciidoc @@ -37,9 +37,9 @@ complete list of all the modifications that are part of this release. `TinkerGraph` is now an interface with two implementations. The in-memory implementation, previously the concrete `TinkerGraph` class, is named `TinkerMemoryGraph`. The transactional implementation, previously named -`TinkerTransactionGraph`, is named `TinkerStorageGraph` in anticipation of it gaining optional disk storage in a -future release. This structure clarifies the relationship between the two implementations and gives them a shared, -stable API. +`TinkerTransactionGraph`, is named `TinkerStorageGraph` because it can now also keep its data on disk, as described +in "TinkerGraph Disk Storage" below. This structure clarifies the relationship between the two implementations and +gives them a shared, stable API. Most code requires no change. `TinkerGraph.open()` still constructs the in-memory graph and the `gremlin.graph` configuration key still accepts `org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph`, which resolves to @@ -71,10 +71,95 @@ which affects providers that extended them. See: link:https://lists.apache.org/thread/2zt62kvfssh6xz5vnf2lk1g7cstq9vod[DISCUSS thread] +==== TinkerGraph Disk Storage + +TinkerGraph has always been a pure in-memory graph with some limited capability to write to disk via the `io()` step +or through that same function on close. This approach kept TinkerGraph simple, but limited its functionality for certain +use cases. For TinkerPop 4, TinkerGraph offers a basic file-based persistence option tied to its transaction capability. +This new feature not only gives TinkerGraph another operational dimension for production use cases, but also makes its +transactional capabilities have more purpose. + +The feature is offered via `TinkerStorageGraph` which gained the optional disk storage hinted by its rename. A storage +engine is selected with the new `gremlin.tinkergraph.storage` configuration key, and +`gremlin.tinkergraph.storage.directory` names the directory that holds the durable data. When a storage engine is +configured, each committed transaction is durably written to disk and the graph is rebuilt from that data when it is +opened again, so a graph survives a restart of the JVM. + +The reference engine, `graphbinary`, records committed transactions as an append-only log serialized with GraphBinary +and folds that log into a compact snapshot on close. The storage layer is pluggable: the value of the storage key may +also be the fully-qualified class name of a custom engine, following the same convention as the `IdManager` selection. +Because it stores values with GraphBinary, the `graphbinary` engine accepts only the types GraphBinary supports. Some +types that a Gryo file could hold, such as `java.util.Date` or `byte[]`, fail the commit and must be converted first, +for example to `OffsetDateTime` or `ByteBuffer`. The settings that shape stored data, such as the id managers and +the default vertex property cardinality, are recorded when a store is created and cannot change afterward. A graph +reopened without them uses the recorded values, and changing one requires exporting the graph and importing it into a +new store. + +[source,groovy] +---- +conf = new BaseConfiguration() +conf.setProperty('gremlin.graph', 'org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph') +conf.setProperty('gremlin.tinkergraph.storage', 'graphbinary') +conf.setProperty('gremlin.tinkergraph.storage.directory', '/data/mygraph') + +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +g.addV('person').property('name','marko').iterate() +g.tx().commit() +graph.close() + +// reopening the same location restores the committed data +graph = TinkerStorageGraph.open(conf) +g = traversal().with(graph) +c = g.V().count().next() +---- + +The in-memory `TinkerMemoryGraph` no longer persists to disk. Earlier versions of TinkerGraph would automatically read +from `gremlin.tinkergraph.graphLocation` on open and write back to it on close, using the `gremlin.tinkergraph.graphFormat` +interchange format. That automatic behavior is removed, and `TinkerMemoryGraph` now ignores both keys and reports +`FEATURE_PERSISTENCE` as `false`. Note that `gremlin.tinkergraph.graphLocation` named a *file* to be read and written +in an interchange format, while the new `gremlin.tinkergraph.storage.directory` names a *directory* managed by a +storage engine. They are deliberately different keys because they mean different things. Durable persistence is the +responsibility of `TinkerStorageGraph` and its storage engine, while moving data in and out of any graph in an +interchange format remains the job of the `io()` step: + +[source,groovy] +---- +// interchange, on demand, for any graph +g.io('/tmp/graph.kryo').write().iterate() +g.io('/tmp/graph.kryo').read().iterate() +---- + +Configurations that previously relied on the in-memory graph loading itself from `graphLocation` on open must either +call `io().read()` explicitly or switch to `TinkerStorageGraph` with a storage engine. Both +`gremlin.tinkergraph.graphLocation` and `gremlin.tinkergraph.graphFormat` are retired as TinkerGraph settings; no graph +implementation reads either one. They survive only in Gremlin Server's `SimpleAuthenticator`, which uses them to read +its credential store at startup. + +A configuration that still sets the retired keys opens without error and simply ignores them, so check older +configuration files and code for the following, all removed in this release: + +* `gremlin.tinkergraph.graphLocation`. Use `gremlin.tinkergraph.storage.directory` with `gremlin.tinkergraph.storage` +on a `TinkerStorageGraph` for durable data, or the `io()` step to read or write a file. +* `gremlin.tinkergraph.graphFormat`. Give the format to the `io()` step instead. +* The `TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION` and `TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT` constants, so +code that referenced them no longer compiles. +* The Gremlin Console sample `conf/tinkergraph-gryo.properties`, replaced by `conf/tinkergraph-storage.properties`, +which opens a `TinkerStorageGraph` with disk storage. + +See: link:https://tinkerpop.apache.org/docs/4.0.0/reference/#tinkergraph-gremlin-persistence[TinkerGraph Persistence] + === Upgrading for Providers ==== Graph System Providers +===== TinkerGraph Persistence Hooks Removed + +Code that extends `AbstractTinkerGraph` loses the hooks for the retired whole-graph save described in "TinkerGraph Disk +Storage" above: the protected `graphLocation` and `graphFormat` fields and the `loadGraph()` and `saveGraph()` methods +are removed, as is `requiresPersistence()` on the test provider `TinkerMemoryGraphProvider`. Durable storage is now +the job of a `TinkerStorage` engine on `TinkerStorageGraph`. + ==== Graph Driver Providers [llms-summary="Upgrade notes for the TinkerPop 4.0.0-beta.3 release: user-facing changes including declarative pattern matching improvements and other 4.0 features."] diff --git a/gremlin-console/conf/tinkergraph-gryo.properties b/gremlin-console/conf/tinkergraph-storage.properties similarity index 60% rename from gremlin-console/conf/tinkergraph-gryo.properties rename to gremlin-console/conf/tinkergraph-storage.properties index 4c2684237c6..9ade10d3d93 100644 --- a/gremlin-console/conf/tinkergraph-gryo.properties +++ b/gremlin-console/conf/tinkergraph-storage.properties @@ -15,7 +15,12 @@ # specific language governing permissions and limitations # under the License. -gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph +# Sample configuration for a durable, transactional TinkerStorageGraph. Opening a graph with this configuration +# (for example via GraphFactory) produces a TinkerStorageGraph that persists committed transactions to the +# configured storage directory. +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph -gremlin.tinkergraph.graphFormat=gryo -gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo +# built-in storage engine; without this key the graph is transactional but in-memory only +gremlin.tinkergraph.storage=graphbinary +# directory holding the durable data (created if absent; a location may be opened by only one graph at a time) +gremlin.tinkergraph.storage.directory=/tmp/tinkergraph diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/types/BinarySerializer.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/types/BinarySerializer.java index 1880b866c4a..a429b1b8324 100644 --- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/types/BinarySerializer.java +++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/types/BinarySerializer.java @@ -27,6 +27,14 @@ import java.nio.ByteBuffer; /** + * Serializes a {@link ByteBuffer} as the GraphBinary {@code binary} type. The value written is the buffer's entire + * backing array, as returned by {@link ByteBuffer#array()}, regardless of the buffer's position, limit or array + * offset, so a buffer that is a view of part of a larger array, such as one from + * {@link ByteBuffer#wrap(byte[], int, int)} or {@link ByteBuffer#slice()}, is written with every byte of that array. + * The buffer must also be backed by an accessible array, so a direct or read-only buffer cannot be written. Wrap + * exactly the bytes to send, as with {@link ByteBuffer#wrap(byte[])}. A value that is read back is a new heap buffer + * positioned at its first byte. + * * @author Stephen Mallette (http://stephen.genoprime.com) */ public class BinarySerializer extends SimpleTypeSerializer { diff --git a/gremlin-server/conf/tinkergraph-credentials.properties b/gremlin-server/conf/tinkergraph-credentials.properties index 4597fabcd3e..a39ff4158f1 100644 --- a/gremlin-server/conf/tinkergraph-credentials.properties +++ b/gremlin-server/conf/tinkergraph-credentials.properties @@ -16,5 +16,9 @@ # under the License. gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph gremlin.tinkergraph.vertexIdManager=LONG +# This credential store is an in-memory TinkerGraph. TinkerGraph no longer auto-loads from disk on open, so +# SimpleAuthenticator reads the store explicitly from graphLocation using graphFormat at startup. Both keys are +# retired as TinkerGraph settings and are honoured only for this load; they name an interchange FILE, not the +# storage DIRECTORY that a durable TinkerStorageGraph configures with gremlin.tinkergraph.storage.directory. gremlin.tinkergraph.graphLocation=data/credentials.kryo gremlin.tinkergraph.graphFormat=gryo \ No newline at end of file diff --git a/gremlin-server/conf/tinkerstoragegraph-persistent.properties b/gremlin-server/conf/tinkerstoragegraph-persistent.properties new file mode 100644 index 00000000000..207647b95b7 --- /dev/null +++ b/gremlin-server/conf/tinkerstoragegraph-persistent.properties @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Sample configuration for a durable, transactional TinkerStorageGraph on Gremlin Server. +gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph + +# Enables durable persistence. "graphbinary" is the built-in storage engine; the fully qualified class name of a +# custom TinkerStorage implementation may be used instead. Without this key the graph is transactional but in-memory. +gremlin.tinkergraph.storage=graphbinary +# Directory holding the durable data. Required when a storage engine is set. It is created if absent and may be +# opened by only one graph at a time (single writer, whether in this JVM or another process). +gremlin.tinkergraph.storage.directory=/tmp/tinkerstoragegraph + +# id managers and cardinalities are fixed when the store is created and recorded in its settings.properties +gremlin.tinkergraph.vertexIdManager=LONG +gremlin.tinkergraph.edgeIdManager=LONG +gremlin.tinkergraph.vertexPropertyIdManager=LONG + +# Optional storage tuning (defaults shown, uncomment to change): +# "commit" forces each transaction to disk (survives OS crash/power loss); "os" is faster but survives only a JVM crash +#gremlin.tinkergraph.storage.sync=commit +# auto-compact the append log once it grows past this many bytes (default 67108864 = 64MB; 0 disables) +#gremlin.tinkergraph.storage.compactThreshold=67108864 +# persist vertex-property ids, generated or explicit, across a reopen (default false; fixed when the store is created) +#gremlin.tinkergraph.storage.preserveVertexPropertyIds=false diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java index afdae0d2de7..28c82e93983 100644 --- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java +++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/auth/SimpleAuthenticator.java @@ -21,14 +21,17 @@ import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversal; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalDsl; import org.apache.tinkerpop.gremlin.groovy.jsr223.dsl.credential.CredentialTraversalSource; +import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.util.GraphFactory; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.mindrot.jbcrypt.BCrypt; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.File; import java.net.InetAddress; import java.nio.charset.StandardCharsets; import java.util.Arrays; @@ -80,12 +83,63 @@ public void setup(final Map config) { // have to create the indices because they are not stored in gryo final TinkerGraph tinkerGraph = (TinkerGraph) graph; tinkerGraph.createIndex(PROPERTY_USERNAME, Vertex.class); + + // TinkerGraph no longer auto-loads from graphLocation on open, so read the credential store here from + // the configured location/format. TinkerStorageGraph (which persists via its own storage engine) manages + // its own data and is left untouched. + loadCredentialStore(tinkerGraph); } credentialStore = graph.traversal(CredentialTraversalSource.class); logger.info("CredentialGraph initialized at {}", credentialStore); } + /** + * Reads the credential store into the supplied in-memory {@link TinkerGraph} from the + * {@code gremlin.tinkergraph.graphLocation} and {@code gremlin.tinkergraph.graphFormat} entries of its + * configuration, if present. Earlier versions of TinkerGraph read those keys themselves on open, so the + * credential store loaded as a side effect of {@code GraphFactory.open}; that automatic behaviour was removed + * and this method preserves it for the credential store alone. + *

+ * No TinkerGraph reads either key any more, which is why they are named here as literals rather than through + * constants. Despite the {@code gremlin.tinkergraph} prefix they are in effect settings of this authenticator, + * and belong in its own {@code config} block beside {@code credentialsDb} rather than in the graph's properties + * file. They are left in place here only to keep existing credential configurations working. A graph with a + * storage engine manages its own data and is skipped. + */ + private static void loadCredentialStore(final TinkerGraph graph) { + final Configuration conf = graph.configuration(); + final String location = conf.getString("gremlin.tinkergraph.graphLocation", null); + // a storage engine manages its own persistence and is not an interchange-format load + final String storage = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, null); + if (null == location || storage != null) + return; + + final File f = new File(location); + if (!f.exists() || !f.isFile()) + return; + + final String format = conf.getString("gremlin.tinkergraph.graphFormat", "gryo"); + try { + switch (format) { + case "graphml": + graph.io(IoCore.graphml()).readGraph(location); + break; + case "graphson": + graph.io(IoCore.graphson()).readGraph(location); + break; + case "gryo": + graph.io(IoCore.gryo()).readGraph(location); + break; + default: + graph.io(IoCore.createIoBuilder(format)).readGraph(location); + break; + } + } catch (Exception ex) { + throw new IllegalStateException(String.format("Could not load credential store at %s with format %s", location, format), ex); + } + } + @Override public SaslNegotiator newSaslNegotiator(final InetAddress remoteAddress) { return new PlainTextSaslAuthenticator(); diff --git a/tinkergraph-gremlin/DESIGN.md b/tinkergraph-gremlin/DESIGN.md new file mode 100644 index 00000000000..8abf6f32006 --- /dev/null +++ b/tinkergraph-gremlin/DESIGN.md @@ -0,0 +1,294 @@ + + +# TinkerStorageGraph Storage Design + +How `TinkerStorageGraph` keeps its data on disk, written for contributors and for anyone who needs to reason about +what is on disk after a failure. What a user can rely on, and how to configure and recover a store, is in the +Persistence section of the TinkerGraph reference documentation. This document covers the machinery behind those +guarantees: the files, their formats, the order of writes, and the checks made when a store is opened. + +## Model + +The graph in memory is authoritative. Storage is a write-ahead mirror of it: each commit is written to disk before it +is applied in memory, and opening a store rebuilds the in-memory graph from what is on disk. A persisted graph must +therefore fit in memory. + +A storage engine implements `TinkerStorage`: `open`, `replay`, `persist`, `flush`, `compact`, `maybeCompact`, `clear`, +`isReadOnly`, `isFailed` and `close`. `AbstractLogStorage` implements everything that does not depend on how an +element is encoded: the files, framing, checksums, replay, sync modes, compaction, recovery and failure handling. +`GraphBinaryStorage`, the `graphbinary` engine, supplies only the codec (`encodeCommit`, `decodeFrame`, +`writeSnapshot`, `beginReplay`). + +## Files + +A storage directory holds: + +| File | Contents | Written | +|---|---|---| +| `LOCK` | a random 16-byte token written by the graph that holds it; held with a file lock while a graph has the store open | at open | +| `VERSION` | magic `TGSB`, format version byte (currently `1`), store id (16-byte UUID), CRC32 of those bytes | when the store is created | +| `settings.properties` | the data-shaping settings the store was created with, as Java properties | when the store is created | +| `snapshot.gbin` | file header, then a dictionary frame and one frame per element | by compaction | +| `log.gbin` | file header, then one frame per commit since the last compaction | appended by every commit | +| `INDEXES` | one indexed key per line, `Vkey` or `Ekey`, with `\`, tab and newlines escaped | when an index is created or dropped | + +Every file other than the log is replaced whole: written to `.tmp`, forced to the device, renamed over the +target with an atomic move, and followed by a sync of the directory. A crash therefore leaves either the old file or +the new one. A `.tmp` file left behind by an interrupted write holds nothing the store needs, and a normal open removes +any stale `snapshot.gbin.tmp`, `log.gbin.tmp` or `VERSION.tmp`. A directory that cannot be synced on some network and +FUSE mounts is logged as a warning rather than failing the write, since only the rename's durability is in doubt. + +The snapshot and the log begin with the same 32-byte file header: the magic, an 8-byte generation, the store id and a +CRC32 of those bytes. The store id ties the files to one store, so a snapshot or log copied in from another store is +detected. The generation counts compactions and ties a log to the snapshot it follows. The checksum keeps a damaged +header from reading as a believable one, since a flipped bit in the generation would otherwise make an intact log look +as if it belonged to another snapshot. A snapshot whose header is damaged fails the open, recovery included, because +nothing in it can be trusted to belong to the store. A damaged log header fails a normal open, and recovery reads the +snapshot alone. When the snapshot and log agree on their store but `VERSION` names another, the error names `VERSION` +as the file that does not belong. `VERSION` carries a checksum of its own for the same reason, so a damaged marker is +reported as damaged rather than as one from another store. + +## Frames + +A snapshot frame is a 4-byte payload length, a 4-byte CRC32 of the payload, and the payload. A log frame is a 4-byte +payload length, a 4-byte CRC32 of that length, a 4-byte sequence number, a 4-byte CRC32 of the sequence number and +payload together, and the payload, so a log frame header is 16 bytes. Checksumming the length lets replay tell a frame +whose length was damaged apart from an append that was cut short: without it, a flipped bit that made a length claim +more bytes than remain would read as an interrupted append and silently drop every later commit. The sequence number +is the frame's position in its log, counting from 0 after every new log is installed. Replay requires it to match, so +an intact record that was repeated, reordered or copied in from another log fails a normal open, and a recovery open +stops at it, rather than replaying a transaction out of place. + +A payload is a varint entry count followed by that many entries, each an op byte and its body: + +| Op | Body | +|---|---| +| `1` put vertex | vertex record | +| `2` delete vertex | id | +| `3` put edge | edge record | +| `4` delete edge | id | +| `5` dictionary append | varint id, varint length, UTF-8 bytes | + +A vertex record is the id, a varint label count and the label refs, a byte saying whether vertex-property ids follow +each value, and then for each property key its ref, a varint value count, and per value the value, its id when the +flag is set, and a varint count of meta-properties, each a key ref and a value. An edge record is the id, the label +ref, the out-vertex id, the in-vertex id, and a varint property count, each a key ref and a value. + +Ids and values are GraphBinary scalars: a one-byte type code and the value without its value flag, with +`UNSPECIFIED_NULL` for a null. Labels and property keys are dictionary refs instead of strings. Replay applies records +in order with last write wins per element id, the snapshot first and then the log. + +### Dictionary + +Labels, property keys and meta-property keys share one dictionary of dense ids assigned in order of first appearance. +A commit that introduces a string emits its dictionary append inside the same frame, ahead of the entries that use it, +so a torn frame loses an entry and its users together. Numbering never changes. Compaction writes the whole +dictionary as the snapshot's first frame and keeps the existing ids. No log is replayed against a newer snapshot, since +the old log left by an interrupted compaction is set aside unread, so the numbering would survive a renumbering too; it +is kept because doing so costs nothing. If encoding a commit fails, +for any exception or error, the entries it added are removed again so that a later commit cannot refer to an id that +never reached disk. + +### Bounded decoding + +A frame's length is checked against the file before its payload is allocated. Inside a payload every length a value +declares, for strings, binary values, `BigInteger`, lists, sets and maps, is checked against the bytes left in the +record before it is allocated, at the top level, inside collections and in the parts a serializer reads with their +type already known, such as the label list of a vertex or edge value, and a bulked list, which the engine never +writes, is rejected. A value cut off by the end of its record, or one the serializers cannot read, is reported as a +corrupt frame naming the file rather than surfacing as a runtime exception or exhausting memory. + +## Commit + +A transaction that changed nothing does not touch storage. Otherwise, under the graph's `storageCommitLock`: + +1. `persist` encodes the changeset into one frame and appends it to the log, opening the log if needed. +2. `flush` flushes the stream and, under `sync=commit`, forces it to the device, then checks that the store is still in + place. +3. The indexes are committed, then the elements are published in memory. +4. `maybeCompact` compacts if the log has grown past the threshold. + +The in-memory publish is held inside the lock because compaction snapshots the graph and then discards the log. A +compaction landing between a persisted and a published commit would snapshot without that commit and delete the +record that held it. Holding the publish under the lock closes that window and makes commits become visible in the +order they were recorded. Indexes are committed only after the write succeeds, because rollback can undo only a +transaction's own index changes, not a shared index already committed. + +A commit that cannot be encoded fails on its own and the engine carries on, since nothing was written. A failure to +write or flush the log stops the engine (fail-stop): what reached disk is unknown, so every later commit is refused +until the graph is reopened and replays what is actually there. A failed automatic compaction does not fail the commit +that triggered it. The next attempt waits for another threshold's worth of log. An `Error` from it, such as running out +of memory while writing the snapshot, is not swallowed: `commit()` throws it as it is, and the transaction is committed +and the engine still accepts commits. A failure of any kind after the new snapshot is renamed into place, while the +empty log is installed, stops the engine, since the old log can then only stay as it is. + +Each frame is held in one buffer, so a single commit, or a single element in a snapshot, is limited to +`Integer.MAX_VALUE - 8` bytes. A larger one fails during encoding, before anything is written. + +## Compaction + +Compaction closes the log, writes `snapshot.gbin.tmp` with the next generation, the dictionary frame and one frame per +committed element, forces and renames it over the snapshot and syncs the directory. It then installs a header-only log +of the same new generation through `log.gbin.tmp`, and records the new log's identity. The log is never deleted, so a +store with data always has a log, and a missing log is always damage. + +A crash after the snapshot rename but before the new log is installed leaves a snapshot one generation ahead of its +log. That log's commits are already in the snapshot, so open completes the compaction by installing the empty log. + +Compaction reads committed state only (`getUnmodified`), never a thread's open transaction. With nothing logged since +the last compaction it does nothing, since the snapshot already holds every commit, so a `close()` after a session that +only read leaves the store untouched. + +## Opening a Store + +1. The directory is locked: exclusively, or, for a recover open of a directory the process cannot write, with a shared + lock on an existing `LOCK` file or with no lock at all. +2. The configured data-shaping settings are validated as the graph will read them, then reconciled with + `settings.properties`: a new store records them, an existing store fills unset ones from the record and refuses + any that differ, and a record missing a fixed key, or holding a value no store is created with (not a built-in + name, or for the id managers and the engine, not a class name), is treated as damaged. A first open that fails + removes the settings and `VERSION` it wrote, so the corrected open is again a first open. +3. `VERSION` is created for a new store, or read and checked. Data without a `VERSION` fails a normal open. +4. The snapshot and log headers are compared: + + | Snapshot | Log | Result | + |---|---|---| + | absent | absent | a new store gets an empty log of generation 0; a store created by an earlier open has lost its log and fails | + | present | absent | the log was removed and fails | + | absent | generation 0 | the log alone holds the data | + | absent | generation above 0 | the snapshot was removed and fails | + | generation G | generation G | normal | + | generation G + 1 | generation G | compaction crash window, completed by installing an empty log | + | other | other | the files do not belong together and fail | + + A store id that differs from `VERSION` fails the open as well. +5. The snapshot and then the log are replayed into the graph, and the recorded indexes are rebuilt over the result. + +### Torn Log Tails + +A crash during an append leaves a log that ends partway through a frame. That commit was never acknowledged, so +replay stops at the end of the last complete frame and the log is cut back to it, forced to the device, before +anything new is appended. Otherwise new commits would land after the torn bytes and the next replay would stop there +and drop them. Replay treats these endings as torn: + +- fewer bytes than a frame header remain; +- a header whose length passes its checksum but claims more bytes than remain; +- every byte after the last complete frame is zero, which some filesystems leave when a crash extends a file before its + data is written (an all-zero header can never be valid, since the CRC32 of length 0 is `0x2144DF1C`); +- the last frame fails its payload checksum, ends exactly at the end of the log, and every byte from the log's last + 512-byte boundary to its end is zero and part of that frame's payload, the state a crash leaves when the header + reached the device but the end of the payload did not. Devices write whole sectors, so data that never reached one + reads back as zeros from a sector boundary on. + +Anything else that fails a checksum is damage and fails a normal open, including a damaged final frame that merely ends +in a zero byte, as most vertex and edge records do. The cost is that a payload torn within its last sector, with the +zeros starting past the boundary or the whole payload shorter than a sector, also fails the open and needs a recover +open. Only the final frame is ever cut by the last rule, and under `sync=commit` an acknowledged commit is lost that +way only if the device zeroed the last sector of a record it had already been forced to hold. A log damaged so that it ends exactly on a frame boundary, such as one +cut back to its header, reads as a log with fewer commits, and the loss is not detected. The sequence numbers do not +change that, since what remains is still numbered from 0. + +## Recovery Mode + +An open with `gremlin.tinkergraph.storage.recover` writes nothing: no `VERSION`, no settings, no truncation, no +compaction, and no commits. Each log frame is applied atomically through a journaled map, and replay stops at the first +frame that cannot be read, keeping every transaction before it. An unreadable snapshot frame is skipped, except the +first, which holds the dictionary every other record depends on, and edges that lose an endpoint as a result are +dropped. A log that is missing, or belongs to another snapshot or store, is ignored and the snapshot read alone. A +`VERSION` that is missing or damaged is tolerated: the store is then read as the current format, with the store id of +its snapshot, so a log from another store is still set aside. What was left out is reported in one warning. + +A recovered graph still accepts in-memory changes and refuses them at commit, and `createIndex`, `dropIndex` and +`clear()` work in memory without being recorded. Reads, and commits of transactions that only read, work as usual. + +## Removed or Replaced Stores + +The log's file identity (`BasicFileAttributes.fileKey()`) is recorded when a store is opened and again whenever +compaction installs a new log. It is checked before the log is opened for append, after each flush, and at +compaction, along with the presence of the directory and the log. A change means the store was removed or replaced +while the graph was open, possibly by another graph that has since created a store at the same path, and the engine +fail-stops with a `StoreRemovedException`. From then on nothing is written to the directory, not even by `close()`, +and the committed data stays readable in memory for export. On a platform without file keys only the presence checks +apply. + +The `LOCK` file's identity is recorded at open and checked with the log's, and also before each commit is appended, so +a graph whose `LOCK` was deleted or replaced, which lets a second graph lock a new file at that path and open the +store, fail-stops before it writes anything more. Acquiring the lock writes a random token through the locked channel +and reads it back through the path. A graph whose open fails removes the `LOCK` it created, so without that check a +graph could hold a lock on a file that has just been unlinked and keep no one out. A recovery open skips the token, +since it leaves every file as it found it and writes nothing to the store. The handle the token is read back through +stays open as long as the lock is held, because closing any handle on a file releases every POSIX lock the process +holds on it while the JVM goes on reporting the lock held. The lock covers a single byte past the token rather than +the whole file, because on Windows a lock keeps every other handle from reading the bytes it covers. For the same +POSIX reason a second open of a directory a graph in this JVM already holds is refused before any handle on its `LOCK` +is opened, by a process-wide record of the directories held: a refused open that opened a handle would have to close +it, releasing the holder's lock. + +A commit that finds the store moved fails and is rolled back in memory, although its frame may already be durable at +the store's new location, because the check follows the forced write. + +## Fail-Stop + +`isFailed()` reports that the engine has stopped after a write failure or a removed store. A fail-stopped graph keeps +its data readable for export. It refuses commits that change anything, explicit `compact()` and `clear()`, and it +changes indexes in memory only, without writing `INDEXES`. + +The graph also fail-stops the engine through `failStop()` when a commit's changes were persisted and flushed but +applying them in memory (the index or element commit) then failed. The durable frame is the commit point, as a commit +record in a database's log is, so the transaction is not reported as rolled back: `doCommit` throws a +`TransactionException` saying that it was recorded and will be present after a reopen. What was applied before the +failure, an index or some of the elements, is not undone, so memory may hold part of the transaction until the +reopen. It no longer matches the log, so nothing more may be written from it. A fail-stopped engine +neither commits nor compacts, `close()` included, and the reopen replays the frame. + +## `clear()` + +`clear()` keeps its in-memory meaning: the whole graph is emptied at once, outside any transaction. Under the commit +lock it rolls back the calling thread's open transaction, marks every element container removed so that a +transaction on another thread that still holds one fails to commit with a conflict, clears memory, and then calls +`TinkerStorage.clear`, which for `AbstractLogStorage` compacts the now-empty graph and rewrites `INDEXES`. That +compaction runs even with nothing logged since the last one, since the snapshot then still holds the data the graph +no longer has. If emptying +the store fails, the engine fail-stops, since the store then holds either the old data or nothing and later commits +must not be appended to data the graph no longer has. The graph calls `failStop()` itself when `TinkerStorage.clear` +throws, so this holds for an engine that does not stop itself, and the SPI has no default `clear`, so that no engine +inherits one built on a compaction that skips a store with nothing new in its log. A closed or fail-stopped graph refuses `clear()`, and a +recovered graph clears memory only. + +Adding an element does not take the commit lock, so `clear()` empties memory one container at a time, marking each +before removing it. An element added meanwhile is either marked and removed, and its commit conflicts, or missed and +kept, and its commit lands in the emptied store. Either way memory and the store agree. + +## Index Definitions + +Indexed keys are recorded in `INDEXES` beside the store rather than in the log. An index changes how fast a lookup +runs, never its result, so a definition lost to a crash costs a rebuild and no data. An unreadable `INDEXES` file is +logged and the graph opens without indexes. Writes to the file are serialized under the commit lock and stop once the +graph is closed, recovered or fail-stopped. + +## Settings Fixed at Creation + +The storage engine, the three id managers, `defaultVertexPropertyCardinality`, `vertexLabelCardinality`, +`allowNullPropertyValues` and `preserveVertexPropertyIds` shape what is stored and how it is read back, so they are +recorded once and enforced on every open. `preserveVertexPropertyIds` is among them even though vertex records say +whether they carry vertex-property ids: every compaction rewrites the store under the setting it was opened with, and +`close()` compacts after any commit, so one open with the setting off would strip the ids from a store that kept them. +Settings that only affect how the engine runs (`sync`, `compactThreshold`, `recover`) may change between opens. +Cardinality is not recorded per property: replay applies the graph-wide default, as the `io()` step does. diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java index fac8d87b2df..51654406f0b 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/AbstractTinkerGraph.java @@ -28,7 +28,6 @@ import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.structure.io.Io; -import org.apache.tinkerpop.gremlin.structure.io.IoCore; import org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONVersion; import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoVersion; import org.apache.tinkerpop.gremlin.structure.util.StringFactory; @@ -36,15 +35,23 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.computer.TinkerGraphComputerView; import org.apache.tinkerpop.gremlin.gql.GqlDeclarativeMatchStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DefaultStorage; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; -import java.io.File; import java.lang.reflect.InvocationTargetException; +import java.util.Arrays; import java.util.Collections; import java.util.Iterator; +import java.util.Locale; import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.locks.ReentrantLock; +import java.util.stream.Collectors; /** * Base class for {@link TinkerMemoryGraph} and {@link TinkerStorageGraph}. @@ -55,6 +62,8 @@ */ public abstract class AbstractTinkerGraph implements TinkerGraph { + private static final Logger logger = LoggerFactory.getLogger(AbstractTinkerGraph.class); + protected AtomicLong currentId = new AtomicLong(-1L); protected Map vertexProperties = new ConcurrentHashMap<>(); @@ -76,8 +85,42 @@ public abstract class AbstractTinkerGraph implements TinkerGraph { protected TinkerServiceRegistry serviceRegistry; protected Configuration configuration; - protected String graphLocation; - protected String graphFormat; + + /** + * The filesystem directory backing the storage engine, from {@code gremlin.tinkergraph.storage.directory}, or + * {@code null} when the graph holds data only in memory. + */ + protected String storageDirectory; + + /** + * The pluggable durable storage engine, or {@code null} when the graph holds data only in memory. Only set by + * transactional implementations that support persistence. + */ + protected TinkerStorage storage; + + /** + * Exclusive lock on the storage directory, held for the graph's lifetime so no second graph — in this or another + * process — can open the same location and corrupt its files. {@code null} when the graph is purely in-memory, and + * once it is closed. Released and cleared under {@link #storageCommitLock}. + */ + protected DirectoryLock directoryLock; + + /** + * Serializes the durable write of a committing transaction. TinkerGraph transactions lock only their own changed + * elements, so two commits touching disjoint elements run their commit paths concurrently; without this lock they + * would both write to the storage engine's single append log at once and interleave (corrupt) its records. Held + * from the engine's persist/flush through the index commit, the in-memory apply and any automatic compaction, so + * a compaction never snapshots a graph missing a transaction the log already holds, and transactions become + * visible in the order they were recorded. Commits of disjoint elements still proceed in parallel up to that + * point. Fair, so committers are served in arrival order and none is starved. + */ + protected final ReentrantLock storageCommitLock = new ReentrantLock(true); + + /** + * Guard set while a graph is replaying its storage log on open. While {@code true}, mutations must not be + * re-persisted, otherwise replay would append the loaded data back to the log. + */ + protected volatile boolean loading = false; /** * {@inheritDoc} @@ -243,54 +286,6 @@ public boolean hasVertexProperty(final Object id) { return vertexProperties.containsKey(id); } - protected void loadGraph() { - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) { - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).readGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).readGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).readGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).readGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not load graph at %s with %s", graphLocation, graphFormat), ex); - } - } - } - - protected void saveGraph() { - final File f = new File(graphLocation); - if (f.exists()) { - f.delete(); - } else { - final File parent = f.getParentFile(); - - // the parent would be null in the case of an relative path if the graphLocation was simply: "f.gryo" - if (parent != null && !parent.exists()) { - parent.mkdirs(); - } - } - - try { - if (graphFormat.equals("graphml")) { - io(IoCore.graphml()).writeGraph(graphLocation); - } else if (graphFormat.equals("graphson")) { - io(IoCore.graphson()).writeGraph(graphLocation); - } else if (graphFormat.equals("gryo")) { - io(IoCore.gryo()).writeGraph(graphLocation); - } else { - io(IoCore.createIoBuilder(graphFormat)).writeGraph(graphLocation); - } - } catch (Exception ex) { - throw new RuntimeException(String.format("Could not save graph at %s with %s", graphLocation, graphFormat), ex); - } - } - - @Override public I io(final Io.Builder builder) { if (builder.requiresVersion(GryoVersion.V1_0) || builder.requiresVersion(GraphSONVersion.V1_0)) @@ -337,15 +332,54 @@ public void clear() { } /** - * This method only has an effect if the {@link TinkerGraph#GREMLIN_TINKERGRAPH_GRAPH_LOCATION} is set, in which case the - * data in the graph is persisted to that location. This method may be called multiple times and does not release - * resources. + * Closes the graph, releasing any resources held by its {@link TinkerServiceRegistry}. This method may be called + * multiple times and is a no-op with respect to graph data for the in-memory implementation. Transactional + * implementations that are backed by a {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} + * engine flush, compact and close that engine here. A failure to flush or compact is logged rather than thrown, + * since the next open replays the log, and the engine and its directory lock are released regardless. */ @Override public void close() { - if (graphLocation != null) saveGraph(); - serviceRegistry.close(); - GqlDeclarativeMatchStrategy.evict(this); + try { + if (storage != null) { + // serialize against concurrent commit writes: close flushes, compacts, and closes the log, which must + // not interleave with a transaction appending to it. + storageCommitLock.lock(); + try { + try { + storage.flush(); + storage.compact(this); + } catch (RuntimeException ex) { + // every acknowledged commit was flushed when it committed and compaction leaves the snapshot + // and log intact, so the next open replays what this one could not fold. The engine must still + // be closed, or a later commit on this graph would reopen a log whose lock is released below. + logger.warn(String.format("Could not flush and compact storage at %s on close", + storageDirectory), ex); + } finally { + storage.close(); + } + } finally { + // release the exclusive directory lock last, so the location is only reopenable once the engine + // has fully released its files. Still under the commit lock, so nothing that writes into the + // directory while holding it, such as recording index definitions, can follow the release. + try { + if (directoryLock != null) { + // a store opened for recovery is left exactly as it was found, LOCK file included + if (storage.isReadOnly()) + directoryLock.closeAndRemoveIfCreated(); + else + directoryLock.close(); + directoryLock = null; + } + } finally { + storageCommitLock.unlock(); + } + } + } + } finally { + serviceRegistry.close(); + GqlDeclarativeMatchStrategy.evict(this); + } } @Override @@ -555,6 +589,52 @@ protected static IdManager selectIdManager(final Configur } } + ///////////// Storage engine /////////////// + /** + * The committed vertices of the graph, for a storage engine to snapshot during compaction. Unlike {@link #vertices()}, + * this view excludes any uncommitted transaction-local state, so compaction never persists changes that a caller has + * not committed. The base implementation, which has no transactional isolation, is equivalent to {@link #vertices()}; + * a transactional subclass overrides it to read only committed element state. + */ + public Iterator committedVertices() { + return vertices(); + } + + /** + * The committed edges of the graph, for a storage engine to snapshot during compaction. The edge counterpart of + * {@link #committedVertices()}. + */ + public Iterator committedEdges() { + return edges(); + } + + /** + * Construct a {@link TinkerStorage} engine from the TinkerGraph {@code Configuration}, or return {@code null} when + * no storage engine is configured. The configuration value is either a {@link DefaultStorage} enum name (matched + * case-insensitively, e.g. {@code graphbinary}) or the fully-qualified class name of a {@link TinkerStorage} + * implementation with a public no-argument constructor. Mirrors {@link #selectIdManager}. + */ + protected static TinkerStorage selectStorage(final Configuration config, final String configKey) { + final String configured = config.getString(configKey, null); + if (null == configured) + return null; + final String storageConfigValue = configured.trim(); + try { + return DefaultStorage.valueOf(storageConfigValue.toUpperCase(Locale.ROOT)).get(); + } catch (IllegalArgumentException iae) { + try { + return (TinkerStorage) Class.forName(storageConfigValue).getDeclaredConstructor().newInstance(); + } catch (Exception ex) { + throw new IllegalStateException(String.format( + "Could not configure TinkerGraph storage engine with '%s': it is neither a built-in engine " + + "(%s) nor the class name of a TinkerStorage implementation with a public no-argument " + + "constructor", + storageConfigValue, Arrays.stream(DefaultStorage.values()) + .map(e -> e.name().toLowerCase(Locale.ROOT)).collect(Collectors.joining(", "))), ex); + } + } + } + protected TinkerServiceRegistry.TinkerServiceFactory instantiate(final String className) { try { return (TinkerServiceRegistry.TinkerServiceFactory) Class.forName(className).getConstructor(AbstractTinkerGraph.class).newInstance(this); diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerElementContainer.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerElementContainer.java index 8c742f33ce7..761f3713251 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerElementContainer.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerElementContainer.java @@ -38,7 +38,7 @@ final class TinkerElementContainer { /** * Used to separate deleted elements from {@code null} ones. */ - private boolean isDeleted = false; + private volatile boolean isDeleted = false; /** * Value of elements updated in current transaction. */ @@ -193,6 +193,14 @@ public void setDraft(final T transactionElement, final TinkerTransaction tx) { tx.markChanged(this); } + /** + * Mark the element as removed by {@code TinkerGraph.clear()}, so a transaction that still holds this container + * fails to commit with a conflict instead of bringing the element back. + */ + void markCleared() { + isDeleted = true; + } + /** * Used to understand if elements was changed by other transaction. */ diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java index dc931b9061a..bc7a4bbd693 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java @@ -45,8 +45,55 @@ public interface TinkerGraph extends Graph { String GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER = "gremlin.tinkergraph.edgeIdManager"; String GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER = "gremlin.tinkergraph.vertexPropertyIdManager"; String GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY = "gremlin.tinkergraph.defaultVertexPropertyCardinality"; - String GREMLIN_TINKERGRAPH_GRAPH_LOCATION = "gremlin.tinkergraph.graphLocation"; - String GREMLIN_TINKERGRAPH_GRAPH_FORMAT = "gremlin.tinkergraph.graphFormat"; + /** + * Selects the pluggable storage engine used by {@link TinkerStorageGraph} to durably persist transactions to the + * {@link #GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY} directory. The value is either a + * {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DefaultStorage} enum name (e.g. + * {@code graphbinary}) or the fully-qualified class name of a + * {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} implementation. When unset, the + * graph holds data only in memory. Ignored by {@link TinkerMemoryGraph}, which logs a warning when this or any + * other {@code gremlin.tinkergraph.storage.*} key is set. + */ + String GREMLIN_TINKERGRAPH_STORAGE = "gremlin.tinkergraph.storage"; + /** + * The filesystem directory that a {@link TinkerStorageGraph} storage engine uses for its durable data. Ignored, with + * a warning, by {@link TinkerMemoryGraph}, which is purely in-memory. Only meaningful when + * {@link #GREMLIN_TINKERGRAPH_STORAGE} is also set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY = "gremlin.tinkergraph.storage.directory"; + /** + * The durability mode a {@link TinkerStorageGraph} storage engine applies on commit. Either {@code commit} + * (default) to {@code fsync} every commit so acknowledged commits survive an OS crash or power loss, or {@code os} + * to only flush to the operating system so commits survive a JVM process crash but may be lost on OS crash or + * power loss. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is set. See + * {@code org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.SyncMode}. + */ + String GREMLIN_TINKERGRAPH_STORAGE_SYNC = "gremlin.tinkergraph.storage.sync"; + /** + * The size in bytes at which a {@link TinkerStorageGraph} storage engine automatically compacts its append log on + * commit, bounding the log growth (and restart replay cost) of a long-running graph that is never explicitly + * closed. Defaults to 67108864 (64 MB). Set to {@code 0} to disable automatic compaction and rely on + * {@code close()} or an explicit {@code compact()}. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is + * set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD = "gremlin.tinkergraph.storage.compactThreshold"; + /** + * Whether a {@link TinkerStorageGraph} storage engine persists vertex-property ids so they are stable across a + * close and reopen. Defaults to {@code false}: every vertex-property id, whether generated or given explicitly, is + * regenerated on load, which keeps the store smaller. Vertex and edge ids are always preserved regardless of this + * setting. Fixed when the store is created, like the other settings that shape stored data, since every + * compaction rewrites the store under the setting it was opened with. Only meaningful when + * {@link #GREMLIN_TINKERGRAPH_STORAGE} is set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS = "gremlin.tinkergraph.storage.preserveVertexPropertyIds"; + + /** + * Whether a {@link TinkerStorageGraph} opens a damaged store by recovering what it can rather than failing. + * Defaults to {@code false}. When enabled, damaged records are skipped where that is safe, the graph is read-only, + * and nothing in the storage directory is changed, so the recovered graph can be exported with {@code g.io()} into + * a new storage directory. Only meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is set. + */ + String GREMLIN_TINKERGRAPH_STORAGE_RECOVER = "gremlin.tinkergraph.storage.recover"; String GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES = "gremlin.tinkergraph.allowNullPropertyValues"; String GREMLIN_TINKERGRAPH_SERVICE = "gremlin.tinkergraph.service"; String GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY = "gremlin.tinkergraph.vertexLabelCardinality"; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java index fc1afcad33e..6e071258639 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraph.java @@ -36,7 +36,10 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphStepStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; @@ -50,8 +53,10 @@ import java.util.concurrent.atomic.AtomicInteger; /** - * The in-memory (with optional persistence on calls to {@link #close()}) implementation of the {@link TinkerGraph} - * interface and the reference implementation of the property graph interfaces provided by TinkerPop. + * The purely in-memory implementation of the {@link TinkerGraph} interface and the reference implementation of the + * property graph interfaces provided by TinkerPop. This graph holds no data across JVM restarts; use + * {@code g.io(...).write()} / {@code g.io(...).read()} for interchange, or {@link TinkerStorageGraph} for durable + * persistence. * * @author Marko A. Rodriguez (http://markorodriguez.com) * @author Stephen Mallette (http://stephen.genoprime.com) @@ -62,6 +67,8 @@ @Graph.OptIn(Graph.OptIn.SUITE_PROCESS_EMBEDDED_COMPUTER) public class TinkerMemoryGraph extends AbstractTinkerGraph { + private static final Logger logger = LoggerFactory.getLogger(TinkerMemoryGraph.class); + static { TraversalStrategies.GlobalCache.registerStrategies(TinkerMemoryGraph.class, TraversalStrategies.GlobalCache.getStrategies(Graph.class).clone().addStrategies( TinkerGraphStepStrategy.instance(), @@ -86,6 +93,7 @@ public class TinkerMemoryGraph extends AbstractTinkerGraph { */ TinkerMemoryGraph(final Configuration configuration) { this.configuration = configuration; + warnOfStorageConfiguration(configuration); vertexIdManager = selectIdManager(configuration, GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, Vertex.class); edgeIdManager = selectIdManager(configuration, GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER, Edge.class); vertexPropertyIdManager = selectIdManager(configuration, GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, VertexProperty.class); @@ -99,20 +107,23 @@ public class TinkerMemoryGraph extends AbstractTinkerGraph { defaultVertexLabel = Vertex.DEFAULT_LABEL; defaultEdgeLabel = Edge.DEFAULT_LABEL; - graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); - - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); - serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); } + /** + * Storage settings are ignored here like any other key the in-memory graph does not use, but they ask for the data + * to outlive the graph, which it then silently does not, so a configuration that sets them is warned about. + */ + private static void warnOfStorageConfiguration(final Configuration configuration) { + final List keys = new ArrayList<>(); + configuration.getKeys(GREMLIN_TINKERGRAPH_STORAGE).forEachRemaining(keys::add); + if (!keys.isEmpty()) + logger.warn("{} keeps its data only in memory and ignores the storage configuration {}; use {} to persist " + + "the graph", TinkerMemoryGraph.class.getSimpleName(), keys, TinkerStorageGraph.class.getName()); + } + /** * Open a new {@link TinkerMemoryGraph} instance. *

@@ -447,6 +458,11 @@ public boolean supportsThreadedTransactions() { return false; } + @Override + public boolean supportsPersistence() { + return false; + } + @Override public boolean supportsServiceCall() { return true; diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java index c74c863e349..cceb500b921 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java @@ -35,8 +35,14 @@ import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphCountStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphStepStrategy; import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.IndexDefinitions; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.StorageSettings; import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; import java.util.Arrays; import java.util.Collections; import java.util.Iterator; @@ -45,11 +51,20 @@ import java.util.Objects; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Function; /** - * The transactional implementation of the {@link TinkerGraph} interface, in-memory with optional persistence on - * calls to {@link #close()}. It is planned that this implementation will optionally support simple storage to disk - * built on its transaction functionality. + * The transactional implementation of the {@link TinkerGraph} interface. It provides {@code read committed} + * transaction isolation with optimistic locking and, when a storage engine is configured, durable persistence to + * disk. With no storage engine configured it is an in-memory transactional graph that retains nothing across + * restarts. + *

+ * Persistence is pluggable through the {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage} + * SPI and enabled with the {@code gremlin.tinkergraph.storage} and {@code gremlin.tinkergraph.storage.directory} + * configuration keys. Each committed transaction is written through to the storage engine before the in-memory commit + * is applied, and reopening the same location replays the persisted commits to rebuild the graph. A storage location + * is single-writer: it is guarded by an exclusive {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock} + * so a second open of the same directory fails rather than corrupting the data. * * @author Valentyn Kahamlyk */ @@ -74,6 +89,12 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { private final TinkerTransaction transaction = new TinkerTransaction(this); + /** + * Set while indexes recorded for the store are being recreated on open, so applying them does not rewrite the + * file they were just read from. + */ + private boolean restoringIndexes = false; + private final Map> vertices = new ConcurrentHashMap<>(); private final Map> edges = new ConcurrentHashMap<>(); @@ -81,6 +102,116 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph { * An empty private constructor that initializes {@link TinkerStorageGraph}. */ private TinkerStorageGraph(final Configuration configuration) { + storageDirectory = configuration.getString(GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null); + storage = selectStorage(configuration, GREMLIN_TINKERGRAPH_STORAGE); + + // a blank directory would otherwise resolve against the working directory and quietly create a store there + if (storage != null && (null == storageDirectory || storageDirectory.trim().isEmpty())) + throw new IllegalStateException(String.format("The %s must be specified when %s is set", + GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, GREMLIN_TINKERGRAPH_STORAGE)); + // a directory without an engine asks for durability the graph would not provide, so its commits would be lost + // on restart without a word + if (null == storage && storageDirectory != null && !storageDirectory.trim().isEmpty()) + throw new IllegalStateException(String.format("The %s must be specified when %s is set", + GREMLIN_TINKERGRAPH_STORAGE, GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + + if (null == storage) { + initialize(configuration); + return; + } + + // take an exclusive lock on the storage directory before anything touches its files, so a second graph on the + // same location fails fast rather than corrupting it. The directory must exist to hold the lock. + final File dir = new File(storageDirectory); + final boolean recovering = configuration.getBoolean(GREMLIN_TINKERGRAPH_STORAGE_RECOVER, false); + // a recovery open never changes the storage directory, so it does not create one either + if (recovering && !dir.isDirectory()) + throw new IllegalStateException(String.format("There is no storage directory at %s to open with %s", + dir, GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + if (!dir.isDirectory()) { + try { + Files.createDirectories(dir.toPath()); + } catch (IOException ex) { + // the cause names the problem, such as a missing parent or a file where a directory should be + throw new IllegalStateException(String.format("Could not create storage directory %s: %s", dir, ex), + ex); + } + } + directoryLock = recovering ? DirectoryLock.acquireForRecovery(dir) : DirectoryLock.acquire(dir); + final boolean settingsRecorded = new File(dir, StorageSettings.SETTINGS_FILE).exists(); + try { + // a value the graph cannot use fails here, before a new store records it as a setting it was created + // with, which would then refuse the corrected value on the next open + validateDataShapingSettings(configuration); + // the settings that shape stored data are fixed when the store is created, so read them before anything + // that depends on them is set up + initialize(StorageSettings.reconcile(configuration, dir)); + storage.open(this, this.configuration); + loading = true; + try { + storage.replay(this); + } finally { + loading = false; + } + // recreate the recorded indexes now that replay has rebuilt the elements they cover, so + // createKeyIndex backfills over the restored data rather than only over writes that follow + restoreIndexes(dir); + } catch (RuntimeException | Error ex) { + // don't leak the lock if the engine fails to open or replay, nor leave a LOCK file behind in a directory + // that turned out not to be a store this graph can open + try { + // a first open that failed leaves the directory as it found it, without settings it recorded + if (!settingsRecorded && !recovering) + StorageSettings.removeIfOnlySettings(dir); + directoryLock.closeAndRemoveIfCreated(); + } catch (RuntimeException cleanup) { + ex.addSuppressed(cleanup); + } + directoryLock = null; + // nor the configured services or what the engine holds. Closing the engine only releases it, since the + // graph's close() is what compacts, so nothing is written from the partly opened graph. + try { + if (serviceRegistry != null) + serviceRegistry.close(); + } catch (RuntimeException cleanup) { + ex.addSuppressed(cleanup); + } + try { + storage.close(); + } catch (RuntimeException cleanup) { + ex.addSuppressed(cleanup); + } + throw ex; + } + } + + /** + * Check the configured id managers and cardinalities the way {@link #initialize} reads them, without keeping + * anything, so an unusable value fails the open before it can be recorded for a new store. + */ + private static void validateDataShapingSettings(final Configuration configuration) { + selectIdManager(configuration, GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, Vertex.class); + selectIdManager(configuration, GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER, Edge.class); + selectIdManager(configuration, GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, VertexProperty.class); + parseSetting(configuration, GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, + VertexProperty.Cardinality.single.name(), VertexProperty.Cardinality::valueOf, + VertexProperty.Cardinality.values()); + parseSetting(configuration, GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY, LabelCardinality.ONE.name(), + LabelCardinality::valueOf, LabelCardinality.values()); + } + + private static void parseSetting(final Configuration configuration, final String key, final String defaultValue, + final Function parser, final Object[] allowed) { + final String value = configuration.getString(key, defaultValue); + try { + parser.apply(value); + } catch (IllegalArgumentException ex) { + throw new IllegalStateException(String.format("Could not configure TinkerGraph %s with %s; it must be one " + + "of %s", key, value, Arrays.toString(allowed)), ex); + } + } + + private void initialize(final Configuration configuration) { this.configuration = configuration; vertexIdManager = selectIdManager(configuration, GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, Vertex.class); edgeIdManager = selectIdManager(configuration, GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER, Edge.class); @@ -95,15 +226,6 @@ private TinkerStorageGraph(final Configuration configuration) { defaultVertexLabel = Vertex.DEFAULT_LABEL; defaultEdgeLabel = Edge.DEFAULT_LABEL; - graphLocation = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null); - graphFormat = configuration.getString(GREMLIN_TINKERGRAPH_GRAPH_FORMAT, null); - - if ((graphLocation != null && null == graphFormat) || (null == graphLocation && graphFormat != null)) - throw new IllegalStateException(String.format("The %s and %s must both be specified if either is present", - GREMLIN_TINKERGRAPH_GRAPH_LOCATION, GREMLIN_TINKERGRAPH_GRAPH_FORMAT)); - - if (graphLocation != null) loadGraph(); - serviceRegistry = new TinkerServiceRegistry(this); configuration.getList(String.class, GREMLIN_TINKERGRAPH_SERVICE, Collections.emptyList()).forEach(serviceClass -> serviceRegistry.registerService(instantiate(serviceClass))); @@ -282,11 +404,104 @@ public void removeEdge(final Object edgeId) { container.markDeleted((TinkerTransaction) tx()); } + /** + * {@inheritDoc} + *

+ * Like the in-memory graph, this clears the whole graph at once, outside any transaction: it needs no commit and + * cannot be rolled back. The calling thread's uncommitted changes are discarded with the rest of the graph, and a + * transaction still open on another thread fails to commit any element it changed. With storage configured the + * store is emptied in the same call, so a reopen finds the graph as {@code clear()} left it. A graph opened with + * {@code gremlin.tinkergraph.storage.recover} is cleared in memory only and its store is left unchanged, like any + * other change made to it. + * + * @throws IllegalStateException if the graph's storage is closed, or stopped accepting writes after a storage + * failure, in which case the graph is not cleared + * @throws java.io.UncheckedIOException if the graph was cleared in memory but its store could not be emptied, in + * which case the storage accepts no further commits until the graph is + * reopened, and the cleared data may return then + */ @Override public void clear() { + if (null == storage) { + clearInMemory(); + return; + } + // the same lock as the commit path, so no commit lands between emptying memory and emptying the store + storageCommitLock.lock(); + try { + if (null == directoryLock) + throw new IllegalStateException(String.format("Storage at %s is closed and cannot be cleared", + storageDirectory)); + if (storage.isFailed()) + throw new IllegalStateException(String.format("Storage at %s stopped accepting writes after a " + + "storage failure, so the graph is not cleared; its data remains readable for export with " + + "g.io() until the graph is closed", storageDirectory)); + clearInMemory(); + if (storage.isReadOnly()) + return; + try { + storage.clear(this); + } catch (RuntimeException ex) { + // memory is already empty, so later commits must not be appended to whatever the store still holds + storage.failStop(ex); + throw ex; + } + recordIndexes(); + } finally { + storageCommitLock.unlock(); + } + } + + private void clearInMemory() { + // the caller's own uncommitted changes are part of the graph being cleared + if (tx().isOpen()) + tx().rollback(); + // a transaction on another thread still holds the containers it changed, so mark them removed: its commit + // then fails with a conflict rather than writing cleared elements back into the store. Each container is + // removed only after it is marked, because adds do not wait for clear(): a container added meanwhile is + // either marked and removed here, or missed and kept, and in both cases memory and the store agree. + clearContainers(vertices); + clearContainers(edges); super.clear(); - this.vertices.clear(); - this.edges.clear(); + } + + private static void clearContainers(final Map> elements) { + elements.forEach((id, container) -> { + container.markCleared(); + elements.remove(id, container); + }); + } + + /** + * Fold the durable storage log into a compact snapshot of the current committed state, reclaiming space. Has no + * effect when no storage engine is configured, or when nothing was committed since the last compaction. + * + * @throws IllegalStateException if the graph is closed, was opened read-only to recover a damaged store, or its + * storage stopped accepting writes after a storage failure + */ + public void compact() { + if (storage != null) { + // hold the same lock as the commit write path: compaction closes the log, rewrites the snapshot, and + // empties the log, which must not interleave with a concurrent transaction appending to that log. + storageCommitLock.lock(); + try { + // an explicit request that cannot run says so, rather than returning as if the store were compacted + if (null == directoryLock) + throw new IllegalStateException(String.format("Storage at %s is closed and cannot be compacted", + storageDirectory)); + if (storage.isReadOnly()) + throw new IllegalStateException(String.format( + "Storage at %s was opened with %s and is read-only, so it cannot be compacted", + storageDirectory, GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + if (storage.isFailed()) + throw new IllegalStateException(String.format("Storage at %s stopped accepting writes after a " + + "storage failure, so it cannot be compacted; close and reopen the graph", storageDirectory)); + storage.flush(); + storage.compact(this); + } finally { + storageCommitLock.unlock(); + } + } } @Override @@ -311,6 +526,20 @@ public boolean hasVertex(Object id) { Map> getVertices () { return vertices; } + /** + * {@inheritDoc} + *

+ * Reads the committed value of each container ({@link TinkerElementContainer#getUnmodified()}), so a compaction + * snapshot reflects only committed state regardless of the calling thread's open transaction. A container whose + * committed value is {@code null} (added but not yet committed, or committed and then deleted) is skipped. + */ + @Override + public Iterator committedVertices() { + return IteratorUtils.map( + IteratorUtils.filter(vertices.values().iterator(), c -> c.getUnmodified() != null), + c -> (Vertex) c.getUnmodified()); + } + @Override public int getEdgesCount() { return (int) edges.entrySet().stream().filter(v -> v.getValue().get() != null).count(); @@ -339,6 +568,19 @@ public boolean hasEdge(Object id) { Map> getEdges () { return edges; } + /** + * {@inheritDoc} + *

+ * The edge counterpart of {@link #committedVertices()}: reads only committed container state and skips containers + * whose committed value is {@code null}. + */ + @Override + public Iterator committedEdges() { + return IteratorUtils.map( + IteratorUtils.filter(edges.values().iterator(), c -> c.getUnmodified() != null), + c -> (Edge) c.getUnmodified()); + } + @Override public TinkerServiceRegistry getServiceRegistry() { return serviceRegistry; @@ -504,6 +746,12 @@ public class TinkerGraphGraphFeatures implements Features.GraphFeatures { private TinkerGraphGraphFeatures() { } + /** + * A persistent {@link TinkerStorageGraph} is a single-writer store: {@code DirectoryLock} permits only one + * graph instance to open a given storage directory at a time. This feature denotes multiple connections / + * instances sharing the same data — not the intra-instance, multi-thread transaction access that the + * thread-local {@link TinkerTransaction} already provides — so it is {@code false}. + */ @Override public boolean supportsConcurrentAccess() { return false; @@ -519,6 +767,11 @@ public boolean supportsTransactions() { return true; } + @Override + public boolean supportsPersistence() { + return storage != null; + } + @Override public boolean supportsServiceCall() { return true; @@ -540,6 +793,7 @@ public boolean supportsServiceCall() { */ @Override public void createIndex(final String key, final Class elementClass) { + checkIndexesCanChange("created"); if (Vertex.class.isAssignableFrom(elementClass)) { if (null == this.vertexIndex) this.vertexIndex = new TinkerTransactionalIndex<>(this, TinkerVertex.class); this.vertexIndex.createKeyIndex(key); @@ -549,6 +803,7 @@ public void createIndex(final String key, final Class ele } else { throw new IllegalArgumentException("Class is not indexable: " + elementClass); } + recordIndexes(); } /** @@ -560,6 +815,7 @@ public void createIndex(final String key, final Class ele */ @Override public void dropIndex(final String key, final Class elementClass) { + checkIndexesCanChange("dropped"); if (Vertex.class.isAssignableFrom(elementClass)) { if (null != this.vertexIndex) this.vertexIndex.dropKeyIndex(key); } else if (Edge.class.isAssignableFrom(elementClass)) { @@ -567,5 +823,57 @@ public void dropIndex(final String key, final Class eleme } else { throw new IllegalArgumentException("Class is not indexable: " + elementClass); } + recordIndexes(); + } + + /** + * Recreate the indexes recorded for this store. Runs after replay so that {@code createKeyIndex} backfills over + * the elements it has just rebuilt. The definitions are already on disk, so recording is suppressed while they + * are applied. + */ + private void restoreIndexes(final File directory) { + final IndexDefinitions definitions = IndexDefinitions.read(directory); + if (definitions.isEmpty()) + return; + restoringIndexes = true; + try { + definitions.vertexKeys().forEach(key -> createIndex(key, Vertex.class)); + definitions.edgeKeys().forEach(key -> createIndex(key, Edge.class)); + } finally { + restoringIndexes = false; + } + } + + /** + * Refuse an index change on a closed durable graph, which could no longer record it. An open graph whose storage + * is read-only or has failed still changes its indexes in memory, where they speed up the reads an export needs, + * without recording them. + */ + private void checkIndexesCanChange(final String change) { + if (storage != null && null == directoryLock) + throw new IllegalStateException(String.format("Storage at %s is closed, so an index cannot be %s", + storageDirectory, change)); + } + + /** + * Record the current set of indexed keys beside the engine's files, so a reopen restores them. Index definitions + * are not part of the transactional log; see {@link IndexDefinitions} for why that is sound. A graph with no + * storage engine keeps everything in memory and writes nothing. + */ + private void recordIndexes() { + // a failed engine writes nothing more to its directory, which may by now belong to another graph + if (null == storage || restoringIndexes || storage.isReadOnly() || storage.isFailed()) + return; + // serialized with other index changes and with close(), so the file always holds the latest set of keys and is + // never written once close() has released the directory, which another graph may then own + storageCommitLock.lock(); + try { + if (null == directoryLock || storage.isFailed()) + return; + new IndexDefinitions(getIndexedKeys(Vertex.class), getIndexedKeys(Edge.class)) + .write(new File(storageDirectory)); + } finally { + storageCommitLock.unlock(); + } } } diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java index 0bfcf09e469..53c6a93cab0 100644 --- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerTransaction.java @@ -21,9 +21,14 @@ import org.apache.tinkerpop.gremlin.structure.Transaction; import org.apache.tinkerpop.gremlin.structure.util.AbstractThreadLocalTransaction; import org.apache.tinkerpop.gremlin.structure.util.TransactionException; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorageMutation; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; +import java.util.List; import java.util.Set; import java.util.concurrent.atomic.AtomicLong; @@ -32,6 +37,8 @@ */ final class TinkerTransaction extends AbstractThreadLocalTransaction { + private static final Logger logger = LoggerFactory.getLogger(TinkerTransaction.class); + private static final String TX_CONFLICT = "Conflict: element modified in another transaction"; /** @@ -128,10 +135,13 @@ protected void markRead(TinkerElementContainer contain * 2. verify if any elements already changed, throw {@link TransactionException} if any * 3. try to lock all containers to prevent other tx from making changes * 4. one more time verify elements versions - * 5. update indices - * 6. commit all changes - * On {@link TransactionException}: + * 5. persist the changes when a storage engine is configured + * 6. update indices + * 7. commit all changes + * 8. compact the storage when its log has grown past the threshold + * On {@link TransactionException} or any other failure before the changes are applied (such as a storage write): * rollback all changes + * An {@code Error} from the compaction in step 8 is thrown as it is, and the transaction is committed. * Lastly: * cleanup transaction intermediate variables. * @@ -147,6 +157,12 @@ protected void doCommit() throws TransactionException { Set> changedEdges = txChangedEdges.get(); if (null == changedEdges) changedEdges = Collections.emptySet(); + // set once the changes are durable, after which a failure cannot undo the transaction, only keep it from + // being seen until the graph is reopened + boolean recorded = false; + // set once the changes are applied in memory as well, after which the transaction is committed in full and + // nothing that fails later may roll it back or report it unapplied + boolean applied = false; try { // Double-checked locking to reduce lock time if (changedVertices.stream().anyMatch(v -> v.updatedOutsideTransaction()) || @@ -168,16 +184,83 @@ protected void doCommit() throws TransactionException { changedEdges.stream().anyMatch(e -> e.updatedOutsideTransaction())) throw new TransactionException(TX_CONFLICT); - // update indices - final TinkerTransactionalIndex vertexIndex = (TinkerTransactionalIndex) graph.vertexIndex; - if (vertexIndex != null) vertexIndex.commit(changedVertices); - final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; - if (edgeIndex != null) edgeIndex.commit(changedEdges); + // write-ahead: durably persist the changeset before applying the in-memory commit. A failure here aborts + // the commit and the catch below rolls back memory. If the failure was in writing or + // flushing the log, the engine also stops accepting commits until the graph is reopened, because what + // reached disk is then unknown: the failed transaction may or may not be in the log, and reopening + // replays only what is actually there. Skipped while the graph is replaying its storage log on open. + // Serialized by storageCommitLock because commits of disjoint elements otherwise reach the engine's single + // append log concurrently and interleave its records. + // + // The in-memory apply is held inside that same lock. Compaction builds its snapshot by reading the graph + // and then discards the log, which is only sound while the graph reflects everything the log holds. That + // is false for exactly as long as a changeset sits persisted but not yet applied, so a compaction landing + // in that window snapshots without the transaction and then deletes the record that held it, losing an + // acknowledged commit. Publishing under the lock closes the window, and also makes the order in which + // transactions become visible match the order they were recorded in. + // + // A transaction that changed nothing has nothing to record, so it neither takes the lock nor touches the + // engine. Every read opens a transaction that is later committed, so writing for it would make each read + // append and sync a frame, and would fail reads on a graph whose storage refuses commits: one opened + // to recover a damaged store, or one stopped after a storage failure, whose data must stay exportable. + final boolean durable = graph.storage != null && !graph.loading && + !(changedVertices.isEmpty() && changedEdges.isEmpty()); + if (durable) graph.storageCommitLock.lock(); + try { + if (durable) { + // clear() empties the graph under this lock and marks every element container removed, so a commit + // that passed the checks above while waiting here must look again before it writes anything + if (changedVertices.stream().anyMatch(v -> v.updatedOutsideTransaction()) || + changedEdges.stream().anyMatch(e -> e.updatedOutsideTransaction())) + throw new TransactionException(TX_CONFLICT); + graph.storage.persist(txVersion, toMutations(changedVertices), toMutations(changedEdges)); + graph.storage.flush(); + recorded = true; + } + + // update indices only once the storage write has succeeded. Committing an index changes the shared + // index that every transaction reads, and rollback can only discard this transaction's own index + // changes, so an index committed ahead of a storage write that then failed would keep the failed + // transaction's view until the graph is reopened. + final TinkerTransactionalIndex vertexIndex = (TinkerTransactionalIndex) graph.vertexIndex; + if (vertexIndex != null) vertexIndex.commit(changedVertices); + final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; + if (edgeIndex != null) edgeIndex.commit(changedEdges); + + // commit all changes + changedVertices.forEach(v -> v.commit(txVersion)); + changedEdges.forEach(e -> e.commit(txVersion)); + applied = true; + + // bound log growth for a long-running graph that is never explicitly closed; no-op unless the + // engine's accumulated log has crossed its threshold. Runs after the apply so the snapshot it may + // write includes this transaction rather than omitting it and then truncating the log that held it. + // The transaction is already applied and durable by now, so a failed compaction must not fail it. + // Compaction is crash-safe (the old snapshot and log stay intact) and is tried again after later + // commits. An Error, such as running out of memory while writing the snapshot, is not swallowed: it + // reaches the caller as itself, with the transaction committed. + if (durable) { + try { + graph.storage.maybeCompact(graph); + } catch (RuntimeException ex) { + if (graph.storage.isFailed()) + logger.error("Storage compaction failed after a successful commit and the storage now " + + "accepts no further commits until the graph is reopened", ex); + else + logger.warn("Storage compaction failed after a successful commit; it will be retried", ex); + } + } + } finally { + if (durable) graph.storageCommitLock.unlock(); + } + } catch (RuntimeException | Error ex) { + // only an Error from compaction gets here once the transaction is applied, and there is nothing to undo + if (applied) { + logger.error("Storage compaction failed with an error after a successful commit; the transaction is " + + "committed and compaction will be retried", ex); + throw ex; + } - // commit all changes - changedVertices.forEach(v -> v.commit(txVersion)); - changedEdges.forEach(e -> e.commit(txVersion)); - } catch (TransactionException ex) { // rollback on error changedVertices.forEach(v -> v.rollback()); changedEdges.forEach(e -> e.rollback()); @@ -188,11 +271,24 @@ protected void doCommit() throws TransactionException { final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; if (edgeIndex != null) edgeIndex.rollback(); + // the transaction is committed once it is durable, as a database counts it once its commit record is in + // the log, so a failure applying it afterwards is not reported as a rollback. Whatever was applied before + // the failure, an index or some of the elements, cannot be undone, so memory may hold part of the + // transaction and no longer matches the log. The storage must take nothing more from it, and reopening + // shows the whole commit. + if (recorded) { + graph.storage.failStop(ex); + throw new TransactionException(String.format("The transaction was recorded durably in storage at %s " + + "but could not be applied in memory, so it will be present when the graph is reopened. Until " + + "then the graph may show only part of it, and no further commits are accepted", + graph.storageDirectory), ex); + } throw ex; } finally { - // remove elements from graph if not used in other tx's - changedVertices.stream().filter(v -> v.canBeRemoved()).forEach(v -> graph.getVertices().remove(v.getElementId())); - changedEdges.stream().filter(e -> e.canBeRemoved()).forEach(e -> graph.getEdges().remove(e.getElementId())); + // remove elements from graph if not used in other tx's. Only the container itself is removed: after clear() + // the id may already belong to a new container, added and committed since this transaction began + changedVertices.stream().filter(v -> v.canBeRemoved()).forEach(v -> graph.getVertices().remove(v.getElementId(), v)); + changedEdges.stream().filter(e -> e.canBeRemoved()).forEach(e -> graph.getEdges().remove(e.getElementId(), e)); final Set readElements = txReadElements.get(); if (readElements != null) @@ -209,6 +305,21 @@ protected void doCommit() throws TransactionException { } } + /** + * Convert the changed element containers into the storage-facing {@link TinkerStorageMutation} view. Called during + * commit, before {@code commit()} is applied to the containers, so the modified value is still available via + * {@link TinkerElementContainer#getModified()}. + */ + private static List> toMutations(final Set> changed) { + final List> mutations = new ArrayList<>(changed.size()); + for (final TinkerElementContainer c : changed) { + mutations.add(c.isDeleted() + ? new TinkerStorageMutation<>(c.getElementId(), null) + : new TinkerStorageMutation<>(c.getElementId(), c.getModified())); + } + return mutations; + } + /** * Rollback all changes made in current transaction. * Workflow: @@ -230,13 +341,13 @@ protected void doRollback() throws TransactionException { final TinkerTransactionalIndex vertexIndex = (TinkerTransactionalIndex) graph.vertexIndex; if (vertexIndex != null) vertexIndex.rollback(); final TinkerTransactionalIndex edgeIndex = (TinkerTransactionalIndex) graph.edgeIndex; - if (vertexIndex != null) edgeIndex.rollback(); + if (edgeIndex != null) edgeIndex.rollback(); - // cleanup unused containers + // cleanup unused containers, removing only the container itself, as in doCommit if (null != changedVertices) - changedVertices.stream().filter(v -> v.canBeRemoved()).forEach(v -> graph.getVertices().remove(v.getElementId())); + changedVertices.stream().filter(v -> v.canBeRemoved()).forEach(v -> graph.getVertices().remove(v.getElementId(), v)); if (null != changedEdges) - changedEdges.stream().filter(e -> e.canBeRemoved()).forEach(e -> graph.getEdges().remove(e.getElementId())); + changedEdges.stream().filter(e -> e.canBeRemoved()).forEach(e -> graph.getEdges().remove(e.getElementId(), e)); final Set readElements = txReadElements.get(); if (readElements != null) diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java new file mode 100644 index 00000000000..9276de8c41c --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java @@ -0,0 +1,1562 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.util.Attachable; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.EOFException; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.RandomAccessFile; +import java.io.UncheckedIOException; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.BasicFileAttributes; +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.zip.CRC32; + +/** + * Log-structured durable storage machinery shared by {@link TinkerStorage} engines, independent of how an element is + * encoded. It persists a {@code TinkerStorageGraph} as an append-only commit log ({@code log.gbin}) plus an optional + * folded {@code snapshot.gbin}; on open the snapshot is read followed by the log, last-write-wins per element id. + *

+ * This base owns everything that is not the element codec: the on-disk file layout, the single-source-of-truth + * {@code VERSION} marker, the store id and compaction generation that tie a log to its snapshot, length+CRC frame + * framing (which tells an interrupted trailing append apart from genuine corruption, with log frames also checksumming + * their length), the replay fold loop, durability via {@link SyncMode}, and crash-safe atomic compaction with a size + * threshold. Concrete engines supply only the codec through {@link #encodeCommit}, {@link #decodeFrame}, + * {@link #writeSnapshot}, and (optionally) {@link #beginReplay} for per-replay decode state. + *

+ * The in-memory graph remains authoritative (write-through); this machinery does not support graphs larger than memory. + */ +public abstract class AbstractLogStorage implements TinkerStorage { + + private static final Logger logger = LoggerFactory.getLogger(AbstractLogStorage.class); + + /** + * Magic bytes ("TGSB" — TinkerGraph Storage Binary) at the start of every storage file, so a file can be + * identified as one written by this engine (and a foreign or corrupt file rejected) before any record is read. + */ + static final byte[] MAGIC = { 'T', 'G', 'S', 'B' }; + + /** + * On-disk format version of the store. Recorded once per store in the {@link #VERSION_FILE} marker rather than in + * every file, so a store has a single unambiguous version even when it momentarily holds a snapshot and a log + * written at different times. A future format bump is detected against this marker so an older store is rejected + * (never silently misread); the supported migration path is to export via {@code g.io()} before upgrading. + */ + static final byte FORMAT_VERSION = 1; + + /** + * Store-level version marker file, holding {@link #MAGIC}, the one-byte {@link #FORMAT_VERSION}, the store's + * {@link #STORE_ID_SIZE}-byte id and a 4-byte CRC32 of those bytes, {@link #VERSION_SIZE} bytes in all. The + * checksum lets a damaged marker be reported as damage rather than as a marker from another store. + */ + static final String VERSION_FILE = "VERSION"; + + /** + * Bytes of the random id a store is given when it is created: a UUID, as two 8-byte longs. The id is recorded in + * the {@link #VERSION_FILE} and in every snapshot and log header, so a file copied in from another store is + * recognized as foreign. The generation alone can't do that, since two stores can be at the same one. + */ + static final int STORE_ID_SIZE = 2 * Long.BYTES; + + /** + * Bytes of the {@link #VERSION_FILE} marker. + */ + static final int VERSION_SIZE = MAGIC.length + 1 + STORE_ID_SIZE + Integer.BYTES; + + /** + * Bytes of the per-file header: {@link #MAGIC}, the file's 8-byte compaction generation, the id of the store it + * belongs to (see {@link #STORE_ID_SIZE}), then a 4-byte CRC32 of those bytes. The format version lives in the + * store-level {@link #VERSION_FILE}, not in each file. The checksum keeps a damaged header from reading as a + * believable one: a flipped bit in the generation would otherwise make an intact log look as if it belonged to + * another snapshot, and a recovery open would then set it aside. + *

+ * The generation is what lets an open tell a damaged store from a legitimate one. Each compaction writes a snapshot + * of the next generation and then replaces the log with an empty one of that same generation, so a store with data + * always has a log, the log's generation always names the snapshot it builds on (or {@code 0} when there is none), + * and only the crash window between those two renames leaves the snapshot one generation ahead of the log. Any other + * combination, such as a missing log or a log whose snapshot is gone, means files were lost. + */ + static final int HEADER_SIZE = MAGIC.length + Long.BYTES + STORE_ID_SIZE + Integer.BYTES; + + /** + * Bytes ahead of each snapshot frame's payload: a 4-byte length and a 4-byte CRC32 of the payload. + */ + static final int FRAME_HEADER_SIZE = 2 * Integer.BYTES; + + /** + * Bytes ahead of each log frame's payload: a 4-byte length, a 4-byte CRC32 of that length, a 4-byte sequence + * number, and a 4-byte CRC32 of the sequence number and payload together. The log is the only file whose tail is + * ever cut back, and a length that claims more bytes than remain is what marks a torn tail, so the length must be + * verifiable on its own. Without that, one corrupted length in the middle of the log would read as a torn tail and + * every commit after it would be cut off. The sequence number is the frame's position in its log, counting from 0, + * so a record that was repeated, reordered or copied in from another log is caught rather than replayed. The + * snapshot is only ever replaced whole, so its frames do without either. + */ + static final int LOG_FRAME_HEADER_SIZE = 4 * Integer.BYTES; + + /** + * The smallest unit a device writes. Data an interrupted append never wrote reads back as zeros in whole sectors, + * which is what separates a final frame torn by a crash from one that was damaged. + */ + static final int SECTOR_SIZE = 512; + + static final String SNAPSHOT_FILE = "snapshot.gbin"; + static final String LOG_FILE = "log.gbin"; + + /** + * Default automatic-compaction threshold: 64 MB of appended log since the last compaction. + */ + static final long DEFAULT_COMPACT_THRESHOLD_BYTES = 64L * 1024 * 1024; + + private File directory; + private File snapshotFile; + private File logFile; + private File versionFile; + + private DataOutputStream logOut; + private FileOutputStream logFos; + + /** + * The identity of the log file the open {@link #logOut} writes to (the inode on POSIX), or {@code null} when the + * filesystem does not report one. On POSIX a removed or replaced file stays writable through a stream that already + * has it open, so the log is checked against its path after each sync (see {@link #checkStoreInPlace}). + */ + private Object logFileKey; + + /** + * The identity of the directory's {@code LOCK} file when the store was opened, or {@code null} when there is none + * (the engine opened on its own, without a graph holding the directory) or the filesystem reports no identity. The + * lock held on that file is what keeps a second graph out, and a lock on a file that is no longer at its path keeps + * out nothing, so the path is checked along with the log's after each sync. + */ + private Object lockFileKey; + + /** + * The sequence number the next frame appended to the log carries: the number of frames the log already holds, as + * counted by replay or reset when a new log is installed. + */ + private int nextLogSequence = 0; + private SyncMode syncMode = SyncMode.COMMIT; + private long compactThresholdBytes = DEFAULT_COMPACT_THRESHOLD_BYTES; + private long logBytesSinceCompaction = 0; + + /** + * After an automatic compaction fails, the log size it must reach before the next attempt. Each attempt rewrites + * the whole graph while every commit waits on the commit lock, so a compaction that keeps failing (a full disk, say) + * is retried after another threshold's worth of log rather than on every commit. Reset by a compaction that + * succeeds. + */ + private long compactionDeferredUntil = 0; + + /** + * The compaction generation of the current log, and of the snapshot it builds on when there is one. + */ + private long generation = 0; + + /** + * The id of this store, read from or written to the {@link #VERSION_FILE} on open, or {@code null} when a recovery + * open finds no marker to read it from and so can't check which store the files belong to. + */ + private UUID storeId; + + /** + * Whether the store's version marker was already present when this open began, which means the store was created + * by an earlier open and so was given a log then. + */ + private boolean storeExisted; + + /** + * Whether replay should fold the log. Only a recovery open that found the log missing, or not belonging to the + * snapshot, leaves it out. + */ + private boolean replayLog = true; + + /** + * What a recovery open had to leave behind, or {@code null} for a normal open. Created on open so the checks made + * there can add to what replay later reports. + */ + private Recovery recovery; + + private boolean closed = false; + + /** + * The first failure to write or flush the log, or {@code null} while the log is sound. Once set, the engine + * fail-stops: no further commit is accepted, nothing still buffered is written, and only a reopen (which replays + * what actually reached disk) recovers. After a failed write or fsync, neither the buffered bytes nor the + * operating system's view of the file can be trusted, so carrying on could make a failed transaction durable. + * Accessed only under the graph's storage commit lock. + */ + private IOException failure; + + /** + * Whether the store was opened with {@code gremlin.tinkergraph.storage.recover}. Replay then recovers what it can + * from a damaged snapshot or log instead of failing, and the engine stays read-only so that nothing on disk + * changes: commits are refused, nothing is compacted or truncated, and no version marker is written. + */ + private boolean recovering = false; + + // ----------------------------------------------------------------------------------------- codec hooks + + /** + * Encode a committing transaction's changeset into a single record payload (the framing is added by the caller). + * A failed encode fails only its own transaction and later commits are still accepted, so on failure an + * implementation must undo any codec state it changed while encoding, such as dictionary entries. + */ + protected abstract byte[] encodeCommit(long txVersion, + Collection> changedVertices, + Collection> changedEdges) throws IOException; + + /** + * Decode one record payload, folding its puts and deletes into the supplied maps (last-write-wins per id). + */ + protected abstract void decodeFrame(byte[] record, + Map vertices, + Map edges) throws IOException; + + /** + * Write the entire current committed state of the graph to {@code out} as framed records (via {@link #writeFrame}), + * for compaction. The file header has already been written to {@code out}. + */ + protected abstract void writeSnapshot(AbstractTinkerGraph graph, DataOutputStream out) throws IOException; + + /** + * Reset any per-replay decode state (e.g. a dictionary) before a fold begins. Default is a no-op. + */ + protected void beginReplay() { + // no-op by default + } + + // ----------------------------------------------------------------------------------------- lifecycle + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { + final String location = config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null); + if (null == location) + throw new IllegalStateException(String.format("%s must be set to use a durable storage engine", + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + this.directory = new File(location); + this.snapshotFile = new File(directory, SNAPSHOT_FILE); + this.logFile = new File(directory, LOG_FILE); + this.versionFile = new File(directory, VERSION_FILE); + this.recovering = config.getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, false); + this.syncMode = SyncMode.fromConfigValue(config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, null)); + this.compactThresholdBytes = config.getLong( + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, DEFAULT_COMPACT_THRESHOLD_BYTES); + if (compactThresholdBytes < 0) + throw new IllegalArgumentException(String.format( + "%s must be 0, to disable automatic compaction, or a positive number of bytes, but was %d", + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, compactThresholdBytes)); + // seed the counter with any pre-existing log so a graph reopened with a large log still compacts promptly + this.logBytesSinceCompaction = logFile.exists() ? Math.max(0, logFile.length() - HEADER_SIZE) : 0; + this.recovery = recovering ? new Recovery() : null; + configureCodec(config); + ensureDirectory(); + // temporary files are only ever renamed into place once complete, so any left behind belong to a write that + // was interrupted and hold nothing the store needs + if (!recovering) + removeStaleTemporaryFiles(); + establishStoreVersion(); + establishGeneration(); + // remember which file the log is from the start, so a store removed or replaced before the first commit is + // caught too, not only one replaced while the log is open for append + if (!recovering) { + recordLogIdentity(); + recordLockIdentity(); + } + } + + /** + * Read any codec-specific configuration. Called once during {@link #open}. Default is a no-op. + */ + protected void configureCodec(final Configuration config) { + // no-op by default + } + + @Override + public void replay(final AbstractTinkerGraph graph) { + beginReplay(); + // Fold snapshot then log into final state: last write per id wins, deletes remove. + final Map vertices = new LinkedHashMap<>(); + final Map edges = new LinkedHashMap<>(); + + // the snapshot is only ever replaced by an atomic rename, so only the log can end in an interrupted append + if (snapshotFile.exists()) + foldRecords(snapshotFile, vertices, edges, false, recovery); + if (replayLog && logFile.exists()) { + final long completeEnd = foldRecords(logFile, vertices, edges, true, recovery); + // a recovery open never changes the files, so a torn tail is left in place + if (recovery == null) + truncateTornLogTail(completeEnd); + else if (logFile.length() > completeEnd && recovery.logStoppedAt < 0) + recovery.logStopped(completeEnd, logFile.length(), "the log ends in an interrupted append"); + } + + if (recovery != null) { + // getOrCreate would invent a bare vertex for a missing endpoint, so drop edges that lost one instead + final Iterator it = edges.values().iterator(); + while (it.hasNext()) { + final DetachedEdge e = it.next(); + if (!vertices.containsKey(e.outVertex().id()) || !vertices.containsKey(e.inVertex().id())) { + it.remove(); + recovery.danglingEdges++; + } + } + recovery.report(directory); + } + + if (vertices.isEmpty() && edges.isEmpty()) + return; + + // Attach vertices first so edges can find their endpoints, then commit once. + for (final DetachedVertex v : vertices.values()) + v.attach(Attachable.Method.getOrCreate(graph)); + for (final DetachedEdge e : edges.values()) + e.attach(Attachable.Method.getOrCreate(graph)); + + graph.tx().commit(); + } + + @Override + public void persist(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) { + checkNotClosed(); + checkNotRecovering(); + checkNotFailed(); + ensureLogOpen(); + // a graph that has lost its lock may share the directory with another, so it must not add to the log at all + checkLockIdentity(); + final byte[] frame; + try { + frame = encodeCommit(txVersion, changedVertices, changedEdges); + } catch (IOException ex) { + // nothing has been written yet, so the log is still sound and only this transaction fails + throw new UncheckedIOException("Could not encode transaction for storage log: " + ex.getMessage(), ex); + } + try { + writeLogFrame(logOut, nextLogSequence, frame); + } catch (IOException ex) { + throw fail("Could not append transaction to storage log", ex); + } + nextLogSequence++; + logBytesSinceCompaction += LOG_FRAME_HEADER_SIZE + frame.length; + } + + @Override + public void flush() { + if (closed || recovering || failure != null) + return; + if (logOut != null) { + try { + // flush the JVM buffer into the OS page cache; durable against a JVM process crash + logOut.flush(); + // in COMMIT mode also force the OS page cache to the device, so an acknowledged commit is durable + // against an OS crash or power loss. OS mode stops at the flush above and accepts that weaker guarantee. + if (syncMode == SyncMode.COMMIT) + logFos.getFD().sync(); + } catch (IOException ex) { + throw fail("Could not flush storage log", ex); + } + // the sync succeeds even on a log that was removed from the directory, so check the commit actually + // landed in the store before it is acknowledged + checkStoreInPlace(); + } + } + + @Override + public void compact(final AbstractTinkerGraph graph) { + compact(graph, false); + } + + /** + * Every compaction, explicit, automatic or for {@link #clear}, runs through here. + * + * @param force rewrite the store even when nothing was logged since the last compaction, as {@link #clear} must, + * since the graph it writes then no longer matches the snapshot + */ + protected void compact(final AbstractTinkerGraph graph, final boolean force) { + // after a write failure the disk is suspect and the log's buffered tail must not be written, so leave the + // files as they are for the next open to replay + if (closed || recovering || failure != null) + return; + // Write a fresh snapshot of the current committed state at the next generation, then replace the log with an + // empty one of that generation. This must be crash-safe: at no point may a crash leave the store without a + // readable snapshot-or-log covering the committed state. Ordering is write-tmp -> fsync tmp -> atomically + // rename tmp over the snapshot -> fsync dir (the rename is now durable) -> install the empty log the same way. + // Each file is only ever replaced by an atomic rename, so a crash at any step leaves either the old snapshot + // and log, the new snapshot with the old log (which open recognizes by its generation), or the new pair. + // compaction writes into the directory it was opened on and never recreates it, so a store removed while the + // graph is open stays removed (the open log is the store's only file the engine holds, so check it first) + checkStoreInPlace(); + // with nothing logged since the last compaction the snapshot already holds everything the graph has + // committed, so rewriting it would only cost a full write and sync, which close() would pay every time + if (logBytesSinceCompaction == 0 && !force) + return; + closeLog(); + final long nextGeneration = generation + 1; + final File tmp = new File(directory, SNAPSHOT_FILE + ".tmp"); + try (final FileOutputStream fos = new FileOutputStream(tmp); + final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(fos))) { + writeHeader(out, nextGeneration); + writeSnapshot(graph, out); + out.flush(); + // force the snapshot's bytes to the device before it is renamed into place + fos.getFD().sync(); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw new UncheckedIOException("Could not write storage snapshot", ex); + } catch (RuntimeException | Error ex) { + StorageFiles.deleteQuietly(tmp); + throw ex; + } + + try { + // atomically replace the snapshot; no delete-then-rename window where the snapshot is briefly absent + try { + StorageFiles.atomicMove(tmp, snapshotFile); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw ex; + } + // fsync the directory so the rename survives a crash before we touch the log + StorageFiles.syncDirectory(directory); + } catch (IOException ex) { + throw new UncheckedIOException("Could not finalize storage snapshot", ex); + } + + // empty the log now that the snapshot durably reflects the committed state. Once the snapshot is in place, the + // old log may only stay as it is, since open replaces it unread as an interrupted compaction. Appending to it + // would put commits there that the next open discards, so a failure here stops further commits instead. + try { + installEmptyLog(nextGeneration); + } catch (IOException ex) { + throw fail("Could not replace storage log after writing a new snapshot", ex); + } catch (RuntimeException | Error ex) { + // whatever the failure, the old log can now only stay as it is, so it must take no more commits + if (failure == null) + failure = new IOException("Could not replace storage log after writing a new snapshot", ex); + throw ex; + } + generation = nextGeneration; + // the new log is a new file, and from now on it is the one that must still be in place + recordLogIdentity(); + + // the log is now empty; the accumulated state lives in the snapshot + logBytesSinceCompaction = 0; + compactionDeferredUntil = 0; + } + + @Override + public void maybeCompact(final AbstractTinkerGraph graph) { + if (closed || compactThresholdBytes <= 0) + return; + if (logBytesSinceCompaction < compactThresholdBytes || logBytesSinceCompaction < compactionDeferredUntil) + return; + try { + compact(graph); + } catch (RuntimeException | Error ex) { + compactionDeferredUntil = logBytesSinceCompaction + compactThresholdBytes; + throw ex; + } + } + + @Override + public void clear(final AbstractTinkerGraph graph) { + checkNotClosed(); + checkNotRecovering(); + checkNotFailed(); + try { + compact(graph, true); + } catch (RuntimeException ex) { + // compaction is crash-safe, so the store still holds either the old data or nothing; which one is unknown + // after a failure, and later commits must not be appended to data the graph no longer has. The graph is + // already empty in memory, so unlike other failures there is nothing left to export. + final IOException cause = ex.getCause() instanceof IOException ? + (IOException) ex.getCause() : new IOException(ex); + if (failure == null) + failure = cause; + throw new UncheckedIOException(String.format("Could not clear storage at %s after the graph was cleared " + + "in memory, so the cleared data may return when the graph is reopened; no further commits are " + + "accepted until then", directory), cause); + } + } + + @Override + public boolean isReadOnly() { + return recovering; + } + + @Override + public boolean isFailed() { + return failure != null; + } + + @Override + public void failStop(final Throwable cause) { + if (failure == null) + failure = new IOException("The graph's memory no longer matches the storage: " + cause.getMessage(), + cause); + } + + @Override + public void close() { + try { + if (failure instanceof StoreRemovedException) { + logger.error("Closing storage at {}, which was removed or replaced while the graph was open; nothing " + + "more is written to it", directory); + discardLog(); + } else if (failure != null) { + logger.warn("Closing storage at {} after an earlier write failure; anything not yet written is " + + "discarded and the next open recovers from what is on disk", directory); + discardLog(); + } else { + closeLog(); + } + } finally { + closed = true; + } + } + + // ----------------------------------------------------------------------------------------- version marker + + /** + * Read and validate the store-level version marker, or create it for a new store. This is the single source of + * truth for the store's format version: a marker naming an unsupported version, or bad magic, fails the open + * loudly rather than risking a misread. + */ + private void establishStoreVersion() { + final boolean storeHasData = snapshotFile.exists() || logFile.exists(); + storeExisted = versionFile.exists(); + if (!versionFile.exists()) { + if (storeHasData) { + final String problem = String.format("Storage location %s has data but no %s marker, so its format " + + "version cannot be confirmed", directory, VERSION_FILE); + if (!recovering) + throw new IllegalStateException(problem + String.format( + "; open it with %s to read it as format version %d", + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, FORMAT_VERSION)); + recovery.note(problem + String.format("; it was read as format version %d, checking its snapshot " + + "and log against each other.", FORMAT_VERSION)); + } else if (!recovering) { + storeId = UUID.randomUUID(); + writeStoreVersion(); + } + return; + } + final byte[] marker; + try { + marker = Files.readAllBytes(versionFile.toPath()); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage version marker %s: %s", versionFile, + ex.getMessage()), ex); + } + final String damage = versionMarkerDamage(marker); + if (damage != null) { + // the marker holds nothing the snapshot and log headers do not also carry, except the format version, so + // a recovery open reads past it, taking the store id the snapshot and log share + final String problem = String.format("Storage version marker %s is damaged (%s)", versionFile, damage); + if (!recovering) + throw new IllegalStateException(problem + String.format("; open the store with %s to read it as " + + "format version %d", TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, FORMAT_VERSION)); + recovery.note(problem + String.format("; it was read as format version %d, checking its snapshot and " + + "log against each other.", FORMAT_VERSION)); + return; + } + final byte version = marker[MAGIC.length]; + if (version != FORMAT_VERSION) { + final String unsupported = String.format("Unsupported storage format version %d at %s (this build " + + "writes %d); open it with the TinkerPop version that wrote it and export it with g.io()", + version, directory, FORMAT_VERSION); + throw new UncheckedIOException(String.format("Could not read storage version marker %s: %s", versionFile, + unsupported), new IOException(unsupported)); + } + final ByteBuffer id = ByteBuffer.wrap(marker, MAGIC.length + 1, STORE_ID_SIZE); + storeId = new UUID(id.getLong(), id.getLong()); + } + + /** + * What is wrong with a version marker's bytes, or {@code null} if they are whole. A marker that a later TinkerPop + * wrote in another layout reads as damaged too, and the reason then names its format byte. + */ + private static String versionMarkerDamage(final byte[] marker) { + if (marker.length < MAGIC.length || !Arrays.equals(Arrays.copyOf(marker, MAGIC.length), MAGIC)) + return "it does not start with the storage magic"; + final String newer = marker.length > MAGIC.length && marker[MAGIC.length] != FORMAT_VERSION ? + String.format(", or it was written in format version %d", marker[MAGIC.length]) : ""; + if (marker.length != VERSION_SIZE) + return String.format("it is %d bytes rather than %d%s", marker.length, VERSION_SIZE, newer); + if (ByteBuffer.wrap(marker).getInt(VERSION_SIZE - Integer.BYTES) != versionMarkerCrc(marker)) + return "its checksum does not match" + newer; + return null; + } + + /** + * The CRC32 of a version marker's bytes ahead of the checksum itself. + */ + static int versionMarkerCrc(final byte[] marker) { + final CRC32 crc = new CRC32(); + crc.update(marker, 0, VERSION_SIZE - Integer.BYTES); + return (int) crc.getValue(); + } + + private void writeStoreVersion() { + // written whole through a temporary file and a rename, so a crash leaves either no marker or a complete one, + // never a short marker that would fail every later open + final File tmp = new File(directory, VERSION_FILE + ".tmp"); + try { + final ByteBuffer marker = ByteBuffer.allocate(VERSION_SIZE); + marker.put(MAGIC).put(FORMAT_VERSION) + .putLong(storeId.getMostSignificantBits()).putLong(storeId.getLeastSignificantBits()); + marker.putInt(versionMarkerCrc(marker.array())); + try (final FileOutputStream fos = new FileOutputStream(tmp); + final DataOutputStream out = new DataOutputStream(fos)) { + out.write(marker.array()); + out.flush(); + fos.getFD().sync(); + } + StorageFiles.atomicMove(tmp, versionFile); + StorageFiles.syncDirectory(directory); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw new UncheckedIOException(String.format("Could not write storage version marker %s", versionFile), ex); + } + } + + /** + * Check that the snapshot and log belong together, using the generation in each header (see {@link #HEADER_SIZE}), + * and settle the generation the engine continues from. A new store gets its empty log here. A snapshot one + * generation ahead of the log is a compaction interrupted between its two renames: the snapshot already holds + * everything the old log does, so the empty log that compaction would have installed is installed now. Any other + * mismatch means files were lost and fails the open. A recovery open changes nothing, reports the problem, and + * reads what remains, except that a log whose snapshot is gone still fails, since its records refer to the + * snapshot's dictionary. + */ + private void establishGeneration() { + final boolean hasSnapshot = snapshotFile.exists(); + final boolean hasLog = logFile.exists(); + if (!hasSnapshot && !hasLog) { + // a store is given its log when it is created and keeps one from then on, so a store created by an + // earlier open with neither file has lost its log, and with it every transaction committed before its first + // compaction. Opening it as a new store would hide that. + if (storeExisted) { + final String problem = String.format("Storage log %s is missing and there is no snapshot, so every " + + "transaction committed to the store is lost", logFile); + if (!recovering) + throw new IllegalStateException(problem + String.format("; open the store with %s to confirm " + + "it is empty, or, if nothing was ever committed to it (its first open was interrupted), " + + "delete the directory to start a new store", TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + recovery.note(problem + "; the graph is empty."); + return; + } + if (!recovering) + installNewLog(0); + return; + } + + final FileHeader snapshotHeader = hasSnapshot ? readHeader(snapshotFile) : null; + // with no readable version marker, which only a recovery open gets past, the store is the one its snapshot + // belongs to, so a log from another store is still told apart + if (null == storeId && snapshotHeader != null) + storeId = snapshotHeader.storeId; + if (snapshotHeader != null && !snapshotHeader.belongsTo(storeId)) { + // a snapshot and log that agree with each other point at the version marker as the file that is foreign + final FileHeader otherLog = hasLog ? readHeaderQuietly(logFile) : null; + if (otherLog != null && snapshotHeader.storeId.equals(otherLog.storeId)) + throw new IllegalStateException(String.format("Storage version marker %s belongs to another store " + + "(%s), while snapshot %s and log %s both belong to store %s; if the marker was copied in, " + + "remove it and open the store with %s", versionFile, storeId, snapshotFile, logFile, + snapshotHeader.storeId, TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + throw new IllegalStateException(String.format("Storage snapshot %s belongs to another store (%s, not %s)", + snapshotFile, snapshotHeader.storeId, storeId)); + } + final long snapshotGeneration = hasSnapshot ? snapshotHeader.generation : -1; + if (!hasLog) { + final String problem = String.format("Storage log %s is missing, so any transactions committed after the " + + "last compaction are lost", logFile); + if (!recovering) + throw new IllegalStateException(problem + String.format( + "; open the store with %s to read its snapshot", TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + recovery.note(problem + "; only the snapshot was read."); + replayLog = false; + return; + } + + final FileHeader logHeader; + try { + logHeader = readHeader(logFile); + } catch (UncheckedIOException ex) { + // a recovery open lets the fold report an unreadable log header and carry on with the snapshot + if (!recovering) + throw ex; + generation = Math.max(0, snapshotGeneration); + return; + } + + if (!logHeader.belongsTo(storeId)) { + final String problem = String.format("Storage log %s belongs to another store (%s, not %s)", + logFile, logHeader.storeId, storeId); + if (!recovering) + throw new IllegalStateException(problem + String.format("; open the store with %s to read its snapshot", + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + recovery.note(problem + "; only the snapshot was read."); + generation = Math.max(0, snapshotGeneration); + replayLog = false; + return; + } + final long logGeneration = logHeader.generation; + + if (!hasSnapshot) { + if (logGeneration != 0) + throw new IllegalStateException(String.format("Storage snapshot %s is missing: log %s was written " + + "after compaction %d and its records depend on that snapshot", snapshotFile, logFile, logGeneration)); + generation = 0; + return; + } + + if (logGeneration == snapshotGeneration) { + generation = snapshotGeneration; + } else if (logGeneration == snapshotGeneration - 1) { + generation = snapshotGeneration; + // compaction wrote the new snapshot but was interrupted before replacing the log, so the snapshot already + // holds every transaction in the log, and replaying the log over it would add nothing + replayLog = false; + if (!recovering) { + logger.warn("Storage at {} was interrupted while compacting; completing it by emptying the log, " + + "whose transactions are all in the new snapshot", directory); + installNewLog(snapshotGeneration); + logBytesSinceCompaction = 0; + } else { + recovery.note(String.format("Compaction was interrupted before log %s was replaced; the snapshot " + + "holds all of its transactions, so the log was not read.", logFile)); + } + } else { + final String problem = String.format("Storage log %s (generation %d) does not belong to snapshot %s " + + "(generation %d)", logFile, logGeneration, snapshotFile, snapshotGeneration); + if (!recovering) + throw new IllegalStateException(problem + String.format( + "; open the store with %s to read its snapshot", TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + recovery.note(problem + "; only the snapshot was read."); + generation = snapshotGeneration; + replayLog = false; + } + } + + /** + * Read the generation and store id from the header of {@code file}, failing if the header is short or is not one + * of ours. + */ + private static FileHeader readHeader(final File file) { + try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(file)))) { + return readAndVerifyHeader(in, file, file.length()); + } catch (IOException ex) { + // the cause's message says what is wrong, such as a checksum mismatch, so it belongs in this one too + throw new UncheckedIOException(String.format("Could not read storage file %s: %s", file, ex.getMessage()), ex); + } + } + + private static FileHeader readHeaderQuietly(final File file) { + try { + return readHeader(file); + } catch (UncheckedIOException ex) { + return null; + } + } + + private void installNewLog(final long logGeneration) { + try { + installEmptyLog(logGeneration); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not create storage log %s", logFile), ex); + } + } + + /** + * Atomically replace the log with one holding only its header at {@code logGeneration}. The log is never deleted, + * so its absence is always damage, and never written in place, so it is never shorter than its header. + */ + private void installEmptyLog(final long logGeneration) throws IOException { + final File tmp = new File(directory, LOG_FILE + ".tmp"); + try (final FileOutputStream fos = openEmptyLog(tmp); + final DataOutputStream out = new DataOutputStream(fos)) { + writeHeader(out, logGeneration); + out.flush(); + fos.getFD().sync(); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw ex; + } + try { + StorageFiles.atomicMove(tmp, logFile); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw ex; + } + nextLogSequence = 0; + StorageFiles.syncDirectory(directory); + } + + private void removeStaleTemporaryFiles() { + for (final String name : new String[]{ SNAPSHOT_FILE, LOG_FILE, VERSION_FILE }) { + final File tmp = new File(directory, name + ".tmp"); + if (tmp.exists()) { + logger.warn("Removing {}, left by a storage write that was interrupted", tmp); + StorageFiles.deleteQuietly(tmp); + } + } + } + + private void ensureDirectory() { + if (directory.exists()) { + if (!directory.isDirectory()) + throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory)); + } else if (!directory.mkdirs()) { + throw new IllegalStateException(String.format("Could not create storage directory %s", directory)); + } + } + + // ----------------------------------------------------------------------------------------- fold / framing + + /** + * Fold every complete frame of {@code file} into the given maps, returning the byte offset at which the last + * complete frame ends. Anything past that offset is an interrupted trailing append. + *

+ * With a {@code recovery} in progress, damage that would otherwise fail the open is tolerated where that is safe. + * In the log, each frame is one transaction, so replay stops at the first frame that can't be read or decoded and + * keeps the consistent prefix before it. In the snapshot, element frames are independent, so an unreadable one is + * skipped. The snapshot's header and its first frame, the dictionary every other frame depends on, still fail. + */ + private long foldRecords(final File file, final Map vertices, + final Map edges, final boolean log, final Recovery recovery) { + final long fileLength = file.length(); + try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(file)))) { + long remaining; + try { + // the generation and store id were already checked on open + readAndVerifyHeader(in, file, fileLength); + remaining = fileLength - HEADER_SIZE; + } catch (IOException ex) { + if (recovery == null || !log) + throw ex; + recovery.logStopped(0, fileLength, ex.getMessage()); + return 0; + } + long completeEnd = fileLength - remaining; + int frameIndex = 0; + while (true) { + final byte[] record; + try { + record = readFrame(in, remaining, log, frameIndex); + } catch (CorruptFrameException ex) { + // a crash can also leave the last frame's header written but its payload reading back as zeros + // where the data never reached the device, which fails the payload's checksum. A final frame that + // ends exactly at the end of the log in such unwritten sectors is therefore cut as an interrupted + // append. + if (log && completeEnd + LOG_FRAME_HEADER_SIZE + ex.payloadLength == fileLength && + endsInUnwrittenSector(file, completeEnd + LOG_FRAME_HEADER_SIZE, fileLength)) + break; + // the frame's bytes were all present and consumed, so a skippable snapshot frame can be passed over + if (recovery == null || log || frameIndex == 0) + throw recoveryStops(recovery, log, ex, completeEnd, fileLength); + final long frameLength = (long) FRAME_HEADER_SIZE + ex.payloadLength; + remaining -= frameLength; + completeEnd += frameLength; + frameIndex++; + recovery.skippedSnapshotFrames++; + continue; + } catch (IOException ex) { + // some filesystems leave a log that crashed during an append longer than what was written, the + // extra bytes reading back as zeros. No frame the engine writes starts with a zero header, since + // the checksum of length 0 is not 0, so an all-zero rest of the log is never a frame, let alone a + // corrupted one: it is cut off like any other interrupted append. + if (log && isZeroFrom(file, completeEnd)) + break; + // a frame header that can't be trusted (such as a negative length) leaves no way to find the next + if (recovery == null || (!log && frameIndex == 0)) + throw ex; + if (log) + recovery.logStopped(completeEnd, fileLength, ex.getMessage()); + else + recovery.snapshotStopped(completeEnd, fileLength, ex.getMessage()); + break; + } + if (record == null) { + // a snapshot is only ever replaced whole by an atomic rename, so bytes it can't read are damage + // rather than an interrupted append + if (!log && remaining > 0) { + final String reason = String.format("%d trailing bytes do not form a complete record", remaining); + if (recovery == null || frameIndex == 0) + throw new IOException(String.format("Corrupt storage file %s: %s", file, reason)); + recovery.snapshotStopped(completeEnd, fileLength, reason); + } + break; + } + final long frameLength = (long) (log ? LOG_FRAME_HEADER_SIZE : FRAME_HEADER_SIZE) + record.length; + if (recovery == null) { + try { + decodeFrame(record, vertices, edges); + } catch (RuntimeException ex) { + // reported as the damage it is, naming the file, rather than as an unexplained runtime error + throw new IOException(String.format("Corrupt storage frame %d: %s", frameIndex, ex), ex); + } + } else { + try { + decodeFrameAtomically(record, vertices, edges); + } catch (IOException ex) { + if (log || frameIndex == 0) + throw recoveryStops(recovery, log, ex, completeEnd, fileLength); + recovery.skippedSnapshotFrames++; + } + } + remaining -= frameLength; + completeEnd += frameLength; + frameIndex++; + } + // the frames before completeEnd are all the log keeps, since a torn tail is cut back to it + if (log) + nextLogSequence = frameIndex; + return completeEnd; + } catch (RecoveryStop stop) { + return stop.completeEnd; + } catch (IOException ex) { + // the cause's message says what is wrong, such as a checksum mismatch, so it belongs in this one too + throw new UncheckedIOException(String.format("Could not read storage file %s: %s", file, ex.getMessage()), ex); + } + } + + /** + * Decide what a damaged frame means under recovery. In the log, replay stops there (signalled by the returned + * {@link RecoveryStop}). In the snapshot's dictionary frame, or with no recovery at all, the damage is rethrown. + */ + private static IOException recoveryStops(final Recovery recovery, final boolean log, final IOException damage, + final long completeEnd, final long fileLength) { + if (recovery == null || !log) + return damage; + recovery.logStopped(completeEnd, fileLength, damage.getMessage()); + return new RecoveryStop(completeEnd); + } + + /** + * Decode a frame so that it is applied entirely or not at all. {@link #decodeFrame} applies a frame's entries one + * by one, so a frame that fails partway would otherwise leave its first entries behind. That doesn't matter when + * any damage fails the open, but recovery keeps going and must not keep half of a transaction. + */ + private void decodeFrameAtomically(final byte[] record, final Map vertices, + final Map edges) throws IOException { + final JournaledMap v = new JournaledMap<>(vertices); + final JournaledMap e = new JournaledMap<>(edges); + try { + decodeFrame(record, v, e); + } catch (IOException | RuntimeException ex) { + e.undo(); + v.undo(); + throw ex instanceof IOException ? (IOException) ex : new IOException(ex.toString(), ex); + } + } + + /** + * Whether {@code file} ends as an append the device never finished leaves it: zero from its last + * {@link #SECTOR_SIZE}-byte boundary to its end, all within the payload that starts at {@code payloadStart}. Storage + * writes whole sectors, so a payload whose end never reached the device reads back as zeros from a sector boundary + * on. Damage to a record that was written, even one whose payload ends in a zero byte as most do, does not. + */ + private static boolean endsInUnwrittenSector(final File file, final long payloadStart, final long fileLength) + throws IOException { + final long lastSector = (fileLength - 1) / SECTOR_SIZE * SECTOR_SIZE; + return lastSector >= payloadStart && isZeroFrom(file, lastSector); + } + + /** + * Whether {@code file} holds at least one byte from {@code offset} on, and every one of them is zero. + */ + private static boolean isZeroFrom(final File file, final long offset) throws IOException { + try (final RandomAccessFile raf = new RandomAccessFile(file, "r")) { + if (offset >= raf.length()) + return false; + raf.seek(offset); + final byte[] chunk = new byte[8192]; + int read; + while ((read = raf.read(chunk)) > 0) { + for (int i = 0; i < read; i++) { + if (chunk[i] != 0) + return false; + } + } + return true; + } + } + + /** + * Cut the log back to {@code completeEnd}, the end of its last complete frame, so that the next append starts on + * a frame boundary. Without this, new commits would be written after the torn bytes and a later replay would stop + * at the old torn frame (dropping them) or read into them (failing the CRC check). Only bytes that never formed a + * complete frame, that read as zeros past the last complete frame, or a final frame that ends in unwritten sectors + * are removed. Under {@code sync=commit} an acknowledged commit is lost this way only if the device zeroed the last + * sector of a record it had already been forced to hold. + */ + private void truncateTornLogTail(final long completeEnd) { + final long length = logFile.length(); + if (length <= completeEnd) + return; + logger.warn("Storage log {} ends in an interrupted append; discarding its last {} bytes", logFile, length - completeEnd); + try (final FileChannel channel = FileChannel.open(logFile.toPath(), StandardOpenOption.WRITE)) { + channel.truncate(completeEnd); + channel.force(true); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not truncate interrupted append from storage log %s", logFile), ex); + } + } + + /** + * Read and validate the magic at the start of the per-file header, returning the number of bytes that follow the + * whole header. The generation that completes the header is left for the caller to read. + */ + private static FileHeader readAndVerifyHeader(final DataInputStream in, final File file, final long fileLength) + throws IOException { + if (fileLength < HEADER_SIZE) + throw new IOException(String.format("Corrupt storage file %s: shorter than its %d-byte header", file, HEADER_SIZE)); + final byte[] header = new byte[HEADER_SIZE]; + readFully(in, header); + if (!Arrays.equals(Arrays.copyOf(header, MAGIC.length), MAGIC)) + throw new IOException(String.format("%s is not a TinkerGraph storage file (bad magic)", file)); + final ByteBuffer buf = ByteBuffer.wrap(header); + if (buf.getInt(HEADER_SIZE - Integer.BYTES) != headerCrc(header)) + throw new IOException(String.format("Corrupt storage file %s: its header is damaged (checksum mismatch)", + file)); + buf.position(MAGIC.length); + final long fileGeneration = buf.getLong(); + return new FileHeader(fileGeneration, new UUID(buf.getLong(), buf.getLong())); + } + + /** + * The CRC32 of a header's bytes ahead of the checksum itself. + */ + private static int headerCrc(final byte[] header) { + final CRC32 crc = new CRC32(); + crc.update(header, 0, HEADER_SIZE - Integer.BYTES); + return (int) crc.getValue(); + } + + /** + * The header of a snapshot or log at {@code fileGeneration} in the store with {@code id}. + */ + static byte[] header(final long fileGeneration, final UUID id) { + final ByteBuffer buf = ByteBuffer.allocate(HEADER_SIZE); + buf.put(MAGIC).putLong(fileGeneration) + .putLong(id.getMostSignificantBits()).putLong(id.getLeastSignificantBits()); + final byte[] header = buf.array(); + buf.putInt(headerCrc(header)); + return header; + } + + private void ensureLogOpen() { + // never reopen the log of a closed engine: its directory lock is released, so the files may now belong to + // another graph + checkNotClosed(); + if (logOut == null) { + // open and compaction always leave a log in place, so appending to a missing one would write frames with + // no header + if (!logFile.isFile()) + throw removed(String.format("its log %s is missing", logFile)); + // the log may have been replaced since open or the last compaction, while it was not held open + checkLogIdentity(); + try { + // retain the FileOutputStream so flush() can reach its FileDescriptor for fsync + logFos = openLogForAppend(logFile); + logOut = new DataOutputStream(new BufferedOutputStream(logFos)); + } catch (IOException ex) { + throw new UncheckedIOException("Could not open storage log for append", ex); + } + } + } + + /** + * Open the log for appending. Exists so tests can substitute a stream that fails on demand. + */ + protected FileOutputStream openLogForAppend(final File file) throws IOException { + return new FileOutputStream(file, true); + } + + /** + * Open the temporary file an empty log is written to before it replaces the log. A seam for tests that fail + * installing a log, as for {@link #openLogForAppend}. + */ + protected FileOutputStream openEmptyLog(final File tmp) throws IOException { + return new FileOutputStream(tmp); + } + + private void closeLog() { + if (logOut == null) + return; + try { + logOut.flush(); + logOut.close(); + } catch (IOException ex) { + discardLog(); + throw fail("Could not close storage log", ex); + } + logOut = null; + logFos = null; + } + + /** + * Close the log file without writing anything still buffered for it. + */ + private void discardLog() { + if (logFos != null) { + try { + logFos.close(); + } catch (IOException ignored) { + // best effort: the log already failed and the next open recovers from what is on disk + } + } + logOut = null; + logFos = null; + } + + /** + * Fail-stop if the store was removed or replaced since the log was opened: the directory or log is gone, or the + * log's path now names a different file than the one being written. A commit synced to such a log would be + * acknowledged yet lost, and a second graph could create the store again beside this one. + */ + private void checkStoreInPlace() { + if (!directory.isDirectory()) + throw removed(String.format("its directory %s is missing", directory)); + // open and compaction always leave a log in place, so a missing one was removed whether or not it is open + if (!logFile.exists()) + throw removed(String.format("its log %s is missing", logFile)); + checkLogIdentity(); + checkLockIdentity(); + } + + private void recordLockIdentity() { + final File lockFile = new File(directory, DirectoryLock.LOCK_FILE); + try { + lockFileKey = lockFile.exists() ? fileKey(lockFile) : null; + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage lock %s", lockFile), ex); + } + } + + /** + * Fail-stop if the {@code LOCK} file the graph holds was removed or replaced, since another graph can then lock a + * new file at that path and open the store beside this one. + */ + private void checkLockIdentity() { + if (null == lockFileKey) + return; + final File lockFile = new File(directory, DirectoryLock.LOCK_FILE); + final Object current; + try { + current = fileKey(lockFile); + } catch (IOException ex) { + throw removed(String.format("its lock %s is missing", lockFile)); + } + if (!lockFileKey.equals(current)) + throw removed(String.format("its lock %s was replaced by another file", lockFile)); + } + + /** + * Remember the file now at the log's path as the store's log. Platforms whose files have no identity key record + * nothing, and the identity checks then pass. + */ + private void recordLogIdentity() { + try { + logFileKey = logFile.exists() ? fileKey(logFile) : null; + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage log %s", logFile), ex); + } + } + + /** + * Fail-stop if the log's path now names a different file than the one recorded as the store's log. + */ + private void checkLogIdentity() { + if (null == logFileKey) + return; + final Object current; + try { + current = fileKey(logFile); + } catch (IOException ex) { + throw removed(String.format("its log %s cannot be read: %s", logFile, ex.getMessage())); + } + if (!logFileKey.equals(current)) + throw removed(String.format("its log %s was replaced by another file", logFile)); + } + + private static Object fileKey(final File file) throws IOException { + return Files.readAttributes(file.toPath(), BasicFileAttributes.class).fileKey(); + } + + /** + * Fail-stop because the store was removed or replaced while the graph was open. Nothing more is written to the + * directory, close() included, and the graph's committed data remains readable in memory until it is closed. + */ + private UncheckedIOException removed(final String what) { + final StoreRemovedException ex = new StoreRemovedException(String.format( + "Storage at %s was removed or replaced while the graph was open (%s), so no further commits are " + + "accepted. The committed data is still readable in memory: export it with g.io() before " + + "closing the graph", directory, what)); + if (failure == null) + failure = ex; + return new UncheckedIOException(ex.getMessage(), ex); + } + + /** + * Record the first log write failure, putting the engine into its fail-stop state, and return the exception to + * throw for this one. + */ + private UncheckedIOException fail(final String message, final IOException cause) { + if (failure == null) + failure = cause; + return new UncheckedIOException(message + "; no further commits are accepted until the graph is reopened", cause); + } + + /** + * Refuse a commit once the engine is closed. The graph sets {@code closed} under its storage commit lock, the same + * lock {@link #persist} runs under, so a commit that was waiting on {@code close()} fails here instead of writing + * to a log that is no longer this graph's to write. + */ + private void checkNotClosed() { + if (closed) + throw new IllegalStateException(String.format( + "Storage at %s is closed and accepts no further commits; open the graph again", directory)); + } + + private void checkNotRecovering() { + if (recovering) + throw new IllegalStateException(String.format( + "Storage at %s was opened with %s and is read-only; export the graph with g.io() into a new " + + "storage directory", directory, TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } + + private void checkNotFailed() { + if (failure instanceof StoreRemovedException) + throw new UncheckedIOException(failure.getMessage(), failure); + if (failure != null) + throw new UncheckedIOException(new IOException(String.format( + "Storage at %s stopped accepting commits after an earlier failure; close and reopen the graph", + directory), failure)); + } + + /** + * Write the per-file header ({@link #MAGIC}, the file's generation and the store id) at the start of a storage + * file. + */ + private void writeHeader(final DataOutputStream out, final long fileGeneration) throws IOException { + out.write(header(fileGeneration, storeId)); + } + + /** + * Write a snapshot frame: a 4-byte big-endian payload length, a 4-byte CRC32 of the payload, then the payload. + * The checksum lets a reader tell a bit-flip inside a complete frame from an incomplete one. Available to codec + * subclasses writing per-element snapshot frames; log frames are written by {@link #writeLogFrame}. + */ + protected static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException { + final CRC32 crc = new CRC32(); + crc.update(payload); + out.writeInt(payload.length); + out.writeInt((int) crc.getValue()); + out.write(payload); + } + + /** + * Write a log frame: a 4-byte big-endian payload length, a 4-byte CRC32 of that length, the 4-byte sequence number, + * a 4-byte CRC32 of the sequence number and payload, then the payload. See {@link #LOG_FRAME_HEADER_SIZE} for what + * the length's own checksum and the sequence number are for. + */ + private static void writeLogFrame(final DataOutputStream out, final int sequence, final byte[] payload) + throws IOException { + out.writeInt(payload.length); + out.writeInt(lengthCrc(payload.length)); + out.writeInt(sequence); + out.writeInt(logPayloadCrc(sequence, payload)); + out.write(payload); + } + + private static int logPayloadCrc(final int sequence, final byte[] payload) { + final CRC32 crc = new CRC32(); + crc.update(ByteBuffer.allocate(Integer.BYTES).putInt(sequence).array()); + crc.update(payload); + return (int) crc.getValue(); + } + + private static int lengthCrc(final int length) { + final CRC32 crc = new CRC32(); + crc.update(ByteBuffer.allocate(Integer.BYTES).putInt(length).array()); + return (int) crc.getValue(); + } + + /** + * Read a framed record, or return {@code null} at end of the readable log. A frame only partially present is + * treated as an interrupted trailing append (truncation) and ends reading; a fully-present frame whose stored CRC + * does not match is genuine corruption and is raised. In the log ({@code log} set), a length that fails its own + * checksum is likewise corruption, so only a verified length that claims more bytes than remain reads as a torn + * tail, and an intact frame whose sequence number is not {@code frameIndex} is raised as one that does not belong + * where it is. + */ + private static byte[] readFrame(final DataInputStream in, final long remaining, final boolean log, + final int frameIndex) throws IOException { + final int headerSize = log ? LOG_FRAME_HEADER_SIZE : FRAME_HEADER_SIZE; + if (remaining == 0) + return null; // clean end of file, exactly on a frame boundary + if (remaining < headerSize) + return null; // not even a full header left — interrupted append + + final int length = in.readInt(); + if (log) { + final int storedLengthCrc = in.readInt(); + if (storedLengthCrc != lengthCrc(length)) + throw new IOException(String.format( + "Corrupt storage frame: length %d fails its checksum (stored %08x, computed %08x)", + length, storedLengthCrc, lengthCrc(length))); + } + final int sequence = log ? in.readInt() : 0; + final int storedCrc = in.readInt(); + if (length < 0) + throw new IOException("Corrupt storage frame: negative payload length " + length); + if ((long) length > remaining - headerSize) + return null; // frame claims more bytes than remain — truncated trailing append + + final byte[] payload = new byte[length]; + try { + readFully(in, payload); + } catch (EOFException eof) { + return null; // partial trailing payload from an interrupted append + } + + final int computedCrc; + if (log) { + computedCrc = logPayloadCrc(sequence, payload); + } else { + final CRC32 crc = new CRC32(); + crc.update(payload); + computedCrc = (int) crc.getValue(); + } + if (computedCrc != storedCrc) + throw new CorruptFrameException(String.format( + "Corrupt storage frame: CRC mismatch (stored %08x, computed %08x) in a fully-present %d-byte record", + storedCrc, computedCrc, length), length); + // an intact frame out of place was put there, since the engine only ever appends the next one + if (log && sequence != frameIndex) + throw new IOException(String.format("Storage log record %d is numbered %d, so a record was repeated, " + + "reordered or copied in from another log", frameIndex, sequence)); + return payload; + } + + private static void readFully(final InputStream in, final byte[] dst) throws IOException { + int off = 0; + while (off < dst.length) { + final int read = in.read(dst, off, dst.length - off); + if (read < 0) + throw new EOFException(); + off += read; + } + } + + /** + * A fully-present frame whose checksum does not match. Its bytes have been consumed, so a reader that can afford to + * lose the frame may continue with the next one. + */ + private static final class CorruptFrameException extends IOException { + private final int payloadLength; + + private CorruptFrameException(final String message, final int payloadLength) { + super(message); + this.payloadLength = payloadLength; + } + } + + /** + * The generation and store id read from a storage file's header. + */ + private static final class FileHeader { + private final long generation; + private final UUID storeId; + + private FileHeader(final long generation, final UUID storeId) { + this.generation = generation; + this.storeId = storeId; + } + + /** + * Whether the file belongs to the store with {@code id}. Always true when the store's id is unknown. + */ + private boolean belongsTo(final UUID id) { + return null == id || id.equals(storeId); + } + } + + /** + * The store was removed or replaced while the graph was open. + */ + private static final class StoreRemovedException extends IOException { + private StoreRemovedException(final String message) { + super(message); + } + } + + /** + * Ends a recovery fold of the log at the last frame that was applied. + */ + private static final class RecoveryStop extends IOException { + private final long completeEnd; + + private RecoveryStop(final long completeEnd) { + super("recovery stopped replaying the log at byte " + completeEnd); + this.completeEnd = completeEnd; + } + } + + /** + * What a recovery open had to leave behind, reported once replay is done. + */ + private static final class Recovery { + private int skippedSnapshotFrames = 0; + private int danglingEdges = 0; + private long logStoppedAt = -1; + private long logLength = 0; + private String logStopReason; + private long snapshotStoppedAt = -1; + private long snapshotLength = 0; + private String snapshotStopReason; + private final List notes = new ArrayList<>(); + + private void note(final String problem) { + notes.add(problem); + } + + private void snapshotStopped(final long offset, final long length, final String reason) { + this.snapshotStoppedAt = offset; + this.snapshotLength = length; + this.snapshotStopReason = reason; + } + + private void logStopped(final long offset, final long length, final String reason) { + this.logStoppedAt = offset; + this.logLength = length; + this.logStopReason = reason; + } + + private void report(final File directory) { + final StringBuilder sb = new StringBuilder(String.format( + "Opened storage at %s with %s; the graph is read-only.", directory, + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + if (notes.isEmpty() && skippedSnapshotFrames == 0 && danglingEdges == 0 && logStoppedAt < 0 && + snapshotStoppedAt < 0) + sb.append(" No damage was found."); + for (final String n : notes) + sb.append(' ').append(n); + if (skippedSnapshotFrames > 0) + sb.append(String.format(" Skipped %d unreadable element record(s) in the snapshot.", skippedSnapshotFrames)); + if (snapshotStoppedAt >= 0) + sb.append(String.format(" Stopped reading the snapshot at byte %d of %d (%s), so any elements after " + + "that point are not included.", snapshotStoppedAt, snapshotLength, snapshotStopReason)); + if (logStoppedAt >= 0) + sb.append(String.format(" Stopped replaying the log at byte %d of %d (%s), so any transactions after " + + "that point are not included.", logStoppedAt, logLength, logStopReason)); + if (danglingEdges > 0) + sb.append(String.format(" Dropped %d edge(s) whose endpoint vertex was lost.", danglingEdges)); + sb.append(" Export the graph with g.io() into a new storage directory."); + logger.warn(sb.toString()); + } + } + + /** + * A map view that records each change it makes to its backing map so that the changes can be undone. + */ + private static final class JournaledMap extends AbstractMap { + private final Map backing; + private final List journal = new ArrayList<>(); + + private JournaledMap(final Map backing) { + this.backing = backing; + } + + @Override + public V put(final K key, final V value) { + journal.add(new Object[]{ key, backing.containsKey(key), backing.get(key) }); + return backing.put(key, value); + } + + @Override + public V remove(final Object key) { + if (backing.containsKey(key)) + journal.add(new Object[]{ key, true, backing.get(key) }); + return backing.remove(key); + } + + @Override + public V get(final Object key) { + return backing.get(key); + } + + @Override + public boolean containsKey(final Object key) { + return backing.containsKey(key); + } + + @Override + public Set> entrySet() { + return Collections.unmodifiableMap(backing).entrySet(); + } + + @SuppressWarnings("unchecked") + private void undo() { + for (int i = journal.size() - 1; i >= 0; i--) { + final Object[] change = journal.get(i); + if ((Boolean) change[1]) + backing.put((K) change[0], (V) change[2]); + else + backing.remove(change[0]); + } + journal.clear(); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java new file mode 100644 index 00000000000..49c4a2becf4 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DefaultStorage.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.util.function.Supplier; + +/** + * The built-in {@link TinkerStorage} engines that can be selected by name via the + * {@code gremlin.tinkergraph.storage} configuration key. A fully-qualified class name may be used instead of one of + * these names to plug in a custom engine. + */ +public enum DefaultStorage implements Supplier { + + /** + * A durable, append-only commit log ("write-ahead log") that serializes each committed transaction with + * GraphBinary. See {@link GraphBinaryStorage}. + */ + GRAPHBINARY { + @Override + public TinkerStorage get() { + return new GraphBinaryStorage(); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java new file mode 100644 index 00000000000..e9430276d37 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLock.java @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.OverlappingFileLockException; +import java.nio.file.AccessDeniedException; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.StandardOpenOption; +import java.util.Arrays; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; + +/** + * An exclusive, whole-process lock on a {@code TinkerStorageGraph} storage directory. A persistent, transactional + * TinkerGraph is a single-writer embedded store: two graphs opened on the same directory — whether in one JVM or + * across processes — would both append to and compact the same files, corrupting them. This holds an OS advisory lock + * ({@link FileLock}) on a {@code LOCK} file in the directory for the lifetime of the graph, so a second open fails + * fast rather than silently corrupting data. + *

+ * An OS lock (rather than a mere marker file) is used so the kernel releases it automatically if the JVM dies, + * avoiding a stale lock that would wedge the store after a crash. + *

+ * Note: {@link FileLock} semantics are unreliable on some network filesystems (notably NFS); this guarantee holds on + * local filesystems. + */ +public final class DirectoryLock implements AutoCloseable { + + private static final Logger logger = LoggerFactory.getLogger(DirectoryLock.class); + + static final String LOCK_FILE = "LOCK"; + + /** + * The byte of the {@code LOCK} file that is locked, past the token {@link #stillAtPath} writes at its start. On + * Windows a lock keeps every other handle from reading the bytes it covers, even within one process, so locking the + * whole file would leave the token unreadable. + */ + private static final long LOCKED_BYTE = 1024; + + /** + * The directories a graph in this JVM holds or is acquiring the lock of. A second acquisition of one is refused here, + * before any handle on its {@code LOCK} file is opened, because closing that handle afterwards, as a refused + * acquisition must, would release every POSIX lock the process holds on the file, the holder's included, while the + * JVM goes on reporting the holder's lock held. + */ + private static final Set HELD = ConcurrentHashMap.newKeySet(); + + private final FileChannel channel; + private final FileLock lock; + + /** + * The handle {@link #stillAtPath} read the token back through, held open as long as the lock. Closing any handle on + * a file releases every POSIX lock the process holds on it, so closing this one would silently release the lock + * while the JVM still reports it held, and a graph in another process could then open the store. + */ + private final FileChannel pathChannel; + + /** + * This lock's entry in {@link #HELD}, removed when the lock is released. + */ + private final String heldKey; + private final File lockFile; + + /** + * Whether acquiring the lock created the {@code LOCK} file, rather than finding it left by an earlier graph. + */ + private final boolean createdFile; + + private DirectoryLock(final FileChannel channel, final FileLock lock, final FileChannel pathChannel, + final File lockFile, final boolean createdFile, final String heldKey) { + this.channel = channel; + this.lock = lock; + this.pathChannel = pathChannel; + this.heldKey = heldKey; + this.lockFile = lockFile; + this.createdFile = createdFile; + } + + /** + * Acquire an exclusive lock on the {@code LOCK} file within {@code directory}. + * + * @param directory the storage directory, which must already exist + * @return the held lock, released by {@link #close()} + * @throws IllegalStateException if another graph (in this or another process) already holds the lock + */ + public static DirectoryLock acquire(final File directory) { + return acquire(directory, true); + } + + private static DirectoryLock acquire(final File directory, final boolean verify) { + final String heldKey = hold(directory); + boolean acquired = false; + try { + final DirectoryLock held = acquire(directory, verify, heldKey); + acquired = true; + return held; + } finally { + if (!acquired) + HELD.remove(heldKey); + } + } + + private static DirectoryLock acquire(final File directory, final boolean verify, final String heldKey) { + final File lockFile = new File(directory, LOCK_FILE); + FileChannel channel = null; + final boolean createdFile = !lockFile.exists(); + try { + channel = FileChannel.open(lockFile.toPath(), + StandardOpenOption.CREATE, StandardOpenOption.WRITE); + final FileLock lock = channel.tryLock(LOCKED_BYTE, 1, false); + if (lock == null) { + channel.close(); + throw lockedByAnother(directory, null); + } + final FileChannel pathChannel = verify ? stillAtPath(channel, lockFile) : null; + if (verify && null == pathChannel) { + lock.release(); + channel.close(); + throw lockedByAnother(directory, null); + } + return new DirectoryLock(channel, lock, pathChannel, lockFile, createdFile, heldKey); + } catch (OverlappingFileLockException ofle) { + // another graph in *this* JVM already holds (or is acquiring) the lock on this file + closeQuietly(channel); + throw lockedByAnother(directory, ofle); + } catch (AccessDeniedException ex) { + closeQuietly(channel); + throw new UncheckedIOException(String.format("Could not acquire storage lock for %s, which this process " + + "cannot write; a store there can only be opened with %s", directory, + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER), ex); + } catch (IOException ex) { + closeQuietly(channel); + throw new UncheckedIOException(String.format("Could not acquire storage lock for %s", directory), ex); + } + } + + /** + * Lock {@code directory} for a recovery open, which only reads it. A writable directory is locked exclusively, + * as by {@link #acquire}. A directory the process cannot write, such as a read-only mount or copy of a damaged + * store, can hold no new lock file: an existing one is locked shared, which still fails while a writer holds it, + * and with none the directory is read unlocked, since recovery writes nothing to it. + * + * @param directory the storage directory to lock + * @return the held lock + * @throws IllegalStateException if another graph holds the lock + */ + public static DirectoryLock acquireForRecovery(final File directory) { + // recovery leaves every file as it found it, so it does not write the token that verifies the lock, and since + // it never writes to the store, a lock it holds on a file that was replaced cannot cost a commit + if (Files.isWritable(directory.toPath())) + return acquire(directory, false); + final String heldKey = hold(directory); + boolean acquired = false; + try { + final DirectoryLock held = acquireShared(directory, heldKey); + acquired = true; + return held; + } finally { + if (!acquired) + HELD.remove(heldKey); + } + } + + private static DirectoryLock acquireShared(final File directory, final String heldKey) { + final File lockFile = new File(directory, LOCK_FILE); + if (!lockFile.exists()) { + logger.warn("Storage directory {} is not writable, so it is read for recovery without a lock; make sure " + + "no other graph writes to it", directory); + return new DirectoryLock(null, null, null, lockFile, false, heldKey); + } + FileChannel channel = null; + try { + channel = FileChannel.open(lockFile.toPath(), StandardOpenOption.READ); + final FileLock lock = channel.tryLock(LOCKED_BYTE, 1, true); + if (lock == null) { + channel.close(); + throw lockedByAnother(directory, null); + } + return new DirectoryLock(channel, lock, null, lockFile, false, heldKey); + } catch (OverlappingFileLockException ofle) { + closeQuietly(channel); + throw lockedByAnother(directory, ofle); + } catch (IOException ex) { + closeQuietly(channel); + throw new UncheckedIOException(String.format("Could not acquire storage lock for %s", directory), ex); + } + } + + /** + * Record that this JVM is acquiring the lock of {@code directory}, failing if a graph in it already holds or is + * acquiring that lock. The directory is named by its real path, so a symbolic link to it is the same directory. + */ + private static String hold(final File directory) { + String key; + try { + key = directory.toPath().toRealPath().toString(); + } catch (IOException ex) { + key = directory.getAbsoluteFile().toPath().normalize().toString(); + } + if (!HELD.add(key)) + throw lockedByAnother(directory, null); + return key; + } + + /** + * Whether {@code lockFile} still names the file {@code channel} holds. A graph whose open fails removes the LOCK file + * it created, so between opening that file and locking it, another graph could have unlinked it and created a new + * one at the path, leaving this lock on a file that keeps no one out. A random token written through the channel + * and read back through the path tells the two apart, which comparing file keys cannot, since the channel does not + * report the key of the file it has open. + * + * @return the handle the token was read back through, which the caller must keep open as long as it holds the lock + * (see {@link #pathChannel}), or {@code null} if the path names another file or none + */ + private static FileChannel stillAtPath(final FileChannel channel, final File lockFile) throws IOException { + final UUID token = UUID.randomUUID(); + final byte[] bytes = ByteBuffer.allocate(2 * Long.BYTES) + .putLong(token.getMostSignificantBits()).putLong(token.getLeastSignificantBits()).array(); + channel.truncate(0); + channel.write(ByteBuffer.wrap(bytes), 0); + channel.force(false); + final FileChannel pathChannel; + try { + pathChannel = FileChannel.open(lockFile.toPath(), StandardOpenOption.READ); + } catch (NoSuchFileException ex) { + return null; + } + final ByteBuffer read = ByteBuffer.allocate(bytes.length + 1); + try { + while (read.hasRemaining() && pathChannel.read(read, read.position()) >= 0) { + // read until the buffer is full or the file ends + } + } catch (IOException ex) { + closeQuietly(pathChannel); + throw ex; + } + if (read.position() == bytes.length && Arrays.equals(bytes, Arrays.copyOf(read.array(), bytes.length))) + return pathChannel; + // the path names another file, so this handle holds nothing of the lock and the open is refused anyway + closeQuietly(pathChannel); + return null; + } + + private static IllegalStateException lockedByAnother(final File directory, final Throwable cause) { + return new IllegalStateException(String.format( + "Storage location %s is already in use by another TinkerStorageGraph (in this or another process); " + + "a persistent TinkerStorageGraph allows only a single writer", directory), cause); + } + + private static void closeQuietly(final FileChannel channel) { + if (channel != null) { + try { + channel.close(); + } catch (IOException ignored) { + // best effort on the failure path + } + } + } + + /** + * Release the lock and, if acquiring it created the {@code LOCK} file, delete that file, leaving the directory as + * it was found. For a graph that must not change the directory: one whose open failed, or one opened for recovery. + */ + public void closeAndRemoveIfCreated() { + // delete while the lock is still held where the platform allows it, so no other graph can take the lock on this + // file in between and then have its LOCK file deleted under it. Windows refuses to delete an open file, so + // there the file is removed once the lock is released. + final boolean deleted = createdFile && lockFile.delete(); + try { + release(); + if (createdFile && !deleted && lockFile.exists() && !lockFile.delete()) + throw new UncheckedIOException(new IOException(String.format("Could not remove storage lock %s", + lockFile))); + } finally { + // kept until the file is gone, so no graph in this JVM can create one that this then deletes + HELD.remove(heldKey); + } + } + + /** + * Release the lock and close the channel. Idempotent-friendly: safe to call once per acquired lock. + */ + @Override + public void close() { + try { + release(); + } finally { + // only once every handle on the file is closed may another graph in this JVM open one + HELD.remove(heldKey); + } + } + + private void release() { + // a recovery open of a directory it cannot write holds no lock at all + if (null == channel) + return; + try { + if (lock.isValid()) + lock.release(); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not release storage lock %s", lockFile), ex); + } finally { + closeQuietly(pathChannel); + closeQuietly(channel); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java new file mode 100644 index 00000000000..0bc79c5adeb --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java @@ -0,0 +1,634 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Property; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.structure.io.Buffer; +import org.apache.tinkerpop.gremlin.structure.io.binary.DataType; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader; +import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter; +import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializer; +import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedProperty; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.io.DataOutputStream; +import java.io.IOException; +import java.math.BigInteger; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * The GraphBinary {@link TinkerStorage} codec on top of {@link AbstractLogStorage}. The base owns the durable + * log-structured machinery (file layout, {@code VERSION} marker, CRC framing, replay fold, {@link SyncMode} + * durability, crash-safe and threshold compaction); this class supplies only how an element is encoded and decoded. + *

+ * Rather than serialize a whole {@code DetachedVertex}/{@code DetachedEdge} (which repeats every property key and + * label as a full string on every element and wraps each property in a {@code VertexProperty} envelope), this codec + * writes element components directly: element ids and property values go through GraphBinary's scalar + * serializers (a one-byte {@link DataType} tag plus the raw value), while labels, property keys, and meta-property + * keys are dictionary-encoded to small integer refs. The dictionary is a single dense namespace built by first + * appearance; new entries are emitted as {@code OP_DICT_APPEND} records inside the same frame, before the entries + * that reference them, so a torn trailing frame drops a ref and its user atomically. + *

+ * Vertex-property ids are auto-generated and, by default, not persisted (they are regenerated on load); element and + * edge ids are always preserved. The snapshot streams one element per frame, so compaction never holds a second full + * copy of the graph in heap. + */ +public final class GraphBinaryStorage extends AbstractLogStorage { + + private static final byte OP_PUT_VERTEX = 1; + private static final byte OP_DEL_VERTEX = 2; + private static final byte OP_PUT_EDGE = 3; + private static final byte OP_DEL_EDGE = 4; + private static final byte OP_DICT_APPEND = 5; + + private final TypeSerializerRegistry registry = TypeSerializerRegistry.INSTANCE; + private final GraphBinaryWriter writer = new GraphBinaryWriter(registry); + private final GraphBinaryReader reader = new BoundedReader(registry); + + /** + * Shared string dictionary for labels, property keys, and meta-property keys. Dense ids assigned by first + * appearance; grows monotonically within a write session and is rebuilt fresh on compaction and on replay. + */ + private final Map keyToId = new HashMap<>(); + private final List idToKey = new ArrayList<>(); + + /** + * When true, persist auto-generated vertex-property ids so they are stable across reopen. Written per vertex + * record so a store reopens correctly regardless of the reader's setting. + */ + private boolean preserveVertexPropertyIds = false; + + @Override + protected void configureCodec(final Configuration config) { + this.preserveVertexPropertyIds = config.getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, false); + } + + @Override + protected void beginReplay() { + keyToId.clear(); + idToKey.clear(); + } + + // --------------------------------------------------------------------------------------------- encode + + @Override + protected byte[] encodeCommit(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) throws IOException { + // the dictionary entries this commit adds only reach disk inside its frame, so if encoding fails they must be + // forgotten. Otherwise a later commit would reference them without defining them, or define the next entry + // after a gap, and the log could no longer be replayed. + final int dictionarySize = idToKey.size(); + try { + return encodeCommitFrame(changedVertices, changedEdges); + } catch (IOException | RuntimeException | Error ex) { + // an Error such as OutOfMemoryError counts too: the transaction is rolled back and the graph carries on + while (idToKey.size() > dictionarySize) + keyToId.remove(idToKey.remove(idToKey.size() - 1)); + throw ex; + } + } + + private byte[] encodeCommitFrame(final Collection> changedVertices, + final Collection> changedEdges) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + // register the strings introduced by this commit (deletes carry only an id, no strings) + final List appends = new ArrayList<>(); + for (final TinkerStorageMutation m : changedVertices) + if (!m.isDeleted()) registerVertexStrings(m.element(), appends); + for (final TinkerStorageMutation m : changedEdges) + if (!m.isDeleted()) registerEdgeStrings(m.element(), appends); + + writeVarInt(buf, appends.size() + changedVertices.size() + changedEdges.size()); + // dictionary appends first, so every ref below resolves during the fold + for (final String s : appends) { + buf.writeByte(OP_DICT_APPEND); + writeVarInt(buf, keyToId.get(s)); + writeString(buf, s); + } + for (final TinkerStorageMutation m : changedVertices) { + if (m.isDeleted()) { + buf.writeByte(OP_DEL_VERTEX); + writeScalar(buf, m.id()); + } else { + buf.writeByte(OP_PUT_VERTEX); + writeVertexRecord(buf, m.element()); + } + } + for (final TinkerStorageMutation m : changedEdges) { + if (m.isDeleted()) { + buf.writeByte(OP_DEL_EDGE); + writeScalar(buf, m.id()); + } else { + buf.writeByte(OP_PUT_EDGE); + writeEdgeRecord(buf, m.element()); + } + } + return buf.toWrittenArray(); + } + + @Override + protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException { + // Preserve the existing dictionary numbering rather than renumbering. No log is ever replayed against a newer + // snapshot, since an open sets aside the old log of an interrupted compaction, so this is not needed for + // correctness, but it costs nothing. Register any not-yet-seen live strings (this only extends the + // dictionary), then emit the whole dictionary as a self-contained header frame so a snapshot-only replay + // resolves every ref. + // committed*() rather than vertices()/edges(): the snapshot must reflect only committed state, never the + // calling thread's uncommitted transaction-local mutations, which the transaction-aware iterators would expose. + final List ignored = new ArrayList<>(); + Iterator vertices = graph.committedVertices(); + while (vertices.hasNext()) + registerVertexStrings(vertices.next(), ignored); + Iterator edges = graph.committedEdges(); + while (edges.hasNext()) + registerEdgeStrings(edges.next(), ignored); + + final TinkerByteBuffer dictBuf = new TinkerByteBuffer(); + writeVarInt(dictBuf, idToKey.size()); + for (int id = 0; id < idToKey.size(); id++) { + dictBuf.writeByte(OP_DICT_APPEND); + writeVarInt(dictBuf, id); + writeString(dictBuf, idToKey.get(id)); + } + writeFrame(out, dictBuf.toWrittenArray()); + + // one element per frame; all keys are already in the dictionary header, so no per-frame appends + vertices = graph.committedVertices(); + while (vertices.hasNext()) + writeElementFrame(out, OP_PUT_VERTEX, vertices.next()); + edges = graph.committedEdges(); + while (edges.hasNext()) + writeElementFrame(out, OP_PUT_EDGE, edges.next()); + } + + /** + * Write one element as its own single-entry put frame. One element is held in memory at a time, so a large graph + * is never buffered whole. + */ + private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + writeVarInt(buf, 1); + buf.writeByte(op); + if (op == OP_PUT_VERTEX) writeVertexRecord(buf, (Vertex) element); + else writeEdgeRecord(buf, (Edge) element); + writeFrame(out, buf.toWrittenArray()); + } + + private void registerVertexStrings(final Vertex v, final List appends) { + for (final String label : v.labels()) + register(label, appends); + final Iterator> vps = v.properties(); + while (vps.hasNext()) { + final VertexProperty vp = vps.next(); + register(vp.key(), appends); + final Iterator> metas = vp.properties(); + while (metas.hasNext()) + register(metas.next().key(), appends); + } + } + + private void registerEdgeStrings(final Edge e, final List appends) { + register(e.label(), appends); + final Iterator> props = e.properties(); + while (props.hasNext()) + register(props.next().key(), appends); + } + + private void register(final String s, final List appends) { + if (!keyToId.containsKey(s)) { + final int id = idToKey.size(); + keyToId.put(s, id); + idToKey.add(s); + appends.add(s); + } + } + + private void writeVertexRecord(final TinkerByteBuffer buf, final Vertex v) throws IOException { + try { + writeScalar(buf, v.id()); + } catch (IOException ex) { + throw cannotStore("the id of a vertex", v.id(), ex); + } + final Set labels = v.labels(); + writeVarInt(buf, labels.size()); + for (final String label : labels) + writeVarInt(buf, keyToId.get(label)); + + // self-describing flag: whether each value below carries a persisted vertex-property id + buf.writeByte(preserveVertexPropertyIds ? 1 : 0); + + // group vertex properties by key so multi-properties (list/set) round-trip + final Map>> groups = new LinkedHashMap<>(); + final Iterator> vps = v.properties(); + while (vps.hasNext()) { + final VertexProperty vp = vps.next(); + groups.computeIfAbsent(vp.key(), k -> new ArrayList<>()).add(vp); + } + writeVarInt(buf, groups.size()); + for (final Map.Entry>> group : groups.entrySet()) { + writeVarInt(buf, keyToId.get(group.getKey())); + final List> values = group.getValue(); + writeVarInt(buf, values.size()); + for (final VertexProperty vp : values) { + try { + writeScalar(buf, vp.value()); + } catch (IOException ex) { + throw cannotStore(String.format("property '%s' of vertex %s", vp.key(), v.id()), vp.value(), ex); + } + if (preserveVertexPropertyIds) { + try { + writeScalar(buf, vp.id()); + } catch (IOException ex) { + throw cannotStore(String.format("the id of property '%s' of vertex %s", vp.key(), v.id()), + vp.id(), ex); + } + } + final List> metas = new ArrayList<>(); + vp.properties().forEachRemaining(metas::add); + writeVarInt(buf, metas.size()); + for (final Property meta : metas) { + writeVarInt(buf, keyToId.get(meta.key())); + try { + writeScalar(buf, meta.value()); + } catch (IOException ex) { + throw cannotStore(String.format("meta-property '%s' of property '%s' of vertex %s", + meta.key(), vp.key(), v.id()), meta.value(), ex); + } + } + } + } + } + + private void writeEdgeRecord(final TinkerByteBuffer buf, final Edge e) throws IOException { + try { + writeScalar(buf, e.id()); + } catch (IOException ex) { + throw cannotStore("the id of an edge", e.id(), ex); + } + writeVarInt(buf, keyToId.get(e.label())); + writeScalar(buf, e.outVertex().id()); + writeScalar(buf, e.inVertex().id()); + final List> props = new ArrayList<>(); + e.properties().forEachRemaining(props::add); + writeVarInt(buf, props.size()); + for (final Property p : props) { + writeVarInt(buf, keyToId.get(p.key())); + try { + writeScalar(buf, p.value()); + } catch (IOException ex) { + throw cannotStore(String.format("property '%s' of edge %s", p.key(), e.id()), p.value(), ex); + } + } + } + + /** + * Describe a value that could not be encoded by what it belongs to and its type, which the serializer's own + * message does not say. + */ + private static IOException cannotStore(final String what, final Object value, final IOException cause) { + return new IOException(String.format("Could not store %s, a %s: %s", + what, value.getClass().getTypeName(), cause.getMessage()), cause); + } + + // --------------------------------------------------------------------------------------------- decode + + @Override + protected void decodeFrame(final byte[] record, + final Map vertices, + final Map edges) throws IOException { + final TinkerByteBuffer buf = new TinkerByteBuffer(record); + final int entryCount = readVarInt(buf); + for (int i = 0; i < entryCount; i++) { + final byte op = buf.readByte(); + switch (op) { + case OP_DICT_APPEND: { + final int id = readVarInt(buf); + final String s = readString(buf); + // idempotent: a snapshot header defines the whole dictionary, and re-appending an existing id with + // the same string is a no-op; a mismatch or a gap is corruption. + if (id < idToKey.size()) { + if (!idToKey.get(id).equals(s)) + throw new IOException(String.format("Corrupt storage: dictionary id %d redefined ('%s' vs '%s')", id, idToKey.get(id), s)); + } else if (id == idToKey.size()) { + idToKey.add(s); + } else { + throw new IOException(String.format("Corrupt storage: dictionary append gap (got %d, expected <= %d)", id, idToKey.size())); + } + // rebuild the write-side mapping too, first appearance wins, so a write session after this replay + // resumes the existing numbering instead of re-appending every live key as a duplicate on each reopen + keyToId.putIfAbsent(s, id); + break; + } + case OP_PUT_VERTEX: { + final DetachedVertex v = readVertexRecord(buf); + vertices.put(v.id(), v); + break; + } + case OP_DEL_VERTEX: { + vertices.remove(readScalar(buf)); + break; + } + case OP_PUT_EDGE: { + final DetachedEdge e = readEdgeRecord(buf); + edges.put(e.id(), e); + break; + } + case OP_DEL_EDGE: { + edges.remove(readScalar(buf)); + break; + } + default: + throw new IOException("Unknown storage op code: " + op); + } + } + } + + private DetachedVertex readVertexRecord(final TinkerByteBuffer buf) throws IOException { + final Object id = readScalar(buf); + final DetachedVertex.Builder b = DetachedVertex.build().setId(id); + final int labelCount = readVarInt(buf); + if (labelCount == 1) { + b.setLabel(resolveKey(buf)); + } else if (labelCount > 1) { + final Set labels = new LinkedHashSet<>(); + for (int i = 0; i < labelCount; i++) + labels.add(resolveKey(buf)); + b.setLabels(labels); + } + final boolean hasVpIds = buf.readByte() != 0; + final int keyGroupCount = readVarInt(buf); + for (int g = 0; g < keyGroupCount; g++) { + final String key = resolveKey(buf); + final int valueCount = readVarInt(buf); + for (int j = 0; j < valueCount; j++) { + final Object value = readScalar(buf); + final DetachedVertexProperty.Builder vpb = DetachedVertexProperty.build().setLabel(key).setValue(value); + if (hasVpIds) + vpb.setId(readScalar(buf)); + final int metaCount = readVarInt(buf); + for (int m = 0; m < metaCount; m++) { + final String metaKey = resolveKey(buf); + final Object metaValue = readScalar(buf); + vpb.addProperty(new DetachedProperty<>(metaKey, metaValue)); + } + b.addProperty(vpb.create()); + } + } + return b.create(); + } + + private DetachedEdge readEdgeRecord(final TinkerByteBuffer buf) throws IOException { + final Object id = readScalar(buf); + final String label = resolveKey(buf); + final Object outVId = readScalar(buf); + final Object inVId = readScalar(buf); + final DetachedEdge.Builder b = DetachedEdge.build().setId(id).setLabel(label) + .setOutV(DetachedVertex.build().setId(outVId).create()) + .setInV(DetachedVertex.build().setId(inVId).create()); + final int propCount = readVarInt(buf); + for (int i = 0; i < propCount; i++) { + final String key = resolveKey(buf); + final Object value = readScalar(buf); + b.addProperty(new DetachedProperty<>(key, value)); + } + return b.create(); + } + + // --------------------------------------------------------------------------------------------- primitives + + /** + * Write a value as a one-byte {@link DataType} tag followed by the raw value (no value-flag byte). A {@code null} + * is a single {@link DataType#UNSPECIFIED_NULL} tag. + */ + @SuppressWarnings({"unchecked", "rawtypes"}) + private void writeScalar(final TinkerByteBuffer buf, final Object value) throws IOException { + if (value == null) { + buf.writeByte(DataType.UNSPECIFIED_NULL.getCodeByte()); + return; + } + final TypeSerializer serializer; + try { + serializer = registry.getSerializer(value.getClass()); + } catch (IOException ex) { + throw new IOException(String.format("the graphbinary storage engine has no serializer for %s; see the " + + "TinkerGraph persistence documentation for the types it can store", + value.getClass().getTypeName()), ex); + } + buf.writeByte(serializer.getDataType().getCodeByte()); + serializer.writeValue(value, buf, writer, false); + } + + @SuppressWarnings({"unchecked", "rawtypes"}) + private Object readScalar(final TinkerByteBuffer buf) throws IOException { + final int code = Byte.toUnsignedInt(buf.readByte()); + final DataType dataType = DataType.get(code); + if (dataType == null) + throw new IOException(String.format("Corrupt storage: unknown value type code 0x%02X", code)); + if (dataType == DataType.UNSPECIFIED_NULL) + return null; + final TypeSerializer serializer = registry.getSerializer(dataType); + checkDeclaredSize(buf, dataType, buf.readerIndex()); + try { + return serializer.readValue(buf, reader, false); + } catch (RuntimeException ex) { + // a value that runs past the end of its record, or declares a negative size, is corruption + throw new IOException(String.format("Corrupt storage frame: unreadable %s value (%s)", dataType, ex), ex); + } + } + + /** + * Fail if the size the value at {@code offset} declares for itself cannot fit in what remains of the record. The + * GraphBinary serializers allocate from that size before reading, so a corrupt record that declares gigabytes in + * a frame of a few bytes would otherwise raise {@code OutOfMemoryError} or {@code NegativeArraySizeException} + * instead of the {@code IOException} a corrupt frame is contracted to produce. Each element of a collection takes + * at least two bytes, its type code and value flag, and each map entry two such elements. + */ + private static void checkDeclaredSize(final TinkerByteBuffer buf, final DataType dataType, final int offset) + throws IOException { + final int bytesPerItem; + switch (dataType) { + case STRING: + case BINARY: + case BIGINTEGER: + bytesPerItem = 1; + break; + case LIST: + case SET: + bytesPerItem = 2; + break; + case MAP: + bytesPerItem = 4; + break; + default: + return; + } + final long available = (long) buf.writerIndex() - offset - Integer.BYTES; + if (available < 0) + throw new IOException(String.format("Corrupt storage frame: %s value is cut off before its size", dataType)); + final int declared = buf.getInt(offset); + if (declared < 0 || (long) declared * bytesPerItem > available) + throw new IOException(String.format("Corrupt storage frame: %s value declares a size of %d with only %d " + + "bytes left in the record", dataType, declared, available)); + } + + /** + * Reads nested values, such as the elements of a list, with the same size checks that {@link #readScalar} + * applies to a top-level value, and rejects bulked lists, which this engine never writes. + */ + private static final class BoundedReader extends GraphBinaryReader { + private BoundedReader(final TypeSerializerRegistry registry) { + super(registry); + } + + @Override + public T read(final Buffer buffer) throws IOException { + // fully-qualified: {type_code}{value_flag}{value} + if (buffer instanceof TinkerByteBuffer && buffer.readableBytes() >= 2) { + final TinkerByteBuffer buf = (TinkerByteBuffer) buffer; + final DataType type = DataType.get(Byte.toUnsignedInt(buf.getByte(buf.readerIndex()))); + final byte valueFlag = buf.getByte(buf.readerIndex() + 1); + if (type != null && (valueFlag & 1) == 0) { + if (type == DataType.LIST && (valueFlag & 2) == 2) + throw new IOException("Corrupt storage frame: bulked list, which this engine never writes"); + checkDeclaredSize(buf, type, buf.readerIndex() + 2); + } + } + return super.read(buffer); + } + + @Override + public T readValue(final Buffer buffer, final Class type, final boolean nullable) throws IOException { + // serializers read some parts of a value with its type already known, such as the unscaled BigInteger of a + // BigDecimal or the label list of a vertex or edge, so those sizes are checked here rather than in read() + final DataType sized = sizedTypeOf(type); + if (buffer instanceof TinkerByteBuffer && sized != null) { + final TinkerByteBuffer buf = (TinkerByteBuffer) buffer; + int offset = buf.readerIndex(); + if (nullable) { + if (buf.readableBytes() < 1) + throw new IOException(String.format("Corrupt storage frame: %s value is cut off", sized)); + // a null carries no size to check + if ((buf.getByte(offset) & 1) == 1) + return super.readValue(buffer, type, nullable); + offset++; + } + checkDeclaredSize(buf, sized, offset); + } + return super.readValue(buffer, type, nullable); + } + + private static DataType sizedTypeOf(final Class type) { + if (type == String.class) return DataType.STRING; + if (type == ByteBuffer.class) return DataType.BINARY; + if (type == BigInteger.class) return DataType.BIGINTEGER; + if (type == List.class) return DataType.LIST; + if (type == Set.class) return DataType.SET; + if (type == Map.class) return DataType.MAP; + return null; + } + } + + private static void writeString(final TinkerByteBuffer buf, final String s) { + final byte[] bytes = s.getBytes(StandardCharsets.UTF_8); + writeVarInt(buf, bytes.length); + buf.writeBytes(bytes); + } + + /** + * Resolve the next dictionary ref in {@code buf} to its string. A ref that names an entry the dictionary does not + * hold is corruption, and is reported as such rather than raised as an {@code IndexOutOfBoundsException} from the + * backing list. + */ + private String resolveKey(final TinkerByteBuffer buf) throws IOException { + final int id = readVarInt(buf); + if (id >= idToKey.size()) + throw new IOException(String.format( + "Corrupt storage frame: dictionary ref %d with only %d entries defined", id, idToKey.size())); + return idToKey.get(id); + } + + private static String readString(final TinkerByteBuffer buf) throws IOException { + final int length = readVarInt(buf); + // check the declared length against what the frame actually holds before allocating. The frame itself is + // already bounded against the file by AbstractLogStorage.readFrame, but a length inside the frame is not, + // so an unchecked allocation here would let a small corrupt record demand gigabytes and raise + // OutOfMemoryError instead of the IOException a corrupt frame is contracted to produce. + if (length > buf.readableBytes()) + throw new IOException(String.format( + "Corrupt storage frame: string of %d bytes declared with only %d readable in the record", + length, buf.readableBytes())); + final byte[] bytes = new byte[length]; + buf.readBytes(bytes); + return new String(bytes, StandardCharsets.UTF_8); + } + + /** + * Unsigned LEB128 varint. Counts and dictionary refs are small and non-negative, so they cost one byte in the + * common case. + */ + private static void writeVarInt(final TinkerByteBuffer buf, final int value) { + int v = value; + while ((v & ~0x7F) != 0) { + buf.writeByte((v & 0x7F) | 0x80); + v >>>= 7; + } + buf.writeByte(v & 0x7F); + } + + private static int readVarInt(final TinkerByteBuffer buf) throws IOException { + int result = 0; + int shift = 0; + byte b; + do { + // Java masks a shift count to five bits, so without this bound an over-long encoding wraps around and + // yields an arbitrary (possibly negative) value rather than failing. Every count, length and dictionary + // ref in the format is non-negative, so anything that does not fit in five groups is corruption. + if (shift >= Integer.SIZE) + throw new IOException("Corrupt storage frame: over-long varint encoding"); + b = buf.readByte(); + result |= (b & 0x7F) << shift; + shift += 7; + } while ((b & 0x80) != 0); + if (result < 0) + throw new IOException("Corrupt storage frame: negative varint value " + result); + return result; + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java new file mode 100644 index 00000000000..c3baafba21d --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/IndexDefinitions.java @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.Writer; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; + +/** + * The set of property keys a persistent graph indexes, held in a small file beside the storage engine's own files. + *

+ * Index definitions are deliberately kept out of the transactional log. An index is purely an optimization, so losing + * a definition to a crash costs a rebuild rather than any data: a crash between {@code createIndex} and the write here + * drops a definition the caller can recreate, and one after {@code dropIndex} resurrects an index that is dropped + * again. Neither can change a query result, which is what makes a file outside the write-ahead log an honest place to + * keep this. Nothing that affects correctness may be stored this way. + *

+ * The file is line oriented and readable, one record per line: {@code V} or {@code E}, a tab, then the property key + * with backslash, tab, newline and carriage return escaped, so a key containing any of them survives a round trip. + */ +public final class IndexDefinitions { + + private static final Logger logger = LoggerFactory.getLogger(IndexDefinitions.class); + + static final String INDEX_FILE = "INDEXES"; + + private static final String VERTEX = "V"; + private static final String EDGE = "E"; + + private final Set vertexKeys; + private final Set edgeKeys; + + public IndexDefinitions(final Set vertexKeys, final Set edgeKeys) { + this.vertexKeys = new LinkedHashSet<>(vertexKeys); + this.edgeKeys = new LinkedHashSet<>(edgeKeys); + } + + public Set vertexKeys() { + return vertexKeys; + } + + public Set edgeKeys() { + return edgeKeys; + } + + public boolean isEmpty() { + return vertexKeys.isEmpty() && edgeKeys.isEmpty(); + } + + /** + * Read the definitions recorded in {@code directory}, or an empty set if none have been recorded. + *

+ * A file that cannot be read is reported and treated as empty rather than raised. The graph then opens with no + * indexes, which is exactly how it behaved before definitions were recorded at all, so an unreadable file can + * never make a store less openable than the data it holds. + */ + public static IndexDefinitions read(final File directory) { + final File file = new File(directory, INDEX_FILE); + if (!file.isFile()) + return new IndexDefinitions(new LinkedHashSet<>(), new LinkedHashSet<>()); + + final Set vertexKeys = new LinkedHashSet<>(); + final Set edgeKeys = new LinkedHashSet<>(); + try { + final List lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); + for (final String line : lines) { + if (line.isEmpty() || line.charAt(0) == '#') + continue; + final int tab = line.indexOf('\t'); + if (tab < 0) + throw new IOException("Malformed index definition line: " + line); + final String key = unescape(line.substring(tab + 1)); + switch (line.substring(0, tab)) { + case VERTEX: vertexKeys.add(key); break; + case EDGE: edgeKeys.add(key); break; + default: throw new IOException("Unknown index element type in line: " + line); + } + } + } catch (IOException ex) { + logger.warn("Could not read index definitions from {}; opening with no indexes. " + + "Recreate them with createIndex() if they are wanted.", file, ex); + return new IndexDefinitions(new LinkedHashSet<>(), new LinkedHashSet<>()); + } + return new IndexDefinitions(vertexKeys, edgeKeys); + } + + /** + * Replace the definitions recorded in {@code directory}. Written to a temporary file, forced to the device and + * renamed into place, so a crash leaves either the previous set or the new one and never a partial file. + */ + public void write(final File directory) { + final File file = new File(directory, INDEX_FILE); + if (isEmpty()) { + try { + Files.deleteIfExists(file.toPath()); + StorageFiles.syncDirectory(directory); + } catch (IOException ex) { + logger.warn("Could not remove index definitions file {}", file, ex); + } + return; + } + + final File tmp = new File(directory, INDEX_FILE + ".tmp"); + try { + try (final FileOutputStream fos = new FileOutputStream(tmp); + final Writer out = new OutputStreamWriter(fos, StandardCharsets.UTF_8)) { + out.write("# TinkerGraph index definitions; recreated on open\n"); + for (final String key : vertexKeys) + out.write(VERTEX + '\t' + escape(key) + '\n'); + for (final String key : edgeKeys) + out.write(EDGE + '\t' + escape(key) + '\n'); + out.flush(); + fos.getFD().sync(); + } + StorageFiles.atomicMove(tmp, file); + StorageFiles.syncDirectory(directory); + } catch (IOException ex) { + logger.warn("Could not record index definitions in {}; they will not survive a reopen", file, ex); + } + } + + private static String escape(final String key) { + final StringBuilder sb = new StringBuilder(key.length()); + for (int i = 0; i < key.length(); i++) { + final char c = key.charAt(i); + switch (c) { + case '\\': sb.append("\\\\"); break; + case '\t': sb.append("\\t"); break; + case '\n': sb.append("\\n"); break; + case '\r': sb.append("\\r"); break; + default: sb.append(c); + } + } + return sb.toString(); + } + + private static String unescape(final String value) throws IOException { + final StringBuilder sb = new StringBuilder(value.length()); + for (int i = 0; i < value.length(); i++) { + final char c = value.charAt(i); + if (c != '\\') { + sb.append(c); + continue; + } + if (++i == value.length()) + throw new IOException("Index definition ends with a dangling escape: " + value); + switch (value.charAt(i)) { + case '\\': sb.append('\\'); break; + case 't': sb.append('\t'); break; + case 'n': sb.append('\n'); break; + case 'r': sb.append('\r'); break; + default: throw new IOException("Unknown escape in index definition: " + value); + } + } + return sb.toString(); + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFiles.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFiles.java new file mode 100644 index 00000000000..9fefb30ab8c --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFiles.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.IOException; +import java.nio.channels.FileChannel; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; + +/** + * File operations shared by the files a {@code TinkerStorageGraph} keeps in its storage directory, which are all + * replaced whole by writing a temporary file and renaming it into place. + */ +final class StorageFiles { + + private static final Logger logger = LoggerFactory.getLogger(StorageFiles.class); + + private StorageFiles() { + } + + /** + * Atomically move {@code source} onto {@code target}, replacing any existing target. Falls back to a non-atomic + * replacing move on filesystems that do not support atomic moves. + */ + static void atomicMove(final File source, final File target) throws IOException { + try { + Files.move(source.toPath(), target.toPath(), + StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); + } catch (AtomicMoveNotSupportedException anse) { + Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING); + } + } + + /** + * fsync {@code directory} so that recent namespace changes (a rename into place, a file deletion) are durable. + */ + static void syncDirectory(final File directory) { + final FileChannel dirChannel; + try { + dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ); + } catch (IOException ex) { + // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the + // durability guarantee there, so treat inability to sync the directory as non-fatal + return; + } + // some filesystems (certain network and FUSE mounts) open a directory but refuse to force it, so a failure is + // reported rather than fatal: the rename has happened, and only its durability against a crash is in doubt + try (final FileChannel channel = dirChannel) { + channel.force(true); + } catch (IOException ex) { + logger.warn("Could not force storage directory {} to disk, so a file just renamed into it may not survive " + + "a crash", directory, ex); + } + } + + /** + * Delete {@code file} if it exists, ignoring a failure. For temporary files that a later write overwrites anyway. + */ + static void deleteQuietly(final File file) { + try { + Files.deleteIfExists(file.toPath()); + } catch (IOException ignored) { + // best effort: the next write of the same temporary file overwrites it + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettings.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettings.java new file mode 100644 index 00000000000..7626e275b3c --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettings.java @@ -0,0 +1,314 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.commons.configuration2.ConfigurationUtils; +import org.apache.commons.configuration2.convert.PropertyConverter; +import org.apache.commons.configuration2.ex.ConversionException; +import org.apache.tinkerpop.gremlin.structure.LabelCardinality; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.io.Reader; +import java.io.UncheckedIOException; +import java.io.Writer; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Properties; +import java.util.Set; + +/** + * The settings a storage directory was created with, recorded in {@link #SETTINGS_FILE} beside the engine's own files. + *

+ * Some settings shape the data itself: the id managers decide the type of every stored id, the cardinalities and + * null handling decide how stored properties read back, and whether vertex-property ids are preserved decides whether + * the store holds them at all, since every compaction rewrites the store with the setting it was opened with. A store is only ever read with the settings that wrote it, so + * these are fixed when the store is created. Changing one is a migration: export the graph with {@code g.io()} and + * import it into a new store. The record is written once, holding the effective value of each such setting (defaults + * included), and never rewritten. + *

+ * On open, a setting the configuration leaves unset takes the recorded value, so a store opens with nothing but its + * storage engine and directory and the original configuration can be lost without harm. A setting that is set to a + * different value fails the open. Settings that only affect how the engine runs, such as the sync mode or compaction + * threshold, are not recorded and may change freely. + */ +public final class StorageSettings { + + private static final Logger logger = LoggerFactory.getLogger(StorageSettings.class); + + public static final String SETTINGS_FILE = "settings.properties"; + + /** + * The files the built-in engine and the graph write beside the settings. + */ + private static final Set STORAGE_FILES = new HashSet<>(Arrays.asList(AbstractLogStorage.SNAPSHOT_FILE, + AbstractLogStorage.LOG_FILE, AbstractLogStorage.VERSION_FILE, IndexDefinitions.INDEX_FILE)); + + /** + * The settings fixed at creation, each with the value TinkerGraph uses when it is not configured. + */ + private static final Map FIXED; + + static { + final Map fixed = new LinkedHashMap<>(); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, null); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.ANY.name()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER, TinkerGraph.DefaultIdManager.ANY.name()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, TinkerGraph.DefaultIdManager.ANY.name()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, + VertexProperty.Cardinality.single.name()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY, LabelCardinality.ONE.name()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, Boolean.FALSE.toString()); + fixed.put(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, Boolean.FALSE.toString()); + FIXED = Collections.unmodifiableMap(fixed); + } + + private StorageSettings() { + } + + /** + * Reconcile {@code config} with the settings recorded in {@code directory}, returning the configuration the graph + * should use. The supplied configuration is not modified. A new store records the effective settings now. An + * existing one fills in each unset fixed setting from the record and fails on any that differ. A recovery open + * changes nothing on disk and reads with the recorded settings, warning where the configuration disagrees. + * + * @param config the configuration the graph was opened with + * @param directory the storage directory, already locked by the opening graph + * @return a copy of {@code config} with the store's fixed settings applied + */ + public static Configuration reconcile(final Configuration config, final File directory) { + final boolean recovering = config.getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, false); + final Configuration effective = new BaseConfiguration(); + ConfigurationUtils.copy(config, effective); + + final File file = new File(directory, SETTINGS_FILE); + if (!file.isFile()) { + if (isNewStore(directory)) { + if (!recovering) + write(file, fixedSettingsOf(effective)); + } else if (!recovering) { + if (!holdsStorageFiles(directory)) + throw new IllegalStateException(String.format( + "Storage directory %s is not empty and is not a store, so no new store is created in " + + "it; use an empty or new directory", directory)); + throw new IllegalStateException(String.format( + "Storage at %s already holds storage files but no %s recording the settings it was created " + + "with; open it with %s to read it with the configured settings", + directory, SETTINGS_FILE, TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } else { + logger.warn("Storage at {} has no {}; reading it with the configured settings", + directory, SETTINGS_FILE); + } + return effective; + } + + final Map recorded = read(file); + for (final Map.Entry setting : FIXED.entrySet()) { + final String key = setting.getKey(); + final String recordedValue = recorded.get(key); + if (null == recordedValue || !isValid(key, recordedValue)) { + // every store records each of these when it is created, and only values it accepted, so a missing or + // unusable one means the file is damaged; defaulting it could read the store with the wrong settings + // and lose data at the next compaction + final String problem = null == recordedValue ? + String.format("Storage settings %s do not record %s, which every store records when it is " + + "created, so the file is damaged", file, key) : + String.format("Storage settings %s record %s=%s, which no store is created with, so the file " + + "is damaged", file, key, recordedValue); + if (!recovering) + throw new IllegalStateException(problem + String.format("; open the store with %s to read it " + + "with the configured settings", TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + logger.warn("{}; reading the store with the configured setting", problem); + continue; + } + if (effective.containsKey(key)) { + final String given = effective.getString(key); + if (normalize(key, given).equals(recordedValue)) + continue; + if (!recovering) + throw new IllegalStateException(String.format( + "Storage at %s was created with %s=%s and cannot be opened with %s. To change it, export " + + "the graph with g.io() and import it into a new storage directory", + directory, key, recordedValue, given)); + logger.warn("Storage at {} was created with {}={}; reading it with that rather than the configured {}", + directory, key, recordedValue, given); + } + effective.setProperty(key, recordedValue); + } + return effective; + } + + /** + * Remove the settings file a failed first open recorded, and the version marker if the engine had written one, when + * the directory holds nothing else of a store, so that the next open is a first open again and can record the + * settings it is given. Call only for an open that found no settings file when it began, which means the store + * did not exist before it and the marker, too, is that open's. + * + * @param directory the storage directory + */ + public static void removeIfOnlySettings(final File directory) { + final List removable = Arrays.asList(SETTINGS_FILE, AbstractLogStorage.VERSION_FILE); + final String[] names = directory.list(); + if (null == names || !Arrays.stream(names).allMatch(name -> name.equals(DirectoryLock.LOCK_FILE) || + removable.contains(name) || (name.endsWith(".tmp") && + removable.contains(name.substring(0, name.length() - 4))))) + return; + for (final String name : removable) { + StorageFiles.deleteQuietly(new File(directory, name)); + StorageFiles.deleteQuietly(new File(directory, name + ".tmp")); + } + } + + /** + * Whether {@code directory} holds any file an engine writes, as opposed to only files that belong to something + * else. + */ + private static boolean holdsStorageFiles(final File directory) { + final String[] names = directory.list(); + return null != names && Arrays.stream(names).anyMatch(name -> STORAGE_FILES.contains(name) || + (name.endsWith(".tmp") && STORAGE_FILES.contains(name.substring(0, name.length() - 4)))); + } + + private static boolean isNewStore(final File directory) { + final String[] names = directory.list(); + return null == names || Arrays.stream(names).allMatch(name -> + name.equals(DirectoryLock.LOCK_FILE) || name.equals(SETTINGS_FILE + ".tmp")); + } + + private static Map fixedSettingsOf(final Configuration config) { + final Map settings = new LinkedHashMap<>(); + for (final Map.Entry setting : FIXED.entrySet()) { + final String value = config.containsKey(setting.getKey()) ? + normalize(setting.getKey(), config.getString(setting.getKey())) : setting.getValue(); + if (value != null) + settings.put(setting.getKey(), value); + } + return settings; + } + + /** + * Whether {@code value} is one a store could have recorded for the fixed setting {@code key}: a built-in name, or + * for the id managers and the engine, the name of a class in a package. Damage that leaves a well-formed class name + * still fails the open, though only when the graph cannot load the class. + */ + private static boolean isValid(final String key, final String value) { + if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE)) + return Arrays.stream(DefaultStorage.values()).anyMatch(e -> e.name().toLowerCase(Locale.ROOT).equals(value)) || + isClassName(value); + if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER) || + key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_EDGE_ID_MANAGER) || + key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER)) + return Arrays.stream(TinkerGraph.DefaultIdManager.values()).anyMatch(m -> m.name().equals(value)) || + isClassName(value); + if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY)) + return Arrays.stream(VertexProperty.Cardinality.values()).anyMatch(c -> c.name().equals(value)); + if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY)) + return Arrays.stream(LabelCardinality.values()).anyMatch(c -> c.name().equals(value)); + try { + PropertyConverter.toBoolean(value); + return true; + } catch (ConversionException ex) { + return false; + } + } + + private static boolean isClassName(final String name) { + final String[] parts = name.split("\\.", -1); + return parts.length > 1 && Arrays.stream(parts).allMatch(part -> !part.isEmpty() && + Character.isJavaIdentifierStart(part.codePointAt(0)) && + part.codePoints().allMatch(Character::isJavaIdentifierPart)); + } + + /** + * Put a configured value in the form it is recorded in, so that spellings TinkerGraph treats alike compare equal. + */ + private static String normalize(final String key, final String value) { + final String trimmed = value.trim(); + if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE)) { + for (final DefaultStorage engine : DefaultStorage.values()) { + if (engine.name().equalsIgnoreCase(trimmed)) + return engine.name().toLowerCase(Locale.ROOT); + } + } else if (key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES) || + key.equals(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS)) { + // the spellings Configuration.getBoolean reads as true or false (yes/no, on/off, in any case); anything + // else is left as given, so it differs from the record and fails the open rather than being guessed at + try { + return PropertyConverter.toBoolean(trimmed).toString(); + } catch (ConversionException ex) { + return trimmed; + } + } + return trimmed; + } + + private static Map read(final File file) { + final Properties properties = new Properties(); + // read back in the encoding write() stores it in, since Properties.load(InputStream) assumes ISO-8859-1 + try (final Reader in = new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)) { + properties.load(in); + } catch (IOException ex) { + throw new UncheckedIOException(String.format("Could not read storage settings %s", file), ex); + } catch (IllegalArgumentException ex) { + // Properties.load reports a malformed escape this way, without naming the file + throw new IllegalStateException(String.format("Storage settings %s are malformed: %s", file, + ex.getMessage()), ex); + } + final Map settings = new LinkedHashMap<>(); + for (final String key : properties.stringPropertyNames()) + settings.put(key, properties.getProperty(key)); + return settings; + } + + private static void write(final File file, final Map settings) { + final Properties properties = new Properties(); + properties.putAll(settings); + final File tmp = new File(file.getParentFile(), SETTINGS_FILE + ".tmp"); + try { + try (final FileOutputStream fos = new FileOutputStream(tmp); + final Writer out = new OutputStreamWriter(fos, StandardCharsets.UTF_8)) { + properties.store(out, "Settings this TinkerStorageGraph store was created with. They cannot be " + + "changed; export and import the graph to use different ones."); + out.flush(); + fos.getFD().sync(); + } + StorageFiles.atomicMove(tmp, file); + StorageFiles.syncDirectory(file.getParentFile()); + } catch (IOException ex) { + StorageFiles.deleteQuietly(tmp); + throw new UncheckedIOException(String.format("Could not record storage settings %s", file), ex); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java new file mode 100644 index 00000000000..00615c41bfb --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncMode.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.util.Locale; + +/** + * The durability mode a {@link TinkerStorage} engine applies when flushing a committed transaction to disk, selected + * with the {@code gremlin.tinkergraph.storage.sync} configuration key. Each value is a complete, mutually-exclusive + * choice; they are ordered from strongest to weakest durability. + *

+ * The name of a mode describes when data is made durable, not what action is taken: {@link #COMMIT} + * performs an {@code fsync} so acknowledged commits survive an OS crash or power loss, whereas {@link #OS} only pushes + * bytes into the operating system's page cache, so commits survive a crash of the JVM process but not of the OS. + */ +public enum SyncMode { + + /** + * {@code fsync} on every commit. An acknowledged commit is durable against process crash, OS crash, and power + * loss. This is the default and the mode that honors the "each committed transaction is durably written to disk" + * contract. + */ + COMMIT, + + /** + * Flush to the operating system on every commit, but do not {@code fsync}. An acknowledged commit survives a crash + * of the JVM process but may be lost on an OS crash or power loss, since the data can still be sitting in the OS + * page cache. Faster than {@link #COMMIT}; use only when that weaker guarantee is acceptable. + */ + OS; + + // TODO: add an INTERVAL mode (group commit) — a peer value on this same key, encoded as "interval:", that + // fsyncs at most once per window rather than once per commit, bounding the crash-loss window by time while + // amortizing fsync cost across commits. It implies fsync (a batched COMMIT), so it slots in as a third + // mutually-exclusive mode without changing the meaning of COMMIT or OS. + // + // This is coupled to the commit lock that serializes concurrent commits (see the persist/flush step in + // TinkerTransaction.doCommit): COMMIT holds it across the fsync, so every commit serializes on disk-sync latency. + // INTERVAL is the fix: hold the lock only for the buffer append (fast) and fsync one batch for many transactions, + // the performance pass that makes serialized commits cheap. + + /** + * Resolve a configuration value to a {@link SyncMode}, matched case-insensitively, defaulting to {@link #COMMIT} + * when unset. + * + * @param configValue the raw configuration value, or {@code null} when unset + * @return the resolved mode + * @throws IllegalArgumentException if the value does not name a known mode + */ + public static SyncMode fromConfigValue(final String configValue) { + if (null == configValue) + return COMMIT; + try { + return SyncMode.valueOf(configValue.trim().toUpperCase(Locale.ROOT)); + } catch (IllegalArgumentException iae) { + throw new IllegalArgumentException(String.format( + "Unknown storage sync mode '%s'; valid values are 'commit' and 'os'", configValue), iae); + } + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java new file mode 100644 index 00000000000..2c0fa34c030 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBuffer.java @@ -0,0 +1,371 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.io.Buffer; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; + +/** + * A self-contained, heap {@code byte[]}-backed implementation of the gremlin-core {@link Buffer} abstraction, used to + * drive {@code GraphBinaryWriter}/{@code GraphBinaryReader} without depending on gremlin-util's Netty-backed buffer. + * All multi-byte values are big-endian, matching the wire order the Netty implementation uses. The backing array grows + * as needed on write. This class is not thread-safe; a buffer is used by a single thread for a single + * serialize/deserialize. + */ +public final class TinkerByteBuffer implements Buffer { + + private static final int DEFAULT_CAPACITY = 256; + + /** + * The most bytes one buffer, and so one storage frame, can hold: the largest array size JVMs reliably allocate. + */ + static final int MAX_CAPACITY = Integer.MAX_VALUE - 8; + + private byte[] array; + private int readerIndex = 0; + private int writerIndex = 0; + private int markedWriterIndex = 0; + private int referenceCount = 1; + + public TinkerByteBuffer() { + this(DEFAULT_CAPACITY); + } + + public TinkerByteBuffer(final int initialCapacity) { + this.array = new byte[Math.max(initialCapacity, 1)]; + } + + /** + * Wraps an existing array for reading. The writer index is positioned at the end of the supplied data. + */ + public TinkerByteBuffer(final byte[] data) { + this.array = data; + this.writerIndex = data.length; + } + + /** + * Returns a copy of the readable-region bytes (from reader index to writer index). Does not change indexes. + */ + public byte[] toReadableArray() { + return Arrays.copyOfRange(array, readerIndex, writerIndex); + } + + /** + * Returns a copy of all written bytes (index 0 to writer index). Does not change indexes. + */ + public byte[] toWrittenArray() { + return Arrays.copyOfRange(array, 0, writerIndex); + } + + private void ensureWritable(final int additional) { + // computed in a long so a buffer nearing the int range neither wraps to a negative size nor doubles forever + final long required = (long) writerIndex + additional; + if (required <= array.length) + return; + if (required > MAX_CAPACITY) + throw new IllegalStateException(String.format("Storage frame would be %d bytes, more than the %d bytes " + + "one frame can hold: a single transaction, or a single element when compacting, cannot encode " + + "to more than that, so commit the changes in smaller transactions", required, MAX_CAPACITY)); + long newCapacity = Math.max(array.length, 1); + while (newCapacity < required) + newCapacity <<= 1; + array = Arrays.copyOf(array, (int) Math.min(newCapacity, MAX_CAPACITY)); + } + + private void checkReadable(final int length) { + if (readerIndex + length > writerIndex) + throw new IndexOutOfBoundsException(String.format( + "Not enough readable bytes: need %d at index %d but writer index is %d", length, readerIndex, writerIndex)); + } + + @Override + public int readableBytes() { + return writerIndex - readerIndex; + } + + @Override + public int readerIndex() { + return readerIndex; + } + + @Override + public Buffer readerIndex(final int readerIndex) { + if (readerIndex < 0 || readerIndex > writerIndex) + throw new IndexOutOfBoundsException("readerIndex: " + readerIndex); + this.readerIndex = readerIndex; + return this; + } + + @Override + public int writerIndex() { + return writerIndex; + } + + @Override + public Buffer writerIndex(final int writerIndex) { + if (writerIndex < readerIndex) + throw new IndexOutOfBoundsException("writerIndex: " + writerIndex); + ensureWritable(writerIndex - this.writerIndex); + this.writerIndex = writerIndex; + return this; + } + + @Override + public Buffer markWriterIndex() { + this.markedWriterIndex = writerIndex; + return this; + } + + @Override + public Buffer resetWriterIndex() { + this.writerIndex = markedWriterIndex; + return this; + } + + @Override + public int capacity() { + return array.length; + } + + @Override + public boolean isDirect() { + return false; + } + + @Override + public boolean readBoolean() { + return readByte() != 0; + } + + @Override + public byte readByte() { + checkReadable(1); + return array[readerIndex++]; + } + + @Override + public short readShort() { + checkReadable(2); + return (short) (((array[readerIndex++] & 0xFF) << 8) | (array[readerIndex++] & 0xFF)); + } + + @Override + public int readInt() { + checkReadable(4); + return ((array[readerIndex++] & 0xFF) << 24) | + ((array[readerIndex++] & 0xFF) << 16) | + ((array[readerIndex++] & 0xFF) << 8) | + (array[readerIndex++] & 0xFF); + } + + @Override + public long readLong() { + checkReadable(8); + long value = 0; + for (int i = 0; i < 8; i++) + value = (value << 8) | (array[readerIndex++] & 0xFF); + return value; + } + + @Override + public float readFloat() { + return Float.intBitsToFloat(readInt()); + } + + @Override + public double readDouble() { + return Double.longBitsToDouble(readLong()); + } + + @Override + public Buffer readBytes(final byte[] destination) { + return readBytes(destination, 0, destination.length); + } + + @Override + public Buffer readBytes(final byte[] destination, final int dstIndex, final int length) { + checkReadable(length); + System.arraycopy(array, readerIndex, destination, dstIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final ByteBuffer dst) { + final int length = dst.remaining(); + checkReadable(length); + dst.put(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer readBytes(final OutputStream out, final int length) throws IOException { + checkReadable(length); + out.write(array, readerIndex, length); + readerIndex += length; + return this; + } + + @Override + public Buffer writeBoolean(final boolean value) { + return writeByte(value ? 1 : 0); + } + + @Override + public Buffer writeByte(final int value) { + ensureWritable(1); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeShort(final int value) { + ensureWritable(2); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeInt(final int value) { + ensureWritable(4); + array[writerIndex++] = (byte) (value >>> 24); + array[writerIndex++] = (byte) (value >>> 16); + array[writerIndex++] = (byte) (value >>> 8); + array[writerIndex++] = (byte) value; + return this; + } + + @Override + public Buffer writeLong(final long value) { + ensureWritable(8); + for (int i = 56; i >= 0; i -= 8) + array[writerIndex++] = (byte) (value >>> i); + return this; + } + + @Override + public Buffer writeFloat(final float value) { + return writeInt(Float.floatToIntBits(value)); + } + + @Override + public Buffer writeDouble(final double value) { + return writeLong(Double.doubleToLongBits(value)); + } + + @Override + public Buffer writeBytes(final byte[] src) { + return writeBytes(src, 0, src.length); + } + + @Override + public Buffer writeBytes(final ByteBuffer src) { + final int length = src.remaining(); + ensureWritable(length); + src.get(array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public Buffer writeBytes(final byte[] src, final int srcIndex, final int length) { + ensureWritable(length); + System.arraycopy(src, srcIndex, array, writerIndex, length); + writerIndex += length; + return this; + } + + @Override + public boolean release() { + return --referenceCount <= 0; + } + + @Override + public Buffer retain() { + referenceCount++; + return this; + } + + @Override + public int referenceCount() { + return referenceCount; + } + + @Override + public int nioBufferCount() { + return 1; + } + + @Override + public ByteBuffer[] nioBuffers() { + return new ByteBuffer[] { nioBuffer() }; + } + + @Override + public ByteBuffer[] nioBuffers(final int index, final int length) { + return new ByteBuffer[] { nioBuffer(index, length) }; + } + + @Override + public ByteBuffer nioBuffer() { + return nioBuffer(readerIndex, readableBytes()); + } + + @Override + public ByteBuffer nioBuffer(final int index, final int length) { + return ByteBuffer.wrap(Arrays.copyOfRange(array, index, index + length)); + } + + @Override + public Buffer getBytes(final int index, final byte[] dst) { + System.arraycopy(array, index, dst, 0, dst.length); + return this; + } + + /** + * The byte at {@code index}, without moving the reader index. + */ + byte getByte(final int index) { + checkIndex(index, 1); + return array[index]; + } + + /** + * The big-endian int at {@code index}, without moving the reader index. + */ + int getInt(final int index) { + checkIndex(index, 4); + return ((array[index] & 0xFF) << 24) | + ((array[index + 1] & 0xFF) << 16) | + ((array[index + 2] & 0xFF) << 8) | + (array[index + 3] & 0xFF); + } + + private void checkIndex(final int index, final int length) { + if (index < 0 || (long) index + length > writerIndex) + throw new IndexOutOfBoundsException(String.format( + "Not enough written bytes: need %d at index %d but writer index is %d", length, index, writerIndex)); + } +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java new file mode 100644 index 00000000000..56204a61afe --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorage.java @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.util.Collection; + +/** + * A pluggable durable storage engine for a transactional {@code TinkerStorageGraph}. Implementations persist the + * changeset of each committed transaction to disk and rebuild the in-memory graph on open. The graph remains the + * authoritative in-memory copy (write-through); the engine is a durable mirror. + *

+ * The lifecycle is: + *

    + *
  1. {@link #open(AbstractTinkerGraph, Configuration)} — resolve the backing location and ready the store.
    2. + *
  2. {@link #replay(AbstractTinkerGraph)} — rebuild in-memory state from what was previously persisted.
    3. + *
  3. {@link #persist(long, Collection, Collection)} then {@link #flush()} — called on each transaction commit, + * before the indexes and in-memory state are committed, so a failure aborts the commit and leaves the graph as it + * was. Whether a commit that failed in the engine reached disk is up to the engine; one that cannot say should + * refuse further commits until the graph is reopened.
    4. + *
  4. {@link #compact(AbstractTinkerGraph)} — optionally fold accumulated changes into a compact snapshot.
    5. + *
  5. {@link #close()} — release resources.
    6. + *
+ * A new engine can be selected by name (see {@link DefaultStorage}) or by fully-qualified class name via the + * {@code gremlin.tinkergraph.storage} configuration key. Implementations must provide a public no-argument constructor. + */ +public interface TinkerStorage extends AutoCloseable { + + /** + * Prepare the storage engine for use, resolving its backing location from the supplied configuration. Called once + * during graph construction before {@link #replay(AbstractTinkerGraph)}. + * + * @param graph the graph that owns this engine + * @param config the graph configuration, including {@code gremlin.tinkergraph.storage.directory} + */ + void open(AbstractTinkerGraph graph, Configuration config); + + /** + * Rebuild the in-memory state of the graph from previously persisted data. Called once during graph construction. + * Implementations should re-apply persisted elements through the graph's own mutation API; the graph sets its + * {@code loading} guard for the duration so this does not re-persist. + * + * @param graph the graph to populate + */ + void replay(AbstractTinkerGraph graph); + + /** + * Durably record the changeset of a committing transaction. Called from within the transaction commit, while the + * changed elements are locked, before the in-memory commit is applied. Each mutation is either a put (added or + * modified element) or a delete (see {@link TinkerStorageMutation}). + * + * @param txVersion the version number of the committing transaction + * @param changedVertices the vertex mutations in this transaction + * @param changedEdges the edge mutations in this transaction + */ + void persist(long txVersion, + Collection> changedVertices, + Collection> changedEdges); + + /** + * Force any buffered writes to durable storage. Called after {@link #persist(long, Collection, Collection)} as the + * commit's durability point. + */ + void flush(); + + /** + * Fold accumulated changes into a compact representation of the current committed state, reclaiming space. Safe to + * call at any time; typically invoked on {@link #close()} or on a size/commit-count threshold. + * + * @param graph the graph whose current committed state should be snapshotted + */ + void compact(AbstractTinkerGraph graph); + + /** + * Compact if the engine's accumulated data has grown past its own threshold, otherwise do nothing. Called on the + * commit path after {@link #flush()} (while the graph's commit lock is held) so a long-running graph that is never + * explicitly closed does not grow its backing storage without bound. The default is a no-op, leaving compaction + * entirely under the control of {@link #compact(AbstractTinkerGraph)} and {@link #close()}. + * + * @param graph the graph whose current committed state would be snapshotted + */ + default void maybeCompact(final AbstractTinkerGraph graph) { + // no-op by default; engines that accumulate an on-disk log override this to bound its growth + } + + /** + * Replace everything the engine holds with the graph's current committed state, which the caller has just + * emptied, so that the store holds nothing and a reopen finds an empty graph. Called by + * {@code TinkerGraph.clear()} under the graph's commit lock, and never on a read-only engine. If the store cannot + * be emptied, the engine must refuse further commits until the graph is reopened: appending them to the old data + * would bring the cleared elements back on reopen, mixed with the new ones. The graph fail-stops the engine + * through {@link #failStop(Throwable)} when this throws. Rewriting the store from the emptied graph, as a + * compaction does, is one way to empty it, but it must then run even when nothing was written since the last + * compaction, since the store still holds the data the graph no longer has. + * + * @param graph the graph, already emptied, whose committed state the store should now hold + */ + void clear(final AbstractTinkerGraph graph); + + /** + * Whether this engine accepts no writes at all, such as when a damaged store was opened for recovery. The graph + * then also leaves its other files in the storage directory, such as index definitions, unchanged. + */ + default boolean isReadOnly() { + return false; + } + + /** + * Whether this engine has stopped accepting writes after a storage failure, such as a failed log write or the + * store being removed while the graph was open. It stays so until the graph is reopened. The graph's committed + * data remains readable in memory for export, so operations that would discard or rewrite it are refused. + */ + default boolean isFailed() { + return false; + } + + /** + * Stop accepting writes, as after a storage failure, until the graph is reopened. The graph calls this when its + * memory no longer matches the store, so that nothing written from that memory, by a commit or a compaction, may + * reach the store: when a transaction's changes were persisted and flushed but could not then be applied in + * memory, so the transaction is durable and returns on the next open, and when {@link #clear} fails after the + * graph was emptied in memory. {@link #isFailed()} reports {@code true} afterwards. + * + * @param cause why the graph's memory no longer matches the store + */ + void failStop(Throwable cause); + + /** + * {@inheritDoc} + *

+ * Flush and release resources. Does not delete persisted data. + */ + @Override + void close(); +} diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java new file mode 100644 index 00000000000..726c6862fd1 --- /dev/null +++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerStorageMutation.java @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.tinkerpop.gremlin.structure.Element; + +/** + * A single element change within a committing transaction, handed to a {@link TinkerStorage} engine to persist. This + * is the stable, public view of a change that decouples storage engines from TinkerGraph's internal transactional + * containers. A mutation is either a put (the element was added or modified, {@link #element()} is non-null) + * or a delete ({@link #element()} is {@code null} and {@link #isDeleted()} is {@code true}). + * + * @param the element type ({@code TinkerVertex} or {@code TinkerEdge}) + */ +public final class TinkerStorageMutation { + + private final Object id; + private final T element; + + /** + * Create a mutation for the given element id. + * + * @param id the element identifier (never {@code null}) + * @param element the committed element for a put, or {@code null} for a delete + */ + public TinkerStorageMutation(final Object id, final T element) { + this.id = id; + this.element = element; + } + + /** + * The identifier of the changed element. + */ + public Object id() { + return id; + } + + /** + * The committed element to persist, or {@code null} when this mutation is a deletion. + */ + public T element() { + return element; + } + + /** + * Returns {@code true} when this mutation deletes the element rather than adding or modifying it. + */ + public boolean isDeleted() { + return element == null; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java index fe17fd7d71e..170b43f8f55 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphProvider.java @@ -21,7 +21,6 @@ import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.AbstractGraphProvider; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.GraphTest; import org.apache.tinkerpop.gremlin.structure.VertexProperty; @@ -39,7 +38,6 @@ import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertexProperty; -import java.io.File; import java.util.HashMap; import java.util.HashSet; import java.util.Map; @@ -73,10 +71,6 @@ public Map getBaseConfiguration(final String graphName, final Cl put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } @@ -84,13 +78,6 @@ public Map getBaseConfiguration(final String graphName, final Cl public void clear(final Graph graph, final Configuration configuration) throws Exception { if (graph != null) graph.close(); - - // in the even the graph is persisted we need to clean up - final String graphLocation = null != configuration ? configuration.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; - if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); - } } @Override @@ -98,13 +85,6 @@ public Set getImplementations() { return IMPLEMENTATION; } - /** - * Determines if a test requires TinkerGraph persistence to be configured with graph location and format. - */ - protected static boolean requiresPersistence(final Class test, final String testMethodName) { - return test == GraphTest.class && testMethodName.equals("shouldPersistDataOnClose"); - } - /** * Determines if a test requires a different cardinality as the default or not. */ diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java index 8a127e8330c..ef37147838f 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerMemoryGraphUUIDProvider.java @@ -20,13 +20,11 @@ package org.apache.tinkerpop.gremlin.tinkergraph; import org.apache.tinkerpop.gremlin.LoadGraphWith; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.structure.Graph; import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerMemoryGraph; -import java.io.File; import java.util.HashMap; import java.util.Map; @@ -47,10 +45,6 @@ public Map getBaseConfiguration(final String graphName, final Cl put(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_PROPERTY_ID_MANAGER, idMaker); if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); - if (requiresPersistence(test, testMethodName)) { - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); - } }}; } } diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java index 6f777a630c2..3d235b284b6 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerStorageGraphProvider.java @@ -70,8 +70,9 @@ public Map getBaseConfiguration(final String graphName, final Cl if (requiresListCardinalityAsDefault(loadGraphWith, test, testMethodName)) put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); if (requiresPersistence(test, testMethodName)) { - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION,TestHelper.makeTestDataFile(test, "temp", testMethodName + ".kryo")); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + put(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, + TestHelper.makeTestDataDirectory(test, "temp", testMethodName)); } }}; } @@ -81,11 +82,10 @@ public void clear(final Graph graph, final Configuration configuration) throws E if (graph != null) graph.close(); - // in the even the graph is persisted we need to clean up - final String graphLocation = null != configuration ? configuration.getString(TinkerStorageGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null) : null; - if (graphLocation != null) { - final File f = new File(graphLocation); - f.delete(); + // in the event the graph is persisted we need to clean up the storage directory + final String storageDirectory = null != configuration ? configuration.getString(TinkerStorageGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, null) : null; + if (storageDirectory != null) { + deleteDirectory(new File(storageDirectory)); } } diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java index 536f5b5ee81..5a9fda456f4 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerMemoryGraphTest.java @@ -18,10 +18,12 @@ */ package org.apache.tinkerpop.gremlin.tinkergraph.structure; +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; import org.apache.commons.configuration2.BaseConfiguration; import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.GraphHelper; -import org.apache.tinkerpop.gremlin.TestHelper; import org.apache.tinkerpop.gremlin.process.computer.Computer; import org.apache.tinkerpop.gremlin.process.traversal.P; import org.apache.tinkerpop.gremlin.process.traversal.Traversal; @@ -39,7 +41,6 @@ import org.apache.tinkerpop.gremlin.structure.T; import org.apache.tinkerpop.gremlin.structure.Vertex; import org.apache.tinkerpop.gremlin.structure.VertexProperty; -import org.apache.tinkerpop.gremlin.structure.io.Io; import org.apache.tinkerpop.gremlin.structure.io.GraphReader; import org.apache.tinkerpop.gremlin.structure.io.GraphWriter; import org.apache.tinkerpop.gremlin.structure.io.IoCore; @@ -61,13 +62,11 @@ import org.apache.tinkerpop.shaded.kryo.io.Input; import org.apache.tinkerpop.shaded.kryo.io.Output; import org.junit.Test; +import org.slf4j.LoggerFactory; import java.awt.Color; -import java.io.BufferedOutputStream; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.FileOutputStream; import java.io.InputStream; import java.util.ArrayList; import java.util.Arrays; @@ -80,7 +79,6 @@ import java.util.Set; import java.util.UUID; import java.util.concurrent.TimeUnit; -import java.util.function.Consumer; import java.util.function.Supplier; import static org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource.traversal; @@ -95,7 +93,6 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import static org.junit.Assume.assumeThat; -import static org.mockito.Mockito.mock; /** * @author Marko A. Rodriguez (http://markorodriguez.com) @@ -103,6 +100,33 @@ */ public class TinkerMemoryGraphTest { + @Test + public void shouldWarnWhenConfiguredWithStorageSettings() { + final ch.qos.logback.classic.Logger logger = + (ch.qos.logback.classic.Logger) LoggerFactory.getLogger(TinkerMemoryGraph.class); + final ListAppender appender = new ListAppender<>(); + appender.start(); + logger.addAppender(appender); + try { + TinkerGraph.open().close(); + assertTrue("no storage settings, no warning", appender.list.isEmpty()); + + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, "/data/mygraph"); + TinkerGraph.open(conf).close(); + assertEquals(1, appender.list.size()); + final ILoggingEvent warning = appender.list.get(0); + assertEquals(Level.WARN, warning.getLevel()); + assertTrue(warning.getFormattedMessage(), + warning.getFormattedMessage().contains(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + assertTrue(warning.getFormattedMessage(), + warning.getFormattedMessage().contains(TinkerStorageGraph.class.getName())); + } finally { + logger.detachAppender(appender); + } + } + @Test public void shouldManageIndices() { final TinkerGraph g = TinkerGraph.open(); @@ -390,13 +414,6 @@ public void shouldSerializeTinkerGraphToGraphSONWithTypes() throws Exception { } } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphLocationIfFormatIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - TinkerGraph.open(conf); - } - @Test(expected = IllegalStateException.class) public void shouldNotModifyAVertexThatWasRemoved() { final TinkerGraph graph = TinkerGraph.open(); @@ -431,137 +448,6 @@ public void shouldNotReadValueOfPropertyOnVertexThatWasRemoved() { v.value("name"); } - @Test(expected = IllegalStateException.class) - public void shouldRequireGraphFormatIfLocationIsSet() { - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, TestHelper.makeTestDataDirectory(TinkerMemoryGraphTest.class)); - TinkerGraph.open(conf); - } - - @Test - public void shouldPersistToGraphML() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphML.xml"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphml"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, true); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGraphSON() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGraphSON.json"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "graphson"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryo() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryo.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToGryoAndHandleMultiProperties() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToGryoMulti.kryo"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateTheCrew(graph); - graph.close(); - - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.toString()); - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertCrewGraph(reloadedGraph, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistWithRelativePath() { - final String graphLocation = TestHelper.convertToRelative(TinkerMemoryGraphTest.class, - TestHelper.makeTestDataPath(TinkerMemoryGraphTest.class)) - + "shouldPersistToGryoRelative.kryo"; - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo"); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - graph.close(); - - final TinkerGraph reloadedGraph = TinkerGraph.open(conf); - IoTest.assertModernGraph(reloadedGraph, true, false); - reloadedGraph.close(); - } - - @Test - public void shouldPersistToAnyGraphFormat() { - final String graphLocation = TestHelper.makeTestDataFile(TinkerMemoryGraphTest.class, "shouldPersistToAnyGraphFormat.dat"); - final File f = new File(graphLocation); - if (f.exists() && f.isFile()) f.delete(); - - final Configuration conf = new BaseConfiguration(); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, TestIoBuilder.class.getName()); - conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, graphLocation); - final TinkerGraph graph = TinkerGraph.open(conf); - TinkerFactory.generateModern(graph); - - //Test write graph - graph.close(); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - - try (BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(f))){ - os.write("dummy string".getBytes()); - } catch (Exception e) { - e.printStackTrace(); - } - - //Test read graph - final TinkerGraph readGraph = TinkerGraph.open(conf); - assertEquals(TestIoBuilder.calledOnMapper, 1); - assertEquals(TestIoBuilder.calledGraph, 1); - assertEquals(TestIoBuilder.calledCreate, 1); - } - @Test public void shouldSerializeWithColorClassResolverToTinkerGraph() throws Exception { final Map colors = new HashMap<>(); @@ -1060,41 +946,6 @@ public Registration getRegistration(final Class clazz) { } } - public static class TestIoBuilder implements Io.Builder { - - static int calledGraph, calledCreate, calledOnMapper; - - public TestIoBuilder(){ - //Looks awkward to reset static vars inside a constructor, but makes sense from testing perspective - calledGraph = 0; - calledCreate = 0; - calledOnMapper = 0; - } - - @Override - public Io.Builder onMapper(final Consumer onMapper) { - calledOnMapper++; - return this; - } - - @Override - public Io.Builder graph(final Graph graph) { - calledGraph++; - return this; - } - - @Override - public Io create() { - calledCreate++; - return mock(Io.class); - } - - @Override - public boolean requiresVersion(final Object version) { - return false; - } - } - @Test public void shouldGroupMultiPropertyValuesUnderSingleKeyForPropertyMap() throws Exception { try (final TinkerGraph graph = TinkerGraph.open()) { diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java index af5e79124a1..fa5ea114bec 100644 --- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java @@ -18,7 +18,13 @@ */ package org.apache.tinkerpop.gremlin.tinkergraph.structure; +import java.io.File; +import java.io.UncheckedIOException; +import java.lang.reflect.Field; +import java.nio.file.Files; import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; import java.util.HashSet; import java.util.Iterator; import java.util.List; @@ -26,18 +32,29 @@ import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import java.util.stream.Collectors; import java.util.stream.IntStream; import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource; +import org.apache.tinkerpop.gremlin.structure.Direction; import org.apache.tinkerpop.gremlin.structure.Edge; import org.apache.tinkerpop.gremlin.structure.Property; import org.apache.tinkerpop.gremlin.structure.T; import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.service.Service; +import org.apache.tinkerpop.gremlin.tinkergraph.services.TinkerServiceRegistry; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorageMutation; import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; import org.apache.tinkerpop.gremlin.structure.util.TransactionException; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -46,9 +63,428 @@ import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.spy; public class TinkerStorageGraphTest { + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + @Test + public void shouldFailACommitThatWaitedOnTheStorageLockWhileTheGraphWasCleared() throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // hold the commit lock so the other thread's commit passes its conflict checks and then waits on it + final TinkerStorageGraph g = graph; + final AtomicReference otherCommit = new AtomicReference<>(); + graph.storageCommitLock.lock(); + final Thread other = new Thread(() -> { + g.vertices(1).next().property("name", "late"); + try { + g.tx().commit(); + } catch (Throwable t) { + otherCommit.set(t); + } + }); + try { + other.start(); + // bounded, so a thread that fails before it reaches the lock fails the test instead of hanging the build + final long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (!graph.storageCommitLock.hasQueuedThreads() && other.isAlive() && System.nanoTime() < deadline) + Thread.sleep(1); + assertTrue("the other commit should be waiting on the commit lock: " + otherCommit.get(), + graph.storageCommitLock.hasQueuedThreads()); + graph.clear(); + } finally { + graph.storageCommitLock.unlock(); + } + other.join(10_000); + + assertTrue("the waiting commit should conflict: " + otherCommit.get(), + otherCommit.get() instanceof TransactionException); + graph.close(); + graph = TinkerStorageGraph.open(conf); + assertFalse(graph.vertices().hasNext()); + graph.close(); + } + + @Test + public void shouldKeepAVertexAddedAfterClearWhenAnOlderTransactionOnItsIdConflicts() throws Exception { + assertVertexAddedAfterClearSurvivesTheOlderTransaction(true); + } + + @Test + public void shouldKeepAVertexAddedAfterClearWhenAnOlderTransactionOnItsIdRollsBack() throws Exception { + assertVertexAddedAfterClearSurvivesTheOlderTransaction(false); + } + + /** + * A transaction on another thread changes vertex 1 and is still open when the graph is cleared and vertex 1 is added + * again and committed. Ending that older transaction must leave the new vertex 1 in place, in memory and on disk. + */ + private void assertVertexAddedAfterClearSurvivesTheOlderTransaction(final boolean commit) throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1, "name", "old"); + graph.tx().commit(); + + final TinkerStorageGraph g = graph; + final CountDownLatch changed = new CountDownLatch(1); + final CountDownLatch end = new CountDownLatch(1); + final AtomicReference older = new AtomicReference<>(); + final Thread other = new Thread(() -> { + try { + g.vertices(1).next().property("name", "older"); + changed.countDown(); + end.await(); + if (commit) + g.tx().commit(); + else + g.tx().rollback(); + } catch (Throwable t) { + older.set(t); + } + }); + other.start(); + assertTrue("the other transaction should change vertex 1", changed.await(10, TimeUnit.SECONDS)); + + graph.clear(); + graph.addVertex(T.id, 1, "name", "new"); + graph.tx().commit(); + end.countDown(); + other.join(10_000); + assertFalse("the other transaction should have ended", other.isAlive()); + if (commit) + assertTrue("the older commit should conflict: " + older.get(), older.get() instanceof TransactionException); + else + assertNull(older.get()); + + assertEquals("new", graph.vertices(1).next().value("name")); + graph.close(); + graph = TinkerStorageGraph.open(conf); + assertEquals("new", graph.vertices(1).next().value("name")); + graph.close(); + } + + @Test + public void shouldCloseTheServicesAndTheEngineWhenTheOpenFails() throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, ReplayFailingStorage.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_SERVICE, CloseRecordingServiceFactory.class.getName()); + CloseRecordingServiceFactory.closed.set(false); + ReplayFailingStorage.closed.set(false); + try { + TinkerStorageGraph.open(conf).close(); + fail("the replay failure should fail the open"); + } catch (IllegalStateException expected) { + assertEquals("replay failure", expected.getMessage()); + } + assertTrue("the configured service should be closed", CloseRecordingServiceFactory.closed.get()); + assertTrue("the storage engine should be closed", ReplayFailingStorage.closed.get()); + } + + /** + * A storage engine whose replay fails, recording whether it is closed. Static because it is instantiated by name. + */ + public static final class ReplayFailingStorage implements TinkerStorage { + static final AtomicBoolean closed = new AtomicBoolean(); + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { } + + @Override + public void replay(final AbstractTinkerGraph graph) { + throw new IllegalStateException("replay failure"); + } + + @Override + public void persist(final long txVersion, final Collection> changedVertices, + final Collection> changedEdges) { } + + @Override + public void flush() { } + + @Override + public void compact(final AbstractTinkerGraph graph) { } + + @Override + public void clear(final AbstractTinkerGraph graph) { } + + @Override + public void failStop(final Throwable cause) { } + + @Override + public void close() { + closed.set(true); + } + } + + /** + * A service factory that records whether it is closed. Static because it is instantiated by name. + */ + public static final class CloseRecordingServiceFactory extends TinkerServiceRegistry.TinkerServiceFactory { + static final AtomicBoolean closed = new AtomicBoolean(); + + public CloseRecordingServiceFactory(final AbstractTinkerGraph graph) { + super(graph, "close.recording"); + } + + @Override + public Set getSupportedTypes() { + return Collections.singleton(Service.Type.Start); + } + + @Override + public Service createService(final boolean isStart, final Map params) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + closed.set(true); + } + } + + @Test + public void shouldStopAcceptingCommitsWhenTheEngineFailsToClearWithoutStoppingItself() throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // an engine whose clear() throws without putting itself into the failed state + final TinkerStorage storage = spy(graph.storage); + doThrow(new IllegalStateException("clear failure")).when(storage).clear(any()); + graph.storage = storage; + try { + graph.clear(); + fail("the failure emptying the store should be reported"); + } catch (IllegalStateException expected) { + assertEquals("clear failure", expected.getMessage()); + } + assertTrue(graph.storage.isFailed()); + assertFalse(graph.vertices().hasNext()); + + graph.addVertex(T.id, 2); + try { + graph.tx().commit(); + fail("a commit after the failed clear should be refused"); + } catch (UncheckedIOException expected) { + // the storage stopped accepting commits + } + graph.close(); + + // the store was never emptied, so the cleared data returns, but nothing was appended to it + graph = TinkerStorageGraph.open(conf); + assertEquals(Collections.singleton(1), elementIds(graph, false)); + graph.close(); + } + + @Test + public void shouldAgreeInMemoryAndOnDiskAboutAVertexAddedWhileTheGraphIsCleared() throws Exception { + assertElementsAddedWhileClearingAgreeInMemoryAndOnDisk(false); + } + + @Test + public void shouldAgreeInMemoryAndOnDiskAboutAnEdgeAddedWhileTheGraphIsCleared() throws Exception { + assertElementsAddedWhileClearingAgreeInMemoryAndOnDisk(true); + } + + /** + * Another thread adds vertex 5 (and, with {@code withEdge}, vertex 6 and edge 100 between them) while clear() is + * emptying the element map, and then commits. Whether that commit conflicts or succeeds, memory, the store as a + * crash would leave it, and the store after a clean close must all show the same elements. + */ + private void assertElementsAddedWhileClearingAgreeInMemoryAndOnDisk(final boolean withEdge) throws Exception { + final File directory = tempFolder.newFolder(); + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, directory.getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + final Vertex v1 = graph.addVertex(T.id, 1); + v1.addEdge("knows", graph.addVertex(T.id, 2), T.id, 99); + graph.tx().commit(); + + // pause clear() inside its pass over the element map, after it has started emptying it + final PausingMap paused = new PausingMap<>(); + final Field field = TinkerStorageGraph.class.getDeclaredField(withEdge ? "edges" : "vertices"); + field.setAccessible(true); + paused.putAll((Map) field.get(graph)); + field.set(graph, paused); + + final TinkerStorageGraph g = graph; + final AtomicReference addCommit = new AtomicReference<>(); + final Thread other = new Thread(() -> { + try { + if (!paused.paused.await(10, TimeUnit.SECONDS)) + throw new AssertionError("clear() did not reach the element map"); + final Vertex v5 = g.addVertex(T.id, 5); + if (withEdge) v5.addEdge("knows", g.addVertex(T.id, 6), T.id, 100); + paused.resume.countDown(); + g.tx().commit(); + } catch (Throwable t) { + addCommit.set(t); + } finally { + paused.resume.countDown(); + } + }); + other.start(); + graph.clear(); + other.join(10_000); + assertFalse("the other thread should have finished", other.isAlive()); + final Throwable failure = addCommit.get(); + assertTrue("the commit should succeed or conflict: " + failure, + null == failure || failure instanceof TransactionException); + + final Set inMemory = elementIds(graph, withEdge); + if (null == failure) + assertTrue(inMemory.contains(withEdge ? 100 : 5)); + else + assertTrue(inMemory.isEmpty()); + + // the store as a crash right now would leave it, read from a copy since the graph still holds the directory + final File crashed = tempFolder.newFolder(); + for (final File file : directory.listFiles()) { + if (!file.getName().equals("LOCK")) + Files.copy(file.toPath(), new File(crashed, file.getName()).toPath()); + } + graph.close(); + final Configuration crashedConf = new BaseConfiguration(); + crashedConf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + crashedConf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, crashed.getAbsolutePath()); + graph = TinkerStorageGraph.open(crashedConf); + assertEquals("after a crash", inMemory, elementIds(graph, withEdge)); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals("after a clean close", inMemory, elementIds(graph, withEdge)); + graph.close(); + } + + private static Set elementIds(final TinkerStorageGraph graph, final boolean edges) { + final Set ids = new HashSet<>(); + (edges ? graph.edges() : graph.vertices()).forEachRemaining(e -> ids.add(e.id())); + return ids; + } + + /** + * Pauses the first removal from the map, which is how clear() empties it, until another thread lets it go on. + */ + private static class PausingMap extends ConcurrentHashMap { + final CountDownLatch paused = new CountDownLatch(1); + final CountDownLatch resume = new CountDownLatch(1); + private final AtomicBoolean first = new AtomicBoolean(true); + + @Override + public boolean remove(final Object key, final Object value) { + pauseOnce(); + return super.remove(key, value); + } + + @Override + public void clear() { + pauseOnce(); + super.clear(); + } + + private void pauseOnce() { + if (!first.getAndSet(false)) return; + paused.countDown(); + try { + resume.await(10, TimeUnit.SECONDS); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + } + } + } + + @Test + @SuppressWarnings("unchecked") + public void shouldReportACommitThatFailsAfterItsStorageWriteAsDurable() throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "a"); + graph.tx().commit(); + + // the index commit runs once the transaction is durable, so a failure there comes after the write + final TinkerTransactionalIndex index = spy((TinkerTransactionalIndex) graph.vertexIndex); + doThrow(new IllegalStateException("index failure")).when(index).commit(any()); + graph.vertexIndex = index; + + graph.addVertex(T.id, 2, "name", "b"); + try { + graph.tx().commit(); + fail("the failure applying the transaction should be reported"); + } catch (TransactionException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("will be present when the graph is reopened")); + } + assertTrue(graph.storage.isFailed()); + assertFalse("the failure came before any of the transaction was applied", graph.vertices(2).hasNext()); + + graph.addVertex(T.id, 3); + try { + graph.tx().commit(); + fail("no further commits are accepted until the graph is reopened"); + } catch (UncheckedIOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("stopped accepting commits")); + } + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertTrue("the durable transaction is present after the reopen", graph.vertices(2).hasNext()); + assertFalse(graph.vertices(3).hasNext()); + graph.close(); + } + + @Test + public void shouldReportACommitThatFailsPartwayThroughItsApplyAsDurable() throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder().getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.createIndex("name", Vertex.class); + graph.createIndex("weight", Edge.class); + graph.addVertex(T.id, 1, "name", "a"); + graph.tx().commit(); + + // the vertex index is committed first, so a failure in the edge index comes after part of the apply + final TinkerTransactionalIndex index = spy((TinkerTransactionalIndex) graph.edgeIndex); + doThrow(new IllegalStateException("index failure")).when(index).commit(any()); + graph.edgeIndex = index; + + graph.vertices(1).next().addEdge("knows", graph.addVertex(T.id, 2, "name", "b"), T.id, 10, "weight", 0.5d); + try { + graph.tx().commit(); + fail("the failure applying the transaction should be reported"); + } catch (TransactionException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("will be present when the graph is reopened")); + assertTrue(expected.getMessage(), expected.getMessage().contains("may show only part of it")); + } + assertTrue(graph.storage.isFailed()); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(1L, graph.traversal().V().has("name", "b").count().next().longValue()); + assertEquals(1L, graph.traversal().E().has("weight", 0.5d).count().next().longValue()); + assertEquals(2, graph.vertices(1).next().edges(Direction.OUT).next().inVertex().id()); + graph.close(); + } + private static final String PROPERTY_NAME = "name"; private static final String PROPERTY_WEIGHT = "weight"; private static final String EDGE_LABEL = "tests"; @@ -81,6 +517,17 @@ public void shouldCountVerticesByLabelAccountingForMultipleLabels() { assertEquals(1L, g.countVerticesByLabel("employee")); } + @Test + public void shouldRollbackWithVertexIndexButNoEdgeIndex() { + final TinkerStorageGraph g = TinkerStorageGraph.open(); + g.createIndex("name", Vertex.class); + g.addVertex("name", "marko"); + g.tx().rollback(); + + assertEquals(0L, (long) g.traversal().V().has("name", "marko").count().next()); + g.tx().rollback(); + } + @Test public void shouldReturnSameVertexInstanceInsideTransaction() { final TinkerStorageGraph g = TinkerStorageGraph.open(); @@ -1415,14 +1862,16 @@ public void shouldHandleConcurrentChangeForVertexMetaProperty() throws Interrupt try { gtx.tx().commit(); - //fail("should throw TransactionException"); + fail("should throw TransactionException"); } catch (TransactionException e) { } - // should be only 1 vertex with updated property + // tx2 committed first and wins; tx1 conflicted and rolled back, so its meta1 was never persisted assertEquals(1L, (long) gtx.V().count().next()); - assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta1", "meta2").next()); + assertEquals(1L, (long) gtx.V(v1.id()).properties("test").properties().count().next()); + assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta2").next()); + assertFalse(gtx.V(v1.id()).properties("test").values("meta1").hasNext()); } @Test diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java new file mode 100644 index 00000000000..209d56dd751 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java @@ -0,0 +1,715 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.math.BigDecimal; +import java.math.BigInteger; +import java.time.Duration; +import java.time.OffsetDateTime; +import java.io.File; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Engine-agnostic conformance suite ("TCK") for pluggable {@link TinkerStorage} engines. A concrete engine is tested + * by subclassing this and returning the {@code gremlin.tinkergraph.storage} value that selects it (an engine name or a + * fully-qualified class name). Every test opens a {@link TinkerStorageGraph} backed by a fresh temporary directory, + * mutates it, reopens from the same configuration, and asserts the data survived. A new engine drops in by adding one + * subclass. + */ +public abstract class AbstractTinkerStorageConformanceTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + /** + * The {@code gremlin.tinkergraph.storage} configuration value that selects the engine under test. + */ + protected abstract String storageEngine(); + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + protected Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, storageEngine()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + protected TinkerStorageGraph open() { + return TinkerStorageGraph.open(config()); + } + + @Test + public void shouldPersistVerticesAndEdgesAcrossReopen() { + TinkerStorageGraph graph = open(); + final Vertex marko = graph.addVertex(T.id, 1, T.label, "person", "name", "marko", "age", 29); + final Vertex lop = graph.addVertex(T.id, 3, T.label, "software", "name", "lop", "lang", "java"); + marko.addEdge("created", lop, T.id, 9, "weight", 0.4); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(1, countOf(graph.edges())); + final Vertex reMarko = graph.vertices(1).next(); + assertEquals("marko", reMarko.value("name")); + assertEquals(Integer.valueOf(29), reMarko.value("age")); + final Edge reCreated = graph.edges(9).next(); + assertEquals("created", reCreated.label()); + assertEquals(0.4, reCreated.value("weight"), 0.0001); + assertEquals(Integer.valueOf(1), reCreated.outVertex().id()); + assertEquals(Integer.valueOf(3), reCreated.inVertex().id()); + graph.close(); + } + + @Test + public void shouldPersistAcrossMultipleCommits() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 10; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.close(); + + graph = open(); + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("value")); + graph.close(); + } + + @Test + public void shouldPersistModificationsWithLastWriteWinning() { + TinkerStorageGraph graph = open(); + final Vertex v = graph.addVertex(T.id, 1, "name", "original"); + graph.tx().commit(); + v.property("name", "updated"); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals("updated", graph.vertices(1).next().value("name")); + graph.close(); + } + + @Test + public void shouldNotPersistRemovedElements() { + TinkerStorageGraph graph = open(); + final Vertex a = graph.addVertex(T.id, 1); + final Vertex b = graph.addVertex(T.id, 2); + final Edge e = a.addEdge("knows", b, T.id, 10); + graph.tx().commit(); + e.remove(); + b.remove(); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldNotPersistRolledBackTransaction() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "committed"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "rolledback"); + graph.tx().rollback(); + graph.close(); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertNotNull(graph.vertices(1).next()); + assertFalse(graph.vertices(2).hasNext()); + graph.close(); + } + + @Test + public void shouldKeepOnlyTheLastMultiPropertyValueOnReopenOfASingleCardinalityStore() { + // the documented sharp edge kept until TinkerPop has schema: a store created with the default single + // cardinality holds list values in memory but rebuilds each key with only its last value on reopen + TinkerStorageGraph graph = open(); + final Vertex v = graph.addVertex(T.id, 1); + v.property(VertexProperty.Cardinality.list, "name", "a"); + v.property(VertexProperty.Cardinality.list, "name", "b"); + v.property(VertexProperty.Cardinality.list, "name", "c"); + graph.tx().commit(); + assertEquals(3, countOf(graph.vertices(1).next().properties("name"))); + graph.close(); + + graph = open(); + assertEquals(Arrays.asList("c"), valuesOf(graph.vertices(1).next(), "name")); + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); // compacts, so the dropped values are gone from the store as well + + graph = open(); + assertEquals(Arrays.asList("c"), valuesOf(graph.vertices(1).next(), "name")); + graph.close(); + } + + private static List valuesOf(final Vertex v, final String key) { + final List values = new ArrayList<>(); + v.properties(key).forEachRemaining(p -> values.add(p.value())); + return values; + } + + @Test + public void shouldPersistMetaPropertiesAndMultiProperties() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, VertexProperty.Cardinality.list.name()); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + final Vertex v = graph.addVertex(T.id, 1); + final VertexProperty vp = v.property(VertexProperty.Cardinality.list, "name", "marko"); + vp.property("acl", "public"); + v.property(VertexProperty.Cardinality.list, "name", "marko a. rodriguez"); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + final Vertex reV = graph.vertices(1).next(); + assertEquals(2, countOf(reV.properties("name"))); + final Iterator> props = reV.properties("name"); + boolean foundAcl = false; + while (props.hasNext()) { + final VertexProperty p = props.next(); + if (p.properties("acl").hasNext()) { + assertEquals("public", p.properties("acl").next().value()); + foundAcl = true; + } + } + assertTrue("meta-property should survive persistence", foundAcl); + graph.close(); + } + + @Test + public void shouldPreserveStateAfterCompact() { + TinkerStorageGraph graph = open(); + for (int i = 0; i < 5; i++) { + graph.addVertex(T.id, i, "value", i); + graph.tx().commit(); + } + graph.compact(); + // keep writing after compaction to exercise the truncated log + graph.addVertex(T.id, 100, "value", 100); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(6, countOf(graph.vertices())); + assertEquals(Integer.valueOf(100), graph.vertices(100).next().value("value")); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("value")); + graph.close(); + } + + @Test + public void shouldReopenEmptyGraph() { + TinkerStorageGraph graph = open(); + graph.close(); + + graph = open(); + assertEquals(0, countOf(graph.vertices())); + assertEquals(0, countOf(graph.edges())); + graph.close(); + } + + @Test + public void shouldReportPersistenceFeature() { + final TinkerStorageGraph graph = open(); + assertTrue(graph.features().graph().supportsPersistence()); + graph.close(); + } + + @Test + public void shouldPersistConcurrentCommitsWithoutLossAcrossReopen() throws Exception { + // End-to-end companion to StorageCommitSerializationTest: many threads commit disjoint vertices at once and, + // on reopen, every record must survive. This exercises the real engine but cannot by itself *prove* the lock + // works — log corruption from interleaving is scheduling-dependent — so the deterministic guarantee is + // asserted separately by StorageCommitSerializationTest via a probe engine. + final int threads = 8; + final int commitsPerThread = 50; + final TinkerStorageGraph writeGraph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); // release all threads together to maximize contention on the commit path + for (int i = 0; i < commitsPerThread; i++) { + final int id = threadId * commitsPerThread + i; + writeGraph.addVertex(T.id, id, "value", id); + writeGraph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + writeGraph.close(); + } + + // reopen from disk: a corrupt (interleaved) log frame would throw or drop records here + final TinkerStorageGraph reopened = open(); + try { + final int expected = threads * commitsPerThread; + assertEquals(expected, countOf(reopened.vertices())); + for (int id = 0; id < expected; id++) + assertEquals(Integer.valueOf(id), reopened.vertices(id).next().value("value")); + } finally { + reopened.close(); + } + } + + @Test + public void shouldRoundTripDiverseValueTypes() { + final Map values = new LinkedHashMap<>(); + values.put("int", 42); + values.put("long", 42L); + values.put("float", 1.5f); + values.put("double", 2.5d); + values.put("bool", true); + values.put("byte", (byte) 7); + values.put("short", (short) 9); + values.put("char", 'x'); + values.put("string", "hello"); + values.put("uuid", new UUID(12L, 34L)); + values.put("bigint", new BigInteger("123456789012345678901234567890")); + values.put("bigdec", new BigDecimal("3.14159265358979")); + values.put("datetime", OffsetDateTime.parse("2020-01-02T03:04:05Z")); + values.put("duration", Duration.ofSeconds(90)); + + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + values.forEach(v::property); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + values.forEach((k, expected) -> assertEquals(k, expected, v.value(k))); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripCollectionValuedProperties() { + final List list = Arrays.asList(1, "two", 3.0d); + final Map map = new LinkedHashMap<>(); + map.put("a", 1); + map.put("b", "two"); + final Set set = new LinkedHashSet<>(Arrays.asList("x", "y", "z")); + + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("list", list); + v.property("map", map); + v.property("set", set); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + assertEquals(list, v.value("list")); + assertEquals(map, v.value("map")); + assertEquals(set, v.value("set")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripNullPropertyValue() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, true); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("maybe", null); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.vertices(1).next().properties("maybe").next(); + assertTrue(vp.isPresent()); + assertNull(vp.value()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripHeterogeneousSameKeyTypes() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "k", 42); // Integer + graph.addVertex(T.id, 2, "k", 42L); // Long + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("k")); + assertEquals(Long.valueOf(42L), graph.vertices(2).next().value("k")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripUuidElementIds() { + roundTripElementIds(new UUID(0L, 1L), new UUID(0L, 2L), new UUID(0L, 10L)); + } + + @Test + public void shouldRoundTripStringElementIds() { + roundTripElementIds("v-1", "v-2", "e-10"); + } + + // uses the default ANY id manager so any id type is accepted verbatim; the point is that the storage codec + // round-trips non-Long element ids through its scalar id encoding + private void roundTripElementIds(final Object outId, final Object inId, final Object edgeId) { + TinkerStorageGraph graph = open(); + try { + final Vertex a = graph.addVertex(T.id, outId, "name", "a"); + final Vertex b = graph.addVertex(T.id, inId, "name", "b"); + a.addEdge("knows", b, T.id, edgeId, "weight", 0.5d); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals("a", graph.vertices(outId).next().value("name")); + assertEquals(outId, graph.vertices(outId).next().id()); + final Edge e = graph.edges(edgeId).next(); + assertEquals("knows", e.label()); + assertEquals(outId, e.outVertex().id()); + assertEquals(inId, e.inVertex().id()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripLargeSchemaAcrossVarintBoundary() { + // >127 distinct keys and >127 values under one key push dictionary refs and counts past the single-byte + // LEB128 range, exercising the multi-byte varint path that small graphs never reach + final int n = 200; + // list cardinality so the >127 values under "multi" survive reopen (reconstruction takes cardinality from + // graph config, not the stored record) + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "list"); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.addVertex(T.id, 1); + for (int i = 0; i < n; i++) + v.property("key" + i, i); + for (int i = 0; i < n; i++) + v.property(VertexProperty.Cardinality.list, "multi", i); + graph.tx().commit(); + graph.compact(); // also exercises a dictionary header with >127 entries + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final Vertex v = graph.vertices(1).next(); + for (int i = 0; i < n; i++) + assertEquals("key" + i, Integer.valueOf(i), v.value("key" + i)); + assertEquals(n, countOf(v.properties("multi"))); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripUnicodeKeysAndValues() { + TinkerStorageGraph graph = open(); + try { + final Vertex v = graph.addVertex(T.id, 1); + v.property("naïve", "café"); + v.property("日本語", "テスト"); + v.property("emoji", "party🎉"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final Vertex v = graph.vertices(1).next(); + assertEquals("café", v.value("naïve")); + assertEquals("テスト", v.value("日本語")); + assertEquals("party🎉", v.value("emoji")); + } finally { + graph.close(); + } + } + + @Test + public void shouldSurviveRepeatedCompactionCycles() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "a", 1); + graph.tx().commit(); + graph.compact(); + graph.addVertex(T.id, 2, "b", 2); + graph.tx().commit(); + graph.compact(); + graph.addVertex(T.id, 3, "c", 3); + graph.tx().commit(); + graph.compact(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(3, countOf(graph.vertices())); + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("a")); + assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("b")); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("c")); + } finally { + graph.close(); + } + } + + @Test + public void shouldSurviveMultipleOpenCloseSessions() { + TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "n", "one"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + graph.addVertex(T.id, 2, "n", "two"); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + graph.addVertex(T.id, 3, "n", "three"); + graph.tx().commit(); + graph.compact(); + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(3, countOf(graph.vertices())); + assertEquals("one", graph.vertices(1).next().value("n")); + assertEquals("two", graph.vertices(2).next().value("n")); + assertEquals("three", graph.vertices(3).next().value("n")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripConcurrentCommitsWithDistinctKeys() throws Exception { + // concurrent commits that each introduce a distinct property key stress dictionary growth under the + // commit-write lock; on reopen every distinct key must resolve + final int threads = 8; + final int perThread = 25; + final TinkerStorageGraph writeGraph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); + for (int i = 0; i < perThread; i++) { + final int id = threadId * perThread + i; + writeGraph.addVertex(T.id, id, "k_" + threadId + "_" + i, id); + writeGraph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + writeGraph.close(); + } + final TinkerStorageGraph reopened = open(); + try { + final int expected = threads * perThread; + assertEquals(expected, countOf(reopened.vertices())); + for (int t = 0; t < threads; t++) { + for (int i = 0; i < perThread; i++) { + final int id = threadId(t, i, perThread); + assertEquals(Integer.valueOf(id), reopened.vertices(id).next().value("k_" + t + "_" + i)); + } + } + } finally { + reopened.close(); + } + } + + private static int threadId(final int t, final int i, final int perThread) { + return t * perThread + i; + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } + + @Test + public void shouldNotPersistUncommittedDataOnClose() { + // close() compacts, and compaction must snapshot only committed state. Uncommitted mutations left on the + // closing thread must not reach disk, matching how a crash drops them. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "marko"); + graph.addVertex(T.id, 2, "name", "vadas"); + // deliberately no commit + graph.close(); + + graph = open(); + assertEquals("uncommitted vertices must not survive a graceful close", 0, countOf(graph.vertices())); + graph.close(); + } + + @Test + public void shouldPersistOnlyCommittedPortionOnClose() { + // a committed vertex plus later uncommitted work: only the committed portion is durable across a graceful close. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "vadas"); // uncommitted + graph.vertices(1).next().property("age", 29); // uncommitted modification to a committed vertex + graph.close(); + + graph = open(); + assertEquals("only the committed vertex is durable", 1, countOf(graph.vertices())); + final Vertex marko = graph.vertices(1).next(); + assertEquals("marko", marko.value("name")); + assertFalse("the uncommitted property must not survive", marko.properties("age").hasNext()); + graph.close(); + } + + @Test + public void shouldRejectCommitAfterClose() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.close(); + + // the storage is closed and its directory lock released, so a further commit must neither succeed nor + // touch any file another graph could now own + final Map filesAtClose = directoryContents(); + graph.addVertex(T.id, 2, "name", "vadas"); + try { + graph.tx().commit(); + fail("a commit on a closed durable graph should fail"); + } catch (IllegalStateException expected) { + // expected + } + assertFalse("the rejected commit must not be visible", graph.vertices(2).hasNext()); + graph.tx().rollback(); + assertEquals("a rejected commit must not change the storage directory", filesAtClose, directoryContents()); + + graph = open(); + assertEquals(1, countOf(graph.vertices())); + assertEquals("marko", graph.vertices(1).next().value("name")); + graph.close(); + } + + private Map directoryContents() { + final Map contents = new LinkedHashMap<>(); + final File[] files = new File(location).listFiles(); + if (files != null) { + Arrays.sort(files); + for (final File f : files) + contents.put(f.getName(), f.length()); + } + return contents; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java new file mode 100644 index 00000000000..4eb20334083 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/ConcurrencyProbeStorage.java @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.util.Collection; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * A {@link TinkerStorage} test double that reliably detects whether the commit path serializes writes. It + * persists nothing; instead {@link #persist} tracks how many threads are inside it at once and records a violation if + * ever more than one is. It also sleeps briefly while "inside" to widen the window, so an unserialized commit path + * trips the detector reliably rather than racily. Selected by fully-qualified class name via + * {@code gremlin.tinkergraph.storage}. State is static because the engine is instantiated reflectively. + */ +public final class ConcurrencyProbeStorage implements TinkerStorage { + + static final AtomicInteger inFlight = new AtomicInteger(0); + static final AtomicInteger maxObserved = new AtomicInteger(0); + static final AtomicInteger concurrentEntries = new AtomicInteger(0); + + static void reset() { + inFlight.set(0); + maxObserved.set(0); + concurrentEntries.set(0); + } + + @Override + public void open(final AbstractTinkerGraph graph, final Configuration config) { } + + @Override + public void replay(final AbstractTinkerGraph graph) { } + + @Override + public void persist(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) { + final int concurrent = inFlight.incrementAndGet(); + try { + maxObserved.accumulateAndGet(concurrent, Math::max); + if (concurrent > 1) + concurrentEntries.incrementAndGet(); + // widen the window so an unserialized path is caught reliably rather than by luck; it can miss, but a + // serialized path never trips it + try { + Thread.sleep(1); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + } + } finally { + inFlight.decrementAndGet(); + } + } + + @Override + public void flush() { } + + @Override + public void compact(final AbstractTinkerGraph graph) { } + + @Override + public void clear(final AbstractTinkerGraph graph) { } + + @Override + public void failStop(final Throwable cause) { } + + @Override + public void close() { } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockHolder.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockHolder.java new file mode 100644 index 00000000000..1e710fb5c77 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockHolder.java @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import java.io.File; + +/** + * Holds a {@link DirectoryLock} from a separate JVM so tests can check the lock against another process rather than + * another channel in their own. Prints {@code locked} once the lock is held and releases it when standard input + * closes. + */ +public final class DirectoryLockHolder { + + public static void main(final String[] args) throws Exception { + try (final DirectoryLock lock = DirectoryLock.acquire(new File(args[0]))) { + System.out.println("locked"); + System.out.flush(); + while (System.in.read() != -1) { + // wait for the test to close standard input + } + } + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java new file mode 100644 index 00000000000..8d96f6a96c6 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/DirectoryLockTest.java @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.BufferedReader; +import java.io.File; +import java.io.InputStreamReader; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.charset.StandardCharsets; +import java.nio.file.StandardOpenOption; +import java.util.concurrent.TimeUnit; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Verifies that a {@link TinkerStorageGraph} takes an exclusive lock on its storage directory so a second opener on + * the same location fails fast rather than corrupting the store. + */ +public class DirectoryLockTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + @Test + public void shouldReleaseLockOnCloseAndAllowReopen() { + // sequential open/close must not leave the directory wedged + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(config()); + assertNotNull(graph.vertices(1).next()); + graph.close(); + } + + @Test + public void shouldRejectSecondOpenWhileLocationIsLocked() throws Exception { + // simulate another process holding the directory lock by taking the OS lock on the LOCK file directly + final File dir = new File(location); + dir.mkdirs(); + final File lockFile = new File(dir, DirectoryLock.LOCK_FILE); + try (final FileChannel channel = FileChannel.open(lockFile.toPath(), + StandardOpenOption.CREATE, StandardOpenOption.WRITE); + final FileLock held = channel.lock()) { + assertNotNull(held); + try { + TinkerStorageGraph.open(config()); + fail("expected open to fail while the storage location is locked"); + } catch (IllegalStateException expected) { + assertTrue("message should name the location: " + expected.getMessage(), + expected.getMessage().contains(location)); + } + } + + // once the simulated holder releases (try-with-resources above), the location opens normally + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.close(); + } + + @Test + public void shouldRejectOpenWhileAnotherProcessHoldsTheLock() throws Exception { + // a lock held by another JVM is the case the directory lock exists for, and one a second channel in this JVM + // cannot stand in for, since the JVM tracks its own locks apart from the operating system's + final File dir = new File(location); + dir.mkdirs(); + final String java = new File(new File(System.getProperty("java.home"), "bin"), "java").getAbsolutePath(); + final Process holder = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"), + DirectoryLockHolder.class.getName(), location).redirectErrorStream(true).start(); + try { + final BufferedReader out = new BufferedReader( + new InputStreamReader(holder.getInputStream(), StandardCharsets.UTF_8)); + assertEquals("the other process should report holding the lock", "locked", out.readLine()); + try { + TinkerStorageGraph.open(config()); + fail("expected open to fail while another process holds the storage lock"); + } catch (IllegalStateException expected) { + assertTrue("message should name the location: " + expected.getMessage(), + expected.getMessage().contains(location)); + } + } finally { + holder.getOutputStream().close(); + if (!holder.waitFor(30, TimeUnit.SECONDS)) + holder.destroyForcibly(); + } + + // once the other process releases the lock, the location opens normally + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.close(); + } + + @Test + public void shouldKeepTheLockAgainstOtherProcessesAfterARefusedOpenInTheSameJVM() throws Exception { + // a refused open must not touch the LOCK file the holder has locked: closing any handle on it would release + // the holder's operating system lock, while this JVM still reports it held, and let another process in + final TinkerStorageGraph holder = TinkerStorageGraph.open(config()); + try { + try { + TinkerStorageGraph.open(config()); + fail("expected a second open in the same JVM to fail"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(location)); + } + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + try { + TinkerStorageGraph.open(recover); + fail("expected a recover open in the same JVM to fail"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(location)); + } + + final String java = new File(new File(System.getProperty("java.home"), "bin"), "java").getAbsolutePath(); + final Process other = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"), + DirectoryLockHolder.class.getName(), location).redirectErrorStream(true).start(); + other.getOutputStream().close(); + try { + assertTrue("the other process should give up rather than wait", + other.waitFor(30, TimeUnit.SECONDS)); + final String output = new String(readAll(other), StandardCharsets.UTF_8); + assertNotEquals("the other process must not get the lock: " + output, 0, other.exitValue()); + assertFalse(output, output.startsWith("locked")); + } finally { + other.destroyForcibly().waitFor(30, TimeUnit.SECONDS); + } + } finally { + holder.close(); + } + } + + private static byte[] readAll(final Process process) throws Exception { + final java.io.ByteArrayOutputStream out = new java.io.ByteArrayOutputStream(); + final byte[] buffer = new byte[4096]; + int read; + while ((read = process.getInputStream().read(buffer)) != -1) + out.write(buffer, 0, read); + return out.toByteArray(); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/FaultInjectingStorage.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/FaultInjectingStorage.java new file mode 100644 index 00000000000..e99e368e251 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/FaultInjectingStorage.java @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex; + +import java.io.DataOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.Collection; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * A {@link TinkerStorage} test double that writes the real {@code graphbinary} format but can be told to fail. While + * {@link #failLogWrites} is set, every write that reaches the log file throws, as a failing disk would. Small frames + * are buffered, so for them the failure surfaces at flush; a frame larger than the buffer fails during the append + * itself. While {@link #failCompaction} is set, compaction throws before touching any file, and while + * {@link #failSnapshotWrites} is set it fails partway through writing the new snapshot, or with an {@code Error} + * while {@link #failSnapshotWritesWithError} is set. Installing an empty log fails with an {@code IOException} while + * {@link #failEmptyLogInstallWithIOException} is set, or throws {@link #failEmptyLogInstall} when that is set. {@link #compactionAttempts} + * counts every compaction started, whether or not it fails. When {@link #compactionGate} is set, compaction counts + * down {@link #compactionEntered} and then waits for the gate, holding the graph's commit lock as its caller does, so + * a test can make another thread wait on that lock. Selected by + * fully-qualified class name via {@code gremlin.tinkergraph.storage}. The switches are static because the engine is + * instantiated reflectively. + */ +public final class FaultInjectingStorage extends AbstractLogStorage { + + static volatile boolean failLogWrites = false; + static volatile boolean failCompaction = false; + static volatile boolean failSnapshotWrites = false; + static volatile boolean failSnapshotWritesWithError = false; + static volatile RuntimeException failEmptyLogInstall = null; + static volatile boolean failEmptyLogInstallWithIOException = false; + static final AtomicInteger compactionAttempts = new AtomicInteger(); + static volatile CountDownLatch compactionEntered = null; + static volatile CountDownLatch compactionGate = null; + + static void reset() { + failLogWrites = false; + failCompaction = false; + failSnapshotWrites = false; + failSnapshotWritesWithError = false; + failEmptyLogInstall = null; + failEmptyLogInstallWithIOException = false; + compactionAttempts.set(0); + compactionEntered = null; + compactionGate = null; + } + + private final GraphBinaryStorage codec = new GraphBinaryStorage(); + + @Override + protected FileOutputStream openLogForAppend(final File file) throws IOException { + return new FileOutputStream(file, true) { + @Override + public void write(final int b) throws IOException { + failIfArmed(); + super.write(b); + } + + @Override + public void write(final byte[] b, final int off, final int len) throws IOException { + failIfArmed(); + super.write(b, off, len); + } + + private void failIfArmed() throws IOException { + if (failLogWrites) + throw new IOException("injected log write failure"); + } + }; + } + + @Override + protected void compact(final AbstractTinkerGraph graph, final boolean force) { + compactionAttempts.incrementAndGet(); + final CountDownLatch gate = compactionGate; + if (gate != null) { + compactionEntered.countDown(); + try { + // bounded, so a test that never opens the gate fails instead of hanging the build + if (!gate.await(30, TimeUnit.SECONDS)) + throw new IllegalStateException("compaction gate was never opened"); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + } + } + if (failCompaction) + throw new UncheckedIOException(new IOException("injected compaction failure")); + super.compact(graph, force); + } + + @Override + protected FileOutputStream openEmptyLog(final File tmp) throws IOException { + if (failEmptyLogInstallWithIOException) + throw new IOException("injected empty log install failure"); + final RuntimeException failure = failEmptyLogInstall; + if (failure != null) + throw failure; + return super.openEmptyLog(tmp); + } + + @Override + protected void configureCodec(final Configuration config) { + codec.configureCodec(config); + } + + @Override + protected void beginReplay() { + codec.beginReplay(); + } + + @Override + protected byte[] encodeCommit(final long txVersion, + final Collection> changedVertices, + final Collection> changedEdges) throws IOException { + return codec.encodeCommit(txVersion, changedVertices, changedEdges); + } + + @Override + protected void decodeFrame(final byte[] record, + final Map vertices, + final Map edges) throws IOException { + codec.decodeFrame(record, vertices, edges); + } + + @Override + protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException { + if (failSnapshotWrites) { + out.write(new byte[]{ 0x01, 0x02, 0x03 }); + throw new IOException("injected snapshot write failure"); + } + if (failSnapshotWritesWithError) + throw new OutOfMemoryError("injected snapshot write error"); + codec.writeSnapshot(graph, out); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java new file mode 100644 index 00000000000..94456587c67 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java @@ -0,0 +1,1489 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.structure.util.TransactionException; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge; +import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Test; + +import java.io.DataInputStream; +import java.io.File; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicReference; +import java.util.zip.CRC32; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Runs the shared {@link AbstractTinkerStorageConformanceTest} suite against the {@link GraphBinaryStorage} engine and + * adds engine-specific tests for the on-disk log layout, sync modes, auto-compaction, snapshot streaming, and + * corruption detection. + *

+ * Not covered here (deliberately): true {@code fsync} durability against OS crash or power loss. The {@code commit} + * vs. {@code os} sync-mode tests verify configuration and a graceful round-trip, but a JVM unit test cannot prove that + * an acknowledged commit survives a kernel crash — that needs OS-level fault injection (e.g. a FUSE layer that drops + * un-synced writes, or {@code dm-flakey}), which is out of scope. Crash-*consistency* of the file layout (as opposed + * to device-level durability) is covered deterministically by {@link StorageCrashConsistencyTest}. + */ +public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest { + + @Override + protected String storageEngine() { + return "graphbinary"; + } + + @Test + public void shouldWriteSnapshotAndLogFiles() { + final TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + assertTrue(new File(location, GraphBinaryStorage.LOG_FILE).exists()); + graph.close(); + // close compacts, producing a snapshot and truncating the log + assertTrue(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + } + + @Test + public void shouldNotWriteToTheLogForATransactionThatChangedNothing() { + final TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final long length = logFile.length(); + + // a read opens a transaction, and committing it must leave the log alone + assertEquals(1, countOf(graph.vertices())); + graph.tx().commit(); + graph.tx().commit(); + assertEquals("a transaction that changed nothing must not append to the log", length, logFile.length()); + graph.close(); + } + + @Test + public void shouldPersistWithOsSyncMode() { + // 'os' is a weaker durability mode (no fsync); a graceful close/reopen must still round-trip the data. The + // OS-crash-loss window that distinguishes it from 'commit' cannot be exercised in a unit test. + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, "os"); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1, "value", 42); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(1, countOf(graph.vertices())); + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("value")); + graph.close(); + } + + @Test + public void shouldPersistWithDefaultCommitSyncMode() { + // with no sync mode configured the engine defaults to 'commit' (fsync per commit); data must round-trip. + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1, "value", 42); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("value")); + graph.close(); + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectUnknownSyncMode() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, "bogus"); + TinkerStorageGraph.open(conf); + } + + @Test + public void shouldLeaveNoTempSnapshotAfterCompaction() { + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.compact(); + // the atomic rename must consume the temp file, leaving a durable snapshot and no leftover .tmp + assertTrue(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + assertTrue(!new File(location, GraphBinaryStorage.SNAPSHOT_FILE + ".tmp").exists()); + graph.close(); + } + + @Test + public void shouldStreamSnapshotAsOneFramePerElement() throws Exception { + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final Vertex a = graph.addVertex(T.id, 1, "name", "a"); + final Vertex b = graph.addVertex(T.id, 2, "name", "b"); + final Vertex c = graph.addVertex(T.id, 3, "name", "c"); + a.addEdge("knows", b, T.id, 10); + b.addEdge("knows", c, T.id, 11); + graph.tx().commit(); + graph.compact(); + + // the snapshot must be streamed: a dictionary header frame plus one framed record per element (3 vertices + + // 2 edges = 5), rather than a single whole-graph frame, so compaction never buffers the entire graph at once + final File snapshotFile = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + assertEquals(1 + 5, countFrames(snapshotFile)); + graph.close(); + + // and the streamed snapshot must reopen to exactly the same graph + final TinkerStorageGraph reopened = open(); + assertEquals(3, countOf(reopened.vertices())); + assertEquals(2, countOf(reopened.edges())); + assertEquals("a", reopened.vertices(1).next().value("name")); + assertEquals("knows", reopened.edges(10).next().label()); + reopened.close(); + } + + @Test + public void shouldStreamSnapshotFrameByFrameAtScale() { + // Bounded-memory proxy for the streaming snapshot path: rather than measure heap (flaky, JVM-dependent), assert + // the observable streaming property holds at scale — a large graph is written as a dictionary header frame plus + // one frame per element, never one whole-graph frame — and round-trips intact. This is the property that keeps + // compaction from materializing a second full copy of the graph in memory; it is not a hard OOM assertion. + final int vertexCount = 500; + final int edgeCount = 499; + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + try { + for (int i = 0; i < vertexCount; i++) + graph.addVertex(T.id, i, "value", i); + for (int i = 0; i < edgeCount; i++) + graph.vertices(i).next().addEdge("next", graph.vertices(i + 1).next(), T.id, 1_000_000 + i); + graph.tx().commit(); + graph.compact(); + + try { + assertEquals(1 + vertexCount + edgeCount, countFrames(new File(location, GraphBinaryStorage.SNAPSHOT_FILE))); + } catch (Exception ex) { + throw new RuntimeException(ex); + } + } finally { + graph.close(); + } + + final TinkerStorageGraph reopened = open(); + try { + assertEquals(vertexCount, countOf(reopened.vertices())); + assertEquals(edgeCount, countOf(reopened.edges())); + assertEquals(Integer.valueOf(499), reopened.vertices(499).next().value("value")); + } finally { + reopened.close(); + } + } + + /** + * Count the framed records in a storage file: a fixed header ({@code HEADER_SIZE} bytes) followed by frames of a + * 4-byte big-endian payload length, a 4-byte CRC, then that many payload bytes. + */ + private static int countFrames(final File file) throws Exception { + int frames = 0; + try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new java.io.FileInputStream(file)))) { + if (!skipFully(in, GraphBinaryStorage.HEADER_SIZE)) return 0; + while (true) { + final int len; + try { + len = in.readInt(); + } catch (java.io.EOFException eof) { + break; + } + in.readInt(); // CRC + if (!skipFully(in, len)) break; + frames++; + } + } + return frames; + } + + /** + * Skip exactly {@code n} bytes, reading in a loop because a single {@link DataInputStream#skip} may skip fewer. + * Returns false if EOF is reached first. + */ + private static boolean skipFully(final DataInputStream in, final long n) throws Exception { + long left = n; + while (left > 0) { + final long s = in.skip(left); + if (s <= 0) { + if (in.read() < 0) return false; // genuine EOF + left -= 1; + } else { + left -= s; + } + } + return true; + } + + @Test + public void shouldAutoCompactWhenLogExceedsThreshold() { + // a small threshold makes automatic compaction fire mid-run, without any explicit compact()/close() + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 2048L); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + for (int i = 0; i < 200; i++) { + graph.addVertex(T.id, i, "value", i, "pad", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); + graph.tx().commit(); + } + // auto-compaction should have folded the log into a snapshot and truncated it well below the total + // bytes written, so the live log stays bounded rather than growing with every commit + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final File snapshotFile = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + assertTrue("expected a snapshot from auto-compaction", snapshotFile.exists()); + assertTrue("expected the live log to stay bounded, was " + logFile.length(), + logFile.length() < 8192); + } finally { + graph.close(); + } + + // data must survive across reopen despite the mid-run compactions + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(200, countOf(graph.vertices())); + assertEquals(Integer.valueOf(199), graph.vertices(199).next().value("value")); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotAutoCompactWhenThresholdIsZero() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + for (int i = 0; i < 50; i++) { + graph.addVertex(T.id, i, "pad", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"); + graph.tx().commit(); + } + // with auto-compaction disabled, no snapshot appears until close()/compact() + assertTrue(!new File(location, GraphBinaryStorage.SNAPSHOT_FILE).exists()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRecoverFromTruncatedTrailingFrame() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + graph.tx().close(); + + // capture the raw log (two good frames) while the graph holds it, then close to release the directory lock. + // close() compacts, folding the log into a snapshot, so we reconstruct a "crashed" on-disk state below. + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] goodLog = Files.readAllBytes(logFile.toPath()); + graph.close(); + + // recreate the pre-crash layout: no snapshot, a log of the two good frames plus a torn trailing frame (a + // length prefix promising 100 bytes with only a couple following), as an interrupted append would leave. + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.setLength(0); + raf.write(goodLog); + raf.write(tornLogFrame(100)); + } + + // reopening must recover the two fully-committed vertices and ignore the torn frame + graph = open(); + assertEquals(2, countOf(graph.vertices())); + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("value")); + assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("value")); + graph.close(); + } + + @Test + public void shouldKeepCommitsWrittenAfterRecoveringFromTornFrameShorterThanClaimed() throws Exception { + // the torn frame promises far more bytes than the later commit adds, so without truncation a second replay + // still sees "not enough bytes" at the torn frame and silently drops the later commit + assertCommitsSurviveSecondCrashAfterTornFrame(100_000); + } + + @Test + public void shouldKeepCommitsWrittenAfterRecoveringFromTornFrameLongerThanClaimed() throws Exception { + // the torn frame promises only a few bytes, so without truncation a second replay reads into the later + // commit as if it were the torn frame's payload, fails the CRC check and refuses to open + assertCommitsSurviveSecondCrashAfterTornFrame(4); + } + + @Test + public void shouldFailOnLogShorterThanItsHeader() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.close(); + + // the log is only ever replaced whole by a rename, so one holding part of its header is damage + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + Files.write(logFile.toPath(), Arrays.copyOf(AbstractLogStorage.MAGIC, 2)); + try { + open(); + fail("a log shorter than its header should not open"); + } catch (Exception expected) { + // expected + } + assertEquals(2, logFile.length()); + } + + @Test + public void shouldRejectShortLogThatIsNotAHeaderPrefix() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.close(); + + // two bytes that cannot be the start of the header are corruption, not an interrupted append + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + Files.write(logFile.toPath(), new byte[]{ 'X', 'Y' }); + try { + open(); + fail("a short log that is not a prefix of the header should not open"); + } catch (Exception expected) { + // expected + } + assertEquals(2, logFile.length()); + } + + @Test + public void shouldRejectBlankStorageDirectory() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, " "); + try { + TinkerStorageGraph.open(conf).close(); + fail("a blank storage directory should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), + expected.getMessage().contains(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + } + assertTrue("no store may be created in the working directory", !new File(" ").exists()); + } + + @Test + public void shouldRejectStorageDirectoryWithoutAnEngine() throws Exception { + final File dir = new File(tempFolder.getRoot(), "no-engine"); + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, dir.getAbsolutePath()); + try { + TinkerStorageGraph.open(conf).close(); + fail("a storage directory without a storage engine should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE)); + } + assertFalse("no directory may be created", dir.exists()); + } + + @Test + public void shouldRejectNegativeCompactThreshold() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, -1); + try { + TinkerStorageGraph.open(conf).close(); + fail("a negative compaction threshold should not open"); + } catch (IllegalArgumentException expected) { + assertTrue(expected.getMessage(), + expected.getMessage().contains(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD)); + } + + // the failed open released the directory, so a valid configuration can open it + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0); + TinkerStorageGraph.open(conf).close(); + } + + @Test + public void shouldNameThePropertyAndTypeThatCannotBeStored() throws Exception { + final TinkerStorageGraph graph = open(); + try { + graph.addVertex(T.id, 1, "created", new Date()); + assertCommitFailsWith(graph, "property 'created' of vertex 1, a java.util.Date", + "no serializer for java.util.Date"); + + final Vertex a = graph.addVertex(T.id, 2); + a.addEdge("knows", a, T.id, 10, "data", new byte[]{ 1, 2 }); + assertCommitFailsWith(graph, "property 'data' of edge 10, a byte[]", "no serializer for byte[]"); + } finally { + graph.close(); + } + } + + private static void assertCommitFailsWith(final TinkerStorageGraph graph, final String... fragments) { + try { + graph.tx().commit(); + fail("a value with no serializer should fail the commit"); + } catch (RuntimeException expected) { + for (final String fragment : fragments) + assertTrue(expected.getMessage(), expected.getMessage().contains(fragment)); + } + } + + @Test + public void shouldReplayCommitsAfterATransactionThatCouldNotBeEncoded() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, T.label, "person", "name", "marko"); + graph.tx().commit(); + + // a new label and a new key whose value has no serializer, so the commit fails while it is being encoded + graph.addVertex(T.id, 2, T.label, "software", "created", new Date()); + try { + graph.tx().commit(); + fail("a value with no serializer should fail the commit"); + } catch (RuntimeException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("no serializer for java.util.Date")); + } + + // reuse the key from the failed commit and add a new one, then crash before close() can compact + graph.addVertex(T.id, 3, T.label, "software", "created", 2024, "lang", "java"); + graph.tx().commit(); + final Map crashed = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crashed); + + graph = open(); + assertEquals(2, countOf(graph.vertices())); + final Vertex v = graph.vertices(3).next(); + assertEquals("software", v.label()); + assertEquals(Integer.valueOf(2024), v.value("created")); + assertEquals("java", v.value("lang")); + graph.close(); + } + + /** + * Crash with a torn trailing frame, reopen, commit more, crash again without a clean close, then reopen and + * check that every acknowledged commit is still there. + */ + private void assertCommitsSurviveSecondCrashAfterTornFrame(final int tornFrameClaimedLength) throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + final Map firstCrash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, firstCrash); + + // the first crash interrupted a third append, leaving a length prefix and a couple of payload bytes + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.seek(raf.length()); + raf.write(tornLogFrame(tornFrameClaimedLength)); + } + + graph = open(); + assertEquals(2, countOf(graph.vertices())); + graph.addVertex(T.id, 3, "value", 3); + graph.tx().commit(); + final Map secondCrash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, secondCrash); + + graph = open(); + assertEquals(3, countOf(graph.vertices())); + assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("value")); + graph.close(); + } + + /** + * The start of a log frame as an interrupted append leaves it: a complete header, whose length passes its checksum, + * claiming {@code claimedLength} payload bytes, followed by only two of them. + */ + private static byte[] tornLogFrame(final int claimedLength) { + final CRC32 lengthCrc = new CRC32(); + lengthCrc.update(ByteBuffer.allocate(Integer.BYTES).putInt(claimedLength).array()); + return ByteBuffer.allocate(GraphBinaryStorage.LOG_FRAME_HEADER_SIZE + 2) + .putInt(claimedLength) + .putInt((int) lengthCrc.getValue()) + .putInt(0) // sequence number + .putInt(0) // payload checksum, never reached + .put(new byte[]{ 0x01, 0x02 }) + .array(); + } + + /** + * Copy the snapshot and log as they are on disk while the graph is still open, so that restoring them after + * close() reproduces the state a crash at this point would leave (close() would otherwise compact them away). + */ + private static Map captureStorageFiles(final String location) throws IOException { + final Map files = new HashMap<>(); + for (final String name : new String[]{ GraphBinaryStorage.SNAPSHOT_FILE, GraphBinaryStorage.LOG_FILE }) { + final File f = new File(location, name); + files.put(name, f.exists() ? Files.readAllBytes(f.toPath()) : null); + } + return files; + } + + private static void restoreStorageFiles(final String location, final Map files) throws IOException { + for (final Map.Entry entry : files.entrySet()) { + final File f = new File(location, entry.getKey()); + if (entry.getValue() == null) + Files.deleteIfExists(f.toPath()); + else + Files.write(f.toPath(), entry.getValue()); + } + } + + @Test + public void shouldCutAZeroFilledTailLikeATornAppend() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.addVertex(T.id, 2); + graph.tx().commit(); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + // a crash during an append can leave the file longer than what was written, reading back as zeros + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final long complete = logFile.length(); + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.seek(complete); + raf.write(new byte[4096]); + } + + graph = open(); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + assertEquals("the zero-filled tail is cut back to the last complete record", complete, logFile.length()); + graph.addVertex(T.id, 3); + graph.tx().commit(); + graph.close(); + + graph = open(); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldNotCutATailThatIsNotAllZero() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + // zeros followed by anything else could be a damaged record, so the open must still refuse it + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] tail = new byte[64]; + tail[40] = 1; + try (final RandomAccessFile raf = new RandomAccessFile(logFile, "rw")) { + raf.seek(raf.length()); + raf.write(tail); + } + try { + open().close(); + fail("a tail that is not all zeros should not be cut"); + } catch (RuntimeException expected) { + assertTrue(rootMessage(expected), rootMessage(expected).contains("fails its checksum")); + } + } + + @Test + public void shouldRemoveTemporaryFilesLeftByAnInterruptedWrite() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + for (final String name : new String[]{ GraphBinaryStorage.SNAPSHOT_FILE, GraphBinaryStorage.LOG_FILE, + GraphBinaryStorage.VERSION_FILE }) + Files.write(new File(location, name + ".tmp").toPath(), new byte[]{ 1, 2, 3 }); + + graph = open(); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + for (final String name : new String[]{ GraphBinaryStorage.SNAPSHOT_FILE, GraphBinaryStorage.LOG_FILE, + GraphBinaryStorage.VERSION_FILE }) + assertFalse(name + ".tmp should be removed", new File(location, name + ".tmp").exists()); + graph.close(); + } + + @Test + public void shouldUndoDictionaryEntriesWhenEncodingFailsWithAnError() throws Exception { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // nesting deep enough that encoding the value overflows the stack, an Error rather than an exception, after + // the commit has already registered the new key "deep" in the dictionary + List deep = new ArrayList<>(); + final List root = deep; + for (int i = 0; i < 200_000; i++) { + final List next = new ArrayList<>(); + deep.add(next); + deep = next; + } + graph.addVertex(T.id, 2, "deep", root); + try { + graph.tx().commit(); + fail("encoding the nested value should fail"); + } catch (StackOverflowError expected) { + // expected + } + + // a later commit using the same key must define it, or the log could not be replayed + graph.addVertex(T.id, 3, "deep", 1); + graph.tx().commit(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + graph = open(); + assertEquals(Arrays.asList(1, 3), vertexIds(graph)); + assertEquals(Integer.valueOf(1), graph.vertices(3).next().value("deep")); + graph.close(); + } + + @Test + public void shouldKeepAClearedGraphClearedAfterACrash() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "a"); + graph.addVertex(T.id, 2, "name", "b"); + graph.tx().commit(); + + // clear() takes effect at once, with no commit, and a crash right after a later commit must not bring the + // cleared vertices back beside the new one + graph.clear(); + assertEquals(0, countOf(graph.vertices())); + graph.addVertex(T.id, 3, "name", "c"); + graph.tx().commit(); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + graph = open(); + assertEquals(Collections.singletonList(3), vertexIds(graph)); + assertTrue("clear() drops the indexes too", graph.getIndexedKeys(Vertex.class).isEmpty()); + graph.close(); + } + + @Test + public void shouldClearAStoreReopenedWithNothingLoggedSinceItsLastCompaction() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); // compacts, so the reopened store has nothing in its log + + graph = open(); + graph.clear(); + graph.close(); + graph = open(); + assertEquals("the cleared data must not return", 0, countOf(graph.vertices())); + + // nor return beside what is committed after the clear, should the graph crash before closing + graph.addVertex(T.id, 1); + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); + graph = open(); + graph.clear(); + graph.addVertex(T.id, 3); + graph.tx().commit(); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + graph = open(); + assertEquals(Collections.singletonList(3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldClearAStoreRightAfterACompaction() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.compact(); + graph.clear(); + graph.close(); + + graph = open(); + assertEquals("the cleared data must not return", 0, countOf(graph.vertices())); + graph.close(); + } + + @Test + public void shouldKeepAClearedGraphClearedWithoutACommit() { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.clear(); + graph.tx().rollback(); + graph.close(); + + graph = open(); + assertEquals("clear() cannot be rolled back", 0, countOf(graph.vertices())); + graph.close(); + } + + @Test + public void shouldNotPersistChangesThatWereOpenWhenTheGraphWasCleared() throws Exception { + TinkerStorageGraph graph = open(); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // another thread changed an element and has not committed when the graph is cleared + final CountDownLatch changed = new CountDownLatch(1); + final CountDownLatch cleared = new CountDownLatch(1); + final AtomicReference otherCommit = new AtomicReference<>(); + final TinkerStorageGraph g = graph; + final Thread other = new Thread(() -> { + g.vertices(1).next().property("name", "late"); + g.addVertex(T.id, 5); + changed.countDown(); + try { + cleared.await(); + g.tx().commit(); + } catch (Throwable t) { + otherCommit.set(t); + g.tx().rollback(); + } + }); + other.start(); + changed.await(); + + // and so has this thread + graph.addVertex(T.id, 2); + graph.clear(); + graph.tx().commit(); + cleared.countDown(); + other.join(10_000); + assertTrue("the other thread's commit should conflict: " + otherCommit.get(), + otherCommit.get() instanceof TransactionException); + assertEquals(0, countOf(graph.vertices())); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final Map crash = captureStorageFiles(location); + graph.close(); + restoreStorageFiles(location, crash); + + graph = open(); + assertEquals("nothing open at clear() may reach the store", 0, countOf(graph.vertices())); + graph.close(); + } + + @Test + public void shouldRefuseToClearAClosedGraph() { + final TinkerStorageGraph graph = open(); + graph.close(); + try { + graph.clear(); + fail("a closed graph should refuse to clear"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("closed")); + } + } + + private static List vertexIds(final TinkerStorageGraph graph) { + final List ids = new ArrayList<>(); + graph.vertices().forEachRemaining(v -> ids.add(((Number) v.id()).intValue())); + Collections.sort(ids); + return ids; + } + + @Test + public void shouldRestoreIndexDefinitionsOnReopen() throws Exception { + TinkerStorageGraph graph = open(); + graph.createIndex("name", Vertex.class); + graph.createIndex("weight", Edge.class); + final Vertex marko = graph.addVertex(T.id, 1, "name", "marko"); + final Vertex josh = graph.addVertex(T.id, 2, "name", "josh"); + marko.addEdge("knows", josh, T.id, 10, "weight", 0.5d); + graph.tx().commit(); + graph.close(); + + graph = open(); + try { + // the definitions come back, and the index covers data written before the restart rather than only + // writes that follow it + assertEquals(Collections.singleton("name"), graph.getIndexedKeys(Vertex.class)); + assertEquals(Collections.singleton("weight"), graph.getIndexedKeys(Edge.class)); + assertEquals(1L, (long) graph.traversal().V().has("name", "josh").count().next()); + assertEquals(1L, (long) graph.traversal().E().has("weight", 0.5d).count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotRestoreADroppedIndex() throws Exception { + TinkerStorageGraph graph = open(); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.dropIndex("name", Vertex.class); + graph.close(); + + graph = open(); + try { + assertTrue("a dropped index must not come back", graph.getIndexedKeys(Vertex.class).isEmpty()); + // data is unaffected by the index being gone; the lookup just falls back to a scan + assertEquals(1L, (long) graph.traversal().V().has("name", "marko").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldOpenWithoutIndexesWhenDefinitionsAreUnreadable() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.close(); + + // a corrupt sidecar must degrade to the behaviour before definitions were recorded at all, never block a + // store whose data is perfectly readable + Files.write(new File(location, IndexDefinitions.INDEX_FILE).toPath(), + "this is not an index definition".getBytes(StandardCharsets.UTF_8)); + + graph = open(); + try { + assertTrue(graph.getIndexedKeys(Vertex.class).isEmpty()); + assertEquals(1L, (long) graph.traversal().V().has("name", "marko").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldRoundTripIndexKeysNeedingEscapes() throws Exception { + final String awkward = "a\tb\nc\\d"; + TinkerStorageGraph graph = open(); + graph.createIndex(awkward, Vertex.class); + graph.addVertex(T.id, 1, awkward, "value"); + graph.tx().commit(); + graph.close(); + + graph = open(); + try { + assertEquals(Collections.singleton(awkward), graph.getIndexedKeys(Vertex.class)); + assertEquals(1L, (long) graph.traversal().V().has(awkward, "value").count().next()); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotRecordIndexesAfterClose() throws Exception { + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.createIndex("name", Vertex.class); + graph.close(); + final File indexFile = new File(location, IndexDefinitions.INDEX_FILE); + final byte[] recorded = Files.readAllBytes(indexFile.toPath()); + + // close() released the directory, which another graph may now own, so a closed graph must not write to it, + // and it refuses the change rather than making one it cannot record + for (final Runnable change : Arrays.asList(() -> graph.createIndex("age", Vertex.class), + () -> graph.dropIndex("name", Vertex.class))) { + try { + change.run(); + fail("a closed graph should refuse an index change"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("closed")); + } + } + assertArrayEquals(recorded, Files.readAllBytes(indexFile.toPath())); + + final TinkerStorageGraph reopened = open(); + try { + assertEquals(Collections.singleton("name"), reopened.getIndexedKeys(Vertex.class)); + } finally { + reopened.close(); + } + } + + @Test + public void shouldNotRecordIndexesWithoutAStorageEngine() { + // a TinkerStorageGraph with no engine is transactional but in-memory, so it must touch no disk at all + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + assertEquals(Collections.singleton("name"), graph.getIndexedKeys(Vertex.class)); + } finally { + graph.close(); + } + } + + @Test + public void shouldFailOnCorruptFrameWithBadCrc() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1, "value", 1); + graph.tx().commit(); + graph.addVertex(T.id, 2, "value", 2); + graph.tx().commit(); + graph.tx().close(); + + // preserve the raw log, then reconstruct it with a bit flipped inside the first frame's payload — a complete + // frame whose CRC no longer matches (distinct from a short trailing frame, which is tolerated as truncation) + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] log = Files.readAllBytes(logFile.toPath()); + graph.close(); + // header, then the first frame's length, length CRC and payload CRC, then payload — flip the first payload byte + final int firstPayloadByte = GraphBinaryStorage.HEADER_SIZE + GraphBinaryStorage.LOG_FRAME_HEADER_SIZE; + log[firstPayloadByte] ^= 0x01; + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(logFile.toPath(), log); + + try { + open(); + fail("expected reopen to fail on a CRC mismatch"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("CRC mismatch")); + } + } + + @Test + public void shouldFailOnStringLongerThanItsFrame() throws Exception { + // a dictionary-append entry declaring a ~2GB string in a record holding no such bytes. The declared length + // must be checked before the array is allocated, otherwise this is an OutOfMemoryError rather than a + // reportable corrupt frame. + final byte[] payload = new byte[] { + 0x01, // entry count = 1 + 0x05, // OP_DICT_APPEND + 0x00, // dictionary id = 0 + (byte) 0xF0, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, 0x07 // varint length = 0x7FFFFFF0 + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on a string longer than its frame"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("string of")); + } + } + + @Test + public void shouldFailOnALabelListLongerThanItsFrame() throws Exception { + // a vertex record whose id is itself a vertex value, whose label list declares 0x7FFFFFFF entries. The + // vertex serializer reads that list with its type already known, so the size must be checked there too, + // otherwise this is an OutOfMemoryError that even a recovery open could not get past. + final byte[] payload = new byte[] { + 0x01, // entry count = 1 + 0x01, // OP_PUT_VERTEX + 0x11, // id: GraphBinary VERTEX tag + 0x01, 0x00, 0x00, 0x00, 0x00, 0x2A, // its id: fully-qualified INT 42 + 0x7F, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF // its label list's size + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on a label list longer than its frame"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("LIST value declares a size of 2147483647")); + } + } + + @Test + public void shouldFailOnOverLongVarInt() throws Exception { + // a varint whose continuation bits run past the width of an int. Java masks a shift count to five bits, so + // without a bound this wraps and yields an arbitrary, possibly negative, value instead of failing. + final byte[] payload = new byte[] { + (byte) 0x80, (byte) 0x80, (byte) 0x80, (byte) 0x80, + (byte) 0x80, (byte) 0x80, (byte) 0x80, 0x00 + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on an over-long varint"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("over-long varint")); + } + } + + @Test + public void shouldFailOnDictionaryRefWithNoSuchEntry() throws Exception { + // a vertex record whose label names dictionary entry 5 when the dictionary is empty. The ref must be + // validated rather than dereferenced straight into the backing list. + final byte[] payload = new byte[] { + 0x01, // entry count = 1 + 0x01, // OP_PUT_VERTEX + 0x01, 0x00, 0x00, 0x00, 0x2A, // id: GraphBinary INT tag then 42 + 0x01, // label count = 1 + 0x05 // dictionary ref = 5, nothing defined + }; + + try { + openWithSyntheticFrame(payload); + fail("expected reopen to fail on an undefined dictionary ref"); + } catch (Exception expected) { + assertTrue("cause should report corruption: " + rootMessage(expected), + rootMessage(expected).contains("dictionary ref")); + } + } + + /** + * The store id recorded in the {@code VERSION} marker at {@code location}, which follows the magic and the version + * byte. + */ + private static byte[] storeId(final String location) throws IOException { + final byte[] marker = Files.readAllBytes(new File(location, GraphBinaryStorage.VERSION_FILE).toPath()); + final int start = GraphBinaryStorage.MAGIC.length + 1; + return Arrays.copyOfRange(marker, start, start + GraphBinaryStorage.STORE_ID_SIZE); + } + + /** + * Replace the store's log with a single well-formed frame (correct length prefix and checksums) carrying {@code + * payload}, then reopen. The framing must be valid so that replay reaches the codec rather than stopping at the + * frame checks, which are covered separately. + */ + private void openWithSyntheticFrame(final byte[] payload) throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.tx().close(); + graph.close(); + + final CRC32 lengthCrc = new CRC32(); + lengthCrc.update(ByteBuffer.allocate(Integer.BYTES).putInt(payload.length).array()); + final CRC32 crc = new CRC32(); + crc.update(ByteBuffer.allocate(Integer.BYTES).putInt(0).array()); // the first frame's sequence number + crc.update(payload); + final ByteBuffer frame = ByteBuffer.allocate( + GraphBinaryStorage.HEADER_SIZE + GraphBinaryStorage.LOG_FRAME_HEADER_SIZE + payload.length); + final ByteBuffer id = ByteBuffer.wrap(storeId(location)); + // generation 0, a log with no snapshot, in this store + frame.put(GraphBinaryStorage.header(0, new UUID(id.getLong(), id.getLong()))); + frame.putInt(payload.length); + frame.putInt((int) lengthCrc.getValue()); + frame.putInt(0); + frame.putInt((int) crc.getValue()); + frame.put(payload); + + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(new File(location, GraphBinaryStorage.LOG_FILE).toPath(), frame.array()); + + open().close(); + } + + @Test + public void shouldFailOnForeignFileWithBadMagic() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.tx().close(); + final File logFile = new File(location, GraphBinaryStorage.LOG_FILE); + final byte[] log = Files.readAllBytes(logFile.toPath()); + graph.close(); + + // corrupt the magic so the file no longer identifies as a TinkerGraph storage file + log[0] ^= 0xFF; + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(logFile.toPath(), log); + + try { + open(); + fail("expected reopen to fail on bad magic"); + } catch (Exception expected) { + assertTrue("cause should report a bad storage file: " + rootMessage(expected), + rootMessage(expected).contains("not a TinkerGraph storage file")); + } + } + + @Test + public void shouldFailOnStoreWithUnsupportedVersionMarker() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + // rewrite the store-level version marker to name a future, unsupported format version, checksummed as the + // TinkerPop that wrote it would have + final File versionFile = new File(location, GraphBinaryStorage.VERSION_FILE); + final byte[] marker = Files.readAllBytes(versionFile.toPath()); + marker[GraphBinaryStorage.MAGIC.length] = 99; // version byte follows the magic + ByteBuffer.wrap(marker).putInt(AbstractLogStorage.VERSION_SIZE - Integer.BYTES, + AbstractLogStorage.versionMarkerCrc(marker)); + Files.write(versionFile.toPath(), marker); + + try { + open(); + fail("expected reopen to fail on an unsupported store version"); + } catch (Exception expected) { + assertTrue("cause should report the unsupported version and migration path: " + rootMessage(expected), + rootMessage(expected).contains("Unsupported storage format version") + && rootMessage(expected).contains("g.io()")); + } + } + + @Test + public void shouldRegenerateVertexPropertyIdsByDefault() { + // default: vertex-property ids are not persisted; the property still round-trips (value + meta), the id is + // simply reassigned on load + TinkerStorageGraph graph = open(); + try { + final VertexProperty vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko"); + vp.property("since", 2010); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = open(); + try { + final VertexProperty vp = graph.vertices(1).next().properties("name").next(); + assertEquals("marko", vp.value()); + assertEquals(Integer.valueOf(2010), vp.property("since").value()); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotRewriteTheSnapshotWhenNothingWasCommitted() throws Exception { + TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final File snapshot = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + final byte[] compacted = Files.readAllBytes(snapshot.toPath()); + + // a session that only reads, and an explicit compact() with nothing new, leave the snapshot as it was, where a + // rewrite would at least have moved its header to the next generation + graph = open(); + assertEquals(1, countOf(graph.vertices())); + graph.tx().commit(); + graph.compact(); + graph.close(); + assertArrayEquals(compacted, Files.readAllBytes(snapshot.toPath())); + + graph = open(); + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); + assertFalse("a commit makes the next compaction rewrite it", + Arrays.equals(compacted, Files.readAllBytes(snapshot.toPath()))); + } + + @Test + public void shouldPreserveVertexPropertyIdsWhenConfigured() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, true); + final Object vpId; + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko"); + vpId = vp.id(); + graph.tx().commit(); + } finally { + graph.close(); + } + graph = TinkerStorageGraph.open(conf); + try { + final VertexProperty vp = graph.vertices(1).next().properties("name").next(); + assertEquals("marko", vp.value()); + assertEquals("vertex-property id should be preserved across reopen", vpId, vp.id()); + } finally { + graph.close(); + } + } + + @Test + public void shouldReplayDictionaryGrowthAcrossLogCommits() throws Exception { + // each commit introduces a new property key, so the dictionary grows via OP_DICT_APPEND across successive log + // frames. Reopening from a log with no snapshot must rebuild the dictionary incrementally and resolve all refs. + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); // keep the log, no auto-compaction + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + for (int i = 0; i < 10; i++) { + graph.addVertex(T.id, i, "key" + i, i); + graph.tx().commit(); + } + graph.tx().close(); + final byte[] log = Files.readAllBytes(new File(location, GraphBinaryStorage.LOG_FILE).toPath()); + graph.close(); + + // restore a log-only store (no snapshot) and reopen + Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath()); + Files.write(new File(location, GraphBinaryStorage.LOG_FILE).toPath(), log); + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i)); + } finally { + graph.close(); + } + } + + @Test + public void shouldRewriteDictionaryOnCompactionAndReopen() { + TinkerStorageGraph graph = open(); + try { + for (int i = 0; i < 10; i++) + graph.addVertex(T.id, i, "key" + i, i); + graph.tx().commit(); + graph.compact(); // writes a fresh full-dictionary snapshot header, then element frames + } finally { + graph.close(); + } + graph = open(); + try { + assertEquals(10, countOf(graph.vertices())); + for (int i = 0; i < 10; i++) + assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i)); + } finally { + graph.close(); + } + } + + @Test + public void shouldNotGrowDictionaryAcrossReopenCycles() throws Exception { + // With committed data and its key vocabulary held constant, repeated reopen+compact cycles must not grow the + // store. The dictionary numbering is rebuilt on replay, so a write session after a reopen resumes it rather + // than re-appending every live key as a duplicate. Before that fix the snapshot grew by the vocabulary size + // on every cycle. + final Configuration conf = config(); + final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + final File snapshot = new File(location, GraphBinaryStorage.SNAPSHOT_FILE); + + // seed a single vertex whose keys (person/name/age) are the entire vocabulary, then fold to a snapshot + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1, T.label, "person", "name", "marko", "age", 29); + graph.tx().commit(); + graph.compact(); + graph.close(); + final long seededSize = snapshot.length(); + + // reopen and re-compact repeatedly without changing the data; the snapshot must stay byte-for-byte the same size + for (int cycle = 0; cycle < 6; cycle++) { + graph = TinkerStorageGraph.open(conf); + graph.compact(); + final long size = snapshot.length(); + graph.close(); + assertEquals("snapshot grew on reopen+compact cycle " + cycle + " with unchanged data (dictionary bloat)", + seededSize, size); + } + + // and the data is still intact + graph = TinkerStorageGraph.open(conf); + try { + assertEquals(1, countOf(graph.vertices())); + assertEquals("marko", graph.vertices(1).next().value("name")); + } finally { + graph.close(); + } + } + + @Test + public void shouldStoreFewBytesPerElement() { + // regression guard: the dictionary-encoded format must stay well under the ~168 bytes/element the old + // whole-object format cost for a comparable graph (3 vertex props, 2 edge props, E=V). + final int vertexCount = 200; + final TinkerStorageGraph graph = open(); + final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY); + try { + for (int i = 0; i < vertexCount; i++) + graph.addVertex(T.id, i, "name", "v" + i, "age", i % 100, "score", i * 1.5d); + for (int i = 0; i < vertexCount; i++) + graph.vertices(i).next().addEdge("knows", graph.vertices((i + 1) % vertexCount).next(), + T.id, 1_000_000 + i, "weight", i * 0.5d, "count", i % 7); + graph.tx().commit(); + graph.compact(); + final long bytes = new File(location, GraphBinaryStorage.SNAPSHOT_FILE).length(); + final double perElement = (double) bytes / (2 * vertexCount); + assertTrue("expected well under 168 bytes/element (whole-object baseline), got " + perElement, perElement < 100.0); + } finally { + graph.close(); + } + } + + // --- decode-error paths ----------------------------------------------------------------------------------------- + // A frame that clears CRC framing can still be internally malformed (a bug in an older writer, or a targeted flip + // that happens to keep the checksum valid). The codec must reject each such frame with a clear IOException rather + // than a silent wrong answer or an opaque runtime crash. These drive decodeFrame directly with hand-built frames; + // values below 128 are single-byte varints, so the frames are written as literal bytes. + + private static final byte OP_DEL_VERTEX = 2; + private static final byte OP_DICT_APPEND = 5; + + /** Decode one frame payload against a fresh codec, returning nothing — the maps are throwaway. */ + private static void decode(final GraphBinaryStorage codec, final byte[] frame) throws IOException { + final Map vertices = new HashMap<>(); + final Map edges = new HashMap<>(); + codec.decodeFrame(frame, vertices, edges); + } + + @Test + public void shouldRejectValuesThatDeclareMoreThanTheirRecordHolds() { + // each is one OP_PUT_VERTEX whose id is a value declaring a size its few-byte record cannot hold. The + // GraphBinary serializers allocate from that size, so unchecked these raise OutOfMemoryError or + // NegativeArraySizeException rather than reporting a corrupt frame. + final byte[][] payloads = new byte[][] { + { 1, 1, 0x03, 0x7F, (byte) 0xFF, (byte) 0xFF, (byte) 0xF0 }, // string of ~2GB + { 1, 1, 0x03, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF }, // string of -1 bytes + { 1, 1, 0x09, 0x7F, (byte) 0xFF, (byte) 0xFF, (byte) 0xF0 }, // list of ~2G elements + { 1, 1, 0x0A, 0x10, 0x00, 0x00, 0x00 }, // map of 268M entries + { 1, 1, 0x25, 0x7F, (byte) 0xFF, (byte) 0xFF, (byte) 0xF0 }, // binary of ~2GB + { 1, 1, 0x09, 0, 0, 0, 1, 0x03, 0x00, 0x7F, (byte) 0xFF, (byte) 0xFF, (byte) 0xF0 } // nested string + }; + for (final byte[] payload : payloads) { + try { + decode(new GraphBinaryStorage(), payload); + fail("expected an IOException for " + Arrays.toString(payload)); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("declares a size of")); + } + } + } + + @Test + public void shouldRejectABulkedList() { + // a list holding one bulked list, a form the engine never writes and whose bulk count drives an unbounded loop + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, 1, 0x09, 0, 0, 0, 1, 0x09, 0x02, 0, 0, 0, 1 }); + fail("expected an IOException for a bulked list"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("bulked list")); + } + } + + @Test + public void shouldReportAValueCutOffByTheEndOfItsRecordAsCorruption() { + // an int id with two of its four bytes + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, 1, 0x01, 0x00, 0x00 }); + fail("expected an IOException for a truncated value"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("unreadable INT value")); + } + } + + @Test + public void shouldRejectFrameWithUnknownOpCode() { + // entryCount=1, op=99 (no such op) + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, 99 }); + fail("expected an IOException for an unknown op code"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Unknown storage op code")); + } + } + + @Test + public void shouldRejectDictionaryAppendGap() { + // entryCount=1, OP_DICT_APPEND, id=5 into an empty dictionary (expected next id is 0) -> gap + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DICT_APPEND, 5, 1, (byte) 'x' }); + fail("expected an IOException for a dictionary append gap"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("dictionary append gap")); + } + } + + @Test + public void shouldRejectDictionaryRedefinition() { + // append id 0 = "a", then re-append id 0 = "b" on the same codec -> redefinition (a mismatch, not an + // idempotent re-append of the same string) + final GraphBinaryStorage codec = new GraphBinaryStorage(); + try { + decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'a' }); + decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'b' }); + fail("expected an IOException for a redefined dictionary id"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("redefined")); + } + } + + @Test + public void shouldRejectUnknownValueTypeCode() { + // OP_DEL_VERTEX reads a scalar id; feed it a value type code (0xFF) no serializer claims. Without a guard this + // is an opaque NullPointerException; the codec must surface it as corruption instead. + try { + decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DEL_VERTEX, (byte) 0xFF }); + fail("expected an IOException for an unknown value type code"); + } catch (final IOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("unknown value type code")); + } + } + + private static String rootMessage(final Throwable t) { + Throwable cur = t; + while (cur.getCause() != null && cur.getCause() != cur) + cur = cur.getCause(); + return String.valueOf(cur.getMessage()); + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java new file mode 100644 index 00000000000..1749b0ea7ea --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCommitSerializationTest.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * Deterministically verifies that {@link TinkerStorageGraph} serializes the durable write of concurrent commits. + * Because TinkerGraph transactions lock only their own changed elements, commits touching disjoint elements run their + * commit paths concurrently; the storage engine's single append log would interleave (corrupt) without the + * commit-write lock. Rather than rely on a race actually corrupting the log, this drives commits through + * {@link ConcurrencyProbeStorage}, which records whether two threads are ever inside {@code persist()} at once. With + * the lock that count is exactly zero; without it the probe trips reliably. + */ +public class StorageCommitSerializationTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + ConcurrencyProbeStorage.reset(); + } + + private TinkerStorageGraph open() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, ConcurrencyProbeStorage.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return TinkerStorageGraph.open(conf); + } + + @Test + public void shouldSerializeConcurrentCommitWrites() throws Exception { + final int threads = 8; + final int commitsPerThread = 20; + final TinkerStorageGraph graph = open(); + try { + final ExecutorService pool = Executors.newFixedThreadPool(threads); + final CountDownLatch start = new CountDownLatch(1); + final List> futures = new ArrayList<>(); + for (int t = 0; t < threads; t++) { + final int threadId = t; + futures.add(pool.submit(() -> { + start.await(); + for (int i = 0; i < commitsPerThread; i++) { + graph.addVertex(T.id, threadId * commitsPerThread + i); + graph.tx().commit(); + } + return null; + })); + } + start.countDown(); + for (final Future f : futures) + f.get(60, TimeUnit.SECONDS); + pool.shutdown(); + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS)); + } finally { + graph.close(); + } + + // the lock must have kept persist() strictly single-threaded + assertEquals("commits entered persist() concurrently", 0, ConcurrencyProbeStorage.concurrentEntries.get()); + assertEquals("more than one thread was inside persist() at once", 1, ConcurrencyProbeStorage.maxObserved.get()); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java new file mode 100644 index 00000000000..5cab45a837d --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java @@ -0,0 +1,231 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Iterator; + +import static org.junit.Assert.assertEquals; + +/** + * Crash-consistency tests for {@link GraphBinaryStorage}. Rather than kill a JVM mid-operation — which is slow and + * non-deterministic — these reconstruct the exact on-disk states a crash would leave at each step of the two durable + * sequences (the write-ahead commit and compaction) and assert that reopening recovers the correct graph. The + * invariant under test: at no step may a crash leave the store unable to recover the last committed state. + */ +public class StorageCrashConsistencyTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + private File snapshotFile; + private File logFile; + private File tmpSnapshotFile; + + @Before + public void setUp() throws Exception { + final File dir = tempFolder.newFolder("storage"); + location = dir.getAbsolutePath(); + snapshotFile = new File(dir, GraphBinaryStorage.SNAPSHOT_FILE); + logFile = new File(dir, GraphBinaryStorage.LOG_FILE); + tmpSnapshotFile = new File(dir, GraphBinaryStorage.SNAPSHOT_FILE + ".tmp"); + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + // disable auto-compaction so tests control exactly when compaction happens + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); + return conf; + } + + private TinkerStorageGraph open() { + return TinkerStorageGraph.open(config()); + } + + /** + * Captured building blocks of valid on-disk files, used to assemble crash states: a snapshot holding {1}, a log + * holding a later commit of {2}, and a compacted snapshot holding {1,2}. + */ + private byte[] snapshotV1; + private byte[] logV2; + private byte[] snapshotV12; + + private void captureBuildingBlocks() throws IOException { + // snapshot holding vertex 1 (compaction on close folds the single commit into the snapshot) + TinkerStorageGraph g = open(); + g.addVertex(T.id, 1, "value", 1); + g.tx().commit(); + g.close(); + snapshotV1 = Files.readAllBytes(snapshotFile.toPath()); + + // a valid log holding a later commit of vertex 2, captured before the closing compaction folds it away + g = open(); // replays {1} from the snapshot + g.addVertex(T.id, 2, "value", 2); + g.tx().commit(); + g.tx().close(); + logV2 = Files.readAllBytes(logFile.toPath()); + g.close(); // compacts {1,2} into the snapshot + snapshotV12 = Files.readAllBytes(snapshotFile.toPath()); + + // reset the directory to a clean slate for the state under test + resetFiles(); + } + + private void resetFiles() throws IOException { + Files.deleteIfExists(snapshotFile.toPath()); + Files.deleteIfExists(logFile.toPath()); + Files.deleteIfExists(tmpSnapshotFile.toPath()); + } + + private void assertReopensWithTheOldLogReplacedTo(final int... expectedIds) { + final TinkerStorageGraph graph = open(); + try { + assertEquals("the old log is replaced by an empty one", AbstractLogStorage.HEADER_SIZE, logFile.length()); + assertEquals(expectedIds.length, countOf(graph.vertices())); + for (final int id : expectedIds) + assertEquals(Integer.valueOf(id), graph.vertices(id).next().value("value")); + } finally { + graph.close(); + } + } + + private void assertReopensTo(final int... expectedIds) { + final TinkerStorageGraph graph = open(); + try { + assertEquals(expectedIds.length, countOf(graph.vertices())); + for (final int id : expectedIds) + assertEquals(Integer.valueOf(id), graph.vertices(id).next().value("value")); + } finally { + graph.close(); + } + } + + @Test + public void shouldRecoverDurableCommitWithNoSnapshot() throws Exception { + // WAL guarantee: a commit durably written to the log with no compaction (no snapshot) recovers on reopen. On + // a fresh store the dictionary starts empty, so the commit frame is self-contained (it carries its own + // dictionary appends), which is exactly the real "log only, never compacted" case. + final TinkerStorageGraph g = open(); + g.addVertex(T.id, 2, "value", 2); + g.tx().commit(); + g.tx().close(); + final byte[] selfContainedLog = Files.readAllBytes(logFile.toPath()); + g.close(); // compaction on close would fold the log away; the raw log was captured above + resetFiles(); + Files.write(logFile.toPath(), selfContainedLog); + assertReopensTo(2); + } + + @Test + public void shouldRecoverFromSnapshotPlusLog() throws Exception { + // steady pre-compaction state: a snapshot holding earlier commits and a log holding later ones. Reopen must + // fold snapshot-then-log into the union. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV1); + Files.write(logFile.toPath(), logV2); + assertReopensTo(1, 2); + } + + @Test + public void shouldIgnoreStrayTempSnapshotFromCrashBeforeRename() throws Exception { + // crash after writing snapshot.gbin.tmp but before the atomic rename: the old snapshot + log are intact and + // the stray .tmp must be ignored, so the last committed state still recovers. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV1); + Files.write(logFile.toPath(), logV2); + Files.write(tmpSnapshotFile.toPath(), new byte[]{ 0x00, 0x01, 0x02, 0x03 }); // garbage half-written temp + assertReopensTo(1, 2); + } + + @Test + public void shouldUseTheNewSnapshotAloneWhenACompactionCrashedBeforeReplacingTheLog() throws Exception { + // crash after the rename installed the new snapshot but before the log was replaced: snapshot holds {1,2} and + // the old log still holds {2}. The snapshot is a generation ahead of the log, so the open completes the + // compaction by replacing the old log unread, and the snapshot alone gives exactly {1,2}. + captureBuildingBlocks(); + Files.write(snapshotFile.toPath(), snapshotV12); + Files.write(logFile.toPath(), logV2); + assertReopensWithTheOldLogReplacedTo(1, 2); + } + + @Test + public void shouldUseTheNewSnapshotAloneWhenACompactionWithADeadKeyCrashedBeforeReplacingTheLog() throws Exception { + // The same crash window with a snapshot whose dictionary holds a key ("alpha") that no element uses any more, + // beside a key ("gamma") added after it. The open replaces the old log unread, so the new snapshot alone must + // give back the surviving vertex with both of its keys. + final TinkerStorageGraph g = open(); + g.addVertex(T.id, 1, "alpha", 1); + g.tx().commit(); + g.addVertex(T.id, 2, "beta", 2); + g.tx().commit(); + g.compact(); // snapshot holds alpha and beta in the dictionary at stable ids + + g.vertices(1).next().remove(); // alpha becomes a dead key, still in the dictionary + g.tx().commit(); + g.vertices(2).next().property("gamma", "g"); // a key added after the dead one + g.tx().commit(); + final byte[] logNotYetDeleted = Files.readAllBytes(logFile.toPath()); + g.compact(); // new snapshot whose dictionary still holds alpha, then the log is replaced + final byte[] newSnapshot = Files.readAllBytes(snapshotFile.toPath()); + g.close(); + + // reconstruct the crash window: new snapshot in place, old log not yet replaced + Files.write(snapshotFile.toPath(), newSnapshot); + Files.write(logFile.toPath(), logNotYetDeleted); + + final TinkerStorageGraph reopened = open(); + try { + assertEquals("the old log is replaced by an empty one", AbstractLogStorage.HEADER_SIZE, logFile.length()); + assertEquals(1, countOf(reopened.vertices())); // only the surviving vertex 2 + assertEquals(0, countOf(reopened.vertices(1))); // alpha's vertex was deleted + final Vertex v = reopened.vertices(2).next(); + assertEquals(Integer.valueOf(2), v.value("beta")); + assertEquals("g", v.value("gamma")); + } finally { + reopened.close(); + } + } + + private static long countOf(final Iterator it) { + long count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFailureTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFailureTest.java new file mode 100644 index 00000000000..a97666448a1 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageFailureTest.java @@ -0,0 +1,778 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.commons.lang3.SystemUtils; +import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils; +import org.junit.After; +import org.junit.Assume; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * How a {@link TinkerStorageGraph} behaves when its storage engine fails while the process keeps running, using + * {@link FaultInjectingStorage} to make the log or compaction fail on demand. + */ +public class StorageFailureTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private Configuration conf; + + @Before + public void setUp() throws Exception { + FaultInjectingStorage.reset(); + conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, FaultInjectingStorage.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, + tempFolder.newFolder("storage").getAbsolutePath()); + } + + @After + public void tearDown() { + FaultInjectingStorage.reset(); + } + + @Test + public void shouldRollBackMemoryAndIndexWhenLogFlushFails() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "a", "value", 1); + graph.tx().commit(); + + FaultInjectingStorage.failLogWrites = true; + graph.vertices(1).next().property("value", 2); + graph.addVertex(T.id, 2, "name", "b"); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + // the same thread must not keep seeing the failed transaction's changes, and the index must not either + assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("value")); + assertEquals(1, IteratorUtils.count(graph.vertices())); + assertEquals(0L, (long) graph.traversal().V().has("name", "b").count().next()); + graph.tx().rollback(); + graph.close(); + } + + @Test + public void shouldKeepIndexesUnchangedWhenEncodeFails() { + final TinkerStorageGraph graph = openWithIndexedData(); + + // an unstorable value fails the encode, which fails only this commit and leaves the engine accepting more + changeIndexedValues(graph); + graph.vertices(2).next().property("unstorable", new Object()); + assertCommitFails(graph); + graph.tx().rollback(); + + assertIndexesHoldCommittedValues(graph); + + // a later successful commit must not expose the failed transaction's index changes either + graph.addVertex(T.id, 3, "name", "z"); + graph.tx().commit(); + assertIndexesHoldCommittedValues(graph); + graph.close(); + } + + @Test + public void shouldKeepIndexesUnchangedWhenLogFlushFails() { + final TinkerStorageGraph graph = openWithIndexedData(); + + FaultInjectingStorage.failLogWrites = true; + changeIndexedValues(graph); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + graph.tx().rollback(); + + // the engine now refuses commits, but the graph stays readable for export and its indexes must still agree + // with the committed data + assertIndexesHoldCommittedValues(graph); + graph.close(); + } + + @Test + public void shouldRefuseToCompactAClosedGraph() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.close(); + try { + graph.compact(); + fail("a closed graph should refuse to compact"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("closed")); + } + } + + @Test + public void shouldRejectACommitThatWaitedForClose() throws Exception { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // close() holds the commit lock while it compacts; a commit arriving then waits for the lock and, once it + // has it, must find the storage closed rather than write to a directory the graph no longer owns + FaultInjectingStorage.compactionEntered = new CountDownLatch(1); + FaultInjectingStorage.compactionGate = new CountDownLatch(1); + final Thread closer = new Thread(graph::close); + closer.start(); + assertTrue(FaultInjectingStorage.compactionEntered.await(10, TimeUnit.SECONDS)); + + final AtomicReference failure = new AtomicReference<>(); + final Thread committer = new Thread(() -> { + graph.addVertex(T.id, 2); + try { + graph.tx().commit(); + } catch (Throwable t) { + failure.set(t); + graph.tx().rollback(); + } + }); + committer.start(); + final long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (committer.getState() != Thread.State.WAITING && System.nanoTime() < deadline) + Thread.sleep(1); + assertEquals("the commit should be waiting on the commit lock", Thread.State.WAITING, committer.getState()); + + FaultInjectingStorage.compactionGate.countDown(); + closer.join(10_000); + committer.join(10_000); + assertTrue(String.valueOf(failure.get()), failure.get() instanceof IllegalStateException); + assertTrue(failure.get().getMessage(), failure.get().getMessage().contains("closed")); + + FaultInjectingStorage.reset(); + final TinkerStorageGraph reopened = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1), vertexIds(reopened)); + reopened.close(); + } + + @Test + public void shouldRefuseToChangeIndexesOnAClosedGraph() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.close(); + for (final Runnable change : Arrays.asList(() -> graph.createIndex("name", Vertex.class), + () -> graph.dropIndex("name", Vertex.class))) { + try { + change.run(); + fail("a closed graph should refuse an index change it can no longer record"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("closed")); + } + } + } + + @Test + public void shouldRefuseToCompactAfterAStorageFailure() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + FaultInjectingStorage.failLogWrites = true; + graph.addVertex(T.id, 1); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + try { + graph.compact(); + fail("a graph whose storage failed should refuse to compact rather than return as if it had"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("storage failure")); + } + graph.close(); + } + + @Test + public void shouldKeepIndexChangesInMemoryAfterAStorageFailure() throws Exception { + final File indexes = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), + IndexDefinitions.INDEX_FILE); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.createIndex("name", Vertex.class); + graph.addVertex(T.id, 1, "name", "a"); + graph.tx().commit(); + final byte[] recorded = Files.readAllBytes(indexes.toPath()); + + FaultInjectingStorage.failLogWrites = true; + graph.addVertex(T.id, 2); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + // an index still helps the reads an export does, but a failed engine writes nothing more to its directory + graph.createIndex("value", Vertex.class); + assertTrue(graph.getIndexedKeys(Vertex.class).contains("value")); + assertArrayEquals(recorded, Files.readAllBytes(indexes.toPath())); + graph.close(); + } + + @Test + public void shouldRefuseCommitsAfterLogFlushFailsAndRecoverOnReopen() { + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + FaultInjectingStorage.failLogWrites = true; + graph.addVertex(T.id, 2); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + // the disk works again, but the engine must not carry on: a later flush would also write the failed + // transaction's buffered frame and make it durable + graph.addVertex(T.id, 3); + assertCommitFails(graph); + + // reads stay possible, including committing the transaction they open, so the data can still be exported + assertEquals(Arrays.asList(1), vertexIds(graph)); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1), vertexIds(graph)); + graph.addVertex(T.id, 4); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1, 4), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldRefuseCommitsAfterLogAppendFails() { + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // a frame larger than the log's buffer reaches the file during the append itself, before any flush + FaultInjectingStorage.failLogWrites = true; + final char[] large = new char[64 * 1024]; + Arrays.fill(large, 'x'); + graph.addVertex(T.id, 2, "value", new String(large)); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + graph.addVertex(T.id, 3); + assertCommitFails(graph); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldKeepCommitWhenCompactionAfterItFails() { + // compact after every commit so the failure lands right after the transaction is applied + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 1); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + + FaultInjectingStorage.failCompaction = true; + graph.addVertex(T.id, 1); + graph.tx().commit(); + FaultInjectingStorage.failCompaction = false; + assertEquals(Arrays.asList(1), vertexIds(graph)); + + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldKeepACommitWhenAutomaticCompactionThrowsAnError() { + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 1024); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + + // the commit is durable and applied before compaction runs, so an Error from compaction is reported as itself + // rather than as a commit that could not be applied, and leaves the storage accepting commits + FaultInjectingStorage.failSnapshotWritesWithError = true; + graph.addVertex(T.id, 1, "name", String.join("", Collections.nCopies(2048, "x"))); + try { + graph.tx().commit(); + fail("the Error from compaction should reach the caller"); + } catch (OutOfMemoryError expected) { + assertEquals("injected snapshot write error", expected.getMessage()); + } + FaultInjectingStorage.failSnapshotWritesWithError = false; + assertEquals(Arrays.asList(1), vertexIds(graph)); + + // the failed compaction waits for another threshold's worth of log before it is tried again + final int attempts = FaultInjectingStorage.compactionAttempts.get(); + graph.addVertex(T.id, 2); + graph.tx().commit(); + assertEquals(attempts, FaultInjectingStorage.compactionAttempts.get()); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldStopCommitsWhenTheEmptyLogCannotBeInstalledAfterTheSnapshot() { + FaultInjectingStorage.failEmptyLogInstallWithIOException = true; + assertEmptyLogInstallFailureStopsCommits(); + } + + @Test + public void shouldStopCommitsWhenInstallingTheEmptyLogThrowsAfterTheSnapshot() { + FaultInjectingStorage.failEmptyLogInstall = new IllegalStateException("injected empty log install failure"); + assertEmptyLogInstallFailureStopsCommits(); + } + + /** + * Once a compaction's snapshot is renamed into place the old log can only stay as it is, since the next open + * replaces it unread, so a failure installing the new empty log must stop commits that would be appended to it. + */ + private void assertEmptyLogInstallFailureStopsCommits() { + // armed after the open, which installs the first empty log through the same path + final boolean ioFailure = FaultInjectingStorage.failEmptyLogInstallWithIOException; + final RuntimeException failure = FaultInjectingStorage.failEmptyLogInstall; + FaultInjectingStorage.reset(); + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + FaultInjectingStorage.failEmptyLogInstallWithIOException = ioFailure; + FaultInjectingStorage.failEmptyLogInstall = failure; + try { + graph.compact(); + fail("compaction should fail when its empty log cannot be installed"); + } catch (RuntimeException expected) { + final Throwable injected = null == expected.getCause() ? expected : expected.getCause(); + assertEquals("injected empty log install failure", injected.getMessage()); + } + FaultInjectingStorage.reset(); + + graph.addVertex(T.id, 2); + assertCommitFails(graph); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals("the snapshot holds every acknowledged commit", Arrays.asList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldRefuseCommitsWhenClearCannotEmptyTheStore() { + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + FaultInjectingStorage.failCompaction = true; + try { + graph.clear(); + fail("clear() should report that the store could not be emptied"); + } catch (RuntimeException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Could not clear storage")); + } + FaultInjectingStorage.failCompaction = false; + + // the cleared data is still on disk, so a commit now would mix it with new data on reopen + graph.addVertex(T.id, 2); + assertCommitFails(graph); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldRefuseToClearAfterAStorageFailure() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + FaultInjectingStorage.failLogWrites = true; + graph.addVertex(T.id, 2); + assertCommitFails(graph); + FaultInjectingStorage.failLogWrites = false; + + try { + graph.clear(); + fail("a graph whose storage failed should refuse to clear"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("g.io()")); + } + assertEquals("the data must stay readable for export", Arrays.asList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldKeepAcceptingCommitsWhenATransactionCannotBeEncoded() { + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // a value with no serializer fails before anything is written, so the log is still sound + graph.addVertex(T.id, 2, "value", new Object()); + assertCommitFails(graph); + assertEquals(Arrays.asList(1), vertexIds(graph)); + + graph.addVertex(T.id, 3); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1, 3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldCloseStorageWhenCompactionFailsOnClose() { + TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // the log still holds every acknowledged commit, so a compaction failure must not keep the graph from closing + FaultInjectingStorage.failCompaction = true; + graph.close(); + FaultInjectingStorage.failCompaction = false; + + // the closed graph must not write to a directory whose lock it has released + graph.addVertex(T.id, 2); + assertCommitFails(graph); + + final TinkerStorageGraph reopened = TinkerStorageGraph.open(conf); + assertEquals(Arrays.asList(1), vertexIds(reopened)); + reopened.close(); + } + + @Test + public void shouldNotRetryAFailedCompactionOnEveryCommit() { + // a threshold well above one commit's frame, so that backing off by a threshold spans several commits + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 512); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + FaultInjectingStorage.failCompaction = true; + int id = 0; + while (FaultInjectingStorage.compactionAttempts.get() == 0 && id < 1000) { + graph.addVertex(T.id, ++id); + graph.tx().commit(); + } + assertEquals(1, FaultInjectingStorage.compactionAttempts.get()); + + // the log is still over the threshold, but each compaction rewrites the whole graph under the commit lock, + // so after a failure the next attempt waits for another threshold's worth of log + for (int i = 0; i < 3; i++) { + graph.addVertex(T.id, ++id); + graph.tx().commit(); + } + assertEquals(1, FaultInjectingStorage.compactionAttempts.get()); + + FaultInjectingStorage.failCompaction = false; + while (FaultInjectingStorage.compactionAttempts.get() == 1 && id < 2000) { + graph.addVertex(T.id, ++id); + graph.tx().commit(); + } + assertEquals(2, FaultInjectingStorage.compactionAttempts.get()); + assertTrue(new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), + AbstractLogStorage.SNAPSHOT_FILE).exists()); + graph.close(); + } + + @Test + public void shouldRemoveThePartialSnapshotWhenCompactionFails() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + FaultInjectingStorage.failSnapshotWrites = true; + try { + graph.compact(); + fail("compaction should have failed"); + } catch (RuntimeException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Could not write storage snapshot")); + } + FaultInjectingStorage.failSnapshotWrites = false; + + final File directory = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + assertFalse("a failed compaction must not leave its partial snapshot behind", + new File(directory, AbstractLogStorage.SNAPSHOT_FILE + ".tmp").exists()); + graph.close(); + } + + @Test + public void shouldStopCommittingWhenTheStorageDirectoryIsRemoved() throws Exception { + assumeNotWindows(); + final File directory = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + deleteRecursively(directory); + + // the open log would otherwise keep accepting writes to a file that no longer has a name + graph.addVertex(T.id, 2); + try { + graph.tx().commit(); + fail("a commit after the storage directory is removed should fail"); + } catch (RuntimeException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("g.io()")); + } + graph.addVertex(T.id, 3); + assertCommitFails(graph); + + // what was committed before the removal is still readable, so it can be exported + assertEquals(Arrays.asList(1), vertexIds(graph)); + graph.close(); + assertFalse("close() must not bring the removed directory back", directory.exists()); + } + + @Test + public void shouldNotRecreateARemovedStorageDirectoryOnClose() throws Exception { + assumeNotWindows(); + final File directory = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.compact(); + + // no commit runs between the removal and close(), so only close() itself can notice + deleteRecursively(directory); + graph.close(); + assertFalse("close() must not bring the removed directory back", directory.exists()); + } + + @Test + public void shouldStopCommittingWhenTheLogIsReplaced() throws Exception { + assumeNotWindows(); + final File directory = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + final File log = new File(directory, AbstractLogStorage.LOG_FILE); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + + // a different file now sits at the log's path, so the open log is no longer the store's + final File copy = new File(directory, "copy"); + Files.copy(log.toPath(), copy.toPath()); + Files.move(copy.toPath(), log.toPath(), StandardCopyOption.REPLACE_EXISTING); + + graph.addVertex(T.id, 2); + assertCommitFails(graph); + graph.close(); + } + + @Test + public void shouldStopCommittingWhenTheLogIsReplacedBeforeTheFirstCommit() throws Exception { + final File log = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), + AbstractLogStorage.LOG_FILE); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + + // the log is not held open until the first commit, so this is when only its recorded identity can tell + final byte[] replacement = replaceWithCopy(log); + graph.addVertex(T.id, 1); + assertCommitFailsWith(graph, "was replaced by another file"); + graph.close(); + assertArrayEquals("nothing may be appended to the file that replaced the log", replacement, + Files.readAllBytes(log.toPath())); + } + + @Test + public void shouldStopCommittingWhenTheLogIsReplacedAfterACompaction() throws Exception { + final File log = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), + AbstractLogStorage.LOG_FILE); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.compact(); + + final byte[] replacement = replaceWithCopy(log); + graph.addVertex(T.id, 2); + assertCommitFailsWith(graph, "was replaced by another file"); + graph.close(); + assertArrayEquals("nothing may be appended to the file that replaced the log", replacement, + Files.readAllBytes(log.toPath())); + } + + @Test + public void shouldStopCommittingWhenTheLockIsRemovedAndAnotherGraphOpensTheStore() throws Exception { + assumeNotWindows(); + final File lock = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), + DirectoryLock.LOCK_FILE); + final TinkerStorageGraph first = TinkerStorageGraph.open(conf); + first.addVertex(T.id, 1); + first.tx().commit(); + + // with its LOCK file gone, nothing keeps a graph in another process from locking a new one and opening the + // store; a second graph in this JVM is refused before it reaches the file + assertTrue(lock.delete()); + final String java = new File(new File(System.getProperty("java.home"), "bin"), "java").getAbsolutePath(); + final Process second = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"), + StoreWriter.class.getName(), FaultInjectingStorage.class.getName(), + conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY), "3") + .redirectErrorStream(true).start(); + try { + final BufferedReader out = new BufferedReader( + new InputStreamReader(second.getInputStream(), StandardCharsets.UTF_8)); + assertEquals("the other process should commit to the store", "committed", out.readLine()); + + first.addVertex(T.id, 2); + assertCommitFailsWith(first, "its lock"); + first.close(); + } finally { + second.getOutputStream().close(); + assertTrue("the other process should close its graph", second.waitFor(30, TimeUnit.SECONDS)); + } + + final TinkerStorageGraph reopened = TinkerStorageGraph.open(conf); + assertEquals("only commits that were acknowledged survive", new HashSet<>(Arrays.asList(1, 3)), + reopened.traversal().V().id().toSet()); + reopened.close(); + } + + @Test + public void shouldNotCompactOnCloseAfterTheLogIsRemoved() throws Exception { + final File directory = new File(conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY)); + final File log = new File(directory, AbstractLogStorage.LOG_FILE); + final File snapshot = new File(directory, AbstractLogStorage.SNAPSHOT_FILE); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.compact(); + final byte[] snapshotBefore = Files.readAllBytes(snapshot.toPath()); + + // no commit runs between the removal and close(), so only close() itself can notice + assertTrue(log.delete()); + graph.close(); + assertFalse("close() must not recreate a removed log", log.exists()); + assertArrayEquals("close() must not compact into a store whose log was removed", snapshotBefore, + Files.readAllBytes(snapshot.toPath())); + } + + /** + * Put a different file with the same bytes at {@code file}'s path, as another graph recreating the store or a + * restore from backup would, and return those bytes. + */ + private static byte[] replaceWithCopy(final File file) throws IOException { + final File copy = new File(file.getParentFile(), "copy"); + Files.copy(file.toPath(), copy.toPath()); + Files.move(copy.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING); + return Files.readAllBytes(file.toPath()); + } + + private static void assertCommitFailsWith(final TinkerStorageGraph graph, final String fragment) { + try { + graph.tx().commit(); + fail("commit should have failed"); + } catch (RuntimeException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(fragment)); + } + } + + /** + * Windows refuses to delete or replace a file another handle holds open, which these tests do to a store whose + * lock and log are open, as an operator could on other platforms. + */ + private static void assumeNotWindows() { + Assume.assumeFalse("Windows cannot remove or replace files that are open", SystemUtils.IS_OS_WINDOWS); + } + + private static void deleteRecursively(final File file) { + final File[] children = file.listFiles(); + if (children != null) { + for (final File child : children) + deleteRecursively(child); + } + assertTrue("could not delete " + file, file.delete()); + } + + /** + * Open a graph with a vertex index on "name" and an edge index on "weight", holding vertex 1 named "a", vertex 2 + * named "b" and an edge 10 between them with weight 1. + */ + private TinkerStorageGraph openWithIndexedData() { + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + graph.createIndex("name", Vertex.class); + graph.createIndex("weight", Edge.class); + final Vertex a = graph.addVertex(T.id, 1, "name", "a"); + final Vertex b = graph.addVertex(T.id, 2, "name", "b"); + a.addEdge("knows", b, T.id, 10, "weight", 1); + graph.tx().commit(); + return graph; + } + + private static void changeIndexedValues(final TinkerStorageGraph graph) { + graph.vertices(1).next().property("name", "c"); + graph.edges(10).next().property("weight", 2); + } + + /** + * Assert that indexed lookups find exactly the committed values from {@link #openWithIndexedData()}, as a scan of + * the graph does, and nothing from a failed change. + */ + private static void assertIndexesHoldCommittedValues(final TinkerStorageGraph graph) { + final GraphTraversalSource g = graph.traversal(); + assertEquals("a", graph.vertices(1).next().value("name")); + assertEquals(Integer.valueOf(1), graph.edges(10).next().value("weight")); + assertEquals(Collections.singletonList(1), g.V().has("name", "a").id().toList()); + assertEquals(Collections.emptyList(), g.V().has("name", "c").id().toList()); + assertEquals(Collections.singletonList(10), g.E().has("weight", 1).id().toList()); + assertEquals(Collections.emptyList(), g.E().has("weight", 2).id().toList()); + } + + private static void assertCommitFails(final TinkerStorageGraph graph) { + try { + graph.tx().commit(); + fail("commit should have failed"); + } catch (RuntimeException expected) { + // a storage refusal or failure, not an unrelated bug such as a NullPointerException + assertTrue(expected.toString(), expected instanceof UncheckedIOException || + expected instanceof IllegalStateException); + } + } + + private static List vertexIds(final TinkerStorageGraph graph) { + final List ids = new ArrayList<>(); + graph.vertices().forEachRemaining(v -> ids.add(((Number) v.id()).intValue())); + Collections.sort(ids); + return ids; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageRecoveryTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageRecoveryTest.java new file mode 100644 index 00000000000..254a5bf1370 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageRecoveryTest.java @@ -0,0 +1,895 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.process.traversal.IO; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Assume; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.ByteBuffer; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; +import java.util.UUID; +import java.util.zip.CRC32; +import java.util.function.UnaryOperator; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Opening a damaged {@code graphbinary} store with {@code gremlin.tinkergraph.storage.recover}: what is recovered, + * what is left out, and that the store on disk is never changed. + */ +public class StorageRecoveryTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + @Test + public void shouldRecoverTheLogUpToAFrameWithABadChecksum() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final List frames = logFrames(log); + log[frames.get(1)[2]] ^= 0x01; // flip a payload byte of the second commit + writeStore(null, log); + + assertNormalOpenFails(); + final Map before = directoryContents(); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + assertReadOnly(graph); + graph.createIndex("name", Vertex.class); + graph.close(); + + assertEquals("recovery must not change the store", before, directoryContents()); + assertNormalOpenFails(); + } + + @Test + public void shouldRecoverFromADirectoryItCannotWrite() throws Exception { + crashedLogOfThreeCommits(); + final File directory = new File(location); + // a read-only copy or mount of a damaged store is a natural place to recover it from + for (final boolean keepLockFile : new boolean[]{ true, false }) { + final File lockFile = new File(directory, DirectoryLock.LOCK_FILE); + if (!keepLockFile) + Files.deleteIfExists(lockFile.toPath()); + else if (!lockFile.exists()) + Files.createFile(lockFile.toPath()); + assertTrue(directory.setWritable(false)); + try { + Assume.assumeFalse("the process can still write the directory, as root can", directory.canWrite()); + final Map before = directoryContents(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + assertEquals("recovery must not change the store", before, directoryContents()); + } finally { + assertTrue(directory.setWritable(true)); + } + } + } + + @Test + public void shouldPointANormalOpenOfADirectoryItCannotWriteToRecovery() throws Exception { + crashedLogOfThreeCommits(); + final File directory = new File(location); + Files.deleteIfExists(new File(directory, DirectoryLock.LOCK_FILE).toPath()); + assertTrue(directory.setWritable(false)); + try { + Assume.assumeFalse("the process can still write the directory, as root can", directory.canWrite()); + TinkerStorageGraph.open(config()).close(); + fail("a store in a directory that cannot be written must not open normally"); + } catch (UncheckedIOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains( + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } finally { + assertTrue(directory.setWritable(true)); + } + } + + @Test + public void shouldCutALastRecordWhosePayloadReadsBackAsZeros() throws Exception { + final byte[] log = crashedLogEndingInALargeCommit(); + // the last record's header reached the device but the sectors at the end of its payload did not, reading back + // as zeros from a sector boundary on + Arrays.fill(log, lastSectorOf(log), log.length, (byte) 0); + writeStore(null, log); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + assertEquals("the torn last commit was never acknowledged, so it is cut", Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldFailOnALastRecordThatEndsInZerosStartingPastASectorBoundary() throws Exception { + final byte[] log = crashedLogEndingInALargeCommit(); + // zeros at the end of the payload that do not reach back to a sector boundary were not left by the device + Arrays.fill(log, lastSectorOf(log) + 1, log.length, (byte) 0); + log[lastSectorOf(log)] = (byte) 0x7F; + writeStore(null, log); + + assertNormalOpenFails(); + } + + @Test + public void shouldFailOnADamagedLastRecordThatEndsInAZeroByte() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final int[] last = logFrames(log).get(2); + // a vertex whose last property has no meta-properties ends its record in a zero count, so a flipped byte + // elsewhere in it leaves a record that fails its checksum and still ends in zero + assertEquals("the record ends in a zero byte as written", 0, log[log.length - 1]); + log[last[2] + last[1] / 2] ^= 0x01; + writeStore(null, log); + + assertNormalOpenFails(); + } + + @Test + public void shouldStillFailOnALastRecordThatIsDamagedButDoesNotEndInZero() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final int[] last = logFrames(log).get(2); + // a flipped byte in the middle of the last record, which still ends as it was written + log[last[2] + last[1] / 2] ^= 0x01; + log[log.length - 1] = (byte) 0x7F; + writeStore(null, log); + + assertNormalOpenFails(); + } + + @Test + public void shouldClearARecoveredGraphInMemoryOnly() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + log[logFrames(log).get(1)[2]] ^= 0x01; + writeStore(null, log); + final Map before = directoryContents(); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + graph.clear(); + assertEquals(Collections.emptyList(), vertexIds(graph)); + graph.close(); + + assertEquals("clearing a recovered graph must not change the store", before, directoryContents()); + } + + @Test + public void shouldNotTruncateTheLogAtACorruptedFrameLength() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + // a flipped high bit makes the second commit's length claim far more bytes than remain, which without a + // checksum on the length would read as an interrupted append and cut the second and third commits off + log[logFrames(log).get(1)[0]] ^= 0x40; + writeStore(null, log); + + final Map before = directoryContents(); + assertNormalOpenFails(); + assertEquals("a failed open must not change the store", before, directoryContents()); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldFailWhenTheLogIsMissing() throws Exception { + crashAfterCommitOnCompactedStore(); + Files.delete(logFile().toPath()); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals("recovery reads the snapshot", Collections.singletonList(1), vertexIds(graph)); + graph.close(); + assertTrue("recovery must not recreate the log", !logFile().exists()); + } + + @Test + public void shouldFailWhenTheSnapshotAndLogAreBothMissing() throws Exception { + // what a store that lost its log before its first compaction is left with, since until then the log is all + // it has + crashedLogOfThreeCommits(); + writeStore(null, null); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a store that lost its only log should not open as an empty graph"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("is missing and there is no snapshot")); + } + assertTrue("a failed open must not create a log", !logFile().exists()); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.emptyList(), vertexIds(graph)); + graph.close(); + assertTrue("recovery must not create a log", !logFile().exists()); + } + + @Test + public void shouldFailWhenTheSnapshotIsMissing() throws Exception { + crashAfterCommitOnCompactedStore(); + Files.delete(snapshotFile().toPath()); + + // the log's records refer to the snapshot's dictionary, so not even recovery can read them + for (final Configuration conf : Arrays.asList(config(), recoverConfig())) { + try { + TinkerStorageGraph.open(conf).close(); + fail("a log whose snapshot is gone should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("snapshot")); + } + } + } + + @Test + public void shouldFailWhenTheVersionMarkerIsMissing() throws Exception { + crashAfterCommitOnCompactedStore(); + Files.delete(new File(location, AbstractLogStorage.VERSION_FILE).toPath()); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + assertTrue("recovery must not write the marker", !new File(location, AbstractLogStorage.VERSION_FILE).exists()); + } + + @Test + public void shouldRecoverPastATruncatedVersionMarker() throws Exception { + assertRecoversPastADamagedVersionMarker(marker -> Arrays.copyOf(marker, marker.length - 3)); + } + + @Test + public void shouldRecoverPastAVersionMarkerWithAFlippedMagicByte() throws Exception { + assertRecoversPastADamagedVersionMarker(marker -> { + marker[0] ^= 0x01; + return marker; + }); + } + + @Test + public void shouldRecoverPastAVersionMarkerWithAFlippedStoreIdByte() throws Exception { + assertRecoversPastADamagedVersionMarker(marker -> { + marker[AbstractLogStorage.MAGIC.length + 1] ^= 0x01; + return marker; + }); + } + + @Test + public void shouldRecoverTheLogOfAStoreWithNoSnapshotPastADamagedVersionMarker() throws Exception { + // with no snapshot the store id the log carries is all there is, and the log must still be read + final byte[] log = crashedLogOfThreeCommits(); + writeStore(null, log); + damageVersionMarker(marker -> { + marker[AbstractLogStorage.MAGIC.length + 1] ^= 0x01; + return marker; + }); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldNotReadALogFromAnotherStoreWhenTheVersionMarkerIsMissing() throws Exception { + crashAfterCommitOnCompactedStore(); + Files.write(logFile().toPath(), withForeignStoreId(Files.readAllBytes(logFile().toPath()))); + Files.delete(new File(location, AbstractLogStorage.VERSION_FILE).toPath()); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals("the snapshot alone is read", Arrays.asList(1), vertexIds(graph)); + graph.close(); + } + + private void assertRecoversPastADamagedVersionMarker(final UnaryOperator damage) throws Exception { + crashAfterCommitOnCompactedStore(); + damageVersionMarker(damage); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a damaged version marker must fail a normal open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("is damaged")); + assertTrue(expected.getMessage(), expected.getMessage().contains( + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } + final Map before = directoryContents(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + assertEquals("recovery must not change the store", before, directoryContents()); + } + + private void damageVersionMarker(final UnaryOperator damage) throws IOException { + final File versionFile = new File(location, AbstractLogStorage.VERSION_FILE); + Files.write(versionFile.toPath(), damage.apply(Files.readAllBytes(versionFile.toPath()))); + } + + @Test + public void shouldFailWhenTheLogBelongsToAnotherSnapshot() throws Exception { + crashAfterCommitOnCompactedStore(); + final byte[] log = Files.readAllBytes(logFile().toPath()); + ByteBuffer.wrap(log).putLong(AbstractLogStorage.MAGIC.length, 7); + Files.write(logFile().toPath(), resign(log)); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals("recovery reads only the snapshot", Collections.singletonList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldFailWhenTheLogBelongsToAnotherStore() throws Exception { + crashAfterCommitOnCompactedStore(); + // same generation as the snapshot, but stamped by a different store, as a log copied in from one would be + Files.write(logFile().toPath(), withForeignStoreId(Files.readAllBytes(logFile().toPath()))); + + final Map before = directoryContents(); + try { + TinkerStorageGraph.open(config()).close(); + fail("a log from another store should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("belongs to another store")); + } + assertEquals("a failed open must not change the store", before, directoryContents()); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals("recovery reads only the snapshot", Collections.singletonList(1), vertexIds(graph)); + graph.close(); + assertEquals("recovery must not change the store", before, directoryContents()); + } + + @Test + public void shouldFailWhenTheSnapshotBelongsToAnotherStore() throws Exception { + crashAfterCommitOnCompactedStore(); + Files.write(snapshotFile().toPath(), withForeignStoreId(Files.readAllBytes(snapshotFile().toPath()))); + + // none of the snapshot's data is this store's, and the log's records refer to this store's snapshot + for (final Configuration conf : Arrays.asList(config(), recoverConfig())) { + try { + TinkerStorageGraph.open(conf).close(); + fail("a snapshot from another store should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("belongs to another store")); + } + } + } + + @Test + public void shouldReportADamagedSnapshotHeaderRatherThanSetTheLogAside() throws Exception { + crashAfterCommitOnCompactedStore(); + final byte[] snapshot = Files.readAllBytes(snapshotFile().toPath()); + snapshot[AbstractLogStorage.MAGIC.length] ^= 0x01; // a flipped bit in the generation + Files.write(snapshotFile().toPath(), snapshot); + final Map before = directoryContents(); + + for (final Configuration conf : Arrays.asList(config(), recoverConfig())) { + try { + TinkerStorageGraph.open(conf).close(); + fail("a damaged snapshot header cannot be recovered from and must fail the open"); + } catch (UncheckedIOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("header is damaged")); + assertTrue(expected.getMessage(), expected.getMessage().contains(AbstractLogStorage.SNAPSHOT_FILE)); + } + } + assertEquals("the intact log is left in place", before, directoryContents()); + } + + @Test + public void shouldNameTheVersionMarkerWhenOnlyItBelongsToAnotherStore() throws Exception { + crashAfterCommitOnCompactedStore(); + final String other = tempFolder.newFolder("other").getAbsolutePath(); + final Configuration otherConfig = config(); + otherConfig.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, other); + TinkerStorageGraph.open(otherConfig).close(); + Files.copy(new File(other, AbstractLogStorage.VERSION_FILE).toPath(), + new File(location, AbstractLogStorage.VERSION_FILE).toPath(), StandardCopyOption.REPLACE_EXISTING); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a version marker from another store must fail the open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("version marker")); + assertTrue(expected.getMessage(), expected.getMessage().contains("remove it")); + } + } + + @Test + public void shouldCompleteACompactionInterruptedBeforeTheLogWasReplaced() throws Exception { + crashAfterCommitOnCompactedStore(); + final byte[] oldLog = Files.readAllBytes(logFile().toPath()); + + // close() compacts, moving the snapshot a generation ahead. Putting back the log it replaced leaves the store as + // a crash between the snapshot's rename and the log's would. + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.close(); + Files.write(logFile().toPath(), oldLog); + + graph = TinkerStorageGraph.open(config()); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + assertEquals("the old log is replaced by an empty one", AbstractLogStorage.HEADER_SIZE, logFile().length()); + graph.addVertex(T.id, 3); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(config()); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldNotReadTheOldLogOfAnInterruptedCompactionWhenRecovering() throws Exception { + crashAfterCommitOnCompactedStore(); + final byte[] oldLog = Files.readAllBytes(logFile().toPath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.close(); + + // the old log's transactions are all in the new snapshot, so even a damaged copy of it costs nothing + oldLog[logFrames(oldLog).get(0)[2]] ^= 0x01; + Files.write(logFile().toPath(), oldLog); + final Map before = directoryContents(); + + graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + assertEquals("recovery must not change the store", before, directoryContents()); + } + + @Test + public void shouldRecoverTheLogUpToAFrameThatCannotBeDecoded() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + // a well-formed frame with a valid checksum whose single entry has an unknown op code + writeStore(null, replacePayload(log, logFrames(log).get(1), new byte[]{ 1, 99 })); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldNotKeepPartOfATransactionThatFailsToDecode() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.addVertex(T.id, 2); + graph.addVertex(T.id, 3); + graph.tx().commit(); + final byte[] log = Files.readAllBytes(logFile().toPath()); + graph.close(); + + // drop the last byte of the second commit's payload, so its first vertex decodes and its second does not + final int[] second = logFrames(log).get(1); + final byte[] payload = Arrays.copyOfRange(log, second[2], second[2] + second[1] - 1); + writeStore(null, replacePayload(log, second, payload)); + + graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals("the damaged transaction must be left out entirely", Collections.singletonList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldFailWhenALogRecordIsRepeated() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final int[] first = logFrames(log).get(0); + final int firstLength = first[2] + first[1] - first[0]; + + // an intact copy of the first commit's record appended after the third: every checksum still passes + final byte[] repeated = Arrays.copyOf(log, log.length + firstLength); + System.arraycopy(log, first[0], repeated, log.length, firstLength); + writeStore(null, repeated); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a repeated log record must fail the open rather than replay"); + } catch (UncheckedIOException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("record 3 is numbered 0")); + } + + // recovery reads up to the record that is out of place + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldFailWhenLogRecordsAreReordered() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final int[] second = logFrames(log).get(1); + final int[] third = logFrames(log).get(2); + + // swap the last two records, each intact + final byte[] reordered = Arrays.copyOf(log, log.length); + final int secondLength = second[2] + second[1] - second[0]; + final int thirdLength = third[2] + third[1] - third[0]; + System.arraycopy(log, third[0], reordered, second[0], thirdLength); + System.arraycopy(log, second[0], reordered, second[0] + thirdLength, secondLength); + writeStore(null, reordered); + + assertNormalOpenFails(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldNotTruncateATornTailWhenRecovering() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + final byte[] torn = Arrays.copyOf(log, log.length + 6); + ByteBuffer.wrap(torn, log.length, 6).putInt(100); // a length prefix promising far more than follows + writeStore(null, torn); + + final TinkerStorageGraph graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2, 3), vertexIds(graph)); + graph.close(); + assertEquals("the torn tail is left in place", torn.length, logFile().length()); + } + + @Test + public void shouldSkipAnUnreadableSnapshotElementAndDropEdgesThatLostAnEndpoint() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + final Vertex v1 = graph.addVertex(T.id, 1); + final Vertex v2 = graph.addVertex(T.id, 2); + final Vertex v3 = graph.addVertex(T.id, 3); + v1.addEdge("knows", v2, T.id, 10); + v2.addEdge("knows", v3, T.id, 11); + v1.addEdge("knows", v3, T.id, 12); + graph.tx().commit(); + graph.close(); // compacts everything into the snapshot + + // frame 0 is the dictionary, frames 1 to 3 are the vertices (in no guaranteed order), then the edges + final byte[] snapshot = Files.readAllBytes(snapshotFile().toPath()); + snapshot[snapshotFrames(snapshot).get(2)[2]] ^= 0x01; + writeStore(snapshot, null); + + assertNormalOpenFails(); + graph = TinkerStorageGraph.open(recoverConfig()); + final List recovered = vertexIds(graph); + assertEquals(2, recovered.size()); + final List expectedEdges = new ArrayList<>(); + if (recovered.contains(1) && recovered.contains(2)) expectedEdges.add(10); + if (recovered.contains(2) && recovered.contains(3)) expectedEdges.add(11); + if (recovered.contains(1) && recovered.contains(3)) expectedEdges.add(12); + assertEquals("only edges whose endpoints both survived", expectedEdges, edgeIds(graph)); + assertEquals("no vertex is invented for a lost endpoint", 2, vertexIds(graph).size()); + graph.close(); + } + + @Test + public void shouldFailRecoveryWhenTheSnapshotDictionaryIsDamaged() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.close(); + + final byte[] snapshot = Files.readAllBytes(snapshotFile().toPath()); + snapshot[snapshotFrames(snapshot).get(0)[2]] ^= 0x01; + writeStore(snapshot, null); + + try { + TinkerStorageGraph.open(recoverConfig()); + fail("nothing can be decoded without the dictionary, so recovery should fail"); + } catch (Exception expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(AbstractLogStorage.SNAPSHOT_FILE)); + } + } + + @Test + public void shouldFailOnTrailingBytesInTheSnapshotUnlessRecovering() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.addVertex(T.id, 2); + graph.tx().commit(); + graph.close(); + + // a snapshot is written whole, so bytes that don't form a complete record are damage, not an interrupted append + final byte[] snapshot = Files.readAllBytes(snapshotFile().toPath()); + writeStore(Arrays.copyOf(snapshot, snapshot.length + 3), null); + + assertNormalOpenFails(); + graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + graph.close(); + } + + @Test + public void shouldExportARecoveredGraphWithIo() throws Exception { + final byte[] log = crashedLogOfThreeCommits(); + log[logFrames(log).get(2)[2]] ^= 0x01; + writeStore(null, log); + + final String export = new File(tempFolder.getRoot(), "export.json").getAbsolutePath(); + final TinkerStorageGraph recovered = TinkerStorageGraph.open(recoverConfig()); + recovered.traversal().io(export).with(IO.writer, IO.graphson).write().iterate(); + recovered.close(); + + final Configuration fresh = config(); + fresh.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, tempFolder.newFolder("fresh").getAbsolutePath()); + TinkerStorageGraph graph = TinkerStorageGraph.open(fresh); + graph.traversal().io(export).with(IO.reader, IO.graphson).read().iterate(); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(fresh); + assertEquals(Arrays.asList(1, 2), vertexIds(graph)); + assertEquals("marko", graph.vertices(1).next().value("name")); + graph.close(); + } + + @Test + public void shouldOpenAnUndamagedStoreReadOnlyWhenRecovering() { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(recoverConfig()); + assertEquals(Collections.singletonList(1), vertexIds(graph)); + assertReadOnly(graph); + graph.close(); + } + + /** + * Three commits, one vertex each, left as a log with no snapshot, as a crash before any compaction would leave. + */ + /** + * A log of three commits whose last one is large enough that its payload spans whole sectors. + */ + private byte[] crashedLogEndingInALargeCommit() throws IOException { + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "vadas"); + graph.tx().commit(); + graph.addVertex(T.id, 3, "name", String.join("", Collections.nCopies(4 * AbstractLogStorage.SECTOR_SIZE, "x"))); + graph.tx().commit(); + final byte[] log = Files.readAllBytes(logFile().toPath()); + graph.close(); + final int[] last = logFrames(log).get(2); + assertTrue("the last payload must start before the log's last sector", last[2] < lastSectorOf(log)); + return log; + } + + private static int lastSectorOf(final byte[] file) { + return (file.length - 1) / AbstractLogStorage.SECTOR_SIZE * AbstractLogStorage.SECTOR_SIZE; + } + + private byte[] crashedLogOfThreeCommits() throws IOException { + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1, "name", "marko"); + graph.tx().commit(); + graph.addVertex(T.id, 2, "name", "vadas"); + graph.tx().commit(); + graph.addVertex(T.id, 3, "name", "lop"); + graph.tx().commit(); + final byte[] log = Files.readAllBytes(logFile().toPath()); + graph.close(); + assertEquals(3, logFrames(log).size()); + return log; + } + + /** + * Leave the store as a crash would after one compaction and one more commit: a snapshot holding vertex 1 and a log + * of the same generation holding vertex 2. + */ + private void crashAfterCommitOnCompactedStore() throws IOException { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 2); + graph.tx().commit(); + final byte[] snapshot = Files.readAllBytes(snapshotFile().toPath()); + final byte[] log = Files.readAllBytes(logFile().toPath()); + graph.close(); + writeStore(snapshot, log); + } + + /** + * Copy a storage file with the store id in its header replaced by a random one, leaving its generation and records + * as they are. + */ + private static byte[] withForeignStoreId(final byte[] file) { + final byte[] copy = Arrays.copyOf(file, file.length); + final UUID other = UUID.randomUUID(); + ByteBuffer.wrap(copy, AbstractLogStorage.MAGIC.length + Long.BYTES, AbstractLogStorage.STORE_ID_SIZE) + .putLong(other.getMostSignificantBits()) + .putLong(other.getLeastSignificantBits()); + return resign(copy); + } + + /** + * Recompute the header checksum of a storage file whose header was edited, so that it reads as a genuine file of + * the generation and store its header now names rather than as a damaged one. + */ + private static byte[] resign(final byte[] file) { + final ByteBuffer buf = ByteBuffer.wrap(file); + final long generation = buf.getLong(AbstractLogStorage.MAGIC.length); + final UUID id = new UUID(buf.getLong(AbstractLogStorage.MAGIC.length + Long.BYTES), + buf.getLong(AbstractLogStorage.MAGIC.length + 2 * Long.BYTES)); + System.arraycopy(AbstractLogStorage.header(generation, id), 0, file, 0, AbstractLogStorage.HEADER_SIZE); + return file; + } + + private void writeStore(final byte[] snapshot, final byte[] log) throws IOException { + Files.deleteIfExists(snapshotFile().toPath()); + Files.deleteIfExists(logFile().toPath()); + if (snapshot != null) + Files.write(snapshotFile().toPath(), snapshot); + if (log != null) + Files.write(logFile().toPath(), log); + } + + private void assertNormalOpenFails() { + try { + TinkerStorageGraph.open(config()).close(); + fail("a normal open of the damaged store should fail"); + } catch (Exception expected) { + // a reported damage, not an unrelated bug such as a NullPointerException + assertTrue(expected.toString(), expected instanceof UncheckedIOException || + expected instanceof IllegalStateException); + } + } + + private static void assertReadOnly(final TinkerStorageGraph graph) { + // reads still work through a transaction that is committed, as Gremlin Server commits after every request + graph.vertices().forEachRemaining(v -> v.properties().forEachRemaining(p -> {})); + graph.tx().commit(); + + graph.addVertex(T.id, 99); + try { + graph.tx().commit(); + fail("a recovered graph should refuse commits"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage().contains(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } + assertTrue("the refused commit must not be visible", !graph.vertices(99).hasNext()); + graph.tx().rollback(); + try { + graph.compact(); + fail("a recovered graph should refuse to compact"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage().contains("read-only")); + } + } + + private static List logFrames(final byte[] log) { + return frames(log, AbstractLogStorage.LOG_FRAME_HEADER_SIZE); + } + + private static List snapshotFrames(final byte[] snapshot) { + return frames(snapshot, AbstractLogStorage.FRAME_HEADER_SIZE); + } + + /** + * The start offset, payload length and payload offset of each complete frame after the file header. + */ + private static List frames(final byte[] file, final int frameHeaderSize) { + final List frames = new ArrayList<>(); + final ByteBuffer buf = ByteBuffer.wrap(file); + int offset = AbstractLogStorage.HEADER_SIZE; + while (offset + frameHeaderSize <= file.length) { + final int length = buf.getInt(offset); + if (length < 0 || offset + frameHeaderSize + length > file.length) + break; + frames.add(new int[]{ offset, length, offset + frameHeaderSize }); + offset += frameHeaderSize + length; + } + return frames; + } + + /** + * Rebuild {@code log} with {@code frame}'s payload replaced, giving it a matching length, length checksum and + * payload checksum, and keeping its sequence number. + */ + private static byte[] replacePayload(final byte[] log, final int[] frame, final byte[] payload) { + final int frameEnd = frame[2] + frame[1]; + final int sequence = ByteBuffer.wrap(log).getInt(frame[0] + 2 * Integer.BYTES); + final CRC32 lengthCrc = new CRC32(); + lengthCrc.update(ByteBuffer.allocate(Integer.BYTES).putInt(payload.length).array()); + final CRC32 crc = new CRC32(); + crc.update(ByteBuffer.allocate(Integer.BYTES).putInt(sequence).array()); + crc.update(payload); + final ByteBuffer out = ByteBuffer.allocate(log.length - frame[1] + payload.length); + out.put(log, 0, frame[0]); + out.putInt(payload.length); + out.putInt((int) lengthCrc.getValue()); + out.putInt(sequence); + out.putInt((int) crc.getValue()); + out.put(payload); + out.put(log, frameEnd, log.length - frameEnd); + return out.array(); + } + + private Map directoryContents() throws IOException { + final Map contents = new TreeMap<>(); + final File[] files = new File(location).listFiles(); + if (files != null) { + for (final File f : files) { + if (f.getName().equals(DirectoryLock.LOCK_FILE)) continue; + final byte[] bytes = Files.readAllBytes(f.toPath()); + contents.put(f.getName(), bytes.length + ":" + Arrays.hashCode(bytes)); + } + } + return contents; + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + private Configuration recoverConfig() { + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + return conf; + } + + private File logFile() { + return new File(location, AbstractLogStorage.LOG_FILE); + } + + private File snapshotFile() { + return new File(location, AbstractLogStorage.SNAPSHOT_FILE); + } + + private static List vertexIds(final TinkerStorageGraph graph) { + final List ids = new ArrayList<>(); + graph.vertices().forEachRemaining(v -> ids.add(((Number) v.id()).intValue())); + Collections.sort(ids); + return ids; + } + + private static List edgeIds(final TinkerStorageGraph graph) { + final List ids = new ArrayList<>(); + graph.edges().forEachRemaining((Edge e) -> ids.add(((Number) e.id()).intValue())); + Collections.sort(ids); + return ids; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettingsTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettingsTest.java new file mode 100644 index 00000000000..b0b749ce4d8 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageSettingsTest.java @@ -0,0 +1,489 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.Graph; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Arrays; +import java.util.Collections; +import java.util.Iterator; +import java.util.Locale; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Tests for the settings a {@link TinkerStorageGraph} store records when it is created and holds fixed after that. + */ +public class StorageSettingsTest { + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private String location; + + @Before + public void setUp() throws Exception { + location = tempFolder.newFolder("storage").getAbsolutePath(); + } + + @Test + public void shouldAdoptTheRecordedSettingsWhenTheyAreNotConfigured() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.LONG.name()); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, + VertexProperty.Cardinality.list.name()); + TinkerStorageGraph graph = TinkerStorageGraph.open(created); + graph.addVertex(T.id, 1L, "name", "a", "name", "b"); + graph.tx().commit(); + graph.close(); + + // only the engine and directory: everything else comes from the store + graph = TinkerStorageGraph.open(config()); + assertEquals(TinkerGraph.DefaultIdManager.LONG.name(), + graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER)); + assertEquals("list cardinality keeps both values", 2, + countOf(graph.vertices(1L).next().properties("name"))); + final Vertex added = graph.addVertex(); + assertTrue("new ids come from the recorded LONG manager", added.id() instanceof Long); + graph.tx().rollback(); + graph.close(); + } + + @Test + public void shouldFailToOpenWithADifferentFixedSetting() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.LONG.name()); + TinkerStorageGraph.open(created).close(); + + final Configuration changed = config(); + changed.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.UUID.name()); + try { + TinkerStorageGraph.open(changed).close(); + fail("a store must not open with a different id manager than it was created with"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("vertexIdManager=LONG")); + assertTrue(expected.getMessage(), expected.getMessage().contains("g.io()")); + } + + // the failed open released the directory + TinkerStorageGraph.open(config()).close(); + } + + @Test + public void shouldTreatSpellingsOfTheSameSettingAlike() { + TinkerStorageGraph.open(config()).close(); + + final Configuration respelled = config(); + respelled.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "GRAPHBINARY"); + respelled.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, "FALSE"); + respelled.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.ANY.name()); + TinkerStorageGraph.open(respelled).close(); + } + + @Test + public void shouldAllowOperationalSettingsToChange() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + final byte[] recorded = Files.readAllBytes(settingsFile().toPath()); + + final Configuration changed = config(); + changed.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, "os"); + changed.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0); + graph = TinkerStorageGraph.open(changed); + assertEquals(1, countOf(graph.vertices())); + graph.close(); + + assertEquals("the record is written once and never rewritten", + new String(recorded), new String(Files.readAllBytes(settingsFile().toPath()))); + } + + @Test + public void shouldKeepPreservedVertexPropertyIdsWhenTheSettingIsLeftOut() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, true); + TinkerStorageGraph graph = TinkerStorageGraph.open(created); + graph.addVertex(T.id, 1).property("name", "marko", T.id, "vp-explicit"); + graph.tx().commit(); + graph.close(); + + // an open that leaves the setting out adopts the recorded one, so its close-time compaction keeps the ids + graph = TinkerStorageGraph.open(config()); + assertTrue(graph.configuration().getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS)); + graph.close(); + + graph = TinkerStorageGraph.open(config()); + assertEquals("vp-explicit", graph.vertices(1).next().properties("name").next().id()); + graph.close(); + } + + @Test + public void shouldFailToOpenWithADifferentPreserveVertexPropertyIdsSetting() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, true); + TinkerStorageGraph.open(created).close(); + + final Configuration changed = config(); + changed.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, false); + try { + TinkerStorageGraph.open(changed).close(); + fail("a store that preserves vertex-property ids must not open with the setting turned off"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("preserveVertexPropertyIds=true")); + } + + // spellings Configuration.getBoolean reads as true name the recorded setting + final Configuration yes = config(); + yes.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, "yes"); + TinkerStorageGraph.open(yes).close(); + } + + @Test + public void shouldOpenAStoreRegardlessOfTheDefaultLocale() { + // a Turkish default locale upper-cases the "i" of graphbinary to a dotted capital I, which names no engine + final Locale original = Locale.getDefault(); + Locale.setDefault(Locale.forLanguageTag("tr-TR")); + try { + TinkerStorageGraph.open(config()).close(); + TinkerStorageGraph.open(config()).close(); + } finally { + Locale.setDefault(original); + } + } + + @Test + public void shouldReadBackARecordedSettingThatIsNotLatin1() { + // the record is written as UTF-8, so it must be read as UTF-8 rather than the ISO-8859-1 Properties assumes + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, "com.example.\u03a9IdManager"); + final File directory = new File(location); + StorageSettings.reconcile(created, directory); + assertEquals("com.example.\u03a9IdManager", StorageSettings.reconcile(created, directory) + .getString(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER)); + } + + @Test + public void shouldNotModifyTheSuppliedConfiguration() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, TinkerGraph.DefaultIdManager.LONG.name()); + TinkerStorageGraph.open(created).close(); + + final Configuration reopened = config(); + TinkerStorageGraph.open(reopened).close(); + assertFalse(reopened.containsKey(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER)); + } + + @Test + public void shouldFailWhenTheSettingsRecordIsMissing() { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + assertTrue(settingsFile().delete()); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a store with data but no settings record should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(StorageSettings.SETTINGS_FILE)); + } + + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + graph = TinkerStorageGraph.open(recover); + assertEquals(1, countOf(graph.vertices())); + graph.close(); + assertFalse("recovery must not write the record", settingsFile().exists()); + } + + @Test + public void shouldFailWhenTheSettingsRecordLacksAFixedSetting() throws Exception { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "list"); + TinkerStorageGraph graph = TinkerStorageGraph.open(created); + graph.addVertex(T.id, 1, "a", 1, "a", 2); + graph.tx().commit(); + graph.close(); + + // a damaged record that lost a line must not silently fall back to the default cardinality, which would keep + // only the last value of each multi-property and write that back at the next compaction + final String key = TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY; + final StringBuilder kept = new StringBuilder(); + for (final String line : Files.readAllLines(settingsFile().toPath(), StandardCharsets.ISO_8859_1)) { + if (!line.startsWith(key)) + kept.append(line).append('\n'); + } + Files.write(settingsFile().toPath(), kept.toString().getBytes(StandardCharsets.ISO_8859_1)); + final byte[] damaged = Files.readAllBytes(settingsFile().toPath()); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a settings record missing a fixed setting should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("do not record " + key)); + } + + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + recover.setProperty(key, "list"); + graph = TinkerStorageGraph.open(recover); + assertEquals(2, countOf(graph.vertices(1).next().properties("a"))); + graph.close(); + assertTrue("recovery must not rewrite the record", + Arrays.equals(damaged, Files.readAllBytes(settingsFile().toPath()))); + } + + @Test + public void shouldFailWhenTheSettingsRecordAnInvalidIdManager() throws Exception { + assertDamagedRecordedValueFailsTheOpen(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, "LONF"); + } + + @Test + public void shouldFailWhenTheSettingsRecordAnInvalidCardinality() throws Exception { + assertDamagedRecordedValueFailsTheOpen(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "lisu"); + } + + @Test + public void shouldFailWhenTheSettingsRecordAnInvalidBoolean() throws Exception { + assertDamagedRecordedValueFailsTheOpen(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, "fals"); + } + + /** + * A damaged record whose value for {@code key} became {@code damaged} must fail a normal open naming the settings + * file, and a recover open must read the store with the configured value instead. + */ + private void assertDamagedRecordedValueFailsTheOpen(final String key, final String damaged) throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + + final StringBuilder edited = new StringBuilder(); + for (final String line : Files.readAllLines(settingsFile().toPath(), StandardCharsets.ISO_8859_1)) + edited.append(line.startsWith(key + "=") ? key + "=" + damaged : line).append('\n'); + Files.write(settingsFile().toPath(), edited.toString().getBytes(StandardCharsets.ISO_8859_1)); + final byte[] record = Files.readAllBytes(settingsFile().toPath()); + + try { + TinkerStorageGraph.open(config()).close(); + fail("a settings record with an unusable value should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(key + "=" + damaged)); + assertTrue(expected.getMessage(), expected.getMessage().contains(StorageSettings.SETTINGS_FILE)); + assertTrue(expected.getMessage(), expected.getMessage().contains( + TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER)); + } + + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + graph = TinkerStorageGraph.open(recover); + assertEquals(1, countOf(graph.vertices())); + graph.close(); + assertTrue("recovery must not rewrite the record", + Arrays.equals(record, Files.readAllBytes(settingsFile().toPath()))); + } + + @Test + public void shouldNotRecordAnUnusableSettingForANewStore() { + final Configuration wrong = config(); + wrong.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "LIST"); + try { + TinkerStorageGraph.open(wrong).close(); + fail("an unknown cardinality should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("must be one of")); + } + assertFalse("a failed first open must not record its settings", settingsFile().exists()); + + final Configuration wrongIds = config(); + wrongIds.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_VERTEX_ID_MANAGER, "long"); + try { + TinkerStorageGraph.open(wrongIds).close(); + fail("an unknown id manager should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("id manager with long")); + } + assertFalse("a failed first open must not record its settings", settingsFile().exists()); + + // the corrected value then creates the store as a first open + final Configuration right = config(); + right.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "list"); + TinkerStorageGraph.open(right).close(); + final TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + assertEquals("list", graph.configuration().getString( + TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY)); + graph.close(); + } + + @Test + public void shouldReadAllowNullPropertyValuesAsConfigurationDoes() { + final Configuration created = config(); + created.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, true); + TinkerStorageGraph.open(created).close(); + + // Configuration.getBoolean reads "yes" as true, so it names the recorded setting + final Configuration yes = config(); + yes.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, "yes"); + TinkerStorageGraph.open(yes).close(); + + final Configuration unreadable = config(); + unreadable.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, "maybe"); + try { + TinkerStorageGraph.open(unreadable).close(); + fail("a value that is neither true nor false must not match the record"); + } catch (IllegalStateException expected) { + assertTrue("the error should give the configured value: " + expected.getMessage(), + expected.getMessage().contains("cannot be opened with maybe")); + } + } + + @Test + public void shouldNameTheSettingsFileWhenItIsMalformed() throws Exception { + TinkerStorageGraph.open(config()).close(); + // a truncated unicode escape, which Properties.load rejects + Files.write(settingsFile().toPath(), + "gremlin.tinkergraph.vertexIdManager=\\u12".getBytes(StandardCharsets.ISO_8859_1)); + try { + TinkerStorageGraph.open(config()).close(); + fail("a malformed settings record should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(settingsFile().getPath())); + } + } + + @Test + public void shouldLeaveNoLockFileInADirectoryThatIsNotAStore() throws Exception { + final File foreign = new File(location); + Files.write(new File(foreign, "notes.txt").toPath(), "not a store".getBytes(StandardCharsets.UTF_8)); + try { + TinkerStorageGraph.open(config()).close(); + fail("a directory holding other files should not open as a store"); + } catch (IllegalStateException expected) { + // the directory holds no storage files, so pointing at recovery would mislead + assertTrue(expected.getMessage(), expected.getMessage().contains("is not empty and is not a store")); + } + assertEquals(Collections.singletonList("notes.txt"), Arrays.asList(foreign.list())); + } + + @Test + public void shouldSayWhyTheStorageDirectoryCannotBeCreated() throws Exception { + final File file = tempFolder.newFile("plain-file"); + final Configuration conf = config(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, new File(file, "store").getPath()); + try { + TinkerStorageGraph.open(conf).close(); + fail("a directory under a regular file cannot be created"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Could not create storage directory")); + assertTrue("the cause should say why: " + expected.getMessage(), null != expected.getCause()); + } + } + + @Test + public void shouldTrimTheStorageEngineName() { + final Configuration padded = config(); + padded.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, " graphbinary "); + TinkerStorageGraph.open(padded).close(); + } + + @Test + public void shouldListTheBuiltInEnginesForAnUnknownEngine() { + final Configuration unknown = config(); + unknown.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "rocksdb"); + try { + TinkerStorageGraph.open(unknown).close(); + fail("an unknown storage engine should not open"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("'rocksdb'")); + assertTrue(expected.getMessage(), expected.getMessage().contains("graphbinary")); + } + } + + @Test + public void shouldNotCreateADirectoryToRecover() throws Exception { + final File missing = new File(location, "missing"); + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, missing.getAbsolutePath()); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + try { + TinkerStorageGraph.open(recover).close(); + fail("recovery of a directory that does not exist should fail"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains(missing.getPath())); + } + assertFalse("recovery must not create the directory", missing.exists()); + } + + @Test + public void shouldNotLeaveALockFileAfterRecovery() throws Exception { + TinkerStorageGraph graph = TinkerStorageGraph.open(config()); + graph.addVertex(T.id, 1); + graph.tx().commit(); + graph.close(); + // a store copied without its LOCK file, as a backup might be + Files.delete(new File(location, DirectoryLock.LOCK_FILE).toPath()); + + final Configuration recover = config(); + recover.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_RECOVER, true); + graph = TinkerStorageGraph.open(recover); + assertEquals(1, countOf(graph.vertices())); + graph.close(); + assertFalse("recovery must leave the directory as it found it", + new File(location, DirectoryLock.LOCK_FILE).exists()); + } + + private Configuration config() { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(Graph.GRAPH, TinkerStorageGraph.class.getName()); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, "graphbinary"); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, location); + return conf; + } + + private File settingsFile() { + return new File(location, StorageSettings.SETTINGS_FILE); + } + + private static int countOf(final Iterator it) { + int count = 0; + while (it.hasNext()) { + it.next(); + count++; + } + return count; + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StoreWriter.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StoreWriter.java new file mode 100644 index 00000000000..af43cafd1f6 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StoreWriter.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.apache.commons.configuration2.BaseConfiguration; +import org.apache.commons.configuration2.Configuration; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph; +import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph; + +/** + * Opens a store from a separate JVM, as another process would, and commits one vertex to it. Prints {@code committed} + * once the commit is acknowledged and keeps the graph open until standard input closes. + */ +public final class StoreWriter { + + public static void main(final String[] args) throws Exception { + final Configuration conf = new BaseConfiguration(); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE, args[0]); + conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_DIRECTORY, args[1]); + final TinkerStorageGraph graph = TinkerStorageGraph.open(conf); + try { + graph.addVertex(T.id, Integer.parseInt(args[2])); + graph.tx().commit(); + System.out.println("committed"); + System.out.flush(); + while (System.in.read() != -1) { + // wait for the test to close standard input + } + } finally { + graph.close(); + } + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java new file mode 100644 index 00000000000..2af41ade1c1 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/SyncModeTest.java @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.junit.Test; + +import java.util.Locale; + +import static org.junit.Assert.assertEquals; + +public class SyncModeTest { + + @Test + public void shouldDefaultToCommitWhenUnset() { + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue(null)); + } + + @Test + public void shouldResolveCaseInsensitivelyAndTrim() { + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue("commit")); + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue("COMMIT")); + assertEquals(SyncMode.OS, SyncMode.fromConfigValue("os")); + assertEquals(SyncMode.OS, SyncMode.fromConfigValue(" Os ")); + } + + @Test + public void shouldResolveRegardlessOfTheDefaultLocale() { + // a Turkish default locale upper-cases "i" to a dotted capital I, which names no sync mode + final Locale original = Locale.getDefault(); + Locale.setDefault(Locale.forLanguageTag("tr-TR")); + try { + assertEquals(SyncMode.COMMIT, SyncMode.fromConfigValue("commit")); + } finally { + Locale.setDefault(original); + } + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectUnknownValue() { + SyncMode.fromConfigValue("interval:1000"); + } +} diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java new file mode 100644 index 00000000000..a6584e8d4f9 --- /dev/null +++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/TinkerByteBufferTest.java @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage; + +import org.junit.Test; + +import java.io.ByteArrayOutputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Unit tests for {@link TinkerByteBuffer}, the heap {@code byte[]} implementation of the gremlin-core + * {@code Buffer} abstraction that backs the GraphBinary storage codec. The byte order and bounds behaviour asserted + * here are relied on by the on-disk format, so a change that breaks them changes what a store means. + */ +public class TinkerByteBufferTest { + + @Test + public void shouldRoundTripEveryPrimitive() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeBoolean(true).writeBoolean(false) + .writeByte(0x7F).writeShort(-2) + .writeInt(Integer.MIN_VALUE).writeLong(Long.MAX_VALUE) + .writeFloat(0.5f).writeDouble(-1.25d); + + assertTrue(buf.readBoolean()); + assertFalse(buf.readBoolean()); + assertEquals(0x7F, buf.readByte()); + assertEquals((short) -2, buf.readShort()); + assertEquals(Integer.MIN_VALUE, buf.readInt()); + assertEquals(Long.MAX_VALUE, buf.readLong()); + assertEquals(0.5f, buf.readFloat(), 0.0f); + assertEquals(-1.25d, buf.readDouble(), 0.0d); + assertEquals(0, buf.readableBytes()); + } + + @Test + public void shouldWriteMultiByteValuesBigEndian() { + // the on-disk format is big-endian, matching the Netty-backed buffer this replaces, so the byte order is + // part of the storage contract rather than an implementation detail + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeShort(0x0102).writeInt(0x03040506).writeLong(0x0708090A0B0C0D0EL); + + assertArrayEquals(new byte[] { + 0x01, 0x02, + 0x03, 0x04, 0x05, 0x06, + 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E + }, buf.toWrittenArray()); + } + + @Test + public void shouldRefuseToGrowPastTheMaximumFrameSize() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1); + // asking for more than a frame can hold fails at once, before anything is allocated, rather than doubling the + // capacity until it overflows and loops forever + try { + buf.writerIndex(Integer.MAX_VALUE); + fail("a buffer must not grow past the maximum frame size"); + } catch (IllegalStateException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("smaller transactions")); + } + assertEquals("the buffer is unchanged", 4, buf.writerIndex()); + } + + @Test + public void shouldGrowFromAnEmptyArray() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[0]); + buf.writeLong(42L); + assertEquals(42L, buf.readLong()); + } + + @Test + public void shouldGrowBeyondInitialCapacity() { + final TinkerByteBuffer buf = new TinkerByteBuffer(1); + final byte[] payload = new byte[1000]; + Arrays.fill(payload, (byte) 7); + buf.writeBytes(payload); + + assertTrue("capacity should have grown to hold the payload", buf.capacity() >= 1000); + assertEquals(1000, buf.readableBytes()); + assertArrayEquals(payload, buf.toWrittenArray()); + } + + @Test + public void shouldWrapAnExistingArrayReadyForReading() { + final byte[] data = { 0x00, 0x00, 0x00, 0x2A }; + final TinkerByteBuffer buf = new TinkerByteBuffer(data); + + assertEquals(4, buf.writerIndex()); + assertEquals(0, buf.readerIndex()); + assertEquals(4, buf.readableBytes()); + assertEquals(42, buf.readInt()); + } + + @Test + public void shouldTrackReaderAndWriterIndexes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1).writeInt(2); + assertEquals(8, buf.writerIndex()); + assertEquals(8, buf.readableBytes()); + + buf.readInt(); + assertEquals(4, buf.readerIndex()); + assertEquals(4, buf.readableBytes()); + + buf.readerIndex(0); + assertEquals(1, buf.readInt()); + } + + @Test + public void shouldDistinguishReadableFromWrittenBytes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1).writeInt(2); + buf.readInt(); + + // written covers everything from index 0; readable covers only what is left ahead of the reader + assertArrayEquals(new byte[] { 0, 0, 0, 1, 0, 0, 0, 2 }, buf.toWrittenArray()); + assertArrayEquals(new byte[] { 0, 0, 0, 2 }, buf.toReadableArray()); + assertEquals("neither view may move the indexes", 4, buf.readerIndex()); + } + + @Test + public void shouldResetWriterIndexToTheMark() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1); + buf.markWriterIndex(); + buf.writeInt(2); + assertEquals(8, buf.writerIndex()); + + buf.resetWriterIndex(); + assertEquals(4, buf.writerIndex()); + assertArrayEquals(new byte[] { 0, 0, 0, 1 }, buf.toWrittenArray()); + } + + @Test + public void shouldRejectReadingPastTheWriterIndex() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeByte(1); + buf.readByte(); + + try { + buf.readByte(); + fail("expected a read past the writer index to be rejected"); + } catch (IndexOutOfBoundsException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Not enough readable bytes")); + } + } + + @Test + public void shouldRejectReadingMoreBytesThanRemain() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3 }); + try { + buf.readBytes(new byte[4]); + fail("expected a bulk read longer than the readable region to be rejected"); + } catch (IndexOutOfBoundsException expected) { + assertTrue(expected.getMessage(), expected.getMessage().contains("Not enough readable bytes")); + } + } + + @Test + public void shouldRejectAnOutOfRangeReaderIndex() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeInt(1); + try { + buf.readerIndex(5); + fail("expected a reader index beyond the writer index to be rejected"); + } catch (IndexOutOfBoundsException expected) { + // expected: the readable region may never extend past what has been written + } + } + + @Test + public void shouldReadBytesIntoAnArraySlice() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + final byte[] dst = new byte[6]; + buf.readBytes(dst, 1, 4); + + assertArrayEquals(new byte[] { 0, 1, 2, 3, 4, 0 }, dst); + assertEquals(0, buf.readableBytes()); + } + + @Test + public void shouldReadAndWriteThroughNioBuffers() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + buf.writeBytes(ByteBuffer.wrap(new byte[] { 9, 8, 7 })); + assertEquals(3, buf.readableBytes()); + + final ByteBuffer dst = ByteBuffer.allocate(3); + buf.readBytes(dst); + assertArrayEquals(new byte[] { 9, 8, 7 }, dst.array()); + } + + @Test + public void shouldReadBytesIntoAnOutputStream() throws Exception { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 4, 5, 6, 7 }); + final ByteArrayOutputStream out = new ByteArrayOutputStream(); + buf.readBytes(out, 3); + + assertArrayEquals(new byte[] { 4, 5, 6 }, out.toByteArray()); + assertEquals(1, buf.readableBytes()); + } + + @Test + public void shouldCopyAbsoluteBytesWithoutMovingIndexes() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + final byte[] dst = new byte[2]; + buf.getBytes(2, dst); + + assertArrayEquals(new byte[] { 3, 4 }, dst); + assertEquals("an absolute read is positional and must not consume", 0, buf.readerIndex()); + } + + @Test + public void shouldExposeAnNioViewOfTheReadableRegion() { + final TinkerByteBuffer buf = new TinkerByteBuffer(new byte[] { 1, 2, 3, 4 }); + buf.readByte(); + + assertEquals(1, buf.nioBufferCount()); + final ByteBuffer view = buf.nioBuffer(); + assertArrayEquals(new byte[] { 2, 3, 4 }, view.array()); + assertEquals("the view is a copy, so consuming it must not move the buffer", 3, buf.readableBytes()); + assertArrayEquals(new byte[] { 2, 3 }, buf.nioBuffer(1, 2).array()); + assertEquals(1, buf.nioBuffers().length); + } + + @Test + public void shouldCountReferences() { + final TinkerByteBuffer buf = new TinkerByteBuffer(); + assertEquals(1, buf.referenceCount()); + + buf.retain(); + assertEquals(2, buf.referenceCount()); + assertFalse("still referenced, so release does not report the last one", buf.release()); + assertEquals(1, buf.referenceCount()); + assertTrue("the final release reports that the buffer is done", buf.release()); + } + + @Test + public void shouldReportAsHeapBacked() { + assertFalse(new TinkerByteBuffer().isDirect()); + } +}