Skip to content

[C++] Bloom filter hash does not match Java for non-ASCII paths, silently dropping series #1077

Closed

Description

@ColinLeeo

Summary

The C++/Python reader can silently drop a whole time series when the device path contains non-ASCII characters. No error is reported: metadata APIs still list the series, but the query returns an all-NULL column (or omits it).

Root cause

Murmur128Hash differs between Java and C++ for the hash tail:

  • Java (java/.../utils/Murmur128Hash.java) uses ((long) key[i]) << shift on byte, which is signed and sign-extends bytes >= 0x80.
  • C++ (cpp/src/common/container/murmur_hash3.cc) uses ((uint64_t)(uint8_t)buf[i]) << shift, which zero-extends.

The two only diverge when the hashed bytes device + "." + measurement have a byte >= 0x80 in the tail (len % 16), i.e. exactly for multi-byte UTF-8 device path segments.

The bloom filter layout is persisted in the file. When C++ reads a Java-written file (or vice versa), it computes different bit positions, so BloomFilter::contains() returns false and TsFileIOReader::alloc_ssi() returns E_NO_MORE_DATA (cpp/src/file/tsfile_io_reader.cc:79). The series is then dropped without any error.

Reproduction

A Java-written tree-model file with device root.sensors.宿舍.TH and measurements humidity / temperature:

  • root.sensors.宿舍.TH.humidity (31 bytes, tail contains the bytes of 舍) -> 0 rows / all NULL
  • root.sensors.宿舍.TH.temperature -> reads correctly

Renaming the device path to ASCII (e.g. root.sensors.dorm.TH) and re-writing the same data makes both series readable. The reverse direction is symmetric: a C++-written file with such a path loses the series when read from Java.

Fix

Hash bloom path entries with Java semantics (signed tail) on both the write and read paths. No backward-compatibility shim: files written by older C++ releases with non-ASCII device paths are not readable by the fixed reader.

PR: #1079

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions