You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] Bloom filter hash does not match Java for non-ASCII paths, silently dropping series #1077
The C++/Python reader can silently drop a whole time series when the device path contains non-ASCII characters. No error is reported: metadata APIs still list the series, but the query returns an all-NULL column (or omits it).
Root cause
Murmur128Hash differs between Java and C++ for the hash tail:
Java (java/.../utils/Murmur128Hash.java) uses ((long) key[i]) << shift on byte, which is signed and sign-extends bytes >= 0x80.
C++ (cpp/src/common/container/murmur_hash3.cc) uses ((uint64_t)(uint8_t)buf[i]) << shift, which zero-extends.
The two only diverge when the hashed bytes device + "." + measurement have a byte >= 0x80 in the tail (len % 16), i.e. exactly for multi-byte UTF-8 device path segments.
The bloom filter layout is persisted in the file. When C++ reads a Java-written file (or vice versa), it computes different bit positions, so BloomFilter::contains() returns false and TsFileIOReader::alloc_ssi() returns E_NO_MORE_DATA (cpp/src/file/tsfile_io_reader.cc:79). The series is then dropped without any error.
Reproduction
A Java-written tree-model file with device root.sensors.宿舍.TH and measurements humidity / temperature:
root.sensors.宿舍.TH.humidity (31 bytes, tail contains the bytes of 舍) -> 0 rows / all NULL
root.sensors.宿舍.TH.temperature -> reads correctly
Renaming the device path to ASCII (e.g. root.sensors.dorm.TH) and re-writing the same data makes both series readable. The reverse direction is symmetric: a C++-written file with such a path loses the series when read from Java.
Fix
Hash bloom path entries with Java semantics (signed tail) on both the write and read paths. No backward-compatibility shim: files written by older C++ releases with non-ASCII device paths are not readable by the fixed reader.
Activity
[C++] Drop legacy bloom hash fallback
[C++][Java] Cover non-ASCII device and measurement names in tests
[C++] Fix spotless formatting in non-ASCII path test
[C++] Match Java bloom filter hash for non-ASCII device paths (#1079)