Repository navigation
perf: 多 shard 锁数组按 64B 对齐,修复 >8 线程吞吐断崖(10–20 线程 1.6–3.7×) - #15
Merged
carsontung666 merged 1 commit intoOct 31, 2026
Merged
Conversation
v0.1.7 的锁数组起点是 56 + 64*48 = 3128(mod 64 = 56),每把锁的锁字和 上一个 shard 的 hot_id1/hot_base 在同一条 cache line 上。home shard 按 步长 9 分配,8 线程以内恰好不相邻;从第 9 个线程起相邻 shard 同时有人 用,吞吐从 8 线程的 157 Mops/s 掉到 10 线程的 63 Mops/s。 - 多 shard 时锁数组起点对齐到 64B(相对 meta;mmap 得到的 meta 天然满足), 头部多 8 字节,sbo_meta_size / sbo_init 同步。 - sbo_meta_t 用原 _pad 的 1 字节记 layout;v0.1.7 创建的多 shard meta (layout 0)由 sbo_meta_ok 拒绝,不会被误读。 - 单 shard 布局逐字节不变(kvspace-c 依赖这一点)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
多 shard(数据区 ≥ 128MB,64 个 shard)时,v0.1.7 的锁数组从 meta 偏移
56 + 64×48 = 3128开始(mod 64 = 56)。sbo_lock_t虽然填充到 64 字节,但每把锁的锁字和上一个 shard 的hot_id1/hot_base落在同一条 cache line 上,而后者每次分配都会写。home shard 按
计数 × 9 & 63分配,8 个线程以内恰好互不相邻(0, 9, 18, …, 63)。从第 9 个线程起开始出现相邻 shard,吞吐断崖下跌:alloc+free 从 8 线程的 157 Mops/s 掉到 10 线程的 63 Mops/s。改动
sbo_meta_size/sbo_init同步。sbo_meta_t用原_pad中的 1 个字节记layout,sizeof仍是 56。v0.1.7 创建的多 shard meta(layout 0)由sbo_meta_ok拒绝(alloc 失败、free 不动作),不会被按新布局误读。SBO_HEAD_FIXED,所以不受影响。验证
正确性
18_model_check 16 50000PASS。SBO_HEAD_FIXED);64 shard 时lock[i] mod 64 = 0,最后一个 pool 不越过 head。perf/big-opt分支,out-of-tree 构建,KVSPACE_BUILD_TESTS=ON)用 main 头和本分支头各跑一遍 ctest,都是 11/11。性能
Xeon w7-3455(24 核 48 线程,机器同时有其他负载),gcc 13.2
-O3 -DNDEBUG。数据区 128MB / 64 shard,meta 16MB,每个线程绑一个物理核。main、本分支、main 副本(A/A)交替运行 15 次取中位数。{8,64,512,4096}B8 线程 alloc+free 那一格的 0.89 没能复现。换一组更空闲的核、每线程 400 万次操作再测,6/8 线程的本分支/main 是 0.95–1.17,而同一组的 A/A 是 0.98–1.30,四分位区间完全重叠。≤8 线程两边的布局没有伪共享差异,这里看到的是机器噪声。≥10 线程的提升远大于噪声,四分位区间不重叠(例如 10 线程 main [59, 69],本分支 [152, 204])。