[v-zhangxiaomeng5]

Summary

A GCC toolchain configured with --disable-tls compiles a thread_local or __thread variable into a call to __emutls_get_address(). The libgcc of such a toolchain is built with the single thread model, so it keeps one copy of each variable for the whole program and every thread reads and writes that same copy, which breaks the meaning of the keyword.

This adds libs/libc/tls/tls_emutls.c, which gives one copy per thread. A pthread key holds the copy and frees it when the thread exits. Because a pthread key belongs to one task group while the __emutls_object the compiler emits is a single global object shared by every task group of a flat build, the key is reached through two levels: obj->loc.offset holds a task local storage index allocated once for the variable, and the task local storage holds the pthread key of the task group, created on first use.

__emutls_register_common() is provided as well, with the same largest-wins merge as libgcc.

Impact

A thread_local variable now has one copy per thread on a toolchain without native TLS.

Known limitation: each variable costs one task local storage slot and one pthread key per task group, so the count is bounded by those two symbols, and a failed task_tls_alloc() is fatal because the caller the compiler emits cannot report an error.

Testing

Built on Ubuntu 22.04.5 (x86_64) with Arm GNU Toolchain 13.2.Rel1 (--disable-tls, the case this targets) and arm-none-eabi-gcc 10.3.1 (--enable-tls), NuttX master and nuttx-apps master.

lm3s6965-ek: libc.a contains tls_emutls.o exporting T __wrap___emutls_get_address and T __wrap___emutls_register_common, and both --wrap options appear in LDFLAGS and in build.ninja.

[github-actions]

MemBrowse Memory Report · Explore what changed →

No memory changes detected for:

• arduino-mega2560
• esp32-devkitc
• hifive1-revb
• mirtoo
• qemu-armv8a
• qemu-intel64
• rx65n-rsk2mb
• s698pm-dkit
• stm32-nucleo-f103rb

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[casaroli]

Thank you for this PR! A per-thread __emutls_get_address() is something I need too, and the --wrap idea is a nice way to make sure libc's copy is used in a flat image.

I tried it on master fc91db1 with qemu-armv8a (QEMU 11) and the Arm GNU Toolchain 14.2.Rel1 for aarch64-none-elf, which is built with --disable-tls, so __thread becomes calls to __emutls_get_address(). With large enough limits it works well: with TLS_NELEM=16 and TLS_TASK_NELEM=32, a program with 12 __thread variables in two threads passes. I found a few things that I think are worth a look. I hope the details help:

1. A kernel build does not compile. TLS_TASK_NELEM depends on !BUILD_KERNEL, so it is not defined there, and then ifneq ($(CONFIG_TLS_TASK_NELEM),0) in Make.defs and tools/Config.mk is true (and the CMake condition NOT CONFIG_TLS_TASK_NELEM EQUAL 0 too). tls_emutls.c is then compiled without the task TLS functions:

./tools/configure.sh qemu-armv8a:knsh
kconfig-tweak --set-val TLS_NELEM 4 && make olddefconfig && make
tls/tls_emutls.c:169:11: error: implicit declaration of function 'task_tls_get_value'
tls/tls_emutls.c:183:15: error: implicit declaration of function 'task_tls_set_value'
tls/tls_emutls.c:233:26: error: implicit declaration of function 'task_tls_alloc'

2. The sim link fails in CI. When the link goes through gcc or c++, as in the sim, the driver does not know --wrap=... without -Wl,. This is the Linux (sim-01) failure of this PR, sim:citest:

c++: error: unrecognized command-line option '--wrap=__emutls_get_address'
c++: error: unrecognized command-line option '--wrap=__emutls_register_common'

3. The task TLS slots can run out. Each variable takes one task TLS slot, and in a flat build the slots belong to the whole system and are not given back. With TLS_NELEM=4 and TLS_TASK_NELEM=4 on qemu-armv8a:nsh, a program with 5 __thread variables stops at an assertion:

nsh> tls5
dump_assert_info: Assertion failed : at file: tls/task_tls_destruct.c:109 task: tls5

The same happens across programs. Five small programs with one __thread variable each, run one after the other in the same boot: the first four pass and exit, and the fifth stops, because the first four still hold their slots:

tls1a: PASS
tls1b: PASS
tls1c: PASS
tls1d: PASS
dump_assert_info: Assertion failed : at file: tls/task_tls_destruct.c:109 task: tls1e
tls1a: PASS

(tls1a passes again at the end because it still has its slot.) I think a loaded ELF module would also take new slots each time it is loaded, because its control objects start at zero again, but I did not test that one.

4. The pthread keys can run out too. Each variable also takes one pthread key in each task group, from the same TLS_NELEM keys that the program uses for its own pthread_key_create(). With TLS_NELEM=4 and TLS_TASK_NELEM=32, the same tls5 program stops here:

nsh> tls5
dump_assert_info: Assertion failed panic: at file: tls/tls_emutls.c:179 task: tls5

This can come quickly in C++: libstdc++ built with thread support has thread_local variables of its own (std::__once_call, std::__once_callable).

One idea, in case it helps: one pthread key per task group that holds an array of the thread's copies, with an index per variable given out once from a global counter. That needs no task TLS slot and has no fixed limit, so it would also work in a kernel build.

Happy to test a new version!

Test programs (built-in apps, PROGNAME = tls5 tls1a tls1b tls1c tls1d tls1e)

/* tls5.c */
#include <pthread.h>
#include <stdio.h>

static __thread int a = 1, b = 2, c = 3, d = 4, e = 5;

static void *worker(void *arg)
{
  a = 10;
  return (void *)(long)(a + b + c + d + e);
}

int main(int argc, char **argv)
{
  pthread_t t;
  void *r;

  pthread_create(&t, NULL, worker, NULL);
  pthread_join(t, &r);
  printf("tls5: %s\n", (long)r == 24 && a == 1 ? "PASS" : "FAIL");
  return 0;
}

/* tls1a.c; tls1b.c to tls1e.c are the same with their own name */
#include <stdio.h>

static __thread int v = 1;

int main(int argc, char **argv)
{
  v++;
  printf("tls1a: %s\n", v == 2 ? "PASS" : "FAIL");
  return 0;
}

[v-zhangxiaomeng5]

@casaroli
Great thanks for the testing and reply. You're right for issues #3 & #4, your solution is better!

I will drop this one and apply your solution.