[nickakt]

kernelize_one() takes each sink's lock with a trylock, and kernelize() starts over at once when one is busy. kernelize() is the only place that holds the locks of two streams at once, and streams that are each other's sinks hold their own lock while they try the other's, so two kernelize() of one call start over against each other. After a change of subscriptions all of them are kernelized on their next packet, at the same moment. With about eight or more such streams the restarts do not end: the threads spin with the call's master lock held, the streams' sockets are no longer read, the log fills with "Kernelizing media stream" lines, and commands for the call are not answered.

A lock in the call, taken ahead of the stream's own lock, runs one kernelize() of the call at a time. A sink's trylock then only meets a holder of that single lock, which waits for nothing while it holds it, so the restart ends. Streams that are each other's sinks are in one call, as connect, inject and mesh merge their calls, and kernelize() of other calls does not wait.

A stream with more than RTPE_MAX_FORWARD_DESTINATIONS outputs, two per sink, was pushed to the kernel anyway: the module answered EINVAL, each output ENOENT, and the stream was still marked kernelized. It now takes the no_kernel path.

The limit itself is left as it is. It came with 01a10c0, where it sized the array of outputs in struct rtpengine_list_entry; that struct is gone, and until now the module's check was its only use.

[rfuchs]

Global mutex for data that is not global is a no-no

[nickakt]

Moved it into the call: kernelize() is the only place that holds the locks of two streams at once, and streams that are each other's sinks are always in one call (connect, inject and mesh merge the calls), so call->kernelize_lock serializes exactly those, and other calls no longer wait.