| Age | Commit message (Collapse) | Author |
|---|
| We never freed any resources of rb_native_thread at fork because it would cause it to hang. This is because it called rb_native_cond_destroy for condition variables. We can't call rb_native_cond_destroy here because according to the specs of pthread_cond_destroy: Attempting to destroy a condition variable upon which other threads are currently blocked results in undefined behavior. Specifically, glibc's pthread_cond_destroy waits on all the other listeners. Since after forking all the threads are dead, the condition variable's listeners will never wake up, so it will hang forever. This commit changes it to only free the memory and none of the condition variables. Notes: Merged: https://.com/ruby/ruby/pull/13591 |
| * Added `Ractor::Port` * `Ractor::Port#receive` (support multi-threads) * `Rcator::Port#close` * `Ractor::Port#closed?` * Added some methods * `Ractor#join` * `Ractor#value` * `Ractor#monitor` * `Ractor#unmonitor` * Removed some methods * `Ractor#take` * `Ractor.yield` * Change the spec * `Racotr.select` You can wait for multiple sequences of messages with `Ractor::Port`. ```ruby ports = 3.times.map{ Ractor::Port.new } ports.map.with_index do |port, ri| Ractor.new port,ri do |port, ri| 3.times{|i| port << "r#{ri}-#{i}"} end end p ports.each{|port| pp 3.times.map{port.receive}} ``` In this example, we use 3 ports, and 3 Ractors send messages to them respectively. We can receive a series of messages from each port. You can use `Ractor#value` to get the last value of a Ractor's block: ```ruby result = Ractor.new do heavy_task() end.value ``` You can wait for the termination of a Ractor with `Ractor#join` like this: ```ruby Ractor.new do some_task() end.join ``` `#value` and `#join` are similar to `Thread#value` and `Thread#join`. To implement `#join`, `Ractor#monitor` (and `Ractor#unmonitor`) is introduced. This commit changes `Ractor.select()` method. It now only accepts ports or Ractors, and returns when a port receives a message or a Ractor terminates. We removes `Ractor.yield` and `Ractor#take` because: * `Ractor::Port` supports most of similar use cases in a simpler manner. * Removing them significantly simplifies the code. We also change the internal thread scheduler code (thread_pthread.c): * During barrier synchronization, we keep the `ractor_sched` lock to avoid deadlocks. This lock is released by `rb_ractor_sched_barrier_end()` which is called at the end of operations that require the barrier. * fix potential deadlock issues by checking interrupts just before setting UBF. https://bugs.ruby-lang.org/issues/21262 Notes: Merged: https://.com/ruby/ruby/pull/13445 |
| Although it almost certainly works in this case, volatile is best not used for multi-threaded code. Using atomics instead avoids warnings from TSan. This also simplifies some logic, as system_working was previously only ever assigned to 1, so --system_working <= 0 should always return true (unless it underflowed). Notes: Merged: https://.com/ruby/ruby/pull/13333 |
| |
| [Feature #20590] For better of for worse, fork(2) remain the primary provider of parallelism in Ruby programs. Even though it's frowned uppon in many circles, and a lot of literature will simply state that only async-signal safe APIs are safe to use after `fork()`, in practice most APIs work well as long as you are careful about not forking while another thread is holding a pthread mutex. One of the APIs that is known cause fork safety issues is `getaddrinfo`. If you fork while another thread is inside `getaddrinfo`, a mutex may be left locked in the child, with no way to unlock it. I think we could reduce the impact of these problem by preventing in for the most notorious and common cases, by locking around `fork(2)` and known unsafe APIs with a read-write lock. Notes: Merged: https://.com/ruby/ruby/pull/10864 |
| * Fix unused functions when no `mmap`. ``` shape.c:285:1: warning: unused function 'redblack_insert' [-Wunused-function] 285 | redblack_insert(redblack_node_t * tree, ID key, rb_shape_t * value) | ^~~~~~~~~~~~~~~ ``` * Fix unknown warning group '-Wmaybe-uninitialized' with clang. ``` thread_win32.c:596:1: warning: unknown warning group '-Wmaybe-uninitialized', ignored [-Wunknown-warning-option] 596 | COMPILER_WARNING_IGNORED(-Wmaybe-uninitialized) | ^ ``` Co-authored-by: Nobuyoshi Nakada <[email protected]> Notes: Merged: https://.com/ruby/ruby/pull/11376 Merged-By: nobu <[email protected]> |
| |
| Introduce `rb_thread_lock_native_thread()` to allocate dedicated native thread to the current Ruby thread for M:N threads. This C API is similar to Go's `runtime.LockOSThread()`. Accepted at https://.com/ruby/dev-meeting-log/blob/master/2023/DevMeeting-2023-08-24.md (and missed to implement on Ruby 3.3.0) |
| Where a local variable is used as part of the stack bounds detection, it has to actually be on the stack. ASAN can put local variable on "fake stacks", however, with addresses in different memory mappings. This completely destroys the stack bounds calculation, and can lead to e.g. things not getting GC marked on the machine stack or stackoverflow checks that always fail. The __asan_addr_is_in_fake_stack helper can be used to get the _real_ stack address of such variables, and thus perform the stack size calculation properly [Bug #20001] |
| This commit changes how stack extents are calculated for both the main thread and other threads. Ruby uses the address of a local variable as part of the calculation for machine stack extents: * pthreads uses it as a lower-bound on the start of the stack, because glibc (and maybe other libcs) can store its own data on the stack before calling into user code on thread creation. * win32 uses it as an argument to VirtualQuery, which gets the extent of the memory mapping which contains the variable However, the local being used for this is actually too low (too close to the leaf function call) in both the main thread case and the new thread case. In the main thread case, we have the `INIT_STACK` macro, which is used for pthreads to set the `native_main_thread->stack_start` value. This value is correctly captured at the very top level of the program (in main.c). However, this is _not_ what's used to set the execution context machine stack (`th->ec->machine_stack.stack_start`); that gets set as part of a call to `ruby_thread_init_stack` in `Init_BareVM`, using the address of a local variable allocated _inside_ `Init_BareVM`. This is too low; we need to use a local allocated closer to the top of the program. In the new thread case, the lolcal is allocated inside `native_thread_init_stack`, which is, again, too low. In both cases, this means that we might have VALUEs lying outside the bounds of `th->ec->machine.stack_{start,end}`, which won't be marked correctly by the GC machinery. To fix this, * In the main thread case: We already have `INIT_STACK` at the right level, so just pass that local var to `ruby_thread_init_stack`. * In the new thread case: Allocate the local one level above the call to `native_thread_init_stack` in `call_thread_start_func2`. [Bug #20001] fix |
| This reverts commit 4ba8f0dc993953d3ddda6328e3ef17a2fc2cbde5. |
| This reverts commit 6185cfdf38e26026c6d38220eeca48689e54cdcf. |
| Where a local variable is used as part of the stack bounds detection, it has to actually be on the stack. ASAN can put local variable on "fake stacks", however, with addresses in different memory mappings. This completely destroys the stack bounds calculation, and can lead to e.g. things not getting GC marked on the machine stack or stackoverflow checks that always fail. The __asan_addr_is_in_fake_stack helper can be used to get the _real_ stack address of such variables, and thus perform the stack size calculation properly [Bug #20001] |
| The implementation of `native_thread_init_stack` for the various threading models can use the address of a local variable as part of the calculation of the machine stack extents: * pthreads uses it as a lower-bound on the start of the stack, because glibc (and maybe other libcs) can store its own data on the stack before calling into user code on thread creation. * win32 uses it as an argument to VirtualQuery, which gets the extent of the memory mapping which contains the variable However, the local being used for this is actually allocated _inside_ the `native_thread_init_stack` frame; that means the caller might allocate a VALUE on the stack that actually lies outside the bounds stored in machine.stack_{start,end}. A local variable from one level above the topmost frame that stores VALUEs on the stack must be drilled down into the call to `native_thread_init_stack` to be used in the calculation. This probably doesn't _really_ matter for the win32 case (they'll be in the same memory mapping so VirtualQuery should return the same thing), but definitely could matter for the pthreads case. [Bug #20001] |
| With M:N thread scheduler, the native thread (NT) related resources should be freed when the NT is no longer needed. So the calling `native_thread_destroy()` at the end of `is will be freed when `thread_cleanup_func()` (at the end of Ruby thread) is not correct timing. Call it when the corresponding Ruby thread is collected. |
| This introduce M:N thread scheduler for Ractor system. In general, M:N thread scheduler employs N native threads (OS threads) to manage M user-level threads (Ruby threads in this case). On the Ruby interpreter, 1 native thread is provided for 1 Ractor and all Ruby threads are managed by the native thread. From Ruby 1.9, the interpreter uses 1:1 thread scheduler which means 1 Ruby thread has 1 native thread. M:N scheduler change this strategy. Because of compatibility issue (and stableness issue of the implementation) main Ractor doesn't use M:N scheduler on default. On the other words, threads on the main Ractor will be managed with 1:1 thread scheduler. There are additional settings by environment variables: `RUBY_MN_THREADS=1` enables M:N thread scheduler on the main ractor. Note that non-main ractors use the M:N scheduler without this configuration. With this configuration, single ractor applications run threads on M:1 thread scheduler (green threads, user-level threads). `RUBY_MAX_CPU=n` specifies maximum number of native threads for M:N scheduler (default: 8). This will be reverted soon if non-easy issues are found. [Bug #19842] |
| |
| for future extension Notes: Merged: https://.com/ruby/ruby/pull/7639 |
| * Remove `waitpid_lock` and related code. * Remove un-necessary test. * Remove `rb_thread_sleep_interruptible` dead code. Notes: Merged-By: ioquatix <[email protected]> |
| * Revert "Remove special handling of `SIGCHLD`. (#7482)" This reverts commit 44a0711eab7fbc71ac2c8ff489d8c53e97a8fe75. * Revert "Remove s for functions that are no longer used. (#7497)" This reverts commit 4dce12bead3bfd91fd80b5e7195f7f540ffffacb. * Revert "Remove SIGCHLD `waidpid`. (#7476)" This reverts commit 1658e7d96696a656d9bd0a0c84c82cde86914ba2. * Fix change to rjit variable name. Notes: Merged-By: ioquatix <[email protected]> |
| * Remove `waitpid_lock` and related code. * Remove un-necessary test. * Remove `rb_thread_sleep_interruptible` dead code. Notes: Merged-By: ioquatix <[email protected]> |
| ``` 1) Failure: TestThreadInstrumentation#test_thread_instrumentation [/tmp/ruby/src/trunk-repeat20-asserts/test/-ext-/thread/test_instrumentation_api.rb:33]: Call counters[4]: [3, 4, 4, 4, 0]. Expected 0 to be > 0. ``` We fire the EXIT hook after the call to `thread_sched_to_dead` which mean another thread might be running before the `EXIT` hook have been executed. Notes: Merged: https://.com/ruby/ruby/pull/7249 |
| GCC warns of empty format strings, perhaps because they have no effects in printf() and there are better ways than sprintf(). However, ruby_debug_log() adds informations other than the format, this warning is not the case. |
| [Misc #18891] Notes: Merged: https://.com/ruby/ruby/pull/6094 |
| It's very flaky for some unknown reason. Something we have an extra EXITED event. I suspect some other test is causing this. Notes: Merged: https://.com/ruby/ruby/pull/6133 |
| * Use PRIxSIZE instead of "z" * Fix sign-compare warning * Suppress unused-but-set-variable warning |
| |
| [Feature #18339] After experimenting with the initial version of the API I figured there is a need for an exit event to cleanup instrumentation data. e.g. if you record data in a {thread_id -> data} table, you need to free associated data when a thread goes away. Notes: Merged: https://.com/ruby/ruby/pull/6029 |
| follow up https://.com/ruby/ruby/pull/6006 |
| Ref: https://bugs.ruby-lang.org/issues/18339 Design: - This tries to minimize the overhead when no hook is registered. It should only incur an extra unsynchronized boolean check. - The hook list is protected with a read-write lock as to cause contention when some hooks are registered. - The hooks MUST be thread safe, and MUST NOT call into Ruby as they are executed outside the GVL. - It's simply a noop on Windows. API: ``` rb_internal_thread_event_hook_t * rb_internal_thread_add_event_hook(rb_internal_thread_event_callback callback, rb_event_flag_t internal_event, void *user_data); bool rb_internal_thread_remove_event_hook(rb_internal_thread_event_hook_t * hook); ``` You can subscribe to 3 events: - READY: called right before attempting to acquire the GVL - RESUMED: called right after successfully acquiring the GVL - SUSPENDED: called right after releasing the GVL. The hooks MUST be threadsafe, as they are executed outside of the GVL, they also MUST NOT call any Ruby API. Notes: Merged: https://.com/ruby/ruby/pull/5500 |
| This macro is no longer used ([GH-5933]). Notes: Merged: https://.com/ruby/ruby/pull/5935 |
| `thread_debug()` was introduced to print debug messages on `THREAD_DEBUG > 0` but `RUBY_DEBUG_LOG()` is more controllable. Notes: Merged: https://.com/ruby/ruby/pull/5933 |
| `rb_thread_t` contained `native_thread_data_t` to represent thread implementation dependent data. This separates them and rename it `rb_native_thread` and point it from `rb_thraed_t`. Now, 1 Ruby thread (`rb_thread_t`) has 1 native thread (`rb_native_thread`). Notes: Merged: https://.com/ruby/ruby/pull/5836 |
| Now GVL is not process *Global* so this try to use another words. * `rb_global_vm_lock_t` -> `struct rb_thread_sched` * `gvl->owner` -> `sched->running` * `gvl->waitq` -> `sched->readyq` * `rb_gvl_init` -> `rb_thread_sched_init` * `gvl_destroy` -> `rb_thread_sched_destroy` * `gvl_acquire` -> `thread_sched_to_running` # waiting -> ready -> running * `gvl_release` -> `thread_sched_to_waiting` # running -> waiting * `gvl_yield` -> `thread_sched_yield` * `GVL_UNLOCK_BEGIN` -> `THREAD_BLOCKING_BEGIN` * `GVL_UNLOCK_END` -> `THREAD_BLOCKING_END` * removed * `rb_ractor_gvl` * `rb_vm_gvl_destroy` (not used) There are GVL functions such as `rb_thread_call_without_gvl()` yet but I don't have good name to replace them. Maybe GVL stands for "Greate Valuable Lock" or something like that. Notes: Merged: https://.com/ruby/ruby/pull/5814 |
| Notes: Merged: https://.com/ruby/ruby/pull/5407 |
| Notes: Merged: https://.com/ruby/ruby/pull/4445 |
| |
| Raise `NotImplementedError` on unsupported platforms regardless the argument consistently. |
| |
| USE_WIN32_MUTEX flag may not be defined. Notes: Merged: https://.com/ruby/ruby/pull/4457 |
| |
| To make some kind of Ractor related extensions, some functions should be exposed. * include/ruby/thread_native.h * rb_native_mutex_* * rb_native_cond_* * include/ruby/ractor.h * RB_OBJ_SHAREABLE_P(obj) * rb_ractor_shareable_p(obj) * rb_ractor_std*() * rb_cRactor and rm ractor_pub.h and rename srcdir/ractor.h to srcdir/ractor_core.h (to avoid conflict with include/ruby/ractor.h) Notes: Merged: https://.com/ruby/ruby/pull/3775 |
| The timer function used on windows system set timer interrupt flag of current main ractor's executing ec and thread can detect the end of time slice. However, to set all ec->interrupt_flag for all running ractors, it is requires to synchronize with other ractors. However, timer thread can not acquire the ractor-wide lock because of some limitation. To solve this issue, this introduces USE_VM_CLOCK compile option to introduce rb_vm_t::clock. This clock will be incremented by the timer thread and each thread can check the incrementing by comparison with previous checked clock. At last, on windows platform this introduces some overhead, but I think there is no critical performance issue because of this modification. Notes: Merged: https://.com/ruby/ruby/pull/3754 |
| |
| This commit introduces Ractor mechanism to run Ruby program in parallel. See doc/ractor.md for more details about Ractor. See ticket [Feature #17100] to see the implementation details and discussions. [Feature #17100] This commit does not complete the implementation. You can find many bugs on using Ractor. Also the specification will be changed so that this feature is experimental. You will see a warning when you make the first Ractor with `Ractor.new`. I hope this feature can help programmers from thread-safety issues. Notes: Merged: https://.com/ruby/ruby/pull/3365 |
| |
| |
| |
| |
| We need another native thread to call some unblocking functions which aren't RUBY_UBF_IO or RUBY_UBF_PROCESS. Instead of a permanent thread in <= 2.5, we can now rely on the thread cache feature to perform interrupts. [ruby-core:90865] [Bug #15499] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66708 b2dd03c8-39d4-4d8f-98ff-823fe69b080e |
