May https://.com/nobu/ruby/tree/rehash-after-delete help it?

Right, @nobu's approach seems much better than reintroducing that weird behavior for .dup.

Ideally we wouldn't rehash as in calling key.hash methods again, but instead just shrink the internal data structure (and same when growing it).

So apparently some applications were relying on Set#dup/Hash#dup to do like C++ shrink_to_fit.
Ruby does not have such a method and it feels quite low-level, so it seems better to resize the internal data structure when removing elements/entries and going below some threshold.

As a note, this repro code is very "lucky" to trigger a dup after removing 99.99% of the elements.
I suppose it's done that way to make the effect very clear though.
Without the - [0] the same problem occurs on 3.0:

$ ruby -v -rset -e 's1 = Set.new(10000.times); s2 = Set.new(9999.times); a=Array.new(10000) { s1 - s2 }; GC.start; puts `ps -o rss= -p #{$$}`.to_i' 
ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
3015808
$ ruby -v -rset -e 's1 = Set.new(10000.times); s2 = Set.new(9999.times); a=Array.new(10000) { s1 - s2 - [0] }; GC.start; puts `ps -o rss= -p #{$$}`.to_i'
ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
74552

If a Set is kept alive a long time, one way to ensure it uses the minimum amount of space is Set#reset, at the cost of extra time to reset/rehash (which notably calls #hash for every key), it's a time vs memory trade-off, can be worth it for big long-lived sets:

$ ruby -v -rset -e 's1 = Set.new(10000.times); s2 = Set.new(9999.times); a=Array.new(10000) { s=s1 - s2 - [0]; s.reset; s }; GC.start; puts `ps -o rss= -p #{$$}`.to_i'
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
62992

Automatic shrinking (PR at https://.com/ruby/ruby/pull/8748) should help the worst cases like the repro so that seems good anyway.

Thanks nobu and nagachika.

I confirmed to resolve this regrassion with ruby_3_2 branch.

# Before
$ ruby -v -rset -e 's1 = Set.new(10000.times); s2 = Set.new(9999.times); Array.new(10000) { s1 - s2 - [0] }; puts `ps -o rss= -p #{$$}`.to_i'
ruby 3.2.2 (2023-07-05 revision 2f603bc4d7) +YJIT [arm64-darwin23]
4564304

# After
$ ruby -v -rset -e 's1 = Set.new(10000.times); s2 = Set.new(9999.times); Array.new(10000) { s1 - s2 - [0] }; puts `ps -o rss= -p #{$$}`.to_i'
ruby 3.2.2 (2023-11-19 revision d9f4f321c6) +YJIT [arm64-darwin23]
40864