Random shuffle of lists #10281

RaimoNiskanen · 2025-10-14T14:54:00Z

This PR adds functions rand:shuffle/1 and rand:shuffle_s/2 due to a discussion on ErlangForums: https://erlangforums.com/t/random-sort-should-be-included-in-the-lists-module/5125

There are 4 algorithms in the first commit. The suggested winner is the one remaining in the second commit.

Documentation and test cases are still missing...

github-actions · 2025-10-14T14:54:48Z

CT Test Results

2 files 97 suites 1h 6m 59s ⏱️
2 220 tests 2 169 ✅ 51 💤 0 ❌
2 607 runs 2 551 ✅ 56 💤 0 ❌

Results for commit a5dbc7b.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

bjorng

Nitpick: I don't know whether you intend to keep the first commit. In case you do, the last paragraph is missing a closing parenthesis, and the word "ridiculous" is misspelled.

lib/stdlib/src/rand.erl

RaimoNiskanen · 2025-10-16T14:18:15Z

New algorithm selected. "Quickshuffle"?

lib/stdlib/src/rand.erl

RaimoNiskanen · 2025-10-17T08:45:50Z

I wrote a longer explanation of the algorithm

RaimoNiskanen · 2025-10-20T08:45:37Z

Pushed some optimizations

lib/stdlib/src/rand.erl

RaimoNiskanen · 2025-10-24T13:58:05Z

I have tests (and previously documentation), and backed out some optimization attempts.

With the measurement function in the test case in place it turned out that
the leaf function for permuting a list of 4 was too clumsy and gave nothing. The permutation of 3 elements could be a bit simpler.

The measurement test function compares with the previously best function; decorate, sort, undecorate and shuffle duplicates. It also compares fast and slow PRNG:s.

Now this might be ready to merge...

bjorng · 2025-10-27T12:42:32Z

Please do some squashing of the commits.

RaimoNiskanen · 2025-10-27T14:14:03Z

Squashed into 3 commits.

Experiments, with 6 algorithms
Selection of a winner
Finalizing the selected

RaimoNiskanen · 2025-10-27T16:52:04Z

Force pushed suppression of Dialyzer warnings for improper lists

Write a few shuffle algorithms for comparison. shuffle1: random decorate, sort, undecorate and recursively deduplicate ------- I have found no formal statement that it is bias free, but have tried to reason around it. The algorithm should be equivalent to generating more random decimals to decide the shuffle order for elements with the same random number. It should make no difference if the random decimals are generated always and ignored, or when needed. Speed: 1.2 s for 2^20 integers on my laptop. shuffle2: Fisher-Yates with map as array ------- The classical textbook shuffle. Speed: 5 s for 2^20 integers on my laptop. shuffle3: random decorates, avoid duplicates, sort and undecorate with gb_trees ------- Quite a beautiful algorithm since the `gb_tree` has all the functionality in itself. Speed: 5 s for 2^20 integers on my laptop. shuffle4: random decorate, avoid duplicates, sort and undecorate with a map ------- The same as the `gb_tree` above, but with a map. Uses the map key order instead of the general term order, which works just fine. Speed: 2 s for 2^20 integers on my laptop. shuffle5: random hidden decorate by split, implicit sort ------- Suggested by Richard A. O'Keefe on ErlangForums as "a random variant of Quicksort", probably misunderstood by me into this algorithm. Shall we name it Quickshuffle? Really fast. Uses random numbers efficiently by looking at individual bits for the random split. Has no overhead for tagging. Just creates intermediate lists as garbage. This generator appears to actually be equivalent with shuffle1, using a random number generator with 1 bit which goes into almost exclusively deduplication recursion. Speed: 0.8 s for 2^20 integers on my laptop. shuffle6: Fisher-Yates with the `array` module ------- The classical textbook shuffle. Our standard `array` module here outperforms map, probably because keys do not have to be stored, they are implicit. Speed: 2 s for 2^20 integers on my laptop. Discussion ------- shuffle3 and shuffle4 have the theoretical limitation that when the length of the list approaches the generator size, it will take catastrophically much longer time to generate a random number that has not been used. There is no check for the list length being larger than the generator size in which case it will be impossible to generate unique random numbers for all list elements, and the algorithm will simply keep on failing forever. This is for now a theoretical problem since already for a list length with log half the generator size (e.g 2^28 with a generator size 2^56), my laptop runs out of memory with a VM of about 30 GB. shuffle1 and shuffle5 avoids that limitation. shuffle1 by recursing over the duplicates sublists so it is not affected much by fairly long lists of duplicates, shuffle5 by using only individual bits and ranges 2, 6, or 24. The classical Fisher-Yates algorithm in shuffle2 and shuffle6 does not have this limitation, but generating random numbers of unlimited length gets increasingly expensive, which should not be any problem for 2 or even 4 times the generator length, that is list lengths of well over 2^200, which is well over ridiculous.

* Explain in comments * Optimize * Document * Write test cases * Write measurement test case to compare with runner-up algorithm

RaimoNiskanen · 2025-10-28T12:50:36Z

I found out that the algorithm headings in the first commit had been lost so the commit comment was impossible to understand. I fixed that commit comment and nothing else...

RaimoNiskanen added this to the OTP-29.0 milestone Oct 14, 2025

RaimoNiskanen requested a review from bjorng October 14, 2025 14:54

RaimoNiskanen self-assigned this Oct 14, 2025

RaimoNiskanen added team:VM Assigned to OTP team VM team:PS Assigned to OTP team PS feature in progress priority:medium labels Oct 14, 2025

bjorng reviewed Oct 15, 2025

View reviewed changes