Skip to content

Make `—order random` more deterministic.

gitlab-qa-bot requested to merge stable-random-ordering into master

Created by: myronmarston

When you’re troubleshooting an order dependent failure, you want to get the repro case down to a minimal run that loads and runs as few specs as possible. With the old random ordering implementation, that was hard to achieve because while rerunning with a given seed produced the same order when the exact same set of examples were loaded, the ordering would be completely different when a subset was loaded.

By ordering by md5(seed + example_id) it ensures that the ordering of any two examples should stay consistent regardless of how many other examples are loaded.

MD5 is significantly slower than shuffle, and it loads an additional part of our stdlib (which we want to minimize) but I think the tradeoff is worth it here.

This will help us be able to implement the --bisect option discussed in #1767 (closed).

What do others think? Is the tradeoff worth it? Is there some other hashing function we should consider using? If there was a hashing function that had similar distribution characteristics to MD5 and was faster and easy to use w/o loading part of the stdlib, that would be ideal, but I can't think of one.

/cc @rspec/rspec

Merge request reports