Randomized testing, as provided by the various variants of Haskell's quickcheck test framework and fuzz testing tools, has proven surprisingly potent at probing areas of a program's parameter space which a developer would not think about manually testing. I personally found a number of interesting bugs in RHist by using it, and therefore think that RHist could benefit from using this technique in its test suite too.
I am not proposing to completely replace RHist's test suite with such tests, because that would require using a pseudorandom data generator that is specifically tuned to adversarially favor known-tricky data, which is hard to implement and does not really have a good pre-built implementation in C++. Instead, I am proposing to complement RHist's existing hand-written tests with a simple randomized test suite which just pokes across the parameter space with a uniform law, much like the test suite of my work-in-progress RHist -> THx conversion tool is doing.
Test reproducibility concerns may be alleviated by either...
- Using a fixed PRNG seed. We don't really need true randomness to benefit from randomized testing, as we only care about the good N-dimensional parameter space coverage properties of pseudo-random numbers.
- Picking a random PRNG seed by default, but printing it on test startup and allowing the dev to replay a test with a specific PRNG seed, as e.g. the verrou floating-point error checker does