🚀 Feature: Re-run a test multiple times: `this.repeat`

[guyb7]

this.retries() allows to run a test multiple times, but the test will pass on the first successful run.

I'm suggesting to add this.rerun() for tests that might randomly succeed (false positives). It will run the test n times and will pass only if all of the reruns have passed successfully. It's useful when testing behavior that is probabilistic.

At the moment I'm running the test in a loop:

for (var i = 1; i <= 100; i++) {
  it('tests something #' + i, function(done) {
    //
  });
}

The problem with this approach is that it outputs 100 different tests and messes up the test counting.

[boneskull]

Hmm. What's the problem with putting the loop inside your test?

[guyb7]

It's just not as convenient - I have to manually call beforeEach() on each iteration and call the next iteration (and call done() on the last one).

Compare this: (can be improved)

describe('#multiple-runs', function() {
  var beforeEachFunction = function() {
    // Code
  };

  beforeEach(beforeEachFunction);

  it('runs 15 times', function(done) {
    var recursive_test = function(n) {
      if (n > 15) {
        done();
        return;
      }
      beforeEachFunction();
      // Code
      recursive_test(n + 1);
    }
    recursive_test(1);
  });
});

With this:

describe('#multiple-runs', function() {
  beforeEach(function(){
    // Code
  });

  it('runs 15 times', function(done) {
    this.rerun(15);
    // Code
  });
});

i think a better sintax could be something like this

it.rerun(15, 'runs 15 times', function(done) {
   //Code
});

[fantasyRqg]

did this function realize ?

[Bamieh]

@fantasyRqg no.

[plroebuck]

No. But in case someone wants to mess with it, here's some code that does testing manually.
Tests if a random integer is divisible by subset of prime numbers over a series of 5 runs.

Save the following as "multirun.spec.js".

var debuglog = require('util').debuglog('multirun');

describe('#multiple-runs', function () {
  var NRUNS = 5;

  // Returns vector of prime numbers (up to max) [Sieve of Eratosthenes]
  function getPrimes(max) {
    var primes = [], sieve = [];

    for (var i = 2; i <= max; ++i) {
      if (!sieve[i]) {
        // i has not been marked -- it is prime
        primes.push(i);
        for (var j = i << 1; j <= max; j += i) {
          sieve[j] = true;
        }
      }
    }
    return primes;
  }

  // Returns random integer (up to max)
  function getRandomInt(max) {
    return Math.floor(Math.random() * Math.floor(max));
  }

  // Randomize array element order in-place. [Durstenfeld]
  function shuffle(array) {
    for (var i = array.length - 1; i > 0; i--) {
      // Pick a remaining element
      var j = Math.floor(Math.random() * (i + 1));

      // Swap it with current element
      var temp = array[i];
      array[i] = array[j];
      array[j] = temp;
    }
    return;
  }

  //=========================================================
  it('should succeed ' + NRUNS + ' times', function (done) {
    (function _suite(nruns) {
      var candidate, primes, somePrimes;

      function _suiteSetup() {
        var MAXPRIME = 31;
        primes = getPrimes(MAXPRIME);
        debuglog('primes[' + primes.length + '] =', primes, '(orig)');

        // :NOTE: Remove '2' from list to improve odds of test success
        primes = primes.slice(1);
      }

      function _setup() {
        var MAX = 100;
        candidate = getRandomInt(MAX);

        // (Re)shuffle primes
        shuffle(primes);
        debuglog('primes[' + primes.length + '] =', primes, '\n');

        // Use first NKEEP prime #s for test
        var NKEEP = 3;
        somePrimes = primes.slice(0, NKEEP);
        debuglog('somePrimes[' + somePrimes.length + '] =', somePrimes);
      }

      // Code being tested [throw to indicate failure]
      function _test(run) {
        somePrimes.forEach(function (prime) {
          debuglog('candidate:', candidate + ', prime:', prime);
          if (candidate % prime === 0) {
            throw new Error('' + candidate + ' is divisible by ' + prime);
          }
        });
      }

      function _teardown() {
        candidate = undefined;
        somePrimes = undefined;
      }

      try {
        _suiteSetup();
        for (var n = 0; n < nruns; n++) {
          debuglog('==========');
          _setup();
          _test(n);
          _teardown();
        }
      }
      catch (err) {
        return done(new Error('run ' + (n+1) + ' failed: ' + err.message));
      }

      return done();
    })(NRUNS);
  });
});

Execute it, expecting both failure/success:

$ mocha multirun.spec.js

Same as above, with verbose output.

$ NODE_DEBUG=multirun mocha multirun.spec.js

[AidanWelch]

This is also useful for testing that the first run didn't break any future usages of whatever your testing without requiring a restart.

[sriharikapu]

I am facing the similar issues with tests as well cant run the same test multiple times

[shunjizhan]

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

[TierK]

Any date prognosis of realization?

[kberg]

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

This is what I'm facing. I'll just wrap it in a shell script and run it a couple hundred times.

[mmomtchev]

Does anyone know if there is any reason why this hasn't been implemented besides no one actually doing it?

My use case is testing for leaks in C++ modules.