🛠 Repo: Test failures

[mark-wiemer]

Tooling Suggestion Checklist

• I have tried restarting my IDE and the issue persists.
• I have pulled the latest main branch of the repository.
• I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
• I have searched for related issues and issues with the faq label, but none matched my issue.
• I want to provide a PR to resolve this

Overview

The GitHub action to run all tests on various versions of Node and various operating systems has started returning intermittent test failures again.

• Passing run
• Failing run

The only difference (netlify.toml changed)

[mark-wiemer]

Will investigate this tomorrow if time/energy allows. Triggered a run on main branch hoping for a clean repro. Most recent runs on main do seem to have at least one failure :(

[jdmarshall]

Looks like a timing bug in the watch tests.

I wonder if this relates to #5372

If you want this all to go faster it probably needs changes so that mocha uses 'process.send' for lifecycle events like the watch operation being ready to run. That would likely shave a lot of time off of your tests as well, instead of waiting 4 seconds for all of them when it looks like even that is not quite enough.

[mark-wiemer]

Thanks, the goal for now is getting the tests to just pass consistently, then we can look at performance. I'm focusing on another issue right now but hope to come back to this one soon (TM)

[jdmarshall]

I was poking at it, but out of the box I’m getting consistent failures of the integration tests, not intermittent.

[mark-wiemer]

Rerunning a batch of 10 test runs and will check tomorrow, but the May 11th batch was intermittent. There have been some good bugfixes since then but nothing specifically addressing these. I was on a break but I'm back now and will actually really truly work on this tomorrow :)

[mark-wiemer]

@jdmarshall if you're still interested I'm curious if the consistent failures you're getting are for the same tests that are intermittently failing in GitHub Actions? Any failures are a problem, consistent or intermittent! Feel free to open a separate bug if you think they're unrelated

[jdmarshall]

It appears to be the same two every time.

For sure it has happened at least twice in my scroll buffer for that window.

M2 Mac, so who knows what cpu specific timing differences there might be.

  340 passing (2m)
  1 pending
  2 failing

  1) root hooks
       when mocha run in serial mode
         support ESM via .js extension w/o type=module
           should fail due to ambiguous file type
             with --no-experimental-detect-module:
     
expected Promise when fulfilled to contain output /SyntaxError: Unexpected token/
  expected
  {
    output: '\n\n  0 passing (1ms)\n\n',
    code: 0,
    args: [
      '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js',
      '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
      '--no-experimental-detect-module',
      '--no-color',
      '--no-bail',
      '--no-parallel'
    ],
    command: '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js --require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js --no-experimental-detect-module --no-color --no-bail --no-parallel'
  }
  to contain output /SyntaxError: Unexpected token/

  
  expected Promise when fulfilled to contain output /SyntaxError: Unexpected token/
    expected
    {
      output: '\n\n  0 passing (1ms)\n\n',
      code: 0,
      args: [
        '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
        '--no-experimental-detect-module',
        '--no-color',
        '--no-bail',
        '--no-parallel'
      ],
    }
    to contain output /SyntaxError: Unexpected token/
      at Array.forEach (<anonymous>)
  From previous event:
      at Context.<anonymous> (test/integration/plugins/root-hooks.spec.js:146:18)
      at callFn (lib/runnable.js:112:584)
      at Runnable.run (lib/runnable.js:112:57)
      at Runner.runTest (lib/runner.js:220:1054)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:248:654
      at next (lib/runner.js:197:422)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:197:809
      at next (lib/runner.js:185:579)
      at Immediate.<anonymous> (lib/runner.js:189:1253)
      at process.processImmediate (node:internal/timers:485:21)
      set UNEXPECTED_FULL_TRACE=true to see the full stack trace

  2) root hooks
       when mocha run in serial mode
         support ESM via .js extension w/o type=module
           should fail due to ambiguous file type
             with --experimental-detect-module:
     
expected Promise when fulfilled to contain output /Cannot require\(\) ES Module/
  expected
  {
    output: '\n\n  0 passing (1ms)\n\n',
    code: 0,
    args: [
      '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js',
      '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
      '--experimental-detect-module',
      '--no-color',
      '--no-bail',
      '--no-parallel'
    ],
    command: '/Users/jasonmarshall/Projects/cobbler/mocha/bin/mocha.js --require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js --experimental-detect-module --no-color --no-bail --no-parallel'
  }
  to contain output /Cannot require\(\) ES Module/

  
  expected Promise when fulfilled to contain output /Cannot require\(\) ES Module/
    expected
    {
      output: '\n\n  0 passing (1ms)\n\n',
      code: 0,
      args: [
        '--require=/Users/jasonmarshall/Projects/cobbler/mocha/test/integration/fixtures/plugins/root-hooks/root-hook-defs-esm-broken.fixture.js',
        '--experimental-detect-module',
        '--no-color',
        '--no-bail',
        '--no-parallel'
      ],
    }
    to contain output /Cannot require\(\) ES Module/
      at Array.forEach (<anonymous>)
  From previous event:
      at Context.<anonymous> (test/integration/plugins/root-hooks.spec.js:166:18)
      at callFn (lib/runnable.js:112:584)
      at Runnable.run (lib/runnable.js:112:57)
      at Runner.runTest (lib/runner.js:220:1054)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:248:654
      at next (lib/runner.js:197:422)
      at /Users/jasonmarshall/Projects/cobbler/mocha/lib/runner.js:197:809
      at next (lib/runner.js:185:579)
      at Immediate.<anonymous> (lib/runner.js:189:1253)
      at process.processImmediate (node:internal/timers:485:21)
      set UNEXPECTED_FULL_TRACE=true to see the full stack trace



ERROR: "test-node:integration" exited with 2.
ERROR: "test-node" exited with 1.

[jdmarshall]

I wonder if I should try plopping it into a docker image.