[Feature] Improve performance of `yarn run`

[SimenB]

• I'd be willing to implement this feature (contributing guide)
• This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

Running yarn jest has a significant performance overhead in Yarn v2 vs Yarn v1 (or npx). The following commands are run in a repo installed using yarn v2, with the node-linker. Yarn v1 is run from the brew installed version, no local version in the repo. Note that all runs are without running yarn or npm in between - they all run against a repo installed using Yarn v2

$ hyperfine -w 5 'yarn jest --version' 'npx jest --version' 'npm exec jest -- --version'
Benchmark #1: yarn jest --version
  Time (mean ± σ):      1.298 s ±  0.031 s    [User: 1.597 s, System: 0.149 s]
  Range (min … max):    1.270 s …  1.381 s    10 runs

Benchmark #2: npx jest --version
  Time (mean ± σ):     469.4 ms ±   9.7 ms    [User: 397.0 ms, System: 91.6 ms]
  Range (min … max):   451.8 ms … 484.1 ms    10 runs

Benchmark #3: npm exec jest -- --version
  Time (mean ± σ):     466.9 ms ±  14.8 ms    [User: 396.8 ms, System: 90.6 ms]
  Range (min … max):   446.9 ms … 490.0 ms    10 runs

Summary
  'npm exec jest -- --version' ran
    1.01 ± 0.04 times faster than 'npx jest --version'
    2.78 ± 0.11 times faster than 'yarn jest --version'

Then for yarn v1 (by doing rm .yarnrc*)

$ hyperfine -w 5 'yarn jest --version' 'npx jest --version' 'npm exec jest -- --version'
Benchmark #1: yarn jest --version
  Time (mean ± σ):     376.0 ms ±  22.0 ms    [User: 265.8 ms, System: 62.9 ms]
  Range (min … max):   352.1 ms … 423.0 ms    10 runs

Benchmark #2: npx jest --version
  Time (mean ± σ):     480.6 ms ±  52.4 ms    [User: 403.2 ms, System: 93.1 ms]
  Range (min … max):   408.4 ms … 582.2 ms    10 runs

Benchmark #3: npm exec jest -- --version
  Time (mean ± σ):     491.8 ms ±  25.8 ms    [User: 412.4 ms, System: 99.7 ms]
  Range (min … max):   451.5 ms … 535.4 ms    10 runs

Summary
  'yarn jest --version' ran
    1.28 ± 0.16 times faster than 'npx jest --version'
    1.31 ± 0.10 times faster than 'npm exec jest -- --version'

While it doesn't matter too much in the case of Jest (although perceived startup performance is noticeably worse when running tests since there's no output), it's quite annoying when running tools as git hooks.

$ hyperfine -w 5 'yarn lint-staged' 'npx lint-staged' 'npm exec lint-staged'
Benchmark #1: yarn lint-staged
  Time (mean ± σ):      1.473 s ±  0.069 s    [User: 1.780 s, System: 0.192 s]
  Range (min … max):    1.367 s …  1.562 s    10 runs

Benchmark #2: npx lint-staged
  Time (mean ± σ):     540.3 ms ±  32.6 ms    [User: 468.0 ms, System: 114.4 ms]
  Range (min … max):   493.2 ms … 579.3 ms    10 runs

Benchmark #3: npm exec lint-staged
  Time (mean ± σ):     589.5 ms ±  59.5 ms    [User: 509.0 ms, System: 124.3 ms]
  Range (min … max):   540.7 ms … 692.1 ms    10 runs

Summary
  'npx lint-staged' ran
    1.09 ± 0.13 times faster than 'npm exec lint-staged'
    2.73 ± 0.21 times faster than 'yarn lint-staged'

And with yarn v1

$ hyperfine -w 5 'yarn lint-staged' 'npx lint-staged' 'npm exec lint-staged'
Benchmark #1: yarn lint-staged
  Time (mean ± σ):     545.4 ms ±  72.8 ms    [User: 411.8 ms, System: 104.0 ms]
  Range (min … max):   465.7 ms … 695.8 ms    10 runs

Benchmark #2: npx lint-staged
  Time (mean ± σ):     583.3 ms ±  31.2 ms    [User: 503.4 ms, System: 123.1 ms]
  Range (min … max):   543.0 ms … 631.4 ms    10 runs

Benchmark #3: npm exec lint-staged
  Time (mean ± σ):     574.3 ms ±  14.1 ms    [User: 497.9 ms, System: 123.1 ms]
  Range (min … max):   556.7 ms … 596.9 ms    10 runs

Summary
  'yarn lint-staged' ran
    1.05 ± 0.14 times faster than 'npm exec lint-staged'
    1.07 ± 0.15 times faster than 'npx lint-staged'

When rebasing lots of commits, that almost 1.5s of overhead (or 1s in the case of yarn v1) adds a lot of time spent. It's gotten to the point where I pass -n if I'm making lots of commits (or rebase) as it takes way too long to run.

For reference, I've been running these benchmarking runs in https://github.com/jest-community/eslint-plugin-jest

Describe the solution you'd like

I'd like it to be faster 😀

I expect some overhead of yarn v2 simply due to the fact it needs to first spawn yarn v1, find the config, then load yarn v2. But almost 3x time spent in execution is way more than I expected.

I assume you've already tried to optimize the overhead added by yarn when running scripts, but maybe some checks for "PnP compliance" or whether the lockfile is in sync with all package.jsons can be dropped for the node linker? Possibly via some flag which we can then use when we want, although a flag saying "just do it, don't check" might not be feasible? And a flag saying "give me perf" is probably weird and few people will know about it

Describe the drawbacks of your solution

I don't think there's any drawbacks to improving performance, but I don't know enough about Yarn's innards to comment on technical drawbacks of whatever optimization is applied.

Describe alternatives you've considered

Stop using yarn as binary runner and use npx or npm exec instead, at least in commit hooks and such.

[arcanis]

Timing are always funny 😄 we got a very similar two days ago, and some improvements have landed: #2560

However my fix should mostly have an effect on the PnP linker, so perhaps there's something similar that needs to be done for the nm one? We should get a CPU stack sample to have a better idea.

[SimenB]

Hah, perfect 😀

No real difference running from sources, tho.

$ yarn set version from sources && yarn
$ yarn --version
2.4.0-git.20210306.hash-fea486ce
$ hyperfine -w 5 'yarn lint-staged' 'npx lint-staged' 'npm exec lint-staged'
Benchmark #1: yarn lint-staged
  Time (mean ± σ):      1.412 s ±  0.063 s    [User: 1.705 s, System: 0.200 s]
  Range (min … max):    1.353 s …  1.526 s    10 runs

Benchmark #2: npx lint-staged
  Time (mean ± σ):     627.2 ms ± 122.9 ms    [User: 520.5 ms, System: 133.5 ms]
  Range (min … max):   561.3 ms … 969.8 ms    10 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark #3: npm exec lint-staged
  Time (mean ± σ):     549.7 ms ±  29.4 ms    [User: 476.2 ms, System: 120.0 ms]
  Range (min … max):   518.9 ms … 613.5 ms    10 runs

Summary
  'npm exec lint-staged' ran
    1.14 ± 0.23 times faster than 'npx lint-staged'
    2.57 ± 0.18 times faster than 'yarn lint-staged'

[SimenB]

Fresh run after the 2 PRs (same sequence of commands as above)

Benchmark #1: yarn lint-staged
  Time (mean ± σ):     995.2 ms ±  30.3 ms    [User: 1.054 s, System: 0.150 s]
  Range (min … max):   966.7 ms … 1070.2 ms    10 runs

Benchmark #2: npx lint-staged
  Time (mean ± σ):     525.0 ms ±  15.9 ms    [User: 458.8 ms, System: 108.9 ms]
  Range (min … max):   512.2 ms … 557.1 ms    10 runs

Benchmark #3: npm exec lint-staged
  Time (mean ± σ):     537.8 ms ±  31.8 ms    [User: 466.9 ms, System: 111.7 ms]
  Range (min … max):   508.3 ms … 605.8 ms    10 runs

Summary
  'npx lint-staged' ran
    1.02 ± 0.07 times faster than 'npm exec lint-staged'
    1.90 ± 0.08 times faster than 'yarn lint-staged'

so it has definitely improved! 👏 1.9x of npx and npm exec now instead of the 2.6-2.8 I saw before

[andreialecu]

Somewhat related discussion: #2117

A big part of the startup overhead is node having to parse the entire yarn bundle and launch the WebAssembly libzip implementation.

If nodejs/node#36812 would ever be implemented in node, performance in such cases should hugely improve. Yarn could then be distributed as separate .js files (within a packaged application), and only a subset of it would need to be parsed for common commands, such as yarn run

[pastelsky]

We hit this too, unfortunately with ~ 25 packages, the overhead of running workspace foreach is often 15-20 seconds just for an echo. This overhead sometimes exceeds the time needed for the command being run itself, and if you're running multiple foreachs, this quickly adds up. For large projects, this becomes prohibitively expensive.

We worked around this by switching runs to a different tool like — https://www.npmjs.com/package/workspaces-run
which isn't great because it comes with its own set of syntax.

[TrySound]

We run node_modules/.bin directly to workaround this.