Causes webpack to run out of heap when building

[dllz]

Hi there.

When I include your plugin in my webpack.config.js file and run build I get a Javascript heap out of memory exception.

When I provide the --max_old_space_size argument and give it 10GB of ram it just builds to what seems infinity.

Stacktrace:
WEBPACK_ENV=build webpack

Warning (worker-plugin): output.globalObject is set to "window". It must be set to "self" to support HMR in Workers.
[BABEL] Note: The code generator has deoptimised the styling of /home/daniel/Documents/Dynex/Canvas/canvas.iron.js/src/components/data/changeFocusHolder.js as it exceeds the max of 500KB.

<--- Last few GCs --->
art of marking 194 ms) (average mu = 0.198, current mu = 0.079) allocation[10962:0x3598210] 290153 ms: Mark-sweep 2045.5 (2052.8) -> 2043.0 (2050.6) MB, 163.7 / 0.0 ms (+ 0.0 ms in 31 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 178 ms) (average mu = 0.144, current mu = 0.079) allocation[10962:0x3598210] 290433 ms: Mark-sweep 2043.0 (2050.6) -> 2043.0 (2050.6) MB, 248.2 / 0.0 ms (average mu = 0.126, current mu = 0.113) allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x13162b9]
1: StubFrame [pc: 0x13718dc]

Security context: 0x35a54a940911
2: walkExpressions [0x2d890b3a71d1] [/home/daniel/Documents/Dynex/Canvas/canvas.iron.js/node_modules/webpack/lib/Parser.js:1475] [bytecode=0x20b4258b8f81 offset=69](this=0x29546e2436a1 ,0x3fec23d59ed9 <JSArray[1]>)
3: walkCallExpression [0x2d890b3a76a1] [/home/daniel/Documents/Dynex/Canvas/can...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x9aedf0 node::Abort() [node]
2: 0x9aff86 node::OnFatalError(char const*, char const*) [node]
3: 0xb078ce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb07c49 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xce4ae5 [node]
6: 0xcf032b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
7: 0xcf1047 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
8: 0xcf3b78 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
9: 0xcc1b5f v8::internal::Factory::NewByteArray(int, v8::internal::AllocationType) [node]
10: 0xc4419a v8::internal::TranslationBuffer::CreateByteArray(v8::internal::Factory*) [node]
11: 0x191fe07 v8::internal::compiler::CodeGenerator::GenerateDeoptimizationData() [node]
12: 0x1920502 v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
13: 0x1993abd v8::internal::compiler::PipelineImpl::FinalizeCode(bool) [node]
14: 0x1994b60 v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl(v8::internal::Isolate*) [node]
15: 0xc06ea0 v8::internal::Compiler::FinalizeOptimizedCompilationJob(v8::internal::OptimizedCompilationJob*, v8::internal::Isolate*) [node]
16: 0xc00766 v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
17: 0xc8ae5c v8::internal::StackGuard::HandleInterrupts() [node]
18: 0xf93ba7 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
19: 0x13162b9 [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! canvas.iron@1.0.85 build: WEBPACK_ENV=build webpack --max_old_space_size=10000
npm ERR! Exit status 134
npm ERR!
npm ERR! Failed at the canvas.iron@1.0.85 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/daniel/.npm/_logs/2019-07-05T11_46_34_789Z-debug.log

[developit]

Hi there,

In order to figure this one out I'm going to need your Node and Webpack versions, as well as the snippet of code you're using to instantiate the Worker, and ideally the relevant portions of your Webpack config. Without those, all I can tell you is that this is an infinite loop likely caused by duplicated Webpack plugins.

FWIW it seems like you are using some very large .js files as data sources. That's okay, but I'd recommend setting Webpack's module.noParse option to prevent parsing of those files, which will reduce your build times (and memory consumption) considerably.

[dllz]

Hey,

Here is the webpack file and some of the code in question:https://jsfiddle.net/dllze/dsvg2x3L/4/

I will look into noparse thanks but those files have imports in them and exports. Luckily they are only used for development/testing purposes and dont ship with the live product and just consist of some test json and code to wire it into callbacks that projects dependent on this one would use

[kfeinUI]

Ran into this as well. Worked fine a very simple test case while proving out my config. However, quickly scaling up to something much more sizeable and complex (trying to migrate existing stuff over to a worker) would just stall out.

Had just as much luck with worker-loader, so you're in good company. Doesn't seem terribly active though.

That the plugins aren't applied consistently seems strange too. Maybe I'm over simplifying it, but I would think that this is a matter of dynamically generating a new entry point with a worker target. Webpack should in theory be able to cope with that like another.

[dllz]

Yeah I tested this last night with a much simpler file and got exactly the same issue:

import {controls, scene, camera, renderer} from "./setup";
import TWEEN from '@tweenjs/tween.js';
let requestAnimationFrameId;
self.addEventListener('message', function(e) {
    var data = e.data;
    switch (data.cmd) {
        case 'run':
            console.log("run mainloop");
            mainLoop();
            break;
        case 'stop':
            console.log("stop mainloop");
            cancelAnimationFrame(requestAnimationFrameId);
            break;
        case 'close':
            console.log("Closing worker");
            cancelAnimationFrame(requestAnimationFrameId);
            self.close();
            break;
        default:
            self.postMessage('Unknown command: ' + data.msg);
    }
});

export default function mainLoop(){
    controls.update();
    TWEEN.update();
    renderer.clear();
    renderer.render(scene, camera);
    requestAnimationFrameId = requestAnimationFrame(mainLoop);
};

[developit]

Plugin application in child compilers is handled by Webpack, there aren't great defaults since plugins can do anything they want. As an example, it would be logical to copy a plugin that enables minification, but illogical to copy a plugin that handles CSS. There is unfortunately no way to infer the purpose of a given plugin.

@dllz thanks for the fiddle. I'm going to try setting that up as a test case to see if it's a version compatibility thing, since the code looks totally fine to me.

[dllz]

@developit So I got this kinda working using rollup, I have not yet got it perfectly transpiling (missing some plugins still) but I noticed a warning in the output that I thought might be helpful.
Both of my failing test cases have circular dependencies, it would make sense that that is causing the heap error and the building forever issue.

Does the plugin have circular dependency detection?

@kfeinUI can you check if your project that failed has circular dependencies as well?

[Nikki1993]

Greetings,

Been hit by the same issue, the command fails on heap out of memory. I tried checking with https://www.npmjs.com/package/circular-dependency-plugin for circular dependencies but the plugin does not seem to find any at some point webpack build just stops processing files with eventual out of memory issue.

I am using
node 13.1.0
webpack 4.41.2
webpack-dev-server 3.9.0

[FishOrBear]

I had the same problem and I suspect it was circular dependency.

[Nikki1993]

In my case, it was me trying to import a JSON file. I just used fetch to do the same instead of importing and the problem resolved itself!

[developit]

If anyone has a way to reproduce the supposed circular dependency issue that would be potentially useful. This plugin doesn't modify Webpack's handling of dependencies at all, so if the issue is related to circular deps it's likely a bug in webpack itself.

Some context on performance for folks who are interested: Webpack (and Rollup, actually) does not support specifying different output types for each entry module in a configuration. It also doesn't support dynamically adding entries during compilation, presumably because that would make it very difficult to determine when it was okay to start reducing the module graph (a new entry invalidates the graph).

So instead, all current solutions for workers in Webpack make use of a "Child Compiler", which is essentially a nested Webpack run that handles compiling your Worker starting from the module you specified. It's not quite as much overhead as compiling your application twice, but it's also not cheap. The whole module graph has to be constructed again just for each Child Compiler, since caching is done at the compiler level.

My recommendation for folks is to try to find ways to keep your workers reasonably sized. Personally, I like to use module.noParse to "quarantine" some of the the huge modules I typically end up importing (Babel, Marked, etc).

Another option would be to use importScripts to load huge libraries from a CDN or basic file (you can copy them to your build directory using copy-webpack-plugin). Though I would caution against this approach since importScripts() is actually removed in real Module Workers.

[nonoroazoro]

I met the same issue here with the following simple code and didn't figure out why yet.

[greyivy]

Having the same issue as well while importing @google/blockly.

[ricoarisandyw]

I ever have same issues. one of my library is use global window function. I remember we cannot use global window function inside web worker. then I don't call that library inside my web worker.

[Tofandel]

I had the same issue after messing around with node version, webpack build stuck at 28% and eventually crashing out of memory, I solved it by removing the node_modules folder and the rerunning npm install, the heap issue was then gone