manticore
Mythical multi-process worker pool
Fork node.js multi-core workers and crunch legendary workloads.
The core concept is you got some code in a function that does some heavy work and you want to run it many times with maximum benefit of your multi-core CPU, and without the overhead of re-spawning piles single-use sub-processes.
⚠️ Early release 😎
Why yet another worker module?
The worker modules I found on npm all have their problems: they either lack functionality, use external dependencies or make all kinds of weird assumptions that get in the way.
Instead of trying to wrangle my app to fit those unsatisfactory modules I build Manticore to be simple and effective with the features you need to get big things done at hyperspeed without jumping through crazy hoops.
How to use?
You put your code in a function that accepts a single parameter, then add a bunch of them in a worker module. In this module you register the functions to expose them as tasks.
In your main app you setup the pool for that module and execute the methods via the pool with your data parameter and Manticore will spawn (and despawn) workers as needed and distribute the work and return the result as a Promise.
You can use a function that returns a value synchronously, or go asynchronous and either use the node.js-style callback or return a Promise.
By default each worker works on only one job at a time, but there is an option to allow workers to process multiple jobs simultaneously that allows a extra boost for IO-bound tasks (of course assuming you use async IO).
Return value
The return value of the pool is always a ES6-style Promise so you easily use fancy logic like Promise.all() or Promise.race().
For some next level setups you can leverage Promise-glue helpers from modules like Q, Bluebird etc. To get creative and pass the Promises into more exotic modules like React, Baconjs, Lazy.js, Highland and all the other cool utility modules with Promise support.
Keep in mind the parameter object and return value are serialised so you cannot pass functions or prototype based objects, only simple JSON-like data.
Notes
- Returns a ES6 Promise.
- Transfers data between threads using pipes (eg: non-blocking).
- Data gets serialised so only primitive JSON-like data can be transferred.
- Makes sure you configure concurrent/paralel to suit your app for best performance
Todo
- Swap JSON serialisation for something that supports Buffers.
- Separate settings per function.
Usage
Setup worker
Put the worker methods in their own module where they are registered to Manticore:
var mc = ; // directly add named function { return ;}mc; // add anonymous functionmc;
There are different ways to return values:
// does it run syncronous? { return ;} // maybe use the node-style callback? { ;} // or return a Promise? { return
Register in bulk:
// add named functions as arraymc; // register the methods as an object to redefine the name// - protip: use the module.exports objectmc;
Use the pool
Create a pool in the main app:
var mc = ; var pool = mc;
Then run the methods by name, pass a parameter value and get a Promise:
pool;
For convenience get a curried function:
var func1 = pool; ;
Pro-tip: for serious bulk processing use Promise.all()
(in Bluebird this is fun with Promise.map()
etc).
PromiseallmyArray;
That's it! 👍
Pool options
;
Development
Manticore is written in TypeScript and compiled with Grunt.
For TypeScript user there is a .d.ts
file both in the repo and bundled in the npm package (also exported in package.json).
Build
Install development dependencies in your git checkout:
$ npm install
Build and run tests using grunt:
$ grunt test
See the Gruntfile.js
for additional commands.
Contributions
They are welcome but please discuss in the issues before you commit to large changes. If you send a PR make sure you code is idiomatic and linted.
History
- 0.2.0 - Transfer data over non-blocking pipes, renamed
modulePath
option toworker
. - 0.1.0 - First release.
License
Copyright (c) 2014 Bart van der Schoor @ Bartvds
Licensed under the MIT license.