Writable object stream on top of MongoDb using bulk mode for speed.
Depending on the nature of data, the gain in speed is about 8x
npm install bulk-mongo
var bulkMongo = require('bulk-mongo');
var factory_function = bulkMongo(dbInstance);
var bulkWriter = factory_function(targetCollection);
sourceStream.pipe(bulkWriter);Of course the code above can be compressed into just two lines.
Is used to create actual stream instance for a particular database connection it was created with. Factory arguments are:
collection is a collection name (a string) or a MongoDb Collection instance.
options are explained below but usually they are not needed.
- bulkSize defaults to
1000. This means that bulk writes will be committed in chunks of 1000 records (documents). Setting this value to 1 or less forces non-bulk mode using ordinary MongoDb connection#insert() to be used. Other options are those eligible for stream.Writable. - objectMode defaults to
true. - highWaterMark defaults to
16.
Returns an instance of stream.Writable.
This event will follow the 'finish' event, indicating there is no more data to be written into the database. This event is sent in non-bulk mode as well.
Because the asynchronous bulk write operation might be triggered by 'finish' event, the 'finish' itself can not guarantee the data being synced.
resultis an instance of BulkWriteResult. It's members nInserted and getInsertedIds() can be particularly useful.
The event is being emitted every time after UnorderedBulkOperation#execute() has been successful. In case of failure, 'error' event is emitted.
This event will follow the 'finish' event, indicating there is no more data to be written into the database.
Because the asynchronous bulk write operation might be triggered by 'finish' event, the 'finish' itself can not guarantee the data is synced.
To run tests or examples, make sure you have mongodb running - probably you'll need something like this:
mongod > mongod.log &
before running tests. It could be handy to run mongod in a separate terminal window.
See http://nodejs.org/api/stream.html#stream_class_stream_writable for other elements of consumer API.
MongoDb Js driver API description is here: http://mongodb.github.io/node-mongodb-native/1.4/
MIT