qtopology
NPM package: https://www.npmjs.com/package/qtopology
Travis CI: https://travis-ci.org/qminer/qtopology
Installation
npm install qtopology
Intro
QTopology is a distributed stream processing layer, written in node.js
.
It uses the following terminology, originating in Storm project:
- Topology - Organization of nodes into a graph that determines paths where messages must travel.
- Bolt - Node in topology that receives input data from other nodes and emits new data into the topology.
- Spout - Node in topology that reads data from external sources and emits the data into the topology.
- Stream - When data flows through the topology, it is optionaly tagged with stream ID. This can be used for routing and filtering.
When running in distributed mode, qtopology
also uses the following:
- Coordination storage - must be resilient, receives worker registrations and sends them the initialization data. Also sends shutdown signals. Implementation is custom.
QTopology
providesREST
-based service out-of-the-box, but the design is similar for other options likeMySQL
storage etc. - Worker - Runs on single server. Registers with coordination storage, receives initialization data and instantiates local topologies in separate subprocesses.
- Leader - one of the active workers is announced leader and it performs leadership tasks such as assigning of topologies to workers, detection of dead or inactive workers.
Quick start
Read this quick-start guide to quickly create your own topology.