nimbus_difference_jstorm_vs_storm

Nimbus Difference

• Nimbus HA

  • JStorm: stable, running more than 2 years
  • Storm: hasn't been released, no online test.
  • Suggestion: using JStorm's HA solution

• Security Feature

  • JStorm doesn't have Security Check, It depends on the high level Web-UI-Porter to control security
  • Storm contain Security Module, * credential, nimbus need credential plugin, topology need set credential setting * SUPERVISOR-RUN-WORKER-AS-USER, * ZK Authentical
  • Suggestion: using Storm's security solution

• Scheduler

  • JStorm hard code using scheduler com.alibaba.jstorm.schedule.default_assign.DefaultTopologyScheduler, using the interface com.alibaba.jstorm.schedule.IToplogyScheduler
  • Storm support scheduler interface backtype.storm.scheduler.IScheduler, which can be setting in conf by "storm.scheduler"
  • Suggestion: using JStorm scheduler, because JStorm scheduler is more powerful, providing feature such as user-define scheduler/old assignment/isolation assignment.

• system-topology

  • Storm add SystemBolt/MetricsConsumerBolt, which receive METRICS-STREAM-ID/SYSTEM-STREAM-ID, send SYSTEM-TICK-STREAM-ID,METRICS-TICK-STREAM-ID,CREDENTIALS-CHANGED-STREAM-ID
  • JStorm add TopologyMaster, receive hearbeat/metrics/flow-ctrol
  • Suggestion: using JStorm's TopologyMaster, it is do more things, not only metrics, but also hearbeat, flow ctrol, the only complement is add validate-structure／validate-topology-size when submit topology

• restart topology

  • JStorm provide restart command, this command is very useful for Cluster Operator, it support reload configuration, including changing the number of worker/ any component's parrel
  • Storm doesn't support restart
  • Suggestion: using JStorm restart

• Rebalance

  • Storm rebalance support change worker/any component executor's number
  • JStorm rebalance support change worker/task number
  • Suggestion: using the JStorm rebalance function, which is more powerful, it can scale in and scale out, the Storm rebalance support change worker number, but the number can't exceed the sum of tasks, the executor can't execeed the task's number. JStorm rebalance support don't kill old workers when scale out.

• update topology

  • JStorm support updateTopology, this thrift api can update topology binary (support dynamic update topology binary by Alibaba Pandora Plugin), update configuration
  • Storm doesn't support this function
  • Suggestion: use this thrift api

• lib jar

  • Storm doesn't support lib jar
  • JStorm support lib jar
  • Suggestion: using support lib jar, it is easy for application to update dependency jar without rebuild package. Sometimes, reduce the package size, user doesn't upload some commond jars, the gateway contains common lib jar

• getTopologyId thrift api

  • Storm doesn't support
  • JStorm support getTopologyId by topologyName
  • Suggestion: using JStorm getTopologyId

• getTopologyInfo thrift API

  • Storm: 2 api getTopologyInfo/getTopologyInfoWithOpts, the option can be used to chose all/none/one error message
  • JStorm: getTopologyInfo/getTopologyInfoByName
  • Suggestion: merge them, support api getTopologyInfo/getTopologyInfoByName/getTopologyInfoWithOpts

• getSupervisorWorkers

  • Storm: no
  • JStorm: this thrift api can get current node's workers' status/metrics
  • Suggestion: using JStorm method

• getVersion

  • Storm: no
  • JStorm: get cluster's jstorm version, which has been largely used to check whether nimbus is alive or not
  • Suggestion: using JStorm method

• Metrics thrift api

  • Storm: no
  • JStorm: getSummarizedTopologyMetrics/getTopologyMetrics/getMetrics/getNettyMetrics/getNettyMetricsByHost/getPagingNettyMetrics/getNettyMetricSizeByHost/getTaskMetrics/getTaskAndStreamMetrics
  • Suggestion: using JStorm method

• TopologyValidator

  • JStorm don't use TopologyValidator nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"
  • Storm use nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator", it will do validate when submit topology
  • Suggestion: the validator is empty, The new Storm had better use the validator, even when it is empty

• Topology cleanup

  • Storm do topology cleanup after monitor topology every 10s
  • JStorm do topology cleanup after finish one scheduler task.
  • Suggestion: follow JStorm method, because this can reduce lock time

• SubmitTopology

  • Storm: * it will call validate-configs-with-schemas validate the topology configuration

• setup-default-uncaught-exception-handler

  • all of JStorm process catch exception by himself, unify all cleanup, shutdown hook and unexpected exception handler into one handler
  • Storm using setup-default-uncaught-exception-handler to catch Exception
  • Suggestion: Wrapper setup-default-uncaught-exception-handler function, let process register cleanup handler, and using this function