normalian blog

Let's talk about Microsoft Azure, ASP.NET and Java!

Get started with Apache Storm on HDInsight for your jar files

HDInsight provides you to create Apache Storm clusters easily. Please read reference articles in this post if you don't know overview of Apache Storm.

Create Storm Cluster on HDInsight

Follow https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-storm-tutorial-get-started-linux article until "Create a Storm cluster" section. It takes about 15 minutes to create your Storm cluster, and pick up below info to connect your Storm cluster.

  • SSH login URL: engiyoistorm02-ssh.azurehdinsight.net
  • Dashboard URL: https://"your cluster name".azurehdinsight.net/
  • StormUI URL: https://"your cluster name".azurehdinsight.net/stormui/index.html

Deploy your jar files into your Storm cluster

Create your jar file including Topology class to deploy into your Storm Cluster. Please refer below example if you don't have such Java projects.
https://github.com/apache/storm/tree/master/examples/storm-starter

After making your jar file, try to connect to your cluster via ssh. Here is a sample connecting to your cluster using WinSCP.
f:id:waritohutsu:20170724143720p:plain
Transfer your jar file from your computer into your cluster. Now, you can run your jar file into your cluster.

Connect to you cluster via ssh. Here is a sample connecting to your cluster using putty.
f:id:waritohutsu:20170724143739p:plain
Follow below commands to run your jar file. Specify second argument as topology class and third argument as topology name.

sshuser@xxxxxxxx:~$ storm jar /home/sshuser/hellostorm-0.0.1-SNAPSHOT.jar com.mydomain.hellostorm.HelloTopology hello-topology
sshuser@xxxxxxxx:~$ storm list
6244 [main] INFO  o.a.s.u.NimbusClient - Found leader nimbus : 10.0.0.10:6627
Topology_name        Status     Num_tasks  Num_workers  Uptime_secs
-------------------------------------------------------------------
hello-topology       ACTIVE     8          3            8758

Monitor your application

Open https://"your cluster name".azurehdinsight.net/stormui/index.html via your browser. You can find you topology in Storm UI.
f:id:waritohutsu:20170724143827p:plain