Content tagged storm

Up and running with Clojure and Storm
posted on 2015-02-08 15:57:41

So you want to build some kind of real-time processing pipeline, you want to use Storm, you don't want to use Java, I went through a couple of tutorials recently which seem to stop at the final step of getting your topology running on a proper storm cluster. Instead they opt for running with lein run -m foo.bar.run! which is not perfect.

First of all, they tell you to explicitly not to include the storm dependencies into your regular project dependencies. With the (outdated) storm project template giving you:

(defproject vektor "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :aot [vektor.TopologySubmitter]
  ;; include storm dependency only in dev because production storm cluster provides it
  :profiles {:dev {:dependencies [[storm "0.8.1"]]}})

However, running lein uberjar or lein jar on this, in order to be able to submit the jar to a running Storm cluster gives:

~/dev/vektor lein jar
Compiling vektor.TopologySubmitter
java.io.FileNotFoundException: Could not locate backtype/storm/clojure__init.class or backtype/storm/clojure.clj on classpath: , compiling:(topology.clj:1)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3387)
    at clojure.lang.Compiler.compile1(Compiler.java:7035)
    at clojure.lang.Compiler.compile1(Compiler.java:7025)

    [thousands of lines of tyranny elided...]

    at clojure.lang.Compiler.eval(Compiler.java:6501)
    at clojure.lang.Compiler.load(Compiler.java:6952)
    at clojure.lang.Compiler.loadFile(Compiler.java:6912)
    at clojure.main$load_script.invoke(main.clj:283)
    at clojure.main$init_opt.invoke(main.clj:288)
    at clojure.main$initialize.invoke(main.clj:316)
    at clojure.main$null_opt.invoke(main.clj:349)
    at clojure.main$main.doInvoke(main.clj:427)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:419)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)

Hmm, that's not good. So we shouldn't include the Storm dependencies but we can't compile without them. Great.

After much googling and messing around you should add them as a :provided profile in project.clj such as:

(defproject vektor "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :aot [vektor.TopologySubmitter]
  ;; include storm dependency only in dev because production storm cluster provides it
  :profiles {:dev {:dependencies [[storm "0.8.1"]]}
             :provided {:dependencies [[storm "0.8.1"]]}})

This will now allow the jars to be created and you will be able to submit them to the storm cluster.

Side notes

Ensure all directories which Storm and Nimbus wish to use are readable/writable by the users you are wanting to use for them.

By default:

/var/lib/storm and /var/log/storm should be writable/readable by the storm user.

This blog covers amazon, browser, cider, clojure, common lisp, distributed computing, ec2, emacs, erlang, fresh, haproxy, kafka, kubernetes, lisp, new, storm, webkit, websockets, zookeeper

View content from 2017-03, 2016-02, 2015-02, 2014-08, 2014-07


Unless otherwise credited all material Creative Commons License by Aaron France