Content tagged clojure

Up and running with Clojure and Storm
posted on 2015-02-08 15:57:41

So you want to build some kind of real-time processing pipeline, you want to use Storm, you don't want to use Java, I went through a couple of tutorials recently which seem to stop at the final step of getting your topology running on a proper storm cluster. Instead they opt for running with lein run -m foo.bar.run! which is not perfect.

First of all, they tell you to explicitly not to include the storm dependencies into your regular project dependencies. With the (outdated) storm project template giving you:

(defproject vektor "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :aot [vektor.TopologySubmitter]
  ;; include storm dependency only in dev because production storm cluster provides it
  :profiles {:dev {:dependencies [[storm "0.8.1"]]}})

However, running lein uberjar or lein jar on this, in order to be able to submit the jar to a running Storm cluster gives:

~/dev/vektor lein jar
Compiling vektor.TopologySubmitter
java.io.FileNotFoundException: Could not locate backtype/storm/clojure__init.class or backtype/storm/clojure.clj on classpath: , compiling:(topology.clj:1)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3387)
    at clojure.lang.Compiler.compile1(Compiler.java:7035)
    at clojure.lang.Compiler.compile1(Compiler.java:7025)

    [thousands of lines of tyranny elided...]

    at clojure.lang.Compiler.eval(Compiler.java:6501)
    at clojure.lang.Compiler.load(Compiler.java:6952)
    at clojure.lang.Compiler.loadFile(Compiler.java:6912)
    at clojure.main$load_script.invoke(main.clj:283)
    at clojure.main$init_opt.invoke(main.clj:288)
    at clojure.main$initialize.invoke(main.clj:316)
    at clojure.main$null_opt.invoke(main.clj:349)
    at clojure.main$main.doInvoke(main.clj:427)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at clojure.lang.Var.invoke(Var.java:419)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)

Hmm, that's not good. So we shouldn't include the Storm dependencies but we can't compile without them. Great.

After much googling and messing around you should add them as a :provided profile in project.clj such as:

(defproject vektor "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :aot [vektor.TopologySubmitter]
  ;; include storm dependency only in dev because production storm cluster provides it
  :profiles {:dev {:dependencies [[storm "0.8.1"]]}
             :provided {:dependencies [[storm "0.8.1"]]}})

This will now allow the jars to be created and you will be able to submit them to the storm cluster.

Side notes

Ensure all directories which Storm and Nimbus wish to use are readable/writable by the users you are wanting to use for them.

By default:

/var/lib/storm and /var/log/storm should be writable/readable by the storm user.

Clojure for the Emacs user
posted on 2014-08-17 13:46:28

One of the great things about Clojure is that it's a Lisp, and not just for the fact that this brings a lot of linguistic power over typical languages. Lisps have a very intertwined history with Emacs.

In this post I will show you what I think is the perfect environment for composing Clojure code.

First of all you will need Emacs version >=24, this ensures that you have the proper package.el support by default. If not; you will need to install package.el manually. This page is a great resource on package.el.

Now, we can install the basic clojure-mode, which will provide proper font-locking and basic code highlighting.

To install: M-x package-install RET clojure-mode RET.

Personally, I prefer a live editing environment over the archaic write/compile/test/repeat cycle that a lot of languages enforce. To this end, we need to install the exceptional CIDER mode. CIDER provides a SLIME-like environment for Clojure. CIDER is a mode which enables a powerful embedded REPL inside emacs. This embedded REPL can do code evaluation, code introspection, code-completion and much more. It's very difficult to return to languages which do not have this rich-editing experience.

To install: M-x package-install RET cider RET.

Finally, clj-refactor is a package which provides a few helpful code transformations. There are gems such as "introduce-let" which moves a form into an enclosing let form, or sort-namespace, which will alphabetically sort the namespace (great for keeping tidy code). There are many more so I suggest that you browse the repository.

To install: M-x package-install RET clj-refactor RET

Enjoy!

This blog covers amazon, browser, cider, clojure, common lisp, distributed computing, ec2, emacs, erlang, fresh, haproxy, kafka, kubernetes, lisp, new, storm, webkit, websockets, zookeeper

View content from 2017-03, 2016-02, 2015-02, 2014-08, 2014-07


Unless otherwise credited all material Creative Commons License by Aaron France