TL;DR - This is not a comprehensive guide to Clojure, but it is enough to get you started with Riemann. This is also an excerpt from my forthcoming book - The Art of Monitoring. It'll also be available in the Riemann documentation at some point too.
Riemann is configured using a Clojure-based configuration file. This means your configuration file is actually processed as a Clojure program. So to process events and send alerts and metrics you'll be writing Clojure. Don't panic! You don't need to become a fully fledged Clojure developer to use Riemann. I can teach you what you need to know in order to use Riemann. Additionally, Riemann comes with a lot of helpers and shortcuts that make it easier to write Clojure to do what we need to process our events.
Let's learn a bit more about Clojure and help you get started with Riemann. Clojure is a dynamic programming language that targets the Java Virtual Machine. It's a dialect of Lisp and is largely a functional programming language.
Functional programming is a programming style that focuses on the evaluation of mathematical functions and steers away from changing state and mutable data. It's highly declarative, meaning you build programs from expressions that describe "what" a program should accomplish rather than "how" it accomplishes it.
Note Languages that describe more of the "how" are called imperative languages.
Examples of declarative programming languages include SQL, CSS, regular expressions and configuration management languages like Puppet and Chef. Let's take a simple example.
SELECT user_id FROM users WHERE user_name = "Alice"
In this SQL query we're asking for the user_id for user_name of Alice from the users table. The statement is asking a declarative "what" question. We don't really care about the "how", the database engine takes care of those details.
In addition to their declarative nature, functional programming languages try to eliminate all side effects from changing state. In a functional language when you call a function its output value depends only on the inputs to the function. So if you repeatedly call function f with the same value for argument x, f(x), it will produce the same result every time. This makes functional programs very easy to understand, test and predict. Functional programming languages call functions that operate like this "pure" functions.
The best way to get started with Clojure is to understand the basics of its syntax and types. Let's get a crash course now.
Warning This is going to be a very high level and not very nuanced introduction to Clojure. It's designed to give you the knowledge and recognition of various syntax and expressions to allow you to work with Riemann. It is not an article that will teach you how to develop in Clojure.
A brief introduction to Clojure
Let's step through the Clojure basic syntax and types. We'll also show you a tool called REPL that can help you test and build your Clojure snippets. REPL (short for read–eval–print loop) is an interactive programming shell that takes single expressions, evaluates them and returns the results. It's a great way to get to know Clojure.
Note If you're from the Ruby world then REPL is just like
irb. Or in Python when you launch the
python binary interactively.
We can install REPL via a tool called Leiningen. Leiningen is an automation tool for Clojure that helps you automate the build and management of Clojure projects.
Installing Leiningen
In order to install Leiningen we'll need to have Java installed on the host. The prerequisite Java packages on Ubuntu and Red Hat for Reimann will also be sufficient for Leiningen too.
We're going to download a Leiningen binary called lein to install it. Let's download that into a bin directory under our home directory.
$ mkdir -p ~/bin
$ cd ~/bin
$ curl -o lein https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
$ chmod a+x lein
$ export PATH=$PATH:$HOME/bin
Here we've created a new directory called ~/bin and changed into it. We've then used the curl command to download the lein binary and the chmod command to make it executable. Lastly, we've added our ~/bin directory to our path so that we can find the lein binary.
Tip The addition of the
~/bin directory assumes you're in a Bash shell. It's also temporary to your current shell. You'd need to add the path to your
.bashrc or the similar setup for your shell.
Next we need to run lein to auto-install its supporting libraries.
$ lein
. . .
This will download Leiningen's supporting Jar file.
Finally, we can run REPL using the lein repl sub-command.
$ lein repl
. . .
user=>
This will download Clojure itself (in the form of its Jar file) and launch our interactive Clojure shell.
Clojure syntax and types
Let's use this interactive shell to look at some of the syntax and functions we've just learnt about. Let's start by opening our shell.
user=>
Now let's try a simple expression.
user=> nil
nil
The nil expression is the simplest value in Clojure. It represents literally nothing.
We can also specify an integer value.
user=> 1
1
Or a string.
user=> "hello Ms Event"
"hello Ms Event"
Or Boolean values.
user=> true
true
user=> false
false
Clojure functions
Whilst interesting these values aren't very exciting on their own. To do some more interesting things we can use Clojure functions. A function is structured like this:
(function argument argument)
Tip If you're used to the Ruby or Python world a function is broadly the equivalent of a method.
Let's look at a function in action by doing something with some values: adding two integers together.
user=> (+ 1 1)
2
In this case we've used the + function and added 1 and 1 together to get 2.
But there's something about this structure that might look familiar to you if you've used other programming languages. Our function looks just like a list. This is because it is! Our expression might add two numbers together but it’s also a list of three items in a valid list data structure.
Note Technically it's an
s-expression.
This is a feature of Clojure called homoiconicity, sometimes described as: "code is data, data is code". This concept is inherited from Clojure's parent language: Lisp.
Homoiconicity means that the program's structure is similar to its syntax. In this case Clojure programs are written in the form of lists. Hence you can gain insight into the program's internal workings by reading its code. This also makes metaprogramming really easy because Clojure's source code is a data structure and the language can treat it like one.
Now let's look more closely at the + function. Each function is a symbol. A symbol is a bare string of characters, like + or inc. Symbols have short names and full names. The short name is used to refer to it locally, for example +. The full name, or perhaps more accurately the fully qualified name, gives you a way to refer to the symbol unambiguously from anywhere. The fully qualified name of the + symbol is clojure.core/+. The clojure.core being the fundamental library of the Clojure language. We can refer to + in it's fully qualified form here:
user=> (clojure.core/+ 1 1)
2
Symbols refer to other things; generally they point to values. Think about them as a name or identifier that points to a concept: + is the name, "adding" is the concept. When Clojure encounters a symbol it evaluates it by looking up its meaning. If it can't find a meaning it'll generate an error message, for example:
user=> (bob 1 2)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: bob in this context, compiling:(NO_SOURCE_PATH:1:1)
Clojure also has a syntax for stopping that evaluation. This is called quoting and it is achieved by prefixing the expression with a quotation mark: '.
user=> '(+ 1 1)
(+ 1 1)
This returns the symbol itself without evaluating it. This is important because often we want to do things, review things, or test things without evaluating.
For example, if we need to determine what type of thing something is in Clojure we can use the type function and quote the function like so:
user=> (type '+)
clojure.lang.Symbol
Here we can see that + is a Clojure language symbol.
Lists
Clojure also has a variety of data structures. Especially useful to us will be collections. Collections are groups of values, for example a list or a map.
Let's start by looking at lists. Lists are core to all Lisp-based languages (Lisp means "LISt Processing"). As we discovered above Clojure programs are essentially lists. So we're going to see a lot of them!
Lists have zero or more elements and are wrapped in parentheses.
user=> '(a b c)
(a b c)
Here we've created a list containing the elements a, b and c. We've quoted it because we don't want it evaluated. If we didn't quote it then evaluation would fail because none of the elements, a, b, etc are defined. Let's see that now.
user=> (a b c)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: a in this context, compiling:(NO_SOURCE_PATH:1:1)
We can do a few neat things with lists, for example add an element using the conj function.
user=> (conj '(a b c) 'd)
(d a b c)
You can see we've added a new element, d, to the front of the list. Why the front? Because a list is really a linked list and focusses on providing immediate access to the first value in the list. Lists are most useful for small collections of elements and when you need to read elements in a linear fashion.
We can also return values from a list using a variety of functions.
user=> (first '(a b c))
a
user=> (second '(a b c))
b
user=> (nth '(a b c) 2)
c
Here we've pulled out the first element, second element, and using the nth function, the third element.
This last, nth, function shows us a multi-argument function. The first argument is the list, '(a b c), and the second argument is the index value of the element we want to return, here 2.
Tip Like most programming languages Clojure starts counting from
0.
We can also create a list with the list function.
user=> (list 1 2 3)
(1 2 3)
Vectors
Another collection available to us is the vector. Vectors are like lists but they are optimized for random access to the elements by index. Vectors are created by adding zero or more elements inside square brackets.
Tip Most of the time, given the choice between a list and a vector, you should use a vector for data access. It's generally faster.
user=> '[a b c]
[a b c]
Like lists, we can again use conj to add to a vector.
user=> (conj '[a b c] 'd)
[a b c d]
You'll note the d element is added at the end because a vector isn't focussed on sequential access like a list.
There are some other useful functions we can use on lists and vectors, for example to get the last element in a list or vector.
user=> (last '[a b c d])
d
Or count the elements.
user=> (count '[a b c d])
4
Because vectors are designed to look up elements by index, we can also use them directly as functions, for example:
user=> ([1 2 3] 1)
2
Here we've retrieved the value, 2, at index 1.
We can create or convert an existing structure, like a list, into a vector with the vector function.
user=> (vector 1 2 3)
[1 2 3]
user=> (vector (list 1 2 3))
[1 2 3]
Sets
There's a final collection related to lists and vectors called a set. Sets are unordered collections of values, prefixed with # and wrapped in curly braces, { }. They are most useful for collections of values where you want to check a value or values is present.
user=> '#{a b c}
#{a c b}
You'll notice the set was returned in a different order. This is because sets are focussed on presence lookups so order doesn't matter quite so much.
Like lists and vectors we can use the conj function to add an element to a set.
user=> (conj '#{a b c} 'd)
#{a c b d}
Sets can never contain an element more than once, so adding an element which is already present does nothing. You can remove elements with the disj function.
user=> (disj '#{a b c d} 'd)
#{a c b}
The most common operation with a set is to check for the presence of a specific value, for this we use the contains? function.
user=> (contains? '#{a b c} 'c)
true
user=> (contains? '#{a b c} 'd)
false
Like a vector, you can also use the set itself as a function. This returns the value if it is present or nil if it is not.
user=> ('#{a b c} 'c)
c
user=> ('#{a b c} 'd)
nil
You can make a set out of any other collection with the set function.
user=> (set '[a b c])
#{a c b}
Here we've made a set out of a vector.
Maps
The last data structure we're going to look at is the map. Maps are key/value pairs enclosed in braces. You can think about them as being equivalent to a hash.
user=> {:a 1 :b 2}
{:b 2, :a 1}
Here we've defined a map with two key/value pairs: :a 1 and :b 2.
You'll note each key is prefixed with a :. This denotes another type of Clojure syntax: the keyword. A keyword is much like a symbol but instead of referencing another value it is merely a name or label. It's highly useful in data structures like maps to do lookups, you look up the keyword and return the value.
We can use the get function to retrieve a value.
(get {:a 1 :b 2} :a)
1
Here we've specified the keyword :a and asked Clojure if it is inside our map. It's returned the value in the key/value pair, 1.
If the key doesn't exist in the map then Clojure returns nil.
user=> (get {:a 1 :b 2} :c)
nil
The get function can also take a default value to return instead of nil, if the key doesn’t exist in that map.
user=> (get {:a 1 :b 2} :c :novalue)
:novalue
We can also use the map itself as a function.
user=> ({:a 1 :b 2} :a)
1
We can also use keywords as functions to look themselves up in a map.
user=> (:a {:a 1 :b 2})
1
To add a key/value pair to a map we use the assoc function.
user=> (assoc {:a 1 :b 2} :c 3)
{:c 3, :b 2, :a 1}
If a key isn't present then assoc adds it. If the key is present then assoc replaces the value.
user=> (assoc {:a 1 :b 2} :b 3)
{:b 3, :a 1}
To remove a key we use the dissoc function.
user=> (dissoc {:a 1 :b 2} :b)
{:a 1}
Note If you've come from the Ruby or Python world the terms list, set, vector and map might be a little new. But the syntax probably looks familiar. You can think about lists, vectors and sets as being very similar to arrays and maps being hashes.
Strings
We can also work with strings. Clojure lets you turn pretty much any value into a string using the str function.
user=> (str "holiday")
"holiday"
The str function turns anything specified into a string. We can also use it concatenate strings.
user=> (str "james needs " 2 " holidays")
"james needs 2 holidays"
Creating our own functions
Up until now we've run functions as stand-alone expressions, for example here's the inc function which increments arguments passed to it:
user=> (inc 1)
2
This isn't overly practical, except to demonstrate how a function works. If we want do more with Clojure we need to be able to define our own functions. To do this Clojure provides a function called fn. Let us construct our first function.
user=> (fn [a] (+ a 1))
So what's going on here? We've used the fn function to create a new function. The fn function takes a vector as an argument. This vector contains any arguments being passed to our function. Then we specify the actual action our function is going to perform. In our case we're mimicking the behavior of the inc function. The function will take the value of a and add 1 to it.
If we run this code now nothing will happen because a is currently unbound as we haven't defined a value for it. Let's run our function now.
user=> ((fn [x] (+ x 1)) 2)
3
Here we've evaluated our function and passed in an argument of 2. This is assigned to our a symbol and passed to the function. The function adds a, now set to 2, and 1 and returns the resulting value: 3.
There's also a shorthand for writing functions that we'll see occasionally in Riemann configurations.
user=> #(+ % 1)
This shorthand function is the equivalent of (fn [x] (+ x 1)) and we can call it to see the result.
user=> (#(+ % 1) 2)
3
Creating variables
But we're still a step from a named function and we're missing an important piece, how do we define our own variables to hold values? Clojure has a function called def that allows us to do this.
user=> (def smoker "joker")
#'user/smoker
The def function does two things:
- It creates a new type of object called a var. Vars, like symbols, are references to other values. You can see our new var
#'user/smoker returned as output of the def function.
- It binds a symbol to that var, here the symbol
smoker is bound to a var with a value of the string "joker".
When we evaluate a symbol pointing to a var it is replaced by the var's value. But because def also creates a symbol we can refer to our var like that too.
user=> user/smoker
"joker"
user=> smoker
"joker"
Where did this user/ come from? It's a Clojure namespace. Namespaces are a way Clojure organizes code and program structure. In this case the REPL creates a namespace called user/ by default. Remember we learnt earlier that a symbol has a short name, for example smoker that can be used locally to refer to it, and a full name. That full name, here user/smoker, would be used to refer to this symbol from another namespace.
We'll talk more about namespaces and use them to organize our Riemann configuration in the HOWTO. If you'd like to read more about them then there is an excellent explanation at http://www.braveclojure.com/organization/.
We can also use the type function to see the type of value the symbol references.
user=> (type smoker)
java.lang.String
Here we can see that the value smoker resolves to is a string.
Creating named functions
Now with the combination of def and fn we can create our own named functions.
user=> (def grow (fn [number] (* number 2)))
#'user/grow
Firstly, we've defined a var (and symbol) called grow. Inside that we've defined a function. Our function takes a single argument, number, and passes that number to the * function, the mathematical multiplication operator in Clojure, and multiplies it by 2.
Let's call our function now.
user=> (grow 10)
20
Here we've called the grow function and passed it a value of 10. The grow function multiplies that value and returns the result: 20. Pretty awesome eh?
But the syntax is a little cumbersome. Thankfully Clojure offers a shortcut to creating a var and binding it to a function called defn. Let's rewrite our function using this form.
user=> (defn grow [number] (* number 2))
#'user/grow
That's a little neater and easier to read. Now how about we add a second argument? Let's make both the number to be multiplied and the multiplier arguments.
user=> (defn grow [number multiple] (* number multiple))
#'user/grow
Let's call our grow function again.
user=> (grow 10)
ArityException Wrong number of args (1) passed to: user/grow clojure.lang.AFn.throwArity (AFn.java:429)
Ooops not enough arguments. Let's add the second argument.
user=> (grow 10 4)
40
We can also add a doc string to our function to help us articulate what it does.
(defn grow
"Multiplies numbers - can specify the number and multplier"
[number multiple]
(* number multiple)
)
We can access a function's doc string using the doc function.
user=> (doc grow)
-------------------------
user/grow
([number multiple])
Multiplies numbers - can specify the number and multplier
nil
The doc function tells us the full name of the function, the arguments it accepts, and returns the docstring.
That's the end of our crash course.
Learning more Clojure
I recommend trying to get an understanding of the basics of Clojure to get the most out of Riemann. If you'd like to start to learn a bit about Clojure then Kyle Kingsbury's excellent Clojure from the ground up series is an great place to start. This section is very much an abbreviated crash-course of sections of that tutorial and I can't thank Kyle enough for writing it. A reading of this tutorial will add signicantly to the knowledge we've shared here. I recommend at least a solid reading of the first three posts in the series:
Tip Another resource if you're interested in learning a bit more about the basics of Clojure is
http://learn-clojure.com/.
Just Enough Clojure for Riemann was originally published by James Turnbull at Kartar.Net on April 12, 2015.