This is a blog post that was written in 2013, but somehow was forgotten about. So here is a bit of history!
— Andrew Parker
Last month I got the chance to attend the Monitorama conference.
This was out and out the best conference I’ve visited so far this year for learning. The conference was organised as a day of lectures by notable people in the field, followed by a day of workshops (practical ‘follow the talk on your laptop’ style sessions) on various monitoring and visualisation tools in parallel with a day of ‘Hackathon’ (working on projects).
The talks I attended covered some topics I’m already very familiar with (E.g. logstash), and some topics which I’m much less familiar with (E.g. the R workshop). The level of technical detail was generally great – not overfacing if you were a beginner, but also with something to learn if you’re a seasoned user.
In the second day, in the morning, I teamed up with Jason from TIM Group, and we were a little selfish – working on scratching our own itch. In the afternoon we broke off and attended some of the workshops, which were interesting. I was extremely surprised (and pleased!) that our project won the 3rd prize for a group project at the conference – that was unexpected given that we’d gone off and solved our own reporting issue, rather than tackling a more generic monitoring problem which would have helped a larger subset of people! We also managed to build something functional for our needs, and get it deployed into production within the 5 hours we had to work on it.
I hope that readers will forgive me if I spend a few paragraphs telling you about what we built (and why!):
We used to deploy Foreman, which we used to view / search through puppet run reports. Unfortunately, with our recent upgrade to Puppet 3.0, our Foreman installation needed upgrading too.
The version of Foreman we were running was ancient, and it’s since gained a massive number of features – however the only feature we were using from it was the report browser. This was causing us to have mysql on our monitoring machines, just to support this application, and re-packaging the latest version to our internal standards proved to be a non-trivial exercise.
We’d basically just disabled it to go ahead with the puppet 3.0 upgrade, with a plan to experiment with (try writing a proof of concept) using our logstash/Elasticsearch solution for the data transport and storage. I was able to very quickly hack up a reporting plugin for puppet, based off of some earlier work I’d found on github, and I’d been playing with the angularjs framework on the plane on the way over.
So, after about 5 hours hacking, we had bolted together Norman (excuse the bad pun).
This is, of course, still a simple and barely functional prototype; however it’s useable enough that after a couple more hours work we have unit tests (and green builds in Travis) and so that it was possible to deploy as an Elasticsearch plugin. There is still missing functionality from what we replaced in Foreman, however none of it is truely essential and we should be able to add that gradually as we have time.