Jul 10 2013

Using broken development frameworks , or why we don't use Zurmo

People often wonder why DBA's used to hate developers, and with DBA's also the System Engineers,
(note that I just expanded devops by adding dba's to the picture..)

So let me tell you a story ..

A couple of weeks ago one of our customers wanted to start experimenting with a new type of CRM. A gamified CRM.
Zurmo ...

So we set this thing up in a dev environment and started playing with it , while at first it looks nice ..
the application actually felt pretty slow.. however given that is a low resource development environment we looked no further.

Yet the next step is that we run into missing features, such as the fact that every contact you create by default is
set to private .. which really isn't productive for a CRM system where you want to be able to follow up on different
customer and share information.

So we tried figuring out what the database changes to do this in bulk would mean, surely it had to be a flag on the contact record .
Wrong, Zurmo uses an ORM for their database connectivity ...so their data model wasn't really trivial.

So we decided to look at the MySQL log file to figure out what db changes happened when updating the record
Yes there's better approaches but this one learned us a lot ..
The procedure I followed was pointing my browser to the page where I wanted to switch the checkbox,
log on to the mysql box, set global logging on . Clicked the checkbox and stopped global logging.

This gave me a log file with all the database actions required to make that one single change.
I had to cross check a number of times ... the file created by this short and small action was.
about 70K

Puzzled you start looking at the queries ...
The query list was full with "SELECT * FROM " stanza's ..
70K whopping K of queries that make your hair turn grey ...

I figured I'd file a bug .. but I couldn't find no bugtracker for Zurmo, only a forum (and forums are the most broken form of communication imvho) , yet the developers responded on Twitter.

The feedback wasn't really satisfying so we quickly decided that supporting this application was not something we would like to do..
and abandonned it..

The real question is who needs a Gamified CRM anyhow...

PS. So while finishing up this article on a late evening this week I might not have put in clear enough that the generated logfile was 70Kb .. I fear some people misunderstood that it generated 70.000 queries. Obviously a huge difference. But still the log file shouldn't have been bigger han 1Kb There should have been 2-3 queries max (https://github.com/KrisBuytaert/snippets/tree/master/zurmo)

But imvho if the size of the queries you are generating is bigger than the page you are generating you are most often doing it wrong.

Jul 03 2013

Done Ignite

Last month I gave an 5 minute Ignite talk at #devopsdays Amsterdam.

To summarize .. it was a rant against people that think that the end of a sprint their work is done, an idea they might have gotten from a broken definition of done.

Both the video recording and the slides are embedded below.

May 22 2013

Why you shouldn't hire a devops

Lately there have been a lot of organisations trying to hire a devops engineer.
I myselve have been asked to fill in devops roles ..

There's a number of issues with that.

The biggest problem is that I always have to ask what exactly the organisation is looking for.

So you want a devops engineer with experience in Linux, MongoDB, MySQL and Java , does that mean you want a Java developer who is familiar with MySQL and Linux and breaths a devops Culture.
Or a Linux expert who understands Java developers and knows how to tune Mongo and MySQL ?

It's absolutely unclear what you want when you are hiring "A devops engineer"

The second problem is that you are trying to hire people who are knowledgeable about devops,

Yet a lot of those people know that you can't do devops on your own , devops is not a jobtitle. devops is not a new devops team you create.

To some of them you are even making a fool out of yourselve, as to them you show that you don't understand devops

On top .. the ones that do apply for this fancy new devops role, are the ones that might not get the fact that the problem isn't about tooling but about people working together and helping eachother , so you end up hiring the wrong people.

Even in todays devops culture a system engineer is still a system engineer, and a developer is still a developer.
You might have developers supporting the build tool chain, or system engineers focussing on infrastructure automation.

But as John said almost 3 years ago they are good at their job.

Devops is not a word you slap onto a tool, a team or a person and expect magic to happen

Let's face it .. devops is hard, you can't do this on your own .. you need to find the right people ..

May 22 2013

Testing the RoboMow 510

A couple of weeks ago I noted fellow geek Peter Vanwelkenhuysen was talking about his #robomowtest experiences. obviously I wanted to know how he got to test one.

Some tweets and mails later I was making arrangements with Lennert van der Pols from Friendly Robotics too.
I've been thinking about buying a RoboMower for years . always have been delaying the purchase since we were planning to move
Since we moved, my wife has been a bit sceptic

So last week Lennert came by and gave us almost an hour of explanation on , he guided us trough the basic functions of the Robomow, gave us some advise on where to lay the cable.
And how to operate, the device. He also explained us that we could make a hole in the fence and have the Robomow , mow the front part too , rather than pick up the mower and move it manually.

Don't be surprised we got an Albino one :)

I didn't really have time to install the wire till later that evening.

Pinning down the back part of our lawn (we opted not to do both front and back during the test period) , took me about an hour. Some of the pins broke during the installation

We have a cherry tree with some big trunks and they could be troublesome .. so we evaded that part. as it was near the back of the garden and
Also in the left back behind the swing there is more moss than grass so we opted to lay the cable before the left back pole of the swing.

So after the first attempt of pinning down the cable in the ground , it was time to power on the Robomower , I was greeted with a pretty straight forward menu to configure it.
I`m one of those persons who really doesn't like user interfaces in my natvive language , Dutch. So I have pretty much all of my electronics in English... the problem with the
Robomow and English is that it assumes you live in a country that doesn't understand the metric system. So I had to convert the size of my garden to sft.
(I checked for options in the menu later, but Lennert confirmed there's no option to put to the metric system )

It takes a couple of times to align the basestation with the wire, but after the Robomow starts testing if it can crawl along your garden.

I have to admit that the frustrating part of the installation starts here. Around the trees we had some parts where the trunks were creating a little to steep trail, it took me about 4-5 runs before I eventually got
it right. The Robomow really takes its time to get stuff right. One of the problems we had was that one of the corners didn't really have that much grass, and each time the
mower passed it made the ground softer so after a couple of runs it actually got stuck I had to tighten in the corner.

So after installing the Robomow it was a good time to let it mow for the very first time, it ran for about an hour.

I was in the middle of a Hangout with some collegues as I suddenly noticed that the Robomow had stopped.
Apparently even while I had been very carefully putting the cable in I hadn't been carefull enough... The Robot had cut it's own wire and decided to call it a day.
(this was wednesday evening) .

On thursday morning I tried quickly tried fixing the cable as described in the video below...

However at my first attempt the blue isolater failed to actually lock the cables inside and I had to rush to work.

Fast forward to saturday evening.. the first timeslot I had to actually fix the cable... with a bit more patience I actually managed to secure the cables again , and the robot started doing it's work again.
I reprogrammed it to leave it's base every day at 8 in the morning rather than the default 1300, that way by the time the weather is good enough for the kids to play outside the lawn has been mown.

Monday and tuesday morning it rained ... on tuesday I saw the robot come out for a short while.. then decide to do the smart thing and go back to it's base station.
Today however the weather was dry.. so the robot had time to go on a suicide mission again ... yes that's right .. it managed to cut its cable again ...


Apr 19 2013

Evolution Woes and yum magic

I`m an oldschool guy .. I still love pop3(s) to get my mails locally and read them with my fat email client. Evolution.

So when gmail breaks their pop/imap infra I`m screwed for a while. I hate reading mail from a web gui and the collapsed threading model gmail uses makes me nauseus.

So I fiddled with my config .. disabled it.. deleted the account.. created it again. But even after gmail was up again . I couldn't access my mail from my favourite client. Yet from other clients it seemed to work.

This obviously is real fun when you are travelling and trying to keep an eye on a number of different email threads ..

So I`m back home from Paris now and spend some 10 minutes figuring out what could be wrong.

I ran into https://bugzilla.redhat.com/show_bug.cgi?id=949180 which points out that for newly created there is a problem with the keyring prompting

And refers to https://bugzilla.redhat.com/show_bug.cgi?id=953641 accounts Which states that gcr-3.6.2-3 breaks password prompt/keyring unlocks.

And indeed ..

  1. yum shell
  2. Loaded plugins: langpacks, presto, ps, puppetverify, refresh-packagekit
  3. > remove gcr
  4. > install gcr-3.6.2-1.fc18
  5. adobe-linux-x86_64 | 951 B 00:00
  6. fedora/18/x86_64/metalink | 31 kB 00:00
  7. google-chrome | 951 B 00:00
  8. google-earth | 951 B 00:00
  9. google-talkplugin | 951 B 00:00
  10. rpmfusion-free-updates | 3.3 kB 00:00
  11. rpmfusion-nonfree-updates | 3.3 kB 00:00
  12. updates/18/x86_64/metalink | 24 kB 00:00
  13. rpmfusion-free-updates/primary_db | 279 kB 00:01
  14. > run
  15. --> Running transaction check
  16. ---> Package gcr.x86_64 0:3.6.2-1.fc18 will be installed
  17. ---> Package gcr.x86_64 0:3.6.2-3.fc18 will be erased
  18. --> Finished Dependency Resolution
  20. ================================================================================
  21. Package Arch Version Repository Size
  22. ================================================================================
  23. Installing:
  24. gcr x86_64 3.6.2-1.fc18 fedora 627 k
  25. Removing:
  26. gcr x86_64 3.6.2-3.fc18 @updates 2.3 M
  28. Transaction Summary
  29. ================================================================================
  30. Install 1 Package
  31. Remove 1 Package
  33. Total download size: 627 k
  34. Is this ok [y/N]: y
  35. Downloading Packages:
  36. gcr-3.6.2-1.fc18.x86_64.rpm | 627 kB 00:02
  37. Running Transaction Check
  38. Running Transaction Test
  39. Transaction Test Succeeded
  40. Running Transaction
  41. Installing : gcr-3.6.2-1.fc18.x86_64 1/2
  42. Cleanup : gcr-3.6.2-3.fc18.x86_64 2/2
  43. Verifying : gcr-3.6.2-1.fc18.x86_64 1/2
  44. Verifying : gcr-3.6.2-3.fc18.x86_64 2/2
  45. Removed:
  46. gcr.x86_64 0:3.6.2-3.fc18
  47. Installed:
  48. gcr.x86_64 0:3.6.2-1.fc18
  50. Finished Transaction
  51. > quit
  52. > Leaving Shell

Solved the problem

Mar 19 2013

Initial loadays speakers announced

Loadays is coming up soon .. 6 and 7 april ..
Loadays is the Linux and Open Administration conference of the low lands , held in Antwerp, Belgium

We've just published the initial batch of speakers

  • Being a Sysadmin at a company full of Sysadmins (Cody Herriges)
  • OpenNebula Fundamentals (Jaime Melis)
  • Integrate UEFI into REAR (Gratien D'haese)
  • Puppet v3 and Hiera (Garrett Honeycutt)
  • Integrating Linux into an Active Directory domain (Gábor Nyers)
  • Normalised instance provisioning for dev, on-premise and public clouds (Karanbir Singh)
  • C.R.E.A.M : Cache Rules Everything Around Me (Thijs Feryn)
  • OpenLDAP's Lightning Memory-Mapped DB (Howard Chu)
  • What's new in syslog-ng (Peter Czanik)
  • Introduction to Ansible (Jan Piet Mens)
  • Network Block Device: network-based block storage for Linux systems (Wouter Verhelst)


  • Building your Enterprise Cloud with OpenNebula (Jaime Melis)
  • Puppet tutorial (Garrett Honeycutt)
  • Automated Everything - Setting up an openQRM Cloud (Matthias Rechenburg)
  • Tutorial about provisioning and management using Ansible (Dag Wieers - Jeroen Hoekx)

There's more to come !

Feb 13 2013

Love, MonitoringLove

Last year we were pretty negative about Monitoring, We shouted out that MonitoringSucked ... A year has passed and a lot has changed ... most importantly our new found love for monitoring, thanks to an inspirational Ignite talk by Ulf Mansson at devopsdays Rome.

Right after Fosdem about 20 people showed up at the #monitoringlove hacksessions hosted at the Inuits.eu offices to work on Open Source monitoring projects and exchange ideas. Some completely new people, some people with already a lot of experience.

Amongst the projects that were worked on was Maciej working on Packaging graphite for Debian, Ohter people were fixing bugs in Puppet , I spent some time with a vagrant box to deploy Sensu using Puppet. Last time I was playing with Sensu was on the flight back from PuppetCon , I gave up the fight with
RabbitMQ and SSL because I had no internet connection .. and now Ulf just pointed out that I could disable SSL at all, which resulted in having a POC up and running in no time.

Patrick was hacking on the Chef counterpart of the vagrant-puppet sensu setup a part of #monigusto. Ulf Mansson was getting dashing to display on a Raspberry Pi ... pretty cool stuff
And Jelle Smet was working on Pyseps a Python based Simple Event Processing Server framework that consume JSON docs from RabbitMQ and forwards them real time to other queues using MongoDB query syntax.

One of the more interesting discussion was around the topic of alerting and modeling business rules and input from a lot of different sources
in order to send the right alerts to the right people.

We explored different ideas like using BPM tools such as Activity or Rules engines like Ruby Rools. There exist some Saas providers that try to solve this need like PagerDuty and friends but obviously there is still a lot of work that needs to be done in order to create a viable alerting system based on different input sources.

The monitoring problem is not solved yet .. and it will stay around for a couple of years .. but with the advent of event such as Monitorama its clear
that an event like our #monitoring love hackessions is needed .. and is probably here to stay for a couple of years.

Feb 05 2013


During my Puppetcamp Gent talk last week, I explained how to get alerts based on trends from graphite. A number of people asked ,e how to do that.

First lets quickly explain why you might want to do that .
Sometimes you don't care about the current value of a metric..as an example take a Queing system .. there is no problem if there are messages added to the queue, not even if there are a lot of messages on the queue, there might however be a problem if over a certain period the number of messages on a queue stays to high.

In this example I`m monitoring the queue length of a hornetq setup which is exposed by JMX.
On the server runnnig HornetQ I have an exported resource that tells the JMXTrans server to send the MessageCount to graphite
(you could also do this using collectd plugins)

  1. @@jmxtrans::graphite {"MessageCountMonitor-${::fqdn}":
  2. jmxhost => hiera('hornetqserver'),
  3. jmxport => "5446",
  4. objtype => 'org.hornetq:type=Queue,*',
  5. attributes => '"MessageCount","MessagesAdded","ConsrCount"',
  6. resultalias => "hornetq",
  7. typenames => "name",
  8. graphitehost => hiera('graphite'),
  9. graphiteport => "2003",
  10. }

This gives me a computable url on which I can get the graphite view

The next step then is to configure a nagios check that verifies this data. For that I need to use the check_graphite plugin from Datacratic ..

Which can work with an nrpe config like

  1. ### File managed with puppet ###
  2. ### Served by: '<%= scope.lookupvar('::servername') %>'
  3. ### Module: '<%= scope.to_hash['module_name'] %>'
  4. ### Template source: '<%= template_source %>'
  6. command[check_hornetq]=/usr/lib64/nagios/plugins/check_graphite -u "http://<%= graphitehost%>/render?target=servers.<%= scope.lookupvar('::fqdn').gsub(/\./,'_')%>_5446.hornetq.docstore_private_trigger_notification.MessageCount&from=-30minutes&rawData=true" -w 2000 -c 20000

I define this check on the host where HornetQ is running as it then will map to that host on Icinga/Nagios rather than throw a host error on an unrelated host.

Jan 13 2013

#devops Conference Season heating up...

My last post has been a while ... in that I announced that there would be another event right before FOSDEM ... I totally forgot to announce it here but I`m sure that most of you already know. Yes. PuppetCamp Europe is coming back to it's roots... it's coming back to the city where we hosted it for the first time on this side of the ocean.. Gent. (that's 31/1 and 1/2 )

There is still time to register for the event http://puppetcampghent2013.eventbrite.com/ The schedule for the event will be published soonish (given that the selection was done on Friday evening and the speakers already received their feedback)

Co-located with PuppetCamp there will another Build and Open Source cloud day
Build a Cloud day with interesting topics such as Cloudstack, Ceph, devops and a really interesting talk on how the Spotify crowd is using Cloudstack.

So after those 2 days in Ghent, a lot of people will be warmed up for the open source event of the year FOSDEM.

And right after FOSDEM a bunch of people will gather at the Inuits office for 2 days of discussing, hacking and evangelizing around #monitoringlove (see previous post)

I almost forgot but even before the FOSDEM week-long there is the 2013 PHP Benelux Conference where I`ll be running a fresh version of the 7 Tools for your devops stack

There is a ****load of #DevopsDays events being planned this year .... the 2012 edition of New York will be taking place next week .
Austin and London have been announced and have opened up their CFP and Registration but different groups are organizing themselves to host events in Berlin, Mountain View, Tokyo, Barcelona, Paris, Amsterdam , Australia , Atlanta and many more ..

And there's even more to come .. April 6 and 7 will be the dates for the Linux Open Administration Days (Loadays 2013) in Antwerp again ... a nice small conference where people gather to discuss different interesting Linux topics .... Call For Presentations is still open ..Submit here

On the other side of the ocean there's DrupalCon Portland which once again is featuring a #devops track , and also the folks over at Agile 2013 (Nashville)
have a #devops track now. Both events are still looking for speakers ..

So if by the end of this year you still don't know what devops is all about .. you probably don't care and shouldn't be in the IT industry anyhow.

And those are only the events I`m somehow involved in for the next couple of months

Nov 13 2012

#monitoringlove hackfest

The age of #monitoringsucks is over, we're now transitioning into a #monitoringlove period.

That however doesn't mean al the work is done, we still need to do a lot of work and a lot of people are working on a lot of stuff.

Therefore like last year we are opening up our offices again right after Fosdem for a #monitoringlove hackfest

That's right on february 4 and 5 a bunch of people interrested to fix the problem will be meeting , discussing and hacking stuff together in Antwerp. In short a #monitoringlove hackathon

Inuits is opening up their offices for everybody who wants to join the effort Please let us (@KrisBuytaert) know if you want to join us in Antwerp. We'll provide caffeine, wireless, chairs and some snacks.

Please register upfront at : http://monitoringlove2013.eventbrite.com/

Obviously if you can't make it to Antwerp you can join the effort on ##monitoringsucks on Freenode or on Twitter.

The location will be Duboistraat 50 , Antwerp
It is about 10 minutes walk from the Antwerp Central Trainstation
Depending on Traffic Antwerp is about half an hour north of Brussels and there are hotels at walking distance from the venue.

Plenty of parking space is available on the other side of the Park

Read last years report http://www.krisbuytaert.be/blog/we-didnt-fix-it to get an idea of what will happen...

PS. Yes I`m trying to get another event of the ground the days before Fosdem but I`m still awaiting confirmation of the venue ..