jenkins

Jun 13 2016

Jenkins DSL and Heisenbugs

I`m working on getting even more moving parts automated, those who use Jenkins frequently probably also have Love - Hate relationship with it.

The love coming from the flexibility , stability and the power you get from it, the hate from it's UI. If you've ever had to create a new Jenkins job or even pipeline based on one that already existed you've gone trough the horror of click and paste errors , and you know where the hate breeds.

We've been trying to automate this with different levels of success, we've puppetized the XML jobs, we've used the Buildflow Plugin (reusing the same job for different pipelines is a bad idea..) We played with JJB running into issues with some plugins (Promoted Build) and most recently we have put our hope in the Job DSL.

While toying with the DSL I ran into a couple of interresting behaviours. Imagine you have an entry like this which is supposed to replace the $foldername with the content of the variable and actually take the correct upstream

  1. cloneWorkspace('${foldername}/dashing-dashboard-test', 'Successful')

You generate the job, look inside the Jenkins UI to verify what the build result was .. save the job and run it .. success ..
Then a couple of times later that same job gives an error ... It can't find the upstream job to copy the workspace from. You once again open up the job in the UI, look at it .. save it , run it again and then it works.. a typical case of Heisenbug ..

When you start looking closer to the XML of the job you notice ..

  1. <parentJobName>${foldername}/dashing-dashboard-test</parentJobName>

obviously wrong .. I should have used double quotes ..

But why doesn't it look wrong in the UI ? That's because the UI autoselects the first option from it's autogenerated pull down list .. Which actually contains the right upstream workplace I wanted to trigger (that will teach me to use 00 as a prefix for the foldername for all my tests..)

So when working with the DSL .. review the generated XML .. not just if the job works ..

Jun 04 2014

Jenkins, Puppet, Graphite, Logstash and YOU

This is a repost of an article I wrote for the Acquia Blog some time ago.

As mentioned before, devops can be summarized by talking about culture, automation, monitoring metrics and sharing. Although devops is not about tooling, there are a number of open source tools out there that will be able to help you achieve your goals. Some of those tools will also enable better communication between your development and operations teams.

When we talk about Continuous Integration and Continuous Deployment we need a number of tools to help us there. We need to be able to build reproducible artifacts which we can test. And we need a reproducible infrastructure which we can manage in a fast and sane way. To do that we need a Continuous Integration framework like Jenkins.

Formerly known as Hudson, Jenkins has been around for a while. The open source project was initially very popular in the Java community but has now gained popularity in different environments. Jenkins allows you to create reproducible Build and Test scenarios and perform reporting on those. It will provide you with a uniform and managed way to , Build, Test, Release and Trigger the deployment of new Artifacts, both traditional software and infrastructure as code-based projects. Jenkins has a vibrant community that builds new plugins for the tool in different kinds of languages. People use it to build their deployment pipelines, automatically check out new versions of the source code, syntax test it and style test it. If needed, users can compile the software, triggering unit tests, uploading a tested artifact into a repository so it is ready to be deployed on a new platform level.

Jenkins then can trigger an automated way to deploy the tested software on its new target platform. Whether that be development, testing, user acceptance or production is just a parameter. Deployment should not be something we try first in production, it should be done the same on all platforms. The deltas between these platforms should be managed using a configuration management tool such as Puppet, Chef or friends.

In a way this means that Infrastructure as code is a testing dependency, as you also want to be able to deploy a platform to exactly the same state as it was before you ran your tests, so that you can compare the test results of your test runs and make sure they are correct. This means you need to be able to control the starting point of your test and tools like Puppet and Chef can help you here. Which tool you use is the least important part of the discussion, as the important part is that you adopt one of the tools and start treating your infrastructure the same way as you treat your code base: as a tested, stable, reproducible piece of software that you can deploy over and over in a predictable fashion.

Configuration management tools such as Puppet, Chef, CFengine are just a part of the ecosystem and integration with Orchestration and monitoring tools is needed as you want feedback on how your platform is behaving after the changes have been introduced. Lots of people measure the impact of a new deploy, and then we obviously move to the M part of CAMS.

There, Graphite is one of the most popular tools to store metrics. Plenty of other tools in the same area tried to go where Graphite is going , but both on flexibility, scalability and ease of use, not many tools allow developers and operations people to build dashboards for any metric they can think of in a matter of seconds.

Just sending a keyword, a timestamp and a value to the Graphite platform provides you with a large choice of actions that can be done with that metric. You can graph it, transform it, or even set an alert on it. Graphite takes out the complexity of similar tools together with an easy to use API for developers so they can integrate their own self service metrics into dashboards to be used by everyone.

One last tool that deserves our attention is Logstash. Initially just a tool to aggregate, index and search the log files of our platform, it is sometimes a huge missed source of relevant information about how our applications behave.. Logstash and it's Kibana+ElasticSearch ecosystem are now quickly evolving into a real time analytics platform. Implementing the Collect, Ship+Transform, Store and Display pattern we see emerge a lot in the #monitoringlove community. Logstash now allows us to turn boring old logfiles that people only started searching upon failure into valuable information that is being used by product owners and business manager to learn from on the behavior of their users.

Together with the Graphite-based dashboards we mentioned above, these tools help people start sharing their information and communicate better. When thinking about these tools, think about what you are doing, what goals you are trying to reach and where you need to improve. Because after all, devops is not solving a technical problem, it's trying to solve a business problem and bringing better value to the end user at a more sustainable pace. And in that way the biggest tool we need to use is YOU, as the person who enables communication.

Sep 13 2012

Gentwerpen Devops Meetup & Conference Season Update

A couple of us have been taking about it a lot already .. we wanted to host a one day #devops event in .be already last year.. then talks about starting a meetup group started again with @wonko_be but it was @fs111 pushing the final button and calling the rest of the .be community to order, we've set a date
and the first session will take place (agenda still needs to be detirmined)

So all you Belgian devops enthousiasts, maark October 11th in your calendar and go register here

We already have 2 other venues (Gent, Boom) lined up .. but let's get this first one started :)

Next to that here's an update for the rest of my upcoming Conference Season :

  • Later this month I`ll be heading to San Francisco for a talk at PuppetConf 2012. I'll probably be around in the valley a bit earlier so if you anyone wants to meet up I`m open for suggestions.
    (Yes I asked Nick Stielau of Pantheon to host a #monitoringsucks / #devops meetup about Sensu but I should have predicted it was about to clash with the PuppetConf speakers dinner :((
  • I was thinking to swing by the MySQL- Connect conference but given the pricetag I don't think I`ll bother ... I am however thinking about crashing the hallway track , or tricking the Foreman Meetup to be colocated with a MySQL event again just like at Fosdem earlier this year
  • I will be attending the Jenkins User conference in San Francisco however before flying back to Europe
  • If you haven't noticed yet , Devopsdays is going to be in Rome this year on october 5 and 6. Registration
    is still open !
  • During the last weekend of october it's time for t-dose.org again. No news on the program yet.
  • And one week later I`ll be heading to Barcelona to speak at LinuxCon Europe I`m really looking forward to that last one again as it looks like the good old LinuxKongresses in Germany .. deep technical topics !
Aug 08 2012

Ruby Gems Yum Repo

For those of you that are looking for my old build-gems Github repo, given that lots of other Inuits collegues are using it too I've transferred ownership of that repo to the Inuits group.

It can now be found on https://github.com/Inuits/build-gems/ (Yes directory indexes are disabled on purpose..)

The build result has also been moved. We've replaced our static repo.inuits.be with a Pulp powered yum repo.

The new location of the rubygems rpm repo now is at http://pulp.inuits.eu/pulp/repos/rubygems/

Which can be used as

  1. [rubygems]
  2. name=RubyGems at Inuits
  3. baseurl=http://pulp.inuits.eu/pulp/repos/rubygems
  4. gpgcheck=0

or even

  1. yumrepo { 'rubygems':
  2. baseurl => 'http://pulp.inuits.eu/pulp/repos/rubygems',
  3. descr => 'RubyGems at Inuits',
  4. gpgcheck => '0',
  5. }

More repos are moving .. I`ll be updating Vagrant trees as I go ..

Oct 30 2011

A different shade of green

Back in late 1997 I had spent way too much time helping people to build websites and was fed up with customers wanting a different shade of green for the background of their website. I was fed up with the graphic artists that didn't want to understand the concept of a color pallet and browser safe colors and didn't understand the differences between print and web. So I decided to try not to work for the wannabe webexperts anymore and doing some real software.

Fast forward 15 years and I find myselve discussing the different shades of green with developers ... maybe it's time for some radical change again :)

You got to love Geek & Poke