Sep 20 18:52

On Systemd and devops

If it's not broken , don't fix it.
Those who don't understand Unix are doomed to reinvent it, poorly
Complexity is the enemy of reliability.

Are some of the more frequently heard arguments in the systemd discussion. Indeed I see and hear a lot of senior Linux people react openly and probably way to late against the introduction of systemd in a lot of our favorite Linux distributions.

To me this is a typical example of the devops gap. The gap between developers writing code and operations needing to manage that code on production platforms at scale.
Often developers writing code that they think is useful and relevant while they are not listening to their target audience , in this case not the end users of the systems but the people that are maintaining the platforms. The people that work on a daily base with these tools.

I have had numerous conversations with people in favor and against systemd, till today I have not found a single general purpose use case that could convince me of the relevance of this large change in our platforms. I've found edge cases where it might be relevant. but not mainstream ones. I've also seen much more people against it than in favor. I've invited speakers to conference to come and teach me. I've probably spoken to the wrong people,

But this is not supposed to be yet another systemd rant.. I want to tackle a bigger problem. The problem that this change and some others have been forced upon us by distributions that should be open, and listen to their users, apparently both Debian and Fedora/RHEL failed largely but somehow fail to listen to their respective communities. Yes we know that e.g Fedora is the development platform and acts as a preview of what might come up in RHEL and thus CentOS later , but not everything eventually ends up in RHEL. So it's not like we didn't have an 'acceptance' platform where we could play with the new technology. The main problem here is that we had no simple way to stop the pipeline, it really feels like that long ago Friday evening rush deploy. Not like a good conversation between developers and actual ops on the benefits and problems of implementing these changes. This feels like the developers of the distributions deciding what goes in from their own little silo and voting in 'private' committee.

It also feels like the ops people being to busy to react, "Someone else will respond to this change, it's trivial this change is wrong , someone else will block this for sure",

And the fact that indeed Operating System developers, like Fedora and Debian friends kinda live in their own silo. (specifically not listing CentOS here..)

So my bigger question is .. how do we prevent this from happening again.. how do we make sure that distributions actually listen to their core users and not just the distribution developers.

Rest assured, Systemd is not the only case with this problem .. there's plenty of cases where features that were used by people, sometimes even the something people considered the core feature of a project got changed or even got ripped out by the developers because they didn't realize they were being used, sometimes almost killing that Open Source project by accident.
And I don't want that to happen to some of my favourite open source projects ..

Aug 12 20:23

Upcoming Conferences

After not being able to give my planned Ignite at #devopsdays Amsterdam because I was down with the flu here's some fresh opportunities to listen to my rants :)

In September I`ll be talking at PuppetConf 2014, San Francisco , USA about some of the horror stories we went trough over the past couple of years when deploying infrastructure the automated fasion.

Just one week later I`ll be opening the #devops track at DrupalCon Amsterdam together with @cyberswat (Kevin Bridges) where we'll talk about the current state of #drupal and #devops , We'll be reopening the #drupal and #devops survey shortly, more info about that later here..

Just a couple of weeks later I will be ranting about Packaging software on Linux at LinuxConf Europe in Dusseldorf, Germany

And in November , I`m headed to Nuremberg, Germany where I will be opening the Open Source Monitoring Conference tinkering about the current state of Open Source Monitoring, do we love it .. or does it still suck :)

That's all ..
for now ..

Aug 12 20:18

Fedora 20 Annoyancies

  1. yum install docker
  2.  
  3. docker run --rm -i -t -e "FLAPJACK_BUILD_REF=6ba5794" \
  4. > -e "FLAPJACK_PACKAGE_VERSION=1.0.0~rc3~20140729T232100-6ba5794-1" \
  5. > flapjack/omnibus-ubuntu bash -c \
  6. > "cd omnibus-flapjack ; \
  7. > git pull ; \
  8. > bundle install --binstubs ; \
  9. > bin/omnibus build --log-level=info flapjack ; \
  10. > bash"
  11. docker - version 1.5
  12. Copyright 2003, Ben Jansens <ben@orodu.net>
  13.  
  14. Usage: docker [OPTIONS]
  15.  
  16. Options:
  17. -help Show this help.
  18. -display DISLPAY The X display to connect to.
  19. -border The width of the border to put around the
  20. system tray icons. Defaults to 1.
  21. -vertical Line up the icons vertically. Defaults to
  22. horizontally.
  23. -wmaker WindowMaker mode. This makes docker a
  24. fixed size (64x64) to appear nicely in
  25. in WindowMaker.
  26. Note: In this mode, you have a fixed
  27. number of icons that docker can hold.
  28. -iconsize SIZE The size (width and height) to display
  29. icons as in the system tray. Defaults to
  30. 24.
  31.  
  32.  
  33. [root@mine ~]# rpm -qf /bin/docker
  34. docker-1.5-10.fc20.x86_64
  35. [root@mine ~]# rpm -qi docker
  36. Name : docker
  37. Version : 1.5
  38. Release : 10.fc20
  39. Architecture: x86_64
  40. Install Date: Sun 03 Aug 2014 12:41:57 PM CEST
  41. Group : User Interface/X
  42. Size : 40691
  43. License : GPL+
  44. Signature : RSA/SHA256, Wed 07 Aug 2013 10:02:56 AM CEST, Key ID 2eb161fa246110c1
  45. Source RPM : docker-1.5-10.fc20.src.rpm
  46. Build Date : Sat 03 Aug 2013 10:30:23 AM CEST
  47. Build Host : buildvm-07.phx2.fedoraproject.org
  48. Relocations : (not relocatable)
  49. Packager : Fedora Project
  50. Vendor : Fedora Project
  51. URL : <a href="http://icculus.org/openbox/2/docker/<br />
  52. Summary" title="http://icculus.org/openbox/2/docker/<br />
  53. Summary">http://icculus.org/openbox/2/docker/<br />
  54. Summary</a> : KDE and GNOME2 system tray replacement docking application
  55. Description :
  56. Docker is a docking application (WindowMaker dock app) which acts as a system
  57. tray for KDE and GNOME2. It can be used to replace the panel in either
  58. environment, allowing you to have a system tray without running the KDE/GNOME
  59. panel or environment.
  60. [root@mine ~]# yum remove docker
  61. [root@mine ~]# yum install docker-io
  62.  
  63. Installed:
  64. docker-io.x86_64 0:1.0.0-9.fc20
  65.  
  66. Complete!

Aug 12 20:17

Ubuntu 14.04 on an Dell XPS 15 9530

So I had the chance to unbox and install a fresh Dell XPS-15 9530
Dell ships the XPS13's with Ubuntu, but for some crazy reason it does not do the same with the XPS15 which imvho is much more
appropriate for a developer as it actually has a usable size screen.

That means they also force you to boot into some proprietary OS the very first time before you can even try get into the boot options menu.
which is where you need to be.

The goal was to get a fresh Ubuntu on the box , nothing more.

There's a large number of tips scattered on the internet such as

- Use the USB 2 connector (that's the one furthest to the back on the right side of the device)
- Disable secure booting
- Put the box in Legacy mode
- Do not enable external repositories, and do not install updates during the installation.

The combination of all of them worked. I booted form USB and 20 minutes later stuff worked. But it really was after trying them all ..

Profit.

wireless works, touchscreen works, touchpad works, hdmi works,

Screen resolution is awesome 3000x1940

PS. For those who wonder .. I`m still on Fedora 20 :)

Jun 04 15:55

Jenkins, Puppet, Graphite, Logstash and YOU

This is a repost of an article I wrote for the Acquia Blog some time ago.

As mentioned before, devops can be summarized by talking about culture, automation, monitoring metrics and sharing. Although devops is not about tooling, there are a number of open source tools out there that will be able to help you achieve your goals. Some of those tools will also enable better communication between your development and operations teams.

When we talk about Continuous Integration and Continuous Deployment we need a number of tools to help us there. We need to be able to build reproducible artifacts which we can test. And we need a reproducible infrastructure which we can manage in a fast and sane way. To do that we need a Continuous Integration framework like Jenkins.

Formerly known as Hudson, Jenkins has been around for a while. The open source project was initially very popular in the Java community but has now gained popularity in different environments. Jenkins allows you to create reproducible Build and Test scenarios and perform reporting on those. It will provide you with a uniform and managed way to , Build, Test, Release and Trigger the deployment of new Artifacts, both traditional software and infrastructure as code-based projects. Jenkins has a vibrant community that builds new plugins for the tool in different kinds of languages. People use it to build their deployment pipelines, automatically check out new versions of the source code, syntax test it and style test it. If needed, users can compile the software, triggering unit tests, uploading a tested artifact into a repository so it is ready to be deployed on a new platform level.

Jenkins then can trigger an automated way to deploy the tested software on its new target platform. Whether that be development, testing, user acceptance or production is just a parameter. Deployment should not be something we try first in production, it should be done the same on all platforms. The deltas between these platforms should be managed using a configuration management tool such as Puppet, Chef or friends.

In a way this means that Infrastructure as code is a testing dependency, as you also want to be able to deploy a platform to exactly the same state as it was before you ran your tests, so that you can compare the test results of your test runs and make sure they are correct. This means you need to be able to control the starting point of your test and tools like Puppet and Chef can help you here. Which tool you use is the least important part of the discussion, as the important part is that you adopt one of the tools and start treating your infrastructure the same way as you treat your code base: as a tested, stable, reproducible piece of software that you can deploy over and over in a predictable fashion.

Configuration management tools such as Puppet, Chef, CFengine are just a part of the ecosystem and integration with Orchestration and monitoring tools is needed as you want feedback on how your platform is behaving after the changes have been introduced. Lots of people measure the impact of a new deploy, and then we obviously move to the M part of CAMS.

There, Graphite is one of the most popular tools to store metrics. Plenty of other tools in the same area tried to go where Graphite is going , but both on flexibility, scalability and ease of use, not many tools allow developers and operations people to build dashboards for any metric they can think of in a matter of seconds.

Just sending a keyword, a timestamp and a value to the Graphite platform provides you with a large choice of actions that can be done with that metric. You can graph it, transform it, or even set an alert on it. Graphite takes out the complexity of similar tools together with an easy to use API for developers so they can integrate their own self service metrics into dashboards to be used by everyone.

One last tool that deserves our attention is Logstash. Initially just a tool to aggregate, index and search the log files of our platform, it is sometimes a huge missed source of relevant information about how our applications behave.. Logstash and it's Kibana+ElasticSearch ecosystem are now quickly evolving into a real time analytics platform. Implementing the Collect, Ship+Transform, Store and Display pattern we see emerge a lot in the #monitoringlove community. Logstash now allows us to turn boring old logfiles that people only started searching upon failure into valuable information that is being used by product owners and business manager to learn from on the behavior of their users.

Together with the Graphite-based dashboards we mentioned above, these tools help people start sharing their information and communicate better. When thinking about these tools, think about what you are doing, what goals you are trying to reach and where you need to improve. Because after all, devops is not solving a technical problem, it's trying to solve a business problem and bringing better value to the end user at a more sustainable pace. And in that way the biggest tool we need to use is YOU, as the person who enables communication.

Jun 04 15:54

Why Does DevOps Matter?

This is a repost of an article I wrote for the Acquia Blog some time ago.

People often ask, why does DevOps matter?

The honest answer to that question is...because having the development and operations team work together is the only way IT is successful.

Over the past few decades I've worked in different environments that include: small web start ups, big pharmaceutical companies, hardware engineering shops and large software companies and banks. All were trying different approaches to deliver quality software to their end users, customers, but most of them were failing badly.

Operations people were being pulled in at the last minute. A marketing campaign needed to go live at 5 p.m. because that's when the first radio commercial was scheduled to be broadcasted. At 11 a.m., the operations people still didn't know the campaign existed.

It was always the other person’s fault. Waterfall projects and large PID documents were the solution to all the problems. But people learned; they figured out that we can't expect humans to predict how long it would take to implement something they have never done before. Unfortunately, even today, only a small set of people understand the value of being agile and that we cannot break a project down to its granular details without factoring in the “unpredictable.” The key element here is the “uncertainty” of the many project pieces.

So on came the agile movement and software development became much smoother.
People agreed on time boxing a reasonable set of work that would result in delivering useful functionality in frequent batches. Yet, on the day of deployment, all hell breaks loose because someone forgot to loop in the Ops team.

This is where my personal experience differs from a lot of others, because I was part of a development team building a product where the developers were sitting right next to the system administration team. Within sprints, our DevOps team was building both system features and application features, making the application highly available was a story on the board next to an actual end user feature.

In the old days, a new feature that was scheduled for Friday couldn't be brought online for a couple of days because it couldn't be deployed to production. In the new setup, deploying to production was a no brainer as we had already tested the automated deployment to the acceptance platform.

This brings us to the first benefit : Actually being able to go live.

The next problem came on a Wednesday evening. A major security issue had popped up in Drupal and an upgrade needed to be performed, however nobody dared to perform the upgrade as they were afraid of breaking the site. Some people had made changes, they hadn't put their config back in code base, and thus the site didn't get updated. This is the typical state of the majority of any type of website where people build something, deploy it and never look back. This is the case until disaster strikes and it hits the evening news.

Teams then learn that not only do they need to implement features and put their config changes in code, but also do continuous integration testing on their sites.

From doing continuous integration, they go to continuous delivery and continuous deployment, where an upgrade isn't a risk anymore but a normal event which happens automatically when all the tests are green. By implementing infrastructure as code, they now have achieved 2 goals. By implementing tests, we build the confidence that the code was working, but also made sure that the number of defects in that code base went down so the number of times people needed to dig back into old code to fix issue also came down.

By delivering better software in a much more regular way, it enables the security issues to be fixed faster, but also brings new features to market faster. With faster, we often mean that there is an change from releasing software on a bi-yearly basis to a release each sprint, to a release whenever a commit has passed a number of test criteria.

Because they started to involve other stakeholders, the value of their application grew as they had faster feedback and better usage statistics. The faster feedback meant that they weren't spending as much time on features nobody used, but focusing their efforts on things that mattered.

Having other stakeholders like systems and security teams involved with early metrics and taking in the non functional requirements into the backlog planning meant that the stability of the platform was growing. Rather than people spending hours and nights fixing production problems, Potential issues are now being tackled upfront because of the
communication between devs and ops. Also, scale and high availability have been built into the application upfront, rather than afterwards -- when it is too late.

So, in the end it comes down to the most important part, which is that devops creates more happiness. It creates more happy customers, developers, operations teams, managers, and investors and for a lot of people it improves not only application quality, but also their life quality.

Jun 04 15:49

The Rise of the DevOps movement

This is a repost of an article I wrote for the Acquia Blog some time ago.

DevOps, DevOps, DevOps … the whole world is talking about DevOps, but what is DevOps?

Since Munich 2012, DrupalCon had a dedicated devops track. After talking to
a lot of people in Prague last month, I realized that the concept of DevOps is still very unclear to a lot of developers. To a large part of the development community, DevOps development still means folks working on 'the infrastructure part' of the development life cycle and for some it just means simply deploying Drupal, being concerned about purely keeping the site alive etc.

Obviously that's not what DevOps is about, so let's take a step back and find out how it all started.

Like all good things, Drupal included, DevOps is a Belgian thing!

Back in 2009 DevopsDays Europe was created because a group of people met over and over again at different conferences throughout the world and didn’t have a common devops conference to go to. These individuals would talk about software delivery, deployment, build, scale, clustering, management, failure, monitoring and all the important things one needs to think about when running a modern web operation. These folks included Patrick Debois, Julian Simpson, Gildas Le Nadan, Jezz Humble, Chris Read, Matt Rechenburg , John Willis, Lindsay Holmswood and me - Kris Buytaert.

O’Reilly created a conference called, “Velocity,” and that sounded interesting to a bunch of us Europeans, but on our side of the ocean we had to resort to the existing Open Source, Unix, and Agile conferences. We didn't really have a common meeting ground yet. At CloudCamp Antwerp, in the Antwerp Zoo, I started talking to Patrick Debois about ways to fill this gap.

Many different events and activities like John Allspaw and Paul Hammond’s talk at “Velocity”, multiple twitter discussions influenced Patrick to create a DevOps specific event in Gent, which became the very first ‘DevopsDays'. DevopsDays Gent was not your traditional conference, it was a mix between a couple of formal presentations in the morning and open spaces in the afternoon. And those open spaces were where people got most value. The opportunity to talk to people with the same complex problems, with actual experiences in solving them, with stories both about success and failure etc. How do you deal with that oldskool system admin that doesn’t understand what configuration management can bring him? How do you do Kanban for operations while the developers are working in 2 week sprints? What tools do you use to monitor a highly volatile and expanding infrastructure?

From that very first DevopsDays in Gent several people spread out to organize other events John Willis and Damon Edwards started organizing DevopsDays Mountain View, and the European Edition started touring Europe. It wasn’t until this year that different local communities started organizing their own local DevopsDays, e.g in Atlanta, Portland, Austin, Berlin, Paris, Amsterdam, London, Barcelona and many more.

From this group of events a community has grown of people that care about bridging the gap between development and operations, a community of people that cares about delivering holistic business value to their organization.

As a community, we have realized that there needs to be more communication between the different stakeholders in an IT project lifecycle - business owners, developers, operations, network engineers, security engineers – everybody needs to be involved as soon as possible in the project in order to help each other and talk about solving potential pitfalls ages before the application goes live. And when it goes live the communication needs to stay alive too.. We need to talk about maintaining the application, scaling it, keeping it secure . Just think about how many Drupal sites are out there vulnerable to attackers because the required security updates have never been implemented. Why does this happen? It could be because many developers don't try to touch the site anymore..because they are afraid of breaking it.

And this is where automation will help.. if we can do automatic deployments and upgrades of a site because it is automatically tested when developers push their code, upgrading won't be that difficult of a task. Typically when people only update once in 6 months, its a painful and difficult process but when its automated and done regularly, it makes life so much easier.

This ultimately comes down to the idea that the involvement of developers doesn’t end at their last commit. Collaboration is key which allows every developer to play a key role in keeping the site up and running, for more happy users. After all software with no users has no value. The involvement of the developers in the ongoing operations of their software shouldn't end before the last end user stops using their applications.

In order to keep users happy we need to get feedback and metrics, starting from the very first phases of development all the way up to production. It means we need to monitor both our application and infrastructure and get metrics from all possible aspects, with that feedback we can learn about potential problems but also about successes.

Finally, summarizing this in an acronym coined by John Willis and Damon Edwards
- CAMS. CAMS says Devops is about Culture, Automation, Measurement and Sharing.
Getting the discussion going on how to do all of that, more specifically in a Drupal environment, is the sharing part .

Jan 07 21:34

Do you want a Panel dicussion at CfgMgmtCamp.eu ? If so comment who should be in it !

Yes
68% (28 votes)
No
32% (13 votes)
Total votes: 41
Dec 22 21:04

FOSDEM 2014 is coming

and with that almost a full week of side events.
For those who don't know FOSDEM, (where have you been hiding for the past 13 years ? ) Fosdem is the annual Free and Open Source Developers European meeting. If you are into open source , you just can't mis this event where thousands of likeminded people will meet.

And if 2 days of FOSDEM madness isn't enough people organise events around it.

Last year I organised PuppetCamp in Gent, the days before Fosdem and a MonitoringLove Hackfest in our office the 2 days after FOSDEM This year another marathon is planned.

On Friday (31/1/2014) the CentOs community is hosting a Dojo in Brussels at the IBM Forum. (Free, but registration required by the venue)

After the success of PuppetCamp in Gent last year we decided to open up the discussion and get more Infrastructure as Code people involved in a CfgMgmtCamp.eu

The keynotes for CfgMgmtCamp will include the leaders of the 3 most popular tools around , both Mark Burgess, Luke Kanies and Adam Jacob will present at the event which will take place in Gent right after Fosdem. We expect people from all the major communities including, but not limited to , Ansible, Salt, Chef, Puppet, CFengine, Rudder, Foreman and Juju (Free but registration required for catering)

And because 3 events in one week isn't enough the RedHat Community is hosting their Infrastructure.next conference after CfgMgmtCamp at the same venue. (Free but registration required for catering)

cya in Belgium next year..

Dec 12 2013

IPv4 Shortage can make you a billionaire

A couple of weeks ago I had this mail conversation...

I just had to share it ..


From:  sales@ipaddressconsultancy.com
To:  Kris Buytaert
Subject:  IPaddress Requirement
Date:  Tue, 05 Nov 2013 11:50:41 -0600 (11/05/2013 06:50:41 PM)

Hi Kris,

Thanks for your response. Please share your contact details including
skype ID so we can discuss further.

Regards

On 05.11.2013 08:26, Kris Buytaert wrote:
> On Mon, 2013-11-04 at 04:28 +0000, Sales wrote:
>> Hello,
>>
>> We are looking for IP addresses as we are growing Internet Solutions
>> Provider.
>>
>> We are looking for Ip addresses(Ipv4) anywhere from /22 to /16 Ipv4
>> to
>> host it. We look forward for your response to discuss further on the
>> pricing terms to take this forward.
>>
>> If you are not the concerned person to discuss about this, please
>> forward it to the appropriate department of your company .
>>
>> Please get back to us as soon as possible.
>>
>> Regards,
>
> Hi there,
>
> we can offer you a whole range of IP addresses..
>
> We can provide you with 127.0.0.1/24 , 192.168.0.0/16 ,172.16.0.0/12 and 10.0.0.0/8
>
> Please send us your best offer.
>
> greetings
>
> Kris Buytaert

Fatal error: Call to undefined function drupal_get_path() in /var/vhosts/www.krisbuytaert.be/htdocs/blog/includes/module.inc on line 285