I have been working as a sys admin with systemd for a while – technically since Centos 7 (or Debian Jessie). I have also built several rpm/deb packages using systemd.

After several months of working with systemd I can say I like it.

There is the only thing I (as a linux systems engineer) am missing. I loved to use TAB to get suggestions when using init.d scripts. Unfortunately, this doesn’t work anymore. Therefore I have to either remember the names of the services or list and grep all of them. This is very unhandy when I need to restart some service via my phone.

Except this only bug, systemd is very good way to go.


Why shared storage environment sucks

In our environment everything run on dell servers with vmware on them, connected to a redundant netapp storage with plenty of fast SAS drives. Typical HA vmware solution with plenty of performance.

After a bug in glibc was announced, I found a sufficient fix and decided to apply it globaly. Therefore I just executed yum update glibc on all 70 VMs we have and suddenly everything stoped responding.

The infrastructure just wasn’t able to handle such an io load and everything stoped working.

I then tested the same scenario on very similar environment based on local hard drives and kvm and although the io load was high, everything kept working properly.

I am now waiting for the official report about what has happened as the only response I got from their support was “We don’t know, let us know when it happenes again”.

[Update #1]: Yep, just another lag on Netapp. Shared storage environments really do not have future! Guys, it’s time to leave enterprice solutions and start using wicked technologies.

Gitlab-ci runner automatic deployment

I have been playing with gitlab for a while. As I mentioned several times before, they do amazing product that fits most of the situations. However, their deployment process is not what I would have expected. I understand that requiring ruby 2+ is a big challenge as it is not default ruby in debian word.

Anyway, Centos/RHEL released version 7 and there is default version of ruby 2+, therefore we can nicely automate the deploy of gitlab.

You can find prepared rpm packaging system here:

Feel free to get in touch if anything is not working.

Server hosting environment is not about uptime anymore

We have been using private cloud services based on some shared storage and vmware for a while. It is very stable though. However, we reached limit of available capacity and we needed to deploy few more servers. Not a big deal, we just asked our provider to add another host and expected to be able continue installing next day. We have very good payment history and don’t pay them small amount of money. Therefore I didn’t expect any issues. The answer shocked me – it took them 1 week to prepare an offer and another 2 weeks to deploy the server. I don’t understand that to be honest. We didn’t want anything special, just another host.

My country is small and competition in the server bussiness is high. In my previous work when we asked our rack hosting provider to lend us 10 servers, beacuse we had performance issues, we got them within ONE HOUR. That means – managed payment stuff, racked, cabled, prepared for deploy with IPMI information.

I got a call from our cloud provider today informing us that the server is ready in the datacenter, but they connected it with not working cable and it will be fixed another day. I am not going to blame them for not testing it, but I am wondering what is wrong with them. Cable change can be done even by a stupid monkey. What problem is to call the datacenter and use remote hands service?

The same thing is with spare hardware. Everytime I see some hosting company that they have hardware in their office, I am asking why hell on earth don’t you have it racked and prepared to deploy? It will greatly decrease deployment time when a customer orders server. You can rack all the servers when guy in the datacenter has time, perfectly prepare cables to decrease mess. When an order comes, you just change memory/hdds, configure correct VLAN (if it is not done automatically) and everything is deployed within minutes.

Last issue I had was with their support. Everytime I call them I have to go through customer check which usualy takes 5 minutes, because they can’t find us in the database. Even phones few years ago supported importing contacts from some external service. They have my phone number in their database. The only thing they have to do is translate data from the database to the phone. It would be amazing if I called them and the second side answered me: “Hello Peter, Joe from ServerHosting speaking. How can I help you?”

Hosting environment is small and even if you are a big player, small things like this can impress your customers and make them feel better with you.


I love using tools made by other people, because there is no reason to reinvent wheel. However, some times I can’t find anything usable. To help the community grow, I have decided to publish some of my work on github. Feel free to check my github.

There is not so much currently, but I am planning to publish some zabbix plugins I made, deb repository management tools and plenty other handy tools I use every day.

Scaling PHP

Before we started our project, I am personaly proud of the most, we didn’t know much about how to scale php applications. At the beggining there were only 2 of us – 2 linux linux guys with some php knowledge and few servers. Nevertheless, we weren’t afraid of facing strange issues. 4 years later, we have one of the biggest sites in our country.

You can’t even imagine how many downtimes we had due to lack of knowledge. I regret not having read “Scaling PHP book” earlier. There is described everything we went through – which also means everything that caused our downtimes.

Even though I don’t read much, I spent 30 minutes a day when I was in a tram/tube to/from work and I finished this book in 3 weeks.

If you still work on yourself (which you definetely might) this book is worth reading.

Thank you Steven for writing such a great book.