Warning: Could not retrieve fact ipaddress

We use puppet heavily and we really like it. We use it in master-less setup. I will write about it later some day.

Nevertheless as we move more to IPv6 only server deployment, we started receiving strange messages after one of our migrations.

Warning: Could not retrieve fact ipaddress

Of course, server doesn’t have IPv4 address except loopback, we still haven’t been brave enough to use 100% IPv6 setup with IPv4 disabled.

We weren’t able to find where it comes from, so we just workaround our problem in our puppet-run.sh script. As we don’t actually use IP address in our puppet configurations, we just pretended to have one:

# Hotfix: "Warning: Could not retrieve fact ipaddress" on IPv6 only servers

ipaddress=$(facter ipaddress)
if [ "x$ipaddress" == "x" ]; then
    export FACTER_ipaddress="127.0.1.1"
fi

And error message disappeared.

If you know where it comes from, feel free to let me know.

Advertisements

Dell N3000 (N3048/N3024) – Unsupported LACP fallback

Dell N3000 series looks nice – it’s relatively cheap, supports 2x10G uplink ports, has two power supplies and many features in their spec sheet. Therefore I bought one to test it.

There are however some downsides, which made  this switch useless for me. One of them is missing LACP fallback. With this switch, you can’t configure LACP and have PXE provisioning working at the same time. The switch is in blocking state when there is no LACP connection established.

Unfortunately, lack of this feature is not mentioned on their site.

r1-sw1>enable
r1-sw1#configure terminal
r1-sw1(config)#interface Gigabitethernet 1/0/12
r1-sw1(config-if-Gi1/0/12)#channel-group 2 ?
mode       Configure the interface to port-channel mode.
r1-sw1(config-if-Gi1/0/12)#channel-group 2 mode ?
active       Force the port to port-channel with LACP.
on              Force the port to port-channel without LACP.
r1-sw1(config-if-Gi1/0/12)#channel-group 2 mode

Leason learnd: Before you buy a new switch, check the command reference in order to find out that the device you are about to buy supports everything you need.

How to ZFS on Debian Jessie

Just a quick note about how to install ZFS on Debian Jessie.

Almost every guide just suggest:

wget http://...zfsonlinux...deb
dpkg -i zfsonlinux.deb
apt-get update
apt-get install debian-zfs

Unfortunately it doesn’t work. What works for me is:

wget http://...zfsonlinux...deb
dpkg -i zfsonlinux.deb
apt-get update
apt-get install build-essential
apt-get install spl-dkms
apt-get install zfs-dkms
apt-get install debian-zfs

Error: Checksumfile /var/lib/puppet/state/state.yaml is corrupt

Error: Checksumfile /var/lib/puppet/state/state.yaml is corrupt ((/var/lib/puppet/state/state.yaml): could not find expected ':' while scanning a simple key at line 556 column 3); replacing

The problem was caused by very long title – in my case like this:

    php::fpm::config { 'disable_functions=... a lot of stuff in here ...': }

So I replaced that with:

 php::fpm::config { 'fpm - disable_functions':
    section => 'PHP',
    setting => 'disable_functions',
    value => '... a lot of stuff in here ...'
}

And the problem has been fixed.

Dualstack puppet firewall IPv4 and IPv6 with hiera

As IPv6 deployment grows around the world, we decided to prepare our infrastructure as well. Besides the fact that we have to buy a lot of new hardware as Nehalem Supermicro motherboards don’t support IPv6 on IPMI and IPv6 PXE boot, we also have to care about software layer of our servers.

All our servers are managed only and we are responsible for their availability and correct configuration. We use puppet with hiera to keep at least basic server configuration (such as firewall, sssd, installed basic packages, sysctl tunning, …) managed the same everywhere. And of course we use the great puppetlabs firewall module.

However, to achieve simple IPv6 and IPv4 firewall configuration we had to write a wrapper around it. I am going to share simplified version of our configuration.

hiera.conf

:hierarchy:
  - "data/9_local/%{module_name}"
  - "data/0_default/%{module_name}"

Directory 0_default is used for default configuration (such as allow icmp, ssh, loopback, … which we want to have everywhere the same. Directory 9_local is used for local server exceptions (external mysql connections, …)

site.pp

...
include service_firewall
...

modules/service_firewall/manifests/init.pp

# == Class: Service_firewall
#
# Firewall configuration service
#
# If enabled is 0, all rules are removed and server accepts everything
#
class service_firewall (
  $enabled,
) {
  # Load iptables resource
  resources { 'firewall': purge => true }
  # Set up firewall when enabled
  if $enabled == 1 {
    # Load the firewall
    Firewall {
      require => Class['service_firewall::include::pre'],
      before => Class['service_firewall::include::post'],
    }
    class { '::firewall': }
    # Run additional rules - dualstack (IPv4 and IPv6 shared rules)
    $dualstack = hiera_hash('service_firewall::dualstack')
    service_firewall::dualstack { 'dualstack_hiera': hash => $dualstack }
    # Run additional rules - IPv4 specific
    $ipv4 = suffix_hash_title(hiera_hash('service_firewall::ipv4', {}), 'IPv4 only')
    create_resources('firewall', $ipv4)
    # Run additional rules - IPv6 specific
    $ipv6 = suffix_hash_title(hiera_hash('service_firewall::ipv6', {}), 'IPv6 only')
    create_resources('firewall', $ipv6, { 'provider' => 'ip6tables' })
    # Pre and post
    class { 'service_firewall::include::pre': }
    class { 'service_firewall::include::post': }
  }
}

modules/service_firewall/manifests/dualstack.pp

define service_firewall::dualstack (
  $hash,
) {
  $ipv4 = suffix_hash_title($hash, ' IPv4 dualstack')
  $ipv6 = suffix_hash_title($hash, ' IPv6 dualstack')
  create_resources('firewall', $ipv4)
  create_resources('firewall', $ipv6, { 'provider' => 'ip6tables' })
}

I think there is nothing exceptional except the suffix_hash_title function, which I had to write in order to prevent 2 the same titles to occure. It only takes the hash and adds suffix to every key.

modules/service_firewall/lib/puppet/parser/functions/suffix_hash_title.rb

module Puppet::Parser::Functions
  newfunction(:suffix_hash_title, :type => :rvalue) do |args|
    result = {}
    if args[0].class == Hash and args[1].class == String
      args[0].each do |title, values|
        result[title + args[1]] = values
      end
    end
    return result
  end
end

And that’s it. service_firewall::include::pre and post are the same as in the puppetlabs example.

Let’s now take a look at yaml files.

data/0_default/service_firewall.yaml

---
service_firewall::enabled: 1
service_firewall::dualstack:
  '800 allow SSH connections':
    action: 'accept'
    proto: 'tcp'
    port: 22
service_firewall::ipv6: {}
service_firewall::ipv4: {}

data/9_local/service_firewall.yaml

---
service_firewall::dualstack:
  '500 http and https':
    action: 'accept'
    proto: 'tcp'
    port: [ 80, 443 ]
service_firewall::ipv4:
  '600 allow access from the office':
    action: 'accept'
    source: '1.2.3.4'

LXC Debian Jessie DHCP address not leased

I am currently playing around with LXC containers for development environment. It is lightweight container virtualization, which is great for it’s low resource requirements.

In our environment, we have dedicated one server for containers and we want to have everyting on it. It means, we will have our own DNS and DHCP server based on dnsmasq in there, which will automatically assign IP addreses to containers and also will manage internal DNS for the development VMs.

The problem, I encountered was that after a clear instalation, my container didn’t want to accept any address from DHCP. When running dhcpdump on the host, I could see that client sent DHCPDISCOVER and the server replied with DHCPOFFER. Unfortunately, the reply was not accepted by the VM. VM then sent another request after timeout expired.

After some googling I found there is a bug isc-dhcp-client described here: http://uli-heller.github.io/blog/2013/07/26/lxc-networking/.

Using iptables command on host solved the problem and container accepts the address correctly.

iptables -A POSTROUTING -t mangle -p udp –dport bootpc -j CHECKSUM –checksum-fill

However, the isc-dhcp-client package version mentioned in the article above differs from mine (perhaps the difference between ubuntu and debian):

root@lxc-test:~# dpkg -l | grep dhcp
ii isc-dhcp-client 4.3.1-6 amd64 DHCP client for automatically obtaining an IP address
ii isc-dhcp-common 4.3.1-6 amd64 common files used by all of the isc-dhcp packages
root@lxc-test:~#

systemd

I have been working as a sys admin with systemd for a while – technically since Centos 7 (or Debian Jessie). I have also built several rpm/deb packages using systemd.

After several months of working with systemd I can say I like it.

There is the only thing I (as a linux systems engineer) am missing. I loved to use TAB to get suggestions when using init.d scripts. Unfortunately, this doesn’t work anymore. Therefore I have to either remember the names of the services or list and grep all of them. This is very unhandy when I need to restart some service via my phone.

Except this only bug, systemd is very good way to go.