[IPv6] No support from Supermicro ipmi protocol

As we slowly move towards IPv6 only deployment, we sometimes reach a situation where there is no simple solution for our problem.

This time we found that supermicro out of band management cards are not accessible via ipmi protocol over IPv6. Instead of getting the asnwer we just received icmp6 message port unreachable. Web interface works well over IPv6 though.

After a while of digging in our firewall, we tried google and the answer was found very quickly: https://www.supermicro.com/support/faqs/faq.cfm?faq=21475

We use zabbix to monitor our infrastructure, which uses openipmi library to access ipmi. For some reason (I didn’t dig into it, so I don’t know why) it doesn’t fall back to ipv4 when IPv6 device is not accessible.

Instead of spending a lot of time by reading C code, which we don’t understand to, we fixed our problem on DNS side. All our devices are being accessed via DNS. Therefore we installed powerdns-recursor (3.6.2-2+deb8u3) on our monitoring server and wrote a simple lua filter, which drops AAAA records in the response when accessing to ipmi.

/etc/powerdns/recursor.conf

lua-dns-script=/etc/powerdns/dns-script.lua

/etc/powerdns/dns-script.lua

function postresolve(remoteip, domain, qtype, records, origrcode)
    -- ipmi.srw.cz returns only ipv4 records
    if (string.find(domain, ".ipmi.srw.cz.$")) then
        for key, value in pairs(records) do
            if (value.qtype == pdns.AAAA) then
                table.remove(records, key)
            end
        end
        return 0, records
    end

    -- Nothing has changed
    return -1,{}
end

Even though it’s not the best solution, it works. We hope we will get ipmi IPv6 support soon. Or perhaps it is time to switch over to snmp (yes, supermicro should support snmp these days) or their new rest api (supermicro added redfish protocol support), which is unfortunately paid feature.

Advertisements

Dell N3000 (N3048/N3024) – Unsupported LACP fallback

Dell N3000 series looks nice – it’s relatively cheap, supports 2x10G uplink ports, has two power supplies and many features in their spec sheet. Therefore I bought one to test it.

There are however some downsides, which made  this switch useless for me. One of them is missing LACP fallback. With this switch, you can’t configure LACP and have PXE provisioning working at the same time. The switch is in blocking state when there is no LACP connection established.

Unfortunately, lack of this feature is not mentioned on their site.

r1-sw1>enable
r1-sw1#configure terminal
r1-sw1(config)#interface Gigabitethernet 1/0/12
r1-sw1(config-if-Gi1/0/12)#channel-group 2 ?
mode       Configure the interface to port-channel mode.
r1-sw1(config-if-Gi1/0/12)#channel-group 2 mode ?
active       Force the port to port-channel with LACP.
on              Force the port to port-channel without LACP.
r1-sw1(config-if-Gi1/0/12)#channel-group 2 mode

Leason learnd: Before you buy a new switch, check the command reference in order to find out that the device you are about to buy supports everything you need.

How to ZFS on Debian Jessie

Just a quick note about how to install ZFS on Debian Jessie.

Almost every guide just suggest:

wget http://...zfsonlinux...deb
dpkg -i zfsonlinux.deb
apt-get update
apt-get install debian-zfs

Unfortunately it doesn’t work. What works for me is:

wget http://...zfsonlinux...deb
dpkg -i zfsonlinux.deb
apt-get update
apt-get install build-essential
apt-get install spl-dkms
apt-get install zfs-dkms
apt-get install debian-zfs

Error: Checksumfile /var/lib/puppet/state/state.yaml is corrupt

Error: Checksumfile /var/lib/puppet/state/state.yaml is corrupt ((/var/lib/puppet/state/state.yaml): could not find expected ':' while scanning a simple key at line 556 column 3); replacing

The problem was caused by very long title – in my case like this:

    php::fpm::config { 'disable_functions=... a lot of stuff in here ...': }

So I replaced that with:

 php::fpm::config { 'fpm - disable_functions':
    section => 'PHP',
    setting => 'disable_functions',
    value => '... a lot of stuff in here ...'
}

And the problem has been fixed.

Dualstack puppet firewall IPv4 and IPv6 with hiera

As IPv6 deployment grows around the world, we decided to prepare our infrastructure as well. Besides the fact that we have to buy a lot of new hardware as Nehalem Supermicro motherboards don’t support IPv6 on IPMI and IPv6 PXE boot, we also have to care about software layer of our servers.

All our servers are managed only and we are responsible for their availability and correct configuration. We use puppet with hiera to keep at least basic server configuration (such as firewall, sssd, installed basic packages, sysctl tunning, …) managed the same everywhere. And of course we use the great puppetlabs firewall module.

However, to achieve simple IPv6 and IPv4 firewall configuration we had to write a wrapper around it. I am going to share simplified version of our configuration.

hiera.conf

:hierarchy:
  - "data/9_local/%{module_name}"
  - "data/0_default/%{module_name}"

Directory 0_default is used for default configuration (such as allow icmp, ssh, loopback, … which we want to have everywhere the same. Directory 9_local is used for local server exceptions (external mysql connections, …)

site.pp

...
include service_firewall
...

modules/service_firewall/manifests/init.pp

# == Class: Service_firewall
#
# Firewall configuration service
#
# If enabled is 0, all rules are removed and server accepts everything
#
class service_firewall (
  $enabled,
) {
  # Load iptables resource
  resources { 'firewall': purge => true }
  # Set up firewall when enabled
  if $enabled == 1 {
    # Load the firewall
    Firewall {
      require => Class['service_firewall::include::pre'],
      before => Class['service_firewall::include::post'],
    }
    class { '::firewall': }
    # Run additional rules - dualstack (IPv4 and IPv6 shared rules)
    $dualstack = hiera_hash('service_firewall::dualstack')
    service_firewall::dualstack { 'dualstack_hiera': hash => $dualstack }
    # Run additional rules - IPv4 specific
    $ipv4 = suffix_hash_title(hiera_hash('service_firewall::ipv4', {}), 'IPv4 only')
    create_resources('firewall', $ipv4)
    # Run additional rules - IPv6 specific
    $ipv6 = suffix_hash_title(hiera_hash('service_firewall::ipv6', {}), 'IPv6 only')
    create_resources('firewall', $ipv6, { 'provider' => 'ip6tables' })
    # Pre and post
    class { 'service_firewall::include::pre': }
    class { 'service_firewall::include::post': }
  }
}

modules/service_firewall/manifests/dualstack.pp

define service_firewall::dualstack (
  $hash,
) {
  $ipv4 = suffix_hash_title($hash, ' IPv4 dualstack')
  $ipv6 = suffix_hash_title($hash, ' IPv6 dualstack')
  create_resources('firewall', $ipv4)
  create_resources('firewall', $ipv6, { 'provider' => 'ip6tables' })
}

I think there is nothing exceptional except the suffix_hash_title function, which I had to write in order to prevent 2 the same titles to occure. It only takes the hash and adds suffix to every key.

modules/service_firewall/lib/puppet/parser/functions/suffix_hash_title.rb

module Puppet::Parser::Functions
  newfunction(:suffix_hash_title, :type => :rvalue) do |args|
    result = {}
    if args[0].class == Hash and args[1].class == String
      args[0].each do |title, values|
        result[title + args[1]] = values
      end
    end
    return result
  end
end

And that’s it. service_firewall::include::pre and post are the same as in the puppetlabs example.

Let’s now take a look at yaml files.

data/0_default/service_firewall.yaml

---
service_firewall::enabled: 1
service_firewall::dualstack:
  '800 allow SSH connections':
    action: 'accept'
    proto: 'tcp'
    port: 22
service_firewall::ipv6: {}
service_firewall::ipv4: {}

data/9_local/service_firewall.yaml

---
service_firewall::dualstack:
  '500 http and https':
    action: 'accept'
    proto: 'tcp'
    port: [ 80, 443 ]
service_firewall::ipv4:
  '600 allow access from the office':
    action: 'accept'
    source: '1.2.3.4'

LXC Debian Jessie DHCP address not leased

I am currently playing around with LXC containers for development environment. It is lightweight container virtualization, which is great for it’s low resource requirements.

In our environment, we have dedicated one server for containers and we want to have everyting on it. It means, we will have our own DNS and DHCP server based on dnsmasq in there, which will automatically assign IP addreses to containers and also will manage internal DNS for the development VMs.

The problem, I encountered was that after a clear instalation, my container didn’t want to accept any address from DHCP. When running dhcpdump on the host, I could see that client sent DHCPDISCOVER and the server replied with DHCPOFFER. Unfortunately, the reply was not accepted by the VM. VM then sent another request after timeout expired.

After some googling I found there is a bug isc-dhcp-client described here: http://uli-heller.github.io/blog/2013/07/26/lxc-networking/.

Using iptables command on host solved the problem and container accepts the address correctly.

iptables -A POSTROUTING -t mangle -p udp –dport bootpc -j CHECKSUM –checksum-fill

However, the isc-dhcp-client package version mentioned in the article above differs from mine (perhaps the difference between ubuntu and debian):

root@lxc-test:~# dpkg -l | grep dhcp
ii isc-dhcp-client 4.3.1-6 amd64 DHCP client for automatically obtaining an IP address
ii isc-dhcp-common 4.3.1-6 amd64 common files used by all of the isc-dhcp packages
root@lxc-test:~#

systemd

I have been working as a sys admin with systemd for a while – technically since Centos 7 (or Debian Jessie). I have also built several rpm/deb packages using systemd.

After several months of working with systemd I can say I like it.

There is the only thing I (as a linux systems engineer) am missing. I loved to use TAB to get suggestions when using init.d scripts. Unfortunately, this doesn’t work anymore. Therefore I have to either remember the names of the services or list and grep all of them. This is very unhandy when I need to restart some service via my phone.

Except this only bug, systemd is very good way to go.