2021-09-07

sudo apt-get update && sudo apt-get dist-upgrade

Debian 11 (codename Bullseye) was recently released. This was the smoothest upgrade I've experienced in some 20 years as a Debian user. In my haste, I completely forgot to first upgrade dpkg and apt, doing a straight dist-upgrade. Nonetheless, everything worked out of the box. No unresolved dependency cycles. Via my last-mile Gigabit connection, it took about 5 minutes to upgrade and reboot. Congratulations to everyone who made this possible!

Since the upgrade, only a handful of bugs were found. I filed bug reports. Over these past few days, maintainers have started responding. In once particular case, my report exposed a CVE caused by copy-pasted code between two similar packages. The source package fixed their code to something more secure a few years ago, while the destination package missed it. The situation has been brought to Debian's security team's attention and should be fixed over the next few days.

Afterthoughts

Having recently experienced hard-disk problems on my main desktop, upgrading to Bullseye made me revisit a few issues. One of these was the possibility of transiting to BTRFS. Last time I investigated the possibility was back when Ubuntu briefly switched their default filesystem to BRTFS. Back then, my feeling was that BRTFS wasn't ready for mainstream. For instance, the utility to convert an EXT2/3/4 partition to BTRFS corrupted the end of the partition. No thanks. However, in recent years, many large-scale online services have migrated to BRTFS and seem to be extremely happy with the result. Additionally, Linux kernel 5 added useful features such as background defragmentation. This got me pondering whether now would be a good time to migrate to BRTFS. Sadly it seems that the stock kernel shipping with Bullseye doesn't have any of these advanced features enabled in its configuration. Oh well.

Geode

The only point that has become problematic is my Geode hosts. For one things, upstream Rust maintainers have decided to ignore the fact that i686 is a specification and arbitrarily added compiler flags for more recent x86-32 CPUs to their i686 target. While Debian Rust maintainers have purposely downgraded the target, RustC still produces binaries that the Geode LX (essentially an i686 without PAE) cannot process. This affects fairly basic packages such as librsvg, which breaks SVG image support for a number of dependencies. Additionally, there's been persistent problems with systemd crashing on my Geode hosts whenever daemon-reload is issued. Then, a few days ago, problems started occurring with C++ binaries, because GCC-11 upstream enabled flags for more recent CPUs in their default i686 target. While I realize that SSE and similar recent CPU features produce better binaries, I cannot help but feel that treating CPU targets as anything else than a specification is a mistake. i686 is a specification. It is not a generic equivalent to x86-32.

2021-05-19

WebRTC fails tests on Firefox 78.10.0esr (64-bit)

Having to participate in many online events since the COVID crisis started, I've come to notice that few of the online clients work properly on the current Firefox ESR found in Debian. A quick visit at WebRTC Test confirmed that none of the tests in the Network and Connectivity section pass. Meanwhile, a Windows 10 laptop running Edge via the same network works just fine, so I have to assume that either a Firefox or Debian packaging issue is to blame, but I wouldn't know where to start. Any help? Thanks!

2021-02-28

Shipping Debian with GNOME X.XX.0 is an extremely bad idea

Since the freeze has slowly crept in, now is the time to revisit my pet peeve with Debian's release process: to publish a new Debian release as soon as GNOME published a new X.XX.0 version. This is an extremely bad idea: X.XX.0 releases tend to lack polish, their translations are not up-to-date and several silly bugs that hamper the user experience (what the Ubuntu guys call "paper cuts") exist. Those issues tend to be fixed later when GNOME X.XX.1, X.XX.2, etc. bugfix releases are published. However, Debian has a policy of not pushing non-security releases onto a stable distribution. In this particular case, there are only two valid alternatives: either release Bullseye with GNOME 3.38.X or change the Debian policy to allow pushing 3.40.X bugfix releases via bullseye-updates.

2021-02-15

OpenWRT: WRT54GL: Backfire: IPv6 issues

While having a Debian boxen as a router feels nice, I kept on longing for something smaller and quieter. I then remembered that I still had my old WRT54GL somewhere. After upgrading the OpenWRT firmware to the latest supported version for that hardware (Backfire 10.03.1, r29592), I installed radvd and wide-dhcpv6-client. Configuring radvd to deliver consistent results was easy enough.

The issue I keep on experiencing is the external interface (wan) dropping the IPv6 address it received from the ISP via router advertisement, which in turn kills the default IPv6 route to the outside world. Logging in via SSH and manually running "rdisc6 eth0.1" restores the IPv6 gateway. I just honestly wished I didn't have to do this every time I need to reboot the router.

Does this issue sound familiar to anyone? What was the solution?

PS: No, I won't just go and ditch this WRT54GL just because new toys exist on the market. This is obviously a software issue, so I need a software solution.

PPS: IPv6 pretty much works out of the box on the Debian boxen I had been using as my router. I previously wrote about this on my blog. Basically, it's unlikely to be an ISP issue.

2021-01-02

Help needed: clean up and submit KMS driver for Geode LX to LKML

Ever since X.org switched to rootless operation, the days of the Geode X.org driver have been numbered. The old codebase dates back from Geode's early days at Cyrix, was then updated by NSC to add support for their new GX2 architecture, from which AMD dropped GX1 support and added support for their new LX architecture. To put it mildly, that codebase is a serious mess.

However, at least the LX code comes with plenty of niceties, such as being able to detect when it runs on an OLPC XO-1 and to probe DCC pins to determine the optimal display resolution on other hardware. This still doesn't make the codebase cruft-free.

Anyhow, most Linux distributions have dropped support for anything older than i686 with PAE, which essentially means that the GX2 code is just for show. Debian is one of very few distributions whose x86-32 port still ships with i686 without PAE. In fact, the lowest common denominator kernel on i386 is configured for Geode (LX).

A while back, someone had started working on a KMS driver for the Geode LX. Through word of mouth, I got my hands on a copy of their Git tree. The driver worked reasonably well, but the codebase needs some polishing before it could be included in the Linux kernel tree.

Hence this call for help:

Is there anyone with good experience of the LKML coding standards who would be willing to clean up the driver's code and submit the patch to the LKML?

2020-11-03

Adding IPv6 support to my home LAN

A couple of year ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.

This also meant changing the settings on my router box for my new ISP.

I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.

Adding IPv6 support to my home LAN

I have been following the evolution of IPv6 ever since the KAME project produced the first IPv6 implementation. I have also been keeping track of the IPv4 address depletion.

Around the time the IPv6 Day was organized in 2011, I started investigating the situation of IPv6 support at local ISPs.

Well, never mind all those rumors about Finland being some high-tech mecca. Back then, no ISP went beyond testing their routers for IPv6 compatibility and producing white papers on what their limited test deployments accomplished.

Not that it matters much, in practice. Most IPv6 documentation out there, including Debian's own, still focuses on configuring transitional mechanisms, especially how to connect to a public IPv6 tunnel broker.

Relocating to a new flat and rethinking my home network to match gave me an opportunity to revisit the topic. Much to my delight, my current ISP offers native IPv6.

This prompted me to go back and read up on IPv6 one more time. One important detail:

IPv6 hosts are globally reachable.

The implications of this don't immediately spring to mind for someone used to IPv4 network address translation (NAT):

Any network service running on an IPv6 host can be reached by anyone anywhere.

Contrary to IPv4, there is no division between private and public IP addresses. Whereas a device behind an IPv4 NAT essentially is shielded from the outside world, IPv6 breaks this assumption in more than one way. Not only is the host reachable from anywhere, its default IPv6 address is a mathematical conversion (EUI-64) of the network interface's MAC address, which makes every connection forensically traceable to a unique device.

Basically, if you hadn't given much thought to firewalls until now, IPv6 should give you enough goose bumps to get around it. Tightening the configuration of every network service is also an absolute must. For instance, I configured sshd to only listen to private IPv4 addresses.

What /etc/network/interfaces might look like on an dual-stack (IPv4 + IPv6) host:

allow-hotplug enp9s0

iface enp9s0 inet dhcp
iface enp9s0 inet6 auto
	privext 2
	dhcp 1

The auto method means that IPv6 will be auto-configured using SLAAC; privext 2 enables IPv6 privacy options and specifies that we prefer connecting via the randomly-generated IPv6 address, rather than the EUI-64 calculated MAC specific address; dhcp 1 enables passive DHCPv6 to fetch additional routing information.

The above works for most desktop and laptop configurations.

Where things got more complicated is on the router. I decided early on to keep NAT to provide an IPv4 route to the outside world. Now how exactly is IPv6 routing done? Every node along the line must have its own IPv6 address... including the router's LAN interface. This is accomplished using the sample script found in Debian's IPv6 prefix delegation wiki page. I modified mine as follows (the rest of the script is omitted for clarity):

#Both LAN interfaces on my private network are bridged via br0
IA_PD_IFACE="br0"
IA_PD_SERVICES="dnsmasq"
IA_PD_IPV6CALC="/usr/bin/ipv6calc"

Just put the script at the suggested location. We'll need to request a prefix on the router's outside interface to utilize it. This gives us the following interfaces file:

allow-hotplug enp2s4 enp2s8 enp2s9
auto br0

iface enp2s4 inet dhcp
iface enp2s4 inet6 auto
	request_prefix 1
	privext 2
	dhcp 1

iface enp2s8 inet manual
iface enp2s8 inet6 manual

iface enp2s9 inet manual
iface enp2s9 inet6 manual

iface br0 inet static
	bridge_ports enp2s8 enp2s9
	address 10.10.10.254

iface br0 inet6 manual
	bridge_ports enp2s8 enp2s9
	# IPv6 from /etc/dhcp/dhclient-exit-hooks.d/prefix_delegation

The IPv4 NAT and IPv6 Bridge script on my router looks as follows:

#!/bin/sh
PATH="/usr/sbin:/sbin:/usr/bin:/bin"
wan=enp2s4
lan=br0
########################################################################
# IPv4 NAT
iptables -F; iptables -t nat -F; iptables -t mangle -F
iptables -X; iptables -t nat -X; iptables -t mangle -X
iptables -Z; iptables -t nat -Z; iptables -t mangle -Z
iptables -t nat -A POSTROUTING -o $wan -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
########################################################################
# IPv6 bridge
ip6tables -F; ip6tables -X; ip6tables -Z
# Default policy DROP
ip6tables -P FORWARD DROP
# Allow ICMPv6 forwarding
ip6tables -A FORWARD -p ipv6-icmp -j ACCEPT
# Allow established connections
ip6tables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
# Accept packets FROM LAN to everywhere
ip6tables -I FORWARD -i $lan -j ACCEPT
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 1 > /proc/sys/net/ipv6/conf/default/forwarding
# IPv6 propagation via /etc/dhcp/dhclient-exit-hooks.d/prefix_delegation

The above already provided enough IPv6 connectivity to pass the IPv6 test on my desktop inside the LAN.

To make things more fun, I enabled DHCPv6 support for my LAN on the router's dnsmasq by adding the last 3 lines to the configuration:

dhcp-hostsfile=/etc/dnsmasq-ethersfile
bind-interfaces
interface=br0
except-interface=enp2s4
no-dhcp-interface=enp2s4
dhcp-range=tag:br0,10.10.10.0,static,infinite
dhcp-range=tag:br0,::1,constructor:br0,ra-names,ra-stateless,infinite
enable-ra
dhcp-option=option6:dns-server,[::],[2606:4700:4700::1111],[2001:4860:4860::8888]

The 5 first lines (included here for emphasis) are extremely important: they ensure that dnsmasq won't provide any IPv4 or IPv6 service to the outside interface (enp2s4) and that DHCP will only be provided for LAN hosts whose MAC address is known. Line 6 shows how dnsmasq's DHCP service syntax differs between IPv4 and IPv6. The rest of my configuration was omitted on purpose.

Enabling native IPv6 on my LAN has been an interesting experiment. I'm sure that someone could come up with even better ip6tables rules for the router or for my desktop hosts. Feel free to mention them in the blog's comment.

Migrating to Predictable Network Interface Names

A couple of years ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.

This also meant changing the settings on my router box for my new ISP.

I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.

Migrating to Predictable Network Interface Names

Ever since Linus decided to flip the network interface enumeration order in the Linux kernel, I had been relying on udev's persistent network interface rules to maintain some semblance of consistency in the NIC naming scheme of my hosts. It has never been a totally satisfactory method, since it required manually editing the file to list the MAC addresses of all Ethernet cards and WiFi dongles likely to appear on that host to consistently use an easy-to-remember name that I could adopt for ifupdown configuration files.

Enter predictable interface names. What started as a Linux kernel module project at Dell was eventually re-implemented in systemd. However, clear documentation on the naming scheme had been difficult to find and udev's persistent network interface rules gave me what I needed, so I postponed the transition for years. Relocating to a new flat and rethinking my home network to match gave me an opportunity to revisit the topic.

The naming scheme is surprisingly simple and logical, once proper explanations have been found. The short version:

  • Ethernet interfaces are called en i.e. Ether Net.
  • Wireless interfaces are called wl i.e. Wire Less. (yes, the official documentation call this Wireless Local but, in everyday usage, remembering Wire Less is simpler)

The rest of the name specifies on which PCI bus and which slot the interface is found. On my old Dell laptop, it looks like this:

  • enp9s0: Ethernet interface at PCI bus 9 slot 0.
  • wlp12s0: Wireless interface at PCI bus 12 slot 0.

An added bonus of the naming scheme is that it makes replacing hardware a breeze, since the naming scheme is bus and slot specific, not MAC address specific. No need to edit any configuration file. I saw this first-hand when I got around upgrading my router's network cards to Gigabit-capable ones to take advantage of my new home's broadband LAN. All it took was to power off the host, swap the Ethernet cards and power on the host. That's it. systemd took care of everything else.

Still, migrating looked like a daunting task. Debian's wiki page gave me some answers, but didn't provide a systematic approach. I came up with the following shell script:

#!/bin/sh
lspci | grep -i -e ethernet -e network
sudo dmesg | grep -i renamed
for n in $(ls -X /sys/class/net/ | grep -v lo);
do
  echo $n: && udevadm test-builtin net_id /sys/class/net/$n 2>/dev/null | grep NAME;
  sudo rgrep $n /etc
  sudo find /etc -name '*$n*'
done

This combined ideas found on the Debian wiki with a few of my own. Running the script before and after the migration ensured that I hadn't missed any configuration file. Once I was satisfied with that, I commented out the old udev persistent network interface rules, ran dpkg-reconfigure on all my Linux kernel images to purge the rules from the initrd images, and called it a day.

... well, not quite. It turns out that with bridge-utils, bridge_ports all no longer works. One must manually list all interfaces to be bridged. Debian bug report filed.

PS: Luca Capello pointed out that Debian 10/Buster's Release Notes include migration instructions.

GRUB fine-tuning

A couple of years ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.

This also meant changing the settings on my router box for my new ISP.

I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.

GRUB fine-tuning

One thing that had been annoying me ever since Debian migrated to systemd as /sbin/init is that boot message verbosity hasn't been the same. Previously, the cmdline option quiet merely suppressed the kernel's output to the bootscreen, but left the daemon startup messages alone. Not anymore. Nowadays, quiet produces a blank screen.

After some googling, I found the solution to that:

GRUB_CMDLINE_LINUX_DEFAULT="noquiet loglevel=5"

The former restores daemon startup messages, while the later makes the kernel output only significant notices or more serious messages. On most of my hosts, it mostly reports inconsistencies in the ACPI configuration of the BIOS.

Another setting I find useful is a reboot delay in case a kernel panic happens:

GRUB_CMDLINE_LINUX="panic=15"

This gives me enough time to snap a picture of the screen output to attach to the bug report that will follow.

2017-12-29

Jackpot

I have no idea whatsover of how I achieved this, but there you go. This citizen's legal draft is moving forward to the Finnish parliament.