Install Ubuntu from ISO on IPv6-only KVM Server in SolusIO

I recently obtained a KVM virtual server on SolusIO platform, and I want to install Ubuntu 20.04 Server from the official ISO image. This is not as easy as I hoped, but I figured it out.

Note: if you are in a hurry, skip the "Background" and start from "Part 1" section.

Background: SolusIO cannot Mount ISO Image

SolusIO is a virtual infrastructure management solution published by Plesk International Gmbh, the same company behind the popular SolusVM software. They describe SolusIO to be the successor of SolusVM, with more focus on the self-service approach for end users.

SolusIO inherits the same clean user interface from SolusVM, and is easy to use. However, as a power user, I notice several features are missing in SolusIO. One of these features is the ability to install the operating system from an ISO image.

Disabling VNC in Virtualizor ⇒ Lost Connectivity?

The KVM server hosting my website went offline last month. Thinking the server might have crashed, I went to Virtualizor, the VPS control panel, to reboot the VPS. It did not solve the problem, so I proceeded with my disaster recovery plan.

The hosting provider, Spartan Host, explained that it was a router bug. They fixed the router after 4 hours, but my server did not come online.

Symptom

To investigate what went wrong with my VPS, I came back to Virtualizor to enable VNC access. Having VNC access is like attaching a monitor and a keyboard to the server. It would allow me to see any error messages printed on the screen and login to check whether there are configuration errors.

I didn't see any error through VNC connection. Thinking it might be a routing problem, I logged in with username and password, and ran a traceroute. To my surprise, the traceroute was able to reach Internet destination. Moreover, I can SSH into this server again.

IPv6 Neighbor Discovery Responder for KVM VPS

I Want IPv6 for Docker

I'm playing with Docker these days, and I want IPv6 in my Docker containers. The best guide for enabling IPv6 in Docker is how to enable IPv6 for Docker containers on Ubuntu 18.04. The first method in that article assigns private IPv6 addresses to containers, and uses IPv6 NAT similar to how Docker handles IPv4 NAT. I quickly got it working, but I noticed an undesirable behavior: Network Address Translation (NAT) changes the source port number of outgoing UDP datagrams, even if there's a port forwarding rule for inbound traffic; consequently, a UDP flow with the same source and destination ports is being recognized as two separate flows.

$ docker exec nfd nfdc face show 262
    faceid=262
    remote=udp6://[2001:db8:f440:2:eb26:f0a9:4dc3:1]:6363
     local=udp6://[fd00:2001:db8:4d55:0:242:ac11:4]:6363
congestion={base-marking-interval=100ms default-threshold=65536B}
       mtu=1337
  counters={in={25i 4603d 2n 1179907B} out={11921i 14d 0n 1506905B}}
     flags={non-local permanent point-to-point congestion-marking}
$ docker exec nfd nfdc face show 270
    faceid=270
    remote=udp6://[2001:db8:f440:2:eb26:f0a9:4dc3:1]:1024
     local=udp6://[fd00:2001:db8:4d55:0:242:ac11:4]:6363
   expires=0s
congestion={base-marking-interval=100ms default-threshold=65536B}
       mtu=1337
  counters={in={11880i 0d 0n 1498032B} out={0i 4594d 0n 1175786B}}
     flags={non-local on-demand point-to-point congestion-marking}

The second method in that article allows every container to have a public IPv6 address. It avoids NAT and the problems that come with it, but requires the host to have a routed IPv6 subnet. However, routed IPv6 is hard to come by on KVM servers, because virtualization platform such as Virtualizor does not support routed IPv6 subnets, but can only provide on-link IPv6.

On-Link IPv6 vs Routed IPv6

yoursunny.com Disaster Recovery Plan: 104 Minutes Downtime, No Tears

The OVH fire taught us the importance of having a disaster recovery plan for your website and online services. In 2017, I rebuilt yoursunny.com and moved everything from configuration to content into git repositories. One of the reasons was that, the git repository could serve as a backup of the website, so that I can recover the site from a data loss.

uptime last 24 hours, vps4 server, yoursunny.com website

Today, I was forced to execute (part of) my disaster recovery plan. The result was: website is successfully recovered after 1 hour and 44 minutes of downtime.

🟥 Down

When I waked up this morning, there were several alert emails from UptimeRobot telling me that my website was down, up, down, and up again. At the same time, I also received alerts that the VPS hosting the website was not responding to ping. I ignored those alerts, thinking that they would resolve itself in a few minutes.

OVH Strasbourg: Halt and Catch Fire, Data Uploaded to the Cloud

2021-03-10, an OVH Cloud data center in Strasbourg, France caught on fire. Thousands of servers have been destroyed by fire. Thousands more are currently unavailable due to power cut, and will remain offline for several more days.

Data stored in those servers have been uploaded to the cloud via black smoke.

a building is burning with black smoke rising to the sky

According to Hacker News, a Dev mistakenly invoked the Halt and Catch Fire instruction on an Uninterrupted Power Supply unit, causing this incident.

Chairman of OVH cloud advised clients to activate their disaster recovery plans, such as restoring off-site backups to a new cloud server. Some clients are crying because they did not have backups, or they stored their backups on another machine in the same data center. Other clients experienced no downtime because they designed their systems for datacenter scale redundancy.

yoursunny.com is Served by Caddy

The last rebuild of yoursunny.com was in Spring 2017, when I moved the whole website into git repositories. It's been more than 3 years, and I think I should share an update on a few changes in the stack that serves this website.

History of HTTP Servers Behind yoursunny.com

Since 2011, my HTTP server of choice was lighttpd. Then, I have PHP running in FastCGI mode to serve the dynamic pages. It works, but I don't really like the lighttpd's script-like configuration structure. Moreover, there were suspected memory leaks in my setup, so that I had to use a cron job to restart the HTTP server weekly.

I keep hearing good words about nginx, as well as the benefits of running PHP in FPM mode. In 2013, I made the switch to nginx and PHP-FPM. The declarative configuration of nginx is easy to understand and makes sense to me.

HTTPS came to yoursunny.com at the end of 2015. Like many other websites, the TLS certificates were issued by Let's Encrypt, and requested through certbot command line client. One problem is that, my VPS at the time had only 64MB memory, and certbot would not work in such a small amount of memory. I had to request the TLS certificate on my laptop, and then upload it to the VPS.

How to Host a Website in Oracle Cloud Free Tier

Oracle Cloud is a cloud computing service offered by Oracle Corporation. Oracle Cloud has a generous free tier that offers two "always free" virtual machine (VM) instances with the following specification:

  • KVM virtualization
  • 1/8 CPU cores (AMD EPYC 7551)
  • 1GB memory
  • 45GB disk storage
  • 1 IPv4 address, no IPv6
  • 48Mbps Internet bandwidth

I signed up for Oracle Cloud, so that I can have some more free computing resources to play with. The sign-up procedure requires a credit card for identity confirmation purpose, but the credit card will not be charged. During sign-up, there's a choice of home region, which determines the location of VM instances; once selected, it cannot be changed in the future.

A common use case for a virtual machine is to host a website. Due to the firewalls, hosting a website on Oracle Cloud needs a few more steps. Here's exactly how to deploy a website in a Oracle Cloud Free Tier VM instance.

Create a VM Instance

Enable IPv4 Access in EUserv IPv6-only VS2-free

EUserv is a virtual private server (VPS) provider in Germany. Notably, they offer a container-based Linux server, VS2-free, free of charge. VS2-free comes with one 1GHz CPU core, 1GB memory, and 10GB storage. Although I already have more than enough servers to play with, who doesn't like some more computing resources for free?

There's one catch: the VS2-free is IPv6-only. It neither has a public IPv4 address, nor offers NAT-based IPv4 access. All you can have is a single /128 IPv6 address.

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
546: eth0@if547: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:77:4b:c0:eb:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 2001:db8:6:1::6dae/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::5ed4:d66f:bd01:6936/64 scope link
       valid_lft forever preferred_lft forever

If I attempt to access an IPv4-only destination, a "Network is unreachable" error appears:

$ host lgger.nexusbytes.com
lgger.nexusbytes.com has address 46.4.199.225
$ ping -n -c 4 lgger.nexusbytes.com
connect: Network is unreachable

How to Select Default IPv6 Source Address for Outbound Traffic in OpenVZ 7

I bought a few Virtual Private Servers (VPS) on Black Friday, and have been busy setting them up. Nowadays, most VPS comes with an IPv6 subnet that contains millions of possible addresses. Initially, only one IPv6 address is assigned to the server, but the user can assign additional addresses as desired. Given that I plan to run multiple services within a server, I added a few more IPv6 addresses so that each service can have a unique IPv6 address.

One of my servers is using OpenVZ 7 virtualization technology, in which I installed Debian 10 operating system. Commonly, OpenVZ 7 uses virtual network device (venet) that does not have a MAC address. venet devices are not fully IPv6 compliant, but still works if you statically assign IPv6 addresses. Moreover, every IP address used in a container must be configured from the host node, because venet would drop ip-packets from the container with a source address, and in the container with the destination address, which is not corresponding to an ip-address of the container. Therefore, I must use the VPS control panel, in this case SolusVM, to assign IPv6 addresses to my server:

IPv6 Subnet management in SolusVM

In the Add IP section, the IPv6 subnet prefix 2001:db8:f1c1:8454:0964: is already shown. Notice that I am putting a colon (:) in front of the suffix 1337, so that they concatenate to the full address 2001:db8:f1c1:8454:0964::1337. Forgetting this colon would cause "Invalid Entry" error.

After making this change in the SolusVM control panel, the /etc/network/interface file on my server is updated automatically:

How to Select Default IPv6 Source Address for Outbound Traffic with Netplan

I bought a few Virtual Private Servers (VPS) on Black Friday, and have been busy setting them up. Nowadays, most VPS comes with an IPv6 subnet that contains millions of possible addresses. Initially, only one IPv6 address is assigned to the server, but the user can assign additional addresses as desired. Given that I plan to run multiple services within a server, I added a few more IPv6 addresses so that each service can have a unique IPv6 address.

One of my servers is using KVM virtualization technology, in which I installed Ubuntu 20.04 operating system manually from an ISO image. Unlike a template-based installation, an ISO-installed Ubuntu 20.04 system manages its networks using Netplan, a backend-agnostic network configuration utility that generates network configuration from YAML files. Most VPS control panels, including SolusVM and Virtualizer, are unable to generate the YAML files needed by Netplan. IPv4 works out of box via DHCP, but IPv6 has to be configured manually. To assign two IPv6 addresses to my server, I need to write the following in /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
      addresses:
        - 2001:db8:30fa:5877::1/64
        - 2001:db8:30fa:5877::beef/64
      routes:
        - to: ::/0
          via: 2001:db8:30fa::1
          on-link: true
      nameservers:
        addresses:
        - 2001:4860:4860::8888
        - 2606:4700:4700::1111

I intend to host my secret beef recipes on its unique IPv6 address 2001:db8:30fa:5877::beef, and use the other address 2001:db8:30fa:5877::1 for outbound traffic such as pings and traceroutes. However, I noticed that the wrong address is being selected for outgoing packets:

$ ping 2001:db8:57eb:8479::2

$ sudo tcpdump -n icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:44:48.704099 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 1, length 64
00:44:48.704188 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 1, length 64
00:44:49.704011 IP6 2001:db8:30fa:5877::beef > 2001:db8:57eb:8479::2: ICMP6, echo request, seq 2, length 64
00:44:49.704099 IP6 2001:db8:57eb:8479::2 > 2001:db8:30fa:5877::beef: ICMP6, echo reply, seq 2, length 64