Proxmox on Debian at Hetzner with multiple IP addresses

I’ve spent most of my day today trying a plethora of different ways on how to configure Proxmox on a Hetzner server with multiple IP addresses. Most of the tutorials I found online gave a good deal of information but where lacking in one or two crucial details. Shortly before I was ready to throw my computer out of the window, I had success and managed to get everything set up the way I wanted it.

This aims to be the definitive guide on how to accomplish the aforementioned task. When ready the setup includes the following features:

  • Host bound to main IPv4 address, that comes with the server (and one of the 18,446,744,073,709,551,616 included IPv6 addresses)
  • Every IPv4 address of a separately delegated subnet usable for virtual machines
  • Internal private network for inter-virtual machine communication and non publicly accessible VMs

Disclaimer

I am describing here what WORKED FOR ME. I don’t know if each of the steps is according to best practice or even most optimal in terms of security and/or performance. If you have any further information, or can spot a mistake I made, I urge you to reach out in the comment section or via Twitter and I’ll be happy to amend this article for the betterment of all of human knowledge.

Installing Proxmox

Installing Proxmox isn’t very hard. Just make sure to make a clean Debian install on a reasonably beefy machine matching your Debian version to the latest recommended one by Proxmox. After initial setup is done and your base system is running create the file ‘/etc/apt/sources.list.d/proxmox.list’ and paste in the following line to update your apt sources:

deb http://download.proxmox.com/debian jessie pve-no-subscription

Then add the repository key to your system with the following command:

wget -O- "http://download.proxmox.com/debian/key.asc" | apt-key add -

Next update and upgrade your system:

apt update && apt upgrade

Now it’s time to install the Proxmox kernel and headers. Don’t worry if the ones here are outdated when you read this, those packages will be automatically upgraded during the whole installation process.

apt install pve-firmware pve-kernel-4.4.8-1-pve pve-headers-4.4.8-1-pve

Now reboot your system to load the new kernel.

After the system is up and running again you can install Proxmox VE with the following command:

apt-get install proxmox-ve

After this step is done, reboot again.

Great now you have Proxmox running and could in theory start creating VMs and containers like crazy. But the part that took me the longest to figure out is till to come. I wish I had this blog post a few hours ago.

Configuring the network

To use all available IP addresses of your delegated IPv4 subnet on a Hetzner dedicated server you need to set up a bridge on your host computer.

Edit the file ‘/etc/network/interfaces’ and fill in the data you received from Hetzner. Ignore the first two definitions for loopback and IPv6 loopback and match your ‘eth0’ configuration to the following:

auto eth0
    iface eth0 inet static
    address <YOUR MAIN IP>
    netmask 255.255.255.224
    gateway <YOUR GATEWAY>
    up route add -net <YOUR NET> netmask 255.255.255.224 gw <YOUR GATEWAY> eth0

Now these settings should all be pretty much already be in there from Hetzner’s automatic installimage configuration during the initial operating system installation. Next is the host’s IPv6 configuration:

iface eth0 inet6 static
    address <ONE OF YOUR IPv6 ADDRESSES>
    netmask 128
    gateway fe80::1

Again these lines should already be existent in your network configuration courtesy of some nice Hetzner engineer that pre-seeds all standard installs with automatic IP configurations. Only you should lower the netmask, because this interface now only participates in the subnet and doesn’t swallow up all of the IPv6 addresses. The whole subnet will be assigned to the bridge below.

Now it’s time to create our first bridge that will connect our host to any virtual machines running on it and them to the outside world.

auto vmbr0
iface vmbr0 inet static
    address <YOUR MAIN IP>
    netmask 255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0
    pre-up brctl addbr vmbr0
    up ip route add <FIRST IP FROM YOUR SUBNET>/32 dev vmbr0
    up ip route add <SECOND IP FROM YOUR SUBNET>/32 dev vmbr0
    ...

Add one ‘up ip route add …’ line per IPv4 address out of your delegated subnet. That way all of them will be available on the bridge and everyone can talk to everyone.

Also add the following to be able to route IPv6 addresses to and from your virtual machines:

iface vmbr0 inet6 static
    address <YOUR MAIN IPv6 ADDRESS>
    netmask 64

This takes care of the all publuc IPv4 and IPv6 addresses at your disposal on the host level. Later we’ll examine how to properly configure your guests to match these settings. But first we will create another bridge on the host to…

Set up internal networking

It’s quite nice to outfit each virtual machine with two virtual network interfaces and have the second one connected to a private network that’s only reachable from your host and all virtual machines. That way you can, for instance, create database, caching or worker machines that need no public IP address and will never be seen on the open internet.

So still in ‘/etc/network/interfaces’ add the following at the end:

auto vmbr1
iface vmbr1 inet static
    address 10.20.30.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up iptables -t nat -A POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE

This configures the internal communication. The last two lines take care of NAT (Network Address Translation), so your “private” VMs can be connect to the internet, for instance to install software via apt and download updates. This way virtual machines can connect to the outside from within, but can’t be directly reached from the internet, exactly like your home computer behind a router.

Since our host acts as a router we have to make sure it’s kernel has all IP packet forwarding features activated. Take a look at ‘/etc/sysctl.conf’ and make sure that the following two lines aren’t commented out:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Lastly make sure your host won’t send ICPM “redirect” messages to guests, telling them to find the gateway by themselves. This won’t work with our particular network setup. Add the following to ‘/etc/sysctl.conf’:

net.ipv4.conf.all.send_redirects=0

This concludes the host configuration. You can now reboot the server one last time, or activate the new settings by writing the sysctl settings directly and restarting the network stack. Now on to the last step.

Guest settings

Now this is different depending on the method you choose to create a VM. Proxmox offers LXC containers and fully virtualised machines. Depending on your needs both have their advantages and drawbacks, but discussing them is far outside the scope of these lines.

Setting up a container

First we’ll inspect creating a container from within Proxmox’s webinterface. Point your browser to ‘https://<YOUR MAIN IP>:8006’ (take care to actually type the https part, or your browser won’t know how to connect) and log in with your Linux user credentials. You need to download a container template (like a system image) initially before you can create your first container.

  • Expand the “Datacenter” node in the left sidebar menu
  • Click on the “local” storage group
  • Find the button named “Templates” in the content area and click it
  • Choose your preferred template, “debian-8.0-standard” for instance.
  • Click the “Download” button

Now that your template is available let’s move on to actually spinning up the container.

  • Click on “Create CT” in the upper right corner menu
  • Choose a good hostname
  • Set up a root password
  • On the “Template” tab choose your just downloaded template
  • On “Root Disk” choose an appropriate disk size
  • Under “CPU” select the desired number of CPUs
  • Assign enough “Memory”

The next Tab is the most interesting one: “Network”

  • Make sure you’re configuring ‘eth0’
  • Leave the MAC address field empty to get a new randomly created from Proxmox
  • Choose bridge “vmbr0”
  • Leave “VLAN Tag” and “Rate limit” alone for now
  • Choose “static” IPv4
  • Insert one of your subnet’s IPv4 addresses into “IPv4/CDR” and add the suffix “/32” behind it (e.g. 192.0.2.2/32)
  • Type in your hosts main IP address into the “Gateway (IPv4)” filed. The one we set up waaaay back in our eth0 configuration in ‘/etc/network/interfaces’ on the host
  • Choose “static” IPv6
  • Insert one of your 18,446,744,073,709,551,616 IPv6 addresses, if you can find one you haven’t used yet
  • As “Gateway (IPv6)” insert the IPv6 address you assigned to eth0 and vmbr0 on the host.

The next two Tab “DNS” and Confirm don’t have any interesting settings and you can leave them pretty much alone. Now you have a container that’s ready to run and can be accessed directly via public IP over the internet.

Now if you want and need private communication, just add a second network interface through the web GUI to your container (click on your container on the left menu and choose network -> add) and give it an IP address in the configured subnet, 10.20.30.2 for example. For the Gateway type in your host’s bridge private IP, 10.20.30.1 in our example. Lastly bind that interface to the private bridge ‘vmbr1’.

Setting up a virtual machine

This is a bit more involved and I’ll describe the way I did it. As mentioned before, there are probably better ways, so please don’t hold back if you have anything to add.

Since there are no downloadable templates for virtual machines, you have to reach out to your preferred operating system’s install media and acquire it as an ISO disk image file. I choose Debian 8 minimal, which is downloadable via https://www.debian.org/CD/netinst

  • Again select the “local” storage node in the left hand menu
  • Click on “Upload” in the main content area
  • Upload your ISO image
  • Click on “Create VM” in the upper right corner
  • Start with choosing a witty hostname
  • On “OS” choose “Linux 4.X/3.X/2.6 Kernel” if you’re following along with a Debian guest system
  • On “CD/DVD” select “Use CD/DVD disc image file (iso)” and choose your just uploaded ISO image
  • “Hard Disk”, “CPU” and “Memory” should be pretty self explanatory. Choose whatever you deem necessary for your VM
  • On the network tab make sure to choose “Bridged mode” and select “vmbr0”
  • Under “Model” select “VirtIO” if your guest OS supports it
  • Confirm your settings and click “Finish”

Now comes the tricky part. Start you newly created virtual machine and select “Console” in the upper menu. This will start a virtual console (Java browser plugin unfortunately required) which let’s you start the installation. Now Debians minimal install is pretty easy to follow, the only thing is: we can’t configure the network during installation because our gateway will be outside of the configured subnet and the installer doesn’t provide for this setup. I found a very old discussion on Debian’s bugtracking about that topic, but the conclusion was pretty much not to alter the installer because people that need this, will find other ways.

So finish the installation without any network access and reboot the virtual machine. After logging in via the virtual console again, edit ‘/etc/network/interfaces’ again – this time on the guest system and fill it with the following values:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address <ONE OF YOUR SUBNET'S IPs>
    netmask 255.255.255.255
    dns-nameservers 213.133.100.100 213.133.98.98 213.133.99.99
    post-up ip route add <YOUR MAIN IP> dev eth0
    post-up ip route add default via <YOUR MAIN IP> dev eth0
    pre-down ip route del default via <YOUR MAIN IP> dev eth0
    pre-down ip route del <YOUR MAIN IP> dev eth0

iface eth0 inet6 static
    address <ONE OF YOUR IPv6 ADDRESSES>
    netmask 64
    gateway <YOUR MAIN IPv6 ADDRESS>

auto eth1
iface eth1 inet static
    address 10.20.30.3
    netmask 255.255.255.0
    gateway 10.20.30.1

After you reboot your VM you should now be able to reach it from the internet as well as be able to communicate with your host and other VMs and containers via your private network. Now if you want a private network only VM or container, just remove eth0 from the web GUI or delete it from ‘/etc/network/interfaces’ and you should be left with a fully NATed machine.

Conclusio

This article has become much longer than I anticipated but that holds true for the whole setup of Proxmox with public IPs that I went through today as well, so I guess it’s only fitting.

If you followed this description you should now have a fully IPv4 and IPv6 capable VM cluster that’s neatly routed via your host and out to the internet in case of the public bridge or segregated via NAT in case of the private bridge. I guess this setup is flexible enough to accommodate a broad range of virtual computing needs. Anything more involved would probably herald a dedicated routing VM using Vayatta/vOS, pfSense or something similar. I decided that this would be overkill for my requirements and I’d rather spend the time figuring out how to properly use Linux’ internals to set up everything.

Please feel free to chime in on how you solved those problems or if you spotted anything in my words that could be improved, optimised or amended.

34 comments

  1. you rock!
    thank you very very very much for this article! i spent (literally) all day yesterday trying every possible mixture of configurations to get the secondary ip’s at hetzner to work.

    thank you so much for taking your time to write this up!!!!!!!!!!!!!!!

  2. very helpful post.. thank you very much,
    but… I’m not understood the definition in the virbr bridge… Why U’re defining the same subnet ‘post-up iptables -t nat -A POSTROUTING -s ‘10.20.30.0/24′ -o eth0 -j MASQUERADE’ twice???

    1. Thanks for pointing that out. It should of course be the matching post-down line the second time:
      post-down iptables -t nat -D POSTROUTING -s '10.20.30.0/24' -o eth0 -j MASQUERADE

      I’ve updated the post.

          1. Florian… You rule!!!!!
            Your config works like a charm in both internal and external networks (VM) and you saved me a lot of time trying to find the best config for my needs.
            BRAVO 🙂

  3. Hi there.
    I´m getting crazy trying to connect my windows virtual machines inside a hetzner host. Do you have any idea on how to do this? i´ve tried every setting possible in hyper v virtual switch but no luck at all. I don´t know the cause. I´m not an expert but i would be extremely grateful if you give me a hint about this.

    Thanks a lot in advance

    1. Sorry, I don’t have any experience in running Windows VMs. I’m pretty much a *nix-only guy. But if I’d attempt to do something like that I’d probably assign one VM with pfSense on it and connect all other VMs through this one machine to the outside world. pfSense has a pretty nice webinterface, so it should be relatively easy to run if you have basic networking knowledge.

    2. If using Single IP’s you need to add a MAC address generated from robot for each IP you use.

      If you have a Subnet you need to create a DHCP server up too.

      1. Sorted it simply by add ing the ip to :

        up ip route add /32 dev vmbr0 then in Proxmox adding the IP with <xxx.xxx.xxx.xxx/32> and my Main IP as Gateway 🙂

        Thank you very much for the Tutorial. Helped me a lot.

        1. Florian do you have any experience with EXSI 6 ?

          I am testing both Proxmox and ESXi and trying to get this setup to work with ESXi 6 if I can. Wonder if you could help ?

          Thanks.

          1. I do have some experience with ESXi but very limited production-wise at the moment. You’d better seek out their support forums or mailinglist if available for specific questions.

  4. Got it working fine thanks, could not use Proxmox as 3 of my additional IP’s were in a different subnet and got too confusing to try get all them working. Main IP and 1 additional were in 176...* and other 3 were in 136...*

    With ESXi it was just a matter of putting the Mac Addresses in the Network Card config.

  5. where should i add this:

    Now it’s time to create our first bridge that will connect our host to any virtual machines running on it and them to the outside world.

    auto vmbr0
    iface vmbr0 inet static
    address
    netmask 255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0
    pre-up brctl addbr vmbr0
    up ip route add /32 dev vmbr0
    up ip route add /32 dev vmbr0

  6. Hi,

    thank you for this.

    I am a bit confused as you write in eth0 AND vmbr0 “” and you use on the vmbr0 /32?

    So two interfaces have the same IP, in Hetzner’s case public ip?

    And what do you mean by “/32 ” – you mean the private addresses or do you have subnet from Hetzner (publicly routed)?

    1. I guess there could be a better way, but having the same public IP address for eth0 and vmbr0 works for me.

      And yes, this article assumes you have a public IPv4 subnet, so each virtual machine can get one of those IPs.

    1. As it says in the sentences just before the code sample: fill in the data you received from Hetzner. Or if you deploy this somewhere else, fill in the subnet you delegated to this host that you want to use for your VMs.

  7. Hey Florian,

    This is a very basic question but I couldn’t find any info online. When you install Hetzner debian image upon which you install proxmox later, how do you partition your disks? Is there a recommended layout for proxmox? Should you use LVM underneath?

    Thanks,
    Antonio

    1. It really depends on what exactly you want to do with the server. In general going with LVM and a separate boot partition on today’s fast disks seems reasonable most of the time.

  8. First of all thank you so much for this guide!
    I was really stuck with my network configuration but you saved my day!

    However when I apt update it shows an ipv6 address behind the repo and remains at 0% until it’s done (but extremely slow). Do you know why this happens?
    I’m currently experiencing the same thing on another machine trying to wget the latest plex server & again ipv6. Now it’s done and shows me some timeouts however then it changed to ipv4 and it worked. Is there any way to force ipv4? I’m kind of a noob so hope you can help me.. also I can’t add the eth1 to a ubuntu VM (had to flush and change the name to ens18 to get eth0 working)

    Thanks in Advance!

    1. You’re welcome, I’m alwaysy happy if my site is useful to people!
      You can append -o Acquire::ForceIPv4=true to your apt calls if you want to restrict them to IPv4 only. But if I was you, I’d dive deeper into why IPv6 connections time out. Maybe there’s no IPv6 forwarding on the host?
      Regarding network device names: As far as I know, Ubuntu changed the naming scheme when switching to udev/systemd.

  9. Hello and thank you for this guide. I used the official Hetzner guide to setup our Proxmox server a while ago, and added a /29 subnet to it. It all works great.
    However, we ran out of IP’s in the /29 and wanted to purchase an additional /29.

    I then set it up in the same way as the first /29, just using a different bridge, vmbr3, so now I have vmbr0, which uses the main IP to activate the routed network, vmbr1 which uses the main IP of the first /29, to be used as gateway in the VMs, vmbr2 which has 192.168.0.1 for our NAT network and vmbr3 which has the main IP of the second /29.

    The initial setup still works great, and from the HV I can use the main IP for vmbr3, but when setting up a new VM using this new bridge, it doesn’t route properly.
    Any ideas?

    Thank you very much.

  10. Hi, i am trying to set this up with the pre installed hetzner proxmox image, version 5… and i cannot get the second ip to work.

    I want to use pfsense instead of a debian host… there you cannot add a route outside your subnet.. not sure what is going on besides that

  11. Hi Florian,
    thank you for that detailed post. It was helpful for a Ganeti installation at Hetzner.

    One issue with your setup:
    I’ve got two Hetzner servers for HA. If I hardcode the hosts main IP in the guest routing, I cannot switch over to the secondary server without changing the routing on the guest.

    Any idea?

    1. I’m not sure if this is even possible with Hetzner servers. I believe you’d need some kind of link aggregation on the switch or maybe use a proper routing protocol. Or am I thinking to complicated? Anyone else have an idea?

  12. Great setup! Works great.
    Some tiny typos:
    Mid section: “gateway by themselves. This won’t work with our particular network setup. Add the following to ‘/etc/sysctl.con’: ”
    This should probably be ‘/etc/sysctl.conf’ (note the f)
    Same for conclusio -> conclusion?

    My question: did you try the 2015 guide in their wiki here: https://wiki.hetzner.de/index.php/Proxmox_VE/en
    Did that work for you as well?
    Difference is you do not use pointopoint and add to your bridge:
    bridge_maxwait 0
    pre-up brctl addbr vmbr0

    Any specific reason?

    1. Hi, thanks for your message. I’m glad these steps work great for you.
      I’ll correct the missing “f”, the second one though, “conclusio” – is on purpose: https://en.wiktionary.org/wiki/conclusio

      I don’t remember if I also used that particular guide and settings. If I did, I couldn’t get it to work for me. If I didn’t then that could be an option to try and maybe improve the setup. If you ever do, please report back and tell us if there are any improvements when doing it this way. I am always happy if I can optimize.

Leave a Reply

Your email address will not be published. Required fields are marked *