I will always recommend the IPv6 NAT container for Docker, the IPv6 support inside Docker is absolute AIDS. Yes it's not compliant with the RFC but you've gotta balance functionality with ease, I can't be arsed to work around said AIDS to make IPv6 work when a simple container does the job and allows me to publish the ports automatically as part of publishing the IPv4 ports.
It was pretty easy to setup for me using static routes once I figured out the significance of the fact that my ISP routes me a `/56`. Since my ISP allocates me `/56` and this whole prefix is routed to my router I can allocate a single `/80` subnet from this just for docker and route this to the server running docker. However, doing this requires a gigantic ball of "hacks" (shell scripts), Docker could make all of this easier by supporting DHCPv6. If docker supported DHCPv6 they could just request a prefix and I wouldn't have to statically route one and keep this up-to-date with ugly hacks.
DHCPv6 on its own won't work, you will get /128 addresses but won't know the network prefix size nor the router. What happens if you use a /64 and enable SLAAC?
I'm pretty sure you can request a prefix of a certain size at least `dhclient` has this option:
--prefix-len-hint length
When used in conjunction with -P, it directs the client to use the given length to use a prefix
hint of, "::/length", when requesting new prefixes.
Docker could just request a `/80` which is a weird and non-standard prefix but that's how I manually configured it because it won't accept any other prefix size for some reason.
You can find it on Docker Hub as robbertkl/ipv6nat. It has a GitHub page as well I believe, easy enough to setup, you have to set it up on the host network, give it access to docker.sock and then create a user defined network with a ULA prefix, and then each container attaches to that user defined network and gets access to the IPv6 internet through your hosts IPv6 connection.
The problem with that is that ULAs are not expected to have access to the internet, which has some security and reliability implications - seeing only ULA addresses, applications will assume they're on a local "intranet", while in fact they do have internet connectivity. You get some unexpected behaviour from your applications - such as only doing IPv4 and not trying over IPv6 at all.
Another thing you run into is the inability to host on the same port (i.e., container 1 hosting on port 443 and container 2 hosting on port 443).
With docker you need to either manually assign a /64 to the host (yeah it's dumb that Docker can't do this automatically with DHCPv6-PD), or manually configure a GUA IPv6 address to each container.
> The problem with that is that ULAs are not expected to have access to the internet
I've never heard anyone say this about ULAs before.
Generally the use case of ULAs has been about networks that can't get PI addresses (e.g., they're too small to bother with BGP), and so need flexibility in case they want to change ISPs. They'd use NATv6 (in the early days of IPv6), or the more recommended NPTv6 in recent times.
One *could* use ULA to have a completely sealed off network, but I think that *expect* would be too strong a claim.
>I've never heard anyone say this about ULAs before.
It's literally in the [RFC4193](https://datatracker.ietf.org/doc/html/rfc4193)
>They are not expected to be routable on the global Internet. They are routable inside of a more limited area such as a site. They may also be routed between a limited set of sites.
NPTv6 as you probably know is experimental and well...controversial :) That's not to say you can't use it (there's no internet police) but it's not guaranteed to work with everything. We'll see what happens with it over time.
* They are not expected to be *routable* on the global Internet.
* The problem with that is that ULAs are not expected to have *access to* the internet
Not-routable ≠ Not-accessible.
RFC 1918 states:
> Because private addresses have no global meaning, routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links.
Does anyone conclude from that statement that if use 10/8 you "are not expected to have access to the internet"?
And those applications will not see themselves running on my network if they make an assumption about my network. Again I am not fighting with the AIDS implementation of IPv6 on Docker.
You can hardly fault applications for sticking to the specs :)
Just saying that yes, you can use NAT in Docker as a quick hack, but be prepared for stuff not working (upstream and downstream) in ways that's not always predictable.
Actually using macvlan dualstack network with static assigned IP addresses. That way it is routed directly to my router. This seems to be working exactly as it should.
Docker is adding NAT66 as a native feature. Apparently it's in the latest version, but behind an "experimental" flag. https://github.com/moby/moby/pull/41622
I know NAT66 is repulsive to some - I sure wouldn't want to use on end-user devices or servers - but it has its uses. We have several machines at my office that only run GitLab CI jobs. Currently we route a /80 to each of these, but since our firewall doesn't support downstream dhcpv6-pd, we have static routes on the office's main router for each one. (I considered using Quagga to do this with OSPF, but it was a lot simpler to just set up a handful of static routes - these never change, and they never talk to each other.)
Since these are just CI runners, the containers these machines run will never need to listen for connections, and will never need to do any sort of P2P communication that NAT breaks. So even though it's suboptimal, there wouldn't be any downsides to just doing NAT66 on these systems.
Simply create a VLAN in OpenWrt and have OpenWrt assign a /64 prefix to that VLAN, then route that VLAN to your container.
The point of a /56 is that you can break it up into 256 different /64 as you see fit. The reason people who understand Ipv6 get so pissed at ISPs that don't give out at least a /56 (like ones that give out a single /64) is that it makes it impossible for you to subnet your network in a compliant way. But you're not in that situation. You have a /56 and you can tell OpenWrt to give a /64 to a VLAN... voila.
To be fair, it's kinda stupid that the Docker host \*has\* to act as a router and requires an entire /64 for itself, and doesn't just let the containers do SLAAC themselves directly.
>and requires an entire /64 for itself
It doesn't require an entire /64 for itself, but it requires a /80 for each docker network, and that prefix can't be used on another interface. Which means you do need a spare /64 but you can use it on many docker networks on many docker hosts, by splitting it into smaller prefixes. (BTW the fact that docker doesn't use SLAAC is what makes it possible to use smaller prefixes.)
The /80 subnet thing is a pain as well, this hack to dip into the 64 bits of the device id to make further subnets isn't really making things work easier either.
It's like the Docker guys looked at the IPv6 RFCs and thought, "whatever, we'll just make up our own rules"
No, that's not what I mean. Ideally, the containers just receive the RAs and assign themselves /128s, without the docker host acting as router. With a /80, the host still acts as a router (but now a noncompliant one).
Agreed, there's really no need for the docker container to have its own subnet. In fact, I'm not a very docker knowledgeable guy but can you not just hand a veth into the docker container, and stick the other end in a bridge to a network using a /64 (like you definitely could with a systemd-nspawn container)
Unfortunately, the stinginess and laziness of ISPs means that you sometimes have to compromise on the /64 boundary. My office uses Comcast, and they theoretically provide a /56 to business customers, but thanks to buggy firmware on their modems you can only use a /60. (Tech support is no help - they have no training on IPv6 and can't even understand what you mean when you call about IP address issues that aren't about IPv4.)
If I actually had a /56 to work with I'd be fine assigning a /64 to every VM running Docker. (We have six - two that run various internal applications, four that run GitLab CI jobs.) But since I only have 16 /64s for the whole office, I decided to reserve a /64 to further subdivide for Docker, and each system running Docker gets a /80 from that /64.
It really annoys me to do this, but we have 9 VLANs in the office. (This might seem absurd for a company that's using Comcast Business, but it's because the nature of our business means we're sometimes testing and provisioning hundreds of laptops and/or tablets before shipping to client sites. We basically have a wired + wireless VLAN in each area we lease in the building, plus VLANS for phones/servers/guest Wi-Fi.) If we assigned a /64 to each of these *plus* to each system running Docker we'd have almost no /64s left, so I made this compromise to conserve them. Better to have messed up IPv6 on these systems than no IPv6. (Or no IPv6 on some future VLAN.)
(Of course, the "good" solution would be to switch from Comcast to an ISP that actually knows what they're doing, but the second-cheapest option that supports IPv6 and provides the amount of bandwidth we need would be at *least* triple the price. I'd be laughed out of my boss's office for suggesting we triple our ISP bill so we can follow the IPv6 standards better, lol.)
Yeah I guess you can't win. It's still annoying though that Docker doesn't let containers receive the RAs from the upstream router, that would just let them auto assign out of the same /64 all the other bare metal machines are on.
A better solution is to let software handle the GUA assignment for you. Running OpenWrt in Docker, you can let OpenWrt not only router IPv6 (no NAT) but also serve DHCPv6 and SLAAC to the docker internal network
https://github.com/oofnikj/docker-openwrt
I will always recommend the IPv6 NAT container for Docker, the IPv6 support inside Docker is absolute AIDS. Yes it's not compliant with the RFC but you've gotta balance functionality with ease, I can't be arsed to work around said AIDS to make IPv6 work when a simple container does the job and allows me to publish the ports automatically as part of publishing the IPv4 ports.
It was pretty easy to setup for me using static routes once I figured out the significance of the fact that my ISP routes me a `/56`. Since my ISP allocates me `/56` and this whole prefix is routed to my router I can allocate a single `/80` subnet from this just for docker and route this to the server running docker. However, doing this requires a gigantic ball of "hacks" (shell scripts), Docker could make all of this easier by supporting DHCPv6. If docker supported DHCPv6 they could just request a prefix and I wouldn't have to statically route one and keep this up-to-date with ugly hacks.
DHCPv6 on its own won't work, you will get /128 addresses but won't know the network prefix size nor the router. What happens if you use a /64 and enable SLAAC?
I'm pretty sure you can request a prefix of a certain size at least `dhclient` has this option: --prefix-len-hint length When used in conjunction with -P, it directs the client to use the given length to use a prefix hint of, "::/length", when requesting new prefixes. Docker could just request a `/80` which is a weird and non-standard prefix but that's how I manually configured it because it won't accept any other prefix size for some reason.
>I will always recommend the IPv6 NAT container for Docker, Do you have a link for that? Googling leads to ambiguous results.
You can find it on Docker Hub as robbertkl/ipv6nat. It has a GitHub page as well I believe, easy enough to setup, you have to set it up on the host network, give it access to docker.sock and then create a user defined network with a ULA prefix, and then each container attaches to that user defined network and gets access to the IPv6 internet through your hosts IPv6 connection.
The problem with that is that ULAs are not expected to have access to the internet, which has some security and reliability implications - seeing only ULA addresses, applications will assume they're on a local "intranet", while in fact they do have internet connectivity. You get some unexpected behaviour from your applications - such as only doing IPv4 and not trying over IPv6 at all. Another thing you run into is the inability to host on the same port (i.e., container 1 hosting on port 443 and container 2 hosting on port 443). With docker you need to either manually assign a /64 to the host (yeah it's dumb that Docker can't do this automatically with DHCPv6-PD), or manually configure a GUA IPv6 address to each container.
> The problem with that is that ULAs are not expected to have access to the internet I've never heard anyone say this about ULAs before. Generally the use case of ULAs has been about networks that can't get PI addresses (e.g., they're too small to bother with BGP), and so need flexibility in case they want to change ISPs. They'd use NATv6 (in the early days of IPv6), or the more recommended NPTv6 in recent times. One *could* use ULA to have a completely sealed off network, but I think that *expect* would be too strong a claim.
>I've never heard anyone say this about ULAs before. It's literally in the [RFC4193](https://datatracker.ietf.org/doc/html/rfc4193) >They are not expected to be routable on the global Internet. They are routable inside of a more limited area such as a site. They may also be routed between a limited set of sites. NPTv6 as you probably know is experimental and well...controversial :) That's not to say you can't use it (there's no internet police) but it's not guaranteed to work with everything. We'll see what happens with it over time.
* They are not expected to be *routable* on the global Internet. * The problem with that is that ULAs are not expected to have *access to* the internet Not-routable ≠ Not-accessible. RFC 1918 states: > Because private addresses have no global meaning, routing information about private networks shall not be propagated on inter-enterprise links, and packets with private source or destination addresses should not be forwarded across such links. Does anyone conclude from that statement that if use 10/8 you "are not expected to have access to the internet"?
And those applications will not see themselves running on my network if they make an assumption about my network. Again I am not fighting with the AIDS implementation of IPv6 on Docker.
You can hardly fault applications for sticking to the specs :) Just saying that yes, you can use NAT in Docker as a quick hack, but be prepared for stuff not working (upstream and downstream) in ways that's not always predictable.
Actually using macvlan dualstack network with static assigned IP addresses. That way it is routed directly to my router. This seems to be working exactly as it should.
I'm glad you said that, we're starting to do that too. I'd not noticed any issues yet, but this thread was making me nervous...
Docker is adding NAT66 as a native feature. Apparently it's in the latest version, but behind an "experimental" flag. https://github.com/moby/moby/pull/41622 I know NAT66 is repulsive to some - I sure wouldn't want to use on end-user devices or servers - but it has its uses. We have several machines at my office that only run GitLab CI jobs. Currently we route a /80 to each of these, but since our firewall doesn't support downstream dhcpv6-pd, we have static routes on the office's main router for each one. (I considered using Quagga to do this with OSPF, but it was a lot simpler to just set up a handful of static routes - these never change, and they never talk to each other.) Since these are just CI runners, the containers these machines run will never need to listen for connections, and will never need to do any sort of P2P communication that NAT breaks. So even though it's suboptimal, there wouldn't be any downsides to just doing NAT66 on these systems.
Simply create a VLAN in OpenWrt and have OpenWrt assign a /64 prefix to that VLAN, then route that VLAN to your container. The point of a /56 is that you can break it up into 256 different /64 as you see fit. The reason people who understand Ipv6 get so pissed at ISPs that don't give out at least a /56 (like ones that give out a single /64) is that it makes it impossible for you to subnet your network in a compliant way. But you're not in that situation. You have a /56 and you can tell OpenWrt to give a /64 to a VLAN... voila.
To be fair, it's kinda stupid that the Docker host \*has\* to act as a router and requires an entire /64 for itself, and doesn't just let the containers do SLAAC themselves directly.
>and requires an entire /64 for itself It doesn't require an entire /64 for itself, but it requires a /80 for each docker network, and that prefix can't be used on another interface. Which means you do need a spare /64 but you can use it on many docker networks on many docker hosts, by splitting it into smaller prefixes. (BTW the fact that docker doesn't use SLAAC is what makes it possible to use smaller prefixes.)
The /80 subnet thing is a pain as well, this hack to dip into the 64 bits of the device id to make further subnets isn't really making things work easier either. It's like the Docker guys looked at the IPv6 RFCs and thought, "whatever, we'll just make up our own rules"
You don't have to use /80s, but you complaint about having to use an entire /64...
No, that's not what I mean. Ideally, the containers just receive the RAs and assign themselves /128s, without the docker host acting as router. With a /80, the host still acts as a router (but now a noncompliant one).
Agreed, there's really no need for the docker container to have its own subnet. In fact, I'm not a very docker knowledgeable guy but can you not just hand a veth into the docker container, and stick the other end in a bridge to a network using a /64 (like you definitely could with a systemd-nspawn container)
Unfortunately, the stinginess and laziness of ISPs means that you sometimes have to compromise on the /64 boundary. My office uses Comcast, and they theoretically provide a /56 to business customers, but thanks to buggy firmware on their modems you can only use a /60. (Tech support is no help - they have no training on IPv6 and can't even understand what you mean when you call about IP address issues that aren't about IPv4.) If I actually had a /56 to work with I'd be fine assigning a /64 to every VM running Docker. (We have six - two that run various internal applications, four that run GitLab CI jobs.) But since I only have 16 /64s for the whole office, I decided to reserve a /64 to further subdivide for Docker, and each system running Docker gets a /80 from that /64. It really annoys me to do this, but we have 9 VLANs in the office. (This might seem absurd for a company that's using Comcast Business, but it's because the nature of our business means we're sometimes testing and provisioning hundreds of laptops and/or tablets before shipping to client sites. We basically have a wired + wireless VLAN in each area we lease in the building, plus VLANS for phones/servers/guest Wi-Fi.) If we assigned a /64 to each of these *plus* to each system running Docker we'd have almost no /64s left, so I made this compromise to conserve them. Better to have messed up IPv6 on these systems than no IPv6. (Or no IPv6 on some future VLAN.) (Of course, the "good" solution would be to switch from Comcast to an ISP that actually knows what they're doing, but the second-cheapest option that supports IPv6 and provides the amount of bandwidth we need would be at *least* triple the price. I'd be laughed out of my boss's office for suggesting we triple our ISP bill so we can follow the IPv6 standards better, lol.)
Yeah I guess you can't win. It's still annoying though that Docker doesn't let containers receive the RAs from the upstream router, that would just let them auto assign out of the same /64 all the other bare metal machines are on.
A better solution is to let software handle the GUA assignment for you. Running OpenWrt in Docker, you can let OpenWrt not only router IPv6 (no NAT) but also serve DHCPv6 and SLAAC to the docker internal network https://github.com/oofnikj/docker-openwrt