https://www.endpointdev.com/blog/tags/ipv6/2017-04-04T00:00:00+00:00End Point DevLinode IPv6 issues with NetworkManager on CentOS 7https://www.endpointdev.com/blog/2017/04/linode-ipv6-issues-with-networkmanager/2017-04-04T00:00:00+00:00Marco Matarazzo
<p>In End Point, we use different hosting providers based on the specific task needs. One provider we use extensively with good results is <a href="https://www.linode.com">Linode</a>.</p>
<p>During a routine CentOS 7 system update, we noticed a very strange behavior where our IPv6 assigned server address was wrong after restarting the server.</p>
<h3 id="ipv6-on-linode-and-slaac">IPv6 on Linode and SLAAC</h3>
<p>Linode is offering IPv6 on all their VPS, and IPv6 dynamic addresses are assigned to servers using <a href="https://en.wikipedia.org/wiki/IPv6#Stateless_address_autoconfiguration_.28SLAAC.29">SLAAC</a>.</p>
<p>In the provided CentOS 7 server image, this is managed by NetworkManager by default. After some troubleshooting, we noticed that during the update the NetworkManager package was upgraded from 1.0.6 to 1.4.0.</p>
<p>This was a major update, and it turned out that the problem was a change in the configuration defaults between the two version.</p>
<h3 id="privacy-stable-addressing">Privacy stable addressing</h3>
<p>Since 1.2, NetworkManager added the Stable Privacy Addressing feature. This allows for some form of tracking prevention, with the IPv6 address to be stable on a network but changing when entering another network, and still remain unique.</p>
<p>This new interesting feature has apparently become the default after the update, with the ipv6.addr-gen-mode property set to “stable-privacy”. Setting it to “eui64” maintains the old default behavior.</p>
<h3 id="privacy-extension">Privacy Extension</h3>
<p>Another feature apparently also caused some problems on our VPS: the Privacy Extension. This is a simple mechanism that somewhat randomizes the network hardware’s (MAC) address, to add another layer of privacy. Alas, this is used in address generation, and that randomization seemed to be part of the problem we were seeing.</p>
<p>This too has become the default, with the ipv6.ip6-privacy property set to 1. Setting it to 0 turns off the feature.</p>
<h3 id="to-sum-it-up">To sum it up</h3>
<p>In the end, after the update, we could restore the old behavior and resolve our issues by running, in a root shell:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">nmcli connection modify <span style="color:#d20;background-color:#fff0f0">"Wired connection 1"</span> ipv6.ip6-privacy <span style="color:#00d;font-weight:bold">0</span>
nmcli connection modify <span style="color:#d20;background-color:#fff0f0">"Wired connection 1"</span> ipv6.addr-gen-mode eui64
</code></pre></div><p>After a reboot, the IPv6 address finally matched the one actually assigned by Linode, and everything was working ok again.</p>
<p>If you want to know more on Privacy Extensions and Privacy Stable Addressing, <a href="https://blogs.gnome.org/lkundrak/2015/12/03/networkmanager-and-privacy-in-the-ipv6-internet/">this great blog post</a> by Lubomir Rintel helped us a lot understanding what was going on.</p>
Postfix IPv6 preferencehttps://www.endpointdev.com/blog/2014/08/postfix-ipv6-preference/2014-08-19T00:00:00+00:00Jon Jensen
<p>On a Debian GNU/Linux 7 (“wheezy”) system with both IPv6 and IPv4 networking setup, running Postfix 2.9.6 as an SMTP server, we ran into a mildly perplexing situation. The mail logs showed that outgoing mail to MX servers we know have IPv6 addresses, the IPv6 address was only being used occasionally, while the IPv4 address was being used often. We expected it to always use IPv6 unless there was some problem, and that’s been our experience on other mail servers.</p>
<p>At first we suspected some kind of flaky IPv6 setup on this host, but that turned out not to be the case. The MX servers themselves are fine using only IPv6. In the end, it turned out to be a Postfix configuration option called <a href="http://www.postfix.org/postconf.5.html#smtp_address_preference">smtp_address_preference</a>:</p>
<blockquote>
<p>smtp_address_preference (default: any)</p>
<p>The address type (“ipv6”, “ipv4” or “any”) that the Postfix SMTP client will try first, when a destination has IPv6 and IPv4 addresses with equal MX preference. This feature has no effect unless the inet_protocols setting enables both IPv4 and IPv6. With Postfix 2.8 the default is “ipv6”.</p>
<p>Notes for mail delivery between sites that have both IPv4 and IPv6 connectivity:</p>
<p>The setting “smtp_address_preference = ipv6” is unsafe. It can fail to deliver mail when there is an outage that affects IPv6, while the destination is still reachable over IPv4.</p>
<p>The setting “smtp_address_preference = any” is safe. With this, mail will eventually be delivered even if there is an outage that affects IPv6 or IPv4, as long as it does not affect both.</p>
<p>This feature is available in Postfix 2.8 and later.</p>
</blockquote>
<p>That documentation made it sound as if the default had changed to “ipv6” in Postfix 2.8, but at least on Debian 7 with Postfix 2.9, it was still defaulting to “any”, thus effectively randomly choosing between IPv4 and IPv6 on outbound SMTP connections where the MX record pointed to both.</p>
<p>Changing the option to “ipv6” made Postfix behave as expected.</p>
DAD Troublehttps://www.endpointdev.com/blog/2014/06/dad-trouble/2014-06-15T00:00:00+00:00Josh Williams
<p>I never thought I’d say it, but these days technology is simply moving too fast for DAD. It’s just the way it is. Of course it’s not DAD’s fault, it’s just the world doesn’t want to wait.</p>
<p>Before I get to that, I want to mention some trouble we’d recently started seeing with nginx failing to start on boot. It’s just been on our most recently obtained servers, both Debian-based (including Ubuntu) and RHEL-based installations. Some were Linode VM’s, others were bare metal hardware systems. After boot and once we got in to try and see what was happening, nginx would happily start manually. The only clue was one line that had been left in the error log:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">2014/06/14 23:33:20 [emerg] 2221#0: bind() to [2607:f0d0:2001:103::8]:80 failed (99: Cannot assign requested address)
</code></pre></div><p>And it wasn’t just nginx; Apache httpd in one instance gave us similar trouble:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address [2600:3c00::f03c:91ff:fe73:687f]:80
no listening sockets available, shutting down
</code></pre></div><p>As an interim fix, since at the moment these systems only had one IPv6 each, we told nginx or httpd to listen on all addresses. But not liking to leave a mystery unsolved, once we were able to schedule a long enough maintenance window on a system to reboot it a few times and see what’s going on, we found that the interface was in a “tentative” state for a short interval.</p>
<p>That was the clue we needed. For some reason, the boot process was allowed to continue before DAD (Duplicate Address Detection) has a chance to decide that if the interface is allowed to use the provided IPv6 address. It’s probably been doing this all along, but the servers that were affected just didn’t boot fast enough to try binding before the interface was ready. Now, things are faster, and service start-up was winning the race.</p>
<p>For us, the addresses are either static or autoconfigured, and we’re confident that a duplicate address situation won’t be a problem. So we turned off dad_transmits by setting this in sysctl.conf:</p>
<pre tabindex="0"><code>net.ipv6.conf.all.dad_transmits = 0
</code></pre><p>Success! No more bind problems on boot preventing a service from starting.</p>
<p>Incidentally, I believe this <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705996">has been solved in Debian</a> by making the interface wait until it’s out of the “tentative” state, but it doesn’t look like it’s been backported to current stable. It should be in Ubuntu as of the current LTS (14.04) however.</p>
<p>Happy Father’s Day!</p>
Proxmox and the fun maze of IPv6 connectivityhttps://www.endpointdev.com/blog/2013/07/proxmox-and-fun-maze-of-ipv6/2013-07-08T00:00:00+00:00Emanuele “Lele” Calò
<p>While working on the Proxmox machine setup and specifically on the IPv6 connectivity I found a problem where after a reboot I always kept getting the *** net.ipv6.conf.all.forwarding*** and all related variable set to ***0***, thus giving lots of IPv6 network connectivity issues on the guests.</p>
<p>While brainstorming with a colleague on this, we discovered in the boot logs these few messages which are quite indicative of something horrible happening at boot:</p>
<pre tabindex="0"><code># less /var/log/boot.0
[..]
Mon Jul 8 18:38:59 2013: Setting kernel variables ...sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/forwarding: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/autoconf: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_dad: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_defrtr: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_rtr_pref: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra_pinfo: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_source_route: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_redirects: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/autoconf: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_dad: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_defrtr: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_rtr_pref: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_ra_pinfo: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_source_route: No such file or directory
Mon Jul 8 18:38:59 2013: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/accept_redirects: No such file or directory
Mon Jul 8 18:38:59 2013: done.
[..]
</code></pre><p>The following steps would be to either crawl through the “inextricable” maze of the ProxMox (PVE) boot initrd image and probably came up with the solution or find a quick way to deal with this in a clean way without touching the boot process.</p>
<p>Since it was all due to <em><strong>sysctl</strong></em> being called too early in the boot process and then not finding proper IPv6 module already loaded calling it again <em>later</em> would suffice. So I simply added the following line to <em><strong>/etc/network/interfaces</strong></em></p>
<pre tabindex="0"><code>iface eth0 inet6 static
address YOUR:IPV6:IS:HERE
netmask 64
up ip -6 route add default via fe80::1 dev eth0
up sysctl -p # <------ ADDED THIS LINE TO FIX IPv6 CONNECTIVITY ISSUES
</code></pre><p>And there it goes. Reboot once again to verify and you should be all set.</p>
RHEL 6 glibc IPv6 DNS resolution bughttps://www.endpointdev.com/blog/2012/07/rhel-6-glibc-ipv6-dns-resolution-bug/2012-07-17T00:00:00+00:00Jon Jensen
<p>We ran into an unpleasant bug in a Red Hat Enterprise Linux 6 glibc update a couple of weeks ago. It since has made its way into CentOS 6 as well.</p>
<p>The problem manifested itself in our particular case with this error from the Postfix mailer:</p>
<pre tabindex="0"><code>Jun 29 01:55:23 sl37 kernel: smtp[7093]: segfault at 1 ip 00007ffc0e455596 sp 00007fff99948f60 error 6 in libresolv-2.12.so[7ffc0e449000+16000]
</code></pre><p>But it affects all DNS resolution on the host, not just for mail.</p>
<p>If you have any IPv6 resolvers at all listed in /etc/resolv.conf, all your DNS resolution is likely to be broken with this version of glibc:</p>
<pre tabindex="0"><code>glibc-2.12-1.80.el6.x86_64
</code></pre><p>To work around the problem, you can either:</p>
<ul>
<li>Use only IPv4 DNS resolvers (comment out the IPv6 resolvers for now)</li>
<li>or downgrade to the previous version of glibc using yum downgrade</li>
</ul>
<p>Red Hat is aware of the bug and you can track progress toward a resolution in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=835090">Bugzilla bug #835090</a>.</p>
<p>If you’re using IPv6, watch out for this! If not, you’re fine.</p>
IPv6 Basics by Josh Williamshttps://www.endpointdev.com/blog/2012/06/ipv6-basics-by-josh-williams/2012-06-15T00:00:00+00:00Brian Gadoury
<p>Josh Williams reviewed World IPv6 Launch that occurred on 2012-06-06. Google, Yahoo, Bing and Facebook are some of the larger companies that launched on that date. While not exactly on the same scale, <a href="/">endpoint.com</a> did a little over a year earlier.</p>
<img src="/blog/2012/06/ipv6-basics-by-josh-williams/image-0.jpeg"/>
<p>As many of you may already know, the pool of (32-bit) IPv4 IP addresses that we’re all familiar with, has officially run out. IPv6, defined in 1998, has 128-bit addresses that are constructed by 8 hexadecimal segments.</p>
<p>What are some of the advantages of having the huge pool of addresses? Well, there’s really no longer any need for NAT—everything becomes directly routable. IPv6 also gives us stateless auto-configuration that uses your MAC address to create your IPv6 address, so this means you’re that much more likely to be able to successfully get the same network address. IPsec is optionally built in to IPv6. (Optional because not everything truly needs IPsec.)</p>
<p>Not sure if your computer or mobile device is using IPv6 yet? Hit <a href="http://testipv6.com">http://testipv6.com</a> and it’ll tell you. Welcome to the future, finally.</p>
Stateful IPv6 tracking in RHEL 5: Failhttps://www.endpointdev.com/blog/2012/04/stateful-ipv6-tracking-in-rhel-5-fail/2012-04-05T00:00:00+00:00Josh Williams
<p>Are you a RHEL 5 user? Or CentOS or Scientific Linux, for that matter? Have you started deploying IPv6 on RHEL 5? If you’re using ip6tables as a firewall in this environment, you may want to double check its configuration.</p>
<p>The short version: The 2.6.18 kernel RHEL 5 ships doesn’t have a working conntrack module for IPv6. The conntrack module is what ip6tables uses for stateful packet tracking. You may already be familiar with it from the IPv4 version of iptables, looking something like this in your firewall config:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">-m state --state ESTABLISHED,RELATED -j ACCEPT
</code></pre></div><p>ip6tables will accept that as well, it just doesn’t do much. The rule is effectively skipped, and the processing eventually gets down to where ip6tables is set to drop or reject, unless it matches something else along the way. Thus it appears that outgoing connections are blocked for most servers, but maybe not everywhere, even if you don’t have any rules in your OUTPUT chain.</p>
<p>Incoming connections will of course work fine, as those don’t rely on the state match (at least initially) and instead match explicitly defined rules for public ports, specific source addresses, etc. RHEL 6 is also fine, as the more recent kernel has switched to something different to implement stateful tracking.</p>
<p>See this <a href="https://bugzilla.redhat.com/show_bug.cgi?id=232933">Red Hat Bugzilla bug</a> for a few details, but essentially this version of the kernel sets all packets to the INVALID state. The SYN packet goes out to the remote server, it just doesn’t match the return SYN/ACK reply. So instead of doing stateful tracking, we have to resort to stateless. We’ll lose that fancy connection tracking which can be useful for protocols like FTP that make multiple connections. Rather, in allowing in anything that’s not starting a new connection, this should do the job:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">-A INPUT -p tcp -m tcp ! --syn -j ACCEPT
</code></pre></div><p>We encountered this little kernel bug in the course of moving our old employee shell server to an existing server that already had IPv6 enabled, only to find out going connections mysteriously not working. It took a little digging, but I’m sure we won’t be the last to encounter it, so hopefully this will help someone else out.</p>
World IPv6 Launch: 6 June 2012https://www.endpointdev.com/blog/2012/02/world-ipv6-launch-6-june-2012/2012-02-17T00:00:00+00:00Jon Jensen
<p>For any of our readers who don’t know: The classic Internet Protocol (IPv4) supported around 4 billion IP addresses, but it recently ran out of free addresses. That the addresses would eventually all be used was no surprise. For more than a decade, a replacement called IPv6 has been under development and allows practically unlimited addresses.</p>
<p>Last year there was a one-day trial run called World IPv6 Day. Our own Josh Williams <a href="/blog/2011/06/june-8-2011-world-ipv6-day/">wrote about it here</a>. It was the first major attempt for mainstream websites to enable dual-stack IPv4 and IPv6 networking, so that both the “old” and “new” Internet could access the same site. It was intended to bring to the surface any problems, and it went very well – most people never knew it was happening, which was the goal.</p>
<p><a href="http://www.worldipv6launch.org"><img align="right" alt="World IPv6 Launch is 6 June 2012 – The Future is Forever" height="128" src="/blog/2012/02/world-ipv6-launch-6-june-2012/image-0.png" title="World IPv6 Launch is 6 June 2012 – The Future is Forever" width="128"/></a></p>
<p>This year there’s a much bigger event planned: <a href="http://www.worldipv6launch.org/">World IPv6 Launch</a>, and this time IPv6 is meant to stay on. Google, Facebook, Yahoo!, Bing, and many other major sites are participating. A big advance over last year’s event is that many ISPs and vendors of home networking gear are also participating. That means it won’t just be a test that classic IPv4 still works for people alongside IPv6, but that for some customers, native IPv6 starts working end to end.</p>
<p>Last year, Josh mused that it would have been most appropriate for the day to be June 6, 6-6, for IPv6 and sixes everywhere. I suspect that wasn’t done because it fell on a Monday in 2011, and there are enough new support complaints backlogged from the weekend without adding IPv6 to the mix! But this year it is on June 6.</p>
<p>We got our own <a href="/">www.endpoint.com</a> website running on IPv6 in time for last year’s World IPv6 Day. A few months ago, we switched on IPv6 for <a href="http://bucardo.org/">bucardo.org</a> and a few new customer sites as well. For this year’s event we plan to prepare most of our remaining internal infrastructure to be dual-stack and to be ready to enable IPv6 for any customer sites upon request.</p>
<p>What can you do to join in and help free the Internet of its address shortage? Try visiting <a href="http://test-ipv6.com/">test-ipv6.com</a> to see how compatible your own current Internet access is. If your ISP doesn’t yet offer IPv6, ask them when they will. Demand drives support. And visit <a href="http://www.tunnelbroker.net/">Hurricane Electric’s IPv6 Tunnel Broker</a> to set up a free tunnel from your location to the IPv6 Internet. It’s a very nice service.</p>
<p>Have fun, and if all goes well, the beginning of widespread adoption of IPv6 starts on June 6!</p>
June 8, 2011: World IPv6 Dayhttps://www.endpointdev.com/blog/2011/06/june-8-2011-world-ipv6-day/2011-06-07T00:00:00+00:00Josh Williams
<img alt="This post has 6 a lot" border="0" src="http://joshwilliams.name/ipv6/hev6certshirt.jpg"/>
<p>I’m a little surprised they didn’t do it today. 06-06, what better day for IPv6? Oh well, at least Hurricane Electric was awesome enough to send a Sage certification shirt just in time!</p>
<p>June 8th, 2011 is the day! In just a couple days time <a href="https://web.archive.org/web/20110606225039/http://www.worldipv6day.org/">World IPv6 Day</a> begins. Several of the largest and most popular sites on the Internet, <a href="https://web.archive.org/web/20110613121940/http://www.worldipv6day.org/participants/">and many others</a>, turn on IPv6 addresses for a 24-hour interval. Many of them already have it, but you have to seek it out on a separate address. Odds are if you’re seeking that out specifically you’re configured well enough to not have any problems. But with IPv6 configured on the primary addresses of some of the largest Internet sites, people that don’t specifically know they’re testing something become part of the test. That’s important to track down exactly what composes those <a href="https://www.facebook.com/notes/facebook-engineering/world-ipv6-day-solving-the-ip-address-chicken-and-egg-challenge/484445583919">1-problem-in-2,000 configurations</a>, and assess if that’s even an accurate number these days.</p>
<p>Not sure about your own connection? <a href="https://test-ipv6.com/">https://test-ipv6.com/</a> is an excellent location to run a number of tests and see how v6-enabled you are. Odds are you’ll end up at one end of the spectrum or the other. But if there’s a configuration glitch that could help you track it down.</p>
<p>At End Point we decided to get a bit of a head start. For the last 24 hours or so <a href="http://www.endpoint.com">www.endpoint.com</a> has been running with an IPv6 AAAA record. And it was pleasantly surprising, the first IPv6 hits started showing up nearly instantaneously! Our visitors are in general likely to be more on the technical side of the scale, but so far the results have been promising. By all accounts everything works as it should. Soon, we’ll likely begin offering to enable it for some of our customer sites.</p>
<p>A part of me wants to express some disappointment that an event like this is even necessary. My favorite database project got IPv6 support way back in the <a href="https://www.postgresql.org/docs/9.0/static/release-7-4.html">PostgreSQL 7.4 release</a>, now long under EOL. But at the same time I know what a huge undertaking the IPv6 migration is on a number of levels. Encouragement by providers and end user awareness go a long way to help things along. So I applaud those sites taking part in World IPv6 Day, helping pave the way for the next generation protocol.</p>
<p>I kinda feel for the tunnel providers, now that I think about it. I’ve had a <a href="https://tunnelbroker.net/">Hurricane Electric</a> tunnel that’s been carrying my home traffic for quite a while now. So far, that’s primarily been things like IRC over to Freenode, ssh traffic to the servers that have IPv6 addresses, a few random hits to sites that have it, etc. But with high bandwidth sites like YouTube and frequently hit CDN providers on board for World IPv6 Day I’d bet that those tunnels will see traffic spike dramatically.</p>
<p>Granted a number of the tunnel providers are running portions of the Internet backbone anyway, but there’s only a handful of tunnel endpoints for that traffic to go through. It’s also forcing it to travel over sections of the provider’s network before it can find its way out to a peer. Both of these are pretty substantial barriers to route optimization, at least compared to native traffic. Don’t get me wrong, even as it is the performance of the tunnel has been great. I just hope providing an arguably necessary (and free!) service isn’t too painful to these providers while end-user deployments are still occurring.</p>