2014 in IPv4 addresses: closing the books on IPv4

▼ Ten years ago, I published my first "IPv4 address use report" over 2005. After that, I did 2006, 2007, 2008, 2009, and 2010. Today, I'm going back to the well one last time and provide an overview of what happened with the IPv4 addresses the past decade, which will close the book on the IPv4 address space as far as I'm concerned.

At the end of 2005, 2056.30 million of the 3706.65 usable IPv4 addresses had been given out: 55%. Today, it's 3592.99 million, which is 97%. Can you imagine driving a car with a tank filled to 3%? Or work on a computer with a drive that's 97% full?

All the numbers I present here are based on those published on the FTP servers of the five Regional Internet Registries that give out IP addresses. So an address is "in use" if an RIR has given it to an ISP or directly to an end-user organization. Whether that ISP or that organization actually uses the address is something we can't tell from these numbers.

Let me first explain the 3706.65 million number. IP addresses are 32 bits long, which means there's 2³² = 4,294,967,296 of them: 0.0.0.0 to 255.255.255.255, or 256 blocks of 16,777,216 addresses that start with the same number. We call one of those a "/8" in the business. 0/8 (or 0.0.0.0/8, which is 0.0.0.0 - 0.255.255.255) and 127/8 can't be used because they each contain one special address. 224/4 (everything starting with 224 to 239) is used for multicast and 224/4 (240 - 255) is "reserved for future use". This prompted some programmers who are a bit too fond of input validation to reject those addresses. As a result, those addresses can't be used on many systems, some of which are no longer updated, but still used. So 268 million perfectly good IPv4 addresses will probably remain unused forever. Last but not least, there's the 17,891,328 addresses set aside for private use: 10/8,172.16/12 and 192.168/16. So the total usable IPv4 address space is 222 × 16,777,216 - 17,891,328 = 3,706,650,624.

By downloading the old address delegation records from the RIR FTP servers it's possible to see how many addresses were given out at the end of each year. (Looking at just the latest files gives different results because addresses that have been returned in the meantime aren't listed in those.) This is the number of IPv4 addresses given out as of the end of the last ten years:

This is the net difference between each year and the previous year:

In this graph, the year 2014 looks strange, as two old class A blocks (16.78 million addresses each) that are used in Europe now fall under the RIPE NCC, while they were previously listed in the ARIN statistics. If we exclude all the "legacy" /8 blocks, the trend makes more sense:

(However, now there's a spike in 2012 for ARIN as a few of these blocks changed their status.)

APNIC and the RIPE NCC have been running on fumes for the past several years: ISPs in their region can only get one last block of 1024 addresses after those two dipped below the 16.78 million address mark (a /8 block). This happened in 2011 for APNIC and in 2012 for the RIPE NCC. This year, ARIN and LACNIC also reached a similar status, although ARIN doesn't use the "one last block" policy.

As a result, organizations that need IPv4 addresses now have to buy them from others who can spare some. IPv4 address transfers were especially big in the RIPE region in 2014. In the ARIN region, the number of transfers was limited at 180 in five years, but the size of the blocks transferred tended to be large; apparently here transfers were mainly used to get larger amounts of address space than what could (then still) be obtained directly from ARIN. Inter-RIR transfers are now also possible, and happen exclusively from the ARIN region to the APNIC region. See the RIPE and ARIN transfer statistics on their websites and APNIC's on their FTP server.

This is how three kinds of numbers compare: the addresses given out with a datestamp of 2014, the net difference between the total number of IPv4 addresses given out, and the number of addresses transferred in 2014:

                   AfriNIC APNIC   ARIN    LACNIC  RIPE NCC
 
Given out 2014:    12.45    3.77   26.06   19.11    2.61
 
Intra-RIR
transfers 2014:             3.85    2.98            9.61
   
Inter-RIR
transfers 2014:             1.10   -1.10
 
Net difference
2014-01-01 -       12.72    4.62   18.82   18.74    0.27
2015-01-01:    
   

When I compiled my first statistics over 2005, this was the list of the 15 biggest IPv4 address users:

Country   Addresses
 
US      1324.93 M        United States
JP       143.00 M        Japan
EU       113.87 M        Multi-country in Europe
CN        74.39 M        China
CA        67.43 M        Canada
DE        51.13 M        Germany
FR        45.16 M        France
KR        41.91 M        Korea
UK        40.18 M        United Kingdom
GB        33.63 M        Great Britain
AU        26.87 M        Australia
IT        18.39 M        Italy
BR        17.17 M        Brazil
NL        16.40 M        Netherlands
ES        16.29 M        Spain

Nine years later, the list looks like this:

Country   Addresses
 
US       1596.02 M        United States
CN        331.99 M        China
JP        202.61 M        Japan
GB        123.75 M        United Kingdom
DE        118.91 M        Germany
KR        112.32 M        South Korea
FR         84.05 M        France
BR         81.02 M        Brazil
CA         80.41 M        Canada
IT         53.44 M        Italy
AU         48.48 M        Australia   
NL         45.94 M        Netherlands
RU         45.55 M        Russian Federation
IN         36.36 M        India
TW         35.47 M        Taiwan

Surprisingly little has changed, although the BRIC countries have claimed a bigger piece of the pie. I don't know why India has barely a tenth of the number of IPv4 addresses that China has.

So long IPv4, it's been good knowing you.

Permalink - posted 2015-01-08

iPhone 6 network speed

▼ It was only a few months ago that I first got 4G/LTE working on my iPhone 5 using a Vodafone NL prepaid SIM, and the results weren't very spectacular: at home, I got 12 Mbps down, which is actually slightly slower than the 3G I get from KPN. Up speed was respectable at 5 or 6 Mbps. In the city center I got slightly better speeds at 20/20 Mbps.

So I was very curious what the iPhone 6, which supports many more LTE frequency bands, would do on the Vodafone LTE network. So I popped in the SIM... and nothing happened. Despite my best efforts and many reboots, I was stuck on 3G for almost a day. Or even GPRS on the 2G network:

Not sure if that did the trick, but I finally got 4G after resetting my network settings. When I first popped in the SIM, I got a text message with a link to a provisioning profile, but that seems to give me the same settings that also appear automatically. However, when I later put my KPN SIM back in the phone, I couldn't get any data service until I removed the Vodafone provisioning profile.

I did some testing as I was traveling home by train, and I got various speeds, starting at about 20/20 Mbps down and up and going as high as 68/36 Mbps:

It's too bad that data plans are so limited and/or expensive, as you can burn though a gigabyte of data in two minutes at these speeds. Vodafone NL apparently doesn't support Voice over LTE yet, as I dropped down to 3G as soon as I made a voice call.

Wi-Fi has also been improved—but only on the 5 GHz band. On the 2.4 GHz band, you still get a maximum of 72 Mbps 802.11n:

The iPhone 5 and 5S did 150 Mbps 802.11n on the 5 GHz band; the iPhone 6 adds 433 Mbps 802.11ac:

These speeds are a good deal lower than what you get on a computer because computers usually have three antennas, which means they can be three times as fast if three data streams can reach the antennas independently. With only one Wi-Fi antenna, the iPhone is limited to 433 Mbps, but it'll reach that maximum more often.

Transferring files between my computer and the iPhone using iTunes, I got about 7 megabytes per second on the 2.4 GHz band, which completely saturates the channel. I got 15 to 22 Mbps on 5 GHz, which doesn't come close to saturating the channel. So if you're still using a 2.4 GHz only Wi-Fi base station or home router, you may want to consider upgrading it to something that supports at least 802.11n—preferably 802.11ac—on the 5 GHz band and you'll get better speed more consistently.

I would have loved to test Bluetooth 4.0 performance, but the iPhone doesn't support file transfers over Bluetooth. Annoying. Nokia could do this more than ten years ago. Apparently they want us to use their proprietary AirDrop system instead. I've seen this work very quickly between two Macs recently, but between my Mac and my iPhone, it takes forever for the two to "see" each other and the transfers then happen at about 5 megabytes per second.

Bottom line: the iPhone 6's networking can be very fast, but it all depends on the SIMs, Wi-Fi base stations, frequency bands and transfer protocols. So I'm glad it still comes with a USB cable, even if it's only USB 2.

Permalink - posted 2014-12-28

Internet packet sizes part 2: IPv4 path MTU discovery is dead

▼ Please read yesterday's post Maximum packet sizes on the internet first. There, I looked at the maximum supported packet sizes that are included in the TCP MSS option in HTTP requests to my server. Today I'll look at the values in ICMP(v6) "too big" messages.

(If you don't need the history lesson, skip ahead to "IPv4 measurements".)

The computer making that HTTP request doesn't necessarily know about the MTU sizes supported along the entire path that the data flows over. If the hops in between support larger packets, there is no issue, although it would be more efficient if we could make use of that ability.

The problems start if there's one or more hops along the way that can't support the packet size that the two computers that are communicating want to use. These days, pretty much everything is Ethernet, so a 1500-byte MTU would be expected. (Ironically, most of the technologies that are now replaced by Ethernet, such as IP over ATM and IP over FDDI, support packet sizes much larger than 1500 bytes.)

However, various types of tunnels (including VPNs) as well as PPP over Ethernet may reduce the the MTU for a given hop to less than 1500. The original way that IP(v4) handled this was through fragmentation. So if a host (computer) sends a 1500-byte IPv4 packet, but the next hop is a PPPoE link that can only handle 1492 bytes, the router will break the original packet into two fragments. Almost always the first packet will be a 1492-byte one, and the second packet holds the remaining 8 bytes It has its own copy of the IP header, so its total size is 28 bytes. (It would be much better to make two 760-byte packets...)

Fragmentation creates a whole bunch of problems. First of all, it takes a lot of extra time and processing for the router to fragment packets. Then the receiving host has to spend extra CPU cycles to reassemble the packet. If the communication rate is high enough, a lost fragment can easily lead to the situation where fragments of two different packets are reassembled. Usually, the TCP/UDP checksum will catch this, but that checksum isn't very strong, so one in every 65000 or so instances, it won't catch the problem and the communication will be corrupted. Last but not least, because the TCP/UDP port numbers are only present in the first fragment, NATs and firewalls have a hard time dealing with fragments.

The solution is path MTU discovery (PMTUD), which was specified in 1990. In order to avoid fragmentation in routers, hosts try to figure out the largest packets they can send to another host without fragmentation. They do this by setting the DF (don't fragment) bit in the IPv4 header. With that bit set, routers aren't allowed to fragment the packet. Instead, they send back a "packet too big but fragmentation needed" message. The PMTUD specification updated this message's format to include the supported packet size. This makes PMTUD very easy: simply send packets as large as you like with the DF bit set, and if you get "too big" messages back, lower your packet size to the value in the too big message.

However, this introduced a new problem: if the too big message doesn't make it back, either because the router didn't generate it, it got lost along the way or because it was filtered by a firewall, the host keeps resending large packets that never make it to their destination. To add insult to injury, the initial TCP handshake uses small packets, so that part works, but then the packets that actually contain data never make it to the other side. This is a PMTUD black hole.

PMTUD is optional with IPv4, although it's universally used, even by people who filter the ICMP "too big" messages. With IPv6, if you want to send packets larger than 1280 bytes, PMTUD is mandatory, as routers aren't allowed to fragment IPv6 packets. And unlike with IPv4, PMTUD also works for non-TCP protocols (such as UDP) with IPv6, as the source host's networking stack will fragment non-TCP packets before transmission. Enough background, on to the...

IPv4 measurements

Over the course of more than five days, my server received 52758 IPv4 ICMP messages. Those were:

• 52047 echo request (someone pinging me)
• 1 echo reply (me pinging someone)
• 645 unreachable
• 61 time exceeded in transit (traceroute)
• 3 unknown

In almost a week, I received zero IPv4 "too big" messages.

So it seems in the IPv4 world, path MTU discovery is dead. Turns out that so many people filter ICMP messages, that if you rely on PMTUD for IPv4, there's just too much breakage. So what (home) routers that sit in front of a reduced-MTU link do is "MSS clamping". They rewrite the value in the TCP MSS option to what's supported on the interface they're about to transmit the packet over.

(Please don't read this as "it's ok to filter ICMP "too big" messages. It could easily be that some users still depend on these.)

IPv6 measurements

Over the same five days, my server received 57244 ICMPv6 messages:

• 41471 neighbor solicitations (similar to an IPv4 ARP request)
• 15288 neighbor advertisements (similar to an IPv4 ARP reply)
• 17 echo request (ping)
• 17 echo reply (ping)
• 84 too big
• 366 unknown

The 84 ICMPv6 too big messages came from 9 unique sources, although one of those is a tunnel gateway that returned two different sizes for (presumably) two different tunnels. So that's 10 values, with the following distribution:

• 1280: 3 (30%)
• 1428: 1 (10%)
• 1472: 1 (10%)
• 1480: 5 (50%)

One of those 1480 results is my own connection at home, which uses an IPv6-in-IPv4 tunnel terminated on my home router. So my computers at home don't know the path MTU is 1480 and depend on PMTUD, which seems to work without obvious problems. Maybe two or three times a week I encounter a page that won't load, which may or may not be an IPv6-related issue, which in turn may or may not be a PTMUD issue.

Hopefully, IPv6 won't lose PMTUD to ICMP filtering like IPv4 did. MSS clamping is effective for TCP, but it doesn't work for non-TCP protocols or IPsec-protected communication. It's also a burden on routers.

Stéphane Bortzmeyer replied to yesterday's post with a link to this 20-year-old (to the day!) message, which has results for very similar measurements. The results are different in interesting ways, with the real stunner being that in 1994, 94% of all systems could handle 1500 bytes, but in 2014, this is down to 65%.

Permalink - posted 2014-12-18

Maximum packet sizes on the internet

▼ After some heated discussions about packet sizes on the mailinglist of the IETF v6ops working group, I decided to do some measurements to find out what maximum packet sizes are supported on today's internet. I did this by capturing two types of packets: the ICMP "too big" messages that routers send to tell a computer to send smaller packets, and the first packet of a TCP session, which contains the MSS option. The maximum segment size (MSS) option is used in TCP sessions to tell the other side what the maximum packet size is that we can receive. This depends on the maximum transfer unit (MTU) of the hardware, which may be further reduced by system administrators.

The Ethernet standard uses an MTU of 1500 bytes, although a lot of Ethernet hardware can support more, such as 9000-byte "jumboframes". Wi-Fi also uses 1500 bytes to be compatible with Ethernet. However, sometimes one protocol needs to be tunneled over another protocol, such as IPv6 over IPv4 (over Ethernet) or PPP over Ethernet, which reduces the supported packet size to 1480 or 1492, respectively. The IPv6 specifications require that a minimum MTU of 1280 bytes is supported. IPv4 has no minimum MTU. Note that all of this is about the maximum packet size, it is of course perfectly fine to send smaller packets.

TCP (and UDP) use segments which are put inside IP packets that are then transmitted inside Ethernet frames. A 1500-byte IPv4 packet supports 1460-byte TCP frames (1500 bytes minus the 20-byte IPv4 header and the 20-byte TCP header). This 1500-byte IP packet is transmitted as a 1518-byte Ethernet frame, although some people only count 14 bytes for the Ethernet header, ignoring the 4-byte checksum that's at the end of the Ethernet frame. Because the IPv6 header is 40 bytes, a 1500-byte IPv6 packet can hold a 1440-byte TCP segment, while a 1500-byte IPv4 packet can hold a 1460-byte TCP segment. I'll be talking about IP MTU sizes rather than segment/MSS sizes to make it easier to compare IPv4 and IPv6 results.

Over the better part of a week, my server received 41753 incoming TCP SYN packets with an MSS option on port 80. Another 140 packets didn't have the MSS option, and looked like they were mostly TCP-based traceroute packets. 24246 packets were IPv4 packets, coming from 4164 unique IP addresses. 17507 were IPv6 packets, which I reduced down to 227 unique IP addresses. Turns out, most of the IPv6 traffic on my server is from bots that check if I've added any new content to the site. Some of them use the same address each time, but others keep using different addresses, but, strangely, the same source port number (12000 - 12006). I removed these addresses to keep them from drowning out the real data.

The data showed no fewer than 72 different MTU sizes for IPv4, ranging from 576 to 9198 bytes. However, both of these extremes only showed up once, and other values below 1280 and above 1500 are also quite rare:

• 576: 1 (0.02%)
• 1200: 2 (0.05%)
• 1240: 4 (0.10%)
• 1242: 1 (0.02%)
• 1252: 4 (0.10%)
• 1272: 1 (0.02%)
• [...]
• 1504: 4 (0.10%)
• 3000: 1 (0.02%)
• 9000: 5 (0.12%)
• 9001: 47 (1.13%)
• 9198: 1 (0.02%)

I found the 9001 value quite curious; computers really like to work on nice round multiples of 2, 4 or 8 bytes. 9001, on the other hand, is a prime number. Turns out that 9001 bytes is used Amazon's datacenters, where some of the bots that index my website reside. These are the more common MTU sizes advertised in the TCP MSS option:

• 1300: 46 (1.10%)
• 1400: 78 (1.87%)
• 1420: 152 (3.65%)
• 1440: 99 (2.38%)
• 1460: 96 (2.31%)
• 1470: 233 (5.60%)
• 1492: 370 (8.89%)
• 1500: 2672 (64.17%)

1300 and 1400 look like someone set them manually; 1300 is also a common VPN MTU. 1440 bytes seems to be hardcoded in some home routers. 1460 could indicate IPv4-over-IPv6 tunneling. 1470 seems to be used by a number of broadband ISPs and 1492 results from PPP over Ethernet (PPPoE). Last but not least, just under two thirds of IPv4 visitors support the Ethernet MTU of 1500.

These are the results for IPv6 with the < 1% values removed (there were no values below 1280 and above 1500):

• 1280: 13 (5.73%)
• 1426: 3 (1.32%)
• 1428: 9 (3.96%)
• 1446: 14 (6.17%)
• 1472: 19 (8.37%)
• 1480: 5 (2.20%)
• 1492: 14 (6.17%)
• 1500: 142 (62.56%)

1280 and 1480 are probably IPv6-in-IPv4 tunnels and 1428 AYIYA tunnels. 1472 could be IPv6-in-UDP-in-IPv4 tunnels or IPv6-in-IPv4-over-PPPoE. The image below shows the cumulative frequency of MTU sizes for both IPv4 (red) and IPv6 (blue), where the line shows how many systems support a given MTU value, starting at 99.98% for 1200 and ending at 65.56% for 1500 (for IPv4).

The 90th percentile MTU size is 1428 for IPv6 and 1440 for IPv4. Obviously 100% of IPv6 systems support 1280, but 99.7% of IPv4 systems also support this size.

The MSS reflects the maximum size that the systems at both ends of a connection think they can use. However, there may be a bottleneck somewhere along the path. In that case, routers send back an ICMP Packet Too Big message. Tomorrow, I'll look at those.

Permalink - posted 2014-12-17

What makes for an undeployable protocol?

▼ I'm in Honolulu for the IETF meeting this week. As always, on sunday morning before the meeting proper starts, there's the IEPG, where there's always interesting stuff being presented, usually from the operational side of networking.

Today, there were talks about IPv6 packets with extension headers being dropped, routing table and packet size issues by Geoff Huston, and a discussion on Shim6 and Multipath TCP (MPTCP) failure recovery by Brian Carpenter. All good stuff. However, at the end of Brian's presentation, Lorenzo Colitti thanked Brian for the interesting presentation about the performance of undeployable protocol A vs undeployable protocol B.

I kind of get why Shim6 and MPTCP are considered undeployable, because you need to have addresses from two different ISPs, and you need to make sure that packets with addresses from ISP 1 go to ISP 1 and those with addresses from ISP 2 to ISP 2. If not, BCP 38 ingress filtering will block the packets. The trouble is that the RIRs started giving out provider independent IPv6 addresses shortly before Shim6 was finished so larger networks simply use those, Shim6 never got any traction and so if you want to use it now you'll find that nobody else uses it, and you need it at both ends for it to work. It's still somewhat early days for MPTCP, but it doesn't seem to be setting the world on fire, either.

But Lorenzo was talking about the fact that MPTCP uses TCP options that are filtered out by firewalls. Brian already mentioned that the Shim6 extension header is also often filtered, and suggested that probe packets should look like normal data packets.

However, when both of these were designed, those issues were considered. Obviously it would have been great if we could have implemented these two protocols without the need for additional options or headers, but I don't see how that would have been possible. So the next best thing was to make sure that if the options, or the packets containing options, are filtered, communication still works without the benefits of Shim6 or MPTCP. This means the protocols were never undeployable: you can turn them on by default without any issues. If the headers/options are filtered, you simply don't get any benefit, but everything still works. For paths where the options/headers are left alone, Shim6 and MPTCP get to do their thing and you benefit from being able to use additional paths. Over time, hopefully firewall operators realize these protocols don't cause any harm and stop filtering them.

Unfortunately, there are protocols that really do turn out to be undeployable, because firewalls or bad implementations break any communication that uses those protocols.

Permalink - posted 2014-11-09