With North America out of IPv4, do we trade, translate or transition to IPv6?

▼ It's official: for all intents and purposes, North America is out of IPv4 addresses. Two of the four other regions have been in that situation for some time, and Latin America will be joining North America, Europe and Asia in about a month. That leaves Africa, which still has almost 50 million IPv4 addresses, which are used up at a rate of 5 - 10 million a year.

Does this mean that if you're running a business that requires a regular infusion of new IP addresses, it's time to relocate to Africa?

Not quite. The Internet Engineering Task Force saw all of this coming twenty years ago—which they should, having hobbled TCP/IP with a meager 32-bit address space for a maximum of about 4 billion IPv4 addresses. But they fixed that with IPv6, which increases the address length to no less than 128 bits for a near-infinite number of addresses.

Unfortunately, a system that only speaks the new IPv6 can't directly talk to a system that only speaks the current IPv4. So giving people just IPv6 means they can't talk to the 97% of internet users or the 90% of top 500 web sites that haven't adopted the new protocol yet.

But don't start looking for apartments in Nairobi or Casablanca just yet: it's also possible to trade IPv4 addresses. The big question is: to what degree will that solve the lack of available IPv4 addresses? We know that Microsoft and Amazon have bought large numbers of IPv4 addresses, and there have also been larger numbers of small exchanges. The going rate seems to be around $10 an address, which is expected to go up to about $30.

Microsoft and Amazon run cloud services where large numbers of virtual machines need IP addresses, which they probably couldn't get the "normal" way. In the Asia-Pacific region, there have also been exchanges of small and medium-sized IPv4 blocks. Obviously, paying $2560 for a "/24" (256 addresses), which is the smallest usable block of IPv4 addresses, is not a big deal for content providers.

However, the big ISPs and telcos that use IP addresses by the hundreds of thousands or millions haven't entered this market so far. The big question is: will they? Until now, a company like Comcast paid pennies per address for its multi-million blocks of IPv4 addresses. Having to pay $30 for an IPv4 address to give to every new customer would be a big hit to their bottom line. Worse, what if a year later the going rate is $60? And then $120, and so on? (And I'm not even talking about the headache of having to acquire more and more smaller and smaller blocks as the years progress.)

Rather than spend $30,000,000 on a million IPv4 addresses, a big ISP could probably spend a lot less on Carrier Grade NAT (CGN) equipment that makes it possible for multiple users to share a single IPv4 address. They still 100,000 addresses to connect a million new customers, but they probably have that kind of address space in reserve, or if not, they can take it from existing users and put those behind the CGN, too.

In the meantime, more and more ISPs are rolling out IPv6 for their users. Today, IPv6-only connectivity is not good enough to replace regular IPv4 connectivity, but the day that it will be is approaching. Even today being limited to just IPv6 connectivity for an hour or two wouldn't be too inconvenient for me. Sure, I wouldn't be able to read many news sites or post to Twitter, but I can get my email over IPv6, visit Google, Facebook, Wikipedia.

So my advice to HP: if you can get a billion dollars for those 33 million IPv4 addresses you're holding (after buying DEC), take it. You'll wish you had when IPv6 reaches critical mass and people start asking themselves why they're still bothering with that arcane 1980s network protocol. However, I'm not an economist, but I would think having millions of IPv4 addresses enter the market would lower the price.

Bottom line: small amounts of IPv4 address space will always be available in some form. But assuming you'll be able to either buy low or sell high many thousands or even millions of IPv4 addresses at any point in the future is a risky bet.

This article also appeared on the Kalorama blog. Also see this post on the topic by Louis Sterchi over there.

Permalink - posted 2014-04-28 - 🇳🇱 Nederlandse versie

Ethernet energy efficiency

▼ How power-hungry are various permutations of Ethernet on modern MacBook Pros? Tests performed and written up by Jeroen van der Ham and Iljitsch van Beijnum.

For some reason, we thought it would be a good idea to make the students of the System and Network Engineering master program test whether various permutations of IPv4/IPv6, energy-efficient Ethernet (eee) and jumboframes use more or less power. While the students were busy measuring, we were bored so we decided to run the same tests ourselves using our MacBook Pros (with Retina Display®)—Jeroen the 15" mid-2012 model and Iljitsch the 13" late 2013 model. These computers are too thin to house an Ethernet port, so we used Apple's Thunderbolt - Gigabit Ethernet adapter. We looked at the number of milliamps taken from the battery to see how much power the entire system used during various tests.

We didn't realize it at first, but the most surprising result was actually the first one, looking at system power use with the adapter plugged into the computer, but without attaching a network cable. This increased system power use by 300 mA, or about 3.4 Watts. On the 15" MBP this is already significant, increasing power use from 1250 to 1550 mA. But the 13" MBP only used 370 mA without the adapter plugged in, so its power use almost doubled after plugging in the adapter!

Could it be that the secret to the long battery life of Apple's recent laptops is the fact that they come without an Ethernet port?

We continued our testing trying different possibilities with Energy Efficient Ethernet (eee) turned on or off. Students in 2012 already looked at the general impact of eee on energy usage for clusters. They found that it provides an energy saving of up to 40% in switches, depending on the amount of traffic. Energy usage goes up when traffic goes up, but it already maxes out when you are using roughly 40% of the bandwidth. We confirmed this in our testing: there is not much difference with eee when the interface is idle, or with full traffic.

More significant energy savings started to appear when we enabled jumbo-frames (MTU 9000). The amount of data that goes through the adapter remains virtually identical, but with only one-sixth as many packets per second, the impact of per-packet overhead is greatly reduced, and this shows in the energy consumption. With full traffic in one direction this saves you over 10%. These savings also add up if you have traffic in both directions, in our experiments we saw a savings 16-19% for a bidirectional fully-utilized link.

Not surprisingly, the biggest difference in energy usage is to scale down the interface speed. When we forced the interface down to 10Mbps we saw a saving of up to 49% with full utilisation. Then again, if you're sending a large file, it is going to take a lot longer, and you basically use a lot more energy, only more distributed over time.

The results of our measurements are in the table below, but note that our measurements (using the System Profiler battery current figures) weren't very precise. The numbers give a pretty rough indication. If you want to do something serious with this data, perform your own experiments.

Total system power use	MBP15	MBP13
Interface attached vs baseline	+24%	+92%
Baseline vs idle eee	+27%	+110%
Idle eee vs no eee	-6%	-1%
Idle eee vs iperf 1 Gbps TCP eee	+74%	+97%
iperf 1 Gbps TCP eee vs no eee	-3%	-1%
iperf 1 Gbps TCP jumbo eee vs no jumbo eee	-13%	-12%
iperf 0.5 Gbps UDP eee vs no eee	+1%	+1%
iperf 1 Gbps TCP bi-di eee vs unidirectional eee	-0%	-3%
iperf 1 Gbps TCP bi-di jumbo eee vs no jumbo bi-di eee	-19%	-16%
iperf 10 Mbps TCP no eee vs 1 Gbps	-38%	-49%

We did some extra tests to look at the impact of jumboframes on CPU use. Please note that our Thunderbolt-to-Gigabit Ethernet adapters don't support TCP segmentation offloading (TSO). With 1500-byte packets the CPU utilization on the MBP13 was about 16%, with 9000-byte packets about 90%. Despite being idle more than 80% of the time, the CPU still ran at its advertised 2.4 GHz clock speed:

So the difference in CPU power use was 18% (7.9 W for 1500 bytes vs 6.4 W for 9000 bytes), despite the difference in CPU utilization only being 5%.

Permalink - posted 2014-03-03

Beware of the DFS band if you want gigabit Wi-Fi

▼ I got an Apple Time Capsule 802.11ac "gigabit Wi-Fi" base station this summer, and then a new 802.11ac-capable MacBook Pro (read my review) a few months ago. Those two can communicate wirelessly at a blistering 1.3 Gbps. Sweet!

However, after my initial testing I would never see speeds faster than 878 Mbps when clicking on the Wi-Fi symbol in the menu bar while holding the alt/option key. For a while I thought there was a software issue and I even waited to see if the MacOS 10.9.1 update would resolve the issue. It didn't.

But then I noticed something. I had my Time Capsule set for automatic channel selection, and it selected channel 100 (DFS). Then at some point it switched to one of the lower channels. And suddenly I was back to 1300 Mbps! (Although fallback to 1053 Mbps still happens regularly.)

I then tried switching channels a few more times, with fairly conclusive results: I would never get more than 878 Mbps on a DFS channel, but on channels 36 - 48 I'd often get 1053 or 1300 Mbps. Also, on DFS I would sometimes have really poor network throughput, which disappeared after switching to 36 - 48.

DFS is dynamic frequency selection. As you can see here, in the US, on all available channels except 36, 40, 44 and 48, dynamic frequency selection is required so the Wi-Fi equipment changes channels if it detects other stuff, most likely radar, getting in the way. Apparently Apple enforces this in Europe, too. So it could be that some radar installation is keeping me from getting my full 1.3 Gbps. Or perhaps the antennas or radios are less efficient at these higher frequencies.

In any event, I'm using channels 36 - 48 from now on, even though two of them are also used by neighbors.

Bonus tip: if you alt-click on the Wi-Fi icon in the menu bar and then hover over a network name, you get to see additional information:

(Yikes, I wrote "Time Machine" instead of "Time Capsule" twice originally!)

Permalink - posted 2014-02-14

The Mac Pro needs 10 Gigabit Ethernet

▼ AnandTech has a great review of the new Mac Pro. The past decade every new model looked virtually identical to the previous one, even when Apple migrated from PowerPC to x86 CPUs. Apple seems to be overcompensating for that stagnation now. The old Mac Pro was the only machine that was still expandable internally: it had tons of room for many drives and PCIe cards. With the new one, you'll have to use the external Thunderbolt and USB ports instead. I think that's not unreasonable for the most part, but the way I see it, the new Mac Pro really needs an internal 10 Gigabit Ethernet (10GE) interface.

Yes, without PCIe slots you can't put any HDMI capture cards and the like in a new Mac Pro. That's annoying right now, but this problem will go away fairly quickly as these applications move to Thunderbolt and/or USB 3. This is a much bigger market anyway, because all of Apple's computers now have Thunderbolt and USB 3.

I can see why Apple wanted to get rid of the drive bays. Those take up a lot of room. Perhaps moving to 2.5" drives rather than the traditional 3.5" ones would have helped, but 3.5" drives are still cheaper for the same capacity, and I assume also faster. Only having two 3.5" drive bays or only 2.5" drive bays would have meant that many Mac Pro users would have had to use some external enclosure for big, fast storage anyway, and this way Apple gets to build a super small system, which seems to be their goal in life.

The AnandTech review shows that using USB 3 for external storage isn't great, as the bandwidth to the USB busses is limited. And of course Thunderbolt is 10 or 20 Gbps while USB 3 is only 5 Gbps. A Thunderbolt-connected RAID array is great if you only have a single Mac Pro, but what if you're a business that has a number of them? Having separate RAID arrays for each Mac Pro is not a great solution: it would be much better to use network-attached storage. But that's not really an option over the Mac Pro's built-in Gigabit Ethernet ports. These only provide 10% of the bandwidth that Thunderbolt provides.

With the older Mac Pros, you'd simply put in a 10GE card. With the new Mac Pro, you have to use one of those cards with a TB-PCIe converter or a TB-10GE adapter. Both solutions require and additional (beefy) power supply.

10GE cards aren't cheap. Perhaps Apple didn't want to increase the cost of the Mac Pro. But as it is right now, it's a typical v1.0 product: amazing in many ways, but lacking one or two crucial features. I hope that in the next iteration, Apple puts in 10 Gigabit Ethernet. A good way to do this would be with an SFP+ interface. Currently, there are many different fiber SFP+ modules, and hopefully there will be UTP SFP+ modules in the future.

Permalink - posted 2014-01-03