Vendors unite to develop next generation Ethernet

This article originally appeared in the July/August 2008 issue of FibreSystems Europe magazine.

The standardization of higher data rates is vital if Ethernet is to continue as a ubiquitous end-to-end protocol. Pauline Rigby finds out how standards are progressing.

Ethernet has traditionally evolved in multiples of 10, from the first successful commercial version of Ethernet at 10 Mbit/s through Fast Ethernet (100 Mbit/s), Gigabit Ethernet (1 Gbit/s) to 10 Gigabit Ethernet (10 Gbit/s) – the highest speed available today. But the next multiple — 100 Gigabit Ethernet (100 GbE) — hit a speed bump when disagreement arose between different interests within the Institute of Electrical and Electronic Engineers (IEEE) Higher Speed Study Group (HSSG).

Eventually vendors reached an agreement, deciding to add 40 Gbit/s data rate to the proposed standard for 100 GbE, which allowed things to move on to the next stage. On 5 December 2007, the HSSG officially became the IEEE P802.3ba Task Force, doing the technical work to define the standard. Here Pauline Rigby talks to John D’Ambrosia, chair of the Task Force, whose full-time job is to guide the process and ensure the work finishes on time.

PR: What’s driving the demand for higher data rates?
JD’A: Quite simply traffic was growing at such a rate in network aggregation applications, especially with the proliferation of 10 GbE, that it needed a higher-speed solution. People were talking about aggregating 16 or 32 links of 10 GbE to handle all of the servers they were using. Link aggregation was becoming very unwieldy; we needed something faster.

Naturally the Ethernet switch vendors were the most vocal about the need for 100 GbE, but it wasn’t just them. One of the big differences between this project and its predecessors was the involvement with the end-user community. We had people from Google, EDS, Microsoft Networks and AMS-IX [Amsterdam Internet Exchange]. We had content providers coming in because video drives the bandwidth through the roof.

One of the examples that really blew people away was from Yahoo! Asia Pacific. They were offering a service for streaming classic major-league baseball games, and filled a 40 Gbit/s pipe. They don’t know what the capacity request actually was because they had to cap the service at 40 Gbit/s.

I was talking with an individual who was at Google at the time of the HSSG, and he said that: “People point at Google as the anomaly and say well that’s just Google. I can tell you that where Google was a year and a half ago is where all of the other data centres are coming to now, they’re all starting to hit these problems.”

If the demand for bandwidth is growing so rapidly, how did the need for an intermediate rate at 40 Gbit/s arise?
There’s one chart that really helps to explain this question (figure 1). This is a case where a picture is worth a thousand words. This chart was based on input from a server vendor. He was talking about Moore’s Law and how the server I/O [input/output] doubles every 24 months. When the growth curve for network aggregation was added, what we see is two different growth rates for two different applications. Core networking doubles every 18 months, server I/O doubles every 24 months. When you plot this out over time you can see this leads to a huge difference.

We mapped different Ethernet standards across the lines, and looked at how that relates to the needs. You can see why Gigabit Ethernet was pretty much an instant success when it was released in 1998. It instantly hit the requirements for network aggregation, and a couple of years later it hit the servers.

Computing and networking bandwidth needs for 40 and 100?GbE.

In general people say Gigabit Ethernet was a huge success, but when you ask about 10 GbE, it’s a different story. If you were talking to someone who was judging 10 GbE by the value it brought to networking, those individuals say it was a raging success, and you can see that from the chart. However, if you were talking to people who were judging by ports shipped, i.e. server vendors, they say they don’t really need it and won’t need it until around 2010.

To move forward within the IEEE you need to have 75% consensus. Considering the two different backgrounds of the people present in the room, you can imagine the difficulty the group at that time had in coming to grips with this problem so that it could move ahead.

You must have been very happy that a consensus was reached. Is it unusual to have two different rates being developed for a single standard?
Yes it is unusual; it’s the first time history that we’d had two new rates at one time, so saying it’s a big deal is an understatement. But while it was unusual to do the two rates, when you look at the chart it becomes apparent that two rates was the right decision.
Some people are saying that 100 GbE is going to be late. Is that the case?

There are many people that will tell you that 100 GbE is going to be late, and if you look at the chart you can understand why they think that. When you look at 100 GbE in relation to the networking applications, it just barely meets the curve. We’re going to finish the specification at around the time the need is there, so there’s going to be a lot of pressure on the group to produce this thing as fast as we can. On the other hand, if you look at the relation to the computing applications, 100 GbE will be seven to eight years early.

Is there a risk that by trying to please everyone you end up pleasing no-one?
Let me answer that from the standards approach. There was essentially a one meeting delay before we were able to become a task force, which impacted on the schedule. However, as the chair my primary job is to make sure that a standard gets done in the proper time frame, and as a result we’ve scheduled accordingly. It really hasn’t been an issue since we made a decision to do it this way.

In fact, we’ve come up with a unique architecture that will satisfy both rates at the same time, and we’ve adopted most of the baseline proposals around that architecture. There are a few more pieces that we need to fill in, but we are making incredible progress.

That’s interesting. Tell us how the architecture can support both rates.
Several of our key objectives concern the physical layer specifications, or PMD [physical medium dependent]. Right now the solutions that are being considered are based on ganging four lanes or 10 lanes: four lanes of 10 Gbit/s, 10 lanes of 10 Gbit/s or four lanes of 25 Gbit/s. Those lanes could be physical lanes or they could be wavelengths. There are also some individuals from a transport background who proposed 40 Gbit/s serial, and others who suggested two lanes.

The next layer in the structure is the PMA, which means physical medium attachment, and that’s where the multiplexing happens. We came up with a unique approach that allows us to adjust the multiplexing to support 10 lanes, four lanes, two lanes — whatever we need.

One of the main areas we had a lot of debate on was in the physical coding. We have just adopted an approach called MLD — multilane distribution — which is based on the standard 64B/66B encoding that we do now in Ethernet. Simply put, it’s a mechanism to distribute the data across those different lanes. So now we have a lane-distribution mechanism that carves up the signal into the right number of lanes for whatever PMD we’re going to target.

The task force met in May. How did that move things along?
The May meeting was phenomenal, and I attribute that to the hard work that the task force is doing in terms of the consensus building and due diligence with the technical issues. We actually wound up making quite a few baseline proposal decisions at that point, which is ahead of schedule.

On the multimode fibre objective for 40 GbE and 100 GbE we’ve adopted a proposal that’s n × 10.3125 Gbit/s across parallel fibres. For the 40 km 100 GbE and the 10 km by 100 GbE we’ve adopted a 4 × 25 Gbit/s architecture. The decision on 40 GbE over 10 km hasn’t been made yet.

There are two proposals out at this point — one is for 40 Gbit/s serial and the second is for a 4 × 10 Gbit/s CWDM [coarse wavelength division multiplexed] approach. That will be one of the key decisions we’ll need to make at the July meeting so that we can move ahead.

What’s the timetable for completing the 40/100 GbE standard?
To keep to our schedule we aim to generate draft 1.0 after the September 2008 meeting. The next phase will be a Working Group Ballot — that’s where you open it up to the 802.3 Ethernet Working Group for comment, which is scheduled for the March meeting in 2009. After that comes the Sponsor Ballot in November 2009, where you open it up to the members of the IEEE Standards Association. The standards release is scheduled for June 2010.

For more information visit www.ieee802.org/3/ba/.

Reproduced with permission. © Institute of Physics and IOP Publishing Ltd.