interesting packet reordering data

Neal Cardwell (cardwell@cs.washington.edu)
Mon, 21 Jun 1999 01:34:38 -0700 (PDT)

In verifying the SOSP paper scripts tonight i noticed that packet
reordering is extremely rare along most paths (.1% of flows along most
paths), but chronic (70-80% of flows) along a few paths in our data set.
The hot servers with reordering were:

Sportsline -> UW
Amazon -> UW
Sony.com -> UW

Sportsline -> Duke
Amazon -> Duke

Altavista -> UCB
Amazon -> UCB
Sony.com -> UCB

For example, for hot servers to UW, may 20:

site reordered flows
============================== ========= ======
cbs sportsline (216.32.88.54) 127 of 149 = 85%
www.amazon.com (208.216.182.15) 121 of 151 = 80%
www.sony.com (209.0.216.83) 18 of 147 = 20%
22 others 5 of 3325 = 0.1%
===
all destinations 271 of 3772 = 7%

o UCB was the only site with reordering for www.altavista.com
(204.152.190.11): 54 of 138 = 40% of flows

The trend looked similar for UW tail sites; only one tail site had
reordering, and 144 of 156 flows from that site had reordering.

Since the reordering looks different from the three client sites, it looks
like packet reordering happens a lot in some routers in the middle of the
network, and not at the servers or clients (at least FreeBSD+3com
clients).

This data also illustrates how extremely dangerous it is to take averages
over data like this (reordering, packet loss, etc). The average
reordering rate is 7% of flows, but what's really happening is that 70% of
our paths have 0% reordering and a couple have 80% reordering. Wacky
stuff.

Interestingly the reordering doesn't seem to be causing any performance
problems, because only it's slight reordering (typical case: middle packet
of a burst of three arrives last). Thus the sender rarely gets 3 dupacks
and rarely goes into fast retransmit without cause.

neal