empirical server loss rates due to exclusion

John Snell (geigudr@cs.washington.edu)
Tue, 26 May 1998 11:06:44 -0700 (PDT)

Europeans:

# Of 29 slaves at the time, 4 removed -- 13.8% loss rate.

Americans:

# 46 servers at the time, 10 removed -- 21.74% loss rate.
# Of educational, 9 total (non-UW), 2 lost -- 22.2% loss rate.

Stanford was one educational site removed (SDSC the other); happy guys
that they are, they even included a comment in the file:

# go away
User-agent: *
Disallow: /

For the math impaired, that means we have 36 US servers to play with right
now, before I write the Harvester (or do any promo'ing of the beauty of
network research). Measurements begin... now. <beeps watch>

For the truly curious (Neal), here's the specifics:

#
#Removed 5/26/98, due to robots.txt exclusion of /cgi-bin
# 46 servers at the time, 10 removed -- 21.74% loss rate.
# Of educational, 9 total (non-UW), 2 lost -- 22.2% loss rate.
#
#"SD SuperComputing Center", "San Diego, CA",
"http://www.sdsc.edu/~hutton/cgi-bin/tracert.cgi?", true, 40, 30,
"198.202.75.101"
#"Stanford", "Stanford, CA",
"http://www.slac.stanford.edu/cgi-bin/nph-traceroute.pl?", true, 40, 30,
"134.79.18.11"
#
# Contents of Stanford file:
# # go away
# User-agent: *
# Disallow: /
#
#"Yahoo", "Santa Clara, CA",
"http://net.yahoo.com/cgi-bin/trace.sh?", true, 40, 30, "206.251.17.74"
#"Illuminati Online", "Austin, TX",
"http://www.tpoint.net/cgi-bin/trace?", true, 40, 30, "199.170.88.39"
#"Mag-Net BBS Systems", "Prince George, BC, Canada",
"http://www.mag-net.com/cgi-bin/nph-traceroute?", true, 40, 30,
"207.102.83.2"
#"Peterson Data Mgmt", "Vancounver, BC",
"http://www.petersen.net/cgi-bin/nph-trace?", true, 40, 30,
"204.244.192.12"
#"PICS On-line", "Mount Holly, NJ",
"http://www.pics.com/cgi-bin/trace-route?", true, 40, 30, "192.135.189.20"
#"AboveNet", "San Jose, CA", "http://www.above.net/cgi-bin/trace?", true,
38, 30, "207.126.96.163"
#"NovaGate Communications", "West Michigan",
"http://www.novagate.com/cgi-bin/trace.cgi?", true, 40, 30,
"205.138.138.2"
#"Call America", "San Luis Obispo, CA",
"http://www.callamer.com/cgi-bin/cgiwrap/pbowden/trace.sh?", true, 40, 30,
"199.74.141.2"

#Removed 5/26/98, due to robots.txt exclusion of /cgi-bin
#
# Of 29 slaves at the time, 4 removed -- 13.8% loss rate.
#
#
#"Internet Vision", "England",
"http://stingray.ivision.co.uk:8080/~azeem/cgi-bin/nph-traceroute.cgi?",
true, 40, 30, "195.50.91.40"
#"Euro InterSoft", "Zelenograd, Moscow, Russia",
"http://www.orgland.ru/cgi-bin/trace?", true, 40, 30, "194.190.194.33"
#"New Space", "Czech Republic",
"http://www.space.cz/cgi-bin/nph-traceroute?", true, 40, 30,
"194.213.40.3"
#"I-Com ISP", "Prague, Czech Republic",
"http://home.icom.cz:8080/cgi-bin/traceroute.cgi?", true, 40, 30,
"194.196.42.3"

_____________________________________________________________________________
"The human mind is a 400,000-year-old legacy application...and you expected
to find structred programming?" -- Randall Davis, 1996 AAAI Pres. Address