[SAGE] TCP Window Tuning for High Bandwidth/High Latency WAN links?

["John Stoffel" <john@stoffel.org>]

Hi all,

We're getting ready to deploy the next step of a high speed WAN VLAN
network between a bunch of remote offices for interoffice networking
needs.  It's all going to be DS3s of various speeds with the vendor
providing us with an ethernet port (100 mbs/full duplex) to plug into
a router.  

It's all up and working it our test setup, except that we're not
seeing the speedups we'd expect with this network.  After doing some
research, I've come to learn more about the Bandwidth Delay Product
(BDP) and how it affects TCP/IP sessions, specifically in our case
FTP, NFS and SSH/SCP sessions.  Since our users send big files back
and forth between sites, sometimes via ftp, sometimes via NFS (we
automount all the data directories across the WAN), it's imperitive
that we get the most bandwidth usage on a per-connection basis.  I
realize that I can do a bunch of connections at once to get the
throughput I need (see iperf and the -P option), that's not acceptable
for sending a 300gb file across the WAN.

Some of the web pages I've been look at include:

    http://www.psc.edu/networking/projects/tcptune/
    http://www.psc.edu/networking/projects/hpn-ssh/theory
    http://dast.nlanr.net/Guides/GettingStarted/TCP_window_size.html
    http://dast.nlanr.net/Projects/Iperf/iperfdocs_1.7.0.html

Which explain the issue pretty clearly, though some of these articles
are a bit dated and don't talk about Solaris 8 much, which is our
dominant Unix OS, along with a bunch of Linux boxes in LSF queues.  

So the options for fixing or improving performance seem to come down
to:

1. tune the TCP settings on each host for all connections, which may
   impact memory usage and won't do much of LAN connections if at all.

2. turn the TCP window size on a per-server (ftp, ssh) and per-client
   basis (ncftp, scp, etc).  Then training the users if the tuning
   isn't automatic how to use it...

3. Get a network box which will do this for us (yet more money...)
   auto-magically at each site.  One option, which I have pricing on
   at all is:

	http://www.internap.com/products/FCP-solution.htm

   Though we won't be multi-homed in our WAN/VLAN setup at this time,
   too much money.  

So it all comes down to what other people have done and/or are doing
in this type of situation?  What solutions have your deployed?  As WAN
links get faster, yet the RTT time doesn't shrink, TCP is going to
need some interesting hacks to make it work better in this situation.

Thanks,
John