For Linux 2.4.x you should think about compiling lighttpd with the option
``--disable-lfs`` to disable the support for files larger than 2GB. lighttpd will
fall back to the ``writev() + mmap()`` network calls which is ok, but not as
fast as possible but support files larger than 2GB.

Disabling the TCP options reduces the overhead of each TCP packet and might
help to get the last few percent of performance out of the server. Be aware that
disabling these options most likely decreases performance for high-latency and lossy
links.
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

Note: Be carefull with net.ipv4.tcp_timestamps. It caused massive problems for me under benchmark load with a high count of concurrent connections.

Increasing the TCP send and receive buffers will increase the performance a
lot if (and only if) you have a lot of large files to send.
net.ipv4.tcp_wmem = 4096 65536 524288
net.core.wmem_max = 1048576

If you have a lot of large file uploads, increasing the receive buffers will help.
net.ipv4.tcp_rmem = 4096 87380 524288
net.core.rmem_max = 1048576

Some things that a high-traffic site found useful:

  # These ensure that TIME_WAIT ports either get reused or closed fast.
  net.ipv4.tcp_fin_timeout = 1
  net.ipv4.tcp_tw_recycle = 1
  # TCP memory
  net.core.rmem_max = 16777216
  net.core.rmem_default = 16777216
  net.core.netdev_max_backlog = 262144
  net.core.somaxconn = 262144

  net.ipv4.tcp_syncookies = 1
  net.ipv4.tcp_max_orphans = 262144
  net.ipv4.tcp_max_syn_backlog = 262144
  net.ipv4.tcp_synack_retries = 2
  net.ipv4.tcp_syn_retries = 2

  # you shouldn't be using conntrack on a heavily loaded server anyway, but these are 
  # suitably high for our uses, insuring that if conntrack gets turned on, the box doesn't die
  net.ipv4.netfilter.ip_conntrack_max = 1048576
  net.nf_conntrack_max = 1048576


Keep in mind that every TCP connection uses the configured amount of memory for socket
buffers. If you've got many connections this can quickly drain the available memory.

See http://www.acc.umu.se/~maswan/linux-netperf.txt for more information on these parameters.