TCP Chatter

The netstat command lies. If you run it, you'll see packets larger than your MTU (maximum transmission unit - you can see this if you run the ifconfig command). This is because netstat is reporting what's happening at the OS level and "the operating system is passing packets larger than MTU to the network adapter, and the network adapter driver is breaking them up so that they fit within the MTU." [1]

This is called TCP Large Segment Offload (TSO or LSO) and is an optimization but makes seeing what is really going on harder. You can turn this off using the ethtool command [2].

Capturing the conversations between clients that are uploading 100MB of data to a server running JStringServer (work in progress), focussing on a particular conversation (by looking at the ports chosen in the exchange) and cleaning the data (with sed) such that data looks like:

34.565659  SERVER > CLIENT
34.606133  SERVER > CLIENT
34.606143  SERVER > CLIENT
.
.

(where the number is the time in seconds) for the server-to-client and

34.562933  CLIENT > SERVER
34.600927  CLIENT > SERVER
34.603579  CLIENT > SERVER
.
.

for the client-to-server allows me to generate a histogram of packets per time interval.

Using a little bit of the R-language:

server2Client <- read.table="" span="">"~/Documents/Temp/43645_server2Client.txt", header=FALSE, sep=" ")

client2Server <- read.table="" span="">"~/Documents/Temp/43645_client2Server.txt", header=FALSE, sep=" ")

numBreaks = 20

hist(c(client2Server$V1), breaks= numBreaks, col=rgb(1,0,0,0.5), xlab="time", main="Packets")

hist(c(server2Client$V1), breaks= numBreaks, col=rgb(0,0,1,0.5), xlab="time", main="Packets", add=TRUE)

yields this:

Packet Histogram. Red is client-to-server. Blue is server-to-client.

Red are the packets from the client uploading its data to the server. Blue is the server packets back to the client (note: these are almost entirely ACK packets). (Purple are the two superimposed on each other. Note, there is only one time interval where the number of server-to-client packets exceeds the client-to-server).

The upshot of all this is that TCP is terribly chatty. The server returns nothing but the string "OK" when the exchange is complete. But this doesn't stop it from sending lots of packets to the client ACKnowledging receipt of a chunk of data. That's a lot of redundant bandwidth.

Further Reading

Interesting paper on TCP burstiness here.

[1] Wireshark docs.
[2] Segmentation and Checksum Offloading - Steven Gordon.