Friday, July 6, 2007 21:25:17 EDT
Speeding up UDP

I've spent a long time at work, off and on, trying to get decent performance out of a UDP app—both speed and reliability. But as soon as we degraded the Ethernet connection past a certain point, our receiving side would block trying to get the next UDP datagram—even though zillions were coming into the machine according to Ethereal.

The light at the end of the tunnel came when one of our networking gurus swore to us that 1032-byte datagrams shouldn't slow things down more than a few percent. Now, our experience had been that a 1032-byte datagram takes about as much time to send as a 10-KB datagram—cutting transmission rate by a factor of 10.

So I hit Google for the umpteenth time, using some obvious-yet-forgotten search string, and stumbled across this thread, which pointed to a solution from comp.protocols.tcp-ip.

The solution is that Microsoft's TCP/IP stack has a magic threshold; for UDP datagrams whose size is above that threshold, it takes a much longer time to send than for those below it. The threshold is 1024, so our datagrams were just missing, and going through the slow processing path, rather than the fast one.

It's not a hard technical limit, though: they picked the number 1024 by doing a lot of realistic testing, seeing how various kinds of traffic mixes were effected, and pulling a nice round number out of their ass. Fortunately, you can change it: by adding a key to the Registry and rebooting. And it's very well documented, if you know exactly what to search for:

FastSendDatagramThreshold

. Which, of course, is the first thing that leaps to my mind when I'm sending UDP datagrams all-out and only using up 8% of the network capacity.

If it were Take Your Child to Work Day, my kids would have greatly expanded their vocabulary.

We set it to 1500, which is our MTU size, and got about an eightfold increase in throughput, enough to meet our need for speed. Then we could just solve our hanging problem by the simple expedient of making sure our UDP datagrams each fit in an IP datagram. (I still want to know exactly what was going on with that.)

So if you're forced to use Windows, and your UDP code sends small datagrams slowly, do a search for FastSendDatagramThreshold.