Friday, July 6, 2007 21:25:17 EDT
Speeding up UDP

I've spent a long time at work, off and on, trying to get decent performance out of a UDP app—both speed and reliability. But as soon as we degraded the Ethernet connection past a certain point, our receiving side would block trying to get the next UDP datagram—even though zillions were coming into the machine according to Ethereal.

The light at the end of the tunnel came when one of our networking gurus swore to us that 1032-byte datagrams shouldn't slow things down more than a few percent. Now, our experience had been that a 1032-byte datagram takes about as much time to send as a 10-KB datagram—cutting transmission rate by a factor of 10.

So I hit Google for the umpteenth time, using some obvious-yet-forgotten search string, and stumbled across this thread, which pointed to a solution from comp.protocols.tcp-ip.

The solution is that Microsoft's TCP/IP stack has a magic threshold; for UDP datagrams whose size is above that threshold, it takes a much longer time to send than for those below it. The threshold is 1024, so our datagrams were just missing, and going through the slow processing path, rather than the fast one.

It's not a hard technical limit, though: they picked the number 1024 by doing a lot of realistic testing, seeing how various kinds of traffic mixes were effected, and pulling a nice round number out of their ass. Fortunately, you can change it: by adding a key to the Registry and rebooting. And it's very well documented, if you know exactly what to search for:

FastSendDatagramThreshold

. Which, of course, is the first thing that leaps to my mind when I'm sending UDP datagrams all-out and only using up 8% of the network capacity.

If it were Take Your Child to Work Day, my kids would have greatly expanded their vocabulary.

We set it to 1500, which is our MTU size, and got about an eightfold increase in throughput, enough to meet our need for speed. Then we could just solve our hanging problem by the simple expedient of making sure our UDP datagrams each fit in an IP datagram. (I still want to know exactly what was going on with that.)

So if you're forced to use Windows, and your UDP code sends small datagrams slowly, do a search for FastSendDatagramThreshold.

Thursday, December 13, 2006 16:17:10 EST
Quicksilver Meets Emacs

Merlin Mann over at 43 Folders regularly sings the praises of Quicksilver, especially for quickly adding lines of text to text files. I've tried it, and never really incorporated it into my workday, because I already have "one app to rule them all": emacs.

But it's still not quite as easy to append to a text file in emacs as it could be: you have to actually switch buffers, then go to the end, then type your addition. So I wrote a little elisp to handle the chore:

 (setq char-to-buffer-alist
       '((?t . "todo")
         (?p . "todo_personal")
         (?b . "Blotter")
         (?a . "agendas")
         ))
 
 (defun char-to-buffer-name (c)
   (let ((entry (assoc c char-to-buffer-alist)))
     (if (null entry)
         "*scratch*"
       (cdr entry))))
 
 (defun tobuffer (bufc msg) (interactive "cChar: \nsAppend line: ")
   (if (> (length msg) 0)
       (save-excursion
         (let ((bname (char-to-buffer-name bufc)))
           (set-buffer bname)
           (end-of-buffer)
           (insert-string "\n__ ")
           (insert-string msg)
           (insert-string "\n")
           (message "Appended to %s" bname)))
     (switch-to-buffer (char-to-buffer-name bufc))))

 (global-set-key 'f8 'tobuffer)

Now I can just hit f8, then a sensible mnemonic for the buffers/files I use frequently (or RETURN for *scratch*), and throw a line of text in at the end, without losing my current context (though it does change where point is in the target buffer).

Or, if I just want to go to that buffer, I enter a blank line of text (i.e. just hit RETURN).

It's been a month or so, and I don't use this all that much, and most of my use is to go to the target buffer. But maybe I just haven't marketed it very well to myself.

Sunday, February 26, 2006 11:34:48 CST
PyCon, Part Two

Saturday was fun: Guido's keynote was technical; Jim Fulton's State of Zope talk cleared a couple things up for me; Chad's Un-Reinventing the Wheel was cool.

I hung out developing in the quiet room for the beginning of the afternoon. I wanted to position bullets well for fonts other than Chalkboard; it turns out 2/3 x-height works well. One problem: there's no get_xheight() call in Font. After trying unsuccessfully to fake it with half the average of height and ascender-height (and some less-obvious secret formulae), I realized I could automatically determine a font's x-height: render a white "x", then scan down from the top for the highest white pixel, and up from the bottom for the lowest. The difference is, by definition, the x-height.

(OK, you have to make sure to not use anti-aliasing, and there's a quirk where some fonts put the lowest pixel of the "x" at the baseline while others put it just above, but all I really want out of this is good-looking bullets.)

Then I went to the re-run of Ian Bicking's "Building Pluggable Software with Eggs," which was deservedly popular. I tossed what sounds in retrospect like a softball question: Is there anyone distributing Python code who shouldn't use Eggs? The answer turns out to be "No."

The Lightning Talks rocked, and were plenary, which is a great idea. I might try to do one on PyGame today.

I missed most of the book party, but did get calories out of it, and bought a book on grid computing. I hit the bar, and drank with the same group from eleven until midnight, and turned in early (all right, the bar closed).

Friday, February 24, 2006 11:10:48 CST
PyCon, Part One

Things have gone great so far. In spite of some minor glitches (such as discovering in mid-air that my verion of Keynote was a 30-day trial, and it was up), my PyGame tutorial yesterday afternoon went well: the students seemed happy, and were actually absorbing the material. I was mildly disappointed that, when we ran short of time, the class voted the Fonts topic off the island — I had a good joke on the topic.

I was somewhat more disappointed that none of the students showed up at the PyGame Lab afterwards, but a few other people dropped in, including Anna Ravenscroft (with a Martelli in there, I forget where), so I had fun and spread the PyGame gospel a little further.

I ran into Mike Orr last night on my way to find some food, and fortunately remembered a bunch of places south of Beltline Road that are open until 10:00. I grabbed an avacado-muenster sandwich at The Great Outdoors, a deli that would have fit right in in Ithaca.

Familiar faces continued to arrive last night (Holger, Stephen Waterbury, Guido, Ian Bicking, Mike, Itamar), and this morning I finally found Jeff Elkner (without students in tow this year).

This morning's keynote was kind of interesting, but (like all keynotes, and as Limi and Runyan themselves admitted) not technical enough. I got interrupted halfway through by a phone call from Molly; James got on the line and informed me that I was at a meeting in Chicago, which I had not previously been aware of.

Now I'm in "All Grown Up: Mission-Critical Python", EWT's Open Space, which is more like a long Lightning Talk. I'm probably going to go drift around after I post this.

Saturday, February 4, 2006 9:27:27 EST
Doors and Locks, Materials and Labor

Two or three weeks ago, Joe managed to lock everyone (including himself) out of the boys' bedroom. There was no real urgency, so Isabel and Molly left it to me when I got home.

No problem: bedroom doors all have privacy locks, not serious deadbolts or exterior knob locks. So I just need a small screwdriver or an awl and I'm in business.

Except.

For some reason, one and only one bedroom door in our house has an actual key lock on it, which requires either a locksmith or a much better lock-picker than me.

Can you guess which one?

After a couple attempts to gently kick in the door (it seemed like a coherent thought at the time) that succeeded only in completely freaking out Joe, I called Brendan, who recommended either carding the door (already tried it) or slipping a scraper straight through the trim to push back the latch (no luck whatsoever; I now think I may have been getting caught up on the not-very-flush latch plate).

At this point, he asked, "Do you have a sledgehammer?" He recommended knocking the knob off to get to the inside part of the latch. He also pointed out that a replacement knob set would cost around $15, and having a locksmith even show up would cost at least $100.

I don't own a sledgehammer (I don't even want to imagine what the boys would do with one), but I have a couple two-by-fours kicking around, and two dozen whacks with one of those persuaded the outer knob to fall off, leaving only some quick disassemly before I gained entry.

So there's been a big hole in the boys' bedroom door for a while. I finally got over to the home center today to rectify the situation, and discovered Brendan's estimate was off: the exact same knob set cost a whopping $10.97. Once I got it home, it took about five minutes to install. Joe still hasn't noticed it's there.

There's a valuable lesson here about how incredibly efficient manufacturing can substitute very efficiently for labor, but its elaboration is left as an exercise for the reader.