The TicketStumbler guys write on switching their spider to Twisted and learning from the experience:

Twisted is Event Driven and Asychronous meaning that things don’t necessarily happen in sequential order and code should be written to respond to events such as “the data is ready” or “his ass is grass.” Despite knowledge of over a dozen programming languages and familiarity with using threads and processes to run sequential code blocks beside one another, this was a very new concept to me and one I initially found difficult to grasp enough to write code for it (though using it was quite straight forward).

twisted