Using a plain hash for the temporary data, and the standard hash setter method like this, gives an instant 25 percent speedup.
I submitted a simple patch that does this, but it was rejected by Tadayoshi without a clear explanation as to why.