For each firm in the dataset we are eagerly loading, we first set the cached invoices association to the empty array.
Then we get all invoices for all clients of all of the firms in the dataset, using the keys of the id map.
For each of those invoices, we associate it back to the related firm using the values of the id map for the invoice's client's firm_id, adding it to the existing array of invoices
After you are finished processing all of the invoices, each firm will have the all related invoices in the association cache, so calling the invoices method on any firm object returned will not cause any additional database queries.
Note that there is nothing inherently specific about this approach, creating a generic plugin that supported any has_many through has many association is possible and probably not even all that complex.
There is already a plugin that does something similar for polymorphic associations, using basically what I've already explained.