This use of would is apparently regional. It is totally natural to me, so it is a bit hard to explain, but basically, this is a subjunctive construction that is used to signal that the outcome depends on somebody else's choice, decision, or action. In other words, the speaker has played his role enabling a situation, now the result depends on the other's actions.
Sometimes Toby would have come back, and there would be loud music in the drawing room;
So in the quote, the speaker is present and able to hear the music, and if Toby had come back, then he would in fact hear loud music. If Toby hadn't come back, presumably the music would not be on, or not be as loud. Would here establishes an unambigous causal relation between Toby and loud music.
Another way of doing the same thing is
Sometimes, should Toby have come back, there would be loud music in the drawing room;
I've looked for information on this usage before but haven't found anything. There are several other questions here in ELU about similar usages. None attracted much attention or got a researched answer.
Looking at the two sentences above, they aren't quite identical. The second is about the fact that Toby came home. The first is about him choosing to come home.