This is one of the trickiest parts of punctuation, and there's not really a good answer and different style guides may even suggest different solutions. The best solution I've been able to come up with is to write so that the punctuation fits seamlessly within the enclosing sentence. The rationale I work to has its origin in a more basic sentence structure where the rule is pretty much universal.
"I like cows," Jane said.
Here, the comma isn't really part of Jane's sentence, which, as a quotation on its own would be expressed as
"I like cows."
So the key principle is that while the post quotation punctuation remains within the quotation marks, it is selected to make sense in context of the sentence as a whole.
The problem you are having is because we expect questions to end with a question mark, whereas we're used to the convention whereby full stops (periods to the American types) are replaced by commas, to the extent its done without really thinking about it.
You have a couple of options that would probably fit within common usage - using either a comma or a question mark before the closing quote. Neither is entirely satisfactory.
The way I solve this problem in practice is to rephrase the sentence so as to make it go away, thus your sentence would become:
He asked me to get him a drink, but I didn't want to.
If the exact words that were used were important, I'd probably go with:
He said "will you get me a drink?" However, I didn't want to.
Note that "will" doesn't get capitalised here because it's in the middle of a sentence, and takes its structure from the sentence around it. Some style manuals suggest using a comma after "said" in this context, but I prefer not to because I feel it interrupts the grammatical flow of the sentence.
If this sort of sentence is giving you difficulty, the key thing to remember is that the point of written text is communication. Try to find a construction that communicates your meaning as clearly as possible. It helps to remember that the way inline quotations are punctuated is largely down to convention rather than any particular logic.