When asked a question that could be confusing, answer in a complete sentence that stands on it's own.
Try to understand the question as meaning "can you tell me about what you did?" instead of either "did you?" or "didn't you".
I do not believe that people asking questions in this way are hoping for either a "yes or no" answer.. I believe they would appreciate more information about what you did or what you were thinking.
"Did you not go to the store?" A "yes" or a "no" would be a slightly anti-social answer on top of the ambiguity. A polite answer could be "I did go to the store an hour ago and picked up some strawberries and the milk you mentioned we needed."
"Don't you want to come to my house?" To a large degree, they are asking "why" you do or don't want to come to their house, not merely "if".
"I'd like to but maybe next time, I need prepare for an early meeting tomorrow."
Why not simply ask "why"? Because asking "didn't you?" gives the person answering the possibility of deferring more vaguely without forcing them into a more complicated explanation than might be needed, depending on what their unknown answer will be. It let's the other person say "I'm sorry, I can't today" without explaining themselves while still expressing an invitation to share if a person wanted.
There is no ambiguity or trouble if you answer in a sentence that stands on it's own without a "yes" or "no" you will not come across as flippant or be misunderstood.