It's contextual. At first read, I'd say that the term could indeed be considered impolite. Depending on how it's uttered, that impoliteness could range anywhere between playfully impersonal to callously derogatory.
Generally speaking, I believe my wife would prefer that I call her "my wife," as opposed to "the wife," especially in a context like:
He: Can you go to the baseball game with me on Friday?
Me: I don't know; I'll have to ask the wife about that.
However, there are instances where the words could be used in an almost complimentary fashion:
He: That was a smart investment you made!
Me: Well, I pretty much always listen to the wife about those kinds of things; she's pretty smart about that.
So, it very much depends on how you use the term. In written communication, I'd say that "my wife" is the safer phrasing, because "the wife" could easily be construed as insensitve, even if you didn't mean it that way.