Should I use "the wife" or "my wife"?

It's contextual. At first read, I'd say that the term could indeed be considered impolite. Depending on how it's uttered, that impoliteness could range anywhere between playfully impersonal to callously derogatory.

Generally speaking, I believe my wife would prefer that I call her "my wife," as opposed to "the wife," especially in a context like:

He: Can you go to the baseball game with me on Friday?
Me: I don't know; I'll have to ask the wife about that.

However, there are instances where the words could be used in an almost complimentary fashion:

He: That was a smart investment you made!
Me: Well, I pretty much always listen to the wife about those kinds of things; she's pretty smart about that.

So, it very much depends on how you use the term. In written communication, I'd say that "my wife" is the safer phrasing, because "the wife" could easily be construed as insensitve, even if you didn't mean it that way.

Both are right, but in the context I think you mean, My wife is almost always preferred. My wife, his wife, (heteronormativity off) her wife, your wife, etc. is the standard use when you want to say something about your wife. This holds for many nouns, not just wife.

In other situations "the wife" would be totally normal such as 'The wife of a dead man is called a widow.'

However, you are very right that the wife can be idiomatic; it has a very specific meaning with a specific connotation. It is not impolite or derogatory although it could be used in a negative sense.

The wife alludes to the generic idea of a wife rather than a specific person. It's a good phrase to use if what you are trying to say might apply to wives in general rather than just your wife or the wife of someone present. For example, 'If you're coming to my party, you can bring along the wife.' or 'I don't think the girlfriend will let me get away with that.' This phrase is very specific to a few select phrases such as wife and girlfriend and wouldn't be used with most other nouns.

My ex husband always referred to me as "the wife" in a very negative tone when I was married to him. I resented it. Saying "the wife" rather than "my wife" takes the person out and objectifies her; it devalues her. The only time that I personally feel that "the wife" would be proper is if someone else is introducing you; for example "This is Mary, the wife of John Adams." Otherwise, the proper way for a husband to refer to his wife is "This is Mary, my wife."

It may express annoyance with the wife. I found the SAME poster use "my wife" and "the wife" in different posts in an audio forum today, as in "the treadmill is used mostly by my wife",(later) "my wife has real issues with infinite-baffle installations" and (in the third post) "the wife saw my design and she thinks it's too big." Hope this answers your question.