There is ambiguity in both phrases. The first is only due to loose usage, the second is due to lack of explicit reference.
"Next Monday," in the strictest interpretation, means whatever Monday will happen closest in the future, excluding the one you currently occupy if it happens to be Monday. I think of it as similar to saying "the next red stepping stone;" if you are on a blue stone, you're talking about the first subsequent red stone you will encounter, while if you're on a red stone, you're talking about whatever stone you will encounter after the one you currently occupy. If the speaker intends to reference the Monday after the closest one in the future, the appropriate phrase is probably "the Monday after next."
"This Monday," however, is ambiguous. "This" could be interpreted as "this week's," "this past," or "this coming." Unless one is pointing at a square on a calendar, it would probably be best to explicitly use one of the two latter phrases to eliminate any confusion.