Your example is a colloquial statement, and the "rules" don't apply easily to colloquial statements
The preposition "of" is a function word that often identifies or sets off a possessive relationship. Let's start with an example of no relationship.
That's too much time.
I've heard this before and it sounds normal to me, but we also don't know whose "time" it is. We only know that whatever the statement refers to, the amount of required time exceeds the available time. The phrase "That's too much of time" sounds awkward and makes little sense. However...
That's too much of my time.
Now we have a posessive relationship. It's not just any old time, it's "my time." It's worth noting that an adjective doesn't create a posessive relationship. Thus:
That's too much of quality time.
is incorrect. "Quality" is just an adjective.
Therefore...
The "rules" suggest that you should NOT have an "of" in the phrase "not too bad an idea."
However...
Colloquialisms don't obey rules as they often come from common slips-of-the-tongue as people babble along. Frankly, folks in my area toss an extra "of" into phrases all the time. If you're hearing the phrase rather than reading the phrase, the "of" is there because the speaker either doesn't know better or isn't paying that much attention to what they're saying.
If it is written, then whether or not the "of" belongs there is an issue of the rules of writing, which include the rules of dialog, which would need an explanation from you of where/how you heard the phrase. Because if the written use was dialog and the writer was intending to reflect the colloquial use of the phrase, then I'd expect to see the "of."
Language — you gotta love it