Mathematics, at least when I was at school, would be notionally attached to the famous (or infamous) Platonic dicta 'let no-one ignorant of geometry enter here' which turns out, to stretch a term, an urban myth; and possibly more due to the nature and vagaries of the English pedagogical tradition.
There is, in fact a Platonic Dialogue, Theatatus where Socrates asks Theodorus, a mathematician that hailed from Cyrene, a colony in North Africa, to recommend any students of his that would make promising philosophers; he does - the eponymous Theatetus.
Given his description:
Snub-nosed with protruding eyes
Which is not far off descriptions of Socrates himself; it's hard not to think that it's Socrates counselling his younger self. The subject of the dialogue, surprisingly, is the nature of knowledge - Socrates asks Theaetatus is there a simple formula for it.
This, in essence, is the gist of your own question - to establish a simple formula for knowledge; such formulas have been established already - the Descartian axiom, the Turing Test and Wittgensteins Logic - but of course these are at best partial answers; it's also in the context of this dialogue that Godels incompleteness theorem displays it's special interest, at least philosophically.
I read books every now and again ... [but] I haven't simply learnt from them.
Reading helps, but did only reading help with mathematics? Perhaps it did, particularly early on, but to enter into the spirit of mathematics today, especially as it is today, takes a certain serious devotion: Questions must be posed, and answered; the real structure of a proof understood, as well as variants and all this must come, somehow naturally - this is the process of subjective dialectic as dramatised in the dialogue above.
It's no different to any other field that has rigour - political philosophy included; this is if real knowledge is sought (and I mean real in the sense of your own questions); however it is true that strong opinions strongly and articulately held often count as knowledge - it's this that Socrates skewers in the Sophist (which doesn't mean that the skill of polemic and rhetoric if done rightly is not a virtue - this is in part, the nature of good journalism).
Should we refrain from speaking about something about we have no high knowledge of
In the particular instance that one is called upon to be an expert, then no, of course not; but then again you will not be called to be an expert witness (this being no disparagement of you abilities) so this situation is nothing to fear.
In the general situation this will depend on the company - people are various; as you've noticed it's often worth asking questions - and besides philosophy on SE, there is history and politics; quite often I have seen good questions and answers get low votes and views and quite commonly too; this given the nature of the site should be taken as a given - so one needs to learn to recognise what counts as a good question and answer; yours, by the way, was a good question.
