It has nothing to do with the k. It's essentially because modern English orthography is a mess.
The two words have always had different vowels, and before the Great Vowel Shift, these two different vowels were written differently as well.
In Old English, ‘know’ was cnāwan, and ‘now’ was nū. As you can see from the GVS link above, the regular outcome of /aː/ and /uː/ in Old English is /oʊ/ and /aʊ/, respectively, and these two words fit that: know is /noʊ/ and now is /naʊ/.
The trouble is that the orthography didn't quite keep up. Know had an original /w/ after its vowel, and in late Middle English when they started writing the word with an /o/ to better match what the pronunciation was drifting towards, that /w/ was still there, so the word ended up having <ow>. But that's really kind of a coincidence (the same coincidence is true with throw < þrāwan). If there hadn't been a /w/ in Old English, it would more likely have ended up being spelt <kno> instead, and ‘know’ and ‘now’ wouldn't have been homographs (apart from the k, of course).
There is a smallish group of monosyllabic words that end in -ow which rhyme with ‘go’: know, throw, slow, grow, stow (and probably some more that don't come to mind right now). If you look all these up on Etymonline, you'll see that they all come from Old English words that have /āw/ (Etymonline doesn't write length, though, so it'll just look like /aw/ on there), or /ow/, as in the case of grow and stow.