Over and to him should not be linked.
The combination of "to him" = preposition + pronoun. This can be explained as (i) an indirect object (ii) an adverbial modifier (iii) a (dative) complement
I...handed......it...over...to...him
S...…V…......O...Adv...prep...pronoun
Note how "quickly" or "reluctantly" can substitute for "over", or "over" can be removed altogether.
I...handed...over...it...to.......him
S...…V…......Adv.…O…prep…pronoun
Note how "quickly" or "reluctantly" cannot substitute for "over", although "over" can be removed altogether. This tends to indicate that I handed it over to him should be the preferred form.
Compare:
I...handed...the meal...over... the counter....to......him
S.......V............NP........prep..........NP.........prep…pronoun
I...handed...........him..............the meal...over... the counter.
S.......V..........Indirect Object........O.........[prep.........NP]
S.......V.........Adverbial Modifier......O......Adverbial Modifier
S.......V........Dative Complement...O.........Adverbial Modifier