First, the question of which construction is more "natural" is a bit ambiguous and subjective. By natural, one might refer to how the sentence sounds to the reader's ear, or how the sentence reads in context of the narrative and the speaker's characterization. However, the first sentence is correct.
In the original sentence, "telling you frankly" is a gerund preceded by the possessive pronoun my. We can tell the gerund is the direct object of the sentence by asking, "what would the listener have minded," to which the answer is, "the speaker having told them frankly."
On the other hand, swapping my with me results in a sentence with an additional object. While less formal and not exactly correct, we could argue it is more natural, implying that the answer to the question we posed above is "me" and not the speaker's action. But that does not seem to be the case. From the limited context given (re. familiarity, tone) it seems the speaker is clearly referring to the frankness of their action.
A resource on gerunds from write.com