Its okay, in English. For example I would happily email someone to say "I'm visiting [your city] to see potential customers,do you want to meet up later". One might even say it to the potential customers. Its not actually offensive or wrong.
But some people might not see themselves that way, or might feel it assumes what they will do (it is presumptuous). It is like if you described someone you like and have seen a couple of times and might end up.close to, as a "potential bedmate": people like to be wooed and not assumed what they will do.
Perhaps emailing them "The purpose of this business trip is to visit people who might be interested in our services", or "I'm visiting to explore business prospects with some contacts" (calling other business people "contacts" doesn't assume whether they will buy or suggest they are only seen through a lens of "potential buyers")