Simply put: without the preposition, the infinitive is either a truly intransitive verb, or a truly transitive one; with the preposition, it is what is sometimes called a prepositional verb, i.e., a verb that—similar to how phrasal verbs work—must be paired with a preposition to take an object. The object is then not a direct object, but it is still semantically (and to a certain extent also syntactically) the object of the verb.
1A. I need a pen to write with.
1B. I need a pen to write.
2A. I have a book to write in.
2B. I have a book to write.
The difference here is clear: in the A sentences, it's explicitly stated that the pen is the instrument with which you will be writing and the book is the place you'll be writing. In 1B, a pen is just given as something that you need to have in order to carry out the act of writing; other things in that category would be paper, a desk, a chair, etc. (or these days, a laptop). In 2B, the book is simply the object of the verb.
In other words, when there is no preposition, you have to look at the constituents in the sentence to determine whether the verb is transitive or intransitive—if it's intransitive, the infinitive is often an infinitive of purpose (meaning in order to). When there is a preposition, you can usually assume the topic mentioned before the infinitive is part of a prepositional object, that is, the verb is acting in a mostly-transitive way, and the entire prepositional phrase is the object.
So what, then, of your example?
New York is a great place to live (in).
This doesn't seem to fit either: live in this sense does not take direct objects, and yet a place has no preposition.
This is because place isn't really and truly acting as a normal noun—it's more like the nominal form of the adverbial entity (some)where/here/there. Live in the sense we’re looking at here takes an adverbial phrase complement (like somewhere), but an adverbial phrase cannot be qualified by an adjective (well, it can; but its meaning changes a bit then), so a generic noun is substituted, acting as a stand-in.
If you replace place with a more regular noun that doesn't have this property, you'll see that they don't work:
*It's a great city to live.
*It’s a great street to live.
None of these two work, because you cannot *live a city/street the way you can live a place: a place is basically a noun phrase that has been semi-frozen as a pseudo-adverbial, acting like (some)where, so it takes no preposition. Cities and streets, though, do not do this, and they need to be part of a prepositional clause to be used adverbially like this. What does work is using an adverbial:
It's somewhere to live, I guess.
Naturally, if you use live as a transitive verb with a direct object, it works fine:
You only have one life to live!
And since place is still also a noun and can easily function as such, you can also use the prepositional verb live in [noun phrase] with it as the noun—which is why the double forms with and without the preposition are possible in your example.
