Require is a transitive verb, which means that in either case, we can look for an object later in the sentence. In the first sentence,
Professor required his students to return their papers typed.
We can diagram everything after "required",
his students to return their papers typed
as part of the object phrase, which in this case is an infinitive phrase functioning as a noun. The base infinitive is "to return," and we can add the object to expand the infinitive phrase, "to return their papers," and add an adverb (or postpositional adjective, but dynamically here it's probably just an adverb) to make it "to return their papers typed." But you can still go further. Since this infinitive form doesn't have an explicit subject, you can add the subject, "his students to return their papers typed." In fact, this verb, "require," happens to require infinitive objects to have an explicit subject. This is because in the absence of an explicit subject, the subject is assumed to be the subject of the clause. For example,
Professor wanted to return their papers typed.
In this sentence, since "his students" is not included as the explicit subject, the implicit subject for the infinitive phrase must be "professor."
The object of the sentence can also be a relative clause (signaled by "that") as in
Professor required that his students return their papers typed.
In this case, the object, what the professor required, is in the relative clause. Just about anything is acceptable as long as the phrase signaled by "that" is an independent clause in the subjunctive mood. You could even drop the word "that" and it would still be correct in some contexts.
My conclusion is that both of these sentences are perfectly acceptable in English. Hope that helps.