The reason why many fruit-related terms are associated with craziness is that most of these slang terms were previously related to homosexuality. In the early 20th century homosexuality was considered a mental illness, as a result this led to a shift in meaning that continued until the 1960s.
By the 1930s both fruit and fruitcake terms are seen as not only negative but also to mean male homosexual, although probably not universally. It should be noted that LGBT people were widely diagnosed as diseased with the potential for being cured, thus were regularly "treated" with castration, lobotomies, pudic nerve surgery, and electroshock treatment so transferring the meaning of fruitcake, nutty, to someone who is deemed insane, or crazy, may have seemed rational at the time and many apparently believed that LGBT people were mentally unsound. In the United States, psychiatric institutions ("mental hospitals") where many of these procedures were carried out were called fruitcake factories while in 1960s Australia they were called fruit factories.
http://en.wikipedia.org/wiki/Fruit_%28slang%29
Homosexuality was associated with fruits because of the effeminacy of homosexuals, thus softness. Fruits are soft also and there you go. You can refer to Cassell's Dictionary of Slang (By Jonathon Green) for the origins of fruit-themed slang terms with connotations of homosexuality:
As you can see below, fruitcake had the meaning of eccentric person first but gained the meaning of crazy in US slang after gaining the connotation of homosexual man. It must be the craziest food because it contains both fruits and nuts. Hence, you can define the craziest person as "nutty as a fruitcake" and even "nuttier than a fruitcake".
Banana also meant homosexual in the past:
But the phrase go bananas is said to be originated from zoos. Monkeys in the zoo go wild when they see bananas coming. Below is excerpt from the book Food: A Dictionary of Literal and Nonliteral Terms (By Robert Allen Palmatier):
Cupcake is not necessarily related to fruits but it connoted homosexuality also in the 1970s:
Though, in the case of "nuts", people were actually crazy for the food and the food became crazy. People were nuts about nuts.
In the late 19th century, the British used "nuts" as slang for something they found enjoyable: (This usage may have originated in an old cliché—"sweet as a nut.") Being nuts on something meant you really liked it, but so did being "crazy on something." It's possible that "nuts" became a synonym for "crazy" because of this similarity.
http://www.slate.com/articles/news_and_politics/explainer/2006/04/what_makes_nuts_so_crazy.html
Crackers is a bit different and trickier though. It seems it is derived from cracked and it shares the same meaning with other similar terms: crackerbarrel, cracko, cracky:
cracked adj. 1. [17C+] insane, crazy, eccentric; thus cracked about/on, obsessed with, infatuated with (cf. CRACKED IN THE FILBERT phr.; CRACKERBARREL adj.; crackers adj.; cracko adj.; cracky adj.).
It might be associated with mental breakdown and thus, falling to pieces. It seems it shares the same or similar roots with craze:
From Middle English crasen (“to crush, break, break to pieces, shatter, craze”), from Old Norse *krasa (“to shatter”). Cognate with Danish krase (“to crack, crackle”), Swedish krasa (“to crack, crackle”), Norwegian krasa (“to shatter, crush”), Icelandic krasa (“to crackle”).
http://en.wiktionary.org/wiki/craze
Etymonline says that it is still used in its original sense in crazy quilt:
Original sense preserved in crazy quilt pattern and in reference to cracking in pottery glazing (1815). Mental sense (by 1620s) perhaps comes via transferred sense of "be diseased or deformed" (mid-15c.), or it might be an image.
It is used in the phrase drive someone crackers also. Below is excerpt from the book Food: A Dictionary of Literal and Nonliteral Terms (By Robert Allen Palmatier):
