J.S. Farmer & W.E. Henley, Slang and Its Analogues, volume 1 (1890), has an entry for "turn out no bottle," which it says comes from "sporting":
BOTTLE. To TURN OUT NO BOTTLE, phr. (sporting).—Not to turn out well ; to fail.
Given that John C. Hotten, A Dictionary of Modern Slang, Cant, and Vulgar Words, second edition (1860) has an entry for bottle-holder:
BOTTLE-HOLDER, an assistant to a "Second,"—Pugilistic ; an abettor ; also, the bridegroom's man at a wedding.
it is tempting to suppose that the sporting context of the Farmer & Henley term is also pugilism. But the bottle that the "assistant to the Second" holds is presumably an actual bottle, and why it should be associated with the fighter's courage is not obvious—particularly when more-obvious symbolic gestures of defeat occur in the boxing trade—specifically, throwing in the towel, or throwing in the sponge—either of which may actually be done to concede a fight.
Eric Partridge, A Dictionary of Slang and Unconventional English, eighth edition (1984) provides a useful chronology of the different slang senses of bottle. here are the ones old enough to have influenced lost [one's] bottle and bottle out:
- turn out no bottle (later C.19 [citation to 1887], but obsolete by 1930)
- bottle meaning the monetary take from an entertainment (late C.19 [citation to 1893])
- no bottle meaning no good; not 'classy' (ca. 1920)
- not much bottle meaning not much good (ca. 1920)
- bottle meaning spirits guts, courage (by 1965)
Though this is only speculation on my part, I wonder whether the bottle in "turn out no bottle" might have emerged from bottle in the sense of earnings or profit from an entertainment. It wouldn't be a great stretch to extend the notion of earnings as a share of ticket sales to include earnings from successful bets and (perhaps) prize money for winning a fight. In that case, "turn out no bottle" might originally have meant "turn out no monetary winnings"—the common fate of a losing fighter in those days.
If so, that sense of bottle might have given rise to the 1920s notions of "no bottle" and "not much bottle," which in turn might have encouraged an equating of bottle as "success" with bottle as "courage."
Then again, if we think of a "bottle of spirits" as a source of "spirit" rather than of "spirits," the connection of bottle (that is, spirit) to courage is direct and unconditional.