If, as is often implied, we're all blind men in some sense, wouldn't
this be a fatal flaw in the argument?
No. If you only consider one subjectivity valid, your picture will be incomplete, and you will find the experiences irreconcilable. But by entering into the experiences of others, a larger perspective can be found, capable of encompassing the disparate experiences. To get our best picture of the world, we must at least tentatively consider the experiences of others as like our own - this is pointing at intersubjectivity, which another ancient metaphor also first recorded in Buddhist texts illustrates too, Indra's Net.
We find this all the time in science, different ways of approaching the same knowledge, with for instance the four (now maybe five) fundamental forces and associated energy levels, which it is thought were unified in one force near the big bang, which will probably require energies that can probe the Planck scale to prove (ie black holes). Consider the necessary knowledge to get to unify the 3 quantum forces: nuclear synthesis (mainly from cosmology), deep inelastic scattering and nuclear decay, and relativistic electrodynamics that followed from precise photon energy tunings. Parts of the elephant - we know they fitted together or interacted deeply at or after the big bang but now are at separate scales, we know they are part of the 'big picture' that can encompass all the data (Hawking suggested Godel's incompleteness theorems mean no final theory is possible, I would suggest only no computationally enumerable final theory - ie, it could be a mind, a strange-loop).
We also have many cases of top-down versus bottom-up ways to approach knowledge, like psychology versus neuroscience - we know they are both trying to understand the same thing, just in different ways.
Probably the most common purpose the story has been turned to, is different religions approaching one greater truth. I like the Kurzgesacht short story The Egg, as a playful example of how a bigger perspective could encompass the cosmologies of Buddhism, Hinduism, and Abrahamic ideas.
We are in fact, a community of the blind - in science, in religion. We don't, cannot have, access to the transcendental supersubjectivity without actually being god (we might participate or contribute though), and we can't have a final theory without that theory elaborating reality beyond itself - there can be no 'final vocabulary'. There will always be an elephant in the room, you might say.