There are multiple directions to take this answer.
The first is that Singer is not a Vegan, and his animal welfare position is not absolutist like Veganism is. https://www.abolitionistapproach.com/peter-singer-oh-my-god-these-vegans/ Singer is not opposed to eating free range eggs, or honey, or use of feathers/wool/silk provided the animals are treated humanely, or the eating of meat or production of leather/feathers from animals that die naturally.
Singer's argument for this position is based on utilitarianism, where the object of utilitarian calculus should be any entity that can experience suffering.
There are ways to reach different moral conclusions, based on different assumptions about moral value and principles.
Some of these moral approaches are MORE extreme than Singer. Tom Regan, who wrote The Case for Animal Rights, takes a "rights" rather than utilitarian approach to ethics, and argues for the extension of full rights to all adult mammals. This still does not get one to veganism, or even vegetarianism (one can eat naturally dying mammals, and silkworms, bees, fish, and birds are not adult mammals), but it does get one closer to both, as animal farming even with no mistreatment (for wool, and milk) would be prohibited.
Another somewhat extreme approach is Deep Ecology, which treats the object of utilitarian calculus as the biosphere, rather than individuals. Deep Ecology does not care so much about our eating individual animals -- after all, LOTS of other animals eat other animals too. Deep Ecology cares about the way we have damaged the robustness of the global ecosystem, by consuming so much of the biosphere in the biomass of humans. This has weakened the robustness of the biosphere against global stressors (solar flares, global climate change, magnetic pole reversals, etc.). Under Deep Ecology we are obligated morally to PROTECT the biosphere, rather than parasitize upon it. Reducing human biomass, and restoring as many of the wild ecosystems as is feasible while humans maintaining our technologic capability to defend Gaia against geological and astronomic threats -- is a very different moral prescription from Veganism.
The human centric moral rebuttals tend to be based on Darwinian ethics of varying types. The most violent of these encourage the morality of "Nature Red in Tooth and Claw", and consider the morality of killing to be validated by evolutionary success. However, within biological ethics, the more aggressive/selfish moral arguments have been mostly refuted by the relative success of eusocial creatures, and even more interestingly, by the way we multi-cellular animals are ourselves eusocial relative to cells. Under eusociality, the community needs to be the object of moral utility, as opposed to individuals. The eusocial argument against Vegetarianism is that we humans are a eusocial community, and we humans are in competition with other species, whom we justifiably treat as a resource, to promote our community.
Note that the Deep Ecology view basically extends Eusociality to INCLUDE all ecosystems.
There are problems with most of these views. For both Veganism and Vegetarianism, we humans would not be able to do any farming. Farming requires clearing land of its vegetation, and doing this will kill adult mammals. So will plowing, mining, any construction, or any vehicle operation. The Vegan position, as well as that of Animal Rights, requires that we humans engage in no disturbing of the planetary surface beyond what low impact gathering would impose. Animal Liberation, a la Singer, would not be quite as extreme. But we could not justify any farming that would kill any significant quantity of animals -- the death rate we cause should not exceed what we would accept in child-deaths to do the same thing. And that would prohibit most farming.
Eusociality leaves a massive moral hole where rights need to exist, even if ABSOLUTE rights are probably not viable as a moral theory.
Of these alternatives, Deep Ecology, despite its apparent extremism, has the least apparent faults. It could be adopted as a guideline rather than a dogma, with an addendum to promote human welfare, and a secondary addendum to promote animal welfare, and this pragmatic approximation could satisfy most moral intuitions.