It is definitely philosophically and logically defensible to claim that morality is not universal. There are some good comments made here which make reference to how this is the accepted view in meta-ethics. Meta-ethics is the study of what ethical statements mean, as opposed to what ethical acts are and how to be ethical. The latter is discussed in applied ethics and ethics more broadly when we think about different ethical systems such as utilitarianism, consequentialism, etc.
It's great to see more-and-more acceptance of the evolutionary origins of our morality in philosophy and society. Many still disagree with this point of view: that the origins of our morality are wholly evolved (or a result of evolution). The counter-view is that morality is universal and we either: 1) don't know the origin of this universal morality or 2) we do know the origin of this universal morality — such as divine, rational inquiry, etc. I will not talk about the possibility of a universal morality, only arguments that could contradict this view.
Yes, objectively we can't say anything is absolutely right or wrong as all claims about moral acts are value judgments. However, most humans don't seem to have a choice in the matter of making moral judgments and acting on them. We make moral judgments because we can't not make them. Hence why there is a sharp focus in some areas of moral philosophy on the evolutionary origins of morality. This will hopefully shed more light on why it is we act morally and don't have a choice in the matter for the most part even though we think we absolutely do.
Some humans, on the other hand, don't act how most of us would think is moral. Somebody whose brain does not function neuro-typically (due to developmental differences in the brain) may lack the capacity to be empathetic. They understand that their actions are wrong on a rational level, but do not feel their actions are good or bad. They may not feel that killing or stealing for their own gain is good or bad even if they know perfectly well that this is not acceptable behavior.
There's a difference then between people who know what's morally wrong and still act in this way but feel guilty about it afterward; and those who know what's considered morally wrong to others, still act in this way but feel zero guilt about it afterward. Neuroscientists have studied how the brain functions, for example, in psychopaths in prison who lack empathy and compared this data with those who don't have this type of disorder (see: Probing psychopathic brains). It seems that psychopaths can switch on and off their empathy when they choose to. Whereas, most of us, don't have a choice in that. If we see something we consider morally wrong, we immediately feel a negative emotional response to it.
There are many reasons why this would be advantageous to us in an evolutionary context. Some of the reasons include group bonding and fitness, evading confrontation with enemies and rivals, rearing children so that they have the best chance of survival and so on (see: Morality and Evolutionary Biology). We are not the only animals to display what could be considered moral behavior and have a sense of fairness. Other animals cannot articulate they know this of course, so we can't say they "know" they are acting morally.
As a society, if we want to think about what makes something moral or not we should consider how the brain evolved to feel emotions which direct our moral behavior. This knowledge can also be used when we think about ethical dilemmas such as the trolly problem. We may feel bad about sacrificing 1 person to save 5 if we are emotionally isolated from this, but if we have to feel the discomfort of physically pushing somebody to their death, this is much more of an emotional experience and a much harder decision to make. And when asked to sacrifice your loved one to save the 5 people, not many would save the strangers — all things being equal. A utilitarian may argue that which places human happiness as the highest good ought to be done. Therefore, these are all morally equivalent scenarios and you should sacrafice your own family to save the strangers. There are specific logical and rational defenses as to why your own family has more utility to you and not to others etc. But it is clear emotions (which are determined wholly by the brain) play a big role in making moral decisions and we should, therefore, heavily consider how emotions work to understand morality.
If someone's brain is not capable of feeling the same emotions as others, such as empathy, should we place the same amount of blame on them as somebody who does? There is an example in neuroscience of a man who was convicted of being a pedophile who also had a brain tumor in the orbitofrontal cortex. This region of the brain is responsible for judgment, impulse control, and social behavior. The man knew his behavior was morally indefensible but continued to act in this way regardless. The brain tumor he had was eventually cut-out and then he stopped feeling these pedophilic impulses. But eventually the tumor returned, and so did the pedophilic impulses (see: Brain tumour causes uncontrollable paedophilia). This is quite a shock and is highly controversial. Nearly all of us think and know why this act is so morally repugnant and would never and couldn't even imagine actually doing so, let alone exacting this crime. But to some, it appears that even in such an extreme scenario, perhaps they have no real "choice" in the matter.
If emotions can overpower moral, logical, and rational decision making, what does that say about the absolutism or universalism of our morals? Not only does this go beyond morality being relative to cultures, it also questions whether people that know the same things are morally indefensible all act in the same ways morally. How should we then think about blaming or praising moral actions? Therefore, these questions are highly linked with notions of free will. Do we have it? What is it? Is it real? If we do or do not have free will, what does this mean for moral decision making?
More on this by the philosopher Derk Pereboom:
http://joelvelasco.net/teaching/hum9/pereboom-nofreewill.pdf
https://www.youtube.com/watch?v=bObzpWrhH-Q
And I recommend checking out the neurobiologist Robert Sapolsky. He talks about how morality, evolutionary biology, free will, and morality intersect from a scientific point of view:
https://www.amazon.com/Behave-Biology-Humans-Best-Worst/dp/1594205078
https://www.youtube.com/watch?v=Cx8xEUYrb74
