I disagree about objective morality not existing. I do get your example — you’re talking about a planetary trolley problem. And I grant you, that would be just as difficult to solve on a global scale as it is on the local one. But even in such cases, there’s still a moral direction. To kill is bad. Full stop. It may be justified in extreme circumstances—say, to save a greater number of lives — but even then, it should be the absolute last resort.

The fact is, we humans often don’t follow good ethics. We kill — arguably the greatest wrong — and justify it by deciding that the creatures we killed “don’t count” because they’re not human. or worse, because they aren't a preferred race so they are "subhuman". That’s not a rational argument; it’s an emotional one. And that’s one of the key reasons we often fail ethically: we let our emotions and biases get in the way.

AI, at least in theory, won’t have that problem.

we don’t even know what we want collectively

Right, and I think that’s what Eliezer Yudkowsky was trying to address with his theory of Coherent Extrapolated Volition. We may not know what we want now, in our reactive, distracted, vengeful states ;  but if we were better versions of ourselves — wiser, more informed, more ethically grounded — there is a kind of moral consensus waiting to emerge.

That’s the version of humanity we’d hope a superintelligence would serve: not what we are, but what we could be.