

I have no idea how alignment works. I can well imagine that it might make sense to train artificial intelligence using values that have been determined through democratic consultation with large population groups. But let’s assume that the AI is already perfectly aligned. Could it be that it knows much better than humans themselves what is good for humans? If you ask a small child what it wants, it might say: a mountain of chocolate. But the parents would know very well that this mountain of chocolate is not good for the child. Another question, of course, is under what circumstances one could trust such an AI.

Yes, I agree with you. I would also like to see the process of AI alignment be a democratic process that is regularly adjusted to reflect people’s values.