These prisoners live in ignorance with their idea of reality being simplistic shadows cast upon walls, but they know nothing else.
Is it wrong that their perception of life is different than those who have been let out of the cave?
That all depends on whether or not you think ignorance is bliss.
By living in ignorance and leading a shallow life these prisoners don’t know what they’re missing out on in the world above; the shadows are all they know. They may have a superficial glimpse of the world, but they may be happy in their shallow mindset.