There is nothing bad about wearing shoes without insoles. Doing so occasionally wouldn’t do much serious damage. However, it is not a very good idea to wear shoes without insoles all the time unless the shoe is slightly smaller and will fit better without the insole.