I get a box with link to image on my public page

When adding a picture in a textbox I get a box with link options on my public page. How do I fix this? Sounds like a bug. It shows the images correctly when Im logged in to my WP-ADMIN account though.

See screenshot.