Deleting a plugin: what happens to the files?

For example, if I install a plugin, use it for a while, but then find a plugin to replace it...what happens to the files from the first plugin after the plugin is deleted? Are they all deleted?

Does it depend on the plugin?

I'm curious about how this stuff works behind the scenes. I feel like if a plugin is installed and it creates code to be used on my site, does some of its code still exist somewhere in my website's files once its deleted?