Clean Cache

i've downloaded and installed so many plugins i would like to clean out all the old tables left behind... i've noticed if i install a plugin, activate it, change settings, deactivate, delete and reinstall... the changes i made in settings stayed... so there must be something that's being saved... and would this cause my site to slow down at all? is there away to remove such "cache"