Best practice for using the Directory plugin

I'd like to use it in my main blog but I'm worried since my MS is already so plugin heavy, it might just render it to a snail's pace. The other option would be in another directory with a separate WP install...but I sure would like the SEO of the directory to bring people to the main idea of joining up for a site. Can anyone tell me if I have a lot of directory categories, will basically kill my MS site speed wise? Thanks.