control pages that should be indexed by search engine

What is the best way to control the pages on your website that get indexed by search engines?

I want my website content to show up like the attached search result for CNN. In my case I want the base to the portal sub-domain and other sub-domains to show up as categories. These sub-domains are setup purely as CNAME aliases and are currently not part of WP.