Directory plugin is performing badly lots of queries fired

In my performance monitoring it came clear that the directory plugin is a performance hog, firing around 50 queries to launch the content.
But a real performance penalty on my site loading.

How can this be fixed?