Domain Mapping has got too many rows

I've got a question about the domain mapping plugin. I'm running a Query Monitor plugin because we've been experiencing slower than normal load times and the one plugin it always pulls up is the Domain Mapping Plugin as taking huge amounts of resources.

It's saying that the plugin has almost 55,000 rows in the database and we've only got about 800 total sites and maybe 250 or so that are using custom domains.

So my question is about whether it's possible to clean that out somehow without losing current data. The network has been around since 2011, so we do have legacy sites that have been deleted etc, but maybe only 1000 at most.

Please Advise