So this is kinda strange. I enabled the Ultimate Color Schemes module in the Ultimate Branding plugin and I noticed my admin pages slowed down considerably. I didn't even realize it was due to this till I started working backwards and figuring it out.
Turns out when I enable it, it takes almost 13 seconds to load an admin page and with it disabled it takes about 5 seconds or less.
Why would this happen? I really like the module. I suppose I could do everything I want with CSS but it's much easier with the module.