Gravatar caching exhausts the memory limits

When I enable Gravatar caching, Gravatar caching exhausts the memory limits and gives 500 error on product pages.