Removing Add Media Button in Directory

In the WP Directory I would like to remove the "Add Media" button. Is there a way to do that?