We have 125,000 members and are growing rapidly. When our authors & editors go to review posts, we are getting 504 errors because edit.php calls wp_list_table which calls _get_list_table('wp_posts_list_table') to get all members and displays them in a dropdown of authors. Only a few members (<100 editors&admin) need to be in the list even though we allow all members to post.
users.php does the same to re-assign posts when deleting a member. The same list works here.