Optimal PHP memory size?

Just wondering the best route for determining PHP memory size from a best practices point of view. We're currently at around 64 MB and have no issues unless somebody tries to delete a lot of pages or posts as once, which is rare.

While memory isn't an issue on our server, we only use 20% of allotted, my understanding is that you don't want to set excessively high values for PHP as it will allow runaway processes to run even longer and get excessively out of control.... is this understanding correct?

Is the best route to use a memory limit that works and if you get errors, bump it up? Or is there an optimal range we should stick in for a growing MU install?

Any feedback appreciated!