Dual server + loadbalancer setup

I'm setting up a dual MU-wordpress server behind a loadbalancer (+ mysql server). Everything seems to be working. wp-content/uploads is on an NFS share and accessible on both servers, the loadbalancer is forwarding the traffic to both servers, I've tested them both individually.
However, as soon as both WP-servers are up, I'm logged out of the wp-admin pages (sometimes immediately, sometimes after 1 or 2 clicks), and I cannot log in anymore (sometimes it works after 3-4 tries, but then I'm logged out again, and can't possibly log in anymore). The only thing that works is shutting down apache on one of both wp-servers.
Any suggestions? The apache loadbalancer is set to stickysession=BALANCEID & the php sessions directory is shared between the two wp-servers too (although Wordpress shouldn't be using it).