Slow site, despite many things have done

My site is very slow (10-20 secs on Godaddy, shared, hosted more sites) despite I increased physical memory to 1024MB.

I tried to copy the site to Hostgator (where there are not so much sites) and it loads in 5-6 secs, but it still slow.

I tested it with and it shows that loading the site itself takes long (5-6 secs) plus some must have plugins

Tips for site speed up

I do not know which one of the above would really take effect espcially on this site. Pls have a look on it and try to help me. Thanks