memory exhausted, tried everything I can think of

I am using a CentOS VPS with godaddy, so I have complete control over the server. While importing databases of posts i get this error:

“Allowed memory size of 67108864 bytes exhausted (tried to allocate 7680 bytes)”

I know youve all seen that before, who hasnt. The confusing thing is, I have tried every known trick in the book.

I am using APC and FastCGI as well, so I dont understand why I am having this issue. I have edited php.ini, .htaccess and wp-config to no avail. see the following:

this is the contents of my php.ini:

tidy.clean_output = Off

include_path = “.:”

mysql.allow_persistent = On

mysql.max_links = -1

safe_mode_allowed_env_vars = PHP_

session.bug_compat_42 = Off

mysql.connect_timeout = 60

session.use_only_cookies = 1

register_argc_argv = Off

open_basedir = “/var/www/vhosts/fallriverhydro.com/:disappointed:tmp/”

session.name = PHPSESSID

session.hash_function = 0

magic_quotes_runtime = Off

log_errors_max_len = 1024

mssql.secure_connection = Off

pgsql.max_links = -1

variables_order = “GPCS”

sybct.allow_persistent = On

max_input_time = 300

odbc.max_links = -1

session.save_handler = files

session.save_path = “/var/lib/php/session”

session.cookie_lifetime = 0

pgsql.auto_reset_persistent = Off

session.cache_expire = 180

error_reporting = E_ALL & ~E_DEPRECATED

safe_mode_gid = Off

auto_prepend_file =

sendmail_path = /usr/sbin/sendmail -t -i

sybct.min_client_severity = 10

pgsql.max_persistent = -1

auto_globals_jit = On

soap.wsdl_cache_ttl = 86400

allow_url_fopen = On

smtp_port = 25

register_globals = Off

display_startup_errors = Off

user_dir =

mssql.min_error_severity = 10

mysqli.max_links = -1

default_socket_timeout = 60

apc.ttl = “60?;

session.serialize_handler = php

session.hash_bits_per_character = 5

unserialize_callback_func =

default_mimetype = “text/html”

magic_quotes_gpc = Off

safe_mode_include_dir =

max_execution_time = 600

mail.add_x_header = On

upload_max_filesize = 128M

register_long_arrays = Off

safe_mode = off

zlib.output_compression = Off

safe_mode_protected_env_vars = LD_LIBRARY_PATH

odbc.max_persistent = -1

mssql.compatability_mode = Off

file_uploads = on

mysqli.reconnect = Off

session.cookie_domain =

apc.shm_size = “512”

mssql.allow_persistent = On

mysql.max_persistent = -1

mssql.max_links = -1

session.use_trans_sid = 0

mysql.default_socket =

mysqli.default_pw =

odbc.defaultbinmode = 1

sybct.max_persistent = -1

output_buffering = 4096

doc_root =

log_errors = On

mysql.default_host =

request_order = “GP”

display_errors = on

mysqli.default_socket =

safe_mode_exec_dir =

html_errors = Off

session.entropy_length = 0

y2k_compliance = On

post_max_size = 32M

asp_tags = Off

memory_limit = 1G

short_open_tag = On

SMTP = localhost

precision = 14

session.bug_compat_warn = Off

sybct.min_server_severity = 10

mssql.min_message_severity = 10

session.gc_maxlifetime = 1440

session.gc_probability = 1

allow_url_include = Off

mysqli.default_host =

mysqli.default_user =

session.referer_check =

pgsql.log_notice = 0

mysql.default_port =

odbc.defaultlrl = 4096

pgsql.ignore_notice = 0

mysql.trace_mode = Off

engine = On

odbc.allow_persistent = On

track_errors = Off

pgsql.allow_persistent = On

session.auto_start = 0

auto_append_file =

disable_classes =

mysql.default_password =

url_rewriter.tags = “a=href,area=href,frame=src,input=src,form=fakeentry”

allow_call_time_pass_reference = Off

ignore_repeated_source = Off

sql.safe_mode = Off

session.cookie_path = /

session.entropy_file =

expose_php = Off

report_memleaks = On

session.gc_divisor = 1000

mssql.max_persistent = -1

serialize_precision = 100

odbc.check_persistent = On

sybct.max_links = -1

define_syslog_variables = Off

session.cookie_httponly =

session.cache_limiter = nocache

enable_dl = Off

ignore_repeated_errors = Off

mysqli.default_port = 3306

disable_functions =

magic_quotes_sybase = Off

soap.wsdl_cache_enabled = 1

soap.wsdl_cache_dir = “/tmp”

session.use_cookies = 1

mysql.default_user =

implicit_flush = Off

bcmath.scale = 0

Here are the relevant lines in wp-config:

define(‘WP_MEMORY_LIMIT’, ‘256M’:wink:;

define( ‘WP_MAX_MEMORY_LIMIT’, ‘256M’ );

and in .htaccess:

php_value memory_limit 256M

what else could it be?

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hi there, PhilMay!

    What’s the physical RAM on your server? And do you know what kind of RAM it is? I can see here that they offer several packages with varying amounts of physical RAM, is your plan one of these? https://www.godaddy.com/pro/managed-vps

    There’s a chance, if you’re on the 2GB or 3GB RAM VPS, you may not have the hardware to support this upload, even if you’ve got all the software properly configured.

    Have you taken a peek at your server error logs for the time of one of these upload failures to see if there are any clues there?

    Thanks!

  • PhilMay
    • Site Builder, Child of Zeus

    the file uploads, but when i select the fields for each column of the csv file and actually perform the import of data is when the error occurs. also there is only 1000 rows with 10 columns of data per row. I split a csv that has 268k rows into 268 smaller files.

    I would really love to not need to break it down any further, it is not even one of the larger database, and there are 50 databases total i need to import. is it somehow possible to handle the import directly through mysql or phpmyadmin?

  • PhilMay
    • Site Builder, Child of Zeus

    I guess I am confused about multi-db. Installing 4096 databases seems to just have used up all my ram on the vps.

    I see everywhere that people say its good for sites/blogs. What about a single multi site install that has an enormous database though? Is 4096 overkill to handle 20+million posts with or without images?

    I guess I am just looking to use multi-db to enhance performance when all that data is uploaded.

    here are 2 rows of the data so you see that it is not even alot of data per “post”

    COMPANY NAME	MAILING ADDRESS	CITY	STATE	ZIP CODE	Location ID	PHONE NUMBER	WEB ADDRESS	Category	Sub Category	Tags	level id
    1-Stop Package 458 N Main St Palmer MA 1069 29 4132832022 Liquors-Retail Liquors-Retail 1

    I guess the difference between my install, and say edublogs, is that we will not have individual sites/blogs for each user. Rather it is just a lot of overall data.

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hi there, PhilMay!

    Ultimately, MultiDB, and especially the option to use 4096 databases for a network that’s not hosting millions of sites on the same network is usually overkill, by quite a lot.

    MultiDB is great for gigantic networks, but there are usually simpler, and ultimately better ways to improve you site performance that I would typically recommend before MultiDB, at least if the end goal is not to host millions of sites.

    I hope this helps!

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hey there PhilMay!

    I’ve been thinking a bit about your particular case, and I wondered if you’d done any research on Varnish as a tool for optimizing your site? I’ve got some links for you about how it works.

    http://code.tutsplus.com/tutorials/optimizing-wordpress-with-varnish-and-w3-total-cache–cms-21136

    http://systemsarchitect.net/boost-wordpress-performance-with-varnish-cache/

    http://z9.io/2011/08/09/speed-up-wordpress-with-apache-and-varnish/

    These should give you an overview of how it works, I think this may give you the performance boost you need without having to use MultiDB.

    Let me know if you’d like more info on this, or if this isn’t an option, and we’ll keep looking.

  • PhilMay
    • Site Builder, Child of Zeus

    l actually do not have multi-db installed anymore. I think the vps in question is juist underpowered. Ive purchased a Dell Poweredge 2950 Gen III server with 2x 3ghz quad core xeons, 32GB ram, 256mb raid cache on card, and 6x 1tb SAS drives. that I am debating running in raid 60 or raid 10. Hopefully this hardware can handle at least one state worth of listings to launch this directory. I am not going to be multisiting as I can not see how multisite would actually be a benefit, therefore multi-db is useless.

  • Tyler Postle
    • CGO

    Hey Phil,

    Thanks for getting back to us!

    Are you able to upload any size CSV? Even a small one? Is it only failing on large files? Just wanting to confirm that it really is the size of the file and not something else that is coming into play here!

    Look forward to hearing back!

    All the best,

    Tyler

  • PhilMay
    • Site Builder, Child of Zeus

    Yes i was able to upload csv files. After playing with some settings in apache and APC I was able to get everything to work, the only thing that timed out was when i used the importer to geocode via the api, rather than upload pre geocoded. This being said, it is only google maps api at this moment that is timing out on me (or at least apache is not waiting long enough for maps api to respond). Either way, as stated above, I have reinstalled the site without multisite and without multi-db.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.