memory exhausted, tried everything I can think of

I am using a CentOS VPS with godaddy, so I have complete control over the server. While importing databases of posts i get this error:
"Allowed memory size of 67108864 bytes exhausted (tried to allocate 7680 bytes)"

I know youve all seen that before, who hasnt. The confusing thing is, I have tried every known trick in the book.

I am using APC and FastCGI as well, so I dont understand why I am having this issue. I have edited php.ini, .htaccess and wp-config to no avail. see the following:

this is the contents of my php.ini:
tidy.clean_output = Off
include_path = ".:"
mysql.allow_persistent = On
mysql.max_links = -1
safe_mode_allowed_env_vars = PHP_
session.bug_compat_42 = Off
mysql.connect_timeout = 60
session.use_only_cookies = 1
register_argc_argv = Off
open_basedir = "/var/www/vhosts/fallriverhydro.com/:disappointed:tmp/"
session.name = PHPSESSID
session.hash_function = 0
magic_quotes_runtime = Off
log_errors_max_len = 1024
mssql.secure_connection = Off
pgsql.max_links = -1
variables_order = "GPCS"
sybct.allow_persistent = On
max_input_time = 300
odbc.max_links = -1
session.save_handler = files
session.save_path = "/var/lib/php/session"
session.cookie_lifetime = 0
pgsql.auto_reset_persistent = Off
session.cache_expire = 180
error_reporting = E_ALL & ~E_DEPRECATED
safe_mode_gid = Off
auto_prepend_file =
sendmail_path = /usr/sbin/sendmail -t -i
sybct.min_client_severity = 10
pgsql.max_persistent = -1
auto_globals_jit = On
soap.wsdl_cache_ttl = 86400
allow_url_fopen = On
smtp_port = 25
register_globals = Off
display_startup_errors = Off
user_dir =
mssql.min_error_severity = 10
mysqli.max_links = -1
default_socket_timeout = 60
apc.ttl = “60?;
session.serialize_handler = php
session.hash_bits_per_character = 5
unserialize_callback_func =
default_mimetype = "text/html"
magic_quotes_gpc = Off
safe_mode_include_dir =
max_execution_time = 600
mail.add_x_header = On
upload_max_filesize = 128M
register_long_arrays = Off
safe_mode = off
zlib.output_compression = Off
safe_mode_protected_env_vars = LD_LIBRARY_PATH
odbc.max_persistent = -1
mssql.compatability_mode = Off
file_uploads = on
mysqli.reconnect = Off
session.cookie_domain =
apc.shm_size = “512”
mssql.allow_persistent = On
mysql.max_persistent = -1
mssql.max_links = -1
session.use_trans_sid = 0
mysql.default_socket =
mysqli.default_pw =
odbc.defaultbinmode = 1
sybct.max_persistent = -1
output_buffering = 4096
doc_root =
log_errors = On
mysql.default_host =
request_order = "GP"
display_errors = on
mysqli.default_socket =
safe_mode_exec_dir =
html_errors = Off
session.entropy_length = 0
y2k_compliance = On
post_max_size = 32M
asp_tags = Off
memory_limit = 1G
short_open_tag = On
SMTP = localhost
precision = 14
session.bug_compat_warn = Off
sybct.min_server_severity = 10
mssql.min_message_severity = 10
session.gc_maxlifetime = 1440
session.gc_probability = 1
allow_url_include = Off
mysqli.default_host =
mysqli.default_user =
session.referer_check =
pgsql.log_notice = 0
mysql.default_port =
odbc.defaultlrl = 4096
pgsql.ignore_notice = 0
mysql.trace_mode = Off
engine = On
odbc.allow_persistent = On
track_errors = Off
pgsql.allow_persistent = On
session.auto_start = 0
auto_append_file =
disable_classes =
mysql.default_password =
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
allow_call_time_pass_reference = Off
ignore_repeated_source = Off
sql.safe_mode = Off
session.cookie_path = /
session.entropy_file =
expose_php = Off
report_memleaks = On
session.gc_divisor = 1000
mssql.max_persistent = -1
serialize_precision = 100
odbc.check_persistent = On
sybct.max_links = -1
define_syslog_variables = Off
session.cookie_httponly =
session.cache_limiter = nocache
enable_dl = Off
ignore_repeated_errors = Off
mysqli.default_port = 3306
disable_functions =
magic_quotes_sybase = Off
soap.wsdl_cache_enabled = 1
soap.wsdl_cache_dir = "/tmp"
session.use_cookies = 1
mysql.default_user =
implicit_flush = Off
bcmath.scale = 0

Here are the relevant lines in wp-config:
define('WP_MEMORY_LIMIT', '256M');
define( 'WP_MAX_MEMORY_LIMIT', '256M' );

and in .htaccess:
php_value memory_limit 256M

what else could it be?

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hi there, PhilMay!

    What's the physical RAM on your server? And do you know what kind of RAM it is? I can see here that they offer several packages with varying amounts of physical RAM, is your plan one of these? https://www.godaddy.com/pro/managed-vps

    There's a chance, if you're on the 2GB or 3GB RAM VPS, you may not have the hardware to support this upload, even if you've got all the software properly configured.

    Have you taken a peek at your server error logs for the time of one of these upload failures to see if there are any clues there?

    Thanks!

  • PhilMay
    • Site Builder, Child of Zeus

    the file uploads, but when i select the fields for each column of the csv file and actually perform the import of data is when the error occurs. also there is only 1000 rows with 10 columns of data per row. I split a csv that has 268k rows into 268 smaller files.

    I would really love to not need to break it down any further, it is not even one of the larger database, and there are 50 databases total i need to import. is it somehow possible to handle the import directly through mysql or phpmyadmin?

  • PhilMay
    • Site Builder, Child of Zeus

    I guess I am confused about multi-db. Installing 4096 databases seems to just have used up all my ram on the vps.

    I see everywhere that people say its good for sites/blogs. What about a single multi site install that has an enormous database though? Is 4096 overkill to handle 20+million posts with or without images?

    I guess I am just looking to use multi-db to enhance performance when all that data is uploaded.

    here are 2 rows of the data so you see that it is not even alot of data per "post"

    COMPANY NAME	MAILING ADDRESS	CITY	STATE	ZIP CODE	Location ID	PHONE NUMBER	WEB ADDRESS	Category	Sub Category	Tags	level id
    1-Stop Package	458 N Main St	Palmer	MA	1069	29	4132832022		Liquors-Retail	Liquors-Retail		1

    I guess the difference between my install, and say edublogs, is that we will not have individual sites/blogs for each user. Rather it is just a lot of overall data.

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hi there, PhilMay!

    Ultimately, MultiDB, and especially the option to use 4096 databases for a network that's not hosting millions of sites on the same network is usually overkill, by quite a lot.

    MultiDB is great for gigantic networks, but there are usually simpler, and ultimately better ways to improve you site performance that I would typically recommend before MultiDB, at least if the end goal is not to host millions of sites.

    I hope this helps!

  • Michelle Shull
    • DEV MAN’s Apprentice

    Hey there PhilMay!

    I've been thinking a bit about your particular case, and I wondered if you'd done any research on Varnish as a tool for optimizing your site? I've got some links for you about how it works.

    http://code.tutsplus.com/tutorials/optimizing-wordpress-with-varnish-and-w3-total-cache--cms-21136

    http://systemsarchitect.net/boost-wordpress-performance-with-varnish-cache/

    http://z9.io/2011/08/09/speed-up-wordpress-with-apache-and-varnish/

    These should give you an overview of how it works, I think this may give you the performance boost you need without having to use MultiDB.

    Let me know if you'd like more info on this, or if this isn't an option, and we'll keep looking.

  • PhilMay
    • Site Builder, Child of Zeus

    l actually do not have multi-db installed anymore. I think the vps in question is juist underpowered. Ive purchased a Dell Poweredge 2950 Gen III server with 2x 3ghz quad core xeons, 32GB ram, 256mb raid cache on card, and 6x 1tb SAS drives. that I am debating running in raid 60 or raid 10. Hopefully this hardware can handle at least one state worth of listings to launch this directory. I am not going to be multisiting as I can not see how multisite would actually be a benefit, therefore multi-db is useless.

  • Tyler Postle
    • CGO

    Hey Phil,

    Thanks for getting back to us!

    Are you able to upload any size CSV? Even a small one? Is it only failing on large files? Just wanting to confirm that it really is the size of the file and not something else that is coming into play here!

    Look forward to hearing back!

    All the best,
    Tyler

  • PhilMay
    • Site Builder, Child of Zeus

    Yes i was able to upload csv files. After playing with some settings in apache and APC I was able to get everything to work, the only thing that timed out was when i used the importer to geocode via the api, rather than upload pre geocoded. This being said, it is only google maps api at this moment that is timing out on me (or at least apache is not waiting long enough for maps api to respond). Either way, as stated above, I have reinstalled the site without multisite and without multi-db.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.