Snap MuLtiDB 4096 + Home + Global

Hey guys

Please let me know if any one has trying to use this on a BIG Setup

100GB Data Set

I have tried it on BlogLines.co.za and no luck.

I need to know what settings to add to PHP.INI To run it

Thx
Mark

  • Mustafa
    • Syntax Hero

    Please let me know if any one has trying to use this on a BIG Setup

    100GB Data Set

    Hey dude, Are you kidding me :slight_smile:

    Trust me, you should use bash script for backup.

    Best scenario for you: Use a bash script and add cron job. Then mount your backup dir to amazon s3 :wink: You can check s3fs for it.

    Cheers,

  • Paul
    • The Green Incsub

    @Mark de Scande, Not sure I follow your question and how it relates to Snapshot. Using the MultiDB plugin with the 4096 setup is your option. The Snapshot plugin does not backup the entire database in one step. Within you 4096 setup you would still only use Snapshot to backup one site at a time.

  • Mark de Scande
    • Syntax Hero

    @Paul I dont want to back up all the DB one shot WHM CPanel Backup covers that.

    I would have like to use the plugin on a site by site basis but this don't seem to work well on a 100GB db set

    Let me know if you available on Skype then i will have a chat with you.

    My Sype paintballcraze Mark de Scande

    Thx

  • Paul
    • The Green Incsub

    @Mark de Scande, Sorry, No Skype for my this morning. Can you just explain more on your needs? Keeping it here on the thread also provides historical information for others who want to use big setups.

    I would have like to use the plugin on a site by site basis but this don't seem to work well on a 100GB db set

    Again, not sure I follow what you have written here. Do you have a single blog in your Multisite which has 100G database? When you state it does not work well. What have you tried? What errors of anything have you experienced?

  • Mark de Scande
    • Syntax Hero

    Setup

    4096 + Home + Global MultiDB Setup Total size 100GB

    Here are the steps

    1) Made sure there is no err logs on the server

    2) http://bloglines.co.za/wp-admin/network/admin.php?page=wpmudev-plugins look for Snapshot click install i have not network activate it

    3) Go to main site http://bloglines.co.za/wp-admin/plugins.php plugin is not listed

    4) Back to Network http://bloglines.co.za/wp-admin/network/plugins.php
    Snapshot Network Activate

    5) I see Snapshot in the left pane

    6) Had a look if any errs server is clean

    7) New Dir /home/blogline/public_html/wp-content/uploads/snapshots

    8) Click Network -> Snapshot -> Settings
    http://bloglines.co.za/wp-admin/network/admin.php?page=snapshots_settings_panel

    Takes for ever to load Still waiting

    Side note

    PHP Settings
    memory_limit 98
    max_execution_time 140

    Page came up i have added a screen shot
    http://www.webpagescreenshot.info/img/858371-17201332023PM

    It looks like i am missing setting

    Thx
    Mark de Scande

  • Mark de Scande
    • Syntax Hero
    [07-Jan-2013 13:10:51 UTC] PHP Warning:  fopen(/index.php) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 957
    
    [07-Jan-2013 13:39:29 UTC] PHP Warning:  fopen(/index.php) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 957
    [07-Jan-2013 13:39:29 UTC] PHP Warning:  fopen(/.htaccess) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 964
  • Paul
    • The Green Incsub

    allow_url_fopen has been set to on

    Snapshot does not open URLs. Only local files and database tables.

    php_value max_execution_time 800
    php_value max_input_time 800
    php_value memory_limit 512M

    Funny thing about WordPress regardless of how much memory you allow in your php.ini it still limits the memory usage to 30Mb for Single WordPress and 64M in Multisite. So you will need to increase the memory allowed by WordPress via the wp-config.php http://codex.wordpress.org/Editing_wp-config.php#Increasing_memory_allocated_to_PHP

  • Paul
    • The Green Incsub

    On the fopen errors these are somewhat strange. In the lines of code referenced they are from one of the setup functions within Snapshot. The specific function's job is to secure the folders within the Snapshot archive directory tree.

    Normally in a default Snapshot installation the archives are stored into the folder /wp-content/uploads/snapshots/ Beneath this folder there are many other 'work' folders used. and so this utility function will create a blank index.php file into these 'work' folders.

    But what is curious on the errors is the path of the file. Note the first line would indicate it is trying to open a file in the server root which obviously it does not have permissions. If you have for some reason changed the default snapshot archive folder please change this back until we can get thing working.

    [07-Jan-2013 13:10:51 UTC] PHP Warning: fopen(/index.php)

  • Mark de Scande
    • Syntax Hero

    Just FYI

    I have set these for testing

    define('WP_MEMORY_LIMIT', '198M'); i have change that to 1198M for testing

    php_value max_execution_time 800
    php_value max_input_time 800
    php_value memory_limit 512M

    I have not change /wp-content/uploads/snapshots/

    1) Activated plug Network wide

    2) Settings Page takes for ever to load well we have giving it way more ram then it needs :slight_smile: Still wating

    The Log

    [07-Jan-2013 15:51:11 UTC] PHP Warning:  fopen(/index.php) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 957
    [07-Jan-2013 15:51:11 UTC] PHP Warning:  fopen(/.htaccess) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 964
    [07-Jan-2013 15:51:12 UTC] PHP Warning:  fopen(/index.php) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 957
    [07-Jan-2013 15:51:12 UTC] PHP Warning:  fopen(/.htaccess) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 964
    [07-Jan-2013 15:51:12 UTC] PHP Warning:  fopen(/index.php) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 957
    [07-Jan-2013 15:51:12 UTC] PHP Warning:  fopen(/.htaccess) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: Permission denied in /home/blogline/public_html/wp-content/plugins/snapshot/lib/snapshot_utilities.php on line 964

    My Server is a open book if you would like to log in just get me on Google Chat Mark de Scande paintballcrazecc@gmail.com or let me know what other tests i can do for you.

  • Paul
    • The Green Incsub

    At this point I'm not sure. I don't guess you tried Snapshot prior to setting up the MultiDB. I can; think what would cause the Settings page to take more than a few seconds to load. There is not much dynamics to that page. But I'm assuming it does eventually load, yes? Can you provide a screenshot of the settings page?

    Also in your wp-config.php can you enable WP_DEBUG?

    Is this possibly a server I can log into? email me at paul [at] incsub.com

  • Paul
    • The Green Incsub

    @Mark de Scande, So you have uncovered something that I've had in the back of my mind about Snapshot. Once big issue with the current code is it doesn't work well for very large Multisite installations. This is more a scaling issue within WordPress functions when you want to get a list of all sites.

    Give me 24 hours. There are some places in the code I want to patch this afternoon to possibly address the bathing of the site information. There are two places. One is on the Snapshot > Settings page where it tries to determined how many sites. This is an easy fix. The second issue is somewhat harder on the Add New Snapshot page. On that page is a dropdown where the admin would normally select the blog to backup. In your case since there are 110K blogs this dropdown will basically prevent the page from loading. So I want to change this to in input field where the admin entered the blog domain or blog number as a lookup field. So give me until the morning to get this code reworked. Then we can start testing.

  • Mark de Scande
    • Syntax Hero

    @Paul your the man Sorry for giving you so much work

    All i was thinking of was to get a snap shot of a "blog" not the hole system but the way you will be doing this will surly help out BIG time

    Thank you again.

    Ps The server is your play ground, Have some Fun

    Pss yes i know it is is huge Multi Site Setup :slight_smile:

    Side Note
    There is one piece of custom coding that @Mustafa for the upload dir..
    /home/blogline/public_html/wp-includes/ms-default-constants.php I just wish hi will on eday make it in to a file that can go in to the mu-plugins so i dont have to every time hack the core files :slight_smile:

    function ms_upload_constants(  ) {
    	global $wpdb;
    
    	if ( !defined( 'UPLOADBLOGSDIR' ) )
    		define( 'UPLOADBLOGSDIR', 'wp-content/blogs.dir' );
    
    	if( $wpdb->blogid <81925) {
    //first don't change default upload pacth
    		define( 'UPLOADS', UPLOADBLOGSDIR . "/{$wpdb->blogid}/files/" );
    		if ( 'wp-content/blogs.dir' == UPLOADBLOGSDIR )
    			define( 'BLOGUPLOADDIR', WP_CONTENT_DIR . "/blogs.dir/{$wpdb->blogid}/files/" );
    	}
    	else if ( ($wpdb->blogid > 81925) && ($wpdb->blogid < 113000) ) {
    //change path for new uploads just deal with ext3 32k limitation
    		define( 'UPLOADS', UPLOADBLOGSDIR . "/assets/00/{$wpdb->blogid}/files/" );
    		if ( 'wp-content/blogs.dir' == UPLOADBLOGSDIR )
    			define( 'BLOGUPLOADDIR', WP_CONTENT_DIR . "/blogs.dir/assets/00/{$wpdb->blogid}/files/" );
    	}
    	else if ( ($wpdb->blogid > 81925) && ($wpdb->blogid < 145000) ) {
    //sector 2
    		define( 'UPLOADS', UPLOADBLOGSDIR . "/assets/01/{$wpdb->blogid}/files/" );
    		if ( 'wp-content/blogs.dir' == UPLOADBLOGSDIR )
    			define( 'BLOGUPLOADDIR', WP_CONTENT_DIR . "/blogs.dir/assets/01/{$wpdb->blogid}/files/" );
    	}
    }
  • Paul
    • The Green Incsub

    @Mark de Scande, No, Thank You! As I mentioned this has been on my mind for a while knowing I will need to build a solution for it. So now I have that opportunity. No problem on the work. I love to write code all day!

    I'll take a look at the custom code segment you provided and see if I can make this a little easier to manage.

  • Paul
    • The Green Incsub

    @Mark de Scande,

    Ok. I've pushed up a beta version of snapshot which has the lookup functionality for selecting a blog to snapshot. And the Settings page now loads as expected.

    I've setup a snapshot archive on a random blog. Seemed to work. I've tried doing a full table backup on the primary site and am getting HTTP 500 error but no details on the exact error. About to drop off for the evening. Will pick tis up in the morning.

  • Mark de Scande
    • Syntax Hero

    @Paul I will leave this up to you :slight_smile:

    Super Thank you :slight_smile:

    Ps Just thinking "Now just to get a setting to run trough all the blogs and send back ups to Drop Box mmm or even push the /backup/cpbackup/... to dropbox

    It will happen soon i know it :slight_smile:

  • Mark de Scande
    • Syntax Hero

    @Paul My Wife asks :slight_smile:

    1) She has SuperBlogs.co.za Multi DB about 100 blogs +-

    2) She did a snap of one site and it was sent to here DropBox

    3) Logged in to drop box and there was a new file with a funky name

    So the question is: Is there no way to give the file name the name of the site that was backed up in the first place

    snapshot-1357589295-130107-200815-bed4c9ea.zip to extremebootcamp.co.za

    I know this sounds easy to say but not easy to implement.

    Thank you for still working on BlogLines.co.za

  • Paul
    • The Green Incsub

    @Mark de Scande, At this time no there is no way to have the destination file be some custom name. But you can easily setup the snapshot item to go into a custom directory on the destination.

    There are two parts. First on the Destination itself (Snapshot > Destinations) you setup the link to Dropbox or wherever. As part of this setup you can define a destination path. Realize the base bath for all Snapshot files on your Dropbox will be '/Apps/WPMU DEV Snapshot ' so the destination path you add will just append to the end of the base path. This is handy is for example you manage many multiple servers.

    Second part is at the snapshot item itself. when you setup a snapshot item to use a destination there are some options you can user to override or supplement the destination configuration. Refer to the attached image for the Destination box. Via some tokens you can supplement the destination path. For example you can append the domain of the snapshot item to the destination path by setting it to '[DEST_PATH]/[SITE_DOMAIN]' Hope this helps.

    Will probably come up with some way to manage the filenames. There are some concerns that some users may chose a filename which is not unique enough thus causing issues with file being lost. But we will eventually address that.

  • Mark de Scande
    • Syntax Hero

    @Paul Just FYI i was picking this up on some random blogs on the system

    ERROR: Cannot create snapshot folder. Check that the parent folder is writable /home/blogline/public_html/wp-content/blogs.dir/81589/files/snapshotsERROR: Cannot create snapshot Log folder. Check that the parent folder is writeable /_backup

  • aecnu
    • WP Unicorn

    Greetings Mark de Scande,

    Just checking if this issue was eventually resolved in another thread? Or by yourself separately to us? Or by us over email with you? Or using our live support?

    If so, no need to reply, that's great news.

    If not, or you have any more questions related to this thread, please feel free to post them below including any new symptoms or errors and tick the 'Mark as Not Resolved (re-open)' box below the post area (or else we'll miss it!)

    Otherwise, happy days, glad you got it sorted :slight_smile:

    Thank you for being a WPMU DEV Community Member!

    Cheers, Joe

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.