snapshot not importing properly

I am creating a snapshot backup in our old server location and installing it in a new server location with a different URL.

When the snapshot restores, it is creating "double layers" of directories. As a result, it cannot find files.

For example, the link to the file it is creating in the import is http://customer.server.com/wp-content/uploads/sites/5/sites/5/2013/02/Lorax.jpg

Do you see where it is repeating the /sites/5 twice?

This is happening on all images imported into a site and it is a mess. How do we fix this?

Thanks a bunch!

  • johnol

    The snapshot is being saved to Amazon S3 by server 1.

    We are making the file public and providing the URL for Snapshot to "import" the file into server 2.

    I would also point out that while the Media Gallery has this odd reflection of /sites/5/sites/5/ when I perform a search regex in the site data itself, all of the "html" properly points to just /sites/5/.

    We have to do it this way because of how security is set on the new server. We do not allow direct upload of files via http://FTP...

    Also, in the course of a few trial imports we've seen different experiences in the directory structure being created. In a prior import, it create /sites/2/sites/2/ and stored some files in that deep directory. We didn't even catch the issue at first. With this go-round, it did not create the /sites/5/sites/5/ sub-directory - just the top level /sites/5/.

  • Paul

    @johnol,

    I am creating a snapshot backup in our old server location and installing it in a new server location with a different URL.

    How old was the WP system of the original backup?

    When you went through the restore processing on the last screen you should have seen the split part where on the left it shows the details from the backup like site URL, database prefix, etc and on the right it shows what the new local values would be. Can you go back through the restore process and capture that section and upload the image so I can review.

    I would also point out that while the Media Gallery has this odd reflection of /sites/5/sites/5/ when I perform a search regex in the site data itself, all of the "html" properly points to just /sites/5/.

    I'm not sure what you are trying to communicate other than again the site prefix is repeated.

    We have to do it this way because of how security is set on the new server. We do not allow direct upload of files via http://FTP...

    Again, I'm not sure what you are attempting to communicate. Are you trying some security scheme? Are the files to be moved to some non-standard directory structure?

    Any chance you can make the archive on AWS available so I can test on my own server environment? Do not post the URL here as this is a public forum.

  • johnol

    Sorry if I'm not being clear.

    Most of my responses were intended to answer previous questions by Ashok.

    -------------

    1. Both systems are running same version of WP with same version of all plugins. There is complete parity from a version perspective. That being stated, the source installation is several years old versus the new which is a clean install.

    The older version of WP has a different mapping of files:
    /wp-content/blogs.dir/blog#/files

    The newer version of WP uses:
    /wp-content/uploads/sites/blog#

    One comment: I noticed that in the imported site's Settings, it is referencing the full URL which I've not seen before - in other words, the File Upload URL setting is being shown as http://subdomain.server.com/wp-content/uploads/sites/blog#

    Not sure if that has an impact but it is a difference.

    -------------

    2. I'm trying to clarify that when I look at the media via the Library, it is repeating the site prefix. However, when I'm looking at the "content" all of the media is properly referenced without the site prefix being repeated. I was hoping this might be helpful in your analysis of where things are potentially going wrong.

    -------------

    3. Ashok asked specifically if we can download a direct copy of the snapshot and then upload that directly to the new server. I was saying we are using Amazon S3 as our backup store because our new server does not allow FTP of files and there's no way to "upload" a backup snapshot via the snapshot import function... it expects the file is already somewhere on the server. Nothing is "non standard" from a directory structure perspective. We just don't support direct FTP to the server.

    I would be happy to give you the opportunity to try the archive. You didn't specify where I could send the link to the file...

  • johnol

    UPDATE:

    As requested, I re-ran the restore from the snapshot and here's some info:

    Information from Archive
    Blog ID: 39
    Site URL: http://subdomain.oldserver.com
    Database Name: abc
    Database Base Prefix: wp_
    Database Prefix: wp_oldblog#_
    Upload Path: wp-content/blogs.dir/oldblog#/files

    Will be restored to
    Blog ID: 5
    Site URL: Insurance Brokers (subdomain.newserver.com)
    Database Name: 123
    Database Base Prefix: srvr_
    Database Prefix: srvr_newblog#_
    Upload Path: wp-content/uploads/sites/newblog#/sites/newblog#

    So basically, the insertion of the odd upload path is happening right at this stage.

    When I verify the information in the site settings for the new site we see:

    Fileupload Url: http://subdomain.newserver.com/wp-content/uploads/sites/newblog#

    Upload Path: wp-content/uploads/sites/newblog#

  • johnol

    Sorry for all the notes - there doesn't appear to be a way to edit what I've already posted.

    I want to add that I created a clean "test" site and started a new restore from the source.

    As it turns out, the initial restore shows a "proper" pathway for the restore. The incorrect pathway shows up when I go to re-restore the snapshot

    So, on a hunch, I ran another restore on the same test site.

    It is now showing the following for the upload path:

    Upload Path: wp-content/uploads/sites/blog#/sites/blog#/sites/blog#

    Basically each time the restore process is run, it changes the upload path and adds a new /sites/blog# layer.

    Hopefully I've provided enough information for you to chase down the bug in the code.

    Thanks!

    Just wanted you to know...

  • Paul

    @johnol,

    I would be happy to give you the opportunity to try the archive. You didn't specify where I could send the link to the file...

    Please use the contact us link https://premium.wpmudev.org/contact/ on the drop down select 'I have a different question'. Then in the body reference this thread URL and my name. Include a link to the public AWS archive. I'll be able to download and test the import/restore on my local system.

    Hopefully I've provided enough information for you to chase down the bug in the code.

    Yes. You have.Thanks. There is some 'funky' code going on in order for Snapshot to attempt to handle the various paths which can be used by WP Multisite. Pre-3.5 WPMU there was one version of the URL used when you add an image to your post. After 3.5 there was another. So somewhere in that logic is the issue. Hopefully once I get the archive I'll be able to do some quick changes to the plugin code. Thanks.

  • Paul

    @johnol,

    Just wanted to reply to some of your previous remarks/questions before I get to the final issue.

    One comment: I noticed that in the imported site's Settings, it is referencing the full URL which I've not seen before - in other words, the File Upload URL setting is being shown as http://subdomain.server.com/wp-content/uploads/sites/blog#

    Not sure if that has an impact but it is a difference.

    As of WordPress 3.5 this was the change on new installs. Sites which were installed prior to 3.5 kept their legacy URL format. So images linked within posts for example would show a shorter url which involved some rewrite rules. All this was removed in WP 3.5

    2. I'm trying to clarify that when I look at the media via the Library, it is repeating the site prefix. However, when I'm looking at the "content" all of the media is properly referenced without the site prefix being repeated. I was hoping this might be helpful in your analysis of where things are potentially going wrong.

    It is. Thanks. Keep reading.

    So today I've been tracing though the WP code to try and figure out the issue. I mentioned in one of my replies there is some 'funky' logic within Snapshot to try and handle the pre-3.5 attachment path and convert them to the post-3.5 format which is the uploads/sites/#ID format.

    As part of the snapshot processing it calls a function wp_upload_dir() which will return information about the URL and PATH of uploads for the given site. As part of this function it reads the entry for 'upload_path' from the options database table. In new installs of WP this option is blank. But some older installs, even if they were upgraded , kept this value.

    This is what is happening on your site. So try this. On the destination system. Log into wp-admin and go to the Network > Sites. Then edit the site you restored the snapshot to. You will see the tabs at the top. Go to the Settings tab. Search for 'Upload Path'. I'm hoping you will see there is a non-blank value for this field. For testing copy the value and save it somewhere. Then blank it out and save the settings.

    Now go to the site dashboard where you restored the snapshot archive to. Go to Media. Are your images now showing?

  • johnol

    BRILLIANT. Here's the issue. When I created the site to receive the snapshot import in the new location, the Upload Path was empty in the Settings area. After the snapshot import, it was populated. Furthermore, after each and every snapshot import the path was updated to repeat the /sites/#ID.

    By emptying the Upload Path field after the snapshot import was completed, it does seem to find the files.

    However, the Snapshot restore is still creating a structure within the web site public folders that also has this /sites/#ID/sites/#ID/sites/#ID structure. So do I just administratively remove those unneeded folders?

    Is there any way to fix this so I don't have to run through all of these added manual steps after a restoration?

  • Paul

    However, the Snapshot restore is still creating a structure within the web site public folders that also has this /sites/#ID/sites/#ID/sites/#ID structure. So do I just administratively remove those unneeded folders?

    I need specific clarification on this. When I ran the restore on my local Multisite sub-site it set the paths per the Media panel with the double sites/ID#/sites/ID# but not the physical path on the server. This is actually the issue as far as I could tell the URL path which is calculated by WordPress per the Media page did not match the physical location of the files on the server. In my case the files were properly written to /wp-content/uploads/sites/6/2013/etc not /wp-content/uploads/sites/6/sites/6/2013/ etc.

    But please confirm the file are physically in the double path.

    Is there any way to fix this so I don't have to run through all of these added manual steps after a restoration?

    That is what I'm attempting to figure out if there is something in Snapshot that I can do to remove the 'upload_path' setting. But there are many variables to check so the logic gets somewhat twisted. Mainly I need to determine the configuration of the restored Multisite system.

  • johnol

    I don't think you are understanding what I'm saying. The plugin is physically adding a /sites/ID# directory on my physical hosting space. When I re-run the import, I can cd into the /sites/ID# directory and find a new /sites/ID# directory within it. When I re-run the import again, a new /sites/ID# directory is created in the server, creating a nesting effect.

    So not only is Snapshot adding this structure to the Upload Path, it is physically creating it within the server. Yet the files are stored at the root of the given site where I would expect it.

    So I'm asking if I can remove these unwanted directories and if there's a way to fix Snapshot so I don't have to manually update the Upload Path field as well as removing the unwanted directories it is creating in the restore process?

  • johnol

    I would add that perhaps I'm not understand what you are saying - just to be fair.

    The bottom line is that two things are happening:

    1) Snapshot is adding in the Upload Path. When I clear this added value, the gallery can see the images properly.

    2) The files that are being imported are being brought in properly. However snapshot is *also* creating a separate directory structure to match the Upload Path it is identifying. So it gets rather messy.

    I'm acknowledging I can manually fix this but with several hundred sites to deal with, this is going to be a royal pain so I was hoping it could be addressed in the programming.

    Thanks!

  • Paul

    @johnol,

    So I'm asking if I can remove these unwanted directories and if there's a way to fix Snapshot so I don't have to manually update the Upload Path field as well as removing the unwanted directories it is creating in the restore process?

    1) Snapshot is adding in the Upload Path. When I clear this added value, the gallery can see the images properly.

    From your perspective maybe. But technically this is WordPress. Snapshot just calls the WP function to determined the Path and URL for media files for a given site. It is WordPress which is providing the incorrect information. But will try and teach Snapshot to be more intelligent about removing the double sites/#ID sets.

  • Paul

    @johnol,

    Here is a link to a beta version I worked up last evening. The logic was adjusted to check the value of the 'upload_path' option before the restore starts. After the restore has finished the original value of the 'upload_path' option is then reapplied to the option in the database table.

    https://dl.dropboxusercontent.com/u/2616987/WPMUDev/snapshot/beta/snapshot-2.4.1.1-beta4.zip

    As you have already run prior snapshot restores you need to first manually clear out the 'upload_path' option in your destination database before running the beta. This is only needed the one time.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.