Batch Create Plugin Automation

I have a cron job exporting members from an Expression Engine CMS hourly. The CSV exported works perfectly for manually importing using the batch create plugin.

What I'm trying to do next however is automate the process by somehow passing this hourly exported file to the batch create plugin script to have it import but I'm having trouble.

Wondering if the plugin creator may have some insight on doing this? I've tried several angles but would like to get your thoughts if possible.

Thanks!

  • Ignacio
    • HummingBird

    Hi @dub3325.

    There's not an easy way to do this though I'm thinking that a little API for this plugin would be great so I'll try to reorganize code in next version in order to make it possible but I cannot give you an ETA.

    Anyway, the function that creates blogs/users is inside class Incsub_Batch_Create_Creator and is called process_ajax_queue(). Right now, the best way is to do your own plugin adding that function and adapting it to your file/code.

    For instance, the sentence $queue_item = $model->get_queue_item(); gets an object that saves all the data to be inserted, you'd need to create your own function for that and instead of getting the the item from database you could get it from your file. Then you'd need to get rid of those constants that belong to the Batch Create plugin.

    Is not such easy and will need some work, but I think it would be possible.

    Regards.
    Ignacio.

  • PC
    • WPMU DEV Initiate

    Thanks for the guidance and considering something like this for future.

    Moving it to "Features and Feedback" so that others looking can vote too.

    dub3325 : You are most certainly welcome :slight_smile: We will be around to hear your results !

    Cheers, PC

  • dub3325
    • Flash Drive

    Hey guys,

    I have an update and also a question if you can help me out as I hopefully wrap this functionality up here shortly!

    I was able to pass in a file path to a revised "process_file_auto()" function I created within the Batch Create plugin with just one minor modification to your original plugin and it lets me externally insert a full queue. So that's working great!

    Then my next step currently is executing do_action('wp_ajax_process_queue');

    And it works great! Except it only processes two entries at a time. If I run this function multiple times it appears to work but doesn't seem like the best way to do it.

    Any thoughts? Please let me know if I can provide any more insight.

    Thanks again!

  • dub3325
    • Flash Drive

    Actually guys I think I got it. Just added a while statement in there to detect the total rows of the file and execute do_action('wp_ajax_process_queue') until the total rows are reached! I think we're in good shape.

    However, it's not working exactly as expected. I tried another CSV file with 164 rows for example instead of 3 rows and I'm having an issue where it's only processing 56 of the 164 entries. Here's my statement at the end of my process_file_auto() function for your reference in case you can offer some insight; I'd greatly appreciate it.

    // Process continually until total rows is reached
    $totalRows = count($tmp_new_blogs);
    $this->log( __( '--- Starting Automated Batch Create Queue Processing ---', INCSUB_BATCH_CREATE_LANG_DOMAIN ) );
    $this->log( __( '--- TOTAL ROWS = ' . $totalRows, INCSUB_BATCH_CREATE_LANG_DOMAIN ) );
    while ($i <= $totalRows) {
        $i++;
        do_action('wp_ajax_process_queue');
    }
    
    return true;
    }
  • dub3325
    • Flash Drive

    @Ignacio,

    Thanks again for taking a look and providing some insight. Attached is my new plugin work in progress called "batch-create-automation" that interacts with your batch-create plugin. To get a good idea of how it works it probably would be best to install the plugin, activate it and definitely view the options page for more information before you dive in.

    Aside from that, the only file I modified so far within your batch-create plugin is /inc/creator.php. There are a few extraneous lines of code in this version attached that I was testing out but the only modification that is required at this point is the function process_file_auto().

    To recap the issue I'm encountering above, I'm able to successfully get the contents of a CSV file into the queue, but it's the processing that isn't working fully. It's processing only a couple at a time or timing out possibly at a certain point. Need to figure out the way to do this.

    Please let me know if you have any questions. Thank you kindly for your assistance!

    **EDIT**

    Had issues uploading anything other than a screenshot image file. Download the source at the link below please and let me know when I can remove it from my server. Thanks!

    http://www.heyviet.com/batch-create-automation.zip

  • Ignacio
    • HummingBird

    Hi @dub3325.

    I think I have something. Attached to this post you'll see your plugin and a new version of Batch Create. I've changed some little things in your plugin so it should work a little better.

    Here's the thing: You enter the file (and path) in your plugin, save it (you were saving it in wp_options table but as it is a network plugin now it is saved in sitemeta table).

    If you use the ajax action, Batch Create processes only one item. That's why there's a progress bar when you upload a file and process a queue in Batch Create. This is made this way to prevent timeouts so with your plugin you should be careful with that, do not add too many blogs at the same time.

    Could you test it and tell my how was it?

    Regards.
    Ignacio.

  • dub3325
    • Flash Drive

    @Ignacio,

    Your modifications appear to have made things work exactly as we would like! Thank you for taking the time to review my code and modify to integrate best with the main plugin. I'm sure there is room for improvement within this patch of a plugin I wrote, but it will definitely do for now.

    Thanks again for your help!

  • PC
    • WPMU DEV Initiate

    Thanks for posting back and letting us know it worked @dub3325

    Glad to help :slight_smile:

    Thanks @Ignacio for the awesome help !

    Sorry for the delay in acknolwedgment as I have been away due to bad health.

    Cheers, PC

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.