Using MultiDB using manual setup

Hi,..
We are test to using multidb rightnow.
The strategy is to shard more than 72000 blogs. 10000 blog IDs/one database. For 72000 IDs, so we have 9 databases.
1 global databases. 8 blogs databases.

We're using RDS from amazon. 1 multiaz for global + blog ID's 1-10000.
2 RDS for the others. Let say we separated the others blogs in a half each other.

I prefer dont use move-blog.php for the migration process. Because the data have been separated like strategy above. So, i need to set up and db-config the database to match our needs.

I have try to manually put the db-config like this:

define ('DB_SCALING', '16');
add_dc_ip('10.138.178.1xx', 'dc1');
add_global_table('dgdgdg');

//	Note: you can also place this section in a file called db-list.php in wp-content
add_db_server('global', 'dc1', 1, 1,'dgdgdg.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg', 'dbdgdgdg',  '#');
add_db_server('0', 'dc1', 1, 1,'dgdgdg.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg_1', 'dbdgdgdg',  '#');
add_db_server('1', 'dc1', 1, 1,'dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg2_2', 'dbdgdgdg',  '#');
add_db_server('2', 'dc1', 1, 1,'dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg2_3', 'dbdgdgdg',  '#');
add_db_server('3', 'dc1', 1, 1,'dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg2_4', 'dbdgdgdg',  '#');
add_db_server('4', 'dc1', 1, 1,'dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg2.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg2_5', 'dbdgdgdg',  '#');
add_db_server('5', 'dc1', 1, 1,'dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg3_6', 'dbdgdgdg',  '#');
add_db_server('6', 'dc1', 1, 1,'dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg3_7', 'dbdgdgdg',  '#');
add_db_server('7', 'dc1', 1, 1,'dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com','dgdgdg3.cmpuwbp11vb.ap-southeast-1.rds.amazonaws.com', 'dgdgdg3_8', 'dbdgdgdg',  '#');

The setting above give me error, db-config coludn't find the global database.
Let me know if i do some wrong there.

**Some info striped and change in the codes, for security reason.

  • aecnu
    • WP Unicorn

    Greetings leksa,

    This is way beyond my familiar knowledge of this plugin while using it outside of the normal parameters and ways for this plugin.

    How is it going to react to the one statement define ('DB_SCALING', '16'); you have above?

    When the obvious scaling is 8?

    Rather then get into the fray here and asking questions beyond my understanding on how and if this will work at all, I am going to see if I can get the lead developer in here for advice/advise on the possible issues we are facing trying to make this work as it was not crafted to do.

    Though this may take a bit more time than a normal ticket, I will try to get him in here asap.

    Cheers, Joe

  • leksa
    • WPMU DEV Initiate

    Thanks Joe,..
    DB scaling in WPDB was defined in 3 options. And there is one point to ask about.

    I'm here trying here to get some clue also, tracking and debugging the code. If i found any directions i will report here.
    But really hope you guys are the expert, that can give me some clues.

    Again, thanks

  • aecnu
    • WP Unicorn

    Greetings leksa,

    Thank you for your confidence and I am sure there is an answer for this layout, but I do not know enough about the interactivity of the code and therefore I am not taking the chance of a guess.

    The wrong answer could potentially mess up the databases, therefore I have pinged the lead developer to see about obtaining his invaluable advice to this situation. Hopefully we will get a comment sooner then later.

    Cheers, Joe

  • Barry
    • DEV MAN’s Mascot

    The number of databases *has* to be 16, 256 or 4096, plus one global database. The blogs tables are spread amongst the databases based on a hash of the id of the blog.

    It is done this way because it provides an easy and more importantly quick method of finding the correct database to use without havin to use any type of lookup or extra database queries.

    If you wanted to use the strategy you highlighted then you would need to code a custom db library to handle it, as ours is heavily reliant on the method above.

  • leksa
    • WPMU DEV Initiate

    Hi Baryy,..
    I'm setting up for 16 because the option.
    What I'm thinking is the database could be growing. I could divide 72000 blogs to 16 databases. But how about if our community blog become 80001 blogs?

    I dont know if it's possible to setup the empty databases for 9-16. So, we're stick with 10000blogs per databases.

    Thank you for the reply

  • aecnu
    • WP Unicorn

    Greetings leksa,

    Thank you for your feedback and letting us know your concerns, it is appreciated.

    This kind of feedback helps us stay in touch with our members needs and challenges which in many cases shapes our plugin solutions.

    Thank you for being a WPMU Dev Member!

    Cheers, Joe

  • Barry
    • DEV MAN’s Mascot

    Hi, the splitting is done mathamatically, so the tables are spread around, so there isn't a concept of having empty databases 9-16. Move-blogs aims to do this for you for the initial move, but the system doesn't place blogs 1-x in db 1 and then move to the next, if that's the impression you have.

    If you don't think 16 db's will be enough then the only option really is to start with 256 or 4096. Whilst starting is relaitevly easy, switching to a higher number of db's at a later date is extremely complex.

  • leksa
    • WPMU DEV Initiate

    *please correct me if i'm wrong.

    So,
    first (1) i have to collect and join all separated databases (in 8 databases + 3 rds server) into one databases. The last time our backup running we have 280K - 300K tables.
    2. after that we run the process like the installation instruction. using move-blog.php. And we will have 16 (or 256) databases in one RDS.
    3. And voila, hope it would be work.

    It has lots of realibility question here. First about how many mysql could handle tables in one databases.
    Then second question, how long the move-blog.php could handle the process,meanwhile i'm worry about the fail execution and timeout.

    If you guys have another direction about how to handle this big process, please let me know.

    Thanks you Barry - Joe

  • Barry
    • DEV MAN’s Mascot

    As, I wrote above. Moving from split db's to a different number / config is extremely complex and can take days of prep and work. Last one I did took 2 days solid. If you already have your db's split, then do you already have a system that works?

    Move-blogs is really for the person on the first step from one db.

  • leksa
    • WPMU DEV Initiate

    Yes, we have working with high tweak DB in server level before -but it was one databases :slight_smile:
    Yes, the engineer worked good at this one on the past. But lots of realibility issues comes now. We have to move on :smiley:

    Well, its clear, looks like i need to take a time for deep analyze on our issue now.

    Thank you for the nice discussion and directions here, Barry. Also Joe there. :slight_smile:

    Greeting from Indonesia

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.