Hello,
I am also trying to plan our scaling of our site.
Do you have all your databases on one machine?
Or do you split them across a couple database servers?
Amazon S3 it tempting for the user upload storage however it seems like their bandwidth can quickly scale well beyond what we are charging for hosting. It just takes one viral video to get highly expensive.
I was looking for a storage solution for that data that could be decentralized over inexpensive machines. Much like a cassandra topology. So I could just quickly through more hardware at the solution to grow it our for performance. Haven't found anything yet though :-)
Yes, I had 16db's setup, but have them converted back to one and InnoDB
Amazon S3 it tempting for the user upload storage however it seems like their bandwidth can quickly scale well beyond what we are charging for hosting.
Yes, looks cheap at first, well is for video at least, but have read of people getting hit with big bills.
I've seen @drmike talk about just an additional HDD, which seems the way to go.
Another question here..
The guy that fixed my db mentioned this about scaling to MultiDB:
With InnoDB tables there is only one situation when MultiDB will add some profit to performance - list all tables. All other queries will run equally well with one or many databases.
Are there any known problems running MultiDB with InnoDB?
6593 pointsCommunity rank #9Supreme DeityMember of the MonthLifetime member
Keeper of the Dark Chocolate
(joined July 2007)
Likes (0)
I've seen @drmike talk about just an additional HDD, which seems the way to go.
Rep Points for listening to me. :D
I know we had a thread this morning that I couldn;t get to where I was looking at the poster's site and it was going back and forth and back and forth between the site and their cdn. Looked like some fancy redirect was going on. About 30 seconds of nothing but blank screen and that back and forth, I just closed out the browser pane and went onto the next thread. Not sure what they were doing but it wasn;t working.
With a hard drive, you can usually get that for a one time cost. Your bandwidth is already included with your box so in theory there's no extra fees involved afterwards.
edit: Granted though I would like to fiddle with setting up a cdn someday. Our large setup just has the uploads on a second box. (or is it the database? Can never remember.) That may be something to consider as well.
For your database I would recommend a raid 10 for performance, Honestly I would be worried about disk IO operations with that many open files in your database. I am interested to hear how others are scaling 4096 whether they are splitting them up on to a couple different database servers.
6593 pointsCommunity rank #9Supreme DeityMember of the MonthLifetime member
Keeper of the Dark Chocolate
(joined July 2007)
Likes (0)
What's easier for you to do? Keep 256 pages open in a book or 4096? :)
Each timne you open up a file, the processor has to assign resources to it to keep track of what's going on. You use less resources with 256 databases than you do with 4096.
Responses (11)
Member (joined April 2009) Likes (0)
Hiya george,
I'll ask a couple of our server experts to chime in with some ideas here.
Thanks!
Member (joined August 2010) Likes (0)
Hello,
I am also trying to plan our scaling of our site.
Do you have all your databases on one machine?
Or do you split them across a couple database servers?
Amazon S3 it tempting for the user upload storage however it seems like their bandwidth can quickly scale well beyond what we are charging for hosting. It just takes one viral video to get highly expensive.
I was looking for a storage solution for that data that could be decentralized over inexpensive machines. Much like a cassandra topology. So I could just quickly through more hardware at the solution to grow it our for performance. Haven't found anything yet though :-)
Member (joined August 2009) Likes (0)
Hi @jozwikjp
Yes, I had 16db's setup, but have them converted back to one and InnoDB
Yes, looks cheap at first, well is for video at least, but have read of people getting hit with big bills.
I've seen @drmike talk about just an additional HDD, which seems the way to go.
Another question here..
The guy that fixed my db mentioned this about scaling to MultiDB:
Are there any known problems running MultiDB with InnoDB?
Thanks! :)
Keeper of the Dark Chocolate (joined July 2007) Likes (0)
Rep Points for listening to me. :D
I know we had a thread this morning that I couldn;t get to where I was looking at the poster's site and it was going back and forth and back and forth between the site and their cdn. Looked like some fancy redirect was going on. About 30 seconds of nothing but blank screen and that back and forth, I just closed out the browser pane and went onto the next thread. Not sure what they were doing but it wasn;t working.
With a hard drive, you can usually get that for a one time cost. Your bandwidth is already included with your box so in theory there's no extra fees involved afterwards.
edit: Granted though I would like to fiddle with setting up a cdn someday. Our large setup just has the uploads on a second box. (or is it the database? Can never remember.) That may be something to consider as well.
Member (joined August 2009) Likes (0)
Thanks.. 100 Points added! :)
If I'm doing 4096dbs, would it be best to have a separate drive for them or for uploads?
Member (joined August 2010) Likes (0)
For your database I would recommend a raid 10 for performance, Honestly I would be worried about disk IO operations with that many open files in your database. I am interested to hear how others are scaling 4096 whether they are splitting them up on to a couple different database servers.
Member (joined August 2009) Likes (0)
@jozwikjp
Can you please clarify?
My host has offered to setup an additional server for me which is a little extra, but might be worth it by the looks of things.
Keeper of the Dark Chocolate (joined July 2007) Likes (0)
What's easier for you to do? Keep 256 pages open in a book or 4096? :)
Each timne you open up a file, the processor has to assign resources to it to keep track of what's going on. You use less resources with 256 databases than you do with 4096.
Member (joined August 2009) Likes (0)
Got it. But I would hate to do this process again so 4096 it has to be.
I might setup another server to handle this then.
Member (joined August 2010) Likes (0)
You could split the 4096 and migrate databases to another db server to alleviate some load.
Member (joined August 2009) Likes (0)
@jozwikjp
Thanks, getting a new server setup and moving all the db's there. Let's see how it goes thanks!
Become a member