Tag Archives: Scaling Out

Please scale your storage too

October 6, 2011 2:33 pm /

Everyone understands the concept of scaling out compute when building applications for Windows Azure, but many times I’m seeing storage left out of the equation. The results are applications that don’t scale because they are tied to a single database. In this post I will explain how you need to start thinking about storage in the cloud.

Storage on-premises

If you think about how you build big applications on premises and how you approach data in general is very different. Typically you have a database or storage standard. This could be SQL Server, or some other relational database. The law of the enterprise is usually “all data shall be stored in <insert your enterprise database here>”. There is no other choice.

The second big difference is how you approach scale. Typically to get scale, you build the biggest most elaborate database server you can. I remember years ago, tuning everything from the hardware, through to how the disks were partitioned and where the file groups in SQL Server were located and used. Machines typically had multiple CPU’s with multiple cores, multiple disk controllers, disks and a bucket full of RAM.

If you also needed high availability, that was easy. You typically built a second monster machine and configured an Active/Passive setup, or hot standby,

How the cloud changes storage

The first thing about using storage in the cloud is that there are zero setup costs. You can just as easily create a SQL Azure database or 7. Create multiple storage accounts, tables etc. You pay for what you consume. Setup is free. This means you have less reason to put all your data in one place. You can start thinking about which data source is right for what data. It is not unusual to have multiple data sources used.

The second big thing is that we don’t have that big huge server with lots of cores and ram and fancy stuff. At least not tuned to the extent you would for your workload. Instead we have “commodity hardware”. To achieve “scale” you have to think about using lots of little storage things, rather than a single huge one. This is the exact same thing you do with compute.

If you have 50GB of data, you may be better served by 50 x 1GB databases than 1 x 50GB. The cost is comparable too. Except you now have 50 of those servers running your database. Guess which model will scale better?

There is a tax in thinking about, designing and implementing this – however if done right, you can scale out as far as your compute (and credit card) will let you.

This might have been a rant. Apologies.

THIS POSTING IS PROVIDED “AS IS” WITH NO WARRANTIES, AND CONFERS NO RIGHTS

Posted in: Windows Azure / Tagged: , ,

Scaling out Azure

April 19, 2010 12:00 am /

<Rant Warning/>

In previous blog posts, I’ve talked about some of the patterns you can use to build your apps for the cloud, including Task-Queue-Task and de-normalizing your data using that pattern. But now something on scaling out.

When you are building apps in the cloud, you have to remember you are running in a shared environment and have no control over the hardware.

Let’s think about that for a moment.

In Windows Azure and SQL Azure we run you on hardware. Information on roughly what to expect can be found here (scroll down and expand compute instances), but here is a table of the compute part of Windows Azure:

Compute Instance Size CPU Memory Instance Storage I/O Performance
Small 1.6 GHz 1.75 GB 225 GB Moderate
Medium 2 x 1.6 GHz 3.5 GB 490 GB High
Large 4 x 1.6 GHz 7 GB 1,000 GB High
Extra large 8 x 1.6 GHz 14 GB 2,040 GB High

So how fast is the memory? What kind of CPU caching do we have? How fast are the drives? What about the network?

For SQL Azure we don’t even tell you what it’s running on, although you can watch this to get a better idea of how “shared” you are.

The point I’m going to make is that when you control the hardware, you can figure out lots of things like the throughput of disk controllers, CPU & Memory and based on that knowledge create filegroups for databases that span multiple drives, install more cores, faster drives, more memory, faster networking – all to improve performance. You are scaling up.

In the cloud, things work differently – you have to scale out. You have lots of little machines doing little chunks of work. No more 32-way servers at your disposal to crank through that huge workload. Instead you need 32 x 1 way servers to crank through that workload. There are no file groups, no 15,000 rpm drives. Just lots of cheap little servers ready for you whenever you need them.

I get a lot of questions about the performance of this, that and the other and while I understand that information can be useful, I think they somewhat miss one of the points and the potential of using the cloud.

I don’t need 15,000rpm drives and 8 cores to handle my anticipated peak workload. Instead I can have 20 servers working my data at peak times, and 5 servers the rest of the time. So stop thinking about how fast the memory is, and start figuring out how you can use as many servers as you need – when you need it.

Remember OUT not UP.

THIS POSTING IS PROVIDED “AS IS” WITH NO WARRANTIES, AND CONFERS NO RIGHTS, EVEN IF YOU SAY PLEASE

Posted in: Windows Azure / Tagged: , ,