Until recently our remote backup service for our ULS servers has been based purely on the amount of storage consumed, whereas the actual real big cost is bandwidth. Where the bandwidth component was often closer to 10% that of the consumed storage, the pricing has been rather steep all things considered. During the last month or so we’ve been working hard to build a bandwidth accounting mechanism surrounding the actual backup protocol and during the week we finally succeeded. This means that, as of the last few days, a new billing structure has been put in place.
Separation of bandwidth and storage allows us to provide an overall pricing that is significantly (in the order of up to 80%) cheaper than was previously possible. The mechanisms implemented to measure the required bandwidth simply isn’t available on the tools that we utilize, yet, they are designed to significantly reduce the required bandwidth. In particular rsync can use a reference file (from the previous backup) to generate checksum based deltas on, resulting in it effectively only transferring portions of the file that has changed. On top of this a run-length encoded zlib compression is utilized in order to further reduce the transmitted size of the deltas of the file. We have seen cases where this reduces the standard bandwidth requirements by as much as 98% (so instead of copying over 1GB worth of files only 20MB gets transferred over the Internet) in extreme cases, the average bandwidth saving seems to be just under 90 %.
Since we are now able to make this distinction, we will be billing bandwidth at a rate of R 15.00/GB for actual transfer, and R 2.00/GB worth of storage required. As an example, from a few of our own servers, the following table shows the storage and bandwidth requirements for it’s backups for the last couple of days, along with what the previous billing scheme would have cost, and what the new billing scheme will cost:
|4.66 GB||550.87 MB (11.8 %)||R 55.92||R 17.58|
|4.66 GB||758.23 MB (16.3 %)||R 55.92||R 20.69|
|4.63 GB||136.50 MB (2.9 %)||R 55.44||R 11.29|
|3.74 GB||77.17 MB (2.1 %)||R 44.88||R 8.64|
|6.66 GB||106.08 MB (15.9 %)||R 79.92||R 14.91|
As can be seen this is a significant reduction. What we have noticed is that the more database-driven the load is the more significant the saving. In particular, loads with large, ‘slow’ changing files (such as innodb files that are typically seen on many of our MySQL servers) sees the most significant results. Also very handy for email servers where the storage requirements are those of the raw text of the emails but the transfer itself is relatively small due to the zlib compression.
For more information please contact firstname.lastname@example.org. The base scheme is primarily targeted at ULS installed servers (utilizing LVM), but can be adjusted for most other Linux servers (and the scripts has even been used for Windows servers before).