MongoDB Backup Strategies with Write-Heavy Applications

The MongoDB Management Service Backup announced earlier in May provides excellent backup and restore capabilities for your MongoDB replica set cluster.

Setting up MongoDB backup is quite easy:
  • Sign Up for service,
  • Provide credit card details
  • Install and configure agent
In ~ 10 minutes your replica set is backed up and backup agent starts transferring the oplog data to backup mothership. Price for snapshot creating and storage is negligible, compared to the price ($2/GB) for oplog processing. For example, previous invoice was $0.89 for 330GB Snapshot storage, $0.65 for 71GB for snapshot create and $155 for 77GB of oplog traffic. 
After analyzing my application, I figured out that most of the traffic was caused by two collections:
  • metalog - 1GB capped collection which stores every API request with details
  • opcounters - this stores various opcounters; every minute new document is created and $inc operator increases various counter(s) by one, depending on action, e.g. db.opcounters.update({$_id:1234}, $set:{read:{$inc:1},write:{$inc:1}});
Luckily, MongoDB backup allows you to exclude namespaces when backing up. I decided to skip my metalog and opcounters and not to include them to backup. Even though my application gets more hits daily, the backup cost has decreased 10 times. Because of replica set, I need not to worry about data being lost; there are 3 members and if primary fails, secondary will still have the data.

I still do regular daily backups of my collections to S3 using mongodump with --oplog, which first allows me to have quick access to data snapshot in case I need to analyse data on my own developer machine. Run mongorestore with --oplogReplay option to get the data with latest oplog.

Sometimes you need to re-sync the backup; this used to happen more often with older version of backup agent. Newest backup agent seems to be more solid and re-syncing the backup is not required that often (in fact, after upgrading the agent, I have not done re-sync anymore).

Depending on the nature of your app using MongoDB, I would suggest skipping collections which are merely for logging purposes (for example capped collections) - these are 99% of the time write-only and can cause your oplog to quickly fill up and this can hit your wallet pretty heavily. Skipping other collections depend how critical these collections are for you.

Other considerations:
  • Run your replica set cluster members in geographically different locations, or different availability zones (for example, when running your replica set in the AWS cloud) to avoid downtime in case of power outage or network failure in data center.
  • Consider running Delayed Replica Set Member for rolling backup. This one makes backups based on oplog with a defined delay, and following feature is extremely useful for human errors. Make sure to set it's priority=0, hidden, set slaveDelay in seconds (3600 is pretty ok). Restoring replica set at this point is promoting this replicaSet member as primary.
More reading:


Location: New York, NY, USA

Comments

Post a Comment

Popular posts from this blog

Stubbing and Mocking Static Methods with PHPUnit

Image
PHPUnit has ability to stub and mock static methods. Consider the class Foo : <?php class Foo { public static function doSomething() { return static::helper(); } public static function helper() { return 'foo'; } } ?> To test the static helper() function with PHPUnit, you can write you test like that: <?php class FooTest extends PHPUnit_Framework_TestCase { public function testDoSomething() { $class = $this->getMockClass( 'Foo', /* name of class to mock */ array('helper') /* list of methods to mock */ ); $class::staticExpects($this->any()) ->method('helper') ->will($this->returnValue('bar')); $this->assertEquals( 'bar', $class::doSomething() ); } } ?> The new staticExpects() method works similar to the non-static expects...
Read more

MongoDB: Remove an Arbiter From a Replica Set

Image
Removing an arbiter from MongoDB Replica Set can be quite tricky, and official MongoDB documentation does not say it directly how to do it. I assume that you have auth enabled (just like I have), so lets start with connecting to your ReplicaSet "repl" master as an administrator: mongo mongo.domain.com/admin -u admin -p pass You will see the following prompt: repl:PRIMARY> Type in conf=rs.conf() repl:PRIMARY> conf = rs.conf() { "_id" : "repl", "version" : 8, "members" : [ { "_id" : 0, "host" : "mongo.server.com:27017", "priority" : 2 }, { "_id" : 1, "host" : "mongo1.server.com:27017" }, { "_id" : 2, "host" : "arbiter.server.com:27017", "arbiterOnly" : true }, { "_id" : 3, "host" : "mongo2.server.com:27017" } ] } Now lets adjus...
Read more

Enable HTTP/2 Support in AWS ELB

Image
This thread  was started more than a year ago, asking for when Amazon Web Services will add support for HTTP/2  into their ELB. In addition to that, people are also asking, how to configure ELB to support multiple SSL certificates. AWS support team has been reluctant on giving out any specific details on when both of these features will be available, citing: I've verified that the ELB team is aware of the interest in ELB supporting HTTP/2. Please keep an eye on What's New from Amazon Web Services: http://aws.amazon.com/new/ for any updates. What is HTTP/2? HTTP/2 is a replacement for how HTTP is expressed “on the wire.” It is not a ground-up rewrite of the protocol; HTTP methods, status codes and semantics are the same, and it should be possible to use the same APIs as HTTP/1.x (possibly with some small additions) to represent the protocol. HTTP/2 protocol is supported by major modern browsers including IE11, Edge 13, Firefox 47, Chrome 52, Safari 9.1, Opera 38, ...
Read more