15 thoughts on - File System Replication

• drbd is the live replication system, this does block level replication of logical volumes. rsync doesn’t handle changing-on-the-fly stuff very well, especially stuff like a database file system which is undergoing constant random write activity.

  but note, the drbd slave copy can’t be mounted until drbd is stopped, or all heck breaks loose.

• You can try lsyncd.

  https://code.google.com/p/lsyncd/

  It’s basically a daemon monitoring inotify events and sending them to an rsync, so it can keep things up to date more easily. (I have never used it myself, so caveat emptor.)

  –keith

• With databases you can not make sync of their data directories (on a filesystem level) while the daemon is running and accessing them in general. You may think of mysql redundant server or replication… Take a look here:

  https://dev.mysql.com/doc/refman/5.0/en/replication.html

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• rsync is only safe to use on something like a database server if you stop the database each time you run rsync, otherwise the files will get out of sync.

  I suppose you could setup mysql replication, and exclude mysql from the lsyncd/rsync, that would be one approach.

• This would be a perfect use case for ZFS snapshot send and receive!

• Its not perfect if you don’t want to lose all database transactions, new files, and email between the last update and the point of hardware failure.

  drbd really is the best answer as its continuous in-order replication.

• Assuming of course that you dont need any kind of performance from your filesystem drdb is indeed the answer.

  ZFS snapshots can however be run on a per second basis which is quite neat!
  What ever you choose to do you would probably want to use mysql binary replication anyway so having a completely sync filesystem replica is probably not too much of a problem.

  I must point out that virtualisation made these kinds of problems mostly go away (assuming of course that you have cash for some kind of High Availability iSCSI device or something.

• Am 09.10.2014 um 20:11 schrieb John R Pierce :

  +1

  in conjunction with pacemaker and corosync it would also switch over when some hardware issues appears.

• Hi John,

  ​Just to clarify. What state is the slave in?

  If master goes down, how does the slave become active? Just reboot it and let it come up? The wording “slave copy can’t be mounted until drdb is stopped” leads me to believe the slave is in some alternate state to be receiving the blocks of data.
  ​

• the slave is up and running, but the file systems you’re replicating are unmounted, and its services are stopped, so you could consider this to be a ‘standby’ state.

  yes, to use drbd, its important that you put your email spools, databases, etc, on dedicated file system(s), NOT on the OS root file system. I generally use lvm for all this.

  a cluster management package, such as the ones suggested by another poster, would take care of all this for you (once you have things setup properly), if the master fails, it would ‘activate’ the slave, switch its IP[*] over to be the ‘production’ system, and mount its file systems, starting its services (mysqld, postfix, etc) per your configuration.

  [*] typically, you use THREE IP addresses for a HA cluster. a unique IP for each system, used only for management, and a ‘service’ IP used for the production accesses, which is held by the currently active system. when the master fails, the slave adopts this service IP.

  —
  john r pierce 37N 122W
  somewhere on the middle of the left coast

• Hi John,

  ​Thank you for this info. This clears up a lot and is very helpful.

  Jason

• Ah, yes, I missed the OP’s reference to mysql. Sorry about that!

  mysql/mariadb replication is a reasonable way to go, and probably has more eyes on it than lsyncd. MariaDB’s somewhat new GTID replication might be more reliable than the old query/row/mixed replication scheme they had before.

  –keith

• Most people forget that DRDB is a ‘clustered’ storage that creates a new block device on top of existing block devices, over the network.

  There are issues like SPLIT BRAIN and a possible data inconsistency, should there be multiple network timeouts, congestions (since it is all network based) etc. This may not be the best solution for your problem!

• those are potential issues with ANY high availability clustering solution that need to be carefully engineered around. HA clusters commonly use fencing technology to prevent split brain, dedicated network adapters just for replication, and redundant connections for status monitoring, etc.

• And you still need backups. Other things besides hardware failures can wipe your data and with mirrors/clusters, software or operator errors will take all the copies at once.