8 thoughts on - Drbd

• oh. this is running on a pair of CentOS 6.latest boxes, each dual xeon x5650 w/ 48GB ram, with LSI SAS2 raid card hooked up to a whole lotta sas/sata drives.

• That the backing device of one node fell out of sync is a cause concern.
  “Weekly” scan might be a bit much, but monthly or so isn’t unreasonable. Of course, as you’re seeing here, it’s a lengthy process and it consumes non-trivial amounts of bandwidth and adds a fair load to the disks.

  How long was it in production before this verify?

  I can’t speak to BackupPC, but I am curious how you’re managing the resources. Are you using cman + rgmanager or pacemaker?

• What version of DRBD? If you’re using a cluster resource manager, which and what version? How is DRBD configured and, if you are using a resource manager, what is it’s config?

• its the drbd8.3 package from elrepo.

  /etc/drbd.d/global_common.conf:

  global { usage-count no; }

  common {
  syncer { rate 200M; }
  }

  /etc/drbd.d/main.res:

  resource main {

  protocol C;
  startup { wfc-timeout 0; degr-wfc-timeout 120; }
  disk { on-io-error detach; }

  syncer { verify-alg crc32c; }

  on sg1.domain.com {
  device /dev/drbd0;
  disk /dev/vg_sg1data/lvdata;
  meta-disk internal;
  address 10.5.160.70:7788;
  }

  on sg2.domain.com {
  device /dev/drbd0;
  disk /dev/vg_sg2data/lvcopy;
  meta-disk internal;
  address 10.5.160.71:7788;
  }

  }

• OK. So what’s your start-up procedure? How should things start automatically then and where exactly is BackupPC hanging up? Is it trying to start before DRBD is Primary?

  So 8.3.16?

  Is this a 10 Gbps + very fast storage setup? If not, that is probably
  *way* too high.

  What does ‘drbdadm dump’ show? That will give a better idea of the actual setup. I’m interested specifically in the start parameters (ie:
  become-primary-on, etc).

• Just autostarting the demon with chkconfig on, didn’t do anything else special. when I reboot the master, its coming up secondary/secondary, and I manually have to kick it to primary then mount the file system, then start the BackupPC service.

  yeah. drbd83-utils-8.3.16-1.el6.elrepo.x86_64

  it is fairly fast storage, and gigE. I had temporarily tweaked that up to 200M to speed up the verify process, it was set to 50M before, and I
  just now set it back.

  # /etc/drbd.conf common {
  syncer {
  rate 50M;
  }
  }

  # resource main on svfis-sg1.netsys.stsv.seagate.com: not ignored, not stacked resource main {
  protocol C;
  on sg1.domain.com {
  device /dev/drbd0 minor 0;
  disk /dev/vg_sg1data/lvdata;
  address ipv4 10.x.x.70:7788;
  meta-disk internal;
  }
  on sg2.domain.com {
  device /dev/drbd0 minor 0;
  disk /dev/vg_sg2data/lvcopy;
  address ipv4 10.x.x.71:7788;
  meta-disk internal;
  }
  disk {
  on-io-error detach;
  }
  syncer {
  verify-alg crc32c;
  }
  startup {
  wfc-timeout 0;
  degr-wfc-timeout 120;
  }
  }

• Ok, so you might want to set ‘become-primary-on ‘. Then on start, the named node should auto-promote to Primary automatically.

  If it’s only 1 Gbps, then the maximum sustainable write speed is ~120
  MB/sec (at most; the slower of network or disk determines the max speed). You want the sync rate to be ~30% of maximum speed, or else you will choke out the apps using the DRBD resource and cause them to suffer a huge performance hit. App performance is (max write – sync rate) when a background sync is performed. This is why the default speed is very low.

  In your case, you’ve set the sync rate way above what is possible, and I’ve seen this dramatically hurt performance and even full-on stall the sync operation. I would set this to ’40M’ at most.

  This contradicts your configuration, I wonder if DRBD dropped it for you.

  That’s it? Nothing more? I would expect a lot more, normally. In any case, given it’s a totally manual config, I support that is enough.

• verify apparently uses this same speed to limit how fast it runs… since its doing checksums on each side, its not actually transferring any data across the network, hence why I boosted it to 200M (that left the node that initiated the verify with one process at 90% CPU, whihc I
  figure is as fast as it can go)… as I said, I’ve set it back to 50M
  now that the verify completed.

  btw, originally I had the verify-alg as sha1… my xeon X5650 2.7Ghz could only verify about 23MB/sec. setting it to crc32c sped up to
  180-200MB/sec, which is more reasonable considering this is a 16TB
  volume built from 2 8x3TB drive raid6’s striped together, behind a lsi
  9261-8i megaraid sas2 card with 512MB BBU cache.