11 thoughts on - Problem With Mdadm, Raid1 And Automatically Adds Any Disk To Raid

• I didn’t check but are you really sure you’re cleaning up the end of the drive? Maybe you should clean the end of every partition first because metadata may be written there.

  Noooooo, you’re wiping it wrong :-)

  
  Nothing easier than that, just terminate systemd while doing the disk management and restart it after you’re done. BTW, PID is 1.
  

  Seriously, there is certainly some systemd unit you may be able to deactivate before doing such things. However, I don’t know which one it is.

  I’ve been fighting a similar crap: On HPE servers when running cciss_vol_status through the disk monitoring system, whenever cciss_vol_status is run and reports hardware RAID status, systemd scans all partition tables and tries to detect LVM2 devices and whatever. Kernel log is just filled with useless scans and I have no idea how to get rid of it. Nice new systemd world.

  Regards, Simon

• Mmmmhhh, not sure. I run fdisk on it, basically re-creating everything from the start.

  The “trying to re-create the MDX’s” happens when I use “w” in fdisk. As soon as I hit the “w” it starts re-creating the MDx!

  Thats the annoying part.

  [snip]

  I am not ;-), it’s @ work.

  Jobst

• In article <20190225050144.GA5984@button.barrett.com.au>, Jobst Schmalenbach wrote:

  The superblock is a property of each partition, not just of the whole disk.

  So I believe you need to do:

  mdadm –zero-superblock /dev/sdb1
  mdadm –zero-superblock /dev/sdb2
  mdadm –zero-superblock /dev/sdb3
  mdadm –zero-superblock /dev/sdc1
  mdadm –zero-superblock /dev/sdc2
  mdadm –zero-superblock /dev/sdc3

  Cheers Tony

• Clearing the initial sectors doesn’t do anything to clear the data in the partitions.  They don’t become blank just because you remove them.

  Partition your drives, and then use “wipefs -a /dev/sd{b,c}{1,2,3}”

  What makes you think this has *anything* to do with systemd? Bitching about systemd every time you hit a problem isn’t helpful.  Don’t.

• If it’s not systemd, who else does it? Can you elaborate, please?

  Regards, Simon

• I’ll wager it’s the mdadm.service unit. You’re seeing systemd in the log because systemd has a unit loaded that’s managing your md devices. The package mdadm installs these files:

  /usr/lib/systemd/system/mdadm.service
  /usr/lib/systemd/system/mdmonitor-takeover.service
  /usr/lib/systemd/system/mdmonitor.service

  Perhaps if you turn off these services, you’ll be able to manage your disks without interference. I do not use mdadm on my system, I’m just looking at the content of the rpm file on rpmfind.net. That said, systemd isn’t the culprit here. It’s doing what it’s supposed to (starting a managed service on demand).

  I do concede the logs are confusing. For example, this appears in my logs:

  Feb 26 05:10:03 demeter systemd: Starting This service automatically renews any certbot certificates found…

  While there is no indication in the log, this is being started by:

  [cbell@demeter log]$ systemctl status certbot-renew.timer
  ● certbot-renew.timer – This is the timer to set the schedule for automated renewals
  Loaded: loaded (/usr/lib/systemd/system/certbot-renew.timer; enabled;
  vendor preset: disabled)
  Active: active (waiting) since Thu 2019-02-21 17:54:43 CST; 4 days ago

  Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
  [cbell@demeter log]$

  And you can see the log message through the service unit using journalctl:

  [cbell@demeter log]$ journalctl -u certbot-renew.service | grep “This service” | tail -1
  Feb 26 05:10:07 demeter.home systemd[1]: Started This service automatically renews any certbot certificates found.
  [cbell@demeter log]$

  You can see there’s no indication in /var/log/messages that it’s the certbot-renewal service (timer) that’s logging this. So it’s easy to misinterpret where the messages are coming from, like your mdadm messages. Perhaps having the journal indicate which service or timer is logging a message is a feature request for Lennart!

  Hope this helps!

  —
  Chris

  “If you wish to make an apple pie from scratch, you must first invent the Universe.” — Carl Sagan

• I’m not sure what your box runs but it’s at least not CentOS 7.

  CentOS 7 contains these md related units:
  /usr/lib/systemd/system/mdadm-grow-continue@.service
  /usr/lib/systemd/system/mdadm-last-resort@.service
  /usr/lib/systemd/system/mdadm-last-resort@.timer
  /usr/lib/systemd/system/mdmonitor.service
  /usr/lib/systemd/system/mdmon@.service

  The only md related daemon running besides systemd is mdadm. I’ve never seen such behavior with EL6 and the mdadm there so I don’t think it will ever do such things.

  The message produced comes from mdadm-last-resort@.timer. Whatever triggers it it’s either systemd or something like systemd-udevd.

  How is it not systemd doing it? Such things didn’t happen with pre systemd distributions.

  Regards, Simon

• [snip]

  Becasue of this.

  Feb 25 15:38:32 webber systemd: Started Timer to wait for more drives before activating degraded array md2..

• I just had a hardware failure of a Raid controller (well they fail thats why we have backups). This means putting the drives onto a new controller I have to (re-) format them.

  In CentOS6 times this took me under an hour to fix this, mostly due to the rsyncing time. Yesterday it took me over 6 hours to move a system.

  Jobst

• I actually deleted the partitions, at first using fdisk than parted (read a few ideas on the internet). Also from the second try onwards I also changed the partition sizes, filesystems. Also I tried with one disk missing (either sda or sdb).

  Jobst

• The following log is from a CentOS 6 system.  I created RAID devices on two drives.  I then stopped the RAID devices and ‘dd’ over the beginning of the drive.  I then re-partition the drives.

  At that point, the RAID devices auto-assemble.  They actually partially fail, below, but the behavior that this thread discusses absolutely is not systemd-specific.

  What you’re seeing is that you’re wiping the partition, but not the RAID
  information inside the partitions.  When you remove and then re-create the partitions, you’re hot-adding RAID components to the system.  They auto-assemble, as they have (or should have) for a long time.  It’s probably more reliable under newer revisions, but this is long-standing behavior.

  The problem isn’t systemd.  The problem is that you’re not wiping what you think you’re wiping.  You need to use “wipefs -a” on each partition that’s a RAID component first, and then “wipefs -a” on the drive itself to get rid of the partition table.

  [root@localhost ~]# dd if=/dev/zero of=/dev/vdb bs=512 count=1024
  1024+0 records in
  1024+0 records out
  524288 bytes (524 kB) copied, 0.0757563 s, 6.9 MB/s
  [root@localhost ~]# dd if=/dev/zero of=/dev/vdc bs=512 count=1024
  1024+0 records in
  1024+0 records out
  524288 bytes (524 kB) copied, 0.0385181 s, 13.6 MB/s
  [root@localhost ~]# kpartx -a /dev/vdb
    Warning: Disk has a valid GPT signature but invalid PMBR.
    Assuming this disk is *not* a GPT disk anymore.
    Use gpt kernel option to override.  Use GNU Parted to correct disk.
  [root@localhost ~]# kpartx -a /dev/vdc
    Warning: Disk has a valid GPT signature but invalid PMBR.
    Assuming this disk is *not* a GPT disk anymore.
    Use gpt kernel option to override.  Use GNU Parted to correct disk.
  [root@localhost ~]# cat /proc/mdstat Personalities : [raid1]
  unused devices:
  [root@localhost ~]# parted /dev/vdb -s mklabel gpt   mkpart primary ext4
  1M 200M   mkpart primary ext4 200M 1224M   mkpart primary ext4 1224M 100%
  [root@localhost ~]# parted /dev/vdc -s mklabel gpt   mkpart primary ext4
  1M 200M   mkpart primary ext4 200M 1224M   mkpart primary ext4 1224M 100%
  [root@localhost ~]# cat /proc/mdstat Personalities : [raid1]
  unused devices:
  [root@localhost ~]# cat /proc/mdstat Personalities : [raid1]
  md2 : active raid1 vdc3[1] vdb3[0]
        19775360 blocks super 1.0 [2/2] [UU]

  md1 : active raid1 vdb2[0]
        999360 blocks super 1.0 [2/1] [U_]

  md0 : active raid1 vdb1[0]
        194496 blocks super 1.0 [2/1] [U_]

  unused devices: