7 thoughts on - Systemd-journald Corruption

• Determined with ‘journalctl –verify’ or another way?

  (which stops ALL logging, including syslog).

  No. Everything reports to journald and rsyslog gets what it wants from journald.

  If you are referring to native journald logs corrupting, that should not affect rsyslog. If you remove /var/log/journal then systemd-journald logs will be stored volatile in /run.

  Sortof, but not in a way that affects rsyslog. Usually journalctl just skips over corrupt parts and systemd-journald will rotate logs when it detects corruption to isolate corrupt files.

  Chris Murphy

• Once upon a time, Chris Murphy said:

  I get messages like this in dmesg:

  [4756650.489117] systemd-journald[21364]: Failed to write entry (21 items, 637 bytes), ignoring: Cannot assign requested address

  I’ll check journalctl –verify when it happens next (seems to happen every day on at least one of the servers). It does only seem to be happening only on my spam-scanning VMs. Some of them do have a relatively high log rate (40-50 messages per second sometimes).

  I dug into it a little more after my original message, and it appears to be a recent issue; I have some VMs that were set up a little longer ago
  (still running CentOS 7.1 I believe) that have not had this problem.

  That appears to be where they’re going (I don’t have a /var/log/journal, but I didn’t do anything to remove it). I have had to remove files from
  /run/log/journal to get systemd-journald working again.

  When it happens, all logs just stop; rsyslogd appears to not get any more log entries (I have rsyslogd logging to central log hosts and they get nothing).

• I haven’t seen this. When I plug this text into a google search field, no quotes, there are 360 results.

  systemd-journald failed to write entry cannot assign requested address

  There’s also this patch as a suggested fix:
  https://bugzilla.redhat.com/show_bug.cgi?id92447#c9

  What version of systemd and rsyslog? systemd-219-19.el7_2.7 and rsyslog-7.4.7-12 are current.

  If you’re there already you could ry editing
  /etc/systemd/journald.conf and uncommenting Compress=yes and changing it to no.

• Once upon a time, Chris Murphy said:

  One system did get into this state overnight, and that said:

  [root@spamscan3 ~]# journalctl –verify
  15bd478: invalid object
  File corruption detected at /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal:15bd478 (of 25165824 bytes, 90%). FAIL: /run/log/journal/f8ade260c5f84b8aa04095c233c041e0/system.journal (Cannot assign requested address)
  (and then a bunch of passes on the rest of the files)

  I’ll take a look at that.

  Those are the versions I have.

  Thanks, I’m trying that on these servers.

• Also I wonder if merely restarting the journal daemon solves it:

  systemctl restart systemd-journald

  What should happen is it realizes its own logs are corrupt and ignores them, and starts working on new copies. And journalctl should still try to read the old ones but skips the corrupt entries.

  If that works you could schedule a restart of the journal periodically as a goofy hack work around until it gets fixed. Clearly Red Hat knows about this problem.

• Once upon a time, Chris Murphy said:

  I tried that and it didn’t work (at least not every time). I had to actually remove the journal file to get it functional again.

  So far, turning off the compression appears to have worked (but I’ll have to watch it for a day or two to really see).

• Once upon a time, Chris Adams said:

  Just to follow up: turning off journald compression does appear to have fixed the corruption problem I was seeing. I’ll watch for an updated systemd package.