Urgent Help On Replacing /var

Home » CentOS » Urgent Help On Replacing /var
CentOS 5 Comments

In a moment of epic stupidity, having ran out of space on the root partition of a server due to /var chewing up the space, I added a separate drive for the purpose of mounting it as /var

To do so, I mounted the new drive as /var2, cp -R (in hindsight should had rsync to preserve attributes), deleted the original /var to free up space, edited fstab and rebooted… unsurprisingly to a fubar’d server.

The thing is it still boots, I can get into single user mode but a full init get stuck at starting the syslogger. What is the best way to rescue the server now from my own stupidity and keeping all the existing configuration and data?

5 thoughts on - Urgent Help On Replacing /var

  • 2012/8/3 Emmanuel Noobadmin :

    Did you rewrite the selinux policy on /var or have you tried disabling selinux if you haven’t do so already?

  • Thank you so much!

    Turning off selinux allowed me get the system running. However, after running fixfiles to restore the context for /var, I
    still cannot boot to init 5, with the choke point now NFS statd.

    I’ll probably have to slowly hunt down the relevant selinux context one by one when nobody’s screaming about the server being down.

  • Hello Emmanuel,

    If you copied a live /var over to the new partition you might have issues from stale lock files that prevent services from starting. /var/lock/subsys/nfsd comes to mind in this particular case. If the service ain’t running that file shouldn’t be there.

    I don’t think stale pid files are as troublesome but it doesn’t hurt to have a look at /var/run as well.

    Regards, Leonard.

  • Would restorecon not help get this bootrapped ? and then with selinux in permissive mode, watch the audit log like a hawk.

    – KB

  • By extension:

    rpm -qa | while read line; do echo $line && rpm –setugids $line; done

    should handle ownerships. Then, reenable selinux in permissive mode, and set it to relabel on the next boot.