How To Determine Why A Server Is Not Responding

Home » CentOS » How To Determine Why A Server Is Not Responding
CentOS 12 Comments

Hi to all!

We’re using CentOS 5.5 64bits for our Plesk 11.

This week we had the following problem 3 times…

Suddenly, the server stops responding in all services (SSH, Apache, Postfix, …) but ping works!

After wait a few minutes (or 2 hours some times) the server continues unresponsive until we reboot. After reboot we search on /var/log/messages but cannot find useful information…

Apr 11 14:56:05 s1 postfix/smtpd[8263]: SQL engine ‘intentionally disabled’
not supported Apr 11 14:56:05 s1 postfix/smtpd[8263]: auxpropfunc error no mechanism available Apr 11 14:56:42 s1 postfix/smtpd[8370]: SQL engine ‘intentionally disabled’
not supported Apr 11 14:56:42 s1 postfix/smtpd[8370]: auxpropfunc error no mechanism available Apr 11 14:56:47 s1 postfix/smtpd[8391]: SQL engine ‘intentionally disabled’
not supported Apr 11 14:56:47 s1 postfix/smtpd[8391]: auxpropfunc error no mechanism available Apr 11 14:56:47 s1 postfix/smtpd[8392]: SQL engine ‘intentionally disabled’
not supported Apr 11 14:56:47 s1 postfix/smtpd[8392]: auxpropfunc error no mechanism available Apr 11 16:55:42 s1 syslogd 1.4.1: restart. Apr 11 16:55:42 s1 kernel: klogd 1.4.1, log source = /proc/kmsg started. Apr 11 16:55:42 s1 kernel: Bootdata ok (command line is ro root=/dev/xvda1
console=xvc0 console=hvc0 xencons=hvc)
Apr 11 16:55:42 s1 kernel: Linux version 2.6.18-194.26.1.el5xen (
mockbuild@builder10.CentOS.org) (gcc version 4.1.2 20080704 (Red Hat
4.1.2-48)) #1 SMP Tue Nov 9 13:35:30 EST 2010
Apr 11 16:55:42 s1 kernel: BIOS-provided physical RAM map:
Apr 11 16:55:42 s1 kernel: Xen: 0000000000000000 – 0000000080000000
(usable)
Apr 11 16:55:42 s1 kernel: No mptable found. Apr 11 16:55:42 s1 kernel: Built 1 zonelists. Total pages: 524288
Apr 11 16:55:42 s1 kernel: Kernel command line: ro root=/dev/xvda1
console=xvc0 console=hvc0 xencons=hvc Apr 11 16:55:42 s1 kernel: Initializing CPU#0
Apr 11 16:55:42 s1 kernel: PID hash table entries: 4096 (order: 12, 32768
bytes)
Apr 11 16:55:42 s1 kernel: Xen reported: 2009.260 MHz processor. Apr 11 16:55:42 s1 kernel: Console: colour dummy device 80×25
Apr 11 16:55:42 s1 kernel: Dentry cache hash table entries: 262144 (order:
9, 2097152 bytes)
Apr 11 16:55:42 s1 kernel: Inode-cache hash table entries: 131072 (order:
8, 1048576 bytes)
Apr 11 16:55:42 s1 kernel: Software IO TLB disabled Apr 11 16:55:42 s1 kernel: Memory: 2043384k/2097152k available (2513k kernel code, 53108k reserved, 1395k data, 184k init)
Apr 11 16:55:42 s1 kernel: Calibrating delay using timer specific routine..
5025.13 BogoMIPS (lpj=10050261)
Apr 11 16:55:42 s1 kernel: Security Framework v1.0.0 initialized Apr 11 16:55:42 s1 kernel: SELinux: Initializing. Apr 11 16:55:42 s1 kernel: selinux_register_security: Registering secondary module capability Apr 11 16:55:42 s1 kernel: Capability LSM initialized as secondary Apr 11 16:55:42 s1 kernel: Mount-cache hash table entries: 256
Apr 11 16:55:42 s1 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache
64K (64 bytes/line)
Apr 11 16:55:42 s1 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Apr 11 16:55:42 s1 kernel: CPU: Physical Processor ID: 0
Apr 11 16:55:42 s1 kernel: CPU: Processor Core ID: 0
Apr 11 16:55:42 s1 kernel: (SMP-)alternatives turned off Apr 11 16:55:42 s1 kernel: Brought up 1 CPUs Apr 11 16:55:42 s1 kernel: checking if image is initramfs… it is Apr 11 16:55:42 s1 kernel: Grant table initialized Apr 11 16:55:42 s1 kernel: NET: Registered protocol family 16
Apr 11 16:55:42 s1 kernel: Brought up 1 CPUs Apr 11 16:55:42 s1 kernel: PCI: setting up Xen PCI frontend stub Apr 11 16:55:42 s1 kernel: ACPI: Interpreter disabled. Apr 11 16:55:42 s1 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay Apr 11 16:55:42 s1 kernel: pnp: PnP ACPI: disabled Apr 11 16:55:42 s1 kernel: xen_mem: Initialising balloon driver. Apr 11 16:55:42 s1 kernel: usbcore: registered new driver usbfs Apr 11 16:55:42 s1 kernel: usbcore: registered new driver hub Apr 11 16:55:42 s1 kernel: PCI: System does not support PCI
Apr 11 16:55:42 s1 kernel: PCI: System does not support PCI
Apr 11 16:55:42 s1 kernel: NetLabel: Initializing Apr 11 16:55:42 s1 kernel: NetLabel: domain hash size = 128
Apr 11 16:55:42 s1 kernel: NetLabel: protocols = UNLABELED CIPSOv4
Apr 11 16:55:42 s1 kernel: NetLabel: unlabeled traffic allowed by default Apr 11 16:55:42 s1 kernel: NET: Registered protocol family 2
Apr 11 16:55:42 s1 kernel: IP route cache hash table entries: 65536 (order:
7, 524288 bytes)
Apr 11 16:55:42 s1 kernel: TCP established hash table entries: 262144
(order: 10, 4194304 bytes)
Apr 11 16:55:42 s1 kernel: TCP bind hash table entries: 65536 (order: 8,
1048576 bytes)
Apr 11 16:55:42 s1 kernel: TCP: Hash tables configured (established 262144
bind 65536)
Apr 11 16:55:42 s1 kernel: TCP reno registered Apr 11 16:55:42 s1 kernel: audit: initializing netlink socket (disabled)
Apr 11 16:55:42 s1 kernel: type=2000 audit(1365692095.507:1): initialized Apr 11 16:55:42 s1 kernel: VFS: Disk quotas dquot_6.5.1
Apr 11 16:55:42 s1 kernel: Dquot-cache hash table entries: 512 (order 0,
4096 bytes)
Apr 11 16:55:42 s1 kernel: Initializing Cryptographic API
Apr 11 16:55:42 s1 kernel: alg: No test for crc32c (crc32c-generic)
Apr 11 16:55:42 s1 kernel: ksign: Installing public key data Apr 11 16:55:42 s1 kernel: Loading keyring Apr 11 16:55:42 s1 kernel: – Added public key 12AFE3EA6A14161C
Apr 11 16:55:42 s1 kernel: – User ID: CentOS (Kernel Module GPG key)
Apr 11 16:55:42 s1 kernel: io scheduler noop registered Apr 11 16:55:42 s1 kernel: io scheduler anticipatory registered Apr 11 16:55:42 s1 kernel: io scheduler deadline registered Apr 11 16:55:42 s1 kernel: io scheduler cfq registered (default)
Apr 11 16:55:42 s1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Apr 11 16:55:42 s1 kernel: rtc: IRQ 8 is not free. Apr 11 16:55:42 s1 kernel: Non-volatile memory driver v1.2
Apr 11 16:55:42 s1 kernel: Linux agpgart interface v0.101 (c) Dave Jones Apr 11 16:55:42 s1 kernel: brd: module loaded Apr 11 16:55:42 s1 kernel: Xen virtual console successfully installed as xvc0
Apr 11 16:55:42 s1 kernel: Event-channel device installed. Apr 11 16:55:42 s1 kernel: Uniform Multi-Platform E-IDE driver Revision:
7.00alpha2
Apr 11 16:55:42 s1 kernel: ide: Assuming 50MHz system bus speed for PIO
modes; override with idebus=xx Apr 11 16:55:42 s1 kernel: ide-floppy driver 0.99.newide Apr 11 16:55:42 s1 kernel: usbcore: registered new driver hiddev Apr 11 16:55:42 s1 kernel: usbcore: registered new driver usbhid Apr 11 16:55:42 s1 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver Apr 11 16:55:42 s1 kernel: PNP: No PS/2 controller found. Probing ports directly. Apr 11 16:55:42 s1 kernel: i8042.c: No controller found. Apr 11 16:55:42 s1 kernel: mice: PS/2 mouse device common for all mice Apr 11 16:55:42 s1 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
Apr 11 16:55:42 s1 kernel: md: bitmap version 4.39
Apr 11 16:55:42 s1 kernel: TCP bic registered Apr 11 16:55:42 s1 kernel: Initializing IPsec netlink socket Apr 11 16:55:42 s1 kernel: NET: Registered protocol family 1
Apr 11 16:55:42 s1 kernel: NET: Registered protocol family 17
Apr 11 16:55:42 s1 kernel: XENBUS: Device with no driver: device/vbd/51712
Apr 11 16:55:42 s1 kernel: XENBUS: Device with no driver: device/vif/0
Apr 11 16:55:42 s1 kernel: Initalizing network drop monitor service Apr 11 16:55:42 s1 kernel: Write protecting the kernel read-only data: 483k Apr 11 16:55:42 s1 kernel: USB Universal Host Controller Interface driver v3.0
Apr 11 16:55:42 s1 kernel: Fusion MPT base driver 3.04.13rh Apr 11 16:55:42 s1 kernel: Copyright (c) 1999-2008 LSI Corporation Apr 11 16:55:42 s1 kernel: SCSI subsystem initialized Apr 11 16:55:42 s1 kernel: Fusion MPT SPI Host driver 3.04.13rh Apr 11 16:55:42 s1 kernel: device-mapper: uevent: version 1.0.3
Apr 11 16:55:42 s1 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12)
initialised: dm-devel@redhat.com Apr 11 16:55:42 s1 kernel: device-mapper: dm-raid45: initialized v0.2594l Apr 11 16:55:42 s1 kernel: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled Apr 11 16:55:42 s1 kernel: SGI XFS Quota Management subsystem Apr 11 16:55:42 s1 kernel: Registering block device major 202
Apr 11 16:55:42 s1 kernel: xvda: xvda1 xvda2 xvda3
Apr 11 16:55:42 s1 kernel: netfront: Initialising virtual ethernet driver. Apr 11 16:55:42 s1 kernel: netfront: device eth0 has copying receive path. Apr 11 16:55:42 s1 kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Apr 11 16:55:42 s1 kernel: EXT3-fs: write access will be enabled during recovery. Apr 11 16:55:42 s1 kernel: kjournald starting. Commit interval 5 seconds Apr 11 16:55:42 s1 kernel: EXT3-fs: recovery complete. Apr 11 16:55:42 s1 kernel: EXT3-fs: mounted filesystem with ordered data mode. Apr 11 16:55:42 s1 kernel: SELinux: Disabled at runtime. Apr 11 16:55:42 s1 kernel: type=1404 audit(1365692123.036:2): selinux=0
auid=4294967295 ses=4294967295
Apr 11 16:55:42 s1 kernel: input: PC Speaker as /class/input/input0
Apr 11 16:55:42 s1 kernel: Floppy drive(s): fd0 is unknown type 15 (usb?), fd1 is unknown type 15 (usb?)
Apr 11 16:55:42 s1 kernel: Failed to obtain physical IRQ 6
Apr 11 16:55:42 s1 kernel: floppy0: no floppy controllers found Apr 11 16:55:42 s1 kernel: lp: driver loaded but no devices found Apr 11 16:55:42 s1 kernel: md: Autodetecting RAID arrays. Apr 11 16:55:42 s1 kernel: md: autorun … Apr 11 16:55:42 s1 kernel: md: … autorun DONE. Apr 11 16:55:42 s1 kernel: device-mapper: multipath: version 1.0.5 loaded Apr 11 16:55:42 s1 kernel: EXT3 FS on xvda1, internal journal Apr 11 16:55:42 s1 kernel: Filesystem “dm-0”: Disabling barriers, trial barrier write failed Apr 11 16:55:42 s1 kernel: XFS mounting filesystem dm-0
Apr 11 16:55:42 s1 kernel: Starting XFS recovery on filesystem: dm-0
(logdev: internal)
Apr 11 16:55:42 s1 kernel: Ending XFS recovery on filesystem: dm-0 (logdev:
internal)
Apr 11 16:55:42 s1 kernel: Filesystem “dm-1”: Disabling barriers, trial barrier write failed Apr 11 16:55:42 s1 kernel: XFS mounting filesystem dm-1
Apr 11 16:55:42 s1 kernel: Starting XFS recovery on filesystem: dm-1
(logdev: internal)
Apr 11 16:55:42 s1 kernel: Ending XFS recovery on filesystem: dm-1 (logdev:
internal)
Apr 11 16:55:42 s1 kernel: Filesystem “dm-2”: Disabling barriers, trial barrier write failed Apr 11 16:55:42 s1 kernel: XFS mounting filesystem dm-2
Apr 11 16:55:42 s1 kernel: Adding 1959920k swap on /dev/xvda2. Priority:-1
extents:1 across:1959920k Apr 11 16:55:43 s1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Apr 11 16:55:43 s1 kernel: Netfilter messages via NETLINK v0.30. Apr 11 16:55:43 s1 kernel: ip_conntrack version 2.4 (8192 buckets, 65536
max) – 304 bytes per conntrack Apr 11 16:55:44 s1 kernel: NET: Registered protocol family 10
Apr 11 16:55:44 s1 kernel: lo: Disabled Privacy Extensions Apr 11 16:55:44 s1 kernel: IPv6 over IPv4 tunneling driver Apr 11 16:55:44 s1 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team Apr 11 16:55:46 s1 xinetd[1426]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in. Apr 11 16:55:46 s1 xinetd[1426]: Started working: 2 available services

What can we do? what can we test?

More information…

Kernel:

Linux 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64
x86_64 x86_64 GNU/Linux

RAM (in MB):

total used free shared buffers cached Mem: 2048 1662 385 0 13 506
-/+ buffers/cache: 1142 905
Swap: 1913 0 1913

Disc usage:

S. fitxers Mida En

12 thoughts on - How To Determine Why A Server Is Not Responding

  • nan del bosc wrote:

    A quick google shows me that the postfix messages are just that, and you might want to fix it so it’s not asking for it.

    HOWEVER, the important thing is that it appears to have just gone completely unresponsive. I’ve seen that happen to some servers here, and we’ve never found any clues…. On the other hand, IIRC, they tended to be boxes that we’ve had other problems with, and have had a number rebuilt under warranty (mostly Penguins, and the problems I’ve had with them, as they’re all Supermicro m/b’s, told me to NEVER buy a Supermicro m/b).

    The only thing I can suggest trying might be to use ipmitool (assuming you don’t want to bring them down and look in the BIOS) to read the SEL
    (system event log), to look for hardware errors.

    mark

  • Thank’s for your quick answer!

    I can’t use ipmi in this machine…

    # ipmitool sel Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get SEL Info command failed

    # modprobe ipmi_si FATAL: Error inserting ipmi_si
    (/lib/modules/2.6.18-194.26.1.el5xen/kernel/drivers/char/ipmi/ipmi_si.ko):
    No such device

    # lsmod |grep ipmi ipmi_watchdog 52641 0
    ipmi_devintf 44753 0
    ipmi_msghandler 73369 2 ipmi_watchdog,ipmi_devintf

    This is a Virtual Server from 1and1, I cannot access the BIOS…

    any other idea?

    2013/4/11

  • Are you running sysstat / sar ?

    Perhaps the sa / sar database that’s left after reboot can show if some resource was over capacity.

  • Could be something related to disk access or RAM, runaway process or whatever. Do you have any system monitoring tools installed? Like munin, atop, sysstat?
    Any kernel errors in the logs?

  • Am 11.04.2013 17:36, schrieb nan del bosc:

    That’s insane! Why on earth do you run a 2,5 years old unpatched public system? You are asking for trouble and innocent third will be the victims of your hacked system.

    That’s a Xen Domain. So IPMI, as suggested by others, will not work.

    [ … ]

    First, update your system to the latest 5.9 + updates!

    Talk to your hoster. If your Xen VM has issues other guests on the same hardware may have too. Or another VM on the hosts consumes so much resources that your VM does not respond any longer.

    [ … ]

    Alexander

  • nan del bosc wrote:

    Um, no: yum install OpenIPMI
    service ipmi start ll /dev/ipmi0…..

    Wait – this is hosted, not something you can lay your hands on? In that case, you need to call the hosting provider and complain.

    mark

  • Nan del bosc wrote on Thu, 11 Apr 2013 17:59:58 +0200:

    You should have said this in the beginning!
    Can you be sure that this is a standard CentOS and not a version catered by the provider? It may just be a problem with the virtualizing software. You should talk to them. They also have means to access it in this hanging state from the host machine that you don’t have.

    Kai

  • If this is a virtual server, the actual hardware may just be running other virtual servers and you’re not getting any resources. If that’s true, nothing you do from your server will help you. You’ll need to get system stats from the actual hardware provider.

    Sounds like the hardware is over-committed. Do you have some kind of service guarantee?

  • Alexander Dalloz, true, but I’m only a system administrator guy who do the work that tells the boss… ;)

    Kai Schaetzl, yes! you are right, I should have said this in the beginning!
    Sorry, it’s because of the hurry… ;)

    Thank’s for all your answer! I wrote a mail to my provider. I keep you informed!

    2013/4/11 Dale Dellutri

  • Hello there, Is there anything in the Apache logs indicating a problem? I think the logs might be in a non-standard location when Plesk is installed so you’ll need to check for where those are. I think I remember having a problem, on a similar configuration, with a PHP limit and a user trying to send a large attachment through Horde that resulted in an almighty slowdown. Anyway, my twopenny’orth. All the best, Nic

  • I don’t think thats the only nonstandard thing when Plesk is in use, in fact, I think it makes so many changes to CentOS thats its very hard to provide CentOS support for the Plesk system.

  • No, it isn’t, yet there’s things available that’ll help you find what you need.

    Or on any system that has been touched by human hands?
    I know what you mean, but you gotta lend a hand.

    Nic

LEAVE A COMMENT