Memory Leak – How To Investigate
My web & name server runs out of memory from time to time, to the point where it’s completely unresponsive to anything. At that point reset is the only alternative. (Or, as this is a virtual guest, I just say “virsh destroy”).
But why this happens – I would like to know.
The host in question is a KVM guest, and runs CentOS 6.4.
From “top” (situation now):
Mem: 1361564k total, 1264324k used, 97240k free, 8428k buffers Swap: 3014648k total, 64852k used, 2949796k free, 358676k cached
At or before the last crash I got a long error message on the console.
“Oom-killer” was called repeatedly by httpd and named.
I paste below the first error message, which is long. There were several long entries like this. I
– Jussi
[root@ns1 ~]# httpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
httpd cpuset=/ mems_allowed=0
Pid: 2962, comm: httpd Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 89
active_anon:201535 inactive_anon:68173 isolated_anon:3424
active_file:130 inactive_file:284 isolated_file:160
unevictable:0 dirty:0 writeback:245 unstable:0
free:14234 slab_reclaimable:2891 slab_unreclaimable:13218
mapped:239 shmem:14 pagetables:28858 bounce:0
Node 0 DMA free:8252kB min:340kB low:424kB high:508kB active_anon:2168kB
inactive_anon:4624kB active_file:4kB inactive_file:144kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB
dirty:0kB writeback:28kB mapped:8kB shmem:0kB slab_reclaimable:12kB
slab_unreclaimable:128kB kernel_stack:8kB pagetables:376kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:1424 all_unreclaimable? no lowmem_reserve[]: 0 1956 1956 1956
Node 0 DMA32 free:48684kB min:44712kB low:55888kB high:67068kB
active_anon:803972kB inactive_anon:268068kB active_file:516kB
inactive_file:992kB unevictable:0kB isolated(anon):13696kB
isolated(file):640kB present:2003828kB mlocked:0kB dirty:0kB
writeback:888kB mapped:948kB shmem:56kB slab_reclaimable:11552kB
slab_unreclaimable:52744kB kernel_stack:3344kB pagetables:115056kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7552
all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 25*4kB 11*8kB 6*16kB 3*32kB 9*64kB 3*128kB 1*256kB 3*512kB
1*1024kB 2*2048kB 0*4096kB = 8252kB
Node 0 DMA32: 977*4kB 401*8kB 320*16kB 179*32kB 58*64kB 63*128kB
32*256kB 7*512kB 3*1024kB 0*2048kB 1*4096kB = 48684kB
21463 total pagecache pages
20882 pages in swap cache Swap cache stats: add 1523887, delete 1503005, find 201987/297332
Free swap = 0kB
Total swap = 3014648kB
511996 pages RAM
43605 pages reserved
66036 pages shared
446256 pages non-shared
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 449] 0 449 2675 0 0 -17 -1000 udevd
[ 1080] 0 1080 6909 6 0 -17 -1000 auditd
[ 1105] 0 1105 62271 1 0 0 0 rsyslogd
[ 1142] 25 1142 40455 262 0 0 0 named
[ 1184] 0 1184 16563 0 0 -17 -1000 sshd
[ 1195] 0 1195 13036 0 0 0 0 vsftpd
[ 1231] 0 1231 27041 1 0 0 0 mysqld_safe
[ 1333] 27 1333 187111 4825 0 0 0 mysqld
[ 1430] 0 1430 20216 21 0 0 0 master
[ 1438] 89 1438 20236 18 0 0 0 pickup
[ 1439] 89 1439 20279 22 0 0 0 qmgr
[ 1440] 0 1440 79250 368 0 0 0 httpd
[ 1448] 0 1448 29313 34 0 0 0 crond
[ 1459] 48 1459 88738 727 0 0 0 httpd
[ 1460] 48 1460 89244 494 0 0 0 httpd
[ 1461] 48 1461 89729 803 0 0 0 httpd
[ 1462] 48 1462 88987 1055 0 0 0 httpd
[ 1463] 48 1463 89796 2560 0 0 0 httpd
[ 1464] 48 1464 91371 3558 0 0 0 httpd
[ 1465] 48 1465 88596 1283 0 0 0 httpd
[ 1466] 48 1466 90794 1253 0 0 0 httpd
[ 1469] 0 1469 106619 198 0 0 0
fail2ban-server
[ 1497] 0 1497 1015 1 0 0 0 mingetty
[ 1499] 0 1499 1015 1 0 0 0 mingetty
[ 1501] 0 1501 1015 1 0 0 0 mingetty
[ 1503] 0 1503 1015 1 0 0 0 mingetty
[ 1505] 0 1505 1015 1 0 0 0 mingetty
[ 1507] 0 1507 1015 1 0 0 0 mingetty
[ 1512] 0 1512 2674 0 0 -17 -1000 udevd
[ 1513] 0 1513 2674 0 0 -17 -1000 udevd
[ 1514] 0 1514 14249 1 0 0 0 login
[ 1520] 48 1520 89586 1139 0 0 0 httpd
[ 1522] 48 1522 90606 773 0 0 0 httpd
[ 1523] 48 1523 88343 2527 0 0 0 httpd
[ 1524] 0 1524 27075 1 0 0 0 bash
[ 1549] 48 1549 88937 1738 0 0 0 httpd
[ 1550] 48 1550 89721 1736 0 0 0 httpd
[ 1579] 48 1579 91106 600 0 0 0 httpd
[ 1581] 48 1581 88218 1491 0 0 0 httpd
[ 1582] 48 1582 88559 595 0 0 0 httpd
[ 1755] 48 1755 88115 1716 0 0 0 httpd
[ 2376] 48 2376 90860 1318 0 0 0 httpd
[ 2562] 48 2562 89680 2348 0 0 0 httpd
[ 2697] 48 2697 87846 2082 0 0 0 httpd
[ 2709] 48 2709 86589 3201 0 0 0 httpd
[ 2710] 48 2710 86056 2467 0 0 0 httpd
[ 2713] 48 2713 84422 1068 0 0 0 httpd
[ 2714] 48 2714 85544 2688 0 0 0 httpd
[ 2715] 48 2715 84748 1505 0 0 0 httpd
[ 2716] 48 2716 84569 1318 0 0 0 httpd
[ 2721] 48 2721 84815 1718 0 0 0 httpd
[ 2722] 48 2722 84502 1256 0 0 0 httpd
[ 2723] 48 2723 87014 4123 0 0 0 httpd
[ 2724] 48 2724 84502 1246 0 0 0 httpd
[ 2725] 48 2725 84420 1203 0 0 0 httpd
[ 2726] 48 2726 86782 4034 0 0 0 httpd
[ 2727] 48 2727 84502 1262 0 0 0 httpd
[ 2728] 48 2728 86054 2340 0 0 0 httpd
[ 2737] 48 2737 84815 1743 0 0 0 httpd
[ 2738] 48 2738 84748 1639 0 0 0 httpd
[ 2739] 48 2739 86589 3300 0 0 0 httpd
[ 2740] 48 2740 85202 2201 0 0 0 httpd
[ 2741] 48 2741 86054 1742 0 0 0 httpd
[ 2742] 48 2742 87014 4231 0 0 0 httpd
[ 2743] 48 2743 84301 711 0 0 0 httpd
[ 2744] 48 2744 84502 1233 0 0 0 httpd
[ 2745] 48 2745 84815 1614 0 0 0 httpd
[ 2746] 48 2746 84420 1113 0 0 0 httpd
[ 2747] 48 2747 84432 913 0 0 0 httpd
[ 2748] 48 2748 84301 724 0 0 0 httpd
[ 2749] 48 2749 86251 3541 0 0 0 httpd
[ 2750] 48 2750 84301 764 0 0 0 httpd
[ 2751] 48 2751 84422 1004 0 0 0 httpd
[ 2752] 48 2752 84291 588 0 0 0 httpd
[ 2758] 48 2758 84301 687 0 0 0 httpd
[ 2759] 48 2759 84301 725 0 0 0 httpd
[ 2760] 48 2760 84432 890 0 0 0 httpd
[ 2761] 48 2761 84301 788 0 0 0 httpd
[ 2765] 48 2765 84420 930 0 0 0 httpd
[ 2766] 48 2766 84430 868 0 0 0 httpd
[ 2769] 48 2769 84432 946 0 0 0 httpd
[ 2770] 48 2770 87526 8319 0 0 0 httpd
[ 2774] 48 2774 84291 582 0 0 0 httpd
[ 2775] 48 2775 84430 897 0 0 0 httpd
[ 2779] 48 2779 84301 692 0 0 0 httpd
[ 2780] 48 2780 84301 659 0 0 0 httpd
[ 2781] 48 2781 84430 1098 0 0 0 httpd
[ 2782] 48 2782 84420 2692 0 0 0 httpd
[ 2783] 48 2783 84430 823 0 0 0 httpd
[ 2784] 48 2784 84422 1062 0 0 0 httpd
[ 2785] 48 2785 84432 899 0 0 0 httpd
[ 2786] 48 2786 84502 1259 0 0 0 httpd
[ 2787] 48 2787 84301 733 0 0 0 httpd
[ 2788] 48 2788 84434 906 0 0 0 httpd
[ 2789] 48 2789 84432 1000 0 0 0 httpd
[ 2790] 48 2790 87526 8311 0 0 0 httpd
[ 2791] 48 2791 84430 1067 0 0 0 httpd
[ 2792] 48 2792 84504 1259 0 0 0 httpd
[ 2793] 48 2793 84301 749 0 0 0 httpd
[ 2794] 48 2794 84422 935 0 0 0 httpd
[ 2795] 48 2795 84303 740 0 0 0 httpd
[ 2796] 48 2796 84560 1417 0 0 0 httpd
[ 2797] 48 2797 84420 1091 0 0 0 httpd
[ 2798] 48 2798 84420 1023 0 0 0 httpd
[ 2799] 48 2799 84301 678 0 0 0 httpd
[ 2800] 48 2800 84301 643 0 0 0 httpd
[ 2809] 48 2809 84422 1017 0 0 0 httpd
[ 2811] 48 2811 84301 643 0 0 0 httpd
[ 2812] 48 2812 84301 620 0 0 0 httpd
[ 2813] 48 2813 84432 958 0 0 0 httpd
[ 2814] 48 2814 84432 928 0 0 0 httpd
[ 2815] 48 2815 84420 1126 0 0 0 httpd
[ 2816] 48 2816 87014 7940 0 0 0 httpd
[ 2817] 48 2817 84301 725 0 0 0 httpd
[ 2818] 48 2818 84301 695 0 0 0 httpd
[ 2819] 48 2819 84432 800 0 0 0 httpd
[ 2820] 48 2820 84420 1209 0 0 0 httpd
[ 2821] 48 2821 84502 1351 0 0 0 httpd
[ 2822] 48 2822 84815 1684 0 0 0 httpd
[ 2823] 48 2823 84301 702 0 0 0 httpd
[ 2824] 48 2824 84430 1416 0 0 0 httpd
[ 2825] 48 2825 84432 824 0 0 0 httpd
[ 2826] 48 2826 84432 899 0 0 0 httpd
[ 2827] 48 2827 84432 884 0 0 0 httpd
[ 2828] 48 2828 84301 794 0 0 0 httpd
[ 2829] 48 2829 84301 723 0 0 0 httpd
[ 2830] 48 2830 84301 1413 0 0 0 httpd
[ 2831] 48 2831 85331 3528 0 0 0 httpd
[ 2832] 48 2832 84420 3300 0 0 0 httpd
[ 2833] 48 2833 84430 1353 0 0 0 httpd
[ 2834] 48 2834 84432 897 0 0 0 httpd
[ 2835] 48 2835 84291 572 0 0 0 httpd
[ 2836] 48 2836 85460 3295 0 0 0 httpd
[ 2852] 48 2852 83773 4497 0 0 0 httpd
[ 2853] 48 2853 84814 2177 0 0 0 httpd
[ 2857] 48 2857 84420 1140 0 0 0 httpd
[ 2858] 48 2858 84432 925 0 0 0 httpd
[ 2859] 48 2859 84485 1568 0 0 0 httpd
[ 2860] 48 2860 84552 1749 0 0 0 httpd
[ 2861] 48 2861 84291 582 0 0 0 httpd
[ 2862] 48 2862 84420 3627 0 0 0 httpd
[ 2868] 48 2868 84301 804 0 0 0 httpd
[ 2869] 48 2869 84487 1268 0 0 0 httpd
[ 2870] 48 2870 84301 803 0 0 0 httpd
[ 2874] 48 2874 84422 1033 0 0 0 httpd
[ 2875] 48 2875 84301 718 0 0 0 httpd
[ 2876] 48 2876 84485 1125 0 0 0 httpd
[ 2877] 48 2877 84420 924 0 0 0 httpd
[ 2878] 48 2878 84748 2217 0 0 0 httpd
[ 2885] 48 2885 84301 2638 0 0 0 httpd
[ 2886] 48 2886 84420 905 0 0 0 httpd
[ 2887] 48 2887 84420 1558 0 0 0 httpd
[ 2888] 48 2888 84301 697 0 0 0 httpd
[ 2889] 48 2889 84502 1228 0 0 0 httpd
[ 2904] 48 2904 84430 791 0 0 0 httpd
[ 2915] 48 2915 84748 1518 0 0 0 httpd
[ 2916] 48 2916 85331 3691 0 0 0 httpd
[ 2917] 48 2917 84569 1634 0 0 0 httpd
[ 2918] 48 2918 84239 4498 0 0 0 httpd
[ 2919] 48 2919 84502 2969 0 0 0 httpd
[ 2920] 48 2920 86763 4460 0 0 0 httpd
[ 2921] 48 2921 84420 1133 0 0 0 httpd
[ 2922] 48 2922 86381 5779 0 0 0 httpd
[ 2928] 48 2928 84502 2436 0 0 0 httpd
[ 2938] 48 2938 85208 2426 0 0 0 httpd
[ 2939] 48 2939 86055 5438 0 0 0 httpd
[ 2940] 48 2940 84487 4186 0 0 0 httpd
[ 2944] 48 2944 84432 2877 0 0 0 httpd
[ 2945] 48 2945 84291 1446 0 0 0 httpd
[ 2946] 48 2946 84818 3977 0 0 0 httpd
[ 2960] 48 2960 84239 4311 0 0 0 httpd
[ 2961] 48 2961 89636 9983 0 0 0 httpd
[ 2962] 48 2962 87955 8183 0 0 0 httpd
[ 2970] 48 2970 84241 4777 0 0 0 httpd
[ 2973] 48 2973 84239 4759 0 0 0 httpd
[ 2974] 48 2974 80053 1296 0 0 0 httpd
[ 2975] 48 2975 80012 1410 0 0 0 httpd
[ 2976] 48 2976 80185 1606 0 0 0 httpd
[ 2978] 48 2978 79284 414 0 0 0 httpd
[ 2980] 48 2980 80185 1387 0 0 0 httpd
[ 2981] 48 2981 80021 1076 0 0 0 httpd
[ 2982] 48 2982 80181 1383 0 0 0 httpd
[ 2983] 48 2983 80138 1348 0 0 0 httpd
[ 2984] 48 2984 80202 1273 0 0 0 httpd
[ 2985] 48 2985 80010 418 0 0 0 httpd
[ 2986] 48 2986 79990 1409 0 0 0 httpd
[ 2988] 48 2988 80202 1540 0 0 0 httpd
[ 2989] 48 2989 79996 756 0 0 0 httpd
[ 2990] 48 2990 80010 1412 0 0 0 httpd
[ 2991] 48 2991 79986 789 0 0 0 httpd
[ 2992] 48 2992 79284 401 0 0 0 httpd
[ 2993] 48 2993 80189 1562 0 0 0 httpd
[ 2994] 48 2994 80119 1316 0 0 0 httpd
[ 3010] 48 3010 79283 425 0 0 0 httpd
[ 3013] 48 3013 80046 1291 0 0 0 httpd
[ 3016] 48 3016 79250 412 0 0 0 httpd
[ 3018] 48 3018 79283 451 0 0 0 httpd
[ 3022] 0 3022 33480 63 0 0 0 crond Out of memory: Kill process 1333 (mysqld) score 12 or sacrifice child Killed process 1333, UID 27, (mysqld) total-vm:748444kB, anon-rss:19232kB, file-rss:68kB
15 thoughts on - Memory Leak – How To Investigate
Jussi Hirvi wrote:
That’s a *lot* of apache. Is that really correct? Do you really need that many threads? How heavily is the webserver used?
Also, I see mysql running – does the website use it?
mark
—– Original Message —
Sever things could be occurring. The first thing I notice is that you have many httpd processes running. This can be useful if you have many simultaneous hits. If you don’t, you can tune the number of processes down
(search on MAX_CHILD in the httpd.conf). Don’t quote me on this, but you can lower the number of simultaneous processes and reduce the number of requests that each processes before exiting. Though much of the memory is shared, a lot isn’t, so reducing the process count helps improve the memory situation. Cycling them more rapidly can help clean up any that have memory leaks.
That doesn’t look like a lot of memory.. Possible to add another .5G or so?
Kwan Lowe wrote:
We’ve got a number of websites on one of our production servers, and they get hit moderately (it’s not Amazon… but they are US gov’t scientific research sites), and I think we’ve got 25 threads running, total, to server *all* of them.
Ah! I missed that. Is it actually the case that your server doesn’t even have 2G of RAM? That’s a *real* problem. If you’re not running it on a five year old desktop, you need to add – I’d say you shouldn’t be running with under 4GB of real memory.
mark “got 8G on my home ssytem, and 6G on my workstation at work”
I have 4 web servers. Every day I read the Logwatches, the ‘home-made’
web activity analysis reports and the instant emails created for every web access error (can’t seem to trap 500 though).
I allow the major crawlers like M$, Google, Yahoo, Yandex (Russian) and Facebook. I don’t block crawlers in robots.txt because updating it is time-consuming. Instead I block data centre IP ranges.
Every non-standard web access initiates a spontaneous emailed alert. 403
and 404 requests are automatically matched against a list of know hacking names. Identified matches causes the requesting IP to be automatically added to the monthly IP blocked list. The generated email, comprehensively full of technical details, is ready for copy and pasting into an email complaint if necessary.
Some well-known hacking names result in Apache re-directs to Chinese web sites.
PUTs are specifically allowed. Anyone trying OPTION, PUT in unauthorised, therefore unnecessary, sites and directories get their IP
added to the monthly blocked list. Usually hackers instantly switch to other compromised IPs and they get blocked too.
If you are serious about running a web server you have to know, daily, what is happening so you can react at the time. Waiting until everything grinds to a halt means you have failed. The good news is your awareness and monitoring can improve.
For every web site hosted my daily activity report shows summary totals for HTML and PHP pages accessed per site. It also lists, for every site, every IP address and the total of HTML and PHP pages individual IPs have accessed. Wading through long lists is boring but you can instantly spot potential problems.
Being a computer programmer means with HTML, CSS and PHP, I can know what is happening and respond to abuses with a full range of instantly deployable ‘tools’.
Its a learning curve and it does take time, but you’ll get better :-)
Small RAM limits with strange values like 1.3 GB are normal for VMs.
Rather than give the VM more dedicated RAM, have you tried adding more swap, Jussi? Your system may be well-tuned, not I/O bound all the time swapping to disk, but that doesn’t mean swap isn’t useful. Modern OSes pretty much depend on having some swap space. If nothing else, it lets the OS move some little-used bits of code out of RAM, so the RAM can be used for the computer’s real work.
Many web stacks are RAM pigs. I don’t mean “they can use a lot of RAM
productively,” I mean they’re based on inefficient or misapplied technologies that load a bunch of pointless things into RAM. Unless these things get swapped back out, they’re ballooning your VM for no useful purpose.
One web stack I used in the past had *GUI libraries* linked into its core executable. This, for software designed to run on headless VPSes!
Another thing to look into is how many forks or threads your web stack uses. More is better for speed, up to the point where you run out of RAM, at which point your web stack slows to a crawl or dies. If each fork takes 500 MB, and you’ve got it set to use 3 forks, you’re already running into swap space, unless there’s some serious RAM sharing going on among the forks.
That’s another aspect of web stacks being RAM pigs: it is *possible* to make a pre-forking web stack share a lot of RAM among the forks, but it doesn’t happen by itself. If you use naive defaults and naive development practices, you can end up with each fork being essentially independent copies of the whole web stack. This not only chews up RAM
to no good end, it means each fork takes longer to start, which hurts site performance.
Web site tuning is hard.
Yes, giving it a few dollars worth of RAM is the real fix. If you have to squeeze by for a while without it, try setting MaxRequestsPerChild to some reasonable (low thousands?) number in httpd.conf to clear out modules that have memory leaks or a lot of internal cache sooner. A new child process will share almost all memory with the parent, slowly growing as values change. This is especially bad for mod_perl or other embedded language modules if the language does reference counting.
Even so, modules that have reference-counted variables and objects will force blocks to copy-on-write as the references change even if the data values remain unchanged. But, just on general principles I’d blame mysql or something caching queries or result sets in the http clients as the real underlying culprit here. I’ve seen it go crazy on a 3 table join.
Looking at the non-shared memory in the list of procs posted, however, it looks like that server at least is running pretty thin.
Rather than give the VM more dedicated RAM, have you tried adding more
Interesting:
21463 total pagecache pages
20882 pages in swap cache Swap cache stats: add 1523887, delete 1503005, find 201987/297332
Free swap = 0kB
Total swap = 3014648kB
In this case I’m more inclined to limit the number of processes rather than increasing page space. Adding swap will delay the OOM, which is good, but performance will suffer.
Yes! You don’t know how many times I’ve heard, “I have 64G of memory! Why do I need swap?” There are some benefits to running swapless though. I’ve dinked around with the overcommit and other memory options for some workloads.
“increase performance”…
:P
If you don’t mind me asking, what are your fork/child settings like for those and what sort of workload?
Octo-core AMD and 64G DDR3, Nvidia 780. :D
Kwan Lowe wrote:
For a very crude estimate, in /var/log/httpd, I did grep GET access_*log |
grep -c 03/Feb, and got 178388, and that’s with 23 workers (as in, ps -ef
| grep -c httpd).
mark
You could try tunning apache.. Start with MaxRequestPerChild, whichs sets a number of requests for child process before it is stopped. When a child is stopped, memory is freed. This could be your protection before running out of memory. KeepAlive is enabled? If yes, maybe you could try disabling it. KeepAlive speeds up your website, but uses twice as much of memory, because child processes need to keep connections opened, while waiting for new requests from established connections.
Cheers, Barbara
>>Mem: 1361564k total
> That doesn’t look like a lot of memory.. Possible to
> add another .5G or so?
Ok, I added RAM (remember, this is a VM). Now I got Mem: 2365180k total Swap: 3014648k total
This might help me some way. My experiences so far indicate that a webserver/nameserver should be ok with 2 gigs RAM plus some swap. I am not sure how much it depends on the amount of traffic.
– Jussi
Others have mentioned tuning httpd.conf parameters. Problem is apache doesn’t give you the math to know what to set those without lots of trial-and-error. The best guide for this math is from F5:
http://www.f5.com/pdf/deployment-guides/f5-apache-dg.pdf
In the past I’ve used webalizer logs to get an idea of the peak number of hits per minute. This works fine for certain types of accesses. For monitoring memory usage, I’ve used top in batch mode to grab the actual process memory usage. If the memory usage grows quickly and there are many spare httpd processes then the sum of these numbers will climb rapidly. In this case, tuning down the number of httpd processes and/or cycling them sooner can help. I’ve found that performance is not measurably affected in my workloads but mileage varies widely.