Hi everyone!
My problem started for no reason. Just at some moment httpd started dying with segfault.
First I upgraded to the 2.2.21 version (I was using 2.2.20) - it didn't help.
Second, I noticed that all my Perl sites are alive on that same server, so I've downloaded and recompiled PHP. It didnt help. Neither helped commenting PHP out of httpd.conf
Third, I enabled core file generation and applied gdb:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0 0xb7b08c11 in strstr () from /lib/libc.so.6
(gdb) where
#0 0xb7b08c11 in strstr () from /lib/libc.so.6
#1 0xb7a5a25d in sch_out_filter () from /usr/local/apache2/modules/mod_log.so
#2 0x08084068 in ap_remove_output_filter ()
#3 0x0807248e in default_handler ()
#4 0x08078926 in ap_run_child_init ()
#5 0x08079068 in ap_invoke_handler ()
#6 0x08095bd7 in ap_die ()
#7 0x08092a21 in ap_process_http_async_connection ()
#8 0x0808027f in ap_run_create_connection ()
#9 0x080806c0 in ap_lingering_close ()
#10 0x080add18 in make_child ()
#11 0x080aded5 in perform_idle_server_maintenance ()
#12 0x080ae0f1 in ap_mpm_run ()
#13 0x080ae55c in ap_mpm_run ()
#14 0x08062ea9 in main ()
The problem is, I have analysed about 10 different types of core file. One kind of them had only 1 string in stack, about "memory cannot be accessed".
Some of them were mentioning different functions, meaningless to me as to non-developer. But the one I've given above is what httpd produces in 100% of cases for the las 6 hours.
I became suspicious of that mod_log.so, so I just commented it out in httpd.conf - an voila! Everything's working now, and besides all logs are alive despite absent mod_log
Then I remembered that when I was updating from 2.2.10 to 2.2.20 `apachectl -t` told me about 10+ modules that were not DSO any more, so I had to comment them out.
My guess is that for some reason my installation became unaware of module conflict and I came to situation when I had both embedded and shared mod_log at the same time.
But, of course, I may be wrong.
Anyways, my problem is gone, so this is more of FYI if someone finds this useful. Also I attach httpd-generated stack trace from error_log, when it dies with signal 6
P.S. still wondering why all the Perl sites were OK while all PHP sites were dead. Long live mod_perl for its memory management?
BR
*** glibc detected *** /usr/local/apache2/bin/httpd: malloc(): memory corruption: 0x0867da78 *** ======= Backtrace: ========= /lib/libc.so.6[0xb7b02357] /lib/libc.so.6(__libc_malloc+0x7e)[0xb7b03b7e] /usr/lib/libapr-1.so.0(apr_palloc+0x179)[0xb7c45ec9] /usr/local/apache2/modules/mod_deflate.so[0xb7a77019] /usr/local/apache2/bin/httpd(ap_pass_brigade+0x90)[0x8084068] /usr/local/apache2/modules/mod_log.so[0xb7a5a777] /usr/local/apache2/bin/httpd(ap_pass_brigade+0x90)[0x8084068] /usr/local/apache2/bin/httpd(ap_filter_flush+0x1a)[0x8084206] /usr/lib/libaprutil-1.so.0(apr_brigade_write+0x94)[0xb7f47554] /usr/local/apache2/bin/httpd[0x806b70e] /usr/local/apache2/bin/httpd(ap_rwrite+0x32)[0x806b806] /usr/local/apache2/modules/libphp5.so[0xb78cea2d] /usr/local/apache2/modules/libphp5.so(php_ub_body_write_no_header+0x31)[0xb7828711] /usr/local/apache2/modules/libphp5.so(php_end_ob_buffer+0x197)[0xb7828db7] /usr/local/apache2/modules/libphp5.so(zif_ob_end_flush+0x39)[0xb78291f9] /usr/local/apache2/modules/libphp5.so[0xb787db38] /usr/local/apache2/modules/libphp5.so(execute+0x129)[0xb7872609] /usr/local/apache2/modules/libphp5.so[0xb787d42a] /usr/local/apache2/modules/libphp5.so(execute+0x129)[0xb7872609] /usr/local/apache2/modules/libphp5.so(zend_execute_scripts+0x168)[0xb7856178] /usr/local/apache2/modules/libphp5.so(php_execute_script+0x1ae)[0xb78191de] /usr/local/apache2/modules/libphp5.so[0xb78cf2dc] /usr/local/apache2/bin/httpd(ap_run_handler+0x46)[0x8078926] /usr/local/apache2/bin/httpd(ap_invoke_handler+0x147)[0x8079068] /usr/local/apache2/bin/httpd(ap_process_request+0x61)[0x8095bd7] /usr/local/apache2/bin/httpd[0x8092a21] /usr/local/apache2/bin/httpd(ap_run_process_connection+0x46)[0x808027f] /usr/local/apache2/bin/httpd(ap_process_connection+0x58)[0x80806c0] /usr/local/apache2/bin/httpd[0x80add18] /usr/local/apache2/bin/httpd[0x80aded5] /usr/local/apache2/bin/httpd[0x80ae0f1] /usr/local/apache2/bin/httpd(ap_mpm_run+0x423)[0x80ae55c] /usr/local/apache2/bin/httpd(main+0x9f1)[0x8062ea9] /lib/libc.so.6(__libc_start_main+0xd8)[0xb7aafdf8] /usr/local/apache2/bin/httpd(apr_dso_error+0x41)[0x8061ca1] ======= Memory map: ======== 08048000-080cc000 r-xp 00000000 09:00 359058 /usr/local/apache2/bin/httpd (deleted) 080cc000-080cf000 rw-p 00084000 09:00 359058 /usr/local/apache2/bin/httpd (deleted) 080cf000-08706000 rw-p 080cf000 00:00 0 [heap] b6e00000-b6e21000 rw-p b6e00000 00:00 0 b6e21000-b6f00000 ---p b6e21000 00:00 0 b6f68000-b6f72000 r-xp 00000000 09:00 593566 /usr/lib/libgcc_s.so.1 b6f72000-b6f73000 rw-p 00009000 09:00 593566 /usr/lib/libgcc_s.so.1 b6f73000-b70b9000 r-xp 00000000 09:00 361795 /usr/local/apache2/modules/mod_perl.so b70b9000-b70be000 rw-p 00145000 09:00 361795 /usr/local/apache2/modules/mod_perl.so b70be000-b70c0000 rw-p b70be000 00:00 0 b70c0000-b70f3000 r--p 00000000 09:00 719846 /usr/lib/locale/en_US/LC_CTYPE b70f3000-b70fc000 r-xp 00000000 09:00 65284 /lib/libnss_files-2.5.so b70fc000-b70fe000 rw-p 00008000 09:00 65284 /lib/libnss_files-2.5.so b70fe000-b7106000 r-xp 00000000 09:00 71312 /lib/libnss_nis-2.5.so b7106000-b7108000 rw-p 00007000 09:00 71312 /lib/libnss_nis-2.5.so b7108000-b710f000 r-xp 00000000 09:00 65314 /lib/libnss_compat-2.5.so b710f000-b7111000 rw-p 00006000 09:00 65314 /lib/libnss_compat-2.5.so b7186000-b71b9000 r--p 00000000 09:00 719898 /usr/lib/locale/ru_RU.cp1251/LC_CTYPE b71b9000-b71ba000 r--p 00000000 09:00 721175 /usr/lib/locale/ru_RU.cp1251/LC_NUMERIC b71ba000-b71bb000 r--p 00000000 09:00 721172 /usr/lib/locale/ru_RU.cp1251/LC_TIME b71bb000-b71c1000 r--p 00000000 09:00 721168 /usr/lib/locale/ru_RU.cp1251/LC_COLLATE b71c1000-b7242000 rw-p b71c1000 00:00 0 b7242000-b7252000 rw-s 00000000 00:07 466882 /dev/zero (deleted) b7252000-b7254000 r-xp 00000000 09:00 71288 /lib/libutil-2.5.so b7254000-b7256000 rw-p 00001000 09:00 71288 /lib/libutil-2.5.so b7256000-b7369000 r-xp 00000000 09:00 593477 /usr/lib/libxml2.so.2.6.28 b7369000-b736e000 rw-p 00113000 09:00 593477 /usr/lib/libxml2.so.2.6.28 b736e000-b736f000 rw-p b736e000 00:00 0 b736f000-b738e000 r-xp 00000000 09:00 593243 /usr/lib/libjpeg.so.62.0.0 b738e000-b738f000 rw-p 0001e000 09:00 593243 /usr/lib/libjpeg.so.62.0.0 b738f000-b73a2000 r-xp 00000000 09:00 71282 /lib/libnsl-2.5.so b73a2000-b73a4000 rw-p 00012000 09:00 71282 /lib/libnsl-2.5.so b73a4000-b73a6000 rw-p b73a4000 00:00 0 b73a7000-b73a8000 r--p 00000000 09:00 721173 /usr/lib/locale/ru_RU.cp1251/LC_MONETARY b73a8000-b73bb000 r-xp 00000000 09:00 592732 /usr/lib/libz.so.1.2.3 b73bb000-b73bc000 rw-p 00012000 09:00 592732 /usr/lib/libz.so.1.2.3 b73bc000-b73bd000 r--p 00000000 09:00 721167 /usr/lib/locale/ru_RU.cp1251/LC_MESSAGES/SYS_LC_MESSAGES b73bd000-b73be000 r--p 00000000 09:00 719910 /usr/lib/locale/ru_RU.cp1251/LC_PAPER b73be000-b73bf000 r--p 00000000 09:00 721171 /usr/lib/locale/ru_RU.cp1251/LC_NAME b73bf000-b73c0000 r--p 00000000 09:00 721169 /usr/lib/locale/ru_RU.cp1251/LC_ADDRESS b73c0000-b73c1000 r--p 00000000 09:00 721170 /usr/lib/locale/ru_RU.cp1251/LC_TELEPHONE b73c1000-b73c2000 r--p 00000000 09:00 719907 /usr/lib/locale/ru_RU.cp1251/LC_MEASUREMENT b73c2000-b73c9000 r--s 00000000 09:00 658501 /usr/lib/gconv/gconv-modules.cache b73c9000-b73ca000 r--p 00000000 09:00 721174 /usr/lib/locale/ru_RU.cp1251/LC_IDENTIFICATION b73ca000-b73cb000 r--p 00000000 09:00 719976 /usr/lib/locale/en_US/LC_NUMERIC b73cb000-b73cc000 r--p 00000000 09:00 720064 /usr/lib/locale/en_US/LC_TIME b73cc000-b73cd000 r--p 00000000 09:00 720065 /usr/lib/locale/en_US/LC_MONETARY b73cd000-b73ce000 r--p 00000000 09:00 719970 /usr/lib/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES b73ce000-b73cf000 r--p 00000000 09:00 720068 /usr/lib/locale/en_US/LC_PAPER b73cf000-b73d0000 r--p 00000000 09:00 720063 /usr/lib/locale/en_US/LC_NAME b73d0000-b73d1000 r--p 00000000 09:00 720061 /usr/lib/locale/en_US/LC_ADDRESS b73d1000-b73d2000 r--p 00000000 09:00 720062 /usr/lib/locale/en_US/LC_TELEPHONE b73d2000-b73d3000 r--p 00000000 09:00 720066 /usr/lib/locale/en_US/LC_MEASUREMENT b73d3000-b73d4000 r--p 00000000 09:00 720067 /usr/lib/locale/en_US/LC_IDENTIFICATION b73d4000-b73f6000 r-xp 00000000 09:00 462107 /usr/local/lib/libpng12.so.0.22.0 b73f6000-b73f7000 rw-p 00021000 09:00 462107 /usr/local/lib/libpng12.so.0.22.0 b73f7000-b74d5000 r-xp 00000000 09:00 466357 /usr/local/lib/libiconv.so.2.5.0 b74d5000-b74d6000 rw-p 000de000 09:00 466357 /usr/local/lib/libiconv.so.2.5.0 b74d6000-b751d000 r-xp 00000000 09:00 32912 /usr/local/lib/mysql/libmysqlclient.so.16.0.0 b751d000-b754d000 rw-p 00047000 09:00 32912 /usr/local/lib/mysql/libmysqlclient.so.16.0.0 b754d000-b754f000 rw-p b754d000 00:00 0 b754f000-b79bf000 r-xp 00000000 09:00 359071 /usr/local/apache2/modules/libphp5.so b79bf000-b79e1000 rw-p 00470000 09:00 359071 /usr/local/apache2/modules/libphp5.so b79e1000-b79ea000 rw-p b79e1000 00:00 0 b79ea000-b79f8000 r-xp 00000000 09:00 359083 /usr/local/apache2/modules/mod_rewrite.so b79f8000-b79f9000 rw-p 0000d000 09:00 359083 /usr/local/apache2/modules/mod_rewrite.so b79f9000-b79fb000 r-xp 00000000 09:00 359088 /usr/local/apache2/modules/mod_speling.so b79fb000-b79fc000 rw-p 00001000 09:00 359088 /usr/local/apache2/modules/mod_speling.so b79fc000-b79ff000 r-xp 00000000 09:00 359095 /usr/local/apache2/modules/mod_imagemap.so b79ff000-b7a00000 rw-p 00002000 09:00 359095 /usr/local/apache2/modules/mod_imagemap.so b7a00000-b7a02000 r-xp 00000000 09:00 359089 /usr/local/apache2/modules/mod_vhost_alias.so b7a02000-b7a03000 rw-p 00001000 09:00 359089 /usr/local/apache2/modules/mod_vhost_alias.so b7a03000-b7a0d000 r-xp 00000000 09:00 359104 /usr/local/apache2/modules/mod_dav_fs.so b7a0d000-b7a0e000 rw-p 0000a000 09:00 359104 /usr/local/apache2/modules/mod_dav_fs.so b7a0e000-b7a12000 r-xp 00000000 09:00 359074 /usr/local/apache2/modules/mod_info.so b7a12000-b7a13000 rw-p 00003000 09:00 359074 /usr/local/apache2/modules/mod_info.so b7a13000-b7a28000 r-xp 00000000 09:00 359114 /usr/local/apache2/modules/mod_dav.so b7a28000-b7a29000 rw-p 00015000 09:00 359114 /usr/local/apache2/modules/mod_dav.so b7a29000-b7a4f000 r-xp 00000000 09:00 359106 /usr/local/apache2/modules/mod_ssl.so b7a4f000-b7a51000 rw-p 00025000 09:00 359106 /usr/local/apache2/modules/mod_ssl.so b7a51000-b7a53000 rw-p b7a51000 00:00 0 b7a53000-b7a55000 r-xp 00000000 09:00 359093 /usr/local/apache2/modules/mod_unique_id.so b7a55000-b7a56000 rw-p 00001000 09:00 359093 /usr/local/apache2/modules/mod_unique_id.so b7a56000-b7a58000 r-xp 00000000 09:00 359092 /usr/local/apache2/modules/mod_usertrack.so b7a58000-b7a59000 rw-p 00001000 09:00 359092 /usr/local/apache2/modules/mod_usertrack.so b7a59000-b7a5b000 r-xp 00000000 09:00 1207696 /usr/local/apache2/modules/mod_log.so b7a5b000-b7a5c000 rw-p 00001000 09:00 1207696 /usr/local/apache2/modules/mod_log.so b7a5c000-b7a5e000 r-xp 00000000 09:00 359096 /usr/local/apache2/modules/mod_ident.so b7a5e000-b7a5f000 rw-p 00001000 09:00 359096 /usr/local/apache2/modules/mod_ident.so b7a5f000-b7a62000 r-xp 00000000 09:00 359082 /usr/local/apache2/modules/mod_headers.so b7a62000-b7a63000 rw-p 00002000 09:00 359082 /usr/local/apache2/modules/mod_headers.so b7a63000-b7a65000 r-xp 00000000 09:00 359111 /usr/local/apache2/modules/mod_expires.so b7a65000-b7a66000 rw-p 00001000 09:00 359111 /usr/local/apache2/modules/mod_expires.so b7a66000-b7a68000 r-xp 00000000 09:00 359081 /usr/local/apache2/modules/mod_cern_meta.so b7a68000-b7a69000 rw-p 00001000 09:00 359081 /usr/local/apache2/modules/mod_cern_meta.so b7a69000-b7a6e000 r-xp 00000000 09:00 359122 /usr/local/apache2/modules/mod_mime_magic.so b7a6e000-b7a6f000 rw-p 00004000 09:00 359122 /usr/local/apache2/modules/mod_mime_magic.so b7a6f000-b7a70000 r-xp 00000000 09:00 359119 /usr/local/apache2/modules/mod_logio.so b7a70000-b7a71000 rw-p 00000000 09:00 359119 /usr/local/apache2/modules/mod_logio.so b7a71000-b7a73000 r-xp 00000000 09:00 359101 /usr/local/apache2/modules/mod_log_forensic.so b7a73000-b7a74000 rw-p 00001000 09:00 359101 /usr/local/apache2/modules/mod_log_forensic.so b7a74000-b7a87000 r-xp 00000000 09:00 359077 /usr/local/apache2/modules/mod_deflate.so b7a87000-b7a88000 rw-p 00012000 09:00 359077 /usr/local/apache2/modules/mod_deflate.so b7a88000-b7a8c000 r-xp 00000000 09:00 359112 /usr/local/apache2/modules/mod_ext_filter.so b7a8c000-b7a8d000 rw-p 00003000 09:00 359112 /usr/local/apache2/modules/mod_ext_filter.so b7a8d000-b7a90000 r-xp 00000000 09:00 359091 /usr/local/apache2/modules/mod_dbd.so b7a90000-b7a91000 rw-p 00003000 09:00 359091 /usr/local/apache2/modules/mod_dbd.so b7a91000-b7a97000 r-xp 00000000 09:00 359116 /usr/local/apache2/modules/mod_auth_digest.so b7a97000-b7a98000 rw-p 00006000 09:00 359116 /usr/local/apache2/modules/mod_auth_digest.so b7a98000-b7a9a000 rw-p b7a98000 00:00 0 b7a9a000-b7bd6000 r-xp 00000000 09:00 65288 /lib/libc-2.5.so b7bd6000-b7bd7000 r--p 0013c000 09:00 65288 /lib/libc-2.5.so b7bd7000-b7bd9000 rw-p 0013d000 09:00 65288 /lib/libc-2.5.so b7bd9000-b7bdd000 rw-p b7bd9000 00:00 0 b7bdd000-b7bdf000 r-xp 00000000 09:00 65309 /lib/libdl-2.5.so b7bdf000-b7be1000 rw-p 00001000 09:00 65309 /lib/libdl-2.5.so b7be1000-b7bf4000 r-xp 00000000 09:00 71277 /lib/libpthread-2.5.so b7bf4000-b7bf6000 rw-p 00012000 09:00 71277 /lib/libpthread-2.5.so b7bf6000-b7bf8000 rw-p b7bf6000 00:00 0 b7bf8000-b7bfd000 r-xp 00000000 09:00 71283 /lib/libcrypt-2.5.so b7bfd000-b7bff000 rw-p 00004000 09:00 71283 /lib/libcrypt-2.5.so b7bff000-b7c26000 rw-p b7bff000 00:00 0 b7c26000-b7c2c000 r-xp 00000000 09:00 71285 /lib/librt-2.5.so b7c2c000-b7c2e000 rw-p 00005000 09:00 71285 /lib/librt-2.5.so b7c2e000-b7c30000 r-xp 00000000 09:00 65291 /lib/libuuid.so.1.2 b7c30000-b7c31000 rw-p 00001000 09:00 65291 /lib/libuuid.so.1.2 b7c31000-b7c53000 r-xp 00000000 09:00 593395 /usr/lib/libapr-1.so.0.2.8 b7c53000-b7c54000 rw-p 00022000 09:00 593395 /usr/lib/libapr-1.so.0.2.8 b7c54000-b7c55000 rw-p b7c54000 00:00 0 b7c55000-b7c73000 r-xp 00000000 09:00 593737 /usr/lib/libexpat.so.1.5.2 b7c73000-b7c75000 rw-p 0001d000 09:00 593737 /usr/lib/libexpat.so.1.5.2 b7c75000-b7d6c000 r-xp 00000000 09:00 65348 /lib/libdb-4.4.so b7d6c000-b7d6f000 rw-p 000f6000 09:00 65348 /lib/libdb-4.4.so b7d6f000-b7d7a000 r-xp 00000000 09:00 593285 /usr/lib/liblber-2.3.so.0.2.20 b7d7a000-b7d7b000 rw-p 0000a000 09:00 593285 /usr/lib/liblber-2.3.so.0.2.20 b7d7b000-b7e8d000 r-xp 00000000 09:00 593102 /usr/lib/libcrypto.so.0.9.8 b7e8d000-b7ea2000 rw-p 00111000 09:00 593102 /usr/lib/libcrypto.so.0.9.8 b7ea2000-b7ea5000 rw-p b7ea2000 00:00 0 b7ea5000-b7edc000 r-xp 00000000 09:00 592936 /usr/lib/libssl.so.0.9.8 b7edc000-b7ee0000 rw-p 00036000 09:00 592936 /usr/lib/libssl.so.0.9.8 b7ee0000-b7eef000 r-xp 00000000 09:00 71284 /lib/libresolv-2.5.so b7eef000-b7ef1000 rw-p 0000f000 09:00 71284 /lib/libresolv-2.5.so b7ef1000-b7ef4000 rw-p b7ef1000 00:00 0 b7ef4000-b7f0a000 r-xp 00000000 09:00 592711 /usr/lib/libsasl2.so.2.0.22 b7f0a000-b7f0b000 rw-p 00015000 09:00 592711 /usr/lib/libsasl2.so.2.0.22 b7f0b000-b7f3f000 r-xp 00000000 09:00 593625 /usr/lib/libldap-2.3.so.0.2.20 b7f3f000-b7f40000 rw-p 00034000 09:00 593625 /usr/lib/libldap-2.3.so.0.2.20 b7f40000-b7f56000 r-xp 00000000 09:00 592767 /usr/lib/libaprutil-1.so.0.2.8 b7f56000-b7f57000 rw-p 00016000 09:00 592767 /usr/lib/libaprutil-1.so.0.2.8 b7f57000-b7f7c000 r-xp 00000000 09:00 71329 /lib/libm-2.5.so b7f7c000-b7f7e000 rw-p 00024000 09:00 71329 /lib/libm-2.5.so b7f7e000-b7f80000 r-xp 00000000 09:00 359072 /usr/local/apache2/modules/mod_dumpio.so b7f80000-b7f81000 rw-p 00001000 09:00 359072 /usr/local/apache2/modules/mod_dumpio.so b7f81000-b7f82000 r-xp 00000000 09:00 359117 /usr/local/apache2/modules/mod_authz_owner.so b7f82000-b7f83000 rw-p 00001000 09:00 359117 /usr/local/apache2/modules/mod_authz_owner.so b7f83000-b7f85000 r-xp 00000000 09:00 359086 /usr/local/apache2/modules/mod_authz_dbm.so b7f85000-b7f86000 rw-p 00001000 09:00 359086 /usr/local/apache2/modules/mod_authz_dbm.so b7f86000-b7f87000 r-xp 00000000 09:00 359118 /usr/local/apache2/modules/mod_authn_dbd.so b7f87000-b7f88000 rw-p 00001000 09:00 359118 /usr/local/apache2/modules/mod_authn_dbd.so b7f88000-b7f89000 r-xp 00000000 09:00 359079 /usr/local/apache2/modules/mod_authn_anon.so b7f89000-b7f8a000 rw-p 00000000 09:00 359079 /usr/local/apache2/modules/mod_authn_anon.so b7f8a000-b7f8b000 r-xp 00000000 09:00 359099 /usr/local/apache2/modules/mod_authn_dbm.so b7f8b000-b7f8c000 rw-p 00000000 09:00 359099 /usr/local/apache2/modules/mod_authn_dbm.so b7f8c000-b7f8d000 rw-p b7f8c000 00:00 0 b7f8d000-b7fa8000 r-xp 00000000 09:00 71268 /lib/ld-2.5.so b7fa8000-b7faa000 rw-p 0001b000 09:00 71268 /lib/ld-2.5.so bfdd2000-bfde6000 rwxp bfdd2000 00:00 0 [stack] bfde6000-bfde7000 rw-p bfde6000 00:00 0 ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso] [Mon Sep 19 17:25:59 2011] [notice] child pid 16696 exit signal Aborted (6), possible coredump in /tmp
--------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|