php/zend and apache segmentation faults

[tariq wali <ganaiwali@xxxxxxxxx>]

Hi,

I am new to the group here and would quickly like to understand if this is the right place to share questions on php/zend ? Just to give an idea we at info edge ltd run most of the web properties on LAMP stack and use various versions of PHP(5.2 to latest 5.4.x) with Zend Engine .

Evidently we get a lot of segmentation faults as logged by apache in error logs and when we try to debug them they more or less point issues to zend . Although I am not a programmer and probably do not understand the exact cause of segfaults but I am willing to believe this is something to do with PHP . we have eliminated all doubts on hardware/apache file system and so forth . Here is a quick output of gdb backtrace on one of the cores that apache wrote on a segfault .

This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so..1".

Core was generated by `/usr/local/apache/bin/httpd -k start'.

Program terminated with signal 11, Segmentation fault.

Reading symbols from /usr/lib/libz.so.1...done.

Loaded symbols for /usr/lib/libz.so.1

Reading symbols from /lib/libssl.so.4...done.

Loaded symbols for /lib/libssl.so.4

Reading symbols from /lib/libcrypto.so.4...done.

Loaded symbols for /lib/libcrypto.so.4

Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.

Loaded symbols for /usr/lib/libgssapi_krb5.so.2

Reading symbols from /usr/lib/libkrb5.so.3...done.

Loaded symbols for /usr/lib/libkrb5.so.3

Reading symbols from /lib/libcom_err.so.2...done.

Loaded symbols for /lib/libcom_err.so.2

Reading symbols from /usr/lib/libk5crypto.so.3...done.

Loaded symbols for /usr/lib/libk5crypto.so.3

Reading symbols from /lib/libresolv.so.2...done.

Loaded symbols for /lib/libresolv.so.2

Reading symbols from /usr/local/apache/lib/libaprutil-0.so.0...done.

Loaded symbols for /usr/local/apache/lib/libaprutil-0.so.0

Reading symbols from /usr/lib/libgdbm.so.2...done.

Loaded symbols for /usr/lib/libgdbm.so.2

Reading symbols from /usr/lib/tls/i686/libdb-4.2.so...done.

Loaded symbols for /usr/lib/tls/i686/libdb-4.2.so

Reading symbols from /usr/lib/libexpat.so.0...done.

Loaded symbols for /usr/lib/libexpat.so.0

Reading symbols from /usr/local/apache/lib/libapr-0.so.0...done.

Loaded symbols for /usr/local/apache/lib/libapr-0.so.0

Reading symbols from /lib/tls/librt.so.1...done.

Loaded symbols for /lib/tls/librt.so.1

Reading symbols from /lib/tls/libm.so.6...done.

Loaded symbols for /lib/tls/libm.so.6

Reading symbols from /lib/libcrypt.so.1...done.

Loaded symbols for /lib/libcrypt.so.1

Reading symbols from /lib/libnsl.so.1...done.

Loaded symbols for /lib/libnsl.so.1

Reading symbols from /lib/tls/libpthread.so..0...done.

Loaded symbols for /lib/tls/libpthread.so.0

Reading symbols from /lib/libdl.so.2...done.

Loaded symbols for /lib/libdl.so.2

Reading symbols from /lib/tls/libc.so.6...done.

Loaded symbols for /lib/tls/libc.so.6

Reading symbols from /lib/ld-linux.so.2...done.

Loaded symbols for /lib/ld-linux.so.2

Reading symbols from /lib/libnss_files.so.2...done.

Loaded symbols for /lib/libnss_files.so.2

Reading symbols from /usr/local/apache/modules/libphp5.so....done.

Loaded symbols for /usr/local/apache/modules/libphp5.so

Reading symbols from /usr/local/mysql/lib/mysql/libmysqlclient.so.15...done.

Loaded symbols for /usr/local/mysql/lib/mysql/libmysqlclient.so.15

Reading symbols from /usr/lib/libpng12.so.0...done.

Loaded symbols for /usr/lib/libpng12.so.0

Reading symbols from /usr/lib/libjpeg.so.62...done.

Loaded symbols for /usr/lib/libjpeg.so.62

Reading symbols from /usr/lib/libcurl.so.3...done.

Loaded symbols for /usr/lib/libcurl.so.3

Reading symbols from /usr/lib/libidn.so.11...done.

Loaded symbols for /usr/lib/libidn.so.11

Reading symbols from /usr/lib/libxml2.so.2...done.

Loaded symbols for /usr/lib/libxml2.so.2

Reading symbols from /usr/local/apache/modules/mod_expires.so...done.

Loaded symbols for /usr/local/apache/modules/mod_expires.so

Reading symbols from /usr/local/apache/modules/mod_headers.so...done.

Loaded symbols for /usr/local/apache/modules/mod_headers.so

Reading symbols from /usr/local/apache/modules/mod_rpaf-2.0.so...done.

Loaded symbols for /usr/local/apache/modules/mod_rpaf-2.0.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/xcache.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/xcache.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/pdo.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/pdo.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/pdo_mysql.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/pdo_mysql.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/memcache.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/memcache.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/wordmapping.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/wordmapping.so

Reading symbols from /usr/lib/libstdc++.so.6...done.

Loaded symbols for /usr/lib/libstdc++.so.6

Reading symbols from /lib/libgcc_s.so.1...done.

Loaded symbols for /lib/libgcc_s.so.1

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/oauth.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/oauth.so

Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/mcrypt.so...done.

Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/mcrypt.so

Reading symbols from /usr/local/lib/libmcrypt.so.4...done.

Loaded symbols for /usr/local/lib/libmcrypt.so.4

Reading symbols from /usr/lib/libltdl.so.3...done.

Loaded symbols for /usr/lib/libltdl.so.3

Reading symbols from /lib/libnss_dns.so.2...done.

Loaded symbols for /lib/libnss_dns.so.2

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

832 if (UNEXPECTED(prev->next_free_block != mm_block) || UNEXPECTED(next->prev_free_block != mm_block)) {

(gdb) 

if (UNEXPECTED(prev->next_free_block != mm_block) || UNEXPECTED(next->prev_free_block != mm_block)) {

###########analyse this one from bottom to top

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

#1  0x00582ea1 in destroy_op_array (op_array=0xafbfbf04) at /opt/php-5.2.6/Zend/zend_opcode.c:234

#2  0x005954c4 in zend_hash_destroy (ht=0xafc7408c) at /opt/php-5.2.6/Zend/zend_hash.c:526

#3  0x00583162 in destroy_zend_class (pce=0x8e1fbec) at /opt/php-5.2.6/Zend/zend_opcode.c:186

#4  0x00595698 in zend_hash_apply_deleter (ht=0x8c89850, p=0x8e1fbe0) at /opt/php-5.2.6/Zend/zend_hash.c:611

#5  0x00595947 in zend_hash_reverse_apply (ht=0x8c89850, apply_func=0x57f260 <clean_non_persistent_class>)

    at /opt/php-5.2.6/Zend/zend_hash.c:760

#6  0x0057f8fd in shutdown_executor () at /opt/php-5.2.6/Zend/zend_execute_API.c:291

#7  0x0058c0f8 in zend_deactivate () at /opt/php-5.2.6/Zend/zend..c:860

#8  0x0055454a in php_request_shutdown (dummy=0x0) at /opt/php-5.2..6/main/main.c:1486

#9  0x006089de in php_handler (r=0x8df6230) at /opt/php-5.2.6/sapi/apache2handler/sapi_apache2.c:469

#10 0x0809b54e in ap_run_handler (r=0x8df6230) at config.c:152

#11 0x0809b919 in ap_invoke_handler (r=0x8df6230) at config.c:364

#12 0x0808468d in ap_process_request (r=0x8df6230) at http_request.c:249

#13 0x080801d9 in ap_process_http_connection (c=0x8debff8) at http_core.c:251

#14 0x080a4ae6 in ap_run_process_connection (c=0x8debff8) at connection.c:43

#15 0x08099d29 in child_main (child_num_arg=Variable "child_num_arg" is not available.

) at prefork.c:610

#16 0x08099f53 in make_child (s=Variable "s" is not available.

) at prefork.c:704

#17 0x0809a8b9 in ap_mpm_run (_pconf=0x8c2f0a8, plog=0x8c67188, s=0x8c34618) at prefork.c:839

#18 0x0809f887 in main (argc=3, argv=0xbfe03244) at main.c:618

(gdb) dump_bt executor_globals.current_execute_data

(gdb) bt

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

#1  0x00582ea1 in destroy_op_array (op_array=0xafbfbf04) at /opt/php-5.2.6/Zend/zend_opcode.c:234

#2  0x005954c4 in zend_hash_destroy (ht=0xafc7408c) at /opt/php-5.2.6/Zend/zend_hash.c:526

#3  0x00583162 in destroy_zend_class (pce=0x8e1fbec) at /opt/php-5..2.6/Zend/zend_opcode.c:186

#4  0x00595698 in zend_hash_apply_deleter (ht=0x8c89850, p=0x8e1fbe0) at /opt/php-5.2.6/Zend/zend_hash.c:611

#5  0x00595947 in zend_hash_reverse_apply (ht=0x8c89850, apply_func=0x57f260 <clean_non_persistent_class>)

    at /opt/php-5.2.6/Zend/zend_hash.c:760

#6  0x0057f8fd in shutdown_executor () at /opt/php-5.2.6/Zend/zend_execute_API.c:291

#7  0x0058c0f8 in zend_deactivate () at /opt/php-5.2.6/Zend/zend.c:860

#8  0x0055454a in php_request_shutdown (dummy=0x0) at /opt/php-5.2.6/main/main.c:1486

#9  0x006089de in php_handler (r=0x8df6230) at /opt/php-5.2.6/sapi/apache2handler/sapi_apache2.c:469

#10 0x0809b54e in ap_run_handler (r=0x8df6230) at config.c:152

#11 0x0809b919 in ap_invoke_handler (r=0x8df6230) at config.c:364

#12 0x0808468d in ap_process_request (r=0x8df6230) at http_request.c:249

#13 0x080801d9 in ap_process_http_connection (c=0x8debff8) at http_core.c:251

#14 0x080a4ae6 in ap_run_process_connection (c=0x8debff8) at connection.c:43

#15 0x08099d29 in child_main (child_num_arg=Variable "child_num_arg" is not available.

) at prefork.c:610

#16 0x08099f53 in make_child (s=Variable "s" is not available.

) at prefork.c:704

#17 0x0809a8b9 in ap_mpm_run (_pconf=0x8c2f0a8, plog=0x8c67188, s=0x8c34618) at prefork.c:839

#18 0x0809f887 in main (argc=3, argv=0xbfe03244) at main.c:618

(gdb) bt full

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

prev = Variable "prev" is not available.

(gdb) frame

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

832 if (UNEXPECTED(prev->next_free_block != mm_block) || UNEXPECTED(next->prev_free_block != mm_block)) {

(gdb) frame 2

#2  0x005954c4 in zend_hash_destroy (ht=0xafc7408c) at /opt/php-5.2.6/Zend/zend_hash.c:526

526 ht->pDestructor(q->pData);

(gdb)

(gdb) info locals

p = (Bucket *) 0xafbfbf98

q = (Bucket *) 0xafbfbecc

(gdb) info args

ht = (HashTable *) 0xafc7408c

(gdb) 

(gdb) bt full

#0  0x00574e4d in _zend_mm_free_int (heap=0x8c89600, p=Variable "p" is not available.

) at /opt/php-5.2.6/Zend/zend_alloc.c:832

prev = Variable "prev" is not available

# /opt/php-5.2.6/Zend/zend_alloc.c

current_conn = ap_run_create_connection(ptrans, ap_server_conf, csd, my_child_num, sbh, bucket_alloc);

        if (current_conn) {

            ap_process_connection(current_conn, csd);

   apr_signal(AP_SIG_GRACEFUL, SIG_IGN);

        child_main(slot);

make_child(ap_server_conf, free_slots[i]);

#main.c

if (ap_mpm_run(pconf, plog, server_conf))

            break;

#/opt/php-5.2.6/Zend/zend_hash.c

     p = ht->pListHead;

        while (p != NULL) {

                q = p;

                p = p->pListNext;

                if (ht->pDestructor) {

                        ht->pDestructor(q->pData);

                }

                if (q->pData != &q->pDataPtr) {

                        pefree(q->pData, ht->persistent);

                }

                pefree(q, ht->persistent);

        }

--

Tariq Wali.