Hi,
I’m currently working on a new php formula in the Homebrew package manager project where we have noticed that httpd crashes on macOS High Sierra under certain conditions. My question now is if this is a bug in Apache httpd that I should file in the bug tracker or if you can point me in the right direction to get to the bottom of this.
In short what happens is that httpd (loading libphp7) crashes when the php snmp extension in php is loaded with a reference to boringssl but when the snmp extension is omitted it doesn’t crash.
# httpd -X
[1] 52393 abort httpd -X
#
Using the php binary alone with the snmp extension enabled causes no issues as far as we’ve been able to tell.
The stack trace from the crash looks like this
Process: httpd [52393]
Path: /usr/local/Cellar/httpd/2.4.29_1/bin/httpd
Identifier: httpd
Version: 0
Code Type: X86-64 (Native)
Parent Process: zsh [541]
Responsible: httpd [52393]
User ID: 501
Date/Time: 2018-02-14 10:49:57.591 +0100
OS Version: Mac OS X 10.13.3 (17D47)
Report Version: 12
Anonymous UUID: AC6370DE-1692-0BA1-08A2-BBE6153A8410
Sleep/Wake UUID: F545ED14-2DF3-4149-89EA-86A67EC93AB1
Time Awake Since Boot: 85000 seconds
Time Since Wake: 84000 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
Assertion failed: (ctx), function digest_update, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-109.40.1/apple/crypto/digests.c, line 49.
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff5f670e3e __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff5f7af150 pthread_kill + 333
2 libsystem_c.dylib 0x00007fff5f5cd312 abort + 127
3 libsystem_c.dylib 0x00007fff5f595368 __assert_rtn + 320
4 libboringssl.dylib 0x00007fff5d4b4ddc digest_update + 51
5 libcrypto.35.dylib 0x00007fff5db0fe08 EVP_DigestInit_ex + 920
6 libcrypto.35.dylib 0x00007fff5db0fa6a EVP_DigestInit + 42
7 libnetsnmp.30.dylib 0x000000010c089b8b sc_hash + 202
8 libnetsnmp.30.dylib 0x000000010c088f57 hash_engineID + 116
9 libnetsnmp.30.dylib 0x000000010c088d7f search_enginetime_list + 40
10 libnetsnmp.30.dylib 0x000000010c08909b set_enginetime + 61
11 libnetsnmp.30.dylib 0x000000010c088a82 init_snmpv3_post_config + 150
12 libnetsnmp.30.dylib 0x000000010c08a7a5 snmp_call_callbacks + 432
13 snmp.so 0x000000010bdcf1b0 zm_startup_snmp + 59
14 libphp7.so 0x0000000108ee023a zend_startup_module_ex + 305
15 libphp7.so 0x0000000108ee05bb zend_startup_module_zval + 12
16 libphp7.so 0x0000000108eec6e1 zend_hash_apply + 201
17 libphp7.so 0x0000000108ee04d8 zend_startup_modules + 47
18 libphp7.so 0x0000000108e82b0a php_module_startup + 2215
19 libphp7.so 0x0000000108f9b426 php_apache2_startup + 21
20 libphp7.so 0x0000000108f9ad33 php_apache_server_startup + 105
21 httpd 0x00000001088bd50b ap_run_post_config + 69
22 httpd 0x00000001088c4c2c main + 2076
23 libdyld.dylib 0x00007fff5f521115 start + 1
Thread 1:
0 libsystem_kernel.dylib 0x00007fff5f671562 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff5f7ac06a _pthread_wqthread + 1035
2 libsystem_pthread.dylib 0x00007fff5f7abc4d start_wqthread + 13
Thread 2:
0 libsystem_kernel.dylib 0x00007fff5f671562 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff5f7ac26f _pthread_wqthread + 1552
2 libsystem_pthread.dylib 0x00007fff5f7abc4d start_wqthread + 13
Thread 3:
0 libsystem_pthread.dylib 0x00007fff5f7abc40 start_wqthread + 0
1 ??? 0x000000005e943d01 0 + 1586773249
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff9857a340 rcx: 0x00007ffee7343998 rdx: 0x0000000000000000
rdi: 0x0000000000000307 rsi: 0x0000000000000006 rbp: 0x00007ffee73439d0 rsp: 0x00007ffee7343998
r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x0000000000000000 r11: 0x0000000000000206
r12: 0x0000000000000307 r13: 0x000000010bdff000 r14: 0x0000000000000006 r15: 0x000000000000002d
rip: 0x00007fff5f670e3e rfl: 0x0000000000000206 cr2: 0x00007fff98558148
Logical CPU: 0
Error Code: 0x02000148
Trap Number: 133
(this has been shortened and the full outtake can be found in the gist below)
Looking at that stack trace it seems that there should be an issue with boringssl under macOS however none of the components httpd, php nor the snmp extension has been linked against boringssl or the macos provided ssl binaries.
The shared libraries that the snmp extension uses looks like this
# otool -L /usr/local/opt/php/lib/php/20170718/snmp.so
/usr/local/opt/php/lib/php/20170718/snmp.so:
/usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (compatibility version 31.0.0, current version 31.3.0)
/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
(brew linkage is a homebrew command to list linked libraries across all packaged components)
# brew linkage httpd
System libraries:
/usr/lib/libSystem.B.dylib
/usr/lib/libexpat.1.dylib
/usr/lib/libiconv.2.dylib
/usr/lib/libxml2.2.dylib
/usr/lib/libz.1.dylib
Homebrew libraries:
/usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr)
/usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util)
/usr/local/opt/nghttp2/lib/libnghttp2.14.dylib (nghttp2)
/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)
/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl)
/usr/local/opt/pcre/lib/libpcre.1.dylib (pcre)
# brew linkage php
System libraries:
/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
/usr/lib/libSystem.B.dylib
/usr/lib/libbz2.1.0.dylib
/usr/lib/libc++.1.dylib
/usr/lib/libcurl.4.dylib
/usr/lib/libedit.3.dylib
/usr/lib/libexpat.1.dylib
/usr/lib/libexslt.0.dylib
/usr/lib/libiconv.2.dylib
/usr/lib/libicucore.A.dylib
/usr/lib/libncurses.5.4.dylib
/usr/lib/libpam.2.dylib
/usr/lib/libresolv.9.dylib
/usr/lib/libtidy.A.dylib
/usr/lib/libxml2.2.dylib
/usr/lib/libxslt.1.dylib
/usr/lib/libz.1.dylib
Homebrew libraries:
/usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr)
/usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util)
/usr/local/opt/argon2/lib/libargon2.1.dylib (argon2)
/usr/local/opt/aspell/lib/libaspell.15.dylib (aspell)
/usr/local/opt/aspell/lib/libpspell.15.dylib (aspell)
/usr/local/opt/freetds/lib/libsybdb.5.dylib (freetds)
/usr/local/opt/freetype/lib/libfreetype.6.dylib (freetype)
/usr/local/opt/gettext/lib/libintl.8.dylib (gettext)
/usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
/usr/local/opt/icu4c/lib/libicudata.60.2.dylib (icu4c)
/usr/local/opt/icu4c/lib/libicui18n.60.dylib (icu4c)
/usr/local/opt/icu4c/lib/libicuio.60.dylib (icu4c)
/usr/local/opt/icu4c/lib/libicuuc.60.dylib (icu4c)
/usr/local/opt/jpeg/lib/libjpeg.9.dylib (jpeg)
/usr/local/opt/libpng/lib/libpng16.16.dylib (libpng)
/usr/local/opt/libpq/lib/libpq.5.dylib (libpq)
/usr/local/opt/libsodium/lib/libsodium.23.dylib (libsodium)
/usr/local/opt/libzip/lib/libzip.5.dylib (libzip)
/usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (net-snmp)
/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)
/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl)
/usr/local/opt/unixodbc/lib/libodbc.2.dylib (unixodbc)
/usr/local/opt/webp/lib/libwebp.7.dylib (webp)
➜ homebrew-core git:(master) ✗ brew linkage net-snmp
System libraries:
/usr/lib/libSystem.B.dylib
Homebrew libraries:
/usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmp.30.dylib (net-snmp)
/usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpagent.30.dylib (net-snmp)
/usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpmibs.30.dylib (net-snmp)
/usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmptrapd.30.dylib (net-snmp)
/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)
So concluding that boringssl in not actively linked against any component involved It looks like httpd seems to be dynamically linking against the incorrect library which therefore causes the system to crash.
The workaround that has been found so far is to statically link the Apache httpd SSL module using --enable-mods-static=ssl which make the system not crash but it doesn’t solve the incorrect dynamic loading.
The httpd binary is built with the following options
./configure
--enable-layout=Slackware-FHS
--prefix=/usr/local/Cellar/httpd/2.4.29_1
--sbindir=/usr/local/Cellar/httpd/2.4.29_1/bin
--mandir=/usr/local/Cellar/httpd/2.4.29_1/share/man
--sysconfdir=/usr/local/etc/httpd
--datadir=/usr/local/var/www
--localstatedir=/usr/local/var
--enable-mpms-shared=all
--enable-mods-shared=all
--enable-cgi
--enable-pie
--enable-suexec
--with-suexec-bin=/usr/local/opt/httpd/bin/suexec
--with-suexec-caller=_www
--with-port=8080
--with-sslport=8443
--with-apr=/usr/local/opt/apr
--with-apr-util=/usr/local/opt/apr-util
--with-mpm=prefork
--with-nghttp2=/usr/local/opt/nghttp2
--with-ssl=/usr/local/opt/openssl
--with-pcre=/usr/local/opt/pcre
So to reiterate my question - do you think this is this a bug in Apache httpd that I should file in the bug tracker or can you point me in another directions to find the root cause of this issue ? I’m happy to provide further information and test whatever I can to help troubleshoot this.
Best regards,
Jan
