Hello Everyone,
On 2015/12/30 17:51:01, Transaction was failed on our system.
I checked the pg_log, it states as below:
2015-12-30 17:51:01 CST [11260]: [42-1] LOG: 00000: server process (PID 14272) was terminated by signal 11: Segmentation fault
<part of the pg_log>
================================================
2015-12-30 17:50:52 CST [18954]: [5165-1] LOG: 00000: statement: select se_addlog('1','99','20151230161941915390111750','O','0000','');
2015-12-30 17:50:52 CST [18954]: [5166-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5167-1] LOG: 00000: duration: 0.951 ms
2015-12-30 17:50:52 CST [18954]: [5168-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5169-1] LOG: 00000: statement: unlisten *
2015-12-30 17:50:52 CST [18954]: [5170-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5171-1] LOG: 00000: duration: 0.118 ms
2015-12-30 17:50:52 CST [18954]: [5172-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5173-1] LOG: 00000: statement: select 1
2015-12-30 17:50:52 CST [18954]: [5174-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5175-1] LOG: 00000: duration: 0.160 ms
2015-12-30 17:50:52 CST [18954]: [5176-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5179-1] LOG: 00000: duration: 0.738 ms
2015-12-30 17:50:52 CST [18954]: [5180-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5181-1] LOG: 00000: statement: unlisten *
2015-12-30 17:50:52 CST [18954]: [5182-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5183-1] LOG: 00000: duration: 0.112 ms
2015-12-30 17:50:52 CST [18954]: [5184-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5185-1] LOG: 00000: statement: select 1
2015-12-30 17:50:52 CST [18954]: [5186-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5187-1] LOG: 00000: duration: 0.159 ms
2015-12-30 17:50:52 CST [18954]: [5188-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5190-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5191-1] LOG: 00000: duration: 0.832 ms
2015-12-30 17:50:52 CST [18954]: [5192-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:50:52 CST [18954]: [5193-1] LOG: 00000: statement: unlisten *
2015-12-30 17:50:52 CST [18954]: [5194-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:50:52 CST [18954]: [5195-1] LOG: 00000: duration: 0.111 ms
2015-12-30 17:50:52 CST [18954]: [5196-1] LOCATION: exec_simple_query, postgres.c:1364
★2015-12-30 17:51:02 CST [14276]: [1-1] LOG: 00000: database system was interrupted; last known up at 2015-12-30 17:50:21 CST★
★2015-12-30 17:51:02 CST [14276]: [31-1] LOG: 00000: last completed transaction was at log time 2015-12-30 17:50:52.793516+08★
2015-12-30 17:51:01 CST [14270]: [1-1] LOG: 00000: connection received: host=[local]
2015-12-30 17:51:01 CST [14270]: [2-1] LOCATION: BackendInitialize, postmaster.c:4056
2015-12-30 17:51:01 CST [14270]: [3-1] LOG: 00000: connection authorized: user=enterprisedb database=edb
2015-12-30 17:51:01 CST [14270]: [4-1] LOCATION: PerformAuthentication, postinit.c:265
2015-12-30 17:51:01 CST [14271]: [1-1] LOG: 00000: connection received: host=[local]
2015-12-30 17:51:01 CST [14271]: [2-1] LOCATION: BackendInitialize, postmaster.c:4056
2015-12-30 17:51:01 CST [14271]: [3-1] LOG: 00000: connection authorized: user=enterprisedb database=edb
2015-12-30 17:51:01 CST [14271]: [4-1] LOCATION: PerformAuthentication, postinit.c:265
2015-12-30 17:51:01 CST [14270]: [5-1] LOG: 00000: statement: select now();
2015-12-30 17:51:01 CST [14270]: [6-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [1-1] LOG: 00000: connection received: host=[local]
2015-12-30 17:51:01 CST [14272]: [2-1] LOCATION: BackendInitialize, postmaster.c:4056
2015-12-30 17:51:01 CST [14270]: [7-1] LOG: 00000: duration: 1.523 ms
2015-12-30 17:51:01 CST [14270]: [8-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:51:01 CST [14270]: [9-1] LOG: 00000: statement: select current_timestamp,pid,current_timestamp - query_start as runtime,datname,usename,client_addr,query
2015-12-30 17:51:01 CST [14270]: [10-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [3-1] LOG: 00000: connection authorized: user=enterprisedb database=edb
2015-12-30 17:51:01 CST [14272]: [4-1] LOCATION: PerformAuthentication, postinit.c:265
2015-12-30 17:51:01 CST [14271]: [5-1] LOG: 00000: statement: select CURRENT_TIMESTAMP(0), client_addr, count(client_addr) from pg_stat_activity group by client_addr order by client_addr;
2015-12-30 17:51:01 CST [14271]: [6-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [5-1] LOG: 00000: statement: select now();
2015-12-30 17:51:01 CST [14272]: [6-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [7-1] LOG: 00000: duration: 0.997 ms
2015-12-30 17:51:01 CST [14272]: [8-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:51:01 CST [14272]: [9-1] LOG: 00000: statement: SELECT bl.pid AS blocked_pid,
2015-12-30 17:51:01 CST [14272]: [10-1] LOCATION: exec_simple_query, postgres.c:1082
2015-12-30 17:51:01 CST [14270]: [11-1] LOG: 00000: duration: 5.044 ms
2015-12-30 17:51:01 CST [14270]: [12-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:51:01 CST [14270]: [13-1] LOG: 00000: disconnection: session time: 0:00:00.015 user=enterprisedb database=edb host=[local]
2015-12-30 17:51:01 CST [14270]: [14-1] LOCATION: log_disconnections, postgres.c:5993
2015-12-30 17:51:01 CST [14271]: [7-1] LOG: 00000: duration: 7.072 ms
2015-12-30 17:51:01 CST [14271]: [8-1] LOCATION: exec_simple_query, postgres.c:1364
2015-12-30 17:51:01 CST [14271]: [9-1] LOG: 00000: disconnection: session time: 0:00:00.015 user=enterprisedb database=edb host=[local]
2015-12-30 17:51:01 CST [14271]: [10-1] LOCATION: log_disconnections, postgres.c:5993
★2015-12-30 17:51:01 CST [11260]: [42-1] LOG: 00000: server process (PID 14272) was terminated by signal 11: Segmentation fault★
★2015-12-30 17:51:01 CST [11260]: [43-1] DETAIL: Failed process was running: SELECT bl.pid AS blocked_pid,★
2015-12-30 17:51:01 CST [11260]: [44-1] LOCATION: LogChildExit, postmaster.c:3362
2015-12-30 17:51:01 CST [11260]: [45-1] LOG: 00000: terminating any other active server processes
2015-12-30 17:51:01 CST [11260]: [46-1] LOCATION: HandleChildCrash, postmaster.c:3083
2015-12-30 17:51:01 CST [18954]: [5197-1] WARNING: 57P02: terminating connection because of crash of another server process
2015-12-30 17:51:01 CST [18954]: [5198-1] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-12-30 17:51:01 CST [18954]: [5199-1] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-12-30 17:51:01 CST [18954]: [5200-1] LOCATION: quickdie, postgres.c:3855
2015-12-30 17:51:01 CST [18953]: [4969-1] WARNING: 57P02: terminating connection because of crash of another server process
2015-12-30 17:51:01 CST [18953]: [4970-1] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-12-30 17:51:01 CST [18953]: [4971-1] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-12-30 17:51:01 CST [18953]: [4972-1] LOCATION: quickdie, postgres.c:3855
================================================
The server log states:
Dec 30 17:51:01 egtcdb1 kernel: edb-postgres[14272]: segfault at 30 ip 00000000006a1f5a sp 00007fff70c5e2e0 error 4 in edb-postgres[400000+5e8000]
Server information,
egtcdb1 (Linux release 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013) (0)
[enterprisedb@egtcdb1 ~]$ psql
psql.bin (9.4.1.3)
Type "help" for help.
edb=# select version ;
LOG: statement: select version ;
LOG: duration: 1.603 ms
version
-------------------------------------------------------------------------------------------------------------------
EnterpriseDB 9.4.1.3 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit
(1 row)
[enterprisedb@egtcdb1 ~]$ free -tm
total used free shared buffers cached
Mem: 15888 12506 3381 0 502 11204
-/+ buffers/cache: 799 15088
Swap: 4015 36 3979
Total: 19904 12543 7361
Cpu 4 core, memory 16GB
Postgresql.conf information,
shared_buffers = 4GB
temp_buffers = 8MB
work_mem = 56696kB
maintenance_work_mem = 256MB
effective_cache_size = 5324784kB
On my case, Please give me the instruction in steps on how to figure out the
reason why did this “signal 11” happen and how to solve “signal 11: Segmentation fault” issue.
Thank you in advance.
John
