"ulimit -a" shows the following
socket buffer size (bytes, -b) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 524288
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 11095 <-------------- Samee as kern.maxfilesperproc
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65536
cpu time (seconds, -t) unlimited
max user processes (-u) 5547
virtual memory (kbytes, -v) unlimited
"sysctl -a | grep kern.maxfile" gives the following kernel limits:
kern.maxfiles: 12328
kern.maxfilesperproc: 11095
PostGreSQL seems to have settled around 4800 open file descriptors as counted by "lsof | grep postgres | wc -l" and the number of open files around 3500 as indicated by "sysctl -a | grep kern.openfile"
The only things that's running on the machine is PostGreSQL, hence first suspicion naturally falls on the database.
It just seems odd that it'd start behaving like this after having run with no problems for 6 months.
Any insight or suggestions as to where to start digging would be greatly appreciated
Cheers
Jona
On Wed, Aug 12, 2009 at 7:52 PM, jinson abraham <abraham.jinson@xxxxxxxxx> wrote:
since the message is "out of file descriptors: Too many open files in system; release and retry" i think you should also be checking the system ulimit.
try commands like "ulimit -a" to check the number of files currently open in the system. Also try "ulimit -Hn" to get "The maximum number of open file descriptors (most systems do not allow this value to be set)" see man pages of ulimit for more details.
you could also increase the ulimit by making changes in "ac/etc/security/limits.conf".
However i hope you have already checked the possibility of some other process which might be continuously opening files and not closing it properly. Linux has a fixed number of fd's that can be used and if the process dont relese them it coudl lead to such a problem.
Hope this helps.
Thanks,
- Jinson.On Wed, Aug 12, 2009 at 10:44 PM, Emanuel Calvo Franco <postgres.arg@xxxxxxxxx> wrote:
2009/8/12 Jonatan Evald Buus <jonatan.buus@xxxxxxxxxxxxxxxxxxx>:
> Greetings,Did you request about kern.maxfiles?
> We're seeing numerous "LOG: out of file descriptors: Too many open files in
> system; release and retry" entries as well as quite a few "LOG: could not
> open temporary statistics file "global/pgstat.tmp": Too many open files in
> system"
> Much more alarming however, we're seeing errors such as:
> FATAL: could not open file "global/pg_database": Too many open files in
> system
> ERROR: could not open relation 1663/2219053/2601: Too many open files in
> system
>
> kern.maxfiles is currently set to 12328 and "sysctl -a | grep
> kern.openfiles" showed over 10.000 open files prior to a reboot.
> After the reboot kern.openfiles quickly increased from less than 200 to over
--
Emanuel Calvo Franco
Database consultant at:
www.siu.edu.ar
www.emanuelcalvofranco.com.ar
--
Sent via pgsql-admin mailing list (pgsql-admin@xxxxxxxxxxxxxx)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
--
JONATAN EVALD BUUS
Executive Vice President Open Systems and Telecommunications
Mobile US +1 (305) 331-5242
Mobile DK +45 2888 2861
Telephone +1 (305) 777-0392
Fax. +1 (305) 777-0449
jonatan.buus@xxxxxxxxxxxxxxxxxxx
www.cellpointmobile.com
CellPoint Mobile Inc.
4000 Ponce de Leon Boulevard
Suite 470
Coral Gables, FL 33146
USA
'Mobilizing the Enterprise'
