We have several Oracle instances 10.2 (some RAC, some not, but all are having the same issue) all running on Red Hat 4.5. The binaries and data files are all on the NetApp and mounted via NFS. Sporadically when we shut the database down, we are unable to restart it. /u1/app/oracle/admin/prodrac/bdump/prodrac1_dbw0_10023.trc Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options ORACLE_HOME = /u1/app/oracle/product/10.2.0 System name: Linux Node name: orarac1 Release: 2.6.9-55.0.2.ELsmp Version: #1 SMP Tue Jun 12 17:58:20 EDT 2007 Machine: x86_64 Instance name: prodrac1 Redo thread mounted by this instance: 1 Oracle process number: 8 Unix process pid: 10023, image: oracle@orarac1 (DBW0) *** SERVICE NAME:() 2007-08-18 02:05:27.878 *** SESSION ID:(1194.1) 2007-08-18 02:05:27.878 ORA-01157: cannot identify/lock data file 1 - see DBWR trace file ORA-01110: data file 1: '/u1/oradata/prod/system/system01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 2 - see DBWR trace file ORA-01110: data file 2: '/u1/oradata/prod/temp/undotbs01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 3 - see DBWR trace file ORA-01110: data file 3: '/u1/oradata/prod/system/sysaux01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 4 - see DBWR trace file ORA-01110: data file 4: '/u1/oradata/prod/table/users01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 5 - see DBWR trace file ORA-01110: data file 5: '/u1/oradata/prod/temp/undotbs02.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 6 - see DBWR trace file ORA-01110: data file 6: '/u1/oradata/prod/temp/undotbs03.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 7 - see DBWR trace file ORA-01110: data file 7: '/u1/oradata/prod/table/esis_codes_data01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 8 - see DBWR trace file ORA-01110: data file 8: '/u1/oradata/prod/index/esis_codes_indx01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 9 - see DBWR trace file ORA-01110: data file 9: '/u1/oradata/prod/index/esis_large_indx.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 10 - see DBWR trace file ORA-01110: data file 10: '/u1/oradata/prod/index/esis_temp_indx.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 11 - see DBWR trace file ORA-01110: data file 11: '/u1/oradata/prod/index/esis_indx.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 12 - see DBWR trace file ORA-01110: data file 12: '/u1/oradata/prod/table/school_connects.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 13 - see DBWR trace file ORA-01110: data file 13: '/u1/oradata/prod/table/esis_temp_data01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 14 - see DBWR trace file ORA-01110: data file 14: '/u1/oradata/prod/table/esis_large_data01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 15 - see DBWR trace file ORA-01110: data file 15: '/u1/oradata/prod/table/esis_data01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 ORA-01157: cannot identify/lock data file 16 - see DBWR trace file ORA-01110: data file 16: '/u1/oradata/prod/table/tools01.dbf' ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 11: Resource temporarily unavailable Additional information: 8 The work around that we have found is to copy all of the files off of that mount point and then move them back. We even tried stopping everything and umounting/remounting the file system, but it still has the same errors. Oracle and Netapp are both pointing to other as the root of the problem. But to me it sounds like an NFS locking issue. Has anyone else experienced this, or have any suggestions on a better work around that having to copy the data files off and move them back? Thanks -Troy -- redhat-list mailing list unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe https://www.redhat.com/mailman/listinfo/redhat-list
