Sys-Admin Information's » Backup & Recovery

Troubleshooting for startup inconsistent DB

Hilman-Lupus — Tue, 27 May 2008 03:14:36 +0000

for e.g. database unable up cause of datafile corruption

SQL> startup
ORACLE instance started.
 
Total System Global Area 1610612736 bytes
Fixed Size                  2177912 bytes
Variable Size             396149896 bytes
DATABASE Buffers         1207959552 bytes
Redo Buffers                4325376 bytes
DATABASE mounted.
ORA-01113: file 2 needs media recovery
ORA-01110: DATA file 2: '/oradata/oracle/ts/undotbs01.

commonly the database unable be startup if UNDO or SYSTEM table space need recovery, if these happen we need correct temporary workaround to handle this, all workaround for each tablespace datafile is similar

do the recovery

SQL> recover DATABASE;
ORA-00279: CHANGE 7516226638 generated at 05/05/2008 12:32:11
needed FOR thread 1
ORA-00289: suggestion : /oradata/oracle/ts/arc/1_42_653916655.dbf
ORA-00280: CHANGE 7516226638 FOR thread 1 IS IN sequence #42
 
Specify log: {ret=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00308: cannot open archived log
‘/oradata/oracle/ts/arc/1_42_653916655.dbf’
ORA-27037: unable TO obtain file STATUS
SVR4 Error: 2: No such file OR directory
Additional information: 3

Open database

SQL> ALTER DATABASE open;
ALTER DATABASE open
*
ERROR at line 1:
ORA-01113: file 2 needs media recovery
ORA-01110: DATA file 2: '/oradata/oracle/ts/undotbs01.dbf'

this condition force us to append archivelog or logfile consist appropriate sequence, if there not available... database unable up till to ever

but we have a temporary solution, below :

Edit Init file
Remark parameter undo_management dan undo_tablespace
add this parameter :
UNDO_MANAGEMENT=MANUAL _ALLOW_RESETLOGS_CORRUPTION = TRUE _ALLOW_ERROR_SIMULATION = TRUE _CORRUPTED_ROLLBACK_SEGMENTS =(_SYSSMU1,_SYSSMU2,_SYSSMU3, ...)
get value _CORRUPTED_ROLLBACK_SEGMENTS from [OS] the command cd system_datafile_directory strings system01.dbf | grep _SYSSMU | cut -d $ -f 1 | sort -u

shutdown database , and turn on with that initfile already edited. Pfile must be wrote in startup command .

SQL> startup mount
pfile='/data1/oracle/product/10.2.0/dbs/initts.ora';
ORACLE instance started.
Total System Global Area 1610612736 bytes
Fixed Size                  2177912 bytes
Variable Size             396149896 bytes
DATABASE Buffers         1207959552 bytes
Redo Buffers                4325376 bytes
DATABASE mounted.

Recover database until cancel

SQL> recover DATABASE until cancel;
ORA-00279: CHANGE 7516226638 generated at 05/05/2008 12:32:11
needed FOR thread 1
ORA-00289: suggestion :
/oradata/oracle/ts/arc/1_42_653916655.dbf
ORA-00280: CHANGE 7516226638 FOR thread 1 IS IN sequence #42
Specify log: {ret=suggested | filename | AUTO | CANCEL}
CANCEL
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery TO be consistent
ORA-01110: DATA file 1: ‘/oradata/oracle/ts/system01.dbf’
ORA-01112: media recovery NOT started

Open database

SQL> ALTER DATABASE OPEN RESETLOGS;
DATABASE altered.

after the database up, backup important data using exp/imp

Backup Listener log

Hendrawan — Sun, 09 Mar 2008 05:40:48 +0000

In database production, the log of listener was grow rapidly, some times if you move the "listener.log" while the listener is up and running, listener will stop writing to log file after you move/delete the "listener.log", it won't create new log file.
The listener log status will be 'off', You need to start the "listener log" again manually. For this reason you must do this procedure.

bash> lsnrctl stat

LSNRCTL for Solaris: Version 9.2.0.6.0 - Production on 09-MAR-2008 12:43:22
 
Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Solaris: Version 9.2.0.6.0 - Production
Start Date                11-SEP-2007 03:34:17
Uptime                    180 days 9 hr. 9 min. 4 sec
Trace Level               off
Security                  OFF
SNMP                      OFF
Listener Parameter File   /oracle/app/product/9.2.0/network/admin/listener.ora
Listener Log File         /oracle/app/product/9.2.0/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=HOSTNAME)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "DB_SID.domain.com" has 2 instance(s).
Instance "DB_SID", status UNKNOWN, has 1 handler(s) for this service...
Instance "DB_SID", status READY, has 1 handler(s) for this service...
The command completed successfully

bash> lsnrctl

LSNRCTL> SET log_status off
Connecting TO (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
LISTENER parameter "log_status" SET TO OFF
The command completed successfully

bash> mv /oracle/app/product/9.2.0/network/log/listener.log /backup/listener.log_9-March-2008

LSNRCTL> set log_status on
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
LISTENER parameter "log_status" set to ON
The command completed successfully

Moving & Recreate Temp Tablespace

Hendrawan — Wed, 12 Dec 2007 17:05:18 +0000

I Recreate the temp tablespace coz my temp tablespace was corrupt,
you can do to the following bellow

create temporary tablespace temp2 tempfile '/rdbms/temp01/temp2.dbf' size 10M extent management local uniform size 1M;
alter database default temporary tablespace temp2;
drop tablespace temp including contents and datafiles;

now change back to temp again with following step bellow:

create temporary tablespace temp tempfile '/rdbms/temp01/temp2_01.dbf' size 1024M extent management local uniform size 1M;
alter database default temporary tablespace temp;
drop tablespace temp2 including contents and datafiles;
alter tablespace temp add tempfile '/rdbms/temp01/temp2_02.dbf' size 1024M;

Split Busy Datafile to Another Mountpoint

Hendrawan — Tue, 09 Oct 2007 05:35:28 +0000

If you want to load balance your datafile, you can split your datafile into another mount point,
here's detail step.

1. lsnrctl stop
2. shutdown immediate
3. copy your datafile

cp /$ora_home/$SID/undo/undo01.dbf /$ora_home/$SID/undo1/
cp /$ora_home/$SID/undo/undo02.dbf /$ora_home/$SID/undo2/
cp /$ora_home/$SID/index1/idx01.dbf /$ora_home/$SID/index2/
cp /$ora_home/$SID/index1/idx02.dbf /$ora_home/$SID/index3/

4. startup exclusive mount
5. rename your old datafile path with alter command

alter db rename file '/$ora_home/$SID/undo/undo01.dbf' to '/$ora_home/$SID/undo1/undo1.dbf';
alter db rename file '/$ora_home/$SID/undo/undo02.dbf' to '/$ora_home/$SID/undo2/undo2.dbf';
alter db rename file '/$ora_home/$SID/index1/idx01.dbf' to '/oradata/$SID/index2/idx01.dbf';
alter db rename file '/$ora_home/$SID/index1/idx02.dbf' to '/oradata/$SID/index3/idx02.dbf';

6. shutdown immediate
7. startup
8. lsnrctl start

Note:
DB=database