一、环境描述
oracle 12c rac

二、处理过程

1、实例启动失败
SYS@orcl1> startup
ORACLE instance started.

Total System Global Area 1241513984 bytes
Fixed Size                  2923872 bytes
Variable Size             620757664 bytes
Database Buffers          603979776 bytes
Redo Buffers               13852672 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 8482
Session ID: 1 Serial number: 48907

2、查看日志
部分省略...............
Starting background process ARC2
Starting background process ARC3
ARC2 started with pid=44, OS id=8737
Unable to create archive log file '+DATADG'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:
ORA-19504: failed to create file "+DATADG"
ORA-17502: ksfdcre:4 Failed to create file +DATADG
ORA-15041: diskgroup "DATADG" space exhausted
ARCH: Error 19504 Creating archive log file to '+DATADG'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:
ORA-16038: log 2 sequence# 108 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATADG/ORCL/ONLINELOG/group_2.263.954539539'
ARC3 started with pid=45, OS id=8739
System state dump requested by (instance=1, osid=8659), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8552_20171231165944.trc
USER (ospid: 8659): terminating the instance due to error 16038
Dumping diagnostic data in directory=[cdmp_20171231165944], requested by (instance=1, osid=8659), summary=[abnormal insta
nce termination].
Instance terminated by USER, pid = 8659

3、确认磁盘组空间大小
$asmcmd -p
ASMCMD [+] > lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     10240     5763                0            5763              0             Y  CRSDG/
MOUNTED  EXTERN  N         512   4096  1048576      6144       13                0              13              0             N  DATADG/
ASMCMD [+] > 

4、处理思路
1、使用rm命令清理部分归档日志。
2、启动数据库
3、使用rman crosscheck archivelog all;
4、扩磁盘组

5、注意

需要使用rman进行crosscheck ,然后再删除归档日志在控制文件中的记录。
RMAN> crosscheck archivelog all;

using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238
validation failed for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924
部分省略..................................
validation succeeded for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_109.288.964199163 RECID=132 STAMP=964199165
validation succeeded for archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_110.325.964199595 RECID=135 STAMP=964199601
Crosschecked 124 objects

RMAN> 

RMAN> delete expired archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK
List of Archived Log Copies for database with db_unique_name ORCL
=====================================================================

Key     Thrd Seq     S Low Time            
------- ---- ------- - --------------------
11      1    50      X 07-DEC-2017 15:24:55
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097

13      1    51      X 07-DEC-2017 15:34:57
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355

14      1    52      X 07-DEC-2017 02:55:53
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355

19      1    53      X 07-DEC-2017 15:56:20
部分省略 ...........................................

51      2    66      X 08-DEC-2017 07:41:03
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883

53      2    67      X 08-DEC-2017 20:57:57
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885

57      2    68      X 09-DEC-2017 10:16:05
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785

59      2    69      X 09-DEC-2017 10:16:23
        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133

Do you really want to delete the above objects (enter YES or NO)? yes
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_59.298.962206427 RECID=29 STAMP=962206427
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_60.300.962208543 RECID=31 STAMP=962208544
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_61.302.962223761 RECID=48 STAMP=962177843
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_62.305.962223817 RECID=36 STAMP=962223817
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_63.306.962223899 RECID=37 STAMP=962223899
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_64.309.962224067 RECID=40 STAMP=962224066
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_65.310.962224095 RECID=41 STAMP=962224094
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_66.311.962224101 RECID=42 STAMP=962224100
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_67.314.962224111 RECID=45 STAMP=962224111
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_68.316.962224463 RECID=47 STAMP=962224462
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_69.317.962224863 RECID=49 STAMP=962224862
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_70.320.962225885 RECID=52 STAMP=962225884
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_71.322.962273773 RECID=54 STAMP=962273772
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_72.323.962273775 RECID=55 STAMP=962273775
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_73.324.962273779 RECID=56 STAMP=962273783
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_74.326.962273785 RECID=58 STAMP=962273785
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_47.281.962120099 RECID=12 STAMP=962073300
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_48.284.962121379 RECID=15 STAMP=962121380
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_49.285.962121381 RECID=16 STAMP=962121380
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_50.286.962121383 RECID=17 STAMP=962121382
部分省略 ....................
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 RECID=51 STAMP=962225883
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 RECID=53 STAMP=962225884
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 RECID=57 STAMP=962273784
deleted archived log
archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 RECID=59 STAMP=962274134
Deleted 48 EXPIRED objects

最近也是有客户遇到类似的问题，通过操作系统命令删除了部分归档，然后备份脚本备份失败，发现是因为这个原因引起的，优化一下脚本，添加进去crosscheck 命令，以后就不会出现类似问题了。