We recently upgraded one of our biggest databases to 22.214.171.124 from 126.96.36.199 due to the library cache mutex related bugs on 188.8.131.52. Since I wasn’t the one who did the upgrade I have nothing to say about the process but I was the on call DBA on the next day after the upgrade.
Users started to report the ORA-00600 for the statement below
delete from TABLE_1 where id=49; ERROR at line 1: ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:objnotfound], , , , , , , , , , , 
Checked for the error and found 3 bugs (9774817 10322768 10373381) related with the issue and not just none of them seems to have a workaround but all of them also says restore is the option which was the last choice for us (database server financial markets and we were close to the opening of markets ). It looks like we had some issues with deferred segments.
The table we try to delete from has got segments so it should not be the problem however it has got on delete trigger doing a merge on other tables and looks like one of those tables were causing the issue.
We first raised the Severity 1 ticket. while the engineer was reviewing the SR we decided to trace all the delete operation and see what is actually failing and once again 10046 trace revealed everything
When the error raised last operation was below
select /*+ all_rows */ count(1) from TABLE_2 where id=49
We run the statement manually, issue repeated again and when I check if the table has got any segment in dba_segments table it did not return anything.
Updated the SR with the new information. Support engineer asked us to try to regenerate the objects but he wasn’t hopeful. He suggested probably we needed to restore the db to before upgrade state.
I wasn’t as hopeful as ORacle support so to see which tables are affected I ran the output of the query below
select 'select count(*) from '||owner||'.'||table_name||' ;' from dba_tables where segment_created = 'NO';
All of the segmentless tables were raising the ORA-00600 so we decided to re-create them.
After recreation of all of the tables and their indexes I ran the same output again and none of the tables raised the error. Our workaround worked fine and we got away from restoring the DB and 2 more standbys which would be a nightmare just for tables which never used before. Interesting part is that we did not see the error on our QA db which I am not so sure what was the difference in terms of upgrade.
Hope you won’t hit this problem but better you check your deferred segments before and after 184.108.40.206 upgrade.
Note: I also check the LOB segments but we did not have any so no action taken for them.
select count(*) from dba_lobs where segment_created = 'NO';