96b6hrt8csw55 was doing merge

903wwaubvn68k was doing an update

apart from that your analysis is very close to mine

here is what I see when I checked 961 and 1057

SQL> @usid 961

 SID             PROGRAM              SPID      LASTCALL STATUS
 --------------  -------------------- ------------------ --------
  '961,1'        (DBW0)               26636       255137 ACTIVE
                                 


SQL> @usid 1057

 SID             PROGRAM              SPID        LASTCALL STATUS
 --------------  -------------------- --------  ---------- --------
  '1057,1'       (DBW1)               26638         255143 ACTIVE
                

then I decided to do another check

.. rest is in update

May be i am wrong.

and I would probably check what sessions 961 and 1057 are doing.

free buffer waits occurs when session cannot find free buffers to read data or *to build a consistent image of data block* – I think it’s only a symptom

Script is slightly modified version of SW.sql (modification is for buffer busy reads nothing to do with this post) called for active foreground sessions with a newer name @swact

This server was completely “new” to me apart from the fact that I only knew the massive core table with 500GB size was partitioned with dbms_redefinition and “partitioning” operation was finished reported as success but as I said the main index was missing that was causing other sessions suffer.

All I did was check with @swact and another check and took an action. I am after my action and the other check as answer.

Thanks for stopping by by the way

If I had to make a guess, I would look at the SEC_IN_STATE column and see that there are several really big numbers there, the largest at 174,744. The script seems to be showing that only one session is on the CPU (probably the session executing swact.sql), while all other sessions are listed as waiting.

SQL> SELECT 174744/60/60/24 FROM DUAL;
 
174744/60/60/24
---------------
         2.0225
 
SQL> SELECT TO_CHAR(TO_DATE('27-DEC-2010 11:30','DD-MON-YYYY HH24:MI')-2.0225,'MM/DD HH24:MI') FROM DUAL;
 
TO_CHAR(TO_
-----------
12/25 10:57

So, that puts the first row lock contention wait at roughly 11 AM on December 25. Something, probably a scheduled job, likely started at 11 AM on Christmas day and attempted to insert a row using the SQL statement with SQL_ID 903wwaubvn68k, causing a possible primary key violation if session 1259 committed. Three other sessions also started being blocked when attempting to execute the same SQL statement (all had different CHILD_NUMBER values, one additional session was attempting to execute SQL ID drjnqxby1927s). This might suggest that the primary key is not generated by a sequence, but instead by something like:

SELECT MAX(ID)+1 NEW_ID FROM T1;

The likely cause of that wait is probably an end-user failing to click OK to indicate that a set of documents printed (or something similar) so that a COMMIT could be performed, before leaving for the Christmas holiday.

The ‘enq: KO – fast object checkpoint’ waits typically indicate that a direct path read is about to happen, possibly due to a parallel query execution or a serial direct read of a table. The first wait of that type shown started at about 9:12 AM on December 27, and the last one at 11:20 AM. At roughly 11:20 AM you yelled at the SAN causing it to panic (blogs.sun.com/brendan/entry/unusual_disk_latency) which then triggered additional waits for blocks to be read from the SAN. The decrease in the ‘free buffer waits’ is simply an indication of less pressure on the buffer cache because all of the other sessions are still waiting for their previous IO requests to complete.

How close is my guess?

The first thing i would check is “903wwaubvn68k” sql_id.This particular sql_id is showing “db file scattered read”,”enq: TX – row lock contention” and “free buffer waits”.Also, would have checked session id 961.

“What might be wrong for free buffer event”– free buffer events can be caused due to inefficient sqls.

Session 1442 shows “write complete waits” event which could be cause of I/O staturation.So, I/O is also to be considered for check.

Regards,
Anand

First, I would have checked the contentions and wait events so we have a contention and a wait event in this very example. Row lock contention could happened because of the application itself. Free buffer waits are there because DBWn can not cope with the dirty buffers fast enough. Jonathan Lewis posted a nice article here –> http://jonathanlewis.wordpress.com/2006/11/21/free-buffer-waits/
So either the objects were pinned in the buffer or DBWn was not fast enough. However, i also see in the chart that we had a vicious log file switch (checkpoint incomplete) wait event for a while near Dec 28, 09:00 AM. I believe there is something wrong with the log_buffer and the online redolog sizes. If they can be tuned than the problem may dissapper before digging into a real analysis.

Ogan