Child Listener Problem

A while ago one of our systems faced with a child listener occurrence problem. The listener wasn’t answering the requests., when our clients were trying to reach the server through listener as a new connection.We couldn’t get the status of the listener with lsnrctl status command and nothing was written to the listener logs.

There wasn’t any problem on the database itself because local connections are accepted and database answers your requests. Also already connected users are working normally. Conclusion was “our listener hanged”

When we looked at the listener proceses we saw that there was a a child listener forked from the real listener.

#ps -ef | grep tns
oracle 22480 14188 0 18:33:57 $ORACLE_HOME/bin/tnslsnr listener_DBNAME -inherit
oracle 14188 1 0 Apr 1 $ORACLE_HOME/bin/tnslsnr listener_DBNAME -inherit

We tried to stop the listener with lsnrctl stop command but it didn’t work. (after closing the application servers)

So we killed the listener processes from OS

# kill -15 22480,14188

After the killing operation we opened the listener and everything worked well and all the connection requests were accepted. This fast action must be taken immediately and it was time to find a real solution for the problem.

Simple metalink search explained the problem and the solution. Note:340091.1 says that “This is a known problem addressed via non-published bug:4518443 (Abstract: Listener Gets Hung Up). The issue is that the TNS listener can hang under load while spawning a process” . Note has 2 solutions one is applying patch 4518443 or setting the listener parameter SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name> =OFF. We choose the parameter setting option and did not face with the problem again. There is something to consider about the parameter setting option for RAC systems, when you set parameter to OFF, FAN (fast application notification) will not be possible.

PS: Note says this problem can occur on any platform for releases from 10.1.0.2.0 to 10.2.0.3.0