EM 12c/13c The listener is down: Failed to run lsnrctl

The listener is up and running. However, the lisener target shows as Down in the Oracle Enterprise Manager Cloud Control Console and the incident message as following,

The listener is down: Failed to run lsnrctl.

Agent trace file emagent_perl.trc shows

lsnrresp.pl: Mon Oct 4 10:36:27 2021: DEBUG:  LISTENER :: em_result=|0|Failed to run lsnrctl

This issue can be seen in EM 12c and 13c. In order to obtain the listener status, EM 12c/13c agent runs the following command

  <LISTENER_HOME>/bin/lsnrctl status (ADDRESS=(PROTOCOL=TCP)(HOST=<HOST_NAME>)(PORT=<PORT_NUMBER>))

And writes the output to a temporary file in the temp directory. Therefore, any of following issues can cause the error,

 1. The agent user does not have execute permission on lsnrctl executable

 2. The agent user does not have write permission on the TEMP directory

 3. The TEMP directory is full

For issue 1, the error should have been there since the agent was deployed. It can be fixed by grant execute permission to agent user on lsnrctl executable. The best way is to assign agent user to same primary group as primary group of listener home owner, it is typically oinstall. Actually, it is recommended to install agent software as owner of database home so that this issue will never happen.

If issue 2 or issue 3 is the case, it may be necessary to find out the location of the temp directory that the agent is trying to use. It is kind of complicated to find out the temp directory which agent is using. Before that, we can give a guess by logging into the server as agent user and run command,

   echo $TMPDIR

Output could be the temp directory, if output is empty, use '/tmp' as the guess. Make sure the guess temp directory is not full and agent user has write permission on the guess, then confirm the listener status issue is fixed. Most likely, you will have the luck. If the issue is not fixed, the guess temp directory is not used by agent. We have to use Oracle support recommended steps to determine temp diretory,

1) Take a copy of file <AGENT_HOME>\plugins\oracle.sysman.db.agent.plugin_<plugin_version>\scripts\db\net\listenerUtil.pl

2) Edit listenerUtil.pl and locate the following section

my $filename;

my $fh;

if ( $OSNAME eq 'WIN' )

{

my $TEMP = $ENV{SYSTEMDRIVE} . "\\TEMP";

#A temp solution

&mkDir_lst($TEMP);

$filename = "$TEMP\\" . "net.$$";

}

else

{

( $fh, $filename ) = tempfile( UNLINK => 1 );

}

3) Amend the section by adding a debug statement at the end of it as following

my $filename;

my $fh;

if ( $OSNAME eq 'WIN' )

{

my $TEMP = $ENV{SYSTEMDRIVE} . "\\TEMP";

#A temp solution

&mkDir_lst($TEMP);

$filename = "$TEMP\\" . "net.$$";

}

else

{

( $fh, $filename ) = tempfile( UNLINK => 1 );

}

EMD_PERL_DEBUG(" listenerUtil.pl :: oracletestfilename == $filename");

4) Enable debug trace on the agent by add following line to file <AGENT_INST>/sysman/config/emd.properties

EMAGENT_PERL_TRACE_LEVEL=DEBUG

5) Reload the agent to make the change take effect

emctl reload agent

6) Force the agent to run the collection

emctl control agent runCollection <LISTENER_NAME>:oracle_listener Response

Note: <LISTENER_NAME> is the listener target name in EM, not the name defined in listener.ora.

7) Check the the trace file <agent_inst>/sysman/log/emagent_perl.trc. Look for the word "oracletestfilename" for example

output on Windows:

lsnrresp.pl: Mon Oct 4 10:45:27 2021: DEBUG: listenerUtil.pl :: oracletestfilename == C:\Windows\Temp\net.5224 <-- HERE IS THE LOCATION OF THE TEMP FILE

Output on Linux/Unix:

lsnrresp.pl: Mon Oct 4 10:45:27 2021: DEBUG: listenerUtil.pl :: oracletestfilename == /tmp/gPbuYYiWX4 <-- HERE IS THE LOCATION OF THE TMP FILE