Internal concurrent manager is not starting after crash in the middle of working hours

Today I experienced interesting case when in the middle of working day concurrent manager crashed (investigation of crash is ongoing) and it won’t start up. If we look from application point of view than we see actual running count 1 but target is 0. If we look into process forms then we see that last concurrent process status is terminated. Situation comes even worse if we trying to find any left internal concurrent process running, there isn’t one.

When I tried to start Internal concurrent manager, log shows me error:
 ========================================================================
 Starting TEST_0617@TEST_BALANCE Internal Concurrent Manager -- shell process ID 19306logfile=/u01/ar121/TEST/log/log/TEST_0617.mgr
 PRINTER=noprint
 mailto=applTEST
 restart=N
 diag=N
 sleep=30
 pmon=4
 quesiz=1
 Reviver is ENABLED
 Routine &ROUTINE has attempted to start the internal concurrent manager.  The ICM is already running.  Contact you system administrator for further assistance.afpdlrq received a
 n unsuccessful result from PL/SQL procedure or function FND_DCP.Request_Session_Lock.
 Routine FND_DCP.REQUEST_SESSION_LOCK received a result code of 1 from the call to DBMS_LOCK.Request.
 Possible DBMS_LOCK.Request resultCall to establish_icm failed
 The Internal Concurrent Manager has encountered an error.
 Review concurrent manager log file for more detailed information. : 17-JUN-2011 14:08:04 -
 Shutting down Internal Concurrent Manager : 17-JUN-2011 14:08:04
 List of errors encountered:
 .............................................................................
 1 _
 Routine AFPCSQ encountered an ORACLE error. .
Review your error messages for the cause of the error. (=<POINTER>)
 .............................................................................
List of errors encountered:
 .............................................................................
_ 1 _
 Routine AFPCAL received failure code while parsing or running your
 concurrent program CPMGR
Review your concurrent request log file for more detailed information.
 Make sure you are passing arguments in the correct format.
 .............................................................................
The TEST_0617@TEST_BALANCE internal concurrent manager has terminated with status 1 - giving up.

Next thing I do is search for error in support.oracle.com and usually I get to notes like:

Concurrent Processing – The Concurrent Manager Fails to Start on GSM Enabled Due to DBMS_LOCK.Request ResultCall Failed to Establish ICM [ID 245563.1]

and only solution is restart whole application/database and issue should be gone. But this is middle of day and there is hundreds of people working with system so this is not the solution. Also attempt to kill all running manager process into application tier is not solving the problem even java based concurrent managers.

After only apache, opmn and forms processes left on apps_tier machine I looked into database and issue is that Oracle DB sessions are not gone after apps_tier process kill. There isn’t Internal concurrent manager running to handle oracle DB sessions against application processes. And this is the case, Internal manager DB session was still there and was conflicted with attempt to start new internal manager.

Solution was to kill this left session which can be found with SQL:

select * from gv$session where program like 'ICM%';

After this session is gone Internal manager starts up!

Advertisements
    • Gaurav
    • May 26th, 2012

    We are also facing the same issue, we have GSM enabled. FNDSM on node 2 has died but ICM is running fine but the log file shows

    Routine &ROUTINE has attempted to start the internal concurrent manager. The ICM is already running. Contact you system administrator for further assistance.afpdlrq received an unsuccessful result from PL/SQL procedure or function FND_DCP.Request_Session_Lock.
    Routine FND_DCP.REQUEST_SESSION_LOCK received a result code of 1 from the call to DBMS_LOCK.Request.
    Possible DBMS_LOCK.Request resultCall to establish_icm failed
    The Internal Concurrent Manager has encountered an error.

    Review concurrent manager log file for more detailed information. : 25-MAY-2012 09:42:10 –

    Shutting down Internal Concurrent Manager : 25-MAY-2012 09:42:10

    List of errors encountered:
    …………………………………………………………………..

    _ 1 _
    Routine AFPCSQ encountered an ORACLE error. .

    Review your error messages for the cause of the error.

    As this is our production system we cannot bounce the Application and database every time this happens

      • Gaurav
      • May 26th, 2012

      The ICM log files shows following :

      “Routine AFPCAL received failure code while parsing or running your
      concurrent program CPMGR

      Review your concurrent request log file for more detailed information.
      Make sure you are passing arguments in the correct format.
      …………………………………………………………………..

      The ebsdbprd_0522@ebsdbprd internal concurrent manager has terminated with status 1 – giving up.”

      But my ICM is running and all the request running fine but my FNDSM and some other manager are down and my FNDOPP show 8 process instead of 4 target process.

      Thanks

    • Neeti Hasija
    • April 22nd, 2013

    I am facing similar issue. Oracle Metalink says bounce of DB is also required but i will kill this session first. We have no news of DB crash on our case.

    Will take full restart of application and then hopefully issue should resolve.

    • DB bounce is not required – last thing You want to do.
      In fact – there is quite easy link v$session with fnd_concurrent_* tables – so You can identify and kill any session You want.

  1. I have faced the similar issue, however select * from gv$session where program like ‘ICM%’ did not return any row.
    After some period of time, CM did recover itself automatically and the process was relaunched :)

  2. Andrejs – you know that nothing happens by itself, right?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: