Monday, January 5, 2015

Got 'Not enought space' while 'emctl stop oms'

Got 'Not enought space' message while stopping oms (OEM Cloud Control 12.1.0.4 running on Solaris 10 Sparc 64 bit):
$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 4  
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
Error Occurred: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
Please check /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

Check log for details:
$ tail /u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details
2015-01-02 14:00:35,466 [main] ERROR wls.OMSController main.253 - OMSController failed for stop oms
2015-01-02 14:00:35,469 [main] ERROR wls.OMSController main.254 - OMSController Error: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmn
ctl": error=12, Not enough space
java.io.IOException: Cannot run program "/u01/app/oracle/product/12.1.0/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:369)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1228)
at oracle.sysman.emctl.util.EmctlUtil.execCommand(EmctlUtil.java:1217)
at oracle.sysman.emctl.util.WebTierUtil.stopWebTier(WebTierUtil.java:430)
at oracle.sysman.emctl.commands.StopCommand.stopOMS(StopCommand.java:204)
at oracle.sysman.emctl.commands.StopCommand.execute(StopCommand.java:114)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:234)
Caused by: java.io.IOException: error=12, Not enough space
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 9 more

No extra helpful message was found. What kind of space is not enough, OMS Home, /tmp or SWAP?
$ df -k
Filesystem kbytes used avail capacity Mounted on
/devices 0 0 0 0% /devices
... ...
swap 8609640 4896048 3713592 57% /tmp
swap 3713656 64 3713592 1% /var/run
rootpool/oracle 209715200 102092072 107623127 49% /u01
... ...
$ swap -s
total: 33445792k bytes allocated + 2772072k reserved = 36217864k used, 3711289k available

There is >90GB (102092072k) space free for mount point /u01 which can be used by OMS instance, and >3GB(3713592k) space left for /tmp.
It is not reasonable to say the space is not enough. However, is 3.6GB (3711289k) available (free) SWAP space enough for stopping OMS?
I'd like to say yes unless I found MOS Doc 1452151.1. The document is talking about same issues happened while OMS 12c is being started and
mentions 4-5GB of minimum free SWAP space for starting OMS. Could it be same requirement for stopping OMS? Make sense?

Before we enlarge SWAP, we found that /tmp and /var/run are sharing SWAP space. Here, we can ignore /var/run because it is tiny (64k).

Tried to delete garbage files from /tmp and check available SWAP:
$ swap -s
total: 31272184k bytes allocated + 2552208k reserved = 33824392k used, 6271264k available

Now, we have almost 6G SWAP available space and re-run 'emctl stop oms', it works!

Refrerences:

EM 12c Emctl Start OMS Returns Error Occurred: Cannot Run Program "<Middleware_Home>/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not Enough Space (Doc ID 1452151.1)

No comments: