Patch or Upgrade Single instance Database and Oracle Restart Using OEM 13c Fleet Maintenance

Oracle Enterprise Manager (OEM) 13c Database Fleet Maintenance is a centralized mechanism for patching/upgrading of Oracle homes and maintain consistency. It allows OEM Cloud Control administrators to patch database, Oracle Restart (Standalone Grid Infrastructure) or Grid Infrastructure with minimal downtime.

It could be able to patch/upgrade hundreds (even more) of databases at scheduled time in parallel, though it is not easy to successfully patch/upgrade multiple databases at same time. I am going to demonstrate the steps to patch or upgrade single instance database and Oracle Restart and list the top issues I was asked. Although it only provides command line (emcli verb) option to perform the tasks, job creation commands (i.e. image creation, software deployment, database patching/upgrading) will print out a OEM URL used to track the status of fleet operation and check job running details. The URL looks like:

  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=<Execution ID>

  Where,
  <EM Host> is the host name of OMS server, has to be replaced with name of your oms server
  <EM port> is the port of OMS server, has to be replaced with the port of your oms server
  <Execution ID> will be given by emcli command

Sample URL of tracking the status of fleet operation,

https://oms.lab.dbaplus.ca:7799/em/faces/core-jobs-procedureExecutionTracking?executionGUID=78B5B15D42E170CBE0540010E0BB9924

 
Monitoring operation status through given URL is strongly recommended, especially when the operation failed with issues/problems. All error messages listed in Troubleshooting section of this post are copied from the URL given by emcli command.

1. Create/determine reference environments

Reference environment is a same or higher version of Oracle home deployed with required patches. For example, if we want to use Fleet Maintenance to apply OCT 2018 Release Update 12.2.0.1.181016(Patch 28662603) on existing databases, we have to apply this patch on an existing 12.2.0.1 Oracle Home or deploy a new home with the patch(s) before we can use fleet.

Note: The home has to be an OEM-managed target before it can be used as reference environment.

Then connect to OEM repository database as sysman to find out the patched/new-deployed home and its target name:

sysman@OEMREP> col target_name for a44
sysman@OEMREP> col home_location for a37
sysman@OEMREP> col patch_id for a10
sysman@OEMREP> col description for a80
sysman@OEMREP> break on TARGET_NAME on home_location
sysman@OEMREP> select h.target_name,h.home_location, p.patch_id, p.description
  2    from mgmt$oh_home_info h, mgmt$oh_patch p
  3   where p.target_guid=h.target_guid
  4         and h.host_name='host01.lab.dbaplus.ca';
 
TARGET_NAME                                        HOME_LOCATION                        PATCH_ID   DESCRIPTION
-------------------------------------------- ------------------------------------------ ---------- -------------------------------------------------------------
OraDB12Home1_20_host01.lab.dbaplus.ca_1533   /u01/app/oracle/product/12.2.0/dbhome_1    25087834
                                                                                        28662603   Database Oct 2018 Release Update : 12.2.0.1.181016 (28662603)
                                                                                        28662626   OCW OCT 2018 RELEASE UPDATE 12.2.0.1.181016 (28662626)
OraGI12Home1_17_host01.lab.dbaplus.ca_24     /u01/app/grid/product/12.2.0/grid          26839277   DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)
                                                                                        28566910   TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:180802.1448.S) (28566910)
                                                                                        28662603   Database Oct 2018 Release Update : 12.2.0.1.181016 (28662603)
                                                                                        28662626   OCW OCT 2018 RELEASE UPDATE 12.2.0.1.181016 (28662626)
                                                                                        28662640   ACFS OCT 2018 RELEASE UPDATE 12.2.0.1.181016 (28662640)

Here, Oracle Home OraDB12Home1_20_host01.lab.dbaplus.ca_1533 is installed on host host01.lab.dbaplus.ca with three patches 25087834, 28662603 and 28662626, and will be used as reference environment for database Gold Image creation. And OraGI12Home1_17_host01.lab.dbaplus.ca_24 is installed on same host with five patches will be used for reference environment for Oracle Restart Gold Image creation.



2. Create response file for Gold Image creation command

The response file is a text file with following input variables:

   * IMAGE_NAME:
     The name of the gold image. This name must be unique across images
 
   * IMAGE_DESCRIPTION:
     Description of the image
 
   * REF_TARGET_NAME:
     The Oracle home target that will be used to create this gold image. This is the database Oracle Home from the existing environment on which the needed patches have been applied. Following step 1 to find the reference target name
 
   * IMAGE_SWLIB_LOC:
     Path in software library(Enterprise > Provisioning and Patching > Software Library) where the gold image will be stored
     Note: this is NOT physical directory path of OS file system. Therefore, it could fail image creation to enter the path starting with '/'
 
   * STORAGE_TYPE_FOR_SWLIB:
     Storage type configured for software library, can be found from OEM(Setup > Provisioning and Patching > Software Library)
 
   * STORAGE_NAME_FOR_SWLIB:
     Name of software library, can be found from OEM(Setup > Provisioning and Patching > Software Library)
 
   * REF_HOST_CREDENTIALS:
     OEM Named Credential created for connecting to target host. The format is <Named Credential:Credential owner> where:
     <Named Credential> is the named credentials for the host on which the reference Oracle Home is located. This user must be the owner of Oracle home.
     <Credential Owner> is the Enterprise Manager user who owns this Named Credential.
Note: it is only applicable for database home
 
   * REF_GI_CREDENTIALS:
     OEM Named Credential created for connecting to target host. The format is <Named Credential:Credential owner> where:
     <Named Credential> is the named credentials for the host on which the reference Oracle Home is located. This user must be the owner of Oracle home.
     <Credential Owner> is the Enterprise Manager user who owns this Named Credential.
Note: it is only applicable for Oracle Restart home
 
   * WORKING_DIRECTORY:
     The temporary location on the host of reference Oracle home target.

   * VERSION_NAME:
     Version name. An image can have multiple versions. A default version of the image is created and more versions can be added

Sample response file "/home/oracle/createImage_12.2.0.1_db.linux" is created for database home

IMAGE_NAME=Oracle Database 12.2.0.1 for Linux x86-64
IMAGE_DESCRIPTION=Gold image of Oracle Database 12.2.0.1 for Linux x86-64
REF_TARGET_NAME=OraDB12Home1_20_host01.lab.dbaplus.ca_1533
IMAGE_SWLIB_LOC=Database Provisioning Profiles/12.2.0.1.0/Linux x86-64
REF_HOST_CREDENTIALS=HOST_ORACLE:SYSMAN
WORKING_DIRECTORY=/u01/tmp
STORAGE_TYPE_FOR_SWLIB=OmsShared
STORAGE_NAME_FOR_SWLIB=default_loc
VERSION_NAME=12.2.0.1.181016

 

Sample response file "/home/oracle/createImage_12.2.0.1_restart.linux" is created for Oracle Restart home

IMAGE_NAME=Oracle Restart 12.2.0.1 for Linux x86-64
IMAGE_DESCRIPTION=Gold Image of Oracle Restart 12.2.0.1 for Linux x86-64
REF_TARGET_NAME=OraGI12Home1_17_host01.lab.dbaplus.ca_24
IMAGE_SWLIB_LOC=Oracle Home Provisioning Profiles/12.2.0.1.0/Linux x86-64
REF_GI_CREDENTIALS=HOST_GRID:SYSMAN
WORKING_DIRECTORY=/u01/tmp
STORAGE_TYPE_FOR_SWLIB=OmsShared
STORAGE_NAME_FOR_SWLIB=default_loc
VERSION_NAME=12.2.0.1.181016

Note:

  1) OS user defined in Named Credential HOST_ORACLE (i.e. oracle) must
     * be owner of Oracle Home OraDB12Home1_20_host01.lab.dbaplus.ca_1533 (host: host01.lab.dbaplus.ca, location: /u01/app/oracle/product/12.2.0/dbhome_1)
     * have full access on temporary directory /u01/tmp

  2) OS user defined in Named Credential HOST_ORACLE (i.e. grid) must
     * be owner of Oracle Home OraGI12Home1_17_host01.lab.dbaplus.ca_24 (host: host01.lab.dbaplus.ca, location: /u01/app/grid/product/12.2.0/grid)
     * have full access on temporary directory /u01/tmp

3. Create a Gold Image from the reference environment using response file created in step 2

To create gold image, use following command,

   emcli db_software_maintenance -createSoftwareImage -input_file="data:<response file>"

Create database home image,

[oracle@oms]$ emcli login -username=SYSMAN
[oracle@oms]$ emcli db_software_maintenance -createSoftwareImage -input_file="data:/home/oracle/createImage_12.2.0.1_db.linux"

Create Gold Image operation has been submitted successfully with the instance name : 'CreateGoldImageProfile_SYSMAN_10_20_2018_23_25_PM' and execution_guid='78B5B15D42E170CBE0540010E0BB9924'
You can track the status of operation using the following:
EMCLI:
emcli get_instance_status -exec=78B5B15D42E170CBE0540010E0BB9924
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=78B5B15D42E170CBE0540010E0BB9924

Verb createSoftwareImage completed successfully

This operation will create archive from reference Oracle Home and upload the archive to the location of OEM software library defined in response file in step 2. It could take a while (over 10 minutes sometimes) depending on system and network performance.
 
Check the operation status with EMCLI command and the operation is completed when it shows 'COMPLETED' as following,

[oracle@oms]$ emcli get_instance_status -exec=78B5B15D42E170CBE0540010E0BB9924
78B5B15D42E170CBE0540010E0BB9924, PROFILE, CreateGoldImageProfile_SYSMAN_10_20_2018_23_25_PM, COMPLETED

In same way to create Oracle Restart image by running command (only response file name is different from database home image creation command),

[oracle@oms]$ emcli db_software_maintenance -createSoftwareImage -input_file="data:/home/oracle/createImage_12.2.0.1_restart.linux"

To validate Image Creation,

[oracle@oms]$ emcli db_software_maintenance -getImages

===============================================================================================================================
IMAGE ID                  IMAGE NAME                DESCRIPTION                  VERSION        CREATION DATE          OWNER
===============================================================================================================================
79DFB1B2E13C2880E0540010  Oracle Restart 12.2.0.1   Gold Image of Oracle Res    12.2.0.1.0     2018-10-20 17:57:36.0   SYSMAN
E0BB9924                  for Linux x86-64          tart 12.2.0.1 for Linux
                                                     x86-64
-------------------------------------------------------------------------------------------------------------------------------
78B5B15D42DB70CBE0540010  Oracle Database 12.2.0.1  Gold Image of Oracle Data   12.2.0.1.0     2018-10-20 23:25:50.0   SYSMAN 
E0BB9924                  for Linux x86-64          base 12.2.0.1 for Linux                                                         
                                                    x86-64                                                                         
-------------------------------------------------------------------------------------------------------------------------------
Total Rows: 2

Verb getImages completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -getVersions -image_id=79DFB1B2E13C2880E0540010E0BB9924

==================================================================================================================================================
POSITION  VERSION ID            VERSION NAME       DATE CREATED           EXTERNAL ID           HASH CODE             STATUS
==================================================================================================================================================
1         79DEA9AEFA2EC10AE054  12.2.0.1.181016    2018-10-20 17:59:53.0  oracle:defaultServic  C3599535236:B5B21276  CURRENT
          0010E0BB94EC                                                    e:em:provisioning:1:  92581BA23C9F97E084F5
                                                                          cmp:COMP_Component:S  1D9AF
                                                                          UB_OracleCRS:79DEA9A
                                                                          EFA2AC10AE0540010E0B
                                                                          B94EC:0.1
--------------------------------------------------------------------------------------------------------------------------------------------------
Total Rows: 1

Verb getVersions completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -getVersions -image_id=78B5B15D42DB70CBE0540010E0BB9924

==================================================================================================================================================
POSITION   VERSION ID            VERSION NAME      DATE CREATED           EXTERNAL ID           HASH CODE             STATUS
==================================================================================================================================================
1          7B3475C264BD5ABDE054  12.2.0.1.181016   2018-10-20 23:26:06.0  oracle:defaultServic  C1737222158:B088DFCA  CURRENT
           0010E0BB94EC                                                   e:em:provisioning:1:  D4ED143E14DADE07671E
                                                                          cmp:COMP_Component:S  CFFCC
                                                                          UB_OracleDB:7B3475C2
                                                                          64B95ABDE0540010E0BB
                                                                          94EC:0.1
--------------------------------------------------------------------------------------------------------------------------------------------------
Total Rows: 1

Verb getImages completed successfully

 

To delete an image, which will in turn delete all versions of an image and all metadata related to an image such as subscriptions and software library components, use the following command:

emcli db_software_maintenance –deleteImage –image_id=<image_id>

4. Subscribe the Targets to the Created Image

In order to patch an Oracle 12.2.0.1 database to same patch level as Gold Image created earlier (12.2.0.1.181016), the database target has to be subscribed to the image with command,

   emcli db_software_maintenance -subscribeTarget -target_name=<DB target name> -target_type=oracle_database -image_id=<image_id>

Where:
  target_name: The name of the database target that needs to be patched.
  target_type: oracle_database for database, has for Oracle Restart
  image_id: The ID of the gold image to which this target is to be patched and can be found by running the "emcli db_software_maintenance -getImages command" as in previous step.

In following example, DB01 & DB02 are Oracle database 12.2.0.1 running on server host01.lab.dbaplus.ca and DB11 & DB12 are databases running on server host02.lab.dbaplus,ca,

[oracle@oms]$ emcli db_software_maintenance -subscribeTarget -target_name="DB01" -target_type=oracle_database -image_id=78B5B15D42DB70CBE0540010E0BB9924
Target 'DB01' subscribed successfully.
Verb subscribeTarget completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -subscribeTarget -target_name="DB02" -target_type=oracle_database -image_id=78B5B15D42DB70CBE0540010E0BB9924
Target 'DB02' subscribed successfully.
Verb subscribeTarget completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -subscribeTarget -target_name="DB11" -target_type=oracle_database -image_id=78B5B15D42DB70CBE0540010E0BB9924
Target 'DB11' subscribed successfully.
Verb subscribeTarget completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -subscribeTarget -target_name="DB12" -target_type=oracle_database -image_id=78B5B15D42DB70CBE0540010E0BB9924
Target 'DB12' subscribed successfully.
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -getImageSubscriptions -image_id=78B5B15D42DB70CBE0540010E0BB9924

=========================================================================================================
TARGET NAME            TARGET TYPE                  VERSION NAME                DATE SUBSCRIBED
=========================================================================================================
DB01                   oracle_database              No Data                     2018-10-20 23:32:17.0
---------------------------------------------------------------------------------------------------------
DB02                   oracle_database              No Data                     2018-10-20 23:33:00.0
---------------------------------------------------------------------------------------------------------
DB11                   oracle_database              No Data                     2018-10-20 23:34:03.0
---------------------------------------------------------------------------------------------------------
DB12                   oracle_database              No Data                     2018-10-20 23:35:17.0
---------------------------------------------------------------------------------------------------------
Total Rows: 4

Verb getImageSubscriptions completed successfully

Subscribe Oracle Restart on host host02.lab.dbaplus.ca to Oracle Restart image,

[oracle@oms]$ emcli db_software_maintenance -subscribeTarget -target_name="has_host02.lab.dbaplus.ca" -target_type=has -image_id=79DFB1B2E13C2880E0540010E0BB9924
Target 'DB01' subscribed successfully.
Verb subscribeTarget completed successfully
[oracle@oms]$
[oracle@oms]$ emcli db_software_maintenance -getImageSubscriptions -image_id=79DFB1B2E13C2880E0540010E0BB9924

======================================================================================
TARGET NAME                   TARGET TYPE     VERSION NAME      DATE SUBSCRIBED
======================================================================================
has_host02.lab.dbaplus.ca     has             No Data           2018-10-20 23:37:08.0
--------------------------------------------------------------------------------------
Total Rows: 1

Verb getImageSubscriptions completed successfully

 

To unsubscribe a target from an image, use the following command:

emcli db_software_maintenance -unsubscribeTarget -target_name=<target _name> —target_type=<target_type> —image_id=<image_id>

5. Create response file for deploying software
 
The Response file is text file contains the following variables:

  * NEW_ORACLE_HOME_LIST:
    Physical path of new Oracle home, the user given by -normal_credential option in next step will be the owner of this new home
    Note: DO NOT use double quotes to enclose the new home path, though you, an experienced DBA, did that in response file for database silent installation. Otherwise, UPDATE_SIHA step in step 7 will fail with Error: Destination home not found.

  * workingDir:
    Name of temp directory on the host where new Oracle home will be installed. The user given by -normal_credential option in next step must have read/write access to this location.

  * dispatchLoc:
    Name of directory for dispatch scripts, these scripts can be executed by a root user. The user given by -normal_credential option in next step must have read/write access to this location.

  * SKIP_PREREQUISITE_CHECKS=<true|false>:
    The default value is false.

  * SKIP_CVU_CHECK=<true|false>:
    The default value is false.

  * PREREQS_ONLY=<true|false>:
    The default value is false. This can be used for identify errors during the prerequisite checks. The actual deployment will not happen when the value is set to “true”.
 
Sample response file "/home/oracle/deploySoftware_12.2.0.1.db.linux" is created for database home deployment,

NEW_ORACLE_HOME_LIST=/u01/app/oracle/product/12.2.0/dbhome_181016
workingDir=/u01/tmp/DatabaseFleetMaintenance/workingDir
dispatchLoc=/u01/tmp/DatabaseFleetMaintenance/dispatchLoc
SKIP_PREREQUISITE_CHECKS=true
SKIP_CVU_CHECK=true
PREREQS_ONLY=false

Sample response file "/home/oracle/deploySoftware_12.2.0.1.restart.linux" is created for Oracle Restart home deployment,

NEW_ORACLE_HOME_LIST=/u01/app/grid/product/12.2.0/grid_181016
workingDir=/u01/tmp/DatabaseFleetMaintenance/workingDir
dispatchLoc=/u01/tmp/DatabaseFleetMaintenance/dispatchLoc
SKIP_PREREQUISITE_CHECKS=true
SKIP_CVU_CHECK=true
PREREQS_ONLY=false

6. Deploy the Software

Using following command to deploy software,

emcli db_software_maintenance -performOperation -name=<User specified Operation name> -purpose=DEPLOY_DB_SOFTWARE
      -target_type=<type of target> —target_list=<list of targets>
      —normal_credential=<credential name> —privilege_credential=<privileged credential name>
  —inputfile="data:<location of input file>" [-start_schedule="start_time:yyyy/mm/dd HH:mm"] [standbyAutoDeploy=<true|false>]

where:

  * name:
    This is the unique name of the operation.

  * purpose:
    DEPLOY_DB_SOFTWARE for database home deployment, DEPLOY_SIHA_SOFTWARE for Oracle Restrart home.

  * target_type:
    oracle_database for Oracle single instance database,  has for Standalone Grid Infrastructure

  * target_list:
    This is a comma separated list of targets which need to be patched.

    Targets of homogeneous types are supported in a single fleet operation. Unique list of hosts based on this target list is displayed and start stage of Oracle home software on those hosts. If targets running from same Oracle home are provided in this list, the stage and deploy operation will be triggered only once and not for all targets.

  * normal_credential:
    This must be entered in the format <Named Credential: Credential Owner> where:

    <Named Credential>: Named credential for the host where new Oracle home will be deployed.

    <Credential Owner>: The Enterprise Manager user who owns this Named Credential.

The credential is the owner of new Oracle home.

  * privilege_credential:
    This must be entered in the format <Named Credential: Credential Owner> where:

    <Named Credential>: Named credential for the host where new Oracle home will be deployed.

    <Credential Owner>: The Enterprise Manager user who owns this Named Credential.

    The credential is used to run scripts as root.

  * start_schedule:
    The date on which the stage and deploy is to be started if it is to be started in the future.

    Format: "start_time:yyyy/mm/dd HH:mm"

    Note: This is an optional parameter. If no date is provided, the fleet operation will start immediately.

  * standbyAutoDeploy:
    This is an optional parameter with default value as True.

    true: If the target is a primary database, a new Oracle home with the same gold image version as Primary is deployed automatically on the Standby host.

false: This parameter disables the automatic deployment of software on standby host when specified as false. If this value is false, standby staging/deploy can be performed independently using the above emcli command.

6.1 Deploy Oracle Restart

Deploy Oracle Restart home on host02.lab.dbaplus.ca,

[oracle@oms]$ emcli db_software_maintenance -performOperation -name=Deploy_Restart_12201181016 \
> -purpose=DEPLOY_SIHA_SOFTWARE -target_type=has -target_list="has_host02.lab.dbaplus.ca" \
> -normal_credential="HOST_GRID:SYSMAN" -privilege_credential="HOST_ROOT:SYSMAN" \
> -input_file="data:deploySoftware_12.2.0.1.restart.linux"

Processing target "has_host02.lab.dbaplus.ca"...
Checking image subscription...
Check image subscription passed.
Oracle home target list is 361CD7DFFC02977A8C9AF8D141DE7AE0
Operation 'Deploy_Restart_12201181016' created successfully.

Deploy SIHA Software operation for  has_host02.lab.dbaplus.ca can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7A936BE99972C589E0540010E0BB94EC -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7A936BE99972C589E0540010E0BB94EC

Verb performOperation completed successfully

[oracle@oms]$ get_instance_status -exec=7A936BE99972C589E0540010E0BB94EC
7A936BE99972C589E0540010E0BB94EC, DBPROV, Fleet_DEPLOY_host02_11_13_2018_16_22_58_404_PM, COMPLETED

6.2 Deploy Oracle Database home

Deploy Oracle home for databases DB01, DB02, DB11 and DB12,

[oracle@oms]$ emcli db_software_maintenance -performOperation -name=Deploy_DB_12201181016 \
> -purpose=DEPLOY_DB_SOFTWARE -target_type=oracle_database -target_list="DB01,DB02,DB11,DB12" \
> -normal_credential="HOST_ORACLE:SYSMAN" -privilege_credential="HOST_ROOT:SYSMAN" \
> -input_file="data:/home/oracle/deploySoftware_12.2.0.1.db.linux"

Processing target "DB01"...
Checking image subscription...
Check image subscription passed.
Oracle home target list is 672F55267F9D4A0C7AC5A1D91FC5E02D
Operation 'Deploy_DB_12201181016' created successfully.

Deploy Oracle home software operation for  database DB01 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7A93CF02E0D3F713E0540010E0BB94EC -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7A93CF02E0D3F713E0540010E0BB94EC

Processing target "DB02"...

A deploy operation is already in progress for the host of this target. It will be skipped

Processing target "DB11"...
Checking image subscription...
Check image subscription passed.
Oracle home target list is FD5D3676300683D7A6EFCD9D06FC1851
Operation 'Deploy_DB_12201181016' created successfully.

Deploy Oracle home software operation for  database DB11 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7A93CF02E0F8F713E0540010E0BB94EC -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7A93CF02E0F8F713E0540010E0BB94EC

Processing target "DB12"...

A deploy operation is already in progress for the host of this target. It will be skipped

Verb performOperation completed successfully

DB01 and DB02 are running from same Oracle home on same server. Therefor, deployment operation for DB02 is skipped. And DB12 is also skipped for same reason.

Keep eyes on the operation status through browser or by EMCLI command, and make sure the operations are completed successfully,

[oracle@oms]$ emcli get_instance_status -exec=7A93CF02E0D3F713E0540010E0BB94EC
7A93CF02E0D3F713E0540010E0BB94EC, DBPROV, Fleet_DEPLOY_DB01_11_13_2018_17_04_03_745_PM, EXECUTING
[oracle@oms]$
[oracle@oms]$ emcli get_instance_status -exec=7A93CF02E0D3F713E0540010E0BB94EC
7A93CF02E0D3F713E0540010E0BB94EC, DBPROV, Fleet_DEPLOY_DB01_11_13_2018_17_04_03_745_PM, COMPLETED
[oracle@oms]$
[oracle@oms]$ emcli get_instance_status -exec=7A93CF02E0F8F713E0540010E0BB94EC
7A93CF02E0F8F713E0540010E0BB94EC, DBPROV, Fleet_DEPLOY_DB11_11_13_2018_17_04_03_745_PM, COMPLETED

Grant ASM storage access permission to database oracle binary, this is only for role separation installation (Oracle Restart home and Database home have different owner). If both homes are owned by same user (oracle), following command is not needed,

$GRID_HOME/bin/setasmgidwrap -o $ORACLE_HOME/bin/oracle

Where, $GRID_HOME is Oracle Restart home
       $ORACLE_HOME is new database home deployed by fleet

Sample output,

[oracle@host01]$ ls -l /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
-rwsr-s--x 1 oracle oinstall 408662192 Nov 13 17:36 /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
[oracle@host01]$
[oracle@host01]$ /u01/app/12.2.0/grid/bin/setasmgidwrap -o /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
[oracle@host01]$
[oracle@host01]$ ls -l /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
-rwsr-s--x 1 oracle asmadmin 408662192 Nov 13 17:36 /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle

[oracle@host02]$ ls -l /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
-rwsr-s--x 1 oracle oinstall 408662192 Nov 13 17:37 /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
[oracle@host02]$
[oracle@host02]$ /u01/app/12.2.0/grid/bin/setasmgidwrap -o /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
[oracle@host02]$
[oracle@host02]$ ls -l /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle
-rwsr-s--x 1 oracle asmadmin 408662192 Nov 13 17:37 /u01/app/oracle/product/12.2.0/dbhome_181016/bin/oracle

7. Update (patch/upgrade) Database / Oracle Restart

After the deployment operations has been completed, the targets need to be migrated to the newly deployed Oracle homes. Following command will migrate targets to new home and patch/upgrade the targets to the patch level/version of new home,

emcli db_software_maintenance -performOperation -name=<User Specified Operation>
      -purpose=<Operation Type> —target_type=<target type> —target_list=<list of targets>
  —normal_credential=credential name> -privilege_credential=<credential name>
  [-rolling=<true/false] [—input_file=”data:<location of input file>]

where:

  * name:
    This is the unique name of the operation.

  * purpose:
    UPDATE_DB for databases and UPDATE_SIHA for Oracle Restart (ASM instance, listener, etc.).

  * target_type:
    oracle_database for Oracle single instance database,  has for Standalone Grid Infrastructure.

  * target_list:
    This is a comma separated list of targets which need to be patched.

    Targets of homogeneous types are supported in a single fleet operation.

  * normal_credential:
    This must be entered in the format <Named Credential: Credential Owner> where:

    <Named Credential>: Named credential for the host where new Oracle home will be deployed.

    <Credential Owner>: The Enterprise Manager user who owns this Named Credential.

It is the owner of new Oracle home.

  * privilege_credential:
    This must be entered in the format <Named Credential: Credential Owner> where:

    <Named Credential>: Named credential for the host where new Oracle home will be deployed.

    <Credential Owner>: The Enterprise Manager user who owns this Named Credential.

    It is used to run scripts as root.

  * rolling:
    This is an optional flag with the default value as true. The update procedure works in "Rolling Patch" mode by default but you can override this if necessary. If the gold image has OJVM patches, you should mark this value as ‘false’ which will take all instances of RAC DB down for update. This option is applicable for DB patching only (not for Grid Infrastructure or Oracle Restart).

  * input_file:
    This is an optional parameter. The parameter(s) that can be specified here are: workingDir: =<Name of temp directory>.

7.1 Patch Oracle Restart

[oracle@oms]$ db_software_maintenance -performOperation -name=Update_has_12201181016 -purpose=UPDATE_SIHA \
> -target_type=has -target_list="has_host02.lab.dbaplus.ca" -normal_credential="HOST_GRID:SYSMAN" \
> -privilege_credential="HOST_ROOT:SYSMAN"

Processing target "has_host02.lab.dbaplus.ca"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Operation 'Update_has_12201181016' created successfully.

Update operation for SIHA   has_host02.lab.dbaplus.ca can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7AC0CE24E7FCB691E0540010E0BB94EC -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7AC0CE24E7FCB691E0540010E0BB94EC

Verb performOperation completed successfully

[oracle@oms]$ emcli get_instance_status -exec=7AC0CE24E7FCB691E0540010E0BB94EC
7AC0CE24E7FCB691E0540010E0BB94EC, DBPROV, Fleet_UPDATE_has_host02_11_15_2018_22_45_48_539_PM, COMPLETED

[oracle@oms]$ emcli db_software_maintenance -getImageSubscriptions -image_id=79DFB1B2E13C2880E0540010E0BB9924

======================================================================================
TARGET NAME                   TARGET TYPE     VERSION NAME      DATE SUBSCRIBED
======================================================================================
has_host02.lab.dbaplus.ca     has             12.2.0.1.181016   2018-10-20 23:37:08.0
--------------------------------------------------------------------------------------
Total Rows: 1

Verb getImageSubscriptions completed successfully

7.2 Patch databases

If listener is configured and started in Oracle Restart home, following command may be needed before start UPDATE_DB to patch/upgrade databases,

chmod g+w $GRID_HOME/network/admin/listener.ora

where, $GRID_HOME is current Oracle Restart home.

Sample output,

[grid@host01]$ ls -l /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora
-rw-r--r-- 1 grid oinstall 851 Dec  9 07:53 /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora

[grid@host01]$chmod g+w /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora

[grid@host01]$ ls -l /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora
-rw-rw-r-- 1 grid oinstall 851 Dec  9 07:53 /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora

OS user grid (owner of Oracle Restart home) is owner of file listener.ora, and oracle (owner of database home) only had read permission on the file as member of group oinstall. The UPDATE_DB operation will fail at step "Migrate tnsnames and listener entries" with error 'Could not open file listener.ora'. The 'chmod g+w' command grant write permission to oracle through group oinstall.

Four databases: DB01, DB02, DB11 and DB12 are patched in parallel as following,

[oracle@oms]$ emcli db_software_maintenance -performOperation -name=UPDATE_DB_12201181016 \
> -purpose=UPDATE_DB -target_type=oracle_database -target_list="DB01,DB02,DB11,DB12" \
> -normal_credential="HOST_ORACLE:SYSMAN" -privilege_credential="HOST_ROOT:SYSMAN"

Processing target "DB01"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Operation 'UPDATE_DB_12201181016' created successfully.

Patching operation for database DB01 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7AF708FA77953475E0540010E0BB9924 -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7AF708FA77953475E0540010E0BB9924

Processing target "DB02"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Operation 'UPDATE_DB_12201181016' created successfully.

Patching operation for database DB02 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7AF708FA77A23475E0540010E0BB9924 -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7AF708FA77A23475E0540010E0BB9924

Processing target "DB11"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Operation 'UPDATE_DB_12201181016' created successfully.

Patching operation for database DB11 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7AF708FA77BC3475E0540010E0BB9924 -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7AF708FA77BC3475E0540010E0BB9924

Processing target "DB12"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Operation 'UPDATE_DB_12201181016' created successfully.

Patching operation for database DB12 can be monitored using:
EMCLI:
  emcli get_instance_status -exec=7AF708FA77EA3475E0540010E0BB9924 -details -xml
Browser:
  https://<EM Host>:<EM Port>/em/faces/core-jobs-procedureExecutionTracking?executionGUID=7AF708FA77EA3475E0540010E0BB9924

Verb performOperation completed successfully

[oracle@oms]$ emcli get_instance_status -exec=7AF708FA77953475E0540010E0BB9924
7AF708FA77953475E0540010E0BB9924, DBPROV, Fleet_UPDATE_DB01_11_18_2018_15_14_38_024_PM, EXECUTING

[oracle@oms]$ emcli get_instance_status -exec=7AF708FA77953475E0540010E0BB9924
7AF708FA77953475E0540010E0BB9924, DBPROV, Fleet_UPDATE_DB01_11_18_2018_15_14_38_024_PM, COMPLETED

[oracle@oms]$ emcli get_instance_status -exec=7AF708FA77A23475E0540010E0BB9924
7AF708FA77A23475E0540010E0BB9924, DBPROV, Fleet_UPDATE_DB02_11_18_2018_15_14_44_143_PM, COMPLETED

[oracle@oms]$ emcli get_instance_status -exec=7AF708FA77BC3475E0540010E0BB9924
7AF708FA77BC3475E0540010E0BB9924, DBPROV, Fleet_UPDATE_DB11_11_18_2018_15_14_50_035_PM, COMPLETED

[oracle@oms]$ emcli get_instance_status -exec=7AF708FA77EA3475E0540010E0BB9924
7AF708FA77EA3475E0540010E0BB9924, DBPROV, Fleet_UPDATE_DB12_11_18_2018_15_14_57_010_PM, COMPLETED

[oracle@oms]$ emcli db_software_maintenance -getImageSubscriptions -image_id=78B5B15D42DB70CBE0540010E0BB9924

=========================================================================================================
TARGET NAME            TARGET TYPE                  VERSION NAME                DATE SUBSCRIBED
=========================================================================================================
DB01                   oracle_database              12.2.0.1.181016             2018-10-20 23:32:17.0
---------------------------------------------------------------------------------------------------------
DB02                   oracle_database              12.2.0.1.181016             2018-10-20 23:33:00.0
---------------------------------------------------------------------------------------------------------
DB11                   oracle_database              12.2.0.1.181016             2018-10-20 23:34:03.0
---------------------------------------------------------------------------------------------------------
DB12                   oracle_database              12.2.0.1.181016             2018-10-20 23:35:17.0
---------------------------------------------------------------------------------------------------------
Total Rows: 4

Verb getImageSubscriptions completed successfully

8. Troubleshooting

8.1 Creating Oracle Restart Imaged failed at step [Create Grid Infrastructure Gold Image]

Symptoms

Failed - Job Failed

  Step: sourcePrerequisites (Failed)

     Targets host01.lab.dbaplus.ca
java.io.IOException: java.io.FileNotFoundException: /u01/app/oraInventory/locks/inventory.lock (Permission denied)

Diagnose

$ ls -l /u01/app/oraInventory/locks/
total 0
-rw-r--r--    1 oracle   oinstall          0 Nov  1 17:36 inventory.lock

Cause
Job was run by Oracle Restart Home owner (grid) who does not have write access to the file.

Solution
Grant write permission to group, both database home owner 'oracle' and Restart home owner 'grid' are member of group 'oinstall'

$ chmod g+w /u01/app/oraInventory/locks/inventory.lock
$ ls -l /u01/app/oraInventory/locks/
total 0
-rw-rw-r--    1 oracle   oinstall          0 Nov  1 17:36 inventory.lock

8.2 Creating Oracle Restart Imaged failed at step [Fetch Profile Metadata Configuration Properties]

Symptoms

Step: Inline transfer and execution of the Software Library Entities (Failed)

    Start Date    2018.11.02 16:51:09
Completed Date    2018.11.02 16:51:12
       Targets    host01.lab.dbaplus.ca
 
Running in inline mode

Compression is enabled

runInvoke.pl arguments were:
perl             = /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/perl/bin/perl
invokeDir        = /u01/app/oracle/product/em/agent/agent_13.2.0.0.0/sysman/admin/scripts/jobutil/
executeDirective = true
doCleanup        = true
destDir          = /u01/tmp/OUITMP_2017-04-11_10-54-15-AM/JOB_79B53834449903C0E0540010E0BB9924
emdRoot          = /u01/app/oracle/product/em/agent/agent_13.2.0.0.0


Can not create destination directory /u01/tmp/OUITMP_2017-04-11_10-54-15-AM/JOB_79B53834449903C0E0540010E0BB9924: Permission denied

Diagnose

$ ls -ld /u01/tmp
drwxrwxrwx   10 grid     oinstall       4096 Nov 29 13:53 /u01/tmp
$ ls -ld /u01/tmp/OUITMP_2017-04-11_10-54-15-AM
drwxr-xr-x   12 oracle   oinstall       4096 Oct 20 20:14 /u01/tmp/OUITMP_2017-04-11_10-54-15-AM
$ id grid
uid=60006(grid) gid=60012(oinstall) groups=60012(oinstall),60014(asmadmin),60015(asmdba),60016(asmoper),60018(racdba)
$ id oracle
uid=60004(oracle) gid=60012(oinstall) groups=60012(oinstall),60013(dba),60017(oper),60015(asmdba),60018(racdba)

Cause
The normal credential (owner of reference Oracle Restart home) 'grid' has full access to '/u01/tmp' which is defined in Image-creating response file (in step 2), but does not have write access permission on sub-directory 'OUITMP_2017-04-11_10-54-15-AM'. The interesting thing is that the sub-directory was created by Oracle database home owner 'oracle' weeks ago. Looks like the sub-directory was orphaned by failed fleet operation of creating database home image. Why is the fleet operation of creating Oracle Restart home image running with user 'grid' using same temporary sub-directory name as operation of creating oracle database home image running with user 'oracle'? Why does the sub-directory have a more than one year old time stamp suffix (2017-04-11_10-54-15-AM)? A bug?

Solution
Delete the sub-directory

$ rm -fr /u01/tmp/OUITMP_2017-04-11_10-54-15-AM

Or set group permission of the directory to writable, grid is member of the directory group 'oinstall'

$ chmod g+w /u01/tmp/OUITMP_2017-04-11_10-54-15-AM
$ ls -ld /u01/tmp/OUITMP_2017-04-11_10-54-15-AM
drwxrwxr-x   12 oracle   oinstall       4096 Oct 20 20:14 /u01/tmp/OUITMP_2017-04-11_10-54-15-AM

8.3 Create Gold Image fails at step 'Upload Metadata Properties'

Symptom

Source target name is: OraDB12Home1_20_host01.lab.dbaplus.ca_1533
Source target type is: oracle_home
About to fetch the source database details from Enterprise Manager Repository.
================================================================
Reference target 'OraDB12Home1_20_host01.lab.dbaplus.ca_1533' Details
================================================================
Host: host01.lab.dbaplus.ca
Oracle Home Location null
Fetched the source database from from Enterprise Manager Repository successfully.
ExpressionEvaluationException: An unexpected error encountered while executing this Computational step. An incident was created for this event. Below is the error message
----------------------------------------------------------
null
----------------------------------------------------------

Cause

Incorrect value for IMAGE_SWLIB_LOC in image creation response file created in step 2, it represent 'path' in software library(Enterprise > Provisioning and Patching > Software Library) where the gold image will be stored, it is NOT physical directory path of OS file system. Therefore, it could fail gold image creation if the path starts with slash '/'.

Solution
Edit the response file and replace the IMAGE_SWLIB_LOC to remove leading "/" from the beginning as shown in step 2. Then retry the creation of image.

8.4 ERROR: Operation could not be completed  when update database

Symptoms

Processing target "DB02"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.

ERROR: Operation could not be completed
Verb performOperation unsuccessful

Cause

Database is in migrate mode (startup upgrade).

Solution

Restart database in normal mode.

8.5 UPDATE_DB failed at step "Start DB Instances from Destination Home"

Symptom 1

SQL> SQL> Connected to an idle instance.
SQL> SQL> SP2-0223: No lines in SQL buffer.
SQL> ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/DB11/spfileTEST101.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/DB11/spfileDB11.ora
ORA-15173: entry 'spfiledb11.ora' does not exist in directory 'DB11'
ORA-06512: at line 4
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 504
ORA-06512: at line 2
Disconnected

 Status code is 1
Could not change DB state. Return Code: 1.

Symptom 2

SQL> SQL> Connected to an idle instance.
SQL> SQL> SP2-0223: No lines in SQL buffer.
SQL> ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/TEST02/PARAMETERFILE/spfile.459.991588999'
ORA-17503: ksfdopn:2 Failed to open file +DATA/TEST02/PARAMETERFILE/spfile.459.991588999
ORA-15056: additional error message
ORA-06512: at line 4
ORA-17503: ksfdopn:2 Failed to open file +DATA/TEST02/PARAMETERFILE/spfile.459.991588999
ORA-12578: TNS:wallet open failed
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 391
ORA-06512: at line 2
Disconnected

 Status code is 1
Could not change DB state. Return Code: 1.

Cause

The fleet operation cannot start database with spfile on ASM diskgroup. It should be considered as bug.

Workaround

Make a copy of spfile from ASM diskgroup to new Oracle home '$ORACLE_HOME/dbs' and remove it from Oracle home after fleet operation is completed.

8.6 UPDATE_DB failed at step "Migrate DB Instances for Patching"

Symptoms

Fri Nov  9 16:30:35 2018 - TEST100 is HA Managed

Fri Nov  9 16:30:35 2018 - Running command /u01/app/oracle/product/12.2.0/dbhome_1/bin/srvctl modify database -d TEST100 -o /u01/app/oracle/product/12.2.0/dbhome_181016 to update the New Oracle Home location

/u01/app/oracle/product/12.2.0/dbhome_1/bin/srvctl modify database -d TEST100 -o /u01/app/oracle/product/12.2.0/dbhome_181016
PRCD-1163 : Failed to modify database TEST100
PRCR-1071 : Failed to register or update resource ora.test100.db
CRS-0245:  User doesn't have enough privilege to perform the operation

Cause

Oracle Restart home (where ASM instance and listener are running) is version 12.2.0.1 and owned by grid, and both old database home and new database home are version 12.2.0.1 and owned by oracle. The issue happens while srvctl is executed by oracle to change home of database from old one to new one. This only happened with Oracle database 12.2.0.1.x when Oracle Restart is configured with different owner from database home owner.

Oracle database support team insists that it is expected/designed behavior of srvctl and imply that OEM should not run this way. In another word, it is OEM bug?

Workaround

While the fleet operation stopped at the step because of this failure, manually remove database from old oracle home by running following command from old home

$ srvctl config service -d <db_unique_name>
$ srvcrl remove database -d <db_unique_name>  -f

And add (srvctl add database) database back to new oracle home by running following command from new home

$ srvctl add database -d <db_unique_name> -o $ORACLE_HOME -p <spfile>
$ srvctl add service -d <db_unique_name> -s <service_name> -l PRIMARY -y AUTOMATIC [-pdb <pdb_name>]

Go back to OEM console and open Procedure Activity window of failed fleet job, then check (select) the failed step and job details will display on the right of the window. Click drop-down button 'Actions', then click 'Ignore' and click 'OK' to confirm. the fleet job will skip current step and resume from next step.


8.7  DEPLOY_SIHA_SOFTWARE failed at step "Transfer Common Perl Module"

Symptom

Failed - Job Failed

Step: Execute Command (Error)

    Start Date    2018.11.13 14:55:35
Completed Date    2018.11.13 14:55:35
       Targets    host02.lab.dbaplus.ca
ERROR: Execution failed due to binary missing or permission issues

Diagnose
This step will start script 'root_dispatcher.sh' from directory '/u01/tmp/DatabaseFleetMaintenance/dispatchLoc' and the directory is defined in deployment response file created in step 5. Check the script status,

[root@host02]# ls -l /u01/tmp/DatabaseFleetMaintenance/dispatchLoc
total 16
-rwx------    1 root     oracle         5523 Nov 13 14:52 root_dispatcher.sh
[root@host02]# head /u01/tmp/DatabaseFleetMaintenance/dispatchLoc/root_dispatcher.sh
#!/bin/bash
#
# $Header: emdb/sysman/metadata/swlib/common/directives/root_dispatcher.sh /st_emgc_pt-13.2.0.0.0pg/2 2016/11/18 00:23:58 mhtrived Exp $
[root@host02]# which bash
which: 0652-141 There is no bash in /usr/local/bin /usr/bin /etc /usr/sbin /usr/ucb /home/oracle/bin /usr/bin/X11 /sbin ..

Cause
Script rot_dispatcher.sh needs bash shell which is not installed on the target host.

Solution
Install bash shell on target host and re-submit new fleet job.

8.8 UPDATE_DB Fails With Destination Home Not Found

Symptom

After Oracle Home is deployed successfully (step 6), UPDATE_DB (step 7) fails as following,

[oracle@oms]$ emcli db_software_maintenance -performOperation -name=UPDATE_DB_12201181016 \
> -purpose=UPDATE_DB -target_type=oracle_database -target_list="DB01" \
> -normal_credential="HOST_ORACLE:SYSMAN" -privilege_credential="HOST_ROOT:SYSMAN"
Processing target "DB01"...
Checking if target is already running from the current version of the image...
Check Passed.
Checking image subscription...
Check image subscription passed.
Destination home not found. Possible causes are either the Oracle home was not deployed in case you're trying to do a update or the Oracle home has been cleaned up in case you're trying to do a cleanup.

Cause

Although new Oracle home is deployed successfully with gold image, the Oracle home metadata in OEM repository is not up-to-date.

Solution

In OEM console, go to home page of the new Oracle home and click 'Refresh Configuration' to update the metadata.

8.9 UPDATE_DB fails at step "Migrate tnsnames and listener entries" with error 'Could not open file listener.ora'

Symptom

Final command line to execute is $PERL_HOME/perl "/tmp/JOB_7AF714933A6D3A4CE0540010E0BB9924/configListener.pl" -srcOracleHomeLoc="/u01/app/oracle/product/12.2.0/dbhome_1" -newOracleHomeLoc="/u01/app/oracle/product/12.2.0/dbhome_181016" -tnsAdminLoc="/u01/app/grid/product/12.2.0/grid/network/admin" -newTnsAdminLoc="/u01/app/grid/product/12.2.0/grid/network/admin" -listenerName="LISTENER" -switchbackMode="" -oldListenerName="" -migrateLsnr="false" -dbSIDs="DB01"

The output of the directive is:
Unknown option: oldlistenername
Unknown option: migratelsnr
Could not open file listener.ora: The file access permissions do not allow the specified action. at /tmp/JOB_7AF714933A6D3A4CE0540010E0BB9924/configListener.pl line 734.
ORACLE_HOME set to /u01/app/oracle/product/12.2.0/dbhome_181016

Diagnose

[grid@host01]$ ls -l /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora
-rw-r--r-- 1 grid oinstall 851 Dec  9 07:53 /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora

OS user grid (owner of Oracle Restart home) is owner of file listener.ora. Owner of database home oracle only has read permission on the file as member of group oinstall.

Solution
Grant write permission to oracle through group oinstall with command 'chmod g+w',

[grid@host01]$chmod g+w /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora

[grid@host01]$ ls -l /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora
-rw-rw-r-- 1 grid oinstall 851 Dec  9 07:53 /u01/app/grid/product/12.2.0/grid/network/admin/listener.ora