Zero-Downtime Grid Infrastructure Patching root.sh failed with CLSRSC-675: Cluster-wide Grid Plug and Play profile not found

 When running root.sh with zero-downtime GI patching option -transparent and -nodriverupdate, got following errors

2020/11/05 11:55:01 CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node.

Died at /u01/app/19.9.0/grid/crs/install/crsgpnp.pm line 3436.

The command '/u01/app/19.9.0/grid/perl/bin/perl -I/u01/app/19.9.0/grid/perl/lib -I/u01/app/19.9.0/grid/crs/install /u01/app/19.9.0/grid/srvm/admin/rhpmovegi.pl /u01/app/19.0.0/grid /u01/app/19.9.0/grid true' execution failed

Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_11-55-01AM.log

Using configuration parameter file: /u01/app/19.9.0/grid/crs/install/crsconfig_params

The log of current session can be found at:

  /u01/app/grid/crsdata/rac02/crsconfig/crs_prepatch_apply_oop_rac02_2020-11-05_11-55-01AM.log

2020/11/05 11:55:01 CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node.

Died at /u01/app/19.9.0/grid/crs/install/crsgpnp.pm line 3436.

The command '/u01/app/19.9.0/grid/perl/bin/perl -I/u01/app/19.9.0/grid/perl/lib -I/u01/app/19.9.0/grid/crs/install /u01/app/19.9.0/grid/crs/install/rootcrs.pl  -transparent -nodriverupdate -dstcrshome /u01/app/19.9.0/grid -prepatch' execution failed

Check the log file crs_prepatch_apply_oop_rac02_2020-11-05_11-55-01AM.log

[grid@rac02]$ tail /u01/app/grid/crsdata/rac02/crsconfig/rootcrs_rac02_2020-11-05_11-55-01AM.log

2020-11-05 11:55:01: Checking if GPnP setup exists at the source dir /u01/app/19.0.0/grid

2020-11-05 11:55:01: GPnP directories are /u01/app/19.0.0/grid/gpnp/profiles /u01/app/19.0.0/grid/gpnp/wallets /u01/app/19.0.0/grid/gpnp/profiles/peer /u01/app/19.0.0/grid/gpnp/wallets/root /u01/app/19.0.0/grid/gpnp/wallets/pa /u01/app/19.0.0/grid/gpnp/wallets/peer /u01/app/19.0.0/grid/gpnp/wallets/prdr

2020-11-05 11:55:01: GPnP files are /u01/app/19.0.0/grid/gpnp/wallets/root/ewallet.p12 /u01/app/19.0.0/grid/gpnp/wallets/peer/cwallet.sso /u01/app/19.0.0/grid/gpnp/wallets/prdr/cwallet.sso /u01/app/19.0.0/grid/gpnp/wallets/pa/cwallet.sso /u01/app/19.0.0/grid/gpnp/profiles/peer/profile.xml

2020-11-05 11:55:01: GPnP file /u01/app/19.0.0/grid/gpnp/wallets/root/ewallet.p12 is either not present or of zero size

2020-11-05 11:55:01: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 675

2020-11-05 11:55:01: Executing cmd: /u01/app/19.0.0/grid/bin/clsecho -p has -f clsrsc -m 675

2020-11-05 11:55:01: Command output:

>  CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node. 

>End Command output

2020-11-05 11:55:01: CLSRSC-675: Cluster-wide Grid Plug and Play profile not found on this node.

It is complaining that following GPnP files are missing on current node

  /u01/app/19.0.0/grid/gpnp/wallets/root/ewallet.p12

  /u01/app/19.0.0/grid/gpnp/wallets/peer/cwallet.sso

  /u01/app/19.0.0/grid/gpnp/wallets/prdr/cwallet.sso

  /u01/app/19.0.0/grid/gpnp/wallets/pa/cwallet.sso

  /u01/app/19.0.0/grid/gpnp/profiles/peer/profile.xml

Confirm the files are missing from current node

[grid@rac02]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/root

total 0

[grid@rac02]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/peer

total 0

[grid@rac02]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/prdr

total 0

[grid@rac02]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/pa

total 0

[grid@rac02]$ ls -l /u01/app/19.0.0/grid/gpnp/profiles/peer

total 0

Check if these files exist on other node

[grid@rac01]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/root

total 4

-rwx------ 1 grid oinstall 2408 Sep 10 16:29 ewallet.p12

[grid@rac01]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/peer

total 4

-rwx------ 1 grid oinstall 2933 Sep 10 16:29 cwallet.sso

[grid@rac01]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/prdr

total 4

-rwxr-x--- 1 grid oinstall 1725 Sep 10 16:29 cwallet.sso

[grid@rac01]$ ls -l /u01/app/19.0.0/grid/gpnp/wallets/pa

total 4

-rwx------ 1 grid oinstall 2933 Sep 10 16:29 cwallet.sso

[grid@rac01]$ ls -l /u01/app/19.0.0/grid/gpnp/profiles/peer

total 12

-rw-r--r-- 1 grid oinstall 2153 Sep 10 16:29 profile.old

-rw-r--r-- 1 grid oinstall 2094 Sep 10 16:29 profile_orig.xml

-rw-r--r-- 1 grid oinstall 2153 Sep 10 16:29 profile.xml

Copy the files from remote node rac01 to current host rac02, and re-run root.sh. The issue is gone.