Invalid password while logging 12c ASM instance

Customer complains that he cannot log into 12c ASM instance because of invalid username/password, but he is pretty sure username and password are correct.

$ . oraenv
ORACLE_SID = [+ASM] ? +ASM
The Oracle base remains unchanged with value /u01/app
$
$ sqlplus /nolog

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 2 10:26:38 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

SQL> connect sys as sysasm
Enter password:
ERROR:
ORA-01017: invalid username/password; logon denied

Try to log in through OS authentication and reset the password:

SQL> conn / as sysasm
Connected.

SQL> alter user sys identified by abc123;
alter user sys identified by abc123
                             *
ERROR at line 1:
ORA-01918: user 'INTERNAL' does not exist

SQL> select * from v$pwfile_users;

no rows selected

Looks like the password file is missing:

$ ls -l $ORACLE_HOME/dbs
total 18
-rw-rw----   1 oragrid  oracle      2180 Jul 24 13:32 ab_+ASM.dat
-rw-rw----   1 oragrid  oracle      1544 Oct  2 10:28 hc_+ASM.dat
-rw-r--r--   1 oragrid  oracle      2992 Feb  3  2012 init.ora

Ok, the password file is not there as before. And we know it can be stored in ASM diskgroup from 12c, check if it is the case:

$ srvctl config asm -a
ASM home: 
Password file:
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
$

The entry 'Password file' is empty, it is not expected configuration if password file does not exist in $ORACLE_HOME/dbs. Let us see if there is password file in ASM:

$ asmcmd
ASMCMD> ls -l
State    Type    Rebal  Name
MOUNTED  EXTERN  N      DATA/
MOUNTED  EXTERN  N      FRA/
ASMCMD> cd DATA
ASMCMD> ls -l
Type      Redund  Striped  Time             Sys  Name
                                            Y    ASM/
                                            Y    DB01/
PASSWORD  UNPROT  COARSE   JUL 14 09:00:00  N    orapwasm => +DATA/ASM/PASSWORD/pwdasm.256.885030391
ASMCMD> exit

The password file 'orapwasm' is in 'DATA' diskgroup, update Oracle Restart registry:

$ srvctl modify asm -pwfile +DATA/orapwasm
$ srvctl config asm -a
ASM home: 
Password file: +DATA/orapwasm
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.885907629
ASM diskgroup discovery string:
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:

try logging again:

$ sqlplus sys as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Fri Oct 16 11:16:04 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS SYSBA SYSDG SYSKM     CON_ID
------------------------------ ----- ----- ----- ----- ----- ----- ----------
SYS                            TRUE  TRUE  TRUE  FALSE FALSE FALSE          0
ASMSNMP                        TRUE  FALSE FALSE FALSE FALSE FALSE          0

It works.