Friday, June 23, 2017

Creating ASM disks using ASM Filter Driver (ASMFD)

Starting 12.1.0.2 we can use ASM Filter Drive (ASMFD) to create ASM disks to be used for ASM diskgroups. I have already explain in this article that how we configure ASMFD during GI installation, and this article for migration from ASMLib to ASMFD. In this article I will show you how to create/add new ASM disks using ASMFD so that we can use them for existing or new diskgroups.
Currently I have a disk /dev/sdb1 that is already configured using ASMFD, and now I have a new physical disk /dev/sdc1 that I will be configuring using ASMFD. On asmcmd prompt, if you issue command “help”, you will see list of options starting with “afd” and all these options are available for using ASM Filter Driver

afd_configure, afd_deconfigure, afd_di, afd_dsget, afd_dsset, afd_filter, afd_label, afd_lsdsk, afd_lslbl, afd_refresh,  afd_scan, afd_state, afd_unlabel
You can further issue help command for each of these options to find out the detailed working of that option.
ASMCMD> help afd_label
afd_label
        To set the given label to the specified disk

Synopsis
        afd_label <label> <disk> [--rename | --migrate] [--init]

Description
        The options for the afd_label command are described below.
        '--init' option is not allowed if AFD is already loaded.

      label                 - label for the disk
      disk                  - disk path
      --rename              - to relabel a disk that was labeled earlier
      --migrate             - to label a disk that was provisioned for ASM
      --init                - to set AFD label during initialize stage

Examples
        The following example labels a disk /dev/sdq with label DATA1.

        ASMCMD [+] >afd_label DATA1 /dev/sdq

See Also
       afd_unlabel afd_lsdsk
I will be using afd_label option to label my new disk so that it can be used by ASM.
As root, execute following command
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA2 /dev/sdc1
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_lslbl
--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DATA                                  /dev/sdb1
DATA2                                 /dev/sdc1

-- Labeled disks are also visible at OS level jut like previously we had /dev/oracleasm directory to view disks labeled using ASMLib

[root@salman11 disks]# ls -l /dev/oracleafd/disks/
total 8
-rwxrwx---. 1 grid oinstall 10 Mar 22 12:38 DATA
-rwxrwx---. 1 grid oinstall 10 Mar 22 13:27 DATA2
Immediately after labelling, disk can be seen under V$ASM_DISK view and can be made part of a diskgroup. Log in as SYSASM and execute following. DATA2 is the newly labelled disk.

SQL> conn sys as sysasm
Enter password:
Connected.
SQL> select path from v$asm_disk;

PATH
--------------------------------------------------------------------------------
AFD:DATA2
AFD:DATA

SQL>  alter diskgroup data add disk 'AFD:DATA2';

Diskgroup altered.
If disk is still not visible in V$ASM_DISK, you can execute “afd_scan” and disk should be visible now.
Removing Labels of Disk(s)
If you want to remove a disk from ASM, drop it from the ASM diskgroup first (if it is part of an ASM diskgroup), and then un-label it using ASMFD

-- Drop disk from diskgroup if disk is part of a diskgroup (connect as SYSASM).

SQL> alter diskgroup data drop disk DATA2;

Diskgroup altered.
-- As root, unlabel the disk
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_unlabel DATA2
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_lslbl
--------------------------------------------------------------------------------
Label                     Duplicate  Path
=====================================================================
DATA                                  /dev/sdb1

Saturday, June 17, 2017

ASMCMD-9524: AFD configuration failed 'ERROR: ASMLib deconfiguration failed'

While configuring ASMFD (ASM Filer Driver), I was playing around and started facing this error after issuing “afd_configure” command
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
ASMCMD-9524: AFD configuration failed 'ERROR: ASMLib deconfiguration failed'

I simply stopped oracleasm and acfsload service and after that reissued the “afd_configure” command, and it started working fine.
Point to node is that we need to stop oracleasm and acfsload before configuring ASMFD using “afd_configure”.
[root@salman11 ~]# oracleasm exit
[root@salman11 ~]# $GRID_HOME/bin/acfsload stop

And now configuration command completed successfully
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

Saturday, June 10, 2017

Migrating from ASMLib to Oracle ASM Filter Driver (ASMFD)

Starting 12.1.0.2, ASMFD can be used instead of ASMLib for the ASM disks management. In this article I will explain how to migrate from ASMLib to ASMFD. I already have ASMLib configured on my server and I want to move to using ASMFD. For new GI installation, we can configure ASMFD for our ASM disks from GI installation wizard. Method explained here works for both standalone GI installations as well as for RAC.

As root user, get list of all current disks being used by ASM.
[root@salman11 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@salman11 ~]# oracleasm listdisks
DATA
[root@salman11 ~]# oracleasm querydisk -p DATA
Disk "DATA" is a valid ASM disk
/dev/sdb1: LABEL="DATA" TYPE="oracleasm"

So I have only one disk DATA (/dev/sdb1) right now that I will migrate to ASMFD.
Next step is to check the state of ASMFD, then we will reset the DISKSTRING parameter by using asmcmd command. As grid user, execute following set of commands. Please note that for my ASM disks I am using discovery path as ‘ORCL*’, in your case, you might be using full disk path (/dev/sd*) instead.
For RAC, perform this on all nodes.
[grid@salman11 bin]$ export ORACLE_HOME=/u01/app/12.2.0/grid/
[grid@salman11 bin]$ export PATH=$ORACLE_HOME/bin:$PATH
[grid@salman11 bin]$ export ORACLE_SID=+ASM

-- Check state of ASMFD
[grid@salman11 bin]$ asmcmd afd_state
ASMCMD-9530: The AFD state is 'NOT SUPPORTED'

-- Check current diskstring parameter setting. For me, diskstring set is ORCL:*. For other cases, it may be something like “/dev/oracleasm/disks/*”.
 [grid@salman11 bin]$ asmcmd dsget
parameter:ORCL:*
profile:ORCL:*

-- Add ‘AFD:*’ to the diskstring. It should contain old diskstring path “ORCL:*” (or if your disksting is /dev/oracleasm/disks/*), and also new disk string path for ASMFD which is “AFD:*”
[grid@salman11 bin]$ asmcmd dsset 'ORCL:*','AFD:*'
[grid@salman11 bin]$ asmcmd dsget
parameter:ORCL:*,  AFD:*
profile:ORCL:*,AFD:*

-- Above command will also set disk_String parameter in ASM instance to point to current disk discovery path and also new AFD path.
[grid@salman11 bin]$ sqlplus

SQL*Plus: Release 12.2.0.1.0 Production on Tue Mar 21 13:09:50 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Enter user-name: sys as sysasm
Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string      ORCL:*, AFD:*


As root, stop the “has” services or CRS.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl stop has

-- For RAC, use following command to stop CRS. Execute this on all nodes
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl stop crs

Stop oracleasm driver and acfsload. Execute following as root (on all nodes in case of RAC)
[root@salman11 ~]# oracleasm exit
Unmounting ASMlib driver filesystem: /dev/oracleasm
Unloading module "oracleasm": oracleasm

[root@salman11 ~]# /u01/app/12.2.0/grid/bin/acfsload stop

As root, configure ASMFD. This will automatically de-configure ASMLib also. For RAC, perform this on one node and once it completes, check the state of ASMFD on all nodes. If it is not configured on all nodes, then execute this step on all nodes.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

-- Check the status of ASMFD (on all nodes)
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'salman11.salman.com'

Only if you are using 12.1.0.2, as root user, migrate all your disk(s) to ASMFD. Perform following step for this.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA /dev/sdb1 --migrate

For versions above 12.1.0.2, if you execute this step, you will see an error as bellow because this migration is automatically performed during ASMFD driver configuration and above step for migration is not needed.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA /dev/sdb1 --migrate

label name DATA is already in use
ASMCMD-9513: ASM disk label set operation failed.


List the disks to confirm if disk(s) have been migrated successfully. Disk(s) will also be listed under /dev/oraclefd/disk directory (If you can recall, for ASMLib; disks were listed under /dev/oracleasm/disks)

[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
====================================================================
DATA                        ENABLED   /dev/sdb1

[root@salman11 ~]# ls -l /dev/oracleafd/disks/
total 4
-rwxrwx---. 1 grid oinstall 10 Mar 22 16:30 DATA
[root@salman11 ~]#

As root user, start asfs first, and then “has” for standalone GI installations.  Start “CRS” for RAC installations (On all nodes)
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed


[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl start has

-- For RAC
/u01/app/12.2.0/grid/bin/crsctl start crs

It is better to deinstall oracleasm (Form all nodes in case of RAC)
[root@salman11 ~]# rpm -q oracleasm-support
oracleasm-support-2.1.8-3.1.el7.x86_64
[root@salman11 ~]# rpm -e oracleasm-support-2.1.8-3.1.el7.x86_64
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave

Since ASMFD will be used now, we can set the diskstring to use only ‘AFD:*’, and remove ‘ORCL:*’ from diskstring.
As grid user, execute following command
[grid@salman11 ~]$ asmcmd dsget
parameter:ORCL:*, AFD:*
profile:ORCL:*,AFD:*

[grid@salman11 ~]$ asmcmd dsset 'AFD:*'

[grid@salman11 ~]$ asmcmd dsget
parameter:AFD:*
profile:AFD:*