Friday, June 23, 2017

Creating ASM disks using ASM Filter Driver (ASMFD)

Starting 12.1.0.2 we can use ASM Filter Drive (ASMFD) to create ASM disks to be used for ASM diskgroups. I have already explain in this article that how we configure ASMFD during GI installation, and this article for migration from ASMLib to ASMFD. In this article I will show you how to create/add new ASM disks using ASMFD so that we can use them for existing or new diskgroups.
Currently I have a disk /dev/sdb1 that is already configured using ASMFD, and now I have a new physical disk /dev/sdc1 that I will be configuring using ASMFD. On asmcmd prompt, if you issue command “help”, you will see list of options starting with “afd” and all these options are available for using ASM Filter Driver

afd_configure, afd_deconfigure, afd_di, afd_dsget, afd_dsset, afd_filter, afd_label, afd_lsdsk, afd_lslbl, afd_refresh,  afd_scan, afd_state, afd_unlabel
You can further issue help command for each of these options to find out the detailed working of that option.
ASMCMD> help afd_label
afd_label
        To set the given label to the specified disk

Synopsis
        afd_label <label> <disk> [--rename | --migrate] [--init]

Description
        The options for the afd_label command are described below.
        '--init' option is not allowed if AFD is already loaded.

      label                 - label for the disk
      disk                  - disk path
      --rename              - to relabel a disk that was labeled earlier
      --migrate             - to label a disk that was provisioned for ASM
      --init                - to set AFD label during initialize stage

Examples
        The following example labels a disk /dev/sdq with label DATA1.

        ASMCMD [+] >afd_label DATA1 /dev/sdq

See Also
       afd_unlabel afd_lsdsk
I will be using afd_label option to label my new disk so that it can be used by ASM.
As root, execute following command
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA2 /dev/sdc1
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_lslbl
--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DATA                                  /dev/sdb1
DATA2                                 /dev/sdc1

-- Labeled disks are also visible at OS level jut like previously we had /dev/oracleasm directory to view disks labeled using ASMLib

[root@salman11 disks]# ls -l /dev/oracleafd/disks/
total 8
-rwxrwx---. 1 grid oinstall 10 Mar 22 12:38 DATA
-rwxrwx---. 1 grid oinstall 10 Mar 22 13:27 DATA2
Immediately after labelling, disk can be seen under V$ASM_DISK view and can be made part of a diskgroup. Log in as SYSASM and execute following. DATA2 is the newly labelled disk.

SQL> conn sys as sysasm
Enter password:
Connected.
SQL> select path from v$asm_disk;

PATH
--------------------------------------------------------------------------------
AFD:DATA2
AFD:DATA

SQL>  alter diskgroup data add disk 'AFD:DATA2';

Diskgroup altered.
If disk is still not visible in V$ASM_DISK, you can execute “afd_scan” and disk should be visible now.
Removing Labels of Disk(s)
If you want to remove a disk from ASM, drop it from the ASM diskgroup first (if it is part of an ASM diskgroup), and then un-label it using ASMFD

-- Drop disk from diskgroup if disk is part of a diskgroup (connect as SYSASM).

SQL> alter diskgroup data drop disk DATA2;

Diskgroup altered.
-- As root, unlabel the disk
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_unlabel DATA2
[root@salman11 disks]# /u01/app/12.2.0/grid/bin/asmcmd afd_lslbl
--------------------------------------------------------------------------------
Label                     Duplicate  Path
=====================================================================
DATA                                  /dev/sdb1

Saturday, June 17, 2017

ASMCMD-9524: AFD configuration failed 'ERROR: ASMLib deconfiguration failed'

While configuring ASMFD (ASM Filer Driver), I was playing around and started facing this error after issuing “afd_configure” command
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
ASMCMD-9524: AFD configuration failed 'ERROR: ASMLib deconfiguration failed'

I simply stopped oracleasm and acfsload service and after that reissued the “afd_configure” command, and it started working fine.
Point to node is that we need to stop oracleasm and acfsload before configuring ASMFD using “afd_configure”.
[root@salman11 ~]# oracleasm exit
[root@salman11 ~]# $GRID_HOME/bin/acfsload stop

And now configuration command completed successfully
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

Saturday, June 10, 2017

Migrating from ASMLib to Oracle ASM Filter Driver (ASMFD)

Starting 12.1.0.2, ASMFD can be used instead of ASMLib for the ASM disks management. In this article I will explain how to migrate from ASMLib to ASMFD. I already have ASMLib configured on my server and I want to move to using ASMFD. For new GI installation, we can configure ASMFD for our ASM disks from GI installation wizard. Method explained here works for both standalone GI installations as well as for RAC.

As root user, get list of all current disks being used by ASM.
[root@salman11 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@salman11 ~]# oracleasm listdisks
DATA
[root@salman11 ~]# oracleasm querydisk -p DATA
Disk "DATA" is a valid ASM disk
/dev/sdb1: LABEL="DATA" TYPE="oracleasm"

So I have only one disk DATA (/dev/sdb1) right now that I will migrate to ASMFD.
Next step is to check the state of ASMFD, then we will reset the DISKSTRING parameter by using asmcmd command. As grid user, execute following set of commands. Please note that for my ASM disks I am using discovery path as ‘ORCL*’, in your case, you might be using full disk path (/dev/sd*) instead.
For RAC, perform this on all nodes.
[grid@salman11 bin]$ export ORACLE_HOME=/u01/app/12.2.0/grid/
[grid@salman11 bin]$ export PATH=$ORACLE_HOME/bin:$PATH
[grid@salman11 bin]$ export ORACLE_SID=+ASM

-- Check state of ASMFD
[grid@salman11 bin]$ asmcmd afd_state
ASMCMD-9530: The AFD state is 'NOT SUPPORTED'

-- Check current diskstring parameter setting. For me, diskstring set is ORCL:*. For other cases, it may be something like “/dev/oracleasm/disks/*”.
 [grid@salman11 bin]$ asmcmd dsget
parameter:ORCL:*
profile:ORCL:*

-- Add ‘AFD:*’ to the diskstring. It should contain old diskstring path “ORCL:*” (or if your disksting is /dev/oracleasm/disks/*), and also new disk string path for ASMFD which is “AFD:*”
[grid@salman11 bin]$ asmcmd dsset 'ORCL:*','AFD:*'
[grid@salman11 bin]$ asmcmd dsget
parameter:ORCL:*,  AFD:*
profile:ORCL:*,AFD:*

-- Above command will also set disk_String parameter in ASM instance to point to current disk discovery path and also new AFD path.
[grid@salman11 bin]$ sqlplus

SQL*Plus: Release 12.2.0.1.0 Production on Tue Mar 21 13:09:50 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Enter user-name: sys as sysasm
Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string      ORCL:*, AFD:*


As root, stop the “has” services or CRS.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl stop has

-- For RAC, use following command to stop CRS. Execute this on all nodes
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl stop crs

Stop oracleasm driver and acfsload. Execute following as root (on all nodes in case of RAC)
[root@salman11 ~]# oracleasm exit
Unmounting ASMlib driver filesystem: /dev/oracleasm
Unloading module "oracleasm": oracleasm

[root@salman11 ~]# /u01/app/12.2.0/grid/bin/acfsload stop

As root, configure ASMFD. This will automatically de-configure ASMLib also. For RAC, perform this on one node and once it completes, check the state of ASMFD on all nodes. If it is not configured on all nodes, then execute this step on all nodes.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

-- Check the status of ASMFD (on all nodes)
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'salman11.salman.com'

Only if you are using 12.1.0.2, as root user, migrate all your disk(s) to ASMFD. Perform following step for this.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA /dev/sdb1 --migrate

For versions above 12.1.0.2, if you execute this step, you will see an error as bellow because this migration is automatically performed during ASMFD driver configuration and above step for migration is not needed.
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_label DATA /dev/sdb1 --migrate

label name DATA is already in use
ASMCMD-9513: ASM disk label set operation failed.


List the disks to confirm if disk(s) have been migrated successfully. Disk(s) will also be listed under /dev/oraclefd/disk directory (If you can recall, for ASMLib; disks were listed under /dev/oracleasm/disks)

[root@salman11 ~]# /u01/app/12.2.0/grid/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
====================================================================
DATA                        ENABLED   /dev/sdb1

[root@salman11 ~]# ls -l /dev/oracleafd/disks/
total 4
-rwxrwx---. 1 grid oinstall 10 Mar 22 16:30 DATA
[root@salman11 ~]#

As root user, start asfs first, and then “has” for standalone GI installations.  Start “CRS” for RAC installations (On all nodes)
[root@salman11 ~]# /u01/app/12.2.0/grid/bin/acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed


[root@salman11 ~]# /u01/app/12.2.0/grid/bin/crsctl start has

-- For RAC
/u01/app/12.2.0/grid/bin/crsctl start crs

It is better to deinstall oracleasm (Form all nodes in case of RAC)
[root@salman11 ~]# rpm -q oracleasm-support
oracleasm-support-2.1.8-3.1.el7.x86_64
[root@salman11 ~]# rpm -e oracleasm-support-2.1.8-3.1.el7.x86_64
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave

Since ASMFD will be used now, we can set the diskstring to use only ‘AFD:*’, and remove ‘ORCL:*’ from diskstring.
As grid user, execute following command
[grid@salman11 ~]$ asmcmd dsget
parameter:ORCL:*, AFD:*
profile:ORCL:*,AFD:*

[grid@salman11 ~]$ asmcmd dsset 'AFD:*'

[grid@salman11 ~]$ asmcmd dsget
parameter:AFD:*
profile:AFD:*

Thursday, May 25, 2017

Configuring Oracle ASM Filter Driver (ASMFD) and Labeling Disks


If you already have 12.2.0.1 and above GI/RAC installation and you want to migrate to ASMFD, click here.

Starting 12.2.0.1, downloaded Grid Infrastructure (GI) software zip is extracted directly in to the directly we have already chosen as ORACLE_HOME for GI. After extraction, we execute $ORACLE_HOME/gridSetup.sh script to initiate the setup wizard. If we want to use ASMFD (ASM Filter Driver), then after we have provisioned the disk/partition at OS level, we can configure ASMFD as follows. We no longer need ASMLib to setup ASM disks once we start using ASMFD.

For this example, I already have created a disk partition /dev/sdb1 which I will be labelling using ASMFD during the configuration of GI. As root user, configure the ASMFD as follows. /u01/app/12.2.0/grid is where I have extracted the GI software.
[root@salman11 ~]# export ORACLE_HOME=/u01/app/12.2.0/grid
[root@salman11 ~]# export ORACLE_BASE=/tmp

[root@salman11 ~]# $ORACLE_HOME/bin/asmcmd afd_label DATA /dev/sdb1 --init
[root@salman11 ~]# $ORACLE_HOME/bin/asmcmd afd_lslbl /dev/sdb1
--------------------------------------------------------------------------------
Label                     Duplicate  Path
=====================================================================
DATA                                  /dev/sdb1

[root@salman11 ~]# unset ORACLE_BASE


Now during GI configuration setup (initiate $ORACLE_HOME/gridSetup.sh), you will see following screen where this new disk will be listed. Select this disk and click the checkbox “Configure Oracle ASM Filter Driver”. ASM Filter Driver will be configured automatically during the setup and you can start using ASMFD for your future ASM disks configurations. As stated above, now you are no longer required to use ASMLib.

Thursday, May 18, 2017

Disabling Transparent Huge Pages on Linux 6 and Linux 7

Oracle recommends that Transparent Huge Pages should be disabled before installation of Oracle software. In the following I will explain how to disable Transparent Huge Pages on Linux 6 and Linux 7. I am using Oracle Enterprise Linux where I see that this feature is already disabled. You can check for your release of Linux whether it is disabled or not, and disable it using method explained here. This is the official document for doing this task.

Check if Transparent Huge Pages is Enabled
For Red Hat Linux, issue following command
# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

For other Linux distributions, use issue following command
# cat /sys/kernel/mm/transparent_hugepage/enabled

For Red hat, if you don’t see directory “redhat_transparent_hugepage”, and for other distributions, directory “transparent_hugepage” does not exist, it means Transparent Huge Pages is already disabled
If Transparent Huge Pages is enabled, above command should return following output.
[always] never

Disabling Transparent Huge Pages 
Add following entry in /etc/grub.conf file for Linux 6, and /etc/grub2.cfg for Linux 7. This entry needs to be made for every kernel listed in the grub configuration file which you use for booting the system.
transparent_hugepage=never

For example, on my current Oracle Linux 7, I have 3 kernels that I can use to boot my OS, and Transparent huge Pages is already disabled, but you will need to make this entry if entry is not already there. Following are these entries from my server.

menuentry 'Oracle Linux Server, with Linux 3.10.0-123.el7.x86_64' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnul
inux-3.10.0-123.el7.x86_64-advanced-c0c94d8d-4dac-4926-89a1-12f5ae2659f5' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  9ca469b8-017b-498
e-a5af-5b0378ee8b75
        else
          search --no-floppy --fs-uuid --set=root 9ca469b8-017b-498e-a5af-5b0378ee8b75
        fi
        linux16 /vmlinuz-3.10.0-123.el7.x86_64 root=UUID=c0c94d8d-4dac-4926-89a1-12f5ae2659f5 ro vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrheb-sun1
6 rhgb quiet numa=off transparent_hugepage=never
        initrd16 /initramfs-3.10.0-123.el7.x86_64.img
}


menuentry 'Oracle Linux Server, with Unbreakable Enterprise Kernel 3.8.13-35.3.1.el7uek.x86_64' --class oracle --class gnu-linux --class gnu --class os --unrestric
ted $menuentry_id_option 'gnulinux-3.8.13-35.3.1.el7uek.x86_64-advanced-c0c94d8d-4dac-4926-89a1-12f5ae2659f5' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  9ca469b8-017b-498
e-a5af-5b0378ee8b75
        else
          search --no-floppy --fs-uuid --set=root 9ca469b8-017b-498e-a5af-5b0378ee8b75
        fi
        linux16 /vmlinuz-3.8.13-35.3.1.el7uek.x86_64 root=UUID=c0c94d8d-4dac-4926-89a1-12f5ae2659f5 ro vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrhe
b-sun16 rhgb quiet numa=off transparent_hugepage=never
        initrd16 /initramfs-3.8.13-35.3.1.el7uek.x86_64.img
}


menuentry 'Oracle Linux Server, with Linux 0-rescue-f58182a4af154aa38f194b45455927b1' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menue
ntry_id_option 'gnulinux-0-rescue-f58182a4af154aa38f194b45455927b1-advanced-c0c94d8d-4dac-4926-89a1-12f5ae2659f5' {
        load_video
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  9ca469b8-017b-498
e-a5af-5b0378ee8b75
        else
          search --no-floppy --fs-uuid --set=root 9ca469b8-017b-498e-a5af-5b0378ee8b75
        fi
        linux16 /vmlinuz-0-rescue-f58182a4af154aa38f194b45455927b1 root=UUID=c0c94d8d-4dac-4926-89a1-12f5ae2659f5 ro vconsole.keymap=us crashkernel=auto vconsole.f
ont=latarcyrheb-sun16 rhgb quiet numa=off transparent_hugepage=never
        initrd16 /initramfs-0-rescue-f58182a4af154aa38f194b45455927b1.img
}


Reboot the server so that changes become effective.