Sunday, April 26, 2015

Installing Oracle 12cR1 RAC on Linux 7

This article explains how to install a 2 nodes Oracle 12cR1 Real Application Cluster (RAC) on Oracle Linux 7. I did this installation on Oracle Virtual Box by creating 2 virtual machines with shared storage. OS platform I used is Oracle Enterprise Linux 7 (update 1), and Oracle GI and RDBMS version is 12.1.0.2. Same installation guide should work for Redhat Linux 7 also. There will be no difference in installation steps if there are more than 2 nodes in the RAC setup.

Official document for 12c RAC installation can be found by clicking at the following link.
http://docs.oracle.com/database/121/CWLIN/toc.htm
Prerequisites
Prepare all the nodes by installing Oracle Linux 7. Have private interconnects setup, and shared storage mounted on all the nodes. For this example, I have 3 disks (2G size each) for CRS diskgroup to store OCR and Voting disk, and 1 disk (4G) for DATA diskgroup to store the database.
Please note that you should have external redundancy for these disks, otherwise you should specify at least 3 different locations for storing voting disk, and 2 locations to store OCR .
Have 2 public IPs, 2 virtual IPs, 2 private IPs and 1 SCAN IP (from public subnet), which we will use later during the installation. For this article, node names for are as follows 

Public
salman1.salman.com
salman2.salman.com

Virtual
salman1-vip.salman.com
salman2-vip.salman.com

Private
salman1-priv.salman.com
salman2.priv.salman.com

SCAN
salman-scan.salman.com

Recommended way for SCAN configuration is to have 3 IP addresses which are resolved through DNS. Since I don’t have DNS for my environment, I will be using entry in the host file for the SCAN, and will use a single SCAN IP address

Download and unzip the software 
Download 12c Grid Infrastructure and RDBMS softwares.
First release (12.1.0.1) can be downloaded from www.oracle.com, but since we are installing 12.1.0.2 here, we would need to download patch set 1769377. First 2 zip files are for database and last 2 are for Grid Infrastructure binaries.












For this article, I extracted the software under /u02/12.1.0.2/ directory.
Now we start the installation. Following are the steps to install Oracle 12c RAC on Linux.
1)
On each node
, edit /etc/selinux/config and set value for SELINUX to either “permissive” or “disabled”

SELINUX=permissive

2)
On each node
, configure Shared Memory File System. Add following line in /etc/fstab file for shared memory file system. Modify the value of “size” based on the amount of memory you will be using for your SGA.
tmpfs                                   /dev/shm                tmpfs   rw,exec,size=8g        0 0

3)
On each node, disable the firewall. 
[root@salman1 ~]# systemctl start firewalld.service
[root@salman1 ~]# systemctl stop firewalld.service
[root@salman1 ~]# systemctl disable firewalld.service
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service' 
4)
Reboot all nodes

5)
On each node
, in /etc/host file, add the IP addresses and fully qualified names of each node of RAC, including public IPs; virtual IP; private IPs and SCAN IP.
#Public
192.231.231.40 salman1.salman.com       salman1
192.231.231.41 salman2.salman.com       salman2

#Virtual
192.231.231.50 salman1-vip.salman.com   salman1-vip
192.231.231.51 salman2-vip.salman.com   salman2-vip

#Private
10.10.10.10 salman1-priv.salman.com     salman1-priv
10.10.10.11 salman2-priv.salman.com     salman2-priv

#SCAN
192.231.231.60 salman-scan.salman.com   salman-scan
Make sure that Public interface and Private interfaces have the same name and they list in same order on all the nodes, in ifconfig command (my public interface name is enp0s3 and private interface name is enp0s8)

6)
Test the nodes connectivity with each other using ping command from each node to the other node(s). SCAN IP and Virtual IPs are not required to be tested at this point.

From node salman1
[root@salman1 ~]#  ping salman1
[root@salman1 ~]#  ping salman1-priv
[root@salman1 ~]#  ping salman2
[root@salman1 ~]#  ping salman2-priv

From node salman2
[root@salman2 ~]#  ping salman2
[root@salman2 ~]#  ping salman2-priv
[root@salman2 ~]#  ping salman1
[root@salman2 ~]#  ping salman1-priv


7)
We can perform automatic configuration of the node using “yum” command. If you want to do manual configuration, skip this step and go to next step.
Automatic configuration would perform following tasks
- Installation of required RPM packages
- Setup kernel parameters in /etc/sysctl.conf file
- Creation of OS groups (oinstall, dba) and OS user (oracle)
- Setting limits for installation user “oracle” 
For Oracle Linux, follow the steps mentioned in the following documents to access the online yum repository.
http://public-yum.oracle.com/

On each node, execute following command to perform all prerequisites automatically.

[root@salman1 ~]# yum install oracle-rdbms-server-12cR1-preinstall –y


As already mentioned, above command will install all required packages which are needed for grid infrastructure and/or RDBMS software installation. If you also plan to install a 32-bit client on this x86_64 server, you would also need to install 32-bit version of these RPM packages manually as follows.

[root@salman1 ~]# yum install glibc-2.17-36.el7.i686
[root@salman1 ~]# yum install glibc-devel-2.17-36.el7.i686
[root@salman1 ~]# yum install libaio-0.3.109-9.el7.i686
[root@salman1 ~]# yum install libaio-devel-0.3.109-9.el7.i686
[root@salman1 ~]# yum install libXi-1.7.2-1.el7.i686
[root@salman1 ~]# yum install libXtst-1.2.2-1.el7.i686
[root@salman1 ~]# yum install libgcc-4.8.2-3.el7.i686
[root@salman1 ~]# yum install libstdc++-4.8.2-3.el7.i686
[root@salman1 ~]# yum install libstdc++-devel-4.8.2-3.el7.i686


If you want to install form Oracle Linux installation media, use “rpm -i” command to install the packages.
Automatic configuration also sets the required kernel parameters in /etc/sysctl.conf file.


Automatic configuration would create default OS groups i.e. oinstall and dba (with group ID 54321 and 54322 respectively), and OS user (oracle) with user ID 54321. If you plan to implement role separation by creating a “grid” user to install and manage grid infrastructure, and use “oracle” user for database software, you would need to create more OS groups and “grid” user manually, as follows (on both nodes).

Add groups
[root@salman1 ~]#  groupadd -g 54323 oper
[root@salman1 ~]#  groupadd -g 54325 asmdba
[root@salman1 ~]#  groupadd -g 54328 asmadmin
[root@salman1 ~]#  groupadd -g 54329 asmoper

Add user
[root@salman1 ~]# useradd -u 54322 -g oinstall -G dba,asmdba,asmadmin,asmoper grid

Set passwords for both users (oracle and grid)
[root@salman1 ~]# passwd oracle
[root@salman1 ~]# passwd grid

Make user oracle member of oper and asmdba groups
[root@salman1 ~]#  usermod  -g oinstall -G dba,oper,asmdba oracle


Automatic configuration also sets the resource limits for the default user “oracle”, by creating oracle-rdbms-server-12cR1-preinstall.conf file under /etc/security/limits.d directory. If you have created user “grid” for grid infrastructure installation, you would also need to set these limits manually for grid user. You can create a separate .conf file under /etc/security/limits.d directory (.conf file name can be anything), or append following lines in oracle-rdbms-server-12cR1-preinstall.conf file
grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 16384
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid hard memlock 134217728
grid soft memlock 134217728

8)
Skip this step if you have followed Step 7 above; otherwise perform following tasks on each node.
Install following RPM packages (or latest version) from either yum repository or from Linux 7 media

binutils-2.23.52.0.1-12.el7.x86_64
compat-libcap1-1.10-3.el7.x86_64
gcc-4.8.2-3.el7.x86_64
gcc-c++-4.8.2-3.el7.x86_64
glibc-2.17-36.el7.i686
glibc-2.17-36.el7.x86_64
glibc-devel-2.17-36.el7.i686
glibc-devel-2.17-36.el7.x86_64
libaio-0.3.109-9.el7.i686
libaio-0.3.109-9.el7.x86_64
libaio-devel-0.3.109-9.el7.i686
libaio-devel-0.3.109-9.el7.x86_64
ksh
make-3.82-19.el7.x86_64
libXi-1.7.2-1.el7.i686
libXi-1.7.2-1.el7.x86_64
libXtst-1.2.2-1.el7.i686
libXtst-1.2.2-1.el7.x86_64
libgcc-4.8.2-3.el7.i686
libgcc-4.8.2-3.el7.x86_64
libstdc++-4.8.2-3.el7.i686
libstdc++-4.8.2-3.el7.x86_64
libstdc++-devel-4.8.2-3.el7.i686
libstdc++-devel-4.8.2-3.el7.x86_64
sysstat-10.1.5-1.el7.x86_64
unixODBC-2.3.1

Example (yum)
[root@salman1 ~]# yum install glibc

Example (Linux Media)
[root@salman1 ~]# rpm -i glibc

Example (Check after install)
[root@salman1 ~]# rpm -q glibc
glibc-2.17-55.el7.x86_64


On each node, edit /etc/sysctl.conf add following entries to set kernel parameters

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500 

Execute following command after adding above lines
/sbin/sysctl -p

On each nodes, create a .conf file (file name can be anything) under /etc/security/limits.d directory to set shell limits for grid and oracle users. For example, create file oracleusers.conf with following entries. Alternatively, you can also set the limits in /detc/security/limits.conf. But I would prefer to set under /etc/security/limits.d directory.

# Grid user
grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 16384
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid hard memlock 134217728
grid soft memlock 134217728

# Oracle user
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 134217728
oracle soft memlock 134217728

9)
On each node,
install oracleasm-support and oracleasmlib, and then configure oracleasm.
oracleasm kernel driver is built in Oracle Linux and does not need to be installed. After installing oracleasm-support and oracleasmlib packages, oracleasm driver starts working.
If you are using some other flavour of Linux, for example RedHat Linux, then you would need to install all 3 packages (oracleasm driver, oracleasm-support and oracleasmlib).
Install oracleasm-support from yum repository or from Linux media, then download oracleasmlib package form the following URL, and install.
http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol7-2352094.html

Install oracleasmlib and oracleasm-support
[root@salman1 ~]#  yum install oracleasm-support
[root@salman1 ~]#  rpm -i oracleasm-lib*

Configure oracleasm (highlighted in red are the inputs during configuration)
[root@salman1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Scan for Oracle ASM disks on boot (y/n) [y]:y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Check Configuration
[root@salman1 ~]# /usr/sbin/oracleasm configure
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

10)
On each node
, add following in /etc/pam.d/login file if not present already

session required /lib64/security/pam_limits.so
session required pam_limits.so

11)
On each node,
stop and disable NTP (chronyd) 
During the installation, Cluster Time Synchronization Service would be installed and used instead or NTP.

[root@salman1 ~]# systemctl stop chronyd
[root@salman1 ~]# systemctl disable chronyd 
rm '/etc/systemd/system/multi-user.target.wants/chronyd.service'

12)
On each node,
make sure /etc/resolv.conf file contain entries similar to the following. Replace salman.com with your domain name and 192.231.231.1 with your names server IP address

search salman.com
nameserver 192.231.231.1

13)
On each nodee
, install cvuqdisk RPM
This is required for cluvfy (Cluster Verification Utility) to work properly to discover shared disks, otherwise an error will be reported while running cluvfy. Log in as root and execute following steps

Set the environment variable CVUQDISK_GRP to point to the group what will own cvuqdisk, which is oinstall group
[root@salman1 ~]# export CVUQDISK_GRP=oinstall

Install cvuqdisk which will be found under unzipd Grid Infrastructure software
[root@salman1 ~]# cd /u02/12.1.0.2/grid/rpm
[root@salman1 ~]#  rpm –i cvuqdisk-1.0.9-1.rpm

14)
On each node
, create directories and change ownership to respective users (grid and oracle).

[root@salman1 ~]# mkdir -p /u01/app/12.1.0/grid
[root@salman1 ~]# mkdir -p /u01/app/grid
[root@salman1 ~]# mkdir -p /u01/app/oracle

[root@salman1 ~]# chown -R grid:oinstall /u01
[root@salman1 ~]# chown oracle:oinstall /u01/app/oracle
[root@salman1 ~]# chmod -R 775 /u01/

15)
From any node,
partition the shared disks. As stated in the beginning, I have total 4 disks - 3 for OCR and Voting disk, and 1 for DATA diskgroup. Highlighted disks are the disks I will be using here to create partitions and then creating ASM disks using oracleasm.

[root@salman1 ~]# ls -l /dev/sd*
brw-rw----. 1 root disk 8,  0 Apr  7 11:38 /dev/sda
brw-rw----. 1 root disk 8,  1 Apr  7 11:38 /dev/sda1
brw-rw----. 1 root disk 8,  2 Apr  7 11:38 /dev/sda2
brw-rw----. 1 root disk 8,  3 Apr  7 11:38 /dev/sda3
brw-rw----. 1 root disk 8,  4 Apr  7 11:38 /dev/sda4
brw-rw----. 1 root disk 8,  5 Apr  7 11:38 /dev/sda5
brw-rw----. 1 root disk 8, 16 Apr  7 11:38 /dev/sdb
brw-rw----. 1 root disk 8, 17 Apr  7 11:38 /dev/sdb1
brw-rw----. 1 root disk 8, 32 Apr  7 11:38 /dev/sdc
brw-rw----. 1 root disk 8, 48 Apr  7 11:38 /dev/sdd
brw-rw----. 1 root disk 8, 64 Apr  7 11:38 /dev/sde
brw-rw----. 1 root disk 8, 80 Apr  7 11:38 /dev/sdf


Use following steps to partition each disk (/dev/sdc, /dev/sdd, /dev/sde, /dev/sdf).

[root@salman1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x457c3565.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-25165823, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-25165823, default 25165823):
Using default value 25165823
Partition 1 of type Linux and of size 12 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

For simplicity, after executing command fdisk for each disk, press following sequence keys
n, p, 1, <enter>, <enter>, w to create new partitions for each disk
After the partitioning completes, following are my partitioned disks


[root@salman1 ~]# ls -l /dev/sd*
brw-rw----. 1 root disk 8, 32 Apr  7 12:04 /dev/sdc
brw-rw----. 1 root disk 8, 33 Apr  7 12:04 /dev/sdc1
brw-rw----. 1 root disk 8, 48 Apr  7 12:08 /dev/sdd
brw-rw----. 1 root disk 8, 49 Apr  7 12:08 /dev/sdd1
brw-rw----. 1 root disk 8, 64 Apr  7 12:08 /dev/sde
brw-rw----. 1 root disk 8, 65 Apr  7 12:08 /dev/sde1
brw-rw----. 1 root disk 8, 80 Apr  7 12:08 /dev/sdf
brw-rw----. 1 root disk 8, 81 Apr  7 12:08 /dev/sdf1

16)
From any node, use oracleasm command to create ASM disks.

[root@salman1 ~]# oracleasm createdisk crs1 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@salman1 ~]# oracleasm createdisk crs2 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@salman1 ~]# oracleasm createdisk crs3 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@salman1 ~]# oracleasm createdisk data /dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@salman1 ~]#
[root@salman1 sys]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@salman1 sys]# oracleasm listdisks
CRS1
CRS2
CRS3
DATA


Execute “oracleasm scandisks” and “oracleasm listdisks” command on all other nodes also and you should be able to see all the ASM disks.
From the other node(s), issue following as root

[root@salman2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@salman2 ~]# oracleasm listdisks
CRS1
CRS2
CRS3
DATA

17)

On each node, add IO scheduler for ASM disks. Modify highlighted disks names to match your environment.

[root@salman1 ~]# echo deadline > /sys/block/sdc/queue/scheduler
[root@salman1 ~]# echo deadline > /sys/block/sdd/queue/scheduler
[root@salman1 ~]# echo deadline > /sys/block/sde/queue/scheduler
[root@salman1 ~]# echo deadline > /sys/block/sdf/queue/scheduler

18)
On each node
, add following in /home/<username>/.bash_profile file of both users; “grid” and “oracle”. At this point, we do no need to set other environment variables i.e. ORACLE_BASE or ORACLE_HOME.

umask 022

19)
Run cluvfy (Optional)
In the node where you plan to initiate the installation, open a terminal and log in as “grid” user, and you can optionally execute cluster verification utility (cluvfy). Click here to see the output of cluvfy

[grid@salman1 ~]$ cd /u02/12.1.0.2/grid/
[grid@salman1 ~]$ ./runcluvfy.sh stage -pre crsinst -n salman1 -fixup -verbose

This will fail if user equivalence (passwordless ssh login) has not been configured from each node to other node. Passwordless login using ssh is also optional at this point, because installer will allow us to configure user equivalence during the installation wizard as you will see in a while. To learn how to configure SSH connectivity prior to the installation, click here

20)
Choose any node form where you want to initiate the installation and log in as installation user (“grid” in this example). You would choose desktop logging in to the server. If you did not install any desktop environment, use X Window System.


















21)
Start installation from the extracted software directory (/u02/12.1.0.2/grid), execute “./runInstaller” to begin the installation


22) 
Click Next









































23)
Click Next





























24) 
Select “Advance Installation” and click Next





























25)
Click Next




























26)
Change Cluster Name and SCAN name and SCAN Port, if needed. Make sure that SCAN Name is same as specified in /etc/host file. Uncheck “Configure GNS”. Click Next




























27)
You will see only first node (where you initiated installation) here. Click Add to open a window to provide details of the second node. You would need to add details of every node in your RAC environment by clicking Add button.
Click SSH Connectivity. If you have already done SSH configuration for passwordless login to the remote node(s), you should skip next setup.




























28)
After clicking SSH Connectivity, you will see text boxes to provide username (grid) and password to setup SSH connectivity. Provide password and click on Setup button. It will take a few seconds to configure the SSH.




























29)
Once SSH configuration completes, click OK, then click Next




























30)
Select correct network interfaces for Public and Private interconnects and click Next




























31)
Click Next




























32)
Provide diskgroup name. I am providing CRS as I would store OCR and Voting disk on this diskgroup. I am also selecting “External” redundancy. If your disks are not already mirrored, you should use either Normal (minimum 2 disks required) or High redundancy (minimum 3 disks required) to safeguard against disks failure. Select ASM disks to be part of this diskgroup.




























33)
Provide password for user SYS (SYSASM) and click Next




























34)
Click Next




























35)
Click Next





























36)
Click Next




























37)
If required, modify the paths based on the directories created in a previous step. Or you can go ahead with the default locations. Click Next




























38)
Click Next




























39)
Select this option if you want the required scripts to be executed automatically by OUI rather than prompting you to execute the scripts as “root”, during the installation. 




























40)
If you see fail/warning for any prerequisite check, fix it before clicking Next. In my case, I am ignoring all of them because of the following reasons.
i) I am using virtual Box with 2G RAM, so I ignored Physical Memory Warning
ii) I am not using DNS, so I ignored resolv.conf failure
iii) I am not using DNS, so I ignored DNS/NIS failure
iv) I am also ignoring /dev/shm as my /dev/shm is already set to 8G which is not recognized by the cluvfy because of a bug in cluvfy shipped with 12.1.0.2 Grid Infrastructure software.
Click here for more details on cluvfy
Click Next




























41)
Click Install button to start installation




























42)
Installer will prompt for the permission to execute the scripts as root. Click Yes




























43)
Installation completed. I see one error here and that is because of cluvfy execution failure during post installation steps. Reason of cluvfy failure is same as I explained above in step 40 above, so I just clicked OK to close the box.

























44)
Click Yes



























45)
Click Close, installation has successfully completed.




























46)
On both nodes, add following environment variables in .bash_profile file. On node2, ORACLE_SID value should be +ASM2, and so on.

##grid user environment variables
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/12.1.0/grid; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib; export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp

47)
Check health of CRS. All seems OK and working properly

[grid@salman2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.asm
               ONLINE  ONLINE       salman1                  Started,STABLE
               ONLINE  ONLINE       salman2                  Started,STABLE
ora.net1.network
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.ons
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       salman1                  STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       salman1                  169.254.93.149 
                                             10.1.10.10,STABLE
ora.cvu
      1        ONLINE  ONLINE       salman1                  STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       salman1                  Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       salman1                  STABLE
ora.salman1.vip
      1        ONLINE  ONLINE       salman1                  STABLE
ora.salman2.vip
      1        ONLINE  ONLINE       salman2                  STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       salman1                  STABLE
--------------------------------------------------------------------------------

[grid@salman2 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node salman1

[grid@salman2 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node salman1

48)
Installing Database software
Before we start database software installation, first we create a diskgroup DATA which will be used to store the database; we will be creating during the database software installation.
Open a terminal and log in as OS user “grid”. Now use SQLPLUS to log into the ASM instance with SYSASM privileges, and then create the diskgroup DATA


[grid@salman1 ~]$ sqlplus

SQL*Plus: Release 12.1.0.2.0 Production on Mon Apr 20 12:11:05 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter user-name: sys as sysasm
Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options


SQL> select name, path from v$asm_disk;

NAME                           PATH
------------------------------ ----------------------------------------------------------------------------------
                                        ORCL:DATA
CRS1                               ORCL:CRS1
CRS2                               ORCL:CRS2
CRS3                             ORCL:CRS3

Compatibility for ASM and RDBMS should be specified, otherwise it would default to 10.1.0.0.0 and we will not be able to store 12c database files on it.

SQL>  create diskgroup DATA external redundancy disk  'ORCL:DATA' ATTRIBUTE 
'compatible.asm'='12.1.0.0.0' , 'compatible.rdbms'='12.1.0.0.0';

Diskgroup created.

If you created this diskgroup from node1, use srvctl command to start this new disk resource on all other nodes
[grid@salman1 ~]$ srvctl start diskgroup -diskgroup DATA -node salman2

Since I am using 2 different OS users here, from any node, log in as user “oracle” by using either XWindow or Linux desktop, and initiate installation from the directory where database software has already been extracted.

[oracle@salman1 database]$cd /u02/12.1.02/database

[oracle@salman1 database]$./runInstaller

Click Next




























49) I want a new database to be created during the installation. Click Next. 






























50)

I selected Server class database. Click Next











































51)
We will be doing RAC database installation so that software is installed on all the RAC nodes. Click Next





























52)
Click Next





























53)
By default all nodes are selected for database software installation, click on “SSH Connectivity” button as previously done during Grid infrastructure installation and configure the SSH connectivity before moving forward. This time SSH will be configured for “oracle” user. Once configured, click Next






























54)
Click Next





























55)
Click Next





























56)
Click Next





























57)
Modify paths if required, click Next





























58)
Select database type which you want to create, click Next





























59)
Provide database name and also check the checkbox if you want it to be a container database





























60)
Specify memory allocation details for the database instances, click Next






























61)
Select “Oracle Automatic Storage Management” as we already have an ASM disk created to store the database.




























62)
Click Next




























63)
Click Next




























64)
Select diskgroup which we created to store the database files, click Next




























65)
Specify the passwords for administrator users, click Next




























66)
Click Next




























67)
On last screen, I see 4 warnings which I can ignore because of following reasons.
i) Since I am not using namesserver, I faced warning number 1 and 2, so I can ignore both
ii) Just like during Grid Infrastructure installation, OUI had an issue in recognizing actual value of my /dev/shm files system, same happened here, so I can ignore it.
iii) Limits for oracle user are already set in /etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf which can be confirmed using “ulimit –a” command, but OUI is checking /etc/security/limits.conf file for limits, so I can ignore this warning also. Click Next




























68)
Click Install




























69)
During the installation, you will be prompted to execute the root.sh on both nodes as root. When prompted, executed on both nodes one by one, click OK to close the box so that installation could proceed further.
























70)
Database Creation Assistant would initiate to create the database



























71)
Click OK to close the prompt after database creation completes.





























72)

Click Close.


























73)
Add following environment variables setting in .bash_profile of “oracle” user, on both nodes. Modify value of ORACLE_SID for each node accordingly

ORACLE_SID=salmandb1; export ORACLE_SID
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/local/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib; export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp

74)
RAC Installation completes successfully. Check the status of all resources and database instances
[grid@salman1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.DATA.dg
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.asm
               ONLINE  ONLINE       salman1                  Started,STABLE
               ONLINE  ONLINE       salman2                  Started,STABLE
ora.net1.network
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
ora.ons
               ONLINE  ONLINE       salman1                  STABLE
               ONLINE  ONLINE       salman2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       salman1                  STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       salman1                  169.254.93.149,STABL
                                                             E
ora.cvu
      1        ONLINE  ONLINE       salman1                  STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       salman1                  Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       salman1                  STABLE
ora.salman1.vip
      1        ONLINE  ONLINE       salman1                  STABLE
ora.salman2.vip
      1        ONLINE  ONLINE       salman2                  STABLE
ora.asm
               ONLINE  ONLINE       salman1                  Started,STABLE
               ONLINE  ONLINE       salman2                  Started,STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       salman1                  STABLE
--------------------------------------------------------------------------------


[grid@salman1 ~]$ srvctl status database -d salmndb
Instance salmndb1 is running on node salman1
Instance salmndb2 is running on node salman2


Related Article



19 comments:

Anibal said...

Hello Salman,

Can you explain to the setup of point 5?
As on both VM(s) ORA1 and ORA2 you activated only two network adapters.
One NAT and one Host Only type.

Do you have the setup in writing for both the VM(s) ORA1 and ORA2?


Thx,

Anibal

Salman Ahmed Qureshi said...

Hi Anibal, Sorry for replying you late.
Did you see my post on how to create Virtual machines, if not, please see bellow.

http://salmandba.blogspot.sg/2015/05/creating-virtual-machine-using-oracle.html

For RAC, you can add 2 host only adapters, one for public and virtual and other for private interconnect. Above document also explains how to add the adapters to your VMs
Salma

Baljinder Singh said...

hi Salman....i following your thread to configure RAC 12c in VM. i facing ASM related problem.
below is the error.

ASM failed to start. Check /u02/app/grid/cfgtoollogs/asmca/asmca-150912AM015433.log for details.

2015/09/12 01:56:54 CLSRSC-184: Configuration of ASM failed
2015/09/12 01:56:54 CLSRSC-258: Failed to configure and start ASM
Died at /u02/app/12.1.0/grid_1/crs/install/crsinstall.pm line 2017.
The command '/u02/app/12.1.0/grid_1/perl/bin/perl -I/u02/app/12.1.0/grid_1/perl/lib -I/u02/app/12.1.0/grid_1/
crs/install /u02/app/12.1.0/grid_1/crs/install/rootcrs.pl -auto -lang=en_US.UTF-8' execution failed


can u pls help me to resolve this issue.
Thank you in advance

Salman Ahmed Qureshi said...

Hi Baljinder,
Did you run cluvfy before installation and was it successful?
This message is just showing that asm configuration failed, not the detail of it. Can you check installation logs for more details of error?

Baljinder Singh said...

Hi Salman,
Thank you so much for your promptly response. i really appreciate you. bro i already run cluvfy there was no issue like that.
laters i deconfig the failed Root.sh configuration and created ASM disks again and now it's working fine.
Thanks again.

Salman Ahmed Qureshi said...

Good to know that it is working for you now. Best of luck.

Hasan Mohammad Mahamudul said...

Very Clear , detailed and really helpful. Inspired from you . Thanks for your knowledge sharing

Domenico Squadrito said...

Hello Salam,
The steps that you describes are very clear and detailed. Good work.
Thank you a lot.
Domenico

SouSurDBA said...

Hi Salman,

./runInstaller is not working for me on Linux 7. How can I solve this problem. The problem is with the display.

Please help.

Thanks,
Surya

Salman Ahmed Qureshi said...

Hi,
I usually install Linux desktop. But if you dont have, then install any X server ( I use xming). See bellow link for how to configure xming
https://www.uaf.edu/arsc/knowledge-base/using-xming-x-server-for-/index.xml

SouSurDBA said...

Hi Salman,

Yes, I have the desktop Linux. I feel like I missed something while doing the install.

I used to run this command as root in Oracle Linux 6 and ./runInstaller always worked after that, but in Linux 7 its not.

#xhost + SI:localuser:oracle

Salman Ahmed Qureshi said...

I am talking about the desktop environment of Linux. If you have desktop environment installed, you need to log into the desktop with "oracle" (just like windows) user and initiate the installer. I think you would need to fix your Linux side. Installation log file may also show you any related error message.

SouSurDBA said...

Hi Salman,

Thank you. Will try that.

Goku said...

Hi Salman,

Appreciate your efforts for the Blog,

I have just one concern why have u selected OCR with "EXTERNAL" (3 Disk) redundancy should it not be "NORMAL" ??

Salman Ahmed Qureshi said...

Hi Goku,
If we already have mirrored disk, we can select External Redundancy. I would edit this document and add further detail on why I am using External Redundancy here. Thanks for pointing out.

Salman

Goku said...

Hi Salman,

Thanks for your response, I also request you to just put a note if you have come across any bugs in installation

Salman Ahmed Qureshi said...

Hi,
I did not come across any bugs during this installation.

Salman

teja said...

Hi salman,

Were are we mentioning port in the process. I followed your steps and every thing went good and default port 1521 was setup. But suppose if i want to use a different port, where i need to do that?

Thanks

Salman Ahmed Qureshi said...

Hi Teja,
Please use this document for changing ports of listeners/scan listeners
http://salmandba.blogspot.sg/2016/03/changing-listener-and-scan-listener.html

Salman