OS :Oracle Linux 5.11 （内存分配3G）
RDBMS: 11.2.0.3 (安装11.2.0.4会出现错误，找不到cell)
1 和安装11gR2一样，安装软件包
vi /tmp/req-rpm.txt
elfutils-libelf
elfutils-libelf-devel
cloog-ppl
compat-libcap1
compat-libstdc++-33
cpp
gcc
gcc-c++
glibc-devel
glibc-headers
kernel-headers
ksh
libXmu
libXt
libXv
libXxf86dga
libXxf86misc
libXxf86vm
libaio-devel
libdmx
libstdc++-devel
mpfr
make
ppl
xorg-x11-utils
xorg-x11-xauth
yum install `awk '{print $1}' /tmp/req-rpm.txt`2 网络配置
192.168.2.102 eth0,业务网络
10.10.10.2 eth1 ，infiniband心跳网络
3 配置/etc/hosts
[root@exadb01 ~]# ps -ef | grep pmon
root 4966 3248 0 05:05 pts/2 00:00:00 grep pmon
[root@exadb01 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.2.101 exacell01.demo.com exacell01
10.10.10.1 exacell01-priv.demo.com excell01-priv
192.168.2.102 exadb01.demo.com exadb01
10.10.10.2 exadb01-priv.demo.com exadb01-priv
[root@exadb01 ~]#
4 创建用户和组
groupadd -g 400 oinstall
groupadd -g 401 dba
groupadd -g 402 oper
groupadd -g 403 asmadmin
groupadd -g 404 asmdba
groupadd -g 405 asmoper
useradd -u 400 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
useradd -u 401 -g oinstall -G dba,asmdba,oper oracle
5 配置系统内核
vi /etc/sysctl.conf
fs.aio-max-nr=1048576
fs.file-max=6815744
kernel.shmall=2097152
kernel.shmax=2109009920
kernel.shmni=4096
kernel.sem=250 32000 100 128
net.ip4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=8388608
net.core.wmem_max=4194304sysctl -p
vi /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 2014
oracle hard nofile 65536
vi /etc/pam.d/login
session required pam_limits.so 6 配置rds协议
modprobe rds
modprobe rds_tcp
modprobe rds_rdma
vi /etc/modprobe.d/rds.conf
install rds /sbin/modprobe --ignore -install rds && /sbin/modprobe rds_tcp && /sbin/modprobe rds_rdma7 关闭防火墙和selinux --装机的时候已经关闭
chkconfig iptables off
service iptables stop
chkconfig ip6tables off
service ip6tables stop
vi /etc/selinux/config
SELINUX=disabled 8 创建安装目录
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown grid.oinstall /u01/app/11.2.0/grid
chown grid.oinstall /u01/app/grid
chown oracle.oinstall /u01/app/oracle
chmod -R 775 /u01
chown -R grid.oinstall /u019 准备exadata配置文件
计算节点能否识别存储节点，是通过计算节点上的配置文件cellip.ora来实现的。需要在计算节点上配置cellip.ora 和cellinit.ora
将文件存放在计算节点上的/etc/oracle/cell/network-config目录下
mkdir -p /etc/oracle/cell/network-config
chown -R grid:oinstall /etc/oracle/cell/network-config
chmod -R 775 /etc/oracle/cell/network-config
su - grid
vi cellinit.ora
ipaddress1=10.10.10.2/24 -- 该IP为计算节点的infiniband心跳网络IP
vi cellip.ora
cell="10.10.10.1" -- 该IP为存储节点的infiniband心跳网络IP 10 验证存储节点griddisk
通过kfod工具检测是否可以访问存储节点的griddisk 。
su - grid
export LD_LIBRARY_PATH=/home/grid/grid/stage/ext/lib:/lib:/usr/lib
/home/grid/grid/stage/ext/bin/kfod disks=all dscvgroup=true -- 这个目录是GI解压后的目录
[root@exadb01 ~]# su - grid
[grid@exadb01 ~]$ export LD_LIBRARY_PATH=/home/grid/grid/stage/ext/lib:/lib:/usr/lib
[grid@exadb01 ~]$ /home/grid/grid/stage/ext/bin/kfod disks=all dscvgroup=true
--------------------------------------------------------------------------------
Disk Size Path Disk Group User Group
================================================================================
1: 976 Mb o/10.10.10.1/DATA_CD_disk01_cell1 # <unknown> <unknown>
2: 976 Mb o/10.10.10.1/DATA_CD_disk02_cell1 # <unknown> <unknown>
3: 976 Mb o/10.10.10.1/DATA_CD_disk03_cell1 # <unknown> <unknown>
4: 976 Mb o/10.10.10.1/DATA_CD_disk04_cell1 # <unknown> <unknown>
5: 976 Mb o/10.10.10.1/DATA_CD_disk05_cell1 # <unknown> <unknown>
6: 976 Mb o/10.10.10.1/DATA_CD_disk06_cell1 # <unknown> <unknown>
7: 976 Mb o/10.10.10.1/DATA_CD_disk07_cell1 # <unknown> <unknown>
8: 976 Mb o/10.10.10.1/DATA_CD_disk08_cell1 # <unknown> <unknown>
9: 976 Mb o/10.10.10.1/DATA_CD_disk09_cell1 # <unknown> <unknown>
10: 976 Mb o/10.10.10.1/DATA_CD_disk10_cell1 # <unknown> <unknown>
11: 976 Mb o/10.10.10.1/DATA_CD_disk11_cell1 # <unknown> <unknown>
12: 976 Mb o/10.10.10.1/DATA_CD_disk12_cell1 # <unknown> <unknown>
KFOD-00301: file not found; arguments: [2] [kgxgncin]
[grid@exadb01 ~]$ 计算节点安装完毕后，执行kfod命令，结果如下（后面的2000MB的磁盘，是安装完毕后又创建的）：
[grid@exadb01 ~]$ export LD_LIBRARY_PATH=/home/grid/grid/stage/ext/lib:/lib:/usr/lib
[grid@exadb01 ~]$ /home/grid/grid/stage/ext/bin/kfod disks=all dscvgroup=true
--------------------------------------------------------------------------------
Disk Size Path Disk Group User Group
================================================================================
1: 976 Mb o/10.10.10.1/DATA_CD_disk01_cell1 DATA <unknown> <unknown>
2: 976 Mb o/10.10.10.1/DATA_CD_disk02_cell1 DATA <unknown> <unknown>
3: 976 Mb o/10.10.10.1/DATA_CD_disk03_cell1 DATA <unknown> <unknown>
4: 976 Mb o/10.10.10.1/DATA_CD_disk04_cell1 DATA <unknown> <unknown>
5: 976 Mb o/10.10.10.1/DATA_CD_disk05_cell1 DATA <unknown> <unknown>
6: 976 Mb o/10.10.10.1/DATA_CD_disk06_cell1 # <unknown> <unknown>
7: 976 Mb o/10.10.10.1/DATA_CD_disk07_cell1 # <unknown> <unknown>
8: 976 Mb o/10.10.10.1/DATA_CD_disk08_cell1 # <unknown> <unknown>
9: 976 Mb o/10.10.10.1/DATA_CD_disk09_cell1 # <unknown> <unknown>
10: 976 Mb o/10.10.10.1/DATA_CD_disk10_cell1 # <unknown> <unknown>
11: 976 Mb o/10.10.10.1/DATA_CD_disk11_cell1 # <unknown> <unknown>
12: 976 Mb o/10.10.10.1/DATA_CD_disk12_cell1 # <unknown> <unknown>
13: 2000 Mb o/10.10.10.1/gd13 DATA_ADD <unknown> <unknown>
14: 2000 Mb o/10.10.10.1/gd14 DATA_ADD <unknown> <unknown>
15: 2000 Mb o/10.10.10.1/gd15 DATA_ADD <unknown> <unknown>
16: 2000 Mb o/10.10.10.1/gd16 DATA_ADD <unknown> <unknown>
17: 2000 Mb o/10.10.10.1/gd17 DATA_ADD <unknown> <unknown>
18: 2000 Mb o/10.10.10.1/gd18 DATA_ADD <unknown> <unknown>
19: 2000 Mb o/10.10.10.1/gd19 DATA_ADD <unknown> <unknown>
20: 2000 Mb o/10.10.10.1/gd20 DATA_ADD <unknown> <unknown>
KFOD-00301: Unable to contact Cluster Synchronization Services (CSS). Return code 2 from kgxgncin.
[grid@exadb01 ~]$ 11 安装GI。因为这里模拟的是ASM单节点的。所有在安装GI的时候，Step2 of 9 ,选择“Configure Oracle Grid Infrastructure for a Standalone Server”.在Step 4 of 11的时候，会自动显示出来ASM磁盘的 。ASM磁盘路径为“o/*/*”格式。安装完毕后ASMCA创建磁盘组。
-- root.sh的过程，和普通的ASM安装是一样的。
12 安装DB和DBCA建库(略)。
到此安装结束，其实和正常安装rac是一样的，只不过是，ASM的设置，是通过cell节点来实现的。
13 碰到的一些错误
问题 1，GI安装完毕后，没有ASM实例和磁盘组
在安装GI的最后一步的时候，出现了[INS-20802]错误，之前多次安装GI都出现这个，原因是DNS问题（没有DNS，设置了etc/hosts文件），这次也没有多想，就忽略了。其实真实正的错误是没有ASMCA创建ASM磁盘组失败。[INS-20802]Automatic Storage Management Configuration Assistant failed.
然后使用crsctl status resource -t 命令，看不到ASM实例和磁盘组。然后卸载GI，再次安装，发现ASMCA还是无法创建ASM磁盘组。查看GI的安装日志，发现类似的错误如下：
INFO: Read: Configuring ASM failed with the following message:
INFO: Read: Configuring HA resource failed. The following error occured:
INFO: Read: PRCR-1079 : Failed to start resource ora.asm
INFO: Read: CRS-5017: The resource action "ora.asm start" encountered the following error:
INFO: Read: ORA-56865: Invalid IP address in CELLINIT.ORA
INFO: Read: . For details refer to "(:CLSN00107:)" in "/u01/app/grid/product/11.2.0/grid/log/db/agent/ohasd/oraagent_grid/oraagent_grid.log".
INFO: Read:
INFO: Read: CRS-2674: Start of 'ora.asm' on 'db' failed
ERROR: diskgroup DATA1 was not created
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 1
从上面可以看到,提示ceccinit.ora里面的IP不对，核对了下，里面的IP是正确的。后来查看文档，说是需要安装11.2.0.3的GI。目前安装的是GI 11.2.0.4.更换为GI11.2.0.3后就正常了。
END
-- 2019-12-23 add
关于flash cacle log的一些说明，可以参考文档：里面记录的很详细。
https://www.cnblogs.com/missyou-shiyh/p/6517589.html
官方的白皮书的说明
Exadata Smart Flash Cache Features and the
Oracle Exadata Database Machine
http://www.oracle.com/technetwork/database/exadata/exadata-smart-flash-cache-366203.pdf
Exadata Smart Flash Logging: Flash for Database Logging
In an OLTP workload, fast response time for database log writes is crucial. The Database
Administrator (DBA) configures redo log groups and mirrored log files for availability but slow
disk performance can have a negative impact on redo log write wait time and system
performance – log writes wait for the write to the slowest disk to complete. Additionally, disk
drives themselves can experience occasional “hiccups” in performance. These spikes can have a
huge impact on database performance. In addition, flash technology can have similar
performance hiccups due to erase cycles or wear leveling. An approach that deals with these
issues, and others, has been provided in the Exadata Database Machine.
Smart Flash Log differentiates Exadata for OLTP workloads, and is another example of how
Exadata optimizes database performance by engineering improvements across the software and
hardware stack. The Smart Flash Logging requires Exadata Storage Software version 11.2.2.4 or
later, and Oracle Database version 11.2.0.2 with Bundle Patch 11, Oracle Database version
11.2.0.3 with Bundle Patch 1, or a later version.
Smart Flash Logging leverages the flash hardware in the Exadata Database Machine. Smart Flash
Logging is much more than simply placing the redo log in flash; duplexed and mirrored log files
exist for important reasons. Just adding a flash-based log to the redo log group will not solve the
problems mentioned above, namely that the database will end up waiting for log writes to the
slowest device whether that device is a slow disk or slow flash. In addition, customers have to set
up their redo log groups and mirrored log files to provide maximum availability. Exadata Smart
Flash Logging has been designed such that no changes are required to this configuration to reap
the benefits of low latency log writes. In essence this feature is transparent to the database
software configuration and just as importantly database recovery.
Smart Flash Logging works as follows. When receiving a redo log write request, Exadata will do
parallel writes to the on-disk redo logs as well as a small amount of space reserved in the flash
hardware. When either of these writes has successfully completed the database will be
immediately notified of completion. If the disk drives hosting the logs experience slow response
times, then the Exadata Smart Flash Cache will provide a faster log write response time.
Conversely, if the Exadata Smart Flash Cache is temporarily experiencing slow response times
(e.g., due to wear leveling algorithms), then the disk drive will provide a faster response time.
This algorithm will significantly smooth out redo write response times and provide overall better
database performance.
The Exadata Smart Flash Cache is not used as a permanent store for redo data – it is just a
temporary store for the purpose of providing fast redo write response time. The Exadata Smart
Flash Cache is a cache for storing redo data until this data is safely written to disk. The Exadata
Storage Server comes with a substantial amount of flash storage. A small amount is allocated for
database logging and the remainder will be used for caching user data. The best practices and
configuration of redo log sizing, duplexing and mirroring do not change when using Exadata
Smart Flash Logging. Smart Flash Logging handles all crash and recovery scenarios without
requiring any additional or special administrator intervention beyond what would normally be
needed for recovery of the database from redo logs. From an end user perspective, the system
behaves in a completely transparent manner and the user need not be aware that flash is being
used as a temporary store for redo. The only behavioral difference will be consistently low
latencies for redo log writes.
By default, 512 MB of the Exadata flash is allocated to Smart Flash Logging. Relative to the 3.2
TB of flash in each Exadata cell this is an insignificant investment for a huge performance
benefit. This default allocation will be sufficient for most situations. Statistics are maintained to
indicate the number and frequency of redo writes serviced by flash and those that could not be
serviced, due to, for example, insufficient flash space being allocated for Smart Flash Logging.
For a database with a high redo generation rate, or when many databases are consolidated on to
one Exadata Database Machine, the size of the flash allocated to Smart Flash Logging may need
to be enlarged. In addition, for consolidated deployments, the Exadata I/O Resource Manager
(IORM) has been enhanced to enable or disable Smart Flash Logging for the different databases
running on the Database Machine, reserving flash for the most performance critical databases.
END
