ACFS -- Creating New Oracle Acfs filesystem

 
                                                          

Considering we are  moving  ACFS from  ODA X-5 using 12C ,  i have  tried to consolidate some information  On acfs that might help on  daily basis 

Starting with Oracle Grid Infrastructure 12c (12.1), Oracle Cloud File system (ASM Cluster File System in cluster environment ) supports database files (database version 11.2.0.4 and up) in addition to general purpose files so that entire Oracle databases can be stored inside Oracle Cloud FS and can leverage the Advanced Data Services such as snapshots, tagging and auditing although replication and encryption are not supported in conjunction with datafiles.

The ability to perform snapshots of ACFS file system is a very powerful feature. An Oracle ACFS Snapshot is a read-write or read-only, space efficient, point-in-time copy of a file system which is immediately available for use after it is created and is always online while the file system is mounted.

The snapshot copy is initially sparse as it merely references the storage allocation information maintained by the source file system. A snapshot utilizes a Copy-On-Write (COW) technology and maintains point in time view of the file system. Whenever an extent is modified by the user, the current extent is copied to the snapshot before modifying the source file extent. When a recovery is required, only the changed blocks are replaced by their “before images” stored in the snapshot.

For a database having its files stored on Oracle Cloud file system, Oracle ACFS Snapshots may serve as point-in-time backups of the database which can be used for online recovery of database files.

Moreover, with Oracle Database 12c, Cloud FS supports Snaps-of-Snaps feature so that snapshots of existing snapshot of the same ACFS file system may be created. Any combination of read-only and read-write snapshots is supported i.e. a read-write snapshot can be based on an existing read-only snapshot, and a read-only snapshot can be based on an existing read-write snapshot. Each ACFS file system can support a total of 63 snapshots, including Snaps-of-Snaps.

The funny bit is that /u02/app/oracle/oradata/datastore contains snapshots… This is visible in the data file location where you find an .ACFS/snaps component. I believe the naming convention is to use .ACFS (invisible) followed by the snapshot name. You can view the snapshot detail using acfsutil:

From ODA release 12.1.2 onwards 11.2.0.4 and 12.1.x databases will be created on ACFS by default. 

The ACFS/ADVM product consists of several processes in the ASM instance, three separate kernel device drivers and several command and utility programs. The three kernel drivers that make up the ACFS products are:

- Oracle ACFS – ASM Cluster File system

- Oracle ADVM – ASM Cluster Dynamic Volume Manager

- Oracle OKS – Oracle Kernel Services

Diagnostic information related to ACFS/ADVM is located in several areas including:

System configuration (e.g. OS version, platform type, etc.)

ASM alert log and process trace files

CSS trace files

Operating system log/event files

CRS resource status

ACFS/ADVM specific log files

Operating system crash dumps

ACFS Replication trace and log file

è Create ACFS :

1. Create ASM disk group

2. Change the compatible parameter of disk groups to 11.2.x.x

3. Create ASM dynamic volume(s)

4. mkfs & mount the file system

5. Extend ACFS on other nodes in cluster

select name, state,total_mb, free_mb,usable_file_mb  from v$asm_diskgroup

select compatibility, database_compatibility from v$asm_diskgroup where name = 'DATADG';

alter diskgroup datadg set attribute 'compatible.asm' = '11.2.0.2';

alter diskgroup datadg set attribute 'compatible.rdbms' = '11.2.0.2';

select compatibility, database_compatibility from v$asm_diskgroup where name = 'DATADG';

alter diskgroup datadg add volume datavol1 size 500M;

select volume_name, volume_device from  v$asm_volume;

SQL> !ls -al /dev/asm/datavol*

ASMCMD [+] > volinfo -a

# mkfs -t acfs -b 8k /dev/asm/datavol1-316

# mkfs -t acfs -b 4k /dev/asm/datavol2-316

# mkdir -p /data /data2

# mount -t acfs /dev/asm/datavol1-316 /data

# mount -t acfs /dev/asm/datavol2-316 /data2

è Verify that the file systems are mounted

# df -h

# mount  | grep asm

# chown ora11gr2:dba /data /data2

ASMCMD [+] > volinfo -a

è Extend ACFS on other nodes in cluster

SQL> alter diskgroup datadg mount

ASMCMD [+] > volinfo -a

ASMCMD [+] > volenable -G DATADG DATAVOL1

ASMCMD [+] > volenable -G DATADG DATAVOL2

# mount -t acfs /dev/asm/datavol1-316 /data

# mount -t acfs /dev/asm/datavol2-316 /data2

# chown ora11gr2:dba /data /data2

è Test the ACFS

# uname -n

# cd /data

# ls -al / > test.txt

# ls -al

# uname -n

# cd /data

# ls -al

############### to remove ACFS entry from registry ######################

Fire:-

/sbin/acfsutil info fs

**it should return :-

acfsutil info fs: ACFS-03036: no mounted ACFS file systems

Use the below command for removing entries from ACFS registry.

/sbin/acfsutil registry -d /dev/ofsctl

And check with below command, whethere entries removed from ACFS registry.

/sbin/acfsutil registry -l

Let me know the results if you still get the errors in clusterware alert log

Create a Cloud file system

1.     Create mountpoint

[root@host01 ~]# mkdir -p /mnt/acfs

2.     Modify the DATA diskgroup to ensure that all the new ADVM features in release 12.1 are enabled.

ASMCMD>setattr -G DATA compatible.advm 12.1.0.0.0

3.     Create a volume VOL1 in the DATA diskgroup having a size of 5G.

ASMCMD>volcreate -G DATA -s 5g --column 1 VOL1

4.     Examine the new volume and take note of the volume device associated with it.

ASMCMD> volinfo -G DATA VOL1

Diskgroup Name: DATA

        Volume Name: VOL1

        Volume Device: /dev/asm/vol1-190

        State: ENABLED

        Size (MB): 5120

        Resize Unit (MB): 8

        Redundancy: MIRROR

        Stripe Columns: 1

        Stripe Width (K): 8192

        Usage:

        Mountpath:

Before  Proceeding with Mkfs ,    Verify Advm status 

crsctl stat res ora.DATAC1.COMMON.advm -t

5.     Make a cloud file system on the newly-created volume VOL1 using the volume device identified in step 4.

[root@host01 ~]# mkfs -t acfs /dev/asm/vol1-190

mkfs.acfs: version = 12.1.0.2.0

mkfs.acfs: on-disk version = 39.0

mkfs.acfs: volume = /dev/asm/vol1-199

mkfs.acfs: volume size = 5368709120 ( 5.00 GB )

mkfs.acfs: Format complete.

6.     Mount the ACFS on the mount point created earlier.

[root@host01 ~]# mount -t acfs /dev/asm/vol1-190 /mnt/acfs

7.     Create a new cloud file system resource using the volume device identified above and the mount point created earlier.

[root@host01 ~]# srvctl add filesystem -m /mnt/acfs -d /dev/asm/vol1-190  -u oracle  -fstype ACFS  -autostart always 

8.     Start the new cloud file system.

[root@host01 ~]# srvctl start file system -d /dev/asm/vol1-190

9.     Confirm that the new file system is mounted.

[root@host01 ~]# srvctl status filesystem -d /dev/asm/vol1-190

ACFS file system /mnt/acfs is mounted on nodes host01

[root@host01 ~]# mount | grep acfs

/dev/asm/vol1-190 on /mnt/acfs type acfs (rw)

10.  Modify the access privileges for the new cloud file system to enable access by any user.

# chmod 777 /mnt/acfs

11.  Modify the newly created cloud FS to enable full control by members of OS group dba which corresponds to SYSDBA privilege on database instance.

[root@host01 ~]# crsctl modify resource ora.data.vol1.acfs -attr "ACL='owner:root:rwx,pgrp:dba:rwx,other::r--'" -unsupported

Troubleshooting  Acfs Issues : 

Main Components :

oracleacfs (oracleacfs.ko): manages all ACFS filesystem operations.
oracleavdm (oracleavdm.ko): AVDM module enabling direct interface with the filesystem 
oracleoks (oracleoks.ko): provides memory management, lock and cluster synchronization

--> To check Corruption 
/sbin/acfsutil info fs

/sbin/acfsutil info fs -o iscorrupt /acfs/backup 

--> To check modules loaded 
 /sbin/lsmod | grep oracle

-->  To manually load modules 
/bin/acfsload start

-->  Try Start / Stop  crs and reinstall  the ACFS driver
/u01/GRID/11.2.0.4/bin/crsctl stop crs
/u01/GRID/11.2.0.4/bin/acfsroot install
/u01/GRID/11.2.0.4/bin/crsctl start crs

--> Restart acfs filesytem 
/bin/mount -t acfs /dev/asm/vol_cloudfs-390 /cloudfs



Runing Fsck  
Fsck attempts to validate all of a file system’s metadata. Part of this process includes validation of metadata block free lists. The free block itself has to be checked to ensure that is it the correct type of metadata block and the free list has to be checked to ensure that there are no missing entries and that there are no loops in the list. Since files can be deleted in any order, these free blocks can be located anywhere on the volume. Processing and validating these lists can produce lots of random I/O due to the random nature of file deletion. If fsck takes too long it is ok to interrupt it given it’s running in check mode.

fsck checks and repairs an existing Oracle ACFS. This command can only be run on a dismounted file system. root privileges are required to run fsck. The Oracle ACFS driver must be loaded for fsck to work

IMPORTANT: By default, fsck only checks for and reports any errors. In check mode fsck can be cancelled if it is taking a long time. The -a flag must be specified to instruct fsck to repair errors in the file system. If check mode completed in a reasonable amount of time, and if it reported problems, run fsck in repair mode. In repair mode fsck cannot be interrupted without risk of leaving the file system in a worse state (loss of data depending on the nature of the corruption).


#  /usr/sbin/umountall -F acfs
/sbin/fsck -a -y -t acfs /dev/asm/[VOLUME_NAME]
# /usr/sbin/mount -v acfs /dev/asm/[VOLUME_NAME] [MOUNT_POINT]

Create a read-only snapshot of the cloud File System

[root@host01 .ACFS]# acfsutil snap create ro_dbsnap /mnt/acfs

acfsutil snap create: Snapshot operation is complete.

1.     Verify that the snapshot created above exists and is a read-only snapshot.

[root@host01 .ACFS]# acfsutil snap info /mnt/acfs

snapshot name: ro_dbsnap

snapshot location: /mnt/acfs/.ACFS/snaps/ro_dbsnap

RO snapshot or RW snapshot: RO

parent name: /mnt/acfs

snapshot creation time: Tue Jun 16 11:14:20 2015

number of snapshots: 1

snapshot space usage: 25313280 ( 24.14 MB )

Oracle ACFS snapshot

Oracle ACFS snapshot is an online, read-only or read-write, point in time copy of an Oracle ACFS file system.

The snapshot copy is space-efficient and uses Copy-On-Write functionality. Oracle ACFS snapshots are immediately available

for use after they are created. The snapshots are created in the .ACFS/snaps/ directory of the file system. They are always online

 while the file system is mounted. Oracle ACFS read-write snapshots enable fast creation of a snapshot image that can be both

 read and written without impacting the state of the Oracle ACFS file system

oracrs +ASM1 > /sbin/acfsutil info fs –h

oracrs +ASM1 > /sbin/acfsutil info fs /oracle/dbdump -s

CREATE ACFS SNAPSHOTS

TO create a snapshot in read-write or read-only mode

oracrs +ASM1> /sbin/acfsutil snap create SNAP_PREUPGRADE /oracle/dbadmin

acfsutil snap create: Snapshot operation is complete.

VIEW ACFS SNAPSHOTS

To know about the snapshot for the particular mount in the ACFS, below command will work from 11.2.0.3 onwards

$ /sbin/acfsutil snap info /oracle/dbdump

DELETE ACFS SNAPSHOTS

oracrs +ASM1 > /sbin/acfsutil snap delete SNAP_PREUPGRADE /oracle/dbadmin

acfsutil snap delete: Snapshot operation is complete.

To identify the file system which are allocated through the ACFS

 col FS_NAME format a15

 select * from V$ASM_FILESYSTEM;

To know about the mount point and volumes allocated in the ACFS

 col FS_NAME format a15

 col VOL_DEVICE format a15

 col VOL_LABEL format a15    

 select * from v$asm_acfsvolumes;

TO understand more about the current volume statistics for ACFS volumes

col VOLUME_NAME format a15

select * from V$ASM_VOLUME_STAT;

col VOLUME_NAME format a15

col USAGE format a15

 col MOUNTPATH  format a15

 select VOLUME_NAME,SIZE_MB,STRIPE_COLUMNS,USAGE,MOUNTPATH from v$asm_volume;

 SELECT SUBSTR(fs_name,1,34) FILESYSTEM,SUBSTR(snap_name,1,28) SNAPSHOT, CREATE_TIME TIME FROM V$ASM_ACFSSNAPSHOTS;

In side the ASMCMD tool, we can find about the volumes through the below commands

volcreate

voldelete

voldisable

volenable

volinfo

volresize

volset

volstat

Other  commands :

[oracle@server1 ~]$ srvctl config database -d ron     ( To see  database acfs  information )

[grid@server1 ~]$ acfsutil snap info RON /u02/app/oracle/oradata/datastore/

[root@oracle-one ~]# acfsutil info fs

[root@oracle-one snaps]# acfsutil snap delete acfsvol0_bak /ginf/grid/acfssystem/acfsvol0

ASMCMD> volcreate -G data -s 10G volume1

ASMCMD> volinfo --all

Once the volume is created it needs to be formatted using ACFS. This can be done in this way:

[oracle@server5 ~]$  mkfs -t acfs /dev/asm/volume1-162

acfsutil registry -a /dev/asm/volume1-162 /u01/oradata

SELECT volume_name, volume_device FROM V$ASM_VOLUME  WHERE volume_name ='VOLUME1';

 select volume_name,size_mb,state,volume_device from v$asm_volume;

col vol_label for a20

 col fs_name for a40

select * from v$asm_acfsvolumes;

Clone PDB using ACFS storage snapshot

PDB duplication using the storage snapshot feature.

 # Ensure that the source database – TOOLS is in read-only mode before clone.

This is required for creating a PDB from an existing PDB, when we use the CREATE PLUGGABLE DATABASE…FROM statement.

SQL> alter pluggable database tools close;

Pluggable database altered.

SQL> alter pluggable database tools open read only;

Pluggable database altered.

# Create the snapshot based clone (note the SNAPSHOT COPY syntax below):

The CREATE TABLE DDL is shown below. The command requests a copy of the TOOLS to be created in the same database, with a new name TOOLSCOPY. The SNAPSHOT COPY specifies that the duplication should be done using storage snapshots. The storage snapshot function is provided by the underlying ACFS drivers.

create pluggable database TOOLSCOPY from TOOLS

 file_name_convert=('/u01/app/grid/acfsmounts/data_vol1/oradata/CDB2/TOOLS',

                    '/u01/app/grid/acfsmounts/data_vol1/oradata/CDB2/TOOLSCOPY') SNAPSHOT COPY

/

SQL> @create_snap.sql;

Pluggable database created.

# Open the new db in read-write mode at least once to integrate it with the CDB.

SQL> alter pluggable database toolscopy open read write;

Check the status of the cloned database to make sure its integrated properly into the CDB.

SQL> select pdb_name,status from cdb_pdbs;

Reference : 

1) What diagnostic information to collect for ADVM/ACFS related issues (Doc ID 885363.1)

2) Create a file system with ACFS hang at mkfs -t acfs (Doc ID 2331497.1)

3) https://access.redhat.com/solutions/727333

4) http://www.asktheway.org/official-documents/oracle/E11882_01/server.112/e16102/asmfs_util001.htm

5) https://www.appservgrid.com/documentation111/docs/rdbms18c/ostmg/acfs-advanced-topics.html#GUID-67360198-D34F-4FBE-98A0-FAB1458FAABB