> ## Documentation Index
> Fetch the complete documentation index at: https://docs.futurex.com/llms.txt
> Use this file to discover all available pages before exploring further.

# RAC and RMAN considerations

> Configure Oracle Database 12c TDE with a Futurex HSM across Real Application Clusters nodes, and use RMAN for backup, restore, and recovery of TDE-encrypted databases.

This page extends the Oracle Database 12c TDE integration guide with guidance for Real Application Clusters (RAC) deployments and for backup, restore, and recovery operations using Oracle Recovery Manager (RMAN). It assumes you have already completed single-instance configuration of the Futurex PKCS #11 library (FxPKCS11) and verified that Oracle TDE can generate a Master Encryption Key (MEK) on the HSM.

<Info>
  Before working through this page, complete the single-instance setup steps in the main Oracle Database TDE (12c) integration guide, including FxPKCS11 installation, `fxpkcs11.cfg` configuration, and initial MEK generation. The RAC and RMAN procedures below build on a working single-instance integration.
</Info>

## RAC deployment considerations

Oracle RAC presents Oracle TDE as a single logical keystore across all cluster nodes. For HSM-backed TDE, this means every RAC instance must be able to reach the HSM and open the same keystore at startup. The following requirements apply to every RAC deployment.

### FxPKCS11 library on every node

The FxPKCS11 library must be installed at the same filesystem path on every RAC node, and `fxpkcs11.cfg` must be identical across nodes.

<Steps>
  <Step title="Install the FxPKCS11 library on each node">
    On every RAC node, install FxPKCS11 to the Oracle-expected HSM library location:

    ```shell expandable lines wrap title="Shell" theme={null}
    sudo mkdir -p /opt/oracle/extapi/64/hsm/futurex/<version>
    sudo cp libfxpkcs11.so /opt/oracle/extapi/64/hsm/futurex/<version>/
    sudo chown -R oracle:oinstall /opt/oracle/extapi
    sudo chmod 755 /opt/oracle/extapi/64/hsm/futurex/<version>/libfxpkcs11.so
    ```

    Replace `<version>` with the FxPKCS11 version you downloaded (for example, `4.10`).
  </Step>

  <Step title="Deploy fxpkcs11.cfg to each node">
    Copy your working `fxpkcs11.cfg` file to `/etc/fxpkcs11.cfg` on every RAC node. The file must be byte-identical across nodes, including the `<HSM>` connection details and any TLS client certificate paths.

    ```shell expandable lines wrap title="Shell" theme={null}
    sudo cp fxpkcs11.cfg /etc/fxpkcs11.cfg
    sudo chown oracle:oinstall /etc/fxpkcs11.cfg
    sudo chmod 640 /etc/fxpkcs11.cfg
    ```
  </Step>

  <Step title="Deploy TLS client certificates to each node">
    If your `fxpkcs11.cfg` references TLS client certificates for mutual authentication to the HSM, copy those certificates (and their private keys) to the referenced paths on every node, with identical permissions.
  </Step>

  <Step title="Verify HSM connectivity from each node">
    On each RAC node, verify that the FxPKCS11 library can reach the HSM:

    ```shell expandable lines wrap title="Shell" theme={null}
    /usr/bin/pkcs11-tool --module /opt/oracle/extapi/64/hsm/futurex/<version>/libfxpkcs11.so --list-slots
    ```

    The command must return the HSM slot information from every node before you continue. If any node fails to connect, resolve the network or certificate issue before proceeding — a failed HSM connection on one node will prevent that instance from starting once TDE is configured.
  </Step>
</Steps>

### Keystore location and sqlnet.ora

Oracle RAC requires that every instance reference the same keystore location. The keystore itself must live on shared storage reachable by all nodes.

<Warning>
  The `ENCRYPTION_WALLET_LOCATION` parameter in `sqlnet.ora` must resolve to the same physical location from every RAC node. If nodes see different directory contents at the configured path, TDE will behave unpredictably and may fail to open the keystore cluster-wide.
</Warning>

Place the keystore on one of the following shared storage options:

* **ACFS** (Oracle ASM Cluster File System) — recommended for most RAC deployments.
* **DBFS** (Database File System) — viable if you are already using DBFS in the environment.
* **NFS** — supported, but requires careful mount options (hard mounts, `actimeo=0`, `nolock`) to avoid stale keystore reads.

Configure `$ORACLE_HOME/network/admin/sqlnet.ora` identically on every node:

```none expandable lines wrap title="sqlnet.ora" theme={null}
ENCRYPTION_WALLET_LOCATION =
  (SOURCE =
    (METHOD = HSM)
    (METHOD_DATA =
      (DIRECTORY = /u01/app/oracle/admin/$ORACLE_UNQNAME/wallet)))
```

<Note>
  After modifying `sqlnet.ora` on any RAC node, you must restart the database instance on that node for the change to take effect. This is an Oracle requirement, not specific to the Futurex integration.
</Note>

## Auto-login keystore for RAC

Oracle supports two types of auto-login keystore:

* **Standard auto-login keystore** (`cwallet.sso`) — portable across hosts. Can be opened from any node that can read the file.
* **Local auto-login keystore** — tied to the hostname of the machine that created it. Will not open on any other host.

<Danger>
  Do not use the local auto-login keystore in a RAC deployment. A local auto-login keystore created on one RAC node cannot be opened by any other node. The database will fail to open TDE-encrypted tablespaces on those nodes, and failover will not work correctly. Always create a standard auto-login keystore for RAC.
</Danger>

Create the standard auto-login keystore from the password-protected keystore that contains your HSM credentials:

<Steps>
  <Step title="Connect to a running RAC instance as SYSDBA">
    ```shell expandable lines wrap title="Shell" theme={null}
    sqlplus / as sysdba
    ```
  </Step>

  <Step title="Create the standard auto-login keystore">
    ```sql expandable lines wrap title="Sql" theme={null}
    ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE
      FROM KEYSTORE '/u01/app/oracle/admin/$ORACLE_UNQNAME/wallet'
      IDENTIFIED BY "hsm_user_password";
    ```

    This generates a `cwallet.sso` file in the same directory as the password-protected keystore. Because the keystore directory is on shared storage, the `cwallet.sso` is immediately visible to all RAC nodes.
  </Step>

  <Step title="Verify auto-open from each node">
    Restart a single RAC instance and confirm that the keystore opens automatically without operator intervention:

    ```sql expandable lines wrap title="Sql" theme={null}
    SELECT WRL_TYPE, STATUS, WALLET_TYPE FROM V$ENCRYPTION_WALLET;
    ```

    The `STATUS` column should report `OPEN` and `WALLET_TYPE` should report `AUTOLOGIN`. Repeat this check on every RAC node after each node has been restarted.
  </Step>
</Steps>

<Tip>
  If `V$ENCRYPTION_WALLET` reports `CLOSED` on one node but `OPEN` on another, the most common cause is a difference in the local FxPKCS11 configuration or a network path to the HSM that is only reachable from some nodes. Verify HSM connectivity from the affected node first.
</Tip>

### Updating HSM credentials

If you rotate the HSM user password (the password used by Oracle to authenticate to the HSM), you must regenerate the auto-login keystore. Run the `ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE` command again using the new password. Because the keystore is on shared storage, a single regeneration propagates to all RAC nodes.

## Migrating a software keystore to the HSM

If the customer has an existing TDE deployment using a software keystore (`ewallet.p12`) and is migrating to HSM-backed TDE, Oracle 12c provides a direct migration path. This is the most common reason a customer will engage Futurex on an Oracle TDE deployment.

<Steps>
  <Step title="Verify the software keystore is open">
    ```sql expandable lines wrap title="Sql" theme={null}
    SELECT WRL_TYPE, STATUS FROM V$ENCRYPTION_WALLET;
    ```

    The software keystore must be `OPEN` before migration.
  </Step>

  <Step title="Configure sqlnet.ora for HSM">
    Update `sqlnet.ora` on all RAC nodes to point to the HSM method, as shown in the previous section. Restart each instance so the new configuration is picked up.
  </Step>

  <Step title="Execute the migration">
    ```sql expandable lines wrap title="Sql" theme={null}
    ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY
      IDENTIFIED BY "hsm_user_password"
      MIGRATE USING "software_keystore_password"
      WITH BACKUP;
    ```

    This generates a new MEK on the HSM and re-encrypts all existing table keys against the new HSM-resident MEK. The `WITH BACKUP` clause creates a backup of the original software keystore before migration.
  </Step>

  <Step title="Verify the migration">
    ```sql expandable lines wrap title="Sql" theme={null}
    SELECT WRL_TYPE, STATUS, WALLET_TYPE FROM V$ENCRYPTION_WALLET;
    ```

    After successful migration, `WRL_TYPE` should be `HSM` and `STATUS` should be `OPEN`. You can also confirm on the HSM that a new key has been generated in the configured key group.
  </Step>
</Steps>

<Warning>
  Do not delete the original software keystore after migration. Oracle retains references to historical table keys that may require the software keystore to decrypt. The software keystore should remain on disk (preferably on shared storage alongside the HSM auto-login keystore) for the lifetime of the database.
</Warning>

Reverse migration (HSM back to a software keystore) is also supported using:

```sql expandable lines wrap title="Sql" theme={null}
ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY
  IDENTIFIED BY "software_keystore_password"
  REVERSE MIGRATE USING "hsm_user_password"
  WITH BACKUP;
```

## RMAN backup with TDE

RMAN backup operations against a TDE-enabled database require the keystore to be open on the instance running the backup. No RMAN-specific configuration is required beyond a working TDE integration.

<Info>
  When RMAN backs up TDE-encrypted tablespaces, the data blocks remain encrypted with their original table keys on the backup media. This is the intended design — RMAN does not decrypt and re-encrypt the data during backup. The backup is only readable during a restore on an instance that has access to the HSM-resident MEK.
</Info>

Verify the keystore is open before initiating a backup:

```sql expandable lines wrap title="Sql" theme={null}
SELECT WRL_TYPE, STATUS FROM V$ENCRYPTION_WALLET;
```

Run the backup as you would for any Oracle database:

```shell expandable lines wrap title="Shell" theme={null}
rman target /

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;
```

<Warning>
  If the keystore is closed when RMAN attempts to back up an encrypted tablespace, the backup will fail with an `ORA-28365: wallet is not open` error. Ensure the auto-login keystore is correctly configured before scheduling unattended RMAN backups.
</Warning>

## RMAN restore

RMAN restore operations require that the same MEK used at backup time is still available on the HSM. The keystore must be open on the target instance before or during the restore, depending on whether the control file references a keystore location.

### Restoring to the original host

If you are restoring to the same host that performed the backup, the FxPKCS11 library, `fxpkcs11.cfg`, and network path to the HSM are already in place. Open the keystore and run the restore:

```sql expandable lines wrap title="Sql" theme={null}
ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN
  IDENTIFIED BY "hsm_user_password";
```

```shell expandable lines wrap title="Shell" theme={null}
rman target /

RMAN> STARTUP MOUNT;
RMAN> RESTORE DATABASE;
RMAN> RECOVER DATABASE;
RMAN> ALTER DATABASE OPEN;
```

### Restoring to an alternate host

Restoring to a host that has never run this database requires additional preparation. Before you can restore encrypted backups, the alternate host must be able to reach the HSM and authenticate as the same crypto user used at backup time.

<Steps>
  <Step title="Install FxPKCS11 on the alternate host">
    Install the FxPKCS11 library at the same path used on the original host (`/opt/oracle/extapi/64/hsm/futurex/<version>/libfxpkcs11.so`). Use the same version of the library to avoid potential incompatibilities.
  </Step>

  <Step title="Copy fxpkcs11.cfg to the alternate host">
    Copy the `fxpkcs11.cfg` file from the original host, along with any TLS client certificates it references. Ensure file permissions match the original.
  </Step>

  <Step title="Verify network reachability to the HSM">
    Confirm the alternate host can reach the HSM on the configured port:

    ```shell expandable lines wrap title="Shell" theme={null}
    /usr/bin/pkcs11-tool --module /opt/oracle/extapi/64/hsm/futurex/<version>/libfxpkcs11.so --list-slots
    ```

    If the HSM is behind a firewall, open the required port before proceeding.
  </Step>

  <Step title="Configure sqlnet.ora on the alternate host">
    Create `sqlnet.ora` with an `ENCRYPTION_WALLET_LOCATION` entry matching the original host. The directory path should point to where you intend to place the restored keystore files.
  </Step>

  <Step title="Restore the keystore files">
    Copy the `cwallet.sso` (and `ewallet.p12` if present) from your backup location to the directory configured in `sqlnet.ora`. The keystore files themselves are required because they contain the HSM user credentials needed to authenticate to the HSM.
  </Step>

  <Step title="Restore the database">
    ```shell expandable lines wrap title="Shell" theme={null}
    rman target /

    RMAN> STARTUP NOMOUNT;
    RMAN> RESTORE CONTROLFILE FROM '<backup_piece>';
    RMAN> ALTER DATABASE MOUNT;
    RMAN> RESTORE DATABASE;
    RMAN> RECOVER DATABASE;
    RMAN> ALTER DATABASE OPEN RESETLOGS;
    ```
  </Step>
</Steps>

<Danger>
  The original MEK used at backup time must still be present on the HSM at restore time. If the MEK has been rotated or destroyed since the backup was taken, the restore will succeed at the block level but encrypted tablespaces will be unreadable. Do not rotate or destroy MEKs while backups that depend on them are still within retention.
</Danger>

### Tablespace restore

Tablespace-level restore follows the same requirements as full database restore: the keystore must be open and the MEK present on the HSM. Tablespace point-in-time recovery (TSPITR) is also supported.

```shell expandable lines wrap title="Shell" theme={null}
rman target /

RMAN> SQL 'ALTER TABLESPACE users OFFLINE IMMEDIATE';
RMAN> RESTORE TABLESPACE users;
RMAN> RECOVER TABLESPACE users;
RMAN> SQL 'ALTER TABLESPACE users ONLINE';
```

## Recovery scenarios

### Point-in-time recovery

Database point-in-time recovery (DBPITR) requires the keystore to remain open throughout the entire recovery window. If the recovery spans a period during which the MEK was rotated, all historical MEKs must still be available on the HSM.

<Warning>
  If you rotate the MEK on the HSM while encrypted redo logs or archivelogs are still required for recovery, the HSM must retain all historical keys. Deleting a rotated-out MEK from the HSM before the archivelogs that reference it have been aged out of backup retention will prevent successful point-in-time recovery.
</Warning>

### Media recovery after instance failure

For a standard media recovery scenario (datafile loss on one RAC node), no TDE-specific steps are required beyond ensuring the keystore is open when the recovered instance restarts. With a correctly configured standard auto-login keystore, this happens automatically.

## Disaster recovery considerations

The MEK is the root of trust for all TDE-encrypted data in the database. Loss of the MEK means loss of access to all encrypted data, regardless of how intact the database backup files themselves are.

<Danger>
  If the HSM is destroyed or the MEK is otherwise irrecoverable, no RMAN backup can be restored to a usable state. The encrypted tablespaces in the backup cannot be decrypted without the MEK. Operational backup procedures for the HSM itself are a mandatory component of any TDE-enabled disaster recovery plan.
</Danger>

Follow these practices to ensure MEK recoverability:

* **Run regular Vectera Plus key backups.** See the Excrypt HSM Administrator guide for you version for key backup procedures. Store backups in a physically separate location from the primary HSM.
* **Do not rotate or destroy MEKs during the RMAN retention window.** Any MEK referenced by a backup within retention must remain on the HSM (or be restorable from a Vectera Plus key backup) until that backup has aged out.
* **Test HSM recovery procedures periodically.** Include a full TDE-restore-to-alternate-host drill in DR exercises, not just a database restore. A database restore that succeeds at the block level but fails at the first `SELECT` against an encrypted tablespace is not a successful drill.
* **Document the HSM user and crypto group mapping.** The HSM credentials in `fxpkcs11.cfg` must match the HSM configuration at restore time. Keep this mapping in your DR runbook.

<Check>
  A properly designed TDE disaster recovery plan covers four recoverable assets: the database backup, the software keystore (if present), the HSM key backup, and the HSM configuration (users, groups, TLS certificates). All four must be recoverable for a successful DR outcome.
</Check>