Skip to main content
The Futurex PKCS #11 library writes its log to the <LOG-FILE> path in /etc/fxpkcs11.cfg (this guide uses /var/log/fxpkcs11/fxpkcs11.log). SAP ASE writes Cryptoki failures, with their CKR_ result codes, to the server error log in $SYBASE/ASE-16_0/install/<server>.log. Check both logs first. To see each command that the library sends to the HSM and the HSM response, set <LOG-MODE> to TRAFFIC and <LOG-TRAFFIC> to YES in fxpkcs11.cfg, restart SAP ASE, and reproduce the failure. Each entry below describes a symptom, then the cause and the fix.

​
The configuration test can’t connect to the HSM

Symptom: configTest fails, or reports no token in slot 0. Cause: The library can’t complete the TLS session to the Excrypt port. Solution: Check the following in order:
  1. fxpkcs11.cfg is in /etc, or FXPKCS11_CFG points at it, and the SAP ASE account can read it.
  2. The host can reach the HSM address in fxpkcs11.cfg on TCP port 9100, and no TLS inspection proxy sits in the path.
  3. <PROD-TLS-ANONYMOUS> matches the TLS setting of the Excrypt port. For mutual authentication, the SAP ASE account can read the PKCS #12 file and <PROD-TLS-KEY-PASS> is correct.

​
Login fails with PIN_INCORRECT although the password is correct

Symptom: PKCS11Manager reports Failed to login with: PIN_INCORRECT, or SAP ASE reports CKR_PIN_INCORRECT, with the correct identity password. With traffic logging enabled, the library log shows the HSM response [AOGUSR;CNN;BBLOGIN FAILED;ERMajor key not set.;...]. Cause: The HSM has no major keys loaded, so it rejects every identity login. The library reports the rejection as an incorrect PIN. Solution: Load the major keys as described in Load major keys, then log in again.

​
SAP ASE reports that C_GetFunctionList isn’t found

Symptom: When you create the HSM key, the SAP ASE error log reports Getting function list failed: /opt/sap/ASE-16_0/lib/libfxpkcs11.so: undefined symbol: C_GetFunctionList. Cause: The file at that path isn’t a Futurex PKCS #11 library build that exports the PKCS #11 entry points, for example an incomplete copy or a different file with the same name. Solution: Reinstall the library as described in Install Futurex PKCS #11 (FXPKCS11). Confirm that nm -D /opt/sap/ASE-16_0/lib/libfxpkcs11.so | grep ' C_GetFunctionList$' prints one line, then run the command again.

​
SAP ASE stops with a segmentation fault in SSL_CTX_new or BIO_new

Symptom: SAP ASE stops when you create the HSM key or when it first uses the HSM after a restart. The error log reports a segmentation fault (SIGSEGV) with the program counter in SSL_CTX_new or BIO_new. Cause: The PKCS #11 library is a build that uses OpenSSL symbols from outside the library. Inside SAP ASE, those names resolve to functions of the SAP ASE encryption provider instead of OpenSSL. Solution: Install the latest 6.x version of the Futurex PKCS #11 module, and run the checks in Check the build before you copy it to $SYBASE/ASE-16_0/lib.

​
Encrypted data is unavailable after a restart

Symptom: After SAP ASE restarts, queries on encrypted columns fail, or a fully encrypted database stays offline. The error log reports a failed Cryptoki call, such as C_Login with CKR_PIN_INCORRECT, or Msg 15432: A validation check failed when Adaptive Server decrypted an encryption key. Cause: SAP ASE couldn’t decrypt the master key through the HSM. Common reasons:
  • The stored HSM credential doesn’t match the identity password, for example after a password change.
  • The library can’t reach the HSM. The library log shows a connection or TLS error.
  • The application partition was changed and no longer allows a command that SAP ASE uses. The library log shows an AOERRO response for that command.
Solution: Correct the cause, then store the credential again if needed and bring any offline database online:
SQL

​
A wrong password is accepted when you store it

Symptom: sp_encryption 'hsm_credential' succeeds with a wrong password, and the failure appears only at the next key operation or restart. Cause: SAP ASE stores the credential without logging in to the HSM. Solution: After you store the password, confirm it with a key operation such as alter encryption key master with external key regenerate key.

​
create encryption key reports that a key already exists

Symptom: create encryption key ... on external keystore returns Msg 16166: A key already exists on HSM external keystore for this SAP ASE instance. Cause: SAP ASE supports one HSM key per server instance, and master already has one. Solution: Use the existing HSM key. To replace it, move every master key to password protection, drop the HSM key with drop encryption key hsm_key cascade, and create a new one. Dropping the HSM key with cascade also deletes it from the HSM.

​
Full database encryption doesn’t start

Symptom: alter database <database> encrypt with <key> returns Msg 5086: This operation requires 1 worker processes. Cause: Full database encryption runs as background worker tasks, and number of worker processes is 0. Solution: Run sp_configure 'number of worker processes', 4 and repeat the command.