Skip to main content
The Futurex PKCS #11 library writes its log to the <LOG-FILE> path in /etc/fxpkcs11.cfg (this guide uses /var/log/fxpkcs11/fxpkcs11.log). SAP ASE writes Cryptoki failures, with their CKR_ result codes, to the server error log in $SYBASE/ASE-16_0/install/<server>.log. Check both logs first. To see each request that the library sends to CryptoHub, set <LOG-TRAFFIC> to YES in fxpkcs11.cfg and reproduce the failure. Each entry below describes a symptom, then the cause and the fix.

​
The configuration test cannot connect to CryptoHub

Symptom: configTest fails, or reports no token in slot 0. Cause: The module cannot complete the mutual TLS session to the CryptoHub Host API. Solution: Check the following in order:
  1. fxpkcs11.cfg is in /etc, or FXPKCS11_CFG points at it.
  2. client.p12 and the .cer files are in the same directory as fxpkcs11.cfg, and the SAP ASE account can read them.
  3. The host can reach the CryptoHub hostname in fxpkcs11.cfg on TCP port 2001, and no TLS inspection proxy sits in the path.

​
SAP ASE stops with a segmentation fault in SSL_CTX_new or BIO_new

Symptom: SAP ASE stops when you store the HSM credential or run create encryption key ... on external keystore, or at startup. The error log reports a segmentation fault (SIGSEGV) with the program counter in SSL_CTX_new or BIO_new. Cause: The PKCS #11 library is a build that resolves its OpenSSL functions from the SAP ASE dataserver process, which exports functions with the same names for its own encryption provider. Solution: Use the library from the endpoint ZIP for the OpenSUSE 15.0 platform of your CryptoHub version, installed as $SYBASE/ASE-16_0/lib/libfxpkcs11.so. Don’t substitute a library from another platform or package.

​
Encrypted databases are offline after a restart

Symptom: After SAP ASE starts, sp_helpdb <database> shows the database as offline, and opening it returns Msg 15432: A validation check failed when Adaptive Server decrypted an encryption key. Cause: SAP ASE couldn’t decrypt the master key through CryptoHub during startup. The error log shows the reason:
  • C_Login failed with error code 160 (0xa0) CKR_PIN_INCORRECT: the stored HSM credential doesn’t match the endpoint PIN, for example after a PIN change.
  • A C_Decrypt failure, with an AOERRO permission error in the library log: CryptoHub refused the operation because the endpoint role was changed after deployment.
  • A connection or TLS error in the library log: SAP ASE couldn’t reach CryptoHub.
Solution: Correct the cause, then bring each database online:
SQL
For a refused operation, redeploy the service from the current SAP ASE template and install the new endpoint configuration.

​
A wrong PIN is accepted when you store it

Symptom: sp_encryption 'hsm_credential' succeeds with a wrong PIN, and the failure appears only at the next key operation or restart. Cause: SAP ASE stores the credential without logging in to CryptoHub. Solution: After you store the PIN, confirm it with a key operation such as alter encryption key master with external key regenerate key.

​
create encryption key reports that a key already exists

Symptom: create encryption key ... on external keystore returns Msg 16166: A key already exists on HSM external keystore for this SAP ASE instance. Cause: SAP ASE supports one HSM key per server instance, and master already has one. Solution: Use the existing HSM key. To replace it, move every master key to password protection, drop the HSM key with drop encryption key hsm_key cascade, and create a new one.

​
Full database encryption doesn’t start

Symptom: alter database <database> encrypt with <key> returns Msg 5086: This operation requires 1 worker processes. Cause: Full database encryption runs as background worker tasks, and number of worker processes is 0. Solution: Run sp_configure 'number of worker processes', 4 and repeat the command.