> ## Documentation Index
> Fetch the complete documentation index at: https://docs.futurex.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Appendix B: Troubleshooting
> Diagnosis steps for missing events, rejected SLFW commands, empty fields, wrong timestamps, and Observe endpoint errors.
This section covers the failures most likely to occur while setting up log ingestion with Observe.
## Isolate the stage before changing anything
The pipeline has three independent stages. Test them separately rather than guessing.
| Stage | Test |
| ---------------------------------- | ------------------------------------------------------------------------------------------ |
| HSM emits, collector receives | Run a plain UDP listener on a spare port, point `SLFW` at it, and confirm datagrams arrive |
| Collector parses | Enable the `debug` exporter and inspect the `hsm.` attributes |
| Collector exports, Observe ingests | Send a test payload with `curl` using the same token and endpoint |
A minimal listener is enough for the first stage. On the collector host:
```bash theme={null}
nc -u -l 5516
```
Then set the HSM destination to `5516`. Raw datagrams print directly to the terminal, which confirms the network path and shows the exact wire format without any parsing in the way.
## No events reach Observe
Work through these in order.
**Wait longer.** Allow at least 15 seconds. Checking too early is the most common cause of a false alarm, and it typically sends people investigating firewalls that are working correctly.
**Confirm forwarding is on.** Send `[AOSLFW;FS2;]` and check that the response contains `FS1` with the address and port you expect, not `FS0`.
**Confirm the HSM is generating events.** Produce a failed login on purpose. An idle appliance with no client traffic may genuinely have nothing to report.
**Confirm the collector is listening.** On the collector host, check that the port is bound:
```bash theme={null}
ss -lunp | grep 5515
```
**Confirm the network path.** The HSM initiates the connection, so the collector must be reachable from the HSM, not merely from your workstation. Test with a plain UDP listener as described above. A test sent from the collector host to its own address proves nothing, because that packet never leaves the machine.
**Check the host firewall.** The listener port must accept UDP from each HSM address.
UDP reports nothing when delivery fails. There is no error on the HSM, no rejection, and no gap marker. Absence of events is therefore not evidence about where the fault lies, which is why isolating each stage matters.
## The SLFW command is rejected
| Symptom | Cause |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Permission error | The logged-in identities lack `Diagnostics:Syslog Settings`. Check the `PR` token in the `GUSR` response. |
| Command not accepted on this port | `SLFW` requires the management port. It is not available on the production port. |
| Command accepted but nothing changes | The connection is not fully authorized. Confirm the second `GUSR` returned `LCY` and `UR0`. |
| `[AOGUSR;BBLOGIN FAILED;CNN;ERInvalid login.;]` | Wrong identity name or password. Check `MI` for missed attempts and `TO` for a brute-force lockout in seconds. |
The most frequent mistake is sending the logins and the `SLFW` command over separate connections. Authorization is tracked per connection and does not persist across reconnects.
## Events arrive but the fields are empty
Records reach Observe, but `hsm.device`, `hsm.service`, and `hsm.level` are missing, and the whole event sits in one string beginning with `: |`.
This means the `syslog` receiver is in use instead of `udplog`. The appliance's format is not standard RFC 3164, and a conforming parser discards the structure without reporting an error. Switch to the `udplog` receiver with the regular expression from [Deploy the OpenTelemetry Collector](./Deploy_the_OpenTelemetry_Collector), and see [Appendix A: Futurex syslog message format](./Appendix_A_Futurex_syslog_message_format) for the comparison.
## Some events are classified as `other`
The envelope parsed correctly but the message shape did not match any classification rule. Query for `hsm.event_type` equal to `other` and inspect `hsm.message`, then extend the `add` operators with a new pattern.
This is expected after a firmware upgrade introduces new message text. Because the unmodified datagram is retained in the log body, historical events can be reclassified once the new rule exists.
## Timestamps are wrong
| Symptom | Cause |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Events shifted by a whole number of hours | The parser used the collector's local timezone. Set `location: UTC`. |
| Events all share the collector's receive time | The timestamp parser did not run. Confirm the regular expression captures the microsecond payload timestamp and that `parse_from` points at it. |
| Events drift steadily from other sources | The HSM or collector clock is unsynchronized. Check NTP on both. |
## Observe returns an error
| Response | Cause and fix |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HTTP 404` | The signal suffix is missing. Logs post to `/v2/otel/v1/logs`. The collector's `otlphttp` exporter appends `/v1/logs`, so the configured endpoint must stop at `/v2/otel`. |
| `HTTP 401` | The token is missing, malformed, or disabled. Confirm the header is `Authorization: Bearer` followed by the full token including the colon. |
| `HTTP 429` | Ingest is being throttled. The exporter's `retry_on_failure` handles this. Investigate whether log volume is higher than expected. |
| Connection errors to `collect.observeinc.com` | Outbound HTTPS on port 443 is blocked, or the hostname omits `collect.`, which Observe requires for ingest routing. |
Do not configure the `otlp` exporter. Observe's OTLP endpoint does not support gRPC. The exporter must be `otlphttp`, or `otlp_http` on newer collectors.
## The collector will not start
| Symptom | Cause |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Unknown receiver type `udplog` | The core collector distribution is running. The `udplog` receiver only ships in the Contrib distribution. |
| Configuration unmarshal error | Check indentation. Operator lists under a receiver are indentation-sensitive. |
| Permission denied binding the port | A port below 1024 requires elevated privileges. Use a port above 1024. |
| Environment variable not found | `OBSERVE_CUSTOMER` or `OBSERVE_TOKEN` is not set in the collector's environment. |
## Log delivery stops silently
Because an HSM in service produces continuous authentication and identity traffic, a drop to zero events is meaningful. Alert on the absence of events over a window tuned to the observed baseline. Without that alert, a broken UDP path can go unnoticed indefinitely, since nothing in the pipeline reports the failure.
## Related topics
- [Troubleshooting](/Drivers_and_libraries/CryptoHub_CNG_developer_guide/Troubleshooting.md)
- [Appendix A: Troubleshoot external IdPs](/Integrations/VirtuCrypt/External_Identity_Providers_for_VirtuCrypt/Appendix_A_Troubleshoot_external_IdPs.md)
- [Appendix: Common errors](/Integrations/KMES_Series_3/Certificate_Authority/Venafi_Adaptable_CA/Appendix_Common_errors.md)
- [Installation](/Drivers_and_libraries/CryptoHub_CNG_developer_guide/Installation.md)
- [External Identity Providers for VirtuCrypt](/Integrations/VirtuCrypt/External_Identity_Providers_for_VirtuCrypt/External_Identity_Providers_for_VirtuCrypt.md)