> ## Documentation Index
> Fetch the complete documentation index at: https://docs.futurex.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build datasets, dashboards, and alerts

> Shape the ingested HSM events into an Observe dataset, then build the dashboard cards and monitors that make the data usable.

Ingested events land in the datastream with the parsed attributes nested inside a `FIELDS` column. Before building dashboards or alerts, promote those attributes into real columns by publishing a shaped dataset. Every dashboard card and monitor then shares one definition.

## Shape and publish a dataset

The attributes the collector produced are addressable at `FIELDS.logs.attributes`. Bracket notation is required because the keys contain a dot.

<Steps>
  <Step>
    In Observe, open a worksheet and select your datastream, for example `futurex-hsm-syslog`.
  </Step>

  <Step>
    Open the OPAL editor and enter the following:

    ```text wrap theme={null}
    make_col hsm_device:string(FIELDS.logs.attributes["hsm.device"]), hsm_service:string(FIELDS.logs.attributes["hsm.service"]), hsm_level:string(FIELDS.logs.attributes["hsm.level"]), hsm_event_type:string(FIELDS.logs.attributes["hsm.event_type"]), hsm_message:string(FIELDS.logs.attributes["hsm.message"]), hsm_raw:string(FIELDS.logs.body)
    filter not is_null(hsm_device)
    ```
  </Step>

  <Step>
    Run the query and confirm the six new columns are populated.
  </Step>

  <Step>
    From the query menu, select **Create new dataset**, name it (for example `Futurex HSM Audit Events`), select **\[ Next ]**, then **\[ Publish ]**.
  </Step>
</Steps>

The `filter not is_null(hsm_device)` line matters more than it looks. It discards any record whose envelope did not parse, and it also excludes historical records ingested before a parsing change. Without it, older rows appear in dashboards with every column empty.

<Note>
  `hsm_raw` carries the unmodified syslog datagram. Keep it. It lets you reclassify historical events after you add a new pattern, and it is the evidence an auditor will ask for.
</Note>

## Build the dashboard

Create a dashboard, then add cards that query the published dataset.

For time-series cards, use `timechart` rather than the point-and-click aggregation builder. `timechart` buckets explicitly, so the result always has a clean time axis and one series per group.

| Card                       | Visualization | OPAL                                                                                                                                           |
| -------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Event volume by type       | Line chart    | `timechart 5m, event_count:count(), group_by(hsm_event_type)`                                                                                  |
| Failed logins and lockouts | Line chart    | `filter hsm_event_type = "auth_failure" or hsm_event_type = "auth_lockout"`<br />`timechart 5m, event_count:count(), group_by(hsm_event_type)` |
| Recent security events     | Table         | `filter hsm_event_type = "auth_failure" or hsm_event_type = "auth_lockout" or hsm_event_type = "config_change"`                                |

<Warning>
  After you change a card's OPAL, check the chart's **Y** encoding under **Chart settings**. Observe keeps the previous aggregate column in the encoding, so the card reports `Missing/Invalid X or Y field` and draws nothing until you select the new column and remove the stale one. This is the most common reason a card looks broken when the query is correct.
</Warning>

Give time-series cards enough height. A card only a couple of rows tall has no room for an axis once the legend is drawn, and the plot area renders empty even though the query returned data.

## Create monitors

Use the **Count** monitor type for all four alerts below. Each one queries the published dataset and filters by `hsm_event_type`.

| Monitor                          | OPAL filter                               | Condition                            |
| -------------------------------- | ----------------------------------------- | ------------------------------------ |
| HSM authentication failure burst | `filter hsm_event_type = "auth_failure"`  | count greater than 2 over 10 minutes |
| HSM account lockout              | `filter hsm_event_type = "auth_lockout"`  | count greater than 0 over 10 minutes |
| HSM configuration change         | `filter hsm_event_type = "config_change"` | count greater than 0 over 10 minutes |
| HSM log delivery stopped         | none; all events                          | count less than 1 over 30 minutes    |

To create each one:

<Steps>
  <Step>
    Select **New monitor**, then choose **Count**.
  </Step>

  <Step>
    Select the published dataset as the monitor query input.
  </Step>

  <Step>
    Open the OPAL editor and enter the filter for that monitor. Leave it empty for the log-delivery monitor.
  </Step>

  <Step>
    Under **Alerting rules**, set the comparison, the threshold, and the window.
  </Step>

  <Step>
    Name the monitor, then select **\[ Save changes ]**.
  </Step>
</Steps>

### Why these four

* **Authentication failure burst** uses a threshold rather than any-single-failure, because one mistyped password is not an incident. Two failures inside ten minutes is.
* **Account lockout** fires on a single event. The appliance only emits it after repeated failures, so it is already the summary of an attack pattern and needs no threshold of its own.
* **Configuration change** covers changes to device settings, including a change to the syslog forwarding target itself. That last one is worth watching: it is how someone would redirect the audit trail.
* **Log delivery stopped** is the one alert most integrations forget. Because forwarding is UDP with no acknowledgement, a broken path produces silence, not an error. Without this alert, the audit trail can stop and nothing reports it.

### Tune thresholds against the observed baseline

An HSM in service is never quiet. Client applications log in on a schedule, which produces a steady stream of `auth_success` and `identity_update` events, and appliance clock synchronization produces recurring `time_change` events, roughly every ten minutes in the tested environment.

Two consequences:

* The log-delivery monitor works precisely **because** the baseline is never zero. Set its window comfortably longer than the longest gap you observe between background events, then confirm the monitor does not fire during normal operation.
* Do **not** include `time_change` in the configuration-change monitor. It is routine automation, and grouping it with real configuration changes makes that alert fire continuously and get muted, which defeats its purpose.

Watch the count of events classified as `other`. A rise there means the appliance is emitting a message shape the classification rules do not cover yet, which is most likely after a firmware upgrade.


## Related topics

- [Log ingestion with Observe](/Integrations/VirtuCrypt/Log_ingestion_with_Observe/Log_ingestion_with_Observe.md)
- [Create an encrypted dataset](/Integrations/KMES_Series_3/Data_storage/TrueNAS/Create_an_encrypted_dataset.md)
- [Create an encrypted dataset in TrueNAS](/Integrations/CryptoHub/Data_storage/TrueNAS_Enterprise/Create_an_encrypted_dataset_in_TrueNAS.md)
- [Advanced metrics with Prometheus and Grafana](/Integrations/VirtuCrypt/Advanced_metrics_with_Prometheus_and_Grafana/Advanced_metrics_with_Prometheus_and_Grafana.md)
- [Validate the integration](/Integrations/VirtuCrypt/Log_ingestion_with_Observe/Validate_the_integration.md)