Events() Function

Use the events() function to get the event data. You can retrieve:

  • events as time series
  • event attributes
  • event tags
  • non-MELT properties as generic event fields
  • filtered event data by attributes, tags, and the raw field
  • events without specifying FROM topology part

The structure of the events() function is as follows:


FETCH
   id,
   events(logs)
      [attributes(severity) IN ['WARN', 'ERROR']]
      {timestamp, raw, attributes, eventFields}
FROM
   entities(entity.type)

Parameters, Filters, Fields

In the event() function, you can specify the types of events and the filter criteria as follows:

events(<event-type>)[<filter-expression>]{<field>*}
 
events(<event-type>)[<filter-expression>].<field>

Parameters

  • <event-type>: Identifies one or multiple events. The events can be fetched from event types with different namespaces. This is a mandatory parameter.

    The following example shows how to fetch events with multiple event types:

       FETCH
      events(audit:assign_tenant, k8s:native_event)

Filter expression

The filter expression allows you to select the events with specific values. The following points are applicable while using filters:

  • You can use filters on attributes, tags, raw fields, and recordId fields. Multiple binary filters can be joined with with AND && operator and OR ||operator. Note that || operator can not be combined with recordId conditions.

  • You can use the following operators to filter tags and attributes:

    • equal =
    • not equal !=
    • set IN []

  • Use the full-text operator ~ to filter raw fields.

  • Use the AND operator && to join multiple operators.

  • The attributes and tag values are compared by matching the whole (case-insensitive) value.

  • You can use the following additional operators to filter numerical event attributes. Note that at least one filtered event type specified in events(event_types) must have the attribute type specified as numerical in the schema.

    • greater than >

    • less than <

    • greater than or equal to >=

    • less than or equal to <=

  • A raw field value is matched if it contains all words from the search phrase in a given order next to each other.

  • The full-text operator ~ on a raw field supports wildcard patterns * and ? for matching 0-N characters or a single character respectively at the start or end of the word. The wildcard patterns are bound to match characters in only one word. For example, hell*rl won't match hello world.

Example: The filter expression.

   FETCH
      events(logs)
         [raw ~ 'Important mess*' && attributes(severity) IN ['WARN', 'ERROR'] && tags(cluster) != 'dev']
         {timestamp, raw}
   FROM
      entities(pods)

The filter expression matches all log events with the text containing the phrase Important message with severity WARN or ERROR that is not in the cluster dev.

Example: Numerical attributes comparison.

FETCH
   events(healthrule.violation)
      [attributes(violation_start_time) > 1699527700000 && attributes(violation_start_time) < 1699527800000]
FROM
   entities(pods)

The filter expression fetches all healthrule.violation events of Kubernetes pods with violation_start_time attribute greater than 1699527700000 and less than 1699527800000.

Fields

The fields attribute allows you to specify the exact data that you want to retrieve. You can specify:

  • string representation of the original event message. For example, raw.

  • specific attribute. For example, attributes("severity").

  • all attributes. For example, attributes.

  • specific tag. For example, tags("my-tag-name").

  • all tags. For example, tags.

  • other event properties as event fields. For example,eventFields.

  • timestamp.

  • source.

  • the ID of the entity (entityId) that generated the event.

  • span ID (spanId).

  • trace ID (traceId).

  • aggregation. For example, count, attributeCardinality("attributeName"), and lastEvent.

  • aggregation time dimension timeBucket with size. For example, timeBucket("15m").

The fields example:


FETCH
   events(logs)
      {timestamp, attributes("severity"), raw, eventFields}
FROM
   entities(pod)

The lastEvent aggregation function returns the last event record (the newest available record or the last seen record), which is stored that matches the aggregation criteria of a given query. It must be used in combination with source, tags(my-tag-name), attributes(my-attribute-name), or attributeCardinality("attributName") fields.

At most, there will be only one last event record that matches the aggregation criteria per event type.

Note

The last event record might not be unique among different aggregation criteria.

You can specify which fields you want to retrieve from the last event record by listing them in the lastEvent function. If you do not specify any fields, the function returns the timestamp and raw fields.

Example: No lastEvent fields were requested, returning timestamp and raw by default:

FETCH
   events(logs)
      {attributes("severity"), source, raw, lastEvent }
FROM
   entities(pod)

Example: Some lastEvent fields are requested:

FETCH
   events(logs)
      {tags("cluster"), attributes("severity"), source, count, lastEvent {spandId, traceId, raw} }
FROM
   entities(pod)

There are a few restrictions on the usage of lastEvent function:

  • At least one dimension like source, tags(my-tag-name) or attributes(my-attribute-name) must be specified.

  • It does not support any aggregation function like count, attributeCardinality or timeBucket.

  • It cannot be nested inside another lastEvent function.

  • It cannot be used as a filtering criterion in the events function.

Array Attributes

Array attributes (attributes that have the array schema type) are, by default, returned as a single string value to preserve backward compatibility. This string value is a concatenation of individual array items separated by a comma.

This behavior can be overridden by a new option called resultArrayFormat(format) where format can be:

  • CONCATENATED - concatenate array items into a single string value (default behavior)
  • ARRAY - return arrays as is
  • ANY_ITEM - reduce an array to any of its items

For the following examples let's suppose there is an array attribute called lines defined in the schema for the logs event type and that its value is [1, 42, 127] (an array).

Example: Concatenate array attributes (default behavior) 

FETCH
   events(logs) { attributes(lines) }
OPTIONS
   resultArrayFormat(CONCATENATED)

The above example would return "1,42,127".

NOTE: resultArrayFormat(CONCATENATED) is the default, so it does not need to be specified explicitly.

Example: Return array attributes as arrays:


FETCH
   events(logs) { attributes(lines) }
OPTIONS
   resultArrayFormat(ARRAY)

The example returns [1, 42, 127].

This behavior can also be controlled individually by using the concat, array, and anyItem functions which take precedence over the option value.

Concat Function

To concatenate array attributes regardless of the resultArrayFormat option value, use the concat function.

Example: Concatenate an array attribute when concatenation is disabled:

FETCH
   events(logs) { attributes(lines).concat() }
OPTIONS
   resultArrayFormat(ARRAY)

The example returns "1,42,127" even though the concatenation is disabled.

The concat function can also be used with a custom delimiter or with a custom delimiter, prefix and suffix:

Example: Concatenate with custom delimiter to produce: "1; 42; 127":

FETCH
   events(logs) { attributes(lines).concat("; ") }

Example: Concatenate with custom delimiter, prefix and suffix to produce: "( 1; 42; 127 )":


FETCH
   events(logs) { attributes(lines).concat("; ", "( ", " )") }

Array Function

To return the array value of array attributes regardless of the resultArrayFormat option value, use the array function.

Example: Returns the value of an array attribute when concatenation is enabled:


FETCH
   events(logs) { attributes(lines).array() }
OPTIONS
   resultArrayFormat(CONCATENATED)

The example returns [1, 42, 127] even though the concatenation is used.

Any Item Function

To reduce an array attribute to a single value of the correct array item type, use the anyItem function.

Example: Reduce an array attribute to a single value to produce: 1 or 42 or 127:


FETCH
   events(logs) { attributes(lines).anyItem() }

The returned value is one of the values of the array. The item is returned according to the array item type.

Note

There are no guarantees which item is returned from the array.

Sum Numerical Attributes

To add the values of the specified attributes of an event, use the sum(attributes(<attribute_name>)) function. The attribute must be of a numerical type and available in the schema.

Example: Count the torpedoes that are reloaded during the last 15 minutes in the Spacefleet reference solution.

The Starfleet reference solution sends the events torpedoes_reloaded that contain the number of the reloaded torpedoes in the event attribute count. You can use the sum function to aggregate these numbers and get the count of all reloaded torpedoes.

FETCH
  events(spacefleet:torpedoes_reloaded){ sum(attributes(count)) }

Tags

Tag values are arrays of strings. The same functions that are used for attributes can also be used on tags.

To concatenate tags into a single string, use the concat function. This is the default behavior.

To return the array value with a tag as is, use the array function.

To reduce a tag to a single item from the array, use the anyItem function.

Limits

You can set the count limit to restrict the maximum number of events returned by a query. This limit does not apply when you use the aggregation functions, because the result is not a list of events.

LIMITS events.count(50)

This limit is applied to each set of events separately and not globally. This means if you retrieve 50 events for 20 entities, the total number of events retrieved is 50 x 20 = 1000.

The current default limit on the number of events is 1000.

Control Pagination of Aggregation Results

Two modes that control the behavior of getting aggregated data in the result:

  • LIMITS events.aggregation(TOTAL) - The client receives a dataset of aggregated data once the query service assembles them for the entire time range. The benefit of this mode is that no additional processing needs to be done on the client side. However, the client has to wait until the calculation is complete to receive the data.

  • LIMITS events.aggregation(FRACTIONAL) - The client receives a part of the aggregated data as soon as the query service has them. The dataset contains a link to the next page of the data. However, since the calculation is not yet over, a data bucket with an aggregation key that is already seen in previous pages might appear again. In that case, it's the client's responsibility to add the values of all data buckets with duplicate keys.

The following example shows a client's responsibility to post-process the received aggregated data for FRACTIONAL aggregation mode:

| timeBucket        | severity | count |              | timeBucket        | severity | count |
|===================|==========|=======|              |===================|==========|=======|
| 2022-12-11T09:00Z | INFO     | 100   | post-process | 2022-12-11T09:00Z | INFO     | 100   |
| 2022-12-11T10:00Z | WARN     | 200   | by client    | 2022-12-11T10:00Z | WARN     | 200   |
| 2022-12-11T10:00Z | INFO     | 300   | --------->   | 2022-12-11T10:00Z | INFO     | 700   |
| 2022-12-11T10:00Z | INFO     | 400   |              | 2022-12-11T11:00Z | WARN     | 500   |
| 2022-12-11T11:00Z | WARN     | 500   |              |                   |          |       |

See event aggregation for an explanation of how to aggregate events.

Event Aggregation

Similar to topology aggregation, events could be aggregated using the Events function.

You enable the event aggregation by adding an aggregation function to the list of fetched fields. Events observed on all entities specified in the query FROM block are split into buckets defined by aggregation dimensions and then aggregated using a specified aggregation function.

Supported Aggregation Functions

  • count - counts the number of events in the time interval.

Supported Aggregation Dimensions

  • timeBucket("<duration>") - events are segregated into time intervals with the specified size. The size could be in the ISO duration or the Jira-like notation. See examples in the timerange definition chapter.

If time segregation is not requested, events are aggregated over the whole query time range.

  • attributes(<attribute-name>) - aggregated events are segregated by the value of the specific attribute. Up to one attribute dimension is allowed. Due to implementation details, aggregation currently works only for the attributes that are stored with SORTED data type in Dashbase of Event Store.

The following example shows event aggregation:


FETCH
   events(logs)
      {timeBucket("15m"), attributes("severity"), raw, eventFields}
FROM
   entities(pod)

The query counts events from all pods which are segregated by 15 minutes long intervals and the values of the attribute severity.

Result

The structure of the result is determined by the fields that you specify. Some values are inlined and others are multiline. The multiline values are referenced to other data chunks.

Inlined Values

  • timestamp

  • source

  • raw

  • attributes('single-attribute')

  • tags('single-tag')

  • count

  • timeBucket

Referenced values

  • all attributes

  • all tags

  • all eventFields

Events vs mergedEvents and FROM-less queries

The events function partitions events by entity. Each entity gets its dataset with events observed for that specific entity.

In the case of mergedEvents and FROM-less queries, all events are aggregated into a single dataset and sorted by timestamp.

Example Query Structure Result 
FETCH events(<event-type>)
FROM entities(<entity-id>*)
 Main dataset contains
references to datasets with
events for each individual entity
specified in the FROM clause		 
FETCH mergedEvents(<event-type>)
FROM entities(<entity-id>*)
 Main dataset contains
just one reference to a single
dataset with all events observed
on entities specified in
the FROM clause		 
FETCH events(<event-type>)
 Main dataset contains
just one reference to a single dataset
containing observed events
regardless of any entity		 
FETCH mergedEvents(<event-type>)
 Effectively, same as query above
without the FROM clause