Traces() and TracesByOrigin() Functions

To get traces, use the functions traces() or tracesByOrigin(). Use these functions to retreive and filter traces.

• the traces() function returns traces that passed through the entities resolved in the FROM clause
• the tracesByOrigin() function returns traces that originated in the entities resolved in the FROM clause

The Traces functions have the following fields. You can retrieve these fields using selector functions and filter them.

[^traces-entity-ids] Note that entityIds and originEntityIds represent the real entity that reported a particular span that contributed to the given trace. It also includes selected parent entities of such entity.

---
caption: The example of the traces() function 
emphasize-lines: 2, 3, 4
---
FETCH
   traces
      [name = 'HTTP GET']
      {traceId, duration}
FROM
   entities(container)

Searches for all containers, gets their ID and for each container it gets the traces that passed through that container. Filters only for traces named HTTP GET. For each trace, it returns its trace ID together with its duration.

Traces Without FROM Clause

When using traces without the FROM clause, both traces() and tracesByOrigin() return traces related to any entity in the system.

Structure

traces()[<filter-expression>]{<field>*}
tracesByOrigin()[<filter-expression>]{<field>*}
 
traces()[<filter-expression>].<field>
tracesByOrigin()[<filter-expression>].<field>

Filtering and Selecting Fields

The filter expression allows you to limit traces satisfying given criteria as described at filters.

To retrieve a field value, use a field selector function denoted by the field name.

Example

---
caption: The Filter and Select Field Example 
emphasize-lines: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14
---
FETCH
   traces
      [
         numErrors > 3 
         && duration > 'PT1M' 
         && originEntityIds(service) !IN ['service:abc', 'service:xyz']
         && entityIds >= ['service:1', 'service:2']
         && name ~ 'HTTP*'
         && source != 'apm'
      ] {
         traceId,
         duration,
         name,
         originEntityIds(service_instance)
      }
FROM
   entities(service)

Filter matches all traces passing through the services returned from the FROM that satisfy constraints:

• number of errors is greater than 3
• duration is longer than 1 minute
• the trace did not originate in the 'service:abc' nor 'service:xyz'
• the trace passed through both 'service:1' and 'service:2'; with name starting with HTTP and source value different from apm.

For each trace, it returns its trace ID, duration, name and origin entity IDs.

Filtering on typed entity IDs

When filtering on typed entity IDs, only entity IDs of the declared type are considered for the filter. Any other potential entity types are ignored by the filter as not important for the result.

This applies to traces() and tracesByOrigin() functions; filtering trace origination entities in case of tracesByOrigin().

Example

---
caption: The Filter on a typed entity IDs set 
emphasize-lines: 4, 5, 6, 7, 8
---
FETCH
   traces
      [
         entityIds(apm:request) = [
             apm:request:BYuiAFUiOZiGBqTGALPJFw,
             apm:request:KRfwzHOLOE6S4L+IRBKQ7w,
             apm:request:VLGzn2U+OZCrGXGHuR88Mw
         ]
      ]
FROM
   entities(service)
   

Filter matches all traces reported by the services returned by the FROM clause that contain the three listed apm:request:... entity IDs. No other apm:request:... entity ID can be present for a trace to pass the filter (= operator is used), but there may be any entity IDs of other types on the trace (example, apm:service:..., apm:instance_endpoint:..., any other types).

Note Using entity ID other than the filtered type in the list is not an error! For example, even if the trace does have apm:service:8BwWcAJ5PkupEAJQRTiVeA entity ID, filter entityIds(apm:request) >= [ apm:service:8BwWcAJ5PkupEAJQRTiVeA ] will never pass because the list of all apm:request:... IDs do not contain an ID apm:service:8BwWcAJ5PkupEAJQRTiVeA.

Traces Aggregation

Similar to topology aggregation, traces can be aggregated. To enable trace aggregation, add an aggregation function into the list of fetched fields.

Observed traces are split into buckets defined by aggregation dimensions and then aggregated using specified aggregation function.

This means each unique combination of dimension values is present only once in the result together with a single value aggregated for all traces that share this unique combination of dimensions. The only supported aggregation function, count produces a scalar value.

The aggregation function can also be used with just one dimension field. Or on its own, in which case all observed traces are aggregated to a single value. In case of count aggregation function, it is a number of all traces.

Supported aggregation dimensions

• count - count of Traces that share the dimension values

Example

---
caption: The Trace Aggregation Example 
emphasize-lines: 3, 4, 7
---
FETCH
   traces()
    [count > 10]
    { name, numErrors, count }
FROM
   entities(apm:service)
ORDER traces.desc(count)

The query counts traces from all apm:service entities which are bucketed by shared name and numErrors values. Only buckets containing more than 10 traces are returned. The result is ordered by the count, buckets with higher number of traces first.

Limits

To limit the maximum number of trace results returned by a query, set the count limit:

LIMITS traces.count(500)

Note This limit is applied to each set of traces separately and not globally. Meaning that if you retrieve 500 traces for 20 entity buckets the total number of traces retrieved is 500 x 20 = 10000.

The default limit on the number of traces is: 100

Ordering

You can order the returned traces using the non-complex fields.

ORDER traces.desc(duration)

The default order is ascending by startedAt field.

You can order up to only 1 ordering identifier (for example, ORDER traces.desc(duration).asc(startedAt) is not supported).

You can specify an alias for the field being fetched and use this same alias in the order clause.

---
caption: The ordering with an alias example 
emphasize-lines: 6
---
FETCH
   traces()
    { name, n: numErrors, count }
FROM
   entities(apm:service)
ORDER traces.desc(n)  

Ordering of aggregation queries

Aggregation queries can only be ordered by fetched dimensions or the aggregation function. This means, to use a dimension in an ORDER clause, you must use it in FETCH clause too.

---
caption: Examples of correct and wrong ordering of aggregated Traces 
emphasize-lines: 5
---
FETCH traces() { name, numSpans, count           } ORDER traces.desc(numSpans) // OK
FETCH traces() { name, numSpans, count           } ORDER traces.asc(name)      // OK
FETCH traces() { name, numSpans, count           } ORDER traces.desc(count)    // OK
FETCH traces() { name, numSpans, c: count        } ORDER traces.asc(c)         // OK
FETCH traces() { name, numSpans, count           } ORDER traces.desc(duration) // WRONG - duration not used in FETCH clause
FETCH traces() { name, numSpans, count, duration } ORDER traces.desc(duration) // OK

Result

The structure of the result is determined by what fields are specified. All the returned values, even the complex ones are inlined.

The set field selector functions entityIds and originEntityIds can be decorated with a formatting function, the possibilities are

• .csv() (the default)
• .json()

such as, entityIds(container).json which inlines the container IDs the trace passed through as an JSON array in the result.