Spans() Function

Use the spans() function to retrieve and filter spans. The spans() function returns spans that were reported by the entities resolved in the FROM clause.

Spans () Function Fields

The Spans() function has following fields. You can retrieve these fields using selector functions and filter them.

Field	Type	Description
spanId	string	The ID of the span.
parentId	string	The ID of the parent span.
traceId	string	The trace ID of the span.
name	string	The name of the span.
startedAt	dateTime	The start timestamp of the span.
duration	duration	The duration of the span.
source	string	The source of the span that is reported by the agent.
spanKind	string	The span kind
statusCode	string	The span status code
error message	string	The error message associated with the span status.
attributes([name])	complex	The span attributes; optional parameter `name` resolves to the value of the particular attribute of that name.
tags([name])	complex	The span tags; optional parameter `name` resolves to the value of the particular tag of that name.
entityIds([entityType])	complex	The entities the trace passed through 1; optional parameter entityType limits the entities to a given entity type.
upstream		The upstream-linked spans of the base span.
downstream		The downstream-linked spans of the base span.

spans-entity-ids: The entityIds represents the real entity that reported a particular span. It also includes selected parent entities of such entity.

block}

Copy---
caption: The example of the spans() function 
emphasize-lines: 2, 3, 4
---
FETCH
   spans
      [name = 'HTTP GET']
      {spanId, duration}
FROM
   entities(container)

Searches for all containers, gets their ID and for each container it gets the spans that were reported by that container. Filters only for spans named HTTP GET. For each span, it returns its span ID together with its duration.

Spans without FROM Clause

When using spans() without the FROM clause, it returns spans related to any entity in the system.

Structure

shell

Copyspans()[<filter-expression>]{<field>*}

Filter and Select Fields

The filter expression allows to limit spans satisfying given criteria as described at filters.

To retrieve a field value, use a field selector function denoted by the field name.

Example

block}

Copy---
caption: The Filter and Select Field Example 
emphasize-lines: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14
---
FETCH
   spans
      [
         statusCode = 'Error' 
         && duration > 'PT1M' 
         && entityIds(service) !IN ['service:abc', 'service:xyz']
         && name ~ 'HTTP*'
         && attributes('http.status') IN [400, 500]
         && source != 'apm'
      ] {
         spanId,
         duration,
         name,
         entityIds(service)
      }
FROM
   entities(service)

Filter matches all spans reported by the services returned from the FROM that satisfy the following constraints:

status code is 'Error'
duration is longer than 1 minute
the span was not reported by the 'service:abc' nor 'service:xyz'
name starts with HTTP; with attribute http.status either as 400 or 500 and source value different from apm.

For each span, it returns its span ID, duration, name and the service that reported it.

Filtering on typed entity IDs

When filtering on typed entity IDs, only entity IDs of the declared type are considered for the filter. Any other potential entity types are ignored by the filter as not important for the result.

Example

block}

Copy---
caption: The Filter on a typed entity IDs set 
emphasize-lines: 4, 5, 6, 7, 8
---
FETCH
   spans
      [
         entityIds(apm:request) = [
             apm:request:BYuiAFUiOZiGBqTGALPJFw,
             apm:request:KRfwzHOLOE6S4L+IRBKQ7w,
             apm:request:VLGzn2U+OZCrGXGHuR88Mw
         ]
      ]
FROM
   entities(service)

Filter matches all spans reported by the services returned from the FROM that contain the three listed apm:request:... entity IDs.

No other apm:request:... entity ID can be present for a Span to pass the filter (= operator is used), but there may be any entity IDs of other types on the Span (e.g. apm:service:..., apm:instance_endpoint:..., any other types).

Note Using entity ID of other than the filtered type in the list is not an error. For example, even if the span has apm:service:8BwWcAJ5PkupEAJQRTiVeA entity ID, filter entityIds(apm:request) >= [ apm:service:8BwWcAJ5PkupEAJQRTiVeA ] does not pass because the list of all apm:request:... IDs do not contain an ID apm:service:8BwWcAJ5PkupEAJQRTiVeA.

Spans Aggregation

Similar to topology aggregation, spans can be aggregated. To enable the span aggregation, add an aggregation function into the list of fetched fields.

Observed spans are split into buckets defined by aggregation dimensions and then aggregated using specified aggregation function.

Each unique combination of dimension values will be present only once in the result together with a single value aggregated for all Spans that share this unique combination of dimensions. The aggregated value can be a simple scalar value or a structured value (example, a list) depending on the aggregation function.

The aggregation function can also be used with just one dimension field or on its own, in which case all observed Spans are aggregated to a single value. In case of count aggregation function, it is a number of all spans. In case of attributeNames aggregation function, it is a list of all attribute names and/or their counts (see below).

Supported aggregation dimensions

count - count of Spans that share the dimension values
attributeNames - unique list of attribute names used on all Spans with the respective dimension values, potentially with count of Spans with that attribute

Note You cannot use both count and attributeNames in the same spans query, only one is supported at a time.

Example of count aggregation

block}

Copy---
caption: The Span count Aggregation Example 
emphasize-lines: 3, 4, 7
---
FETCH
   spans()
    [count > 10]
    { attributes("service.name"), spanKind, count }
FROM
   entities(apm:service)
ORDER spans.desc(count)

The query counts spans from all apm:service entities which are bucketed by shared service.name and spanKind values. Only buckets containing more than 10 spans are returned. The result is ordered by the count, buckets with higher number of spans first.

attributeNames aggregation

attributeNames aggregation provides two pieces of data, attribute name and count of Spans on which that attribute is present.

User can request any of these values, or both: attributeNames{name, count}

If no value is specifically declared, both are provided in the result (attributeNames{name, count} is equal to just attributeNames).

The attributeNames name is always used for implicit ordering of the result. There is currently no support to order attributeNames by count.

Example of attributeNames aggregation

block}

Copy---
caption: The Span attributeNames Aggregation Example 
emphasize-lines: 3
---
FETCH
   spans()
    { attributeNames{name, count}, spanKind }

For each distinct value of spanKind, this query produces a list of names of attributes present on all Spans with that spanKind. For each attribute name, there is also a number of how many spans have that attribute.

Aggregation on missing dimension

Although an attribute specification (example, attributes("http.method")) can be used as aggregation dimension, not all spans have all specific attributes, that is, a span may not have a desired dimension. This leads to (potentially many) spans that are left off the result of aggregation queries.

Only spans that have all the desired dimensions are reflected in the query result.

Example

If we have these spans in TraceStore

Code Snippet

Copy| SpanId           | Attributes                                  |
|==================|=============================================|
| 5a6a63784e446c6c | ["http.method=GET" ,"http.status_code=200"] |
| 4e54426a59325530 | ["http.method=POST","http.status_code=200"] |
| 11db40517d78fb4e | [                   "http.status_code=200"] |
| 7a2c6195e847c577 | [                                         ] | (no attributes)
| 8444df2ca1cb2b90 | ["http.method=GET"                        ] |

A non-aggregation query,

block}

CopyFETCH spans() { attributes("http.method") }

returns all spans, even those without the specified attribute. The missing attribute values are represented by null:

Code Snippet

Copy| http_method |
|=============|
| GET         |
| POST        |
| null        |
| null        |
| GET         |

But an aggregation query,

block}

CopyFETCH spans() { attributes("http.method"), count }

omits the spans completely that don't have the desired dimension:

Code Snippet

Copy| http_method | count |
|=============|=======|
| GET         |     2 |
| POST        |     1 |

You can still query to fetch a count of spans that don't have a specific attribute, using a filter:

block}

CopyFETCH spans() [ attributes("http.method") = null ] { count }

Limits

To limit the maximum number of span results returned by a query set the count limit as follows:

block}

CopyLIMITS spans.count(500)

Note This limit is applied to each set of spans separately and not globally. This means if you retrieve 500 spans for 20 entity buckets the total number of spans retrieved is 500 x 20 = 10000.

The default limit on the number of spans is: 100

Order

Order the returned spans by the non-complex fields as follows:

block}

CopyORDER spans.desc(duration)

The default order is ascending by startedAt field.

You can order using only 1 ordering identifier. For example ORDER spans.desc(duration).asc(startedAt) is not supported.

You can specify an alias for the field being fetched and use this same alias in the order clause.

Order using the functions: tags, attributes and attributeNames is not supported.

block}

Copy---
caption: The Ordering with an alias Example 
emphasize-lines: 4
---
FETCH
   spans()
    { a: statusCode, spanKind, count }
ORDER spans.asc(a)

Ordering of aggregation queries

Aggregation queries can only be ordered by fetched dimensions or the aggregation function. That is, to use a dimension in ORDER clause, it must be used in FETCH clause too.

block}

Copy---
caption: Examples of correct and wrong ordering of aggregated Spans 
emphasize-lines: 5
---
FETCH spans() { name, spanKind, count           } ORDER spans.desc(spanKind) // OK
FETCH spans() { name, spanKind, count           } ORDER spans.asc(name)      // OK
FETCH spans() { name, spanKind, count           } ORDER spans.desc(count)    // OK
FETCH spans() { name, spanKind, c: count        } ORDER spans.asc(c)         // OK
FETCH spans() { name, spanKind, count           } ORDER spans.desc(duration) // WRONG - duration not used in FETCH clause
FETCH spans() { name, spanKind, count, duration } ORDER spans.desc(duration) // OK

Note Ordering using the attributeNames function is not supported because the attributeNames are always implicitly order by its name.

Linked Spans

Two spans can have a relation although they don't belong to the same span hierarchy. That is, one span is not the parent or the child of the other span. Also, they don't have to belong to the same trace. So, such type of spans are called linked spans.

You can fetch linked spans in two directions:

upstream: The spans that happened before the base span and are related to it.
downstream: The spans that happened after the base span and are related to it.

UQL allows to fetch linked spans of a base span. The base span indicates the span from which you want to start fetching the related spans on either upstream, downstream or both directions. For example, you start from a span 1111, follow its link upstream direction to fetch a span 2222. Here, span 1111 is the base span and span 2222 is the upstream linked span.

Example:

The following query fetches the span ID, all the upstream linked spans, and all the downstream linked spans of a base span. While fetching the upstream or downstream linked spans, the filter of the base span must be included. In this case, the base span's filter is the trace ID.

Code Snippet

CopyFETCH spans()[ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream,
  downstream
}

Linked Span Filter

You can add filters for the upstream and downstream functions. The filter returns the linked spans that comply the filter criteria.

Example:

The query fetches the upstream linked spans whose duration is greater than five minutes.

Code Snippet

CopyFETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
{ 
  spanId,
  upstream[duration > 5m]
}

Linked Span Fields

The upstream or downstream fields can also have a valid list of fields to fetch. See Spans function for the list of fields that can be fetched. If you don't specify any fields explicitly, the default fields are fetched.

For example, in the following query, the upstream field specifically lists its own fields: name and duration:

Code Snippet

CopyFETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream{ name, duration } 
}

You can also use filters along with the fields to fetch the required data. For example:

Code Snippet

CopyFETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream[duration > 5m]{ name, duration } 
}

Linked Spans Aggregation

The fetched list of linked spans (upstream or downstream) is grouped by base spans’ dimension fields. For example, the following query fetches name of the base spans, spanKind fields, and upstream-linked spans. The linked spans are grouped by name and spanKind fields of their respective base spans.

Code Snippet

CopyFETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
 { 
   name, 
   spanKind, 
   upstream
 }

Spans dimension fields are optional. If they are missing, the result contains a single list of all linked spans.

Example:

Code Snippet

CopyFETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  /* nothing here */
  upstream{ traceId, spanId, statusCode } 
}

Output:

Code Snippet

Copy
| upstream                              |
|=======================================|
| | traceId   | spanId   | statusCode | |
| |===========|==========|============| |
| |traceid1234|spanid1234|UNSET       | |
| |traceid1234|spanid2345|UNSET       | |
| |traceid4567|spanid7894|ERROR       | |

Limitations of Linked Spans Fetching

Multiple level links are not allowed to fetch. For example:

Code Snippet

Copy
   FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
   { 
      upstream{ upstream }  // not allowed 
   }

Multiple upstream or downstream lists are not allowed to fetch:

Code Snippet

Copy   FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
   { 
      upstream, 
      upstream   // not allowed
   }

Different filters for two lists are not allowed to fetch:

Code Snippet

Copy   FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
   { 
      upstream[ duration < 5m ], 
      upstream[ statusCode = 'ERROR']  // still not allowed
   }

However, you can use combination of upstream and downstream lists. For example:

Code Snippet

Copy   FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
   { 
      upstream, 
      downstream
   }

Timerange and Limits Application to Linked Spans

The same time range (SINCE .. UNTIL) is applied on base spans as well as on linked spans to be fetched. If a linked span is out of SINCE..UNTIL time range, it will not be fetched.

The same LIMITS spans.count(xyz) constraint is applied on base spans as well as on linked spans to be fetched.

The limit is applied on the total number of eligible linked spans. In case the linked spans are divided into groups in the result (see Linked Spans Aggregation section), some of the groups may not return all linked spans that there are in Trace Store.

Example:

If there are these spans in the Trace Store:

Code Snippet

Copy
| traceId     | spanId     | linkedSpans                                          |
|=============|============|======================================================|
| traceid1234 | spanid1111 | ["traceid1234:spanid4444", "traceid1234:spanid5555"] |
| traceid1234 | spanid2222 | ["traceid1234:spanid6666", "traceid1234:spanid7777"] |
| traceid1234 | spanid3333 | ["traceid1234:spanid8888", "traceid1234:spanid9999"] |
| traceid1234 | spanid4444 | null                                                 |
| traceid1234 | spanid5555 | null                                                 |
| traceid1234 | spanid6666 | null                                                 |
| traceid1234 | spanid7777 | null                                                 |
| traceid1234 | spanid8888 | null                                                 |
| traceid1234 | spanid9999 | null                                                 |

Then the query:

Code Snippet

CopyFETCH spans() [ traceId = 'traceid1234' ] { spanId, upstream{spanId} }
LIMITS spans.count(3)

Assuming all the spans fit within the default timerange

Will results in:

Code Snippet

Copy| spanId     | upstream                    |
|============|=============================|
| spanid1111 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | | traceid1234 |spanid4444 | |
|            | | traceid1234 |spanid5555 | |
|            |                             |
| spanid2222 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | | traceid1234 |spanid6666 | |  // spanid7777 is missing
|            |                             |
| spanid3333 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | |             |           | |  // no linked spans

Only three linked spans in total are returned, because the LIMITS spans.count(3) limit was hit.

It is undefined which of the linked spans will be missing in such situation, hence it is unknown which group is missing some linked spans, if any.

Ordering of Linked Spans

Linked spans are not ordered. The ORDER spans clause doesn’t apply to linked spans. The actual order of fetched linked spans is undefined.

Result

The structure of the result is determined by the fields specified. All the returned values, even the complex ones are inlined.

The set field selector function entityIds can be combined with a formatting function, the possibilities are:

.csv() (the default)
.json()

For example, entityIds(container).json that inlines the container ID the span was reported at as an JSON array in the result.