Spans() Function

Use the spans() function to retrieve and filter spans. The spans() function returns spans that were reported by the entities resolved in the FROM clause.

Spans () Function Fields

The Spans() function has following fields. You can retrieve these fields using selector functions and filter them.

spans-entity-ids: The entityIds represents the real entity that reported a particular span. It also includes selected parent entities of such entity.

---
caption: The example of the spans() function 
emphasize-lines: 2, 3, 4
---
FETCH
   spans
      [name = 'HTTP GET']
      {spanId, duration}
FROM
   entities(container)

Searches for all containers, gets their ID and for each container it gets the spans that were reported by that container. Filters only for spans named HTTP GET. For each span, it returns its span ID together with its duration.

Spans without FROM Clause

When using spans() without the FROM clause, it returns spans related to any entity in the system.

Structure

spans()[<filter-expression>]{<field>*}

Filter and Select Fields

The filter expression allows to limit spans satisfying given criteria as described at filters.

To retrieve a field value, use a field selector function denoted by the field name.

Example

---
caption: The Filter and Select Field Example 
emphasize-lines: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14
---
FETCH
   spans
      [
         statusCode = 'Error' 
         && duration > 'PT1M' 
         && entityIds(service) !IN ['service:abc', 'service:xyz']
         && name ~ 'HTTP*'
         && attributes('http.status') IN [400, 500]
         && source != 'apm'
      ] {
         spanId,
         duration,
         name,
         entityIds(service)
      }
FROM
   entities(service)

Filter matches all spans reported by the services returned from the FROM that satisfy the following constraints:

• status code is 'Error'
• duration is longer than 1 minute
• the span was not reported by the 'service:abc' nor 'service:xyz'
• name starts with HTTP; with attribute http.status either as 400 or 500 and source value different from apm.

For each span, it returns its span ID, duration, name and the service that reported it.

Filtering on typed entity IDs

When filtering on typed entity IDs, only entity IDs of the declared type are considered for the filter. Any other potential entity types are ignored by the filter as not important for the result.

Example

---
caption: The Filter on a typed entity IDs set 
emphasize-lines: 4, 5, 6, 7, 8
---
FETCH
   spans
      [
         entityIds(apm:request) = [
             apm:request:BYuiAFUiOZiGBqTGALPJFw,
             apm:request:KRfwzHOLOE6S4L+IRBKQ7w,
             apm:request:VLGzn2U+OZCrGXGHuR88Mw
         ]
      ]
FROM
   entities(service)

Filter matches all spans reported by the services returned from the FROM that contain the three listed apm:request:... entity IDs.

No other apm:request:... entity ID can be present for a Span to pass the filter (= operator is used), but there may be any entity IDs of other types on the Span (e.g. apm:service:..., apm:instance_endpoint:..., any other types).

Note Using entity ID of other than the filtered type in the list is not an error. For example, even if the span has apm:service:8BwWcAJ5PkupEAJQRTiVeA entity ID, filter entityIds(apm:request) >= [ apm:service:8BwWcAJ5PkupEAJQRTiVeA ] does not pass because the list of all apm:request:... IDs do not contain an ID apm:service:8BwWcAJ5PkupEAJQRTiVeA.

Spans Aggregation

Similar to topology aggregation, spans can be aggregated. To enable the span aggregation, add an aggregation function into the list of fetched fields.

Observed spans are split into buckets defined by aggregation dimensions and then aggregated using specified aggregation function.

Each unique combination of dimension values will be present only once in the result together with a single value aggregated for all Spans that share this unique combination of dimensions. The aggregated value can be a simple scalar value or a structured value (example, a list) depending on the aggregation function.

The aggregation function can also be used with just one dimension field or on its own, in which case all observed Spans are aggregated to a single value. In case of count aggregation function, it is a number of all spans. In case of attributeNames aggregation function, it is a list of all attribute names and/or their counts (see below).

Supported aggregation dimensions

• count - count of Spans that share the dimension values
• attributeNames - unique list of attribute names used on all Spans with the respective dimension values, potentially with count of Spans with that attribute

Note You cannot use both count and attributeNames in the same spans query, only one is supported at a time.

Example of count aggregation

---
caption: The Span count Aggregation Example 
emphasize-lines: 3, 4, 7
---
FETCH
   spans()
    [count > 10]
    { attributes("service.name"), spanKind, count }
FROM
   entities(apm:service)
ORDER spans.desc(count)

The query counts spans from all apm:service entities which are bucketed by shared service.name and spanKind values. Only buckets containing more than 10 spans are returned. The result is ordered by the count, buckets with higher number of spans first.

attributeNames aggregation

attributeNames aggregation provides two pieces of data, attribute name and count of Spans on which that attribute is present.

User can request any of these values, or both: attributeNames{name, count}

If no value is specifically declared, both are provided in the result (attributeNames{name, count} is equal to just attributeNames).

The attributeNames name is always used for implicit ordering of the result. There is currently no support to order attributeNames by count.

Example of attributeNames aggregation

---
caption: The Span attributeNames Aggregation Example 
emphasize-lines: 3
---
FETCH
   spans()
    { attributeNames{name, count}, spanKind }

For each distinct value of spanKind, this query produces a list of names of attributes present on all Spans with that spanKind. For each attribute name, there is also a number of how many spans have that attribute.

Aggregation on missing dimension

Although an attribute specification (example, attributes("http.method")) can be used as aggregation dimension, not all spans have all specific attributes, that is, a span may not have a desired dimension. This leads to (potentially many) spans that are left off the result of aggregation queries.

Only spans that have all the desired dimensions are reflected in the query result.

| SpanId           | Attributes                                  |
|==================|=============================================|
| 5a6a63784e446c6c | ["http.method=GET" ,"http.status_code=200"] |
| 4e54426a59325530 | ["http.method=POST","http.status_code=200"] |
| 11db40517d78fb4e | [                   "http.status_code=200"] |
| 7a2c6195e847c577 | [                                         ] | (no attributes)
| 8444df2ca1cb2b90 | ["http.method=GET"                        ] |

FETCH spans() { attributes("http.method") }

| http_method |
|=============|
| GET         |
| POST        |
| null        |
| null        |
| GET         |

FETCH spans() { attributes("http.method"), count }

| http_method | count |
|=============|=======|
| GET         |     2 |
| POST        |     1 |

You can still query to fetch a count of spans that don't have a specific attribute, using a filter:

FETCH spans() [ attributes("http.method") = null ] { count }

Limits

To limit the maximum number of span results returned by a query set the count limit as follows:

LIMITS spans.count(500)

Note This limit is applied to each set of spans separately and not globally. This means if you retrieve 500 spans for 20 entity buckets the total number of spans retrieved is 500 x 20 = 10000.

The default limit on the number of spans is: 100

Order

Order the returned spans by the non-complex fields as follows:

ORDER spans.desc(duration)

The default order is ascending by startedAt field.

You can order using only 1 ordering identifier. For example ORDER spans.desc(duration).asc(startedAt) is not supported.

You can specify an alias for the field being fetched and use this same alias in the order clause.

Order using the functions: tags, attributes and attributeNames is not supported.

---
caption: The Ordering with an alias Example 
emphasize-lines: 4
---
FETCH
   spans()
    { a: statusCode, spanKind, count }
ORDER spans.asc(a)    

Ordering of aggregation queries

Aggregation queries can only be ordered by fetched dimensions or the aggregation function. That is, to use a dimension in ORDER clause, it must be used in FETCH clause too.

---
caption: Examples of correct and wrong ordering of aggregated Spans 
emphasize-lines: 5
---
FETCH spans() { name, spanKind, count           } ORDER spans.desc(spanKind) // OK
FETCH spans() { name, spanKind, count           } ORDER spans.asc(name)      // OK
FETCH spans() { name, spanKind, count           } ORDER spans.desc(count)    // OK
FETCH spans() { name, spanKind, c: count        } ORDER spans.asc(c)         // OK
FETCH spans() { name, spanKind, count           } ORDER spans.desc(duration) // WRONG - duration not used in FETCH clause
FETCH spans() { name, spanKind, count, duration } ORDER spans.desc(duration) // OK

Note Ordering using the attributeNames function is not supported because the attributeNames are always implicitly order by its name.

Linked Spans

Two spans can have a relation although they don't belong to the same span hierarchy. That is, one span is not the parent or the child of the other span. Also, they don't have to belong to the same trace. So, such type of spans are called linked spans.

You can fetch linked spans in two directions:

• upstream: The spans that happened before the base span and are related to it.

• downstream: The spans that happened after the base span and are related to it.

UQL allows to fetch linked spans of a base span. The base span indicates the span from which you want to start fetching the related spans on either upstream, downstream or both directions. For example, you start from a span 1111, follow its link upstream direction to fetch a span 2222. Here, span 1111 is the base span and span 2222 is the upstream linked span.

Example:

The following query fetches the span ID, all the upstream linked spans, and all the downstream linked spans of a base span. While fetching the upstream or downstream linked spans, the filter of the base span must be included. In this case, the base span's filter is the trace ID.

FETCH spans()[ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream,
  downstream
}

Linked Span Filter

You can add filters for the upstream and downstream functions. The filter returns the linked spans that comply the filter criteria.

Example:

The query fetches the upstream linked spans whose duration is greater than five minutes.

FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
{ 
  spanId,
  upstream[duration > 5m]
}

Linked Span Fields

The upstream or downstream fields can also have a valid list of fields to fetch. See Spans function for the list of fields that can be fetched. If you don't specify any fields explicitly, the default fields are fetched.

For example, in the following query, the upstream field specifically lists its own fields: name and duration:

FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream{ name, duration } 
}

You can also use filters along with the fields to fetch the required data. For example:

FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  spanId,
  upstream[duration > 5m]{ name, duration } 
}

Linked Spans Aggregation

The fetched list of linked spans (upstream or downstream) is grouped by base spans’ dimension fields. For example, the following query fetches name of the base spans, spanKind fields, and upstream-linked spans. The linked spans are grouped by name and spanKind fields of their respective base spans.

FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
 { 
   name, 
   spanKind, 
   upstream
 }

Spans dimension fields are optional. If they are missing, the result contains a single list of all linked spans.

Example:

FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ]
{
  /* nothing here */
  upstream{ traceId, spanId, statusCode } 
}

Output:

| upstream                              |
|=======================================|
| | traceId   | spanId   | statusCode | |
| |===========|==========|============| |
| |traceid1234|spanid1234|UNSET       | |
| |traceid1234|spanid2345|UNSET       | |
| |traceid4567|spanid7894|ERROR       | |

Limitations of Linked Spans Fetching

• Multiple level links are not allowed to fetch. For example:

  
     FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
     { 
        upstream{ upstream }  // not allowed 
     }
  

• Multiple upstream or downstream lists are not allowed to fetch:

     FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
     { 
        upstream, 
        upstream   // not allowed
     }
  

• Different filters for two lists are not allowed to fetch:

     FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
     { 
        upstream[ duration < 5m ], 
        upstream[ statusCode = 'ERROR']  // still not allowed
     }
  

  However, you can use combination of upstream and downstream lists. For example:

     FETCH spans() [ traceId = 'fcf95cb21f27d2070518cf7b63b2fe18' ] 
     { 
        upstream, 
        downstream
     }
  

Timerange and Limits Application to Linked Spans

The same time range (SINCE .. UNTIL) is applied on base spans as well as on linked spans to be fetched. If a linked span is out of SINCE..UNTIL time range, it will not be fetched.

The same LIMITS spans.count(xyz) constraint is applied on base spans as well as on linked spans to be fetched.

The limit is applied on the total number of eligible linked spans. In case the linked spans are divided into groups in the result (see Linked Spans Aggregation section), some of the groups may not return all linked spans that there are in Trace Store.

Example:

If there are these spans in the Trace Store:

| traceId     | spanId     | linkedSpans                                          |
|=============|============|======================================================|
| traceid1234 | spanid1111 | ["traceid1234:spanid4444", "traceid1234:spanid5555"] |
| traceid1234 | spanid2222 | ["traceid1234:spanid6666", "traceid1234:spanid7777"] |
| traceid1234 | spanid3333 | ["traceid1234:spanid8888", "traceid1234:spanid9999"] |
| traceid1234 | spanid4444 | null                                                 |
| traceid1234 | spanid5555 | null                                                 |
| traceid1234 | spanid6666 | null                                                 |
| traceid1234 | spanid7777 | null                                                 |
| traceid1234 | spanid8888 | null                                                 |
| traceid1234 | spanid9999 | null                                                 |

Then the query:

FETCH spans() [ traceId = 'traceid1234' ] { spanId, upstream{spanId} }
LIMITS spans.count(3)

Assuming all the spans fit within the default timerange

Will results in:

| spanId     | upstream                    |
|============|=============================|
| spanid1111 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | | traceid1234 |spanid4444 | |
|            | | traceid1234 |spanid5555 | |
|            |                             |
| spanid2222 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | | traceid1234 |spanid6666 | |  // spanid7777 is missing
|            |                             |
| spanid3333 | | traceId     | spanId    | |
|            | |=============|===========| |
|            | |             |           | |  // no linked spans

Only three linked spans in total are returned, because the LIMITS spans.count(3) limit was hit.

It is undefined which of the linked spans will be missing in such situation, hence it is unknown which group is missing some linked spans, if any.

Ordering of Linked Spans

Linked spans are not ordered. The ORDER spans clause doesn’t apply to linked spans. The actual order of fetched linked spans is undefined.

Result

The structure of the result is determined by the fields specified. All the returned values, even the complex ones are inlined.

The set field selector function entityIds can be combined with a formatting function, the possibilities are:

• .csv() (the default)
• .json()

For example, entityIds(container).json that inlines the container ID the span was reported at as an JSON array in the result.