Apache SkyWalking – Grafana

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Wed, 08 Apr 2026 00:00:00 +0000

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Apache SkyWalking introduced TraceQL support in version 10.4.0, implementing Grafana Tempo’s HTTP query APIs so that Grafana can query and visualize traces stored in SkyWalking without any additional plugins. This means you can now use the familiar Grafana Tempo data source to search, filter, and drill into both SkyWalking native traces and Zipkin-compatible traces — all served by your existing SkyWalking OAP server.

Architecture Overview

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

The TraceQL Service sits inside the OAP server and exposes the Tempo-compatible HTTP API on port 3200 (default). It converts traces from their native format into Tempo’s format, where the trace detail part (Trace message) reuses OTLP Trace definitions.

Limitations and Supported TraceQL Features

TraceQL is a rich query language, but SkyWalking currently implements a practical subset. The following features are supported:

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

The following features are not yet supported:

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

Important: SkyWalking native trace support in TraceQL is based on the Query Traces V2 API. Currently, only BanyanDB storage implements this API. Other storage backends (e.g., Elasticsearch, MySQL, PostgreSQL) do not support SkyWalking native trace queries via TraceQL. Zipkin-compatible traces are not subject to this restriction.

Trace Format Conversion

Since the trace detail part of Tempo’s format reuses OTLP Trace definitions, the conversion descriptions below refer to OTLP field names (e.g., span kind, status code).

SkyWalking Native Trace

Trace ID Encoding

SkyWalking native trace IDs are arbitrary strings (e.g., 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539), while Grafana Tempo requires pure hex-encoded trace IDs. The TraceQL Service encodes each UTF-8 byte of the original trace ID as two lowercase hex characters:

Original:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
Encoded:   32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

This encoded hex trace ID is what appears in all API responses and in Grafana. When you click a trace ID in Grafana, the TraceQL Service automatically decodes it back to the original SkyWalking trace ID for the internal query.

Span Kind Mapping

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

Status Mapping

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents are converted to OTLP span events, with tags mapped as string attributes and summary mapped as numeric attributes (serialized as strings).

Zipkin Trace

Span Kind Mapping

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

Status Mapping

If the otel.status_code tag is present, it is used directly.
Otherwise, if the error tag equals true, the status is STATUS_CODE_ERROR.
If neither tag is present, the status defaults to STATUS_CODE_UNSET.

Endpoint and Annotation Mapping

Zipkin endpoint fields are mapped to OTLP attributes (e.g., localEndpoint.ipv4 → net.host.ip), and Zipkin annotations are converted to OTLP span events.

For the full conversion details, see the TraceQL Service documentation.

How to Enable TraceQL

Step 1: Enable the TraceQL Module

By default, the TraceQL module is disabled (selector: ${SW_TRACEQL:-}). To enable it, set the selector to default:

# In application.yml
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

Or via environment variables:

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

If you want to query Zipkin traces, you also need to enable the Zipkin receiver so that SkyWalking can ingest Zipkin trace data:

# In application.yml
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

Or via environment variable:

export SW_RECEIVER_ZIPKIN=default

Full Configuration Reference

For the complete list of all configuration options and their default values, see the Configuration section of the TraceQL Service documentation.

Configuring Grafana Tempo Data Source

Prerequisite: Grafana 12 or later is required.

Each trace backend (SkyWalking native / Zipkin) needs its own Tempo data source in Grafana, because each is served under a different context path.

Context Paths

The two backends are served under separate context paths on the same port:

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

Setting Up the SkyWalking Data Source

In Grafana, go to Configuration → Data Sources → Add data source.
Choose Tempo.
Set the URL to http://<oap-host>:3200/skywalking.
Disable the Streaming option (SkyWalking does not support streaming mode).

Save and test the data source.

Setting Up the Zipkin Data Source

Same as above, but set the URL to http://<oap-host>:3200/zipkin.

Configuring Trace List Result Tags

When you search for traces in Grafana, the trace list panel shows a summary of each trace. The tracesListResultTags configuration controls which span tags are included in the search result and displayed as columns in the trace list.

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

Note that service.name and span.kind are always included regardless of this setting.

These tags appear as attribute columns in the Grafana Tempo trace search results, making it easier to identify and group traces at a glance:

SkyWalking native trace list:

Zipkin trace list:

You can customize these tags based on your application’s instrumentation. For example, if your services heavily use messaging, you might add mq.destination or messaging.system to the list.

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Go to the Explore page in Grafana.
Select the Tempo data source you configured for SkyWalking (e.g., SkyWalkingTraceQL).
Run a test query, then click Add to dashboard and save it as SkyWalking Trace.

Step 2: Configure Variables

Add dashboard variables so users can filter traces dynamically (e.g., by service name):

Step 3: Add a Trace Panel

Choose a Table chart (or edit the panel you saved).
Set Query type to Search.
Set the Service Name query condition to the variable $Service.
Add other query conditions as needed (e.g., duration, span name, tags).
Test and save.

Step 4: View Trace Details

Click any trace ID in the trace panel to jump to the Explore page showing the full trace waterfall view with all spans, tags, and events:

Zipkin Trace Dashboard

The setup for Zipkin traces is identical to SkyWalking native traces — just use the Zipkin Tempo data source you configured (e.g., ZipkinTraceQL).

Zipkin trace detail view:

Summary

With TraceQL support in SkyWalking 10.4.0, you can now leverage Grafana’s powerful Tempo data source to query and visualize both SkyWalking native traces and Zipkin-compatible traces. The key points to remember:

Enable the TraceQL module by setting SW_TRACEQL=default and enabling the desired backends.
Configure separate Tempo data sources in Grafana for each backend (/skywalking and /zipkin).
Disable the Streaming option in the Grafana Tempo data source settings.
Customize result tags via SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS and SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS to control what’s shown in search results.
SkyWalking native trace queries require BanyanDB storage (Zipkin traces work with all storage backends).

For the complete API reference and conversion details, see the TraceQL Service documentation. For Grafana integration details, see Use Grafana As The UI.

Zh: 使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Wed, 08 Apr 2026 00:00:00 +0000

使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Apache SkyWalking 在 10.4.0 版本中引入了 TraceQL 支持，实现了 Grafana Tempo 的 HTTP 查询 API，使 Grafana 无需任何额外插件即可查询和可视化 SkyWalking 中存储的链路追踪数据。这意味着你现在可以使用熟悉的 Grafana Tempo 数据源来搜索、过滤和深入分析 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪 —— 所有数据都由现有的 SkyWalking OAP 服务器提供。

架构概览

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

TraceQL Service 位于 OAP 服务器内部，在端口 3200（默认）上暴露 Tempo 兼容的 HTTP API。它将链路追踪数据从原生格式转换为 Tempo 的格式，其中链路追踪详情部分（Trace 消息）复用了 OTLP Trace 定义。

支持的 TraceQL 特性与限制

TraceQL 是一种功能丰富的查询语言，但 SkyWalking 目前实现了一个实用的子集。以下特性已支持：

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

以下特性暂不支持：

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

重要提示：TraceQL 中的 SkyWalking 原生链路追踪支持基于 Query Traces V2 API。目前只有 BanyanDB 存储实现了该 API。其他存储后端（如 Elasticsearch、MySQL、PostgreSQL）不支持通过 TraceQL 查询 SkyWalking 原生链路追踪数据。 Zipkin 兼容链路追踪不受此限制。

Trace 格式转换

由于 Tempo 格式的链路追踪详情部分复用了 OTLP Trace 定义，以下转换描述使用 OTLP 字段名称（如 span kind、status code）。

SkyWalking 原生链路追踪

Trace ID 编码

SkyWalking 原生 trace ID 是任意字符串（例如 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539），而 Grafana Tempo 要求纯十六进制编码的 trace ID。TraceQL Service 将原始 trace ID 的每个 UTF-8 字节编码为两个小写十六进制字符：

原始值:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
编码后:  32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

编码后的十六进制 trace ID 会出现在所有 API 响应和 Grafana 中。当你在 Grafana 中点击 trace ID 时，TraceQL Service 会自动将其解码回原始的 SkyWalking trace ID 进行内部查询。

Span Kind 映射

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

状态映射

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents 被转换为 OTLP span events，其中 tags 映射为字符串属性，summary 映射为数值属性（序列化为字符串）。

Zipkin 链路追踪

Span Kind 映射

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

状态映射

如果存在 otel.status_code 标签，则直接使用。
否则，如果 error 标签等于 true，则状态为 STATUS_CODE_ERROR。
如果以上标签都不存在，则状态默认为 STATUS_CODE_UNSET。

Endpoint 与 Annotation 映射

Zipkin endpoint 字段被映射为 OTLP 属性（例如 localEndpoint.ipv4 → net.host.ip）， Zipkin annotations 被转换为 OTLP span events。

完整的转换详情请参阅 TraceQL Service 文档。

如何启用 TraceQL

步骤 1：启用 TraceQL 模块

默认情况下，TraceQL 模块是禁用的（selector: ${SW_TRACEQL:-}）。要启用它，将 selector 设置为 default：

# 在 application.yml 中
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

或通过环境变量设置：

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

步骤 2：启用 Zipkin 接收器（仅用于 Zipkin 链路追踪）

如果你需要查询 Zipkin 链路追踪数据，还需要启用 Zipkin 接收器，以便 SkyWalking 能够接收 Zipkin 链路追踪数据：

# 在 application.yml 中
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

或通过环境变量设置：

export SW_RECEIVER_ZIPKIN=default

完整配置参考

所有配置选项及其默认值的完整列表，请参阅 TraceQL Service 文档的配置章节。

配置 Grafana Tempo 数据源

前提条件：需要 Grafana 12 或更高版本。

每个链路追踪后端（SkyWalking 原生 / Zipkin）需要在 Grafana 中配置各自独立的 Tempo 数据源，因为它们分别在不同的上下文路径下提供服务。

上下文路径

两个后端在同一端口上使用不同的上下文路径提供服务：

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

配置 SkyWalking 数据源

在 Grafana 中，前往 Configuration → Data Sources → Add data source。
选择 Tempo。
将 URL 设置为 http://<oap-host>:3200/skywalking。
禁用 Streaming 选项（SkyWalking 不支持流式模式）。

保存并测试数据源。

配置 Zipkin 数据源

与上述步骤相同，但将 URL 设置为 http://<oap-host>:3200/zipkin。

配置链路追踪列表结果标签

在 Grafana 中搜索链路追踪时，链路追踪列表面板会显示每条追踪的摘要信息。 tracesListResultTags 配置控制哪些 span 标签会包含在搜索结果中并作为列显示在追踪列表中。

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

注意，无论此设置如何，service.name 和 span.kind 始终包含在结果中。

这些标签在 Grafana Tempo 链路追踪搜索结果中显示为属性列，方便快速识别和分组追踪数据：

SkyWalking 原生追踪列表：

Zipkin 追踪列表：

你可以根据应用程序的埋点情况自定义这些标签。例如，如果你的服务大量使用消息队列，可以在列表中添加 mq.destination 或 messaging.system。

在 Grafana 中构建链路追踪仪表板

SkyWalking 原生追踪仪表板

步骤 1：探索并保存

前往 Grafana 的 Explore 页面。
选择你为 SkyWalking 配置的 Tempo 数据源（例如 SkyWalkingTraceQL）。
运行一个测试查询，然后点击 Add to dashboard 并保存为 SkyWalking Trace。

步骤 2：配置变量

添加仪表板变量，以便用户可以动态过滤追踪数据（例如按服务名称过滤）：

步骤 3：添加追踪面板

选择 Table 图表（或编辑你保存的面板）。
将 Query type 设置为 Search。
将 Service Name 查询条件设置为变量 $Service。
根据需要添加其他查询条件（如 duration、span name、tags）。
测试并保存。

步骤 4：查看追踪详情

点击追踪面板中的任意 trace ID，即可跳转到 Explore 页面，查看完整的追踪瀑布图，包含所有 span、标签和事件：

Zipkin 追踪仪表板

Zipkin 追踪的设置与 SkyWalking 原生追踪完全相同 —— 只需使用你配置的 Zipkin Tempo 数据源（例如 ZipkinTraceQL）。

Zipkin 追踪详情视图：

总结

通过 SkyWalking 10.4.0 中的 TraceQL 支持，你现在可以利用 Grafana 强大的 Tempo 数据源来查询和可视化 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪数据。需要记住的要点：

启用 TraceQL 模块：设置 SW_TRACEQL=default 并启用所需的后端。
在 Grafana 中配置独立的 Tempo 数据源：为每个后端分别配置（/skywalking 和 /zipkin）。
禁用 Streaming 选项：在 Grafana Tempo 数据源设置中关闭流式模式。
自定义结果标签：通过 SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS 和 SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS 控制搜索结果中显示的内容。
SkyWalking 原生追踪查询需要 BanyanDB 存储（Zipkin 追踪支持所有存储后端）。

完整的 API 参考和转换详情，请参阅 TraceQL Service 文档。 Grafana 集成详情，请参阅使用 Grafana 作为 UI。

Blog: Build Grafana dashboards for Apache SkyWalking -- Native PromQL Support

Thu, 16 Mar 2023 00:00:00 +0000

Background

As an application performance monitoring tool for distributed systems, Apache SkyWalking provides monitoring, tracing, diagnosing capabilities for distributed system in Cloud Native architecture. Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem. Especially Prometheus metrics receive widespread support through exporters and integrations. PromQL as Prometheus Querying Language containing a set of expressions and expose HTTP APIs to read metrics.

SkyWalking supports to ingest Prometheus metrics through OpenTelemetry collector and through the aggregate calculation of these metrics to provide a variety of systems monitoring, such as Linux Monitoring and Kubernetes monitoring. SkyWalking already provides native UI and GraphQL API for users. But as designed to provide wider ecological integration capabilities, since 9.4.0, it provides PromQL Service, the third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through it. SkyWalking users will benefit from it when they integrate with different systems.

What is PromQL Service in SkyWalking?

PromQL Service is a query engine on the top of SkyWalking native GraphQL query, with additional query stage calculation capabilities powered by Prometheus expressions. It can accept PromQL HTTP API requests, parse Prometheus expressions, and transform between Prometheus metrics and SkyWalking metrics.

The PromQL Service follows all PromQL’s protocols and grammar and users can use it as they would with PromQL. As SkyWalking is fundamentally different from Prometheus in terms of metric classification, format, storage, etc. PromQL Service doesn’t have to implement the full PromQL feature. Refer to the documentation for the detail.

SkyWalking Basic Concepts

Here are some basic concepts and differences from Prometheus that users need to understand in order to use the PromQL service: Prometheus metrics specify the naming format and structure, the actual metric names and labels are determined by the client provider, and the details are stored. The user aggregates and calculates the metrics using the expression in PromQL. Unlike Prometheus, SkyWalking’s metric mechanism is built around the following core concepts with a hierarchical structure:

Layer: represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). This layer would be the owner of different services detected from different technologies. All Layers definitions can be found here.
Service: Represents a set/group of workloads which provides the same behaviors for incoming requests.
Service Instance: An individual workload in the Service group.
Endpoint: A path in a service for incoming requests.
Process: An operating system process. In some scenarios, a service instance is not a process, such as a pod Kubernetes could contain multiple processes.

The metric name and properties (labels) are configured by the SkyWalking OAP server based on the data source as well as OAL and MAL. SkyWalking provides the ability to down-sampling time series metrics, and generate different time bucket data (minute, hour, day).

The SkyWalking metric stream is as follows:

Traffic

The metadata of the Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation. Include names, layers, properties, relations between them, etc.

Metric

Name: metric name, configuration from OAL and MAL.
Entity: represents the metrics’ belonging and used for the query. An Entity will contain the following information depending on the Scope： Scope represents the metrics level and in query stage represents the Scope catalog, Scope catalog provides high-dimension classifications for all scopes as a hierarchy structure.

Scope	Entity Info
Service	Service(include layer info)
ServiceInstance	Service, ServiceInstance
Endpoint	Service, Endpoint
ServiceRelation	Service, DestService
ServiceInstanceRelation	ServiceInstance, DestServiceInstance
EndpointRelation	Endpoint, DestEndpoint
Process	Service, ServiceInstance, Process
ProcessRelation	Process, ServiceInstance, DestProcess

Value:

single value: long.
labeled value: text, label1,value1|label2,value2|..., such as L2 aggregation,5000 | L1 aggregation,8000.

TimeBucket: the time is accurate to minute, hour, day.

How to use PromQL Service

Setup

PromQL Service is enabled by default after v9.4.0, so no additional configuration is required. The default ports, for example, can be configured by using OAP environment variables:

restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
restPort: ${SW_PROMQL_REST_PORT:9090}
restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}

Use Prometheus expression

PromQL matches metric through the Prometheus expression. Here is a typical Prometheus metric.

To match the metric, the Prometheus expression is as follows:

In the PromQL Service, these reserved labels would be parsed as the metric name and entity info fields with other labels for the query. The mappings are as follows.

SkyWalking Concepts	Prometheus expression
Metric name	Metric name
Layer	Label
Service	Label
ServiceInstance	Label<service_instance>
Endpoint	Label
…	…

For example, the following expressions are used to match query metrics: service_cpm, service_instance_cpm, endpoint_cpm

service_cpm{service='agent::songs', layer='GENERAL'}
service_instance_cpm{service='agent::songs', service_instance='agent::songs_instance_1', layer='GENERAL'}
endpoint_cpm{service='agent::songs', endpoint='GET:/songs', layer='GENERAL'}

Typical Query Example

At here, we take the SkyWalking Showcase deployment as the playground to demonstrate how to use PromQL for SkyWalking metrics.

The following examples can be used to query the metadata and metrics of services through PromQL Service.

Get metrics names

Query:

http://localhost:9099/api/v1/label/__name__/values

Result:

{
    "status": "success",
    "data": [
        "meter_mysql_instance_qps",
        "service_cpm",
        "envoy_cluster_up_rq_active",
        "instance_jvm_class_loaded_class_count",
        "k8s_cluster_memory_requests",
        "meter_vm_memory_used",
        "meter_apisix_sv_bandwidth_unmatched",
        "meter_vm_memory_total",
        ...
    ]
}

Select a metric and get the labels

Query:

http://localhost:9099/api/v1/labels?match[]=service_cpm

Result:

{
  "status": "success",
  "data": [
    "layer",
    "service",
    "top_n",
    "order"
  ]
}

Get services from a specific layer

Query:

http://127.0.0.1:9099/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

Query specific metric for a service

Query:

http://127.0.0.1:9099/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}

Result:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_cpm",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1679559960,
          "6"
        ]
      }
    ]
  }
}

About the range query and different metrics type for query can refer to the document here.

Build Grafana Dashboard

From the above, we know the mechanism and how to query from PromQL Service, now we can build the Grafana Dashboard for the above service example. Note: All the following configurations are based on Grafana version 9.1.0.

SkyWalking Showcase provides dashboards files such as services of General and Service Mesh layers， we can quickly create a dashboard for the General layer service by importing the dashboard JSON file.

After the Grafana application is deployed, follow the steps below:

Configure Data Source

First, we need to create a data source: In the data source config panel, chose Prometheus and set the URL to the OAP server address, the default port is 9090. Here set the data source name SkyWalking in case there are multiple Prometheus data sources.

Import Dashboard File

Create a dashboard folder named SkyWalking.
Import the dashboard file into Grafana, there are two ways to get the file:
1. From SkyWalking Showcase.
2. Go to SkyWaking Demo: Preview metrics on Grafana, and export it from the General Service dashboard.
Done! Now we can see the dashboard is working, the services are in the drop-down list and the metrics are displayed on the panels.

This is an easy way to build, but we need to know how it works if we want to customize it.

How the dashboard works

Dashboard Settings

Open the Settings-Variables we can see the following variables:

Let’s look at what each variable does:

$DS_SkyWalking

This is a data source ty variable that specifies the Prometheus data source which was defined earlier as SkyWalking.
$layer

This is a constant type because in the ‘General Service’ dashboard, all services belong to the ‘GENERAL’ layer, so they can be used directly in each query Note When you customize other layers, this value must be defined in the Layer mentioned above.
$service

Query type variable, to get all service names under this layer for the drop-down list.

Query expression:
```
label_values(service_traffic{layer='$layer'}, service)
```
The query expression will query HTTP API /api/v1/series for service metadata in $layer and fetch the service name according to the label(service).
$service_instance

Same as the $service is a query variable that is used to select all instances of the service in the drop-down list.

Query expression:
```
label_values(instance_traffic{layer='$layer', service='$service'}, service_instance)
```
The query expression here not only specifies the $layer but also contains the variable $service, which is used to correlate with the services for the drop-down list.
$endpoint

Same as the $service is a query variable that is used to select all endpoints of the service in the drop-down list.

Query expression:
```
label_values(endpoint_traffic{layer='$layer', service='$service', keyword='$endpoint_keyword', limit='$endpoint_limit'}, endpoint)
```
The query expression here specifies the $layer and $service which are used to correlate with the services for the drop-down list. And also accept variables $endpoint_keyword and $endpoint_limit as filtering condition.
$endpoint_keyword

A text type variable that the user can input to filter the return value of $endpoint.
$endpoint_limit

Custom type, which the user can select to limit the maximum number of returned endpoints.

Panel Configurations

There are several typical metrics panels on this dashboard, let’s see how it’s configured.

Common Value Metrics

Select Time series chart panel Service Apdex and click edit.

Query expression
```
service_apdex{service='$service', layer='$layer'} / 10000
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. The calculation Divided by 10000 is used for matching the result units. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.

Labeled Value Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_percentile{service='$service', layer='$layer', labels='0,1,2,3,4', relabels='P50,P75,P90,P95,P99'}
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. Add labels='0,1,2,3,4' filter the result label, and addrelabels='P50,P75,P90,P95,P99' rename the result label. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.
Set Legend to {{label}} for show up.

Sort Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_instance_cpm{parent_service='$service', layer='$layer', top_n='10', order='DES'}
```
The expression is used for query the sore metrics under service, so add labels parent_service and layer for the match. Add top_n='10' and order='DES' filter the result. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set the Calculation --> Latest*.
Set Legend to {{service_instance}} for show up.

Conclusion

In this article, we introduced what is the PromQL Service in SkyWalking and its background. Detailed how to use PromQL Service and the basic concepts related to SkyWalking, and show how to use PromQL Service to build Grafana dashboards for SkyWalking.

In the future, there will be more integrations by leveraging this protocol, such as CI/CD, HPA (scaling), etc.

Zh: 为 Apache SkyWalking 构建 Grafana Dashboard —— 原生 PromQL 支持

Thu, 16 Mar 2023 00:00:00 +0000

背景

Apache SkyWalking 作为分布式系统的应用性能监控工具，提供了对云原生架构下的分布式系统的监控、跟踪、诊断能力。Prometheus 是一个开源系统监控和警报工具包，具有活跃的生态系统。特别是 Prometheus 指标通过导出器和集成得到广泛支持。 PromQL 作为 Prometheus 查询语言，包含一组表达式并公开 HTTP API 以读取指标。

SkyWalking 支持通过 OpenTelemetry 收集器摄取 Prometheus 指标，并通过这些指标的聚合计算提供多种系统监控，例如 Linux 监控和 Kubernetes 监控。SkyWalking 已经为用户提供了原生 UI 和 GraphQL API。但为了提供更广泛的生态整合能力，从 9.4.0 开始，它提供了 PromQL 服务，已经支持 PromQL 的第三方系统或可视化平台（如 Grafana），可以通过它获取指标。SkyWalking 用户在与不同系统集成时将从中受益。

SkyWalking 中的 PromQL 服务是什么？

PromQL 服务是 SkyWalking 原生 GraphQL 查询之上的查询引擎，具有由 Prometheus 表达式提供支持的附加查询阶段计算能力。它可以接受 PromQL HTTP API 请求，解析 Prometheus 表达式，并在 Prometheus 指标和 SkyWalking 指标之间进行转换。

PromQL 服务遵循 PromQL 的所有协议和语法，用户可以像使用 PromQL 一样使用它。由于 SkyWalking 在度量分类、格式、存储等方面与 Prometheus 有根本不同，因此 PromQL 服务不必实现完整的 PromQL 功能。有关详细信息，请参阅文档。

SkyWalking 基本概念

以下是用户使用 PromQL 服务需要了解的一些基本概念和与 Prometheus 的区别： Prometheus 指标指定命名格式和结构，实际指标名称和标签由客户端提供商确定，并存储详细信息。用户使用 PromQL 中的表达式聚合和计算指标。与 Prometheus 不同，SkyWalking 的度量机制是围绕以下具有层次结构的核心概念构建的：

层（Layer）：表示计算机科学中的一个抽象框架，如 Operating System（OS_LINUX 层）、Kubernetes（k8s 层）。该层将是从不同技术检测到的不同服务的所有者。可以在此处

找到所有层定义。
服务：表示一组 / 一组工作负载，它为传入请求提供相同的行为。
服务实例：服务组中的单个工作负载。
端点：传入请求的服务路径。
进程：操作系统进程。在某些场景下，service instance 不是一个进程，比如一个 Kubernetes Pod 可能包含多个进程。

Metric 名称和属性（标签）由 SkyWalking OAP 服务器根据数据源以及 OAL 和 MAL 配置。SkyWalking 提供了对时间序列指标进行下采样（down-sampling），并生成不同时间段数据（分钟、小时、天）的能力。

SkyWalking 指标流如下：

流量

Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation 的元数据。包括名称、层、属性、它们之间的关系等。

指标

名称（Name）：指标名称，来自 OAL 和 MAL 的配置。
实体（Entity）：表示指标的归属，用于查询。一个 Entity 根据 Scope 不同会包含如下信息： Scope 代表指标级别，在查询阶段代表 Scope catalog，Scope catalog 为所有的 scope 提供了高维的分类，层次结构。

Scope	实体信息
Service	服务（包括图层信息）
ServiceInstance	服务、服务实例
Endpoint	服务、端点
ServiceRelation	服务，目标服务
ServiceInstanceRelation	服务实例、目标服务实例
EndpointRelation	端点、目标端点
Process	服务、服务实例、流程
ProcessRelation	进程、服务实例、DestProcess

值：

单值：long
标签值：文本，label1,value1|label2,value2|... ，例如 L2 aggregation,5000 | L1 aggregation,8000

TimeBucket：时间精确到分钟、小时、天

如何使用 PromQL 服务

设置

PromQL 服务在 v9.4.0 之后默认开启，不需要额外配置。例如，可以使用 OAP 环境变量配置默认端口：

restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
restPort: ${SW_PROMQL_REST_PORT:9090}
restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}

使用 Prometheus 表达式

PromQL 通过 Prometheus 表达式匹配指标。这是一个典型的 Prometheus 指标。

为了匹配指标，Prometheus 表达式如下：

在 PromQL 服务中，这些保留的标签将被解析为度量名称和实体信息字段以及用于查询的其他标签。映射如下。

SkyWalking 概念	Prometheus 表达
指标名称	指标名称
层	标签
服务	标签
服务实例	标签 <服务实例>
端点	标签
……	……

例如，以下表达式用于匹配查询指标：service_cpm、service_instance_cpm、endpoint_cpm

service_cpm {service='agent::songs', layer='GENERAL'}
service_instance_cpm {service='agent::songs', service_instance='agent::songs_instance_1', layer='GENERAL'}
endpoint_cpm {service='agent::songs', endpoint='GET:/songs', layer='GENERAL'}

典型查询示例

在这里，我们将 SkyWalking Showcase 部署作为 Playground 来演示如何使用 PromQL 获取 SkyWalking 指标。

以下示例可用于通过 PromQL 服务查询服务的元数据和指标。

获取指标名称

查询：

http://localhost:9099/api/v1/label/__name__/values

结果：

{
    "status": "success",
    "data": [
        "meter_mysql_instance_qps",
        "service_cpm",
        "envoy_cluster_up_rq_active",
        "instance_jvm_class_loaded_class_count",
        "k8s_cluster_memory_requests",
        "meter_vm_memory_used",
        "meter_apisix_sv_bandwidth_unmatched",
        "meter_vm_memory_total",
        ...
    ]
}

选择一个指标并获取标签

查询：

http://localhost:9099/api/v1/labels?match []=service_cpm

结果：

{
  "status": "success",
  "data": [
    "layer",
    "service",
    "top_n",
    "order"
  ]
}

从特定层获取服务

查询：

http://127.0.0.1:9099/api/v1/series?match []=service_traffic {layer='GENERAL'}&start=1677479336&end=1677479636

结果：

{
    "status": "success",
    "data": [
        {"__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

查询服务的特定指标

查询：

http://127.0.0.1:9099/api/v1/query?query=service_cpm {service='agent::songs', layer='GENERAL'}

结果：

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {
          "__name__": "service_cpm",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },"value": [
          1679559960,
          "6"
        ]
      }
    ]
  }
}

关于range query和不同的metrics type for query 可以参考这里的文档。

构建 Grafana Dashboard

从上面我们知道了 PromQL 服务的机制和查询方式，现在我们可以为上面的服务示例构建 Grafana Dashboard。注：以下所有配置均基于 Grafana 9.1.0 版本。

SkyWalking Showcase 提供了 General Service 和 Service Mesh 层等 Dashboard 文件，我们可以通过导入 Dashboard JSON 文件快速为层服务创建 Dashboard。

部署 Grafana 应用程序后，请按照以下步骤操作：

配置数据源

首先，我们需要创建一个数据源：在数据源配置面板中，选择 Prometheus 并设置 URL 为 OAP 服务器地址，默认端口为 9090。 SkyWalking 如果有多个 Prometheus 数据源，请在此处设置数据源名称。

导入 Dashboard 文件

创建一个名为 SkyWalking 的 Dashboard 文件夹。
将 Dashboard 文件导入到 Grafana 中，有两种获取文件的方式：
1. 来自 SkyWalking Showcase
2. 转到 SkyWaking Demo：在 Grafana 上预览指标，并将其从 General Service Dashboard 导出。
完毕！现在我们可以看到 Dashboard 正在运行，服务位于下拉列表中，指标显示在面板上。

这是一种简单的构建方式，但是如果我们想要自定义它，我们需要知道它是如何工作的。

Dashboard 的工作原理

Dashboard 设置

打开 Settings-Variables 我们可以看到如下变量：

让我们看看每个变量的作用：

$DS_SkyWalking

这是一个数据源 ty 变量，它指定了之前定义为 SkyWalking 的 Prometheus 数据源。
$layer

这是一个常量类型，因为在 ‘General Service’ Dashboard 中，所有服务都属于 ‘GENERAL’ 层，因此可以在每个查询中直接使用它们。注意，当您自定义其他层时，必须在 Layer 上面定义该值。
$service

查询类型变量，为下拉列表获取该层下的所有服务名称。

查询表达式：
```
label_values (service_traffic {layer='$layer'}, service)
```
查询表达式将查询 HTTP API /api/v1/series，以获取 $layer 中服务元数据，并根据标签（服务）提取服务名称。
$service_instance

与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有实例。

查询表达式：
```
label_values (instance_traffic {layer='$layer', service='$service'}, service_instance)
```
这里的查询表达式不仅指定了 $layer 还包含 $service 变量，用于关联下拉列表的服务。
$endpoint

与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有端点。

查询表达式：
```
label_values (endpoint_traffic {layer='$layer', service='$service', keyword='$endpoint_keyword', limit='$endpoint_limit'}, endpoint)
```
此处的查询表达式指定 $layer 和 $service 用于与下拉列表的服务相关联的。并且还接受 $endpoint_keyword 和 $endpoint_limit 变量作为过滤条件。
$endpoint_keyword

一个文本类型的变量，用户可以输入它来过滤 $endpoint 的返回值。
$endpoint_limit

自定义类型，用户可以选择它以限制返回端点的最大数量。

Dashboard 配置

这个 Dashboard 上有几个典型的指标面板，让我们看看它是如何配置的。

普通值指标

选择 Time series chart 面板 Service Apdex 并单击 edit。

查询表达式
```
service_apdex {service='$service', layer='$layer'} / 10000
```
指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。该计算 Divided by 10000 用于匹配结果单位。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Connect null values --> AlwaysShow points --> Always，因为当查询间隔大于 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。

标签值指标

选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。

查询表达式
```
service_percentile {service='$service', layer='$layer', labels='0,1,2,3,4', relabels='P50,P75,P90,P95,P99'}
```
指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。添加 labels='0,1,2,3,4' 过滤结果标签，并添加 relabels='P50,P75,P90,P95,P99' 重命名结果标签。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Connect null values --> AlwaysShow points --> Always，因为当查询间隔 > 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。
设置 Legend 为 {{label}} 来展示。

排序指标

选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。

查询表达式
```
service_instance_cpm {parent_service='$service', layer='$layer', top_n='10', order='DES'}
```
该表达式用于查询服务下的排序指标，因此添加标签 parent_service 和 layer 进行匹配。添加 top_n='10' 和 order='DES' 过滤结果。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Calculation --> Latest*。
设置 Legend 为 {{service_instance}} 来展示。

结论

在这篇文章中，我们介绍了 SkyWalking 中的 PromQL 服务是什么以及它的背景。详细介绍了 PromQL 服务的使用方法和 SkyWalking 相关的基本概念，展示了如何使用 PromQL 服务为 SkyWalking 构建 Grafana Dashboard。

未来，将会有更多的集成利用这个协议，比如 CI/CD、HPA（缩放）等。