Skip to content

feat(metrics): support dubbo-go metrics in shared Grafana dashboards - #1591

Open
dodjdnh wants to merge 10 commits into
apache:developfrom
dodjdnh:feat/metric-compatibility
Open

dodjdnh wants to merge 10 commits into
apache:developfrom
dodjdnh:feat/metric-compatibility

Conversation

@dodjdnh

@dodjdnh dodjdnh commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adapt Dubbo Admin's existing shared Metric dashboards to consume available dubbo-go RPC metrics while retaining a query path for Java RPC series that do not carry a native IP label. Changes include consumer metrics in service and method discovery, bare-IP instance selection, corrected processing metric names, and removal of the fixture-specific application histogram filter.

This is Admin-side metric compatibility work, not a new SDK exporter, metric collection pipeline, or Trace → Metric feature.

Motivation

The existing Instance Metric flow constructs a guessed IP:QoSPort and uses it to filter Prometheus scrape instances. This does not reliably identify an application instance when the exporter uses a different port or when Java and Go series expose different labels. Provider-only variable discovery also omits consumer-only RPC interfaces and methods.

The shared dashboards should use the selected application's actual metric identities and available labels, rather than assume one SDK or exporter endpoint.

Changes

1. Discover services and methods from both RPC sides

Use provider or consumer request series for the Service dashboard's interface and method variables, updating both variable definitions and query objects:

label_values({__name__=~"dubbo_(provider|consumer)_requests_total"},interface)
label_values({__name__=~"dubbo_(provider|consumer)_requests_total",interface="$service"},method)

This makes consumer-only interfaces and methods selectable; it does not invent provider data for a consumer-only application.

2. Isolate Instance Metric by native IP

  • Pass the selected instance's application name and bare IP to Grafana. Do not append the configured or default QoS port.
  • Keep resource lookup based on the existing Admin resource manager; the Metric implementation and its tests do not require an Instance Trace helper or API.
  • For metrics with ip, select the requested IP directly.
  • For metrics without ip, associate the selected application/IP with its scrape target through dubbo_application_info_total, then match RPC series on (application_name, instance).
  • Apply this selection pattern to the Instance dashboard's existing metric queries and use native ip values in its instance variable.

Representative request metric query selector:

dubbo_provider_requests_total{application_name="$application",ip="$instance"}
or (
  dubbo_provider_requests_total{application_name="$application"}
  and on(application_name,instance)
  dubbo_application_info_total{application_name="$application",ip="$instance"}
)

Prometheus instance is an internal association key, not the product instance identity. If both paths select an identical series, or retains it once. The metadata path requires application-info and RPC series to share consistent application and scrape labels.

3. Correct existing dashboard queries

  • Query dubbo_provider_requests_processing_total and dubbo_consumer_requests_processing_total in the Application, Instance, and Service dashboards. These series represent in-flight requests/concurrency; the name suffix does not turn them into request-rate queries.
  • Remove method="getItem" from the Application's existing provider histogram query so it is not restricted to a sample method.
  • Do not add a Histogram capability/status panel, a new bucket family, or a Go-specific histogram implementation.

4. Add focused tests and deployment guidance

  • Add a self-contained Metric test file covering bare-IP variables and URLs for QoS ports 0 and 22222, existing query-parameter preservation, missing configuration, missing instances, and lookup failures.
  • Document native instance identity, the two query paths, Go Metrics enablement and exporter configuration, and the need for matching Prometheus scrape discovery.
  • Explain why the existing QoS ServiceMonitor does not automatically discover a Go exporter on 9090/metrics.
  • Distinguish optional metric families, compatibility targets, historical runtime evidence, and complete product acceptance.

Verification

Unit tests, static checks, and locally reproducible upstream CI checks on the current independent commit series are all PASS. Runtime verification, the complete Go browser journey, and Java regression checks were manually completed and passed by the submitter; no browser automation was used. Local checks do not replace the GitHub Actions results after PR submission.

Area Check Result
Unit Bare-IP variables / URLs with and without QoS port; configuration and lookup error handling PASS on the independent series
Unit / package regression go test ./pkg/console/service ./pkg/console/handler -count=1 at each of the five independent commit checkpoints PASS at all five checkpoints
Local CI / license License header checks using License Eye and the repository's .licenserc.yaml PASS
Local CI / frontend Node 18.20.8 and Yarn 1.22.22; install locked dependencies, then run yarn build PASS
Local CI / formatting Go 1.24.11; check LF source from Git, with no source differences after frontend build and go fmt ./... PASS
Local CI / full tests Ubuntu 24.04, Go 1.24.11, GCC 13.3, CGO enabled; run upstream make test PASS; exit code 0, atomic coverage file generated, repository-wide statement coverage 13.9%
Local CI / RISC-V GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 go build ./... PASS; cross-compilation check, not a RISC-V runtime test
Static YAML parsing and embedded dashboard JSON parsing PASS
Static Consumer service/method variables, processing metric names, and application histogram method filter PASS
Static All 84 Instance query expressions using $instance contain the direct-IP / metadata selection pattern; instance variable uses ip PASS; structural checks, not PromQL execution
Static / scope Independent series contains only backend Metric changes, Metric tests, dashboard definitions, and compatibility documentation PASS; no unmerged Trace / Log changes required
Static / regression Parsed dashboards outside Application / Instance / Service Metrics match upstream; existing upstream observability test file is unchanged PASS
Static git diff --check b9ee1b7 HEAD PASS
Runtime Deploy the current independent branch and execute real Java / Go queries, direct-IP and metadata paths, consumer-only discovery, and two-instance isolation/deduplication PASS; submitter's manual acceptance
Browser Registered Go application → Admin Application / Instance / Service views → Grafana; real RPC data and A/B isolation PASS; submitter's manual acceptance
Runtime regression Existing Java Metric views after deploying the current independent branch PASS; submitter's manual acceptance

Earlier runtime checks recorded in docs/observability-metric-compatibility.md used Java 3.2.16, Go 3.3.2, Grafana 12.1.0, and Prometheus data. They observed a metadata path for two Java instances and direct-IP queries for Go provider/consumer data. These historical records do not automatically replace acceptance for the current independent branch; the runtime and browser PASS results in the table come from the submitter's separately confirmed manual acceptance.

No Codecov upload or release workflow was run locally; the coverage file was generated and retained. Frontend build and RISC-V cross-compilation were performed in a disposable Windows copy, and full unit tests were performed in WSL Ubuntu, all against the same HEAD cdcb6c6 without modifying product source.

Reproducible backend checks:

go test ./pkg/console/service -run '^TestInstanceMetricDashboard' -count=1
go test ./pkg/console/service ./pkg/console/handler -count=1
make test
git diff --check b9ee1b7 HEAD

Review Focus

  1. Whether application plus bare IP is the correct Instance Metric contract, and whether the backend parameter change and dashboard filters need to be deployed together.
  2. Whether the metadata association correctly isolates scrape targets when RPC series lack ip, especially with multiple instances and consistent relabeling. Review the complete metric label sets rather than assume IP:QoSPort.
  3. Whether combining direct-IP and metadata paths preserves isolation and avoids duplicating an identical series. Structural validation and the submitter's runtime acceptance have passed; review should still focus on these query semantics.
  4. Whether consumer-only variable discovery and processing metric names match actual exported Java / Go series, without promising metric families that the SDK does not export.
  5. Whether the documentation clearly separates query compatibility from exporter enablement, scrape configuration, optional capabilities, and complete product acceptance.

Compatibility / Impact

  • The Instance Metric variable changes from IP:QoSPort to bare IP. Deploy the backend and shared Instance dashboard together; custom dashboards that assume this parameter is a scrape address must adapt to the parameter contract.
  • Application and Service identity variables retain their existing meaning. Existing request and duration metric families are reused; this PR does not align their cross-language semantics or units.
  • Runtime capabilities remain limited to what the application actually exports and Prometheus scrapes. Removing the method filter does not make Go export provider histogram buckets; no universal parity for Java-only metric families is claimed.
  • The documented compatibility targets are Java 3.2.0+ and Go 3.1.1+. Source inspection and selected Java 3.2.16 / Go 3.3.2 runtime checks are not an all-version end-to-end guarantee.
  • The Go exporter must be enabled and exposed with a matching scrape target. This PR does not modify ServiceMonitor, scrape configuration, datasource provisioning, or automatic deployment workflows.
  • The independent branch does not modify Trace / Log APIs, Collector pipelines, Jaeger datasource settings, or existing Trace dashboards. Runtime UI regression cannot be ruled out merely because those files are unchanged.
  • Updating an already-running Grafana installation still requires the normal dashboard provisioning reload/deployment procedure.

Affected areas, retaining the category names from the repository PR template:

  • Docs
  • Installation
  • User Experience
  • Dubboctl
  • Console
  • Core Component

Installation impact here is limited to deploying the updated dashboard and documenting scrape prerequisites; no automatic collection infrastructure is added.

Scope

Included

  • Existing shared Java / Go Metric query compatibility, consumer variable discovery, and instance selection.
  • Processing / Histogram query corrections, focused backend tests, and integration documentation.

Not Included

  • SDK or exporter changes, new metrics, Go error classification or Histogram support, and cross-language duration/error normalization.
  • A new Histogram capability panel, a complete dashboard redesign, or guaranteed support for Java-only metric families in Go.
  • Automatic Go registration/deployment, Prometheus scrape provisioning, or runtime environment changes.
  • Log observability, Trace → Log, Trace → Metric, OTel transforms, Tempo, or spanmetrics.

Related Issue

Fixes #1590

Related to #1524. This PR closes only the dedicated Metric compatibility issue, not the overall OSPP issue or the separate Instance Trace, Log, or Trace → Metric issues.

…ata bridge

Use the native IP instead of a guessed QoS scrape port. Keep backend
and dashboard parameter contracts together with instance documentation.

Resolve Metric instances directly through the existing resource manager
API, without relying on the instance Trace helper. Add a self-contained
Metric test file for bare-IP variables, URLs, and lookup errors. In the
combined branch, keep legacy Trace tests focused only on Trace behavior;
the standalone Metric PR does not require those Trace tests or helpers.
Query provider and consumer requests_processing_total in the Application,
Instance and Service dashboards to match the exported metric names.
…ogram

Remove the sample getItem method restriction so the application histogram
query covers the selected application rather than one fixture method.
Retain the original adjacent line endings to preserve the aggregate diff.
…ance

Describe shared identities, optional metric capabilities, Go exporter
enablement and Prometheus scraping requirements. Combine compatibility
and acceptance guidance while keeping the essential instance identity
contract in the instance feature commit. Distinguish runtime query checks
from unclaimed discovery and browser end-to-end acceptance.

@robocanic robocanic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

整体 LGTM,感谢贡献。CI 已全部通过,几点小意见供参考:

  1. Instance dashboard 过滤不完整:grafana-dashboardDefinitions.yaml 中 Instance metrics 面板里的 dubbo_provider_requests_failed_network_total 和 dubbo_provider_requests_failed_codec_total(约 5932、5945 行,带 or vector(0))只按 application_name 过滤,没有应用 ip 选择模式,因此展示的是整个应用的数据而不是所选实例。这是历史遗留问题,但 PR 描述称已处理所有 Instance 查询,建议一并修复或在说明中注明。
  2. 同一 IP 多个实例:元数据路径按 ip 关联抓取目标 instance,同一主机上端口不同的多个 Java 实例会同时匹配,选中一个 IP 时可能混入其他实例的序列。可以接受,建议在文档中补充一句说明。
  3. 文档:docs/observability-metric-compatibility.md 中出现的 “D009” 是内部跟踪编号,上游读者无法理解,建议删除。
  4. 验证方式:PromQL 仅做了结构性检查,没有自动化测试,主要依赖手动运行验证,请确保 Java 与 Go 两条路径都有实际截图或结果记录。
  5. 发布顺序:var-instance 参数约定已从 IP:port 变为纯 IP,后端与 Grafana dashboard 需要一起部署,建议在 release notes 中说明。

@dodjdnh

dodjdnh commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

好的

整体 LGTM,感谢贡献。CI 已全部通过,几点小意见供参考:

  1. Instance dashboard 过滤不完整:grafana-dashboardDefinitions.yaml 中 Instance metrics 面板里的 dubbo_provider_requests_failed_network_total 和 dubbo_provider_requests_failed_codec_total(约 5932、5945 行,带 or vector(0))只按 application_name 过滤,没有应用 ip 选择模式,因此展示的是整个应用的数据而不是所选实例。这是历史遗留问题,但 PR 描述称已处理所有 Instance 查询,建议一并修复或在说明中注明。
  2. 同一 IP 多个实例:元数据路径按 ip 关联抓取目标 instance,同一主机上端口不同的多个 Java 实例会同时匹配,选中一个 IP 时可能混入其他实例的序列。可以接受,建议在文档中补充一句说明。
  3. 文档:docs/observability-metric-compatibility.md 中出现的 “D009” 是内部跟踪编号,上游读者无法理解,建议删除。
  4. 验证方式:PromQL 仅做了结构性检查,没有自动化测试,主要依赖手动运行验证,请确保 Java 与 Go 两条路径都有实际截图或结果记录。
  5. 发布顺序:var-instance 参数约定已从 IP:port 变为纯 IP,后端与 Grafana dashboard 需要一起部署,建议在 release notes 中说明。

好的

Preserve the upstream Instance Trace resource helper while keeping Metric dashboard variables on bare IPs. Adapt the upstream Metric/Trace regression test to the shared bare-IP contract.
Apply native-IP and application-info metadata selection to provider network and codec failure summaries, retaining the zero fallback. Add a regression check that parses provisioned dashboards and inspects all Instance query expressions, including queries that omit the instance variable entirely.
Document same-application shared-IP limits and the coordinated backend/dashboard rollout. Remove the internal tracking identifier and distinguish recorded Java/Go query observations from the new structural regression check and runtime acceptance evidence still needed for the error summaries.
@dodjdnh

dodjdnh commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

e2e验证结果

1.java metadata联表关联查询
a4defeb4cd61cfea9bff8ed8bab65b2a
2.Java A/B实例隔离
c4311b1d5fb61aacb706cbfc369df838
3.go原生ip查询
7e318017dfde0eadae8e87c6e04f92fa
4.序列化错误java查询语句(network错误同理)
1e57780dfc94df8eecd9ffd226059a8b

@dodjdnh

dodjdnh commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

dubbo-go sdk目前无法正确有效分类codec,network错误,我创建apache/dubbo-go#3774 这个issue用于解决这个问题,不影响admin侧的查询逻辑

@Alanxtl

Alanxtl commented Oct 22, 2026

Copy link
Copy Markdown
Member

cool work

@dodjdnh

dodjdnh commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

E2E测试
1.java metadata关联查询
a4defeb4cd61cfea9bff8ed8bab65b2a
2.Java A/B实例隔离
c4311b1d5fb61aacb706cbfc369df838

@dodjdnh

dodjdnh commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

3.go原生ip查询
7e318017dfde0eadae8e87c6e04f92fa
4.go A/B实例隔离
a70c4051de8de0d64d4c2f81c7a064fd

@dodjdnh

dodjdnh commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

5.java确认实例隔离
37ebe227c2d69496d40cf953e3c1bd42
6.序列化错误java查询语句(network错误同理,go同理)
1e57780dfc94df8eecd9ffd226059a8b

@AlexStocks AlexStocks left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:可以合入,但这是有意的 Breaking 变更,需要发布说明

变更实质

GetInstanceDashboardVariables 与 GetInstanceTraceDashboardVariables 的 var-instance 从 IP:QoSPort(QoS 端口缺省时用 22222)改为裸 IP,以匹配 Prometheus 的 ip 标签。同时修正了多处指标名拼写(dubbo_*_tota → dubbo_*_total、requests_processing → requests_processing_total),这部分是净收益。

已核对

  • Instance Metric 仪表盘的 instance 变量查询已改为 label_values(dubbo_application_info_total{application_name="$application"}, ip),与新后端返回值一致。
  • 文档中提到的另一条路径(RPC 序列缺 ip 时,用 dubbo_application_info_total 按 (application_name, instance) 关联)在查询里以 A or (B and on(application_name,instance) C) 的形式实现,语义正确:or 保证两条路径命中同一序列时不重复。
  • Instance traces 仪表盘的 instance 是 custom 类型、query 为空,且整块 dashboard 的查询没有引用 $instance,因此改成裸 IP 不会影响 Jaeger 查询。
  • 其余 12 处 "name": "instance" 属于导入的 Kubernetes / node-exporter 仪表盘,仍然是 label_values(up{...}, instance),本次未被误改,正确。

P3 · 混合版本部署没有运行时保护

docs/observability-metric-compatibility.md 已明确写出「旧后端 + 新仪表盘」或「新后端 + 旧仪表盘」会得到空结果。但代码里没有任何版本标记或降级提示,用户只能看到空面板。考虑到这是 Grafana 侧配置,后端能做的有限;建议至少在发布说明里把它列为升级步骤(先更新 Admin,再 reload provisioned dashboards),并说明 var-instance 是用户可见的 URL 契约变化。

P3 · var-instance 的语义变化影响自定义看板

任何保存了 var-instance=10.42.1.10:22222 的外部链接都会失效。文档已提示,建议同时在 CHANGELOG / release note 中体现,而不仅放在新增的文档里。

@dodjdnh

dodjdnh commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

结论:可以合入,但这是有意的 Breaking 变更,需要发布说明

变更实质

GetInstanceDashboardVariables 与 GetInstanceTraceDashboardVariables 的 var-instance 从 IP:QoSPort(QoS 端口缺省时用 22222)改为裸 IP,以匹配 Prometheus 的 ip 标签。同时修正了多处指标名拼写(dubbo_*_tota → dubbo_*_total、requests_processing → requests_processing_total),这部分是净收益。

已核对

  • Instance Metric 仪表盘的 instance 变量查询已改为 label_values(dubbo_application_info_total{application_name="$application"}, ip),与新后端返回值一致。
  • 文档中提到的另一条路径(RPC 序列缺 ip 时,用 dubbo_application_info_total 按 (application_name, instance) 关联)在查询里以 A or (B and on(application_name,instance) C) 的形式实现,语义正确:or 保证两条路径命中同一序列时不重复。
  • Instance traces 仪表盘的 instance 是 custom 类型、query 为空,且整块 dashboard 的查询没有引用 $instance,因此改成裸 IP 不会影响 Jaeger 查询。
  • 其余 12 处 "name": "instance" 属于导入的 Kubernetes / node-exporter 仪表盘,仍然是 label_values(up{...}, instance),本次未被误改,正确。

P3 · 混合版本部署没有运行时保护

docs/observability-metric-compatibility.md 已明确写出「旧后端 + 新仪表盘」或「新后端 + 旧仪表盘」会得到空结果。但代码里没有任何版本标记或降级提示,用户只能看到空面板。考虑到这是 Grafana 侧配置,后端能做的有限;建议至少在发布说明里把它列为升级步骤(先更新 Admin,再 reload provisioned dashboards),并说明 var-instance 是用户可见的 URL 契约变化。

P3 · var-instance 的语义变化影响自定义看板

任何保存了 var-instance=10.42.1.10:22222 的外部链接都会失效。文档已提示,建议同时在 CHANGELOG / release note 中体现,而不仅放在新增的文档里。

明白

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support dubbo-go metrics in Dubbo Admin Grafana dashboards

4 participants