[telemetry-subscribers] add RUST_LOG_TAILS for per-target log file tails

[evan-wall-mysten]

Description

Adds RUST_LOG_TAILS to telemetry-subscribers (with a with_log_tail builder / log_tails config) that additionally writes individual tracing targets to dedicated files, on top of the normal log output. A "tail" is an extra file sink: matching events are duplicated to the file, not moved off the console/JSON stream.

Syntax — comma-separated target=file entries (mirroring RUST_LOG's target=level, but assigning a file instead of a level):

• =file is a daily-rolling file (like RUST_LOG_FILE) and is mandatory — RUST_LOG_TAILS only adds file sinks, it never routes to the stream.
• The output format is inferred from the file extension: .jsonl / .ndjson (case-insensitive) → newline-delimited JSON; everything else (including .json and .log) → text.

Entry	Result
graphql_request=/var/log/sui/gql.log	graphql_request → text in that file
graphql_request=/var/log/sui/gql.jsonl	graphql_request → NDJSON in that file (replay corpus)
a=/x.jsonl,b=/y.log	two tails, each to its own file

Behavior:

• Tails are additional sinks — the events still appear on the normal stream as well, including when RUST_LOG_JSON (global JSON output) is set.
• Each tail honors RUST_LOG's per-target level (its layer filter is the target restriction ANDed with an EnvFilter): e.g. with RUST_LOG=warn a bar tail captures only warn+, while RUST_LOG=warn,bar=debug also captures bar debug. Targets are matched by prefix (like RUST_LOG).
• A malformed entry (missing target or =file) or a file whose directory cannot be created hard-fails init() rather than silently degrading.

Example — capture graphql_request as a JSON replay corpus in its own file while normal logs stay human-readable:

RUST_LOG="info,graphql_request=trace" RUST_LOG_TAILS="graphql_request=/var/log/sui/graphql_request.jsonl"

Test plan

Added unit tests (entry parsing, extension→format inference, malformed-entry panics, builder). cargo test -p telemetry-subscribers, cargo xclippy, and cargo fmt -- --check all pass.

---

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

• Protocol:
• Nodes (Validators and Full nodes): New optional RUST_LOG_TAILS env var additionally writes individual tracing targets to dedicated files (format inferred from the file extension — .jsonl/.ndjson for JSON, otherwise text), leaving normal log output unchanged. Backward-compatible — unset by default, so log output is unchanged and no action is required.
• gRPC:
• JSON-RPC:
• GraphQL:
• CLI:
• Rust SDK:
• Indexing Framework: The same RUST_LOG_TAILS env var is available to indexer binaries (e.g. sui-indexer-alt-graphql). Set it alongside RUST_LOG (e.g. RUST_LOG="info,graphql_request=trace" with RUST_LOG_TAILS="graphql_request=/var/log/sui/graphql_request.jsonl") to capture that target as NDJSON for replay while normal logs stay readable. Opt-in; no action required otherwise.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
sui-docs	Ready	Preview, Comment	Jun 29, 2026 3:34pm

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
multisig-toolkit	Ignored	Preview	Jun 29, 2026 3:34pm
sui-kiosk	Ignored	Preview	Jun 29, 2026 3:34pm

[amnn]

This is currently feeling quite ad-hoc to me, especially the way it requires json log output to be disabled in order for json log target to be applicable, and the fact that enabling this causes logs to disappear from the normal format (coupled with the fact that targets are prefix matched, which means you may end up inadvertently capturing logs you didn't mean to).

Instead, can we offer some kind of orthogonal control surface for this? I.e. I'd like to be able to emit logs matching certain targets (separate from the normal filter) to a file (separate from log_file) in addition to whatever normal logging configuration there is.

[evan-wall-mysten]

This is currently feeling quite ad-hoc to me, especially the way it requires json log output to be disabled in order for json log target to be applicable, and the fact that enabling this causes logs to disappear from the normal format (coupled with the fact that targets are prefix matched, which means you may end up inadvertently capturing logs you didn't mean to).

Instead, can we offer some kind of orthogonal control surface for this? I.e. I'd like to be able to emit logs matching certain targets (separate from the normal filter) to a file (separate from log_file) in addition to whatever normal logging configuration there is.

I changed the env var to be less focused on JSON by allowing it to set either the format, the file, or both per target.

Should the logs in this new file be accessed through Grafana like the existing logs or copied out of the k8s pod? I was testing doing it either way in https://github.com/MystenLabs/sui-operations/pull/8197/, but logging to a new file would require either ingesting that file into Grafana (would be a separate change) or getting it from the k8s pod.

[amnn]

The telemetry subscriber is already pretty complex, with lots of flags interacting with each other. If we can avoid introducing new complexity (interactions between flags or multiple ways to do things) then we should.

In that vein, I'd suggest introducing an environment variable for tailing logs, filtered by target, to a file, optionally modifying the format. Key differences from what we have here:

• Just duplicate the logs (don't eliminate logs from the normal stream because they are being tailed elsewhere). Given filters are prefix matches, it can be difficult to predict what they pick up and it's incorrect to assume that the audience for the human-readable logs and the audience for the tailed logs are the same (in a way where they would care that the logs are duplicated).
• The file/path becomes mandatory -- we are always writing to a file, never to the default stream -- but the format is optional. This reduces overlap with other features (like json_log_output, so don't ignore this config if that flag is set), and also the potential to duplicate logs in an obvious way.
• I would also suggest parsing the path as a PathBuf rather than a String, and keying the format on the extension of that path (e.g. if it's .jsonl then you assume json formatting), to avoid having to invent new syntax.
• Name it something other than RUST_LOG_TARGETS -- a "target" is the label for the log and lots of configs we care about accept a target in this file. Perhaps RUST_LOG_TAILS or something?

[evan-wall-mysten]

The telemetry subscriber is already pretty complex, with lots of flags interacting with each other. If we can avoid introducing new complexity (interactions between flags or multiple ways to do things) then we should.

In that vein, I'd suggest introducing an environment variable for tailing logs, filtered by target, to a file, optionally modifying the format. Key differences from what we have here:

• Just duplicate the logs (don't eliminate logs from the normal stream because they are being tailed elsewhere). Given filters are prefix matches, it can be difficult to predict what they pick up and it's incorrect to assume that the audience for the human-readable logs and the audience for the tailed logs are the same (in a way where they would care that the logs are duplicated).
• The file/path becomes mandatory -- we are always writing to a file, never to the default stream -- but the format is optional. This reduces overlap with other features (like json_log_output, so don't ignore this config if that flag is set), and also the potential to duplicate logs in an obvious way.
• I would also suggest parsing the path as a PathBuf rather than a String, and keying the format on the extension of that path (e.g. if it's .jsonl then you assume json formatting), to avoid having to invent new syntax.
• Name it something other than RUST_LOG_TARGETS -- a "target" is the label for the log and lots of configs we care about accept a target in this file. Perhaps RUST_LOG_TAILS or something?

Updated the PR

The telemetry subscriber is already pretty complex, with lots of flags interacting with each other. If we can avoid introducing new complexity (interactions between flags or multiple ways to do things) then we should.

In that vein, I'd suggest introducing an environment variable for tailing logs, filtered by target, to a file, optionally modifying the format. Key differences from what we have here:

• Just duplicate the logs (don't eliminate logs from the normal stream because they are being tailed elsewhere). Given filters are prefix matches, it can be difficult to predict what they pick up and it's incorrect to assume that the audience for the human-readable logs and the audience for the tailed logs are the same (in a way where they would care that the logs are duplicated).
• The file/path becomes mandatory -- we are always writing to a file, never to the default stream -- but the format is optional. This reduces overlap with other features (like json_log_output, so don't ignore this config if that flag is set), and also the potential to duplicate logs in an obvious way.
• I would also suggest parsing the path as a PathBuf rather than a String, and keying the format on the extension of that path (e.g. if it's .jsonl then you assume json formatting), to avoid having to invent new syntax.
• Name it something other than RUST_LOG_TARGETS -- a "target" is the label for the log and lots of configs we care about accept a target in this file. Perhaps RUST_LOG_TAILS or something?

Updated the PR with these changes which ended up simplifying the code a bit.

[amnn]

Looks good, thanks @evan-wall-mysten !