fix(sdk): spawn periodic reader worker at reader build time - #3605
fix(sdk): spawn periodic reader worker at reader build time#3605karan-vk wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3605 +/- ##
=======================================
+ Coverage 82.3% 82.4% +0.1%
=======================================
Files 130 130
Lines 28697 28737 +40
=======================================
+ Hits 23626 23707 +81
+ Misses 5071 5030 -41 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I think #3413 should help out here also and we should aim to get that in soon to clear up all of these weird unexpected deadlocks issues at once |
The async-runtime PeriodicReader stored its spawn closure and only ran it inside register_pipeline, so the worker task landed on whatever tokio runtime was ambient when the meter provider was built, not when the reader taking the runtime argument was built. Building the reader and provider in different contexts could deadlock or panic. Spawn the worker eagerly in build(), matching the async-runtime BatchSpanProcessor and BatchLogProcessor, and reduce ProducerOrWorker to a plain optional producer. Fixes open-telemetry#3601
de07fdf to
ca73a74
Compare
|
Took a proper look through #3413 and I think it's a really nice piece of work. Consolidating For this PR specifically, I think the two compose rather than overlap: #3413 fixes how and where the spawn executes and how shutdown waits, while this one fixes when the worker is captured (at reader build, where the A nice side effect of landing this one first is that the #3413 rebase of this file gets simpler: with the spawn in |
There was a problem hiding this comment.
Pull request overview
This PR fixes periodic_reader_with_async_runtime::PeriodicReader spawning its background worker during meter provider construction (via register_pipeline) instead of at reader build time, which could capture the wrong ambient Tokio runtime and lead to deadlocks or panics when reader/provider are built in different runtime contexts.
Changes:
- Spawn the periodic reader worker eagerly in
PeriodicReaderBuilder::build(), so it runs on the runtime that’s ambient at reader construction time. - Simplify internal state by removing the deferred “spawn worker” closure and storing only an optional producer (
sdk_producer) for collection. - Add a regression test covering the “build reader inside runtime, build provider outside runtime” scenario and update the SDK changelog.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| opentelemetry-sdk/src/metrics/periodic_reader_with_async_runtime.rs | Eagerly spawns the periodic worker at reader build time, simplifies registration state, and adds a regression test for the captured-runtime behavior. |
| opentelemetry-sdk/CHANGELOG.md | Documents the behavior change for PeriodicReader worker spawning timing and its implications. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ## vNext | ||
|
|
||
|
|
||
| - `periodic_reader_with_async_runtime::PeriodicReader` now spawns its background |
|
Hey @karan-vk , thanks for looking more into this! @cijothomas would be great to merge this when you have a moment! |
Fixes #3601
Changes
periodic_reader_with_async_runtime::PeriodicReaderstored its spawn closure and only ran it insideregister_pipeline, so the worker task landed on whatever tokio runtime was ambient when the meter provider was built rather than when the reader, which takes theruntimeargument, was built. Building the reader and the provider in different contexts could deadlock or panic.The worker is now spawned eagerly in
build(), matching the async-runtimeBatchSpanProcessorandBatchLogProcessor, andProducerOrWorkeris reduced to a plainOptionproducer. Added a regression test that builds the reader inside a runtime and the provider outside of one; it panics without this fix.Behavior note: a reader that is built but never attached to a provider now runs its interval ticker and logs collect errors, where before the worker was never spawned. This matches the existing behavior of the async-runtime batch processors.
Merge requirement checklist
CHANGELOG.mdfiles updated for non-trivial, user-facing changes