Columnar encoding for the OpenTelemetry protocol

[lquerel]

Ongoing experiments, research, and draft specification for a multi-variate columnar encoding of OTLP.

At the request of @jmacd

[reyang]

LGTM with some minor wording suggestions.

[reyang]

@jsuereth @zeroshade @mbrobbel @itaykat could you take another look and see if there is anything blocking? If not, I suggest that we give the PR few more days and merge it by end of this week (6/30/2023).

[jgehrcke]

🚀

[dyladan]

The GC voted to accept the donation (open-telemetry/community#1332) on July 6 2023

[infogulch]

Did we ever get results back on the Phase 2 multivariate time series tests?

For multivariate time series, OTel Arrow is 3 to 7 times better in terms of bandwidth reduction while having an end-to-end speed (including conversion to/from OTLP) similar to the univariate time series scenario phase 1. Phase 2 has been yet estimated but similar results are expected.

[lquerel]

@infogulch - The current implementation (Phase 1) includes optimizations for multivariate metrics, which are presented as a set of univariate metrics. We've observed a compression ratio enhancement ranging from 3x to 7x with this setup. Phase 2, which is yet to start, will deepen this support for multivariate metrics, integrating them not just at the protocol layer but also within the client SDK API. I aim to release a beta version of this extended client SDK by the end of the year.
I am, in any case, interested in learning more about your use case if possible, to refine my response if necessary.

[infogulch]

I see, thank you for clarifying. I didn't understand that the phase 2 tests are still in process.

Personally I've been watching otel for a while but am not currently using it; my comment came from a fleeting curiosity about the intersection of telemetry and column-oriented storage/processing formats from the recent hn thread about otel, where someone kindly forwarded me here. From this report it seems they work together quite nicely! (I love when incidental ideas are confirmed by someone that put in the work to test it 6 months ago!)

Thank you for your attention, sorry for the noise, and good luck integrating columnar layouts into otel!