Phase I of the engagement was designed to help the Envoy Team reduce the number of open bugs identified, reduce noise in the existing fuzzers (so the Envoy team spends less time triaging non-issues), and improve fuzzing coverage where possible. OSTIF sourced four senior security experts from X41 D-sec to complete the engagement.

The result of Phase I was the fixing of 68 security bugs. Furthermore, two critical vulnerabilities were found and proactively fixed. At the beginning of the Phase I engagement, there were 76 Total Fuzz Issues. As fixes and improvements were implemented, new fuzz issues were identified for analysis, allowing for further iteration and fine-tuning.

Phase II was focused on improving the number of executions/second and high signal to noise ratio produced by the fuzzers. Additionally to triaging remaining open high priority fuzzers from Phase I, there was an effort to close another 10 health check related fuzz issues. In order to cover more code faster, while remaining highly efficient and controlling output, X41 was able to analyze bugs from the first phase to determine how the fuzzers were running tests and where they were losing speed and validity. Of Envoy’s original 67 fuzzers, 19 were considered high priority due to testing code paths that are exposed to potential attack vectors; by dataplane traffic in the form of QoDs and bugs that can impact reliability and security of Envoy. Because Envoy is also offered as a service, fuzz testing and hardening the configuration interface is crucial. When fuzzers work on data and configurations simultaneously, there can be a tendency for them to move away from configurations that need deeper testing. This results in less code coverage. Envoy’s code base uses debug assertions, which further slowed down the testing process and often returned noise. To combat and remediate these multiple consequences, X41’s team developed a plan. First, create a way to generate valid configuration files by introspecting the configuration description already augmented by expressions supplying context, and then develop a two-step fuzzing process with the configuration files generated then used for fuzzing.

The team of Markus Vervier, Eric Sesterhenn, Dr. Andre Vehreschild and Dr. Robert Femmer had to develop a way to address the set of points they identified as limiting the fuzzers functionality. They aimed to improve the critical fuzzers that run on HTTP decoders and stream management, specifically HTTP/1.1, HTTP/2, and the HTTP/3 codec interfaces.

Dr. Vehreschild and Dr. Femmer not only developed four new fuzzers, but improved the overall security testing environment of Envoy instrumentally. The created fuzzers targeted HTTP decoders for high throughput focused on primary attack surfaces of Http1 (Balsa and http_parser), Http2 (nghttp2 and oghttp2), and Http3 Quic. To address the issues around configuration files, the team experimented with running configurations repeatedly to explore the data plane deeper before generating another configuration. Configurations found could increase fuzzing speed by a factor of 4 to 20 times. Further work resulted in removing debug assertions and reducing the binary size of fuzzers by a factor of 1.5. By developing ways to create valid and long-running configurations across the configuration and data planes, the performance of fuzzers was increased by a factor of 40.

The graph below strongly suggests that, since March 2022 when Phase I of the engagement began, there has been a downward trend in generation of new fuzz issues due to this collective effort.

This work took months to complete and was an open discussion-based project that allowed for all members to collaborate, help each other, and discuss the possibilities and testing done.

We sincerely thank Kirtimaan Rajshiva, Adi Peleg, Yan Avlasov, Harvey Tuch, Joshua Marantz, and the entire Envoy Platform Team for funding and guiding this effort along with Markus Vervier, Eric Sesterhenn, Dr. Andre Vehreschild, Dr. Robert Femmer, and Sofie Seuren of X41 D-Sec for their diligent work and insight.

OSTIF is grateful for the opportunity to collaborate and improve security posture for the betterment of FOSS. We would also like to recognize Google, without whose funding this project would not have been possible.

You can read the audit report here.

You can read more about the work done at X41’s blog here.

For further information on Envoy, see here.

OSTIF collaborates with the Envoy Team to further improve security posture. was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

History

Envoy was released as OSS in the fall of 2016, and much to our amazement quickly gained traction throughout the industry. Users were drawn to many different aspects of the project including its inclusive community, extensibility, API-driven configuration model, powerful observability output, and increasingly extensive feature set.

Although in its early history Envoy became synonymous with “service mesh,” its first use at Lyft was actually as an API gateway / edge proxy, providing in-depth observability output that aided Lyft’s migration from a monolithic to a microservice architecture.

Over the last 5+ years, we have seen Envoy adopted by a tremendous number of end users, both as an API gateway and as a sidecar proxy in “service mesh” roles. At the same time we have seen a large vendor ecosystem spring up around Envoy, providing a multitude of solutions both in the OSS and proprietary domains. Envoy’s vendor ecosystem has been critical to the project’s success; without funding for all of the employees that work part or full-time on Envoy the project would certainly not be what it is today.

The flip side of Envoy’s success as a component of many different architecture types and vendor solutions is that it is inherently low level; Envoy is not an easy piece of software to learn. While the project has had massive success being adopted by large engineering organizations around the world, it is only lightly adopted for smaller and simpler use cases, where nginx and HAProxy are still dominant.

The Envoy Gateway project was born out of the belief that bringing Envoy “to the masses” in the API gateway role requires two primary things:

• A simplified deployment model and API layer aimed at lighter use cases.
• Merging the existing CNCF API gateway projects (Contour and Emissary) into a common core that can provide the best possible onboarding experience, while still allowing vendors to build value-added solutions based on Envoy Proxy and Envoy Gateway.

We strongly believe that if the community converges around a single Envoy branded API gateway core, it will:

• Reduce duplicative efforts around security, control plane technical details, and other shared concerns.
• Allow vendors to focus on layering value added functionality on top of Envoy Proxy and Envoy Gateway in the form of extensions, management plane UI, etc.
• Lead to a “rising tide lifts all boats” phenomenon in which more users around the world enjoy the benefits of Envoy, whether their organization is large or small. More users feed the virtuous cycle of more potential customers, more support for the core Envoy project, and a better overall experience for all.

Project outline

At a high level, Envoy Gateway can be thought of as a wrapper around the Envoy Proxy core. It will not change the core proxy, xDS, go-control-plane, etc. in any way (other than potentially driving features, bug fixes, and general improvements!). It will provide the following functionality:

• A simplified API for the gateway use case. The API will be the Kubernetes Gateway API with some Envoy-specific extensions. This API was chosen because deployment on Kubernetes as an ingress controller is the initial focus for the project and because the API has broad industry buy-in.
• A “batteries included” experience that will enable users to get up and running as fast as possible. This includes lifecycle management functionality that provisions controller resources, control plane resources, proxy instances, etc.
• An extensible API surface. While the project will aim to make common API gateway functionality available out of the box (e.g., rate limiting, authentication, Let’s Encrypt integration, etc.), vendors will be able to provide SaaS versions of all APIs, provide additional APIs and value added functionality such as WAF, enhanced observability, chaos engineering, etc.
• High quality documentation and getting started guides. Our primary goal with Envoy Gateway is to make the most common gateway use cases trivial to stand up for the average user.

On the topic of APIs, we believe that one area that has led to significant confusion is the effective reimplementation of Envoy’s xDS APIs in other projects when targeting advanced use cases. This pattern causes users to have to learn multiple sophisticated APIs (which ultimately translate back to xDS) in order to get their job done. As such, Envoy Gateway is committed to a “hard line in the sand” in which the Kubernetes Gateway API (and any permissible extensions within that API) is the only additional API that will be supported. More advanced use cases will be served by an “xDS mode” in which existing API resources will be automatically translated for the end user, who can then switch to utilizing the xDS APIs directly. This will lead to a crisper primary API while allowing an escape hatch for organizations that might outgrow the expressiveness of the primary API and wish to utilize the full power of Envoy via xDS.

On API standardization

Although a goal of Envoy Gateway is to provide a reference implementation for easily running Envoy in Kubernetes as an ingress controller, possibly the most important contribution of this effort will be standardizing the APIs that are used for this purpose. As the industry converges on specific Envoy Kubernetes Gateway API extensions, it will allow vendors to easily provide alternate SaaS implementations that may be preferable if a user outgrows the reference implementation, wants additional support and features, etc. Clearly, there is much work to do around defining the API extensions, determining which APIs are required versus optional for conformance, etc. This is the beginning of our standardization journey and we are eager to dive in with all interested parties.

Next steps

Today we are thankful for the initial sponsors of Envoy Gateway (Ambassador Labs, Fidelity, Tetrate, and VMware) and are excited to start on this new journey with all of you. The project is very early with the focus so far having been to agree on goals and high level design, so it’s a great time to get involved, either as an end user or as a system integrator.

We also want to make it extremely clear that existing users of Contour and Emissary will not be left behind. The project (and VMware and Ambassador Labs) is completely committed to ensuring a smooth eventual migration path for users of those projects to Envoy Gateway, either via translation and replacement, or via those projects becoming wrappers around the Envoy Gateway core.

We are extremely excited about bringing Envoy to a larger group of users via the Envoy Gateway project, and we hope you will join us on our journey!

Introducing Envoy Gateway was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

“I am excited about Tetrate’s Envoy Fundamentals course and certification. It is well composed with information on the applications of Envoy and practical labs with step-by-step instructions and quizzes,” said Matt Klein, creator of Envoy Proxy and senior engineer at Lyft. “You will be rewarded with a certificate for completing the full course. Best of all, the training is entirely free. I highly recommend this course to people who want to learn and use Envoy Proxy.”

Envoy is the default choice for building a service mesh

The CNCF-graduated project Envoy Proxy is the most popular sidecar and ingress provider. It’s the default sidecar in multiple service mesh projects, including Istio, Open Service Mesh, and Appmesh. As per CNCF’s 2020 survey, the usage of Envoy as an ingress provider increased 116%, with a total of 37% of respondents using Envoy Proxy in production.

Envoy was initially built at Lyft as a proxy to serve as a universal data plane for large-scale microservice service mesh architectures. The idea is to have Envoy sidecars run next to each service in your application, abstracting the network from the application. It works as an edge gateway, service mesh, and hybrid networking bridge. With Envoy, companies can scale their microservices by providing a more flexible release process and highly available and resilient infrastructure.

Envoy is rich in its network-related features such as retries, timeouts, traffic routing and mirroring, TLS termination, observability, and many more. As all network traffic flows through the mesh of Envoy proxies, it becomes necessary to observe traffic and problem areas, fine-tune the performance, and pinpoint any latency sources from a single place. With its many features and vast APIs, it might be overwhelming for users to navigate the extensive and comprehensive documentation, especially for beginners unfamiliar with proxies and just starting with their Envoy journey. Therefore, we decided to create a course that introduces the basic concepts of Envoy and its internals to enable a faster learning curve for users.

“Envoy has seen a rapid increase in adoption over the past years. This means access to easy to learn resources that provide users the ability to scale fast in their learning is crucial to keep Envoy adoption easier,” said Varun Talwar, Tetrate Co-founder.

About the Envoy Fundamentals Course

The free Envoy fundamentals course consists of 8 modules, with multiple video lessons and labs within each module.

The course starts with an introduction to Envoy and explains concepts such as HTTP connection manager filter, clusters, listeners, logging, administrative interface, and extending Envoy. Each module includes practical labs with step-by-step instructions. The labs allow learners to practice explained concepts such as

• Configuration of dynamic Envoy
• Circuit breakers
• Traffic splitting
• Mirror requests
• Global and local rate-limiting
• HTTP tap filter
• Extending Envoy using Lua script and Wasm, and more.

Quizzes after each module help you evaluate your knowledge and gauge your progress. After completing the course and all quizzes, you’ll receive a certificate of completion. Sign up for the free Envoy Fundamentals course on the Tetrate Academy website to start learning.

More resources to learn Envoy

• Get started with Envoy in 5 minutes (blog)
• The basics of Envoy and Envoy extensibility (blog)
• Envoy 101: File-based dynamic configurations (blog)
• Istio Weekly: Envoy fundamentals (video)
• Istio Weekly: Developing Envoy Wasm Extensions (video)
• Envoy deep dive (video)
• Lyft’s Envoy: Embracing a service mesh (video)
• Making Envoy contributions feasible for everyone (video)

About Tetrate

Started by Istio founders and Envoy maintainers to reimagine application networking, Tetrate is the enterprise service mesh company managing the complexity of modern, hybrid cloud application infrastructure. Its flagship product, Tetrate Service Bridge, provides a comprehensive, enterprise-ready service mesh platform built for multi-cluster, multitenancy, and multi-cloud deployments. Customers get consistent, baked-in observability, runtime security, and traffic management in any environment. Tetrate remains a top contributor to the open-source projects Istio and Envoy Proxy, and its team includes senior maintainers of Envoy. Find out more at www.tetrate.io.

Envoy Fundamentals, a training course to enable faster adoption of Envoy Proxy was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Envoy project has always strived to make the network “transparent” to all applications running regardless of the programming language, the platform architecture, and the operating system. Today, we’re excited to announce that Envoy is now generally available for use on the Windows platform! You can start using Envoy on Windows for production workloads starting with version 1.18.3.

Porting Envoy on Windows has been a goal of the community since 2016. Since then, the Envoy-Windows-Development group has made a lot of progress. Primarily composed of developers from VMware and Microsoft, the group has collaborated over the last year to get Envoy from an alpha release in October 2020 to a stable production-ready state today.

You can now use Envoy on Windows to build cloud-native applications, improve the observability of legacy applications, and even deploy Envoy alongside a Windows application as an edge proxy.

Before we delve into the public-facing features that we have built to improve the Windows experience, we would like to thank the Envoy developer community and maintainers for their guidance, support, and patience.

Recently added features

Since the Alpha release of Envoy on Windows we have added more features, enabled continuous integration, and improved the performance and reliability of Envoy on Windows.

Improved polling mechanism with synthetic edge events

Envoy solves the C10K problem on Linux by serving many clients with each thread and using nonblocking I/O and edge-triggered readiness change notifications. However, edge-triggered change notifications are not supported on Windows Server 2019 and this caused Envoy on Windows to spin and drain CPU resources.

To address this issue, we designed synthetic edge events that behave like edge events. Synthetic edge events are level events that are managed by Envoy and behave like edge events. We achieved this by manually disabling event registration when a new event arrives and enabling them again only when needed.

We observe, in the integration tests, that by switching to synthetic edge events, Envoy catches 3 orders of magnitude fewer events. This is a significant improvement that allows Envoy on Windows to scale to multiple concurrent connections. We plan to improve the event mechanism further. The newer version of Windows offers a native edge events API which we plan to integrate into Envoy.

Windows container image

We want operators and developers to be able to get started with Envoy on Windows with minimal friction. Since late October of 2020, we have been publishing developer images on docker hub. These images contain various SDK and tools that are particularly useful for developers looking to extend or experiment with Envoy. With version 1.18 we also publish slimmer images, envoy-windows, which are more suitable for production environments.

Improved Diagnostics

We expect that operators will want to run Envoy on different platforms with the same configuration. The new stream access loggers allow operators to redirect the access logs produced by the listeners and the admin portal to the standard output of the process. Envoy uses the correct native API to write to the standard output/error depending on the platform it runs on.

Add support for Clang compiler

Envoy users leverage Envoy’s versatile extension model to build custom filters and features for their use case. Part of the versatile extension model is the support for different architectures (arm) and compiler toolchains (Clang and GCC) on Linux. Following the spirit of the community, we have added support for Clang on Windows. Since January, the CI builds envoy.exe on every commit both with MSVC and Clang compiler.

Improved process management

The Alpha release focused on functionality more than usability. Since then, we have added features for developers and Windows native operators to manage the lifetime of the Envoy process with ease. Now Envoy terminates gracefully when Ctrl + C and Ctrl + Break events are sent to the console in the same way it handles SIGINT and SIGTERM. Additionally, we have added experimental support to run Envoy as a Windows service.

Contribution Statistics

Although these statistics do not say much on their own, we would like to take a step back to look at what we have accomplished in the past year:

1. The Windows development group has contributed 189 patches to the Envoy repository.
2. 416 out of the 435 Envoy’s tests are run on Windows at every commit. 16 tests are not compiled on Windows due to the lack of platform support for the particular feature and the remaining 3 tests are failing in the newly added QUIC support.
3. We support two compilers (MSVC and Clang), three runtimes (win32 native, SCM, and containers), and multiple versions of the Windows OS (Client and Server versions 1809 and above).

What’s next for Envoy on Windows?

We still have a lot to work to get Envoy on Windows to parity with Linux. We look forward to:

1. Add more sample sandboxes that demonstrate different use cases.
2. Improve the distribution of binaries.
3. Benchmark and improve performance.
4. Integrate with Service Mesh solutions, like OSM, on the upcoming Windows Server 2022 release.

How do I provide feedback and get involved?

We are looking forward to listening to your feedback and your comments. There are multiple ways to reach us and all of them are equally effective so you can choose the one that you prefer.

You can get in touch with the contributors working on Envoy on Windows to ask questions or provide feedback in the Envoy slack workspace #envoy-windows-dev channel. Additionally, we follow and triage all the Github issues. We also follow the envoy-dev and envoy-announce Google groups and reply to questions and issues there. We also maintain a FAQ on the documentation website.

One important note, if you encounter a bug that is causing Envoy to crash, please reach out to envoy-security@googlegroups.com. You might have stumbled upon a security vulnerability that should not be publicly disclosed before we patch it.

Like every CNCF project, we host bi-weekly meetings which you can find on the Envoy CNCF calendar. These meetings are a good place to start engaging and contributing to the Envoy roadmap on Windows.

Envoy Windows Development Group,

Sunjay Bhatia, William A. Rowe Jr (VMware)

Praveen Balasubramanian, Nick Grifka, Randy Miller, Sotiris Nanopoulos, David Schott (Microsoft)

General Availability of Envoy on Windows was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

During March and April this year security researchers from Ada Logics worked on the fuzzing infrastructure of Envoy Proxy sponsored by the Linux Foundation. One of the focus areas was optimisation and in this blog post we will go through some of the findings of this work. The full report is a comprehensive 28-page document that is available here.

In Envoy we have a large code base comprising roughly 1.3 million lines of code including auto-generated code (e.g. Protobuf code), testing infrastructure and also various important dependencies. To cater for security we have a comprehensive fuzzing suite that continuously analyses Envoy by way of OSS-Fuzz. We follow the principles of an ideal integration and to this end we have a variety of fuzzers where some fuzzers resemble unit-tests in that they target specific areas of the Envoy proxy, such as our json fuzzer, and others are closer related to integration-tests in that they target end-to-end concepts of Envoy, such as our HTTP2 end-to-end fuzzer.

The issue we had observed was that our end-to-end fuzzers ran with a low execution speed and, in particular, the execution speed was much lower than our own performance experiments with Envoy Proxy even after considering the slowdown of sanitizers, e.g. AddressSanitizer. Since execution speed is an essential factor for the success of fuzzers we had a desire to resolve the excessive slowdown and our investigations into this slowdown is the focus of this blog post.

Coverage instrumentation and performance in libFuzzer

In order to understand the reason for slowdown we need to understand how coverage-guided fuzzing works in libFuzzer. We describe this in detail in the full report so in this blog post we will keep it short.

From a simplified perspective, coverage-guided fuzzing with libFuzzer works as follows. First, we compile the target with fuzzer-specific instrumentation, which means we instrument the code with coverage-feedback instrumentation. This instrumentation will place counters on each basic block of the target program.

Second, when we run the compiled fuzzer the following process happens. At instantiation the fuzzer identifies the number of counters in the target and creates a “corpus” object as a combination of all the counters. The fuzzer then continues by performing the following infinite loop:

1. Execute the fuzzer entry point. In this case, basic blocks that are executed will trigger an increase of its corresponding counter.
2. Post-process the execution by going through each of the counters in the program and log their state.If any of the counters is different in comparison to previous runs, it means that the execution triggered new coverage. Therefore, update corpus with the new test case and record the state of the counters.

Following the description above, we can observe that there are performance penalties in two places when using libFuzzer, namely in the execution of the target as well as the post-processing after each fuzz iteration. The performance penalty during execution of the target code is fairly obvious for a non-fuzzing expert, however, the performance penalty of the post-processing is less obvious.

To measure the impact of the post-processing, consider the following simple fuzzer harness:

#include <stdint.h>
#include <string.h>
#include <stdlib.h>
#include “target.h”

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size){
  char *new_str = (char *)malloc(size+1);
  if (new_str == NULL){
    return 0;
  }
  memcpy(new_str, data, size);
  new_str[size] = ‘\0’;

  free(new_str);
  return 0;
}

The above is an empty fuzzer that simply does a few operations but essentially explores no code. However, it includes the target.h header file and compiling this fuzzer with an empty “target.h” file, we can observe the amount of counters inserted by the fuzzer-instrumentation into the executable fuzzer:

$ clang -fsanitize=fuzzer ./test.c 
$ ./a.out -runs=0
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3040949426
INFO: Loaded 1 modules   (3 inline 8-bit counters): 3 [0x6ee0c0, 0x6ee0c3), 
...

The output shows us the fuzzer has 3 inline 8-bit counters. Now, if we compile the exact same fuzzer but with a target.h file defined as follows:

int target(char *addr, size_t size) {
  if (addr[0] == ‘A’ && size == 0) return 1;
  return 0;
}

and then load the fuzzer again without execution any iterations, we observe the following:

$ clang -fsanitize=fuzzer ./test.c 
$ ./a.out -runs=0
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3295510585
INFO: Loaded 1 modules   (7 inline 8-bit counters): 7 [0x6ee0c0, 0x6ee0c7),

This time, the fuzzer has 7 inline 8-bit counters, despite nothing in the fuzzer-relevant code has changed. That is, the fuzzers have the exact same coverage and the only difference between the two fuzzers is that one fuzzer is compiled with additional, albeit unreachable, code which in turn is instrumented with coverage-feedback instrumentation.

The question is, what is the impact of these inline 8-bit counters even if the fuzzer does not change? In order to identify this, we can use a small script to generate arbitrarily large target.h files and then run an experiment for each fuzzer that counts the number of inline 8-bit counters and the number of executions per second of the resulting fuzzer. To observe this, we used the following minor Python script to automate creation of the target.h file with an arbitrary number of if-statements similar to the if-statement above as well as a simple bash script:

import os
max_iterations = int(os.environ['N'])
func_impl = "int target(char *addr, size_t size) {\n"
for i in range(max_iterations):
    func_impl += "\tif (addr[0] == 'A' && size == %d) return %d;\n"%(i, i)
func_impl += "\treturn 0;\n}\n"

with open("target.h", "w") as ff:
    ff.write(func_impl)

The Python script generates a target.h file with N number of if-statements and we then use the following bash script to run several experiments with various values of N:

for N in 1 1000 10000 50000 100000 200000 500000 1000000; do
    echo "[+] Starting analysis"
    export MAX_ITER=${N}
    python ./make_p.py
    clang -fsanitize=fuzzer test.c
    ./a.out -runs=0
    ./a.out -max_total_time=60
done

Running this script and noting the counters as well sa the execution speed, we get the following data:

We observe the number of inline 8-bit counters grow linearly with N, which is to be expected. Again, we emphasize that the fuzzer itself is still just a single function and all fuzzers have the same code coverage, the only difference is the amount of extra code in the final executable that is compiled with fuzzer-specific instrumentation, although this code is not reachable by the fuzzer.

The next question now is how do these counters impact the execution speed of the fuzzer? To measure this, we simply ran the fuzzer for 60 seconds (done in the bash script above) and noted the number of executions per second. We observed the following data:

The following Figure visualises the data:

The difference between 7 inline 8-bit counters and 3,000,004 is a slowdown of 2407x despite the fuzzer having the exact same coverage (namely 2) and essentially executing very little code. The slowdown is entirely due to the post-processing of counters, and these counters.

Impact on Envoy

So how did this impact the Envoy fuzzers? As noted above, Envoy consists of a lot of code and the vast majority of this code was built with sanitizers. The HTTP2 end-to-end fuzzer had a staggering 1.3 million inline 8-bit counters and there is a specific reason for instrumenting our codebase in this manner. Our fuzzing is run by OSS-Fuzz, which requires for the fuzzers to be built statically. As such, we build a lot of our dependencies ourselves rather than using pre-compiled binaries, and all of our instrumentation flags are also used when compiling these dependencies.

To get a more meaningful understanding of how much time was spent inside the post-processing we profiled our system using Prodfiler and observed that 26% of samples were observed inside the post-processing logic of our fuzzer, meaning that roughly a quarter of our system processing was spent in this function:

The impact of 1.3 million counters is significant and we observe in the experiment above that 1.5 million inline 8-bit counters causes a slowdown of 1203x on an empty fuzzer. It is also noteworthy here that not all of the code in our resulting fuzz binaries is actually reachable by a given fuzzer. Thus, by default, there is room for improvements in terms of avoiding instrumentation of irrelevant code. However, it may also be that there are improvements to be found by limiting coverage instrumentation of code that is reachable by a fuzzer. This observation raised a set of questions:

1. Should we accept the slowdown of instrumenting all of the code in Envoy with coverage-guided instrumentation and continue in our usual manner?
2. Assuming we should not instrument all code with coverage-guided instrumentation, which parts should we focus on? Are some parts more relevant than others, e.g. parser-specific code?
3. Should we push the issue of selectively instrumenting the code of a fuzz target away from the Envoy codebase and instead focus on adding heuristics about this into libFuzzer?

In essence, between (1) and (2) above, the solution must be based on empirical evidence. In this context, the empirical evidence is the amount of bugs we found, and not necessarily the execution performance of our fuzzers.

Reducing coverage instrumentation in turn reduces the amount of code the fuzzer explores, which can also have the benefit of helping the fuzzer focus on important parts of the codebase. This seems positive as we can focus our fuzzer even more on our threat model. However, it could also leave out specific areas of the code and this might indirectly affect how the fuzzer explores relevant code.

We experimented with reducing coverage instrumentation on certain parts of the code, resulting in decreasing the inline 8-bit counters from 1.3 million to 270,000. We saw a speedup of about 2–3x on our end-to-end fuzzers with this reduction. This was a quick win in terms of performance improvements, however, there is still a lot of room for improvement if we continue to reduce the amount of code instrumented. Unfortunately, this reduction also comes with an increase in our build system, which is already fairly complex in and of itself.

Conclusions and future work

In an investigation into our fuzzing infrastructure we found that we incurred a non-negligible performance penalty in our fuzzers due instrumenting a large codebase with coverage-feedback instrumentation. The post-processing of each fuzz iteration due to the number of coverage-counters had a significant impact on the overall performance.

We observed that we need a more refined policy in terms of which parts of the codebase to instrument instead of instrumenting our entire codebase. Previously our goal was to instrument all of the codebase, whereas now, we have observed the need for instrumenting code on a per-fuzzer basis.

In addition to improving our own build set up, we think it is worth exploring improvements to libFuzzer itself. Specifically, it should be possible by the fuzzer to approximate which coverage-feedback should be used by a given fuzzer. The benefit of this is that there are potential large gains to be achieved across all projects that use libFuzzer.

We thank the Linux Foundation for sponsoring this project as well as the team at Envoy for a fruitful and enjoyable collaboration.

A stroll down fuzzer optimisation lane and why instrumentation policies matter was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today we are excited to announce Alpha support for OpenTelemetry access logging in Envoy, which implements access logging based on the OpenTelemetry 0.7.0 Protocol release. The OpenTelemetry project was first announced by the Cloud Native Computing Foundation (CNCF) in May of 2019; and merged OpenTracing and OpenCensus into a new, unified standard.

With this announcement, users can now configure Envoy to export OpenTelemetry Protocol (OTLP) access logs in a flexible way, utilizing Envoy’s access logging formatter.

Furthermore, by exporting these logs to an OpenTelemetry Collector, the logs can be processed and exported as other telemetry data formats.

What does Alpha support mean?

Alpha support for OTLP in Envoy signifies that the Envoy and OpenTelemetry codebase have reached a stage where both the contributor and maintainer communities are confident it is stable enough for evaluation by the general public. We hope that by announcing this Alpha release, we can accelerate the process of collecting community feedback and contributions to push for a stable release in the near future (see Envoy extension security and stability posture).

The Alpha release signifies that the extension is functional but has not had substantial production burn time.

The road ahead

In the future, additional OpenTelemetry support will be introduced in Envoy, such as tracing and metrics and semantic conventions, to provide a complete OpenTelemetry Observability experience.

Envoy support for OpenTelemetry access logging was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Security Scorecards project is one of my favorite projects I’ve worked on while at Google. We announced it under the OpenSSF umbrella several weeks ago. It auto-generates a “security score” through a number of checks on OSS projects. The reason why I like this project so much is because it’s simple to understand, fully automated, uses objective criteria and has the ability to make a large impact across the OSS ecosystem by driving awareness and inspiring projects to improve their security posture.

Right after the initial announcement, we learned that the Envoy project was looking for a mechanism to understand and enforce policy on the health of projects they take dependencies on. This gave us the opportunity to test Scorecards on a real-world project used in critical systems across the industry!

We helped Harvey Tuch, maintainer of Envoy, try out and evaluate Scorecards for their use case as part of their new policy for external dependencies.

“Until recently, we’ve had no stance on external dependencies or criteria for determining if a new external dependency is acceptable.”

First, for fun, let’s run Scorecards on the Envoy project itself, and then we can run it against all of Envoy’s dependencies.

For Envoy, we get these results:

Not too shabby, and this prompted an issue for signing releases and another fix for the Scorecards project!

Taking this one level deeper, here’s a snippet of the output against Envoy’s external dependencies:

It was awesome to see the conversations taking place amongst the maintainers of those projects on making improvements — “hey, can we get fuzzing integrated into this project?”

It’s working!! 😏

The Envoy project plans to integrate OpenSSF Scorecards into their dependency metadata and enforce in CI policies around their dependencies. Scorecards will reduce the toil and manual effort when maintaining Envoy’s supply chain. A key aspect of their new policy is that automated criteria are applied first, and then where necessary exceptions are made for non-conforming projects. This deliberative process allows maintainers the opportunity to consider relevant scorecard criteria, asking questions around missing criteria and evaluate alternatives. No automated system will be perfect, but Envoy plans to collaborate with OpenSSF Scorecards to improve accuracy and relevancy.

I’m looking forward to seeing more case studies like this. It’s really motivating to see the beginnings of a success story and cross-community collaboration. If you’re a maintainer of an OSS project and interested in trying out Scorecards similar to Envoy, tell me about it! You can find me and others working on projects like this in the Securing Critical Projects Slack Channel.

Now if we can just figure out how to stop bumping up against GitHub’s API limits. ;)

Security Scorecards & Envoy — Automating supply chain analysis was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures. In this blog post we will walk you through using Envoy and Windows containers to perform A/B testing on a website.

At the end of this post you should know:

1. How to get started with Envoy proxy.
2. How to use Envoy proxy as an http proxy inside a Windows Server Container.
3. How to use Envoy proxy as an edge proxy to split the traffic between different containers.

If you are new to Envoy, we would recommend the following material to onboard:

1. Intro: Envoy — Matt Klein & Constance Caramanolis, Lyft
2. The project documentation page at envoproxy.io.

The customer scenario

In this demo we want to split the user traffic of our website to two different services. In practice, traffic split is useful to perform A/B testing and rolling deployments.

The architecture of the system is the following:

To build the system we rely on two types of components. These components are:

1. The Front-end Envoy container that sits at the edge of the network. This container balances the traffic between Service 1 and Service 2.
2. The Service container. This container serves the front-end of our pets website. There are two different instances of the pets website. Each instance runs on a different service container. One instance of the website prints Dog images whereas the other instance prints Cat images.

All the containers are based on the 2019 Windows Server Core image. For the user code here we use Python3 and Flask although any other server technology will also work.

Building and Running the Demo

In this section we will incrementally build the system. First we will build the two types of containers that we need. For each container we will configure the EnvoyProxy and validate that they work in isolation. Finally, we will compose all the containers together in a single network and have the full architecture up and running.

The code is available at github.com/davinci26/windows-envoy-samples.

Requirements

To follow along the demo you will need the following:

1. A Windows (Server) machine running on version greater than 2019. The reason is that internally Envoy uses Unix Domain Sockets. This feature is available on Windows after version 2019. You might not run into issues with an older version but you will be on uncharted waters. Also if you are running this code on virtual machine make sure that you have nested virtualization enabled.
2. Envoy proxy static executable built from source. As Envoy for Windows moves from alpha → beta we will be providing pre-built binaries but currently you will have to build Envoy from source. We have created this document to make the onboarding process a bit easier.
3. Docker for Windows and make sure that your docker engine is switched to Windows containers.

Setup the Service container

The service container is responsible for hosting the application code. For the service container we will use the following Dockerfile:

# Service Container Image

FROM mcr.microsoft.com/windows/servercore:ltsc2019

# Container Variables
ARG servicePath

# Setup Python
COPY ./setup_python.ps1 /

RUN powershell.exe .\\setup_python.ps1
RUN pip3 install -q Flask==0.11.1

# Copy local files for the flask server
RUN powershell -Command mkdir service/
ADD ${servicePath} service/

# Copy envoy and its configuration
RUN powershell -Command mkdir envoy-config/
ADD ./envoy-service-config.yaml ./envoy-config/envoy-service-config.yaml
ADD ./envoy-static.exe ./envoy-static.exe

# Set up the entrypoint
ADD ./service_entrypoint.ps1 ./
ENTRYPOINT powershell ./service_entrypoint.ps1

The setup that we do on the container is to install Python and Flask and copy Envoy and its configuration. The entry point is a PowerShell script that spawns the Python Flask server and Envoy.

For the service container we use the minimal Envoy configuration that allows us to match all the traffic coming to the container port 8000 and forward it to the Python Flask server running on port 8080. The Envoy configuration used is available here.

To build & run the container you need to use:

PS D:\envoy-test> docker build -f "./Dockerfile-service" -t "test:service" --build-arg servicePath=./service1 
PS D:\envoy-test> docker run --publish 3000:8000 --detach --name bb test:service.
PS D:\envoy-test> curl "http:\\localhost:3000" -UseBasicParsing

StatusCode        : 200
StatusDescription : OK

Now if you visit http://localhost:3000/ on your browser you should see a cute dog appearing on the screen.

To run Service2, build the Service2 container by changing the build argument to --build-arg servicePath=./service2.

Setup the Front-end Envoy container

For the front-end Envoy container we will create a container that is similar to the Service container. The Dockerfile that we use is the following:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

# Copy envoy and its configuration
RUN powershell -Command mkdir envoy-config/
ADD ./envoy-frontend.yaml ./envoy-config/envoy-frontend.yaml
ADD ./envoy-static.exe ./envoy-static.exe

# Create a log folder to store the stats
RUN powershell -Command mkdir logs/

ENTRYPOINT ["envoy-static.exe", "-c", "./envoy-config/envoy-frontend.yaml", "--service-cluster", "front-envoy"

In this container we only run an Envoy proxy that handles splitting the traffic between Service 1 and Service 2.

To orchestrate the traffic split, we add the following snippet to Envoy’s configuration:

routes:
 - match:
   prefix: "/"
      runtime_fraction:
       default_value:
         numerator: 50
          denominator: HUNDRED
        runtime_key: routing.traffic_shift.placeholder
     route:
        cluster: service1
   - match:
      prefix: "/"
     route:
      cluster: service2

This script creates two matching rules for the traffic coming to the container. The first matching rule, which applies for 50% of the traffic, routes the traffic to Service 1. Envoy routes the remaining traffic (other 50%) to Service 2. With this routing rule we perform A/B testing on the two services.

To build & run the container you need to use:

PS D:\envoy-test> docker run -f --publish 3005:8001 --detach --name fe test:envoy
PS D:\envoy-test> docker run --publish 3005:8081 --detach --name fe test:envoy
PS D:\envoy-test> curl "http:\\localhost:3005" -UseBasicParsing

StatusCode        : 200
StatusDescription : OK

Now if you visit http://localhost:3005/ on your browser you should see Envoy admin board.

Compose the network together

At this point every piece of the system is working. Now, we only need to assemble the network and deploy all the containers together. To achieve this we have created a docker compose file.

The docker-compose file is the following:

version: "3.7"
services:
  front-envoy:
    build:
      context: .
      dockerfile: Dockerfile-envoy
    networks:
      - envoymesh
    expose:
      - "8000"
      - "8080"
      - "8081"
    ports:
      - "3000:8080"
      - "8081:8081"
    depends_on:
      - dog-service
      - cat-service

dog-service:
    build:
      context: .
      dockerfile: Dockerfile-service
      args:
        - servicePath=./service1/
    expose:
        - "8000"
    networks:
      envoymesh:
        aliases:
          - service1
    environment: 
      - ServiceId=1

cat-service:
    build:
      context: .
      dockerfile: Dockerfile-service
      args:
        - servicePath=./service2/
    expose:
        - "8000"
    networks:
      envoymesh:
        aliases:
          - service2
    environment: 
      - ServiceId=2

networks:
  envoymesh: {}

To build & run the composed multi-container application we run the following commands:

PS D:\envoy-test> docker-compose build --pull
PS D:\envoy-test> docker-compose up -d
PS D:\envoy-test> docker-compose ps
Name                        Command               State                            Ports
----------------------------------------------------------------------------------------------------------------------------
envoy-test_cat-service_1   cmd /S /C powershell ./ser ...   Up      8000/tcp
envoy-test_dog-service_1   cmd /S /C powershell ./ser ...   Up      8000/tcp
envoy-test_front-envoy_1   envoy-static.exe -c ./envo ...   Up      8000/tcp, 0.0.0.0:3000->8080/tcp, 0.0.0.0:8081->8081/tcp

Now if you visit http://localhost:3000/ you should see the output from either Service 1 (dog service) or Service 2 (cat service). Refreshing the page triggers another request that goes through the front-end Envoy proxy. Envoy forwards the request to either to Service 1 or Service 2.

Finally, we can go on Envoy admin page hosted on http://localhost:8081 and see the stats that Envoy collects automatically.

For example we can search for the upstream_rq_completed entry which is available in the stats page. This entry will tell us how many requests each Service completed. You can learn more about how the Envoy stats work in this blog post.

Recap

In this blog post we built a multi-container system to split the traffic our website between two service. To achieve that we relied on Windows Server Core containers and Envoy Proxy. We incrementally built each type of container in our system and then we composed them together.

Envoy on Windows is currently at an Alpha stage and we look forward to hearing your feedback. If you encounter any issue running the code above feel free to open an issue at github.com/envoyproxy/envoy. You can also reach out to the developers on the Envoy proxy slack channel.

Envoy Proxy on Windows Containers was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Porting Envoy to the Windows platform has been a goal of the project since 2016 and today we are excited to announce the Alpha release of Windows-native support for Envoy. The contributor community has been hard at work bringing Envoy’s rich feature set to Windows and this is another step in line with the project’s mission of making the network “transparent” to any application, regardless of language, architecture, or operating system.

Envoy is already in production use by a wide range of companies and Windows support should open up its usage to additional cloud-native services, legacy .NET applications, and a whole host of other application architectures. Particularly promising is the potential for users to deploy Envoy alongside Windows applications running in the datacenter or public cloud on Windows Server, in Windows-based containers, or even alongside desktop applications.

The road to this Alpha announcement has been a long one but we hope we have done our part to improve the Envoy code base with cross platform code, new abstractions, and additional test coverage. If you are interested in a glimpse into the process of porting Envoy to Windows, take a look at this presentation from KubeCon 2019 and look out for the upcoming presentation at EnvoyCon 2020. We would like to thank the Envoy maintainer team and especially Matt Klein and Lizan Zhou for enabling and supporting the Windows contributor group to reach this milestone.

What does Alpha support on Windows mean?

Alpha support for Envoy on Windows signifies the Envoy codebase has reached a stage where the contributor and maintainer community is confident it is stable enough on Windows for evaluation by the general public. General Availability (GA) release is also upcoming. We hope that by announcing this Alpha release, we can accelerate the process of collecting community feedback and contributions to push for a GA release.

As a result of getting to Alpha, Envoy compiles on Windows and tests are now required to pass in CI for every pull request and merged commit. In addition, there is a dedicated group of developers contributing to Windows, spending their time triaging reported issues and bugs, fixing CI failures and test flakes, and working with maintainers to ensure code quality and correctness (if you would like to get involved with this effort, see below!). The Alpha release does not signify that Envoy is suitable or supported for production workloads yet.

How do I get started with Envoy on Windows?

The project considers the master branch of the Envoy source repo to be release candidate quality at all times, and many organizations track and deploy master in production. As such, there is no “tagged” Alpha release commit, rather the master branch should be considered Alpha release quality on Windows until a GA release occurs. In general the Envoy codebase continues to move forward rapidly so we recommend refreshing your source checkouts often to take advantage of the feedback and improvements from the contributor community.

Update 10/20/2020: Windows Docker Image

As of this PR per-master commit Windows Docker image builds containing a statically compiled Envoy binary are now published publicly. The image can be found here. Image entrypoint and configuration mirrors the Linux “dev” image published per-master commit and will run a basic bootstrap configuration. You may use this image as-is to evaluate Envoy or extract the compiled envoy.exe binary to another container image or to run outside of a container.

Building From Source

Documentation on setting up a build environment and compiling a statically linked Envoy executable from source on Windows with Bazel can be found here. We also provide a Windows Server 2019 Server Core based Docker container image with all required tools to build and statically link Envoy, see this document for more details.

Usage Example

Once you have an Envoy binary and want to start getting familiar with using Envoy on Windows, a good place to start is this tutorial. You will run through a modified version of the Front Proxy Sandbox example that demonstrates the advantage of running Envoy collocated with your services: all requests are handled by the service Envoy, and efficiently routed to your services.

Are there any Windows-specific differences to be aware of?

Work on Windows support is still moving rapidly and as of this Alpha release most all core Envoy functionality should have parity with Linux. Service mesh support requires additional platform capabilities and we hope to enable this functionality with an upcoming release of Windows. Envoy configuration and usage should not differ between platforms other than with common platform specific details like file paths, socket options, etc. That said, some existing features of Envoy were designed and implemented with Linux in mind first and as a result may be disabled on Windows or work in a limited capacity. You can find a list of Envoy APIs with degraded or disabled functionality on Windows here.

How do I provide feedback and get involved?

We expect users and new contributors may run into known issues or new bugs others have reported. The area/windows tag in the Envoy issue tracker on GitHub and pulling the latest Envoy source from the master branch are great starting points if you encounter problems. Including “Windows:” in the title of any new issues and following the existing Envoy new issue templates will greatly help with triage. As always, PRs and issues are welcome to improve documentation in addition to Envoy source code.

To get in touch with full-time contributors to Envoy on Windows about how to get more involved with the project, development details, and detailed user scenarios, visit the Envoy slack workspace #envoy-windows-dev channel. We also hold a community meeting specifically for Windows contributors which you can find on the Envoy CNCF calendar here. In addition to Github issues, this weekly meeting is a good place to stay in the loop with and contribute to the Envoy roadmap on Windows. The envoy-dev and envoy-announce Google groups are two other avenues via which we may solicit feedback.

We hope to lean on the community to get as much mileage as we can running Envoy on Windows and grow the community as we push forward to a GA release. Whether you would simply like to evaluate if Envoy suits your needs in a Windows environment or are interested in getting involved in active development on Windows, the project greatly appreciates detailed feedback. We look forward to collaborating with you and hearing how you use Envoy on Windows!

Announcing Alpha Support for Envoy on Windows was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you are considering developing a custom Envoy extension, it is crucial to have a solid understanding of how request processing works in Envoy and the role Envoy extensions play.

Today, we will take a closer look at Network Filters.

We begin with Network Filters partly because they have a simpler model, but also because support for HTTP in Envoy is implemented as yet another Network Filter. It wouldn’t be possible to explain HTTP Filters without referring to Network Filters first.

We’ll start from a general overview and then go over the practical applications.

Let’s get going!

Lifecycle of a Network Filter

Envoy is fundamentally a L3/L4 proxy capable of handling any protocols at or above that level.

The workhorse of Envoy is a Listener — a concept responsible for accepting incoming (also known as “downstream”) connections and kicking off the request processing flow.

Each connection is processed by passing received data through a series of Network Filters collectively referred to as a Filter Chain.

Network Filters intercept data (TCP payloads) flowing in both directions:

The “Downstream > Envoy > Upstream” path is referred to in Envoy as the “read” path, and the opposite direction is referred to as the “write” path.

Unlike the Filter concept you’ve seen in other APIs, Filters in Envoy are stateful. A separate instance of Network Filter is allocated for every connection.

The interface of a Network Filter consists of the following callbacks.

On the “read” path:

On the “write” path:

A Network Filter may also subscribe to connection events:

As mentioned earlier, Envoy processes received data by iterating through the filter chain.

On connect from the Downstream, Envoy will iterate through the filter chain to call onNewConnection().

If a Network Filter returns StopIteration from its callback, Envoy will not proceed to the next filter in the chain.

Beware that StopIteration only means “don’t call filters after me for this particular iteration cycle” as opposed to “don’t do any further processing on that connection until I give a green light”.

For example, even if a Network Filter returns StopIteration from its onNewConnection() callback, once Envoy receives a chunk of request data from the Downstream, it will iterate through the filter chain again, this time calling the onData() callback. Filters that haven’t seen onNewConnection() yet are guaranteed to see it prior to onData().

From a perspective of a single Network Filter, request processing flow looks the following way.

On the “read” path:

On the “write” path:

We will explain the difference between onNewConnection() and onEvent(Connected) in the section on Gatekeeping along with a practical example.

Right now, let’s focus on StopIteration and the effect it has on data buffering in Envoy.

The first thing to know is that “read” and “write” paths in Envoy are not symmetrical and will be described separately.

On the “read” path:

When Envoy receives a new chunk of request data from the Downstream:

• First, it appends the new chunk into the “read” buffer.
• Next, it iterates over the filter chain calling onData() with the entire “read” buffer as a parameter (as opposed to the new chunk only).
• The “read” buffer will normally be drained by the terminal filter in the chain (e.g., TcpProxy).
• However, if one of the filters in the chain returns StopIteration without draining the buffer, the data will remain buffered.

When Envoy receives the next chunk of request data from the Downstream:

• It will again append to the “read” buffer.
• Next, it iterates over the filter chain calling onData() with the entire “read” buffer as parameter.

The important thing to notice is that, due to the presence of the “read” buffer, a Network Filter might observe the same data twice in its onData() callback!

Finally, how safe is it to let the “read” buffer keep growing? Can it grow indefinitely or overflow? The good news is that Envoy takes care of this aspect automatically and will stop reading data from the socket as soon as the size of the “read” buffer exceeds the limit (1MiB by default).

On the “write” path:

There is no equivalent of the “read“ buffer on the “write” path.

When Envoy receives a new chunk of response data from the Upstream:

• It iterates over the filter chain calling onWrite() with the new chunk as parameter.
• If all filters in the chain return Continue, the chunk will be appended to the “write” buffer (response data ready to be sent back to the Downstream).
• If one of the filters returns StopIteration, the chunk will be dropped.

It’s worth noting one more time that StopIteration has effect only on a single iteration cycle through the filter chain. It’s not a signal to “stop further processing until I give a green light“. The next time Envoy receives a chunk of data, it will start calling filters again no matter whether they returned Continue or StopIteration last time. Consequently, if a filter needs to wait for some external event to occur, it has to keep returning StopIteration from onData()/onWrite() callback until that very moment.

To prevent the “write” buffer from overflowing, Envoy implements a concept of flow control (also known as “backpressure”). Its purpose is to stop receiving data from the remote side (i.e., Upstream) if the local buffer is full (i.e., “write” buffer on the “downstream” connection). You can learn more about flow control here.

At this point, we’ve touched most of the request processing flow. The final group of related APIs — continueReading(), injectReadDataToFilterChain(), and injectWriteDataToFilterChain() — will be explained in the section on Reshaping Traffic with a practical example.

Practical Applications

Not every Network Filter has to be as complicated as the HTTP Connection Manager.

Much more often, a Network Filter implements a single very specific action, such as rate limiting or authorization, and relies on other filters in the chain to do routing, load balancing, connecting to the Upstream, etc.

Let’s take a look into such practical applications of Network Filters.

Gatekeeping

One simple application of Network Filters is rejecting unwanted connections.

A few great examples would be RBAC filter, External Authorization filter, Client SSL Auth filter, Rate Limit filter, etc.

These filters remain neutral to the application-level protocol and make use of only a few Envoy APIs.

A typical request flow implemented by this filters looks like this:

Although it might seem intuitive to always initiate an auxiliary external request from the context of onNewConnection() callback, it is not possible in certain cases.

Envoy calls onNewConnection() [at least, on the first filter in the filter chain] as soon as a new connection has been accepted by the Listener. However, in the case of TLS connections, TLS handshake is not yet complete at this point. If a Network Filter depends on information from the TLS handshake, e.g., Client SSL Auth filter, it cannot do much in onNewConnection().

In such a case, a Network Filter should defer the auxiliary external request until onEvent(Connected) is called on that filter. Connected, despite such a general name, is in fact a very specific event that gets fired right after the TLS handshake completes successfully.

Beware of the difference between returning Continue vs StopIteration from onNewConnection() callback, which becomes apparent in the case of gatekeeping filters.

If a filter chain includes the TcpProxy filter (always the last filter in the chain), a listener will not start reading data from that connection until onNewConnection() is called on TcpProxy. Which means that returning StopIteration by a gatekeeping Network Filter might leave the connection in a stalemate (cannot proceed because it is not allowed to read data). Unfortunately, returning Continue has its own side effects. When onNewConnection() is called on TcpProxy, it immediately kicks off connection to the Upstream and then starts proxying the response from it, all happening before the decision whether to allow connection or not has been made by the gatekeeper. This is a good example where Envoy API could be made cleaner and safer by default.

Lastly, you might be wondering what happens to the request/response data when a Network Filter returns StopIteration. On the “read” path, data stays in the “read” buffer and the filter will see it again in the next call to onData(). On the “write” path, data gets dropped.

Collecting Protocol-specific Stats

Support for a new application-level protocol in Envoy typically starts from a single feature — the ability to parse protocol messages and derive some metrics out of them.

Mongo, MySQL, Postgres, Zookeeper, Kafka filters are all good examples.

These filters do not modify data they proxy and leave routing and load balancing up to the TcpProxy filter.

Just having insights into the actual traffic is already a big deal and brings a lot of value on its own.

Network Filters in this category typically operate as follows:

Notice that in the model described above, the Network Filter assumes that onData() callback is always called on a new, previously unseen chunk of data.

However, onData() API does not guarantee that (as described earlier).

For the filter to work correctly, subsequent filter(s) in the chain must drain the “read” buffer (e.g., TcpProxy filter always does that).

In practice, it means that users/operators of Envoy have to be mindful of the configuration they choose. Filters like Mongo, MySQL, Postgres will work correctly when they are immediately followed by TcpProxy. However, injecting an arbitrary filter in between the two might lead to unexpected results.

Here is an example of the correct combination of filters:

And here is an example of the incorrect one:

Feeding Protocol-specific Metadata

As an extension to the previous use case, a Network Filter can expose not only metrics but also fine-grained metadata derived from protocol messages.

Such metadata can later be used to power access decisions (e.g., by RBAC filter), to enrich access logs, etc.

Reshaping Traffic

When speaking about traffic shaping, think of a Fault Injection filter that throttles traffic.

From a technical perspective, throttling implies shifting the time when the next chunk of data is forwarded to the Upstream or Downstream.

Request flow implemented by these filters look the following way.

On the “read” path:

On the “write” path:

In the above example traffic gets reshaped through combined use of onTimer() and injectReadDataToFilterChain() / injectWriteDataToFilterChain() / continueReading().

onTimer() is not the only reason for traffic to change its shape. For example, filters can also invoke these API methods from callbacks after processing the results of HTTP or gRPC Client API calls.

Protocol-specific Routing and Load Balancing

Network Filters that fall into this category are the most advanced ones, e.g. HTTP Connection Manager, Redis, Thrift, Dubbo, etc.

Instead of relying on TcpProxy for protocol-agnostic routing and load balancing, a Network Filter can take over and do this job much more efficiently.

For example, the HTTP Connection Manager filter (which implements support for HTTP/1.1 and HTTP/2 in Envoy) takes advantage of HTTP/2 multiplexing and cuts the number of “upstream” connections to down to 1.

Since these filters are responsible for forwarding data to Upstream, they are also in charge of implementing flow control: if the “write” buffer on the “downstream” connection gets full, then stop receiving data from the Upstream. Similarly, if the “write” buffer on the “upstream” connection gets full, then stop receiving data from the Downstream.

We will touch more on routing, load balancing and flow control in future blog posts.

Native Envoy extensions (C++) vs WebAssembly

The Network Filter model and its practical applications we’ve described so far are applicable to both native (C++) and WebAssembly-based Envoy extensions.

Eventually, you will be able to achieve identical behaviour in both cases.

However, at the time of writing (early September 2020), WebAssembly support in Envoy implements only a subset of APIs available to native (C++) extensions:

• ✅ onNewConnection()
• ✅ onData()
• ✅ onWrite()
• ✅ onEvent(RemoteClose | LocalClose)
• ❌ onEvent(Connected)
• ❌ continueReading()
• ❌ injectReadDataToFilterChain()
• ❌ injectWriteDataToFilterChain()
• ❌ connection.close()
• ❌ setTimer() / onTimer()
• ❌ setDynamicMetadata()

Conclusion

Today you’ve learned enough to be able to develop a Network Filter of your own.

For example, you could add support for a new SQL / NoSQL / object database, a new key-value store, a new message broker, etc.

In the simplest form, all you need to do is to integrate an existing protocol parsing library into Envoy request lifecycle, expose some metrics and profit :).

Take it as a challenge!

Make it even more interesting and do it in Rust 😉.

That’s all for today. See you in the next blog post where we will do a deep dive into HTTP Filters!

Shameless plug

If you’re curious about WebAssembly-based Envoy extensions and willing to learn some Rust, take a look at Envoy SDK for Rust we’ve been working on here at Tetrate.

This community project is a place for experiments on how an idiomatic Envoy SDK could look.

Acknowledgements

A huge thank you to Stephan Zuercher, a senior maintainer of Envoy, who’s been patiently reviewing this blog post and made it much more readable!

References

https://www.envoyproxy.io/docs/envoy/latest/intro/life_of_a_request

https://github.com/envoyproxy/envoy/blob/master/source/docs/flow_control.md

https://blog.envoyproxy.io/envoy-threading-model-a8d44b922310

Taming a Network Filter was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.