Title: Ratelimit under load timeouts
Description:
I have an Envoy edge proxy deployed in Kubernetes, with the ratelimit service. I'm doing some basic load testing (~100rps) expecting latency to increase however I'm seeing rate limit service errors showing up as grpc upstream timeouts to the rate limiting service.
I've reviewed and applied/increased as many timeout settings as makes sense from the docs just trying to get these to go away, but I'm not having any success.
Initially I opened a ticket, #322 in the ratelimit repo but after debugging this feels more like an Envoy configuration issue.
I've included most of the timeout hacks that I've been trying, this is the basic configuration that I have.
...
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
generate_request_id: true
request_id_extension:
typed_config:
"@type": type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig
use_request_id_for_trace_sampling: false
stat_prefix: ingress_http
codec_type: AUTO
common_http_protocol_options:
max_connection_duration: 10s
idle_timeout: 900s
request_timeout: 30s
drain_timeout: 10s
...
http_filters:
- name: envoy.ratelimit
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit"
domain: test_domain
rate_limit_service:
transport_api_version: "v3"
grpc_service:
envoy_grpc:
cluster_name: ratelimit_service
timeout: 90s
failure_mode_deny: true
...
- name: ratelimit_service
connect_timeout: 90s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
common_http_protocol_options:
max_connection_duration: 10s
idle_timeout: 30s
http2_protocol_options:
connection_keepalive:
timeout: 30s
load_assignment:
cluster_name: ratelimit_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ratelimit-ratelimit-service
port_value: 8081Errors in the logs:
'grpc-status', '14'
'grpc-message', 'upstream request timeout'
[2022-02-21 16:39:36.991][13][debug][filter] [source/extensions/filters/common/ratelimit/ratelimit_impl.cc:117] rate limit fail, status=14 msg=upstream request timeout
[2022-02-21 16:39:36.991][13][debug][filter] [source/extensions/filters/http/ratelimit/ratelimit.cc:163] rate limit status, status=1
[2022-02-21 16:39:36.991][13][debug][http] [source/common/http/filter_manager.cc:947] [C271][S10347976980624733423] Sending local reply with details rate_limiter_error
[2022-02-21 16:39:36.991][13][debug][http] [source/common/http/conn_manager_impl.cc:1472] [C271][S10347976980624733423] encoding headers via codec (end_stream=true):
':status', '500'
[2022-02-21 16:39:36.991][13][debug][router] [source/common/router/router.cc:924] [C0][S10762015607805821320] upstream timeout
[2022-02-21 16:39:36.991][13][debug][router] [source/common/router/upstream_request.cc:341] [C0][S10762015607805821320] resetting pool request
[2022-02-21 16:39:36.991][13][debug][client] [source/common/http/codec_client.cc:142] [C53] request reset
[2022-02-21 16:39:36.991][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:209] [C53] destroying stream: 1 remaining
[2022-02-21 16:39:36.991][13][debug][http2] [source/common/http/http2/codec_impl.cc:1128] [C53] sent reset code=0
[2022-02-21 16:39:36.991][13][debug][http2] [source/common/http/http2/codec_impl.cc:1241] [C53] stream closed: 0
[2022-02-21 16:39:36.991][13][debug][http] [source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=true):
Title: Ratelimit under load timeouts
Description:
I have an Envoy edge proxy deployed in Kubernetes, with the ratelimit service. I'm doing some basic load testing (~100rps) expecting latency to increase however I'm seeing rate limit service errors showing up as grpc upstream timeouts to the rate limiting service.
I've reviewed and applied/increased as many timeout settings as makes sense from the docs just trying to get these to go away, but I'm not having any success.
Initially I opened a ticket, #322 in the ratelimit repo but after debugging this feels more like an Envoy configuration issue.
I've included most of the timeout hacks that I've been trying, this is the basic configuration that I have.
... filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager generate_request_id: true request_id_extension: typed_config: "@type": type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig use_request_id_for_trace_sampling: false stat_prefix: ingress_http codec_type: AUTO common_http_protocol_options: max_connection_duration: 10s idle_timeout: 900s request_timeout: 30s drain_timeout: 10s ... http_filters: - name: envoy.ratelimit typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit" domain: test_domain rate_limit_service: transport_api_version: "v3" grpc_service: envoy_grpc: cluster_name: ratelimit_service timeout: 90s failure_mode_deny: true ... - name: ratelimit_service connect_timeout: 90s type: STRICT_DNS lb_policy: ROUND_ROBIN common_http_protocol_options: max_connection_duration: 10s idle_timeout: 30s http2_protocol_options: connection_keepalive: timeout: 30s load_assignment: cluster_name: ratelimit_service endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ratelimit-ratelimit-service port_value: 8081Errors in the logs: