announcement
Announcing Arroyo 0.15.0
Arroyo 0.15 is now available! This release includes a large number of fixes, improvements, and new features, including a complete Iceberg sink, and more.
Micah Wylde CEO of Arroyo
Blog
Updates from the Arroyo team
|
Arroyo is joining Cloudflare
announcement
Arroyo has been acquired by Cloudflare to bring serverless SQL stream processing to the Cloudflare Developer Platfrorm, integrated with Queues, Workers, and R2. The Arroyo Engine will remain open-source and self-hostable.
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.14.0
announcement
Arroyo 0.14 is now available! This release introduces support for lookup joins, more powerful updating SQL, new syntax, structs in DDL, and more!
Micah Wylde CEO of Arroyo
Fast columnar JSON decoding with arrow-rs
engineering
JSON is the most common serialization format used in streaming pipelines, so it pays to be able to deserialize it fast. This post covers in detail how the arrow-json library works to perform very efficient columnar JSON decoding, and the additions we've made for streaming use cases.
Micah Wylde CEO of Arroyo
Building a near-real-time data lake with the LOAD stack
tutorial
The LOAD stack (log storage/object storage/Arroyo/DuckDB) makes it easy to build an affordable real-time data lake with minimal operational overhead. This tutorial will guide you through the process of setting up a complete system on AWS.
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.13.0
announcement
Arroyo 0.13 is now available! This release introduces support for reading source metadata, a RabbitMQ connector, improved CDC support, operator chaining, along with many other improvements.
Micah Wylde CEO of Arroyo
Talk: Latency, Throughput, Fault Tolerance
arroyo
Arroyo creator Micah Wylde recently spoke at P99Conf, discussing how Arroyo achieves low-latency and high-throughput while maintaining fault tolerance and fast recovery times
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.12.0
announcement
Arroyo 0.12 is now available! This release introduces Python UDFs, Protobuf ingestion, JSON syntax, custom state TTLs, and many other features, improvements, and fixes.
Micah Wylde CEO of Arroyo
Serverless Arroyo pipelines on Fly.io
tutorial
Arroyo is the easiest way to build real-time data pipelines, and Fly.io is the easiest way to run them. This tutorial shows how to use the new pipeline cluster feature in Arroyo 0.11 to build a streaming pipeline and a web app that consumes it, all running on Fly's serverless infrastructure.
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.11.0
announcement
Arroyo 0.11 is now available! This release introduces pipeline clusters for lightweight, self-contained job execution, and SQLite support for simplified deployments. It also brings a new configuration system, improved UI for pipeline creation and previewing, SQL enhancements, and more.
Micah Wylde CEO of Arroyo
How to build a plugin system in Rust
engineering
Software used by businesses often needs to be extensible. For Arroyo, a real-time SQL engine, that means supporting user-defined functions (UDFs). But how can we support dynamic, user-written code in a static language like Rust? This post dives deep into the technical details of building a dynamically-linked, FFI-based plugin system in Rust.
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.10.0
announcement
Arroyo 0.10 is now available! This is our biggest release ever, featuring an entirely new SQL engine that's >3x faster and ships as a single binary. Plus NATS and MQTT connectors, more SQL features, and more.
Micah Wylde CEO of Arroyo
We built a new SQL Engine on Arrow and DataFusion
announcement
Arroyo 0.10 has an entirely new SQL engine built with Apache Arrow and DataFusion. It's much faster, smaller, and easier to run. Read on for why and how we're making this change.
Micah Wylde CEO of Arroyo
Confluent & Arroyo: Partnering to Bring Real-time SQL to Kafka
announcement
We are excited to announce that Arroyo is now a Connect with Confluent Partner, making it easier than ever for Confluent customers to integrate with the Arroyo platform. Arroyo extends Kafka with powerful stateful stream processing support, enabling businesses to analyze their data in real-time using SQL.
Micah Wylde CEO of Arroyo
What is stateful stream processing?
explainer
Arroyo is a stateful stream processing engine—which means that it's able to remember information about previously seen events, enabling features like joins, windows, and aggregations. When should you choose a stateless or a stateful streaming system? And how do stateful engines like Arroyo and Flink mitigate the difficulty of dealing with large amounts of state?
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.9.0
announcement
Arroyo 0.9 is now available! This release introduces async UDFs, which allow users to use databases, services, and models from within their pipelines. It also brings support for joining update tables, more control over bad data handling, a redesigned connection profile editor, and more.
Micah Wylde CEO of Arroyo
2023 Year in Review
arroyo
It's been a big year for Arroyo! We launched the company, open-sourced the engine, and did 8 releases. Here's a look back at our very exciting 2023.
Micah Wylde CEO of Arroyo
Using Kafka with Rust
explainer
Apache Kafka is a distributed log that's a great fit for streaming applications, microservice architectures, and more. In this post, we will learn how to use Kafka with applications written in Rust.
Micah Wylde CEO of Arroyo
Parsing custom formats with UDFs
tutorial
User-defined functions (UDFs) allow users to extend Arroyo with new functionality by writing Rust code. In this tutorial, we'll walk through how to use UDFs to parse a custom data format: the Common Log Format used by Apache HTTP and other web servers.
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.8.0
announcement
Arroyo 0.8 is now available, with a new FileSystem source, Delta Lake sink, Redis sink, Avro support, global UDFs, and more.
Micah Wylde CEO of Arroyo
What is streaming SQL?
explainer
What does it mean to apply SQL—a batch-oriented query language—to streams of data that are never complete? Read on for a deep dive into streaming SQL in Arroyo and other engines.
Micah Wylde CEO of Arroyo
Running Arroyo on EKS
tutorial
The easiest way to run a highly-scaled production Arroyo cluster is on Kubernetes. Setting up a Kubernetes cluster used to be a daunting task, but services like Amazon EKS have made it much easier. This post will walk through how to set up an EKS cluster and deploy Arroyo to it.
Micah Wylde CEO of Arroyo
Can you replace Prometheus with a stream processor?
arroyo
Recent versions of Arroyo have added support for HTTP sources, and treating individual lines of a response as streaming messages. So I wondered: could we use Arroyo to directly process metrics?
Micah Wylde CEO of Arroyo
Announcing Arroyo 0.7.0
announcement
Arroyo 0.7.0 is now available, with custom partitioning for s3 writes, message framing, unnest, union, state compaction, and more.
Micah Wylde CEO of Arroyo
