Apache InLong Blog

Version 2.3.0 Released

Thu, 13 Nov 2025 00:00:00 GMT

Apache InLong (应龙) recently released version 2.3.0, which resolved 59 issues, including 3 major features and 50+ optimizations. Mainly completed the support for Transform capabilities in Sort Standalone module for Kafka/HTTP/CLS/ES subscription types, enabled horizontal scaling for Audit Store, and optimized the Transform SDK and DataProxy SDK. This version also optimizes the operational and maintenance experience for Apache InLong. Additionally, numerous other features were implemented in Apache InLong 2.3.0.

About Apache InLong

As the industry's first one-stop, all-scenario massive data integration framework, Apache InLong (Yinglong) delivers automated, secure, reliable, and high-performance data transmission capabilities. It enables businesses to rapidly build stream-based data analysis, modeling, and applications. Currently, InLong is widely used across industries such as advertising, payment, social media, gaming, and artificial intelligence, serving thousands of business use cases. It handles over a million billion records/day in high-performance scenarios and over a hundred billion records/day in high-reliability scenarios.

The core positioning of InLong revolves around three keywords: "one-stop," "all-scenario," and "massive data." For "one-stop," InLong aims to shield technical complexities by providing complete data integration and supporting services for out-of-the-box usability. For "all-scenario," it offers comprehensive solutions covering common data integration scenarios in big data ecosystems. For "massive data," its architecture leverages data pipeline layering, fully extensible components, and built-in multi-cluster management to stably support data scales beyond millions of billions of records/day.

Overview of Version 2.3.0

Apache InLong (应龙) recently released version 2.3.0, which resolved 59 issues, including 3 major features and 50+ optimizations. Key enhancements include:

Sort: Transform capabilities supported in Sort Standalone module for Kafka/HTTP/CLS/ES subscription types
Audit: Audit Store supports horizontal scaling, improving system scalability and stability
SDK: Performance and functional optimizations for Transform SDK and DataProxy SDK

This version also optimizes the operational and maintenance experience for Apache InLong. Other significant features are detailed below.

Dashboard Module

Added grouping and stream switching features to the audit page, improving usability
Fixed the issue of multiple API calls triggered by queries on the audit page, optimizing performance

Manager Module

Added comprehensive audit alarm rule management API, supporting more granular alarm policy configuration
Supported parsing transformation configurations into transformation SQL
Enabled configuration support for source fields in receivers, enhancing flexibility

Agent Module

The agent now supports parallel creation of sender connections to the DataProxy, significantly improving connection initialization efficiency
Added loading functionality for the agent_ext.properties configuration file to prevent personalized configurations from being lost when agent.properties is overwritten during upgrades

Sort Module

Sort Standalone module supports Transform functionality for Kafka/HTTP/CLS/ES subscription types
Deserialization process now supports returning the byte size of data in a single row, improving data processing visualization and accuracy
Upgraded the Pulsar SDK to version 4.0.3, enhancing stability and functional compatibility

SDK Module

TransformSDK now supports array index access; the WHERE clause supports the LIKE operator; the str_to_json function can convert KV format data to JSON format
SortSDK defaults to retrieving GroupId and StreamId from unified metadata if these cannot be obtained from the InLongMsgV0 protocol, enhancing compatibility
Optimized Golang SDK to fix potential data race issues, improving concurrency safety

Audit Module

Audit service now supports enabling and disabling custom caching, meeting performance requirements for different scenarios
Supports audit reconciliation by data stream group dimension, improving verification accuracy
Added end-to-end reconciliation alarm capabilities, enabling timely detection of abnormal situations
Added horizontal scaling for Audit Store, enhancing system load capacity

TubeMQ Module

Fixed TubeMQ image build failure, ensuring stable image generation

Key Features of Version 2.3.0

Dashboard supports audit data reconciliation based on data stream groups

This feature enables querying audit data based on data stream groups to achieve reconciliation functionality

Contributed by @wohainilaodou via INLONG-11894.

Audit Store supports horizontal scaling

When the scale of audit data reaches hundreds of billions, a single Audit Store may face performance pressure. This version introduces horizontal scaling capabilities to effectively enhance the system's capacity

Build a routing topology between Audit Store clusters and AuditId, GroupId, StreamId, supporting flexible routing rule configuration through regular expressions.
Each Audit Store writes audit data to the corresponding storage clusters (ClickHouse/StarRocks/MySQL) based on the routing topology.
Audit Service queries the corresponding audit data based on the routing topology.

Contributed by @doleyzi via INLONG-12009.

Sort supports dynamic Transform capabilities

InLong Transform enhances InLong's Sort distribution capabilities, adapting to complex and diverse data analysis scenarios on the distribution side, improving data quality and data collaboration, simplifying pre-processing operations before data analysis, and focusing on the business value of data. Supports multiple data format parsing (CSV, KV, Protobuf, Json/Bson, Avro, Yaml, XML, Parquet), supports 180+ functions and arithmetic/logical operators. Transform logic supports dynamic changes during Sort runtime without requiring flow interruption or restart, supports seamless migration between Sort tasks, enabling elastic scaling or resource scheduling.

Contributed by @luchunliang via INLONG-11958, INLONG-11937, INLONG-11902, INLONG-11888.

Future Plans

In version 2.3.0, we have enriched and improved our operational capabilities. Welcome everyone to use it. If you have more scenarios and requirements, or encounter any problems during use, please feel free to raise issues and PR. In future versions, the InLong community will continue to:

Support for collecting from more data sources
Real-time synchronization supports more data sources and data targets
Optimized issue of data duplication in weak network environments

We welcome contributions from developers interested in InLong!

Version 2.2.0 Released

Wed, 14 May 2025 00:00:00 GMT

Apache InLong (应龙) recently released version 2.2.0, which resolved 87 issues, including 3 major features and 80+ optimizations. Key enhancements focus on improving Agent stability and resource management in data collection, multi-language SDK support with performance optimizations, and enhanced traffic control capabilities for Sort. This version also optimizes the operational and maintenance experience for Apache InLong. Additionally, numerous other features were implemented in Apache InLong 2.2.0.

About Apache InLong

Overview of Version 2.2.0

Apache InLong (应龙) recently released version 2.2.0, which resolved 87 issues, including 3 major features and 80+ optimizations. Key enhancements include:

Agent: Improved stability in data collection and resource management capabilities
SDK: Multi-language support and performance optimizations
Sort: Enhanced traffic control capabilities

This version also optimizes the operational and maintenance experience for Apache InLong. Other significant features are detailed below.

Dashboard Module

Fixed page refresh issues after switching
Optimized user login verification logic

Manager Module

Added support for SQL data sources
Enhanced JDBC validation
Enabled configuration of Inlong Properties metadata for Pulsar data sources

Agent Module

Optimized instance lifecycle control mechanism
Introduced global instance count limits
Fixed duplicate file collection issues
Enhanced recovery capabilities in exceptional scenarios
Improved accuracy in log collection

Sort Module

Introduced traffic control for SortStandalone
Optimized escaping processing for KV/CSV formats
Added Exactly Once audit reporting for MySQL CDC

SDK Module

Go SDK: Added connection pool lifetime limits to improve stability in high-concurrency scenarios
Optimized JAVA SDK implementations
SortSDK: Shared a single PulsarClient across SortTasks to avoid performance bottlenecks from excessive clients

Audit Module

Supported CDC auditing for MySQL Binlog

TubeMQ Module

Fixed intermittent consumption interruption issues
Saved consumption offsets to local files

Key Features of Version 2.2.0

DataProxy SDK Refactoring

The original SDK provided two separate Sender objects for TCP and HTTP reporting protocols, with independent logic and high maintenance costs. Error returns were coarse-grained without detailed error categorization.

This version reconstructs the DataProxy Java SDK to reduce user complexity:

Decoupled Sender construction and initialization: Heavy operations (e.g., thread pool initialization, metadata fetching) are moved from the constructor to a new start() method, allowing faster object creation and explicit startup control.
TCP reporting with QoS: Three modes are introduced:
- QoS 0: SDK sends continuously without awaiting responses; DataProxy forwards requests silently.
- QoS 1: SDK awaits DataProxy acknowledgment before DataProxy forwards to MQ.
- QoS 2: SDK awaits MQ-level confirmation from DataProxy
Contributed by @gosonzhang via INLONG-11729 and INLONG-11720.

Enhanced Agent Stability and Resource Management

Agent now enforces a global instance limit (default: 100, configurable via agent.instance.limit). Requests to add tasks are rejected if they exceed this limit, preventing resource exhaustion.

Contributed by @justinwwhuang via INLONG-11760 and INLONG-11776.

Sort Traffic Control Enhancements

Previously, SortStandalone allocated up to 1/3 of resources to new tasks, leading to idle resource waste. Version 2.2.0 introduces:

Lazy-loaded Flow Controller: Physical resources are allocated only when a task's data stream is consumed.
Dynamic resource allocation: Resources scale based on actual traffic demand.
Failure handling: Blocks data flow for high-failure-rate tasks to prevent systemic degradation.

Contributed by @luchunliang via INLONG-11836 and INLONG-11841.

Future Plans

In version 2.1.0, we have enriched and improved our operational capabilities. Welcome everyone to use it. If you have more scenarios and requirements, or encounter any problems during use, please feel free to raise issues and PR. In future versions, the InLong community will continue to:

Supporting more data source connectors
Expanding real-time synchronization sources and targets
Optimizing SDK capabilities and usability
Improving Dashboard experience

We welcome contributions from developers interested in InLong!

Release 2.1.0

Fri, 03 Jan 2025 00:00:00 GMT

Apache InLong has recently released version 2.1.0, which has closed over 120 issue, including more than 4 major features and over 110 optimizations. The main accomplishments include Dashboard supports batch operation of nodes, Manager supports multiple scheduling engines, Agent supports COS data sources, Sort supports archiving dirty data through the InLong SDK. Simultaneously optimize the user experience of Apache InLong operations and maintenance. In Apache InLong 2.1.0 version, a large number of other features have also been completed.

About Apache InLong

As the industry's first one-stop, all-scenario massive data integration framework, Apache InLong provides automated, secure, reliable, and high-performance data transmission capabilities, enabling businesses to quickly build stream-based data analysis, modeling, and applications. Currently, InLong is widely used in various industries including advertising, payment, social networking, gaming, and artificial intelligence, serving thousands of businesses, with high-performance scenarios processing over hundreds of billions of records per day and highly reliable scenarios handling over tens of trillions of records per day.

The core keywords for InLong's project positioning are "one-stop," "all-scenario," and "massive data." For "one-stop," we aim to shield technical details, provide complete data integration and supporting services, and achieve out-of-the-box usability; for "all-scenario," we aim to offer comprehensive solutions covering common data integration scenarios in the big data field; for "massive data," we hope to leverage architectural advantages such as layered data links, fully extensible components, and built-in multi-cluster management to stably support even larger data volumes based on hundreds of billions of records per day.

2.1.0 Overview

Apache InLong has recently released version 2.1.0, which has closed over 120 issue, including more than 4 major features and over 110 optimizations. The main accomplishments include

Dashboard supports batch operation of nodes
Manager supports multiple scheduling engines
Agent supports COS data sources
Sort supports archiving dirty data through the InLong SDK.

Simultaneously optimize the user experience of Apache InLong operations and maintenance. In Apache InLong 2.1.0 version, a large number of other features have also been completed.

Dashboard Module

Support COS data source
Support batch operation of agents: restart, upgrade
Support exporting audit data as CSV files
Support sorting of audit data and comparison of differences
Support queries for all types of indicators
Support data preview field segmentation

Manager Module

Support COS data source
Support managing multiple scheduling engines: AirFlow、DolphinScheduler
Support dirty data management and querying
Support querying heartbeat information based on IP
Limit one IP to only belong to one cluster
Provide an API for querying of dirty data archiving

Agent Module

Support COS data source
Support quick startup and shutdown
Support starting multiple instances
Support data supplementation in chronological order
Optimize the logic of the Installer process guardian for Agent
Support supplementary recording based on local data time

Sort Module

Added Elasticsearch connector based on Flink 1.18
Support KV separation on Kafka Sink
Support audit data reporting
Tube Connector source supports dirty data archiving

SDK Module

Transform SDK adds 7 new functions
Add Dirty Data Archiving SDK

Audit Module

Audit Proxy increases metric reporting
Audit Store adds metric reporting
Audit Service increases metric reporting
Add asynchronous flush audit data interface

TubeMQ Module

Write the consumption offset information to a local file
Optimize the load balancing logic of the Go version SDK

Others

Pipeline supports parallel build
Support Manager to configure volumes

2.1.0 Feature Introduction

Dashboard supports batch operation of agents

This feature is mainly used for the operation of Inlong Agent: mainly for upgrading and restarting:

After finding the cluster in cluster management, select multiple nodes to operate on and click on batch operation.
Select the operation type and fill in the required parameters for the corresponding operation, then click OK.

This feature optimizes the operation and maintenance experience of Inlong: interface based operation eliminates the need to operate DB and increases the cohesion of Inlong:

Visual Agent version upgrade, which can be upgraded in batches and at regular intervals to control upgrade risks.
During agent fault recovery, this function can be used to quickly restart.

Thanks to @wohainilaodou for their contributions to this feature. For more details, please refer to INLONG-11187

Manager supports multiple scheduling engines

Previously, for offline data synchronization, Inlong only supported Quartz scheduling engine. This version has added two third-party engines: DolphinScheduler and AirFlow.

AirFlow engine

In order to facilitate the maintenance and expansion of AirFlow interface support in the future, AirflowApi interface and BaseAirflowApi abstract class have been designed, and subsequent extensions only need to be based on this foundation.
Implement a unified request class AirflowServerClient for the interface.
Add two interceptors in OkHttpClient: AirflowAuthInterceptor for unified authorization of interfaces; LoggingInterceptor is used for logging.

Thanks to @Zkplo for their contributions to this feature. For more details, please refer to INLONG-11400

DolphinScheduler engine

-Add the DolphinScheduler package to org.apache.inlong.manager.schedule -Add client and engine for DS, as well as util for operating open APIs for DS -Add pojo class for DS interaction

Thanks to @emptyOVO for their contributions to this feature. For more details, please refer to INLONG-11401

Agent supports COS data source

Create a new COS type node and fill in the corresponding bucket name, credential ID, credential key, and region.
Create a new COS type data source, select the corresponding node, IP, and file path.

This feature supports direct data collection from COS storage, and businesses do not need to download COS files locally for data collection. Thanks to @justinwwhuang for their contributions to this feature. For more details, please refer to INLONG-11187

Sort supports archiving dirty data through the InLong SDK.

Added the ability to report dirty data to specified GroupId and StreamId through the InLong SDK. Users can choose to export dirty data or consume it independently from Pulsar.

The following configuration needs to be added to the Connector:

'dirty.side-output.inlong-sdk.inlong-auth-key' = 'your auth key',

'dirty.side-output.inlong-sdk.inlong-auth-id' = 'your auth id',

'dirty.side-output.enable' = 'true',

'dirty.side-output.inlong-sdk.inlong-group-id' = 'target_inlong_group_id',
'dirty.side-output.inlong-sdk.inlong-stream-id' = 'target_inlong_stream_id',

'dirty.side-output.labels' = 'groupId=xx&streamId=xx&serverType=tube&dataflowId=xx',

'dirty.side-output.inlong-sdk.inlong-manager-addr' = 'xxx',

'dirty.side-output.connector' = 'inlong-sdk',

'dirty.ignore' = 'true',`

Thanks to @vernedeng and @fuweng11 for their contributions to this feature. For more details, please refer to INLONG-11481 and INLONG-11508

Future Plans

Support more data source collection capabilities
Enrich Flink 1.15, 1.18 Connector
Continuously enhance Transform capabilities.
Provide real-time synchronization support for more data sources and targets.
Optimize SDK capabilities and user experience
Optimize Dashboard experience

We also look forward to more developers interested in InLong to contribute and help drive the project's development!

Release 2.0.0

Sun, 20 Oct 2024 00:00:00 GMT

Apache InLong（应龙）has recently released version 2.0.0, which has closed over 315 issues, including more than 6 major features and over 96 optimizations. The main accomplishments include support for the Transform SDK and integrated into the ES Sink of Sort Standalone, OceanBase data source management, adaptive resource configuration for Sort tasks, HTTP output for SortStandalone, community documentation restructuring, and full-link support for offline synchronization.

After the 2.0.0 release, Apache InLong has added transform capabilities, improved support for the Agent Pulsar Source, enriched the capabilities and applicable scenarios of Sort, and optimized the display of the InLong Dashboard, addressing various issues encountered during the operation and maintenance of InLong.

About Apache InLong

From 1.0 to 2.0

In version 1.13.0, InLong added the underlying framework for offline synchronization tasks and supported Flink's stream-batch integration capability. Version 2.0.0 fixed several issues with the built-in scheduler, connected the front and back ends, and enabled the configuration of offline synchronization tasks on the Dashboard page, providing full process support for the configuration and management of offline synchronization tasks. Based on this capability, users can manage both real-time and offline synchronization tasks uniformly.

In historical versions, InLong's standard and lightweight architecture focused on data collection, reporting, and data storage in lakes, with relatively weak support for fine-grained operations on data. Therefore, after version 1.13.0, a very important task for the InLong community was to enhance support for Transform capabilities, allowing users to perform more flexible processing of data at any stage of data integration.

Transform is based on general SQL semantics and has now completed the framework for Transform capabilities, supporting over 180 custom Transform functions while ensuring the extensibility of Transform from a design perspective, allowing users to flexibly define new Transform capabilities.

In addition to the Transform feature, another focus of the InLong community has been the restructuring of community documentation. On one hand, we have filled in missing documentation and updated outdated content; on the other hand, we have reorganized the documentation according to user guidance, core system introduction, development guidance, and system management, providing better explanations from user, developer, and operational perspectives. With the restructured documentation, users can more easily utilize InLong and better understand its operational and management mechanisms, as well as quickly develop custom plugins to meet specific needs.

Overall, by the time version 2.0.0 was released:

InLong has achieved full-link support for offline synchronization tasks, possessing stream-batch integrated data processing capabilities.
InLong has greatly enriched the capabilities of T, laying the foundation for future support of ELT/EtLT Pipelines.
The optimization of documentation has made InLong more user-friendly, better attracting users to understand, use, and co-build InLong.

InLong now supports both standard and lightweight architectures, stream-batch integrated data synchronization capabilities, and flexible data Transform capabilities. Therefore, the community has decided to upgrade InLong to version 2.0.0, marking the official entry of Apache InLong into the 2.0 era.

2.0.0 Overview

Apache InLong has recently released version 2.0.0, which has closed over 315 issues, including more than 6 major features and over 96 optimizations, achieving the following:

Support for Transform SDK and integrated into the ES Sink of Sort Standalone
New capabilities for OceanBase data source management
Adaptive resource configuration for Sort tasks
HTTP output for Sort Standalone
Restructured and optimized community documentation
Stream-batch integration, support for offline synchronization capabilities

In addition to the above features, version 2.0.0 also:

Improved support for the Agent Pulsar Source
Enriched the capabilities and usage scenarios of Sort
Fixed some issues encountered during InLong operation and maintenance
Optimized the display and user experience of the Dashboard

Agent Module

Optimized the Pulsar Source implementation to fix issues with inaccurate consumption offsets.
Added support for data re-insertion filtering capabilities.
Introduced the ability to report Agent status.
Updated the implementations for Redis, Oracle, SQLServer, and MQTT data sources.

Dashboard Module

Added an offline synchronization configuration page for data synchronization.
Optimized the style and structure of data preview.
Introduced a heartbeat display page for cluster node management.
Added cluster name display to data source information.
Supported custom ASCII code options for source data field delimiters.
Merged metric items with other items on the module review page.
Enabled delete operations for cluster management and template management.
Fixed errors in data preview.
Added support for OceanBase data sources.

Manager Module

Added support for OceanBase data source management.
Introduced TubeMQ configuration capabilities for Sort Standalone.
Supported asynchronous installation of Agent and the display of Agent installation logs.
Enabled configuration for HTTP type Sink.
Supported paginated queries for detailed information on sorting tasks.
Data preview now supports KV data types, escape characters, and filtering Tube data by StreamId.
Added data filtering functionality.
Permission optimization: Regular users are not allowed to modify Group information if they are not the Owner.
Fixed issues with offline synchronization updates.
Resolved alignment issues in data preview fields.

SDK Module

Transform supports data partitioning using GroupBy semantics.
Transform can parse Map nodes in JSON or PB data.
Transform JSON data source supports multidimensional arrays.
Transform supports ELT functionality.
Transform supports configuration and parsing of Transform annotations.
Transform supports various data source types: JSON, PB, XML, YAML, BSON, AVRO, ORC, PARQUET, etc.
Transform supports arithmetic functions: ceil, floor, sin, cos, cot, tanh, cosh, asin, atan, mod, etc.
Transform supports date and time functions: year, quarter, month, week, from_unixtime, unix_timestamp, to_timestamp, etc.
Transform supports string functions: substring, replace, reverse, etc.
Transform supports common encoding and encryption functions: MD5, ASCII, SHA.
Transform supports numeral and bitwise operation functions: HEX, Bitwise.
Transform supports compression and decompression functions: GZIP, ZIP, etc.
Transform includes other common functions: case conversion, IN, NOT IN, EXISTS, etc.
DataProxy Java SDK: Shaded Native Library to reduce conflicts with other SDKs.
DataProxy Java SDK: Optimized sending jitter issue during metadata changes.
DataProxy CPP SDK: Improved memory management and optimized build scripts.
DataProxy CPP SDK: Supports multiple protocols.
DataProxy CPP SDK: Added message manager and optimized data reception capabilities.
DataProxy CPP SDK: Supports forking subprocesses in DataProxy CPP SDK.
DataProxy Python SDK: Updated build scripts and supports skipping the CPP SDK build step.

Sort Module

Adjust resources required for Sort tasks based on data scale.
Added support for OceanBase data source.
Flink 1.18 now supports Elasticsearch 6 and Elasticsearch 7 connectors.
SortStandalone Elasticsearch Sink supports Transform.
SortStandalone supports HTTP Sink and batch sorting.
Connector supports OpenTelemetry log reporting.
Optimized producer parameters for Kafka connector.
Added end-to-end test cases for Flink 1.15.

Audit Module

Supports global memory control for the audit SDK.
Optimized daily dimension audit data statistics.
Audit SDK allows custom local IP settings.
Unified audit aggregation interval range.
Resolved Protobuf version conflicts between Audit SDK and other components.

2.0.0 Feature Introduction

New Transform Capabilities

InLong Transform enhances InLong's ability to expand access and distribution capabilities by adapting to a wider range of data protocols and reporting scenarios on the input side, while accommodating complex and diverse data analysis scenarios on the output side. This improves data quality and collaboration, providing computing capabilities such as connection, aggregation, filtering, grouping, value extraction, and sampling, all decoupled from the computation engine.

It simplifies the pre-processing operations for users reporting data, lowers the barriers to data usage, and streamlines the pre-operations required before users can start analyzing data. The focus is on the business value of data, achieving the goal of making data "visible and usable."

Transform has a wide range of application scenarios. Here are some typical examples:

Data Cleaning: During the data integration process, Transform capabilities can effectively eliminate errors, duplicates, and inconsistencies in data from different sources, improving data quality.
Data Fusion: Combining data from different sources for unified analysis and reporting. Transform capabilities can handle various formats and structures of data, achieving data fusion and integration.
Data Standardization: Converting data into a unified standard format for cross-system and cross-platform data analysis. Transform capabilities help enterprises achieve data standardization and normalization.
Data Partitioning and Indexing: To enhance the performance of data queries and analysis, Transform capabilities can dynamically adjust field values for data partitioning and indexing, thereby improving data warehouse performance.
Data Aggregation and Calculation: In the data analysis process, Transform capabilities can perform complex data aggregation and calculations to extract valuable business information, covering multidimensional data analysis.

Main Features of Transform：

Support for Rich Data Protocols: Enables integration with a variety of data protocols.
Decoupled from Computing Engine: Allows flexibility in processing without being tied to a specific computing engine.
Support for Rich Transformation Functions: Provides a wide range of functions for data transformation.
Lossless and Transparent Changes: Ensures that changes can be made without data loss or noticeable impact.
Automatic Scaling: Supports dynamic scaling up and down based on workload.

Currently, Transform supports a variety of data formats and custom functions, allowing users to flexibly process data using SQL. Special thanks to contributors such as @luchunliang, @vernedeng, @emptyOVO, @ying-hua, @Zkplo, @MOONSakura0614, and @Ybszzzziz for their efforts. For more details, please refer to Transform SDK Issues.

Community Documentation Restructuring

With the continuous development of the InLong community, the capabilities of InLong are also constantly enhancing. However, there have been issues with missing or outdated community documentation. To address this, the InLong community has initiated a restructuring of the community documentation to better assist users in understanding and utilizing InLong.

Main Content Includes:

Optimized Document Structure: Better organization of document content.
Enhanced Quick Start Examples:
- Offline synchronization usage examples
- Transform SDK usage examples
- Data subscription usage examples
- HTTP message reporting usage examples
Improved SDK Documentation:
- DataProxy: C++, Java, Golang, Python SDKs, and HTTP data reporting manuals
- TubeMQ SDK: C++, Java, Golang SDK usage manuals
Enhanced Development Guidelines:
- Code compilation guidelines
- Documentation for data protocols of each component
- Documentation for extension development of each component
- REST API documentation
Improved Management Articles: Documentation on user management, approval management, tenant management, node management, cluster management, tag management, template management, and agent management.

Currently, the community documentation has seen significant improvements in usage guidelines, development guidelines, and management guidelines. Special thanks to contributors such as @aloyszhang, @fuweng11, @vernedeng, @luchunliang, @gosonzhang, @doleyzi, @baomingyu, @justinwwhuang, and @wohainilaodou for their contributions to the documentation improvement. For more details, please refer to the official website.

Support for OceanBase Data Source

OceanBase Database is a distributed relational database characterized by high availability and scalability, suitable for large-scale data storage and processing scenarios. InLong version 2.0.0 adds support for OceanBase data sources, allowing data to be imported from data sources into OceanBase.

Managing OceanBase data nodes is similar to MySQL. To create a new OceanBase node, you need to fill in the node name, type (OceanBase), username, password, address, and other key information.

To write data into OceanBase, you first need to create a data target of type OceanBase.

Then fill in the relevant information, including the name, data node information, database and table names of the data target, and the primary key information of the target table.

Thanks to @xxsc0529 for their contributions to this feature. For more details, please refer to INLONG-10700, INLONG-10701, INLONG-10704.

Dynamic Resource Calculation for Sort Tasks

The total resources (task parallelism) for Flink Sort Jobs come from the configuration file flink-sort-plugin.properties, meaning that all submitted sorting jobs will use the same amount of resources. When the data scale is large, resources may be insufficient, and when the data scale is small, resources may be wasted.

Therefore, dynamically calculating the required resource quantity based on data volume is a much-needed feature. InLong now supports dynamically calculating the total resource requirements for tasks based on data volume, involving two core pieces of data:

The data volume of the task: This relies on the audit system and is derived from the average data volume recorded by DataProxy over the past hour.
The processing capacity per core: This depends on the maximum number of messages per core configured in the flink-sort-plugin.properties file.

With these two pieces of data, the total resource requirements for a task can be calculated. This feature supports a switch to enable or disable it as needed.

Thanks to @PeterZh6 for their contributions to this feature. For more details, please refer to INLONG-10916.

SortStandalone Supports HTTP Sink

Inlong SortStandalone is responsible for consuming data from MQ and distributing it to various data storage modules, supporting multiple data stores such as ElasticSearch and CLS.

Compared to SortFlink, SortStandalone offers higher performance and lower latency, making it suitable for scenarios with high performance requirements.

HTTP is a widely used communication protocol, and SortStandalone supports HTTP output, allowing data to be sent to HTTP interfaces without worrying about specific storage implementations, thus adapting more flexibly to different business scenarios.

The processing flow for HTTP output is as follows:

HTTP output has the following features:

SortSDK is responsible for consuming data from MQ
Supports semaphore-based traffic control capabilities
Metadata management relies on Manager, supporting dynamic updates
The output protocol is HTTP, decoupling specific storage implementations
Supports retry strategies

Thanks to @yfsn666 and @fuweng11 for their contributions to this feature. For more details, please refer to INLONG-10831 and INLONG-10884.

Full-Link Management of Offline Synchronization

Version 2.0.0 adds full-link management capabilities for offline data synchronization tasks, with configuration methods for offline synchronization tasks similar to real-time synchronization. The specific process is as follows:

First, create the Group information for the synchronization task.

Note that the "Sync Type" should be selected as "Offline."

The second step is to configure the scheduling information for the offline task.

The conventional scheduling configuration requires setting the following parameters:

Scheduling unit: Supports minute, hour, day, month, year, and single execution (single execution means it will only run once)
Scheduling cycle: Represents the time interval between two task schedules
Delay time: Represents the delay time for task startup
Valid time: Includes start and end times; scheduled tasks will only run within this time range

In addition to conventional scheduling methods, Crontab configuration is also supported.

Crontab scheduling requires setting the following parameters:

Valid time: Includes start and end times; scheduled tasks will only run within this time range
Crontab expression: Represents the task cycle, for example, 0 */5 * * * ?

The third step is to create a Stream and configure data source and target information, which is consistent with real-time synchronization and will not be repeated here. For more details, please refer to Offline Synchronization Pulsar->MySQL.

Summary and Future Plans

Version 2.0.0 is the first version of the 2.x series, and the technical capability framework has been basically established. We welcome everyone to use it. If you have more scenarios and requirements, or encounter issues during use, please feel free to raise issues and PRs. In future versions, the InLong community will continue to:

Support more data source collection capabilities.
Enrich Flink 1.15 and 1.18 connectors.
Continuously enhance Transform capabilities.
Provide real-time synchronization support for more data sources and targets.
Advance offline integration, supporting third-party scheduling engines.
Optimize SDK capabilities and user experience.
Improve Dashboard experience.

We also look forward to more developers interested in InLong to contribute and help drive the project's development!

Release 1.13.0

Thu, 18 Jul 2024 00:00:00 GMT

Apache InLong recently released version 1.13.0, which closed about 275+ issues, including 6+ major features and 100+ optimizations. The main features include supporting SSH installation of agents, managing field templates, support configuring offline synchronization tasks, and collecting PostgreSQL through agents . After the release of 1.13.0, Apache InLong has enriched and optimized Agent function scenarios, enhanced the accuracy of Audit data measurement, and enriched the capabilities and applicable scenarios of Sort, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance.

About Apache InLong

As the industry's first one-stop, full-scenario, open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion lines per day, and the scale of high-reliability scene data exceeds 10 trillion lines per day.

The core keywords of InLong project positioning are "one-stop" and "massive data". For "one-stop", we hope to shield technical details, provide complete data integration and support services, and implement out-of-the-box; With its advantages, such as multi-cluster management, it can stably support larger-scale data volumes based on trillions of lines per day.

1.13.0 Version Overview

Agent Module

Support data version numbers to distinguish between normal data and supplementary data
Location storage supports plugins, currently supporting Rocksdb and Zookeeper
Support configuration version number comparison to prevent repeated configuration
Support minute level file collection
Add PostgreSQL and MongoDB data source collection

Manager Module

Support installing agents through SSH
Switch audit ID query from direct interaction with database to Audit SDK
Offline synchronization supports Pulsar -> MySQL
Support offline synchronous scheduling information management
File collection supports multi IP collection
Support obtaining Agent configuration information
Support automatic synchronization to Sink after modifying Stream field information
Support field template management
Data preview supports KV format
Data preview supports querying based on field filtering criteria

Dashboard Module

Add Source Data Field Template Page
Add monitoring and auditing page
Support installing agents through SSH key based authentication
Audit supports displaying total and variance audit data
File type data stream supports minute level cycles

Audit Module

Unified allocation and management of audit items using the Audit SDK
The Audit SDK supports automatic management of Audit Proxy addresses
Audit Store supports the universal JDBC protocol
Restarting the Audit Store optimization process may lead to data loss issues
Audit Service optimizes thread pool management
Audit Service compatible with historical audit data with empty Audit Tag
Audit Service Optimization for OpenAPI Audit Transmission Delay Calculation

Sort Module

Add JDBC connector on Flink 1.15
Added Pulsar connector on Flink 1.18
Redis connector supports reporting audit information
Kafka connector supports reporting audit information
MongoDB connector supports reporting audit information
PostgreSQL connector supports reporting audit information

1.13.0 Version Feature Introduction

Manager supports installing Agent by SSH

Through this feature, operation and maintenance personnel can install agents through the Dashboard, which currently supports SSH and manual installation methods. Users can create a new Agent cluster on the cluster management page.

Afterwards, enter the node, select the new node, and configure the SSH username and password to achieve SSH installation agent capability. Thanks to @haifxu and @fuweng11. For more information, please refer to INLONG-10409.

Manager supports field template management

With this feature, users can pre-configure field templates, and when creating a new Stream, they can select the pre-configured field templates, thus achieving the purpose of reusing configurations across multiple Streams. Thanks to @kamianlaida and @fuweng11. For more information, please refer to: INLONG-10330.

Support the construction of the underlying framework for configuring offline synchronization tasks

In version 1.13.0, Manager supports the configuration of offline synchronization tasks. Compared to real-time synchronization, offline data synchronization pays more attention to synchronization throughput and efficiency. Real-time synchronization tasks run in the manner of Flink stream tasks, while offline synchronization runs in the manner of Flink batch tasks. This approach can ensure the consistency of real-time and offline synchronization tasks' code as much as possible, reducing maintenance costs. The offline synchronization function of InLong will be combined with the scheduling system to synchronize the complete or incremental data of the data source information to the data target. The offline synchronization task is created by InLong Manager (including scheduling information), and the specific data synchronization logic is implemented through the InLong Sort module.

Key Competency:

Job Type: Support single or periodic offline data synchronization.
Scheduling:The scheduling function is plugin based, with a built-in simple periodic scheduling function that allows customization of third-party scheduling systems to support more complex capabilities, such as task dependencies.
Compute Engine: Flink
Offline Job Operation and Maintenance: Job start,stop and running status monitoring
Special Handling: Dirty Data Processing Capability

The following is the core process:

Thanks to @aloyszhang. For more information, please refer to: INLONG-10054, INLONG-10053, INLONG-10055, INLONG-10069.

Optimize the Sort Standalone configuration process

In version 1.13.0, the distribution process of Sort Standalone configuration was modified. In previous versions, there were the following issues with the distribution of Sort Standalone configuration:

Configuration changes are unreliable. Configuration changes will be updated in real-time to the manager's cache. Once the configuration is changed, Sort Standalone will sense it and write data based on the new configuration without verifying the configuration.
The configuration and construction process is repetitive and cumbersome. When pulling the Sort configuration from the database, it is a full pull and real-time build is performed after the pull is complete.

In the new version, after modifying the data target, the Sort configuration will not take effect in real time, but will need to be built and written into the sort_config table after executing the workflow. The following figure shows a process comparison:

Thanks to @fuweng11, @vernedeng. For more information, please refer to: INLONG-9867, INLONG-10017.

Agent ability for collecting data from PostgreSQL

In version 1.13.0, Agent supports data collection from PostgreSQL. When creating a data source, you can directly select PostgreSQL and fill all relevant data source information to start using it. The parameters include:

Data source name: Used to distinguish between different data sources
Cluster name: The cluster of data source belongs
Data source IP: The IP information of data source
Server Host: PostgreSQL Server Host
Port: PostgreSQL port

Thanks to @haifxu. For more information, please refer to: INLONG-10318.

Sort connectors reporting audit information supports exactly once

Sort Connectors support for exactly once semantics in reporting audit information. When an operator encounters an exception and the snapshot fails, each Connector ensures that the audit information is reported exactly once. As shown in the following diagram, when data is transferred, the current data will be written to the current operator's AuditBuffer and the corresponding checkpointId. When the checkpoint is completed (within the notifyCompleteCheckpoint method), the data written to the buffer will be reported.

During the execution of snapShot, the checkpointId written to the buffer will be changed to the checkpointId in the snapShot, and when the checkpoint is completed, the audit information corresponding to the checkpointId will be uploaded to the Audit Server within the notifyCompleteCheckpoint method. In the Audit SDK, it is already supportedto report all information in the Audit buffer that is less than the specified checkpointId.

Thanks to @XiaoYou201. For more information, please refer to: INLONG-10311, INLONG-10312, INLONG-10317, INLONG-10355, INLONG-10357, INLONG-10358, INLONG-10401.

Future plans

In version 1.13.0, the community refactored the process of Manager issuing Agent, Sort tasks, enriched features such as Flink 1.15 Connector, Inlong Agent collecting PostgreSQL and other functions. In subsequent versions, InLong will continue to enrich Flink 1.15 Connector, enhance Transform capabilities, support offline data integration, unify DataProxy data protocols, optimize Dashboard experience, etc. We look forward to more developers participating and contributing.

Release 1.12.0

Sun, 21 Apr 2024 00:00:00 GMT

Apache InLong recently released version 1.12.0, which closed about 140+ issues, including 7+ major features and 90+ optimizations. The main features include Manager supports for agent install package management and it's self-upgrading processe, Agent ability for self-upgrading process, Agent ability for collecting data from Kafka、Pulsar and MongoDB, Support for Redis connector in Sort module, Optimization for Audit and enhancement of its capabilities . After the release of 1.12.0, Apache InLong has enriched and optimized Agent function scenarios, enhanced the accuracy of Audit data measurement, and enriched the capabilities and applicable scenarios of Sort, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance.

About Apache InLong

1.12.0 Version Overview

Agent Module

Agent ability for self-upgrading process
Optimize initialization logic to reduce IO usage
Optimize message acknowledgment logic to reduce semaphore competition
Increase auditing for sending exceptions and resending
Optimize message recovery logic to avoid data loss caused by too many supplementary files

Manager Module

Add an agent installer module management for agent installation
Support parsing specific field information based on data types such as CSV while previewing data
Supports pulsar multi cluster while previewing data
Supports returning header and specific field information while previewing data
Support adding data and tasks for file collection
Audit data query switches from jdbc to Audit OpenAPI.
Support to set compression type in Pulsar DataNode
Provide OpenAPI for batch saving InLongGroup, InLongStream, and other information.
Support to config Kafka data node

Dashboard Module

Optimize audit data query
Optimize audit data display
Support for underscore "_" in Sink field mapping
Support paginated display of resource details
Support MongoDB data source configuration

Audit Module

Support user-defined ways to obtain Audit proxy information
Audit SDK supports reporting version numbers
Audit SDK supports both singleton and non-singleton usage
Audit SDK supports data reporting in Flink Checkpoint feature
Audit Service supports HA (High Availability) capabilities
Audit Service supports local caching and OpenAPI
Audit Service supports multi-datasource

Sort Module

Supports using state key during StarRocks connector sinitialization
Supports parsing KV and CSV data containing split symbols
Using ZLIB as the default compression type for Pulsar Sink
Pulsar Connector supports authentication configuration
Pulsar Sink supports authentication configuration
Redis Source supports String, Hash, and ZSet data types
Redis Sink supports Bitmap, Hash, and String data types

1.12.0 Version Feature Introduction

Manager supports for agent install package management and it's self-upgrading processe

In version 1.12.0, Operator can manage Agent installation packages through the Dashboard, including Agent installation, upgrade, heartbeat management, etc. Users can create/manage installation packages on the System Operation -> Installation Packages -> Agent page. Thanks to @haifxu and @fuweng11. For more information, please refer to: INLONG-9932.

Agent ability for self-upgrading process

Agents can perform self-upgrades through a pre-deployed Installer.The Installer will obtain the upgrade configuration information from InLong Manager via IP and determine whether to proceed with the upgrade based on the configuration. The main process includes:

ADD: Download the installation package -> Unzip the installation package -> Start the process
DELETE: Stop the process -> Delete the installation files
UPDATE: Download the installation package -> Stop the process -> Delete the installation files -> Unzip the installation package -> Start the process Thanks to @justinwwhuang. For more information, please refer to: INLONG-9801.

Agent ability for collecting data from Kafka

In version 1.12.0, Agent supports data collection from Kafka. When creating a data source, you can directly select Kafka and fill all relevant data source information to start using it. The parameters include:

Data source name: Used to distinguish between different data sources
Cluster name: The cluster of data source belongs
Data source IP: The IP information of data source
Bootstrap Servers: Kafka cluster address
Pulsar namespace: Pulsar namespace
Kafka topic: Kafka topic
Automatic offset reset: Set offset strategy
Partition offset: Set specific partition offset Thanks to @justinwwhuang. For more information, please refer to: INLONG-9741.

Agent ability for collecting data from Pulsar

In version 1.12.0, Agent supports data collection from Pulsar. When creating a data source, you can directly select Pulsar and fill all relevant data source information to start using it. The parameters include:

Data source name: Used to distinguish between different data sources
Cluster name: The cluster of data source belongs
Data source IP: The IP information of data source
Pulsar tenant: Pulsar tenant
Pulsar namespace: Pulsar namespace
Pulsar topic: Pulsar topic
Pulsar admin url: Pulsar admin url
Pulsar service url: Pulsar service url Thanks to @justinwwhuang. For more information, please refer to: INLONG-9804.

Agent ability for collecting data from MongoDB

In version 1.12.0, Agent supports data collection from MongoDB. When creating a data source, you can directly select MongoDB and fill all relevant data source information to start using it. The parameters include:

Data source name: Used to distinguish between different data sources
Cluster name: The cluster of data source belongs
Data source IP: The IP information of data source
Server host: MongoDB address
Username: MongoDB username
Password: MongoDB password
Database name: MongoDB database name
Collection name: MongoDB collection name
Read mode: Optional "Full + Incremental" or "Incremental" Thanks to @justinwwhuang. For more information, please refer to: INLONG-10006。.

Optimization for Audit and enhancement of its capabilities

In version 1.12.0, InLong enhancement the Audit reconciliation scenarios, including support for Agent data supplementation scenarios, Sort on Flink Checkpoint scenarios, etc. Thanks to @doleyzi. For more information, please refer to: INLONG-9904、INLONG-9926、INLONG-9928、INLONG-9957.

Support OpenAPI capabilities In version 1.12.0, Audit has supported the OpenAPI, and each OpenAPI can be elected as the leader through HA. The leader node is responsible for real-time and retroactive aggregation of audit data, and the aggregated results are saved in the DB. The slave node is responsible for caching the data in the DB to memory and providing services externally. The leader node also provides the same service.
Support Agent data supplementation capabilities In version 1.12.0, Audit supports the Agent data supplementation scenario by adding audit-version, which distinguishes the audit reconciliation for each supplementation.
Support Sort Flink Checkpoint capabilities In version 1.12.0, Audit supports the Sort Flink checkpoint scenario, ensuring that audit data is not lost or duplicated when the Flink job restarts or fails over.

Support for Redis connector in Sort module

In version 1.12.0, An additional Flink 1.15-based Redis connector implementation has been added, supporting read and write operations for String, Hash, ZSet, and Bitmap, four common data types in Redis clusters and standalone instances. Schema conversion is supported in the Redis connector, allowing users to specify a Schema that can be converted to different Redis data types. The specific Schema conversion logic is shown in the following figure. In the bitmap conversion logic in the figure below, field1 as the key for the bitmap, while filed2 and field4 as the position (index) in the Bitmap. filed3 and field5 represent the set values (0 or 1). For specifics, refer to the native Redis command SETBIT key index value. Thanks to @XiaoYou201. For more information, please refer to: INLONG-9835、INLONG-8948 .

Future plans

In version 1.12.0, the community refactored InLong Agent, InLong Audit, enriched Flink 1.15 Connector and other functions. In subsequent versions, InLong will continue to enrich Flink 1.15 Connector, enhance Transform capabilities, support offline data integration, unify DataProxy data protocols, optimize Dashboard experience, etc. We look forward to more developers participating and contributing.

Release 1.10.0

Wed, 13 Dec 2023 00:00:00 GMT

Apache InLong recently released version 1.10.0, which closed about 200+ issues, including 6+ major features and 30+ optimizations. The main features include Manager supporting viewing operation logs, supporting Group migration between clusters, Agent supporting periodic collection and task supplementation, Sort real-time synchronization supporting Transform, supporting MySQL to Iceberg whole database synchronization, and supporting automatic table creation. After the release of 1.10.0, Apache InLong has enriched and optimized Agent function scenarios, added the ability of whole database synchronization and automatic table creation, supported viewing operation logs, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance.

About Apache InLong

1.10.0 Version Overview

Agent Module

Support periodic collection and task supplementation
Global memory control to avoid OOM caused by large business data
Achieve final consistency between Agent and Manager tasks
Enrich Agent audit dimensions

Sort Module

Support MySQL to Iceberg whole database synchronization
Support richer Flink 1.15 Connectors: MongoDB, Iceberg, SQLServer, HBase
Support task status management, able to view real-time synchronization task status
Real-time tasks support automatic table creation
Support configuration of Transform

Manager Module

Support Sort Standalone management
Separate transmission protocol and data protocol types
Support Group cluster switching
Support query audit data size
Support viewing operation logs
Apache Iceberg supports automatic Schema information fetching

Dashboard Module

Support displaying resource creators and recent modifiers simultaneously
Support viewing operation logs
Approval directory displays Group corresponding to consumer groups
Support batch parsing of source fields

Audit Module

Support automatic creation of required Kafka Topics when the service starts

SDK Module

Resolve dependency version conflicts in DataProxy Java SDK
Update dependency version in DataProxy Go SDK
Use UUID instead of snowflake algorithm to generate data ID in DataProxy Go SDK
DataProxy C++ SDK supports dynamic load balancing
DataProxy C++ SDK supports multi-dimensional resource isolation
Optimize the ability of DataProxy C++ SDK to access nearby resources in multi-location deployment
DataProxy C++ SDK supports local disaster recovery
DataProxy C++ SDK supports dynamic update of DataProxy node information

Others

Strengthen protection against request forgery attacks
Update Snappy version
Add Master branch protection strategy

1.10.0 Version Feature Introduction

Agent Supports Periodic Collection

In version 1.10.0, InLong adds the ability to perform periodic file collection tasks. The periodic strategy includes daily, hourly, and real-time, and users can specify this strategy when creating a new file data source. At the same time, users can also configure time offsets to delay or advance collection. Thanks to @Justinhuang, @Blue

Support for viewing Agent audit by IP dimension

In order to better monitor the status of InLong Agent and quickly discover problems in live network operation, in version 1.10.0, users can view different Agent audit indicators based on the IP dimension in the system operation and maintenance -> audit module. Thanks to @fuwen11, @Bluewang, and @Justinhuang for their contributions, for more details, please see INLONG-9443, INLONG-9446, and INLONG-9458.

Manager adds Group operation logs

In version 1.10.0, InLong supports viewing operation logs, including the creation and modification of Group/Stream, and the addition and removal of Sink/Source operations. Operation logs can quickly help users track historical behavior, allowing users to quickly view key data stream operations for easier live network maintenance. Users can view all operation logs under a Group in the Data Access -> Group Details -> Operation Logs section.

New ability to switch Group clusters

In order to support resource integration and cost allocation capabilities, InLong introduced the Group cluster switching feature in version 1.10.0. Direct switching will inevitably lead to problems such as unsynchronized metadata between modules and data loss. To achieve seamless cluster switching for businesses, Group cluster switching is divided into three states and two steps. Thanks to @Vernedeng for the contribution, for more details, please see INLONG-9314. Before switching clusters, the data is on Cluster 1. After starting the cluster switch, the original routing configuration is copied, new data is written to Cluster 2, and the unsent data on Cluster 1 continues to be sent.

C++ SDK supports multi-dimensional isolation

In the old version of the C++ SDK, all Groups competed for the internal resources of the SDK. If a particular Group has a particularly large flow, it will inevitably squeeze other Groups, causing small flow Groups to be unable to obtain resources. In version 1.10.0, DataProxy C++ SDK supports resource isolation at the Cluster and Group levels, and users can enable or disable it by configuring enable_isolation. Thanks to @doleyzi for the contribution, for more details, please see INLONG-9213.

Future plans

In version 1.10.0, the community refactored InLong Agent, enriched Flink 1.15 Connector, and completed support for viewing operation logs and other functions. In subsequent versions, InLong will continue to enrich Flink 1.15 Connector, enhance Transform capabilities, unify DataProxy data protocols, support integration with Apache Paimon, optimize Dashboard experience, etc. We look forward to more developers participating and contributing.

Release 1.9.0

Mon, 25 Sep 2023 00:00:00 GMT

Apache InLong recently released version 1.9.0, which closed about 200+ issues, including 2+ major features and 30+ optimizations. The main improvements include building observability capabilities and optimizing DataProxySDK-CPP. After the release of version 1.9.0, Apache InLong has enhanced its observability capabilities in areas such as end-to-end tracing, metric collection, access and visualization, and alerting. This addresses the need for rapid problem diagnosis and performance optimization during development and operations, improving the user experience for Apache InLong's operation and maintenance.

About Apache InLong

Overview of version 1.9.0

Apache InLong recently released version 1.9.0, which closed about 200+ issues, including 5+ major features and 30+ optimizations. The main work completed in this version includes building observability capabilities, optimizing DataProxy C++ SDK, optimizing DataProxy metadata configuration updates, adding command-line tools for TubeMQ, automatically switching write methods for Iceberg, and supporting resource migration between tenants in Manager. After the release of version 1.9.0, Apache InLong has enhanced its observability capabilities in areas such as end-to-end tracing, metric collection, access and visualization, and alerting. This addresses the need for rapid problem diagnosis and performance optimization during development and operations, while also improving the user experience for Apache InLong's operation and maintenance.

In addition, Apache InLong 1.9.0 includes a large number of other features, mainly including:

Agent module

Support end-to-end log tracking and reporting, enhancing observability capabilities
Remove metric reporting deregistration logic during TaskManager initialization
Remove capacity limit for setting blacklist
Optimize Agent's JVM parameters

DataProxy module

Optimize metadata update logic
Optimize DataProxy metric statistics
Optimize retry logic after failed sending

Sort module

Data synchronization supports audit metric reporting
Iceberg supports dynamic switching between append and upsert modes
When writing data to Kafka, support writing to different partitions based on custom fields
PostgreSQL supports multi-concurrent reading
Doris write supports automatic Schema changes

Manager module

Support end-to-end log tracking and reporting, enhancing observability capabilities
Support data targets such as Tencent CLS, Pulsar cluster, Iceberg, and StarRocks
Support Iceberg and StarRocks as data sources on Flink 1.15
Improve multi-tenant capabilities: including tenant status validation before deleting a tenant, OpenAPI support for tenant parameters, and multi-tenant configuration support for data sources
ManagerClient supports paginated queries for data stream Source and Sink resources

Dashboard module

Support management of Pulsar data clusters and support output to Pulsar
Optimize the performance of tenant permission queries
Improve the usability of the interface display

Audit module

Add audit_tag information to distinguish between data sources and data targets
Optimize audit Proxy log output

SDK module

Optimize DataProxySDK-CPP, improving performance and reliability during network instability
SortSDK supports parallel creation of cache layer consumption
Optimize failure retry strategy for DataProxySDK-Java

TubeMQ module

Add TubeMQ command-line tool
TubeMQ Manager adds restart script

Others

Add ASF DOAP file
Add Mysql connector management image
Optimize third-party dependencies to address security risks

1.9.0 Feature Introduction

Observability Capability Building

In the application process of Apache InLong, the following scenario requirements are often encountered:

Locate the problematic code through detailed link call data (Tracing)
Query and analyze the abnormal module and associated logs to find the core error information (Logs)
Open the monitoring dashboard to find abnormal phenomena and locate the abnormal module through queries (Metrics)
Discover anomalies through various preset alarms (Metrics/Logs)

In version 1.9.0, @ZhaoNiuniu has contributed to the complete observability capability building of InLong based on OpenTelemetry. OpenTelemetry provides flexible, convenient, and rich Span customization capabilities, such as adding custom attributes, adding events, etc. By embedding points in the core positions of the code, the data can be displayed in the backend UI. The entire solution mainly includes the following steps:

The application pushes the Trace, Log, and Metric data collected through instrumentation to the otel collector using the otlp-exporter;
The otel collector collects and converts the data, then exports it to Jaeger, Prometheus, and Elasticsearch;
Grafana configures the three data sources for unified display, query, monitoring, and alerting.

Thanks to @ZhaoNiuniu's contribution, see INLONG-8611 and INLONG-8799 for details.

Optimize DataProxy C++ SDK

The older version of DataProxy C++ SDK was developed based on the C language, which had limited extensibility. The use of the singleton pattern resulted in performance limitations, and the handling of failed sending scenarios was unreasonable, which could easily trigger coredump. Therefore, the SDK has been optimized with the following improvements:

Adopt the C++ development mode, refactor DataProxy C++ SDK, no longer limited by the C language, enhancing the extensibility of the code;
A single process supports multiple SDK instances, allowing horizontal scaling by adding SDK instances, breaking through performance limitations;
Decouple data reception and data transmission, allowing separate configuration of the number of threads for receiving and sending, improving the SDK's scalability;
Decouple the SDK's data packaging logic from the business thread, reducing the impact on the business thread while improving the packaging performance;
Resolve the issue of the older version SDK causing coredump due to sending failures from the root.

With these optimizations, the DataProxy C++ SDK becomes more efficient, scalable, and reliable, providing better data processing capabilities for Apache InLong users.

Thanks to @doleyzi's contribution, see INLONG-8747 for details.

Optimize metadata update logic of DataProxy

DataProxy fetches metadata configuration from Manager, but sometimes due to network issues, misoperations, or other reasons, DataProxy may obtain incorrect metadata configuration from Manager. In version 1.9.0, DataProxy has added a protection mechanism for metadata configuration updates, enhancing the reliability of Apache InLong in extreme scenarios. The logic of the metadata configuration update protection mechanism is as follows:

When DataProxy receives a new metadata configuration from Manager, it first checks the configuration's validity, such as checking for null values, incorrect formats, etc., to ensure that the received configuration is correct and complete.
DataProxy compares the new configuration with the current configuration. If there are no significant differences, it ignores the new configuration and continues using the current one. This helps avoid unnecessary updates and reduces the impact on the system.
If the new configuration has significant differences from the current configuration, DataProxy will update its internal data structures and processing logic accordingly, ensuring that the system can adapt to the new configuration without causing errors or data loss.
In case the new configuration causes errors or issues, DataProxy will roll back to the previous configuration and report the problem to Manager, allowing the system to maintain its stability and reliability.

By implementing this protection mechanism for metadata configuration updates, Apache InLong can better handle extreme scenarios and ensure the reliability of its data processing capabilities.

Thanks to @gosonzhang's contribution, see INLONG-8758 and INLONG-8899 for details.

TubeMQ command-line tool

To enable Apache InLong operations and maintenance personnel to perform various operations simply, quickly, and flexibly in terminal windows in server management and DevOps scenarios, TubeMQ provides command-line tools to manage topics, produce and consume messages, and manage consumer groups through command-line parameters or options. For detailed usage documentation, please refer to TubeMQ Command-line Tools. The main features include:

Managing topics: creating, deleting, and querying topics, as well as setting topic attributes such as the number of partitions and replication factor.
Producing messages: sending messages to specified topics, with support for custom message content, key, and partition.
Consuming messages: consuming messages from specified topics and partitions, with support for specifying consumer groups and offsets.
Managing consumer groups: creating, deleting, and querying consumer groups, as well as setting consumer group attributes such as rebalancing and offset management.
Monitoring and statistics: collecting and displaying various statistics and metrics related to topics, producers, consumers, and consumer groups, helping users understand the system's performance and status.

By providing these command-line tools, Apache InLong allows operations and maintenance personnel to manage and operate TubeMQ more easily and efficiently in various scenarios.

$ bin/tubectl [options] [command] [command options]
$ bin/tubectl topic -h
$ bin/tubectl message produce
$ bin/tubectl message consume
$ bin/tubectl cgroup list

Thanks to @fancycoderzf's contribution, see INLONG-4972 for details.

Automatically switch write modes of Iceberg

Currently, the real-time synchronization from MySQL to Iceberg uses the upsert mode by default during the full data phase. The upsert implementation in Iceberg involves a delete operation followed by an add operation, with the delete operation resulting in the creation of a large number of delete files. This significantly impacts query efficiency. This optimization enables the task to use Append mode during the full data phase and automatically switch to Upsert mode during the incremental phase. The implementation involves the source carrying metadata fields in the data to determine which write mode to use in the Iceberg writer. When switching write modes, the data is refreshed, and the metadata fields are removed during the actual write operation.

Thanks to @lordcheng10 and @Emsnap's contribution.

Manager support resource migration between tenants

After InLong supports multi-tenancy, InLong Groups upgraded from older versions will be defined under the public tenant, while newly created InLong Groups will be defined within the tenant under the user's name. This leads to users frequently switching between multiple tenants to use resources. This optimization supports the migration of InLong Groups between different tenants, while also validating and automatically creating corresponding cluster labels, data nodes, and other resources.

Thanks to @vernedeng's contribution.

Follow-up planning

In the 1.9.0 version, the community has also refactored the DataProxy C++ SDK, enriched the Flink 1.15 Connector, and improved data synchronization features. In subsequent versions, InLong will continue to enrich the Flink 1.15 Connector, enhance the scheduling capabilities of data synchronization, and improve the Agent's file collection capabilities. We look forward to more developers participating and contributing.

Release 1.8.0

Mon, 24 Jul 2023 00:00:00 GMT

Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations. The main improvements include multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics.

About Apache InLong

Overview of version 1.8.0

Apache InLong recently released version 1.8.0, which closed about 190+ issues, including 6+ major features and 30+ optimizations, mainly completing multi-tenant management, support for multiple Apache Flink versions, data synchronization in Dashboard, data preview support, and optimization of ultra-long log processing logic. After the release of 1.8.0, Apache InLong has completed the layout of full-scenario data integration around data access, data synchronization, and data subscription, combined with multi-tenant management, multi-cluster management, approval flow management, and full-link audit/metrics. Apache InLong has built a comprehensive data integration solution, achieving out-of-the-box usability:

Data access: Data access is the process of aggregating data from data sources to the same storage service for further data querying and analysis;
Data synchronization: Data synchronization is the process of establishing consistency between data sources and target data storage, which can coordinate data over time;
Data subscription: Data subscription provides subscribers with the data they are authorized to access;

In version 1.8.0 of Apache InLong, many other features have also been completed, mainly including:

Agent module

Optimized ultra-long log processing logic, improving file collection efficiency and stability
Fixed the issue of thread leakage caused by task termination
Adopted flow control to solve the OOM problem caused by file number growth

DataProxy module

Support for Golang SDK
Support for configuring black and white lists based on full IP or CIDR format IP segments
Support for configuring the maximum number of write retries
Support for sending data to the default Topic when write fails
Code refactoring, unified configuration acquisition method

Sort module

Enhanced DDL parsing capability, improving the stability of DDL-aware scenarios
Support for multiple Flink versions
Support for Decimal precision recognition in whole-library scenarios
Hive supports whole-library migration, with the implementation consistent with MySQL whole-library migration
Iceberg supports automatic column updates and column deletions, greatly enriching Schema change capabilities

Manager module

Support for Pulsar, TubeMQ data preview
Support for dynamic configuration of audit data sources
Support for querying audit delay information
Support for multi-tenant management

Dashboard module

Support for data flow preview
Support for viewing InLongGroup resource details
Support for tenant management and tenant switching
Support for data synchronization

Others

Remove conflicting Jsqlparser versions
Upgrade Spring-Boot-Autoconfigure version to 2.6.15
Upgrade the Snappy-Java version to 1.1.10.1
Fix syntax errors in Workflow configuration files

1.8.0 Feature Introduction

Agent optimizes ultra-long log processing logic, improving file collection efficiency and stability

In actual use, due to improper use by users or bugs in data production programs, occasionally, a single data length reaches MB or even GB level. For Agents deployed in low-profile environments, this type of data greatly affects the performance of sending. The lower version Agent reads this type of data directly into memory based on the newline character and then discards it, but is limited by the hardware configuration of the Agent deployment environment, a single ultra-long data is extremely likely to cause OOM exceptions. In version 1.8.0, Agent optimized the processing logic of ultra-long logs, ensuring that data loading does not exceed memory limits through segmented collection and segmented discarding. Thanks to @justinhuang's contribution, see INLONG-8180 for details.

Agent adopts global flow control, solving the OOM problem caused by the growth of file numbers

In previous versions, each file was collected and sent by its thread. Although we limited the maximum collection buffer of each file, with the growth of user traffic, the increase in the number of files is inevitable, leading to the simultaneous collection of too many files, causing OOM exceptions due to memory overflow. InLong supports the feature of Agent configuring global flow control in version 1.8.0. With this feature, Agent can effectively avoid frequent OOM problems caused by the growth of file numbers or the use of small quota servers. Thanks to @justinhuang's contribution, see INLONG-8251 for details. If you need to use this feature, you can add the corresponding configuration in agent.properties.

Support multiple Flink versions

As community users go deeper into using InLong, the scenarios InLong faces become more diverse and complex. To support the needs of users in different Flink environments, InLong has added support for multiple Flink versions in the current version. Users can choose the Flink version to start in the plugins/flink-sort-plugin.properties configuration file in InLong-Manager.

When changing the Flink version required to run the Sort component, you also need to replace the connectors in the InLong-Sort/connector directory with the corresponding version of the jar package. For details, see the InLong official website documentation. Thanks to @Emsnap, @GanfengTan, and @haifxu for their contributions to this capability.

# inLong-manager/plugins/flink-sort-plugin.properties

# Flink version, support [1.13|1.15]
flink.version=1.13

Support multi-tenant management

To address the requirements for permission and resource isolation in multi-user scenarios, InLong introduces a multi-tenant architecture in the current version. The multi-tenant architecture ensures that data and permissions do not interfere with each other among different users within the same group of services. Thanks to @vernedeng and @bluewang for their contributions to this feature, see INLONG-7914 for feature details. The following is the core process:

Tenants are transparent to core logic developers. At the entrance of the request, tenant authentication is added, and requests without access to the tenant's permissions are directly rejected; before accessing the Database, the corresponding tenant filter conditions are added to ensure that the scope of data query and modification is limited within the tenant.

Users can create tenants and assign tenant roles on the Dashboard.

Support real-time synchronization

The new version supports real-time data synchronization. The main difference between real-time synchronization and data access is that it does not require the support of intermediate MQ storage. The Sort component directly stores the source data, greatly enriching the user's usage scenarios.

As shown in the figure below, the Tab page adds a "Data Synchronization" label. After the user configures the basic Group information, they only need to enter the "Data Source" and "Data Target" information, and after submitting the task, the data can be synchronized in real-time.

Thanks to @fuwen11, @bluewang, @Emsnap, and @haifxu for their contributions to this feature.

Support data preview

For businesses just accessing InLong, data preview can help users quickly confirm the accuracy of reported data and locate problems. In this version, the InLong front supports previewing users' real-time reported data. Thanks to @fuwen11 and @bluewang's contributions, users can choose data preview in the operation bar under the data stream after successfully creating a data stream and reporting data.

Support querying transmission delay

Transmission delay is crucial for some real-time consumption scenarios. In this version, InLong audit supports frontend viewing of average transmission delay indicators. Thanks to @fuwen11 and @bluewang's contributions, users can query link transmission delay after successfully creating a data stream and reporting data.

Follow-up planning

In version 1.8.0, the community also restructured the DataProxy code, unified the configuration pull interface, supported complete IP and CIDR-based IP segment configuration of black and white list features, and improved module performance and stability. Sort has improved stability in DDL sensing scenarios and supports whole library migration of Hive, Iceberg automatic column update, and column storage features. In subsequent versions, InLong will refactor DataProxy C++ SDK, enrich Flink 1.15 Connector, and improve data synchronization functions, looking forward to more developers participating in the contribution.

Release 1.7.0

Fri, 19 May 2023 00:00:00 GMT

Apache InLong recently released version 1.7.0, which closed about 150+ issues, including 3+ major features and 40+ optimizations. The main features include support for sending data directly to Kafka, MySQL all-database migration with schema change support, GH-OST awareness for MySQL all-database migration, the addition of 4 batch import modes (CSV, SQL, JSON, and Excel), simplification of command line tool for creating data stream configurations, and refactoring of the Dashboard layout.

About Apache InLong

As the industry's first one-stop open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion lines per day, and the scale of high-reliability scene data exceeds 10 trillion lines per day.

1.7.0 Overview

Agent Module

Support for sending data directly to Kafka, bypassing DataProxy
Agent optimization, improving file collection and transmission performance
Fixed the issue of Reader creation failure during MySQL data collection

DataProxy Module

Simplify common configuration and related control logic
Code optimization, cleaning up invalid configurations in ConfigManager

Manager Module

Add PostgreSQL and Redis data node management
Add heartbeat timeout status for data sources
Add 4 batch import modes: CSV, SQL, JSON, and Excel
Simplify command-line tools, including data stream creation logic
Support restarting and stopping data source tasks in data streams
Add connectivity tests for Redis and Kudu

Sort Module

Mask sensitive information of Flink SQL-related data sources/targets in logs
Optimize the logic for calculating object byte size and related metrics
Support extracting DDL and operations from raw data
Add rate control when writing to Iceberg
Support schema changes in MySQL full-database migration
In full-database migration, MySQL Connector supports detecting GH-OST DDLs

Dashboard Module

Add CSV, SQL, JSON, and Excel batch import pages for 4 import modes
Optimize Clickhouse flow configuration, supporting ttl/engine and other configurations
Refactor Dashboard layout, improving display experience
Optimize the creation process for data sources and data targets

Other

Fix multiple MySQL-related security vulnerabilities
TubeMQ Golang SDK supports production, completing the first phase of development
Optimize InLong development tool support for MacOS and Linux
Optimize Pulsar Client dependency to reduce installation package size

1.7.0 Feature Introduction

Support for sending data directly to Kafka

In previous versions, InLong supported sending data directly from Agent to Pulsar without going through DataProxy. With this design, users with simple data scenarios and who want to ensure data integrity as much as possible can reduce their dependency on DataProxy. For users accustomed to using Kafka, version 1.7.0 supports the feature of sending data directly from Agent to Kafka. Thanks to @wangpeix for the complete contribution, and details can be found in INLONG-7783. If you want to experience this feature, you can choose "Send to MQ, and respond after MQ receives" during the data stream approval process.

MySQL all-database migration with schema change support

As community users delve deeper into using InLong, the drawbacks of not supporting schema changes become increasingly apparent. After the source end changes the DDL, the task needs to modify the configuration and restart, significantly increasing operational costs. In the current version, InLong supports automatic schema change capabilities. Upstream data sources can perceive Create, Alter, Drop, Truncate, and Rename DDL operations and synchronize these operations downstream. Meanwhile, downstream data sources can respond to upstream DDL changes and process them accordingly, supporting different processing strategies. For more details, refer to INLONG-7553. Thanks to @Emsnap, @yunqingmoswu, and @lordcheng10 for contributing to this feature. The following diagram illustrates the core process:

In the database, DDL messages are perceived by Debezium in the CDC. At this point, the data obtained is a single DDL statement, such as "DROP TABLE A". This statement is a field in the Debezium JSON. The DDL statement is then parsed into a DDL model by JSQLParser. This model parses common DDL messages and processes them into a JSON format that is easy for the program to handle. The DDL model will be sent to the Sink Operator in Flink as data, and the Operator will process the DDL model.

GH-OST awareness for MySQL all-database migration

GH-OST (GitHub Online Schema Migration) is a trigger-free online schema migration solution for MySQL released by GitHub. It is testable and provides pause, dynamic control/reconfiguration, auditing, and many operational privileges. Throughout the migration process, it generates minimal workload on the primary server and is separated from the existing work on the migrated tables. By supporting GH-OST-aware DDL, the MySQL Connector can correctly handle table structure changes caused by GH-OST while capturing data changes. Thanks to @e-mhui for the complete contribution, and details of this feature can be found in INLONG-7554. The following diagram illustrates the core process:

First, after enabling the automatic DDL response for MySQL CDC, the ghc, gho, and del tables generated by GH-OST are also captured. Second, when perceiving the changes made by GH-OST to the gho table, the gho table in the DDL statement is replaced with the source table and stored in the state. Finally, after GH-OST completes the entire change process for the source table, the DDL statements previously stored in the state are sent downstream.

Addition of 4 batch import modes (CSV, SQL, JSON, and Excel)

When creating data stream input metadata fields, we need to enter information such as name, type, and description in sequence. If we need to input hundreds or thousands of field information, this processing method is extremely inefficient. In version 1.7.0, InLong has added four batch import modes for CSV/SQL/JSON/Excel formats. Users only need to refer to the template for each format and fill in the selected information to achieve one-time import. Many thanks to @featzhang and @fuweng11 for their participation in the development of this feature. The four batch import modes are now supported by both front-end and back-end, and you can download the latest version for direct use.

Simplification of command line tool for creating data stream configurations

In the before version, when creating a data stream using the command line, the JSON file content required was complex, and the file structure was not clear enough, making the threshold for users to create data streams through the command line very high. Additionally, when users wanted to reuse the file to create a new data stream, they had to modify many repetitive fields, such as inlongGroupID and inlongStreamID. In version 1.7.0, InLong has optimized the data stream configuration JSON structure and field configuration. Users can simply add Source/Sink content according to their data stream requirements, making creating a data stream much simpler than before. For more details, see INLONG-7778, and many thanks to @haifxu for the contribution. The following example is a template for creating a File -> Pulsar -> Clickhouse in the new version:

{
    "groupInfo": {
        "inlongGroupId": "test_group_ctl",
        "inlongClusterTag": "default_cluster",
        "mqType": "PULSAR"
    },
    "streamInfo": {
        "inlongStreamId": "test_stream_ctl",
        "fieldList": [
            {
                "fieldName": "name",
                "fieldType": "string"
            }
        ],
        "sourceList": [
            {
                "sourceType": "FILE",
                "sourceName": "test_source_ctl",
                "agentIp": "127.0.0.1",
                "pattern": "/data/test.txt"
            }
        ],
        "sinkList": [
            {
                "sinkType": "CLICKHOUSE",
                "sinkName": "test_sink_ctl",
                "dataNodeName": "test_clickhouse",
                "dbName": "db_test",
                "tableName": "table_test",
                "flushInterval": 1,
                "flushRecord": 1000,
                "retryTimes": 3,
                "engine": "Log",
                "isDistributed": 1,
                "sinkFieldList": [
                    {
                        "sourceFieldName": "name",
                        "sourceFieldType": "string",
                        "fieldName": "name",
                        "fieldType": "string"
                    }
                ]
            }
        ]
    }
}

Refactoring of the Dashboard layout

In version 1.7.0, the community has refactored the overall layout of the Dashboard, including adjusting the top-bottom layout to a left-right layout (moving the navigation bar to the left), adding a dark theme, adding icons to the main menu, adjusting the data source selection display and process, etc. This adjustment has improved the user experience of the Dashboard. Special thanks to @leezng and @bluewang for their contributions. For more details, see INLONG-7734.

Follow-up planning

In version 1.7.0, the community has also improved the performance and stability of Agent file collection, while TubeMQ has completed the first phase of the Golang SDK production. Additionally, Sort can now consume using the subscription groups allocated by the Manager. In subsequent versions, InLong will support multiple Apache Flink versions, including not only the current Flink 1.13 but also Flink 1.15. Furthermore, tenant management will be added to unify the models of InLong projects, users, and resources. We look forward to more developers participating and contributing.

Release 1.6.0

Thu, 23 Mar 2023 00:00:00 GMT

Apache InLong recently released version 1.6.0, which closed about 202+ issues, including 11+ major features and 80+ optimizations. Mainly completed the addition of Kudu data stream, improvement of Redis data stream, the addition of MQ cache cluster selector strategy, optimization of Audit ID allocation rules, the addition of data node connection testing, optimization of Sort Audit reconciliation benchmark time, and expansion of Audit support for using Kafka to cache audit data.

About Apache InLong

1.6.0 Overview

Agent Module

Improved the file collection stability and fixed multiple collection bugs
Fixed multiple bugs such as MQTT, MongoDB

DataProxy Module

Added MQ cluster Selector strategy to reduce the number of producers
Added Audit report for the new MQ Sink

Manager Module

Optimized the Audit ID distribution rules and supported multiple Load Data Node audit
Optimized Clickhouse data node metadata configuration and management
Added connection test for new Data Node, check the availability of nodes
Added Pulsar Multi-Cluster Topic and Subscription Management
Fixed multiple data stream management and status management bugs

Sort Module

Added data Audit for Kafka Source connector
Added new CSV format and dirty data archive for Doris connector
Supported Array, Map, Struct, and other complex types
Optimized Pulsar Connector to solve the issue of data loss
Fixed the writing in chaos for Canal-JSON metadata fields
Optimized reconciliation benchmark time for Sort Audit

Dashboard Module

Continue to optimize the Dashboard experience and reduce the threshold for first users
Added node management for Redis, Kudu, and other data nodes
Optimized data node parameters such as PostgreSQL, Kafka, Redis, etc.
Simplified the Agent node IP selection strategy
Added connection test pages for data nodes

Other

Supported using Kafka as cache MQ for Audit
Audit uniformly obtains the MQ cluster from Manager
Optimized deployment steps such as Standalone, Docker-Compose, Kubernetes, and other deployment steps

1.6.0 Feature Introduction

Supported Kudu data stream

Apache Kudu is an open-source storage engine by Cloudera, which can provide low-delayed random read and write and efficient data analysis capabilities simultaneously. In version 1.6.0, InLong supports the Kudu data stream, including adding Kudu Connector, metadata management, metrics, etc. Kudu data stream contributed by @featzhang independently. Interested users can make an installation and experience.

Improved the Redis data stream

Redis is a viral open-source memory database with high performance and rich data structure. In version 1.6.0, InLong perfects the Redis data stream, adding SinkFunction, metadata management, indicators, and Dashboard pages in Redis Connector. Supported data formats like Redis's PLAIN, Hash, and Bitmap and realized Redis Schema conversion through the SCHEMAMAPPING mechanism. Redis data streams through the SCHEMA mapping mode, and SCHEMA can be converted into different [Redis Data-Type] (https://redis.io/docs/data-types/tutorial/). Redis data stream is mainly contributed to and fulfilled by @featzhang independently. For details, please refer to [INLONG-7060] (https://github.com/apache/inlong/issues/7060).

Added MQ cluster Selector strategy

In the multi-MQ cluster scene, if DataProxy is connected to all MQ clusters simultaneously, the number of producers in the MQ cluster will surge. At the same time, the amount of metadata of ZooKeeeper is excess, which will cause OutOfMemory. In version 1.6.0, InLong increased the selector strategy of the MQ cache cluster level (mainly for Apache Pulsar). As a result, the DataProxy node can only choose some MQ clusters under the same tag for production, thereby reducing the number of producer connections and Zookeeper metadata. @Luchunliang mainly develops this feature. For details, please refer to [INLONG-7231] (https://github.com/apache/inlong/pull/7236).

Optimized the Audit ID distribution rules

In the original design of InLong Audit, the receiving and sending number of each module is an independent audit ID, which is used to record the receiving and sending number for each module. There is a defect in this scheme. If InLong Sort sorted data to multiple targets at the same time (such as writing Hive and Clickhouse), the audit ID cannot distinguish different data streams for the Sort. In version 1.6.0, the Audit ID distribution rules are optimized, the different data stream has a different audit ID, achieving data audit of multiple sorting targets for the same data stream. This feature also involves changes in Manager and Sort module. It is developed and implemented by @FuWeng11 and @EMSNAP. For details, please refer to [INLONG-7389] (https://github.com/apache/inlong/pull/7390), [INLONG-7232] (https://github.com/apache/inlong/pull/7233), and [INLONG-7503] (https://github.com/apache/inlong/pull/7552).

('audit_sort_hive_input', 'HIVE', 0, '7'),
('audit_sort_hive_output', 'HIVE', 1, '8'),
('audit_sort_clickhouse_input', 'CLICKHOUSE', 0, '9'),
('audit_sort_clickhouse_output', 'CLICKHOUSE', 1, '10'),
('audit_sort_es_input', 'ELASTICSEARCH', 0, '11'),
('audit_sort_es_output', 'ELASTICSEARCH', 1, '12'),
('audit_sort_starrocks_input', 'STARROCKS', 0, '13'),
('audit_sort_starrocks_output', 'STARROCKS', 1, '14'),
('audit_sort_hudi_input', 'HUDI', 0, '15'),
('audit_sort_hudi_output', 'HUDI', 1, '16'),
('audit_sort_iceberg_input', 'ICEBERG', 0, '17'),
('audit_sort_iceberg_output', 'ICEBERG', 1, '18'),
('audit_sort_hbase_input', 'HBASE', 0, '19'),
('audit_sort_hbase_output', 'HBASE', 1, '20'),
('audit_sort_doris_input', 'DORIS', 0, '21'),
('audit_sort_doris_output', 'DORIS', 1, '22')

Added connection test for new Data Node

In the previous version, InLong added data nodes and registered MQ clusters, and did not judge the availability of the cluster. In the new version, InLong adds a connection test for the main data node and InLong system component registration, which enhances the data stream creation. This feature is mainly participated in development by @leosanqing, @Bluewang, and @Fuweng11.

Optimized reconciliation benchmark time for Sort Audit

Before version 1.6.0, the Sort reconciliation benchmark time is the machine time for data processing. Using this time will cause the full-link reconciliation data to be inaccurate. In this version, the Sort audit reconciliation referred to the design of TimestampedCollector in Apache Flink, replaced the Simple Collector in the original Pulsar Connector, and aligned the audit index. The implementation principle is to set the Timestamp field at the Collector. When obtaining the InLongMsg data transmitted by DataProxy, the reset data time is used as the Audit time record metric when sending the message downstream. The optimized Sort audit index can be aligned with other modules, such as DataProxy, this feature was mainly developed by @Emsnap.

Supported using Kafka as cache MQ for Audit

InLong Audit is an independent subsystem that conducts real-time audit reconciliation of the InLong system's Agent, DataProxy, and Sort module's inflow, and outflow. The current amount of audits is minute, hour, and day. In the previous version, InLong Audit only supports the use of Pulsar to cache audit data, and this will increase the cost of user deployment when they chose Kafka. In the entire InLong Audit design, the choice of MQ type should be consistent with the data stream to avoid different use of different use MQ types. In version 1.6.0, in order to achieve the use of the same type of MQ service in Audit modules and data stream, Audit supports the use of Kafka cache audit data to achieve the unity of MQ service selection. This feature is mainly completed by @haifxu and @dockerzhang.

Follow-up planning

In 1.6.0, the Sort module also fixes multiple bugs such as dirty data archives, metrics, and Connectors. The Dashboard continues to optimize display and approval processes and other experience problems. For more details, please refer to the Release [Changelog] (https: // github. com/Apache/InLong/Blob/Master/Changes.md). In the subsequent version, Apache InLong will add Schema dynamic change, Schema batch import, agent installation, adding more data nodes, looking forward to more developers participating in contributions.

Release 1.5.0

Fri, 13 Jan 2023 00:00:00 GMT

Apache InLong recently released version 1.5.0, which closed about 296+ issues, including 12+ major features and 110+ optimizations. Mainly completed the addition of StarRocks, Hudi, Doris, Elasticsearch, and other sinks, optimization of the Dashboard experience, refactor the MQ management model, support dirty data processing, full-link Apache Kafka support, and TubeMQ C++/Python SDK support for production, etc.

About Apache InLong

As the industry's first one-stop open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion/day, and the scale of high-reliability scene data exceeds 10 trillion/day sky.

The core keywords of InLong project positioning are "one-stop" and actual "massive data". For "one-stop", we hope to shield technical details, provide complete data integration and support services, and realize out-of-the-box; With its advantages, such as multi-cluster management, it can stably support larger-scale data volumes on the basis of trillions/day.

1.5.0 Overview

Apache InLong recently released version 1.5.0, which closed about 296+ issues, including 12+ major features and 110+ optimizations. Mainly completed the addition of StarRocks, Hudi, Doris, Elasticsearch, and other data stream sinks, optimization of the Dashboard experience, reconstruction of the MQ management model, addition of dirty data processing, full-link Apache Kafka support, and TubeMQ C++/Python SDK support for production, etc. This version has also completed a large number of other features, mainly including the following:

Agent Module

Support log collection in CVM scenarios
Added direct sending Pulsar, sending DataProxy synchronous and asynchronous strategies

DataProxy Module

Refactor the MQ management model to support the rapid expansion of new MQ types
Optimized caching layer to support Apache Kafka message queue
Added support for BufferQueueChannel

TubeMQ Module

Increase data sending and receiving delay statistics
TubeMQ C++ SDK supports the produce
TubeMQ Python SDK supports the produce

Manager Module

Added Hudi data node and data stream management
Added StarRocks data node and data stream management
Optimize Elasticsearch data node and data stream management
Added data conversion management in Manager Client
Optimize Apache Kafka message queue management

Sort Module

The MySQL Load node inventory phase supports concurrent reading of tables without primary keys
Added StarRocks, Hudi, Doris, Elasticsearch 5.x data flow support
Add dirty data processing for Doris, PostgreSQL, Hive, HBase, Elasticsearch, etc.
Upgraded Iceberg to version 1.1.0
StarRocks, PostgreSQL, Doris, Hudi and other flows support table-level indicators

Dashboard Module

Experience optimization with more than 50 optimization points
Add JSON, Key-Value, and AVRO formats
Support ClickHouse, Iceberg, Elasticsearch, MySQL, and other data node management pages
Added SQLServer, Oracle, MongoDB, and MQTT data source pages

Other

Add Spotless code formatting plugin and response pipeline
Docker-compose comes with Apache Flink environment
Added Grafana indicator display templates for Agent and DataProxy

1.5.0 Feature Introduction

Support StarRocks, Hudi, Doris, Elasticsearch Sinks

In version 1.5.0, InLong expanded the new data node Connector, supported StarRocks, Hudi, Doris, Elasticsearch, and other flow directions for community user scenarios, and expanded the data warehouse and lake scenarios. These new data nodes are mainly contributed by @liaorui, @featzhang, @kuansix, @LvJiancheng, and other developers.

Optimization of the Dashboard Experience

Compared with traditional data integration projects, InLong has added concepts such as Group, Stream, and data nodes. Community users using Dashboard for the first time will be confused about the whole process. To reduce the cost of using Dashboard users, InLong has made a lot of optimizations for the Dashboard front-end page, with more than 50 optimization points, and adjusted the concept, process, and display. The figure below shows the process of creating a Stream in 1.5.0, which is more simplified than the previous version. Special thanks to @leezng, @bluewang, @kinfuy for optimizing the Dashboard and @Charles Zhang for the modification suggestions.

Refactor the MQ Management Model

To quickly support new message queue services (such as RocketMQ) to implement plug-ins, and unify the existing support for Pulsar, Kafka, and TubeMQ, in version 1.5.0, InLong DataProxy refactored the MQ management model, and all MQ types are based on MessageQueueHandler Implement the corresponding Handler. Thanks to @woofyzhao and @luchunliang for the implementation of this feature. If you need to develop a new MQ type, you can refer to the DataProxy plug-in guide.

Support Dirty Data Processing

If there is dirty data that does not meet the data specifications (such as field range exceeding, missing data fields, etc.) when entering the lake into the warehouse, it may cause the user task to fail to write and restart continuously. In version 1.5.0, InLong supports storing unrecoverable dirty data in external storage, including S3 and local logs. At the same time, users can customize the output port of dirty data and can configure "whether to enable dirty data archiving" and "whether to ignore write entry error", as follows to design a UML diagram for dirty data archiving. The realization of this feature is thanks to the support of @yunqingmoswu and @Yizhou-Yang.

Support Apache Kafka Full-link

In version 1.5.0, the DataProxy, Manager, Sort, and Dashboard modules have completed the full-link support for Apache Kafka. The support for Kafka has gone through two versions, and it is available for production in 1.5.0. When users create data streams Just choose Kafka. The implementation of this feature is thanks to @woofyzhao, @fuweng11, @haifxu for their support.

For more details on the 1.5.0 release, please refer to the release notes, which detail the features, enhancements, and bug fixes for this release.

Follow-up planning

In the following versions, Apache InLong will add multi-tenant management, standardize data flow, resources, and permissions of projects, clusters, and users, and optimize the performance and stability of various data sources, Agent management, etc., expect more developers to participate and contribute.

Release 1.4.0

Wed, 16 Nov 2022 00:00:00 GMT

Apache InLong is a one-stop integration framework for massive data that provides automatic, secure and reliable data transmission capabilities. InLong offers great power to build data analysis, modeling and other real-time applications based on streaming data.

1.4.0 version overview

Apache InLong recently released version 1.4.0, which closed about 364+ issues with 16+ features and 120+ optimizations. It mainly completes the real-time synchronization of the entire database to Apache Doris and the real-time synchronization of the entire database to Apache Iceberg, the standard architecture supports HTTP reporting, and the standard architecture adds MongoDB and other collection nodes. This release also completes a number of other features, including:

Agent Module

Refactored sink-sending metrics
Audit report increases data size
Support Redis, MQTT, SQLServer, Oracle, and MongoDB data sources
Enhanced Kubernetes environment file collection capability

DataProxy module

Heartbeat reporting adds service status and supports authentication
Added proxy-send mode to send data
Optimized data link buried point indicators

TubeMQ module

Added client load balancing consumer group control API
C++ SDK fixes multiple bugs

Manager module

Data stream Group and Stream support extended parameters
Client supports updating and deleting data flow through Key
Refactored the way to obtain Sort cluster configuration information
Optimized state management
Client supports cluster addition, deletion, modification, and query
Cluster nodes report new protocol types
Cache layer usage supports using Kafka

Sort module

Support debezium-json format
Kafka data nodes support topic dynamic awareness
Connectors such as Hive/Hbase/Iceberg support indicator status recovery
Elasticsearch 6/7, JDBC connector added indicator status
Iceberg sink supports schema revolution, can automatically build tables, and perceive the increase of fields

Dashboard module

Unified data source, data flow type definition
Added Agent type for cluster management
Add data node management
Support selection of Kafka message type

Other

docker-compose deploys the built-in Flink environment
Fix multiple aarch64 mirror bugs
Fix multiple dependency security bugs

1.4.0 version feature introduction

Agent adds a variety of data sources

In version 1.4.0, Agent supports data sources such as Redis, MQTT, SQLServer, Oracle, MongoDB, etc., so that the collection capabilities of standard architecture and lightweight architecture are basically aligned and users have more choices in massive scenarios. The support of this part of the back-end capabilities is mainly completed by @iamsee123 and @haibo-duan, and the front-end part is completed by @bluewang.

Improve component metrics

In version 1.4.0, the Agent, DataProxy, and Sort modules all have indicators optimized and improved, including the reconstruction of the indicators sent by the Agent, increasing the dimension of the data Group/Stream indicators, and fixing the inaccuracy of the Prometheus Listener indicators. Thanks to @Keylchen, @pocozh, and others for their contributions.

Optimize Docker-compose deployment

There are many InLong service components, and there has always been a problem with high deployment thresholds. In version 1.4.0, the compatibility of docker-compose deployment is optimized, and an Apache Flink environment is built-in to help developers quickly start creating tasks. Thanks to @dockerzhang for optimizing this part.

Optimize heartbeat management

In version 1.4.0, a lot of optimizations have been made for the heartbeat of service components, including adding data protocol when reporting, automatic registration of Agent/DataProxy component reporting, adding heartbeat management API to Manager, and optimizing multiple heartbeat status bugs. Thanks to @gosonzhang, @GanfengTan, @pocozh, @lucaspeng12138 and @haifxu for their contributions.

Support real-time synchronization of the entire database

In version 1.4.0, InLong began to support real-time synchronization of the entire database to follow up on the needs of community users. Currently, it is the first to achieve real-time synchronization of the entire database to Doris and real-time synchronization of the entire database to Iceberg/Kafka/Doris. In the near future, the community will also realize the synchronization of the entire database. Share the details. Thanks to @thesumery, @EMsnap, @yunqingmoswu for their contributions.

For more details on the 1.4.0 release, please refer to the release notes, which detail the features, enhancements, and bug fixes for this release.

Follow-up planning

In the next version, the community will continue to add synchronization scenarios for the entire database, improve task indicators, increase system stability, and conduct stress tests on standard architectures and lightweight architectures.

Release 1.3.0

Mon, 05 Sep 2022 00:00:00 GMT

Apache InLong is a one-stop integration framework for massive data that provides automatic, secure and reliable data transmission capabilities. InLong supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time applications based on streaming data.

1.3.0 Features Overview

The just-released 1.3.0 version closes about 410+ issues, contains 110+ features and 170+ optimizations. Mainly include the following:

Enhance management and control capabilities

Added permission authentication for Open Api
Added cluster heartbeat mechanism for Agent and DataProxy
Manager adapts to two roles such as user and approver
Abstract operations on Load nodes to support easy scaling of Load node resources
Supports creation of databases and tables for SQLServer, Oracle and MySQL
Enhanced functionality of the Manager client, including but not limited to user and data node management

Extended collection node

Support for collecting data in TubeMq
Support for collecting data in Redis
Support for collecting data in Doris
Support to collect data in Pulsar without AdminUrl

Optimize write node

Kafka Sink supports All Changelog Mode
JDBC Sink supports All Changelog Mode

Support data conversion

Support Union operator
Support encrypted Udf
Support Json Udf
Support Temporal Join
Support Lookup Join
Support Interval Join

Strengthen Agent function

Support regular expression custom line break: default "\n" line ending mark, custom regular matching line ending mark can realize multi-line merging and folding
Support K8s log collection and carry cluster information
Supports standard output, node log collection, and will carry container and cluster information for standard output
Support full and incremental collection of file content
Supports automatic heartbeat reporting and registration to Manager
Support custom IP and get IP automatically

Other optimizations

GitHub Action check, pipeline optimization
DataProxy improves monitoring capabilities such as auditing and indicator reporting
DataProxy adds c++ sdk data reporting capability
Sort Support metrics report and audit report

1.3.0 Features Details

Abstracting Load node operations

Manager abstracts Load nodes to support easy expansion of Load node resources and greatly reduce the development time of a Load node This part of the feature was contributed by @ciscozhou

Add permission authentication for Manager Open Api

In the old version, the Manager Open Api can be accessed anonymously, and in the new version, it is implemented using the Apache Shiro framework. Login authentication method based on Basic Access Authentication, this part of the function was contributed by @woofyzhao

Enhanced collection of file data and k8s logs

Version 1.3.0 enhances the collection of file data and k8s data, in which file collection supports regular expression custom line breaks, so that multiple lines can be merged and folded In addition, the new version of Agent supports full and incremental collection of file content. This part of the function was contributed by @ganfengtan

DataProxy adds c++ sdk capability

In addition to the current java client, DataProxy has added c++ client capabilities, which are provided by @pocozh

Supports multiple udf and join operators

The new version of Sort supports three kinds of Temporal Join\ Lookup Join \ Interval Join, this part of the function was contributed by @yunqingmoswu Most community users mentioned the need for encryption and decryption and Json Udf, this part of the function was contributed by @Emsnap and @Emhui

Sort connector supports indicator reporting function

The new version of Sort Connector supports Flink built-in indicator reporting of various Connectors. External indicator systems such as Prometheus can directly obtain the number and rate of task data read and write. In addition, the new version also supports InLong Audit Audit data reporting, which is contributed by @pacigong, @Emsnap, @thesumery @Oneal65 @yunqingmoswu

Manager supports the creation of resources in multiple flow directions

In version 1.3.0, Manager added the creation of some storage resources:

Create Topic for SQLServer
Create Oracle libraries and tables
Create MySQL namespaces and tables

The above are all contributed by community member @haibo-duan, thanks

Other features and bug fixes

For related content, please refer to the Release Notes, which details the features, enhancements and bug fixes of this release.

Apache InLong follow-up planning

In subsequent versions, we will expand more data sources and storages to cover more usage scenarios, and gradually improve the usability and robustness of the system, including:

Agent adds Redis, CloudEvents, MongoDB collection types
Unified DataProxy MQ framework
Full support for Apache Kafka

Release 1.2.0

Wed, 22 Jun 2022 00:00:00 GMT

1.2.0 Features Overview

The just-released 1.2.0-incubating version closes about 410+ issues, contains 30+ features and 190+ optimizations. Mainly include the following:

Enhance management and control capabilities

Dashboard and Manager add cluster management capabilities
Dashboard optimizes the flow creation process
Manager supports plug-in extension of MQ

Extended collection node

Support for collecting data in Pulsar
Support data collection in MongoDB-CDC
Support data collection in MySQL-CDC
Support data collection in Oracle-CDC
Support data collection in PostgreSQL-CDC
Support data collection in SQLServer-CDC

Extended write node

Support for writing data to Kafka
Support for writing data to HBase
Support for writing data to PostgreSQL
Support for writing data to Oracle
Supports writing data to MySQL
Support writing data to TDSQL-PostgreSQL
Support for writing data to Greenplum
Supports writing data to SQLServer

Support data conversion

Support String Split
Support String Regular Replace
Support String Regular Replace First Matched Value
Support Data Filter
Support Data Distinct
Support Regular Join

Enhanced system monitoring function

Support the reporting and management of data link heartbeat

Other optimizations

Supports the delivery of DataProxy multi-cluster configurations
GitHub Action check, pipeline optimization

1.2.0 Features Details

Support multi-cluster management

Manager adds cluster management function, supports multi-cluster configuration, and solves the limitation that only one set of clusters can be defined through configuration files. Users can create different types of clusters on Dashboard as needed.

The multi-cluster feature is mainly designed and implemented by @healchow, @luchunliang, @leezng, thanks to three contributors.

Enhanced collection of file data and MySQL Binlog

Version 1.2.0 supports collecting complete file data, and also supports collecting data from the specified Binlog location in MySQL. This part of the work was done by @Greedyu.

Support whole database migration

Sort supports migration of data across the entire database, contributed by @EMsnap.

Supports writing data in Canal format

Support for writing data in Canal format to Kafka, contributed by @thexiay.

Optimize the HTTP request method in Manager Client

Optimized the way of executing HTTP requests in Manager Client, and added unit tests for Client, which reduces maintenance costs while reducing duplication of code. This feature was contributed by new contributor @leosanqing.

Supports running SQL scripts

Sort supports running SQL scripts, see INLONG-4405, thanks to @gong for contributing this feature.

Support the reporting and management of data link heartbeat

This version supports the heartbeat reporting and management of data grouping, data flow and underlying components, which is the premise of the state management of each link of the subsequent system.

This feature was primarily designed and contributed by @baomingyu, @healchow and @kipshi.

Manager supports the creation of resources in multiple flow directions

In version 1.2.0, Manager added the creation of some storage resources:

Create Topic for Kafka (contributed by @woofyzhao)
Create databases and tables for Iceberg (contributed by @woofyzhao)
Create namespaces and tables for HBase (contributed by @woofyzhao)
Create databases and tables for ClickHouse (contributed by @lucaspeng12138)
Create indices for Elasticsearch (contributed by @lucaspeng12138)
Create databases and tables for PostgreSQL (contributed by @baomingyu)

Sort supports lightweight architecture

Version 1.2.0 of Sort has done a lot of refactoring and improvements. By introducing Flink-CDC, it supports a variety of Extract and Load nodes, and also supports data transformation (ie Transform).

This feature contains many sub-features. The main developers are: @baomingyu, @EMsnap, @GanfengTan, @gong, @lucaspeng12138, @LvJiancheng, @kipshi, @thexiay, @woofyzhao, @yunqingmoswu, thank you all for your contributions.

For more information, please refer to: Analysis of InLong Sort ETL Solution.

Other features and bug fixes

For related content, please refer to the Release Notes, which details the features, enhancements and bug fixes of this release.

Apache InLong follow-up planning

In subsequent versions, we will expand more data sources and storages to cover more usage scenarios, and gradually improve the usability and robustness of the system, including:

Heartbeat report of each component
Status management of data flow
Full link audit support for writing to ClickHouse
Expand more types of acquisition nodes and storage nodes

Analysis of InLong Sort ETL Solution

Thu, 16 Jun 2022 00:00:00 GMT

1. Background

With the increasing number of users and developers of Apache InLong(incubating), the demand for richer usage scenarios and low-cost operation is getting stronger and stronger. Among them, the demand for adding Transform (T) to the whole link of InLong has received the most feedback. After the research and design of @yunqingmoswu, @EMsnap, @gong, @thexiay community developers, the InLong Sort ETL solution based on Flink SQL has been completed. This article will introduce the implementation details of the solution in detail.

Firstly, based on Apache Flink SQL, there are mainly the following considerations：

Flink SQL has high scalability and flexibility brought about by its powerful expression ability. Basically, Flink SQL can support most demand scenarios in the community. When the built-in functions of Flink SQL do not meet the requirements, we can also extend them through various UDFs.
Compared with the implementation of the underlying API of Flink, the development cost of Flink SQL is lower. Only for the first time, the conversion logic of Flink SQL needs to be implemented. In the future, we can focus on the construction of the ability of Flink SQL, such as the extension connector and the UDF.
In general, Flink SQL will be more robust and run more stable. The reason is that Flink SQL shields a lot of the underlying details of Flink, has strong community support, and has been practiced by a large number of users.
For users, Flink SQL is also easier to understand, especially for users who have used SQL, the usage is simple and familiar, which helps users to land quickly.
For the migration of existing real-time tasks, if they are originally SQL-type tasks, especially Flink SQL tasks, the migration cost is extremely low, and in some cases, no changes are even required.

Note: For all codes of this scheme, please refer to Apache InLong Sort, which can be downloaded and used in the upcoming version 1.2.0.

2. Introduction

2.1 Requirements

The main requirements of this solution are the completed inlong sort module transform (T) capability, including:

Transform	Notes
Deduplication in the window	Deduplicate data within a time window
time window aggregation	Aggregate data within a time window
time format conversion	Converts the value of a field to a string in the target time format
field segmentation	Split a field into multiple new fields by a delimiter
string replacement	Replace some or all of the contents of a string field
Data filtering	Discard or retain data that meets the filter conditions
Content extraction	Extract part of a field to create a new field
Join	Support two table join
Value substitution	Given a matching value, if the field's value is equal to that value, replace it with the target value

2.2 Usage Scenarios

Users of big data integration have transform requirements such as data transformation, connection and filtering in many business scenarios.

2.3 Design Goal

This design needs to achieve the following goals:

Functionality: Under InLong Sort's existing architecture and data flow model, it covers basic Transform capabilities and has the ability to expand rapidly.
Compatibility: The new InLong Sort data model is forward compatible to ensure that historical tasks can be configured and run properly.
Maintainability: The conversion of the InLong Sort data model to Flink SQL only needs to be implemented once. When there are new functional requirements later, this part does not need to be changed, even if there are changes, it can be supported with a small amount of changes.
Extensibility: When the open source Flink Connector or the built-in Flink SQL function does not meet the requirements, you can customize the Flink Connector and UDF to achieve its function expansion.

2.4 Basic Concepts

The core concept refers to the explanation of terms in the outline design

Name	Meaning
InLong Dashboard	Inlong front end management interface
InLong Manager Client	Wrap the interface in the manager for external user programs to call without going through the front-end inlong dashboard
InLong Manager Openapi	Inlong manager and external system call interface
InLong Manager metaData	Inlong manager metadata management, including metadata information of group and stream dimensions
InLong Manager task manager	Inlong manager manages the data source collection task module, manages agent task distribution, instruction distribution, and heartbeat reporting
InLong Group	Data flow group, including multiple data flows, one group represents one data access
InLong Stream	Data flow: a data flow has a specific flow direction
Stream Source	There are corresponding acquisition end and sink end in the stream. This design only involves the stream source
Stream Info	Abstract of data flow in sort, including various sources, transformations, destinations, etc. of the data flow
Group Info	Encapsulation of data flow in sort. A group info can contain multiple stream infos
Node	Abstraction of data source, data transformation and data destination in data synchronization
Extract Node	Source side abstraction of data synchronization
Load Node	Destination abstraction of data synchronization
MySQL Extract Node	MySQL data source abstraction
Kafka Load Node	Kafka data destination abstraction
Transform Node	Transformation process abstraction of data synchronization
Aggregate Transform Node	Data synchronization aggregation class transformation process abstraction
Node Relation	Relationship abstraction of nodes in data synchronization
Field Relation	Abstraction of the relationship between upstream and downstream node fields in data synchronization
Function	Abstraction of the relationship between upstream and downstream node fields in data synchronization
Substring Function	Abstraction of string interception function
Filter Function	Abstraction of data filter function
Function Param	Input parameter abstraction of function
Constant Param	Constant parameters
Field Info	Node field
Meta FieldInfo	Node meta information field

2.5 Domain Model

This design mainly involves the following entities:

Group, Stream, GroupInfo, StreamInfo, Node, NodeRelation, FieldRelation, Function, FilterFunction, SubstringFunction, FunctionParam, FieldInfo, MetaFieldInfo, MySQLExtractNode, KafkaLoadNode, etc.

For ease of understanding, this section will model and analyze the relationship between entities. Description of entity correspondence of domain model:

One group corresponds to one group info
A group contains one or more streams
One stream corresponds to one StreamInfo
A GroupInfo contains one or more StreamInfo
A StreamInfo contains multiple nodes
A StreamInfo contains one or more NodeRelations
A NodeRelation contains one or more FieldRelations
A NodeRelation contains 0 or more FilterFunctions
A FieldRelation contains one function or one FieldInfo as the source field and one FieldInfo as the target field
A function contains one or more FunctionParams

The above relationship can be represented by UML object relationship diagram as:

2.6 Function Use-case Diagram

3. System Outline Design

3.1 System Architecture Diagram

Serialization: Serialization Implementation Module
Deserialization: Deserialization Implementation Module
Flink Source: Custom Flink source implementation module
Flink Sink: Custom Flink sink implementation module
Transformation: Custom Transform implementation module
GroupInfo: Corresponding to Inlong group
StreamInfo: Corresponding to inlong stream
Node: Abstraction of data source, data conversion and data destination in data synchronization
FlinkSQLParser: SQL parser

3.2 InLong Sort Internal Operation Flow Chart

3.3 Module Design

This design only adds Flink connector and Flink SQL generator to the original system, and modifies the data model module.

3.3.1 Module Structure

3.3.2 Module Division

Description of important module division:

Name	Description
FlinkSQLParser	Used to generate Flink SQL core classes, including references to GroupInfo
GroupInfo	The internal abstraction of sort for inlong group is used to encapsulate the synchronization related information of the entire inlong group, including the reference to list\>
StreamInfo	The internal abstraction of sort to inlong stream is used to encapsulate inlong stream synchronization related information, including references to list\>, list\>
Node	The top-level interface of the synchronization node. Its subclass implementation is mainly used to encapsulate the data of the synchronization data source and the transformation node
ExtractNode	Data extract node abstraction, inherited from node
LoadNode	Data load node abstraction, inherited from node
TransformNode	Data transformation node abstraction, inherited from node
NodeRelation	Define relationships between nodes
FieldRelation	Define field relationships between nodes
Function	Abstract of T-ability execution function
FilterFunction	Function abstraction for data filtering, inherited from function
SubstringFunction	Used for string interception function abstraction, inherited from function
FunctionParam	Abstraction for function parameters
ConstantParam	Encapsulation of function constant parameters, inherited from FunctionParam
FieldInfo	The encapsulation of node fields can also be used as function input parameters, inherited from FunctionParam
MetaFieldInfo	The encapsulation of built-in fields is currently mainly used in the metadata field scenario of canal JSON, which is inherited from FieldInfo

4. Detailed System Design

The following describes the principle of SQL generation by taking MySQL synchronizing data to Kafka as an example

4.1 Node Described in SQL

4.1.1 ExtractNode Described in SQL

The node configuration is:

 private Node buildMySQLExtractNode() {
        List<FieldInfo> fields = Arrays.asList(
                new FieldInfo("name", new StringFormatInfo()),
                new FieldInfo("age", new IntFormatInfo()));
        return new MySqlExtractNode("1", "mysql_input", fields,
                null, null, "id",
                Collections.singletonList("tableName"), "localhost", "root", "password",
                "inlong", null, null,
                null, null);
    }

The generated SQL is:

CREATE TABLE `mysql_1` (`name` string,`age` int) 
with 
('connector' = 'mysql-cdc-inlong',
'hostname' = 'localhost',
'username' = 'root',
'password' = 'password',
'database-name' = 'inlong',
'table-name' = 'tableName')

4.1.2 TransformNode Described in SQL

The node configuration is:

 List<FilterFunction> filters = Arrays.asList(
                new SingleValueFilterFunction(EmptyOperator.getInstance(),
                        new FieldInfo("age", new IntFormatInfo()),
                        LessThanOperator.getInstance(), new ConstantParam(25)),
                new SingleValueFilterFunction(AndOperator.getInstance(),
                        new FieldInfo("age", new IntFormatInfo()),
                        MoreThanOrEqualOperator.getInstance(), new ConstantParam(18))
        );

The generated SQL is:

SELECT `name` AS `name`,`age` AS `age` FROM `mysql_1` WHERE `age` < 25 AND `age` >= 18

4.1.3 LoadNode Described in SQL

The node configuration is:

 private Node buildKafkaLoadNode(FilterStrategy filterStrategy) {
        List<FieldInfo> fields = Arrays.asList(
                new FieldInfo("name", new StringFormatInfo()),
                new FieldInfo("age", new IntFormatInfo())
        );
        List<FieldRelation> relations = Arrays
                .asList(
                        new FieldRelation(new FieldInfo("name", new StringFormatInfo()),
                                new FieldInfo("name", new StringFormatInfo())),
                        new FieldRelation(new FieldInfo("age", new IntFormatInfo()),
                                new FieldInfo("age", new IntFormatInfo()))
                );
        List<FilterFunction> filters = Arrays.asList(
                new SingleValueFilterFunction(EmptyOperator.getInstance(),
                        new FieldInfo("age", new IntFormatInfo()),
                        LessThanOperator.getInstance(), new ConstantParam(25)),
                new SingleValueFilterFunction(AndOperator.getInstance(),
                        new FieldInfo("age", new IntFormatInfo()),
                        MoreThanOrEqualOperator.getInstance(), new ConstantParam(18))
        );
        return new KafkaLoadNode("2", "kafka_output", fields, relations, filters,
                filterStrategy, "topic1", "localhost:9092",
                new CanalJsonFormat(), null,
                null, "id");
    }

The generated SQL is:

CREATE TABLE `kafka_3` (`name` string,`age` int) 
with (
'connector' = 'kafka-inlong',
'topic' = 'topic1',
'properties.bootstrap.servers' = 'localhost:9092',
'format' = 'canal-json-inlong',
'canal-json-inlong.ignore-parse-errors' = 'true',
'canal-json-inlong.map-null-key.mode' = 'DROP',
'canal-json-inlong.encode.decimal-as-plain-number' = 'true',
'canal-json-inlong.timestamp-format.standard' = 'SQL',
'canal-json-inlong.map-null-key.literal' = 'null'
)

4.2 Field T Described in SQL

4.2.1 Filter operator

See 4.1 node configuration for relevant configurations

The generated SQL is:

INSERT INTO `kafka_3` SELECT `name` AS `name`,`age` AS `age` FROM `mysql_1` WHERE `age` < 25 AND `age` >= 18

4.2.2 Watermark

The complete configuration of GroupInfo is as follows:

private Node buildMySqlExtractNode() {
        List<FieldInfo> fields = Arrays.asList(
                new FieldInfo("name", new StringFormatInfo()),
                new FieldInfo("age", new IntFormatInfo()),
                new FieldInfo("ts", new TimestampFormatInfo()));
        WatermarkField wk = new WatermarkField(new FieldInfo("ts", new TimestampFormatInfo()),
                new StringConstantParam("1"),
                new TimeUnitConstantParam(TimeUnit.MINUTE));
        return new MySqlExtractNode("1", "mysql_input", fields,
                wk, null, "id",
                Collections.singletonList("tableName"), "localhost", "root", "password",
                "inlong", null, null,
                null, null);
    }

    private Node buildKafkaNode() {
        List<FieldInfo> fields = Arrays.asList(
                new FieldInfo("name", new StringFormatInfo()),
                new FieldInfo("age", new IntFormatInfo()),
                new FieldInfo("ts", new TimestampFormatInfo()));
        List<FieldRelation> relations = Arrays
                .asList(new FieldRelation(new FieldInfo("name", new StringFormatInfo()),
                                new FieldInfo("name", new StringFormatInfo())),
                        new FieldRelation(new FieldInfo("age", new IntFormatInfo()),
                                new FieldInfo("age", new IntFormatInfo()))
                );
        return new KafkaLoadNode("2", "kafka_output", fields, relations, null, null,
                "topic", "localhost:9092", new JsonFormat(),
                1, null, "id");
    }

    private NodeRelation buildNodeRelation(List<Node> inputs, List<Node> outputs) {
        List<String> inputIds = inputs.stream().map(Node::getId).collect(Collectors.toList());
        List<String> outputIds = outputs.stream().map(Node::getId).collect(Collectors.toList());
        return new NodeRelation(inputIds, outputIds);
    }

    @Override
    public GroupInfo getTestObject() {
        Node input = buildMySqlExtractNode();
        Node output = buildKafkaNode();
        StreamInfo streamInfo = new StreamInfo("1", Arrays.asList(input, output), Collections.singletonList(
                buildNodeRelation(Collections.singletonList(input), Collections.singletonList(output))));
        return new GroupInfo("1", Collections.singletonList(streamInfo));
    }

Release 1.1.0

Mon, 25 Apr 2022 00:00:00 GMT

1.1.0 Features Overview

The 1.1.0-incubating just released mainly includes the following:

Enhanced management capabilities for manager

Added Manager Client to support the integration of InLong for creating data streams
Added ManagerCtl command-line tool to support viewing and creating data streams
Manager supports dispatching Agent tasks
Manager supports dispatching Sort Flink tasks
Manger data streams management, supports start, restart, pause operations
Manager plugin support
Manager metadata management supports using MySQL
The first phase of cluster management, unified cluster information registration

Rich data nodes

Support Iceberg
Support ClickHouse
Support MySQL Binlog collection
Support Kafka ingestion
Sort Standalone supports multiple type streams

Tools construction

Dashboard plugin support
Kubernetes deployment optimization
Standalone deployment refactoring

System Upgrade

Protocol Buffers upgrade
Unified version Maven version dependencies
Fixed a bunch of dependency CVEs
DataProxy supports PB compression protocol

This version closed about 642+ issues, including four 23 features and 180 improvements.

1.1.0 Features Details

Add Manager Client

The newly added Manager Client defines common InLong Group/Stream operation interfaces, including task creation, viewing and deletion. Through Manager Client, users can build InLong into any third-party platform to achieve unified customized platform construction. The Manager Client is mainly designed and implemented by @kipshi, @gong, @ciscozhou, thanks to three contributors.

Add ManagerCtl command line tool

ManagerCtl is developed based on Manager Client and is a command-line tool for manipulating InLong resources, which can further simplify the use of developers. ManagerCtl was contributed independently by @haifxu and includes guidelines including:

Usage: managerctl [options] [command] [command options]
Options:
-h, --help
Get all command about managerctl.
Commands:
create      Create resource by json file
Usage: create [options]
​
describe      Display details of one or more resources
Usage: describe [options]
​
list      Displays main information for one or more resources
Usage: list [options]

Manager supports issuing Sort tasks

In previous versions, after the user created the data group/stream task, Sort needed to submit it to the Flink cluster through the command line, and then perform data sorting. In version 1.1.0, we unified the start, stop, and suspend operations of Sort tasks to the Manager to complete. Users only need to configure the correct Flink cluster when the Manager is deployed. When the data stream is approved, Sort will be automatically started. This part of the work is mainly contributed by @LvJiancheng.

Manager metadata is saved to ZooKeeper

InLong uses ZooKeeper to save data stream metadata, which increases the deployment threshold and operation and maintenance difficulty for developers and users. In version 1.1.0, we save data stream metadata in DB by default, and ZooKeeper is only an optional solution in large-scale scenarios. Thanks to @kipshi @yunqingmoswu for contributing a PR to ZooKeeper.

Support MySQL Binlog collection

Version 1.1.0 fully supports the collection of data from MySQL, and supports both incremental and full strategies. Users can collect MySQL data with simple configuration in InLong. This part of the work was contributed by @EMsnap.

Sort Added Iceberg, ClickHouse, Kafka

In version 1.1.0, the storage of data nodes for various scenarios has been added, including Iceberg, ClickHouse, and Kafka. The support of these three streams enriches the usage scenarios of InLong. New stream support, mainly contributed by @chantccc @KevinWen007 @healchow.

Sort Standalone supports Hive, Elasticsearch, Kafka

As mentioned in the previous version, for non-Flink environments, we can sort data streams through Sort Standalone. In version 1.1.0, we added support for Hive, ElasticSearch, Kafka, and expanded the usage scenarios of Sort Standalone. Sort Standalone is mainly contributed by @vernedeng @luchunliang.

Protocol Buffers upgrade

All InLong components Protocol Buffers dependencies have been upgraded from 2.5.0 to 3.19.4. Thanks to @gosonzhang @doleyzi for their contributions, a lot of compatibility and testing work for Protocol Buffers upgrades.

InLong on Kubernetes optimization

The optimization work of InLong on Kubernetes mainly includes adding Audit, combing configuration, optimizing the use of message queues, optimizing the use of documents, etc., to facilitate the use of InLong on the cloud. Thanks to @shink for contributing to these optimizations.

Dashboard plugin

In order to facilitate users to quickly build new data stream on Dashboard, Dashboard is support plugin in version 1.1.0. JavaScript developers who understand the plugin development guidelines can quickly expand new data stream. Thanks for this part of the work @leezng

Other features and bug fixes

For related content, please refer to the version release notes, which list the features, enhancements and bug fixes of this version in detail, as well as specific contributors.

Apache InLong(incubating) follow-up planning

In subsequent versions, we will support lightweight Sort, and expand more data sources and targets to cover more usage scenarios, including:

Flink SQL support
Elasticsearch, HBase support

Release 0.12.0

Tue, 04 Jan 2022 00:00:00 GMT

InLong: the sacred animal in Chinese myths stories, draws rivers into the sea, as a metaphor for the InLong system to provide data access capabilities.

Provide automatic, safe, reliable and high-performance data transmission capabilities, while supporting batch and streaming
Refactor the document structure of the official website to facilitate users to consult related documents based on the main line
The whole process supports Pulsar data flow, and completes the whole process coverage of high-performance and high-reliability scenarios
Full-process support indicators, including JMX and Prometheus output
The first phase of data audit and reconciliation, write audit data to MySQL

This version closed about 120+ issues, including four major features and 35 improvements.

Apache InLong(incubating) Introduction

Apache InLong is a one-stop integration framework for massive data donated by Tencent to the Apache community. It provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate the construction of streaming-based data analysis, modeling, and applications.
The Apache InLong project was originally called TubeMQ, focusing on high-performance, low-cost message queuing services. In order to further release the surrounding ecological capabilities of TubeMQ, we upgraded the project to InLong, focusing on creating a one-stop integration framework for massive data.

Apache InLong uses TDBank internally used by Tencent as the prototype, and relies on trillion-level data access and processing capabilities to integrate the entire process of data collection, aggregation, storage, and sorting data processing. It is simple to use, flexible to expand, stable and reliable characteristic.

Apache InLong serves the entire life cycle from data collection to landing, and provides different processing modules according to different stages of data, including the next modules:

inlong-agent, data collection agent, supports reading regular logs from specified directories or files and reporting data one by one. In the future, DB collection and HTTP reporting capabilities will also be expanded.
inlong-dataproxy, a Proxy component based on Flume-ng, supports data transmission blocking, placing retransmission, and has the ability to forward received data to different MQ (message queues).
inlong-tubemq, Tencent's self-developed message queuing service, focuses on high-performance storage and transmission of massive data in big data scenarios and has a relatively good core advantage in mass practice and low cost.
inlong-sort, after consuming data from different MQ services, perform ETL processing, and then aggregate and write the data into Apache Hive, ClickHouse, Hbase, IceBerg, etc.
inlong-manager, provides complete data service management and control capabilities, including metadata, OpenAPI, task flow, authority, etc.
inlong-website, a front-end page for managing data access, simplifying the use of the entire InLong control platform.

What’s New in Apache InLong(incubating) 0.12.0

1. Support Apache Pulsar data cache

In version 0.12.0, we have completed the data reporting capability of FileAgent→DataProxy→Pulsar→Sort. So far, InLong supports high-performance and high-reliability data access scenarios: Compared with the high-throughput TubeMQ, Apache Pulsar can provide better data consistency and is more suitable for scenarios that require extremely high data reliability. For example, finance and billing.

Thanks to @healchow, @baomingyu, @leezng, @bruceneenhl, @ifndef-SleePy and others for their contributions to this feature. For more information, please refer to INLONG-1310incubator-inlong/issues/1310), please refer to [Pulsar usage example](https://inlong.apache. org/zh -CN/docs/next/quick_start/pulsar_example/) to get the usage guide.

2. Support JMX and Prometheus metrics

In addition to the existing file output metrics, the various components of InLong began to gradually support the output of JMX and Prometheus metrics to improve the visibility of the entire system. Currently, modules including Agent, DataProxy, TubeMQ, Sort-Standalone, etc. already support the above metrics, and the documentation of metrics output by each module is being sorted out.

Thanks to @shink, @luchunliang, @EMsnap, @gosonzhang and others for their contributions. For related PRs, please see INLONG-1712, [INLONG-1786] (https://github.com/apache/incubator-inlong/issues/1786), INLONG-1796, [INLONG-1827] (https://github.com/apache/incubator-inlong/issues/1827), INLONG-1851, [INLONG-1926] (https://github.com/apache/incubator-inlong/issues/1926).

3. Function extension of the modules

The Sort module adds support for Apache Doris storage and realizes the ability to load sorted data into Apache Doris, giving InLong one more storage option. In addition, in order to enrich the functions of the entire data access process, the Audit and Sort-Standalone modules have been added:

The Audit module provides the ability to reconcile the entire process of data flow, and monitor the service quality of the system through data flow indicators;
Sort-Standalone module is a non-Flink-based data sorting module. It adds lightweight data sorting capabilities to the system, facilitating the rapid construction and use of businesses.

The Audit and Sort-Standalone modules are still under development and will be released when the version is stable.

Thanks to @huzk8, @doleyzi, @luchunliang and others for their contributions, please refer to INLONG-1821, [INLONG-1738]( https: / /github.com/apache/incubator-inlong/issues/1738), INLONG-1889.

4. Official website document directory reconstruction

In addition to the improvement of functional modules in version 0.12.0, the official website structure and the use of documents have also been deeply adjusted, including the reconstruction of the document directory structure, supplementing and improving the function introduction of each module, adding document translation, and further improving the documentation of the InLong official website. Readability, to achieve quick search and easy reading. You can check the official website to understand this content. The document is still under construction. We welcome your valuable comments or suggestions.

Thanks to @bluewang, @dockerzhang, @healchow and others for their contributions, please refer to INLONG-1711, [INLONG-1740](https: //github.com/apache/incubator-inlong/issues/1740), INLONG-1802, [INLONG-1809](https: //github.com/apache/incubator-inlong/issues/1809), INLONG-1810, [INLONG-1815](https: //github.com/apache/incubator-inlong/issues/1815), INLONG-1822, [INLONG-1840](https: //github.com/apache/incubator-inlong/issues/1840), INLONG-1856, [INLONG-1861](https: //github.com/apache/incubator-inlong/issues/1861), INLONG-1867, [INLONG-1878](https: //github.com/apache/incubator-inlong/issues/1878), INLONG-1901, [INLONG-1939](https: //gith ub.com/apache/incubator-inlong/issues/1939).

5. Other features and bug fixes

For related content, please refer to Version Release Notes, which lists the detailed features of this version, Improvements, bug fixes, and specific contributors.

Apache InLong(incubating) follow-up planning

In subsequent versions, we will further enhance the capabilities of InLong to cover more usage scenarios, including:

Support link for data access ClickHouse
Support DB data collection
The second stage full link indicator audit function

Release 0.11.0

Wed, 10 Nov 2021 00:00:00 GMT

Apache InLong (incubating) has been renamed from the original Apache TubeMQ (incubating) from 0.9.0. With the name change, InLong has also been upgraded from a single message queue to a one-stop integration framework for massive data. InLong supports data collection, aggregation, caching, and sorting, users can import data from the data source to the real-time computing engine or land to offline storage with a simple configuration.

The just-released version 0.11.0-incubating is the third version after the name changed. This version includes next features:

Lower the user's threshold for use further. Support all modules of InLong to be deployed on Kubernetes, and refactor the official website, so that users can more easily access related documents.
Support more usage scenarios, increase the data flow direction of Dataproxy -> Pulsar, and cover scenarios with higher requirements for data integrity and consistency.
Supports SDKs in more languages for TubeMQ. This version opens the production-level TubeMQ Go SDK, which is convenient for users who use the Go language to access

This version closed more than 80 issues, including four significant features and 35 improvements.

Apache InLong(incubating) Introduction

Apache InLong serves the entire life cycle from data collection to landing, and provides different processing modules according to different stages of data, including the next modules:

inlong-agent, data collection agent, supports reading regular logs from specified directories or files and reporting data one by one. In the future, DB collection and HTTP reporting capabilities will also be expanded.
inlong-dataproxy, a Proxy component based on Flume-ng, supports data transmission blocking, placing retransmission, and has the ability to forward received data to different MQ (message queues).
inlong-tubemq, Tencent's self-developed message queuing service, focuses on high-performance storage and transmission of massive data in big data scenarios and has a relatively good core advantage in mass practice and low cost.
inlong-sort, after consuming data from different MQ services, perform ETL processing, and then aggregate and write the data into Apache Hive, ClickHouse, Hbase, IceBerg, etc.
inlong-manager, provides complete data service management and control capabilities, including metadata, OpenAPI, task flow, authority, etc.
inlong-website, a front-end page for managing data access, simplifying the use of the entire InLong control platform.

What’s New in Apache InLong(incubating) 0.11.0

InLong on Kubernetes

Apache InLong includes multiple components such as data collection, data aggregation, data caching, data sorting, and cluster management and control. In order to facilitate users to use and support cloud-native features, all components currently support deployment in Kubernetes. Thanks to @shink for contributing to this feature. For specific details, please refer to INLONG-1308.

Open up dataproxy->pulsar data flow

Before version 0.11.0, InLong's data caching layer could only support TubeMQ. TubeMQ is very suitable for scenarios with huge data volumes, but in extreme scenarios, there may be a small amount of data loss. To provide data reliability, the Inlong added support for Apache Pulsar in version 0.11.0. Now InLong backend can support data flow agent -> dataProxy -> tubeMQ/pulsar -> sort. The introduction of Pulsar makes the application scenarios covered by InLong more abundant, which could meet the needs of more users. Thanks to @baomingyu for his contribution to this feature. For more details, please refer to INLONG-1330.

Go SDK for InLong TubeMQ

Before version 0.11.0, InLong TubeMQ supports SDKs in three languages: Java, C++, and Python. With more and more applications of Go language, the demand for Go language SDK in the community is also increasing. Version 0.11.0 was officially introduced to the Go language SDK of TubeMQ. The multilingual SDK is enriched, and the difficulty of access and use for Go language users is also reduced. Thanks to @TszKitLo40 for contributing to this feature. For more details, please refer to:

refactor the official website

In version 0.11.0, InLong uses Docusaurus to refactor the official website to provide a more concise and intuitive document management and display. Thanks to @leezng for his contribution to this feature. For more details, please refer to:

In addition to the above major features, InLong 0.11.0 version has other key improvements, including but not limited to:

Added contribution guidelines in the main Repo, INLONG-1623
Add Inlong-Manager to DataProxy configuration management, INLONG-1595
Optimized the GitHub issue template, INLONG-1585
Code Checkstyle optimization, INLONG-1571, INLONG-1593, INLONG-1593, INLONG-1662
Agent introduces message filter, INLONG-1641

The 0.11.0 version also fixes ~45 bugs. Thanks to all the contributions who have contributed to the Inlong community (in no particular order):

shink
baomingyu
TszKitLo40
leezng
ruanwenjun
leo65535
luchunliang
pierre94
EMsnap
dockerzhang
gosonzhang
healchow
guangxuCheng
yuanboliu

Apache InLong(incubating) follow-up planning

In the subsequent version planning of InLong, we will further release the capabilities of InLong to cover more usage scenarios, mainly including

Support Apache Pulsar full link data access capabilities, including back-end processing and front-end management capabilities.
Support data flow such as ClickHouse, Apache Iceberg, Apache HBase, etc.

Apache InLong Blog

Version 2.3.0 Released

About Apache InLong​

Overview of Version 2.3.0​

Dashboard Module​

Manager Module​

Agent Module​

Sort Module​

SDK Module​

Audit Module​

TubeMQ Module​

Key Features of Version 2.3.0​

Dashboard supports audit data reconciliation based on data stream groups​

Audit Store supports horizontal scaling​

Sort supports dynamic Transform capabilities​

Future Plans​

Version 2.2.0 Released

About Apache InLong​

Overview of Version 2.2.0​

Dashboard Module​

Manager Module​

Agent Module​

Sort Module​

SDK Module​

Audit Module​

TubeMQ Module​

Key Features of Version 2.2.0​

DataProxy SDK Refactoring​

Enhanced Agent Stability and Resource Management​

Sort Traffic Control Enhancements​

Future Plans​

Release 2.1.0

About Apache InLong​

2.1.0 Overview​

Dashboard Module​

Manager Module​

Agent Module​

Sort Module​

SDK Module​

Audit Module​

TubeMQ Module​

Others​

2.1.0 Feature Introduction​

Dashboard supports batch operation of agents​

Manager supports multiple scheduling engines​

AirFlow engine​

DolphinScheduler engine​

Agent supports COS data source​

Sort supports archiving dirty data through the InLong SDK.​

Future Plans​

Release 2.0.0

About Apache InLong​

From 1.0 to 2.0​

2.0.0 Overview​

Agent Module​

Dashboard Module​

Manager Module​

SDK Module​

Sort Module​

Audit Module​

2.0.0 Feature Introduction​

New Transform Capabilities​

Community Documentation Restructuring​

Support for OceanBase Data Source​

Dynamic Resource Calculation for Sort Tasks​

SortStandalone Supports HTTP Sink​

Full-Link Management of Offline Synchronization​

Summary and Future Plans​

Release 1.13.0

About Apache InLong​

1.13.0 Version Overview​

Agent Module​

Manager Module​

Dashboard Module​

Audit Module​

Sort Module​

1.13.0 Version Feature Introduction​

Manager supports installing Agent by SSH​

Manager supports field template management​

Support the construction of the underlying framework for configuring offline synchronization tasks​

About Apache InLong

Overview of Version 2.3.0

Dashboard Module

Manager Module

Agent Module

Sort Module

SDK Module

Audit Module

TubeMQ Module

Key Features of Version 2.3.0

Dashboard supports audit data reconciliation based on data stream groups

Audit Store supports horizontal scaling

Sort supports dynamic Transform capabilities

Future Plans

About Apache InLong

Overview of Version 2.2.0

Dashboard Module

Manager Module

Agent Module

Sort Module

SDK Module

Audit Module

TubeMQ Module

Key Features of Version 2.2.0

DataProxy SDK Refactoring

Enhanced Agent Stability and Resource Management

Sort Traffic Control Enhancements

Future Plans

About Apache InLong

2.1.0 Overview

Dashboard Module

Manager Module

Agent Module

Sort Module

SDK Module

Audit Module

TubeMQ Module

Others

2.1.0 Feature Introduction

Dashboard supports batch operation of agents

Manager supports multiple scheduling engines

AirFlow engine

DolphinScheduler engine

Agent supports COS data source

Sort supports archiving dirty data through the InLong SDK.

Future Plans

About Apache InLong

From 1.0 to 2.0

2.0.0 Overview

Agent Module

Dashboard Module

Manager Module

SDK Module

Sort Module

Audit Module

2.0.0 Feature Introduction

New Transform Capabilities

Community Documentation Restructuring

Support for OceanBase Data Source

Dynamic Resource Calculation for Sort Tasks

SortStandalone Supports HTTP Sink

Full-Link Management of Offline Synchronization

Summary and Future Plans

About Apache InLong

1.13.0 Version Overview

Agent Module

Manager Module

Dashboard Module

Audit Module

Sort Module

1.13.0 Version Feature Introduction

Manager supports installing Agent by SSH

Manager supports field template management

Support the construction of the underlying framework for configuring offline synchronization tasks

Optimize the Sort Standalone configuration process

Agent ability for collecting data from PostgreSQL

Sort connectors reporting audit information supports exactly once

Future plans

About Apache InLong

1.12.0 Version Overview