From the course: Advanced Data Processing: Batch, Real-Time, and Cloud Architectures for AI

Characteristics of real-time AI

From the course: Advanced Data Processing: Batch, Real-Time, and Cloud Architectures for AI

Start my 1-month free trial Buy for my team

Characteristics of real-time AI

“

- [Instructor] Having discussed batch AI in the previous chapter, let's talk about real-time AI architectures in this chapter. When it comes to real-time AI, real-time inference is popular for a number of use cases that require on-demand predictions. On the other hand, real-time training is limited in usage, only for use cases that require instant model learning and adoption. What are some key characteristics of real-time AI? In real-time AI, we deal with events generated from sources typically one at a time. These could be real-time user inputs or business events. To do low-latency data processing, we need to use stream processing techniques that deal with one event at a time. If we are doing real-time training, then that requires continuous model updates, including testing and validation, and then promoting these models to production. Instead of batch jobs, inference services are deployed as either API-based microservices or as queue consumers. This enables them to respond to events immediately and at scale. In real-time AI, response times and throughput play a key role in generating outcomes with low latency. When failures happen, real-time pipelines typically skip ahead and move to new events rather than trying to reprocess failed events. What are the key architecture focus areas for real-time AI? The first focus area is data processing for feature engineering. Data streaming and stream processing at scale are required to handle events in real time, especially for inference. When databases are accessed, records should be read at low latency. Similarly, updates should be done in low latency. Models should be loaded into memory with low latency or they should be cached in memory. Serving should also be done with low latency. Building real-time inference requires building on-demand interfaces. This includes API-based services that can be called upon when needed and also queue consumers that can consume records from queues whenever they become available. Real-time systems are susceptible to load spikes. To deal with this, the architecture should be capable of auto-scaling and be able to load balance between available resources to maintain low latency. The system should also be resilient to failures. In case of failures, it should be able to recover quickly and proceed with other pending tasks without being blocked on the failed task.

Contents