From the course: Advanced Data Processing: Batch, Real-Time, and Cloud Architectures for AI

The ML life cycle

From the course: Advanced Data Processing: Batch, Real-Time, and Cloud Architectures for AI

Start my 1-month free trial Buy for my team

The ML life cycle

“

- [Instructor] Let's begin this course by reviewing the machine learning lifecycle. A machine learning project, or ML project for short, has many stages and tasks. The machine learning lifecycle represents the end-to-end workflow that connects these tasks. It constitutes building, executing, and maintaining machine learning models and applications. Let's discuss some key characteristics of the ML lifecycle. The ML lifecycle is an iterative process where the sequence of tasks are repeated from time to time to improve the models, it has three key flows. They are feature engineering, model training, and model inference. We will discuss these three flows in the next few videos. While the ML lifecycle focuses on the model, it also integrates into application development, deployment, and operations. Together, they manage the lifecycle of the application. The lifecycle provides a template based on which architectures can be created for artificial intelligence applications. Let's walk through the sequence of steps in a machine learning lifecycle. The cycle begins with the data sources that provide the data sets for model training, testing, and inference. Raw data is acquired from these data sources in various formats and stored in a data lake. Then feature engineering is performed on this data to cleanse and extract features required for machine learning. Then this produces the training dataset and the test dataset. The training dataset is used to train a ML model. The model is then tested using the test dataset. This is an iterative process where the model is refined until satisfactory model performance is achieved. This model is then stored in a model repository. The model is used for inference on the inference dataset. This is typically done in production scenarios. The outcomes of the model are then stored and published to the consumers.

Contents