Artificial intelligence is getting closer every year, only yesterday it was available only to a limited number of people. Today, we can create machine learning models ourselves and use them in our projects. In this article, we will introduce the Create ML tools and try to train the model for classifying images.But before you begin, a little history and explanation. At the annual conference for developers WWDC2018, Apple introduced the tools for working with machine learning Create ML. The model trained in Create ML is the result of applying a machine learning algorithm to a set of training data. Models do not take up much space (about 3Mb), so they can be stored in the project. Initially, it was proposed to train models using Playgrounds in Xcode 10 and supported work with images, text and tables. When launching the playgrounds of the project, it was necessary to import the CreateML library and run MLImageClassifierBuilder (in case we trained the model for classifying images).import CreateMLUI
let builder = MLImageClassifierBuilder()
builder.showInLineView()
In 2019, Apple introduced a completely new tool for working with machine learning models, separating it from Playgrounds. It became possible to train models without a single line of code. The list of data types has been replenished with sounds and activity. In total, 5 data types and 9 model templates are presented. Let's look at each data type and what models we can get as a result.Images
When training using images, two templates are available - Image Classifier (image classifier) and Object Detector (object detector).Image Classifier is used to classify images based on their contents. Having trained the model, we can identify one object in the image and attribute it to some type. For example, to identify animals in images and even distinguish which flower is depicted in the photograph. In Create ML, the Image Classifier template uses transfer training . Transfer training is the ability to combine a pre-trained model with user data, which allows you to train models on a small number of images.Object Detector is a deep learning model. Allows you to identify several objects in the image, such as people, buildings, cars. And all this in one image. It should also be added that when training Object Detector models, the computer GPU is used.Sounds
When working with sounds, only one template is available for creating a model - Sound Classifier (sound classifier). Sound Classifier is a new template and allows you to determine the most dominant sound in the audio stream. For example, you can determine which instrument sounds in audio. Sound Classifier, like Image Classifier, uses transfer training.Activity
Activity is represented by only one template - Activity Classifier. The Activity Classifier is a new template in Create ML and is used to classify motion data from an accelerometer and gyroscope.Such models are deep learning models and are trained using the GPU of a computer.Text
The text is represented by two templates: Text Classifier (text classifier) and Word Tagger (word marker).Text Classifier is used to identify sentences, paragraphs, and even entire articles based on their content. Word Tagger is used for marking words or recognizing any names in the text.Tables
Tables are represented by three templates: Tabular Regressor (tabular regressor), Tabular Classifier (table classifier) and Recommender (recommender).Tabular Regressor is a model that can predict a numerical value based on some data. For example, you can train a model that will predict the weather.Tabular Classifier classifies objects based on their features. For example, you can train a model to determine the size of clothes in an online store based on various user parameters. A model works with types such as integer, double, string, and long if the target is a discrete value.Recommender allows you to recommend content based on user behavior, and all calculations are performed directly on the device.Now that we’ve become familiar with all of the available templates, we can begin to create our first machine learning model. Let's start with Image Classifier.Image classifier
Have you watched the HBO Silicon Valley series? In one of the series of the fourth season, they showed an application that recognizes "hot dogs" and "not hot dogs" in the images. Using the Image Classifier, you can create an analogue of this application. But first you need to prepare the data for training the model. Archive with images here .A few requirements for training the Image Classifier model. Images should not be less than 299 by 299 pixels. Training kits must contain at least 10 images, but the more the better. Now we know that when training Image Classifier, transfer training is used, which allows you to train models on a small number of images. The number of images for each set must be equal, otherwise there will be a shift towards one of the categories. Approximately 80% of the images are used for training the model, and the remaining 20% for testing. Also, images should not be repeated, that is, in the folder with images for training, there should not be images that will be further used for testing. The names of the folders for training will be used as identifiers for the respective classes,when using models. My example uses the names HotDog or NotHotDog. The names of the images themselves do not play any role, they can be called as you like. I designated them with serial numbers.Now that our data is ready, we can start training the model. To work with Create ML, we need Xcode 11 or higher. I will use Xcode 11.3.1. To run the Create ML tool, you need to go to Xcode -> Open Developer Tool -> Create ML. A window will appear where you can select an existing one or create a new project. Select New Document. Next, you need to select a template, in our case we will use Image Classifier. Give a name to our project, click Next and select the place where the project will be stored. A working environment opens for working with the model.
As we can see, this is a completely separate project where you can create and train several models. The name of the project (Project) is displayed in the inspector on the left, clicking on it displays information about the project and allows you to edit it. Further below, Model Sources contains all the models used in our project. Let's now look at the Data Inputs section in the main tool window.Training Data - here you need to add images for training models. Validation Data - here you can add a set of images to test model training (in my case, I left Auto). Testing Data - here you need to add images for testing the model that were not involved in the training of the model.In the Parameters section, you can specify the maximum number of iterations performed on a single image during model training. And in the Augmentations section, you can choose the effects that are superimposed on the image.Now that we’ve become acquainted with Create ML, we can begin training our first model. We simply drag the Training Data folder (the name can be any) into the Training Data tab. Now click Train. The learning process begins.
So, it took me about 27 seconds to train, and the model is almost ready. After training, you can evaluate how well the model classifies images from the Training Data set. Since the model is trained in these images, it classifies them well. In my case, the model identified all the images from the training set and 97% of the images from the test set. Testing time has come.
Go to the Testing tab. As with the data for training the model, just drag the Testing Data folder into the Testing Data window. In fact, we could set up a folder with images for testing before starting the training. But for the linearity of the process, we will do it differently. Click Test Model. According to the test results, the model coped by 98%.
By going to Output, you can verify the model’s performance without adding it to your project. To do this, add images of a hot dog and not a hot dog and check the result. I added an image of a salad and a hot dog and, in both cases, the model did just fine. Of course, serious projects may require more data for training, more iterations, and so on. But for a test project this will be enough. To save the model, simply transfer it from Output to the desired folder.
Now I suggest trying to integrate the model into a test project, which is available here. A model has already been added to the project, but you can use any other classifier. This will not break the application, since the models are interchangeable and you can use several models in the project. Just transfer the finished model to the project.
In the Machine Learning Model section, you can find information about the model. This information can be specified in the Create ML tool project. Please note that the model we trained takes only 17 KB. Further in the Model Class section we are warned that a Swift class will be created with the name MyImageClassifier, I strongly do not recommend touching anything there, as this may affect the application. The Prediction section describes the input parameters and what we get as a result. In case you are using a model that you didn’t teach, in the Experimentation section you can check the model, just transfer the image.
Go to the ImageClassifierService class, here you can see how the model is initialized.private func makeImageClassifierModel() -> VNCoreMLModel? {
return try? VNCoreMLModel(for: MyImageClassifier().model)
}
Simple enough, right? Next, you need to create, process and execute a classification request.private func makeClassifierRequest(for model: VNCoreMLModel, ciImage: CIImage) {
let request = VNCoreMLRequest(model: model) { [weak self] request, error in
self?.handleClassifierResults(request.results)
}
let handler = VNImageRequestHandler(ciImage: ciImage)
DispatchQueue.global(qos: .userInteractive).async {
do {
try handler.perform([request])
} catch {
self.onDidUpdateState?(.requestFailed)
}
}
}
It remains only to process the result. private func handleClassifierResults(_ results: [Any]?) {
guard let results = results as? [VNClassificationObservation],
let firstResult = results.first else {
onDidUpdateState?(.requestFailed)
return
}
DispatchQueue.main.async { [weak self] in
let confidence = (firstResult.confidence * 100).rounded()
let resultModel = ClassifierResultModel(identifier: firstResult.identifier, confidence: Int(confidence))
self?.onDidUpdateState?(.receiveResult(resultModel: resultModel))
}
}
Now we can run the application and check the accuracy of the model.
As we can see, the model is doing well. Now let's try to show our model an image of not a hot dog and see how she copes with it.
As we can see, the model defines the burger image as “not a hot dog”. I propose to confuse the model a bit and add an image of a dog in a hot dog costume.
And here, our trained model does an excellent job.Conclusion
In this short article, we introduced the Create ML tool. Learned what types it works with and what patterns the tool uses. They also tried to train their model. The most difficult part in creating a machine learning model is finding and preparing data for learning. You can not train models, but use already trained ones. For example, there are already ready-made models that work with images and text. There are many custom models on github.Images for training models are also easy to find, for example, archives from Google . But if you need to train a model for narrowly targeted tasks, then you have to collect images and create your archives.