Hey everyone, I need a bit of a general advice. I’m working on a sign language translation application that is being written in C++ and is using the qt framework. I need to train an image classifier and am not sure how to approach it. There are two options for me:
Use fastai (library that I am familiar with) and try to convert that model to a format that can be used in C++. If any of you have done this I’d appreciate some pointers on how to achieve this. Is there a well-tested way to do this?
Use OpenCV (library that I am not familiar with) and try to do the training and inference in C++. This option is a bit overwhelming for me considering I have a deadline.
If any of you has done something similar, I’d appreciate your advice here.
Model deployment with C++ is a fairly standard task, and the first option you have described is the more conventional and straightforward of the two. Assuming you have a trained PyTorch model, there are two routes you can take to perform inference in C++, outlined below.
I) TorchScript: TorchScript enables you to serialize PyTorch models and execute them in other environments such as C++. The official guide walks the reader through this process.
II) ONNX + ONNX Runtime: ONNX is a framework for converting machine learning models from most packages - TensorFlow, PyTorch, etc. - into a common intermediate representation (IR) graph, i.e., the ONNX format. ONNX Runtime provides APIs for running ONNX models in various languages, including C++. PyTorch has a tutorial on exporting a model using ONNX, and here is a repository of C++ sample applications demonstrating ONNX Runtime.
The latter is leaner and tends to be more efficient, whereas the former is simpler and can be more flexible depending on the network architecture. In short, if you are seeking maximal performance and runtime optimizations, ONNX + ONNX Runtime would be the more judicious choice. Otherwise, TorchScript is an excellent alternative.
P.S: fastai may be pre-processing your data during training, e.g., normalization. It’s important that such steps be included during deployment as well.
Please don’t hesitate to reach out if you have further questions.
The only thing I am worried about is the fact that fastai is a high-level API so there might be steps that are need for converting a learner that are not very well documented. I couldn’t find a fastai specific example. But I guess I’ll just start with the links you sent.
You’re very welcome, I am glad you found my response helpful. Bear in mind that you are not converting the fastai Learner - you need to extract the underlying PyTorch model using learn.model, and you can subsequently follow the instructions I have linked. However, data pre-processing conducted by fastai must not be ignored; in the case of image classification, that generally entails resizing, cropping, and normalization, which should be included during deployment as well.
Hey @BobMcDear, sorry for bothering. I just have a couple of questions and was hoping you could help.
When it comes to the pre-processing steps, do those steps need to be applied to the image we want to do inference on, or do the functions of those steps need to be present in the c++ app and be available to the model objects to call? Also where can I check all the preprocessing steps that a fastai learner does to data?
I may be misunderstanding you - are you asking if pre-processing must be conducted on your input images prior to feeding them to the network, or whether it should be included in the definition of the model? The latter can be more expedient since you could simply add a few lines of code to the network’s forward method in Python and TorchScript/ONNX would trace them, meaning you wouldn’t need to concern yourself with image pre-processing in your C++ application. However, I have personally found the former route to be preferable, especially for more advanced data pipelines, because there exist operations TorchScript/ONNX cannot export, and implementing them in C++ using, e.g., OpenCV is ultimately more hassle-free. My advice is to adopt the first approach if the pre-processing steps in question are relatively standard - for example, resizing and normalization - and the second one otherwise.
fastai performs two fundamental types of pre-processing, namely, item transformations (dls.after_item), which are computed over individual data points, and batch transformations (dls.after_batch), which are computed over batches of data points. Note that certain transformations such as data augmentation are generally applied during training only and should not be employed during inference. Luckily, each fastai transformation is accompanied by a split_idx attribute that is 0 when a transformation is reserved exclusively for training and None or 1 otherwise, so differentiating between the two groups is straightforward. For instance, the snippet below iterates through a set of transformations and prints the ones that should be carried over to inference.
# tfms should be dls.after_item or dls.after_batch
for f in tfms.fs:
if f.split_idx != 0:
print(f)
If I could jump in here I’m fairly new to Fast.AI but I’ve been using PyTorch at work successfully to train in Python, export the model to TorchScript, and load it in C++ for inference. We train the model as normal then we export it to TorchScript how it says in the tutorials linked above.
On the C++ side of things we do:
try {
// Deserialize the ScriptModule from a file using torch::jit::load().
auto model = torch::jit::load(modelPath , /* optional device settings */);
}
catch (const c10::Error& e) {
return false;
}
From there we can do things like:
const auto output = model.forward(input).toTensor().to(device);
We have to do the usual things like make sure the input tensor to the forward method is of the correct rank and sizes otherwise model.forward throws a runtime exception.
You could also just implement everything in C++. In that case I found this repo might be useful.
Thanks @BobMcDear and @astroesteban for your responses and apologies if my message was confusing.
I think @BobMcDear answered my question. I can just do all the needed operations in OpenCV separately and at the end simply feed the proper cv::Mat object to the model.