This content originally appeared on Level Up Coding - Medium and was authored by Jeffrey Wolberg
This is part two of a three part series explaining how to use dart:ffi, OpenCV, and Tensorflow to write a computer vision app in Flutter.
Part one discussed how to properly configure the OpenCV library and our custom C++ files to work within a Flutter app and how to access these C++ files from Dart via dart:ffi.
Part two discusses how to pass data between Dart and C++ using the dart:ffi library using pointers. This is especially important for transferring image data to C++ so that it can be manipulated by OpenCV and returned to Dart.
Part three discusses how to use tensorflow in a Flutter app using the tflite_flutter plug in. We will go through proper configuration, common bugs, and how to run inference on our device.
Passing image data to and from C++
There are two main ways to do this:
- Read and write the image data using OpenCV’s imwrite and imread functions.
- Passing a pointer between our Dart and C++ files and accessing our bytes directly from memory.
Let’s go over how to do both!
Using OpenCV’s imwrite and imread functions
Using this is fairly straightforward, and easy to get working. We will pass the image path to C++ and have it read in the image using OpenCV’s imread function.
In Flutter, we may have access to an image from the user’s photo library, in which case an image path will be available to us, or we may only have access to an image’s pixel data. In the latter case, we will have to save our image to a temporary location in order to access it in C++. Let’s show a quick example of how to do that.
import ‘package:path_provider/path_provider.dart’;
Uint8List imgBytes = YOUR_IMAGE_BYTES; // pixel data
XFile img = XFile.fromData(imgBytes);
Directory dir = await getApplicationDocumentsDirectory();
String dirName = dir.path;
String imgPath = dirName + "/tempImg.jpg";
img.saveTo(imgPath);
We obtain the directory to save the file through the method getApplicationDocumentsDirectory, which is offered by the path_provider package. XFile offers a convenient way to write an image to a file using the saveTo method, and therefore I will convert my image pixels to an XFile, though there are other ways of saving images to a file.
Note: I am assuming your pixel data is represented by unsigned 8-bit integers, which is equivalent to the data type unsigned char in C++. I will continue to make this assumption throughout the tutorial. If your images are saved in a different format, please make modifications to my example code accordingly.
When we call our C++ function, we will pass in the imgPath as an argument and use OpenCV’s imread function.
returnType func(char *path, ...) {
cv::Mat image = cv::imread(path);With our image now accessible in C++, we can use OpenCV for our computer vision logic.
We will often want to access the resultant image in Dart after manipulating it with OpenCV. We can pass in another argument, outputImgPath, which will be the path to save our output image, or we can overwrite the original image path. Either way, this is as easy as doing:
cv::imwrite(outputImgPath, outputImg);
Back in our Flutter code, we can access the image through the line
Image img = Image.file(outputImgPath);
Very easy!
cv::imwrite encodes the image that we are writing to disk into the format specified in the outputImgPath (e.g. .jpg, .png). For most cases, this is the intended behavior, however if we would like to have access to raw and uncompressed pixel data we cannot save an image in this way. This is where we must use pointers to point to our raw pixel data via dart:ffi.
Using dart:ffi and pointers to pass data to and from C++
We will make use of the dart:ffi library to call native C APIs to read/write and allocate/deallocate native memory. Specifically, in our dart code, we will allocate a specific number of bytes onto the heap, and then pass this pointer into C++. In C++, we will either access the input image pixel data that was filled in by Dart or fill in these values with the output image pixel data.
If we know the dimensions of the image, this is much easier. If the image size is variable, this is still possible, but slightly more complicated. Dealing with variably sized images is touched upon later in the tutorial. For now, let’s assume we know the dimensions of our input and output image and that it is 500x500. Let’s also assume it is a grayscale image, for simplicity. Therefore the total number of input bytes is 500 x 500 x 1 = 250,000. If we had an RGB image, we would allocate three times as much memory.
In our dart file, make sure to import dart:ffi;
We must allocate memory on the native heap. In Dart, we must write
Pointer<Uint8> imgPtr = malloc.allocate<Uint8>(500*500*1);
Uint8 is dart:ffi’s way of writing unsigned char in C++.
Remember, after we are done using our data pointed to by imgPtr we must free the memory using:
malloc.free(imgPtr);
If we plan on passing in our image pixel data into our C++ function, we can write the following line of code to copy our image data into the memory we just allocated, which will then be passed into C++:
Uint8List imgBytes = YOUR_IMAGE_BYTES; // pixel data
imgPtr.asTypedList(imgBytes.length).setAll(0, imgBytes);
This imgPtr is passed into our C++ function with a signature as follows:
returnType func(..., unsigned char *imagePtr, ...) { If we filled in our imagePtr with actual pixel values, we can now access them by dereferencing this imagePtr. Alternatively, if we just allocated the memory in Dart in order to fill it in C++, we can do the following:
for (int i=0; i < 500*500*1; i++)
imagePtr[i] = img.data[i]; // img is of type cv::Mat
img.data is the pointer to the output image pixel data, and we are merely copying the values that it is pointing to into the block of memory that we allocated.
In our Dart code, after our image processing is completed, we can create a list backed by that memory.
Uint8List imgPixels = imgPtr.asTypedList(500*500*1);
Now imgPixels gives us access to all the pixel data that we saved from C++. We can display it, feed it into a machine learning model, and much more!
Note: The standard Flutter way to display an image from memory using Image.memory(byteData) expects the passed in argument to hold encoded byte data that represents the .jpg or .png format of the image. If you give it raw input pixel data it will not be able to display your image. If this is the case, try using OpenCV’s imencode method to encode raw pixel data into encoded bytes, which can then be successfully passed into Image.memory in Flutter.
Let’s write some code!
We can make all of these concepts crystal clear with an example demonstrating how we would do just that: encoding raw image data in a C++ function that will read in our RGB input data using a pointer and save our encoded output data via another pointer.
In C++ we can write the function encodeIm that returns the length (total number of pixels) of the encoded image:
// uchar is a typedef provided by OpenCV for 'unsigned char'
int encodeIm(int h, int w, uchar *rawBytes, uchar **encodedOutput) {
cv::Mat img = cv::Mat(h, w, CV_8UC3, rawBytes);
vector<uchar> buf;
cv:imencode(".jpg", img, buf); // save output into buf
*encodedOutput = (unsigned char *) malloc(buf.size());
for (int i=0; i < buf.size(); i++)
(*encodedOutput)[i] = buf[i];
return (int) buf.size();Note that encodedOutput is a uchar** (Pointer to a pointer to a uchar). Because we cannot know beforehand in Dart the amount of bytes that the encoded output will take up, we malloc 8 bytes in Dart and save our pointer that was malloced dynamically in C++ here. This C++ pointer points to a variable amount of memory, depending on how much our original image’s size was reduced due to its encoding. This way we do not allocate any memory that is not necessary to our program.
In our Dart code:
Uint8List bytes = YOUR_RAW_IMAGE_BYTES; // raw pixel data;
// allocate memory on the heap for our input data
Pointer<Uint8> bytesPtr = malloc.allocate(bytes.length);
// Allocate just 8 bytes to store a pointer that will be malloced in C++ that points to our variably sized encoded image
Pointer<Pointer<Uint8>> encodedImPtr = malloc.allocate(8);
// copy our input data onto the heap
bytesPtr.asTypedList(bytes.length).setAll(0, bytes);
// Encode our input image and access the outputted pixel data
int encodedImgLen = _encodeIm(h, w, bytesPtr, encodedImPtr);
// Access the ptr malloced in C++ and save the data that it is // pointing to
Pointer<Uint8> cppPointer = encodedOutputPtr.elementAt(0).value;
Uint8List encodedImBytes = cppPointer.asTypedList(encodedImgLen);
Image myImg = Image.memory(encodedImBytes);
// Free the memory that we allocated after we are finished with it
malloc.free(bytesPtr);
malloc.free(cppPointer);
malloc.free(encodedImPtr); // always frees 8 bytes
Note: the bolded function _encodeIm above was loaded in Dart using the methods described at the end of part1 of this tutorial.
For a refresher on how to do this, here is the following code:
// declare _encodeIm for initialization later
late int Function(int height, int width, Pointer<Uint8> bytes, Pointer<Pointer<Uint8>> encodedOutput) _encodeIm;
final DynamicLibrary _dylib = Platform.isAndroid ? DynamicLibrary.open(‘libnative_add.so’) : DynamicLibrary.process();
_encodeIm = _dylib.lookup<NativeFunction<Int32 Function(Int32 height, Int32 width, Pointer<Uint8> bytes, Pointer<Pointer<Uint8>> encodedOutput)>>(‘encodeIm’).asFunction();
Great! We have learned how to connect our C++ functions that contain our computer vision logic with our Dart code, allowing us to compartmentalize our computer vision logic (C++) with our user interface logic (Dart). We have learned how to pass in and retrieve image data using OpenCV’s imread and imwrite methods and through pointers.
In the next part, we will learn how to feed the data returned from C++ into a machine learning model using the Flutter package tflite_flutter. We will go over the proper configuration, common bugs, and other cool details. Stick around!
Building a Flutter Computer Vision App Using Dart:ffi, OpenCV, and Tensorflow (Part 2) was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Jeffrey Wolberg
Jeffrey Wolberg | Sciencx (2022-03-09T02:31:08+00:00) Building a Flutter Computer Vision App Using Dart:ffi, OpenCV, and Tensorflow (Part 2). Retrieved from https://www.scien.cx/2022/03/09/building-a-flutter-computer-vision-app-using-dartffi-opencv-and-tensorflow-part-2/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.