This content originally appeared on DEV Community 👩‍💻👨‍💻 and was authored by Charlie Gerard
The first time I experimented with TensorFlow.js for micro-controllers, I got really excited about the fact that a machine learning model was transferred via bluetooth to my Arduino. In just a few seconds gesture control was enabled on a website! My excitement quickly turned into curiosity; how does it actually work?
To understand it better, I spent time diving into the open-source script tf4micro-motion-kit.js that is used as part of the project Tiny Motion Trainer. In this post, I am going to explain how a machine learning model can be transferred via bluetooth from the browser to an Arduino.
If you’re interested in how this technology can be used to create gesture-controlled web applications, check out the previous blog post I wrote about this.
Bluetooth and GATT
A core concept to understand for the rest of this blog post is that Bluetooth Low Energy (BLE) devices transfer data back and forth using what is called services and characteristics, which are concepts that are part of GATT (Generic ATTribute Profile). GATT relies on the ATT (Attribute Protocol) data protocol, which uses 16-bit or 128-bit UUIDs (Universally Unique Identifiers) to store services and characteristics.
You can think of services and characteristics as variables that will hold data about the functionalities of your device, that can be read or written to from another application or device.
For example, a service you can find on many BLE devices is called the Battery Service. It contains multiple characteristics, for example the battery level and the battery power state characteristic. When you buy an off-the-shelf device, these services and characteristics are already set for you and you might be able to do some “read” or “write” requests to them.
On the Arduino however, you can create your own services and characteristics. When it comes to being able to upload a machine learning model to the board, you might want a generic service with characteristics such as the transfer state, the total length of the file, the settings with which your model was trained, and more.
What do these services and characteristics look like? As mentioned above, they are either 16-bit or 128-bit UUIDs so a service ID could be:
"81c30e5c-0000-4f7d-a886-de3e90749161"
In this example, you have 32 hexadecimal values. A hexadecimal value is represented as 2^4 so 4 bits that can hold a value of either 0 or 1. As a result, 32 x 4 = 128 bits.
When these services and characteristics are created in the program running on the Arduino, they can then be read from the browser using these UUIDs.
Have a look at the Arduino upload program if you want to see how all the services and characteristics are created.
Now, let’s move on to how this works in practice.
Connecting to the board
First of all, the board has to be somehow connected to the browser to be able to send or receive data. In this project, this connection is established via bluetooth. Using the Web Bluetooth API, you need to start by using the requestDevice method on the navigator.bluetooth object. You can pass in some filters if you’d like so the browser will show you only the device(s) you’re interested in when initiating the connection.
For example, in the tf4micro-motion-kit.js script, the main service UUID that is created in the program running on the Arduino is used as a filter to identify the device.
const SERVICE_UUID = "81c30e5c-0000-4f7d-a886-de3e90749161";
const device = await navigator.bluetooth.requestDevice({
filters: [{ services: [SERVICE_UUID] }],
});
The code above returns a promise and the next step is to connect to the device:
const server = await device.gatt.connect();
If this step is successful, you can start to store the reference of these services and characteristics in variables so you can use them later on. For example, storing a reference to the main service is done with the following line:
const service = await server.getPrimaryService(SERVICE_UUID);
And you can define different characteristics with the code below, for example, the file length characteristic using the same ID as the one defined in the program uploaded to the Arduino:
const FILE_LENGTH_UUID = "bf88b656-3001-4a61-86e0-769c741026c0";
const fileLengthCharacteristic = await service.getCharacteristic(FILE_LENGTH_UUID);
The characteristic above is used to refer to the file size of the machine learning model in bytes so, when it is transferred to the Arduino, you can check when the entire file has been transferred.
After the connect() method has been called and you store all the references to the services and characteristics you’re interested in, the board is connected to the browser and you can move on to the next step.
Loading the machine learning model
The model is stored in your application folder as a .tflite file. First, it needs to be loaded using either the fetch API or an XMLHTTPRequest. Second, the response needs to be served as an array buffer.
const loadFile = (modelUrl) => {
return new Promise((resolve, reject) => {
const oReq = new XMLHttpRequest();
oReq.open("GET", modelUrl, true);
oReq.responseType = "arraybuffer";
oReq.onload = function () {
const arrayBuffer = oReq.response;
if (arrayBuffer) {
resolve(arrayBuffer);
} else {
reject(new Error("Failed fetching arrayBuffer"));
}
};
oReq.onerror = reject;
oReq.send(null);
});
};
Once the file is loaded, the next step prepares the file to be transferred.
Splitting the machine learning model
The Arduino Nano 33 BLE Sense is designed around a Nordic Semiconductor chip which has a Maximum Transmission Unit (MTU) of 23 bytes, which represents the largest packet size that can be sent at a time. According to this resource, the maximum data throughput for this size is 128 kbps so you need to split the machine learning model into packets of this size to be able to transfer it over to the Arduino.
You also need to make sure that the total size of the file isn’t too large for the microcontroller to handle.
To do this, you start by reading the characteristic that holds the maximum file size value from the Arduino.
let maximumLengthValue = await fileMaximumLengthCharacteristic.readValue();
let maximumLengthArray = new Uint32Array(maximumLengthValue.buffer);
let maximumLength = maximumLengthArray[0];
// The fileBuffer variable is the model buffer you loaded in the code sample above.
if (fileBuffer.byteLength > maximumLength) {
console.log(
`File length is too long: ${fileContents.byteLength} bytes but maximum is ${maximumLength}.`
);
return;
}
If the file isn’t too long, you can start splitting it and transferring it. The code below details the different steps.
// 1. Start by writing the length value of the model to the file length characteristic on the Arduino.
let fileLengthArray = Int32Array.of(fileContents.byteLength);
await fileLengthCharacteristic.writeValue(fileLengthArray);
// 2. Pass the file buffer and the starting index to the function
sendFileBlock(fileContents, 0)
async function sendFileBlock(fileContents, bytesAlreadySent) {
// 3. Calculate the remaining bytes.
let bytesRemaining = fileContents.byteLength - bytesAlreadySent;
const MAX_BLOCK_LENGTH = 128;
const blockLength = Math.min(bytesRemaining, MAX_BLOCK_LENGTH);
// 4. Transform the content into a Uint8array
const blockView = new Uint8Array(fileContents, bytesAlreadySent, blockLength);
// 5. Write the value to the characteristic on the Arduino
return fileBlockCharacteristic
.writeValue(blockView)
.then((_) => {
// 6. Remove the length of the block already sent from the size of bytes remaining
bytesRemaining -= blockLength;
// 7. If the file has not finished transferring, repeat
if (bytesRemaining > 0 && isFileTransferInProgress) {
console.log(`File block written - ${bytesRemaining} bytes remaining`);
bytesAlreadySent += blockLength;
return sendFileBlock(fileContents, bytesAlreadySent);
}
})
.catch((error) => {
console.log(error);
console.log(
`File block write error with ${bytesRemaining} bytes remaining, see console`
);
});
}
Once this code runs, it should transfer the machine learning model, packet by packet, until there are no bytes remaining. Even though this works, you can add an additional step to ensure that the file received is the same as the file transferred.
Checksum
A checksum can be used to detect errors that may have been introduced when the file was transferred between the browser and the Arduino. The tf4micro-motion-kit.js library uses a CRC32 function to verify data integrity.
It calculates a 32-bit cyclic redundancy checksum for the data transferred and repeats this calculation on the receiving side. If the two values don’t match, the data has somehow been corrupted and an error message can indicate that the transfer has failed.
I’m not going to dive into how the CRC32 function is implemented but you can view the source in the repository.
How the Arduino runs the model
Once the Arduino has received the complete file and the checksum has verified the integrity of the data, the code on the device gets live input data from the built-in sensors using the IMU.readAcceleration method, transforms that data into tensors, and feeds it to the model to predict gestures. This code is written in C++ and is part of the program you need to upload to the Arduino so I won’t go into details, but you can have a look at the code in this repository.
Conclusion
Transferring a machine learning model from the browser to an Arduino via bluetooth is done in 5 steps including connecting the board to the browser using the Web Bluetooth API, splitting the model into chunks, transferring them one by one to the device, running it with live data from the Arduino’s built-in sensors, and notifying the frontend once a gesture has been detected.
If you want to dive deeper into the entire code written for the Arduino sketch, you can find it in this repository and if you want to look at the code written to create the model, the Tiny Motion Trainer project is also open-source!
If you’re building anything interesting with this technology, let us know!
In the meantime, you can stay up to date with Stripe developer updates on the following platforms:
đź“Ł Follow @StripeDev and our team on Twitter
đź“ş Subscribe to our Youtube channel
đź’¬ Join the official Discord server
📧 Sign up for the Dev Digest
About the author
Charlie Gerard is a Developer Advocate at Stripe, a creative technologist and Google Developer Expert. She loves researching and experimenting with technologies. When she’s not coding, she enjoys spending time outdoors, trying new beers and reading.
This content originally appeared on DEV Community 👩‍💻👨‍💻 and was authored by Charlie Gerard
Charlie Gerard | Sciencx (2022-10-28T15:42:47+00:00) Demystifying machine learning via Bluetooth with Arduino. Retrieved from https://www.scien.cx/2022/10/28/demystifying-machine-learning-via-bluetooth-with-arduino/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.