Home Security Engineering Projects Research


This project is an experiment in robot vision (e.g. Open CV) and neural networks applied to the practical purpose of cleaning my apartment.

Although this is a fundamentally "solved" problem (e.g. Roomba), this project presents an opportunity to return to creating robots whose control systems are complex neural networks. The intent is to make use of "deep" learning tools - particularly TensorFlow - for the control system in order to get more intelligent behaviour than would be expected from a Roomba-like device.


After a number of kludgey attempts to retrofit one of my other robots with a vaccum, I decided to outsource the problem. I performed a small amount of research to see what hacks others had done and discovered that iRobot produced a hacker-friendly version of the Roomba (Create2). I thought this was perfect and until I noticed that, to make room for all the hacks, iRobot took out the vacuum... I ultimately chose the Rollibot which was available on the shelf at Canada Computers.

I noticed that there were literally no hacks for this vacuum. After a cursory investigation under the hood, I noted that the controller board is ultimately unhackable - no abusable interfaces were obvious. So this will have to be a wholesale, warranty-busting hack that swaps the existing board with something more malleable...

Platform De-Construction

The figure below shows the top of the Rollibot (less a plastic cover)

The figure below shows the bottom of the Rollibot. The weight of the battery is significantly more than the vacuum motor so the robot stays upright even though there is no forward caster.

The figure below shows the Rollibot after opening. The bumper shield must be taken off before the two halves can be separated. Once open, care needs to be exercised because the top and bottom remain connected through the wires shown in the figure. Once the wires and their connectors were labelled, I was able to fully separate the two halves by pulling out a few connectors.

Once I pulled the top and bottom apart, I set to work on the topside where I will be attaching a new control system. I started this process by splitting the pop-up lid into its component parts as shown in the figure below. There is a black plastic cover that is slightly curved covering the pop-up lid. This cover serves as both a display and a keyboard interface for the user.

Underneath the plastic cover the pop-up lid is nice and flat and can support the attachment of parts. The keyboard/display system is embedded in the pop-up lid as shown below.

I removed the keyboard/display circuit. They were held in place by two screws and connected to the rest of the robot through a convenient connector just underneath. I found the circuit to be instereting in in the way that it used surface mount LEDs for the digital readout - relying on layout of the plastic to channel the light to look like the bars that you would see in a normal LED. Of course, I couldn't reuse either of these so I tossed them in my "to examine later" bin.


The Rollibot controller circuit board is shown in the figure below. The main controller appears at the center. I numbered the male and female headers and also mapped them to the components that they control in the unlikely event that I would need to return the board back to operation.

The Rollibot is controlled by an ARM-based microcontroller (STM32F103-VB) as shown in the figure below. A cursory exam of the remainder of the board indicates two locations that might be used to interface to it and possibly hack new code. One of these is outlined in red in the figure. The other is an unpopulated, four-pin through-hole header that sits some ditance away from the controller. If there had been an easily identified interface that had some hope of being bent to my needs, I would have considered keeping the board in place and modifying it to act as a slave to a new controller. This approach would allow me to keep the sensorimotor interface intact. Unfortunately, it also involves really getting to know the innards of the existing circuit board and resident code.


As shown in the figure below, the motors were surprisingly easy to extract from the frame. They are self-contained and held in place by three screws. Each motor has a circuit board containing a shaft encoder circuit (the black circle on the blue circuit board). Each motor also has a spring that forces the wheel downwards. When the wheel is at its maximum downward reach, there is an embedded switch that opens to tell the Rollibot that it is no longer touching ground. The switch appears above the shaft encoder board (two blue wires) in the figure.


The Rollibot is equipped with five downward facing sensor ports for detecting the absense of floor - an upsetting situation that occurs when crossing over an edge. The locations of these sensors are outlined in red in the figure below.

I removed the edge sensors for some closer examination. These are shown in the figure below. You will note that all of the edge sensors are interfaced to connectors 6 and ?? which can be seen in the figure.

I wanted to know if the edge sensors had any built-in "smarts" - i.e. did they do any processing locally. Given that they were only two-wire interfaces, I didn't hold out any hope. The figure below shows one of the sensors cracked open for a better look. As I suspected, its just a transmitter-receiver LED pair - most likely working in the IR band. Signal modulation and conditioning would have to be done on the circuit board (hence all of the surface mount resistors).

The rollibot also comes with four forward facing IR sensors. These are mounted into dedicated slots along the front edge of the Rollibot similar to the one shown in the figure below. To get access to these sensors, it is necessary to remove the a semicircular plastic mount and split it in half. The two pieces can be seen sandwiched together in the figure.

I opened the plastic mount and removed the sensors. They are all connected to headers XXX and YYY as shown in the figure below.

The forward facing sensors transmit through a tinted, translucent lense attached to the front of the Rollibot. This lense also acts as a bumper and is used to activate bump switches. The bumper reminds me of RoboCop.

There is a final set of four IR sensors on the Rollibot. These are shown in the figure below. They appear to be IR receivers only. There are three on the front and one facing backward. The very front sensor is contained in a plastic box that very narrowly constrains the line-of-site of the sensor. I haven't confirmed this but my working hypothesis is that these are the "go home" sensors that are used to find the recharging base. The base itself is tinted and translucent so I suspect it contains a modulated IR beacon. The algorithm used by Rollibot is likely to turn in the direction that maximizes the IR signal being received on the front, center sensor and then to move in that direction. Because of the shadow being cast by the shell containing the center sensor, it is able to very precisely wiggle onto the rechaging slots. But that's just a hypothesis....

There are six switch sensors. As noted previously, two are attached to the wheel mechanism to determine if the robot has left the ground. As shown in the figure below, three more are attached to the front of the Rollibot (outlined in red) and are activated by depressing the RoboCop bumper. The bumper is pushed back into place after hitting something by the aluminum spring attachment outlined in purple.

There is one final switch sensor used to detect if the vacuum container is in the slot. There is also one final IR sensor that shines across the mouth of the vacuum container which is used to see if there is any garbage obstructing the opening.

Platform Mods

Every one of the IR sensors needs to have a custom circuit on the back end to generate and modulate the detection signal. That is why the circuit board is overrun by resistors. From a cost perspective, I can see why the designers chose this path. The sensors and their circuits are dirt cheap. However, I decided that integrating a custom conditioning circuit would be harder than just replacing the sensors with something having more more intelligence at the front end. This section describes the results of that effort.

Edge Sensors Mods

I shuffled around in my box of parts and found an unopend Sharp distance sensor that I had purchased from RobotShop.com.

It turns out that this sensor has the same width and almost the same height as the original Rollibot edge sensor.

In fact, the Sharp sensor not only fits well, it is also held in place by the original plastic hook when it is seated flush with the top of the holder. All that is needed to get the sensor to sit flush is to clip a small plastic extrusion on the top lip of the holder as shown in the figure below.

Dinstance Sensor Mods

TBD: I haven't attached distance sensors yet.


Controller Hardware - Teensy 3.2

For this project, I swapped out the original controller with the Teensy 3.2. However, this board will only act as the sensorimotor interface. I plan to use a neural network or some other CPU intensive approach for learning and adaptation and the Teensy won't suffice. I just need it to give me a standard interface into the platform.

Also for this project, I wanted to try out MBED OS. While there are a large number of supported platforms identified in the hardware section for MBED, only one device in my drawer of microcontrollers is supported on that list - Teensy 3.1 (3.1 and 3.2 are essentially the same device). A quick look around at the forums indicates that the more capable Teensy versions are not supported and will not likely attain that status in the future. So Teensy 3.2 it is!

Controller Software - MBED OS

There are a large number of embedded communities out there. Barriers to entry vary widely - although not as much as they even ten years ago. If you are looking to get something up and running in a bare minimum of time then a good choice would be something like Arduino. It has a large community and there are plenty of examples to help guide the learning curve. But the single most important reason for its success is that the coding environment and build cycle are as close to "plug and play" as you can get. You download and install one executable, select a target board, select the "Hello World" example (blinking light), hit download and you've overcome the entry barrier. Coding in Arduino is not trivial but it also not mind-bending and there are thousands of examples available to crib from.

Unfortunately, Arduino trades depth of control for breadth of access. If you need the features of an operating system (in particular multi-tasking) and you need real-time response, then off-the-shelf Arduino falls short (IMHO).

There are a number of embedded operating systems that you can choose from. Most of these are cross-platform and require compiler tools to be downloaded and installed. In most cases, getting the compiled code onto your board is *your problem*. However, if you are a fan of ARM, then MBED is a good choice. The biggest advantage of MBED is that you can create and manage your project using just a browswer - including compiling the code to a hex file (however, if you prefer, you can also download the compiler tools and execute them directly). Like Arduino, you can make a target selection in the browser window and have your code compiled directly for that device. And the device pages are well designed. Combined, these create inviting conditions for relative newbies. Browser-based compile tools are the next big thing.

Getting Started

Getting started with MBED OS on Teensy requires you to sign-up for an account at the MBED OS site. Once you have an account, your default interface will look similar to the one in the figure below.

There are two places you want to visit from this page as indicated in red in the figure. The first is to go select your board and the second is to go to the compiler (if you plan to use the online version of the compiler as I am.

Clicking on the "Hardware" dropdown and selecting "Boards" takes you to a location similar to the one in the figure below.

This page identifies all board types compatible with MBED OS. Scrolling down, I found the Teensy 3.1 board.

As I indicated above, it is a community contribution rather than a vendor manufactured board (denoted by CC).

Clicking on the Teensy board brings us to its product page as shown in the figure below. This page contains an overview of the platform and a schematic of the board. This schematic is critically impartant when coding - the MCU references in BLUE on the far left and right are the names you will use when you want to refer to an MCU pin.

The other item to note in the product page is the button that allows you to add it as a target to the online compiler (it is circled in red in the figure above). When you click this button, you get the confirmation shown in the figure below.

Now if you click on the "Compiler" button at the top, you will arrive at a coding interface that targets the Teensy be default. This coding environment is shown in the figure below. The current hardware target is identified in the top right corner.

If you click on the "New" button in the top right corner, you can add and work on a new program. You are asked for a target and program name in a new dialogue as shown in the figure below.

In this example, I named the new program "Myfirst" and when I clicked "OK", the compiler added it to the tree as shown in the figure below.

If you right click on the program name (Myfirst in this case), you are given a number of options as shown in the figure below. In order to get stuff done without re-inventing every wheel, it is helpful to have a library or two available to tap into. So we click "Import Library -> From Import Wizard" as shown in the figure below.

Selecting "Import Library -> From Import Wizard" brings us to a library search page as shown in the figure below. Initially, the tab is empty. I typed the word "mbed" in the search window and the results shown appear after a few seconds. After doing a bit of research on the options available, I've determined that a very useful library would be the top one (the one which is also the most imported and which is authored by the MBED project as opposed to a user contribution). However, as you can see, there are a large numbe of other libraries available which are somehow related to my initial search.

Double-clicking on "mbed" adds it to my project as shown in the figure below. Because I am not sure of what's available in the library, I spent some time browsing the classes.

If you right click on the program name again and select "New File", the new file window dialogue shown below opens up.

I gave my new file the name main.cpp. This file is added to Myfirst and a coding window opens as shown in the figure below. In theory, everything is ready to go!!!

Hello World Blinkenlight

In order to validate the code/compile/run/fix cycle, I created a simple "hello world" program as shown below. The objective here is to blink a light and send a message out a serial port.

		#include "mbed.h"
		DigitalOut led(PTC5);
		Serial pc(PTB17, PTB16);

		int main() {

			while(1) {
				pc.printf("Hello World!!\n");

A few notable things about this program:

If you hit the "Compile" button and it succeeds without error, you will receive feedback similar to the figure shown below.

Take note that when it finishes compiling, it downloads a HEX file for you. This is identified with the arrow. To move this hex file to the Teensy, you will need to get the Teensy Loader Application over at PJRC. Once you have it installed, you will need to point the loader at the HEX file, hit the load button on the Teensy, and then click on the download and reboot buttons in the loader. In my case, I had success.

Initial Sensorimotor Interface Code

The code below shows a polling loop I wrote to test signal connections and operation.

	#include "mbed.h"

	// modified PinNames.h, PeripheralPins.c, PeripheraNames.h
	PwmOut MotorLeftSpeed(PTC3); // Left motor speed
	DigitalOut MotorLeftDir1(PTD3); // Left motor dir
	DigitalOut MotorLeftDir2(PTD2); // Left motor dir

	PwmOut MotorRightSpeed(PTC4); // Right motor speed
	DigitalOut MotorRightDir1(PTC6); // Right motor dir
	DigitalOut MotorRightDir2(PTC7); // Right motor dir

	DigitalIn Down1(PTC5);
	DigitalIn Down2(PTD1);
	DigitalIn Down3(PTC0);
	DigitalIn Down4(PTB0);
	DigitalIn Down5(PTB1);

	AnalogIn AIn1(PTB3);
	AnalogIn AIn2(PTB2);
	AnalogIn AIn3(PTD5);
	AnalogIn AIn4(PTD6);
	AnalogIn AIn5(PTC1);

	int main() {
		while(1) {
			if (MotorLeftSpeed==1) {

This code failed to operate. That is, the motors did not turn and the sensor lights did not activate. I dropped in some cookie crumbs (i.e. I put in a delay and activated the on-board LED). I eventually managed to make it partially work when I disabled activating PWM on PTC4. After toggling the PWM on that pin a few more times, I had a fortuitous accident. I had connected a two-line serial LCD to the default serial port on the Teensy for a previous project - which I never bothered to disconnect. I noticed that when I enabled PWM on PTC4, the LCD displayed a message: "Pinmap not found". After doing some googling, it appears that not all errors will be caught by the MBED compiler and, when this happens, the runtime will push a message out the serial port.

In this case, it appeared that the MBED library might be broken in some way. In order to confirm and fix a problem under the hood, I would need to get access to the sources. This page was useful in that regard. Instead of adding the compiled MBED library into the project, it indicates adding MBED-DEV which will provide all of the source.

Take note: MBED OS 2 is called MBED and its sources are called MBED-DEV. If you want to work with MBED OS 5, you need to install MBED-OS. Not confusing at all....

Fixing the problem required locating the files containing the pinmap. As shown int he figure below, I found a promising file in the library structure under Targets. Freescale makes the K20XX which is the device on the Teensy. It turns out that there was a Teensy specific target directory containing the file peripheralpins.c. The original PWM pin mapping within this file is also shown in the figure below.

Notice that PTC4 is not represented in the mapping. The pinout for the Teensy indicates that PWM5 is connected to PTC4.

Also note that the pin name PWM_5 is missing and that the comments for PTD5 and PTD6 map both ports to the same FTM and channel.

To figure out what required changing, it was necessary to dive into the pinout for the K20DX256 chip. I made a very focussed summary of pin-to-function mapping shown in the table below. Each pin can have "Alternative" functionality. In the perpipheralpins.c file, the alternate function number is the last number in each mapping triplet.

I noted some confution between the PWM naming convention on the MBED pin mapping figure and the PWM naming within the peripheralpins.c file. Specifically, peripheralpins.c refers to PWM using the convention PWM_nn whereas the mapping figure uses PWM nn (where it is assumed that the absence of a number is PWM 0).

Referring to the PeripheralNames.h file provides the actual mappings between ports, pins, and FTM channels. I took this file as ground truth. The mapping is shown below.

	#define TPM_SHIFT   8
		typedef enum {
			PWM_1  = (0 << TPM_SHIFT) | (0),  // FTM0 CH0
			PWM_2  = (0 << TPM_SHIFT) | (1),  // FTM0 CH1
			PWM_3  = (0 << TPM_SHIFT) | (2),  // FTM0 CH2
			PWM_4  = (0 << TPM_SHIFT) | (3),  // FTM0 CH3
			PWM_5  = (0 << TPM_SHIFT) | (4),  // FTM0 CH4
			PWM_6  = (0 << TPM_SHIFT) | (5),  // FTM0 CH5
			PWM_7  = (0 << TPM_SHIFT) | (6),  // FTM0 CH6
			PWM_8  = (0 << TPM_SHIFT) | (7),  // FTM0 CH7
			PWM_9  = (1 << TPM_SHIFT) | (0),  // FTM1 CH0
			PWM_10 = (1 << TPM_SHIFT) | (1),  // FTM1 CH1
		} PWMName;

Using all of this information together, I was able to correct the peripheralpins.c file as follows:

	const PinMap PinMap_PWM[] = {
		// LEDs
		{LED_RED  , PWM_3 , 4}, // PTC3, FTM0 CH2
		{LED_GREEN, PWM_5,  4}, // PTD4, FTM0 CH4
		{LED_BLUE , PWM_8 , 3}, // PTA2, FTM0 CH7	
		{PTA0,  PWM_6 , 3}, // PTA0, FTM0 CH5
		{PTA1,  PWM_7 , 3}, // PTA1, FTM0 CH6
		{PTA3,  PWM_1 , 3}, // PTA3, FTM0 CH0
		{PTA4,  PWM_2 , 3}, // PTA4, FTM0 CH1
		{PTA5,  PWM_3 , 3}, // PTA5, FTM0 CH2
		{PTA12, PWM_9 , 3}, // PTA12, FTM1 CH0
		{PTA13, PWM_10, 3}, // PTA13, FTM1 CH1
		{PTB0,  PWM_9 , 3}, // PTB0, FTM1 CH0
		{PTB1,  PWM_10, 3}, // PTB1, FTM1 CH1
		{PTC1,  PWM_1 , 4}, // PTC1, FTM0 CH0
		{PTC2,  PWM_2 , 4}, // PTC2, FTM0 CH1
		{PTC3,  PWM_3 , 4}, // PTC3, FTM0 CH2
		// Added the following line for PTC4
		{PTC4,  PWM_4 , 4}, // PTC4, FTM0 CH3
		// Changed PTD4 PWM name to PWM_5 (missing PWM name and it applies to Channel 4)
		{PTD4,  PWM_5 , 4}, // PTD4, FTM0 CH4
		{PTD5,  PWM_6 , 4}, // PTD5, FTM0 CH6
		{PTD6,  PWM_7 , 4}, // PTD6, FTM0 CH6
		{PTD7,  PWM_8 , 4}, // PTD7, FTM0 CH7
		{NC , NC    , 0}

I dropped the code into the board and it solved the problem.

Analog Inputs

I turns out that I needed more cowbell (ADC pins) since I had allocated most of them to other functions. So I lifted the Teensy and soldered a 6 pin cable to the pads on the bottom of the board as shown in the figure below:

In theory, his would have given me six more analog inputs from (A15-A20). In practice, I got nothing. Although the code compiled, I received a pinmap error on the serial port again. On a hunch, I commented-out the PTE ports which eliminated the error. A quick check of Peripheralpins.c revealed that the analog capabilities of PTE0 and PTE1 were not mapped. However, even after commenting these out, my inputs were not properly recording analog signals from my sensors.

A quick check of the manual indicated that all of the analog inputs on the underside of the board (PTC8, PTC9, PTC10, PTC11, PTE0, PTE1) are mapped to ADC1. And, as it turns out, ADC1 was not enabled.

According to the pinout in Section 10.3.1 of the K20 manual, the ADC0 and ADC1 pin assignments are as follows:

		1 PTE0 ADC1 SE4a  (SE = Single Ended channel)
		2 PTE1 ADC1 SE5a
		3 PTE2 ADC1 SE6a
		4 PTE3 ADC1 SE7a
		18 DAC0_OUT ADC0 SE23
		35 PTB0 ADC0/ADC1 SE8
		36 PTB1 ADC0/ADC1 SE9
		37 PTB2 ADC0 SE12
		38 PTB3 ADC0 SE13
		43 PTC0 ADC0 SE14
		44 PTC1 ADC0 SE15
		45 PTC2 ADC0 SE4b
		53 PTC8 ADC1 SE4b
		54 PTC9 ADC1 SE5b
		55 PTC10 ADC1 SE6b
		56 PTC11 ADC1 SE7b
		58 PTD1 ADC0 SE5b
		62 PTD5 ADC0 SE6b
		63 PTD6 ADC0 SE7b

Note that PTE0, PTE1 and PTC8, PTC9 are multiplexed on channels 4 and 5. ADCx_CFG2[MUXSEL] bit selects between ADCx_SEn channels a and b. The MUXSEL mask is defined in the MK20DX256.h file (located at target-device-toolchain) as follows:

	#define ADC_CFG2_MUXSEL_MASK                     0x10u	

Since this value is in HEX, it lines up on the fifth bit (MUXSEL). To toggle MUXSEL, we need to:

	#Define ADC_CFG2_MUXSEL_MASK_A 0x00u
	#Define ADC_CFG2_MUXSEL_MASK_B 0x10u

For channel A:

               | ADC_CFG2_ADHSC_MASK    // High-Speed Configuration
               | ADC_CFG2_ADLSTS(0);    // Long Sample Time Select

For channel B:

               | ADC_CFG2_ADHSC_MASK    // High-Speed Configuration
               | ADC_CFG2_ADLSTS(0);    // Long Sample Time Select

So, in the read function, we will re-assign port numbers as follows:

	Number	Ch	MUXSEL
	1		4	0
	2		5	0
	3		4	1
	4		5	1
	5		6	1
	6		7	1

	MUXSEL=Mux select( 0 = Channel a, 1 = Channel b)

In peripheralpins.c, I added two lines to map the PTE0 and PTE1 functionality. The SE names came from the mapping table. Since they are the default behaviour, I set their position to zero. These additions are technically not required because I am addressing the ADC capability directly using a channel number.

	const PinMap PinMap_ADC[] = {
    {PTC2, ADC0_SE4b, 0},
    {PTD1, ADC0_SE5b, 0},
    {PTD5, ADC0_SE6b, 0},
    {PTD6, ADC0_SE7b, 0},
    {PTB0, ADC0_SE8,  0},
    {PTB1, ADC0_SE9,  0},
    {PTB2, ADC0_SE12, 0},
    {PTB3, ADC0_SE13, 0},
    {PTC0, ADC0_SE14, 0},
    {PTC1, ADC0_SE15, 0},
    {PTC8, ADC1_SE4b, 0},
    {PTC9, ADC1_SE5b, 0},
    {PTC10,ADC1_SE6b, 0},
    {PTC11,ADC1_SE7b, 0}, 
    {PTE0, ADC1_SE4a, 0},
    {PTE1, ADC1_SE5a, 0},   
    {NC,   NC,        0}};