Joshua Kettlewell | Projects

Joshua Kettlewell

Ph.D Student,
Singapore University
of Technology and Design

Neural Networks to clean the sink

So one of the reasons I love my housemates I because they do ridiculous things with technology. This what happens when you live with 5 other PhD students and are too poor to do anything outside the house at the weekend.

The latest development is a neural network, which is being used to tell us when the sink is dirty.

Dirty Sink

Previously we have taken a Bayesian model to discern whether we think the sink is full of pots, pans and plates. Based on what we've previously seen, then It probably is.

However - one of my more washing responsible housemates has recently returned from MIT so we are trying to be a little more on top of things to keep him happy.

The previous problem was that no one wanted to be the person to constantly be sending out messages saying the sink is dirty - we are all good friends and trust it will be done whenever the offender has time (we are all often flying in and out of the house), and its not fun to be the house villain.

My housemate Thommen had a very novel solution to this - he automated the task of detecting whether the sink is dirty to a robot system. It uses a neural network to detect whether the sink is dirty and emails everyone is the house and abuses us in Singlish if it's not up to spec.

Thommen Karimpanal George

I would like to make clear that this is entirely the work of my housemate Thommen - whose own personal website can be found here. I didn't do any work developing this! Thommen is a robotics and sensing expert so this was a piece of cake for him - he mostly works with path finding for robots.

The reason why I'm writing it up is because its cool - Thommen hasn't got a write up on it yet and I really thought this is something that the Internet would enjoy (especially a forum such as /r/shittyrobots). I've asked him if it is ok to share online. I also want to remember how the system worked and have a code repository later in case I (or anyone else reading this!) would like to do something similar.

Thommen's github with the code is found here.

The method works as follows:

First there is a webcam attached to the cupboard above the sink and plugged into the power outlet. This is a Wi-Fi camera, which connects directly to Thommens laptop. Now unfortunately Wi-Fi is a 1-1 connection, so he also used a Wi-Fi dongle on the laptop to give Internet access.


The camera can thus give a continuous feed to the laptop if necessary - however this would be a huge pain, also we would only like to receive updates on the sink status periodically

So instead there is a master-file in MatLab, which will in eminently use the feed to capture an image for evaluation. Currently this is set to once every three days.

This image is then used as an input to a neural network program which has be previously trained using catagorised images sets of the clean and dirty sink. To train this Thommen actually needed to have images of the sink in a clean state, and thus do all the washing up (so the project has already payed dividends by this point!) as well as some image of a dirty sink... which were not too hard to procure.

The use of neural network is ideal here. It would also be possible to calculate the difference between a picture of the clean sink and whatever picture is give and output a distance measure - however this would be less robust to the camera being knocked slightly, lighting conditions, and the movement of the arm of the tap which can move from side to side. Obviously it is in our interest for the classification of the image to be as accurate as to avoid false positives (false negatives would mean we still don't clean the sink - so pretty much the status quo).

The network was built and trained in MatLab using the default toolbox. This built upon some of Thommen's previous work from an old masters project where he had used a neural network to identify male and female faces. The output of the network is a true of false value, indicating the state of the sink. If the image tests positive then the image is the sent using directly from MatLab using a dedicated email account.

A future addition to the project would be to change the system from running on a laptop to onto a raspberry Pi. This would be great as, although the system is running automatically, it will only work if Thommen has his laptop at home.

This is especially a problem for him - as this would mean he's at home - and thus is available for domestic chores.

Using a Pi with a built in Wi-Fi dongle and cable webcam (or an extra dongle if the current Wi-Fi cam is still used) then this wouldn't be the case - we could just glue the pi to the cupboard next to the camera. However - getting MatLab on a Pi might be more of a problem. The solution would probably be to do the whole thing again in a different language (such at TensorFlow or Theano for python) but this would mean translating ALL of the code... which might take hours! This isn't going to happen anytime soon I imagine.

dirty sink email

When the image is sent via email, it is accompanied by a short phrase randomly chosen from a list. Usually a bit of Singlish (Walao! So messy ah! THIS CANNOT), urging us actually wash some dishes.

clean sink email

Or if the sink is clean we get a nice email!

The final stage of this project is now going to be evaluating how successful this is in making us do the washing up. Honestly - this might work. However - a more likely scenario, which I envision happening soon, is me or David screwing with the camera. This is likely as it is communicating via Wi-Fi, its open to an ARP poisoning attack, which means I may be able send different images to the network as input.

More coming soon - sending the sink on a psychedelic trip (in progress)

Dirty Sink

So for a while now I've seen many pictures online of people using neural networks to restyle photos (yes, yes, I'm aware theres an app for that now - Prisma in particular comes to mind, and I believe there is one for GIFs also) as well as using googles deep dream for photos. So I really wanted to get it working myself to play around with some images - the results weren't spectaucular I'll admit - mostly due to my lack of GPU processing power on my local machine, but here is a write up for myself and the reader on how to get everything working.

This guide will walk you through the setup for neural-style on Ubuntu to edit images.