Can we save our time in markets using a smarter scale for buying fruits and vegetables?

• Sunday, July 22, 2018

• By Jovica Turcinovic

Some time ago, I was on a vacation with my family in Greece.

We went to the Lidl supermarket a couple of times. It looks as a super organized store with competitive prices and a wide range of products.

We are really happy that our sons like quite a lot of fruits and vegetables. Our visits to supermarkets always end up with buying some of them.

The process I usually go through looks like this:
- pick a bag
- put fruit into the bag
- remember the ID of the given fruit
- put the on the scale
- follow the instructions how to print out the proper label
- stick the label to the bag
- repeat the process for all desired vegetables/fruits

The overall process takes some time and usually consists of certain destructions:
- you can't find the number
- on the way from the place where you found the fruit to the scale, you forget the fruit number
- the scale is occupied by some other person (a granny trying to figure out why a banana label is printed instead of an apricot label)
- the scale is out of function due to printer/paper issues

Sometimes you may be lucky to pick all desired vegetables/fruits and, with the help of an excellent supermarket employee, you complete the entire process very quickly.

On the other hand, on the market there is a solution with scales at cash registers.

Now, back to Lidl in Greece ...
I noticed that everybody just picked fruits and put them in the shopping cart, meaning that the whole process will be completed at the cash register.

I took a watermelon and soon finished at the cash register. I found out that the cashier had a scale in front of her.
She put the watermelon in front of her, touched the monitor few times and that was it.

But, she needed to check every single bag and to clarify the exact product.
After that, she had to click on the correct product from a usually huge list, or to manually insert the appropriate code.

A cashier scans thousands and thousands products on a daily basis. She needs to take care of cash, credit cards, etc.
Putting the cashier in a position where an additional task has to be executed continuously surely makes the cashier tired and become prone to making mistakes.

What if we may improve that process? What if we could minimize the risk?
What if customers do exactly the things which they really want to do at the supermarket
- choosing the desired products and finishing shopping as soon as possible without an unnecessary waste of time?

The idea to have a scale at cash register is really good. In that case, there is no need for additional labeling, all process points I mentioned earlier could be reduced to picking a bag (or some other special kind of packaging case), put the fruit in and that is all.

How could we improve the cashier's overall process to be much easier and to avoid all the unnecessary mistakes (become mistake-proof)?

For that purpose, I would employ the AI and CNN neural network.

Each scale or connected computer may have a camera.
Once the cashier puts fruit on the scale, the camera would be triggered - AI would process the image and recognize the product and, in most cases, nothing would need to be done additionally by the cashier.

The cashier would see the recognized product on the monitor in front of her.

Let’s just imagine ten types of apples which the cashier needs to constantly distinguish. On the other hand, current cutting-edge AI is able to recognize it easily.

In case AI is not certain about the product - the cashier may expect to see a list of products which are most likely to be the correct one.

Of course, using a bag would not be suitable any more - or it could be, but I cannot currently imagine how AI would recognize products properly.

A cardboard box with a lid could be a solution. Stores could have several box sizes.
The boxes could be branded. Some kinds of marketing campaigns could be applied from time to time.

The boxes would be made of recyclable material.

So, a lot of stuff can be combined together within just one small enhancement.

This could be a little “Kaizen” applied in stores.

One more thing worth mentioning is that, at self-checkout lanes, you cannot purchase alcohol and vegetables. Using a smart scale could resolve this problem.

So, let’s try and see how good AI is at recognizing fruits/vegetables.

The camera would take images always from the top, so my set of data would be similar to the real world scenario which I picked using the Google Image Search.
The images taken will be used for educational/testing purposes only.

I tested the following fruits:
- a Fuji apple
- a Golden Delicious apple
- a Granny Smith apple
- a Red Delicious apple
- a banana
- a strawberry

I picked around 20-30 images of each fruit category. I separated the training and validation set of the images.

I used a neural network similar to the one I already explained in more detail in the article.

The result looked very good. I achieved an accuracy of 93-94%. Everything was prepared in less than an hour.

I strongly believe that by having a real-world image set taken in a real environment, we may expect to obtain more accurate results.

What are your experiences regarding buying fruits/vegetables? Do you already have/know some interesting examples/solutions?