Project idea
To create a facial recognition system and automate recording working hours
Previously, KFC restaurant managers kept a paper log of each employee’s clock-in and clock-out times and entered the data into a time clock system manually. This resulted in human errors: managers could miscalculate the amount of time worked or forget to put down the clock-in/clock-out time of an employee.
There have been a few cases of restaurant partners hiring ‘phantoms’ who only existed on paper or people working under someone else’s ID.
Some restaurants had a time clock system equipped with fingerprint scanners. However, it was quite expensive and easy to come around using scotch tape.
A face recognition system is a much better fit for time clock automation. The system can later be expanded to check not only the clock-in/out times of employees but also the quality of their work at specific restaurant stations: the system can be connected to several surveillance cameras and adjusted to identify employees online.
Our task was to develop an app that can:
- Accurately identify restaurant employees,
- Register the actual clock-ins and clock-outs,
- Send the data to the server.
The app needs to be integrated with an ERP system called KFC DSR, as well as a corporate Workplace portal and software specifically used to register employee attendance.
Down the road, the app should become a full-fledged interactive portal. Instead of a whiteboard with paper notes, the employees are using right now, each restaurant will have a digital dashboard displaying key metrics, corporate news, and other useful information.
Solution
Equipment
The system has to be tamper-proof: it has to tell the difference between actual faces, photographs, and video clips. What matched our requirements was the 3D-camera Intel RealSense with a ToF sensor which plots a depth map to distinguish between an actual face and a flat image.
Some restaurants may occasionally have an unstable internet connection, meaning that cloud recognition services won’t work. The process has to take place locally, on the client’s side — the only thing we send to the server is the result. Such tasks can be easily done on an Android tablet.
A tablet has to have both a charger port and a USB port for the camera. Most of the usual tablets don’t meet this requirement. Therefore we found a 22-inch sensor screen tablet with Android OS installed. It doesn’t have a built-in battery but it has a 220V power cord in addition to several USB ports. This means we can connect it to the camera without blocking its access to power.
Interface design
We’ve implemented KFC’s corporate design and did our best at creating a friendly UI. For first-time users, the interface became a wow factor and a source of positive user experience: we talk directly to the users, wish them a great shift and give them extra information they may find useful.
We’ve also put a lot of thought into the UX, making the interface user-friendly and thoroughly covering borderline cases:
- the app detected a face and identified it, but there is no match in the database,
- the face is not detected,
- application error,
- server connection error,
- request timeout error, etc.
We’ve covered all the possible scenarios, identifying what can go wrong and what to tell users if it happens. It’s crucial not to put users on the spot and make them feel uncomfortable since it can easily affect their motivation and performance.
The app is also very informative: whenever it is in standby mode, it shows a carousel of banners, displaying the information on tasks and target values for each employee of a restaurant.
As for the ERP system called KFC DSR, we’ve added reports on actual clock-ins and clock-outs.
Image processing and recognition
Image processing and recognition take place on the client’s side, namely in the Android app. To do that we used a Face SDK library.
We get two frames from a 3D camera: one of them is in color, and the other one is a depth map. Those two are used in further calculations. The frames are transformed into a suitable format of images and passed to the library where the images undergo recognition.
The library identifies a face in the picture, crops the image, and creates a digital impression. With the help of a module called Recognizer, the impression is then compared to the others in the database. Whenever the library finds a match, it identifies whose face is in the picture. After that, a DepthLivenessEstimator module and a depth map help the app check whether the person in the picture is real or a flat image.
All results, whether successful or not, are then sent to the KFC DSR server.
Integration with KFC DSR
KFC DSR is built using a microservices architecture, while the facial recognition system was built as a separate service.
The Android app receives a current list of restaurant employees from DSR. This means that managers must take photographs of restaurant staff with their devices and add those to the database. The resulting photographs will be the ones used for facial recognition.
Whenever a face is successfully recognized, the system submits the actual clock in and clock out to the following:
- the KFC DSR ERP system to calculate performance and other key performance indicators of a restaurant,
- the employee clocking system, which is used to calculate salary.
RGMs and shift managers can now check the reports on actual hours worked via the DSR web interface.
Real-life testing
After it was thoroughly tested at Surf, we gave the app a trial run in a KFC restaurant. To track any failed cases of recognition and trace them back to their causes we supplemented the Reports section with a table for incidents. Over a month of real-life operation, the system had shown to be 100% accurate: there hadn’t been a single case when a face was not detected or identified.
Result
With human errors eliminated, the clocking process became much more efficient.
- The time clock system in restaurants became automated thanks to face recognition.
- The face recognition system runs error-free.
- The system is now tamper-proof.
- The app is the first step towards a full-fledged interactive portal eventually freeing the restaurants from paper notes. The information will become more accurate, timely, accessible, and broadcast in a centralized manner.
- The system leaves a lot of room for experiments: apart from faces it can also identify gender, age, emotions, recognize several faces in the picture and obtain information from surveillance cameras set up in the kitchen, the dining area, etc.
