Smarter Door

Links

Smarter_Door_Log
Amazon Reckognition Tutorial
GitHub
Poster
Website

Team Members

• Jonathan Strek
• Katie Cardwell
• Andrew Koltz
• Ethan Shry (TA)
• Professor Feher

Overview

We are building a door security system that sends the user a picture when an unknown person tries to open the door. The user will receive a text and be able to interact with the door by responding over text. Our security system includes a red light that illuminates when the user says the visitor is not welcome, or when the user has not authorized entry after a set period of time.

Goals

• Photograph anyone who tries to open door
• Recognize owner from an uploaded photo collection
• Text owner if unrecognized face is seen
• Respond to owner's text with security light
• Design plastic housing to make unit presentable

The goal of this project is to build a facial recognition security system for a door. The system will increase security by taking a picture of anyone who tries to open the door. If the person is not recognized by the system, a text will be sent to the user informing him/her that someone has tried to open the door. If the user does not recognize the person in the picture, they can respond via text and a security light will go on. This light should discourage people from trying to open a random door, and it also indicates to law enforcement that someone unknown has tried to enter the door.

Gantt Chart

Challenges

• Learning Python
• Gaining familiarization with Twilio and learning how to interpret incoming texts using Twilio
• Security of user interface
• Building a 3D casing for raspberry pi detector
• Ensuring motion detector only signals to camera to take a picture when there is truly someone in front of the door rather than when people walk by or there is motion in the background

Security Concerns

• Who can upload a picture to be recognized
• Who receives the text from the Raspberry Pi
• Who can text the Raspberry Pi
• Can someone intercept the message
• How easy is it to change the recipient to someone else
• Is the door unit sturdy

Budget

• Raspberry Pi: provided ($35)
• Raspberry Pi Camera: provided ($12)
• Infrared PIR motion Sensor: ($12) (https://www.adafruit.com/product/189?gclid=CjwKCAiAqaTjBRAdEiwAOdx9xo34FxU9DTSkwjdc8BoTUGuFJe6SMUgkO7Ussy_YMhy5Z-2xehN72xoCVYsQAvD_BwE)
• Twilio Messaging ($10)
• Amazon Web Services Recognition: ($5)
• 3D Printing software and materials: ($10)
• Small breadboard ($6)
• a LED
• a 330 ohm resistor

Total: $90

Why this project?

Security in dorms or houses is always a concern, and we hope to alleviate such worries. The increase in dependence on online retailing means more strangers than ever are using your address to find your place of residence. With Smarter Door you will not have to worry about who is at the door. Dorms can also be safer as a red light outside of a dorm will be a sign that an RA or even WUPD should be called.

Design and Solutions

General Overview and Setup:

• Download development environment (we used Visual Studio and Atom)
  • Download Python, Node, CSS, and HTML extensions
• Download MySQL
• Create AWS instance
• Create AWS S3 bucket
  • Connect to raspberry pi
• Create AWS database instance
  • Connect to MySQL
• Purchase Twilio number that can send and receive messages

General Project Diagram:

How Our Product Runs:

Below is the decision tree of our product. The color of the LED described within the parenthesis helps users see where the device is in the facial recognition process.

Door:

Our project uses a raspberry pi to combine the inputs and outputs of a PIR motion sensor, a Raspberry Pi Camera and various colored LEDs. If motion is detected (one of the blue LEDs light up), the camera takes a picture (while the other blue LED is on) and sends the picture to S3 to undergo facial recognition (a yellow light is shown to indicate the picture being processed). This image is labeled with a timestamp of when it was taken called its accessid. This image can be called upon used its accessid and allows us to pull upon certain attempts of entry which will be utilized later as we collect the information from the pi to send to the database which is then utilized to create a website. If Amazon Rekognition recognizes the user, a green light is displayed for 7 seconds before running again. If the user is not recognized, the picture is sent to the owner of the door asking for approval. If the owner texts back a positive response, a green light is displayed. If they text back a negative response, the blue, yellow and green light all go on, indicating that this person is an intruder. A button is placed into our system for demoing purposes, acting as a backup for the motion sensor. Below is the circuit schematic for our physical system and a flowchart to display the process of recognition at the door. The 9V battery powers the system due to collaboration issues we had between the pi and the PIR motion sensor.

Server:

• AWS Lightsail Server alerts the door owner that there is someone at the door, sends picture to user, and receives reply
• These objectives are achieved by utilizing routes and Twilio. The server is continuously listening on an open port, when the Pi sends a picture to the bucket it gives the server the name of the photo. The server then uses the name along with a static ip for the public s3 bucket to send the picture using the Twilio number. The user is prompted to reply to the photo, and the text is sent back to Twilio, which uses a webhook to send the content of the reply back to the server. The server can send that information to the Pi, which changes lights based on the reply.

Twilio:

• Setup:
  • We chose to use Twilio because of the ease to install. Twilio has been fairly widely used and as such has good documentation and trouble shooting help.
  • The installation requires purchasing a Twilio number. Each Twilio number comes with different functionality, but we only needed sms and mms abilities. Once the number is purchased, Twilio needs to be installed in the console, and then a simple script is written. Twilio has a startup guide for different languages, which made it easy to figure out how to get it working. We started by putting Twilio on the Pi in Python to make sure the texting, image texting, and reply texting worked. Then we wrote the same scripts in Javascript and added them to the server. Twilio texts using your phones network provider, meaning it cannot utilize iMessage on an iPhone. As a result we had trouble testing in the lab, but AT&T has stronger reception in Lopata, and when we started sending messages to a phone on AT&T it worked.
• Issues:
  • Sending texts from Twilio to a phone was as easy as following the directions on Twilio's startup page. However, texting back to the number had a few issues. We initially used Ngrok as a webhook on the Pi, which allowed Twilio to forward the message so it was publicly available and the Pi could access it. Ngrok had issues with our Lightsail server, so we used a PM2 server, which was more difficult to install.
    • Once the PM2 server was set up it was easy to change the incoming text portion of the Twilio number to look for the static URL of the server.
  • We also had issues with interpreting the received messages to the Twilio number. We used Marchex Sentiment Analysis to determine if the message was positive or negative, but after installing the extension, it was difficult to access the result of the analysis. We had to parse the object returned by the incoming message and find the result object, which returned a number between 0 and 1, with more positive messages closer to 1, and more negative messages closer to 0.
    • If the analysis was greater than 0.5 we set the reply as allowed, and if the analysis was less than 0.5 we set the reply as denied.

Database:

• Stores information and pictures of who tried to enter

S3:

S3 is a storage "bucket" run by Amazon Web Services. In this project we created a file within our S3 bucket. In this file we saved the images of our 3 faces with the format name.jpg, which we then coded to be the owners of the door. This bucket also holds all of the pictures of those who tried to enter (when motion was detected) with a filename indicative of the time of the occurrence. In the AmazonRekognition tutorial linked to this project page, one can learn how to send images to the S3 bucket and pull pictures from the S3 bucket using your Amazon Web Services credentials and python.

Web UI:

• Allows the user to access the information from the database
• Can see the intruder id, when the intruder came to the door, your decision, and a link to the picture the door took

AWS rekognition:

AWS Rekognition is the program we used to detect and compare faces. It determines a similarity confidence number based on the location and prevalence of facial landmarks. We can compare this confidence number to a chosen threshold to choose to identify them as an owner or not. In this project we used Rekognition to detect a face in the taken picture and compare it to a list of faces that we had uploaded to our S3 bucket as the "owners". A detailed tutorial of how to create an AWS account, how to use S3 buckets and Rekognition and how to utilize the Raspberry Pi Camera in this process is uploaded and linked to this page. An example of a comparison of our three owners Johnny, Andrew and Katie is shown below. ] ]

API:

• Communicates with the door owner using Twilio

3D Casing:

Features & Design Choices:

We used Visual Studio to code in HTML, CSS, Node, and Python to run our server, and our Raspberry Pi. The Amazon Web Service Lightsail server allowed us to communicate between the door, the facial recognition, and the end user. The Raspberry Pi controls the motion sensing and picture taking, and uploads the pictures to AWS to be used in facial recognition.

The majority of the project is comprised of the Raspberry Pi, AWS server, facial recognition, bucket, and Twilio. We chose the Pi because it is easy to use, yet complex enough to handle communications with the server. Amazon Web Service was chosen because it provided us with an affordable and easy to use suite of web services to handle communications. The Lightsail server listens to the pi, contacts the user, receives the decision, and sends the decision back to the Pi. The amazon facial Rekognition was chosen because it made it really easy to utilize our s3 bucket and aws server. It also printed out facial recognition results in a very tangible "confidence level" number which made it easy to analyze. The S3 bucket allowed us to store the new images from the Pi and send them with the server to the user. The S3 bucket was also very easy to upload pictures directly to for the purposes of uploading owners and also is a relatively cheap storage mechanism even for the large amount of pictures that we are storing in it. We chose Twilio because it was a simple way to communicate from a third party to the user. For a small amount anyone can purchase a Twilio number with the ability to send texts, and with minimal console work can write a script to send anything from that message to any number in the United States or Canada.

Results:

Currently the Smarter Door can sense that someone is in front of the door, take their picture, saved with an indicating filename of when it was taken, and upload that picture to an s3 bucket. Then the picture is sent to the user who can respond whether or not they want the person to enter. The door will then display lights differently depending on the reply. A database saves the intruder ID, the time the door was accessed and the image from the access attempt. This database then was used to create a website that includes a table of these different aspects. A video of the hardware is included below and a screenshot of the user log and a link to the user log can be found above.

• Reach Goals:
  • Allow multiple users for the same door
  • Allow users to upload their picture from the website
  • Allow users to add their friends to be approved
  • Create an unlocking mechanism that could let someone in the door instead of approving them

Next Steps:

• One change we would make in transitioning to the next phase of our product is to remove the 9V battery from the design. Due to issues running the motion sensor in connection with the raspberry pi the battery was needed, however, eliminating this battery from the design will decrease the weight and size of the project and will make for a simpler, cheaper product to produce.
• Another goal for the future of this project is to create a website where pictures can be securely uploaded as owners rather than having to manually enter the image into the S3 bucket. If this product were to be mass produced we would need this easy and secure set up to be in place.

References

• Twilio set up PI https://www.twilio.com/docs/sms/quickstart/node#send-an-outbound-sms-message-with-nodejs
• Proposal presentation Proposal Presentation
• Twilio add on installation [1]