Blood_Pressure_Recorder_Mobile_App

Summary: Developed a mobile APP to recognize picture of blood pressure monitor and store the data to sqlite3 database. The use can view, add, edit the records.

Contents:

• 1. Introduction
• 2. Home screen
• 3. Records list screen
• 4. Add records screen
  • 4.1 Base interface
  • 4.2 Adding records by hand
  • 4.3 Adding records by file chooser
  • 4.4 Adding records by camera
• 5. The back-end method
  • 5.1 recog.py: recognize value and extract time
  • 5.2 sql_op.py: database operations
• 6. Improvement ideas
• 7. Reference:

Check the code at here.

1. Introduction

In this repo, I developed a mobile APP to recognize picture of blood pressure monitor and store the data to sqlite3 database. The use can view, add, edit the records.

In my experience, taking a picture is the easiest way to record important information, such as recording blood pressure, recording logs on the running machine.

About the files:

📦Blood Pressure Recorder Mobile App
 ┣ 📂example_img
 ┃ ┣ 📜IMG_20210904_180521.jpg
 ┃ ┣ 📜pick2.jpg
 ┃ ┗ 📜pick5.jpg
 ┣ 📜BP.db
 ┣ 📜bp_img.png
 ┣ 📜kv_str.py
 ┣ 📜line_img.jpg
 ┣ 📜main.py
 ┣ 📜recog.py
 ┣ 📜seg_image.ipynb
 ┗ 📜sql_op.py

The fron-end is designed with kivy, kiviMD, Plyer, back-end is implemented with python, OpenCV, numpy. There are three screens:

• Home screen: show line map the records
• Adding screen, user can add records by filling out form, or uploading file, or taking a picture
• Records list screen, list all records, user can deleting selected record

2. Home screen

In home screen, all records are shown in scatter plot

When new record is inserted or deleted, the figure will be updated.

3. Records list screen

The third creen is Viewing and editting screen, it lists all records ordered by date and time. The user can delete record by clicking deletion button.

In the below picture, the record with id 15 is deleted.

left: before deletion right: after deletion

4. Add records screen

4.1 Base interface

Below is the base interface of the middle screen.

• left one is the base interface
• right one the the camera screen when clicking camera icon

Explaination of the icons:

• black picture: showing the photo chosen or taken by user
• upload icon: choosing photo from local folder
• camera icon: when clicking, it will switch to camera screen, by clicking the left arrow icon, it will switch back to the previous screen
• High, Low, Date, Time: text fields
• calendar icon: Choosing date from pop up window
• time icon: choosing timme from pop up window
• cancel icon: cancel this try, and clear all text fields
• add icon: adding records to database, and clear all text fields

Below is the date picker window(left) and time picker window(right).

4.2 Adding records by hand

The user can add record by filling out the text files.

In the following pictures:

• left one shows the filled text fields
• middle one shows the result after clicking add icon
• right one shows the added result in list

4.3 Adding records by file chooser

The user can add record by choosing photo from local folder. By clicking the upload icon, a folder explorer window will pop up, user can choose photo. Below are some results:

Below pictures show the second added results in scatter plot and list.

The App recognizes the pressure value correctly, it also extracts the date and time information from the file with the help of library exifread.

4.4 Adding records by camera

The user can add record by taking photo with camera. By clicking the camera icon, it will switch to camera screen, by clicking the left arrow icon, it will switch back to the previous screen.

Below are the results:

• Left one is the the camera screen
• Right one is the adding record screen

Note: By using iVCam, I can use my cellphone’s camera when coding on PC.

We can see that the App recognizes the pressure value correctly, and sets current time as it’s date and time.

5. The back-end method

The back-end code is in files:

• recog.py
• sql_op.py

5.1 recog.py: recognize value and extract time

The digit on the blood pressure monitor is seven segment digits, which are not always easy to recognize.

I tried EasyOCR and EAST text detection model.

Both of them can detect the words in the photo except the large digits in the middle, which are the blood pressure values.

Below are the text detection results, left is result of EasyOCR, right is result of EAST.

Both of them are powerful tool in text detection and/or recognition. But in my case, they don’t work.

After trying some deep-learning based methods, I didn’t get a better solution. So my decided to segment the digits and recognize them by their seven segment area.

DIGITS_LOOKUP = {
    (1, 1, 1, 0, 1, 1, 1): 0,
    (0, 0, 1, 0, 0, 1, 0): 1,
    (1, 0, 1, 1, 1, 0, 1): 2,
    (1, 0, 1, 1, 0, 1, 1): 3,
    (0, 1, 1, 1, 0, 1, 0): 4,
    (1, 1, 0, 1, 0, 1, 1): 5,
    (1, 1, 0, 1, 1, 1, 1): 6,
    (1, 0, 1, 0, 0, 1, 0): 7,
    (1, 1, 1, 1, 1, 1, 1): 8,
    (1, 1, 1, 1, 0, 1, 1): 9,
    (0, 0, 0, 0, 0, 0, 0): 0
} #the last zero is for empty at the left-most side of values

The whole process go in this way:

1. Read image, apply Gaussian Blur to clear out noise, apply Canny method to get the contours in the image. Used cv2.GaussianBlur, cv2.Canny.

1. Sort the contours by length, the first four contours shoulb be the outline of the LED screen. Used cv2.findContours, cv2.arcLength, cv2.approxPolyDP.

   

2. Convert the image to white-black picture, used cv2.threshold

   

3. Apply Opening method to decrease noise again, used cv2.getStructuringElement, cv2.morphologyEx
4. Setmenting the digits horizontally and vertically

   

5. Extracting each digit by checking each segment’s area Take number 2 as example,its seven segments matrix is [1, 0, 1, 1, 1, 0, 1], by checking the above DIGITS_LOOKUP matrix, it’s recognized as 2. The seven recognized digits are as follows:1,2,8, ,9,7

   

The details are in the file seg_image.ipynb.

5.2 sql_op.py: database operations

There are very basic SQL operations in this file.

import sqlite3
import time

def connect():
    conn = sqlite3.connect("BP.db") #BP means blood pressure
    cur = conn.cursor()
    #time is text datatype, https://tableplus.com/blog/2018/07/sqlite-how-to-use-datetime-value.html
    cur.execute("CREATE TABLE IF NOT EXISTS bp_table (id INTEGER PRIMARY KEY, time int, high int, low int)")
    conn.commit()
    conn.close()

def insert(time, high, low):
    conn = sqlite3.connect("BP.db")
    cur = conn.cursor()
    cur.execute("INSERT INTO bp_table VALUES (NULL,?,?,?)", (time, high, low))
    conn.commit()
    conn.close()
    
def view():
    conn = sqlite3.connect("BP.db")
    cur = conn.cursor()
    cur.execute("SELECT * FROM bp_table")
    rows = cur.fetchall()
    conn.close()
    return rows
def delete(id):
    conn = sqlite3.connect("BP.db")
    cur = conn.cursor()
    cur.execute("DELETE FROM bp_table WHERE id=?", (id,))
    conn.commit()
    conn.close()
    
def update(id, time, high, low):
    conn = sqlite3.connect("BP.db")
    cur = conn.cursor()
    cur.execute("UPDATE book SET time=?, high=?, low=? WHERE id=?", (time, high, low, id))
    conn.commit()
    conn.close()

6. Improvement ideas

• Gneralize the recognition method, deep-based method is better, or train a model on blood picture pictures
• The UI design
• More interactive function
• More functions: such as recognizing the log on running machine, log on bandsports bracelets, mileage of my car, gallon and price after fueling, etc.
• Start up speed

7. Reference:

• OpenCV
• OpenCV Text Detection (EAST text detector)
• Kivy
• KivyMD
• Plyer
• XCamera
• Buildozer
• Many impressive tutorial videos on Youtube