Raspberry Pi 3 Gesture controlled digital Picture Frame

The first time I realised the possibilities of gesture controlled devices I was dazzled. It was the kick-started project of a smart bike assistant, the Haiku. The Haiku’s idea is to use flick gestures to switch between functions or to handle notifications, independent of possibly covered fingertips. No direct touching is required!

How simple is it to steer something without touching? Without taking off gloves or anything that hampers control as it might with common touch sensitive devices such as smartphones or tablets? Without leaving fingerprints or accidentally painting the unlock pattern on a smudgy touchscreen (take a look at a smartphone in grazing light!)?
Gesture control sounds highly convenient and probably more safe than touch control to me. Although, one could accidentally do something by moving a bit too close on a gesture control input. That is truly a side effect…

During a hacking night with fellow workers I learnt about Pimoroni’s skywriter. Attachable to either an Arduino controller or the Raspberry Pi the skywriter allows to recognise gestures such as flicks and taps as well as X/Y/Z 3D position sensing within a range of 15 cm. I ordered one to test it for another project… so why not use a skywriter to see through photographs in a directory? Displaying images in a digital picture frame with a convenient input method.

Components used

Raspberry Pi 3

Raspberry Pi 7″ Touch Screen (any other monitor for a Pi will do)

Skywriter breakout or HAT

Wiring

The wiring is described in pimoroni’s github repository. The Raspberry Pi 3 pinouts are described here.

Skywriter Raspberry Pi
GND GND
TRFR GPIO 27
RESET GPIO 17
SCL GPIO 3 / SCL
SDA GPIO 2 / SDA
VCC 3 V

Software

At pimoroni’s github repository some very good examples are found for using the skywriter either on an Arduino controller or a Raspberry. On the Raspberry the same libraries have to be installed, e.g. with the given shell command as described in the readme.

For the Raspberry I started with the Python example touch.py. Each recognised gesture and a move’s coordinates are printed on the console. The print statement in the move() method obscures the results of the recognised gestures, it can be commented.

Python’s TKinter toolkit is used to display pictures in a window. The TKinter mainloop runs in a separate thread.

Usage

The usage should be natural and intuitive.

  • Flicking from left to right will display the next image in the directory.
  • Flicking from right to left will switch back to the previous image.
  • Tapping into the centre of the skywriter will close the program
  • Tapping onto the lower end will minimise the image window.
  • Guess how to maximise the image window again!

This is the whole Python script to move through (holiday) pictures using gesture control on the digital picture frame:

#!/usr/bin/python

print('''
Switch between different images in a directory using the skywriter.

Swipe left to right: display next image in directory
Swipe north to south: display next image in directory
Swipe right to left: display previous image in directory
Swipe south to north: display previous image in directory

Tap the lower left corner to minimize the image window
Tap the upper right corner to maximize the image window

Press CTRL+C, ESC or tap the skywriter's center to exit.
''')
# use a Tkinter label as a panel/frame with a background image
# note that Tkinter only reads gif and ppm images
# use the Python Image Library (PIL) for other image formats
# free from [url]http://www.pythonware.com/products/pil/index.htm[/url]
# give Tkinter a namespace to avoid conflicts with PIL
# (they both have a class named Image)

import Tkinter as tk
from PIL import Image, ImageTk
from ttk import Frame, Button, Style
import gtk, pygtk
import time

import sys
import os
import signal
import skywriter
import threading
import random

pathToPictures = "/home/pi/Desktop/images/"

class ImageDisplay(threading.Thread):

    def __init__(self):
        threading.Thread.__init__(self)
        self.root = None
        self.start()

    def callback(self):
        self.root.quit()

    def run(self):
        if self.root == None:
        self.root = tk.Tk()
        self.root.title('My Photographs')

    self.root.mainloop()

    def setImageName(self, name):
        self.name = name

    def showImage(self, path):
        self.original = Image.open(path)

        # make the root window the size of the screen
        screen_size = getScreenSize()
        self.root.geometry("%dx%d+%d+%d" % (screen_size["width"],         screen_size["height"], 0, 0))
        #self.root.attributes("-fullscreen", False)
        #self.root.overrideredirect(True)
                       #self.root.geometry({0}x{1}+0+0".format(self.root.winfo_screenwidth(), self.root.winfo_screenheight()))
        ##self.root.geometry("{0}x{1}+0+0".format(screen_size["width"],       screen_size["height"]))
        self.root.focus_set()  # >-- move focus to this widget
        self.root.bind(">Escape<", lambda e: e.widget.quit())

        self.resized = self.original.resize((screen_size["width"],        screen_size["height"]), Image.ANTIALIAS)
        self.image = ImageTk.PhotoImage(self.resized) # keep a reference, prevent GC

        # root has no image argument, so use a label as a panel
        self.panel1 = tk.Label(self.root, image = self.image)
        self.display = self.image
        self.panel1.pack(side=tk.TOP, fill=tk.BOTH, expand=tk.YES)
        print "Display image " + path

    def updateImage(self, path):
        self.original = Image.open(path)
        # resize
        screen_size = getScreenSize()
        self.root.geometry("%dx%d+%d+%d" % (screen_size["width"], screen_size["height"], 0, 0))

        self.resized = self.original.resize((screen_size["width"], screen_size["height"]), Image.ANTIALIAS)
        self.image = ImageTk.PhotoImage(self.resized) # keep a reference, prevent GC

        # root has no image argument, so use a label as a panel
        self.panel1.configure(image=self.image)
        self.display = self.image
        print "Display image " + path

    def minimize(self):
        self.root.iconify()
    def maximize(self):
        self.root.deiconify()

    def stopThread(self):
        self.root.quit()
        self.do_run = False  # stop thread

#---Utilities---#
def getScreenSize():
    window = gtk.Window()
    screen = window.get_screen()
    print "width = " + str(screen.get_width()) + ", height = " + str(screen.get_height())
    screen_size = {}
    screen_size["width"] = screen.get_width()
    screen_size["height"] = screen.get_height()
    return screen_size

def findImages(directory):
    imageList = []
    for file in os.listdir(directory):
        if file.endswith(('.jpg','.JPG','.jpeg','.JPEG')):
            print(file)
            imageList.append(file)
    return imageList

def increaseIndex():
    global index, images
    index += 1
    # start again with index 0
    if index >= len(images):
	index = 0

def decreaseIndex():
    global index, images
    index -= 1
    # start again with index max
    if index < 0:
	index = len(images) - 1

def nextImage():
    global imageDisplay, images, index, pathToPictures
    increaseIndex()
    print "image " + images[index] + ", index=" + str(index) + "(" + str(len(images)) + ")"
    imageDisplay.updateImage(pathToPictures + images[index])

def previousImage():
    global imageDisplay, images, index, pathToPictures
    decreaseIndex()
    print "image " + images[index] + ", index=" + str(index) + "(" + str(len(images)) + ")"
    imageDisplay.updateImage(pathToPictures + images[index])

#---detect gestures on skywriter---#
@skywriter.flick()
def flick(start,finish):
    print('Got a flick!', start, finish)
    if (start == "west" and finish == "east") or (start == "south" and finish == "north"):
        print "Display next image in directory"
        nextImage()
    if (start == "east" and finish == "west") or (start == "north" and finish == "south"):
        print "Display previous image in directory"
        previousImage()

@skywriter.touch()
def touch(position):
    print('Touched!', position)
    if (position == "center"):
        print "Exit image display"
        imageDisplay.stopThread()
        sys.exit()
    if (position == "south"):
        print "minimize image window"
        imageDisplay.minimize()
    if (position == "north"):
        print "maximize image window"
        imageDisplay.maximize()

# parse picture folder
images = findImages(pathToPictures)

# reset index
index = 0

# launch image window as thread
imageDisplay = ImageDisplay()

def main():
    try:
        print "Skywriter image display launched"
        getScreenSize()
        print "Images found: "
        for i in images:
            print i
        global imageDisplay, pathToPictures
        imageDisplay.showImage(pathToPictures + images[index])
    except KeyboardInterrupt:
        print "Exit"
        imageDisplay.stopThread()
        sys.exit()
    pass
if __name__ == '__main__':
    main()

Result

skywriter_photo
Gesture controlled Picture Frame
Advertisements

Raspberry Pi 3 wears a Display-O-Tron HAT

Two weeks ago I laid my hands on a Raspberry Pi 3. For sure I did not buy the 10.000.000th one. I ordered just a week before this milestone wich was a good excuse for discounts, but anyway, a Raspberry Pi is a nice and versatile thing to play with.

Since there are already a huge number of tutorials on how to set up and configure a Raspberry Pi I will spare the details. For this example of using the Display-O-Tron HAT on a Raspberry lets assume your Pi is already set up with a Linux distribution, knows Python and is connected to the internet.

dothat

An Idea

My idea is to use the Display-O-Tron HAT as display for several purposes. The Display-O-Tron HAT comes with a three line LCD display, a row of very bright LEDs on the right and 6 touch buttons. Each button can trigger the display of different information such as

  • system statistics
  • the number of unread emails, signaling incoming messages
  • process states (dead or alive?)
  • date and time
  • head lines of a news feed

The Sources

Based on the examples for the Display-O-Tron HAT on github I created a simple, straightforward, quick and dirty Python script.

On the push of a button the LCD display will be alighted for a couple of seconds in a different colour and the desired information will be retrieved and displayed. After a couple of seconds the backlight LEDs will be turned off. This is realized using threads that will finish after the desired time has passed.
The functionality that is implemented is a check for unread emails on googlemail’s IMAP servers, a simple date and time display, a quick internet connection and system status check and a status check of certain processes.
If unread emails were detected the LEDs on the (b)right side will blink three times in a row. Every minute the display will switch back to the default view: date and time.

#!/usr/bin/env python
print('''
Switch between different views using the Display-o-Tron HAT buttons

Main button: Display date and time
Right: Display number of unread emails
Left: Display system statistics
Down: Display process status
Up: TODO find something else to display

Press CTRL+C to exit.
''')

import dothat.touch as touch
import dothat.lcd as lcd
import dothat.backlight as backlight
import signal
import sys

import psutil
import urllib2
import subprocess
import imaplib
import time
from datetime import date
import calendar
import threading

'''

Captouch provides the @captouch.on() decorator
to make it super easy to attach handlers to each button.

The handler will receive 'channel'( corresponding to a particular
button ID ) and 'event' ( corresponding to press/release ) arguments.
'''

'''DISPLAY utilities'''
# store time of last button touch
# to be able to reset backlight LEDs
oldTime = time.time()
turnedOff = True

def resetBacklightLEDs():
    global oldTime
    oldTime = time.time()
    global turnedOff
    turnedOff = False

def clearDOT():
    lcd.clear()

def colorDOT(r, g, b):
    backlight.rgb(r, g, b)

def turnOnLEDsDOT():
    # turn on LEDs one by one in a row
    for led in range(6):
        backlight.graph_set_led_state(led, 1)
        time.sleep(0.1)
    pass

def turnOffDOT():
    colorDOT(0, 0, 0)  # backlight off
    backlight.off()
    backlight.graph_off()  # side LEDs off
    global turnedOff
    turnedOff = True

def dotClock():
    clearDOT()

    d = time.strftime("%d.%m.%Y")
    t = time.strftime("%H:%M")
    the_date = date.today()
    day = calendar.day_name[the_date.weekday()]
    print day + ", " + d + " / " + t

    lcd.set_cursor_position(3, 0)
    lcd.write(day)
    lcd.set_cursor_position(3, 1)
    lcd.write(d)
    lcd.set_cursor_position(5, 2)
    lcd.write(t)

def dotMails(login, password):
    nofUnreadEmails = check_googlemail(login, password)
    if nofUnreadEmails > 0:
        lcd.write("Unread Emails: " + str(nofUnreadEmails))
        showNewMessages(200, 0, 0)

    else:
        lcd.write("No new mail")

def dotSystemStats():
    lcd.set_cursor_position(0, 0)
    lcd.write(check_internet())

    lcd.set_cursor_position(0, 1)
    lcd.write("CPU: " + check_CPU())

    lcd.set_cursor_position(0, 2)
    lcd.write("Memory: " + check_memory())

def check_internet():
    try:
        # ping google to check whether internet connection works
        response = urllib2.urlopen('http://www.google.com', timeout=1)
        return "Internet: OK"
    except urllib2.URLError as err: pass
    return "Internet connection broken"

def check_CPU():
    cpu_usage = str(psutil.cpu_percent(interval=None)) + " %"
    print "CPU usage: " + cpu_usage
    return cpu_usage

def check_memory():
    mem = psutil.virtual_memory()
    # print "Memory: " + str(mem)
    memory_used = str(mem.percent) + " %"
    print "Memory used: " + memory_used
    THRESHOLD = 100 * 1024 * 1024  # 100MB
    if mem.available >= THRESHOLD:
        print("Warning, memory low")
        return "Warning, memory low"
    return memory_used

def get_pid(name):
    try:
        pids = subprocess.check_output(["pidof", name])
    except subprocess.CalledProcessError as pids:
        print "error code", pids.returncode, pids.output
        return ""
    return map(int, pids.split())

def get_single_pid(name):
    return int(check_output(["pidof", "-s", name]))

def check_process(name):
    PID = get_pid(name)
    if len(PID) == 1:
        print "PID " + name + ": " + str(PID[0])
        p = psutil.Process(PID[0])
        status = ""
        if p.status == psutil.STATUS_ZOMBIE:
            status = "Process " + name + " died"
            print status
        else:
            status = "Process " + name + " OK"
            print status
            return status
        return ""

def dotProcessStats():
    lcd.set_cursor_position(0, 0)     
    process1 = check_process('geany')
    if len(process1) > 1 :    
        lcd.write(process1)
    else:
        lcd.write("geany is dead.")

    lcd.set_cursor_position(0, 1)
    process2 = check_process('bash')
    if len(process2) > 1 :
        lcd.write(process2)
    else:
        lcd.write("bash is dead.")

    lcd.set_cursor_position(0, 2)
    process3 = check_process('firefox')
    if len(process3) > 1 :
        lcd.write(process3)
    else:
        lcd.write("firefox is dead.")

def check_googlemail(login, password):
    # if new mail return # emails
    obj = imaplib.IMAP4_SSL('imap.gmail.com', '993')
    obj.login(login, password)
    obj.select()
    nofUnreadMessages = len(obj.search(None, 'UnSeen')[1][0].split())
    print "Unread emails: " + str(nofUnreadMessages)
    return nofUnreadMessages

class ShowNewMessagesThread (threading.Thread):
    red, green, blue = 0, 0, 0  # static elements, it means, they belong to the class

    def run (self):
        colorDOT(self.red, self.green, self.blue)
        for i in range(0, 3):
            turnOnLEDsDOT()
            time.sleep(2)
            turnOffDOT()
            time.sleep(2)

            if i == 2:
                print "stop ShowNewMessagesThread"
                self.do_run = False  # stop thread
                break

snmt = ShowNewMessagesThread()
def showNewMessages(r, g, b):
    global snmt
    if snmt.is_alive():
        return
    snmt = ShowNewMessagesThread()
    snmt.red = r
    snmt.green = g
    snmt.blue = b
    snmt.daemon = True  # enable stop of thread along script with Ctrl+C
    snmt.start()

class AlightThread (threading.Thread):
    red, green, blue = 0, 0, 0  # static elements, it means, they belong to the class

    def run (self):
        colorDOT(self.red, self.green, self.blue)
        while True:
        if turnedOff == False:
            if time.time() - oldTime > 5:
                print "stop AlightThread"
                turnOffDOT()
                self.do_run = False  # stop thread
                break

at = AlightThread()
def alightDisplay(r, g, b):
    global at
    if at.is_alive():
        return
    at = AlightThread()
    at.red = r
    at.green = g
    at.blue = b
    at.daemon = True  # enable stop of thread along script with Ctrl+C
    at.start()

class ClockThread (threading.Thread):
    def run (self):
        print "run update clock thread " + str(self)
        dotClock()
        self.do_run = False  # stop thread

    def stopClockThread(self):
        self.do_run = False  # stop thread

ct = ClockThread()
def updateClock():
    global ct
    if ct.is_alive():
        print "Clock thread " + str(ct) + " is alive."
        ct.stopClockThread()
        return
    print "Launching clock thread " + str(ct)
    ct = ClockThread()
    ct.daemon = True  # enable stop of thread along script with Ctrl+C
    ct.start()

'''DOT touch button handler'''
@touch.on(touch.UP)
def handle_up(ch, evt):
    print("Up pressed: TODO find another useful display idea")
    clearDOT()
    alightDisplay(255, 0, 255)
    lcd.write("Up up and away: TODO")
    resetBacklightLEDs()

@touch.on(touch.DOWN)
def handle_down(ch, evt):
    print("Down pressed: display process states")
    clearDOT()
    alightDisplay(255, 0, 0)
    dotProcessStats()
    resetBacklightLEDs()

@touch.on(touch.LEFT)
def handle_left(ch, evt):
    print("Left pressed: display system statistics")
    clearDOT()
    alightDisplay(0, 100, 200)
    dotSystemStats()
    resetBacklightLEDs()

@touch.on(touch.RIGHT)
def handle_right(ch, evt):
    print("Right pressed, check for new email")
    clearDOT()
    alightDisplay(100, 200, 255)
    dotMails('email adress', 'password')
    resetBacklightLEDs()

@touch.on(touch.BUTTON)
def handle_button(ch, evt):
    print("Main button pressed: show date and time")
    clearDOT()
    alightDisplay(255, 255, 255)
    updateClock()
    resetBacklightLEDs()

@touch.on(touch.CANCEL)
def handle_cancel(ch, evt):
    print("Cancel pressed!")
    clearDOT()
    backlight.rgb(0, 0, 0)
    lcd.write("Cancel")
    resetBacklightLEDs()
    alightDisplay(20, 20, 20)

'''main'''
if __name__ == '__main__':
    try:
    while True:
        timeDiff = time.time() - oldTime
        # print "time diff: " + str(timeDiff)

        # update clock every minute
        if timeDiff > 59 or timeDiff < 0.5:
        oldTime = time.time()
        alightDisplay(255, 255, 255)
        updateClock()
        time.sleep(1)
    except KeyboardInterrupt:
        print "exit"
        clearDOT()
        turnOffDOT()
        sys.exit()

 

Links

Raspberry Pi 3 Model B

Display-O-Tron HAT

https://github.com/pimoroni/dot3k

https://www.python.org/