Introducing MyMoves: system-wide gestures for Harmattan

Hiyouall! I’d like to demonstrate the results of the small 3-day home hackathon I ended up having : )

(Links to sources and .debs in the bottom, I recommend reading through first : P )

It’s a thing called MyMoves. MyMoves is an attempt to bring configurable
gestures to Harmattan, written in QML/C++. The idea of MyMoves is to be able
to create a gesture, and attach an arbitrary shell command into it.

The current prototype of MyMoves consists of two parts:
– mymoveserver, handles the touch events via XRecord, saves the gestures and performs the recognition
– Configuration UI for adding new gestures and starting/stopping the gesture observation
The server is controlled by the configuration UI via D-Bus commands

So. What on earth can we do with this? Let’s take a closer look. Here’s the configuration UI:

It consists of three buttons: “New Move“, “Start observing gestures“, and “Stop observing“.
The first thing to do is to is to create new gestures. Pressing “New Move” shows the screen for adding a gesture:

It’s a very simple screen with a box where you draw the gesture (it’s a QmlCanvas btw)
and buttons for saving and clearing the gesture box. Here’s one drawn gesture:

Let’s take a look at the video on how adding gestures goes:

Now that the gestures are set up, it’s time to attach them to shell commands,
which currently means getting dirty with the terminal. The gestures are saved
into files called mymoves[number] under /home/user/MyDocs/moves. Each
gesture has the shell command and the gesture data, so the start of a gesture file looks
something like this:

Add your command here
-128 430
-105 436
-95 441
-76 447
-57 455
-38 460
-21 466
-2 474
14 479

Now we can just edit the file, and add the command we want to run
on that specific gesture, let’s say e.g. the calendar, so we replace
Add your command here
with
/usr/bin/invoker –no-wait –type=m /usr/bin/organiser -showWindow 1 &

The command needs to be added just like to a command to a terminal, so no quotes.
Make sure to add & to the end of the command, so it’ll start in a non-blocking way.
Also, the MyDocs folder is in users possession, so use su – user to get permissions for editing the files.
Do that for each of the mymoves[number] files, and you’re set up!

Now using the gestures is as simple as firing up MyMoves and pressing “Start observing gestures“.
Here’s MyMoves in action:

Enough for the talking! Where can I get it?

Source code: http://gitorious.org/mymoves
Harmattan debs: http://koti.kapsi.fi/~sh8dfwk/mymoves

Installation only via command line. SSH to your phone, copy the .debs under /home/user/MyDocs,
and run dpkg -i mymoveserver_0_0_1_armel.deb mymoves_0_0_1_armel.deb.

Plans for the future:
– Making the server more robust
– Decoupling the server from the configuration UI so that it starts from the boot
– A usable UI for configuring everything (such as dialogs for selecting an application to run on a gesture etc)
Multi-touch gestures : )
– anything else that pops into my mind

Have fun! 🙂

About sandst1
A technology-oriented guy interested in all-round creativity, be it music, art, or putting software together for creating something new and expressing oneself

27 Responses to Introducing MyMoves: system-wide gestures for Harmattan

  1. Pingback: MyMoves app brings system-wide gestures for #N950 | LoveMyNokia.com

  2. Hnry says:

    nice!

  3. Pingback: MyMoves – Programme durch “Gestures” auf dem N9 öffnen!!!

  4. Jed says:

    You sir, are a legend!

    This is roughly the kinda thing I was hoping would happen after seeing this:
    http://felipec.wordpress.com/2011/08/14/n9-swipe-undocumented-feature-activate-sane-behavior/

    Something like quicksilver on OSX, but designed for touchscreens instead of kbs!
    http://qsapp.com/

  5. sandst1 says:

    Thanks!

    Talking about quicksilver, investigating multitouch gestures is next on my list : )

  6. Jed says:

    [Talking about quicksilver, investigating multitouch gestures is next on my list : )]

    Awesome, you legend!

    Are you familiar with apps like quicksilver? They’re truly a god-send.
    And something like them would be a “killer app” on the N9!

    • sandst1 says:

      I’m not too familiar with them but will definitely take a closer look : )

      • Jed says:

        I think you’ll be impressed….
        I think there’s actually some better alternatives to Quicksilver on OSX nowadays.
        But it was the one that started it all.
        Have used Windows at all for about 4yrs, but no doubt there’s something similar.

  7. Martin Alderson says:

    Excellent!

    I don’t have an N9(50) to test this but it looks great. Over on Felipe’s post on configurable swipes I was thinking it would be quite hard for the community to extend it to run arbitrary commands or use extra gestures etc. You’ve made a decent start to it here though which gives me some hope!

    The great thing about swiping from the edge is that it can be global without (usually) conflicting with an applications functionality. With your sample gestures I wonder how much they would get in the way of scrolling etc.

    I would love to see this eventually become a (much more powerful) replacement for the Nokia swipe gestures. To achieve that it would need a way to anchor the start of the gesture to the edge of the screen, a way to specify a hold or kinetic flick at the end of a gesture and ideally some form of animation to show how far along the gesture you are with the ability to retrace your steps to cancel the command. To do all that I imagine would be a fair bit of work but what you have done already has given me some motivation to look into it. I’ll try and have a look at your code tomorrow and maybe have a play around with it.

    • sandst1 says:

      Yeap.. gestures interfering with the other functionality is an important
      issue that should certainly be addressed in a way or another.

      The swipe is so integrated to the compositor/window manager, it’s a bit hard to replace it.
      All the touch events could be grabbed, but then we’d have to act as the window manager too.

      But. Your comment about anchoring to the edge of the screen gave me one idea: instead of replacing/ignoring the swipe functionality, this could be used to improve it: specifying a (configurable) place, like one of the screen corners, where the user places his/her finger to say ‘I’m starting a gesture now” and then draws the actual gesture with another hand.

      Using the corner anchoring would hold a flickable screen still during the panning. The only thing in the system that would be clearly affected is zooming but that’s a smaller problem than random panning around.

      Dang, i have to try that one out today : )

  8. Jed says:

    “Haven’t” used Windows.

  9. CaCO3 says:

    That looks like a really cool application!
    I installed it (dpkg -i …), but how do I start it? There is no icon, also in the command line I couldn’t find a command for it.

    And a suggestion: make the moves folder hidden by adding a leading dot to its folder name.
    A UI for editing the settings for sure will be great as well.
    If you pack it, I suggest to already include some example configurations like shown in the demo.

    Also, please start using the OBS server or another repo, so the app gets automatically updated once there is a new version.

  10. sandst1 says:

    Hi.

    You have to install two .debs:

    First
    dpkg -i mymoveserver_0_0_1_armel.deb

    then
    dpkg -i mymoves_0_0_1_armel.deb

    mymoves_0_0_1_armel.deb is the configuration UI, and it adds a shortcut to the app grid.

    Thanks for the suggestions : )

    It’s a good idea to hide the moves folder. Regarding examples, I’ll consider that once the app is matured enough. They’ll have to be simple enough though so they’ll work with most users.

    Yes, OBS server is in plans also, just hadn’t had the time slot for that yet.

    • CaCO3 says:

      Sorry, I was my bad, I failed the installation 😦
      Now it is there.
      I recorded some gestures and added the command (the one you show as an example, with – replaced by – ).
      How ever it still doesn’t open the program after I pushed “start observing gestures” and dit the gestures. Are there any debug information available?
      Dont bother too much, I will test it again when you are further with the app.

      • sandst1 says:

        No worries : )

        The server pushes debug information to the logs but because the UI currently
        controls the server’s start/stop, the logs are not visible.

        You could try this:
        1. close the configuration UI
        2. log in via SSH
        3. su – user
        4. killall mymoveserver
        5. /opt/mymoveserver/bin/mymoveserver
        6. start the MyMoves UI
        7. Press ‘Start observing gestures’

        And now you should see logs in your SSH window.

  11. CaCO3 says:

    Thank you.

    When I start the server on the terminal, I see this:
    ——————-8<————————-
    loaded the Generic plugin
    loaded the grue plugin
    Loaded the MeeGo sensor plugin
    Setting data rate 0 Hz (interval 0 ms) for "meego.orientationsensor"
    MyMoveServer, portrait: 1
    Server VendorRelease: 10905000
    XRecord for server " :0 " is version 1 . 13 .
    XQueryPointer returned: 1
    Entering XRecord loop…
    ——————-8<————————-

    So that looks quite ok to me.
    Also, when I now start using the touchpad, I see a lot of data like
    ——————-8<————————-
    touchMove: x 517 y 86
    idle or saving, ignoring event
    touchMove: x 517 y 86
    idle or saving, ignoring event
    MyMoveServer::touchRelease
    ——————-8<————————-

    But as soon as I start the UI, the server terminates.
    I can start the server again and it will again log the movements, but no app will be started.

    Please let me know how I can help.

    Just out of interest, as I am a developer myself, what kind of algorithm are you using to match the movements?

    • Martin Alderson says:

      @CaCO3: the source is available on gitorious. I don’t know much about these kind of matching algorithms but the code seems to say it is using pearson. Also, when the UI starts up it does “killall mymoveserver” before re-launching it again itself with “/opt/mymoveserver/bin/mymoveserver &” so I think to debug further you would need to do sandst1’s steps in a different order, i.e.

      1. close the configuration UI
      2. log in via SSH
      3. su – user
      4. start the MyMoves UI
      5. killall mymoveserver
      6. /opt/mymoveserver/bin/mymoveserver
      7. Press ‘Start observing gestures’

      I dunno if that will get you further though.

      • sandst1 says:

        Ah, sorry CaCO3, i accidentally put the commands in the wrong order..

        Thanks Martin for sorting that out.

        Currently the algorithm is quite a novel one, basically just about normalizing the gesture vector, then calculating Pearson correlations against the saved gestures and choosing the best match if it’s good enough. The multitouch-one will need some tweaking and perhaps it’s good to move to a more ‘real’ recognition algorithm in the future : )

  12. Martin Alderson says:

    sandst1 has poured some cold water over me regarding replacing Nokia’s swipe. That’s still something I’d like to look into but perhaps it is outside the bounds of this project.

    I’ve been trying to build and run this project without a device. I can’t build for the desktop as it relies on device sensors (orientation). Qt simulator but doesn’t have support for Harmattan (doesn’t have the new Qt Quick components). Maemo5 emulator also failed with no Qt Quick. The Harmattan emulator is probably my best bet but that (aside from being slow) fails to compile mymoveserver as it can’t find X11/extensions/record.h. It runs mymoves gui but I can’t get it to display in portrait and in landscape the save button isn’t displayed.

    Any help on getting this built and running is appreciated.

    • sandst1 says:

      Hi Martin,

      I’m not against the idea of investigating on how to replace the swipe, it’s just so deeply buried to the closed code that IMO it would require a Big hack to get it working, just ran out of ideas on how that could be done. If you can come up with something, i’m interested in getting it integrated to this project : )

      If you want to build it for the PC, you can basically throw out the sensor-specific code to make it compile. The landscape stuff is relevant only for recognizing gestures given in the landscape mode.

      What’s your IRC nick so we could talk more about the issues you’re seeing..?

      • Martin Alderson says:

        Yeah, I agree. Replacing swipe looks like rewriting chunks of Nokia’s closed source bits. Hopefully there are still some ways it can be enhanced rather than replaced as you mentioned earlier.

        I’ll have a go at removing the sensor specific code but I guess it will be pretty hard to do much interesting stuff without it working in the emulator or a device.

        On freenode I’m equim.

  13. sandst1 says:

    Without the sensor specific code you can do pretty much all the same things on the emulator as in the device, everything’s just in portrait. So that’s really not a blocker for doing interesting things : )

    • Martin Alderson says:

      Thanks for all your help earlier sandst1.

      I now have it all working with the Harmattan emulator. For anyone else having the same problems here are the magic hacks I used:
      1. Grabbed libxtst-dev, libxtst6 and x11proto-record-dev debs from http://harmattan-dev.nokia.com/pool/harmattan-beta/free and extracted the /usr directory from each to QtSDK/Madde/sysroots/harmattan-nokia-meego-arm-sysroot-1122-slim/usr.
      2. Change dh_shlibdeps in mymoveserver/qtc_packaging/debian_harmattan/rules to dh_sh###libdeps (it is already uncommented but seems to run regardless!) to get around a dependency problem with libXtst when deploying.
      3. Make sure the mymoveserver project build target is set to Meego 1.2 Harmattan API.
      4. Start the maemo emulator and wait a few minutes for the icons to show up.
      5. Run mymoveserver from QtCreator.
      6. Add a line containing “rotation: -90” below the “Page {” line in mymoves/qml/NewMove.qml to allow the save button when creating a new gesture to be visible.
      7. Run mymove from QtCreator (which will kill and restart mymoveserver).

      Hopefully I can get a device sooner rather than later!

      With the emulator being so slow and my mouse gesturing skills being quite poor it’s pretty damn hard to get anything other than a tap to be recognised as a gesture, but it’s a start at least.

  14. Pingback: Ce que vous ne savez toujours pas sur le N9 ! Partie 2 | NokiaPhones.fr

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: