Friday, October 10, 2014

Multi-touch and bigger screens

Brace for disclaimer!

Note that this has nothing to do with Jolla. People have been asking me about how could Sailfish OS work on larger touch screens, so here it is folks. Some theoretical design thoughts about Sailfish OS user experience, on a bigger size. This is important for the sake of understanding a mobile operating system design and the effects different touch screen sizes have on it.

Fantastic, let's move on.

Multi-touch makes an excellent parallel subject, when talking about larger touch interfaces. I've personally grown to dislike majority of multi-touch implementations because they seem to be driven by technical capability to track, rather than supporting the way we use our hands.

So what should multi-touch be then?

Let's use another tool example to dig deeper. I like comparing things to tools because of their simple, efficient and purposeful interfaces.

Think about a hammer and a nail. The task is to hammer a nail into a piece of wood. Your other hand holds the nail while the other one hammers it in. When breaking that down, we can recognize multiple smaller operations inside that task. Pinching a nail while holding it perpendicular to the target surface, is one. Whacking it in with a hammer in your other hand, is another. This is a simple multi-touch use case from the real life. With two types of multi-touch.


Yes, I think there's two kinds of multi-touch. The first type is related to the task itself. You need to use both hands simultaneously for the same task to succeed. Hold the nail and use the hammer. The second type is related to individual hand operations inside that task. Pinching a nail with at least two fingers, and gripping a hammer with up to five. The latter is the most common use of multi-touch to implement a pinch/spread-to-zoom for example.

Ok, two types of multi-touch. Two-handed task type and multi-fingered operation type. There is no need to discuss about how many fingers you need to do something, because then you're focusing on wrong things. And I feel many existing interfaces are limited to only the operation type, because they're not focused on user tasks, but in something completely different. Not to mention forgetting totally what our hands are capable of.

So, I ended up with this definition because there wasn't really anything tangible behind existing multi-touch interfaces, in the mobile OS space. At least I haven't ran into anything that made sense. What I've found abundantly, though, is a lot of complexity that multi-touch can help to add. It's very easy by introducing more and more fingers on the screen, and mapping that to do yet another thing in the software. The amount of needed fingers has lately gotten kind of out of hand. Pun intended.

If you need more than 1 finger to move around in your OS, you should seriously look at the interface architecture and feature priorities.

"But with multi-touch , I could have an OS feature to directly alter orbits of celestial objects and.."

No. Stop it. You'd be still browsing, watching videos and gaming. And the only celestial object you know is Starbucks. Stop looking at increasing OS features, and pay more attention to enhancing user potential.

Alright, apologies for the slow intro. This is where some illustrations comes into play, and hopefully make more sense of the multitasking stuff above.

Sailfish OS was designed to be less dependent on display sizes than other mobile operating systems. This is because the most common user interactions are not depending on the user handedness, hand size or thumb reach, so the gesture based interface naturally allowed one-handed use of a smartphone sized devices.

"Hah, you can't really use a huge device comfortably with one hand, so your one-handed use benefit is lost then?"

Yes and no. The same way that Sailfish OS made a small interface fit into a single hand, it makes any larger interfaces fit two. This opens new ways to interact with larger devices due to the analog nature of touch gestures.

We should also understand that people are very liberal in how they use and hold devices in real life environments. In commute, at home or during a holiday trip. Most of the time, it's resting against something, and simply hold in place with one hand.

This a two hands grip, while using a full-screen application. This is a precondition to completing a task. My tool comparison is holding a nail in the other hand and a hammer in another.

The left (or right, it doesn't matter) hand performs the Peek gesture to expose the Home screen. The hand with the nail is placing it against the wood surface.

While keeping finger on the screen, user is able to see what three other applications are doing in Home screen. The nail is ready to be hammered in.

Without releasing the left thumb, user performs a cover action to play/pause/skip a song with the right hand as an example. Releasing left thumb after interacting with any active cover, would keep user in the application 1 (first image). That's a fast way to look into Home (just like on the phone), perform an action (enabled by the larger screen) and get back to the app. All without even really leaving it.

Alternatively, if user would tap another application cover, or trigger a cover action that requires a full-screen state, the screen real-estate would be divided between the two. The nail has been hammered in, and user is back to default state that precedes the next task.

This behavior is a natural progression of Sailfish OS Peek gesture and how application windows are handled. Only the support for the other hand and screen division was added. Both hands performed an individual operation that alone, completed a single task (pinch a nail and hold/carry a hammer). When they're performed together, a different task is completed (a piece of wood is attached to another). Just like we do so many things with in our physical environment. I wanted to focus on illustrating the task type, because there are countless examples about the single hand multi-touch, the operation type.

The value in all of this, is that the entire interaction sequence is built into the same application usage behavior, without any additional windowing modes or mechanisms that need to be separately activated and used. It supports the way we work and enhances our natural potential. After all, you don't turn your hand into a separate mode when you're driving nails into planks of wood. No, it's the same hand, all the time.

Similarly, when you need to reach something from a tool drawer, you will not physically enter the drawer yourself. You wouldn't fit. Instead, you stand next to it, open the desired drawer and pick up the tool you needed, before closing it again. The Sailfish OS peek gesture is doing exactly the same on larger screens because of multi-touch. Exposing another location (Home/events = drawer) with the other hand, to see what's there and perform a task (trigger an action = take a tool) with another. All without actually going to that location.

That's what multi-touch should be.

Something that focuses on enhancing our potential, instead of enhancing features we are required to use.

Button based tablet operating systems (excluding Win 8 etc) are not going to do something like that any time soon. Not only because they treat active applications in a different way (as second class citizens), but it would be challenging to implement the behavior into the way how buttons work. Also the button locations do not support ergonomic use of individual hands when gripping the device from the bezel. On the other hand, the sliding gesture over the screen edge is very natural to perform, because it happens where your thumb is most comfortable any given time.

This conventional button approach, that many Android devices use, exhibit another problem in enforcing a hand preference in controlling the device. As you can see from the image above, the left hand is not able to reach notifications on the right, and similarly the right hand struggles in reaching Home, back and task switcher buttons on the left.

It's not about changing the interface between a phone and a tablet. Tasks are anyway the same. It's how the two-handed use is enhancing our potential through the increased touch screen area.

Don't try fitting an existing multi-touch solution into your interface, but think how an interface can handle both one- and two-handed use.

Then, the rest will find their own places naturally.

Thanks for reading and see you in the next post. In the meantime, agree or disagree, debate or shout. Bring it on and spread the word.


  1. It will be nice to see this feature in a tablet. On my Linux Mint Cinnamon desktop, I love tiling and snapping brought in by their window manager.

    1. Hi Anand,

      I'm excited about it as well. Different linux distros have been ahead of many commercial desktop solutions when it comes to window handling/tiling/snapping. I'm sure Sailfish OS can bring this kind of feature to the market at some point.

      Thanks you for stopping by and commenting :)

  2. tldr; An embedded animated SVG emulator speaks a thousand words.

    1. Yes, it's hard for me to write short, quickly consumed articles. I need to work on that. Thanks for reminding :)

  3. hello Jakko (if you accept that i call you by your name).

    i find your concept of dual screen for the tablet really great.
    It's not a wonder that why the jolla community vote for it.
    I am fan of it too. That's cool.

    I want to ask a question about it. If it don't bother your (i hope).
    Does your concept integrate the possibility to have the other screen view in a kind of PIP-View (Picture In Picture)?

    If not, what do you think about this possibility or option furthermore?
    User could choose between dual-screen
    or PIP-screen, means: one take place on the whole screen, and the other one take place in a little rectangle on a choosen corner (left or right below corner, or left or right above corner). And on a corner of the little rectangle, could be a toggle-button to switch between the both view. Which one goes to the background.

    Thank you very much in advance for your interesting, and your answer.
    And i wish you have a good continuation. And enjoy the development of your concept.

    1. Hi cemoi71, it's great to hear you like it. I've always wanted to have a simple and fast way to handle multiple apps, so it felt as a natural continuation for our work for create this concept.

      I think PIP would work really well for some apps, and it's a next step toward more effortless interaction. Sometimes you need more space, but sometimes even a small fraction of a screen is enough (video playback).

      The real challenge comes from making sure whatever the small window contains, it doesn't obscure important information below it. Splitting the screen vertically solves the issue, but is too much for some cases. Also, the more layout possibilities (PIP, split, full screen) a single application window supports, the more complex those apps also become in terms of how their content is shown.

      All in all, it's definitely an idea worth discovering further. I can see the enthusiams toward making Sailfish OS improve is infectious. I appreciate the time you're using to suggest improvements. I promise to do my best taking them further.

      Take care :)

    2. Hi Jaakko,

      that's it!! i had like a feeling, and can't exactly describe it about the need. And you have described it with simply words.

      You are good i think in your job. It's not a wonder that you and Jolla work together.
      The important magic, is how to combined and control those functions. With the ergonomic.
      I am now really excited after your answer, on how this concept will be presented on the future.
      I know, that it is not so obvious to bring them all inside. Because of development complexity, financing, needs of the majority or from the big-boss.

      Maybe if you have further sketches, you could bring it on your blog as update. And if it's fine for your, to the jolla together community. Then i could put a link on it.

      i have already put a thread on it by tjc, if you have time an interest:

      Thank you very much about your kindness.
      Take care you too, and have a lot on enjoyment on what you do.
      Seems that you have a great fun, on jolla side.

      Best regards

      ps: could tell me please, how could we be aware on any new mind-work from you about this theme? i think we can find here. does your blog works with rss channel? i am extremely interested on your work about it.

    3. Thanks. It's a very rough concept at the moment, and if Jolla ends up implementing something like that for the tablet, more details will for sure emerge. Until then, it's pretty hard to predict what really works or not, as it requires a lot of prototyping and fine tuning the behavior. So much can go wrong with the details.

      Great to see you active in TJC, keep it up. I'll keep writing new posts whenever there's something worth sharing and discussing, so stay tuned. I added a widget to the sidebar you can use to subscribe. I totally forgot that earlier, so thanks for reminding me about it :)

  4. i'm sorry really for the mistyping of your name. was an unwanted backspace action just before sending. Now i can't edit the post for rewrite it right... :-(

    1. No harm done :) ..and it's ok to call me by my given name.

      Yeah, I also miss the edit comment feature. It would've saved me from some mistakes as well. Maybe it comes in the future..

  5. Hello Jaakko,

    that’s me again.
    I was pretty by thinking today about this concept, after the short exchange on it.
    And more over the ergonomic of the gesture. The art of gesture recognition that the user could use to bring the system with one view, or on the other one.

    What do you think about this.
    The beginning of the sequence as you described above is really great. Simple and natural.
    1) user stand in front of app-1.
    2) user peek to home screen the app-1 and hold his finger on it.
    3) appears on display all running apps.
    4-a) user tap on the app-2 that he want to have in split-view, with his right thumb.
    5-a) User have both apps in dual-screen on within the other one.

    4-b) User hold the wished app-2 with is right thumb.
    5-b) User see app-1 in full screen and 4 empty areas with dot-contour for possible pip-view places for app2 (at this state, user may release all his fingers from the screen).
    5-c) User tap on one of the four corner, for the app-2 place.
    5-d) User see app-1 in full screen, and app-2 in pip-view upon. If he tap on the pip-view area, then system toggle each time between both apps. And naturally no action could be done inside (the pip-view one).

    Did you made any thought about the split-view releasing? I mean for the come back to the conventional view?
    What do you think about, if the user make first the same gesture beginning as to activating the split-view. Means, peek to home screen and hold his finger on it. System makes the whole screen a little smaller (because it knows that it is in split-view). Then place his right thumb on the second app and hold it. Then he could release one of the both app, if he drag it with is thumb out of the screen...
    What do you think of this vision/idea? Is there a potential in it for the transcription in an optimal ergonomic for this concept?

    I wish you have a nice week-end.

    1. Hello again cemoi71,

      Your comment is a great example of how our curious minds start working their way through a problem we find interesting. And the funny thing is, that our brains cannot be stopped if they find something they like.

      Now, what you just wrote, brings back a lost idea: what if you could just reserve the other side of split for Home screen (like, do an edge swipe the opposite thumb [right], instead tapping on an app cover)? The minimized apps would then be shown on the right side, just like they're shown on Home right now. That would essentially give you a lot of PIP screens (that's what Home actually is right now). Maybe I need to do an example image at some point :)

      To make it better, the minimized app covers could be larger to enhance viewing experience. Maybe if user would drag the split between the App 1 and Home, the cover sizes would reflect that. A method like that would elimininate another editor state needed to manage the PIP window location (the one with a dot contour), and would just rely on how covers are arranged on Home. Naturally, this would require more work on how app covers are arranged on Home, but that's needed anyway to calm down the busy behavior of covers moving around all the time. It would help Home screen to become better.

      Thanks for reconnecting me with that lost idea. Without you, it could've lost between the sketch book pages :)

    2. Hi Jaakko,

      sorry for the late answer. The week was pretty event full. if you see what i mean. And that is a little bit dump that we could not have a mail notification when you or someone make a reply...

      I'm not sure that i understand the part with the edge swipe (again the bad english on my side ;-) ). Would you mean that on the step 5-b that i have described, instead of waiting of any dot forms. The user after that he hold on the 2nd app, he get just the confirmation of the next step by a changing of app2 appearance, et could simply drag it where he want?
      If it's what you've means, i find it quite brillant, more simple.
      As you said after thinking, the dot contour is just an unnecessary gadget that give the system more job, and the development is more difficult.
      I think (or hope) that will be fine if you make some more example image, as you told. Picture, are always good for the understanding.
      Maybe if you have some decision problem between different concepts, you could ask into the jolla together community to vote between them. Just a stupid idea from me...

      Hey Jaakko, you're welcome. that's a pleasure simply to communicate with you on the subject without taboo. I find great that you let us have a look on your work, and discuss on it. How you''re thinking, on the same level (with no condescending). That is pretty interesting.

      I wish you have lots of fun on it. ;-)

    3. Hi cemoi71, thank you for your message. It's busy times here as well, but the most important part is to get message through :)

      You're correct, this is not the easiest thing to explain through text. I'll try to find time to make some examples images to illustrate the idea a bit better.

      Not at all a stupid idea, I think it would be great to have some concepts revealed in TJC before implementing them.

      Have a great week :)

    4. Good morning.

      you're right for the communication.
      And i know that some things have more priority.
      So i understand that some reaction are not so fast.
      Or that for example, something are impossible, shortly, or in longer time or at least not practicable to do.
      If the argument are clearly given (not just because of technical issue but business or human ones). Than it's easier to accept it.

      Do you have an account in tjc?
      Otherwise I have no problem of making some threads on tjc about it. But i believe that i won't have a big impact on community, in compare with you or a jolla team people.

      maybe there is eric ( one of the jollamen as modo in tjc. Nice guy Maybe you know him. He has a bigger impact on tjc community, or simo (in fact all jolla tjc modo).

      I have already done a thread on it (i think i have already given the link but here it is:
      I have no problem that it be complete reedited or pointed as duplicate if someone make an other one better. I'll make just a short review, because of little new ideas from here...

      Thank you very much, and wish you have a nice week too...

    5. Hi cemoi71, good to hear from you.

      Yes I have TJC account, and I probably should arrange time to comment things there as well. Majority of the time goes to tablet things, and the split screen works is coming a bit later.

      The thread you started will sure be taken into consideration when the topic is processed. We go through tjc for potential items when working on a feature.

      Keep it up and let's hope there's some more news regarding the split screen schedule in MWC2015. Take care :)

    6. Hi Jaakko,

      yes i have shortly seen your account. Someone has made last days a reference of the answer of you about the ambience philosophy on sailfish-os

      I understand it, for the split story. It has naturally a lower prio.
      The tablet should first run stable with the necessary.

      hmhm, do you go physically too at the MWC2015?
      I hope that jolla send to all of us the presentation of it.

      May i ask you to ask for if someone could make a video of the current stand of tablet-handling there (at MWC2015)? and put it on the web.

      I'll try to snap the event when it comes. I'm really curious on which state is the complete current handling of the tablet.

      Have you got the permission to tell if someone has already made some work on the development of it (the split view i mean)?

      Wish you have good continuation and fun with/on it. :-)

    7. Hi cemoi71, it will take time for the ambience story to really develop. It has a lot of potential, but only time will show what comes out of it :)

      I won't be joining Jolla MWC crew this year. Don't worry though, there will be other designers you can chat with. I'm sure there will be a lot of videos about the tablet.

      Sorry, nothing to report yet on the split screen. It's a challenging feature to implement, as it affects so many things.

      Thanks for the support, have a nice week :)

    8. wow... you speak about other designers and not joining the team...does that means that you are not involve any more on the further design of the sailfish? or there is different design-parts that need different people for it?

      Wish you have a nice week too.

      ps: if some things are challenging feature, then maybe it will be not bad for you to make some proposal into the community together channel. But i don't know how will the jolla boss react if a complete concept lands in fully wild wild world, or how could be interest the community people, if the concept is presented in small small restricted info...

    9. Hi cemoi71,

      We take turns. Not everyone from the design team traveled to MWC last year either :)

      At the moment, I'm working on user guide, sales package, browser and some other errands, so less OS design for me at the moment.

      Yes, it would be nice to get visibility to design activities for community. It's been discussed earlier in the past, but it's a company decision, not mine. I hope you understand :)

      I can post general interface design things, but cannot disclose details.

      Thank you for commenting :)

    10. Hi Jaakko,

      yes i understand it. I know how it works, that the same in my company ;-).

      That is quite interesting what you told. Until now i didn't imagine that you are many people on the design. I thought that you were alone.

      Yes yes yes please post something. I think a mini little bit (a micro info) is better than nothing. And curiosity is really high on it.
      And that is clear for me that the limitation are on side of the jolla management. And they have some responsibilities, and i respect it. even If this is on 100% limitation.

      Hmm then as all said. Wait and see.
      I'm curious on how goes the evolution of it.

      Good continuation, and have fun.
      Thanks a lot for the commenting too and discussion. Is quite interesting.

    11. Hello cemoi71,

      Good to hear you understand the situation. It will be interesting to see the reaction to the tablet next week.

      Thanks for being supportive, take care :)

  6. Stop looking at increasing OS features, and pay more attention to enhancing user potential - Awesome.

  7. Hi Jaakko,

    just for curiosity, do you have any information how far is the split-view development?
    Does it come on the first sfos v2 inside? or later?

  8. Hi cemoi71,

    Good question. It's a pretty complex feature from the compositor and application layout perspective. Unfortunately there's too many open issues for me to predict when it'll be ready, but it will not be included in 2.0 release.

    Thanks for asking :)

  9. ok, do you mean that you 8or your workmate) could not be active work on it with the dev team?
    means that's more job to do as expected, doesn't it?
    nevermind, if it doesn't comes in rel-2.0, then we have more time to discuss on it. lol!

    1. Yes, there are other things we have to do before multi-window support is ready :)