A vision for the 10-finger desktop

12 Oct 2009 - 8:25am
4 years ago
22 replies
1195 reads
Clayton Miller
2009

For some time, I've been intrigued by the question of what comes next
after the mouse and the windowed desktop, the Xerox PARC legacy of
desktop human-computer interaction. The paradigm that PARC gave us
has proven amazingly versatile. But even it has its limitations, both
in navigability and interaction bandwidth.

Over a span of about nine months, I pulled together some casual
research here and there, examining other proposals, trying to
determine their shortcomings, and, in the realm of thought
experiment, iterated answers to the problems that I identified. Over
this past summer, I finally produced an eight-minute motion graphic
piece detailing the problem and my proposed solutions.

I'm sure there are yet many shortcomings in my proposal. Still, I
think it's a solid basis for further exploration, and I hope it
inspires some new dialog.

The video and some more background information are here:
http://10gui.com/

What do you think?

Comments

12 Oct 2009 - 11:04am
.pauric
2006

That is a very well thought out design. I like the way you've not
just tackled the hardware issues but approached the problem from
sw/hw interaction point. Great presentation of the design btw. Ship
it!

Some questions
1) how do you see apps with a user defined layout working within your
app switcher. e.g. most of the Adobe suite.
2)I have some widgets permanently on display on a second monitor.
How would that work within your design.
3)Do you feel there are still cases that a user will want a mouse or
trackball to hand, if so, how would they interact with the gestural
mental model of your design?

I've been thinking about the same problem for some time and taken a
most simplistic (and hacky) approach. I do not have your scope of
vision for the OS! The first iteration of my design incorporated a
resistive touchscreen mapped to keypress macros (copy, paste, app
switching, etc) for my left hand and a pen tablet as input for the
right. The next iteration will see the addition of multitouch with
the recently introduced Wacom Bamboo touch series. I start work on
v2 this week and should have a working model early next year to
share. I will continue to split the modes of input between left and
right hands, a fundamental difference to your approach.

Just as a general discussion point. What do you see as the pros &
cons of the two approaches. 10 finger versus 5 5, actually 1 5 as my
embedded display is resistive.
One obvious constraint of the 5 5 is that the virtual buttons are
contextual to the app in focus, thus inducing a level of cognitive
processing when the user switches apps. However, I feel it affords a
great deal of additional input bandwidth. Your thoughts? Do you see
a gestural language being a solid alternative to 'the button'.

btw I am completely in awe at your vision and the clarity in which
you explain it. It puts the recent Microsoft touch book thing demo
to shame! What you've created could ship tomorrow and hit the
ground running in terms of UX imho

regards!
/pauric

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

12 Oct 2009 - 11:05am
.pauric
2006

ooops, forgot to include a link to some pictures of my design. Again,
I'm hacking things together and the first iterative is just a working
mockup
http://www.flickr.com/photos/pauric/sets/72157616093748066/show/with/3488471659/

regards /pauric

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

12 Oct 2009 - 2:43pm
Kim van Poelgeest
2009

First of all thanks for the vid. I think you're making an interesting
point. Have you actually researched the strain involved with such an
interface?

I have a Jazzmutant Lemur http://www.jazzmutant.com/lemur_v20.php,
which I use to make music. I can use all ten fingers in an intuitive
way. However, it costs way more energy it does just using pen tablet,
touchpad and keyboard combined (or another combination). Especially
using everything from a three finger pinch I imagine it to become a
heavy physical strain.

On the topic of Lemur and tablets, industrial designers could surely
learn a thing or two from the electronic music industry. Touch-strips
(sort of sliders) are inexpensive and intuitive (think a large
scrollable area in percentage or 1:1 mode). Or how about the easy to
use combination of interface adaptation with this simple MIDI
controller combined with eg Ableton Live
http://www.novationmusic.com/products/nocturn?option=1. It sure as
hell beats programming a Lemur in simplicity and directness. And what
about

For some reason, we are limited to this day to a single touchpad and
a keyboard on our screens. Why don't we have:
proximity sensors
touchwheels
motion controller (miniwii?), e.g. just slightly waving your hand
from a small distance to skip from one area to another, improving
distant usability (think entertainment centre, scanning through
documents or photo's, or whichever you come up with)

and a dozen other techniques that could improve tangibility of user
interfaces.

But you need adaptable software, maybe some sort of interface
swapping. Again, MIDI/OSC has researched far more advanced heavy duty
interface users over a lengthy period of time. All I know is that if I
ever came to work in the interface industry (researching both OS
possibilities and hardware standards) I would research all those
other areas currently under development.

Again thanks for the vid. I'll be watching closely!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 1:12am
Adam Korman
2004

There are a lot of nice things in the concept. It would be great to
see this on a tall, rather than wide, screen. If you stack the windows
vertically rather than horizontally, you wouldn't need to have that
sideways text, which kind of drives me nuts. Or, like the iPhone, have
a display that you can rotate and have the UI adapt appropriately.
I've always wanted this to become mainstream for desktop displays.

The basis of much of Apple's multi-touch technology (laptop trackpads,
iPhone) was from buying out Fingerworks. They had some interesting
products (like a keyboard replacement that also serves as one, big
gesture input device) that had a lot more power than what Apple has
used in their products so far. You can still get a lot of info about
their old stuff here: www.fingerworks.com. It will be interesting to
see how they integrate this over time and if they start to make more
substantive changes to the desktop OS (like in your proposal), rather
than just handy shortcuts (like they do now with the laptop trackpads).

On Oct 12, 2009, at 6:25 AM, Clayton Miller wrote:

> Over
> this past summer, I finally produced an eight-minute motion graphic
> piece detailing the problem and my proposed solutions.
>
> The video and some more background information are here:
> http://10gui.com/
>
> What do you think?

12 Oct 2009 - 3:52pm
Gordon Barlow
2009

I wanted one when I saw this video, I immediately thought of how
muscle memory could be used to remember how you interact with things.
It could be much more intuitively designed, so that things came
naturally, without thinking.

At the very end of the video you showed a large track pad below the
keyboard, one issue I have had with his setup (like on a 17" MacBook
Pro) is that I would often touch the track pad with my palms while
typing with the keyboard, it became cumbersome to keep my palms
arched above the track pad.

Another concept could be initiated by something I worked with in
Industrial Design, a 3-D Sculpting pen called Freeform
(http://www.sensable.com/products-freeform-systems.htm or
http://en.wikipedia.org/wiki/Digital_sculpting) this may trigger a
multi-motion interface, involving a more fluid interaction involving
the feel of real 3-D life.

It sounds like an exciting project to work on, let us know how it
turns out.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

12 Oct 2009 - 9:47am
Anonymous

This is very intriguing! Obviously some good thought has gone into
this. Nicely done! (Not sure about the cheesy music in the video.) I
tweeted the link, fwiw.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 9:25am
Dave Malouf
2005

Clayton, I think my attempt at a response was too long, so I'm trying
this shorter one. My response has a lot of criticism, but I want to be
sure you see this even if you don't see the blog post.

...

Most importantly though, that putting yourself out there like this to
criticism is HUGE. This is inspirational not only for what it offers
directly but because it offers a point of discussion. I could have
never done this level of articulation with someone to respond to as
well produced as your demonstration. It has great thinking and there
are real problems to address, or more importantly even if the
problems aren't great, there are still places where we can hope to
do better.

All my classes will see this video and all my faculty. Thanx!

blog post: http://davemalouf.com/?p=1730

Enjoy!

-- dave

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 12:30pm
Marc Resnick
2006

To give you some feedback on how it would interest non-IXDA types, I
posted the video on my FB profile and half of my techie friends
reposted it within 24 hours.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 12:48pm
Brian Mila
2009

I think its a great vision! There are some novel ideas in both the
hardware and software concepts. Personally, I think a better
arrangement for the keyboard would be to have the trackpad split and
have the left and right halves be placed left and right of the
keyboard. I really liked the menu display, with the main menu items
in the middle and then submenu items above and below.

However, I'm not sure I agree with the idea that the traditional
overlapping window manager is the problem. Why not combine some of
the gestures with a traditional virtual desktop, so that every app is
maximized in its own virtual window? The zooming part of the
interface would still apply nicely, and it would allow users to form
memories of where a particular application "physically" resides in
a larger virtual workspace. They could easily zoom out and scroll
the workspace using the gestures. In your design, you've
essentially incorporated the memory limitations of a traditional
taskbar...that is, that the organizational structure is simply a
product of the order in which applications are opened. The ability
to rearrange the order of items is nice (I wish XP had that), but
IMHO doesnt address the main problem.

On the subject of still needing a mouse, I wonder, what if you had a
"legacy mouse mode" in the trackpad, so that you rest your hand as
if an invisible mouse were underneath it. Then move your hand
normally and use the index finger to click. It might reduce the
learning curve and may help with accessibility concerns that others
have raised.

All critiquing aside, it was an awesome presentation! Thanks for
sharing!

Brian

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 1:19pm
poomoo
2009

I thought it was a very interesting video. Good work!

I expect Apple, who have the vertical integration you mention, must
have done a lot of thinking on this already. However, the real power
comes when software developers like Adobe support multi-touch in
their apps. It will be interesting to see how they enable us to use
transformation tools and apply filters.

Have you considered how pressure could be integrated into gestures?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 9:53pm
Sean Gerety
2009

I have one of the Fingerworks Touchstream LP keyboards and have been
using it for years. I can perform two hand gestures and navigation
with both pads at the same time and I love it. Here's a link to the
gesture guides for fingerworks.
http://www.fingerworks.com/userguides.html

I think that by using chord modifiers instead of a global and local
app launcher bars at the ends you would reduce the strain of reaching
for the edges. (I only say this because with the touchstream I no
longer have to reach for the shift key on a keyboard anymore.)

Great work on the video.

Cheers,

Sean

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

14 Oct 2009 - 2:49am
Anonymous

Fascinating! I wonder how this will be accepted by actual users. It is something else than users got familiar with over the years, especially to do fast work, such as copy and paste.

14 Oct 2009 - 1:06am
sbisker
2009

First of all, your video is wonderful. It's a polished (and great)
idea - it's clear 10/gui has gone all the way from careful research
to wild brainstorming to careful-once-again visual prototyping
(letting yourself bounce everywhere in between). And what's more
amazing is how you've managed to synthesize everything you've
learned in those 9 months into a video that isn't afraid to
patiently, clearly and *simply* explain both the entire problem space
and your solution. The end result of your polishing is a concept video
that, more than any other concept video I've seen in a very long
time, has left me (and everyone else here) genuinely wanting to "try
it." I mean it sincerely when I say that your design vision abilities
are uncanny - but even more remarkable are your design synthesis and
storytelling abilities, which to me set a standard that many
professional design firms don't even meet. While I'm still a
wet-behind-the-ears design student, I suspect I speak for a lot of
professionals here when I say that I wish and work towards my designs
and deliverables someday being as crisp and professional as this.

That said, my biggest gripe with the video is, in a sense, just how
strongly I still want to try it (and feel I haven't tried it by
watching). Despite the idea being fantastic, and polished, and really
convincing, it doesn't go that one last step of leaving me with a
good sense of "I can/should use this myself". And maybe this is
just a function of my hacking/"build the thing" background, but I
think some of this comes from it feeling like "just a concept."

On his blog, Dave notes that some of that has to do with the fact
that your demonstration use cases don't really try to reflect the
average person's usage of the system - that it's not situational.
I'd actually take that argument a step further and say that your
design is dangerously close to *over designed.* By which I mean,
while your design is wonderful now, any further design (and arguably,
some of your current "finishing touches" like the specificity of
your gesture vocabulary) now has an incredibly high likelihood of
being rendered useless by the simplest of direct user
prototyping/interviewing/testing.

It's clear your design understands the human condition and has a
clear vision on how to improve it, but it's not yet clear to me your
design understands *me* and how I would use it as an individual. And
the only way you're going to convince me you understand me, is to
build some manifestation of 10/gui (be it a "situational video"
that puts me in someone's shoes, or a clickable Flash prototype, or
a full-on beta) and let us imagine using it in daily life. We
understand it, we love it, now put your design to the next,
inevitable test - let us try it.

(Oh, and please ship it - I want one for my stupid laptop and its
tiny windowed screen. :D )

-Solomon

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

13 Oct 2009 - 6:45pm
Daniel Zollman
2009

This is a great concept.

I'm confused about one thing, though that might only be because I
can't try it out. Resizing a window (to allow more content to fit
onto the screen) is different than zooming. Does this system allow
for both actions?

Also, this does a lot for the interaction on the OS level, but what
about applications? Can applications access the

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

14 Oct 2009 - 1:27pm
Oleh Kovalchuke
2006

Nice.

Three comments:

- Why not use a much userable idea of "con10uous zoom" (Jef Raskin),
instead of different finger combinations for different kinds of zoom, which
is not only harder to learn, but also more error prone? A two-finger
"con10uous zoom" would zoom out application content up to a point, after
which the application window and the space around it begin to zoom out. I
would keep the five finger chord for easy access to entire strip of
applications.
- The boundaries of Individual application windows can easily be
expanded/shrunk via vertical bars on both sides of the window -- the
affordances are more obvious.
- Sleep and shutdown buttons are precariously close to search input.

--
Oleh Kovalchuke
Interaction Design is design of time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

14 Oct 2009 - 2:30pm
Oleh Kovalchuke
2006

Comments edited:

- The distinctions among 2-, 3-, 4-, and 5-finger chords are hard to
learn and to remember. Why not use a more userable idea of two finger
"continuous zoom" (Jef Raskin), instead of different finger combinations for
different kinds of zoom. The two-finger pinch would zoom out application
content up to a point, after which the application window and the space
around it would begin to zoom out to reveal the open application strip.
- The five finger chord is a good shortcut to zoom to entire task strip
of applications.
- It makes sense to put favorite applications in the right panel, but I
would use zoomed out, application strip view to access both open apps strip
and all available applications in another strip/view below it:

[ ] [ ] [ ] [ ] [] - open apps strip

[] [] [] [] [] [] [] [] [] - all available applications strip

Interesting possibilities for rearranging the content of the strips.

- The boundaries of individual application windows can easily be
expanded/shrunk via vertical bars on both sides of the window -- the
affordances are more obvious than gestures.
- Sleep and shutdown buttons in the right panel are precariously close to
search input.

--
Oleh Kovalchuke
Interaction Design is design of time
http://www.tangospring.com/IxDtopicWhatIsInteractionDesign.htm

14 Oct 2009 - 9:22pm
Brian Mila
2009

I think the burning question is: Are you going to be able to give the
10 finger desktop the 3 finger salute? haha i couldn't resist.

I think Oleh's comments about the zoomable desktop are right on.
When I was in grad school I did a research project that proved that a
large, zoomable, virtual desktop was far superior to a standard
desktop. The research was focused on a wearable computer with a head
mounted display, but the results should still be applicable to the
desktop.

http://www.brianmila.com/wearable/zoomable.html

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

15 Oct 2009 - 6:06am
Uidude
2009

Nice video there. I sense there could be a difficulty initially in
identifying which of the 5 pointers belonged to its respective finger
while they all are closely placed and look the same round with at
the center. The mouse pointer is just one thing we need to focus on
now which is relatively easy to follow as we make mouse movements.
But, with these 5 pointers, one will have to quickly confirm that
focus is on the right finger and that the pointer is at the right
place before doing anything there :D Can't image doing the right
thing with 10 pointers on screen and you can use any one of them to
click or make selection. How would you get the system to identify
that when only one finger is pressed, say my index finger when I
actually start applying pressure on the pad with little finger?

But nice post, interesting to think about :)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

15 Oct 2009 - 8:38am
.pauric
2006

I have another question Clayton, more of a technical detail than
anything....

With the hybrid resistive-capacitive hardware you propose. And with
the limitation of resistive sensors to only being able to register a
single x-y coord at any point in time. How does the 10gui system
detect the multifinger clicks required for the advanced window
manipulation?

As an aside, for the list's benefit, I've been using the latest
wacom touch-pen tablet for a couple of days now. Its a really
exciting and natural way to interact with the OS. However, with the
capacitive only input one has to hover their fingers above the
surface a lot. Resting a finger on to the surface is an action. In
short, its becomes a little tiring to use a capacitive only
multitouch surface for prolonged heavy duty input.

http://www.wacom.com/bamboo/bamboo_pen_touch.php

I only see touch as a primary method of input working through
Clayton's proposed solution. However, I don't think I fully
understand how multiple clicks are achieved with current technology.

thanks! /pauric

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

15 Oct 2009 - 10:52pm
Clayton Miller
2009

Thanks to everyone for the very insightful comments! I hope I can
address some of the questions in depth soon, and really appreciate
the thought that has gone into your responses. Thanks also to whoever
tipped off TechCrunch early Tuesday... it's been a strange and
exciting week since then!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

16 Oct 2009 - 5:01am
Fergus Neff
2009

Wonderful, provocative stuff Clayton!

Like many others have expressed, I want to get my hands on there and
try the system out!

I'm also interested to see how using multiple monitors comes into
play. Particularly as in our case where we generally use two
widescreen monitors per workstation with one rotated and oriented to
portrait mode (the left screen) and the other in its default
landscape setup.

Intriguing.

Thanks,

Fergus.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

16 Oct 2009 - 3:07pm
Anonymous

I love it!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Posted from the new ixda.org
http://www.ixda.org/discuss?post=46632

Syndicate content Get the feed