“Innovation has nothing to do with how many R&D dollars you have. When Apple came up with the Mac, IBM was spending at least 100 times more on R&D. It's not about money. It's about the people you have, how you're led, and how much you get it.” - Steve Jobs

10/GUI – re-thinking the multi-touch desktop

Watch This! before continuing: (It’s an amazing video prototype that re-thinks multi-touch for the desktop.)

10/GUI from C. Miller on Vimeo.

First Clayton, I’m incredibly impressed by this video. It shows an attention to detail in production value, theoretical analysis, and overall solutioneering that I know many can learn and be inspired by.

I have a host of questions about the video:

1. This feels overly complicated. “It feels” that way b/c it looks like (I can’t play w/ it yet, right?) a piano metaphor and at that the gestures that each finger has to be able to articulate to reach the level of expert is even more complex than that of a pianist. That being said, I just realized that most functionality is available in simple strokes and this can be a way of ramping up someone from novice to expert. I just wonder how many people will be able to make it past chopsticks.

2. I have worked in the industrial design community for quite some time now. Their reliance on tablet profile desktop screens through the use of Wacom Cintiques is quite strong. Other 3D software communities like visual effects artists, etc. are also making great use of this model. The flexibility of the the cinque to articulate between positions is the key to its success b/c it takes advantage of the reality that there are different modes of operation throughout the day. I do Alias and then I answer email. But my main point is that due to the this ability to transform the tablet is not permanent and thus the stress you suggest inherit in that design is not real. BUT! I also want to say that before we had computers we wrote. I mean w/ a pen and this meant looking down and such. We survived that, no? All that is to say that I would re-evaluate your critique of the tablet position, or your limited view of the tablet profile and re-examine it.

3. Who? who is this for? I think your video is trying to generalize experiences which may not be true. For example I have changed my entire mode of operation to be comopletely browser driven. Close to 80% of my daily interactions happen in the browser. This means that due to tabs I have 1 application open at a time with maybe 2 other widgets that I access from a sidebar (music & twitter). This model of interaction feels like it is an extreme case, which is why the person above who talks about music is totally perfect for this, b/c music has such a level of complexity of controls.

Further, is “window” operation/manipulation really that hard in the current model? I probably spend 80% of my time on the keyboard and 20% on the mouse, so what problem are you look at that requires such an intense shift in hardware and software models?

This is to ask plainly, “Is the problem that great or even really there at all?”

But another important point to this is that design needs to not just think about human mechanics, but needs to be situational. I love video prototypes, but I stress to anyone who tries it that video prototypes need to be situational. The narrative of human use is an imperative in making the medium be truly useful to a design process. So I would ask that a 2nd take on this video do just that. place a “real” person in the middle using this in their day to day life. Bumptop a similar attempt at desktop re-design has the same problem, while I think that Aurora by Adapative Path is a great example of changing the browser & hell the desktop too, by embedding real human narrative into their demonstrations.

4. the loss of direct manipulation feels to me to be the crux of the issue. What makes an iphone and other multi-touch systems “work” is direct manipulation. If multi-touch is just a remoted system like a mouse, then all you’ve done is change the point of gesturing and added an arguable level of complexity that is not required. to me it is direct manipulation and not gesturing where the greatest added benefit of touch comes to play and this doesn’t address this.

5. the keyboard. 1 area that is interesting is that you don’t address the transitional moments of shifting between keyboard (well we all know the keyboard is a problem in and of itself) and the “pointing device”. The simple and current elbow articulation to a single handed mouse not only is simple and the muscle memory easy to embed, but it has the added value of leaving 1 hand on the keyboard so that experts (ever play doom?) can gesture with both hands to create unique modes of operation. I.e. control and drag causes a copy.

But the real benefit of this mode of operation is that w/ one hand “always on the keyboard” you end up with increased efficiency of target acquisition b/c the moving hand can always use the reference point of the stationary hand when trying to find home keys when leaving the mouse.

6. the existing trackpad issues where not addressed. The main one being my wife’s pet peeve which is that she always looses her cursor with accidental taps by her wrists.

7. Why choose 1. Going back to my ID studio, the current set up has many devices: critique (pen direct touch), 3D mouse, keyboard and regular mouse.

Most importantly though, that putting yourself out there like this to criticism is HUGE. This is inspirational not only for what it offers directly but because it offers a point of discussion. I could have never done this level of articulation with someone to respond to as well produced as your demonstration. It has great thinking and there are real problems to address, or more importantly even if the problems aren’t great, there are still places where we can hope to do better.

My top list:
1. object management
2. ergonomic data entry
3. mental models of pervasive and transitive computing
4. form factors beyond (too wide, open, eh?)
5. self induced behavioral change through technology

Thanx Clayton for putting this out there! It is going straight in front of all my classes and my entire faculty today!

— dave

Be Sociable, Share!
  • http://designaday.tumblr.com/ Jack Moffett

    A very thorough and thoughtful analysis, Dave. I completely agree with it.

  • http://www.zuschlogin.com/ Michael Zuschlag

    Don’t Mac notebooks already have multitouch touchpads? That said, this is the first use of more than two fingers in multitouch that strikes me as workable when there is no haptic feedback (like on a piano). However, it suffers the same problem found in many that other attempts to expand multitouch to more than a couple commands: poor discoverability and a certain degree of arbitrariness. The local-global levels of multitouch are suitable for expert shortcuts and cannot be the only means of interacting.

    Oversized desktops are an old idea, dating from at least the mid 1990s. Such desktops generally extend beyond the screen in _both_ dimensions. I’m not clear why one dimension should be superior. In any case, they don’t really scale well. It’s like having your work spread out on the dining room table, then working at it in the dark with a penlight. Managing multiple windows (or tabs) is indeed a problem. I spend a fair amount of time selecting, sizing, and moving windows. The main problems are: 1. Finding the right window/tab among dozens of others. 2. Coordinating work between windows (e.g., referring to an email while making an appointment). With most of the windows scrolled off the screen in Con10uum, I don’t see it addressing these problems any better than a thumbnail-capable taskbar/dock.

  • jezza

    download & try it here – http://wpfcon10uum.codeplex.com/


The archives run deep. Feel free to search older content using topic keywords.