How can computers use tacit knowledge?

I'm seven weeks into my Master's, and the clouds are beginning to clear around my research topic. But first, an aside...



In my improv class at Impatient Theatre on Saturday, there was a scene that was described as "incredibly complex". We were playing with the idea of an "analogous scene" that explores and expands the themes of a previous scene in a different situation, with different characters.

The first scene was an old couple going through storage in their basement. As the scene progressed, there was subtle tension between the two, with the woman wanting to bring something up, and the man ignoring her, and instead bringing attention to some old toys. He passed one to her (a plastic Pony) and held one himself (a He-Man). He held the He-Man in front of his face and said "Hey there little Pony!" and so on. They weren't interacting as a couple anymore, but as two little toys. Finally, the Pony said that she wasn't sure if it was working out between her and He-Man, and that maybe they should re-evaluate their relationship. Both the woman and the man froze, the conversation between the toys having become about their relationship. The scene ended.

The second scene was two friends exploring the garage of a house that one of them was house-sitting. Strangely, there were a bunch of sports equipment and other costumes in the garage. The house-sitter started putting on and taking off equipment for football and hockey, etcetera, while the friend starting playing with some Star-Wars costumes. The more equipment the house-sitter had on, the angrier they would be about some issue with the friend (a car getting scratched, poster in their house being thrown out). Meanwhile, the friend thought they were just play-acting scenes from the Star Wars movies. At one point, the friend put a Yoda head on the house-sitter to try to get him out of his angry mode, but the house-sitter took it off, saying it didn't feel natural. Finally, the friend put the Darth Vader costume back on and got really angry about an insignificant housemate issue, attempting to show the house-sitter how he was acting. The house-sitter responded with "Did that happen in the movie?" and the scene ended.

The first scene took about 2 minutes, the second one took 4 minutes. We analyzed both of them for about 5 minutes apiece, and I still find myself thinking about them. When the improvisors were asked why they made certain choices, they couldn't actually verbalize their thinking or reasoning, yet all of them made amazing scenes. It turned out that the improvisors in the second scene had two very different in-the-moment interpretations of one part of the scene, yet they together made something that was interesting to watch from the audience.


In my continuing research about learning by mimicry, I've become interested in the idea of tacit knowledge, knowledge that is defined by its difficulty of communication. Tacit knowledge is usually passed by a learner observing someone else, whether they are explicitly a teacher or not. I assert that interactions that require or use tacit knowledge are more powerful and expressive than those that are not. While my initial motivation to explore this space was miming, a form of physical expression frequently used by people when words are not enough, tacit knowledge exists in the use of any sufficiently complex interaction. I use the metaphor of drafting while riding a bicycle as the way that a user can explore the space of existing tacit knowledge:

1. Advance tentatively
2. Have the space of previous interactions shown in a useful representation
3. Made a choice
4. GoTo 1

A small philosophical problem I have with this is that it seems to suggest that all possible interactions are pre-determined. Well, on a regular basis, the vast majority of interactions are pre-determined. New space can be explored by a different mechanism, but frequently it is built on existing knowledge from different areas that is synthesized, filling in convexities.

I want to focus this down to how tacit knowledge about physical interactions is communicated and processed by users. In particular, I want to focus on newer gestural interfaces, which are currently baffling to people that haven't seen them (I played with one, and it was trial and error). If communicating gestural information is possible, it also means we can move beyond the "list of (x,y) finger points" as the only possible interaction, which is embarassingly limiting given the possibilities afforded.

In the improv scene I wrote about above, the improvisors in the second beat didn't know explicitly what pattern they were following, but they did know it, and they could expand on it. This is a power that is hard to express in existing interfaces. If computers can be made to express tacit knowledge, then I believe we've taken another step to the final goal: making computers worth caring about.

Image above is a hand painting that was made each night for a play I was in this summer, "For a Better World" or "Für eine bessere Welt", for Single Thread Theatre Company. One theme of the play, as I saw it, was the evolution of culture.

Stuck in Real Time

At Friday's DGP party, a projector and a webcam was set up. The webcam was aimed at the party, and the projector was aimed at a large white wall, displaying the party. This is all good and cool, and if you aim the webcam so that it sees part of the projection, then you get fun feedback.



However, the clincher was that the projector displayed what happened on the webcam 2 minutes in the past. This led to the party itself becoming self-aware, where much of the discussion led to what had happened 2 minutes ago. Since the webcam was aimed to produce feedback, then this included what happened 4, 6, 8 minutes ago too.



As the party continued and my own personal inebriation increased, combined with my recent play-through of Braid, it occurred to me that the projection on the wall represented a barrier between this world and the world from 2 minutes ago. I wanted to cross this barrier, and be in both worlds at once.

Me taking a picture of myself taking a picture:


I created a sequence of movement of myself in front of the projector that lasted about 2 minutes, and kept repeating it and repeating it. The more I did it, the more I aligned with the past version of myself. As the party kept going on in parallel worlds, I was the only person in every world at once.

Crazy Dream

So I'm in Toronto now, yada yada yada all is great and shiny and mucho-fantastic. But that's another post. For now, let's discuss a dream I had last night. I'm lucky to have pretty vivid dreams regularily, but this one was too odd to let go. I tried to send it in to my twitter, which I use much more often than this blog, but I couldn't make it concise enough. Here it is, in a sort of half-grammar:

Dream: Was sentenced to hang for mistakenly kicking and killing someone in soccer. Goodbyes to family and friends; neck-strengthening exercises. Dad said I could borrow car to flee to States. Realized this was ridiculous; became lucid; woke up.

A conversation with the Google Earth API

Dustin: Oh hey there, Google Earth, it looks like you have an API so that we can actually talk to each other.
GE: Yeah, hello! It's really useful.
Dustin: Well, we'll see about that. So, I've turned you on. Can you tell me what layers you have?
GE: Of course! I've got the Geographic Web, Roads, 3D Buildings, Street View.. [goes on a long list]
Dustin: Oh that's great. So, can you tell me what is in each layer?
GE: ...
Dustin: Hello?
GE: ...
Dustin: Oh. I see. Well if I zoom in, I can see that panoramas from Street View are now rendered as spheres floating in space where the picture was taken. That's really cool! But, I guess you can't tell me where they are. I'm looking at a few now. Which ones are they?
GE: ...
Dustin: Okay. What about this one? 482 Avenue of the Americas in NY, NY.
GE: Oh yes. I've memorized that one now.
Dustin: Okay. What do you see now?
GE: 482 Avenue of the Americas in NY, NY. That's all.
Dustin: Oh. Well, can you memorize that street view locations for 478, 486, 490 too?
GE: Yep! Done.
Dustin: Okay, what do you see?
GE: 478, 482, 486, 490 Avenue of the Americas in NY, NY.
Dustin: So you'll only tell me you see something if I specifically tell you to look for it?
GE: Yes.
Dustin: Well what about that panorama over there? 94 W 12th Street?
GE: Sorry I don't know what you're talking about.
Dustin: Really. Memorize 94 W 12th Street.
GE: Ah, yes. So now I see 478, 482, 486, 490 Avenue of the Americas and 94 W 12th Street in NY, NY.
Dustin: Okay fine, you're young so it's okay. Now, can you tell me in latitude and longitude where 482 Avenue of the Americas is?
GE: ...
Dustin: No? But its right there on the screen. I can click on it to make you fly to it!
GE: Oh, I can fly there!
Dustin: Where is 482 Avenue of the Americas?
GE: ...
Dustin: I said where is 482 Avenue of the Americas?
GE: ...
Dustin: Please fly to 482 Avenue of the Americas.
GE: Done.
Dustin: Okay, where are you now?
GE: [gives a series of latitude and longitude coordinates of where it's looking]
Dustin: I see. So, you can't tell me where 482 Avenue of the Americas is, but I can ask you to fly right to it and then you'll tell me where you are. It seems like you're trying to make my life difficult.
GE: [shrug]
Dustin: Okay, now that I'm sitting at 482 Avenue of the Americas and looking towards the horizon like I'm driving. Cool! So, let's turn to the left to look over there.
GE: [turns slightly to the left]
Dustin: Wait, hold on, I'm not in the sphere anymore. I've moved over to the right of the road somehow.
GE: Yes, I turned you around the Focus point!
Dustin: What? What is that?
GE: It's where you're looking at.
Dustin: I thought I was looking at the panorama at 482 Avenue of the Americas.
GE: Actually, that's what you told me to do, but I decided to fake it by making you look at a point 60 meters to the north at a really shallow angle. You weren't actually looking at the panorama at all.
Dustin: Can I look directly at the panorama at all?
GE: ...
Dustin: ...
GE: When I look at something, its defined by the point on the earth itself, and then the compass angle, vertical tilt, and distance I am from it. The panorama is just floating above the Earth, so you can never truly look at it.
Dustin: But there's little controls I have on you that let me move you around whatever way I want. However, when I actually want to talk to you I can only tell you to move in certain ways.
GE: Yes.
Dustin: Well okay. Let me think. I know! How about, instead of looking at a point 60 meters to the north of the panorama, I tell you to look at a point 60 meters to the northwest, and then make the compass angle so that the panorama is in the center of your view.
GE: Like this?
Dustin: Wow! It's like I rotated 45 degrees around the panorama.
GE: No it isn't. Not exactly. That's not what I did.
Dustin: Shut up, it looks like I did, and that's all that matters.
GE: Okay.
Dustin: So, in order to rotate around a panorama as if I'm viewing it, I need to exactly tell you the panorama I want (I can't ask which panoramas are available). Then, I have to tell you to fly to it. Then, I have to ask you where I am, and the answer you give back to me is some point 60 meters away from the panorama. Then, I have to do some clever trigonometry to figure out where on earth the actual panorama is. Then, in order to rotate around the panorama to look at it from different angles I have to make up different points on the Earth in latitude and longitude which are on the opposite side of the panorama I want to look at. Then I can tell you to fly to those points, which don't explicitly look at the panorama but panorama happens to exactly be in the way of the point on the ground I told you to look at.
GE: ...
Dustin: Right, you're not much help here are you?

Weekend Roundup

This past weekend:

- My best friend Dave came up from Guelph.
- I volunteered for Admitted Students Day at Queen's.
- I saw Iron Man with Dave, and my other friend Daniel and housemate Jack.
- Visited Free Comic Books Day, saw nothing I really liked.
- Watched three famous movies I had not seen before: Shawshank Redemption, Vertigo by Alfred Hitchcock and Amadeus.
- Started playing with some Image Processing stuff on paper.
- Went for a bike ride.
- Had a recording session for the radio play I'm in.

Generally, I had a pretty sweet weekend. Let there be more!

I broke 50 kilometers!

I just got back from a 54 k bike ride!

Here's the google map.


View Larger Map

Now, my average speed sport of sucks, as it took over two hours, but that can be improved. I think I'll need to buy a bike clock.

Real-Life Spam

Well, the house just received 30 identical looking standard-size envelopes, none of which were addressed to any of us. Most of them were weird variations of either Phillip Van Stone (18) or Andrew Collins (9). I opened one for "Mike ColinManing", which I assumed I couldn't get prosecuted for since it obviously wasn't a real person. It was an invitation to some $20,000 tournament (for which we had OBVIOUSLY already been considered) at PartyPoker.com. Below is a scan of all of the Phillip Van Stone variations: