February 23, 2010
David Cope’s software creates beautiful, original music. Why are people so angry about that?
Ryan Blitstein in Miller-McCune:
She was just a software program, a jumble of code he’d originally dubbed Experiments in Musical Intelligence (EMI, hence “Emmy”). Still — though Cope struggles not to anthropomorphize her — he speaks of Emmy wistfully, as if she were a deceased child.
Emmy was once the world’s most advanced artificially intelligent composer, and because he’d managed to breathe a sort of life into her, he became a modern-day musical Dr. Frankenstein. She produced thousands of scores in the style of classical heavyweights, scores so impressive that classical music scholars failed to identify them as computer-created. Cope attracted praise from musicians and computer scientists, but his creation raised troubling questions: If a machine could write a Mozart sonata every bit as good as the originals, then what was so special about Mozart? And was there really any soul behind the great works, or were Beethoven and his ilk just clever mathematical manipulators of notes?
Cope’s answers — not much, and yes — made some people very angry. He was so often criticized for these views that colleagues nicknamed him “The Tin Man,” after the Wizard of Oz character without a heart. For a time, such condemnation fueled his creativity, but eventually, after years of hemming and hawing, Cope dragged Emmy into the trash folder.
This month, he is scheduled to unveil the results of a successor effort that’s already generating the controversy and high expectations that Emmy once drew.
More here.
Posted by Abbas Raza at 08:15 AM | Permalink
Comments
This is fascinating, but not in the way intended. Cope seems to have hit upon a way to not have to take responsibility for his compositions (which is especially interesting in light of the revelation that he originally turned to computers out of writer's block, and is no doubt related to his present testiness about the subject).
When Haydyn and Mozart wrote their dice music, they were not at liberty to say that the dice "wrote" the pieces, nor blame the dice if the music sucked. The pieces began with the composers' intention to write music in such a fashion. The same is true of John Cage, or John Zorn--if Zorn's Cobra were a flop, we could not fault the cue cards; whatever the defects of Cage's Music of Chance, we cannot look to the writer of the I Ching.
A computer named Emily did not compose any music. "She" just did some of the complex sequencing of notes, because Cope told "her" to. The idea was his, the constraints were his. The idea of deviating from those constraints was his. The determination that the music was good enough to perform publicly, or to record, was his. For Cope to present the result of all this composing as the work of a computer is just sad, not least because our culture's mania for intelligent robots will have many people believing that this computer did something interesting (not least of whom Douglas Hofstadter, who should know better.)
Posted by: Chris Schoen | Feb 23, 2010 1:25:02 PM
I listened to the two short pieces by "Emily Howell" and they are repetitive and derivative imitations of real music. Mozart and Bach can rest in piece.
Posted by: J.Hawkins | Feb 23, 2010 1:32:46 PM
Has anyone been to the Stephen Wolfram site to listen to WolframTones? I think he has set the computer to doing something interesting -- sort of ticking over on its own and coming up with iterations that a human would be having to copy it -- rather than the other way around -- to do. But it sounds just awful, much much worse than the kind of stuff that, halfway up the hi-rise, you run out of the elevator to get away from.
http://tones.wolfram.com/
As for Bach, gee -- he's probably still ahead. It would have taken a musical scribe of his era as much time to copy out his works as it did him to compose them. The computer that beat Gary Kasparov at chess is not quite there yet.
Posted by: Elatia Harris | Feb 23, 2010 1:56:47 PM
J. Hawkins, perhaps you just don't like the style he chose for those Emily Howell pieces...it may be more informative to listen to pieces intentionally composed in the style of a real composer. I remember at a Douglas Hofstadter lecture in college, he played a real Chopin piece alongside a Chopin-style piece composed by "Emmy", and the audience was evenly split on which one was real. Unfortunately not too many of Emmy's pieces seem to be online, but you can preview snippets of tracks from Cope's "virtual Mozart" CD here:
http://itunes.apple.com/us/album/virtual-mozart-experiments/id251530884
and Bach:
http://itunes.apple.com/us/album/bach-by-design-computer-music/id325822075
Of course the best way to really judge the quality of such pieces is in a "blind" test like the one Hofstadter gave, where you don't know in advance whether you're listening to an Emmy piece or the real McCoy...as the article notes, there's a sort of mental bias that creeps in where if you already know the music is computer-composed, you'll convince yourself that you can detect telltale signs of artificiality in it. Consider this anecdote from the article:
At one Santa Cruz concert, the program notes neglected to mention that Emily Howell wasn’t a human being, and a chemistry professor and music aficionado in the audience described the performance of a Howell composition as one of the most moving experiences of his musical life. Six months later, when the same professor attended a lecture of Cope’s on Emily Howell and heard the same concert played from a recording, Cope remembers him saying, “You know, that’s pretty music, but I could tell absolutely, immediately that it was computer-composed. There’s no heart or soul or depth to the piece.”
The human mind is very good at convincing itself that its observations confirm things it was already predisposed to want to believe! Also consider this other paragraph:
When Emmy’s Bach pieces were first performed, at the University of Illinois at Urbana-Champaign in 1987, they were met with stunned silence. Two years later, a series of performances at the Santa Cruz Baroque Festival was panned by a music critic — two weeks before the performance. When Cope played “the game” in front of an audience, asking which pieces were real Bach and which were Emmy-written Bach, most people couldn’t tell the difference. Many were angry; few understood the point of the exercise.
Posted by: Jesse M. | Feb 23, 2010 2:22:52 PM
Thanks for the link, Elatia.
That "music" is so bad it's almost good. But there's no getting away from intention, is there? John Cage meant to have us listen to the world more closely by hearing the music inherent in it, despite the lack of conventional indicators ("rules") we associate with music. His work predicated on human sensitivity. WolframTones is exactly the opposite, all rules, no content, predicated on a complete lack of human sensitivity. Ouch. Ouch!
Posted by: Chris Schoen | Feb 23, 2010 2:32:49 PM
Chris Schoen wrote:
A computer named Emily did not compose any music. "She" just did some of the complex sequencing of notes, because Cope told "her" to. The idea was his, the constraints were his. The idea of deviating from those constraints was his. The determination that the music was good enough to perform publicly, or to record, was his. For Cope to present the result of all this composing as the work of a computer is just sad, not least because our culture's mania for intelligent robots will have many people believing that this computer did something interesting (not least of whom Douglas Hofstadter, who should know better.)
That's a pretty reductive argument. Certainly some of the credit for any music composed by Emmy should go to Cope, but it's not an all-or-nothing matter! Human mathematicians also came up with the equation that, when plotted on a computer, result in all the visual intricacies of the Mandelbrot set, and in any video of a Mandelbrot "zoom" humans also picked what sections to center on during the magnification process, but would you say that means all the intricate forms seen in this video should be credited to the creativity of those humans? (for some more background on the Mandelbrot set and the related "Mandelbulb", you can check out the post I wrote up here) Can't mathematical equations and computer programs have "emergent" results that the humans that wrote them down didn't anticipate and can't claim creative credit for?
For a more science-fictional thought-experiment, you could also imagine designing a simulation of an embryonic brain that would grow and learn from experience gained through its sensory inputs (connected to a robotic body, say) just like a real biological brain...if such a simulated brain eventually showed humanlike creativity, would you argue that all its creative output should be credited to the original human programmer?
Posted by: Jesse M. | Feb 23, 2010 2:53:42 PM
Jesse M.
Thanks for the links. I listened to short excepts of Mozart and Bach. Superficially, they do sound like Mozart or Bach, but after just a few seconds you notice a rigid adherence to formulaic pattern that Mozart or Bach would never have maintained. They know how to vary the pattern and go off into original and unexpected sound combinations. The computer knows the method well, but not how to transcend it. It's ersatz material and therefore lifeless. There is no emotion - after all, what could a computer know of human emotion? In the end, music is not abstract pattern, but emotion in sound. Still, it is an interesting comparison.
Posted by: J.Hawkins | Feb 23, 2010 3:03:26 PM
I listened to short excepts of Mozart and Bach. Superficially, they do sound like Mozart or Bach, but after just a few seconds you notice a rigid adherence to formulaic pattern that Mozart or Bach would never have maintained.
But like I said, people have a tendency to have their perceptions strongly colored by preexisting beliefs without being aware of the extent to which this is happening (consider the anecdote about the two totally different reactions to the same piece by the 'chemistry professor and music aficionado' before and after he knew they were composed by a computer). Without doing a blind test where real compositions by a given composer (ones you aren't familiar with) are played side-by-side with Emmy's faux versions, without your knowing which is which, you can't really be sure if your own perceptions of a "formulaic pattern" are genuine or just an example of your perceptions being colored by beliefs in this way. Hofstadter said at the lecture he had done this side-by-side test with people many times (including with plenty of people quite knowledgeable about classical music, if I'm recalling what he said correctly) and they pretty consistently did poorly at telling the real from faux versions, so you might be surprised at the results of such a test if you did it yourself!
Posted by: Jesse M. | Feb 23, 2010 3:23:46 PM
I also think that while a computer may provide some kind of imitation of classical music forms, I can't imagine a computer coming up with a revolutionary sound like Stravinsky's Rite of Spring. When a computer can do that then humans may have outlived their time on earth.
Posted by: J.Hawkins | Feb 23, 2010 3:28:57 PM
I wouldn't say that computers will never be able to come up with really new sounds that are aesthetically pleasing or interesting, but I agree that this kind of innovation is beyond the ability of programs like Cope's, which are basically just riffing on existing styles...
Posted by: Jesse M. | Feb 23, 2010 3:39:13 PM
Jesse,
As computers can have no experience of being a physical body in a physical world, of having a heartbeat, of feeling the sensations and rhythms of muscle movements, of knowing what it is to sing and dance or even simply of hearing actual sound waves in air, I do not see how they will ever be able to do anything but imitate the creations of humans who experience all these things.
Posted by: J. Hawkins | Feb 23, 2010 4:30:21 PM
Of course the thing isn't as good as Beethoven - Cope himself sounds almost heartbroken he's never created one truly excellent piece. Concede every point made against this program here. I do claim it's still as decent as the average ad jingle, movie soundtrack or jazz album. If instead you care about AI, this is greater success than computers have had with speech recognition. How is that something to sneeze at?
Posted by: prasad | Feb 23, 2010 4:39:59 PM
didn't finish. Most people can make only trashy, derivative art. If the day a computer makes exceptional music is the day people have "outlived their time on earth", for most of us that time was several hundred years ago.
Posted by: prasad | Feb 23, 2010 4:44:07 PM
Prasad,
Who's sneezing?
My point was that a human being scrupulously programmed a piece of software to do a certain thing, which to was credibly emulate the music of an existing composer. This is something slightly less than what both Cope and his hagiographer claim, which is that a computer wrote music. ("...could create a Mozart sonata every bit as good as the originals.") Not quite.
We cannot separate "Emily's" creation of faux Bach and Mozart from Cope's intention of the same. Music without mind is just noise (which, again, was Cage's insight).
Posted by: Chris Schoen | Feb 23, 2010 5:14:47 PM
Prasad, that's well conceived and well said. But you put me weirdly in mind of the early days of Second Wave Feminism. When women _could_ shatter barriers, but only by being many times as competent as the average man. What sisters wanted for each other in those days was not so much for superior women to be recognized and rewarded as for any woman to be just as mediocre as a man, and still be competitive. While this analogy might not take me as far as I would like, I am wondering if those standards can or should be transposed to apply to a middling melody that's AI-generated. If it's not better than the basic human-produced jingle, if it's just more Oobie-doobie-doo, then how is that victorious or worthy? It's not as though the right to work were involved, or the children must be fed by that labor. The only rights I would be concerned with here are those of us potential listeners, who hope very much not to be subjected to AI sounds from a synthesizer with an IQ of 105.
Posted by: Elatia Harris | Feb 23, 2010 5:16:17 PM
As computers can have no experience of being a physical body in a physical world, of having a heartbeat, of feeling the sensations and rhythms of muscle movements, of knowing what it is to sing and dance or even simply of hearing actual sound waves in air, I do not see how they will ever be able to do anything but imitate the creations of humans who experience all these things.
If you're talking about a computer with broad humanlike creativity that would be able to have meaningful conversations, learn all the types of things humans can learn, create all the forms of art humans can create, and so forth, then you'd probably need one as complex as the human brain that did have the experiencing of being embodied and of learning through interactions with the world, including social relationships with other embodied beings (I think the most promising route to this would be mind uploading, where the idea is just to copy an existing human brain in a computer simulation rather than create a form of intelligence from scratch). But musical composition is a much more narrow type of creativity that's based on coming up with interesting patterns from combinations of a small number of notes...there seems to be some evidence that our appreciation of music is not tied too closely to the sounds reminding us of any specific sounds or kinesthetic experiences drawn from life, that it has more to do with a sort of general ability to anticipate abstract patterns in the way the notes unfold over time, see the paper discussed here or this book, for example. In limited domains like chess we know computers can match human creative abilities even without having any sort of broad humanlike intelligence, so it's more plausible to me that a computer without this sort of intelligence could match us in the field of music than it would be in, say, the fields of novel-writing or movie-making. I don't find it that counterintuitive that a computer without this sort of broad intelligence could imitate the style of a human composer, but think how astonishing it would be if a computer could write coherent original novels in the style of Dickens or Kafka!
Posted by: Jesse M. | Feb 23, 2010 5:24:19 PM
Chris and Prasad and Jesse M.,
If I sat down and improvised you some faux-Mozart, and it was really pretty clever, so that you briefly wondered whether I was playing something Schubert wrote on a dull day, you would not think I had an unsuspected gift for musical composition. You would think what I did wasn't a bad party stunt, and that, as useless knacks went, it was just about harmless. Also, after a few moments of reflection, you might feel rather sad -- for me, for everyone -- that knocking off Mozart trivializes the unique and irreplaceable, and that there were better things to be doing with one's skills. So why does Emily get more points and more opprobrium, too? Because she's a computer?
Posted by: Elatia Harris | Feb 23, 2010 5:29:46 PM
Elatia,
I don't follow. I'm saying if the music is good, bad or middling, it's to the programmer not the software. Emily gets no opprobrium from me, any more than a rhyming dictionary would (Mozart's dice, Cage's I Ching, Zorn's cue cards, etc.)
I completely agree that knocking off Mozart or whoever is trivial, though, and a sad way for an actual composer to close out his career. Cope might have said something of his own, or maybe taught students how to. For someone so clearly moved by music, he seems to have lost sight of what it's for.
Posted by: Chris Schoen | Feb 23, 2010 5:49:44 PM
Chris, I was responding to an ethos so to speak, and not criticizing any specific view. Surely one has sparred often enough for that :) Anyway, I'm actually quite excited by precisely the fact that this seems potentially like such a powerful tool for a composer. The passage where Cope describes dashing down from his shower to "code up" an undeveloped and hazy musical idea in fifteen minutes made me sit up. Imagine every composer having at his desktop a (fake, ersatz, foolish, whatever) version of his own musical character, always ready to quickly implement, test and debug new musical ideas...
Incidentally, I don't think Cope *just* carefully hard coded in bits of Bach to make Emmy recreate the sound by cutting and pasting. He wrote generic code that apparently lets Emmy fake any composer, not just Bach, moderately well given large music samples and some user guidance. The Beethoven, Scott Joplin and Chopin items also sounded plausible to me. That sounds like a bigger achievement than you're giving him (or it) credit for, and is easily something worth getting excited about.
Elatia, party stunts can be quite enjoyable in their own right.
More seriously, I think "dull day" and "briefly" drive your argument, and the attendant sense that something precious is being diluted away. If instead a musically literate (yes, maybe those audiences weren't. Doesn't change *that* much for my purposes. Given two things X cannot distinguish between, why should X deem the first sublime and the second tawdry? Or imagine, a few years on, music professors finding it hard to do the thing) college audience really does no better than chance at separating real-Bach from Emmy-Bach - and one assumes said audience likes real-Bach - why should it be dismayed by the existence of the copies? I also find it hard to believe anyone would be disturbed if a human being were doing this thing, or criticizing that person for not being as good as Bach.
At some point it seems like we're in de Beers territory, frantically preserving the worth of scarce, real diamonds against synthetic copies. Surely the value of beautiful music isn't like that! Good music doesn't become bad just because decent music exists as well.
Posted by: prasad | Feb 23, 2010 6:43:58 PM
"Surely the value of beautiful music isn't like that!"
Oh, but it probably is--that's why there's such different reactions in the blind vs non-blind tests. And that's why doubling the price on a bottle of wine will double its flavor, and other things like that. Of course, if you meant "beautiful music shouldn't be like that!" then I think one would be tempted to agree with you...
Posted by: billy | Feb 23, 2010 7:08:50 PM
The idea is simple enough: a computer algorithmically analyzes compositions that are deemed music by us to arrive at a set of heuristics based on the notes, their combinations, juxtapositions, sequences, frequency of occurrence, and more. Some "training" then allows the software to produce sounds similar to the music in the sample set. Cope's new software assigns weights to relationships between notes as in neural nets, which all seems like standard AI stuff.
As with AI in other realms, the real challenge for the computer are things like how to understand shifts in social context and modify the composition in response, incorporate a new instrument into the composition and decide when the result is music, improvise in the face of altered circumstances (the flute broke), etc. Basically it is incapable of doing much outside its training domain, exactly where humans excel because they have social skills, musical imagination, moods, bodily sensations, etc.
AI tackling music is not going to be like AI tackling chess, where it routinely beats grandmasters. The difference is that the number of moves in chess are finite, well-defined, and amenable to be optimized for a clear end (a win). Chess moves also do not benefit from being attuned to social context or mood or the cultural heritage of the players or the audience. Can't say that about music. Where AI is going to perform well is all the stuff that yields to algorithmic analysis. It'll likely go farther with Bach than with Coltrane. Folk musicians can rest easy, so to speak. It'd be good to discover how far AI goes with classical music before it runs into a brick wall. I think Cope's new software is best seen as an expert system, a tool designed to aid humans produce new kinds of music more efficiently.
(Folks might also want to check out my 3QD http://www.3quarksdaily.com/3quarksdaily/2009/06/the-dearth-of-artificial-intelligence.html ">essay on AI.)
Posted by: Namit | Feb 24, 2010 2:23:10 AM
Prasad,
If we're to be impressed, let's be impressed by the clever way Cope devised to emulate works of the classical greats. Let's not anthropomorphize his tools. If my fellow rug merchant gets a good price on a new shipment I do not compliment the ingenuity of his abacus.
If there is a "triumph of the cyborg" (eww) then the cyborg is Cope, and Emily is his bionic leg. But that's no way to look at making music, as we know from our agreement that neither Cope nor Emily have produced anything of artistic value.
Namit,
Good point about chess games not having a social function beyond the realm of the two players. (Yes, chess will occasionally interest a small audience, but this is negligible. We don't generally reify the results of most tasks qua tasks that computers do.)
Good point too about Bach, which Elatia hinted at earlier. Bach's music was made to be analyzed in such a fashion, but the man was a freak.
But even if Cope were to have his computer emulate pieces by, say Lizst (or Coltrane), he would still be reliant on these composers having already lived and written as they did. This project is, at best, mimesis.
To say something new requires something to be said, which requires a mind to want to say it. If "Emily" generated some brilliant new form (which she so far patently has not), the brilliance would be in the recognition of it by the composer-programmer, who would use his/her own intentionality to associate it with a kind of statement. If the composer-programmer did not intend such a result, then the critic or audience member becomes the new "composer" (again, this was Cage's great insight in applying meaning to patterns which arose without one. Or Duchamp's insight of same.)
Perhaps someday computer-cyborgs will develop the kind of agency Cope and others are trying to ascribe to Emily. Until they do, however, we are right to keep inquiring into our seeming desperation to describe such agency where none exists.
Posted by: Chris Schoen | Feb 24, 2010 2:10:27 PM
Chris,
I think the lack of intentionality or agency is definitely one of many reasons (another being lack of acculturation as a human) why AI machines are condemned to keep failing the turing test. However, I don't believe the turing test itself requires the presence of intentionality in the machine. It does not require the machine composer to "intend such a result", simply that it be indistinguishable from what humans create. In any case, when a human offers a composition to the world, say via iTunes, it becomes a standalone product in the world, to be received by the audience as a thing in itself. The audience decides whether or not it is good music—independent of the fact that the composer had "intention" behind it. Or as JM Coetzee wrote about writing:
So I think we can't use the lack of intention or agency as a reason to disparage the results of "Emily". But we can use it to explain why "Emily" falls short where it does. Do you agree?
Posted by: Namit | Feb 24, 2010 4:38:02 PM
Abbas informed us this morning that TV News is being generated more or less in this matter. In that case however, it is wholly intentional.
Posted by: Ruchira | Feb 24, 2010 4:55:11 PM
I agree with this, but consider it a flaw in the Turing Test, not a feature. That is, mind/intention is not a characteristic of intelligence by Turing's definition, which I think is just silly.
As for the Coetzee quote, I agree that the author is not the last word on her own text. This is the nature of communication, and is one of the things Barthes was getting at with his "death of the author." But this is not to say that a work of art does not begin with authorial intention, even if it is shaped along the way in unexpected ways.
I hope I don't sound like a broken record, but my point seems to get repeatedly missed (or I am being unclear). I'm not using the lack of intentionality to disparage Emily's results. I'm saying Emily has no results. Cope does, and they stand or fall with him. My "scratch paper" did not get a [score undisclosed] on the SATs (back in the Reagan era); I did. Any "falling short" was my own.
Posted by: Chris Schoen | Feb 24, 2010 5:04:18 PM
I hear ya, Chris, but wouldn't it be an awesome iPhone app?
Posted by: Vicki Baker | Feb 24, 2010 7:35:07 PM
Namit, what a good article -- good then, good now -- thank you.
Chris, I wasn't disagreeing with you up there. But now I am. Isn't it the intention of the programmer to write in "potential" in the form of possibilities she doesn't utterly foresee? So that things could happen without her knowing they would. I think I read way back when it was possible to program for that. If this is what Cope did, it hardly turns Emily into a prodigy of aleatory music, but it speaks to your view of intentionality. If it were not musical composition, but a program for mechanically tossing lollipop sticks into the air, wouldn't the patterns they made when they fell to the ground be sometimes more beautiful than other times? In a moral universe, intentionality matters; in an aesthetic one, maybe it matters less.
Vicki, yeah! What an app!
Posted by: Elatia Harris | Feb 24, 2010 8:28:27 PM
Elatia,
Yes, but works of art are more than aesthetic objects; otherwise everything would be one. Patterns of lollipop sticks, but also the list of files in your trash folder. Or the gibberish that you get when you open a document with the wrong software.
These can be art, too, perhaps, with the help of a Warhol, Duchamp or Cage. But they are not art simply by being in themselves.
Back to your lollipop sticks: Jackson Pollack didn't--and couldn't--predict the placement of every drip and aplatter. His paintings were partly technique, and partly stochastics. But he knew when he'd done it right or not, or better have. Knew when he was done. (How many of us have watched a child "ruin" a touching drawing she was working on because she didn't know when to stop?)
Which is perhaps a way of saying that art is intrinsically moral, too (but not "moralistic.")
Posted by: Chris Schoen | Feb 24, 2010 9:21:05 PM
Chris,
Stravinsky was once asked what music expressed. Nothing, he said. "Music expresses nothing -- it expresses itself." Then what is mood in music for?, he was asked. "Girls," he replied. Well, at the time (1970s) he was tightly in the grasp of Robert Kraft. And meaninglessness was newsier then. But, if intentionality were supreme, we could never find beautiful the striped rock, the mackerel sky, the sparkle on the water. Don't you think it is from these phenomena, and others like them, that our idea of what is beautiful was formed? (Oh, not to sound like Denis Dutton...)
I am as guilty as anyone of ascribing moral qualities to the beautiful -- just think of the lives that Mozart has saved. But everything I know and feel tells me that our yearning for a moral life allows us to find beauty everywhere -- look! listen! -- but not to create it. If only.
Posted by: Elatia Harris | Feb 24, 2010 9:55:58 PM
Namit wrote:
I think the lack of intentionality or agency is definitely one of many reasons (another being lack of acculturation as a human) why AI machines are condemned to keep failing the turing test.
Are attributions of intentionality/agency just based on qualitative judgments about whether a system has a sufficient level of self-awareness or understanding of its own actions or some other such characteristics, or are these words meant to signify something more precisely-defined and non-subjective? Are you arguing that a computer program could never have intentionality/agency, for example? Do you think insects have these things?
Posted by: Jesse M. | Feb 24, 2010 10:43:17 PM
But how will we know when that day comes, Chris? What is the test for "agency"?
Because it seems to me that you don't think that it's ever possible. From your comparisons of Emily to an abacus and scratch paper, it's clear that you would think that anything that a "computer-cyborg" creates is really created by the persons who created the "computer-cyborg", no? How is such a "computer-cyborg" ever able to earn credit for the things it creates? Does it have to also create itself, by its own bootstraps?
Posted by: billy | Feb 24, 2010 10:56:33 PM
It just seems to me to be a distortion of how we usually talk about things, Chris. Suppose I was visiting a factory (or watching an episode of "How It's Made") where raw materials are dumped in at one end and finished products pop out of the other, and I make the comment,
Would you admonish me then and there, saying,Posted by: billy | Feb 24, 2010 11:50:58 PM
Elatia,
It is possible that Stravinski was just wrong. It's more probable that he didn't want to get into a discussion of what music is saying, which would have been the logical follow up question, broaching a topic where angels fear to tread (but on which I will rush in, if pressed). (But I will never say that music or painting or dance can be translated discursively, which is probably what his interlocutor wanted to hear).
I don't argue that intentionality is "supreme" to the extent that the artist is never receptive, but always and constantly a fecund Dionysian. That's Nietzsche, not me.
There's much to be said for looking and listening, maybe for several lifetimes over. My point is, I hope, more humble; that art cannot exist without passing through the artist's intention (and then the looker/listener's, and beyond). I'm not ascribing a moral component to the beautiful (though I don't deny one). I'm saying art relies upon human agency to come to be. Surely even Stravinsky at his most dyspeptic would agree with that.
Posted by: Chris Schoen | Feb 25, 2010 12:59:32 AM
Billy,
I wrote earlier that I would step aside if a synthetic person created a piece of art.
I don't want to answer spuriously about how we could tell, except to say again I find Turing's answer flawed. I'll think more about it. We might have to rely on intuition.
In the meanwhile, the evidence that "Emily" is far more tool than "person," and that any creative property in "her" music traces back to Cope seems overwhelming to me, and no one has offered much challenge to my analogies.
I might. It's perfectly valid (whether normative or not) to rephrase your comment as
...which after all, would be more accurate.
Posted by: Chris Schoen | Feb 25, 2010 1:08:43 AM
Chris,
"I'm saying art relies upon human agency to come to be."
I agree -- art does do that, but beauty does not. If Emily cooked up something ravishing, it still wouldn't be art. It wouldn't even be birdsong. But it could well be uncannily beautiful, just as if... And there would be a moment of Hoffmannesque pathos, too. Then, the full sorrow of being morally defrauded, made to love a non-being, made to heed a meaningless call. Remember when Captain Kirk fell for a humanoid woman in a distant galaxy, with whom he flirted via that screen? A loooong distance romance that was going very well. Until they met in the flesh, and he was approximately the height of her face. Might be kinda like that.
Posted by: Elatia Harris | Feb 25, 2010 1:29:55 AM
That doesn't seem more accurate to me. The owners of capital may design the machines, and tell them what to make and how to make it--but it's the machines that are still doing the making. What if we replaced the machines and robots with people. Would you then say that the factory-line workers are making the stuff? Or, being tools of the owners of captial, are they incapable of making anything?Let me also say that I don't think Emmy/Emily has agency, although it doesn't seem like I use that term in the same way that you do. I'm perfectly happy with saying that she creates or composes things, and so on. In any event, you might want to reread what Cope said:
I don't know. An abacus, scratch paper, a shovel: they all seem poor analogies to me, since they just sit there. Emmy/Emily actively does stuff--you hit a button and she creates something original.Posted by: billy | Feb 25, 2010 2:16:54 AM
Jesse M,
I've tacked at essay length many of the questions you ask. Take a peek there first. I present many of the best explanations today of why strong AI is so moribund and what it will take for it to revive—but which requires surmounting hurdles that no one has any idea how to. I hope it will also make you question your enthusiasm for "mind uploading" mumbo-jumbo from folks like Kurzweil.
Thanks, Elatia.
Posted by: Namit | Feb 25, 2010 3:43:31 AM
Elatia.
Like that. Or like learning sea monkeys are just brine shrimp.
Billy,
Perhaps you should share your definition of agency, since I have no idea what you mean.
Posted by: Chris Schoen | Feb 25, 2010 10:30:49 AM
Wikipedia says "Human agency is the capacity for human beings to make choices and to impose those choices on the world." Along with this, I usually think of things like beliefs, desires, intentions. Emmy/Emily probably doesn't have these, but I wouldn't restrict agency to just humans.
I am curious what you think about the factory-line workers, if you'd describe them as making things or not. (Hopefully you can see why I find your position to be a bit silly: the exact same inputs, the exact same outputs, described differently depending on whether the thing that converted the inputs to outputs was a person or a machine. The person "created" something, but the machine didn't?)
Posted by: billy | Feb 25, 2010 11:43:44 AM
Namit, I had read that essay of yours, but I don't really see where it directly addresses any of the questions I asked you above (whether 'intentionality/agency' are qualitative subjective judgments that are all in the eye of the beholder or whether they are supposed to point to something more objective, whether you believe that computer programs could never in principle have intentionality/agency, and whether you believe simple organisms like insects have them).
You might also be interested in taking a look at this comments thread from a while back where I defended the idea of "mind uploading" at length...
Posted by: Jesse M. | Feb 25, 2010 2:23:04 PM
Billy,
Your argument employs a bait and switch over what a person is in a given context. (Though I don't think it's intentional).
At any given moment, a human factory worker and a machine may be functionally equivalent in terms of production. This is, indeed, the way the logic of Capitalism would have us view things. Put X amount of raw materials into a factory, and yield Y amount of goods, whether the factory is "staffed" by humans or machines.
Your analogy puts it exactly backwards. In the case of a factory, humans are the form of capital known as labor. They don't make things--they provide a resource for the owner to make things. This is why they get a wage and not a stake in the profits. It's not a question of machines acting like people, but of people acting like machines.
This isn't black and white of course; humans exercise some intentionality on the job. Much less so in the factory than in the office suite, of course. But for the purposes of acquiring capital, these instances are factored out, either by writing off the inefficiency, by instituting Tayorist techniques (today we call them "six sigma"), or by moving to a more automated process.
If we want to consider humans not in the capacity of their function as workers but as fully human agents, we have to say that at any given moment a human factory worker could refuse to perform a task; take a break; choose to speed up its work or slow it down; flirt with a co-workers; check his text messages...
If we talked about a robot in those terms, we would be talking about the robot as a person. But such agency has not come up in reference to a robot in this thread. We all agree a computer (at the present technological moment) cannot act as an agent, and deserves no praise for a job "well done" or opprobrium for a job done poorly.
The fact that this has alarming implications for life in a capitalist system is another conversation, but it doesn't alter the main terms. This has been Jaron Lanier's critique of the Turing Test for many years: not being able to distinguish between a person and a machine might mean the machine has become intelligent; or it might mean the human has become mechanized. Most of the "success" of AI to date obtains from the latter possibility.
Posted by: Chris Schoen | Feb 25, 2010 2:46:28 PM
Compare: If a forger could paint a Vermeer every bit as good as the originals, then what was so special about Vermeer?
We're used to the idea that sometimes even experts might have to rely on forensic techniques like chemical tests, etc, to determine if a painting or objet d'art is a forgery. Even though it is unsettling to think that the ordinary ways we apprehend visual art aren't enough to determine whether a work is "real", our categories of "real" and "fake" still seem clear. I don't think they would be challenged if someone created a cyber-forger app to knock off Vermeers. If the creator of the cyber-forger went around saying that Vermeer was just a manipulator of pigment, nothing very special, would s/he be taken seriously?
Is there something different about music as an art form that makes the corollary more objectionable? I don't know.
Billy:
"the exact same inputs, the exact same outputs, described differently depending on whether the thing that converted the inputs to outputs was a person or a machine"
The "thing" that created the outputs was the factory or assembly line system. Typically, the system works by reducing the choice of each individual on the line to a simple binary - tighten that rivet, or withhold labor altogether. By replacing the workers with robotic controls, the factory owners reduce that binary choice to zero.
Posted by: Vicki Baker | Feb 25, 2010 3:15:34 PM
If not a factory worker, Chris, what about my dad, who's a self-employed carpenter? Would you say that he constructs (or makes, creates) buildings? Or is he just a tool of the architect--or the tool of the person who hired him and the architect?
Or better yet, what about a musical composer, who's commissioned to write an opera? Would you say that he's "creating" the music? Or is the one handing out the cash doing the creating?
Posted by: billy | Feb 25, 2010 3:25:59 PM
Vicki,
I love the Vermeer example. Suppose someone tried to exactly replicate one of his works, then yes, I wouldn't consider the forgery artistically equal to the original, or indeed exceptional at all - given a few tens of millions *and* access to the original, I imagine a machine could squirt paint in the same way as any given painting, to few-micron precision.
Suppose however you came across an entirely new painting, one reminiscent of his style, and whose authorship you are uncertain of. It may have been Vermeer, or maybe it was some no-name contemporary of his. I imagine you don't want to call this work more or less meritorious depending merely upon *who* squirted ink at it. I realize precisely this thing happens in appraisals, but to me that says, roughly, that more than quality enters such (monetary) assessment. But if we've agreed on this - that the work ennobles its creator and not vice-versa - it seems perverse to me call an original work art if made by a person, but rubbish otherwise.
Chris,
"the cyborg is Cope, and Emily is his bionic leg. But that's no way to look at making music, as we know from our agreement that neither Cope nor Emily have produced anything of artistic value"
I've said I don't think this Emmy makes anything reaching these lofty heights, but I don't think the output is garbage either. At least I'd be moderately impressed with music student or entertainer could do the thing. I also reject your notion that the cleverness of the algorithm is exclusively that of its maker. (Anyone who enjoys chess should examine this game, and note that none of the programmers responsible - and possibly no human being at all - would begin to be capable of its aesthetic effect.) As such, I'll fail to carry out my part of this "agreement" :)
Posted by: prasad | Feb 25, 2010 6:41:56 PM
Re: the Vermeer example - I didn't make it up:
Fake Vermeers
In this case, the forger tried to pass the work of as newly discovered works of Vermeer - they weren't simply copies.
But if he hadn't tried to fool Hermann Goering and various art experts, if he had presented the paintings as his own work in the style of Vermeer, would anyone have cared? I don't think so, but maybe Elatia can shed further light.
Posted by: Vicki Baker | Feb 25, 2010 7:11:39 PM
Billy, when Emily starts complaining to Cope about how he takes her for granted and treats her like some kind of mindless robot he can just switch on and off, maybe you'll have a point.
Posted by: Vicki Baker | Feb 25, 2010 11:27:37 PM
Vicki:
Another hypothetical. Suppose I get my nephew a lego set for his birthday, and after a few hours he comes back to me, grinning broadly, saying, "Look what I made!" Should I pat him on the head and say, "No, actually you didn't make anything. You only followed the directions--the ones who designed the instructions are the ones who actually made it. And me, since I paid for it."
It seems to me to be a complete distortion of what a normal person means by the words "make" or "create". Do I need to go to the dictionary?
Nowhere in those definitions is there any mention of, say, complaining about being taken for granted. That's just not relevant to whether thing A created or made thing B.I probably should've learned my lesson by now, to get out of discussions before they drown themselves in silliness. I find myself arguing against positions that are absurd, trying to illustrate their absurdity--and somehow failing! Anyway, the point I was heading towards is that knocking Emmy/Emily down a few (or a ton) of pegs to the level of an abacus, also knocks Bach and Mozart down to that same level (since she produces works indistinguishable from theirs). I'm not a huge fan of classical music, but that seems to me to be a terrible thing to do. Instead we should acknowledge that she's doing some very cool and interesting stuff, and although she's not human and may not have agency, she's much more than scratch paper or a shovel.
Posted by: billy | Feb 26, 2010 2:33:44 AM
(since she produces works indistinguishable from theirs)
No, there is a very big distinction. Bach and Mozart and the other composers Cope has emulated with his computer did not set out to plausibly be taken for someone else. They wrote music that was influenced by preceding forms, but which adhered to, in the end, their own internal aesthetic symbolic analysis and intuition, which is to say their own judgement. To deny this is to deny everything that is great about great art.
Emily has no faculty for such analysis, owing to her lack of sentience and intentionality. Cope is her judge. He decided (after many failures) when the emulations were good enough to potentially fool the educated listener. Emily did not say to him, perhaps we should wait a bit, I haven't got the hang of this yet, because she has no capacity for aesthetic evaluation.
But even then, even fooling music critics at a classical music taste test, as "cool and interesting" as these results are, they are not art; they are facsimiles.
(Prasad, this goes to your comment too. There is a long drop between art and "garbage." I never said that Cope's work with the Emily computer wasn't of interest or social value. I just said it has no *artistic* value.)
Your (Billy) continued functionalist comparisons rely on a radical behaviorism that is completely out of joint with the way we view the world morally and aesthetically. You are welcome to keep insisting that human cognition is just an aggregate of gradations of computational complexity from shovel on up to Stephen Hawking, but without a significant role for mind along the way this is just empty talk.
Posted by: Chris Schoen | Feb 26, 2010 10:44:52 AM
There's nothing to debate: imitation is not art. And it's not even a good imitation from what I heard. That computers can beat humans at chess just illustrates how much music is superior to a game. That's why millions of people would rather listen to Mozart's music than analyze chess moves. If I were religious I would certainly say that Mozart's music has a "spiritual" dimension, but, as I am not, I will repeat what I wrote in another post: When you have Mozart, who needs a god?
Posted by: J. Hawkins | Feb 26, 2010 11:09:14 AM
Billy, I have no problem with saying a product is "machine-made" or that a certain process inside an oyster "creates" a pearl. Nor would Chris, I imagine.
You're fetishizing words and ignoring meanings.
Posted by: Vicki Baker | Feb 26, 2010 11:41:47 AM
I didn't say that Emmy/Emily creates music in the same way that Mozart and Bach do, only that she creates music that's indistinguishable from theirs. Of course, Mozart and Bach don't really know how they create music either--much of our mental processing is at unconscious levels and is only experienced as "aesthetic judgment" or some such.
I also think your distinction between art and non-art is weird, but I won't say more, as I'd only be repeating what prasad's already said.
From your first comment, Chris:
I took this to mean that you thought his work was uninteresting...Posted by: billy | Feb 26, 2010 11:42:56 AM
Vicki:
That's a refreshing dose of sanity, but are you so sure about Chris? I've asked him about a half dozen times and all his answers seem to point in the other direction. If he does think that machines and programs are able to "make" and "create" things, then I wish he'd just say so directly.Posted by: billy | Feb 26, 2010 12:15:12 PM
Jesse,
I read a fair bit of that very interesting thread and now have a sense of where you're coming from. I should say first that for that exchange, you and Chris should get some kind of award, something like "Dogged Persistence Award" (are you reading, Abbas?) but then Chris has done it more than a few times on 3QD, much to the enrichment of this forum. I should say next that I stand in the opposite isle from you on the potential of strong AI and mind-uploading, largely on philosophical grounds, which are the only grounds you have too (that all our mental activity is amenable to computational reduction, that reductionism is possible not just in a physical sense but also for things like qualia and know-how, that functionalism is true, that mind and body are separable, that implementation will be feasible, etc.). But pardon me, I digress.
I can guess where you are going. Let's say that my view that I have agency is my qualitative subjective judgment. I also believe that there is no way for a designer to objectively show agency in a computer (Searle's Chinese room). But then on what grounds can I deny the computer agency if it emulates the behavior of a being with agency? Is agency like porn—I know it when I see it? Simple emulations like Emily apart, these are difficult philosophical questions that I've not yet answered to my satisfaction.
Posted by: Namit | Feb 26, 2010 12:44:10 PM
"are you so sure about Chris? I've asked him about a half dozen times "
Remind me again why it's important for Chris to say that a process inside an oyster "creates" a pearl. It's a word fetish on your part, which is interfering with your ability to understand his argument. Maybe Chris deserves sanction for enabling your word fetish. On the other hand, he has the sanction of 3QD to participate as a commenter and structure his argument as he sees fit.
Posted by: Vicki Baker | Feb 26, 2010 12:57:11 PM
Vicki: It's important in the context of dehumanization that I talked about a few messages ago.
Posted by: billy | Feb 26, 2010 1:18:55 PM
Billy: I don't understand what you're trying to say then.
Posted by: Vicki Baker | Feb 26, 2010 1:38:51 PM
If you're not willing to say that a computer create anything, that it's nothing more than an abacus, and yet it somehow "outputs" works of art that are just as good as some of the greatest (by some accounts) artists who ever lived, then what does that say about the worth of those artists, or art generally? Or if a factory worker (or carpenter, or composer, or small child) doesn't make things, then what's the worth of their job? It seems to me that this is a big problem.
The story that Chris (and you?) is selling, is that the important part about art is how it's created. This seems to me to be completely wrong. Whenever I see a painting or listen to music, the important thing is how it's affecting me--sometimes it's what the artist/composer wanted me to feel, other times not. But the whole experience is between me and the piece of art, and I judge its worth based on the things I take from it. It really doesn't matter whether the artist is male or female, or if they were high when they created it, or when and where they were born--apart from how those characteristics express themselves in the piece. And so it seems to me that it also shouldn't matter whether the artist was human or not.
Posted by: billy | Feb 26, 2010 3:06:00 PM
Vicki Baker wrote:
Remind me again why it's important for Chris to say that a process inside an oyster "creates" a pearl. It's a word fetish on your part, which is interfering with your ability to understand his argument.
But what is his argument, then? It seems to involve devaluing what Emmy does in some way (look at his original comment that it is 'sad' that Emmy's compositions will have 'many people believing that this computer did something interesting'), but it's not clear how--if the argument is just that what she does isn't "art" because Emmy has no intentionality, that seems like word fetishism too! A mountain or a tree can be very beautiful without having been produced by an intentional process, so if a programmer can invent some new type of non-intentional process that also produces novel creations with genuine aesthetic value, for me it doesn't make sense to say that this should be devalued as "sad" or not "interesting" just because it doesn't meet the definition of the word "art".
It also doesn't make sense to say that the outputs of such a process are "really" just the creation of the programmer, as Chris suggested in that first comment with "Cope seems to have hit upon a way to not have to take responsibility for his compositions". I would say Cope doesn't deserve responsibility for the specific compositions that Emmy creates, even though he obviously deserves responsibility for her general architecture. The idea that non-intentional processes (such as the evolutionary processes which created the form of trees) can be responsible for the "creation" of things with aesthetic value is not just word fetishism IMO, it gets to the heart of the question of whether there can be anything independently interesting about the output of a computer program beyond what it tells us about the intentions and creativity of the programmer. This was why I brought up the analogy of the Mandelbrot set in my second comment in this thread, since humans only had to come up with the idea of plotting the results of a very simple recursive equation, but doing so resulted in a vast array of aesthetically fascinating forms (reminiscent of many natural forms like those seen in trees) that the humans never remotely anticipated when they came up with the basic equation.
Posted by: Jesse M. | Feb 26, 2010 3:18:21 PM
It would be far more than I hoped possible from this thread to agree with everyone that we indicate something different by the word "made" when we say that an oyster made a pearl, a falling coconut made a splash in the lagoon, Holly made a big impression on me, and Mozart made "The Magic Flute." I can enter my weekend with a lighter heart, knowing this.
Billy quotes me as saying:
and he responds:
I don't. Cope's work is interesting. In fact, "fascinating," as I started out saying in my first comment.
I should not have given you this impression. What I said was that how art is created is integral to how we understand it (else a forgery is every bit as good as the original), but not that that's the only important fact about it. I also never said that we need to feel what the author intended us to feel.
But there needs to be an author, and that author needs to have intended *something*. This is a bare minimum for a definition of art. As a general principle I will say I don't necessarily care if the author is human or not. If a paintbrush can be shown to start painting of its own volition, or a piano starts playing itself, I'll stand up and applaud with the rest of you. Forgive my radical skepticism in the meanwhile about these things coming to pass, and not just because of the third law of thermodynamics.
Either way, I'll stick to my luddite, elitist yardstick on this: no intention, no art. Maybe we get something beautiful, like Elatia's lollipop sticks, but until a Duchamp, Cage or Warhol recontextualizes them as an artwork (by moving them from the world of extension to the world of meaning) they fail the test, by my lights.
Posted by: Chris Schoen | Feb 26, 2010 4:49:36 PM
Billy - With your examples of the factory worker, the carpenter, the child, and the composer - its important to me that they experience their own agency. It would make a difference to me, and presumably to the workers, if the factory was worker-owned, and the workers themselves decided to invest in robotic controls and allow some of the workforce to choose to be pensioned off or transition to new work roles. It would make a difference to me whether the carpenter was a debt slave or not. I would care whether the child was making legos for the joy of playing with legos, or just to please me. I would care if the composer was Shostakovich being commissioned to write a suite during the height of the Stalinist purges.
With the factory example in particularly, you're eliding all kinds of psychological and sociological facts by insisting that it's just like other forms of human "making".
I'm sure Namit or someone more knowledgeable would correct me, but wouldn't it be theoretically possible to assemble and train a team of human calculators to perform the tasks that Emily performs? In other words, I can make a flow chart diagram of Emily's algorithm, and break down all the steps to their finest granularity, and assign those tasks to humans. Given enough humans and enough time, they would come up with a new work of music. In what way is this analogous to what is usually thought of as art or the creative process?
Posted by: Vicki Baker | Feb 26, 2010 6:05:18 PM
I don't get it, Chris. Cope's work is interesting, but the stuff that his computer did is uninteresting?
Well, that's seemingly very generous of you. Of course the problem is that you deny that it's ever possible for a computer program to have "intention" or "volition", because it would only being following its instructions, with no choices along the way, right? (As if our neurons could choose to disobey physics.)
It's not analogous, Vicki--but at the same time, we really don't know how our creative process works. A lot of it takes place below consciousness, and only bubbles up as vague feelings or thoughts or instincts. If we were exposed to all the inner workings of the brain, and imagined ourselves as manually sending the electrochemical messages from one neuron to the next, I think we'd probably also say that that was far removed from our normal experience of the creative process.Posted by: billy | Feb 26, 2010 6:30:17 PM
"at the same time, we really don't know how our creative process works."
Absolutely. I don't think Chris' requirement for intentionality implies complete conscious control.
Posted by: Vicki Baker | Feb 26, 2010 6:50:28 PM
I don't get it, Chris. Cope's work is interesting, but the stuff that his computer did is uninteresting?
I said that people should not believe the computer "did something interesting." This goes back to your failure to distinguish shades of meaning in multivalent use of the same word. It's moving the goal posts to say that because the computer colloquially "did" something, that this doing is of the same nature as a sentient being doing something like making a piece of art.
Of course the problem is that you deny that it's ever possible for a computer program to have "intention" or "volition", because it would only being following its instructions, with no choices along the way, right?
That's not what I said either. I said we don't have reason to ascribe intention or volition or agency to the computer right now. (You agreed Emily had none of the three). What future computers / shovels / abacuses do is a different story.
Posted by: Chris Schoen | Feb 26, 2010 7:02:23 PM
Hi Namit, thanks for taking the time to wade through some of that long comments thread! You wrote:
I should say next that I stand in the opposite isle from you on the potential of strong AI and mind-uploading, largely on philosophical grounds, which are the only grounds you have too (that all our mental activity is amenable to computational reduction, that reductionism is possible not just in a physical sense but also for things like qualia and know-how, that functionalism is true, that mind and body are separable, that implementation will be feasible, etc.).
Even if you don't agree with all of these, I'd be interested in knowing more about where your breaking point would be (which is sort of what my questions were getting at)--for example, when you summarize my position as "reductionism is possible not just in a physical sense but also for things like qualia", do you find it plausible that reductionism might be possible in a physical sense but not for qualia, i.e. the idea of a philosophical zombie? And if you are inclined to doubt that reductionism is valid even in a physical sense, then does that imply some sort of soul-stuff or élan vital influencing the behavior of particles in the brain in a top-down fashion, so that local interactions between a given pair of molecules would not obey exactly the same laws as the same local interactions happening in a nonliving substance?
Are you familiar with the philosopher David Chalmers, by the way? If not, I recommend his book The Conscious Mind for a lot of interesting thoughts on issues related to reductionism and qualia. He basically takes it as read that the great success of the reductionist scientific paradigm, including modern neuroscience, suggests that it is almost certainly true that human behavior associated with consciousness can in principle be accounted for in a reductionist way (and in general he thinks that the physical world is 'causally closed' so all physical events can in principle be explained in terms of physical causes). But he still thinks there is a "hard problem" associated with first-person aspects of consciousness like qualia, since qualia seem to be logically distinct from third-person phenomena like behavior and computation; he argues that philosophical zombies seem to be at least logically possible, at least if you accept the existence of a physical world separate from the mental world of qualia (which might be rejected in the type of panpsychist metaphysics I discuss below).
At the same time, he also has some interesting arguments suggesting that functionalism is probably true in the broader sense that there'd be a lawlike relationship between physical phenomena and qualia--what he calls "psychophysical laws"--and that one important aspect of these laws would be that different instances of the same computation would be associated with the same qualia, regardless of the physical medium doing the computing. One of his main arguments for this involves the thought-experiment of gradually replacing all the neurons in a brain with functionally identical but materially different replacements, and arguing that that it seems implausible to suggest that the brain's qualia would fade or change as more and more of it was replaced, since behaviorally the individual whose brain was being replaced would act as though he wasn't experiencing any changes. See his Absent Qualia, Fading Qualia, Dancing Qualia paper for more on this line of argument.
The idea of a causally closed "physical world" which gives rise to mental events in a totally passive way may seem overly dualistic, and it makes it questionable how we can coherently talk about qualia since the qualia are not imagined to have any causal influence on our behavior, including speech. (maybe you'd have to adopt a view like Leinbiz's pre-established harmony between the mental and physical realms?) One way out of this is to question whether the notion of purely "physical" events makes sense at all--perhaps instead we could adopt a type of naturalistic panpsychism where all so-called physical events are associated with qualia of some basic type, even if not nearly as "rich" as the qualia in the brain of a biological organism. Then physical laws could be understood as a type of lawlike relationship between different qualia--just as my predictable desire for coffee in the morning can be viewed "from the outside" as a causal relationship (even by me!) as well as "from the inside" as a collection of qualia relating to that desire, so our observations of even the simplest of particle interactions might be an analogous "outside" view of what are really some very elementary types of qualia. Chalmers tentatively suggests that this sort of panpsychist view may be the most elegant way of resolving some of the philosophical problems he brings up, though he doesn't outright endorse it. Personally I wouldn't say that I have any totally convincing arguments for naturalistic panpsychism that could convince a skeptic, but it does make a lot of sense to me and I basically feel like it's the solution most favored by Occam's razor.
So basically, although in some sense it's true that I endorse the view that "reductionism is possible not just in a physical sense but also for things like qualia"--I do think that an uploaded mind that was nearly functionally identical to the original brain would have near-identical qualia--in another sense I don't go as far as others in thinking it's not even meaningful to talk about qualia as something distinct from behavior or phyiscal processes in the brain (eliminative materialsm). These are tricky issues and so I'll understand if you don't have the time to get into your own position on philosophy-of-mind questions, but if you do feel like elaborating on what your position is and what was implied by your comment that you stand "on the opposite aisle" from me, I'd certainly be interested to hear more of your own thoughts on these matters.
Posted by: Jesse M. | Feb 26, 2010 7:07:23 PM
Chris Schoen:
I said that people should not believe the computer "did something interesting." This goes back to your failure to distinguish shades of meaning in multivalent use of the same word. It's moving the goal posts to say that because the computer colloquially "did" something, that this doing is of the same nature as a sentient being doing something like making a piece of art.
But did anyone actually argue that what the computer did was "of the same nature as a sentient being doing something like making a piece of art"? Personally I would say the computer did something interesting in the same sense that the evolutionary process on Earth did something interesting by creating trees and other aesthetically appealing life forms (though of course Emmy's productions are not nearly as interesting).
Also, Vicki's last comment brings up an interesting issue: the degree to which a human's "sentience" is involved in the creation of various aesthetically interesting actions may not be an all-or-nothing affair, especially if the brain works in a somewhat "modular" way. If we watch a slowed-down movie of an athlete engaged in some motion like throwing a discus, the detailed sequence of motions may be beautiful in a way, but isn't it plausible they a result of mostly unconscious brain-processing that the self-aware sentient mind contributes very little to? In contrast I think self-awareness is pretty essential to something like the creation of stories about imaginary (or real) characters. But music is abstract enough that I can buy that music-creation is somewhere in between these two extremes...for instance, I wouldn't be too surprised to learn of a case where a highly autistic individual with no ability to communicate and little to no apparent self-awareness might still be able to come up with some pretty interesting and original music.
Posted by: Jesse M. | Feb 26, 2010 7:26:27 PM
"If we watch a slowed-down movie of an athlete engaged in some motion like throwing a discus, the detailed sequence of motions may be beautiful in a way, but isn't it plausible they a result of mostly unconscious brain-processing that the self-aware sentient mind contributes very little to?"
In this example, the art lies in slowing down the images and considering them aesthetically.
Posted by: Vicki Baker | Feb 26, 2010 7:49:51 PM
What is your meaning of the word "did"? I'm at a complete loss. And I'm still unsure whether you think the computer did interesting stuff or uninteresting stuff. Or maybe you think it didn't "did something" at all?
I was under the impression that your reason for denying intention and volition for Emmy/Emily was that all it was doing was following instructions. If that's the case (is it?), then that logic would force you to conclude that no future computer program could ever possibly have intention or volition. Right?
Posted by: billy | Feb 26, 2010 8:06:01 PM
"In this example, the art lies in slowing down the images and considering them aesthetically."
But none of the credit goes to the athlete? What if it was a dancer? I guess my point is that it's not all-or-nothing, that there's a fuzzy boundary between processes in the brain that produce beauty but are mostly outside of conscious control or awareness, and brain processes that draw more one one's status as a self-aware individual capable of conscious decision-making. And to take a sort of Buddhist perspective on this stuff, I think the ego tends to take far more credit for things the brain and body do than it really deserves!
Posted by: Jesse M. | Feb 26, 2010 8:33:55 PM
"But none of the credit goes to the athlete?"
Obviously I feel that they should at a minimum be a willing participant - but what makes you think the athlete and the videographer have to be different people?
Posted by: Vicki Baker | Feb 26, 2010 8:54:41 PM
"Obviously I feel that they should at a minimum be a willing participant - but what makes you think the athlete and the videographer have to be different people?"
They don't have to be, but making them different will make your position on what it means to create "art" more clear. Do you think there is a sharp dividing line between brain processes that produce beautiful results but are too unconscious to really qualify as the creation of art, and processes that are sufficiently conscious/self-aware? If a great idea for a melody pops into a composer's head all at once (or the composer hears it in a dream, like Coleridge claimed was how he came up with 'Kubla Khan'), is it possible that the composer might not deserve any artistic credit since the processes responsible for the melody seem to have been outside conscious control?
Posted by: Jesse M. | Feb 26, 2010 9:29:25 PM
Jesse M. - A dream came to Coleridge unbidden. He made it into a poem. Same thing could happen in any art. I could rely on random processes at every step in the creation of the work, and it would still be art. I would happily go to Elatia's exhibition of popsicle sticks arranged by her popsicle stick pop-o-matic, especially if she was cooking.
To clarify further:
To put one brick upon another,
Add a third and then a fourth,
Leaves no time to wonder whether
What you do has any worth.
But to sit with bricks around you
While the winds of heaven bawl
Weighing what you should or can do,
Leaves no doubt of it at all.
--Philip Larkin
Posted by: Vicki Baker | Feb 26, 2010 11:23:44 PM
"A dream came to Coleridge unbidden. He made it into a poem."
I think what he claimed is that the exact words of the poem came to him unbidden in the opium dream, he didn't have to do anything to "make it into a poem" afterwards besides writing it down word-for-word...and unfortunately for posterity, he also said the dream-poem was much longer, but as he was writing it down a knocking at the door interrupted him and he forgot the rest! This page has the details:
According to his own account, COLERIDGE had the feeling that during his sleep "he could not have composed less than from two to three hundred lines..." and that "all the images rose up before him as things, with a parallel production of the corresponding expressions, without any sensation or consciousness of effort. On awakening he appeared to himself to have a distinct recollection of the whole, and instantly wrote down the lines... At this moment he was unfortunately called out by a person on business" , which interrupted his recollection of ideas.
Maybe this isn't what really happened, but assume for the moment it is--are you arguing that the poem produced by his dreaming mind was not itself "art" since it involved no conscious thought or effort, but only became art by the act of his making the decision to write it down word-for-word? If so this seems like a sort of fetishizing of the conscious ego over all other parts of the mind...like I said, I prefer a more "Buddhist" approach which doesn't put the ego-self up on a pedestal of its own making. Perhaps this relates in a way to this interesting-looking book about the right and left brain which I saw discussed on 3quarksdaily a while ago and should really get around to reading one of these days...I think the left brain may have a habit of taking credit for the right brain's work, and of seeing its own more self-conscious activity as being what's "really important"!
Posted by: Jesse M. | Feb 27, 2010 12:42:52 AM
Jesse M., Vicki, Chris and others,
Several times that I'm aware of, a highly autistic person has come up with very interesting visual art. How do I mean that? Well, the art would have to be interesting even if the artist were not an autist (groan!). I am thinking in particular of an autistic child of the 1970s named Nadia, who drew magnificently, bafflingly, and who never grew up as an artist, because her autism was treated in such a way that her gift disappeared. She never became neurotypical, for her deficits were no sham, but as she approached more orthodox subnormalcy, she ceased to draw. I have always been sad that this child's story belongs more to neurology than to art.
There are others who have become professionally active as artists, despite the communication difficulties and other cognitive deficits of autism -- Oliver Sacks has written about one such artist. But their art is considered in light of their autism, and that speaks more interestingly to neuroscience than to art.
I think part of what makes us recognize a work of art as art is knowing that it is the result of a struggle, that it is the result of being human, which is a struggle. Art comes from the full depth of human experience, beauty can and does come from anywhere. Whether a painting is hard won seems to matter to us, and we are moved when we see a manuscript with hundreds of changes scribbled in. Reading a Beethoven score is astonishing -- the changes he made, to get it right. This is how we recognize each other, along the carnage strewn path -- "all those moments," as the most famous humanoid said, the one who made us think of what being human entailed. A musical composition machine ought not to be able to take it away, and we should resist. If we don't, we are not better than machines ourselves.
Posted by: Elatia Harris | Feb 27, 2010 1:58:06 AM
Well said, Elatia. I saw your comment as I was nearly done with mine, so I'll just post anyway. Call it a one-two punch at those who claim Emily's work is art. :)
We're not going to settle whether Emily produced "art" or not without looking hard at what we mean by "art" itself. Chris attempted this but didn't go far enough I think, because he said: "As a general principle I will say I don't necessarily care if the author is human or not." I disagree with this. I think our idea of art profoundly assumes that it has been produced by a sentient being very much like us, that it is an expression, however lame or silly, of what it means to be human. It is not important that the work was a product of our conscious mind alone; just that it was produced by someone like us. It is not important that it was produced under duress, or by following a recipe, or motivated by a dream; only that the author was someone like us, and we can imagine trading places with her and doing the same thing.
When we admire a sunset, it is not just our conscious part that plays a role, but also our subconscious and unconscious parts. When we look at a sunset painting, we implicitly assume that the artist was also capable of harboring similar mental activity, both when she looked at a sunset and when she painted it. The very notion that a computer achieved such a painting is profoundly alienating, however good the emulation and however strong its intentionality and other technical jargon. Art requires that it be created by someone we can empathize with by virtue of our shared humanity, even if it is an autistic humanity. No amount of agency in a computer will lead to its output being recognized by us as art, because we cannot relate to the computer as we relate to a human. Call it a fundamental attribute of our minds. What machines make will never be art to us, at least not until we come to think of machines as entirely on par with humans.
Posted by: Namit | Feb 27, 2010 3:25:50 AM
I don't count myself as a big art-lover, so it could be that I really don't understand what it's all about. But I would like clarification, Elatia, if you don't mind.
So you know how every piece of artwork that's in your home was created? Or when you walk through a museum, the pieces don't affect you until you know everything that the artists struggled with in order to create them?You've seen the handwritten early drafts of every novel that you've enjoyed?
You need to know the history of every composer before you can really appreciate their work?
I'm not saying that you're not "right", only that it's not what I usually understand art to be.
Posted by: billy | Feb 27, 2010 5:47:26 AM
Elatia,
Re struggle, hardship and carnage, I had the same reaction Billy did, but let me pose the question differently. I'm not much for medieval art, but do you reject it wholesale? Why not? Surely you've essentially no evidence for the artist struggling through hundreds of drafts, his tears and such.
Fine, you can imagine them. But then consider the great epics, holding in mind Jesse's evolution-type arguments. Everything I know and believe tells me that the Mahabharata was composed over hundreds of years over a broad geographical area by thousands (at least! And surely you must also add in audience responses as the story was settling into equilibrium, of excitement and joy or disappointment and irritation depending on the nature of novelty or performance...). There's little reason to think fifteen thousand *geniuses* worked on the Mahabharata, and little likelihood they all had heroic tears to shed. Do I have to believe blind artist fantasies to consider the Mahabharata or the Iliad "art"? Do your words commit you to the view that in every case of national epic there must have been one or five or fifteen Great Sufferers to do the actual work?
Me, I think epic literature bears at least a passing resemblance to Vicki's fantasy of tons of people following algorithmic procedures to replicate the operation of Emmy - given a reasonably interesting people interested in hearing and telling and repeating stories, over a period of time some often excellent artistic work emerges from their combined, incremental, decentralized efforts, mostly minus creative weeping...
Posted by: prasad | Feb 27, 2010 9:56:22 AM
Aren't we all forgetting the Jesus factor?
You know Bach and that religion thing.
Didn't the Psychopathic Space Daddy help him out? Isn't the Psychopathic One just like a virus infected computer?
Posted by: Dave Ranning | Feb 27, 2010 12:00:06 PM
Wow, I do have answers for you, Prasad and Billy. But I'm going to be offline for the next 10 to 12 hours -- see you at midnight. And, thank you, Namit, I'll get back to you too.
Posted by: Elatia Harris | Feb 27, 2010 12:29:21 PM
Adding to my previous comment. I think we can all agree that behind every human act lies a continuum of mental activity, from the conscious to the unconscious (as Jesse pointed out). It is also clear that some of this mental activity can be modeled adequately well by an algorithm, but some of it not yet and likely never. This is where software has helped penetrate our algorithmic mental work (the 'low hanging fruit'). For example, a human doing basic arithmetic, or searching for a word on a page, or following a recipe to make a cake. Similarly, aspects of group behavior also lend themselves to algorithmic representation. For example, queuing theory can model people going through the DMV. As it turns out some aspects of making music may also yield to algorithmic analysis (whether or not a human does so consciously), and others not (developing a new style, improvisation, factoring in a new instrument, folk music, etc.).
But there is a lot about being human that is not, and may never be, amenable to algorithmic analysis. Can a machine be made to understand the fear of death or existential anxiety? Or can it emulate what it means to be an Indian sister to an Indian boy? I think the worth of AI is not that it is on the road to making humanoids indistinguishable from humans, but that it is revealing that some of our basic mental work may have algorithmic components. If AI can package that in software (for example, a search engine), in a sense it can allow us to concentrate on parts that remain non-algorithmic. I think we can all see that software has had just this kind of an impact in our lives.
Again, the point of my previous comment is key to this debate, that for us to call something "art" profoundly assumes that it was created by someone (individual or as a group) who we think relates to the phenomenal world as we do.
Posted by: Namit | Feb 27, 2010 1:01:21 PM
I could get used to having people speculate on how I might respond rather than having to come with it on my own. But I do want to clarify something Namit wrote:
No argument from me. I was making allowances for the theoretical possibility of an android, upon Billy's repeated insistence that I was ruling it out a priori. But I agree that such an android would need to be nearly indistinguishable from a biological human in sharing the human condition: the biological sensorium, sentience, loss, knowledge of death, which is the main reason behind my skepticism.
Elatia,
You have the Mahabbarata well in hand? I would like to tender my resignation on this thread, maybe hand over my resonses to a bot. Namit is right that the only place this conversation can go is deeper into theory of art and aesthetics. I have thoughts on same but I tremble and quake at how arduous it will be just agreeing on fundamental premises.
Prasad,
My kneejerk response to your question about 15,000 sufferers, multiple authors of the Odyssey or 1,001 Nights, etc. is that these revisions were done artistically, not mechanically. It would be a reductio to envision Vicki's example of millions of people acting out the functional equivalent of one neuron each. Does anyone want to play historian, and talk about the oidoi? Yes, authorship in the modern sense (Homer did it) is often a crude myth, but let's not let our speculation about how the great epics arose lead us to potted theories about hive minds.
Posted by: Chris Schoen | Feb 27, 2010 3:44:53 PM
Chris: Why can't you ever answer my questions?
Posted by: billy | Feb 27, 2010 4:35:47 PM
I have an uneducated question.
Why is the focus here solely on the agency/intention of the creator in defining art? What about the consumer/ beholder? Is it not enough if he or she has a "human and artistic" reaction to the product, no matter how it was created or produced? Why should that not figure in defining art? If a mechanically produced landscape, portrait or piece of music evokes in my mind a sunset, my mother's face or the roaring of the sea, is that not "art" enough as far as I am concerned?
Several times I have seen comments threads where the sleazy side of an artist or author's life comes up for examination. We are usually told that we should be able to ignore that in valuing their artistic or poetic output because art transcends our human failings. Yet now I am being told to take into consideration the sufferings, tears and epiphanies of the creator in evaluating his / her art. So which one is it? We are the sum total of all our emotions - the base and the sublime. Who is to say which ones hone our artistic genius?
Elatia's inside knowledge of art and artists for example, is vastly superior to mine. But that is not enough to accurately predict which one of us is going to have the more authentic artistic experience when confronted by a masterpiece or tawdry piece of kitsch.
Unless we can accept art as a two way street where spontaneous and even untrained human reactions on the other side of the equation are just as valid as the motivation and experience (or the lack of it, as in the case of machines) of the creator, it is then not about aesthetics at all but high minded gate keeping by mavens to an exclusive club.
Posted by: Ruchira | Feb 27, 2010 4:57:04 PM
Jeez Chris, I said 'resemblance'! That people are intelligent agents hasn't escaped my notice. If you see no resemblance at all, shrug. I didn't discuss simulating neurons or solving Schrodinger's equation or whatever because Billy did that already.
I *will* say that a pretty respectable fraction of these "artistic revisions" in Mahabharata or Odyssey or Icelandic saga writing required from the many, many people making them over time, no greater skill, insight or giftedness than is needed to tell a decent bedtime story, or to distinguish a good advertisement from a bad one (supposing for a moment this is the level of artistic ability most people possess). You can tell me if you wish that none of the great epics were in significant measure composed and passed-on descent-with-modification style by large groups of people. I think it's a pretty challenging undertaking, but fine. You can't tell me though those large groups of people were all, or even mostly, chock full of artistic ability and special insight into the human condition. If tons of people did it, the implication that mediocrities did it follows.
In any case, the point was to question the fetish of consciously-planning-and-thinking-and-working-and-suffering-singular-author in place here, because plenty of things it sounds positively mad not to call art lie outside those constraints. The question then becomes why computer art is so suspiciously easy to brand 'non-art', when in general that's such a hard decision to make, and people are sufficiently inclusive to let in blank canvases, John Cage music and the like. It's especially weird considering its suddenness, and just how much grief Denis Dutton got for calling virtuosity integral to art, not that far back. Now suddenly the artist and his birthing pains are back at the center of things.
Anyhow, I'm pretty done too.
Posted by: prasad | Feb 27, 2010 5:00:11 PM
Geez, prasad, I said "kneejerk"! :)
I'm not trying to foist a genius paradigm on anybody, where the genius of a work must be correlative to the genius of the composer. This goes to Ruchira's question of the receiver of the work (which I did touch on earlier.) Greatness can be discovered in places where it was not intended, and much of literary criticism is occupied with this task.
But I think it's safe to suggest at least in a nominal way that even "unintended" meaning can be postulated in many cases to a subconscious grasp of latent meaning that the author wasn't fully conscious of. The best artists rely on their usage of forms having a certain pluripotency that could never be fully plumbed, otherwise they would be trafficking in cardboard allegory of the type we were talking about in the thread on American Beauty. An artist with too much control over her material is not liable to create anything of enduring value. So I agree with you that intentionality in art can be over-argued.
I think I am partly to blame for over-reliance on the word "intention" as though the author could intend everything that happened in his work. I'm really talking about a much more indeterminate thing, a great component of which is receptivity and attention. (I think I discussed this earlier too, but it's a long thread, and I don't think Carlos-the-historian is tracking it).
Posted by: Chris Schoen | Feb 27, 2010 6:24:45 PM
"We're not going to settle whether Emily produced "art" or not without looking hard at what we mean by "art" itself. "
Yes, and Walter Benjamin has gone there before us. The means of production and reproduction of art changes its role in a culture.
The very notion that a computer achieved such a painting is profoundly alienating...
And profound alienation has been a goal of some artists at least since Brecht, which makes me think that the Cope's essential artistic act was to list "Emily Howell" as the composer's name for these pieces.
So is the chemistry professor who first fell in love with Emily's work, and then repudiated it, a character in a Hoffmanesque tragedy, as Eliatia hints, or a poseur, exposed?
Posted by: Vicki Baker | Feb 27, 2010 9:31:34 PM
Ruchira: "Unless we can accept art as a two way street where spontaneous and even untrained human reactions on the other side of the equation are just as valid as the motivation and experience (or the lack of it, as in the case of machines) of the creator"
I agree that Billy's response, that it's still beautiful, seems more genuine than the professor's. But still, I understand the shock of finding out that your favorite singer uses Autotune, or that a favorite painting was a forgery. What you see as gatekeeping I see as the common modern anxiety about what is real vs. fake.
I have more to say about the Mahabharata and "das Volk dichtet", but my family claims that y'all are not "real people" and I should pay attention to them and not the computer.
Posted by: Vicki Baker | Feb 27, 2010 10:00:50 PM
"... a Hoffmanesque tragedy, as Elatia hints, or a poseur, exposed?"
Vicki, I think it's divine comedy!
Jesse,
Thanks for your response and the links. Much food for thought. A considered response will require additional reading and more time than I have for the next 2 weeks. In fact, Pinker's testimonial for this site really applies to me right now, "I couldn't tear myself away from 3 Quarks Daily, to the point of neglecting my work." So I'll defer it to a more opportune time and another thread.
Posted by: Namit | Feb 27, 2010 10:47:48 PM
It seems to me that David Cope IS the computer. Which is to say that to some extent we are all cyborgs now, and Cope more so than most.
So it appears that Cope has used his exceptional musical ability to create these musical copies, as though a composer had spent his life saying 'Look! I can write just like Mozart, or Beethoven, or whoever, and writing compositions devoid of context which prove it. And in order to do so he has become a cyborg, making use of superior computational power and speed to accomplish the thing, rather than holding it in his head and furiously scribbling notes. It is a bit of a so-what. Composition students are routinely set exercises of composing in this or that style. They have to derive some sort of rules to do so. Cope has managed to formalise some of the rules, and encode them in a computer. There will also have to be some pseudorandom component, or it will all sound too predictable. I don't know how many iterations of the program it takes to produce a really good piece 'in the style of...' but presumably there's some selection. The speedy computer can presumably pop these pieces out fairly swiftly.
What we have is, essentially, an algorithm for doing what a second-year composition class might do, and picking the best. I think that's pretty neat, but not all that impressive
Posted by: Mike Cope | Feb 28, 2010 4:11:15 AM
I would welcome the opportunity to do a blind test, as I feel pretty sure I would be able to recognize which is the computer-generated music most of the time. How well I ( or anyone else) could distinguish the two would also depend on how long a sample we were given. The longer the sample, the more obvious becomes the difference, the more obvious becomes the lack of meaningful thematic development in the computer-generated score. I don't want to come off like a jerk, but I have noticed that a lot of people seem to enjoy music, but they don't really hear it very acutely or experience it very deeply. It may be that for them, Emily Howell's compositions would be a reasonable substitute for Bach or Mozart. Based on what I heard, I thought the stylistic mimicry was impressive, but there is no comparison.
By analogy, I am fully aware that I don't "get" art. In terms of abstract expression, for example, you can put up something that is apparently execrable or done at random by a robot and as long as the colors please me, I'll like it and won't be able to tell if it's "good" or not or whether it was done by a trained artist or not. I simply do not have a sensitivity that allows me to distinguish the two, and I realize that. It doesn't mean I don't ever see a painting that I like, but I am aware that my appreciation for it is dim and limited.
Posted by: MoZeu | Feb 28, 2010 10:54:53 AM
A bump for Elatia.
In my senior year of high school, I was in a class called "Modern English" where we read and discussed mostly short stories. I enjoyed the class a lot, trying to figure out all the symbolism and whatnot. One day we took a field trip to hear one of the authors give a talk. I forget who it was, or what the story was about, but the thing that I remember was in the question-and-answer phase, when one girl asked about the meaning of a certain passage. It was about a mountain, and looking up at birds, or something like that, but it was something that I'd spent quite a bit of time thinking about myself. The author replied (rather dismissively) that there wasn't any symbolism there at all, that it was just a mountain, just looking up at birds--you know, one of those "a cigar is just a cigar" things. Most of the audience (since school's uncool, or for other reasons) cheered at this. I thought it was just stupid, and that she was lying (in order to seem cooler). But maybe she wasn't.
Anyway, that's my story about "latent meaning".
Posted by: billy | Mar 1, 2010 12:14:43 PM
Another bump for Elatia.
Posted by: billy | Mar 4, 2010 8:46:05 AM
My last attempt to get that woman's attention.
Posted by: billy | Mar 6, 2010 9:03:07 AM
Post a comment