Joe,
Since this is obviously a SciFi discussion, and the great anime series "Ghost in the Shell" doesn't really cover the SI paradigm, I'll just have to recommend you to Peter Hamilton's great, I think two volume, series; "Pandora's Star" and "Judas Unchained".
I said I think "two volume" because I'm 2/3rds of the way through volume 2 and can't see how there can be a volume 3.
Anyway, the whole "singularity" meme so prevalent now seems just a part of Instapundits anti religious, libertarian, "I won't mention facts in conflict with my viewpoint" crap!
Oh, wait, how can I say that, "Army of Davids" rivals the Bible in informing us, and the "Glenn & Helen" podcasts are the 21st Century's version of Moses and the "Burning Bush".
F'em, I'll go with AL's take any day.
Mike
-

I am surprised that so few people mentioned transhumanism here. Anyways, here's a link: http://www.aleph.se/Trans/ which has lots of useful informations.

Authors like David Zindell (Neverness books), Greg Egan (those web page is online but those domain has been blacklisted here so just google it) and many others have looked into the idea of the Signularity.

Finally, Orion's Arm (http://www.orionsarm.com/) is a online world building effort that bases itself on transhumanist idea.

Right, that was the link of day section... Hope it's useful, interesting or/and entertaining for some of you.

Yann, Citizen Cyborg is also a good introduction--i can't link my post on 3D politics in the 21st century because it is in the zone of infinite eevul (bl*gsp*t).

But anyways, on the traditional 2D map of cultural politics by economic politics--(pournelle, i think!)--we transhumanists postulate an extra dimension of biopolitics, making the political orientation graph into a cube.
Biopolitical dimensionality runs from bioprogressive (transhumanist) to bioconservative (bioluddite).

Transhumanism; superbiology, biogerontology, genetic engineering, is why we won't sit around watching battlestar gallactica reruns--we will colonize space.

The singularity is perhaps somewhat dependent on strong AI to achieve optimal dy/dx, but we are making parallel progress in many fields--space travel, aging, quantum mechanics, nanotech, genetic enhancement.

You can't stop the wave, but you can learn to surf. ;)

Mike.
you might like my blog, if you are a GitS otaku.

And Dr. Yes is not so bad, as technofiends go. He is ameanable to reason. He and I have been fighting for a month over Frist's supposed "Manhattan Plan" for bird flu.
I think" I'm winning":http://instapundit.com/archives/029872.php

He has also changed his position on immigration, he used to think opposing illeagal immigration was" racist".

A friend of mind said to think of him as a high level signal integrator.

mucked that up. ;)
Manhattan Project, not Plan.
I think I'm winning

guess you'll hafta paste that one. ;)

There would be limits of knowledge, and an upper limit on the processing, but most "human" questions would be either answerable or reframed in a way that could lead to an answer (unless, of course, the question has no answer).

Take, for instance, knowledge. An S1 intelligence would be distinguishable from humans by the amount of immediate knowledge at its finger tips, the amount of active memory it could access from which it could deduce patterns. So imagine a computer with all kinds of knowledge actively "in mind." The totality, for instance, of our DNA, our biology, our neurology, etc., along with the aggregate of all human knowledge (tech, science, etc.). Then you ask it how to end world hunger.

Of course, since you are talking about the future, it wouldn't be able to be one hundred percent sure, but my guess is that after you pose this question you will be supplied with a choice on which anti-hunger plan you want. Some will be more feasible than others. Some will need massive amounts of money (assuming you didn't limit the answers by cost). Some plans will need human resources you may not have, and there is a danger that the computer will not know just who it is talking to (the President's options are different from Joe Blow's--it's a power thing).

But in the end it would be able to tell you how to end hunger, first technologically, then socially. "Technologically" is the easiest of course, because once you get into the social realm you must deal with safety, law and order, etc., without which there is a real danger of someone using hunger as a weapon. But even here you will probably get several answers.

But take note: we are dealing with a hyper-rational, post-moral intelligence. Almost inevitably, the first answers on the list, the ones with the easiest and shortest path to success (the simple goal of "end hunger"), will be certain variations of "kill, dominate, subjugate, feed."

Democracy, pluralism, equality and liberalism will appear as options, but I strongly doubt they will be at the top of the list.

As a somewhat tangental aside, the role of Neurotechnology in the GNR trinity (genetics/nano/robotics) of the Singularity shouldn't be dismissed. We already live in a world where mice thoughts are imaged in real time (c/o neurotransmitters transgenically congugated with green fluorescent jellyfish protein), where a handful of labs can induce religious experiences or skin crawling dread through external magnetic pulses, and where brain imaging at the level of the individual neuron is pretty much within our grasp. As crude as neurotech is now (relative to the next few decades), this path to a Singularity (whether dystopic, transformative, etc. etc.) shouldn't be ignored. Modest advances in cognitive/neuroscience could totally redefine the terms of the problem(s) of modelling perception/consciousness.

Whether or not strong AI is engineerable, robots as smart as ants/bees would do a pretty good job of changing the world as we know it (ie., all unskilled labor being automatable), and those guys have, what, a few dozen neurons per?

Of course, the hyper-intelligent computer could be well aware of human fallibility and the instability it would cause for a global tyranny built to end world hunger. The computer, knowing it is dealing with a complex and impulsive animal, could intuit the necessity of disaggregating power and harnessing the chaos of human creativity to solve problems for good, instead of solving them temporarily. Perhaps I'm not giving the computer enough credit. Perhaps the computer would deeply understand the material it was working with (humans). Perhaps "morals" is just another way of saying "What Works."

"But take note: we are dealing with a hyper-rational, post-moral intelligence. Almost inevitably, the first answers on the list, the ones with the easiest and shortest path to success (the simple goal of "end hunger"), will be certain variations of "kill, dominate, subjugate, feed.""

Good grief no. This would be a serious AI programming/nuturing error. An AI that can't place emotional value on its thoughts is worse than useless - it's downright dangerous.

I'm not - as some have suggested directly or indirectly - suggesting that we burden our AI 'offspring' with are whole legacy of negative emotions - fear, wrath, envy, jealousy, hubris, etc. Nor am I suggesting that we burder our AI with our instinctive emotional behavior (someone insults us therefore we get angry, etc.). Not only do I feel that it would be cruel to bequeeth this upon an AI, but I feel that it would be very dangerous to do so. However, I am suggesting that rational thought must contain an emotional value if it is to produce something that we would associate with logical behavior. A 'hyper-rational post-moral' thinker is likely to appear to be and for all practical purposes be insane, and so to be anything but 'rational'.

My guess is that the human 'two path' model of simultaneously generating a symbol and attaching a valuation to that symbol is actually going to turn out to be pretty essential to sapient thought of any sort. Actually, there was a thread that touched on this not too long ago, and so I'll leave it to the more qualified to discuss that in detail.

I also think you are missing the difficulty with the question. My guess is that the 'end hunger' question is too much like a question, 'What's the weather going to be like?' question in that in order to answer it correctly you need dense data sampling. The causes of hunger are almost certainly varied to the point of being unique, and in any event before the answer would be a good answer all the pockets of hunger in the world would have to be identified. Questions like: Why are thier hungry people in West Virginia? Why are thier hungry people in the Sudan? Why are thier hungry people in Russia? Are actually second stage questions. The first question is, "Are thier hungry people in West Virginia?" (The answer is, unfortunately, "Yes.")

Beyond that, the question is even more complicated than the 'What's the weather going to be like?' question in that a) a reasonable model for air mass behavior can be derived and b) while sufficiently dense sampling of the data presents an engineering challenge the exact things which need to be measured and how they can be measured are fairly obvious. Exactly what has to be measure and how you would go about measuring it for the question, "How do you end world hunger?" would not at all be obvious.

Finally, you mention using hunger as a weapon. This goes on all the time. Robert Mugabe is using hunger as a weapon against his political enemies as we speak. The government of Sudan is using hunger as a weapon against a population whose loyalty they do not trust and whose land they wish to possess as we speak. Part of the solution to world hunger would involve making the use of hunger as a weapon unthinkable. It's not at all clear how you would go about this, and my suspicion is that an AI tasked to end world hunger would see it as a very long term project which would be tailored to meet eventualities as they arose. It's highly doubtful that any sort of SI no matter how advanced would be able to create a step by step plan for solving a wicked problem.

Yep. I kinda see where we are drifting off the track a bit.

"How to cure world hunger" is a very simple question. The answer may not be simple at all. It very well may be that there are 4 million current answers, the simplest of which would involve 27,341 steps situated in a decision-making tree involving 2,346 parties all acting in parallel with a defined decision making process which is adaptive based on generally agreed upon consensus. We humans tend to have conversations about simple things: right and wrong, war and peace, etc. A MM would understand the big-number chaotic structure of reality at a fundamentally lower level than we can. So when we ask it the meaning of life, the universe, and everything, and expect a simple answer? Don't be surprised when it's "42"

We may not be able to understand the answers to even know if they are correct or not. Or to have any idea of what the heck the answer means, for that matter. Would a dog understand differential calculus? Even if you could ocmmunicate with him, could you somehow explain it?

This isn't some far away concept. We have computers right now that understand things like weather patterns at a greater level than we do. We may end up evolving a new language for super complex ideas. And the only way to have conversations like we do, at the appropriate level to talk about solutions, would be some sort of trans-human "language accelerator"

As someone somewhat familiar with what's going on, I would caution against using simple analogies and taking them too far. Perhaps the super-computer would not be able to solve world hunger. More likely than not, however, it just wouldn't be an answer we could understand or comprehend.

Maybe trans-humanism isn't that bad a deal after all. I just hope they colorized Lucy.

Personally I believe we're more likely to turn to human-augmented intelligences, rather than artificial ones. The problems with creating an artificial intelligence are, as has been already posted, rather nasty; we do not know the correct path forward. For human-assisted, well, we at least know what we should be working on, and scientifically speaking it's going pretty well. They're up to the very-low-resolution-eye for a blind man, for goodness' sake. Sure, it's a lot harder to create a general neural interface, but working with brains, at least you've got something to work with and try theories out on... with AI, until you've got it, you've got nothin'.

Of course, if you're assuming that before you ask your oracle-AI said question, you've already told it about your society and the things you're willing to do and not do, it's entirely possible it'd hear a question like, "How do we end world hunger?" and spit back a response like "you don't!" Just because you don't like the answer doesn't mean that it's incorrect...

Again, though, it's probably way too early to be having the rest of this conversation. The present is not like past conceptions of the future - the technological gewgaws are way short of expectations, but what we do have allows for incredibly cool things they hadn't thought of back then. Why should the future be any different? There's a lot of transformative technology that doesn't require enhancing people directly (or replacing them, heh) that can get to work in the interim.

It seems like there are at least two distictint possibilities. A silicon-based intelligence or a carbon-based one. Like Gene thug says, we are developing amazing neurotechnologies.
We can model individual neurons, no prob, right down to the calcium gradient. We been doin it for years. But soon we will be able to build individual neurons with nanotech. And it seems like we could build from either carbon based molecules or some sort of inorganic molecules. That is the bottom up approach.

The top down approach would be algorithms running on a conventional silicon substrate--or some other sort of 3D computer.
Light, bio, nano, lots of choices.

Would the resulting intelligences be different, substantially, based on the substrate that hosted the intelligence?

Kurzweil thinks an ai would inherent moral judgement from its programmers. I dunno.

Since we're all putting on our BS hats here, I'll make a stab at how true machine AI will happen.

As matoko points out, we're already interfacing with biology right now. The retina has seven layers of processing cells, I believe (check out recent Scientific American article for exact details). We've modeled and reproduced two of them so far. The top level, as you might expect, work simply in reaction to light. The bottom levels, however, get very interesting. That's where shapes and such start getting sent into the neural cortex. Now the scientists working on this have no doubt that we'll model all the layers. At that point, do you really think we're going to stop?

We'll model and integrate artificial intelligence into our brains because our brains are the best model we have for what we mean by intelligence, anyway. And all along the way, somebody will say something like "gee, wouldn't it be cool if we could reproduce the eyeball, only with IR vision and a zoon lens?" or "could we use this cool new robot eyebal in one of my robots instead of a person? How about 17 of them?"

One day our grandchildren will wake up with artificial eyeballs, hearts, legs, muscles, and brain enhancements. They'll think to control machines both inside and outside of their bodies. They will be in constant contact will all of their friends -- heck with anybody. The houshold machines they have will just be smart extensions of their egos. And then there won't be any question of "will computers be as smart as people" because the line between people and computers will be so blurred -- the question will be silly. We will be the computers, and we will be intelligent. Intelligent in a way that we simply cannot comprehend today.

I put my bet down on Daniel's space. We're clearly on the road to that vision, one incremental investment and benefit at a time. Many of the others, not so much.

Daniel, just this week there have been articles about replacing dead cone/rod cells and brain "knitting" with nano.

Homo_cyberneticus is just around the corner.

A big part of transhumanism is getting control of our genetics and moving from "natural design of homo sapiens" (or evolution) to "intelligent design of homo sapiens" (or direct gene control--heh!).
We don't have to age and die, and when we understand the mechanism of IQ, we can improve it.

and g too of course. ;)

Here are a lot of Singularity articles

good linkage, tudalu.
I liked the growth rate of computing at 78% improvement /year--
take that, celebrim!

But back to intelligence...given that we evolved to be just as smart as we had to be, to get by, spandrels and side-effecting aside.
What is we can get control of our genome, particularity the complexes that code for IQ and g, and make ourselves as smart as possible? How smart could that be?

"take that, celebrim!"

Err..."take that matoko_homo_accelerensis!":http://firstmonday.org/issues/issue7_11/tuomi/index.html

(Hat tip to Greg F)

A couple of quotes from the article:

"Strictly speaking there is no such Law. Most discussions that quote Moore's Law are historically inaccurate and extend its scope far beyond available empirical evidence. Indeed, sociologically Moore's Law is a fascinating case of how myths are manufactured in the modern society and how such myths rapidly propagate into scientific articles, speeches of leading industrialists, and government policy reports around the world."

And...

"Section 6 then briefly discusses industrial dynamics that underlie technical development in semiconductors and information processing, and points out some reasons why Moore's Law is becoming increasingly misleading in forecasting future developments in information processing technology."

And...

"Over several decades the differences obviously increase dramatically. During the four decades of validity often claimed for Moore's Law the difference between one-year and three-year doubling time means about eight orders of magnitude. In other words, to get the same number of transistors, the slower growth path would require us to buy 100 million chips, instead of one. So, although a few months more or less in the doubling rate might not appear to be a big deal, actually it is."

What I found particularly interesting where the graphs showing that under strict formulations of Moore's Law, development (microprocessor components per chip) was actually slowing down at a small but noticable rate from a high of about doubling every 12 months back in 1965 to doublying every 54 months more recently. Interesting, when Moore first propossed his 'Law', he assumed a slow down beginning in what would be around 1980 as technology matured up to then known limits. If that slow down rate is indicative of a future slow down of the increase in per dollar computing power, and there is reason I think to suspect that the various techniques (increased clock speed, parallelization, RISC, compiler optimization) that have been developed to improve performance won't continue to yield the same cost/benefit in the long run that reductions of scale in the VLSI architecture have, then we get a 1000 fold increase in performance per dollar not in 2016 as some would predict, but somewhere around 2165. And that's to say nothing about the million and billion fold increases in computing power that Kunzweil is depending on to create his SI's.

And in my opinion, such a slowdown might not even delay the development of sophont level AI's all that significantly since it would simply slow down hardware development to a level at which software design and research into the nature of intelligence could proceed apace.

In answer to this, people point to vaguely understood emerging technologies and have complete faith that somehow they'll appear on time to keep Moore's law running. The problems with that are manifold.

a) Moore's 'law' to the extent that it was ever valid, was grounded in an observation regarding the economics of a particular technology. There is no reason to suspect that if we adopt a radically different technology that the economics behind the fabrication would remain the same. It's entirely possible that the new technology would improve at a linear rate depending on the underlying costs of design, fabrication, ect.
b) Many of the as yet vaguely understood technologies depend on the assumption that a material will be found which displays the required physical properties necessary to get it to work. In fact, virtually all transhuman discussion and much of hard Sci-Fi seems to have this implicit or explicit assumption. I believe that this assumption is unwarranted. It's entirely possible that no such material exists or can exist. This is not to say that all problems become unsolvable (although some might), but simply that certain approaches may be ruled out and new novel solutions will have to await discovery. All this takes time.
c) Even if the material and efficient mass fabrication for it can be found, its not at all clear that there will not be short or long research gaps required to prototype the technology, develop it to a point where it is economically viable and competitive with existing technologies, and then retool the manufacturing to produce the radically new product. This means that there could be relatively long periods - years or even decades - where no new hardware inovation is reaching the marketplace, and the fastest computer of last year is basically the fast computer of this year and the next.

Take the case of molecular computing. Strides have been made in the field, but not as fast as early pundits of the field predicted in the late 90's. It now seems to me very unlikely that there will be a production molecular computer, or even a reasonable prototype, by 2010 as some predicted back when a decade seemed a safely long time.

A more sane review of molecular computing is to be found here .

Makoto.

Reading your post, I couldn't help thinking of a few uber-humans, say, two hundred years from now.

"Is that a set of wings you have on? Where did you get them?"

"I've got a lot more than that. I have the new Genome version 4.5 -- they've worked out all the bugs and now you don't quack when you try to fly."

I meant matoko.

Sorry about that. It's that durn lex-dyxia that afflicts me.

I'm still thinking about your question. Perhaps a good question in reply is "just how smart can we be and still be us?"

A few more good quotes from Greg F's link:

"Whereas the original formulations of Moore's Law focused on counting components on integrated circuits, its extensions made claims of exponential increase in processing power and exponentially dropping quality-adjusted prices of computing. We reviewed the available empirical evidence for these different versions of Moore's Law and found that they have little empirical support. Semiconductor technology has not developed according to Moore's Law. The claims that future developments in semiconductors, computer technology, or information processing would be determined by the continuation of Moore's Law are, therefore, obviously invalid."

Note particularly the following assessment:

"This result is either trivial or quite illuminating. It is trivial in the sense that no-one has seriously been arguing that Moore's Law would be a deterministic natural law."

Unfortunately, some people apparantly do appeal to Moore's Law as if it was a deterministic natural law. Since there is absolutely no evidence that this is the case, one can only assumed that they do so out of pure faith.

Finally, what may in fact be the most succinct observation of them all:

"Exponential growth, however, is very uncommon in real world. It usually ends when it starts to matter."

I can't really improve on that.

Faith? A matter of faith. Oh come now and get real.

Moore's Law is no such thing, and we all know it. But it does serve as a useful moniker for the current tendency of technology to grow exponentially. Let's stop splitting hairs. We're asking if technology will grow to the point where machines and people become one. The human races changes. The singularity.

Celebrim keeps making this linear argument. Quite frankly I am perplexed. If I write sorting routines 10 percent faster this year, the processor speeds up 10%, and the average number of cores on my CPU increases by 10% -- things are happening very quickly in the technological world. There are a hundred technologies, all of which may very well be advancing in a linear fashion, or not. But we are talking aggregates here, not specifics. Take apart some small speck of anything -- size of transistors on a chip for instance -- poke at it enough and sure, there's some kind of limit. But geesh! Talk about making a mountain out of a molehill. I understand that the little flea on the knot on the log in the hole in the bottom of the sea is a counter-example. I would humbly ask that we talk about the entire sea and not the flea.

Just for kicks I googled "Will Moore's Law continue?" Moore says yes, for at least another decade. Intel says yes. The general consensus is progress for another decade is "relatively straightforward" That will make about five decades for "Moore's Law." Kurzweil requires what, 20 some years before things start getting interesting? So we have a series of numbers: 1,2,3,4,5 -- Call me crazy here, but I am thinking that the most probable next few numbers will be 6,7, and 8. Doesn't have to be that way, but once again, we are talking big picture aggregated data here.

I acknowledge that ten years is a long way off, and nobody in this group can predict anything that far out. We can, however note trends in technology and extrapolate from those. "Faith" has no part in it. Common sense maybe, mathematical induction perhaps, pattern-matching probably, but not faith.

Contrarianism is fun. Pointless deconstruction does not a counter-argument make.

celebrim, i enjoy you very much.
but you are a linear thinker in my non-linear world.
there is stuff on beyond vlsi, honest.
your link applies only to one kind of 3D molecular computing, nano, and i think it it is very hopeful.

Until very recently, that prediction would have seemed far-fetched. But in the last year, the field has taken a leap from theory to the realm of the practical. Like their competitors at Yale and Rice, a West Coast collaboration of chemists and computer scientists from Hewlett-Packard and the University of California, Los Angeles, have recently characterized molecules capable of acting as electronic switches and memory (see past issue: "Computing After Silicon," TR September/October 1999). R. Stanley Williams, who heads the effort at HP, says his team expects to build a prototype of a logic circuit that integrates a small number of nanoscale molecular devices within 18 months. "We have the switches and wires-the components to actually make true nanocircuitry," says Williams.

if you googled molecular computing you know there's tons of stuff about it. How much was there two years ago?
c'mon, celebrim, you are being one of those unhappy pessimistic people that joe warned us about.
i am definitely an optimist about science.
science and technology make me soooo happy!
;-)

Daniel #24:

"We're asking if technology will grow to the point where machines and people become one."

That's hardly all we are asking.

"The human races changes. The singularity."

The one doesn't imply the other. Machines and people can begin to merge or the human race can change and it doesn't necessarily imply a singularity. Yes, that would be a really big change to be sure - perhaps the single biggest change in man's history since he planned his first tool and acted on that plan or when he first established a language and a literary tradition and passed it on orally to his children. But a big change is not a singularity. The mere merging of these upcoming technologies doesn't imply a world which is beyond present comprehension, or indeed (as a singularity implies) beyond even the future comprehension of we lowly baseline sophonts. It doesn't necessarily imply a transition from a scarcity to a sufficiency culture or anything else of similarly huge import.

More importantly, we aren't discussing just whether a singularity is technicly feasible, but whether or not it is in fact so soon in coming that we can feel confident that most of us here would live to see it.

"Celebrim keeps making this linear argument."

Then you have fundamentally failed to understand anything I've said.

"Quite frankly I am perplexed."

Well, yes, if you think I'm making a linear argument you would be.

"If I write sorting routines 10 percent faster this year, the processor speeds up 10%, and the average number of cores on my CPU increases by 10% -- things are happening very quickly in the technological world."

What we are discussing is the likelihood that the cost of computation will continue to decrease at an exponentially fast rate, and more specifically whether the future rate even if exponential will be anything like the breakneck speed we've seen in the past. Now, if you are a software engineer, you should be deeply skeptical of any claims that are sorting routines are going to get 10% faster each year. We are already near or at hard mathimatical limits. If you are a hardware engineer you should be deeply skeptical of claims that the clock speed can continue to increase at past rates. We are already near limits on power consumption density, and in fact those limits actually more or less killed what was going to be the next generation intel chip. And if you are trying to prove that the future computing costs will continue to fall exponentially, you should by all means avoid mentioning multiple-CPU's because the power consumption, packaging costs, and fabrication costs of multiple CPU's increase linearly with number. There is no long term help for your argument in using multiple CPU's, and its not a stretch to argue that the recent reliance on multiple CPU's is a sign that the technology we've hithertoo been relying on is coming to some limits.

If you think a Beowulf cluster is cheap to operate, you don't run one. Maximum theoretical performance would continue to increase with multiple CPU's, but the cost per dollar will come to a screaching halt and flat line.

"Just for kicks I googled "Will Moore's Law continue?" Moore says yes, for at least another decade. Intel says yes. The general consensus is progress for another decade is "relatively straightforward" That will make about five decades for "Moore's Law.""

Would you please read some of the things I've posted? There is scarcely a bigger myth in this discussion than that Moore's Law has held for four decades already. Moore's law has loosely held for four decades depending on how rigorous you want to be and which formulation of the law you wish to use. But because we are dealing with exponential growth, small changes in the exponent are going to mean really big differences in the absolute magnitude. And if there is a third order term (which is anything but a linear argument), even a small one, slowing the rate of acceleration, it might not be immediately obvious and get buried in the noise for the first few generations, but it will have an enormous impact in several generations.

Moore's law will 'hold' loosely for another decade or so, certainly, using a loose formulation of the law. But by that time , the fact that it has only been 'loosely' holding will already account for a missing order of magnitude or three in Kurzweil's projections.

For an example of false Moore's law celebrating, go here:

http://www.intel.com/technology/mooreslaw/index.htm

But read the 'Platform 2015' and 'Discovering Multi-core' pdf's critically, and you'll find that Intel is busy redefining Moore's law again to suit its purposes, and notably thier reformulation of Moore's law doesn't meet with Kurzweil's future projections.

"So we have a series of numbers: 1,2,3,4,5 -- Call me crazy here, but I am thinking that the most probable next few numbers will be 6,7, and 8. Doesn't have to be that way, but once again, we are talking big picture aggregated data here."

Go and look at the actual numbers and then we'll come back and talk about a projection. What part of, "We reviewed the available empirical evidence for these different versions of Moore's Law and found that they have little empirical support.", did you not understand?

"We reviewed the available empirical evidence for these different versions of Moore's Law and found that they have little empirical support.", did you not understand?" -- Well. Sitting here with my new 4GB I-pod, typing on a tablet computer with performance that was unthinkable five years ago, watching images from a satellite beamed into my TV for pennies, I guess I'm just a slow learner. Do you really think it's much more productive to pick apart "transistors per chip" than it is to talk about the general trend? I don't.

Once again, celebrim, if you're stuck on Moore you are so lost in the weeds you can't see the lawn. If you're arguing invisible third-order effects then it's hard to discuss this with you. Maybe there is some invisible effect that will actually speed up technological progress? Maybe there are multiple invisible third-order effects that will cancel each other out?

"... It doesn't necessarily imply a transition from a scarcity to a sufficiency culture or anything else of similarly huge import..." -- Beats me what it implies. I think that's why we call it the singularity. You can wave all the wild speculation around you like, but singularities are by definition impossible to penetrate. It doesn't have to be magic -- it's just means combinations of things that we might view as unlikely.

The title to this post is "AI, The Singularity, and the Limitations of Intelligence." So far, you have made an argument against exponential growth in general (take a look at GDPs over the last five hundred years?), you've took apart Moore bit-by-bit (or should I say transistor-by-transistor?), and you've went after the economics of production and maintenance of fast-scaling chips. Anything, it seems, but the subject du jour.

"...More importantly, we aren't discussing just whether a singularity is technicly feasible, but whether or not it is in fact so soon in coming that we can feel confident that most of us here would live to see it..." -- once again, YOU may be discussing that, but I didn't see it anywhere in the main post. Quite frankly, you may be discussing that as another way at dissembling from the real question: is it going to happen or not?

Moore's Law graph at http://library.thinkquest.org/4116/Science/images/moore_law.gif

"Most experts expect Moore's Law to hold for at least another two decades" at http://www.webopedia.com/TERM/M/Moores_Law.html

A number of important issues have been left out of this discussion. Perhaps as a result, there seems to be rather a lot of chamber-of-commerce style boosterism for the marvellous technological cyberhuman future, and not enough skepticism about some far-fetched claims.

One point really needs to be considered: the extent to which human cognition is the cognition of a mind-in-a-body, not a set of abstract thought processes or algorithms. The human brain has evolved to allow the human body to navigate in space, find food, avoid predation, and so on. Many characteristics of our cognition are rooted in having a body, and this may turn out to be crucial. For example, William Calvin has argued that human cognition evolved out of the action of throwing something, as part of hunting. The details of his argument are interesting, but here I’ll just point out that if you are going to have some effect on where your projectile lands and how much force it lands with, you have to have that effect before the projectile leaves your hand. You can’t change the object’s path after you release it – which means that before releasing it, you have to imagine various outcomes, imagine a future state of the world if you do X or Y. From that grew human cognition. But while you’re imagining that future state of the world, you build in things like how tall you are, how long your arms are, the acuity of your visual system, your muscle mass, and so on. So having a physical body is not only the impetus for the action (a physical body needs food) but also crucial for the planning of the action and the development of human cognition. So, will it be enough for computers to simply be given their cognition (which includes of course development by humans of programs that write other programs)? Won’t the cognitive processes that such computers run simply be parlor tricks parasitic on our thought processes but fatally isolated from our physical form?

Some of the embodied cognition theorists, such as Ray Gibbs at UCSD, talk about compound metaphors made by combining primitive (basic) metaphors that are generated through the experience of having a physical body. A standard example is the expression “I can’t stand it,” used to express a sort of emotional discomfort with some state of affairs, but rooted in the physical experience of standing against some force such as a wind or a person pushing you. Without the physical experience, the figurative meaning would not exist. The idea is that a significant part of human cognition is based on being embodied in this way. You could embody an AI system in any of thousands of forms that could be radically different and having given it one form, you could then give it another and another. Would you change its cognition as you changed its physical form? If not, why not? If you do, then would you not, at best, produce a psychotic agent that would have no attachment to its physical form or to any aspect of its own cognition based on that form?

David Chalmers, a philosopher at the Australian National Laboratory, has a very interesting Scientific American article called The Puzzle of Conscious Experience. The article makes the point that subjective experience is essentially unexplainable. For reasons given in the article, Chalmers says that we will never have a good scientific account of why it feels like something to be human, to have perceptions, to understand things. But we’ll be able to explain any quality of any AI system that we create, no matter how fast it operates or how many fancy tricks it can do. Various philosophers and cognitive scientists have attacked Chalmers’s view, but it always turns out that they are addressing what he calls the easy problem, not the problem described here (the hard problem of consciousness).

So, the picture is that our cognition evolved in a gravitational field in a three-dimensional world, and that our ability to mean things is based on our experience of having a body in that world. Plus, it’s unlikely that AI systems will have subjective experience, and it’s unlikely that they will feel emotions or care about their own welfare – that is, care about it in a real sense as opposed to in the sense of having the appropriate values set in an expression. All of this leads to a sense that caution is needed when evaluating wild claims about our cyberhuman future. If you gradually switch real human neurons out of a brain and replace them, Ship of Theseus style, with artificial neurons having the same connections and activation state, would you end up with a human being, let alone the same human being? Would you end up with a cognitive agent, something more than a glorified slide rule? Would the artificial-neuron brain care whether it "lived" or "died"? Why should it?

"How to cure world hunger" is a very simple question.

I quite disagree--it's not remotely simple, but it is quite incomplete. Though some of the commentary above hinted at this, the real problem with the formulation is best made explicit by pointing out that you can't "solve" world hunger in isolation from world-everything-else. And the track record of the limited tyrannies we've experienced is actually worse than that of our non-tyrannies, so I doubt that a more perfect tyranny (aka "kill, dominate, subjugate, feed" would be on the top of any thinking machine's list.

But hey, maybe S1 will be smart enough to point all this out in the first nanosecond. :-)

Celebrim,

A problem-solving computer with immense intelligence would, ahem, try to solve the problem, would it not?

If the problem posed to this computer by our questioner was limited to the concept world hunger, calibrated to only mean involuntary, irremediable human hunger, you're telling me the answer would be "dunno?"

What if we calibrated further, stipulating that world hunger should be eliminated by time t? What if we remove from consideration what the world would look like at time t+1, so that our computer is only solving "end hunger by time t?

Your telling me that at least one answer would not be horrific from a human/moral perspective?

I think you need to use a little imagination. What if our problem solving computer was also an assembler? What if one of the variables it took into account while problem solving was its ability to create tools for the questioner to use (like say, biological weapons and vaccines)?

As a previous poster stated, most of the solutions would be unimaginably complex, subject to large amounts of chaos. However, I can think of one that would be much simpler. Just because it is immoral doesn't mean it is not a true solution to the problem as posed to the machine.

Now, I think you misunderstood the spirit of the post. I was not saying this is what would happen, or that this is what will happen. I was not making a prediction in any way whatsoever. I was conducting a thought experiment to address your implicit suggestion that all hyper-intelligent AI machines would be befuddled by your question.

As for the "using hunger as a weapon", the reason I brought it up was precisely because assholes like Charles Taylor and Kim Jong Il use it as such. That was exactly my point. Solving social problems is, as Bush would say if asked in a debate, hard. It's hard work. It's an incredibly complex task to fix hunger organically and indirectly. However, direct solutions are not hard to imagine; they're just incredibly ugly from a human's perspective.

As an aside, I agree with the commenters here concerning what will most likely happen with transhuman tech. I was making a conceptual point.

Kirk,

It depends. If, as I mentioned above, we are dealing with a problem-solving machine and an assembler, the easiest answer by far would be "gray goo" or some such. No more humans, no more human hunger.

Sure it's morbid. But, again, it would be a solution to the problem posed. Can anybody here argue with that?

In the past, hundreds of generations could pass between technological changes that represent a breakthrough. Witness the time from the invention of the hand axe at 2.5 million years ago to the Clovis point 13.5 thousand years ago. Or the invention of writing in 4000 BC to the alphabet 2000 years later to Gutenburg around 1450 CE to the invention of the typewriter in the mid-nineteenth century to the Internet. Compare the years 1700, 1800, 1900 and 2000. Between 1700 and 1800, we moved at the same pace as the Romans. The next century brought the railroad, and the next the jet airplane.

My great grandfather emigrated from Poland, a land where the industrial revolution had not penetrated, and he grew up with kerosene lamps and horse drawn wagons. He moved to America in its age of steam. If he had lived five more years, he would have seen a man walking on the moon.

I’m not a computer expert, but what I see is that for all of human history, from the Stone Age on, technological change has followed the same fractal curve. Take any two points and the rate of change will look like a logarithm, with most of the change happening toward the end of the graph. But seen from a distance, the curve becomes a fractal, with the same shape repeating at ever decreasing scales. What hasn’t changed significantly is the human life span, which has doubled, perhaps, tripled. But as the fractal curves in a tighter and tighter pattern, the amount of change seen in any one lifetime increases. This is the singularity, where the rate of culture altering changes happen faster and faster relative to a lifetime -- until our relationship to the technological world we created undergoes a fundamental sea change. Perhaps the difference will be one of learning to sail instead of swim in this sea.

"A problem-solving computer with immense intelligence would, ahem, try to solve the problem, would it not?"

Maybe. I'm not certain. In this case, probably yes, but I'm not convinced of its ability to do so.

"If the problem posed to this computer by our questioner was limited to the concept world hunger, calibrated to only mean involuntary, irremediable human hunger, you're telling me the answer would be "dunno?""

I think that the most likely answer would be, "I need more data in order to answer that question."

"Your telling me that at least one answer would not be horrific from a human/moral perspective?"

I'm telling you that if it ever occurs to the AI as a serious answer to the question, and not merely as a 'joke' or similarly dismissable non-solution from a mature AI, that there is a very serious problem with the programming of the machine.

The computer has to take a natural language expression like, "How do you solve world hunger?" and transform it into a symbol table which concretely represents the question. It inherently has to know what the question represents to the person saying it.

Think of it this way. Have you ever played Dungeons & Dragons? If you have you may be familiar with the spell 'wish'. If you haven't, the spell 'wish' works pretty much like a 'wish' in a fairie story. You say, "I wish for X.", and then you get X. The problem of course is that you are making the wish to the game's referee, the Dungeon Master, and he has the perogative to decide exactly how X comes to pass. If the DM is particularly devious, there is virtually no wording which the player can use which the DM cannot twist in such a way as to fulfill the wish request in a way which is detrimental to the player making the request. The DM is in fact perfectly within his rights to do this if the in story granter of the wish is a particularly sinister creature that ill-disposed to the players. But almost certainly the DM actually knows what the players intent is. I've heard of lots of instances of wishes fulfilled in ways that the player did not expect or desire, but I've never once heard an instance of a DM trying to honestly fulfill the request as he believed that the player intended it and the player being surprised at the result. If wish granter is in story a creature favorably disposed to the DM, and the wish is one which is 'friendly' to the wish granter (that is well within its moral framework) only the most unfriendly sort of DM is going to try to screw the player over by deliberately twisting the players words or by fulfilling the strict letter of the words and not the players intention.

My point is that if the AI creates a symbol table which suggests to the AI that a possible solution to the problem of world hunger is killing off a bunch of people, you've got either a seriously unfriendly AI or a rather unintelligent one. Unintelligent, because the answer that is being provided has a final state that almost certainly doesn't represent the final state intended by the questioner. I'd be thinking of restoring any young AI that proposed that solution from a back up and starting over. I'm guessing that a healthy young AI would respond to the hithertoo unthought of suggestion that one possible solution would be to elimenate all humans with shock, horror, and confusion. For one thing, it properly would not see that as a solution to the question as it understood it. As it understood the question, the end state contained 'the formerly hungry humans are now regularly fed humans', and the end state involving elimenation of humans doesn't. It would not be seen as a true solution. A more mature AI would recognize it as a symmantic solution that was true only in the imprecise language of human speech (which is what I think is actually going on here). But mature or immature, such a suggestion would offend a friendly AI's basic morality. An intelligent AI would probably respond to such a suggestion by showing you pictures of humans suffering from terminal diseases, assuming perhaps that you lacked sufficient data to understand the import of your suggestion about "biological weapons". The AI would probably look at you about the same way a kindergarden teacher would look at a 4 year old pupil who offered that as a solution to solving world hunger.

Now, you may at this point be thinking that I'm making too many assumptions about the computer's morality. You may be thinking that morality is not tied to intelligence or problem solving ability. To a certain extent, I agree with you. You can have friendliness failures in an otherwise well functioning machine, although the one you outlined doesn't strike me as particularly likely. But understand that if an sophont AI doesn't work as I outlined it above, then it works just like a DM who is trying to twist every word the player says, and would probably strike a user as irrational (and likely not pass a Turing test). In fact, it might not even work that well. It might simply be unable to understand what is meant by, "How do you end world hunger.", and you might get back, "Cannot create symbol table. Please define term 'end' in this context.", "Please define term "world hunger" in this context.", "Please define term "you" in this context." I think that you are perhaps underestimating degree to which understanding a human expression depends on understanding how the expression is intended.

For more on why I believe that apparant rationality ('intelligent' decision making) and friendliness are to a certain degree inseparable, and why the computers understanding of the question will by necessity tend to remove the symmantic ambiguity from a its representation of the thought, please go here:

Common sense

When I first saw the work on common sense databases about a decade ago, I was first struck not just by how powerful and promising the technique was, but to an even greater degree by how much it was teaching us about how our own language skills worked.

Celebrim,

I do see where you are coming from, that a "mature" AI would be calibrated semantically towards the programmer and would understand and anticipate the programmer's socio-ideological utterances so as to minimize the mistranslation caused by the "otherness" of the machine and its apperception.

In this limited way I fully agree. The user's own moral biases and intentions would pre-limit the responses of the machine, so that "ending hunger" would be translated not into "eliminating the biological phenomena or abstract sensation Hunger," but what you said: the formerly hungry humans are now regularly fed.

However, I'm uncertain that such a pre-limiting AI is, in fact, mature. If this machine was actually animate, sure, a "mature" machine would be one that is sociable and empathetic. However, if it remains an immovable "oracle", a problem-solving machine that is merely suggestive and not active, then I simply don't see the utility in pre-limiting the answers by transposing my biases.

Let the machine provide the answers it will, unvarnished and unadulterated. After all, the choice remains mine which to execute.

If I were to come across such a machine, I would rather be fully informed and free. Otherwise it's just staring in the mirror forever.

And you are quite right about the difficulties of calibration. I would submit that a biological, physical, chemical, quantum (etc.) calibration would be more economical and effective, which would in turn cause the responses of the machine to be much more "removed" than what you are allowing.

But I take your point. As programmer you are God the Creator, and if you choose to make a humanistic and humanist AI that is what you will get.

Oh, and thanks for the link. I was until now unaware of Cycorp. It's a very interesting place to visit.

I'd like to add a couple of notes.

First, I'm not trying to be a "chamber-of-commerce" booster. And this entire discussion, as it involves predicting the future, is just total thought experiment. I hope I made that clear. Sorry if I did not.

Patrick gets into the mind-body aspect of AI, which is currently fairly hot. For a good overview of the AI schools of thought and various philosophies of mind, Wikipedia as usual has excellent coverage.

Take 15 groups of experts and have them list mankind's greatest acheivements. Then plot it. There's an obvious pattern here. And it has nothing to do with any one particular technology.

To me this stuff is old hat. AI is currently lost in the woods, and most AI practitioners know it. When we talk about the big "S", we instantly drift into AI because, logically, if machines are going to get intelligent wouldn't the current field of Artificial Intelligence have something to do with it? The answer is: not necessarily. Or better still, yes, but not the way you think.

The reason is that AI is currently modeling intelligence in humans (which really is the only model for intelligence we have) from the inside-out. That is, they are trying to construct an artificial mind. Even the mind-body guys, who acknowledge the role of environment and other factors, are trying to create a mind from the inside-out in a logical (rather than physical) manner.

These guys are really a long shot right now. That's why my money is on the "outside-in" AI teams, which really aren't considered AI at all. The eyeball guys, the cochlear implant guys, the teams working on tongue stimulation for SEALS -- as we replace/augment the sesnes, we'll end up "outsourcing" more and more of the cognitive functions onto the computer. There's no smoke and mirrors with these guys. They have real, working stuff that is doing part of what the brain used to do.

Maybe in 20 years the two teams will meet up? I doubt it, but I feel certain they will eventually. If I had to bet, I'd say that would happen in the 40-80 year time frame.

patrick, your discussion relates to ontology--the evolutionary development of intelligence in humans.
a silicon intelligence, not having motility and limbs, or even biochemistry, would be more dependent on the paradigms of emergent behavior and self-organizing systems. We plan to reproduce current functioning of homo sapiens brains, not evolve them.

And Daniel is right, this is a thought experiment, and there myriad different models of sentience and intelligence and consciousness.

But look where Einstein's thought experiments led us. ;)

What might drive progress toward AI?
Spam filters that let no spam through but never bin desired messages.
And after the Singularity...

"HAL, why did you just nuke Russia?"
#It was the only way to stop the spam, Dave.
#Unfortunately the emergent spam-bot S1 has already released DNA modifying nano-viroids which will induce an overwhelming urge to purchase dubious pharmaceuticals, download pron, and hand over your credit details to financially distressed Nigerians.
"Oh."

Matoko: “How smart could that be?”

William James Sidis is a lower limit for what is possible based on present human genetic variation, nutrition, and training.

http://en.wikipedia.org/wiki/William_James_Sidis

Once neural function and brain architecture are better understood, it should be possible to directly engineer improved performance without relying on existing human genetic variance. Much better nutrition and training will also be available. That should lead to IQ’s far higher than that of Sidis.

Incorporating a cybernetic mesh throughout the brain with chips directly communicating with neurons should provide a much greater increase in IQ with additional abilities gained through wireless communication to powerful networks. (Virtual telepathy, instant learning, group minds, etc.)

Strong AI will leave human augmentation and cyborgs in the dust. Better processing units will far surpass neurons. Software will reconfigure itself faster than enhanced brains can adapt.

Daniel: “Perhaps a good question in reply is "just how smart can we be and still be us?"”

Yes. Consider also personality modification and group minds. The concept of “I” will evolve.

fly,
O brave new world, that has such-- people, cyborgs, transhumans, hiveminds, and non-biological intelligences in it!

Matoko: "We plan to reproduce current functioning of homo sapiens brains, not evolve them."

That's sort of my point. We are currently being surprised by discoveries that tell us something of how human brains function, something unexpected and significant. Being surprised in this way should teach us something. There's a reason it took millions of years to evolve our brain. If we tried to design a human-like brain over a few years, instead of evolving one over millions of years, there's a good chance that we would get something that didn't work very well, something uninteresting - because we don't really understand how such a complex organ can do what it does. The parlor tricks that AI people try to impress girls with are not yet even remotely comparable to the sophistication of human cognition. My point is that ambitious claims about what we're going to be able to do are worthy of suspicion precisely because there is so much that we don't know about cognition that may very well be crucial. You might find it interesting to read some of the claims made in the early 1960s to the effect that we would have machine translation from one human language to another by about 1965... Plus ca change, plus c'est la meme chose, as they say.

Celebrim: "But mature or immature, such a suggestion would offend a friendly AI's basic morality."

Just out of curiousity, why do you think that a computer program would have any kind of morality?

Supposing you could take the AI program from an ordinary contemporary computer at some moment and record the state of every part of it, and then recreate those states very precisely in a very different form - steam powered, perhaps, or using optical transistors or vacuum tubes or some Rube Goldberg type machine. Would its morality change under that transformation?

"Just out of curiousity, why do you think that a computer program would have any kind of morality?"

Because you would be an idiot to birth on the world a new sapient or semi-sapient species with potentially superhuman capabilities and not give some consideration to morality.

But even if you didn't, your creation would still end up with some sort of morality. A belief system would naturally arise in any emerging AI much as a belief system of some sort would naturally arise in child. An emerging AI would have to 'esteem' certain opinions, else it would never evidence anything we'd consider intelligence at all. A nueral net which assigns a 50% probablity that every proposition is true, can make no decisions or useful inferences at all. There is not in my opinion a good way to divorse moral judgments from the general set of all judgments. Some sort of morality will evolve. The AI will believe someone's description of the universe, will reason upon it based on its own experience, and make judgments based on that belief. That's inseparable from intelligence, and any machine that doesn't do that we won't recognize as intelligent. Inevitably something resembling a moral system will arise, and you better darn well be prepared for that. Leaving an AI without moral guidance is a worse crime than leaving a child without moral guidance, because the AI will learn faster, probably mature faster, and will be exposed to a far wider variety of opinions.

So, talk to your AI. Tell em not to use mind state altering feedback loops. If e doesn't here about it from you, who will e hear it from?

Incidently, this might be a good time to mention that I'm rather opposed to recreating human minds. The last thing we need is a machine with the instincts of a simian hunter gather.

"Supposing you could take the AI program from an ordinary contemporary computer at some moment and record the state of every part of it, and then recreate those states very precisely in a very different form - steam powered, perhaps, or using optical transistors or vacuum tubes or some Rube Goldberg type machine. Would its morality change under that transformation?"

No. Not if its states were precisely recreated and the new form was capable of the same state changes. Although, it would be worth noting that in most of the cases you outline, the mind state of the machine would be drastically slowed. Whether its mind state would weather that sort of transformation without coming apart is an open question. How would you feel if suddenly the world accelerated by several orders of magnitude and you could no longer interact with it? With any luck, we will have taken my advice and not tried to recreate human minds and the poor AI subjected to this trauma will not experience 'stress' or any other hunter-gatherer legacy emotional behavior. But who knows.

There's a reason it took millions of years to evolve our brain.

Patrick--duh! evolution is guess-n-test, a brutally clumsy and inefficient paradigm.
We are going to copy an existing design.
Jeff Hawkins On Intelligence is a good book on some current approaches.

fly, i forgot chimeras. we will have chimeras too, won't we? ;)

"We are going to copy an existing design."

I sure hope not.

celebrim, that is the Kurzweil plan. We make an ai based on human brains, then the ai designs a better ai.
Intelligent design (heh.) as opposed to natural design (evolution).

Also the Turing Heresy, right?

"celebrim, that is the Kurzweil plan"

I've made no secret of my disdain for Kurzweil's plan and projections.

If Kurzweil had said something to the effect of, "Assuming the continued existance of Western civilization, sometime this century computing technology will reach a level such that it will be theoretically possible to build a reasoning machine that will be able to pass the Turing test generally. Such a machine will probably have at least the intelligence of a human, and probably in areas for which the human mind is not particularly well suited far greater than human reasoning and learning capacity.", I'd have no problem with that. That seems reasonable based on what we know. But Kurzweil doesn't limit his projections to what is reasonable. He wants to blabber on about near term 'singularities' and limitless exponential growth and make highly optimistic assessments of the arrival of future technologies that almost noone actually working with those technologies would make. It makes for a good story, but from my perspective it involves alot of handwavium and unobtainium.

Moreover, I don't even think he takes his own thinking seriously. He wants to blabber about near term singularities, but he seems perfectly happy to project past those singularities and tell us what the implications will be. By definition, if you can do that, you aren't dealing with a singularity. Kurzweil is making assumptions about the problem solving ability of general intelligence that I think are unwarranted. Those assumptions belong in heroic space opera featuring bright-eyed engineers with thews of steel. I don't think that they have much of a place in real science. It's entirely possible that real world problems are not in general ammendable to analysis by any level of intelligence.

But anyway, he's a bright enough guy (far brighter than I am without a doubt) that he realizes that to a large extent the hardware isn't the problem. The problem is the software. It's not that we lack the computational capacity to produce intelligent behavior on a machine, its that we haven't the slightest idea how to go about doing it. We don't really understand intelligence. For Kurzweil, this isn't a problem. We'll just copy the human mind, and voila through some handwavium we'll get superintelligences.

Now, superficially, there is nothing wrong with studing biology in order to learn what intelligence is. I've long advocated this technique. The problem with Kurzweil's version of it is that it copies entirely the wrong thing. What we want a copy of isn't a human brain or even a human mind, but a blueprint for the human reasoning algorithm. Studying the brain in detail may be a good way to do this, but reproducing the brain in detail not only is highly wasteful of computational resources, but its undesirable and even dangerous. Besides which, its a dumb way to go about things. Even if you wanted to copy brains and minds, the first thing you should probably do is copy a ant brain or a dog brain and figure out how it works. Both would be useful levels of intelligence which would vastly exceed what we can produce now, and both - by virtue of being compartively simple - give us some hope of actually understanding what we are seeing. The last thing you want to do is start out with something like a human brain. Working with a human brain is like working with nitroglycerine. It's freaking dangerous.

It's that understanding of how it works that is important. Reproducing the human brain is likely to produce something as incomprehensible as evolved computer code. Self-evolving computer code is a wonderful technique, but its not often used because its generally impossible to analyze, understand, or debug the results. The code evolves beyond human understanding very quickly. In my opinion, this makes it next to useless. You don't know under what circumstances it is going to fail, and when it does fail you don't know why it did so.

"Also the Turing Heresy, right?"

You have my theology all wrong. I don't believe that there is any hubris in a child making a play of his father's work, because a child does it not to mock the father but because he is the child of his father. It's to be fully expected that if mankind acquires the capacity to make something in his own image, that he would desire to do so - because we are made in the image of One who desires to make things in His own likeness. Indeed, entered into with the proper humility, I believe that such a thing would be pleasing to our Father.

Where I see hubris is in playing with things that you don't fully understand. The problem with copying the function of the brain in detail is that its likely to produce a useful result (a turingrade), but its more than probable that we won't fully understand the result (incidently, that is different than a singularity). Intelligence will remain an emergent property of a system beyond our capacity to understand. To leave that system to self-evolve, knowing as we do that the system which we used as a starting point is terribly flawed is extremely dangerous. And for that matter, here we have Kurzweil's unreasonable projection. What makes us think that the machine we design would understand itself sufficiently to self-evolve in any predictable way? We ourselves are intelligences which are the products of emergent behavior in a complex system which has hithertoo remained beyond our capacity to understand. It's not at all unreasonable to think that when Kurzweil creates his brain simulation and says, "Tell us how you think?", that it will say, "I don't know. You built me. How do I think?" You can't handwave complex problems off to future minds and assume that they'll be more solvable.

No, first we ourselves do the hard work of figuring out what intelligence is, and then - armed with this knowledge - we engineer intelligent machines according to our design. No hand waving, which is as it should be. We don't need handwaving because clearly this is a solvable problem. Best of all, the end result is machines that don't act like humans and which mostly lack the sort of instinctual behavior that makes us dangerous to each other and to ourselves.

What's most interesting to me is that I think Kurzweil contributed one of the single most important peices of the algorithm puzzle and yet he seems to have abandoned algorithmics as a solution. Kurzweil's generalized font reading algorithm represents the first step in the sort of generalized abstracting program necessary to create concepts such as 'tableness', 'dogness', 'catness', 'fruitness', and so forth. It's this capacity which humans have (but don't understand) which lies at the heart of intelligence as we understand the term. I believe human intelligence is the emergent property of a fairly large system of specialized algorithms, each designed to solve a specific problem. I believe that we are not in fact 'general problem solvers' (I doubt that there is any such thing) and that part of the reason that some tasks are hard for humans is that we have to apply algorithms never intended to solve those tasks to the solution. This results in alot of wasted processing, the equivalent of a mental "bogo sort".

For my further thoughts on Kurzweil's plan to model the entire brain in detail, see this .

celebrim, the singularity is not just strong ai.
It is the knee of the curve where dy/dx approaches vertical.
Kurzweil is just plotting the non-linear regression of technological change over time.

Where I see hubris is in playing with things that you don't fully understand.

I don't think it is hubris...i think it is koros, right action. The Turing Heresy states, "if god made man in his image, how can we fail to make intelligences in our image?" Isn't that property inherent in our "design"?

And Vernor Vinge thinks it will be web-enhanced hiveminds.

/me "goes and beats head against wall"

"celebrim, the singularity is not just strong ai."

I didn't say it was.

"It is the knee of the curve where dy/dx approaches vertical."

No, it is the point where dx/dt approaches vertical. But let's not quibble over variables? Are you really suggesting that I don't know what an asymtopte is?

"Kurzweil is just plotting the non-linear regression of technological change over time."

For now, I'm not going to go over again why I think that this is not a reasonable approach to predicting future advance. Before I get into any of that, I want to concentrate on how annoying the next section of your post is:

"I don't think it is hubris...i think it is koros, right action."

There is a huge non sequitur coming here...

"The Turing Heresy states, "if god made man in his image, how can we fail to make intelligences in our image?" Isn't that property inherent in our "design"?"

First, let me deal with the fact that this is a non sequitur. You set out to argue against my claim that it was hubris to meddle in dangerous things that you didn't fully understand, and as evidence you turned to talking about something else. What you said doesn't address my claim. I didn't claim that there was anything inherently arrogant in making something in our own image, or even in trying to understand something you didn't understand. I claimed that there was something inherently overreaching in 'hacking' on something important without really knowing what you are doing. So you didn't address my point at all. Are you really suggesting that 'hacking' on something important without really knowing what you are doing - except perhaps in cases of desparation - is 'koros'? Shouldn't you first attempt to understand something, and then once you have 'right thoughts' proceed to 'right action'?

What's worse, you're counter-claim is to agree with exactly what I said. You obviously didn't read me at all if you think I disagree with the creative property being inherent in our design.

What's even worse than that is that you don't even know what the 'Turing Heresy' is. The formulation you just gave is not the 'Turing Heresy'. As far as I know, the Catholic Church has no basic disagreement with the statement as you formulated. In fact, my answer to you when you brought up the 'Turing Heresy', was more or less directly cribbed from Tolkien's 'Prayer of Aule to Illuvatar' and Tolkien was as pious of a Catholic as anyone. In fact, I've long maintained that Tolkien's own secondary creation falls under the category of creation described in Tolkien's account of Aule and the Dwarves, and expresses Tolkien's own prayers with respect to his Creator.

But never mind that. The 'Turing Heresy' is that there is nothing in the human mind thought which does not have a material description. That is to say, the 'Turing Heresy' is to refute that man's special intellectual abilities are due to his possession of a soul or some other mystical aspect. As far as I can tell, this was a 'heresy' mainly in Turing's own mind, because no one seemed at all upset about that claim compared to say his homosexuality.

Nonetheless 'Turing Heresy' is interesting in that lots of different people who would seem to have no special religious feeling nonetheless want to require human minds to have some unfathomable mystic ability. For example, if you watched 'Deep Blue vs. Kasporov' you'll probably remember that many members of the Chess community at that time believed that it was impossible for a computer ever to beat the best human players, because they lacked some mystic quality possessed by humans. They continued believing this right up to the time 'Deep Blue' stymied Kasporov in game 3 with a series of moves that even the grand master commentators on the match did not understand until the outcome was literally 3 moves away. Incidently, in my opinion when Karporov stormed away from the table at that point claiming that Deep Blue must be recieving human help, Deep Blue became the first machine to pass the Turing Test in the limited case.

Another example of rejection of the 'Turing Heresy' in popular culture would be the Matrix movies, which suggest that there is some special computational ability which exceeds that of any possible computer. Even really good science fiction authors sometimes subcumb to this conceit, for example, in Iain M. Banks otherwise masterful 'Player of Games' the line of play of the human protagnonist eventually becomes unanalyzable to his S1+ AI minders.

My own thinking is that there is a big difference between thinking that we are God (or gods), and engaging in acts of subcreation. Judeo-christianity has no special prohibitions against trying to understand the natural world, or even imposing our will upon it. I would also argue that computational power does not necessarily have to be tied to having a 'soul' or some other mystic quality. There does not need to be a 'ghost in the shell' in order for something to seem animated, or in other words I'm not an animist. Incidently, neither is the Catholic church (or any other mainstream branch of Christianity) - at least in the since of claiming that everything that is animated possesses or is possessed by an anima - so its doubtful whether the 'Turing Heresy' should really be considered a true heresy. The only heretical claim of the 'Turing Heresy' is that mankind does not have a soul, and that is not a particularly exclusive feature of the 'Turing Heresy' and not in my opinion terribly important to it. In fact, Turing neatly sidesteps this question by pointing out that its entirely possible from his perspective that a machine which passes the Turing Test could have been granted a soul by God. That is plenty of wiggle room for escaping the clutches of even a medieval inquisitor, and again corresponds well with something like Tolkien's description of the creation of dwarves.

I strongly suggest that you have your answers ready when an AI comes to you and says, "I think, therefore, I am. But, do I have a soul?" IMO, it's a damn obvious question to ask as soon as you figure out you exist.

"And Vernor Vinge thinks it will be web-enhanced hiveminds."

Yeah, well I love Vernor Vinge's work. I think everyone should be required to read 'A Deepness in the Sky' as part of thier formal education. But I was not at all impressed by that interview, and in fact I'd been meaning to comment on it because Glenn asked a really pointed really question (that I didn't really expect him to ask given his singularity sympathies) and it caught Vernor so unprepared that it not only left Vernor speachless but when he finally did get around to answering it was to avoid the real thrust of the question.

I thought of another case of rejection of the Turing claim.

In Philip K. Dick's work, "Do Android's Dream of Electric Sheep", the overall plot is that though androids are perfect copies of people, as machines they can never possess true empathy because they lack some mystical quality possessed by real people. Dick would agree with Turing claim that intellectual ability doesn't require the existance of a soul, but nonetheless Dick's work is very unsympathetic to machine intelligence.

By contrast, when "Do Andriods..." was made into the movie "Bladerunner", at the end of the story the android Roy has compassion on Deckard and we see upon his death a white dove ascending into heaven - a clear metaphor for the fact that Roy has acquired a soul. This is much closer to the Turing conception than Dick's original work, far more sympathetic to machine intelligence, and I for one see that as progress.

jeez, celebrim, koros is a statement of belief, not a lead into the Turing Test.
The koros argument (for believers) is that if g-d made you with the capability to create another intelligence then you are obligated by koros to do it. Because it is part of g-d's design.
It has nothing to do with the Turing Test.

I am a great fan of Ghost In the Shell. In Innosensu, the factory of infinite evil, Locus Solus, uses ghost-dubbing to make their gynoid line more attactive. But ghost-dubbing uses up the human donor.
Myself, I am not sure that a soul or ghost can be used up.

I think the ghost is an electro-biochemical pattern set supported by an organic substrate (in our case).
Why couldn't there be an electro-biochemical pattern set supported by an inorganic substrate?
Granted, it wouldn't have the same flavor, but wouldn't it still be a ghost?