Some arguements on why 'The Singularity' will likely never occur, in this universe or New Eden:
1. People tend to stretch a single idea too far. The idea that processing power is everything, that once we get the equivalent processing power as the brain then AI will be inevitable, seems like a part of this. Processing power is one thing. A modern off-the-shelf calculator has the same processing power as the Apollo spacecraft and yet doesn’t routinely fly to the moon and back. For that to happen you need lots of other things: a well engineered spaceship, a huge amount of money backing you, the political willpower and climate to make it feasible, trained astronauts, an idea of spaceflight set up in the culture, a scientific rationale for going, the blessing and dreams of millions of people who believe it to be necessary, an economic incentive, etc. All of these things (and more) were in place in the 1960s to make space travel possible. Similarly, there will have to be a convergence of more than just technology in order to make AI possible. There are many ways for it to come about. And people like Raymond Kurzweil and Vernon Vinge are engineers. They have a great deal of authority when it comes to technology and engineering. I trust that plenty of Kurzweil’s purely technological predictions can come true. But he downplays or forgets the importance of other factors and so I think his ultimate thesis is wrongheaded. Based purely on technology, we could have colonies on Mars right now. We probably have been able to for twenty or thirty years. And yet, it hasn’t come about. So might it be with AI.
2. The mind is like a computer but it is not a computer. Yes, it works on mechanical, physical principles. But that doesn’t make it identical to a computer. Giving estimates of the “processing power” of the brain is largely going to be guesswork. It’s an estimation based on a metaphor based on a particular view of the universe. Furthermore, the brain is an integrated whole that is integrated into a larger whole. It’s hard to figure out the levels that the brain operates on because everything inside of it is interconnected. It affects itself affecting itself. And it gets affected by the larger organism that it inhabits. I am not just my thoughts nor just my brain. I am me, every part of me, down from my wiggling toes to the split ends of my hair. And I am in constant flux. At any moment, I am taking molecules from my environment and incorporating them into my body. Every cell wall that is a part of me is a jiggling mass of continuous change. I am a standing wave of barely controlled chaos traveling through space and time. Recreating that is hard. In order to simulate a person, we will have to simulate both hardware and software and the fact that the two are really just one. You will have to account for the fact that I am just a collection of atoms borrowed from the universe for a span. I don’t know if perfectly simulating that is possible. Perhaps a different sort of mind is. I believe that we can get AI, it will just take a long time and it will come piecemeal.
3. Intelligence is not a “real” thing. We like to think that intelligence has to do with getting a high SAT score or being good at chess or being able to answer trivia questions. But those are all just views of intelligence. Smartness is more like beauty. We can largely agree on what is beautiful and we can even “enhance” our beauty with modern surgical techniques (but that doesn’t lead to a “beauty explosion” of ever more beautiful people). The best we can do is make human beings more beautiful based on some pre-existing idea of beauty. They start to match up closer to that idea but, ultimately, it’s just vanity. Maybe someday in the future we will discover techniques to “enhance” our intelligence in the same way that we do plastic surgery. But we will have to work with what we’ve got. There isn’t some switch or dial that you can flip to make a person more intelligent. And yes we are very intelligent, in some ways more intelligent than all other creatures on Earth. But all organisms are problem solvers. They model their external reality just as well as every other creature around them, regardless of whether or not they are bacteria, fungi, plants, or primates. If they couldn’t, they would be eliminated. We can design tests of intelligence but they have to match up with some pre-existing idea of intelligence. All they can ever do is show how one person or organism does better on that test than others.
4. Knowledge is wonderful. But in and of itself it can not accomplish the impossible. It can not make a perpetual motion machine. It can not stop entropy. It can not preclude the fact that a wrench will be thrown into even the best laid plans. The universe is not a series of stumbling blocks that we are in charge of solving. We are taught to think this way in our high school classes: find the hypotenuse, derive Kepler’s Laws from Newton’s, tell me how Coleridge and Wordsworth were similar, give me the causes of the Civil War. But we use those illustrations precisely because they have answers (some of them, at least). Life is not a question and answer session. Real problems, the kinds outside of textbooks, don’t have “real” solutions. Even those in physics and math. Answers always end up generating more questions. Usually three or four. Our knowledge can be great and amazing. But nonconscious and unconscious forces are often greater and more amazing. I still trust evolution to manage things much better than we can for the time being. We are still too stuck in linear thinking; that problems only have one cause and one solution. That DNA controls cells. That pills can target specific emotional states. That robots will either love us or hate us. Reality is made up of infinite shades of gray and just thinking about things will not generate a remedy for everything that ails us. The universe does a much better job of shepherding itself because it does not have to deal with questions and answers. Our way of thinking can accomplish many things. But a nonconscious entity does not have to couch itself in this “problem and solution” view. It has a different way of thinking. Perhaps, if we do eventually create AI, it would be smart to make it nonconscious.
5. We do not occupy a special place in time. We already know that the Copernican Principle says that we do not enjoy a special place in space. Why do we insist on thinking that our time is the most important time of all? For most of Western history, people have thought this. They said, “Now the Messiah will come. It will happen in my lifetime.” We don’t think like that anymore but we hold on to certain ways. There is not some special point in time when everything that comes after it will be different (or, to put it a different way, every point in time is such a time). We are not living at the height of civilization nor are we at the lowest point. We are at a point. To insist that things are inevitable, that there is some destiny in the universe, that things are not contingent, and that this is the only way that could be is, to me, an extremely old-fashioned way to think. I always thought the point of atheism was knowing that there isn’t some design or plan in the universe. To say something is inevitable contradicts this, in my opinion. There are patterns. There are interesting places and interesting times. But we are the ones that decide what is interesting. We are the ones who see the patterns. Destiny is in our heads entirely. We are the ones who shape it for ourselves. We have to deal with that.
In and of themselves, do I think these points are enough to say the Singularity won’t happen? No, of course not. These are my thoughts. But there are counter-arguments to nearly every argument. Raymond Kurzweil spent a chapter of "The Singularity is Near" attacking some of the points I just raised (though not well enough to dismiss them, in my opinion). Furthermore, the Singularity is a very powerful meme. I suspect that it will last longer than Kurzweil and Vinge, longer than 2045, longer than plenty of us. People will forget what we’ve already said and then say, “No, really, this time it’s just around the corner. We have better facts, we can prove it more fully now.” It’s what always happens.