Yep. This stuff is genuinely scary. We may find ourselves looking back fondly to the "good old days" when all we had to worry about was nuclear annihilation. At least a nuclear bomb only renders a few, or a few hundred, square miles uninhabitable for a few centuries. But since it's not self-reproducing, once it's blown up, it's gone.

You raise the specter of enemies creating bugs aimed at specific targets. But why stop there? There are people out there who favor the extinction of the human race. Not very many, but with this technology, it doesn't take very many, does it?

Before trying to figure out how to get from here to somewhere safe in twenty years, let's ask what "somewhere safe" might look like.

Visualize a society in which this kind of technology exists, and can be used by specialists for all the promised beneficial purposes. What sort of protection allows this society to protect itself from small groups of radicals who might want to use the same technology to wipe out groups ranging from their particular enemies to the entire human race?

Can we have a democracy? Can we have individual rights? What about freedom of speech, press, and assembly? What about freedom from unreasonable search and seizure? What about privacy?

These are not rhetorical questions. These freedoms are essential to the checks and balances that make our system so successful. Open societies are more robust and adaptable than totalitarian ones, so if we have to accept totalitarian control as the price of survival, our society will become less effective and less safe from other sorts of catastrophes.

I'm a fan of technology (believe it or not!), but this stuff scares me. Perhaps we can come up with a technical fix, but I certainly don't know what it would be like.

You raise the specter of enemies creating bugs aimed at specific targets. But why stop there? There are people out there who favor the extinction of the human race. Not very many, but with this technology, it doesn't take very many, does it?

Terry Gilliam got there first.

Oh no! My ending </blockquote> wasn't. Can the moderators fix it?

Quite. As several people have mentioned here and in other places before, the problem of fundamentalist Islam is made worse by technological advance - as is the problem of various other sorts of nutcase, but Islam probably has the best resources.

Nuclear weapons, within a few years (not more than forty, possibly twenty) will be remembered fondly as a minor problem - if there is anyone left to remember. Not only biotech, but nanotech, could kill us all. At least bacteria and viruses don't have intelligence - potentially, nanoassemblers could.

As an example, imagine a replicating nanoassembler network that destroys the host organism whenever a crucifix appears in its visual field, or it recites the Lord's Prayer. Or just imagine a completely uncontrolled one - the "grey goo" scenario.

Which means that the problem of Islamofascist extremism needs to be solved, one way or another, pretty damned soon. Nukes can't kill everyone, much less everything. However, how many uncontrolled replicating nanoassemblers are needed to destroy all life on Earth?

One.

Smallpox is the obvious case, because it's easy. If you have the sequence and you can build the genes, all that's left is to let them build proteins and self-assemble.

Anybody with the sequence, the equipment, and the skills can rebuild it. Easy in principle. Once it happens we can go back to vaccinating everybody and it won't make much difference in the first world.

The trouble with designing something that's tailored for particular populations etc is that you need to somehow select what you're looking for. We don't understand this stuff nearly well enough to design such thing from scratch. So, say you want a virus that only attacks asians. You start with a wide variety of viruses and grow them on asian tissue culture. Then you take your liters of grown virus and expose it to tissue culture with caucasian, negro, australasian, etc cell membranes. Only the ones that fail to attach survive. You take your survivors and infect a new batch of asian tissue culture, and after repeated cycles you hope to get something that only infects asians. If it infects anybody at all. It's hard to select for just what you think you're selecting for. Maybe you'll wind up with something that only attacks tissue cultures. Etc.

To do this adequately you'd need a bunch of hi-tech concentration camps. Infect actual people, keep them isolated, and kill the survivors in case they're carrying something. There's still the small problem of making sure your product doesn't kill the people you don't want it to....

A small group of nihilists could perhaps make pathogens at random. Take genes that are thought to be lethal from one virus and splice them into something that spreads fast. They could release all their candidates and hope that one of them kills everybody. But they're extremely unlikely to kill everybody this way. They only get one chance. I mean, as long as their methods almost completely fail they can keep trying, but as soon as they get something that works well enough to disrupt the world economy so their electricity etc fails then they have to stop trying. Far more likely they succeed just well enough to sabotage themselves than that they actually drive us to extinction.

Due to the nature of the technology, there is no solution within the context of centralized, large-scale institutions. The threat is inherently decentralized. Thus, any feasable solution must also be decentralized. In this manner, the threat is also the solution.

Biotech and biofabrication technology will not only enable those who want to do the bad things. It will enable those who want to do the good stuff as well. The best approach to cure a designer pathogen is to allow a decentralized network of biohackers to have at it and to develop the antidote. I am certain this approach will be far faster and will save many more lives than reliance on big bureaucracies such as CDC and WHO.

The same decentralized networks of biohackers can deliver us the cure for aging (SENS) as well.

What should be obvious here is the need for a change of mindset. The idea that governments and large institutions can do positive things for us is one mindset that needs to be disguarded. All large institutions are bureaucracies. It is a law of nature that all bureaucracy is disfunctional. Thus, any political ideology that is based on the belief of the efficacy of bureaucracy is inherently flawed and should also be disguarded.

It should be obvious to everyone that the era of centralized control and centralized institutions is coming to a close. Let's bury the institutions and get on with creating the open lives and freedoms that we truely want.

Only with this change of mindset will we have the positive future that we all want.

J: Agreed about the vaccinations, though hopefully we'll be far enough along to create something more tailored and faster acting. There's a window of vulnerability until then, but if anything of the sort happens once (or comes close) there will be a massive program to close up vulnerabilities to 'legacy' pathogens.

Re the concentration camps idea, no, you won't need that in 15-20 years. Look, sequencing a human genome was a national lab scale effort when the US program kicked off in 1990. It had a $3 billion budget. According to a Wired story that just landed on my desk, it now costs a million bucks to do a complete sequence on an arbitrary human. That's over three orders of magnitude shift in cost/performance in a good deal less than twenty years. How much further does that have to run before you just get a 1,000 phenotypic representatives of whatever trait you want to track on, and sequence them all?

This is going to get done, not for genocidal reasons, but because there genetically based and correlated disorders that have strong ethnic and racial links. At some point we'll be stepping from empirical, statistical linkage to analytic understanding of mechanisms. And then those mechanisms become common knowledge and art.

Fletcher: I'm less worried about Bill Joy's 'grey goo' for one specific reason: There is NO observable experience curve running on nano-mechanical systems at this point. It's an unpopular view around the Valley, but I think that whole approach is going to have a lot of trouble getting jump-started. There's a lot of basic science and engineering to be done before getting anything to market, and whatever goes to market is going to be judged as a 'point application' - just valued for whatever particular need it fills. When I see nanomech move beyond the gedankenexperiment stage and start generating some scalable cash flow, I'll start worrying.

Genomics and carbon-based 'nano' get a free ride around a lot of these problems. There's an existence proof and inherent scale. A poorly understood, but huge, 'library' out there. The main application looks at you in the mirror every morning. Healthcare in the broadest sense is an enormous sector and still growing. The economic systems around both innovation and delivery are already in trouble, and are likely to break down completely when genomic and proteomic screening really hit their stride. So you've got companies lining up to buy DNA synthesizers at $300,000 the pop, and financiers eagerly fronting them the money.

Re the concentration camps idea, no, you won't need that in 15-20 years. .... How much further does that have to run before you just get a 1,000 phenotypic representatives of whatever trait you want to track on, and sequence them all?

The trouble is, right now "pathogenicity" is not a simple trait. You might get an organism that attacks at a particular receptor, that's present in one population but not in another. But how pathogenic will it be? You just don't know.

Maybe we'll get technology to understand that stuff better, but right now there's no substitute for clinical trials. You have to actually infect people if you want to find out what the symptoms are like.

The idea that governments and large institutions can do positive things for us is one mindset that needs to be disguarded. All large institutions are bureaucracies. It is a law of nature that all bureaucracy is disfunctional. kurt9 [#6]

Well, then, I guess we have nothing to worry about, because the Human Genome Project, and in fact the entire structure of modern science and medicine, must have failed. There are things to be said for libertarianism, but this bit of knee-jerk foolishness isn't one of them.

I would argue exactly the opposite. Large institutions (corporations both profit-making and non-profit, governments, churches, etc.) have become so successful that they have taken over our world. These large institutions can be considered to be artificially intelligent entities, with their own knowledge, goals, and plans, though probably not consciousness in a meaningful sense. They are legal persons, with many advantages in our economic system over individual humans.

They may be dysfunctional as servants, but they seem to be doing just fine as our masters. Welcome to the future!

J: You did notice the part where I talked about 'twenty years out', right? I'm not even attempting to describe 'right now', except as a way of estimating a rate of change. And I'm making no claims of 'imminent' threat, merely inevitable, if that makes you feel better.

Well, then, I guess we have nothing to worry about, because the Human Genome Project, and in fact the entire structure of modern science and medicine, must have failed. There are things to be said for libertarianism, but this bit of knee-jerk foolishness isn't one of them.

They may be dysfunctional as servants, but they seem to be doing just fine as our masters. Welcome to the future!

I will have to agree with kurt9 and disagree with beard. The technological trends in the twenty-first century overwhelmingly favors networks of individuals and private groups. The advent of decentralized manufacturing (fablabs) and home manufactured bio-tools (or weapons) both create a way and a need (urban centers become increasingly too dangerous to inhabit) to maintain advanced civilization with a more dispersed population. Such a trend would probably make large nation-states increasingly untenable and create more fragmentation (more micro-states). It is really hard for a government institution to control large numbers of people if its army and police keep getting shredded to pieces by homemade WMD's of biotechnological and nanotechnological nature. It will just collapse. The future is chaos.

Sure, Tim. But have you seen even any theoretical concept how we'd test pathogenicity without actual trials? I'm not up on the relevant literature, but I have absolutely no concept how that could be done. Like, how do you figure out how well a new combination of genes will do against an immune system, unless you test them against an actual immune system? Maybe in 20 years we'll have immune systems in a vial to test against, but....

The experimental evolution approach is great for things like bioremediation. When I was an undergraduate I met a pioneer in that field. He had a sewage treatment plant handling highly-alkaline textile waste, and he eventually got a functioning system at pH 12. Lots of bacteria, rotifers, even a nematode growing at pH 12. Of course the effluent was still high-pH after he got it cleaned up, but it was impressive. He grew systems that handled lots of heavy metals, acid, waste from antibiotics factories, lots of things.

But to do that with pathogens don't you have to expose them to lots of humans? And to get it specific, don't you have to reward them for attacking the humans you want them to attack and punish them for attacking the humans you want them to avoid?

Maybe I'm out of date. Have we had advances since the last 20 years that would have seemed theoretically impossible 20 years ago? I don't know of theory that says what you want is impossible, but I don't begin to imagine a methodology for it today. Aren't you depending on technology that we simply have no concept of just yet?

Oh by the way, large networks of individuals possess emergent intelligence not hierarchical bureaucracies. A bureaucracy is only as smart as its boss.

J Thomas #12,

As far as an approach to the problem today, I think your requirements for concentration-camp-style testing are about right. Ten years from now, perhaps not. I can think of some in vitro/"in silico" approaches, making some not-unreasonable guesses about what the current wave of genome characterization work will reveal over then next 2-3 years. It's late, so my explanation would be too fuzzy, and I'm not sure I even want to go into detail (even though I'm not an expert and my speculations probably don't matter).

AMac, I can sort of imagine some things too. Maybe all but the last stage in testing could be done in vitro, and then you find out whether you actually have anything.

And for a nihilist group none of this matters. They might do better to make a few million variant forms and spread them all at the same time, and see what survives. If nothing is infectious then the attack will probably go completely unnoticed and they can try again. If something comes out too infectious and too lethal and too indiscriminate, well, they're nihilists and they don't care who gets killed as long as a lot of the ones they're targeting do.

#7 Tim Oren:

To borrow your comment and twist it around: To see if functioning nanomech is possible, look in the mirror. There are millions of nanoassemblers (otherwise known as ribosomes) in every cell in your body. And nanomechanical devices, albeit made by mass processes, are not more than a couple of years off - I believe IBM is using nanoscale probes making holes in a substrate right now, in some research memory units. There are many possible routes, some of them using methods stolen from biology.

The point is that when someone gets an assembler built, and if it goes rogue, it will be too late. For everybody, and for everything living.

J: While I'll disclaim that I'm not a molecular biologist, I think one can already construct models of toxicity that wouldn't need to be 'evolved' but rather engineered. We already know a lot about metabolic pathways within the cell (de Grey's book is actually a good gloss on this) and increasing amounts about things like the mechanisms of neural signaling.

I have come to a conclusion that I didn't expose well in the post (and maybe deserves a separate one). That's the notion that a direct consequence of the synthetic biology approach I'm outlining is that we are going to be constructing what amount to 'control programs' that would never have been created through evolutionary pressures on a genetic algorithm, due to their statistical nature. Something like "silently replicate until there is a certain level of a metabolite unique to this organism in the surrounding environment, and then express a gene for a protein that binds to and clogs neuro-receptors." I dumped all that in behind the phrase 'escalation of complexity', and I should likely go back and elaborate to the extent I'm able. (Neal Stephenson alludes to ideas of this sort in his writings, though in a hypothetical nanomech context.)

Speaking of which, Fletcher, I think you're following the conventional line in conflating a lot of things that just happen to share the same physical scale. IBM's work is a natural outgrowth of the silicon processing and data storage industries. It's precisely there that you'd expect entirely human engineered nanoscale features to pop up first. They aren't assemblers by the farthest stretch of the imagination. Biology, as you say, already runs at that scale or we wouldn't be having this discussion.

But neither of these bears directly on building the infrastructure for the type of Drexlerian nanomech that lies behind the 'grey goo' vision. Remember, that horror show requires that the assemblers process everything in their path, not just silicon or carbon in particular configurations. Lumping things together just because they share dimension may be fine for promoting conferences or hawking magazines, but it's not a very good basis for analysis.

Just ran across this highly relevant New Yorker article that will have been published on Monday. ;)

Carlson here. A couple of thoughts for you all to consider.

First, interesting post, and interesting comments. You have managed to remain intelligent and thoughtful when most conversations about these issues rapidly go off the rails.

Second, re measuring the "wrong thing": I have tried for many years to figure out what the "right thing" is, and generally failed to come up with anything better than productivity and cost. As the original post above observes, there is no productivity history for gene printers because there are no gene printers. You can plot the longest published synthetic DNA fragment as a function of time, which is now well past most viruses and is into simple bacteria, but I am not sure this is a very interesting thing to do. I published one such curve last year in a report for the DOE, and have an updated version I will put on my blog as soon as it stops snowing and I can get to my office. Gene synthesis has a lot of art in it, and the "length of synthetic DNA vs time" plot only teaches you about methods in the lab, not about commercial technology. This means it is certainly possible to synthesize smallpox, but it is damn hard. And despite published methods for synthesizing flu viruses in the lab, there is no threat presently from artificial constructs of the 1918 flu because it is even harder to make than smallpox.

Third, artificial threats are unlikely to be significant for many years to come. But we do need better technology NOW to deal with natural bugs. There is still no vaccine for SARS. It will take at least 12 months to make a flu vaccine for a pandemic strain, and probably longer, when we have historical examples of the flu escaping vaccines in less than six months. It is possible to make rapid, synthetic vaccines, but the technology isn't widely accessible. More, faster, broad innovation in biological technologies is the only way to increase our safety and security in the short term. In the long term, such innovation is the only source of countermeasures against artificial threats anyway, so we need to invent faster anyway.

Fourth, there is no mechanism for tuning artificial organisms into smart bombs. That is, there is no biological function I am aware of that would make this work. The South African government already tried this, and failed. You could argue that increased knowledge of genomics and better technology might enable an effort to build a pathogen that would attack people with certain mutations, but for any given mutation there are many people alive today who would then also be susceptible to that pathogen. The comments above about the necessity of testing such a bug in people, and more importantly evolving it within people, are right on the money.

Fifth, Venter founded Celera and made it work not based on any dramatic improvement in sequencing hardware, but rather on his contribution to the method known as shotgun sequencing, which enabled rapid sequencing of short pieces of the genome and subsequent electronic reassembly. Moreover, the Celera effort was only feasible because the public effort had already completed a large scale genome map that could be used to place all the short sequences produced by Celera. That large scale map is what consumed most of the resources in the public project. In other words, he cheated. That said, it was a perfectly reasonable thing to do, since the map was public information. It is also true that human sequencing today -- the "million dollar genome" mentioned above -- also use the public large scale map. So genome cost is not a very good metric to use as an "experience curve". The cost of de novo sequencing of any organism is still quite high, though certainly not what it was when the human genome project was started.

Well, 'nuff said for now. There will be more on these topics at my blog synthesis, of course, and I hope to get my book done by the end of this month. Should be out by fall '08 from Harvard Press.

Thanks very much for stopping by, and for the clarifications. I'll stick with my comments on experience curves, however. Like Moore's Law, an experience curve doesn't model HOW improvements occur, just that they do. Knocking off your fellow engineer or scientist's work is part of the game, in carbon or silicon.

I look forward to the book!

Perhaps I should weigh in with another comment: Drexler said that one possible route to nanotech is to create the structures needed out of protein, created from tailored DNA - at least at first. Hence the two different threats (the one brought up by the OP and the one from me) are in reality one and the same. After all, biological systems are capable of creating nanoscale structures from inorganic materials - butterfly wings and diatom shells come to mind.

The issue is construction to at least molecular and ideally atomic specification/precision. Just sloshing molecules around, even in organelles and vacuoles, is not molecular manufacturing -- the name adopted by proponents of classical Feynman-Drexler nanotechnology after the "endless september" of bandwagon hoppers started declaring, e.g., that their face cream had nanotechnology in it.

Dry nano is at one end of a possibly multi-axis continuum; wet nano is at another end, and has a large fuzzy intersection with molecular biology. Drexler, Freitas and others who tend toward the dry camp do so for a number of reasons, not the least of which is that ab initio + in vacuo physical chemistry computation is somewhat more tractable than the protein folding problem.

If your goal is true precision, biotech is at best a path, not a destination.

That in no way changes the risks and opportunities that anticipated biotech presents. Very few people care about true precision yet. And it's not impossible we might run out of time for that, if things turn sufficiently sour.

Nort: We at least agree on definitions. Molecular manufacturing and synthetic biology engineering are not the same thing. My point is simply that the former has no observable experience curve underway, while the latter is already extent in the form of an entire ecosphere and is being rapidly and observably turned into human-useable technology. I'll keep worrying about and investing in the carbon-based bet, thanks very much.

"Thanks very much?" Wow, where did that come from? I was only refining something Fletcher said. Didn't mean to get up your nose about your portfolio.

Crikey.

Perhaps you didn't actually read my final paragraph:

That in no way changes the risks and opportunities that anticipated biotech presents. Very few people care about true precision yet. And it's not impossible we might run out of time for that, if things turn sufficiently sour.

Does this sound like I'm somehow making you wrong, Mr Oren? slow blink in taken-aback-ness

It wasn't meant as a slam, just a slightly flip sign-off. It obviously didn't come across that way, so my apologies. As a matter of fact, I have lost money down the nano-hole, not a lot fortunately, and not on something that I think would at all fit the 'manufacturing' theme.

Concerning "wet" nano, only now is work beginning on how water molecules encase and enable and affect protein molecules. The crystalized fold may have little to do with how it all works in a complex H2O envelope. It would be as though gears and cogs reshaped and reconfigured when oiled.

And as for dry ... Van der Waahls forces and other atomic-level interactions pretty much put paid to the image of nano-sized cogs and gears and levers; things get sticky. And thermal energy randomity plays hob with predictability and even location from pico-second to pico-second. There are new rules for the game down there, physical, chemical, and even quantum interactions that internal combustion engines know nothing of.

And short of operating in a vacuum, even staying "dry" may be a problem. Water wants to go where it isn't, and may take a hand regardless of whether it was invited to the table.

Could it be that nature has already in carbon-water biology found the only reliable model of manipulating nano-scale events? Perhaps that's why we don't have any competition from quartz critters. ;)

Brian:

I think you'll find that every one of your substantive points has been addressed by an entire chapter of the text Nanosystems. Thermal jitter, quantum effects and Van der Waals force, in particular. I can cite chapters if you'd like. Far from ignoring these matters, or handwaving, people at the Institute for Molecular Manufacturing have taken them seriously and have quantitative results that suggest that all are tractable for large classes of possible structures.

Further, researchers of "dry"-style molecular manufacturing have always noted that it's certainly best done under very controlled conditions. Depending on feedstock, "controlled conditions" might mean a vacuum, or it might mean a fluid bath -- say, neon or maybe even something like fluorinert. After completion of construction, the finished products might hold up quite well in the real world.

Re "[T]he only reliable model" -- how systems fail is a very interesting and complex subject. Nobody predicted flash photolysis of carbon nanotubes, AFAIK, yet it's retrospectively obvious.

Lastly, "critters" are a separate thing from devices constructed to molecular specification, and kind of a spurious point. Gas turbine engines aren't critters, and are much simpler than most critters, but they didn't evolve until people built and refined them. Or maybe you can correct me on that: how many gas turbine engines has natural selection produced?

Summing up my views:

It's far from certain that dry nano is a dead end. But without fairly good AI/design-ahead, advances in system design and management will probably be slow in coming. And AI hasn't come forward (yet?). So far, apart from plain old molecules, the results of modeling are all either bulk substrates, or pretty small subsystem elements that look like they'd be stable under some circumstances -- but how to make them is still very hard and unproven, and integrating them into systems is speculative.

The big "win" of wet nano and biosystem approaches comes from the fact that the stuff (or variations on it) is all mostly already out there. Near term, wet stuff has advantages for getting various useful things done, as well as having potential for various sizes of catastrophe.

IFF one really wants to build stuff to molecular specification, wet stuff is probably not the end of the game, either way.