The fact that corporations already have this technology suggests that we need more safeguards, not less. If the corporations have it, and they've outsourced, then foreign governments, transnational political or terror organizations, identity-thieves and outright thieves can invade any home, business or government office in the US.

Note that today the FTC has declined to set up a do-not-spam list because spammers would get hold of it and use it, yes my preciousss. That says much about the state of cyber security.

Some informative sites in this area:
www.eff.org, Electronic Frontier Foundation
http://www.techlawjournal.com/, Tech Law Journal

Larry Niven wrote an interesting novel on this subject: "Oath of Fealty." It's worth reading, it's sufficiently unusual that it actually altered my attitude somewhat.

Bob, it's useful to be more specific about exactly what technology corporations have and what the feds want to do.

Corporations have amassed huge databases of commercial transaction information. Everytime you charge groceries, for instance, detailed info is kept on what you bought and how much you paid for it, scrubbed (maybe) of identifying data like your name, and then combined with the other data in huge data warehouses.

What individual corporations have, and have had for a long time, is software that can search that data for patterns ... what is selling where, at what price. Many food and consumer goods companies use this data to decide how much of which products to put on your local grocery shelf, for instance.

In the last few years, some of that data has been sold to common databases used by multiple companies, again with varying degrees of personal identification info attached.

So far as I can tell from public accounts of TIA, it would have gone beyond the existing corporate use in several modest ways:

1. multiple databases would be spanned, possibly requiring some slightly more sophisticated software to access data in different formats - no big deal, commonly done today for other reasons.

2. software would be programmed to look for specific patterns of transactions other than those that corporations usually care about... airline flights from certain countries followed by large monetary transfers, plus cell phone or calling card calls to those countries &/or to known terror sympathizers, followed by car rentals in sensitive areas ( near nuclear power plants, for instance) - I'm making up the examples but I think we all could come up with reasonable ones.

Given that this is well within the capability of corporations - and similar things are being done by them already - is it reasonable to prevent the government from doing so, for arguably more urgent reasons?

And is it likely that corporate use can be stopped or limited, given that much of the data is collected with consumers' consent (albeit usually casual rather than thoughtful consent)?

As an aside, I've been a software technologist for several decades and appreciate what EFF contributes, but FWIW don't always personally agree with their positions ....

According to From The Wilderness (see above URL), the government has actively pursued advanced databasing and retrieval to mine intel through the use of a commandeered software technology called PROMIS. The software, which was alegedly stolen from its original developer can elegantly collect information across databases of several legacies and languages for the purposes of gaining highly specific intel. CARNIVORE is another well known government global harvestor and with the guidelines established by the Justice Department and their use of the Patriot Act, anything gleaned from these powerful tools is safely hidden away from Freedom of Information queiries and such.

If we should just "get over it", isn't that the same as submit to it out of futility? The Constitution does not appear to firmly establish a right to privacy, but the 3rd, 4th, & 5th amendments allude to it and it is handed down to us from Common Law to some extent. It could be that the reason the nature of this right becomes vulnerable from time to time is that the encroachment is not firmly barred constitutionally. Still, if we don't argue agressively for it with proper vigilance, the chances are we will suffer from an unprincipled intrusion from a government willing to sell us out sooner or later.

When discussing privacy and the Internet it might be helpful to get our analogies straight. Due to the essential technology of the Internet communications (particularly email) sent over it are not like letters sent through the U. S. Mail or even phone calls. They're more like notices posted in the town square. Is there a reasonable expectation of privacy to such notices? We might wish there were but it won't make it so.

Encryption and various other technologies can mitigate this somewhat but they won't change the basic technology.

Obelus, I concur that there are dangers.What guidelines would you propose?

I'd really like to hear people's thoughts on the tradeoffs here. What policies should we advocate? What risks would those policies inevitably entail???

>>It's useful to be more specific about ... what the feds want to do.

Echelon; Carnivore; Total Information Awareness. The last one was a sort of google on everybody.

Dave, you're right of course about the basic design of the Internet protocols.

It would seem that your point, then, is that since any information exchanged electronically is more or less open, we have little (practical? legal?) basis for attempting to limit the use of electronic data once it has been exchanged. Is that a fair reading of your comment?

Would your position change if something like quantum computing were to significantly improve our ability to prevent unintended users from decoding our data?

Bob, Echelon and Carnivore are reportedly able to search intercepted communications.

TIA was reportedly intended to search a very different source of data, namely commercially available data warehouses (among other sources).

But in any case, what policies would you advocate? Which risks would you rather run - the risk of an overly intrusive government or the risk of major loss of life in an attack? Why?

I can see arguments for both sides of this, but I think it's critically important that we all think through our positions carefully. I bemuses me to hear people I know both criticize the US government harshly for not having prevented the attacks of 9/11 and also insist that government officials should not sift through data until there is an incident to investigate. There ain't no such thing as a free lunch here -- just a question of which side we might err on and how to mitigate the worst aspects of that risk ....

Or so it seems to me.

I've mostly given up on protecting privacy. David Brin's The Transparent Society describes in detail how we're going to lose it as technology advances, and that the important thing is to force the government to reveal what it's up to. He's convinced me. As a free society our best defense is to keep watching each other.

Robin Burk:

we have little (practical? legal?) basis for attempting to limit the use of electronic data once it has been exchanged. Is that a fair reading of your comment?

That's precisely my point. I'd go a little further. The problem isn't the Internet. It is what it is. It's people assumptions and expectations that are the problem.

I'd add to that, peoples' conflicting desires too. Note this article about targeted online ads, which will only work if people are willing to have their online browsing habits tracked. My guess is that most people won't take the extensive measures needed to avoid being tracked.

The Ahmed Chalabi/compromised Iranian cryptography story discussed by Dan Darling on WoC two weeks back got me thinking about this issue. Hold on, it's not as off-topic as it seems.

So US ally Chalabi told Iranian Intelligence that the NSA is reading their telexes, causing that gold mine to end. Simple story. Or is it? There was a link on WoC to crypto expert Bruce Schneier and his evaluation of the story. Schneier's conclusion: the story as it's been shopped by Newsweek et al isn't logical, and there are numerous alternative plausible explanations. But if one of them is true, the others necessarily aren't, and those of us with only open-source information can't distinguish the true from the false.

Robin is posing a question about the conflicts between privacy and security that has similar features. Through open sources, we have a good idea of the structure of the internet (per Dave Schuler), and of what DARPA, NSA, HSA, foreign governments, and private actors are likely to be capable of (per Bob Harmon & obelus). What these actors are actually doing isn't--never will be--completely clear. First, there are often 'rogues' operating in the grey zones at the margins of corporate and government accountability. Second, as far as "Homeland Security," unambiguous clarity lets our adversaries know what we aren't doing. One doesn't have to subscribe to "security through obscurity" to see that ambiguity and uncertainty complicates the adversaries' planning and operations.

Personally, I don't want our society to be one step away from the no-privacy dystopia that is telegraphed by "Total Information Awareness." On the other hand, we will never know that an agency or company isn't doing X (any more than we know the NSA/Chalabi truth). The 9-11 Commission is currently reminding us of the political benefits of playing simplistic and even illogical Blame Games. e.g. "Curse you for not doing what you weren't authorized to do!"

So I'll side with Dave Schuler: we should recognize that which is in the technical realm, and ensure that privacy safeguards (laws and policy directives) are reality-based, and don't arise from wishful thinking. Especially when, for the forseeable future, there will be powerful (profit) and sometimes legitimate (security) incentives for various actors to cut corners when it comes to protection of privacy.

I think before we assume that it's a choice between "the risk of an overly intrusive government or the risk of major loss of life in an attack" we should ask whether the overly intrusive government would prevent that attack?

One of the problems in 9/11 is that field FBI offices did get wind of the pilots in two or three different parts of the country. The also had Zacarias Mossauwi. Did this information get shared through the FBI system, not to mention to other agencies? No? So the problem was not information, but how to analyze it and act on it.

So, the more data generated by TIA and its ilk would pose these problems:
1. Data overload. Mountains of data, oceans of data that even computer searches would find daunting.
2. Assessment problems. How do you pick out the hints of the next attack, esp. given an enemy prone to shape-shifting, and, worse yet, apparently has no central command (The Economist notes that al-Qaeda is more like a "franchise")?
3. Security problems. Amass a central database and someone, sure enough, will steal it. Didn't we learn anything from Hanssen and Ames?
4. Belief problems. What if the powers that be are given intelligence that they don't want to hear? Stalin had ample warning, from various sources, about 6-22-41. It didn't get acted on.
5. Accuracy problems. "Hello? Homeland Security? I think my ex-spouse is an al-Qaeda agent."

Just look at the no-fly list. One case the ACLU pursued was of an anti-war activist named Adams, barred from flying. Turned out that anybody named J. Adams would have problems on check-in, Jan Adams, Jay Adams, John Quincy Adams. Meantime, any terrorist worth his salt could buy a new identity on the internet and go right on through.

Before you trust an intrusive government, you might want to ask if it even knows what it's doing.

Fair enough, Bob. But before you list likely obstacles, you might also want to get the mechanisms right ... in the interest of helpful analysis.

Oceans of data are an issue - but much of it already exists. Its security is an issue, but not in the sense you suggest. You are envisioning massive mainframes in the bowels of some secret government facility. Think instead in terms of networks of computers, in a variety of places, sharing the data and the analysis workload. That changes the security issue a lot and some of the relevant technologies to handle that are already being matured for other purposes, such as the networked battlefield of which the 4th Infantry Division's famous equipment is a small precursor.

The software techniques to find relevant connections in the data - to do the analysis you note was missing before 9/11, to connect the dots - is precisely what 'TIA and its ilk' are intended to produce. There are technical challenges, both in terms of semantic analysis and also in terms of computability. (In technical terms, full searches are 'NP hard', i.e. the processing time for full searches goes up exponentially compared to the number of data items to examine.) These aren't issues unique to TIA or its ilk, however ... in fact, they are well-known challenges when constructing intelligent software agents of all types and have been hot topics in addressed in computer science departments / the AI community for a number of years.

Moreover, constructing such software requires analysts to clarify their working assumptions about what might constitute a relevant pattern or connection between dots. The actual execution of that semantic analysis against masses of data is what computers are good for ... the analytic insight always comes from humans, however -- and making that explicit often enhances any human information process when it is automated. Those of us who have reengineered significant corporate processes know just how much of an impact that explicit identification can have ... strategies, assumptions and authority must be addressed openly, which shakes some organizations up rather dramatically, usually for the better.

Belief and accuracy are always issues when data is interpreted to become information, whether manually or by software. There are several models from other walks of life that have been proposed to deal with the accuracy issue, including the approaches taken in criminal trials to establish credibility of witnesses and their testimony.

It certainly is true that software won't automatically cause organizations or people to believe what they don't want to believe. But it can uncover unsuspected patterns and trends and it can produce strong evidence that corroborates or challenges assumptions and beliefs.

Re: the no-fly list, the fact that stupid software was constructed quickly under time pressure is scarcely an argument not to construct smart software that is well thought-out.

Robin, that's a valuable insight, and I defer to your expertise in this field. I continue to worry about the human practicalities: the fact that intelligence users sometimes want to see what they want to see, the tendency to amass data for its own sake, and above all the time consumption required. You quite rightly point out the need for analysts to (1) clarify their assumptions (and methodology) and (2) then, clarify the intelligence they get, using that framework.

The legal Rules of Evidence is something I have been exposed to, and the weighing of relevance, credibility, authenticity and originality of any evidence -- not just witnesses but documents and other media -- is an intensely human method of analysis. They've yet to design a computer program to analyze a witness' character, for example. You're right to use this as a metaphor but remember this requires time. And thought. And the people (not just algorithms) to do it. And higher-ups who won't throw it aside or simply not understand it (e.g., at FBI Washington).

"Only two things are infinite: the Universe and human stupidity -- and I'm not so sure about the Universe." -- A. Einstein

The risk is of the government collecting and concentrating private data is greater than believed.

Assume for a moment that you are in Mr. Ridge's crosshairs for some reason. He can use some derivative of a total information awareness system to gather evidence against you. He would have reasons to gather information to incriminate you, but what happens to any exculpatory evidence? Was it never collected? Is it held in the system somewhere? Can it be deleted? Can it be altered?

Will your defense council have unfettered access to this system to defend you against government charges? Or will charges always result in a fait accompli for the government?

Without the ability to communicate and do business anonymously, I believe we will ultimately be destroyed by our own government. I consider this issue to be of the highest priority, certainly far more important than the threat posed by those Al-Qaeda fools. We have some time, perhaps 15-20 years, before the government's information processing technology reaches the point where effective monitoring -- and thus control -- of all commerce and communications becomes practical. At this point we will all be at the mercy of whoever controls the surveillance apparatus at that point.

If the past is any example (Hoover anyone?), life under the Panopticon will not be particularly free or pleasant.

At this point data collection technology is already sufficient. As some here have pointed out, data analysis is not there yet -- it's really only adequate for harming innocent people right now.