Thanks for the primer on where MT is these days. Used to share offices with a translation bureau (about 2 decades ago) and the first "computer-assisted translations" were beyond funny. But the entire rule-based approach has been a real stumper. It's not just fragments and slang or colloquialisms that present problems -- there's no way to get to any of the "blends" that the cognitive/linguistic types talk about.

I'm interested in learning more about some of the issues. Can you point me to info on the sorts of proposals you referred to here. The more I deal with blogs, I find the more the whole area of taxonomies becomes interesting/essential:
Since all machine translation techniques work better when the domain of discourse is known and used to categorized both the training data and text to be translated, there may be some advantage in at least creating hooks for the various categorization and taxonomy proposals floating around the blogging and open source communities.

Much obliged!

Nadezha, good question, and I will put together a few followup links, maybe a post. But right now I got back from drinks and a big dinner with AL, Blackfive, Jeff Jarvis, Omar and Mohammed and the Spirit of America folks and I am wasted. Tomorrow is airplane and helping my wife with a holiday gathering, so likely Monday before I get to it, just don't want you to think I'm blowing it off. Brain cells shutting down in 5,4,3....

Hmm nice idea.
I can see a few problems (and there is alot of IMHO here):
1) The opensource community maybe this industries worst enemy. Once an opensource project gets going it tends to get a bad case of mission growth, every feature under the sun will be included, and new sub projects will form which basically support the main project. (Cite: Wikipedia, Apache, Linux, etc)
If a general machine translation project gets going, expect it to eventually try to do every in the general area of machine translation, industry would have to either compete, or stay in niche markets (which opensource doesn't seem to do so well).
2) Opensource projects are very long term, and generally quite slow. From the moment something is first done on the project, to the time when it starts to become a 'large' (and useful, one might imagine) project could easily be measured in years. OTOH 'large' OS projects will prolly last decades. (Cite: Emacs, gcc, have both being going for 20odd years, Emacs predated the GPL)
3) OS projects are run by doers, not planners. The founding members will be doing all the grunt work for the start of the project, the ppl that will form the basis of the development group in a mature project will come from the userbase, the only way get a userbase is by developing a useful project, and that takes alot of work.

"The terms of openness may need to include some variant of Creative Commons that allows these texts to be used to train MT engines which will then be sold, but only if the vendor agrees to some useful giveback to the blogosphere, such as availability of a limited functionality, XML enabled server (or service) that can be plugged into blogging or other community software for free."
I can't see this working. I could see it working if the translating volunteers were providing data to an existing OS project, but in that setup it gives the company running the engine too much power, the project would be too much of a cog in the companies machine.

Hmm that turned into a bit of a waffle.

I've worked as a professional technical translator for 30 years and I'll guarantee no software at any price renders a faithful translation. Had I a blog, I would not want it translated by software, because it would surely say things I never intended, some of which would be ******* ******.

PacJim: I'm fully aware of the limits of MT, having used it in projects before. Where the blog situation differs from most document translation projects is the presence of people who can almost always work around the flaws of the translation. I've in fact seen this work in a 'social software' setting, almost a decade ago - something I should probably write up.

Factory - all good issues, and worth thinking about, particularly in terms of bootstrapping, i.e., what is the smallest useful thing that could be done. There's always a chance of mission creep, but doing actual translation software would probably be the last target, considering the lead time to working software it would imply. As far as licensing terms, IANAL, what I wrote was a requirements statement for someone who is an attorney.

...heading to airport...

PacRim Jim: I wouldn't give up hope just yet-- consider the non-trivial [english language] speech recognition problems of the eighties which have been basically solved. Hmm, didn't Robin have some stuff on Star Trek translaters?

I'll look for some DARPA/DLI links, but I remember reading about a proposed machine translation contest, like the autonomous vehicle contest earlier this year.

I personally think we have to jettison the stodgy old Chomsky comptational linguistics models that we've been stuck with for a half century. :(

I think you've missed a couple of points -

For one, the translation in blogspace doesn't have to be perfect (as it would if someone were trying to translate some technical report or business contract and correspondance, or whatnot) - as long as the ideas can come through, even somewhat garbled, we can get the gist of what the person was saying - at least enough to have an argument with them, which is the ultimate goal.

The other is one of the more important aspects of Open technologies - that's community.

I could envision creating technology that enables a community of people, who are trying to translate works back and forth between language pairs, to work collaboratively.

This, in concert with the notion of hypertext (where a word doesn't just have its text, but can 'mean' several things on different levels) has been limited in it's role in the web - we can do better. For instance, if I hover over a word in a translation, I might get several varients of the potential meanings of the translation, some definitions, etc...

Let me give you a real-life example of how this might play out in an Open Source/Community: let's say I want to translate some article from some website from Arabic to English for the noblest of all reasons - I want to find out what they're saying about me (cough).

I speak English. I don't care that the translation is not perfect - nor do I care that the translation output is not proper English and I might have to mouse-over some words or phrases - I'm trying to get the meaning and sense of the article. (This is plenty good enough for blogspace - each person writes in their native language, the reader reads (with the added difficulty of machine translation) in their native language.)

First, we let the machines do what they do best - handling lists and generating several provisional translations for each word, making a 'guess' as to which one was right, but that guess doesn't need to be right with hypertext. Then, syntactical analysis allows one to reorder subject/object/verb and related magic (I'm deliberately hand-waving, but a bad translation is tolerable to make the system start working, for reasons referenced below, and ultimately things should get better).

The software can make a statistical guess, and allow people who have some knowledge of both languages the ability to define better translations and add that to it's statistical analysis in the future.

But, in the more general case, you would let the native speaker in the target language select what makes the most sense - since language has context, it's usually easy enough to go from a syntactical and definition translation to a semantic one. That 'most sense' pick comes back to the server to inform the next set of 'best guess'.

Finally, if something doesn't translate right, or a definition isn't found, the english speaker could flag it, and then, some kind-hearted community-spirited multi-lingual folk (or greedy ones - we can probably pay a per phrase charge) can translate the line, and that translation is fed back in to the system, starting the cycle over.

This distribution of effort with retention of results would make the system self-propagating and limit the impact of the system on any one person, so as time goes on, things should become better and better through greater use.

Obviously, the software folks will continue making the software better/smarter, because that's what they do. The multi-lingual folks will always have a chance to translate or transliterate jargon, and handle euphamism and literary license, and make suggestions for better handling of certain syntactical constructs.

Frankly, and I've recommended this to them, this is a job for Google. Or the State Department. Or both.

Tim, all, some ideas:

For a minimal first step, how about an RSS tag element to indicate that “this post is a translation from language [source] to language [dest] of the post at [url]”. This would allow bots which harvest translations to monitor RSS feeds to get new translations for the purpose of accumulating a corpus. Some HTML scraping tools would be required in order to actually harvest both of the posts.

The point about categorization and taxonomy is dead on.

Consider: while speech recognition has indeed progressed substantially (as jinnderella indicated), natural language understanding has “strong AI” requirements (it’s the “understanding” that’s really hard). Real robust MT will likely require strong natural language understanding. This is A Ways Off (at least it is unwise to count on it soon.).

Further, even with strong understanding, domain context is crucial to creating decent translations. I’m a regular user of web based machine translation – I read the bike racing news from French sources. My knowledge of French is very poor, but when combined with a poor machine translation and my knowledge of the domain (bike racing), is sufficient to render a good translation – likely better, in fact, than a translation from someone who was an expert in French but not bike racing. The idiomatic mappings are sufficiently regular that an approach which combined categorization (based on machine learning / statistical computing) and corpus based machine translation (also based on machine learning / statistical computing) could create very good translations, I believe.

Finally, as indicated above, a human with poor language skills but decent domain expertise can use poor quality MT to create a decent translation, which could be reposted (to be read by humans or used to train a corpus based MT system). Categorization and taxonomy software could be used to “route” posts to the domain experts for translation. In fact, this kind of “taxonomy based routing” could make a boffo service in itself, and may be a great stand-alone “first step”. Taxonomy / classification systems which use Machine Learning also require a corpus of material. This corpus (the “training set”) must be hand categorized by topic. The categorization community uses several such training sets to benchmark their algorithms; on major one is a corpus of Reuters news stories from the 80’s. I’d suggest that one modern and relevant corpus might be the archive of “Winds of Change” itself, having a substantial volume and a detailed taxonomy. Same for other blogs which keep a detailed classification system for their archives.

Tim, I’m not a MT guru but I have some small knowledge of (and substantial interest in) the classification / taxonomy space, and some (IMHO) cool ideas in that space. I’d be very interested in a follow up post about “the various categorization and taxonomy proposals floating around the blogging and open source communities.” Personally I’m casting about for an OSS project to contribute my modest skills to. Color me interested in helping out.

Good to meet you in silly valley today, Tim (as well as, of course, Omar & Mo). The Arabic blogging tool sounds great!

That's great. But you know, machine translation has its own disadvantages, the main is - a machine usually makes a lot of mistakes in translation, which can cause misunderstanding and even serious consequenses.

[Spam. Kapow. --NM]