Dec 6th 2008

The Search for Quality on the Web

by Riza C. Berkan

Riza Berkan is a nuclear scientist with a specialization in artificial intelligence, fuzzy logic, and information systems. He is the founder of Hakia.

NEW YORK - In the not-so-distant future, students will be able to graduate from high school without ever touching a book. Twenty years ago, they could graduate from high school without ever using a computer. In only a few decades, computer technology and the Internet have transformed the core principles of information, knowledge, and education.

Indeed, today you can fit more books on the hard disk of your laptop computer than in a bookstore carrying 60,000 titles. The number of Web pages on the Internet is rumored to have exceeded 500 billion, enough to fill 10 modern aircraft carriers with the equivalent number of 500-page, one-pound books.

Such analogies help us visualize the immensity of the information explosion and ratify the concerns that come with it. Web search engines are the only mechanism with which to navigate this avalanche of information, so they should not be mistaken for an optional accessory, one of the buttons to play with, or a tool to locate the nearest pizza store. Search engines are the single most powerful distribution points of knowledge, wealth, and yes, misinformation.

When we talk about Web search, the first name that pops up is, of course, Google. It is not far-fetched to say that Google made the Internet what it is today. It shaped a new generation of people who are strikingly different from their parents. Baby boomers might be the best placed to appreciate this, since they experienced Rock 'n' Roll as kids and Google as parents.

Google's design was based on statistical algorithms. But search technologies that are based on statistical algorithms cannot address the quality of information, simply because high-quality information is not always popular, and popular information is not always high-quality. You can collect statistics until the cows come home, but you cannot expect statistics to produce an effect beyond what they are good for.

In addition, statistics collection systems are backward-looking. They need time for people to make referrals, and time to collect them. Therefore, new publications and dynamic pages that change their content frequently are already beyond the scope of the popularity methods, and searching this material is vulnerable to rudimentary techniques of manipulation.

For example, the inefficiencies of today's search engines have created a new industry called Search Engine Optimization, which focuses on strategies to make Web pages rank high against the popularity criteria of Google-esque search engines. It is a billion-dollar industry. If you have enough money, your Web page can be ranked higher than many others that are more credible or higher quality. Since the emergence of Google, quality information has never been so vulnerable to the power of commercialism.

Information quality, molded in the shadow of Web search, will determine the future of mankind, but ensuring quality will require a revolutionary approach, a technological breakthrough beyond statistics. This revolution is underway, and it is called semantic technology.

The underlying idea behind semantic technology is to teach computers how the world operates. For example, when a computer encounters the word "bill," it would know that "bill" has 15 different meanings in English. When the computer encounters the phrase "killed the bill," it would deduce that "bill" can only be a proposed law submitted to a legislature, and that "kill" could mean only "stop."

By contrast, "kill bill" would only be the title of the movie by that name. At the end, a series of deductions like these would handle entire sentences and paragraphs to yield an accurate text-meaning representation.

To achieve this level of dexterity in handling languages by computer algorithms, an ontology must be built. Ontology is neither a dictionary nor a thesaurus. It is a map of interconnected concepts and word senses that reflect relationships such as those that exist between the concepts of "bill" and "kill."

Building an ontology encapsulating the world's knowledge may be an immense task, requiring an effort comparable to compiling a large encyclopedia and the expertise to build it, but it is feasible. Several start-up companies around the world, like Hakia, Cognition Search, and Lexxe, have taken on this challenge. The result of these efforts remains to be seen.

But how would a semantic search engine solve the information quality problem? The answer is simple: precision. Once computers can handle natural languages with semantic precision, high-quality information will not need to become popular before it reaches the end user, unlike what is required by Web search today.

Semantic technology promises other means of assuring quality, by detecting the richness and coherence of the concepts encountered in a given text. If the text includes a phrase like "Bush killed the last bill in the Senate," does the rest of the text include coherent concepts? Or is this page a spam page that includes a bunch of popular single-liners wrapped with ads? Semantic technology can discern what it is.

Given humans' limited reading speed (200-300 words per minute) and the enormous volume of available information, effective decision-making today calls for semantic technology in every aspect of knowledge refinement. We cannot afford a future in which knowledge is at the mercy of popularity and money.

Copyright: Project Syndicate, 2008.

If you wish to comment on this article, you can do so on-line.

Should you wish to publish your own article on the Facts & Arts website, please contact us at info@factsandarts.com. Please note that Facts & Arts shares its advertising revenue with those who have contributed material and have signed an agreement with us.

 


This article is brought to you by Project Syndicate that is a not for profit organization.

Project Syndicate brings original, engaging, and thought-provoking commentaries by esteemed leaders and thinkers from around the world to readers everywhere. By offering incisive perspectives on our changing world from those who are shaping its economics, politics, science, and culture, Project Syndicate has created an unrivalled venue for informed public debate. Please see: www.project-syndicate.org.

Should you want to support Project Syndicate you can do it by using the PayPal icon below. Your donation is paid to Project Syndicate in full after PayPal has deducted its transaction fee. Facts & Arts neither receives information about your donation nor a commission.

 

 

Browse articles by author

More Current Affairs

Jul 15th 2019
".....one of the most accurate recession indicators, known as the yield curve, has recently been flashing warning signs. Every postwar recession in the US was preceded by an inversion of the yield curve, meaning that long-term interest rates had fallen below short-term interest rates, some 12 to 18 months before the outset of the economic downturn."
Jul 6th 2019
Extract: ".........growing poverty even when working, the collapse of stable and safe social identities linked to work, the increasing instability of employment security, and the rapid change of local communities due to emigration, migration, collapsing housing affordability, and redevelopment initiatives that displace communities. These provide precise and urgent electoral rallying points. They are particularly effective given that so many mainstream politicians ignore these basic grievances. In recent years, the lineup of politicians opposing the New Right – Hillary Clinton, the Remain campaign, Emmanuel Macron and Matteo Renzi – have been unwilling to even recognise these structural problems. This provided the New Right the opportunity to appear credible, simply by acknowledging them."
Jul 6th 2019
".........an openly Russophilic administration in the US may be one reason why Putin’s domestic support has been declining so sharply."
Jul 3rd 2019
"Extract: .........in a world of rapidly expanding automation potential, demographic shrinkage is largely a boon, not a threat. Our expanding ability to automate human work across all sectors – agriculture, industry, and services – makes an ever-growing workforce increasingly irrelevant to improvements in human welfare. Conversely, automation makes it impossible to achieve full employment in countries still facing rapid population growth........The greatest demographic challenges therefore lie not in countries facing population stabilization and then gradual decline, but in Africa, which still faces rapid population growth."
Jul 1st 2019
Trump’s personal style – vocal, expertise-averse, scandal-prone and driven by a focus on his partisan base – may be unusual, but aspiring Democratic presidential contenders may be making a serious error in allowing Trump’s “Wizard of Oz” act of big claims and small achievements to pass unchallenged. There is a massive gap between the pledges he made to voters and the reality of an outsider presidency thoroughly co-opted by its party. So far, the “Trump revolution” turns out to be an ordinary Republican presidency.
Jun 25th 2019
"Trump’s vindictive bluster has steamrolled economic-policy deliberations – ignoring the lessons of history, rejecting the analytics of modern economics, and undermining the institutional integrity of the policymaking process. Policy blunders of epic proportion have become the rule, not the exception. It won’t be nearly as easy to spin the looming consequences."
Jun 19th 2019
Solar energy is one of the fastest-growing energy sectors in the world, and has the great advantage of producing no carbon dioxide, a greenhouse gas that is raising the average surface temperature of the earth. India is now for the first time in history investing more in solar energy than in coal. There is a simple reason for this. Coal costs roughly 5 cents a kilowatt hour to generate electricity. India just let a bid for 1.2 gigawatts of solar energy and four companies scooped it up at 3.6 cents a kilowatt hour.
Jun 19th 2019
Extract: "Abe has reportedly nominated Trump for a Nobel Peace Prize – at the request of the US – for opening talks with North Korea. And he has offered to mediate in America’s dispute with Iran. (His recent visit to Tehran – where he reportedly asked Iran’s leaders, at Trump’s request, to release detained Americans – made clear that, even squeezed by sanctions, Iran has no interest in negotiating with a serial violator of signed agreements.) What Trump calls an “incredible partnership” is, in reality, a largely one-sided relationship. But, for Abe, appeasing Trump is not so much a choice as a necessity: he must prove to Japan’s people and their neighbors, particularly the Chinese, that he knows how to keep Trump on his side."
Jun 17th 2019
Extarct: "We know well the damage that corrupt leaders do to their people. We should therefore have much more to say about the quintessential corruption entailed by tolerating lies. Such tolerance allows the poison to spread through the body and soul of democracy, undermining democracy’s institutions by attacking the invisible norms and tacit understandings that support them."
Jun 11th 2019
Extract: "I noticed this dynamic firsthand a few years ago in Blagoveshchensk, on the Siberian border, just a half-mile from the Chinese town of Heihe. A century and a half ago, Blagoveshchensk was part of China. Then the Cossacks took control of it, along with many other territories in Chinese Outer Manchuria, on behalf of the Russian czar. Blagoveshchensk’s local history museum presents the development of the town after the Cossack takeover as a civilizing mission. The Russians, it seems, still view themselves as superior Westerners. As for Heihe, it got rich a quarter-century ago, after capitalizing on Russia’s post-Soviet disarray to sell cheap goods to then-starving Russians. Its own history museum presents the Cossacks as “hairy barbarians” (Lao Maozi) and lists the towns of Russia’s far east by their historical Chinese names: Blagoveshchensk is Hailanpao, Vladivostok is Haishenwai, and Sakhalin is Kuye. Local behavior reflects these perspectives. At the ferry port, the Russians sneer at the Chinese traders who bring Russian vodka and chocolate to Heihe, while the Chinese move past the Russians as if they do not exist."
Jun 5th 2019
Extract: "....the Constitution, which established the impeachment process as a check on the president’s behavior between elections, says nothing about using it only when politically convenient. Moreover, given the results in 2018, Democratic Party leaders might well discourage making the disposition of the president the key issue in the next election. Most important, a decision not to initiate an impeachment process against Trump could set a terrible precedent. If Trump isn’t impeached for his numerous criminal acts and abuses of power, would impeachment remain a viable check on the presidency? "
Jun 3rd 2019
Extracts: "Sooner or later, all smaller powers dependent on global markets would have to choose a side, unless they are somehow strong enough to withstand both American and Chinese pressure. With China and the US both demanding clarity, even economic giants like the European Union, India, and Japan would be faced with an intractable economic dilemma."
May 24th 2019
Waging a war against Iran, or even thinking of doing so, is sheer madness. Trump has thus far wisely rejected the warmonger National Security Advisor John Bolton’s outrageous advice. Waging another war in the Mideast, this time against Iran, would have not only disastrous consequences for the US but will also engulf our allies from which they would suffer incalculable human losses and destruction. Bolton was the architect behind the devastating war in Iraq in 2003, which inflicted more than 5,000 US casualties and a cost exceeding two trillion dollars, allowed Iran to entrench itself in Iraq, and gave way to the rise of ISIS.
May 24th 2019
The private Tasnim news agency reports from Iran that in a speech to thousands of university students, Iran’s clerical leader Ali Khamenei made an unusual and extraordinary criticism of president Hassan Rouhani and foreign minister Mohammad Javad Zarif over their handling of the 2015 Joint Comprehensive Plan of Action or deal on limiting Iran’s nuclear enrichment program.
May 21st 2019
Extract: "Brexit, after all, is as much a Kremlin project as it is anyone else’s. Putin wants to divide Europeans, and in the UK, Brexit has succeeded in dividing Britons like nothing since the Corn Law debates almost 200 years ago. Putin wants the EU to fragment, and Brexit is causing the biggest crack yet in the bloc’s history. Putin wants to sow doubt about the legitimacy of traditional news sources; pro-Brexit media consistently promote lies as truth and inveigh against reputable papers like the Financial Times as elitist enemies of the people."
May 16th 2019
Iraq’s population when invaded was 26 million. Iran’s population today is 81 million..........Whereas Iraq’s neighbors– Turkey, Iran and Saudi Arabia in particular– had been mauled by Saddam and so did not strongly oppose Bush’s invasion, Shiite Iraqis, many Syrians, the Hazaras of Afghanistan, and the some 40 million Shiites of Pakistan would support Iran.
May 15th 2019
It’s time that economists, pundits, and politicians start looking holistically at life in our times, and take seriously the long-term structural changes needed to address the multiple crises of health care, despair, inequality, and stress in the US and many other countries. US citizens, in particular, should reflect on the fact that many other countries’ people are happier and less worried, and are living longer. In general, those other countries’ governments are not cutting taxes for the rich and slashing services for the rest. They are attending to the common good, instead of catering to the rich while pointing to illusory economic statistics that hide as much as they reveal.
May 8th 2019
"........Meanwhile, Trump is leaving the door open for Russia to come to his aid again in 2020. The White House and congressional Republican leaders have been blocking a bill to secure US elections against foreign attacks. And administration officials have been instructed not to raise the issue of Russian interference with the president, lest it cast a shadow on his legitimacy.  The next phase in this affair is already coming into focus. Barr, with the help of Trump’s golfing buddy Lindsey Graham, the Republican chair of the Senate Judiciary Committee, is now enlisted in peddling the president’s fantasy that the Mueller investigation was a “witch hunt” orchestrated by “deep-state” supporters of Hillary Clinton. Once again, current and former FBI agents will be targeted, either because they expressed criticism of Trump or because they opened a national security investigation into a hostile power’s meddling in the US presidential election (which continued in the 2018 midterms). FBI director Christopher Wray, commenting on the Mueller report, said that the Russians are “upping their game” for 2020. "
May 7th 2019
We are witnessing the loss of biodiversity at rates never before seen in human history. Nearly a million species face extinction if we do not fundamentally change our relationship with the natural world, according to the world’s largest assessment of biodiversity.