Dec 6th 2008

The Search for Quality on the Web

by Riza C. Berkan

Riza Berkan is a nuclear scientist with a specialization in artificial intelligence, fuzzy logic, and information systems. He is the founder of Hakia.

NEW YORK - In the not-so-distant future, students will be able to graduate from high school without ever touching a book. Twenty years ago, they could graduate from high school without ever using a computer. In only a few decades, computer technology and the Internet have transformed the core principles of information, knowledge, and education.

Indeed, today you can fit more books on the hard disk of your laptop computer than in a bookstore carrying 60,000 titles. The number of Web pages on the Internet is rumored to have exceeded 500 billion, enough to fill 10 modern aircraft carriers with the equivalent number of 500-page, one-pound books.

Such analogies help us visualize the immensity of the information explosion and ratify the concerns that come with it. Web search engines are the only mechanism with which to navigate this avalanche of information, so they should not be mistaken for an optional accessory, one of the buttons to play with, or a tool to locate the nearest pizza store. Search engines are the single most powerful distribution points of knowledge, wealth, and yes, misinformation.

When we talk about Web search, the first name that pops up is, of course, Google. It is not far-fetched to say that Google made the Internet what it is today. It shaped a new generation of people who are strikingly different from their parents. Baby boomers might be the best placed to appreciate this, since they experienced Rock 'n' Roll as kids and Google as parents.

Google's design was based on statistical algorithms. But search technologies that are based on statistical algorithms cannot address the quality of information, simply because high-quality information is not always popular, and popular information is not always high-quality. You can collect statistics until the cows come home, but you cannot expect statistics to produce an effect beyond what they are good for.

In addition, statistics collection systems are backward-looking. They need time for people to make referrals, and time to collect them. Therefore, new publications and dynamic pages that change their content frequently are already beyond the scope of the popularity methods, and searching this material is vulnerable to rudimentary techniques of manipulation.

For example, the inefficiencies of today's search engines have created a new industry called Search Engine Optimization, which focuses on strategies to make Web pages rank high against the popularity criteria of Google-esque search engines. It is a billion-dollar industry. If you have enough money, your Web page can be ranked higher than many others that are more credible or higher quality. Since the emergence of Google, quality information has never been so vulnerable to the power of commercialism.

Information quality, molded in the shadow of Web search, will determine the future of mankind, but ensuring quality will require a revolutionary approach, a technological breakthrough beyond statistics. This revolution is underway, and it is called semantic technology.

The underlying idea behind semantic technology is to teach computers how the world operates. For example, when a computer encounters the word "bill," it would know that "bill" has 15 different meanings in English. When the computer encounters the phrase "killed the bill," it would deduce that "bill" can only be a proposed law submitted to a legislature, and that "kill" could mean only "stop."

By contrast, "kill bill" would only be the title of the movie by that name. At the end, a series of deductions like these would handle entire sentences and paragraphs to yield an accurate text-meaning representation.

To achieve this level of dexterity in handling languages by computer algorithms, an ontology must be built. Ontology is neither a dictionary nor a thesaurus. It is a map of interconnected concepts and word senses that reflect relationships such as those that exist between the concepts of "bill" and "kill."

Building an ontology encapsulating the world's knowledge may be an immense task, requiring an effort comparable to compiling a large encyclopedia and the expertise to build it, but it is feasible. Several start-up companies around the world, like Hakia, Cognition Search, and Lexxe, have taken on this challenge. The result of these efforts remains to be seen.

But how would a semantic search engine solve the information quality problem? The answer is simple: precision. Once computers can handle natural languages with semantic precision, high-quality information will not need to become popular before it reaches the end user, unlike what is required by Web search today.

Semantic technology promises other means of assuring quality, by detecting the richness and coherence of the concepts encountered in a given text. If the text includes a phrase like "Bush killed the last bill in the Senate," does the rest of the text include coherent concepts? Or is this page a spam page that includes a bunch of popular single-liners wrapped with ads? Semantic technology can discern what it is.

Given humans' limited reading speed (200-300 words per minute) and the enormous volume of available information, effective decision-making today calls for semantic technology in every aspect of knowledge refinement. We cannot afford a future in which knowledge is at the mercy of popularity and money.

Copyright: Project Syndicate, 2008.

If you wish to comment on this article, you can do so on-line.

Should you wish to publish your own article on the Facts & Arts website, please contact us at info@factsandarts.com. Please note that Facts & Arts shares its advertising revenue with those who have contributed material and have signed an agreement with us.

 


This article is brought to you by Project Syndicate that is a not for profit organization.

Project Syndicate brings original, engaging, and thought-provoking commentaries by esteemed leaders and thinkers from around the world to readers everywhere. By offering incisive perspectives on our changing world from those who are shaping its economics, politics, science, and culture, Project Syndicate has created an unrivalled venue for informed public debate. Please see: www.project-syndicate.org.

Should you want to support Project Syndicate you can do it by using the PayPal icon below. Your donation is paid to Project Syndicate in full after PayPal has deducted its transaction fee. Facts & Arts neither receives information about your donation nor a commission.

 

 

Browse articles by author

More Current Affairs

Nov 20th 2019
Extract: "Moody’s, one of the big three credit rating agencies, is not upbeat about the prospects for the world’s debt in 2020 – to put it mildly. If we were to try to capture the agency’s view of where we are heading on a palette of colours, we would be pointing at black – pitch black."
Nov 17th 2019
Extract: "Digital money is already a key battleground in finance, with technology firms, payment processing companies, and banks all vying to become the gateway into the burgeoning platform-based economy. The prizes that await the winners could be huge. In China, Alipay and WeChat Pay already control more than 90% of all mobile payments. And in the last three years, the four largest listed payment firms – Visa, Mastercard, Amex, and PayPal – have increased in value by more than the FAANGs (Facebook, Apple, Amazon, Netflix, and Google)."
Nov 14th 2019
Extract: "Trump, who understands almost nothing about governing, made a major mistake in attacking career public officials from the outset of his presidency. He underestimated – or just couldn’t fathom – the honor of people who could earn more in the private sector but believe in public service. And he made matters worse for himself as well as for the government by creating a shadow group – headed by the strangely out-of-control Rudy Giuliani, once a much-admired mayor of New York City, and now a freelance troublemaker serving as Trump’s personal attorney – to impose the president’s Ukraine policy over that of “the bureaucrats.” "
Nov 4th 2019
Extract: "Trump displays repeated and persistent behaviours consistent with narcissistic personality disorder and antisocial personality disorder. These behaviours include craving for adulation, lack of empathy, aggression and vindictiveness towards opponents, addiction to lying, and blatant disregard for rules and conventions, among others." The concern is that leaders with these two disorders may be incapable of putting the interests of the country ahead of their own personal interests. Their compulsive lying may make rational action impossible and their impulsiveness may make them incapable of the forethought and planning necessary to lead the country. They lack empathy and are often motivated by rage and revenge, and could make quick decisions that could have profoundly dangerous consequences for democracy.
Oct 31st 2019
EXTRACT: "......let’s see what happens when we have less money for all the things we want to do as a country and as individuals. Promises and predictions regarding Brexit will soon be tested against reality. When they are, I wouldn’t want to be one of Johnson’s Brexiteers."
Oct 21st 2019
EXTRACT: "Were Israel to be attacked with the same precision and sophistication as the strike on Saudi Arabia, the Middle East would be plunged into war on a scale beyond anything it has experienced so far. Sadly (but happily for Russian President Vladimir Putin), that is the reality of a world in which the US has abandoned any pretense of global leadership."
Oct 20th 2019
EXTRACT: "Europe also stands to lose from Trump’s abandonment of the Kurds. If, in the ongoing chaos, the thousands of ISIS prisoners held by the Kurdish-led Syrian Democratic Forces escape – as some already have – America’s estranged European allies will suffer. Yet Trump is unconcerned. “Well, they are going to be escaping to Europe, that’s where they want to go,” he remarked casually at a press conference. “They want to go back to their homes." "
Oct 15th 2019
EXTRACT: "Assuming the House ultimately votes to impeach Trump, the fact remains that there are far fewer votes in the Senate than will be needed to convict him and remove him from office. But the willingness of Congress – including the Senate – to continue tolerating his dangerous conduct in office, including threats to US national security, is now truly in question."
Oct 7th 2019
EXTRACT: "The problem didn't start with the election of Donald Trump. Nor did it begin with the Democrats launching an impeachment inquiry against Trump. This is a developing crisis that has been growing like a cancer within our polity for at least the past 25 years. Its main symptoms are a lack of civility in our political discourse, a "take no prisoners" mindset, and a denial of the very legitimacy of "the other side." Trump didn't create this crisis; he was the result of it.   When Newt Gingrich took the helm of Congress in 1995, unlike previous Republican leaders, he embarked on a campaign not only to obstruct the efforts of then President Clinton, but to destroy him. Congress launched a series of investigations accusing Clinton of everything from corruption to obstruction of justice – with hints of even more nefarious plots to assassinate those who might pose a problem to his presidency.  "
Oct 4th 2019
EXTRACT: "As the story spreads, it grows darker. Meanwhile, Trump is trying to learn the identity of the whistleblower (who is protected by law), which could expose that person to great danger. And he is accusing some people – including Adam Schiff, the chair of the House Intelligence Committee – of treason. My sense is that Trump fears the tough, focused Schiff. Trump has ominously noted that traitors used to be shot or hanged. And he hasn’t helped himself with members of either party by declaring, in one of his hundreds of febrile tweets, that forcing him from office could lead to a “civil war.” Trump has taken the United States somewhere it’s never been before. His presidency may not survive it."
Sep 24th 2019
EXTRACT: "But regardless of whether the Ukraine scandal remains front-page news, it will haunt the US intelligence community, which has been Trump’s bête noire since the day he took office. Trump has relentlessly attacked US intelligence agencies, cozied up to Russian President Vladimir Putin, and divulged secrets to foreign officials, potentially burning high-value sources. This behavior had already raised serious concerns about whether Trump can be trusted to receive sensitive intelligence at all. Now, intelligence leaders must ask themselves how far they are willing to go in toeing the White House line."
Sep 21st 2019
EXTRACT: "As Lobaczewski pointed out, pathological leaders tend to attract other people with psychological disorders. At the same time, empathetic and fair-minded people gradually fall away. They are either ostracised or step aside voluntarily, appalled by the growing pathology around them.......As a result, over time pathocracies become more entrenched and extreme. You can see this process in the Nazi takeover of the German government in the 1930s, when Germany moved from democracy to pathocracy in less than two years.......In the US, there has clearly been a movement towards pathocracy under Trump. As Lobaczewski’s theory predicts, the old guard of more moderate White House officials – the “adults in the room” – has fallen away. The president is now surrounded by individuals who share his authoritarian tendencies and lack of empathy and morality. Fortunately, to some extent, the democratic institutions of the US have managed to provide some push back."
Sep 16th 2019
EXTRACT: "If the Supreme Court does agree with the Divisional Court that the question is political rather than legal, it will take the UK constitution into quite peculiar territory. Prime ministers will be the new kings and queens. They will be free to suspend parliament at will, and for as long as they wish, without any judicial interference. Parliament will meet not out of constitutional necessity but in the service of the government’s interests – namely, to pass its legislation and to maintain appearances, rather than to hold it to account."
Sep 12th 2019
Extract: "The Republican Party has lashed its fate to an increasingly unhinged leader. Though three other presidential hopefuls for 2020 now stand in Trump’s way, none can defeat him. But they can damage his reelection effort, which is why the Republican Party has been scrapping some primaries and caucuses. How well Trump does in November next year may well depend on how his fragile ego withstands the coming months."
Sep 2nd 2019
EXTRACTS: "Most people think of revolutions as sudden earthquakes or volcanic eruptions that come without warning and sweep away an entire political system. But historians, political scientists, and even the odd politician know that the reality is very different: revolutions happen when systems hollow themselves out, or simply rot from within. Revolutionaries can then brush aside established norms of behavior, or even of truth, as trivialities that should not impede the popular will............ Only time will tell whether we are currently witnessing the hollowing out of British democracy. But Prime Minister Boris Johnson may well have crossed some invisible Rubicon by.......... Whatever happens now, British parliamentary democracy may never be the same again. It will certainly never again be the model that so many people around the world once admired."
Aug 29th 2019
EXTRACT: "Events such as prorogations and dissolutions happen when countries face difficult times. Therefore, because of the disastrous effects of Brexit: sterling in freefall; a recession looming on the horizon and Britain’s international standing at its lowest ebb since Suez, it is no surprise that the country is in this position now. The worrying thing is that using the monarchical power of prorogation does not solve problems – it has a history of turning them into frightening and often violent crises. There is a worrying relationship between the use of such powers and a complete breakdown in government."
Aug 28th 2019
EXTRACT: "Reminiscent of Don Quixote, Trump is tilting at windmills. His administration is flailing at antiquated perceptions of the Old China that only compound the problems it claims to be addressing. Financial markets are starting to get a sense that something is awry. So, too, is the Federal Reserve. Meanwhile, the global economy is fraying at the edges. The US has never been an oasis in such treacherous periods. I doubt if this time is any different. 
Aug 24th 2019
EXTRACT: "In fact, with firms in the US, Europe, China, and other parts of Asia having reined in capital expenditures, the global tech, manufacturing, and industrial sector is already in a recession. The only reason why that hasn’t yet translated into a global slump is that private consumption has remained strong. Should the price of imported goods rise further as a result of any of these negative supply shocks, real (inflation-adjusted) disposable household income growth would take a hit, as would consumer confidence, likely tipping the global economy into a recession."
Aug 21st 2019
EXTRACT: "Climate change is real, and it is a problem. According to the IPCC, the overall impact of global warming by the 2070s will be equivalent to a 0.2-2% loss in average income. That’s not the end of the world, but the same as a single economic recession, in a world that is much better off than today.  The risk is that outsized fear will take us down the wrong path in tackling global warming. Concerned activists want the world to abandon fossil fuels as quickly as possible. But it will mean slowing the growth that has lifted billions out of poverty and transformed the planet. That has a very real cost. "
Aug 20th 2019
EXTRACTS: "It is no exaggeration to say that Johnson has lied his way to the top, first in journalism and then in politics. His ascent owes everything to the growing xenophobia and English nationalism that many Conservatives now espouse................Johnson has chosen a government of like-minded anti-European nationalists. His principal adviser, Dominic Cummings, was described by David Cameron, Britain’s prime minister from 2010 to 2016, as a “career psychopath.” Cummings is, alongside Johnson, the most powerful figure in the new government; he is an unelected wrecker who earlier this year was ruled to be in contempt of parliament. Fittingly, if depressingly, he now is masterminding our departure from the EU with or without parliamentary approval."