What is a full text search?

Full-text search differs from other searches, such as the search for

• Metadata (e.g. author, title, date),
• primary keys in database tables or
• a fixed file name in the file system,

by searching through the entire text of a document.

Full text index

A full-text search can either scan the so-called search word document at the moment it is triggered (on the fly), or it can fall back on an index that has already been created for each text to be searched. Such an index, also known as a full-text index, is a data structure that allows a full-text search to be carried out efficiently. The best-known example is the inverted index, which contains a link between words and the documents in which they occur. The index is created by what is known as indexing . The text is broken down into tokens that are added to the index. Depending on the type of search query, additional information such as the position of a word in the text or its frequency is saved during indexing.

Full text search with index

Searching with an index is usually considerably faster than a cursory search. The disadvantage of the index search is the effort required to create the index. It must either be carried out before the first search or, if the texts to be searched are changed very frequently, it must be carried out again before each search. The search or indexing process can be supplemented by further linguistic analyses such as stemming or lemmatization. These analyses help to trace different inflectional forms of a word back to a basic form in order to improve the search results. For example, the words “gehe”, “ging” and “gegangen” are traced back to the lemma “gehen”. Search results can also be weighted according to relevance, which is known as ranking. Such systems are known as information retrieval systems.

Full text search in relational databases

In relational databases, field indexes are often used to index specific table columns. Although these indexes allow quick searches for exact field contents or for field contents that contain a specific word, they do not support more complex search queries, such as searching for several words that occur at a certain distance in the same text. This requires special full-text indexes that index the entire column and provide corresponding operators for the search query.

History

Providers of commercial full-text systems include IBM (STAIRS, since 1973), OpenText (since 1985) and Verity (since 1988).

Known applications

Well-known applications of a full-text search are web searches with search engines such as Google, Bing or Yahoo, which search in the indexed websites, as well as the Unix programs grep and locate, which search in files or file names. Desktop search engines such as Windows Search or Google Desktop Search also use full-text search to find documents stored on the local computer. Another example is Wikipedia, which enables a full-text search in all articles.

Full text search in databases

Full-text search in databases is a relatively new concept, as traditional relational databases such as Oracle, PostgreSQL or Microsoft SQL Server were not originally designed to search large amounts of text data efficiently. However, extensions were introduced in the 1990s to support this functionality. Nowadays, many database systems offer integrated full-text search functions that make it possible to search large amounts of text efficiently.