The chatbot revolution in internal company search

What are LLMs – and why do they change the way we search for information?

Large language models (LLMs) are AI systems that have been trained with huge amounts of text data. They not only understand individual words, but also understand language in context, recognize meanings and provide independently formulated answers. In this way, they go far beyond traditional search engines: instead of mere lists of hits, they provide precise, context-related information. This capability is based on technological advances such as transformer architectures and semantic vectors (embeddings).

While earlier models such as LSTM were very limited, Google introduced a real LLM with language understanding into search for the first time in 2018 with BERT. With the GPT models from OpenAI – in particular GPT-3 and GPT-4 – this development reached a new level: AI can now process not only text but also images and search the internet in real time. Unsurprisingly, services such as GPT-4 or Microsoft Co-Pilot have been explicitly marketed as search tools since 2024. The idea behind this is that users ask questions in search engines, but instead of a static list of search results in the form of data sources, they receive generated texts that answer the user’s questions directly. Generative search is developing rapidly – with far-reaching changes that will have a lasting impact on internal company data and information management. Companies should keep an eye on these trends and think strategically.

The five disruptive trends in LLM-based search in 2025

1. the search becomes a conversation

The way we search is changing fundamentally – because the classic search bar is a thing of the past. “Search is no longer a search box with static lists of results – it’s a conversation” is how Christoph Wendl, CEO of Iphos IT Solutions GmbH, the manufacturer of the company’s internal search engine searchit to the point. “Instead of typing in individual keywords and hoping for a list of links, users today talk to their search engine – quite naturally, as if in a dialog.” Large language models, so-called LLMs, such as those used in Google Gemini, Microsoft Copilot or ChatGPT with browsing function, make it possible to follow up on questions, refine statements and obtain context-related answers. This not only makes searches more flexible, but also much more intelligent. On this basis, companies are building modern search systems – so-called search stacks – that use vector databases such as Pinecone or Chroma. In combination with open source LLMs such as Mistral or LLaMA, this results in powerful, scalable and, above all, secure and data protection-compliant search solutions, as implemented by the company’s internal search engine searchit.

2nd Retrieval Augmented Generation (RAG)

A key term in this development is Retrieval Augmented Generation – RAG for short. “The special thing about it is that the language model not only accesses its training data, but can also research in real time if required – for example on the internet or in internal company sources,” says Wendl. “This results in answers that not only draw on existing knowledge, but also on current, specific information.” New RAG technologies combine the best of both worlds: The accuracy of classic database searches meets the expressiveness and flexibility of generative AI. This technology can make all the difference, especially in complex areas such as justice, medicine or academia – by providing precise, fact-based information instead of generally generated statements.

3. personalized search assistants with memory

But it’s not just about search queries. Modern AI-supported search systems are increasingly adapting to the individual needs of their users – and even learning with them. Personalized search assistants with “memory” analyse who is searching, in what context and with what goal in mind. For example, a developer can automatically access current Jira tickets by simply entering “Show project status“, while a marketing team member can use the same command to view schedule data. The search adapts to the user – not the other way around. Even more impressive is the ability of these systems to remember past conversations. Anyone who has ever asked about “2024 performance metrics in marketing” can later follow up with, “What has changed since our last conversation?” The search query becomes part of an ongoing exchange – it becomes continuous.

4. search structured and unstructured data

However, the full potential of LLMs is revealed where structured and unstructured data come together. Traditional databases, such as those in SQL, are no longer the only sources. Today, it is also important to be able to search PDFs, emails, chat histories, images or even code documents in very different sources such as network hard disks or websites – all at the same time. A modern query such as “What do customers say about our support?” is no longer answered separately – with figures here, statements there – but provides an integrated answer: with key figures, sentiments and quotes in one report. The development towards multimodal search is particularly exciting: LLMs can now capture and process images, text, audio or even diagrams simultaneously. The image information is translated into descriptive texts by specialized modules. Christoph Wendl cites customer examples such as “Find this logo in all design files” or “What does this graphic show?” This makes the search not only linguistically but also visually accessible – a real game changer.

5. autonomous search agents

Another innovative step is autonomous search agents – a highly topical trend in the LLM environment. Here, the search becomes an action: AI-supported agents independently plan multi-stage tasks, access tools such as Notion, Slack or internal databases, collect information, summarize it and deliver complete reports on request. The technology therefore goes far beyond simple search queries. The agent acts independently – on the basis of natural language. In this context, the term “natural language BI” is also used – i.e. business intelligence through natural language. The answers are not only presented in an understandable text format, but also include automatically generated diagrams, trend analyses and summaries. This is exactly where platforms such as Tableau Pulse or Microsoft Fabric come in – and offer companies completely new possibilities for data-based decision-making.

“In short: today’s search is no longer what it used to be. It talks, thinks, remembers – and acts “, summarizes Wendl. “Companies that miss out on this development run the risk of being left behind. However, those who get to grips with LLMs, RAGs and intelligent agents at an early stage will gain access to a powerful tool for efficiency, knowledge transfer and data-based future strategies. “