How to build a secure LLM chatbot?

Introduction

We work with authorization, that is our bread and butter. LLMs are not. But recently we have been thinking a lot about how to build secure LLM chatbots, with authorization considerations in mind. This post won’t go in-depth into how to build one using Oso Cloud and list filtering, but instead serve as an introductory post to LLM chatbots, how they work, and best practices of building them. 

What are LLM chatbots? 

LLMs, or large language models, are advanced AI models trained on vast amounts of data to understand and generate human-like language by predicting the most likely sequence of words based on context. These models power a wide range of applications, including content creation, text completion, translation, sentiment analysis, and code generation.

One of the most common uses of LLMs is in chatbots—AI-powered tools that simulate human conversation. Chatbots are widely used for customer support, information retrieval, and task automation, interacting with users through text or voice.

LLMs and chatbots are inherently interconnected as LLMs serve as the underlying technology -natural language understanding (NLU) and generation (NLG) capabilities- that powers many modern chatbots (think ChatGPT, Claude, Gemini, etc.) When a user inputs a query, the chatbot sends it to the LLM, which processes the text, considers the context using attention mechanisms, retrieves relevant knowledge (potentially augmented by retrieval-augmented generation (RAG) or external APIs), and generates an appropriate response. 

This architecture allows chatbots to maintain conversation history and context across multiple exchanges. It enables them to understand complex queries, linguistic nuances, and provide accurate answers or perform tasks.

By leveraging LLMs, modern chatbots achieve greater fluency, coherence, and contextual awareness, making them highly effective for applications such as virtual assistants, interactive customer support, and personalized content generation.

Examples of LLM chatbots:

• ChatGPT (OpenAI): A general-purpose AI chatbot based on OpenAI's GPT architecture, designed for diverse text-based tasks like answering questions, generating content, and providing personalized assistance. Its core strength lies in its ability to handle a wide range of topics and generate human-like responses, utilizing the transformer-based GPT model for general-purpose language understanding and generation.
• Claude (Anthropic): An AI chatbot built with Anthropic’s focus on AI alignment and safety, designed to prioritize ethical considerations while providing helpful responses. Unlike other models, Claude emphasizes cautious, aligned behavior, ensuring responses adhere to safety guidelines and ethical standards.
• Google Gemini (formerly Bard): An AI chatbot integrated into Google’s search ecosystem, using Google’s advanced language models to assist with real-time, contextually relevant information. It differs from others by being deeply connected with Google Search, enabling seamless information retrieval within the search environment.
• Meta AI: An AI-powered chatbot embedded in Meta's social media platforms, such as Facebook and WhatsApp. It focuses on personalized responses, recommendations, and assistance, tailored to the social and messaging context, utilizing Meta's proprietary language models fine-tuned for these platforms.

How do LLM chatbots work?

LLMs leverage deep learning techniques, with a particular focus on transformer architecture, to process and generate text. These models are trained on vast and diverse datasets, including books, websites, and other sources, enabling them to learn patterns, grammar, contextual relationships, and semantic structures within language. At the core of their functionality is the attention mechanism, which allows the model to focus on relevant parts of the input text, understanding long-range dependencies and contextual nuances. This mechanism, along with other advanced techniques such as Retrieval-Augmented Generation (RAG), allows LLMs to generate highly coherent, context-aware responses by dynamically attending to relevant information both within the input and from external knowledge sources.

Attention

At the core of transformers is a mechanism called attention, which allows the model to weigh the importance of different words in a sentence relative to one another. This enables the model to capture long-range dependencies and understand the context of words beyond their immediate neighbors. Rather than processing words in isolation, attention helps the model grasp how each word interacts with others in the broader sentence structure.

Training an LLM involves processing billions of parameters (weights), which are adjusted over time to minimize prediction errors. The model is trained on large text datasets and learns to predict the next word in a sequence, refining its ability to generate coherent and contextually appropriate text as it processes more data.

When generating text, the model uses the patterns it has learned to predict the most likely continuation of a given prompt. Each potential next word is scored based on its relevance to the preceding words, with the model selecting the word that is most probable based on its learned context.

For example, in the sentence "The cat sat on the ___," the attention mechanism helps the model prioritize "mat" over less relevant words like "bus" or "computer," as it recognizes that "mat" fits the context and completes the sentence meaningfully.

Thus, LLMs process text by encoding input through layers of attention mechanisms, refining predictions through training, and generating responses by selecting the most probable next words based on the context they’ve learned.

Retrieval augmented generation

RAG (retrieval-augmented generation) is a technique that combines information retrieval with generative models. In a RAG system, when the model receives a prompt, it first retrieves relevant information from an external source (such as a document or database) using a search mechanism. It then uses a generative model (typically based on transformer architecture) to produce a response that incorporates both the original input and the retrieved information.

In RAG:

1. Retrieval: The model queries an external knowledge base or document corpus to find relevant text.
2. Generation: A transformer-based model, like GPT or BART, then generates a response based on both the original input and the retrieved information.

While the transformer architecture powers the generative part of RAG, RAG itself refers to the process of augmenting generation with external, retrieved knowledge. This enhances the model's ability to generate more informed, contextually accurate, and up-to-date responses, allowing it to draw on a broader range of information beyond its pre-trained knowledge.

For example, imagine a user asks an AI-powered assistant, "What are the latest advancements in quantum computing?" Without real-time internet access, a generative model alone might struggle to provide an up-to-date response. However, in a RAG system, the model could first query a relevant database of recent research papers or news articles about quantum computing. It would then generate a response incorporating the latest findings from those sources, ensuring a more informed and accurate answer.

Thus, RAG improves the model’s capability by augmenting its generative process with real-time, external knowledge, making it more effective at answering specific or dynamic queries.

How do LLM chatbots learn?

Unsupervised learning

LLMs learn through unsupervised learning, where they are trained on vast amounts of text data to predict the next word in a sequence. This training process involves feeding the model large, unlabeled datasets. Since there are no explicit human annotations, the model learns patterns, grammar, context, and relationships between words purely by identifying patterns and structures in the text.

At the core of the learning process is gradient descent, an optimization algorithm that adjusts the model's parameters (weights) over time to minimize prediction errors. The model starts with random weights and, as it processes each text input, it predicts the next word based on the context. When the prediction is incorrect, the model updates its weights to reduce future errors. This iterative process is repeated billions of times across massive datasets, enabling the model to generate coherent, contextually appropriate text.

During training, the model learns:

• Syntax: The rules governing the structure of language.
• Semantics: The meaning behind words and phrases.
• Context: How words interact with each other in different situations, allowing the model to handle long-range dependencies and nuances.

A practical example of unsupervised learning is customer segmentation in marketing. Suppose a company has a large dataset of customer behavior, but no predefined categories, like "high-value customers" or "frequent buyers." Using k-means clustering, an unsupervised algorithm, the model can automatically group customers based on their purchasing behaviors (e.g., high spenders, occasional buyers). These segments can then form the basis for more targeted marketing strategies.

Supervised fine-tuning

While LLMs are initially trained through unsupervised learning, they are often fine-tuned using supervised learning to enhance their performance on specific tasks. In this phase, the model is trained on a smaller, labeled dataset where the correct output (e.g., a classification label or answer to a question) is provided. This enables the model to learn more task-specific patterns and improve its accuracy for particular applications, such as translation or question-answering.

For instance, after performing unsupervised customer segmentation, the e-commerce company might want to classify customer reviews as positive, negative, or neutral to optimize email campaigns. The company can fine-tune a pre-trained model using a labeled dataset of customer reviews with sentiment tags. This fine-tuning allows the model to specialize in understanding sentiment, improving its ability to categorize future reviews specific to the company’s product offerings.

Transfer learning

LLMs also use transfer learning, where they apply the knowledge gained from one task to other, often unrelated tasks. This enables models to be effective in various scenarios without requiring explicit training for each specific task.

Building on the sentiment analysis example, suppose the company wants to expand its model’s capabilities to handle a broader range of customer queries, including customer service inquiries. Instead of starting from scratch, the company can leverage transfer learning to apply the sentiment analysis model’s understanding of customer sentiment to a new task, such as responding to customer inquiries.

For example, the company could adapt the fine-tuned sentiment model for a customer service chatbot. The chatbot can now understand the sentiment of customer feedback and respond intelligently based on that sentiment. If a customer expresses frustration about a late delivery, the model uses its knowledge of sentiment and context to generate a sympathetic and helpful response. This approach significantly enhances the chatbot's ability to assist customers without requiring new, extensive training.

Making an internal chatbot: the why’s and the how’s

The whys: Why build a chatbot internally?

1. Shortening sales cycles:
   An internal chatbot can automate the generation of personalized outbound emails by retrieving customer data from CRM systems, helping the sales team craft contextually relevant messages faster.
2. Improving customer support:
   The chatbot can integrate with existing knowledge bases and support ticket systems to provide fast, accurate responses to support teams, reducing response times and improving customer satisfaction.
3. Training new employees:
   Chatbots can automate onboarding by providing immediate access to internal documents, policies, and FAQs, helping new hires learn company processes efficiently.
4. Identifying documentation gaps:
   The chatbot can detect unanswered queries or requests for information not present in the current documentation, helping highlight areas that need updating.

The hows: How to build a chatbot?

1. Define scope and role-based access:
   Establish the chatbot’s purpose—whether for sales, support, or training—and implement role-based access control (RBAC) to ensure that users only get the information relevant to their roles. For instance, a support agent might have access to detailed internal resources, while a general employee can access more basic information.
2. Select technology and integration:
   Use transformer-based models (e.g., GPT) for natural language understanding and integrate with internal APIs, databases, and knowledge bases. Leverage RAG to fetch relevant, real-time information for responses.
3. Implement authorization mechanisms:
   Integrate the chatbot with an authorization service (like Oso) to enforce permissions based on user roles and attributes. This ensures sensitive company data, such as sales figures or employee performance, is only accessible to authorized personnel.
4. Training and fine-tuning:
   Train the chatbot with domain-specific data, integrating internal documents and knowledge bases. Fine-tune the model using supervised learning to ensure accurate responses in context, while also ensuring the model understands which information should be withheld based on user authorization.
5. Test and monitor:
   Regularly test the bot with internal users to identify performance issues and unauthorized access attempts. Continuously update the bot’s knowledge base and fine-tune the model to reflect changes in company processes, policies, and security protocols.

Four ways to build an LLM Chatbot: pros and cons

Pre-trained models and fine-tuning

A common approach for building an LLM-powered chatbot is to utilize pre-trained models such as GPT or BERT, and then fine-tune them on domain-specific datasets. Pre-trained models are trained on large-scale corpora, enabling them to process natural language efficiently. Fine-tuning these models involves training them on smaller, task-specific datasets to adapt them to a particular use case, such as customer support or internal documentation. This method allows the chatbot to leverage the extensive language capabilities of the pre-trained model while focusing on the specifics of the target domain. 

For example, our documentation site has a chatbot helper trained around our internal documentation, which can answer developers' questions about the product and its implementation.

However, fine-tuning can be resource-intensive, requiring substantial computational power and domain-specific data for optimal performance. It may also increase development time if the fine-tuning process requires continuous updates or data refinement.

Retrieval-augmented generation (RAG)

RAG combines the retrieval of relevant information from external sources with the generative capabilities of LLMs. In a RAG-based chatbot, when a user submits a query, the model first queries an external knowledge base or database for relevant information. The retrieved information is then passed to the generative model, which uses it to produce a response. This method improves the accuracy of responses by grounding them in real-time data, making it particularly useful for applications that require up-to-date information. The main advantage of RAG lies in its ability to generate contextually accurate responses without requiring extensive training on domain-specific datasets. 

For example, if Oso's internal chatbot utilizes a RAG based approach to answer developers' questions, it can query the latest internal documentation as well as the GitHub repositories before generating a response, to ensure developers receive the most up-to-date and contextually accurate answers about Oso’s authorization system. The integration of real-time data retrieval with the generative model would allow for more precise and tailored responses, and reduce the need for frequent updates or retraining of the chatbot. 

However, RAG systems can be complex to implement, as they require integrating external data sources and managing retrieval processes. Additionally, ensuring the quality and relevance of the data is essential for maintaining the effectiveness of the system.

Rule-based systems

Rule-based systems operate on predefined sets of rules, where a chatbot follows specific patterns or decision trees to generate responses. This approach is typically used for tasks with a defined scope, such as answering frequently asked questions or executing specific commands. Rule-based systems do not require large-scale training data, making them less resource-intensive than LLM-based systems. However, they are limited in handling unanticipated user inputs or complex conversations, as they can only respond according to the rules defined at the outset. Over time, they may require maintenance to stay aligned with evolving user needs or business requirements.

In Oso, an example of a rule-based chatbot might involve predefined queries like 'Show me the scope for this migration project' or 'Where in the code are we defining the attribute?' The chatbot would respond based on these fixed rules. While this approach is efficient for well-defined queries, it cannot address more complex or unexpected inputs, requiring either a referral to a human representative or updates to the rule set.

Hybrid approaches

Hybrid approaches combine the strengths of LLMs, RAG, and rule-based systems to address the limitations of each individual method. For example, a chatbot may use an LLM to process and understand natural language input, utilize RAG for retrieving relevant information from an external source, and employ rule-based logic for specific tasks or to enforce structured workflows. This approach allows the chatbot to handle both dynamic, context-sensitive queries and tasks requiring rigid, predefined responses. However, hybrid systems introduce complexity in design and maintenance, as they require integration across multiple components and ensure seamless interaction between the different methods.

API-driven solutions

API-driven solutions enable a chatbot to interact with external systems to retrieve data, trigger actions, or execute specific functions. This method is particularly useful for chatbots that need to access live data, perform transactions, or integrate with other software systems. For example, a chatbot could call an API to retrieve customer data or process a payment request. While API-driven solutions extend the chatbot’s capabilities beyond simple conversational tasks, they also introduce dependencies on external services. These dependencies can create potential points of failure if the external APIs experience downtime, have limited availability, or undergo changes that break compatibility with the chatbot. Additionally, using external APIs introduces security and data privacy considerations, particularly when dealing with sensitive or personal information.

Chatbot security risks and best practices

When building an LLM-powered chatbot, it's essential to address potential security risks that could compromise both the integrity of the chatbot and the privacy of sensitive data. One of the primary risks is hallucinations, where the LLM generates inaccurate or misleading information. This can be dangerous, especially in high-stakes contexts like customer support or internal documentation, where erroneous advice can lead to confusion or even financial loss.

Authorization plays a critical role in securing a chatbot, particularly when it comes to restricting access to sensitive information. Without proper access control, there's a significant risk that unauthorized users may obtain information they shouldn't have access to. 

For instance, a chatbot could inadvertently reveal an employee’s personal performance details if proper access restrictions aren’t in place. Similarly, confidential information, such as a coworker's family history, could be exposed if the chatbot lacks the necessary safeguards to prevent unauthorized queries. 

Other potential horror stories could include:

1. An employee searching for other employees' income information to compare with their own.
2. An employee searching for health insurance information but accidentally discovering a coworker's medical condition, such as breast cancer, and asking questions about it.
3. A team looking for a document outside of their department and inadvertently accessing sensitive financial data about the company.
4. An employee in a junior role attempting to retrieve confidential project details from a senior team, gaining access to strategic plans or confidential negotiations that they are not authorized to see.
5. A user searching for information about a specific company policy but accidentally receiving outdated or sensitive versions of the policy that have been superseded or classified.
6. A contractor accessing internal documents containing intellectual property (IP) or trade secrets that are not relevant to their temporary role.

To mitigate these risks, it's essential to implement authorization solutions, such as role-based access control (RBAC), relationship-based access control (ReBAC), or attribute-based access control (ABAC), with list filtering. The implementation involves creating a robust authorization framework that defines roles, permissions, and relationships to limit the scope of data accessible through the chatbot. This ensures that the chatbot only provides responses to users who are authorized to access that information. 

For example, sensitive financial or employee health data should only be accessible to users with specific clearance or within particular roles, such as HR or legal teams. Regular audits and updates to the authorization model should also be conducted to adapt to changing organizational structures and access needs.

Conclusion

In conclusion, building secure LLM chatbots requires a careful balance of advanced AI techniques and robust authorization mechanisms. By integrating retrieval-augmented generation, leveraging pre-trained models, and implementing strong role-based access controls, businesses can ensure that their chatbots are not only efficient but also secure, offering personalized and contextually accurate interactions.

Join us for the O’Reilly SuperStream: Retrieval-Augmented Generation in Production webinar to learn more on how to build your own LLM chatbot, and secure it using authorization.