# Adding AI features to existing software URL: https://brainpowergroup.eu/en/blog/adding-ai-features-to-existing-software Date: 2026-08-12 By: Barry Willems, Brainpower Group Start with one narrow LLM feature on data you already have, test it on fifty to a hundred real examples and track cost and privacy per feature. Start with one narrow feature that works on data you already have, measure whether it is good enough, and only then expand. Calling the language model is the easy part. The work is in retrieving the right data, respecting user permissions, testing the output and keeping an eye on cost. ## Pick a narrow first feature Good first features share two traits: the input already exists in your system, and a person looks at the output before anything happens. Think of summarising a ticket for a support agent, drafting a reply from your knowledge base or labelling incoming documents. In the first version, do not let the model send, change or delete anything on its own. A draft that someone approves already saves time, and it does little damage when it is wrong. ## With RAG the model uses your data A language model knows nothing about your customers, products or contracts. With retrieval augmented generation (RAG) you first look up the relevant text and include it in the prompt. You usually do not need a separate vector database for this. The open source pgvector extension adds vector search to Postgres, with HNSW and IVFFlat indexes. If you already run Postgres, your embeddings live next to the rest of your data. Aleksa explains that choice in [Postgres as the default database](/en/blog/postgres-as-the-default-database). Three things I would get right from day one: - Combine vector search with plain keyword search. Full-text search finds product codes, names and numbers better than embeddings do. - Filter on permissions at retrieval time. A user must never see text through the model that they cannot open in the application itself. - Test retrieval separately from the answer. If the system does not find the right passages, no prompt will fix that. ## With tool use the model takes actions With tool use the model calls functions you define, such as "look up order" or "create draft invoice". Your existing API is the best starting point, because validation and authorisation already live there. The rules I use: - Start with read-only tools. - A tool always runs with the permissions of the logged-in user, never with a shared service account. - Have a person confirm any action that writes data. - Log every call with its input, output and user. If your main application runs on .NET or PHP and you want to build the AI logic in Python, you can run it as a separate service. How to set that up is covered in [Python and FastAPI for AI backends](/en/blog/python-and-fastapi-for-ai-backends). ## Evaluate before you go live Without a test set you cannot tell whether an LLM feature works well. Collect fifty to a hundred real examples from your own data, each with the answer you expect. Run that set on every change to the prompt, the retrieval or the model. Measure separately whether retrieval found the right passages, whether the answer is correct and whether the model declines cleanly when it does not know. A second model can grade answers, but check a sample by hand every time. In production, log the output and let users give a thumbs up or down. Those examples go back into your test set. ## Cost per feature You pay per token, for input and output. With RAG the input grows quickly, since every question carries a few pages of context. So work out per feature what one call costs and how often a user triggers it per day. Prompt caching helps when a large part of the prompt stays the same, such as instructions or a fixed document. At Anthropic, input read from the cache costs a tenth of the normal input price by default. Writing to the cache costs a quarter more than normal input. The cache lives for five minutes by default and is refreshed each time it is used. Beyond that, use a smaller model for simple tasks such as labelling, set a budget per customer or user, and track cost per feature the way you track the rest of your cloud bill. ## Privacy and the AI Act If you send personal data to a model provider, that provider is a processor under the GDPR. You need a data processing agreement, and you want to know where the data is processed and how long it is kept. OpenAI does not use API data for training by default. It keeps abuse monitoring logs for up to 30 days, and API projects can have their data stored and processed in Europe, subject to conditions. Anthropic does not use inputs and outputs from its commercial products, including the API, for training by default. Send no more than you need. If a summary does not need a national ID number or an email address, strip it out first. API keys for model providers belong in your secrets manager, not in the code. Boris covers that in [managing secrets and API keys safely](/en/blog/managing-secrets-and-api-keys-safely). Since 2 August 2026 the transparency rules of the EU AI Act apply. When a user interacts directly with an AI system, such as a chatbot, it has to be clear that it is AI. The AI Omnibus, in force since 27 July 2026, mainly postponed the rules for high-risk systems. Have a lawyer check your own situation. ## Sources - [pgvector on GitHub](https://github.com/pgvector/pgvector) - [Anthropic: prompt caching](https://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching) - [OpenAI: data controls in the OpenAI platform](https://developers.openai.com/api/docs/guides/your-data) - [Anthropic Privacy Center: is my data used for model training?](https://privacy.claude.com/en/articles/7996868-is-my-data-used-for-training) - [Houthoff: AI Act transparency obligations and the AI Omnibus Regulation](https://www.houthoff.com/insights/news/ai-act-transparency-obligations-and-the-ai-omnibus-regulation/) ## Frequently asked questions **What is RAG and do I need it?** With RAG you first retrieve the relevant pieces of your own data and pass them to the language model, so the answer is based on your information. You need it as soon as the model has to know something specific to your company or customers. **Do OpenAI and Anthropic use my API data to train their models?** Not by default. OpenAI does not use data sent through its API for training unless you opt in. Anthropic does not use inputs and outputs from its commercial products, including the API, for training by default either. **Do I have to tell users they are talking to AI?** Yes, if they interact directly with an AI system such as a chatbot. That transparency obligation under the EU AI Act has applied since 2 August 2026. Have a lawyer check the details for your case. --- Source: https://brainpowergroup.eu/en/blog/adding-ai-features-to-existing-software Brainpower Group, contact: https://brainpowergroup.eu/en/contact, hello@brainpowergroup.eu