App: Local large language model (llm2)

目录

The llm2 app is one of the apps that provide text processing functionality using Large language models in Nextcloud and act as a text processing backend for the Nextcloud Assistant app, the mail app and other apps making use of the core Text Processing API. The llm2 app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.

This app uses llama.cpp under the hood and is thus compatible with any model in gguf format.

However, we primarily test with the models listed below. Output quality will differ depending on which model you use and downstream tasks like summarization or Context Chat may not work on other models. We thus recommend the following models:

Multimodal chat

With a multimodal-capable model, llm2 supports multimodal chat in the Nextcloud Assistant app: users can attach images (and with Gemma 4, also audio) to chat messages so the model can reason about their content. Text files can also be attached as context.

The following shipped models support multimodality:

Vision-capable models also register the Analyze images and OCR task types in addition to the usual text-processing tasks.

Multimodal models need a matching multimodal projector (mmproj) gguf file next to the model file. The projector files for the shipped Qwen 3.5 and Gemma 4 models are downloaded automatically with the models. If you supply your own multimodal model, place its mmproj file in /nc_app_llm2_data as well and set mmproj_path in the model's JSON configuration (see Configuring alternate models below).

Multilinguality

This app supports input and output in languages other than English if the underlying model supports the language.

Qwen 3.5 supports the following 201 languages: <https://qwen.ai/blog?id=qwen3.5>

Gemma 4 provides multilingual support, offering high-quality performance in 35 languages and basic support for 140 languages.

Llama 3.1 supports the following languages:

Note that other languages may work as well, but only the above languages are guaranteed to work with Llama 3.1.

Requirements

Installation

  1. Make sure the Nextcloud Assistant app is installed
  2. Install AppAPI and setup a Deploy Demon
  3. Install the "Local large language model" ExApp via the "Apps" page in the Nextcloud web admin user interface

Supplying alternate models

This app allows supplying alternate LLM models as gguf files in the /nc_app_llm2_data directory of the docker container.

  1. Download a gguf model e.g. from huggingface
  2. Copy the gguf file to /nc_app_llm2_data inside the docker container
  3. Restart the llm2 ExApp
  4. Select the new model in the Nextcloud AI admin settings

Configuring alternate models

Since every model requires slightly different inference parameters, you can pass along a configuration file for the alternate model files you supply.

The configuration file for a model file must have the same name as the model file but must end in .json instead of .gguf.

The strings {system_prompt} and {user_prompt} are variables that will be filled in by the app, so they must be part of your prompt template.

Here is an example config file for Llama 2:

{
  "prompt": "<|im_start|> system\n{system_prompt}\n<|im_end|>\n<|im_start|> user\n{user_prompt}\n<|im_end|>\n<|im_start|> assistant\n",
  "loader_config": {
     "n_ctx": 4096,
     "max_tokens": 2048,
     "stop": ["<|im_end|>"]
  }
}

Here is an example configuration for Llama 3:

{
  "prompt": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n{user_prompt}<|eot_id|>\n<|start_header_id|>assistant<|end_header_id|>\n",
  "loader_config": {
      "n_ctx": 8000,
      "max_tokens": 4000,
      "stop": ["<|eot_id|>"],
      "temperature": 0.3
  }
}

Scaling

It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 1000 user requests per hour. However, this number is based on theory and we do appreciate real-world feedback on this. If you would like to scale up your language model usage, we recommend using an AI as a Service provider or hosting a service compatible with the OpenAI API yourself that can be scaled up and connecting nextcloud to it via the integration_openai app.

App store

You can also find the app in our app store, where you can write a review: https://apps.nextcloud.com/apps/llm2

Repository

You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: https://github.com/nextcloud/llm2

Nextcloud customers should file bugs directly with our Support system.

Known Limitations

Addendum: Running with a fully open model

If you would like to use a fully open model that scores a green score on our Ethical AI rating, we recommend the following model:

What makes OLMo a fully open model?

Limitations

版本号 #3
由 cocobook-sync 创建于 2026-10-28 16:28:49 UTC
由 cocobook-sync 更新于 2026-10-29 01:09:22 UTC