|
30 | 30 | - **`tldw_chatbook/Widgets/`**: Reusable UI components used across different screens.
|
31 | 31 | - **`tldw_chatbook/Config.py`**: Contains all configuration settings for the application, including API keys, database paths, and other settings.
|
32 | 32 | - **`tldw_chatbook/Constants.py`**: Contains all constants used throughout the application, such as default values and error messages.
|
33 |
| -- **`tldw_chatbook/Logging_Config.py`**: Contains the logging configuration for the application, setting up loggers, handlers, and formatters. |
| 33 | +- **`tldw_chatbook/Logging_Config.py`**: Contains the logging configuration for the application, setting up loggers, handlers, and formatters. |
| 34 | + |
| 35 | + |
| 36 | + |
| 37 | + |
| 38 | + |
| 39 | +## LLM Backend Integrations |
| 40 | + |
| 41 | +This section details the various Large Language Model (LLM) inference backends integrated into `tldw_chatbook`. |
| 42 | + |
| 43 | +### Llama.cpp Integration |
| 44 | +- |
| 45 | + |
| 46 | +### Llamafile Integration |
| 47 | + |
| 48 | +### Ollama Integration |
| 49 | + |
| 50 | +### vLLM Integration |
| 51 | + |
| 52 | +### Transformers Integration |
| 53 | + |
| 54 | +### ONNX Runtime Integration |
| 55 | + |
| 56 | +### MLX-LM Integration |
| 57 | +- https://github.com/ml-explore/mlx-lm/tree/main |
| 58 | + |
| 59 | +The application now supports MLX-LM for running local language models optimized for Apple Silicon hardware. |
| 60 | +Users can manage MLX-LM instances via the "LLM Management" tab, allowing configuration of: |
| 61 | + |
| 62 | +* **Model Path**: Specify a HuggingFace model ID compatible with MLX or a path to a local MLX model. |
| 63 | +* **Server Host & Port**: Configure the network address for the MLX-LM server. |
| 64 | +* **Additional Arguments**: Pass extra command-line arguments to the `mlx_lm.server` process. |
| 65 | + |
| 66 | +The integration starts a local `mlx_lm.server` process and interacts with it, assuming an OpenAI-compatible API endpoint (typically at `/v1`). This allows for efficient local inference leveraging MLX's performance benefits on supported hardware. |
0 commit comments