Created RAG task list

This commit is contained in:
Thorsten Sommer 2024-11-03 21:38:22 +01:00
parent 3ccb097705
commit d1c4cb499a
Signed by: tsommer
GPG Key ID: 371BBA77A02C0108

View File

@ -5,16 +5,16 @@ Things we are currently working on:
- Since November 2024: Work on RAG (integration of your data and files) has begun. We will support the integration of local and external data sources. We need to implement the following runtime (Rust) and app (.NET) steps: - Since November 2024: Work on RAG (integration of your data and files) has begun. We will support the integration of local and external data sources. We need to implement the following runtime (Rust) and app (.NET) steps:
- Runtime: Restructuring the code into meaningful modules - [ ] Runtime: Restructuring the code into meaningful modules
- App: Metadata for providers (which provider offers embeddings?) - [ ] App: Metadata for providers (which provider offers embeddings?)
- App: Management of data sources (local data) - [ ] App: Management of data sources (local data)
- Runtime: Integration of the vector database [LanceDB](https://github.com/lancedb/lancedb) - [ ] Runtime: Integration of the vector database [LanceDB](https://github.com/lancedb/lancedb)
- Runtime: Extract data from txt / md / pdf / docx / xlsx files - [ ] Runtime: Extract data from txt / md / pdf / docx / xlsx files
- App: Define an interface for the integration of RAG processes in chats - [ ] App: Define an interface for the integration of RAG processes in chats
- Define the [External Data API (EDI)](https://github.com/MindWorkAI/EDI) as a contract for integrating arbitrary external data (PR [#1](https://github.com/MindWorkAI/EDI/pull/1)) - [ ] Define the [External Data API (EDI)](https://github.com/MindWorkAI/EDI) as a contract for integrating arbitrary external data (PR [#1](https://github.com/MindWorkAI/EDI/pull/1))
- App: Implement the process control of vectorizing local data - [ ] App: Implement the process control of vectorizing local data
- App: Integrate data sources in chats - [ ] App: Integrate data sources in chats
- App: Management of data sources (external data via [EDI](https://github.com/MindWorkAI/EDI)) - [ ] App: Management of data sources (external data via [EDI](https://github.com/MindWorkAI/EDI))
- Since September 2024: Experiments have been started on how we can work on long texts with AI Studio. Let's say you want to write a fantasy novel or create a complex project proposal and use LLM for support. The initial experiments were promising, but not yet satisfactory. We are testing further approaches until a satisfactory solution is found. Related PR: [#167](https://github.com/MindWorkAI/AI-Studio/pull/167). - Since September 2024: Experiments have been started on how we can work on long texts with AI Studio. Let's say you want to write a fantasy novel or create a complex project proposal and use LLM for support. The initial experiments were promising, but not yet satisfactory. We are testing further approaches until a satisfactory solution is found. Related PR: [#167](https://github.com/MindWorkAI/AI-Studio/pull/167).