4.7.1-alpha (#1120)

Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>
This commit is contained in:
Archer
2024-04-03 18:14:09 +08:00
committed by GitHub
parent 9ae581e09b
commit 8a46372418
76 changed files with 3129 additions and 2104 deletions

View File

@@ -60,7 +60,7 @@ fastgpt.run 域名会弃用。
- [x] 混合检索 & 重排
- [x] Tool 模块
- [ ] 嵌入 [Laf](https://github.com/labring/laf),实现在线编写 HTTP 模块
- [ ] 插件封装功能
- [ ] 插件封装功能,支持低代码渲染
`2` 知识库能力
- [x] 多库复用,混用
@@ -68,9 +68,8 @@ fastgpt.run 域名会弃用。
- [x] 支持知识库单独设置向量模型
- [x] 源文件存储
- [x] 支持手动输入直接分段QA 拆分导入
- [x] 支持 pdfdocxtxthtmlmdcsv
- [x] 支持。txt 。md 。htmlpdfdocxpptx 。csv 。xlsx (有需要更多可 PR file loader)
- [x] 支持 url 读取、CSV 批量导入
- [ ] 支持 PPT、Excel 导入
- [ ] 支持文件阅读器
- [ ] 更多的数据预处理方案
@@ -114,7 +113,7 @@ fastgpt.run 域名会弃用。
* [多模型配置](https://doc.fastgpt.in/docs/development/one-api/)
* [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading)
* [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/)
* [知识库结构详解](https://doc.fastgpt.in/docs/course/datasetengine/)
* [知识库结构详解](https://doc.fastgpt.in/docs/course/dataset_engine/)
<a href="#readme">
<img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">

View File

@@ -64,7 +64,7 @@ Tips: 可以通过点击上下文按键查看完整的上下文组成,便于
FastGPT 知识库采用 QA 对(不一定都是问答格式,仅代表两个变量)的格式存储,在转义成字符串时候会根据**引用模板**来进行格式化。知识库包含多个可用变量: q, a, sourceId数据的ID, index(第n个数据), source(数据的集合名、文件名)score(距离得分0-1) 可以通过 {{q}} {{a}} {{sourceId}} {{index}} {{source}} {{score}} 按需引入。下面一个模板例子:
可以通过 [知识库结构讲解](/docs/course/datasetEngine/) 了解详细的知识库的结构。
可以通过 [知识库结构讲解](/docs/course/dataset_engine/) 了解详细的知识库的结构。
#### 引用模板

View File

@@ -1,93 +0,0 @@
---
title: "知识库结构讲解"
description: "本节会详细介绍 FastGPT 知识库结构设计,理解其 QA 的存储格式和多向量映射,以便更好的构建知识库。这篇介绍主要以使用为主,详细原理不多介绍。"
icon: "dataset"
draft: false
toc: true
weight: 102
---
## 理解向量
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库,要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
人类的文字、图片、视频等媒介是无法直接被计算机理解的,要想让计算机理解两段文字是否有相似性、相关性,通常需要将它们转成计算机可以理解的语言,向量是其中的一种方式。
向量可以简单理解为一个数字数组,两个向量之间可以通过数学公式得出一个`距离`,距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上,可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
而由于文字是有多种类型,并且拥有成千上万种组合方式,因此在转成向量进行相似度匹配时,很难保障其精确性。在向量方案构建的知识库中,通常使用`topk`召回的方式,也就是查找前`k`个最相似的内容,丢给大模型去做更进一步的`语义判断``逻辑推理``归纳总结`,从而实现知识库问答。因此,在知识库问答中,向量搜索的环节是最为重要的。
影响向量搜索精度的因素非常多,主要包括:向量模型的质量、数据的质量(长度,完整性,多样性)、检索器的精度(速度与精度之间的取舍)。与数据质量对应的就是检索词的质量。
检索器的精度比较容易解决,向量模型的训练略复杂,因此数据和检索词质量优化成了一个重要的环节。
## FastGPT 中向量的结构设计
FastGPT 采用了 `PostgresSQL``PG Vector` 插件作为向量检索器,索引为`HNSW`。且`PostgresSQL`仅用于向量检索,`MongoDB`用于其他数据的存取。
`MongoDB``dataset.datas`表中,会存储向量原数据的信息,同时有一个`indexes`字段会记录其对应的向量ID这是一个数组也就是说一组向量可以对应多组数据。
`PostgresSQL`的表中,设置一个 `index` 字段用于存储向量。在检索时会先召回向量再根据向量的ID`MongoDB`中寻找原数据内容,如果对应了同一组原数据,则进行合并,向量得分取最高得分。
![](/imgs/datasetSetting1.png)
### 多向量的目的和使用方式
在一组向量中内容的长度和语义的丰富度通常是矛盾的无法兼得。因此FastGPT 采用了多向量映射的方式,将一组数据映射到多组向量中,从而保障数据的完整性和语义的丰富度。
你可以为一组较长的文本,添加多组向量,从而在检索时,只要其中一组向量被检索到,该数据也将被召回。
### 提高向量搜索精度的方法
1. 更好分词分段:当一段话的结构和语义是完整的,并且是单一的,精度也会提高。因此,许多系统都会优化分词器,尽可能的保障每组数据的完整性。
2. 精简`index`的内容,减少向量内容的长度:当`index`的内容更少,更准确时,检索精度自然会提高。但与此同时,会牺牲一定的检索范围,适合答案较为严格的场景。
3. 丰富`index`的数量,可以为同一个`chunk`内容增加多组`index`
4. 优化检索词:在实际使用过程中,用户的问题通常是模糊的或是缺失的,并不一定是完整清晰的问题。因此优化用户的问题(检索词)很大程度上也可以提高精度。
5. 微调向量模型:由于市面上直接使用的向量模型都是通用型模型,在特定领域的检索精度并不高,因此微调向量模型可以很大程度上提高专业领域的检索效果。
## FastGPT 构建知识库方案
在 FastGPT 中,整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`,一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`,也就是说,知识库搜索时,是对整个`库`进行搜索,而集合仅是为了对数据进行分类管理,与搜索效果无关。(起码目前还是)
| 库 | 集合 | 数据 |
| --- | --- | --- |
| ![](/imgs/datasetEngine1.jpg) | ![](/imgs/datasetEngine2.jpg) | ![](/imgs/datasetEngine3.webp) |
### 导入数据方案1 - 直接分段导入
选择文件导入时,可以选择直接分段方案。直接分段会利用`句子分词器`对文本进行一定长度拆分,最终分割中多组的`q`。如果使用了直接分段方案,我们建议在`应用`设置`引用提示词`时,使用`通用模板`即可,无需选择`问答模板`
| 交互 | 结果 |
| --- | --- |
| ![](/imgs/datasetEngine4.webp) | ![](/imgs/datasetEngine5.webp) |
### 导入数据方案2 - QA导入
选择文件导入时可以选择QA拆分方案。仍然需要使用到`句子分词器`对文本进行拆分,但长度比直接分段大很多。在导入后,会先调用`大模型`对分段进行学习,并给出一些`问题``答案`,最终问题和答案会一起被存储到`q`中。注意,新版的 FastGPT 为了提高搜索的范围,不再将问题和答案分别存储到 qa 中。
| 交互 | 结果 |
| --- | --- |
| ![](/imgs/datasetEngine6.webp) | ![](/imgs/datasetEngine7.webp) |
### 导入数据方案3 - 手动录入
在 FastGPT 中,你可以在任何一个`集合`中点击右上角的`插入`手动录入知识点,或者使用`标注`功能手动录入。被搜索的内容为`q`,补充内容(可选)为`a`
| | | |
| --- | --- | --- |
| ![](/imgs/datasetEngine8.jpg) | ![](/imgs/datasetEngine9.jpg) | ![](/imgs/datasetEngine10.jpg) |
### 导入数据方案4 - CSV录入
有些数据较为独特,可能需要单独的进行预处理分割后再导入 FastGPT此时可以选择 csv 导入,可批量的将处理好的数据导入。
![](/imgs/datasetEngine11.jpg)
### 导入数据方案5 - API导入
参考[FastGPT OpenAPI使用](/docs/development/openapi)。
## QA的组合与引用提示词构建
参考[引用模板与引用提示词示例](/docs/course/ai_settings/#示例)

View File

@@ -0,0 +1,136 @@
---
title: '知识库搜索方案和参数'
description: '本节会详细介绍 FastGPT 知识库结构设计,理解其 QA 的存储格式和多向量映射,以便更好的构建知识库。同时会介绍每个搜索参数的功能。这篇介绍主要以使用为主,详细原理不多介绍。'
icon: 'language'
draft: false
toc: true
weight: 106
---
## 理解向量
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库,要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
人类的文字、图片、视频等媒介是无法直接被计算机理解的,要想让计算机理解两段文字是否有相似性、相关性,通常需要将它们转成计算机可以理解的语言,向量是其中的一种方式。
向量可以简单理解为一个数字数组,两个向量之间可以通过数学公式得出一个`距离`,距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上,可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
而由于文字是有多种类型,并且拥有成千上万种组合方式,因此在转成向量进行相似度匹配时,很难保障其精确性。在向量方案构建的知识库中,通常使用`topk`召回的方式,也就是查找前`k`个最相似的内容,丢给大模型去做更进一步的`语义判断``逻辑推理``归纳总结`,从而实现知识库问答。因此,在知识库问答中,向量搜索的环节是最为重要的。
影响向量搜索精度的因素非常多,主要包括:向量模型的质量、数据的质量(长度,完整性,多样性)、检索器的精度(速度与精度之间的取舍)。与数据质量对应的就是检索词的质量。
检索器的精度比较容易解决,向量模型的训练略复杂,因此数据和检索词质量优化成了一个重要的环节。
### 提高向量搜索精度的方法
1. 更好分词分段:当一段话的结构和语义是完整的,并且是单一的,精度也会提高。因此,许多系统都会优化分词器,尽可能的保障每组数据的完整性。
2. 精简`index`的内容,减少向量内容的长度:当`index`的内容更少,更准确时,检索精度自然会提高。但与此同时,会牺牲一定的检索范围,适合答案较为严格的场景。
3. 丰富`index`的数量,可以为同一个`chunk`内容增加多组`index`
4. 优化检索词:在实际使用过程中,用户的问题通常是模糊的或是缺失的,并不一定是完整清晰的问题。因此优化用户的问题(检索词)很大程度上也可以提高精度。
5. 微调向量模型:由于市面上直接使用的向量模型都是通用型模型,在特定领域的检索精度并不高,因此微调向量模型可以很大程度上提高专业领域的检索效果。
## FastGPT 构建知识库方案
### 数据存储结构
在 FastGPT 中,整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`,一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`,也就是说,知识库搜索时,是对整个`库`进行搜索,而集合仅是为了对数据进行分类管理,与搜索效果无关。(起码目前还是)
![](/imgs/dataset_tree.png)
### 向量存储结构
FastGPT 采用了`PostgresSQL``PG Vector`插件作为向量检索器,索引为`HNSW`。且`PostgresSQL`仅用于向量检索(该引擎可以替换成其它数据库),`MongoDB`用于其他数据的存取。
`MongoDB``dataset.datas`表中,会存储向量原数据的信息,同时有一个`indexes`字段会记录其对应的向量ID这是一个数组也就是说一组向量可以对应多组数据。
`PostgresSQL`的表中,设置一个`vector`字段用于存储向量。在检索时会先召回向量再根据向量的ID`MongoDB`中寻找原数据内容,如果对应了同一组原数据,则进行合并,向量得分取最高得分。
![](/imgs/datasetSetting1.png)
### 多向量的目的和使用方式
在一组向量中内容的长度和语义的丰富度通常是矛盾的无法兼得。因此FastGPT 采用了多向量映射的方式,将一组数据映射到多组向量中,从而保障数据的完整性和语义的丰富度。
你可以为一组较长的文本,添加多组向量,从而在检索时,只要其中一组向量被检索到,该数据也将被召回。
意味着,你可以通过标注数据块的方式,不断提高数据块的精度。
### 检索方案
1. 通过`问题优化`实现指代消除和问题扩展,从而增加连续对话的检索能力以及语义丰富度。
2. 通过`Concat query`来增加`Rerank`连续对话的时,排序的准确性。
3. 通过`RRF`合并方式,综合多个渠道的检索效果。
4. 通过`Rerank`来二次排序,提高精度。
![](/imgs/dataset_search_process.png)
## 搜索参数
| | | |
| --- |---| --- |
|![](/imgs/dataset_search_params1.png)| ![](/imgs/dataset_search_params2.png) | ![](/imgs/dataset_search_params3.png) |
### 搜索模式
#### 语义检索
语义检索是通过向量距离,计算用户问题与知识库内容的距离,从而得出“相似度”,当然这并不是语文上的相似度,而是数学上的。
优点:
- 相近语义理解
- 跨多语言理解(例如输入中文问题匹配英文知识点)
- 多模态理解(文本,图片,音视频等)
缺点:
- 依赖模型训练效果
- 精度不稳定
- 受关键词和句子完整度影响
#### 全文检索
采用传统的全文检索方式。适合查找关键的主谓语等。
#### 混合检索
同时使用向量检索和全文检索,并通过 RRF 公式进行两个搜索结果合并,一般情况下搜索结果会更加丰富准确。
由于混合检索后的查找范围很大,并且无法直接进行相似度过滤,通常需要进行利用重排模型进行一次结果重新排序,并利用重排的得分进行过滤。
#### 结果重排
利用`ReRank`模型对搜索结果进行重排,绝大多数情况下,可以有效提高搜索结果的准确率。不过,重排模型与问题的完整度(主谓语齐全)有一些关系,通常会先走问题优化后再进行搜索-重排。重排后可以得到一个`0-1`的得分,代表着搜索内容与问题的相关度,该分数通常比向量的得分更加精确,可以根据得分进行过滤。
FastGPT 会使用 `RRF` 对重排结果、向量搜索结果、全文检索结果进行合并,得到最终的搜索结果。
### 搜索过滤
#### 引用上限
每次搜索最多引用`n``tokens`的内容。
之所以不采用`top k`,是发现在混合知识库(问答库、文档库)时,不同`chunk`的长度差距很大,会导致`top k`的结果不稳定,因此采用了`tokens`的方式进行引用上限的控制。
#### 最低相关度
一个`0-1`的数值,会过滤掉一些低相关度的搜索结果。
该值仅在`语义检索`或使用`结果重排`时生效。
### 问题优化
#### 背景
在 RAG 中,我们需要根据输入的问题去数据库里执行 embedding 搜索,查找相关的内容,从而查找到相似的内容(简称知识库搜索)。
在搜索的过程中,尤其是连续对话的搜索,我们通常会发现后续的问题难以搜索到合适的内容,其中一个原因是知识库搜索只会使用“当前”的问题去执行。看下面的例子:
![](/imgs/coreferenceResolution2.webp)
用户在提问“第二点是什么”的时候只会去知识库里查找“第二点是什么”压根查不到内容。实际上需要查询的是“QA结构是什么”。因此我们需要引入一个【问题优化】模块来对用户当前的问题进行补全从而使得知识库搜索能够搜索到合适的内容。使用补全后效果如下
![](/imgs/coreferenceResolution3.webp)
#### 实现方式
在进行`数据检索`前,会先让模型进行`指代消除``问题扩展`,一方面可以可以解决指代对象不明确问题,同时可以扩展问题的语义丰富度。你可以通过每次对话后的对话详情,查看补全的结果。

View File

@@ -19,6 +19,9 @@ llm模型全部合并
```json
{
"feConfigs": {
"lafEnv": "https://laf.dev" // laf环境
},
"systemEnv": {
"vectorMaxProcess": 15,
"qaMaxProcess": 15,
@@ -164,7 +167,7 @@ llm模型全部合并
"model": "bge-reranker-base", // 随意
"name": "检索重排-base", // 随意
"charsPointsPrice": 0,
"requestUrl": "{{host}}/api/v1/rerank",
"requestUrl": "{{host}}/v1/rerank",
"requestAuth": "安全凭证,已自动补 Bearer"
}
]

View File

@@ -44,7 +44,7 @@ weight: 910
### docker 部署
+ 镜像名: `luanshaotong/reranker:v0.2`
+ 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
+ 端口号: 6006
+ 大小约8GB
@@ -56,12 +56,12 @@ ACCESS_TOKEN=mytoken
**运行命令示例**
- 无需GPU环境使用CPU运行
```sh
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
```
- 需要CUDA 11.7环境
```sh
docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
```
**docker-compose.yml示例**
@@ -69,7 +69,7 @@ docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken lu
version: "3"
services:
reranker:
image: luanshaotong/reranker:v0.2
image: registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
container_name: reranker
# GPU运行环境如果宿主机未安装将deploy配置隐藏即可
deploy:

View File

@@ -1,5 +1,5 @@
---
title: 'V4.7(进行中)'
title: 'V4.7'
description: 'FastGPT V4.7更新说明'
icon: 'upgrade'
draft: false
@@ -26,7 +26,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv47' \
## 3. 升级 ReRank 模型
4.7对ReRank模型进行了格式变动兼容 cohere 的格式,可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型,需要修改镜像为:`luanshaotong/reranker:v0.2`
4.7对ReRank模型进行了格式变动兼容 cohere 的格式,可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型,需要修改镜像为:`registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
cohere的重排模型对中文不是很好感觉不如 bge 的好用,接入教程如下:

View File

@@ -0,0 +1,28 @@
---
title: 'V4.7.1'
description: 'FastGPT V4.7.1 更新说明'
icon: 'upgrade'
draft: false
toc: true
weight: 825
---
## 初始化脚本
从任意终端,发起 1 个 HTTP 请求。其中 {{rootkey}} 替换成环境变量里的 `rootkey`{{host}} 替换成FastGPT的域名。
```bash
curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \
--header 'rootkey: {{rootkey}}' \
--header 'Content-Type: application/json'
```
该请求会执行脏数据清理(清理无效的文件、清理无效的图片、清理无效的知识库集合、清理无效的向量)
## V4.7.1 更新说明
1. 新增 - Pptx 和 xlsx 文件读取。但所有文件读取都放服务端,会消耗更多的服务器资源,以及无法在上传时预览更多内容。
2. 新增 - 集成 Laf 云函数,可以读取 Laf 账号中的云函数作为 HTTP 模块。
3. 新增 - 定时器清理垃圾数据。采用小范围清理会清理最近n个小时的所以请保证服务持续运行长时间不允许可以继续执行 clearInvalidData 的接口进行全量清理。)
4. 修改 - csv导入模板取消 header 校验,自动获取前两列。
5. 修复 - 工具调用模块连线数据类型校验错误。

View File

@@ -1,5 +1,7 @@
export const fileImgs = [
{ suffix: 'pdf', src: 'file/fill/pdf' },
{ suffix: 'ppt', src: 'file/fill/ppt' },
{ suffix: 'xlsx', src: 'file/fill/xlsx' },
{ suffix: 'csv', src: 'file/fill/csv' },
{ suffix: '(doc|docs)', src: 'file/fill/doc' },
{ suffix: 'txt', src: 'file/fill/txt' },

View File

@@ -11,5 +11,5 @@ export const formatFileSize = (bytes: number): string => {
};
export const detectFileEncoding = (buffers: string | Buffer) => {
return detect(buffers)?.encoding || 'utf-8';
return (detect(buffers)?.encoding || 'utf-8') as BufferEncoding;
};

View File

@@ -57,6 +57,7 @@ export type FastGPTFeConfigsType = {
uploadFileMaxAmount?: number;
uploadFileMaxSize?: number;
lafEnv?: string;
};
export type SystemEnvType = {

View File

@@ -61,7 +61,8 @@ export enum FlowNodeTypeEnum {
pluginOutput = 'pluginOutput',
queryExtension = 'cfr',
tools = 'tools',
stopTool = 'stopTool'
stopTool = 'stopTool',
lafModule = 'lafModule'
// abandon
}

View File

@@ -20,6 +20,7 @@ import { AiQueryExtension } from './system/queryExtension';
import type { FlowNodeTemplateType, moduleTemplateListType } from '../../module/type.d';
import { FlowNodeTemplateTypeEnum } from '../../module/constants';
import { lafModule } from './system/laf';
/* app flow module templates */
export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
@@ -35,7 +36,8 @@ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
ClassifyQuestionModule,
ContextExtractModule,
HttpModule468,
AiQueryExtension
AiQueryExtension,
lafModule
];
/* plugin flow module templates */
export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
@@ -51,7 +53,8 @@ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
ClassifyQuestionModule,
ContextExtractModule,
HttpModule468,
AiQueryExtension
AiQueryExtension,
lafModule
];
/* all module */
@@ -73,7 +76,8 @@ export const moduleTemplatesFlat: FlowNodeTemplateType[] = [
PluginInputModule,
PluginOutputModule,
RunPluginModule,
AiQueryExtension
AiQueryExtension,
lafModule
];
export const moduleTemplatesList: moduleTemplateListType = [

View File

@@ -0,0 +1,86 @@
import {
FlowNodeInputTypeEnum,
FlowNodeOutputTypeEnum,
FlowNodeTypeEnum
} from '../../node/constant';
import { FlowNodeTemplateType } from '../../type';
import {
ModuleIOValueTypeEnum,
ModuleInputKeyEnum,
ModuleOutputKeyEnum,
FlowNodeTemplateTypeEnum
} from '../../constants';
import {
Input_Template_DynamicInput,
Input_Template_Switch,
Input_Template_AddInputParam
} from '../input';
import { Output_Template_Finish, Output_Template_AddOutput } from '../output';
export const lafModule: FlowNodeTemplateType = {
id: FlowNodeTypeEnum.lafModule,
templateType: FlowNodeTemplateTypeEnum.externalCall,
flowType: FlowNodeTypeEnum.lafModule,
avatar: '/imgs/module/laf.png',
name: 'Laf 函数调用(测试)',
intro: '可以调用Laf账号下的云函数。',
showStatus: true,
isTool: true,
inputs: [
Input_Template_Switch,
{
key: ModuleInputKeyEnum.httpReqUrl,
type: FlowNodeInputTypeEnum.hidden,
valueType: ModuleIOValueTypeEnum.string,
label: '',
description: 'core.module.input.description.Http Request Url',
placeholder: 'https://api.ai.com/getInventory',
required: false,
showTargetInApp: false,
showTargetInPlugin: false
},
Input_Template_DynamicInput,
{
...Input_Template_AddInputParam,
editField: {
key: true,
description: true,
dataType: true
},
defaultEditField: {
label: '',
key: '',
description: '',
inputType: FlowNodeInputTypeEnum.target,
valueType: ModuleIOValueTypeEnum.string
}
}
],
outputs: [
{
key: ModuleOutputKeyEnum.httpRawResponse,
label: '原始响应',
description: 'HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。',
valueType: ModuleIOValueTypeEnum.any,
type: FlowNodeOutputTypeEnum.source,
targets: []
},
{
...Output_Template_AddOutput,
editField: {
key: true,
description: true,
dataType: true,
defaultValue: true
},
defaultEditField: {
label: '',
key: '',
description: '',
outputType: FlowNodeOutputTypeEnum.source,
valueType: ModuleIOValueTypeEnum.string
}
},
Output_Template_Finish
]
};

View File

@@ -9,6 +9,7 @@ import { DispatchNodeResponseKeyEnum } from './runtime/constants';
import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type';
import { UserModelSchema } from 'support/user/type';
import {
ChatItemType,
ChatItemValueItemType,
ToolRunResponseItemType,
UserChatItemValueItemType

View File

@@ -41,7 +41,7 @@ export const str2OpenApiSchema = async (yamlStr = ''): Promise<OpenApiJsonSchema
path,
method,
name: methodInfo.operationId || path,
description: methodInfo.description,
description: methodInfo.description || methodInfo.summary,
params: methodInfo.parameters,
request: methodInfo?.requestBody
};

View File

@@ -1,5 +1,5 @@
import { TeamMemberRoleEnum } from './constant';
import { TeamMemberSchema } from './type';
import { LafAccountType, TeamMemberSchema } from './type';
export type AuthTeamRoleProps = {
teamId: string;
@@ -10,12 +10,14 @@ export type CreateTeamProps = {
name: string;
avatar?: string;
defaultTeam?: boolean;
lafAccount?: LafAccountType;
};
export type UpdateTeamProps = {
teamId: string;
name?: string;
avatar?: string;
teamDomain?: string;
lafAccount?: null | LafAccountType;
};
/* ------------- member ----------- */

View File

@@ -1,5 +1,6 @@
import type { UserModelSchema } from '../type';
import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
import { LafAccountType } from './type';
export type TeamSchema = {
_id: string;
@@ -13,6 +14,7 @@ export type TeamSchema = {
lastExportDatasetTime: Date;
lastWebsiteSyncTime: Date;
};
lafAccount: LafAccountType;
};
export type tagsType = {
label: string;
@@ -58,6 +60,7 @@ export type TeamItemType = {
role: `${TeamMemberRoleEnum}`;
status: `${TeamMemberStatusEnum}`;
canWrite: boolean;
lafAccount?: LafAccountType;
};
export type TeamMemberItemType = {
@@ -74,3 +77,8 @@ export type TeamTagItemType = {
label: string;
key: string;
};
export type LafAccountType = {
token: string;
appid: string;
};

View File

@@ -6,16 +6,9 @@ import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
import { MongoFileSchema } from './schema';
import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
import { readFileRawText } from '../read/rawText';
import { ReadFileByBufferParams } from '../read/type';
import { readMarkdown } from '../read/markdown';
import { readHtmlRawText } from '../read/html';
import { readPdfFile } from '../read/pdf';
import { readWordFile } from '../read/word';
import { readCsvRawText } from '../read/csv';
import { MongoRwaTextBuffer } from '../../buffer/rawText/schema';
import { readPptxRawText } from '../read/pptx';
import { readXlsxRawText } from '../read/xlsx';
import { readFileRawContent } from '../read/utils';
export function getGFSCollection(bucket: `${BucketNameEnum}`) {
MongoFileSchema;
@@ -146,7 +139,7 @@ export const readFileEncode = async ({
return encoding as BufferEncoding;
};
export const readFileContent = async ({
export const readFileContentFromMongo = async ({
teamId,
bucketName,
fileId,
@@ -205,47 +198,14 @@ export const readFileContent = async ({
}
};
const { rawText } = await (async () => {
switch (extension) {
case 'txt':
return readFileRawText(params);
case 'md':
return readMarkdown(params);
case 'html':
return readHtmlRawText(params);
case 'pdf':
return readPdfFile(params);
case 'docx':
return readWordFile(params);
case 'pptx':
return readPptxRawText(params);
case 'xlsx':
const xlsxResult = await readXlsxRawText(params);
if (csvFormat) {
return {
rawText: xlsxResult.formatText || ''
};
}
return {
rawText: xlsxResult.rawText
};
case 'csv':
const csvResult = await readCsvRawText(params);
if (csvFormat) {
return {
rawText: csvResult.formatText || ''
};
}
return {
rawText: csvResult.rawText
};
default:
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
}
})();
const { rawText } = await readFileRawContent({
extension,
csvFormat,
params
});
if (rawText.trim()) {
await MongoRwaTextBuffer.create({
MongoRwaTextBuffer.create({
sourceId: fileId,
rawText,
metadata: {

View File

@@ -2,6 +2,15 @@ import { markdownProcess } from '@fastgpt/global/common/string/markdown';
import { uploadMongoImg } from '../image/controller';
import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants';
import { addHours } from 'date-fns';
import { ReadFileByBufferParams } from './type';
import { readFileRawText } from '../read/rawText';
import { readMarkdown } from '../read/markdown';
import { readHtmlRawText } from '../read/html';
import { readPdfFile } from '../read/pdf';
import { readWordFile } from '../read/word';
import { readCsvRawText } from '../read/csv';
import { readPptxRawText } from '../read/pptx';
import { readXlsxRawText } from '../read/xlsx';
export const initMarkdownText = ({
teamId,
@@ -23,3 +32,50 @@ export const initMarkdownText = ({
expiredTime: addHours(new Date(), 2)
})
});
export const readFileRawContent = async ({
extension,
csvFormat,
params
}: {
csvFormat?: boolean;
extension: string;
params: ReadFileByBufferParams;
}) => {
switch (extension) {
case 'txt':
return readFileRawText(params);
case 'md':
return readMarkdown(params);
case 'html':
return readHtmlRawText(params);
case 'pdf':
return readPdfFile(params);
case 'docx':
return readWordFile(params);
case 'pptx':
return readPptxRawText(params);
case 'xlsx':
const xlsxResult = await readXlsxRawText(params);
if (csvFormat) {
return {
rawText: xlsxResult.formatText || ''
};
}
return {
rawText: xlsxResult.rawText
};
case 'csv':
const csvResult = await readCsvRawText(params);
if (csvFormat) {
return {
rawText: csvResult.formatText || ''
};
}
return {
rawText: csvResult.rawText
};
default:
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
}
};

View File

@@ -55,8 +55,8 @@ export const clearTmpUploadFiles = () => {
fs.stat(filePath, (err, stats) => {
if (err) return;
// 如果文件是在1小时前上传的,则认为是临时文件并删除它
if (Date.now() - stats.mtime.getTime() > 1 * 60 * 60 * 1000) {
// 如果文件是在2小时前上传的,则认为是临时文件并删除它
if (Date.now() - stats.mtime.getTime() > 2 * 60 * 60 * 1000) {
fs.unlink(filePath, (err) => {
if (err) return;
console.log(`Deleted temp file: ${filePath}`);

View File

@@ -0,0 +1,15 @@
export enum TimerIdEnum {
checkInValidDatasetFiles = 'checkInValidDatasetFiles',
checkInvalidDatasetData = 'checkInvalidDatasetData',
checkInvalidVector = 'checkInvalidVector',
clearExpiredSubPlan = 'clearExpiredSubPlan',
updateStandardPlan = 'updateStandardPlan'
}
export const timerIdMap = {
[TimerIdEnum.checkInValidDatasetFiles]: 'checkInValidDatasetFiles',
[TimerIdEnum.checkInvalidDatasetData]: 'checkInvalidDatasetData',
[TimerIdEnum.checkInvalidVector]: 'checkInvalidVector',
[TimerIdEnum.clearExpiredSubPlan]: 'clearExpiredSubPlan',
[TimerIdEnum.updateStandardPlan]: 'updateStandardPlan'
};

View File

@@ -0,0 +1,29 @@
import { connectionMongo, type Model } from '../../mongo';
import { timerIdMap } from './constants';
const { Schema, model, models } = connectionMongo;
import { TimerLockSchemaType } from './type.d';
export const collectionName = 'systemtimerlocks';
const TimerLockSchema = new Schema({
timerId: {
type: String,
required: true,
unique: true,
enum: Object.keys(timerIdMap)
},
expiredTime: {
type: Date,
required: true
}
});
try {
TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 });
} catch (error) {
console.log(error);
}
export const MongoTimerLock: Model<TimerLockSchemaType> =
models[collectionName] || model(collectionName, TimerLockSchema);
MongoTimerLock.syncIndexes();

View File

@@ -0,0 +1,5 @@
export type TimerLockSchemaType = {
_id: string;
timerId: string;
expiredTime: Date;
};

View File

@@ -0,0 +1,25 @@
import { TimerIdEnum } from './constants';
import { MongoTimerLock } from './schema';
import { addMinutes } from 'date-fns';
/*
利用唯一健,使得同一时间只有一个任务在执行,后创建的锁,会因唯一健创建失败,从而无法继续执行任务
*/
export const checkTimerLock = async ({
timerId,
lockMinuted
}: {
timerId: `${TimerIdEnum}`;
lockMinuted: number;
}) => {
try {
await MongoTimerLock.create({
timerId,
expiredTime: addMinutes(new Date(), lockMinuted)
});
return true;
} catch (error) {
return false;
}
};

View File

@@ -210,7 +210,6 @@ export const runToolWithToolChoice = async (
).filter(Boolean) as ToolRunResponseType;
const flatToolsResponseData = toolsRunResponse.map((item) => item.moduleRunResponse).flat();
if (toolCalls.length > 0 && !res.closed) {
// Run the tool, combine its results, and perform another round of AI calls
const assistantToolMsgParams: ChatCompletionAssistantToolParam = {

View File

@@ -37,6 +37,7 @@ import { dispatchRunTools } from './agent/runTool/index';
import { ChatItemValueTypeEnum } from '@fastgpt/global/core/chat/constants';
import { DispatchFlowResponse } from './type';
import { dispatchStopToolCall } from './agent/runTool/stopTool';
import { dispatchLafRequest } from './tools/runLaf';
const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
[FlowNodeTypeEnum.historyNode]: dispatchHistory,
@@ -56,6 +57,7 @@ const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
[FlowNodeTypeEnum.queryExtension]: dispatchQueryExtension,
[FlowNodeTypeEnum.tools]: dispatchRunTools,
[FlowNodeTypeEnum.stopTool]: dispatchStopToolCall,
[FlowNodeTypeEnum.lafModule]: dispatchLafRequest,
// none
[FlowNodeTypeEnum.userGuide]: () => Promise.resolve()

View File

@@ -61,7 +61,7 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H
chatId,
responseChatItemId,
...variables,
histories: histories.slice(0, 10),
histories: histories.slice(-10),
...body
};
@@ -165,6 +165,7 @@ async function fetchData({
'Content-Type': 'application/json',
...headers
},
timeout: 120000,
params: params,
data: ['POST', 'PUT', 'PATCH'].includes(method) ? body : undefined
});

View File

@@ -0,0 +1,212 @@
import type { ModuleDispatchProps } from '@fastgpt/global/core/module/type.d';
import {
DYNAMIC_INPUT_KEY,
ModuleInputKeyEnum,
ModuleOutputKeyEnum
} from '@fastgpt/global/core/module/constants';
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/module/runtime/constants';
import axios from 'axios';
import { valueTypeFormat } from '../utils';
import { SERVICE_LOCAL_HOST } from '../../../../common/system/tools';
import { addLog } from '../../../../common/system/log';
import { DispatchNodeResultType } from '@fastgpt/global/core/module/runtime/type';
type LafRequestProps = ModuleDispatchProps<{
[ModuleInputKeyEnum.httpReqUrl]: string;
[DYNAMIC_INPUT_KEY]: Record<string, any>;
[key: string]: any;
}>;
type LafResponse = DispatchNodeResultType<{
[ModuleOutputKeyEnum.failed]?: boolean;
[key: string]: any;
}>;
const UNDEFINED_SIGN = 'UNDEFINED_SIGN';
export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafResponse> => {
let {
appId,
chatId,
responseChatItemId,
variables,
module: { outputs },
histories,
params: { system_httpReqUrl: httpReqUrl, [DYNAMIC_INPUT_KEY]: dynamicInput, ...body }
} = props;
if (!httpReqUrl) {
return Promise.reject('Http url is empty');
}
const concatVariables = {
appId,
chatId,
responseChatItemId,
...variables,
...body
};
httpReqUrl = replaceVariable(httpReqUrl, concatVariables);
const requestBody = {
systemParams: {
appId,
chatId,
responseChatItemId,
histories: histories.slice(0, 10)
},
variables,
...dynamicInput,
...body
};
try {
const { formatResponse, rawResponse } = await fetchData({
method: 'POST',
url: httpReqUrl,
body: requestBody
});
// format output value type
const results: Record<string, any> = {};
for (const key in formatResponse) {
const output = outputs.find((item) => item.key === key);
if (!output) continue;
results[key] = valueTypeFormat(formatResponse[key], output.valueType);
}
return {
assistantResponses: [],
[DispatchNodeResponseKeyEnum.nodeResponse]: {
totalPoints: 0,
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
httpResult: rawResponse
},
[DispatchNodeResponseKeyEnum.toolResponses]: rawResponse,
[ModuleOutputKeyEnum.httpRawResponse]: rawResponse,
...results
};
} catch (error) {
addLog.error('Http request error', error);
return {
[ModuleOutputKeyEnum.failed]: true,
[DispatchNodeResponseKeyEnum.nodeResponse]: {
totalPoints: 0,
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
httpResult: { error: formatHttpError(error) }
}
};
}
};
async function fetchData({
method,
url,
body
}: {
method: string;
url: string;
body: Record<string, any>;
}): Promise<Record<string, any>> {
const { data: response } = await axios({
method,
baseURL: `http://${SERVICE_LOCAL_HOST}`,
url,
headers: {
'Content-Type': 'application/json'
},
data: body
});
const parseJson = (obj: Record<string, any>, prefix = '') => {
let result: Record<string, any> = {};
if (Array.isArray(obj)) {
for (let i = 0; i < obj.length; i++) {
result[`${prefix}[${i}]`] = obj[i];
if (Array.isArray(obj[i])) {
result = {
...result,
...parseJson(obj[i], `${prefix}[${i}]`)
};
} else if (typeof obj[i] === 'object') {
result = {
...result,
...parseJson(obj[i], `${prefix}[${i}].`)
};
}
}
} else if (typeof obj == 'object') {
for (const key in obj) {
result[`${prefix}${key}`] = obj[key];
if (Array.isArray(obj[key])) {
result = {
...result,
...parseJson(obj[key], `${prefix}${key}`)
};
} else if (typeof obj[key] === 'object') {
result = {
...result,
...parseJson(obj[key], `${prefix}${key}.`)
};
}
}
}
return result;
};
return {
formatResponse:
typeof response === 'object' && !Array.isArray(response) ? parseJson(response) : {},
rawResponse: response
};
}
function replaceVariable(text: string, obj: Record<string, any>) {
for (const [key, value] of Object.entries(obj)) {
if (value === undefined) {
text = text.replace(new RegExp(`{{${key}}}`, 'g'), UNDEFINED_SIGN);
} else {
const replacement = JSON.stringify(value);
const unquotedReplacement =
replacement.startsWith('"') && replacement.endsWith('"')
? replacement.slice(1, -1)
: replacement;
text = text.replace(new RegExp(`{{${key}}}`, 'g'), unquotedReplacement);
}
}
return text || '';
}
function removeUndefinedSign(obj: Record<string, any>) {
for (const key in obj) {
if (obj[key] === UNDEFINED_SIGN) {
obj[key] = undefined;
} else if (Array.isArray(obj[key])) {
obj[key] = obj[key].map((item: any) => {
if (item === UNDEFINED_SIGN) {
return undefined;
} else if (typeof item === 'object') {
removeUndefinedSign(item);
}
return item;
});
} else if (typeof obj[key] === 'object') {
removeUndefinedSign(obj[key]);
}
}
return obj;
}
function formatHttpError(error: any) {
return {
message: error?.message,
name: error?.name,
method: error?.config?.method,
baseURL: error?.config?.baseURL,
url: error?.config?.url,
code: error?.code,
status: error?.status
};
}

View File

@@ -10,7 +10,6 @@ import { MongoTeam } from './teamSchema';
async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> {
const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
if (!tmb) {
return Promise.reject('member not exist');
}
@@ -27,7 +26,8 @@ async function getTeamMember(match: Record<string, any>): Promise<TeamItemType>
role: tmb.role,
status: tmb.status,
defaultTeam: tmb.defaultTeam,
canWrite: tmb.role !== TeamMemberRoleEnum.visitor
canWrite: tmb.role !== TeamMemberRoleEnum.visitor,
lafAccount: tmb.teamId.lafAccount
};
}

View File

@@ -35,6 +35,14 @@ const TeamSchema = new Schema({
lastWebsiteSyncTime: {
type: Date
}
},
lafAccount: {
token: {
type: String
},
appid: {
type: String
}
}
});

View File

@@ -125,7 +125,9 @@ export const iconPaths = {
'file/fill/manual': () => import('./icons/file/fill/manual.svg'),
'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'),
'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'),
'file/fill/ppt': () => import('./icons/file/fill/ppt.svg'),
'file/fill/txt': () => import('./icons/file/fill/txt.svg'),
'file/fill/xlsx': () => import('./icons/file/fill/xlsx.svg'),
'file/html': () => import('./icons/file/html.svg'),
'file/indexImport': () => import('./icons/file/indexImport.svg'),
'file/manualImport': () => import('./icons/file/manualImport.svg'),

View File

@@ -0,0 +1,11 @@
<svg t="1712108366314" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="1576"
width="128" height="128">
<path
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
fill="#E34221" p-id="1577"></path>
<path d="M960 326.4v16H755.2s-100.8-20.8-99.2-108.8c0 0 4.8 92.8 97.6 92.8H960z" fill="#DC3119" p-id="1578"></path>
<path d="M657.6 0v233.6c0 25.6 17.6 92.8 97.6 92.8H960L657.6 0z" fill="#FFFFFF" p-id="1579"></path>
<path
d="M304 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V686.4c0-9.6 8-17.6 17.6-17.6H304c38.4 0 59.2 25.6 59.2 57.6S340.8 784 304 784z m-3.2-94.4h-51.2v73.6h51.2c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8zM480 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-11.2-4.8-11.2-11.2V686.4c0-9.6 6.4-17.6 16-17.6H480c38.4 0 59.2 25.6 59.2 57.6S518.4 784 480 784z m-3.2-94.4h-49.6v73.6h49.6c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8z m225.6 0h-52.8v161.6c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V689.6h-51.2c-6.4 0-11.2-4.8-11.2-11.2 0-4.8 4.8-9.6 11.2-9.6h128c6.4 0 11.2 4.8 11.2 11.2 0 4.8-4.8 9.6-11.2 9.6z"
fill="#FFFFFF" p-id="1580"></path>
</svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

View File

@@ -0,0 +1,11 @@
<svg t="1712108380525" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2654"
width="128" height="128">
<path
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
fill="#45B058" p-id="2655"></path>
<path
d="M374.4 862.4c-3.2 0-6.4-1.6-8-3.2l-59.2-80-60.8 80c-1.6 1.6-4.8 3.2-8 3.2-6.4 0-11.2-4.8-11.2-11.2 0-1.6 0-4.8 1.6-6.4l62.4-81.6-57.6-78.4c-1.6-1.6-3.2-3.2-3.2-6.4 0-4.8 4.8-11.2 11.2-11.2 4.8 0 8 1.6 9.6 4.8l56 73.6 54.4-73.6c1.6-3.2 4.8-4.8 8-4.8 6.4 0 12.8 4.8 12.8 11.2 0 3.2-1.6 4.8-1.6 6.4l-59.2 76.8 62.4 83.2c1.6 1.6 3.2 4.8 3.2 6.4 0 6.4-6.4 11.2-12.8 11.2z m160-1.6H448c-9.6 0-17.6-8-17.6-17.6V678.4c0-6.4 4.8-11.2 12.8-11.2 6.4 0 11.2 4.8 11.2 11.2v161.6h80c6.4 0 11.2 4.8 11.2 9.6 0 6.4-4.8 11.2-11.2 11.2z m112 3.2c-28.8 0-51.2-9.6-67.2-24-3.2-1.6-3.2-4.8-3.2-8 0-6.4 3.2-12.8 11.2-12.8 1.6 0 4.8 1.6 6.4 3.2 12.8 11.2 32 20.8 54.4 20.8 33.6 0 44.8-19.2 44.8-33.6 0-49.6-113.6-22.4-113.6-89.6 0-32 27.2-54.4 65.6-54.4 24 0 46.4 8 60.8 20.8 3.2 1.6 4.8 4.8 4.8 8 0 6.4-4.8 12.8-11.2 12.8-1.6 0-4.8-1.6-6.4-3.2-14.4-11.2-32-16-49.6-16-24 0-40 11.2-40 30.4 0 43.2 113.6 17.6 113.6 89.6 0 27.2-19.2 56-70.4 56z"
fill="#FFFFFF" p-id="2656"></path>
<path d="M960 326.4v16H755.2s-102.4-20.8-99.2-108.8c0 0 3.2 92.8 96 92.8h208z" fill="#349C42" p-id="2657"></path>
<path d="M656 0v233.6c0 25.6 19.2 92.8 99.2 92.8H960L656 0z" fill="#FFFFFF" p-id="2658"></path>
</svg>

After

Width:  |  Height:  |  Size: 1.5 KiB

View File

@@ -36,7 +36,7 @@ export default function Editor({
hasVariablePlugin?: boolean;
hasDropDownPlugin?: boolean;
variables: EditorVariablePickerType[];
onChange?: (editorState: EditorState) => void;
onChange?: (editorState: EditorState, editor: LexicalEditor) => void;
onBlur?: (editor: LexicalEditor) => void;
value?: string;
currentValue?: string;
@@ -119,9 +119,9 @@ export default function Editor({
<HistoryPlugin />
<FocusPlugin focus={focus} setFocus={setFocus} />
<OnChangePlugin
onChange={(e) => {
onChange={(editorState: EditorState, editor: LexicalEditor) => {
startSts(() => {
onChange?.(e);
onChange?.(editorState, editor);
});
}}
/>

View File

@@ -32,15 +32,13 @@ const HttpInput = ({
const [, startSts] = useTransition();
const onChangeInput = useCallback((editorState: EditorState) => {
const text = editorState.read(() => $getRoot().getTextContent());
const formatValue = text.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
setCurrentValue(formatValue);
onChange?.(formatValue);
const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onChange?.(text);
}, []);
const onBlurInput = useCallback((editor: LexicalEditor) => {
startSts(() => {
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onBlur?.(text);
});
}, []);

View File

@@ -7,10 +7,13 @@ import {
useDisclosure,
MenuButton,
Box,
css
css,
Flex
} from '@chakra-ui/react';
import type { ButtonProps, MenuItemProps } from '@chakra-ui/react';
import { ChevronDownIcon } from '@chakra-ui/icons';
import { useLoading } from '../../../hooks/useLoading';
import MyIcon from '../Icon';
export type SelectProps = ButtonProps & {
value?: string;
@@ -20,14 +23,16 @@ export type SelectProps = ButtonProps & {
label: string | React.ReactNode;
value: string;
}[];
isLoading?: boolean;
onchange?: (val: any) => void;
};
const MySelect = (
{ placeholder, value, width = '100%', list, onchange, ...props }: SelectProps,
{ placeholder, value, width = '100%', list, onchange, isLoading = false, ...props }: SelectProps,
selectRef: any
) => {
const ref = useRef<HTMLButtonElement>(null);
const { Loading } = useLoading();
const menuItemStyles: MenuItemProps = {
borderRadius: 'sm',
py: 2,
@@ -78,7 +83,10 @@ const MySelect = (
: {})}
{...props}
>
{selectItem?.alias || selectItem?.label || placeholder}
<Flex alignItems={'center'}>
{isLoading && <MyIcon mr={2} name={'common/loading'} w={'16px'} />}
{selectItem?.alias || selectItem?.label || placeholder}
</Flex>
</MenuButton>
<MenuList

View File

@@ -1,10 +1,10 @@
import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react';
import React, { useEffect, useState } from 'react';
import React from 'react';
import { editorStateToText } from './utils';
import Editor from './Editor';
import MyModal from '../../MyModal';
import { useTranslation } from 'next-i18next';
import { $getRoot, EditorState, type LexicalEditor } from 'lexical';
import { EditorState, type LexicalEditor } from 'lexical';
import { EditorVariablePickerType } from './type.d';
import { useCallback, useTransition } from 'react';
@@ -34,16 +34,12 @@ const PromptEditor = ({
const { t } = useTranslation();
const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
const stringifiedEditorState = JSON.stringify(editorState.toJSON());
const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
const formatValue = editorStateTextString.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
onChange?.(formatValue);
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onChange?.(text);
}, []);
const onBlurInput = useCallback((editor: LexicalEditor) => {
startSts(() => {
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
onBlur?.(text);
});
}, []);

View File

@@ -1,6 +1,5 @@
import { useEffect } from 'react';
import { BLUR_COMMAND, COMMAND_PRIORITY_EDITOR, LexicalEditor } from 'lexical';
import { mergeRegister } from '@lexical/utils';
import { useLexicalComposerContext } from '@lexical/react/LexicalComposerContext';
export default function OnBlurPlugin({ onBlur }: { onBlur?: (editor: LexicalEditor) => void }) {

View File

@@ -209,9 +209,8 @@ export function editorStateToText(editor: LexicalEditor) {
const stringifiedEditorState = JSON.stringify(editor.getEditorState().toJSON());
const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
const compressedText = editorStateTextString.replace(/\n+/g, '\n\n');
return compressedText;
return editorStateTextString;
}
const varRegex = /\{\{([a-zA-Z_][a-zA-Z0-9_]*)\}\}/g;

2857
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,4 +1,7 @@
{
"feConfigs": {
"lafEnv": "https://laf.dev"
},
"systemEnv": {
"openapiPrefix": "fastgpt",
"vectorMaxProcess": 15,

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

View File

@@ -23,7 +23,7 @@ function embedChatbot() {
const ChatBtn = document.createElement('div');
ChatBtn.id = chatBtnId;
ChatBtn.style.cssText =
'position: fixed; bottom: 1rem; right: 1rem; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
'position: fixed; bottom: 30px; right: 60px; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
// btn icon
const ChatBtnDiv = document.createElement('img');
@@ -39,7 +39,7 @@ function embedChatbot() {
iframe.id = chatWindowId;
iframe.src = botSrc;
iframe.style.cssText =
'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 4rem; right: 1rem; width: 24rem; height: 40rem; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 80px; right: 60px; width: 375px; height: 667px; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
iframe.style.visibility = defaultOpen ? 'unset' : 'hidden';
document.body.appendChild(iframe);

View File

@@ -148,6 +148,7 @@
"Status": "Status",
"Submit failed": "Submit failed",
"Submit success": "Update Success",
"Sync success": "",
"System version": "System version",
"Team": "Team",
"Team Tags Set": "Team Tags",
@@ -819,6 +820,7 @@
"Http request props": "Request props",
"Http request settings": "Request settings",
"Input Type": "Input Type",
"Laf sync params": "Sync params",
"Plugin output must connect": "Custom outputs must all be connected",
"QueryExtension": {
"placeholder": "Questions about python introduction and usage, etc. The current conversation is related to the game GTA5.",
@@ -912,6 +914,9 @@
"target": "Target Data",
"textarea": "Textarea"
},
"laf": {
"Select laf function": ""
},
"output": {
"Add Output": "Add Output",
"Output Number": "Output: {{length}}",
@@ -1226,19 +1231,26 @@
"Set Public": "Set to public"
},
"plugin": {
"App": "Choose App",
"Auth Header Prefix": "Auth header prefix",
"Auth Method": "Auth method",
"Auth Type": "Auth Type",
"Confirm Delete": "Confirm to delete the plugin?",
"Create Your Plugin": "Create Plugin",
"Currentapp": "CurrentApp",
"Custom Plugin": "Custom plugin",
"Description": "Description",
"Edit Http Plugin": "Edit HTTP plugin",
"Enter Env": "Enter laf environment",
"Enter PAT": "Please enter personal access token (PAT)",
"Func": "Choose Function",
"Get Plugin Module Detail Failed": "Get plugin detail failed",
"HTTP Plugin": "HTTP plugin",
"Import Plugin": "Import HTTP plugin",
"Import from URL": "Import from URL. https://xxxx",
"Intro": "Plugin Intro",
"Invalid Appid": "Invalid appid",
"Invalid Env": "Invalid Env",
"Invalid Schema": "Invalid Schema",
"Invalid URL": "Invalid URL",
"Key": "Key",
@@ -1248,16 +1260,20 @@
"No Intro": "This plugin is not introduced",
"None": "None",
"Path": "Path",
"Please bind laf accout first": "Please bind laf accout first",
"Plugin List": "Plugin list",
"Plugin Module": "Plugin",
"Privacy Agreement": "privacy agreement",
"Search plugin": "Search plugins",
"Set Name": "Plugin Name",
"Synchronous app": "Sync App",
"Synchronous version": "Sync Version",
"To Edit Plugin": "To Edit",
"Update Your Plugin": "Update Plugin",
"Value": "Value",
"path": ""
"go to laf": "go to laf",
"path": "",
"update params": "update params"
},
"support": {
"account": {
@@ -1303,6 +1319,9 @@
"user": {
"AI point standard": "AI points price",
"Avatar": "Avatar",
"Go laf env": "Click to go to laf to get PAT certificate.",
"Laf account course": "Check out the laf account binding tutorial.",
"Laf account intro": "After binding your laf account, you will be able to use the laf module in your workflow to write code online.",
"Need to login": "Please log in first",
"Price": "Price",
"User self info": "My info",
@@ -1495,10 +1514,13 @@
"Bill Detail": "Bill Detail",
"Change": "Change",
"Copy invite url": "Copy invitation link",
"Current laf Env": "Current laf Env",
"Edit name": "Click to modify nickname",
"Invite Url": "Invite Url",
"Invite url tip": "Friends who register through this link will be permanently bound to you, and you will get a certain balance reward when they recharge. In addition, when friends register with their mobile phone number, you will get 5 yuan reward immediately.",
"Laf Account Setting": "laf account setting",
"Language": "Language",
"Learn More": "Learn More",
"Member Name": "Name",
"Notice": "Notice",
"Old password is error": "Old password is error",
@@ -1513,6 +1535,7 @@
"Promotion rate tip": "You will be rewarded with a percentage of the balance when your friends top up",
"Recharge Record": "Recharge",
"Replace": "Replace",
"Set Laf Account Failed": "set laf accout failed",
"Set OpenAI Account Failed": "Set OpenAI account failed",
"Sign Out": "Sign Out",
"Source": "Source",

View File

@@ -148,6 +148,7 @@
"Status": "状态",
"Submit failed": "提交失败",
"Submit success": "提交成功",
"Sync success": "同步成功",
"System version": "系统版本",
"Team": "团队",
"Team Tags Set": "标签",
@@ -821,6 +822,7 @@
"Http request props": "请求参数",
"Http request settings": "请求配置",
"Input Type": "输入类型",
"Laf sync params": "同步参数",
"Plugin output must connect": "自定义输出必须全部连接",
"QueryExtension": {
"placeholder": "例如:\n关于 python 的介绍和使用等问题。\n当前对话与游戏《GTA5》有关。",
@@ -914,6 +916,9 @@
"target": "外部数据",
"textarea": "段落输入"
},
"laf": {
"Select laf function": "选择laf函数"
},
"output": {
"Add Output": "添加出参",
"Output Number": "出参: {{length}}",
@@ -1228,19 +1233,26 @@
"Set Public": "设为团队可用"
},
"plugin": {
"App": "选择应用",
"Auth Header Prefix": "鉴权头部前缀",
"Auth Method": "鉴权方法",
"Auth Type": "鉴权类型",
"Confirm Delete": "确认删除该插件?",
"Create Your Plugin": "创建你的插件",
"Currentapp": "当前应用",
"Custom Plugin": "自定义插件",
"Description": "描述",
"Edit Http Plugin": "编辑 HTTP 插件",
"Enter Env": "输入 laf 环境",
"Enter PAT": "请输入访问凭证PAT",
"Func": "选择函数",
"Get Plugin Module Detail Failed": "获取插件信息异常",
"HTTP Plugin": "HTTP 插件",
"Import Plugin": "导入 HTTP 插件",
"Import from URL": "从URL导入。https://xxxx",
"Intro": "插件介绍",
"Invalid Appid": "appid 无效",
"Invalid Env": "laf 环境错误",
"Invalid Schema": "Schema 无效",
"Invalid URL": "URL 无效",
"Key": "键",
@@ -1250,16 +1262,20 @@
"No Intro": "这个插件没有介绍~",
"None": "无",
"Path": "路径",
"Please bind laf accout first": "请先绑定 laf 账号",
"Plugin List": "插件列表",
"Plugin Module": "插件模块",
"Privacy Agreement": "隐私协议",
"Search plugin": "搜索插件",
"Set Name": "给插件取个名字",
"Synchronous app": "同步应用",
"Synchronous version": "同步版本",
"To Edit Plugin": "去编辑",
"Update Your Plugin": "更新插件",
"Value": "值",
"path": ""
"go to laf": "去编写",
"path": "",
"update params": "更新参数"
},
"support": {
"account": {
@@ -1305,6 +1321,9 @@
"user": {
"AI point standard": "AI积分标准",
"Avatar": "头像",
"Go laf env": "点击前往 laf 获取 PAT 凭证。",
"Laf account course": "查看绑定 laf 账号教程。",
"Laf account intro": "绑定你的laf账号后你将可以在工作流中使用 laf 模块,实现在线编写代码。",
"Need to login": "请先登录",
"Price": "计费标准",
"User self info": "个人信息",
@@ -1497,10 +1516,13 @@
"Bill Detail": "账单详情",
"Change": "变更",
"Copy invite url": "复制邀请链接",
"Current laf Env": "当前 laf 环境",
"Edit name": "点击修改昵称",
"Invite Url": "邀请链接",
"Invite url tip": "通过该链接注册的好友将永久与你绑定,其充值时你会获得一定余额奖励。\n此外好友使用手机号注册时你将立即获得 5 元奖励。\n奖励会发送到您的默认团队中。",
"Laf Account Setting": "laf 账号配置",
"Language": "语言",
"Learn More": "查看文档",
"Member Name": "昵称",
"Notice": "通知",
"Old password is error": "旧密码错误",
@@ -1515,6 +1537,7 @@
"Promotion rate tip": "好友充值时你将获得一定比例的余额奖励",
"Recharge Record": "支付记录",
"Replace": "更换",
"Set Laf Account Failed": "设置 laf 账号异常",
"Set OpenAI Account Failed": "设置 OpenAI 账号异常",
"Sign Out": "登出",
"Source": "来源",

View File

@@ -214,9 +214,6 @@ export const FlowProvider = ({
if (source?.flowType === FlowNodeTypeEnum.classifyQuestion && !type) {
return ModuleIOValueTypeEnum.boolean;
}
if (source?.flowType === FlowNodeTypeEnum.tools) {
return ModuleIOValueTypeEnum.tools;
}
if (source?.flowType === FlowNodeTypeEnum.pluginInput) {
return source?.inputs.find((input) => input.key === connect.sourceHandle)?.valueType;
}

View File

@@ -52,6 +52,7 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
const router = useRouter();
const [currentParent, setCurrentParent] = useState<RenderListProps['currentParent']>();
const [searchKey, setSearchKey] = useState('');
const { feConfigs } = useSystemStore();
const {
basicNodeTemplates,
@@ -64,7 +65,12 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
const templates = useMemo(() => {
const map = {
[TemplateTypeEnum.basic]: basicNodeTemplates,
[TemplateTypeEnum.basic]: basicNodeTemplates.filter((item) => {
if (item.flowType === FlowNodeTypeEnum.lafModule && !feConfigs.lafEnv) {
return false;
}
return true;
}),
[TemplateTypeEnum.systemPlugin]: systemNodeTemplates,
[TemplateTypeEnum.teamPlugin]: teamPluginNodeTemplates.filter((item) =>
searchKey ? item.pluginType !== PluginTypeEnum.folder : true

View File

@@ -88,7 +88,7 @@ enum TabEnum {
headers = 'headers',
body = 'body'
}
type PropsArrType = {
export type PropsArrType = {
key: string;
type: string;
value: string;
@@ -245,7 +245,7 @@ const RenderHttpMethodAndUrl = React.memo(function RenderHttpMethodAndUrl({
);
});
function RenderHttpProps({
export function RenderHttpProps({
moduleId,
inputs
}: {

View File

@@ -0,0 +1,260 @@
import React, { useCallback, useMemo } from 'react';
import { NodeProps } from 'reactflow';
import NodeCard from '../render/NodeCard';
import { FlowModuleItemType } from '@fastgpt/global/core/module/type.d';
import Container from '../modules/Container';
import { Box, Button, Center, Flex, useDisclosure } from '@chakra-ui/react';
import { ModuleIOValueTypeEnum, ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
import { onChangeNode, useFlowProviderStore } from '../../FlowProvider';
import { useTranslation } from 'next-i18next';
import { getLafAppDetail } from '@/web/support/laf/api';
import MySelect from '@fastgpt/web/components/common/MySelect';
import { getApiSchemaByUrl } from '@/web/core/plugin/api';
import { str2OpenApiSchema } from '@fastgpt/global/core/plugin/httpPlugin/utils';
import { useUserStore } from '@/web/support/user/useUserStore';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { ChevronRightIcon } from '@chakra-ui/icons';
import { useQuery } from '@tanstack/react-query';
import dynamic from 'next/dynamic';
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { useToast } from '@fastgpt/web/hooks/useToast';
import Divider from '../modules/Divider';
import RenderToolInput from '../render/RenderToolInput';
import RenderInput from '../render/RenderInput';
import RenderOutput from '../render/RenderOutput';
import { getErrText } from '@fastgpt/global/common/error/utils';
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
const NodeLaf = (props: NodeProps<FlowModuleItemType>) => {
const { t } = useTranslation();
const { toast } = useToast();
const { feConfigs } = useSystemStore();
const { data, selected } = props;
const { moduleId, inputs } = data;
const requestUrl = inputs.find((item) => item.key === ModuleInputKeyEnum.httpReqUrl);
const { userInfo } = useUserStore();
const token = userInfo?.team.lafAccount?.token;
const appid = userInfo?.team.lafAccount?.appid;
// not config laf
if (!token || !appid) {
return (
<NodeCard minW={'350px'} selected={selected} {...data}>
<ConfigLaf />
</NodeCard>
);
}
const { data: lafData, isLoading: isLoadingFunctions } = useQuery(
['getLafFunctionList'],
async () => {
// load laf app detail
const appDetail = await getLafAppDetail(appid);
// load laf app functions
const schemaUrl = `https://${appDetail?.domain.domain}/_/api-docs?token=${appDetail?.openapi_token}`;
const schema = await getApiSchemaByUrl(schemaUrl);
const openApiSchema = await str2OpenApiSchema(JSON.stringify(schema));
const filterPostSchema = openApiSchema.pathData.filter((item) => item.method === 'post');
return {
lafApp: appDetail,
lafFunctions: filterPostSchema.map((item) => ({
...item,
requestUrl: `https://${appDetail?.domain.domain}${item.path}`
}))
};
},
{
onError(err) {
toast({
status: 'error',
title: getErrText(err, '获取Laf函数列表失败')
});
}
}
);
const lafFunctionSelectList = useMemo(
() =>
lafData?.lafFunctions.map((item) => ({
label: item.description ? `${item.name} (${item.description})` : item.name,
value: item.requestUrl
})) || [],
[lafData?.lafFunctions]
);
const selectedFunction = useMemo(
() => lafFunctionSelectList.find((item) => item.value === requestUrl?.value)?.value,
[lafFunctionSelectList, requestUrl?.value]
);
const onSyncParams = useCallback(() => {
const lafFunction = lafData?.lafFunctions.find((item) => item.requestUrl === selectedFunction);
if (!lafFunction) return;
const bodyParams =
lafFunction?.request?.content?.['application/json']?.schema?.properties || {};
const requiredParams =
lafFunction?.request?.content?.['application/json']?.schema?.required || [];
const allParams = [
...Object.keys(bodyParams).map((key) => ({
name: key,
desc: bodyParams[key].description,
required: requiredParams?.includes(key) || false,
value: `{{${key}}}`,
type: 'string'
}))
].filter((item) => !inputs.find((input) => input.key === item.name));
// add params
allParams.forEach((param) => {
onChangeNode({
moduleId,
type: 'addInput',
key: param.name,
value: {
key: param.name,
valueType: ModuleIOValueTypeEnum.string,
label: param.name,
type: FlowNodeInputTypeEnum.target,
required: param.required,
description: param.desc || '',
toolDescription: param.desc || '未设置参数描述',
edit: true,
editField: {
key: true,
name: true,
description: true,
required: true,
dataType: true,
inputType: true,
isToolInput: true
},
connected: false
}
});
});
toast({
status: 'success',
title: t('common.Sync success')
});
}, [inputs, lafData?.lafFunctions, moduleId, selectedFunction, t, toast]);
return (
<NodeCard minW={'350px'} selected={selected} {...data}>
<Container>
{/* select function */}
<MySelect
isLoading={isLoadingFunctions}
list={lafFunctionSelectList}
placeholder={t('core.module.laf.Select laf function')}
onchange={(e) => {
onChangeNode({
moduleId,
type: 'updateInput',
key: ModuleInputKeyEnum.httpReqUrl,
value: {
...requestUrl,
value: e
}
});
}}
value={selectedFunction}
/>
{/* auto set params and go to edit */}
{!!selectedFunction && (
<Flex justifyContent={'flex-end'} mt={2} gap={2}>
{/* <Button variant={'whiteBase'} size={'sm'} onClick={onSyncParams}>
{t('core.module.Laf sync params')}
</Button> */}
<Button
variant={'grayBase'}
size={'sm'}
onClick={() => {
const lafFunction = lafData?.lafFunctions.find(
(item) => item.requestUrl === selectedFunction
);
if (!lafFunction) return;
const url = `${feConfigs.lafEnv}/app/${lafData?.lafApp?.appid}/function${lafFunction?.path}`;
window.open(url, '_blank');
}}
>
{t('plugin.go to laf')}
</Button>
</Flex>
)}
</Container>
{!!selectedFunction && <RenderIO {...props} />}
</NodeCard>
);
};
export default React.memo(NodeLaf);
const ConfigLaf = () => {
const { t } = useTranslation();
const { userInfo } = useUserStore();
const { feConfigs } = useSystemStore();
const {
isOpen: isOpenLafConfig,
onOpen: onOpenLafConfig,
onClose: onCloseLafConfig
} = useDisclosure();
return !!feConfigs?.lafEnv ? (
<Center minH={150}>
<Button onClick={onOpenLafConfig} variant={'whitePrimary'}>
{t('plugin.Please bind laf accout first')} <ChevronRightIcon />
</Button>
{isOpenLafConfig && feConfigs?.lafEnv && (
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLafConfig} />
)}
</Center>
) : (
<Box>Laf环境</Box>
);
};
const RenderIO = ({ data, selected }: NodeProps<FlowModuleItemType>) => {
const { t } = useTranslation();
const { moduleId, inputs, outputs } = data;
const { splitToolInputs, hasToolNode } = useFlowProviderStore();
const { commonInputs, toolInputs } = splitToolInputs(inputs, moduleId);
return (
<>
{hasToolNode && (
<>
<Divider text={t('core.module.tool.Tool input')} />
<Container>
<RenderToolInput moduleId={moduleId} inputs={toolInputs} canEdit />
</Container>
</>
)}
<>
<Divider text={t('common.Input')} />
<Container>
<Box mb={3}>Body参数</Box>
<RenderInput moduleId={moduleId} flowInputList={commonInputs} />
</Container>
</>
<>
<Divider text={t('common.Output')} />
<Container>
<RenderOutput moduleId={moduleId} flowOutputList={outputs} />
</Container>
</>
</>
);
};

View File

@@ -164,7 +164,7 @@ const NodeCard = (props: Props) => {
top={'-20px'}
right={0}
transform={'translateX(90%)'}
pl={'17px'}
pl={'20px'}
pr={'10px'}
pb={'20px'}
pt={'20px'}

View File

@@ -1,11 +1,25 @@
import { FlowNodeInputItemType } from '@fastgpt/global/core/module/node/type';
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
import { ModuleIOValueTypeEnum } from '@fastgpt/global/core/module/constants';
export const defaultEditFormData: FlowNodeInputItemType = {
valueType: 'string',
type: FlowNodeInputTypeEnum.hidden,
type: FlowNodeInputTypeEnum.target,
key: '',
label: '',
toolDescription: '',
required: true
required: true,
edit: true,
editField: {
key: true,
description: true,
dataType: true
},
defaultEditField: {
label: '',
key: '',
description: '',
inputType: FlowNodeInputTypeEnum.target,
valueType: ModuleIOValueTypeEnum.string
}
};

View File

@@ -44,7 +44,8 @@ const nodeTypes: Record<`${FlowNodeTypeEnum}`, any> = {
[FlowNodeTypeEnum.tools]: dynamic(() => import('./components/nodes/NodeTools')),
[FlowNodeTypeEnum.stopTool]: (data: NodeProps<FlowModuleItemType>) => (
<NodeSimple {...data} minW={'100px'} maxW={'300px'} />
)
),
[FlowNodeTypeEnum.lafModule]: dynamic(() => import('./components/nodes/NodeLaf'))
};
const edgeTypes = {
[EDGE_TYPE]: ButtonEdge

View File

@@ -0,0 +1,185 @@
import React, { useCallback } from 'react';
import { ModalBody, Box, Flex, Input, ModalFooter, Button, Link } from '@chakra-ui/react';
import MyModal from '@fastgpt/web/components/common/MyModal';
import { useTranslation } from 'next-i18next';
import { useForm } from 'react-hook-form';
import { useRequest } from '@fastgpt/web/hooks/useRequest';
import { useQuery } from '@tanstack/react-query';
import MySelect from '@fastgpt/web/components/common/MySelect';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { useToast } from '@fastgpt/web/hooks/useToast';
import { putUpdateTeam } from '@/web/support/user/team/api';
import { useUserStore } from '@/web/support/user/useUserStore';
import type { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
import { postLafPat2Token, getLafApplications } from '@/web/support/laf/api';
import { getErrText } from '@fastgpt/global/common/error/utils';
const LafAccountModal = ({
defaultData = {
token: '',
appid: ''
},
onClose
}: {
defaultData?: LafAccountType;
onClose: () => void;
}) => {
const { t } = useTranslation();
const { register, handleSubmit, setValue, getValues, watch, reset } = useForm({
defaultValues: {
...defaultData,
pat: ''
}
});
const lafToken = watch('token');
const pat = watch('pat');
const appid = watch('appid');
const { feConfigs } = useSystemStore();
const { toast } = useToast();
const { userInfo, initUserInfo } = useUserStore();
const onResetForm = useCallback(() => {
reset({
token: '',
appid: '',
pat: ''
});
}, [reset]);
const { mutate: authLafPat, isLoading: isPatLoading } = useRequest({
mutationFn: async (pat) => {
const token = await postLafPat2Token(pat);
setValue('token', token);
},
errorToast: t('plugin.Invalid Env')
});
const { data: appListData = [] } = useQuery(
['appList', lafToken],
() => {
return getLafApplications(lafToken);
},
{
enabled: !!lafToken,
onSuccess: (data) => {
if (!getValues('appid') && data.length > 0) {
setValue('appid', data[0].appid);
}
},
onError: (err) => {
onResetForm();
toast({
title: getErrText(err, '获取应用列表失败'),
status: 'error'
});
}
}
);
const { mutate: onSubmit, isLoading: isUpdating } = useRequest({
mutationFn: async (data: LafAccountType) => {
if (!userInfo?.team.teamId) return;
return putUpdateTeam({
teamId: userInfo?.team.teamId,
lafAccount: data
});
},
onSuccess() {
initUserInfo();
onClose();
},
successToast: t('common.Update Success'),
errorToast: t('common.Update Failed')
});
return (
<MyModal isOpen iconSrc="/imgs/module/laf.png" title={t('user.Laf Account Setting')}>
<ModalBody>
<Box fontSize={'sm'} color={'myGray.500'}>
<Box>{t('support.user.Laf account intro')}</Box>
<Box textDecoration={'underline'}>
<Link href={`https://doc.laf.run/zh/`} isExternal>
{t('support.user.Laf account course')}
</Link>
</Box>
<Box>
<Link textDecoration={'underline'} href={`${feConfigs.lafEnv}/`} isExternal>
{t('support.user.Go laf env')}
</Link>
</Box>
</Box>
<Flex alignItems={'center'} mt={5}>
<Box flex={'0 0 70px'}>PAT:</Box>
{!lafToken ? (
<>
<Input
flex={'1 0 0'}
size={'sm'}
{...register('pat')}
placeholder={t('plugin.Enter PAT')}
/>
<Button
ml={2}
variant={'whitePrimary'}
isDisabled={!pat}
onClick={() => {
authLafPat(pat);
}}
isLoading={isPatLoading}
>
</Button>
</>
) : (
<Button
variant={'whitePrimary'}
onClick={() => {
onResetForm();
putUpdateTeam({
teamId: userInfo?.team.teamId || '',
lafAccount: { token: '', appid: '' }
});
}}
>
</Button>
)}
</Flex>
{!!lafToken && (
<Flex alignItems={'center'} mt={5}>
<Box flex={'0 0 70px'}>{t('plugin.Currentapp')}</Box>
<MySelect
minW={'200px'}
list={
appListData
.filter((app) => app.state === 'Running')
.map((app) => ({
label: `${app.name}`,
value: app.appid
})) || []
}
placeholder={t('plugin.App')}
value={watch('appid')}
onchange={(e) => {
setValue('appid', e);
}}
{...(register('appid'), { required: true })}
/>
</Flex>
)}
</ModalBody>
<ModalFooter>
<Button mr={3} variant={'whiteBase'} onClick={onClose}>
{t('common.Close')}
</Button>
<Button isLoading={isUpdating} onClick={handleSubmit((data) => onSubmit(data))}>
{t('common.Update')}
</Button>
</ModalFooter>
</MyModal>
);
};
export default LafAccountModal;

View File

@@ -8,7 +8,8 @@ import {
Input,
Link,
Progress,
Grid
Grid,
Image
} from '@chakra-ui/react';
import { useForm } from 'react-hook-form';
import { UserUpdateParams } from '@/types/user';
@@ -41,12 +42,14 @@ import {
} from '@/web/support/wallet/sub/constants';
import StandardPlanContentList from '@/components/support/wallet/StandardPlanContentList';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
const StandDetailModal = dynamic(() => import('./standardDetailModal'));
const TeamMenu = dynamic(() => import('@/components/support/user/team/TeamMenu'));
const PayModal = dynamic(() => import('./PayModal'));
const UpdatePswModal = dynamic(() => import('./UpdatePswModal'));
const OpenAIAccountModal = dynamic(() => import('./OpenAIAccountModal'));
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
const CommunityModal = dynamic(() => import('@/components/CommunityModal'));
const Account = () => {
@@ -518,7 +521,7 @@ const Other = () => {
const { reset } = useForm<UserUpdateParams>({
defaultValues: userInfo as UserType
});
const { isOpen: isOpenLaf, onClose: onCloseLaf, onOpen: onOpenLaf } = useDisclosure();
const { isOpen: isOpenOpenai, onClose: onCloseOpenai, onOpen: onOpenOpenai } = useDisclosure();
const { isOpen: isOpenConcat, onClose: onCloseConcat, onOpen: onOpenConcat } = useDisclosure();
@@ -537,7 +540,6 @@ const Other = () => {
},
[reset, toast, updateUserInfo]
);
return (
<Box>
<Grid gridGap={4} mt={3}>
@@ -582,6 +584,32 @@ const Other = () => {
</Box>
</Link>
{feConfigs?.lafEnv && userInfo?.team.role === TeamMemberRoleEnum.owner && (
<Flex
bg={'white'}
py={4}
px={6}
border={theme.borders.sm}
borderWidth={'1.5px'}
borderRadius={'md'}
alignItems={'center'}
cursor={'pointer'}
userSelect={'none'}
onClick={onOpenLaf}
>
<Image src="/imgs/module/laf.png" w={'18px'} alt="laf" />
<Box ml={2} flex={1}>
laf
</Box>
<Box
w={'9px'}
h={'9px'}
borderRadius={'50%'}
bg={userInfo?.team.lafAccount?.token ? '#67c13b' : 'myGray.500'}
/>
</Flex>
)}
{feConfigs?.show_openai_account && (
<Flex
bg={'white'}
@@ -620,6 +648,9 @@ const Other = () => {
)}
</Grid>
{isOpenLaf && userInfo && (
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLaf} />
)}
{isOpenOpenai && userInfo && (
<OpenAIAccountModal
defaultData={userInfo?.openaiAccount}

View File

@@ -2,12 +2,14 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
import { addHours } from 'date-fns';
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import {
checkInvalidDatasetFiles,
checkInvalidDatasetData,
checkInvalidVector
} from '@/service/common/system/cronTask';
let deleteImageAmount = 0;
async function checkInvalidImg(start: Date, end: Date, limit = 50) {
@@ -60,11 +62,12 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
(async () => {
try {
console.log('执行脏数据清理任务');
const end = addHours(new Date(), -1);
// 360天 ~ 2小时前
const end = addHours(new Date(), -2);
const start = addHours(new Date(), -360 * 24);
await checkFiles(start, end);
await checkInvalidDatasetFiles(start, end);
await checkInvalidImg(start, end);
await checkInvalidCollection(start, end);
await checkInvalidDatasetData(start, end);
await checkInvalidVector(start, end);
console.log('执行脏数据清理任务完毕');
} catch (error) {

View File

@@ -1,98 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { MongoUsage } from '@fastgpt/service/support/wallet/usage/schema';
import { connectionMongo } from '@fastgpt/service/common/mongo';
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
import { addHours } from 'date-fns';
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
let deleteImageAmount = 0;
export async function checkInvalidImg(start: Date, end: Date, limit = 50) {
const images = await MongoImage.find(
{
createTime: {
$gte: start,
$lte: end
},
'metadata.relatedId': { $exists: true }
},
'_id teamId metadata'
);
console.log('total images', images.length);
let index = 0;
for await (const image of images) {
try {
// 1. 检测是否有对应的集合
const collection = await MongoDatasetCollection.findOne(
{
teamId: image.teamId,
'metadata.relatedImgId': image.metadata?.relatedId
},
'_id'
);
if (!collection) {
await image.deleteOne();
deleteImageAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成,共删除 ${deleteImageAmount} 个无效图片`);
}
/* pg 中的数据搬到 mongo dataset.datas 中,并做映射 */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
await authCert({ req, authRoot: true });
// 检查 usage 是否有记录
const totalUsages = await MongoUsage.countDocuments();
if (totalUsages === 0) {
// 重命名 bills 集合成 usages
await connectionMongo.connection.db.renameCollection('bills', 'usages', {
// 强制
dropTarget: true
});
}
(async () => {
try {
console.log('执行脏数据清理任务');
const end = addHours(new Date(), -1);
const start = addHours(new Date(), -360 * 24);
await checkFiles(start, end);
await checkInvalidImg(start, end);
await checkInvalidCollection(start, end);
await checkInvalidVector(start, end);
console.log('执行脏数据清理任务完毕');
} catch (error) {
console.log('执行脏数据清理任务出错了');
}
})();
jsonRes(res, {
message: 'success'
});
} catch (error) {
console.log(error);
jsonRes(res, {
code: 500,
error
});
}
}

View File

@@ -4,7 +4,7 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
@@ -19,7 +19,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
const { teamId } = await authFile({ req, authToken: true, fileId });
const { rawText } = await readFileContent({
const { rawText } = await readFileContentFromMongo({
teamId,
bucketName: BucketNameEnum.dataset,
fileId,

View File

@@ -90,6 +90,7 @@ export async function initSystemConfig() {
// get config from database
const config: FastGPTConfigFileType = {
feConfigs: {
...fileRes?.feConfigs,
...defaultFeConfigs,
...(dbConfig.feConfigs || {}),
isPlus: !!FastGPTProUrl

View File

@@ -1,7 +1,7 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
@@ -36,7 +36,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
});
// 1. read file
const { rawText, filename } = await readFileContent({
const { rawText, filename } = await readFileContentFromMongo({
teamId,
bucketName: BucketNameEnum.dataset,
fileId

View File

@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import {
delFileByFileIdList,
readFileContent
readFileContentFromMongo
} from '@fastgpt/service/common/file/gridfs/controller';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
@@ -47,7 +47,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
});
// 1. read file
const { rawText, filename } = await readFileContent({
const { rawText, filename } = await readFileContentFromMongo({
teamId,
bucketName: BucketNameEnum.dataset,
fileId

View File

@@ -4,7 +4,7 @@ import { connectToDatabase } from '@/service/mongo';
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
import { authFile } from '@fastgpt/service/support/permission/auth/file';
import { PostPreviewFilesChunksProps } from '@/global/core/dataset/api';
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants';
import { parseCsvTable2Chunks } from '@fastgpt/service/core/dataset/training/utils';
@@ -28,7 +28,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
const fileId = String(file._id);
const { rawText } = await readFileContent({
const { rawText } = await readFileContentFromMongo({
teamId,
bucketName: BucketNameEnum.dataset,
fileId,
@@ -53,7 +53,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
if (type === ImportDataSourceEnum.csvTable) {
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
const fileId = String(file._id);
const { rawText } = await readFileContent({
const { rawText } = await readFileContentFromMongo({
teamId,
bucketName: BucketNameEnum.dataset,
fileId,

View File

@@ -0,0 +1,69 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { request } from 'https';
import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';
import url from 'url';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { path = [], ...query } = req.query as any;
const queryStr = new URLSearchParams(query).toString();
const requestPath = queryStr
? `/${path?.join('/')}?${new URLSearchParams(query).toString()}`
: `/${path?.join('/')}`;
if (!requestPath) {
throw new Error('url is empty');
}
const lafEnv = global.feConfigs?.lafEnv;
if (!lafEnv) {
throw new Error('lafEnv is empty');
}
const parsedUrl = url.parse(lafEnv);
delete req.headers?.cookie;
delete req.headers?.host;
delete req.headers?.origin;
const requestResult = request({
protocol: parsedUrl.protocol,
hostname: parsedUrl.hostname,
port: parsedUrl.port,
path: requestPath,
method: req.method,
headers: req.headers,
timeout: 30000
});
req.pipe(requestResult);
requestResult.on('response', (response) => {
Object.keys(response.headers).forEach((key) => {
// @ts-ignore
res.setHeader(key, response.headers[key]);
});
response.statusCode && res.writeHead(response.statusCode);
response.pipe(res);
});
requestResult.on('error', (e) => {
res.send(e);
res.end();
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
}
export const config = {
api: {
bodyParser: false
}
};

View File

@@ -11,7 +11,7 @@ import { MongoTeamMember } from '@fastgpt/service/support/user/team/teamMemberSc
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { avatar, timezone, openaiAccount } = req.body as UserUpdateParams;
const { avatar, timezone, openaiAccount, lafAccount } = req.body as UserUpdateParams;
const { tmbId } = await authCert({ req, authToken: true });
const tmb = await MongoTeamMember.findById(tmbId);
@@ -47,7 +47,8 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
{
...(avatar && { avatar }),
...(timezone && { timezone }),
openaiAccount: openaiAccount?.key ? openaiAccount : null
openaiAccount: openaiAccount?.key ? openaiAccount : null,
lafAccount: lafAccount?.token ? lafAccount : null
}
);

View File

@@ -1,91 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import {
delFileByFileIdList,
getGFSCollection
} from '@fastgpt/service/common/file/gridfs/controller';
import { addLog } from '@fastgpt/service/common/system/log';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { addHours } from 'date-fns';
/*
check dataset.files data. If there is no match in dataset.collections, delete it
可能异常情况
1. 上传了文件,未成功创建集合
*/
let deleteFileAmount = 0;
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
const { startHour = 24, endHour = 1 } = req.body as {
startHour?: number;
endHour?: number;
limit?: number;
};
await authCert({ req, authRoot: true });
await connectToDatabase();
// start: now - maxDay, end: now - 3 day
const start = addHours(new Date(), -startHour);
const end = addHours(new Date(), -endHour);
deleteFileAmount = 0;
console.log(start, end);
await checkFiles(start, end);
jsonRes(res, {
data: deleteFileAmount,
message: 'success'
});
} catch (error) {
addLog.error(`check valid dataset files error`, error);
jsonRes(res, {
code: 500,
error
});
}
}
export async function checkFiles(start: Date, end: Date) {
const collection = getGFSCollection('dataset');
const where = {
uploadDate: { $gte: start, $lte: end }
};
// 1. get all file _id
const files = await collection
.find(where, {
projection: {
metadata: 1,
_id: 1
}
})
.toArray();
console.log('total files', files.length);
let index = 0;
for await (const file of files) {
try {
// 2. find fileId in dataset.collections
const hasCollection = await MongoDatasetCollection.countDocuments({
teamId: file.metadata.teamId,
fileId: file._id
});
// 3. if not found, delete file
if (hasCollection === 0) {
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
console.log('delete file', file._id);
deleteFileAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成,共删除 ${deleteFileAmount} 个无效文件`);
}

View File

@@ -1,96 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { addLog } from '@fastgpt/service/common/system/log';
import { deleteDatasetDataVector } from '@fastgpt/service/common/vectorStore/controller';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { addHours } from 'date-fns';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
/*
检测无效的 Mongo 数据
异常情况:
1. 训练过程删除知识库,可能导致还会有新的数据插入,导致无效。
*/
let deleteAmount = 0;
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
const { startHour = 3, endHour = 1 } = req.body as { startHour?: number; endHour?: number };
await authCert({ req, authRoot: true });
await connectToDatabase();
// start: now - maxDay, end: now - endHour
const start = addHours(new Date(), -startHour);
const end = addHours(new Date(), -endHour);
deleteAmount = 0;
await checkInvalidCollection(start, end);
jsonRes(res, {
data: deleteAmount,
message: 'success'
});
} catch (error) {
addLog.error(`check Invalid user error`, error);
jsonRes(res, {
code: 500,
error
});
}
}
export async function checkInvalidCollection(start: Date, end: Date) {
// 1. 获取时间范围的所有data
const rows = await MongoDatasetData.find(
{
updateTime: {
$gte: start,
$lte: end
}
},
'_id teamId collectionId'
).lean();
// 2. 合并所有的collectionId
const map = new Map<string, { teamId: string; collectionId: string }>();
for (const item of rows) {
const collectionId = String(item.collectionId);
if (!map.has(collectionId)) {
map.set(collectionId, { teamId: item.teamId, collectionId });
}
}
const list = Array.from(map.values());
console.log('total collections', list.length);
let index = 0;
for await (const item of list) {
try {
// 3. 查看该collection是否存在不存在则删除对应的数据
const collection = await MongoDatasetCollection.findOne({ _id: item.collectionId });
if (!collection) {
const result = await Promise.all([
MongoDatasetTraining.deleteMany({
teamId: item.teamId,
collectionId: item.collectionId
}),
MongoDatasetData.deleteMany({
teamId: item.teamId,
collectionId: item.collectionId
}),
deleteDatasetDataVector({
teamId: item.teamId,
collectionIds: [String(item.collectionId)]
})
]);
console.log(result);
console.log('collection is not found', item);
continue;
}
} catch (error) {}
console.log(++index);
}
}

View File

@@ -1,86 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
import { addLog } from '@fastgpt/service/common/system/log';
import {
deleteDatasetDataVector,
getVectorDataByTime
} from '@fastgpt/service/common/vectorStore/controller';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { addHours } from 'date-fns';
/*
检测无效的 Vector 数据.
异常情况:
1. 插入数据时vector成功mongo失败
2. 更新数据,也会有插入 vector
*/
let deletedVectorAmount = 0;
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
const { startHour = 5, endHour = 1 } = req.body as { startHour?: number; endHour?: number };
await authCert({ req, authRoot: true });
await connectToDatabase();
// start: now - maxDay, end: now - endHour
const start = addHours(new Date(), -startHour);
const end = addHours(new Date(), -endHour);
deletedVectorAmount = 0;
await checkInvalidVector(start, end);
jsonRes(res, {
data: deletedVectorAmount,
message: 'success'
});
} catch (error) {
addLog.error(`check Invalid user error`, error);
jsonRes(res, {
code: 500,
error
});
}
}
export async function checkInvalidVector(start: Date, end: Date) {
// 1. get all vector data
const rows = await getVectorDataByTime(start, end);
console.log('total data', rows.length);
let index = 0;
for await (const item of rows) {
if (!item.teamId || !item.datasetId || !item.id) {
console.log('error data', item);
continue;
}
try {
// 2. find dataset.data
const hasData = await MongoDatasetData.countDocuments({
teamId: item.teamId,
datasetId: item.datasetId,
'indexes.dataId': item.id
});
// 3. if not found, delete vector
if (hasData === 0) {
await deleteDatasetDataVector({
teamId: item.teamId,
id: item.id
});
console.log('delete vector data', item.id);
deletedVectorAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
console.log(`检测完成,共删除 ${deletedVectorAmount} 个无效 向量 数据`);
}

View File

@@ -117,3 +117,5 @@ export const RenderUploadFiles = ({
</>
) : null;
};
export default RenderUploadFiles;

View File

@@ -249,7 +249,7 @@ const InputDataModal = ({
return openConfirm(onDeleteData)();
}
if (e === TabEnum.doc) {
return window.open(getDocPath('/docs/course/datasetengine'), '_blank');
return window.open(getDocPath('/docs/course/dataset_engine'), '_blank');
}
setCurrentTab(e);
}}

View File

@@ -1,22 +1,61 @@
import { setCron } from '@fastgpt/service/common/system/cron';
import { startTrainingQueue } from '@/service/core/dataset/training/utils';
import { clearTmpUploadFiles } from '@fastgpt/service/common/file/utils';
import { checkInvalidDatasetFiles, checkInvalidDatasetData, checkInvalidVector } from './cronTask';
import { checkTimerLock } from '@fastgpt/service/common/system/timerLock/utils';
import { TimerIdEnum } from '@fastgpt/service/common/system/timerLock/constants';
import { addHours } from 'date-fns';
export const startCron = () => {
setTrainingQueueCron();
setClearTmpUploadFilesCron();
};
export const setTrainingQueueCron = () => {
const setTrainingQueueCron = () => {
setCron('*/1 * * * *', () => {
startTrainingQueue();
});
};
export const setClearTmpUploadFilesCron = () => {
clearTmpUploadFiles();
const setClearTmpUploadFilesCron = () => {
// Clear tmp upload files every ten minutes
setCron('*/10 * * * *', () => {
clearTmpUploadFiles();
});
};
const clearInvalidDataCron = () => {
setCron('0 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInValidDatasetFiles,
lockMinuted: 59
})
) {
checkInvalidDatasetFiles(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
setCron('10 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInvalidDatasetData,
lockMinuted: 59
})
) {
checkInvalidDatasetData(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
setCron('30 */1 * * *', async () => {
if (
await checkTimerLock({
timerId: TimerIdEnum.checkInvalidVector,
lockMinuted: 59
})
) {
checkInvalidVector(addHours(new Date(), 2), addHours(new Date(), 6));
}
});
};
export const startCron = () => {
setTrainingQueueCron();
setClearTmpUploadFilesCron();
clearInvalidDataCron();
};

View File

@@ -0,0 +1,157 @@
import {
delFileByFileIdList,
getGFSCollection
} from '@fastgpt/service/common/file/gridfs/controller';
import { addLog } from '@fastgpt/service/common/system/log';
import {
deleteDatasetDataVector,
getVectorDataByTime
} from '@fastgpt/service/common/vectorStore/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
/*
check dataset.files data. If there is no match in dataset.collections, delete it
可能异常情况
1. 上传了文件,未成功创建集合
*/
export async function checkInvalidDatasetFiles(start: Date, end: Date) {
let deleteFileAmount = 0;
const collection = getGFSCollection('dataset');
const where = {
uploadDate: { $gte: start, $lte: end }
};
// 1. get all file _id
const files = await collection
.find(where, {
projection: {
metadata: 1,
_id: 1
}
})
.toArray();
addLog.info(`Clear invalid dataset files, total files: ${files.length}`);
let index = 0;
for await (const file of files) {
try {
// 2. find fileId in dataset.collections
const hasCollection = await MongoDatasetCollection.countDocuments({
teamId: file.metadata.teamId,
fileId: file._id
});
// 3. if not found, delete file
if (hasCollection === 0) {
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
console.log('delete file', file._id);
deleteFileAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
addLog.info(`Clear invalid dataset files finish, remove ${deleteFileAmount} files`);
}
/*
检测无效的 Mongo 数据
异常情况:
1. 训练过程删除知识库,可能导致还会有新的数据继续插入,导致无效。
*/
export async function checkInvalidDatasetData(start: Date, end: Date) {
// 1. 获取时间范围的所有data
const rows = await MongoDatasetData.find(
{
updateTime: {
$gte: start,
$lte: end
}
},
'_id teamId collectionId'
).lean();
// 2. 合并所有的collectionId
const map = new Map<string, { teamId: string; collectionId: string }>();
for (const item of rows) {
const collectionId = String(item.collectionId);
if (!map.has(collectionId)) {
map.set(collectionId, { teamId: item.teamId, collectionId });
}
}
const list = Array.from(map.values());
addLog.info(`Clear invalid dataset data, total collections: ${list.length}`);
let index = 0;
for await (const item of list) {
try {
// 3. 查看该collection是否存在不存在则删除对应的数据
const collection = await MongoDatasetCollection.findOne({ _id: item.collectionId });
if (!collection) {
const result = await Promise.all([
MongoDatasetTraining.deleteMany({
teamId: item.teamId,
collectionId: item.collectionId
}),
MongoDatasetData.deleteMany({
teamId: item.teamId,
collectionId: item.collectionId
}),
deleteDatasetDataVector({
teamId: item.teamId,
collectionIds: [String(item.collectionId)]
})
]);
console.log(result);
console.log('collection is not found', item);
continue;
}
} catch (error) {}
console.log(++index);
}
}
export async function checkInvalidVector(start: Date, end: Date) {
let deletedVectorAmount = 0;
// 1. get all vector data
const rows = await getVectorDataByTime(start, end);
addLog.info(`Clear invalid vector, total vector data: ${rows.length}`);
let index = 0;
for await (const item of rows) {
if (!item.teamId || !item.datasetId || !item.id) {
addLog.error('error data', item);
continue;
}
try {
// 2. find dataset.data
const hasData = await MongoDatasetData.countDocuments({
teamId: item.teamId,
datasetId: item.datasetId,
'indexes.dataId': item.id
});
// 3. if not found, delete vector
if (hasData === 0) {
await deleteDatasetDataVector({
teamId: item.teamId,
id: item.id
});
console.log('delete vector data', item.id);
deletedVectorAmount++;
}
index++;
index % 100 === 0 && console.log(index);
} catch (error) {
console.log(error);
}
}
addLog.info(`Clear invalid vector finish, remove ${deletedVectorAmount} data`);
}

View File

@@ -44,8 +44,7 @@ export async function insertData2Dataset({
indexes =
Array.isArray(indexes) && indexes.length > 0
? indexes.map((index) => ({
// @ts-ignore
...index.toObject(),
text: index.text,
dataId: undefined,
defaultIndex: index.text.trim() === qaStr
}))

View File

@@ -1,9 +1,11 @@
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import type { UserModelSchema } from '@fastgpt/global/support/user/type';
import { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
export interface UserUpdateParams {
balance?: number;
avatar?: string;
timezone?: string;
openaiAccount?: UserModelSchema['openaiAccount'];
lafAccount?: LafAccountType;
}

View File

@@ -0,0 +1,169 @@
import axios, {
Method,
InternalAxiosRequestConfig,
AxiosResponse,
AxiosProgressEvent
} from 'axios';
import { useUserStore } from '@/web/support/user/useUserStore';
interface ConfigType {
headers?: { [key: string]: string };
timeout?: number;
onUploadProgress?: (progressEvent: AxiosProgressEvent) => void;
cancelToken?: AbortController;
maxQuantity?: number;
}
interface ResponseDataType {
error: string | null;
data: any;
}
const maxQuantityMap: Record<
string,
{
amount: number;
sign: AbortController;
}
> = {};
function requestStart({ url, maxQuantity }: { url: string; maxQuantity?: number }) {
if (!maxQuantity) return;
const item = maxQuantityMap[url];
if (item) {
if (item.amount >= maxQuantity && item.sign) {
item.sign.abort();
delete maxQuantityMap[url];
}
} else {
maxQuantityMap[url] = {
amount: 1,
sign: new AbortController()
};
}
}
function requestFinish({ url }: { url: string }) {
const item = maxQuantityMap[url];
if (item) {
item.amount--;
if (item.amount <= 0) {
delete maxQuantityMap[url];
}
}
}
/**
* 请求开始
*/
function startInterceptors(config: InternalAxiosRequestConfig): InternalAxiosRequestConfig {
if (config.headers && !config.headers.Authorization) {
config.headers.Authorization = `Bearer ${useUserStore.getState().userInfo?.team?.lafAccount?.token || ''}`;
}
return config;
}
/**
* 请求成功,检查请求头
*/
function responseSuccess(response: AxiosResponse<ResponseDataType>) {
return response;
}
/**
* 响应数据检查
*/
function checkRes(data: ResponseDataType) {
if (data === undefined) {
console.log('error->', data, 'data is empty');
return Promise.reject('服务器异常');
} else if (data.error) {
return responseError(data.error);
}
return data.data;
}
/**
* 响应错误
*/
function responseError(err: any) {
console.log('error->', '请求错误', err);
if (!err) {
return Promise.reject({ message: '未知错误' });
}
if (typeof err === 'string') {
return Promise.reject({ message: err });
}
if (err?.response?.data) {
return Promise.reject(err?.response?.data);
}
return Promise.reject(err);
}
/* 创建请求实例 */
const instance = axios.create({
timeout: 60000, // 超时时间
headers: {
'content-type': 'application/json'
}
});
/* 请求拦截 */
instance.interceptors.request.use(startInterceptors, (err) => Promise.reject(err));
/* 响应拦截 */
instance.interceptors.response.use(responseSuccess, (err) => Promise.reject(err));
function request(
url: string,
data: any,
{ cancelToken, maxQuantity, ...config }: ConfigType,
method: Method
): any {
/* 去空 */
for (const key in data) {
if (data[key] === null || data[key] === undefined) {
delete data[key];
}
}
requestStart({ url, maxQuantity });
return instance
.request({
baseURL: '/api/lafApi',
url,
method,
data: ['POST', 'PUT'].includes(method) ? data : null,
params: !['POST', 'PUT'].includes(method) ? data : null,
signal: cancelToken?.signal,
...config // 用户自定义配置,可以覆盖前面的配置
})
.then((res) => checkRes(res.data))
.catch((err) => responseError(err))
.finally(() => requestFinish({ url }));
}
/**
* api请求方式
* @param {String} url
* @param {Any} params
* @param {Object} config
* @returns
*/
export function GET<T = undefined>(url: string, params = {}, config: ConfigType = {}): Promise<T> {
return request(url, params, config, 'GET');
}
export function POST<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'POST');
}
export function PUT<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'PUT');
}
export function DELETE<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
return request(url, data, config, 'DELETE');
}

View File

@@ -0,0 +1,34 @@
import { GET, POST, PUT } from '@/web/common/api/lafRequest';
export const postLafPat2Token = (pat: string) => POST<string>(`/v1/auth/pat2token`, { pat });
export const getLafApplications = (token: string) =>
GET<
{
appid: string;
name: string;
state: 'Running' | 'Failed' | 'Stopped';
}[]
>(
`/v1/applications`,
{},
{
headers: {
Authorization: `Bearer ${token}`
}
}
);
export const getLafAppDetail = (appid: string) =>
GET<{
appid: string;
name: string;
openapi_token: string;
domain: {
_id: string;
appid: string;
domain: string;
state: string;
phase: string;
};
}>(`/v1/applications/${appid}`);

View File

@@ -38,7 +38,7 @@
**镜像和端口**
+ 镜像名: `luanshaotong/reranker:v0.2`
+ 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
+ 端口号: 6006
```