mirror of
https://github.com/labring/FastGPT.git
synced 2025-07-23 05:12:39 +00:00
4.7.1-alpha (#1120)
Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>
This commit is contained in:
@@ -60,7 +60,7 @@ fastgpt.run 域名会弃用。
|
||||
- [x] 混合检索 & 重排
|
||||
- [x] Tool 模块
|
||||
- [ ] 嵌入 [Laf](https://github.com/labring/laf),实现在线编写 HTTP 模块
|
||||
- [ ] 插件封装功能
|
||||
- [ ] 插件封装功能,支持低代码渲染
|
||||
|
||||
`2` 知识库能力
|
||||
- [x] 多库复用,混用
|
||||
@@ -68,9 +68,8 @@ fastgpt.run 域名会弃用。
|
||||
- [x] 支持知识库单独设置向量模型
|
||||
- [x] 源文件存储
|
||||
- [x] 支持手动输入,直接分段,QA 拆分导入
|
||||
- [x] 支持 pdf,docx,txt,html,md,csv
|
||||
- [x] 支持。txt, 。md, 。html, 。pdf, 。docx,pptx, 。csv, 。xlsx (有需要更多可 PR file loader)
|
||||
- [x] 支持 url 读取、CSV 批量导入
|
||||
- [ ] 支持 PPT、Excel 导入
|
||||
- [ ] 支持文件阅读器
|
||||
- [ ] 更多的数据预处理方案
|
||||
|
||||
@@ -114,7 +113,7 @@ fastgpt.run 域名会弃用。
|
||||
* [多模型配置](https://doc.fastgpt.in/docs/development/one-api/)
|
||||
* [版本更新/升级介绍](https://doc.fastgpt.in/docs/development/upgrading)
|
||||
* [OpenAPI API 文档](https://doc.fastgpt.in/docs/development/openapi/)
|
||||
* [知识库结构详解](https://doc.fastgpt.in/docs/course/datasetengine/)
|
||||
* [知识库结构详解](https://doc.fastgpt.in/docs/course/dataset_engine/)
|
||||
|
||||
<a href="#readme">
|
||||
<img src="https://img.shields.io/badge/-返回顶部-7d09f1.svg" alt="#" align="right">
|
||||
|
@@ -64,7 +64,7 @@ Tips: 可以通过点击上下文按键查看完整的上下文组成,便于
|
||||
|
||||
FastGPT 知识库采用 QA 对(不一定都是问答格式,仅代表两个变量)的格式存储,在转义成字符串时候会根据**引用模板**来进行格式化。知识库包含多个可用变量: q, a, sourceId(数据的ID), index(第n个数据), source(数据的集合名、文件名),score(距离得分,0-1) 可以通过 {{q}} {{a}} {{sourceId}} {{index}} {{source}} {{score}} 按需引入。下面一个模板例子:
|
||||
|
||||
可以通过 [知识库结构讲解](/docs/course/datasetEngine/) 了解详细的知识库的结构。
|
||||
可以通过 [知识库结构讲解](/docs/course/dataset_engine/) 了解详细的知识库的结构。
|
||||
|
||||
#### 引用模板
|
||||
|
||||
|
@@ -1,93 +0,0 @@
|
||||
---
|
||||
title: "知识库结构讲解"
|
||||
description: "本节会详细介绍 FastGPT 知识库结构设计,理解其 QA 的存储格式和多向量映射,以便更好的构建知识库。这篇介绍主要以使用为主,详细原理不多介绍。"
|
||||
icon: "dataset"
|
||||
draft: false
|
||||
toc: true
|
||||
weight: 102
|
||||
---
|
||||
|
||||
## 理解向量
|
||||
|
||||
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库,要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
|
||||
|
||||
人类的文字、图片、视频等媒介是无法直接被计算机理解的,要想让计算机理解两段文字是否有相似性、相关性,通常需要将它们转成计算机可以理解的语言,向量是其中的一种方式。
|
||||
|
||||
向量可以简单理解为一个数字数组,两个向量之间可以通过数学公式得出一个`距离`,距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上,可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
|
||||
|
||||
而由于文字是有多种类型,并且拥有成千上万种组合方式,因此在转成向量进行相似度匹配时,很难保障其精确性。在向量方案构建的知识库中,通常使用`topk`召回的方式,也就是查找前`k`个最相似的内容,丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`,从而实现知识库问答。因此,在知识库问答中,向量搜索的环节是最为重要的。
|
||||
|
||||
影响向量搜索精度的因素非常多,主要包括:向量模型的质量、数据的质量(长度,完整性,多样性)、检索器的精度(速度与精度之间的取舍)。与数据质量对应的就是检索词的质量。
|
||||
|
||||
检索器的精度比较容易解决,向量模型的训练略复杂,因此数据和检索词质量优化成了一个重要的环节。
|
||||
|
||||
## FastGPT 中向量的结构设计
|
||||
|
||||
FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器,索引为`HNSW`。且`PostgresSQL`仅用于向量检索,`MongoDB`用于其他数据的存取。
|
||||
|
||||
在`MongoDB`的`dataset.datas`表中,会存储向量原数据的信息,同时有一个`indexes`字段,会记录其对应的向量ID,这是一个数组,也就是说,一组向量可以对应多组数据。
|
||||
|
||||
在`PostgresSQL`的表中,设置一个 `index` 字段用于存储向量。在检索时,会先召回向量,再根据向量的ID,去`MongoDB`中寻找原数据内容,如果对应了同一组原数据,则进行合并,向量得分取最高得分。
|
||||
|
||||

|
||||
|
||||
### 多向量的目的和使用方式
|
||||
|
||||
在一组向量中,内容的长度和语义的丰富度通常是矛盾的,无法兼得。因此,FastGPT 采用了多向量映射的方式,将一组数据映射到多组向量中,从而保障数据的完整性和语义的丰富度。
|
||||
|
||||
你可以为一组较长的文本,添加多组向量,从而在检索时,只要其中一组向量被检索到,该数据也将被召回。
|
||||
|
||||
### 提高向量搜索精度的方法
|
||||
|
||||
1. 更好分词分段:当一段话的结构和语义是完整的,并且是单一的,精度也会提高。因此,许多系统都会优化分词器,尽可能的保障每组数据的完整性。
|
||||
2. 精简`index`的内容,减少向量内容的长度:当`index`的内容更少,更准确时,检索精度自然会提高。但与此同时,会牺牲一定的检索范围,适合答案较为严格的场景。
|
||||
3. 丰富`index`的数量,可以为同一个`chunk`内容增加多组`index`。
|
||||
4. 优化检索词:在实际使用过程中,用户的问题通常是模糊的或是缺失的,并不一定是完整清晰的问题。因此优化用户的问题(检索词)很大程度上也可以提高精度。
|
||||
5. 微调向量模型:由于市面上直接使用的向量模型都是通用型模型,在特定领域的检索精度并不高,因此微调向量模型可以很大程度上提高专业领域的检索效果。
|
||||
|
||||
## FastGPT 构建知识库方案
|
||||
|
||||
在 FastGPT 中,整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`,一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`,也就是说,知识库搜索时,是对整个`库`进行搜索,而集合仅是为了对数据进行分类管理,与搜索效果无关。(起码目前还是)
|
||||
|
||||
| 库 | 集合 | 数据 |
|
||||
| --- | --- | --- |
|
||||
|  |  |  |
|
||||
|
||||
### 导入数据方案1 - 直接分段导入
|
||||
|
||||
选择文件导入时,可以选择直接分段方案。直接分段会利用`句子分词器`对文本进行一定长度拆分,最终分割中多组的`q`。如果使用了直接分段方案,我们建议在`应用`设置`引用提示词`时,使用`通用模板`即可,无需选择`问答模板`。
|
||||
|
||||
| 交互 | 结果 |
|
||||
| --- | --- |
|
||||
|  |  |
|
||||
|
||||
|
||||
### 导入数据方案2 - QA导入
|
||||
|
||||
选择文件导入时,可以选择QA拆分方案。仍然需要使用到`句子分词器`对文本进行拆分,但长度比直接分段大很多。在导入后,会先调用`大模型`对分段进行学习,并给出一些`问题`和`答案`,最终问题和答案会一起被存储到`q`中。注意,新版的 FastGPT 为了提高搜索的范围,不再将问题和答案分别存储到 qa 中。
|
||||
|
||||
| 交互 | 结果 |
|
||||
| --- | --- |
|
||||
|  |  |
|
||||
|
||||
### 导入数据方案3 - 手动录入
|
||||
|
||||
在 FastGPT 中,你可以在任何一个`集合`中点击右上角的`插入`手动录入知识点,或者使用`标注`功能手动录入。被搜索的内容为`q`,补充内容(可选)为`a`。
|
||||
|
||||
| | | |
|
||||
| --- | --- | --- |
|
||||
|  |  |  |
|
||||
|
||||
### 导入数据方案4 - CSV录入
|
||||
|
||||
有些数据较为独特,可能需要单独的进行预处理分割后再导入 FastGPT,此时可以选择 csv 导入,可批量的将处理好的数据导入。
|
||||
|
||||

|
||||
|
||||
### 导入数据方案5 - API导入
|
||||
|
||||
参考[FastGPT OpenAPI使用](/docs/development/openapi)。
|
||||
|
||||
## QA的组合与引用提示词构建
|
||||
|
||||
参考[引用模板与引用提示词示例](/docs/course/ai_settings/#示例)
|
136
docSite/content/docs/course/dataset_engine.md
Normal file
136
docSite/content/docs/course/dataset_engine.md
Normal file
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: '知识库搜索方案和参数'
|
||||
description: '本节会详细介绍 FastGPT 知识库结构设计,理解其 QA 的存储格式和多向量映射,以便更好的构建知识库。同时会介绍每个搜索参数的功能。这篇介绍主要以使用为主,详细原理不多介绍。'
|
||||
icon: 'language'
|
||||
draft: false
|
||||
toc: true
|
||||
weight: 106
|
||||
---
|
||||
|
||||
## 理解向量
|
||||
|
||||
FastGPT 采用了 RAG 中的 Embedding 方案构建知识库,要使用好 FastGPT 需要简单的理解`Embedding`向量是如何工作的及其特点。
|
||||
|
||||
人类的文字、图片、视频等媒介是无法直接被计算机理解的,要想让计算机理解两段文字是否有相似性、相关性,通常需要将它们转成计算机可以理解的语言,向量是其中的一种方式。
|
||||
|
||||
向量可以简单理解为一个数字数组,两个向量之间可以通过数学公式得出一个`距离`,距离越小代表两个向量的相似度越大。从而映射到文字、图片、视频等媒介上,可以用来判断两个媒介之间的相似度。向量搜索便是利用了这个原理。
|
||||
|
||||
而由于文字是有多种类型,并且拥有成千上万种组合方式,因此在转成向量进行相似度匹配时,很难保障其精确性。在向量方案构建的知识库中,通常使用`topk`召回的方式,也就是查找前`k`个最相似的内容,丢给大模型去做更进一步的`语义判断`、`逻辑推理`和`归纳总结`,从而实现知识库问答。因此,在知识库问答中,向量搜索的环节是最为重要的。
|
||||
|
||||
影响向量搜索精度的因素非常多,主要包括:向量模型的质量、数据的质量(长度,完整性,多样性)、检索器的精度(速度与精度之间的取舍)。与数据质量对应的就是检索词的质量。
|
||||
|
||||
检索器的精度比较容易解决,向量模型的训练略复杂,因此数据和检索词质量优化成了一个重要的环节。
|
||||
|
||||
|
||||
### 提高向量搜索精度的方法
|
||||
|
||||
1. 更好分词分段:当一段话的结构和语义是完整的,并且是单一的,精度也会提高。因此,许多系统都会优化分词器,尽可能的保障每组数据的完整性。
|
||||
2. 精简`index`的内容,减少向量内容的长度:当`index`的内容更少,更准确时,检索精度自然会提高。但与此同时,会牺牲一定的检索范围,适合答案较为严格的场景。
|
||||
3. 丰富`index`的数量,可以为同一个`chunk`内容增加多组`index`。
|
||||
4. 优化检索词:在实际使用过程中,用户的问题通常是模糊的或是缺失的,并不一定是完整清晰的问题。因此优化用户的问题(检索词)很大程度上也可以提高精度。
|
||||
5. 微调向量模型:由于市面上直接使用的向量模型都是通用型模型,在特定领域的检索精度并不高,因此微调向量模型可以很大程度上提高专业领域的检索效果。
|
||||
|
||||
## FastGPT 构建知识库方案
|
||||
|
||||
### 数据存储结构
|
||||
|
||||
在 FastGPT 中,整个知识库由库、集合和数据 3 部分组成。集合可以简单理解为一个`文件`。一个`库`中可以包含多个`集合`,一个`集合`中可以包含多组`数据`。最小的搜索单位是`库`,也就是说,知识库搜索时,是对整个`库`进行搜索,而集合仅是为了对数据进行分类管理,与搜索效果无关。(起码目前还是)
|
||||
|
||||

|
||||
|
||||
### 向量存储结构
|
||||
|
||||
FastGPT 采用了`PostgresSQL`的`PG Vector`插件作为向量检索器,索引为`HNSW`。且`PostgresSQL`仅用于向量检索(该引擎可以替换成其它数据库),`MongoDB`用于其他数据的存取。
|
||||
|
||||
在`MongoDB`的`dataset.datas`表中,会存储向量原数据的信息,同时有一个`indexes`字段,会记录其对应的向量ID,这是一个数组,也就是说,一组向量可以对应多组数据。
|
||||
|
||||
在`PostgresSQL`的表中,设置一个`vector`字段用于存储向量。在检索时,会先召回向量,再根据向量的ID,去`MongoDB`中寻找原数据内容,如果对应了同一组原数据,则进行合并,向量得分取最高得分。
|
||||
|
||||

|
||||
|
||||
### 多向量的目的和使用方式
|
||||
|
||||
在一组向量中,内容的长度和语义的丰富度通常是矛盾的,无法兼得。因此,FastGPT 采用了多向量映射的方式,将一组数据映射到多组向量中,从而保障数据的完整性和语义的丰富度。
|
||||
|
||||
你可以为一组较长的文本,添加多组向量,从而在检索时,只要其中一组向量被检索到,该数据也将被召回。
|
||||
|
||||
意味着,你可以通过标注数据块的方式,不断提高数据块的精度。
|
||||
|
||||
### 检索方案
|
||||
|
||||
1. 通过`问题优化`实现指代消除和问题扩展,从而增加连续对话的检索能力以及语义丰富度。
|
||||
2. 通过`Concat query`来增加`Rerank`连续对话的时,排序的准确性。
|
||||
3. 通过`RRF`合并方式,综合多个渠道的检索效果。
|
||||
4. 通过`Rerank`来二次排序,提高精度。
|
||||
|
||||

|
||||
|
||||
|
||||
## 搜索参数
|
||||
| | | |
|
||||
| --- |---| --- |
|
||||
||  |  |
|
||||
|
||||
### 搜索模式
|
||||
|
||||
#### 语义检索
|
||||
|
||||
语义检索是通过向量距离,计算用户问题与知识库内容的距离,从而得出“相似度”,当然这并不是语文上的相似度,而是数学上的。
|
||||
|
||||
优点:
|
||||
- 相近语义理解
|
||||
- 跨多语言理解(例如输入中文问题匹配英文知识点)
|
||||
- 多模态理解(文本,图片,音视频等)
|
||||
|
||||
缺点:
|
||||
- 依赖模型训练效果
|
||||
- 精度不稳定
|
||||
- 受关键词和句子完整度影响
|
||||
|
||||
#### 全文检索
|
||||
|
||||
采用传统的全文检索方式。适合查找关键的主谓语等。
|
||||
|
||||
#### 混合检索
|
||||
|
||||
同时使用向量检索和全文检索,并通过 RRF 公式进行两个搜索结果合并,一般情况下搜索结果会更加丰富准确。
|
||||
|
||||
由于混合检索后的查找范围很大,并且无法直接进行相似度过滤,通常需要进行利用重排模型进行一次结果重新排序,并利用重排的得分进行过滤。
|
||||
|
||||
#### 结果重排
|
||||
|
||||
利用`ReRank`模型对搜索结果进行重排,绝大多数情况下,可以有效提高搜索结果的准确率。不过,重排模型与问题的完整度(主谓语齐全)有一些关系,通常会先走问题优化后再进行搜索-重排。重排后可以得到一个`0-1`的得分,代表着搜索内容与问题的相关度,该分数通常比向量的得分更加精确,可以根据得分进行过滤。
|
||||
|
||||
FastGPT 会使用 `RRF` 对重排结果、向量搜索结果、全文检索结果进行合并,得到最终的搜索结果。
|
||||
|
||||
### 搜索过滤
|
||||
|
||||
#### 引用上限
|
||||
|
||||
每次搜索最多引用`n`个`tokens`的内容。
|
||||
|
||||
之所以不采用`top k`,是发现在混合知识库(问答库、文档库)时,不同`chunk`的长度差距很大,会导致`top k`的结果不稳定,因此采用了`tokens`的方式进行引用上限的控制。
|
||||
|
||||
#### 最低相关度
|
||||
|
||||
一个`0-1`的数值,会过滤掉一些低相关度的搜索结果。
|
||||
|
||||
该值仅在`语义检索`或使用`结果重排`时生效。
|
||||
|
||||
### 问题优化
|
||||
|
||||
#### 背景
|
||||
|
||||
在 RAG 中,我们需要根据输入的问题去数据库里执行 embedding 搜索,查找相关的内容,从而查找到相似的内容(简称知识库搜索)。
|
||||
|
||||
在搜索的过程中,尤其是连续对话的搜索,我们通常会发现后续的问题难以搜索到合适的内容,其中一个原因是知识库搜索只会使用“当前”的问题去执行。看下面的例子:
|
||||
|
||||

|
||||
|
||||
用户在提问“第二点是什么”的时候,只会去知识库里查找“第二点是什么”,压根查不到内容。实际上需要查询的是“QA结构是什么”。因此我们需要引入一个【问题优化】模块,来对用户当前的问题进行补全,从而使得知识库搜索能够搜索到合适的内容。使用补全后效果如下:
|
||||
|
||||

|
||||
|
||||
#### 实现方式
|
||||
|
||||
在进行`数据检索`前,会先让模型进行`指代消除`与`问题扩展`,一方面可以可以解决指代对象不明确问题,同时可以扩展问题的语义丰富度。你可以通过每次对话后的对话详情,查看补全的结果。
|
@@ -19,6 +19,9 @@ llm模型全部合并
|
||||
|
||||
```json
|
||||
{
|
||||
"feConfigs": {
|
||||
"lafEnv": "https://laf.dev" // laf环境
|
||||
},
|
||||
"systemEnv": {
|
||||
"vectorMaxProcess": 15,
|
||||
"qaMaxProcess": 15,
|
||||
@@ -164,7 +167,7 @@ llm模型全部合并
|
||||
"model": "bge-reranker-base", // 随意
|
||||
"name": "检索重排-base", // 随意
|
||||
"charsPointsPrice": 0,
|
||||
"requestUrl": "{{host}}/api/v1/rerank",
|
||||
"requestUrl": "{{host}}/v1/rerank",
|
||||
"requestAuth": "安全凭证,已自动补 Bearer"
|
||||
}
|
||||
]
|
||||
|
@@ -44,7 +44,7 @@ weight: 910
|
||||
|
||||
### docker 部署
|
||||
|
||||
+ 镜像名: `luanshaotong/reranker:v0.2`
|
||||
+ 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
|
||||
+ 端口号: 6006
|
||||
+ 大小:约8GB
|
||||
|
||||
@@ -56,12 +56,12 @@ ACCESS_TOKEN=mytoken
|
||||
**运行命令示例**
|
||||
- 无需GPU环境,使用CPU运行
|
||||
```sh
|
||||
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
|
||||
docker run -d --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
|
||||
```
|
||||
|
||||
- 需要CUDA 11.7环境
|
||||
```sh
|
||||
docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken luanshaotong/reranker:v0.2
|
||||
docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
|
||||
```
|
||||
|
||||
**docker-compose.yml示例**
|
||||
@@ -69,7 +69,7 @@ docker run -d --gpus all --name reranker -p 6006:6006 -e ACCESS_TOKEN=mytoken lu
|
||||
version: "3"
|
||||
services:
|
||||
reranker:
|
||||
image: luanshaotong/reranker:v0.2
|
||||
image: registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2
|
||||
container_name: reranker
|
||||
# GPU运行环境,如果宿主机未安装,将deploy配置隐藏即可
|
||||
deploy:
|
||||
|
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: 'V4.7(进行中)'
|
||||
title: 'V4.7'
|
||||
description: 'FastGPT V4.7更新说明'
|
||||
icon: 'upgrade'
|
||||
draft: false
|
||||
@@ -26,7 +26,7 @@ curl --location --request POST 'https://{{host}}/api/admin/initv47' \
|
||||
|
||||
## 3. 升级 ReRank 模型
|
||||
|
||||
4.7对ReRank模型进行了格式变动,兼容 cohere 的格式,可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型,需要修改镜像为:`luanshaotong/reranker:v0.2` 。
|
||||
4.7对ReRank模型进行了格式变动,兼容 cohere 的格式,可以直接使用 cohere 提供的 API。如果是本地的 ReRank 模型,需要修改镜像为:`registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2` 。
|
||||
|
||||
cohere的重排模型对中文不是很好,感觉不如 bge 的好用,接入教程如下:
|
||||
|
||||
|
28
docSite/content/docs/development/upgrading/471.md
Normal file
28
docSite/content/docs/development/upgrading/471.md
Normal file
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: 'V4.7.1'
|
||||
description: 'FastGPT V4.7.1 更新说明'
|
||||
icon: 'upgrade'
|
||||
draft: false
|
||||
toc: true
|
||||
weight: 825
|
||||
---
|
||||
|
||||
## 初始化脚本
|
||||
|
||||
从任意终端,发起 1 个 HTTP 请求。其中 {{rootkey}} 替换成环境变量里的 `rootkey`;{{host}} 替换成FastGPT的域名。
|
||||
|
||||
```bash
|
||||
curl --location --request POST 'https://{{host}}/api/admin/clearInvalidData' \
|
||||
--header 'rootkey: {{rootkey}}' \
|
||||
--header 'Content-Type: application/json'
|
||||
```
|
||||
|
||||
该请求会执行脏数据清理(清理无效的文件、清理无效的图片、清理无效的知识库集合、清理无效的向量)
|
||||
|
||||
## V4.7.1 更新说明
|
||||
|
||||
1. 新增 - Pptx 和 xlsx 文件读取。但所有文件读取都放服务端,会消耗更多的服务器资源,以及无法在上传时预览更多内容。
|
||||
2. 新增 - 集成 Laf 云函数,可以读取 Laf 账号中的云函数作为 HTTP 模块。
|
||||
3. 新增 - 定时器,清理垃圾数据。(采用小范围清理,会清理最近n个小时的,所以请保证服务持续运行,长时间不允许,可以继续执行 clearInvalidData 的接口进行全量清理。)
|
||||
4. 修改 - csv导入模板,取消 header 校验,自动获取前两列。
|
||||
5. 修复 - 工具调用模块连线数据类型校验错误。
|
@@ -1,5 +1,7 @@
|
||||
export const fileImgs = [
|
||||
{ suffix: 'pdf', src: 'file/fill/pdf' },
|
||||
{ suffix: 'ppt', src: 'file/fill/ppt' },
|
||||
{ suffix: 'xlsx', src: 'file/fill/xlsx' },
|
||||
{ suffix: 'csv', src: 'file/fill/csv' },
|
||||
{ suffix: '(doc|docs)', src: 'file/fill/doc' },
|
||||
{ suffix: 'txt', src: 'file/fill/txt' },
|
||||
|
@@ -11,5 +11,5 @@ export const formatFileSize = (bytes: number): string => {
|
||||
};
|
||||
|
||||
export const detectFileEncoding = (buffers: string | Buffer) => {
|
||||
return detect(buffers)?.encoding || 'utf-8';
|
||||
return (detect(buffers)?.encoding || 'utf-8') as BufferEncoding;
|
||||
};
|
||||
|
@@ -57,6 +57,7 @@ export type FastGPTFeConfigsType = {
|
||||
|
||||
uploadFileMaxAmount?: number;
|
||||
uploadFileMaxSize?: number;
|
||||
lafEnv?: string;
|
||||
};
|
||||
|
||||
export type SystemEnvType = {
|
||||
|
@@ -61,7 +61,8 @@ export enum FlowNodeTypeEnum {
|
||||
pluginOutput = 'pluginOutput',
|
||||
queryExtension = 'cfr',
|
||||
tools = 'tools',
|
||||
stopTool = 'stopTool'
|
||||
stopTool = 'stopTool',
|
||||
lafModule = 'lafModule'
|
||||
|
||||
// abandon
|
||||
}
|
||||
|
@@ -20,6 +20,7 @@ import { AiQueryExtension } from './system/queryExtension';
|
||||
|
||||
import type { FlowNodeTemplateType, moduleTemplateListType } from '../../module/type.d';
|
||||
import { FlowNodeTemplateTypeEnum } from '../../module/constants';
|
||||
import { lafModule } from './system/laf';
|
||||
|
||||
/* app flow module templates */
|
||||
export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
|
||||
@@ -35,7 +36,8 @@ export const appSystemModuleTemplates: FlowNodeTemplateType[] = [
|
||||
ClassifyQuestionModule,
|
||||
ContextExtractModule,
|
||||
HttpModule468,
|
||||
AiQueryExtension
|
||||
AiQueryExtension,
|
||||
lafModule
|
||||
];
|
||||
/* plugin flow module templates */
|
||||
export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
|
||||
@@ -51,7 +53,8 @@ export const pluginSystemModuleTemplates: FlowNodeTemplateType[] = [
|
||||
ClassifyQuestionModule,
|
||||
ContextExtractModule,
|
||||
HttpModule468,
|
||||
AiQueryExtension
|
||||
AiQueryExtension,
|
||||
lafModule
|
||||
];
|
||||
|
||||
/* all module */
|
||||
@@ -73,7 +76,8 @@ export const moduleTemplatesFlat: FlowNodeTemplateType[] = [
|
||||
PluginInputModule,
|
||||
PluginOutputModule,
|
||||
RunPluginModule,
|
||||
AiQueryExtension
|
||||
AiQueryExtension,
|
||||
lafModule
|
||||
];
|
||||
|
||||
export const moduleTemplatesList: moduleTemplateListType = [
|
||||
|
86
packages/global/core/module/template/system/laf.ts
Normal file
86
packages/global/core/module/template/system/laf.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
import {
|
||||
FlowNodeInputTypeEnum,
|
||||
FlowNodeOutputTypeEnum,
|
||||
FlowNodeTypeEnum
|
||||
} from '../../node/constant';
|
||||
import { FlowNodeTemplateType } from '../../type';
|
||||
import {
|
||||
ModuleIOValueTypeEnum,
|
||||
ModuleInputKeyEnum,
|
||||
ModuleOutputKeyEnum,
|
||||
FlowNodeTemplateTypeEnum
|
||||
} from '../../constants';
|
||||
import {
|
||||
Input_Template_DynamicInput,
|
||||
Input_Template_Switch,
|
||||
Input_Template_AddInputParam
|
||||
} from '../input';
|
||||
import { Output_Template_Finish, Output_Template_AddOutput } from '../output';
|
||||
|
||||
export const lafModule: FlowNodeTemplateType = {
|
||||
id: FlowNodeTypeEnum.lafModule,
|
||||
templateType: FlowNodeTemplateTypeEnum.externalCall,
|
||||
flowType: FlowNodeTypeEnum.lafModule,
|
||||
avatar: '/imgs/module/laf.png',
|
||||
name: 'Laf 函数调用(测试)',
|
||||
intro: '可以调用Laf账号下的云函数。',
|
||||
showStatus: true,
|
||||
isTool: true,
|
||||
inputs: [
|
||||
Input_Template_Switch,
|
||||
{
|
||||
key: ModuleInputKeyEnum.httpReqUrl,
|
||||
type: FlowNodeInputTypeEnum.hidden,
|
||||
valueType: ModuleIOValueTypeEnum.string,
|
||||
label: '',
|
||||
description: 'core.module.input.description.Http Request Url',
|
||||
placeholder: 'https://api.ai.com/getInventory',
|
||||
required: false,
|
||||
showTargetInApp: false,
|
||||
showTargetInPlugin: false
|
||||
},
|
||||
Input_Template_DynamicInput,
|
||||
{
|
||||
...Input_Template_AddInputParam,
|
||||
editField: {
|
||||
key: true,
|
||||
description: true,
|
||||
dataType: true
|
||||
},
|
||||
defaultEditField: {
|
||||
label: '',
|
||||
key: '',
|
||||
description: '',
|
||||
inputType: FlowNodeInputTypeEnum.target,
|
||||
valueType: ModuleIOValueTypeEnum.string
|
||||
}
|
||||
}
|
||||
],
|
||||
outputs: [
|
||||
{
|
||||
key: ModuleOutputKeyEnum.httpRawResponse,
|
||||
label: '原始响应',
|
||||
description: 'HTTP请求的原始响应。只能接受字符串或JSON类型响应数据。',
|
||||
valueType: ModuleIOValueTypeEnum.any,
|
||||
type: FlowNodeOutputTypeEnum.source,
|
||||
targets: []
|
||||
},
|
||||
{
|
||||
...Output_Template_AddOutput,
|
||||
editField: {
|
||||
key: true,
|
||||
description: true,
|
||||
dataType: true,
|
||||
defaultValue: true
|
||||
},
|
||||
defaultEditField: {
|
||||
label: '',
|
||||
key: '',
|
||||
description: '',
|
||||
outputType: FlowNodeOutputTypeEnum.source,
|
||||
valueType: ModuleIOValueTypeEnum.string
|
||||
}
|
||||
},
|
||||
Output_Template_Finish
|
||||
]
|
||||
};
|
1
packages/global/core/module/type.d.ts
vendored
1
packages/global/core/module/type.d.ts
vendored
@@ -9,6 +9,7 @@ import { DispatchNodeResponseKeyEnum } from './runtime/constants';
|
||||
import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type';
|
||||
import { UserModelSchema } from 'support/user/type';
|
||||
import {
|
||||
ChatItemType,
|
||||
ChatItemValueItemType,
|
||||
ToolRunResponseItemType,
|
||||
UserChatItemValueItemType
|
||||
|
@@ -41,7 +41,7 @@ export const str2OpenApiSchema = async (yamlStr = ''): Promise<OpenApiJsonSchema
|
||||
path,
|
||||
method,
|
||||
name: methodInfo.operationId || path,
|
||||
description: methodInfo.description,
|
||||
description: methodInfo.description || methodInfo.summary,
|
||||
params: methodInfo.parameters,
|
||||
request: methodInfo?.requestBody
|
||||
};
|
||||
|
@@ -1,5 +1,5 @@
|
||||
import { TeamMemberRoleEnum } from './constant';
|
||||
import { TeamMemberSchema } from './type';
|
||||
import { LafAccountType, TeamMemberSchema } from './type';
|
||||
|
||||
export type AuthTeamRoleProps = {
|
||||
teamId: string;
|
||||
@@ -10,12 +10,14 @@ export type CreateTeamProps = {
|
||||
name: string;
|
||||
avatar?: string;
|
||||
defaultTeam?: boolean;
|
||||
lafAccount?: LafAccountType;
|
||||
};
|
||||
export type UpdateTeamProps = {
|
||||
teamId: string;
|
||||
name?: string;
|
||||
avatar?: string;
|
||||
teamDomain?: string;
|
||||
lafAccount?: null | LafAccountType;
|
||||
};
|
||||
|
||||
/* ------------- member ----------- */
|
||||
|
8
packages/global/support/user/team/type.d.ts
vendored
8
packages/global/support/user/team/type.d.ts
vendored
@@ -1,5 +1,6 @@
|
||||
import type { UserModelSchema } from '../type';
|
||||
import type { TeamMemberRoleEnum, TeamMemberStatusEnum } from './constant';
|
||||
import { LafAccountType } from './type';
|
||||
|
||||
export type TeamSchema = {
|
||||
_id: string;
|
||||
@@ -13,6 +14,7 @@ export type TeamSchema = {
|
||||
lastExportDatasetTime: Date;
|
||||
lastWebsiteSyncTime: Date;
|
||||
};
|
||||
lafAccount: LafAccountType;
|
||||
};
|
||||
export type tagsType = {
|
||||
label: string;
|
||||
@@ -58,6 +60,7 @@ export type TeamItemType = {
|
||||
role: `${TeamMemberRoleEnum}`;
|
||||
status: `${TeamMemberStatusEnum}`;
|
||||
canWrite: boolean;
|
||||
lafAccount?: LafAccountType;
|
||||
};
|
||||
|
||||
export type TeamMemberItemType = {
|
||||
@@ -74,3 +77,8 @@ export type TeamTagItemType = {
|
||||
label: string;
|
||||
key: string;
|
||||
};
|
||||
|
||||
export type LafAccountType = {
|
||||
token: string;
|
||||
appid: string;
|
||||
};
|
||||
|
@@ -6,16 +6,9 @@ import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
|
||||
import { MongoFileSchema } from './schema';
|
||||
import { detectFileEncoding } from '@fastgpt/global/common/file/tools';
|
||||
import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
|
||||
import { readFileRawText } from '../read/rawText';
|
||||
import { ReadFileByBufferParams } from '../read/type';
|
||||
import { readMarkdown } from '../read/markdown';
|
||||
import { readHtmlRawText } from '../read/html';
|
||||
import { readPdfFile } from '../read/pdf';
|
||||
import { readWordFile } from '../read/word';
|
||||
import { readCsvRawText } from '../read/csv';
|
||||
import { MongoRwaTextBuffer } from '../../buffer/rawText/schema';
|
||||
import { readPptxRawText } from '../read/pptx';
|
||||
import { readXlsxRawText } from '../read/xlsx';
|
||||
import { readFileRawContent } from '../read/utils';
|
||||
|
||||
export function getGFSCollection(bucket: `${BucketNameEnum}`) {
|
||||
MongoFileSchema;
|
||||
@@ -146,7 +139,7 @@ export const readFileEncode = async ({
|
||||
return encoding as BufferEncoding;
|
||||
};
|
||||
|
||||
export const readFileContent = async ({
|
||||
export const readFileContentFromMongo = async ({
|
||||
teamId,
|
||||
bucketName,
|
||||
fileId,
|
||||
@@ -205,47 +198,14 @@ export const readFileContent = async ({
|
||||
}
|
||||
};
|
||||
|
||||
const { rawText } = await (async () => {
|
||||
switch (extension) {
|
||||
case 'txt':
|
||||
return readFileRawText(params);
|
||||
case 'md':
|
||||
return readMarkdown(params);
|
||||
case 'html':
|
||||
return readHtmlRawText(params);
|
||||
case 'pdf':
|
||||
return readPdfFile(params);
|
||||
case 'docx':
|
||||
return readWordFile(params);
|
||||
case 'pptx':
|
||||
return readPptxRawText(params);
|
||||
case 'xlsx':
|
||||
const xlsxResult = await readXlsxRawText(params);
|
||||
if (csvFormat) {
|
||||
return {
|
||||
rawText: xlsxResult.formatText || ''
|
||||
};
|
||||
}
|
||||
return {
|
||||
rawText: xlsxResult.rawText
|
||||
};
|
||||
case 'csv':
|
||||
const csvResult = await readCsvRawText(params);
|
||||
if (csvFormat) {
|
||||
return {
|
||||
rawText: csvResult.formatText || ''
|
||||
};
|
||||
}
|
||||
return {
|
||||
rawText: csvResult.rawText
|
||||
};
|
||||
default:
|
||||
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
|
||||
}
|
||||
})();
|
||||
const { rawText } = await readFileRawContent({
|
||||
extension,
|
||||
csvFormat,
|
||||
params
|
||||
});
|
||||
|
||||
if (rawText.trim()) {
|
||||
await MongoRwaTextBuffer.create({
|
||||
MongoRwaTextBuffer.create({
|
||||
sourceId: fileId,
|
||||
rawText,
|
||||
metadata: {
|
||||
|
@@ -2,6 +2,15 @@ import { markdownProcess } from '@fastgpt/global/common/string/markdown';
|
||||
import { uploadMongoImg } from '../image/controller';
|
||||
import { MongoImageTypeEnum } from '@fastgpt/global/common/file/image/constants';
|
||||
import { addHours } from 'date-fns';
|
||||
import { ReadFileByBufferParams } from './type';
|
||||
import { readFileRawText } from '../read/rawText';
|
||||
import { readMarkdown } from '../read/markdown';
|
||||
import { readHtmlRawText } from '../read/html';
|
||||
import { readPdfFile } from '../read/pdf';
|
||||
import { readWordFile } from '../read/word';
|
||||
import { readCsvRawText } from '../read/csv';
|
||||
import { readPptxRawText } from '../read/pptx';
|
||||
import { readXlsxRawText } from '../read/xlsx';
|
||||
|
||||
export const initMarkdownText = ({
|
||||
teamId,
|
||||
@@ -23,3 +32,50 @@ export const initMarkdownText = ({
|
||||
expiredTime: addHours(new Date(), 2)
|
||||
})
|
||||
});
|
||||
|
||||
export const readFileRawContent = async ({
|
||||
extension,
|
||||
csvFormat,
|
||||
params
|
||||
}: {
|
||||
csvFormat?: boolean;
|
||||
extension: string;
|
||||
params: ReadFileByBufferParams;
|
||||
}) => {
|
||||
switch (extension) {
|
||||
case 'txt':
|
||||
return readFileRawText(params);
|
||||
case 'md':
|
||||
return readMarkdown(params);
|
||||
case 'html':
|
||||
return readHtmlRawText(params);
|
||||
case 'pdf':
|
||||
return readPdfFile(params);
|
||||
case 'docx':
|
||||
return readWordFile(params);
|
||||
case 'pptx':
|
||||
return readPptxRawText(params);
|
||||
case 'xlsx':
|
||||
const xlsxResult = await readXlsxRawText(params);
|
||||
if (csvFormat) {
|
||||
return {
|
||||
rawText: xlsxResult.formatText || ''
|
||||
};
|
||||
}
|
||||
return {
|
||||
rawText: xlsxResult.rawText
|
||||
};
|
||||
case 'csv':
|
||||
const csvResult = await readCsvRawText(params);
|
||||
if (csvFormat) {
|
||||
return {
|
||||
rawText: csvResult.formatText || ''
|
||||
};
|
||||
}
|
||||
return {
|
||||
rawText: csvResult.rawText
|
||||
};
|
||||
default:
|
||||
return Promise.reject('Only support .txt, .md, .html, .pdf, .docx, pptx, .csv, .xlsx');
|
||||
}
|
||||
};
|
||||
|
@@ -55,8 +55,8 @@ export const clearTmpUploadFiles = () => {
|
||||
fs.stat(filePath, (err, stats) => {
|
||||
if (err) return;
|
||||
|
||||
// 如果文件是在1小时前上传的,则认为是临时文件并删除它
|
||||
if (Date.now() - stats.mtime.getTime() > 1 * 60 * 60 * 1000) {
|
||||
// 如果文件是在2小时前上传的,则认为是临时文件并删除它
|
||||
if (Date.now() - stats.mtime.getTime() > 2 * 60 * 60 * 1000) {
|
||||
fs.unlink(filePath, (err) => {
|
||||
if (err) return;
|
||||
console.log(`Deleted temp file: ${filePath}`);
|
||||
|
15
packages/service/common/system/timerLock/constants.ts
Normal file
15
packages/service/common/system/timerLock/constants.ts
Normal file
@@ -0,0 +1,15 @@
|
||||
export enum TimerIdEnum {
|
||||
checkInValidDatasetFiles = 'checkInValidDatasetFiles',
|
||||
checkInvalidDatasetData = 'checkInvalidDatasetData',
|
||||
checkInvalidVector = 'checkInvalidVector',
|
||||
clearExpiredSubPlan = 'clearExpiredSubPlan',
|
||||
updateStandardPlan = 'updateStandardPlan'
|
||||
}
|
||||
|
||||
export const timerIdMap = {
|
||||
[TimerIdEnum.checkInValidDatasetFiles]: 'checkInValidDatasetFiles',
|
||||
[TimerIdEnum.checkInvalidDatasetData]: 'checkInvalidDatasetData',
|
||||
[TimerIdEnum.checkInvalidVector]: 'checkInvalidVector',
|
||||
[TimerIdEnum.clearExpiredSubPlan]: 'clearExpiredSubPlan',
|
||||
[TimerIdEnum.updateStandardPlan]: 'updateStandardPlan'
|
||||
};
|
29
packages/service/common/system/timerLock/schema.ts
Normal file
29
packages/service/common/system/timerLock/schema.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
import { connectionMongo, type Model } from '../../mongo';
|
||||
import { timerIdMap } from './constants';
|
||||
const { Schema, model, models } = connectionMongo;
|
||||
import { TimerLockSchemaType } from './type.d';
|
||||
|
||||
export const collectionName = 'systemtimerlocks';
|
||||
|
||||
const TimerLockSchema = new Schema({
|
||||
timerId: {
|
||||
type: String,
|
||||
required: true,
|
||||
unique: true,
|
||||
enum: Object.keys(timerIdMap)
|
||||
},
|
||||
expiredTime: {
|
||||
type: Date,
|
||||
required: true
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
TimerLockSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 5 });
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
|
||||
export const MongoTimerLock: Model<TimerLockSchemaType> =
|
||||
models[collectionName] || model(collectionName, TimerLockSchema);
|
||||
MongoTimerLock.syncIndexes();
|
5
packages/service/common/system/timerLock/type.d.ts
vendored
Normal file
5
packages/service/common/system/timerLock/type.d.ts
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
export type TimerLockSchemaType = {
|
||||
_id: string;
|
||||
timerId: string;
|
||||
expiredTime: Date;
|
||||
};
|
25
packages/service/common/system/timerLock/utils.ts
Normal file
25
packages/service/common/system/timerLock/utils.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
import { TimerIdEnum } from './constants';
|
||||
import { MongoTimerLock } from './schema';
|
||||
import { addMinutes } from 'date-fns';
|
||||
|
||||
/*
|
||||
利用唯一健,使得同一时间只有一个任务在执行,后创建的锁,会因唯一健创建失败,从而无法继续执行任务
|
||||
*/
|
||||
export const checkTimerLock = async ({
|
||||
timerId,
|
||||
lockMinuted
|
||||
}: {
|
||||
timerId: `${TimerIdEnum}`;
|
||||
lockMinuted: number;
|
||||
}) => {
|
||||
try {
|
||||
await MongoTimerLock.create({
|
||||
timerId,
|
||||
expiredTime: addMinutes(new Date(), lockMinuted)
|
||||
});
|
||||
|
||||
return true;
|
||||
} catch (error) {
|
||||
return false;
|
||||
}
|
||||
};
|
@@ -210,7 +210,6 @@ export const runToolWithToolChoice = async (
|
||||
).filter(Boolean) as ToolRunResponseType;
|
||||
|
||||
const flatToolsResponseData = toolsRunResponse.map((item) => item.moduleRunResponse).flat();
|
||||
|
||||
if (toolCalls.length > 0 && !res.closed) {
|
||||
// Run the tool, combine its results, and perform another round of AI calls
|
||||
const assistantToolMsgParams: ChatCompletionAssistantToolParam = {
|
||||
|
@@ -37,6 +37,7 @@ import { dispatchRunTools } from './agent/runTool/index';
|
||||
import { ChatItemValueTypeEnum } from '@fastgpt/global/core/chat/constants';
|
||||
import { DispatchFlowResponse } from './type';
|
||||
import { dispatchStopToolCall } from './agent/runTool/stopTool';
|
||||
import { dispatchLafRequest } from './tools/runLaf';
|
||||
|
||||
const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
|
||||
[FlowNodeTypeEnum.historyNode]: dispatchHistory,
|
||||
@@ -56,6 +57,7 @@ const callbackMap: Record<`${FlowNodeTypeEnum}`, Function> = {
|
||||
[FlowNodeTypeEnum.queryExtension]: dispatchQueryExtension,
|
||||
[FlowNodeTypeEnum.tools]: dispatchRunTools,
|
||||
[FlowNodeTypeEnum.stopTool]: dispatchStopToolCall,
|
||||
[FlowNodeTypeEnum.lafModule]: dispatchLafRequest,
|
||||
|
||||
// none
|
||||
[FlowNodeTypeEnum.userGuide]: () => Promise.resolve()
|
||||
|
@@ -61,7 +61,7 @@ export const dispatchHttp468Request = async (props: HttpRequestProps): Promise<H
|
||||
chatId,
|
||||
responseChatItemId,
|
||||
...variables,
|
||||
histories: histories.slice(0, 10),
|
||||
histories: histories.slice(-10),
|
||||
...body
|
||||
};
|
||||
|
||||
@@ -165,6 +165,7 @@ async function fetchData({
|
||||
'Content-Type': 'application/json',
|
||||
...headers
|
||||
},
|
||||
timeout: 120000,
|
||||
params: params,
|
||||
data: ['POST', 'PUT', 'PATCH'].includes(method) ? body : undefined
|
||||
});
|
||||
|
212
packages/service/core/workflow/dispatch/tools/runLaf.ts
Normal file
212
packages/service/core/workflow/dispatch/tools/runLaf.ts
Normal file
@@ -0,0 +1,212 @@
|
||||
import type { ModuleDispatchProps } from '@fastgpt/global/core/module/type.d';
|
||||
import {
|
||||
DYNAMIC_INPUT_KEY,
|
||||
ModuleInputKeyEnum,
|
||||
ModuleOutputKeyEnum
|
||||
} from '@fastgpt/global/core/module/constants';
|
||||
import { DispatchNodeResponseKeyEnum } from '@fastgpt/global/core/module/runtime/constants';
|
||||
import axios from 'axios';
|
||||
import { valueTypeFormat } from '../utils';
|
||||
import { SERVICE_LOCAL_HOST } from '../../../../common/system/tools';
|
||||
import { addLog } from '../../../../common/system/log';
|
||||
import { DispatchNodeResultType } from '@fastgpt/global/core/module/runtime/type';
|
||||
|
||||
type LafRequestProps = ModuleDispatchProps<{
|
||||
[ModuleInputKeyEnum.httpReqUrl]: string;
|
||||
[DYNAMIC_INPUT_KEY]: Record<string, any>;
|
||||
[key: string]: any;
|
||||
}>;
|
||||
type LafResponse = DispatchNodeResultType<{
|
||||
[ModuleOutputKeyEnum.failed]?: boolean;
|
||||
[key: string]: any;
|
||||
}>;
|
||||
|
||||
const UNDEFINED_SIGN = 'UNDEFINED_SIGN';
|
||||
|
||||
export const dispatchLafRequest = async (props: LafRequestProps): Promise<LafResponse> => {
|
||||
let {
|
||||
appId,
|
||||
chatId,
|
||||
responseChatItemId,
|
||||
variables,
|
||||
module: { outputs },
|
||||
histories,
|
||||
params: { system_httpReqUrl: httpReqUrl, [DYNAMIC_INPUT_KEY]: dynamicInput, ...body }
|
||||
} = props;
|
||||
|
||||
if (!httpReqUrl) {
|
||||
return Promise.reject('Http url is empty');
|
||||
}
|
||||
|
||||
const concatVariables = {
|
||||
appId,
|
||||
chatId,
|
||||
responseChatItemId,
|
||||
...variables,
|
||||
...body
|
||||
};
|
||||
|
||||
httpReqUrl = replaceVariable(httpReqUrl, concatVariables);
|
||||
|
||||
const requestBody = {
|
||||
systemParams: {
|
||||
appId,
|
||||
chatId,
|
||||
responseChatItemId,
|
||||
histories: histories.slice(0, 10)
|
||||
},
|
||||
variables,
|
||||
...dynamicInput,
|
||||
...body
|
||||
};
|
||||
|
||||
try {
|
||||
const { formatResponse, rawResponse } = await fetchData({
|
||||
method: 'POST',
|
||||
url: httpReqUrl,
|
||||
body: requestBody
|
||||
});
|
||||
|
||||
// format output value type
|
||||
const results: Record<string, any> = {};
|
||||
for (const key in formatResponse) {
|
||||
const output = outputs.find((item) => item.key === key);
|
||||
if (!output) continue;
|
||||
results[key] = valueTypeFormat(formatResponse[key], output.valueType);
|
||||
}
|
||||
|
||||
return {
|
||||
assistantResponses: [],
|
||||
[DispatchNodeResponseKeyEnum.nodeResponse]: {
|
||||
totalPoints: 0,
|
||||
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
|
||||
httpResult: rawResponse
|
||||
},
|
||||
[DispatchNodeResponseKeyEnum.toolResponses]: rawResponse,
|
||||
[ModuleOutputKeyEnum.httpRawResponse]: rawResponse,
|
||||
...results
|
||||
};
|
||||
} catch (error) {
|
||||
addLog.error('Http request error', error);
|
||||
return {
|
||||
[ModuleOutputKeyEnum.failed]: true,
|
||||
[DispatchNodeResponseKeyEnum.nodeResponse]: {
|
||||
totalPoints: 0,
|
||||
body: Object.keys(requestBody).length > 0 ? requestBody : undefined,
|
||||
httpResult: { error: formatHttpError(error) }
|
||||
}
|
||||
};
|
||||
}
|
||||
};
|
||||
|
||||
async function fetchData({
|
||||
method,
|
||||
url,
|
||||
body
|
||||
}: {
|
||||
method: string;
|
||||
url: string;
|
||||
body: Record<string, any>;
|
||||
}): Promise<Record<string, any>> {
|
||||
const { data: response } = await axios({
|
||||
method,
|
||||
baseURL: `http://${SERVICE_LOCAL_HOST}`,
|
||||
url,
|
||||
headers: {
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
data: body
|
||||
});
|
||||
|
||||
const parseJson = (obj: Record<string, any>, prefix = '') => {
|
||||
let result: Record<string, any> = {};
|
||||
|
||||
if (Array.isArray(obj)) {
|
||||
for (let i = 0; i < obj.length; i++) {
|
||||
result[`${prefix}[${i}]`] = obj[i];
|
||||
|
||||
if (Array.isArray(obj[i])) {
|
||||
result = {
|
||||
...result,
|
||||
...parseJson(obj[i], `${prefix}[${i}]`)
|
||||
};
|
||||
} else if (typeof obj[i] === 'object') {
|
||||
result = {
|
||||
...result,
|
||||
...parseJson(obj[i], `${prefix}[${i}].`)
|
||||
};
|
||||
}
|
||||
}
|
||||
} else if (typeof obj == 'object') {
|
||||
for (const key in obj) {
|
||||
result[`${prefix}${key}`] = obj[key];
|
||||
|
||||
if (Array.isArray(obj[key])) {
|
||||
result = {
|
||||
...result,
|
||||
...parseJson(obj[key], `${prefix}${key}`)
|
||||
};
|
||||
} else if (typeof obj[key] === 'object') {
|
||||
result = {
|
||||
...result,
|
||||
...parseJson(obj[key], `${prefix}${key}.`)
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
};
|
||||
|
||||
return {
|
||||
formatResponse:
|
||||
typeof response === 'object' && !Array.isArray(response) ? parseJson(response) : {},
|
||||
rawResponse: response
|
||||
};
|
||||
}
|
||||
|
||||
function replaceVariable(text: string, obj: Record<string, any>) {
|
||||
for (const [key, value] of Object.entries(obj)) {
|
||||
if (value === undefined) {
|
||||
text = text.replace(new RegExp(`{{${key}}}`, 'g'), UNDEFINED_SIGN);
|
||||
} else {
|
||||
const replacement = JSON.stringify(value);
|
||||
const unquotedReplacement =
|
||||
replacement.startsWith('"') && replacement.endsWith('"')
|
||||
? replacement.slice(1, -1)
|
||||
: replacement;
|
||||
text = text.replace(new RegExp(`{{${key}}}`, 'g'), unquotedReplacement);
|
||||
}
|
||||
}
|
||||
return text || '';
|
||||
}
|
||||
function removeUndefinedSign(obj: Record<string, any>) {
|
||||
for (const key in obj) {
|
||||
if (obj[key] === UNDEFINED_SIGN) {
|
||||
obj[key] = undefined;
|
||||
} else if (Array.isArray(obj[key])) {
|
||||
obj[key] = obj[key].map((item: any) => {
|
||||
if (item === UNDEFINED_SIGN) {
|
||||
return undefined;
|
||||
} else if (typeof item === 'object') {
|
||||
removeUndefinedSign(item);
|
||||
}
|
||||
return item;
|
||||
});
|
||||
} else if (typeof obj[key] === 'object') {
|
||||
removeUndefinedSign(obj[key]);
|
||||
}
|
||||
}
|
||||
return obj;
|
||||
}
|
||||
function formatHttpError(error: any) {
|
||||
return {
|
||||
message: error?.message,
|
||||
name: error?.name,
|
||||
method: error?.config?.method,
|
||||
baseURL: error?.config?.baseURL,
|
||||
url: error?.config?.url,
|
||||
code: error?.code,
|
||||
status: error?.status
|
||||
};
|
||||
}
|
@@ -10,7 +10,6 @@ import { MongoTeam } from './teamSchema';
|
||||
|
||||
async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> {
|
||||
const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
|
||||
|
||||
if (!tmb) {
|
||||
return Promise.reject('member not exist');
|
||||
}
|
||||
@@ -27,7 +26,8 @@ async function getTeamMember(match: Record<string, any>): Promise<TeamItemType>
|
||||
role: tmb.role,
|
||||
status: tmb.status,
|
||||
defaultTeam: tmb.defaultTeam,
|
||||
canWrite: tmb.role !== TeamMemberRoleEnum.visitor
|
||||
canWrite: tmb.role !== TeamMemberRoleEnum.visitor,
|
||||
lafAccount: tmb.teamId.lafAccount
|
||||
};
|
||||
}
|
||||
|
||||
|
@@ -35,6 +35,14 @@ const TeamSchema = new Schema({
|
||||
lastWebsiteSyncTime: {
|
||||
type: Date
|
||||
}
|
||||
},
|
||||
lafAccount: {
|
||||
token: {
|
||||
type: String
|
||||
},
|
||||
appid: {
|
||||
type: String
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
|
@@ -125,7 +125,9 @@ export const iconPaths = {
|
||||
'file/fill/manual': () => import('./icons/file/fill/manual.svg'),
|
||||
'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'),
|
||||
'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'),
|
||||
'file/fill/ppt': () => import('./icons/file/fill/ppt.svg'),
|
||||
'file/fill/txt': () => import('./icons/file/fill/txt.svg'),
|
||||
'file/fill/xlsx': () => import('./icons/file/fill/xlsx.svg'),
|
||||
'file/html': () => import('./icons/file/html.svg'),
|
||||
'file/indexImport': () => import('./icons/file/indexImport.svg'),
|
||||
'file/manualImport': () => import('./icons/file/manualImport.svg'),
|
||||
|
11
packages/web/components/common/Icon/icons/file/fill/ppt.svg
Normal file
11
packages/web/components/common/Icon/icons/file/fill/ppt.svg
Normal file
@@ -0,0 +1,11 @@
|
||||
<svg t="1712108366314" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="1576"
|
||||
width="128" height="128">
|
||||
<path
|
||||
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
|
||||
fill="#E34221" p-id="1577"></path>
|
||||
<path d="M960 326.4v16H755.2s-100.8-20.8-99.2-108.8c0 0 4.8 92.8 97.6 92.8H960z" fill="#DC3119" p-id="1578"></path>
|
||||
<path d="M657.6 0v233.6c0 25.6 17.6 92.8 97.6 92.8H960L657.6 0z" fill="#FFFFFF" p-id="1579"></path>
|
||||
<path
|
||||
d="M304 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V686.4c0-9.6 8-17.6 17.6-17.6H304c38.4 0 59.2 25.6 59.2 57.6S340.8 784 304 784z m-3.2-94.4h-51.2v73.6h51.2c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8zM480 784h-54.4v67.2c0 6.4-4.8 11.2-11.2 11.2-6.4 0-11.2-4.8-11.2-11.2V686.4c0-9.6 6.4-17.6 16-17.6H480c38.4 0 59.2 25.6 59.2 57.6S518.4 784 480 784z m-3.2-94.4h-49.6v73.6h49.6c22.4 0 38.4-16 38.4-36.8 0-22.4-16-36.8-38.4-36.8z m225.6 0h-52.8v161.6c0 6.4-4.8 11.2-11.2 11.2-6.4 0-12.8-4.8-12.8-11.2V689.6h-51.2c-6.4 0-11.2-4.8-11.2-11.2 0-4.8 4.8-9.6 11.2-9.6h128c6.4 0 11.2 4.8 11.2 11.2 0 4.8-4.8 9.6-11.2 9.6z"
|
||||
fill="#FFFFFF" p-id="1580"></path>
|
||||
</svg>
|
After Width: | Height: | Size: 1.2 KiB |
11
packages/web/components/common/Icon/icons/file/fill/xlsx.svg
Normal file
11
packages/web/components/common/Icon/icons/file/fill/xlsx.svg
Normal file
@@ -0,0 +1,11 @@
|
||||
<svg t="1712108380525" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2654"
|
||||
width="128" height="128">
|
||||
<path
|
||||
d="M145.6 0C100.8 0 64 36.8 64 81.6v860.8C64 987.2 100.8 1024 145.6 1024h732.8c44.8 0 81.6-36.8 81.6-81.6V324.8L657.6 0h-512z"
|
||||
fill="#45B058" p-id="2655"></path>
|
||||
<path
|
||||
d="M374.4 862.4c-3.2 0-6.4-1.6-8-3.2l-59.2-80-60.8 80c-1.6 1.6-4.8 3.2-8 3.2-6.4 0-11.2-4.8-11.2-11.2 0-1.6 0-4.8 1.6-6.4l62.4-81.6-57.6-78.4c-1.6-1.6-3.2-3.2-3.2-6.4 0-4.8 4.8-11.2 11.2-11.2 4.8 0 8 1.6 9.6 4.8l56 73.6 54.4-73.6c1.6-3.2 4.8-4.8 8-4.8 6.4 0 12.8 4.8 12.8 11.2 0 3.2-1.6 4.8-1.6 6.4l-59.2 76.8 62.4 83.2c1.6 1.6 3.2 4.8 3.2 6.4 0 6.4-6.4 11.2-12.8 11.2z m160-1.6H448c-9.6 0-17.6-8-17.6-17.6V678.4c0-6.4 4.8-11.2 12.8-11.2 6.4 0 11.2 4.8 11.2 11.2v161.6h80c6.4 0 11.2 4.8 11.2 9.6 0 6.4-4.8 11.2-11.2 11.2z m112 3.2c-28.8 0-51.2-9.6-67.2-24-3.2-1.6-3.2-4.8-3.2-8 0-6.4 3.2-12.8 11.2-12.8 1.6 0 4.8 1.6 6.4 3.2 12.8 11.2 32 20.8 54.4 20.8 33.6 0 44.8-19.2 44.8-33.6 0-49.6-113.6-22.4-113.6-89.6 0-32 27.2-54.4 65.6-54.4 24 0 46.4 8 60.8 20.8 3.2 1.6 4.8 4.8 4.8 8 0 6.4-4.8 12.8-11.2 12.8-1.6 0-4.8-1.6-6.4-3.2-14.4-11.2-32-16-49.6-16-24 0-40 11.2-40 30.4 0 43.2 113.6 17.6 113.6 89.6 0 27.2-19.2 56-70.4 56z"
|
||||
fill="#FFFFFF" p-id="2656"></path>
|
||||
<path d="M960 326.4v16H755.2s-102.4-20.8-99.2-108.8c0 0 3.2 92.8 96 92.8h208z" fill="#349C42" p-id="2657"></path>
|
||||
<path d="M656 0v233.6c0 25.6 19.2 92.8 99.2 92.8H960L656 0z" fill="#FFFFFF" p-id="2658"></path>
|
||||
</svg>
|
After Width: | Height: | Size: 1.5 KiB |
@@ -36,7 +36,7 @@ export default function Editor({
|
||||
hasVariablePlugin?: boolean;
|
||||
hasDropDownPlugin?: boolean;
|
||||
variables: EditorVariablePickerType[];
|
||||
onChange?: (editorState: EditorState) => void;
|
||||
onChange?: (editorState: EditorState, editor: LexicalEditor) => void;
|
||||
onBlur?: (editor: LexicalEditor) => void;
|
||||
value?: string;
|
||||
currentValue?: string;
|
||||
@@ -119,9 +119,9 @@ export default function Editor({
|
||||
<HistoryPlugin />
|
||||
<FocusPlugin focus={focus} setFocus={setFocus} />
|
||||
<OnChangePlugin
|
||||
onChange={(e) => {
|
||||
onChange={(editorState: EditorState, editor: LexicalEditor) => {
|
||||
startSts(() => {
|
||||
onChange?.(e);
|
||||
onChange?.(editorState, editor);
|
||||
});
|
||||
}}
|
||||
/>
|
||||
|
@@ -32,15 +32,13 @@ const HttpInput = ({
|
||||
|
||||
const [, startSts] = useTransition();
|
||||
|
||||
const onChangeInput = useCallback((editorState: EditorState) => {
|
||||
const text = editorState.read(() => $getRoot().getTextContent());
|
||||
const formatValue = text.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
|
||||
setCurrentValue(formatValue);
|
||||
onChange?.(formatValue);
|
||||
const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
|
||||
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
|
||||
onChange?.(text);
|
||||
}, []);
|
||||
const onBlurInput = useCallback((editor: LexicalEditor) => {
|
||||
startSts(() => {
|
||||
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
|
||||
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
|
||||
onBlur?.(text);
|
||||
});
|
||||
}, []);
|
||||
|
@@ -7,10 +7,13 @@ import {
|
||||
useDisclosure,
|
||||
MenuButton,
|
||||
Box,
|
||||
css
|
||||
css,
|
||||
Flex
|
||||
} from '@chakra-ui/react';
|
||||
import type { ButtonProps, MenuItemProps } from '@chakra-ui/react';
|
||||
import { ChevronDownIcon } from '@chakra-ui/icons';
|
||||
import { useLoading } from '../../../hooks/useLoading';
|
||||
import MyIcon from '../Icon';
|
||||
|
||||
export type SelectProps = ButtonProps & {
|
||||
value?: string;
|
||||
@@ -20,14 +23,16 @@ export type SelectProps = ButtonProps & {
|
||||
label: string | React.ReactNode;
|
||||
value: string;
|
||||
}[];
|
||||
isLoading?: boolean;
|
||||
onchange?: (val: any) => void;
|
||||
};
|
||||
|
||||
const MySelect = (
|
||||
{ placeholder, value, width = '100%', list, onchange, ...props }: SelectProps,
|
||||
{ placeholder, value, width = '100%', list, onchange, isLoading = false, ...props }: SelectProps,
|
||||
selectRef: any
|
||||
) => {
|
||||
const ref = useRef<HTMLButtonElement>(null);
|
||||
const { Loading } = useLoading();
|
||||
const menuItemStyles: MenuItemProps = {
|
||||
borderRadius: 'sm',
|
||||
py: 2,
|
||||
@@ -78,7 +83,10 @@ const MySelect = (
|
||||
: {})}
|
||||
{...props}
|
||||
>
|
||||
<Flex alignItems={'center'}>
|
||||
{isLoading && <MyIcon mr={2} name={'common/loading'} w={'16px'} />}
|
||||
{selectItem?.alias || selectItem?.label || placeholder}
|
||||
</Flex>
|
||||
</MenuButton>
|
||||
|
||||
<MenuList
|
||||
|
@@ -1,10 +1,10 @@
|
||||
import { Button, ModalBody, ModalFooter, useDisclosure } from '@chakra-ui/react';
|
||||
import React, { useEffect, useState } from 'react';
|
||||
import React from 'react';
|
||||
import { editorStateToText } from './utils';
|
||||
import Editor from './Editor';
|
||||
import MyModal from '../../MyModal';
|
||||
import { useTranslation } from 'next-i18next';
|
||||
import { $getRoot, EditorState, type LexicalEditor } from 'lexical';
|
||||
import { EditorState, type LexicalEditor } from 'lexical';
|
||||
import { EditorVariablePickerType } from './type.d';
|
||||
import { useCallback, useTransition } from 'react';
|
||||
|
||||
@@ -34,16 +34,12 @@ const PromptEditor = ({
|
||||
const { t } = useTranslation();
|
||||
|
||||
const onChangeInput = useCallback((editorState: EditorState, editor: LexicalEditor) => {
|
||||
const stringifiedEditorState = JSON.stringify(editorState.toJSON());
|
||||
const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
|
||||
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
|
||||
|
||||
const formatValue = editorStateTextString.replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
|
||||
onChange?.(formatValue);
|
||||
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
|
||||
onChange?.(text);
|
||||
}, []);
|
||||
const onBlurInput = useCallback((editor: LexicalEditor) => {
|
||||
startSts(() => {
|
||||
const text = editorStateToText(editor).replaceAll('\n\n', '\n').replaceAll('}}{{', '}} {{');
|
||||
const text = editorStateToText(editor).replaceAll('}}{{', '}} {{');
|
||||
onBlur?.(text);
|
||||
});
|
||||
}, []);
|
||||
|
@@ -1,6 +1,5 @@
|
||||
import { useEffect } from 'react';
|
||||
import { BLUR_COMMAND, COMMAND_PRIORITY_EDITOR, LexicalEditor } from 'lexical';
|
||||
import { mergeRegister } from '@lexical/utils';
|
||||
import { useLexicalComposerContext } from '@lexical/react/LexicalComposerContext';
|
||||
|
||||
export default function OnBlurPlugin({ onBlur }: { onBlur?: (editor: LexicalEditor) => void }) {
|
||||
|
@@ -209,9 +209,8 @@ export function editorStateToText(editor: LexicalEditor) {
|
||||
const stringifiedEditorState = JSON.stringify(editor.getEditorState().toJSON());
|
||||
const parsedEditorState = editor.parseEditorState(stringifiedEditorState);
|
||||
const editorStateTextString = parsedEditorState.read(() => $getRoot().getTextContent());
|
||||
const compressedText = editorStateTextString.replace(/\n+/g, '\n\n');
|
||||
|
||||
return compressedText;
|
||||
return editorStateTextString;
|
||||
}
|
||||
|
||||
const varRegex = /\{\{([a-zA-Z_][a-zA-Z0-9_]*)\}\}/g;
|
||||
|
2857
pnpm-lock.yaml
generated
2857
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
@@ -1,4 +1,7 @@
|
||||
{
|
||||
"feConfigs": {
|
||||
"lafEnv": "https://laf.dev"
|
||||
},
|
||||
"systemEnv": {
|
||||
"openapiPrefix": "fastgpt",
|
||||
"vectorMaxProcess": 15,
|
||||
|
BIN
projects/app/public/imgs/module/laf.png
Normal file
BIN
projects/app/public/imgs/module/laf.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 2.7 KiB |
@@ -23,7 +23,7 @@ function embedChatbot() {
|
||||
const ChatBtn = document.createElement('div');
|
||||
ChatBtn.id = chatBtnId;
|
||||
ChatBtn.style.cssText =
|
||||
'position: fixed; bottom: 1rem; right: 1rem; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
|
||||
'position: fixed; bottom: 30px; right: 60px; width: 40px; height: 40px; cursor: pointer; z-index: 2147483647; transition: 0;';
|
||||
|
||||
// btn icon
|
||||
const ChatBtnDiv = document.createElement('img');
|
||||
@@ -39,7 +39,7 @@ function embedChatbot() {
|
||||
iframe.id = chatWindowId;
|
||||
iframe.src = botSrc;
|
||||
iframe.style.cssText =
|
||||
'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 4rem; right: 1rem; width: 24rem; height: 40rem; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
|
||||
'border: none; position: fixed; flex-direction: column; justify-content: space-between; box-shadow: rgba(150, 150, 150, 0.2) 0px 10px 30px 0px, rgba(150, 150, 150, 0.2) 0px 0px 0px 1px; bottom: 80px; right: 60px; width: 375px; height: 667px; max-width: 90vw; max-height: 85vh; border-radius: 0.75rem; display: flex; z-index: 2147483647; overflow: hidden; left: unset; background-color: #F3F4F6;';
|
||||
iframe.style.visibility = defaultOpen ? 'unset' : 'hidden';
|
||||
|
||||
document.body.appendChild(iframe);
|
||||
|
@@ -148,6 +148,7 @@
|
||||
"Status": "Status",
|
||||
"Submit failed": "Submit failed",
|
||||
"Submit success": "Update Success",
|
||||
"Sync success": "",
|
||||
"System version": "System version",
|
||||
"Team": "Team",
|
||||
"Team Tags Set": "Team Tags",
|
||||
@@ -819,6 +820,7 @@
|
||||
"Http request props": "Request props",
|
||||
"Http request settings": "Request settings",
|
||||
"Input Type": "Input Type",
|
||||
"Laf sync params": "Sync params",
|
||||
"Plugin output must connect": "Custom outputs must all be connected",
|
||||
"QueryExtension": {
|
||||
"placeholder": "Questions about python introduction and usage, etc. The current conversation is related to the game GTA5.",
|
||||
@@ -912,6 +914,9 @@
|
||||
"target": "Target Data",
|
||||
"textarea": "Textarea"
|
||||
},
|
||||
"laf": {
|
||||
"Select laf function": ""
|
||||
},
|
||||
"output": {
|
||||
"Add Output": "Add Output",
|
||||
"Output Number": "Output: {{length}}",
|
||||
@@ -1226,19 +1231,26 @@
|
||||
"Set Public": "Set to public"
|
||||
},
|
||||
"plugin": {
|
||||
"App": "Choose App",
|
||||
"Auth Header Prefix": "Auth header prefix",
|
||||
"Auth Method": "Auth method",
|
||||
"Auth Type": "Auth Type",
|
||||
"Confirm Delete": "Confirm to delete the plugin?",
|
||||
"Create Your Plugin": "Create Plugin",
|
||||
"Currentapp": "CurrentApp",
|
||||
"Custom Plugin": "Custom plugin",
|
||||
"Description": "Description",
|
||||
"Edit Http Plugin": "Edit HTTP plugin",
|
||||
"Enter Env": "Enter laf environment",
|
||||
"Enter PAT": "Please enter personal access token (PAT)",
|
||||
"Func": "Choose Function",
|
||||
"Get Plugin Module Detail Failed": "Get plugin detail failed",
|
||||
"HTTP Plugin": "HTTP plugin",
|
||||
"Import Plugin": "Import HTTP plugin",
|
||||
"Import from URL": "Import from URL. https://xxxx",
|
||||
"Intro": "Plugin Intro",
|
||||
"Invalid Appid": "Invalid appid",
|
||||
"Invalid Env": "Invalid Env",
|
||||
"Invalid Schema": "Invalid Schema",
|
||||
"Invalid URL": "Invalid URL",
|
||||
"Key": "Key",
|
||||
@@ -1248,16 +1260,20 @@
|
||||
"No Intro": "This plugin is not introduced",
|
||||
"None": "None",
|
||||
"Path": "Path",
|
||||
"Please bind laf accout first": "Please bind laf accout first",
|
||||
"Plugin List": "Plugin list",
|
||||
"Plugin Module": "Plugin",
|
||||
"Privacy Agreement": "privacy agreement",
|
||||
"Search plugin": "Search plugins",
|
||||
"Set Name": "Plugin Name",
|
||||
"Synchronous app": "Sync App",
|
||||
"Synchronous version": "Sync Version",
|
||||
"To Edit Plugin": "To Edit",
|
||||
"Update Your Plugin": "Update Plugin",
|
||||
"Value": "Value",
|
||||
"path": ""
|
||||
"go to laf": "go to laf",
|
||||
"path": "",
|
||||
"update params": "update params"
|
||||
},
|
||||
"support": {
|
||||
"account": {
|
||||
@@ -1303,6 +1319,9 @@
|
||||
"user": {
|
||||
"AI point standard": "AI points price",
|
||||
"Avatar": "Avatar",
|
||||
"Go laf env": "Click to go to laf to get PAT certificate.",
|
||||
"Laf account course": "Check out the laf account binding tutorial.",
|
||||
"Laf account intro": "After binding your laf account, you will be able to use the laf module in your workflow to write code online.",
|
||||
"Need to login": "Please log in first",
|
||||
"Price": "Price",
|
||||
"User self info": "My info",
|
||||
@@ -1495,10 +1514,13 @@
|
||||
"Bill Detail": "Bill Detail",
|
||||
"Change": "Change",
|
||||
"Copy invite url": "Copy invitation link",
|
||||
"Current laf Env": "Current laf Env",
|
||||
"Edit name": "Click to modify nickname",
|
||||
"Invite Url": "Invite Url",
|
||||
"Invite url tip": "Friends who register through this link will be permanently bound to you, and you will get a certain balance reward when they recharge. In addition, when friends register with their mobile phone number, you will get 5 yuan reward immediately.",
|
||||
"Laf Account Setting": "laf account setting",
|
||||
"Language": "Language",
|
||||
"Learn More": "Learn More",
|
||||
"Member Name": "Name",
|
||||
"Notice": "Notice",
|
||||
"Old password is error": "Old password is error",
|
||||
@@ -1513,6 +1535,7 @@
|
||||
"Promotion rate tip": "You will be rewarded with a percentage of the balance when your friends top up",
|
||||
"Recharge Record": "Recharge",
|
||||
"Replace": "Replace",
|
||||
"Set Laf Account Failed": "set laf accout failed",
|
||||
"Set OpenAI Account Failed": "Set OpenAI account failed",
|
||||
"Sign Out": "Sign Out",
|
||||
"Source": "Source",
|
||||
|
@@ -148,6 +148,7 @@
|
||||
"Status": "状态",
|
||||
"Submit failed": "提交失败",
|
||||
"Submit success": "提交成功",
|
||||
"Sync success": "同步成功",
|
||||
"System version": "系统版本",
|
||||
"Team": "团队",
|
||||
"Team Tags Set": "标签",
|
||||
@@ -821,6 +822,7 @@
|
||||
"Http request props": "请求参数",
|
||||
"Http request settings": "请求配置",
|
||||
"Input Type": "输入类型",
|
||||
"Laf sync params": "同步参数",
|
||||
"Plugin output must connect": "自定义输出必须全部连接",
|
||||
"QueryExtension": {
|
||||
"placeholder": "例如:\n关于 python 的介绍和使用等问题。\n当前对话与游戏《GTA5》有关。",
|
||||
@@ -914,6 +916,9 @@
|
||||
"target": "外部数据",
|
||||
"textarea": "段落输入"
|
||||
},
|
||||
"laf": {
|
||||
"Select laf function": "选择laf函数"
|
||||
},
|
||||
"output": {
|
||||
"Add Output": "添加出参",
|
||||
"Output Number": "出参: {{length}}",
|
||||
@@ -1228,19 +1233,26 @@
|
||||
"Set Public": "设为团队可用"
|
||||
},
|
||||
"plugin": {
|
||||
"App": "选择应用",
|
||||
"Auth Header Prefix": "鉴权头部前缀",
|
||||
"Auth Method": "鉴权方法",
|
||||
"Auth Type": "鉴权类型",
|
||||
"Confirm Delete": "确认删除该插件?",
|
||||
"Create Your Plugin": "创建你的插件",
|
||||
"Currentapp": "当前应用",
|
||||
"Custom Plugin": "自定义插件",
|
||||
"Description": "描述",
|
||||
"Edit Http Plugin": "编辑 HTTP 插件",
|
||||
"Enter Env": "输入 laf 环境",
|
||||
"Enter PAT": "请输入访问凭证(PAT)",
|
||||
"Func": "选择函数",
|
||||
"Get Plugin Module Detail Failed": "获取插件信息异常",
|
||||
"HTTP Plugin": "HTTP 插件",
|
||||
"Import Plugin": "导入 HTTP 插件",
|
||||
"Import from URL": "从URL导入。https://xxxx",
|
||||
"Intro": "插件介绍",
|
||||
"Invalid Appid": "appid 无效",
|
||||
"Invalid Env": "laf 环境错误",
|
||||
"Invalid Schema": "Schema 无效",
|
||||
"Invalid URL": "URL 无效",
|
||||
"Key": "键",
|
||||
@@ -1250,16 +1262,20 @@
|
||||
"No Intro": "这个插件没有介绍~",
|
||||
"None": "无",
|
||||
"Path": "路径",
|
||||
"Please bind laf accout first": "请先绑定 laf 账号",
|
||||
"Plugin List": "插件列表",
|
||||
"Plugin Module": "插件模块",
|
||||
"Privacy Agreement": "隐私协议",
|
||||
"Search plugin": "搜索插件",
|
||||
"Set Name": "给插件取个名字",
|
||||
"Synchronous app": "同步应用",
|
||||
"Synchronous version": "同步版本",
|
||||
"To Edit Plugin": "去编辑",
|
||||
"Update Your Plugin": "更新插件",
|
||||
"Value": "值",
|
||||
"path": ""
|
||||
"go to laf": "去编写",
|
||||
"path": "",
|
||||
"update params": "更新参数"
|
||||
},
|
||||
"support": {
|
||||
"account": {
|
||||
@@ -1305,6 +1321,9 @@
|
||||
"user": {
|
||||
"AI point standard": "AI积分标准",
|
||||
"Avatar": "头像",
|
||||
"Go laf env": "点击前往 laf 获取 PAT 凭证。",
|
||||
"Laf account course": "查看绑定 laf 账号教程。",
|
||||
"Laf account intro": "绑定你的laf账号后,你将可以在工作流中使用 laf 模块,实现在线编写代码。",
|
||||
"Need to login": "请先登录",
|
||||
"Price": "计费标准",
|
||||
"User self info": "个人信息",
|
||||
@@ -1497,10 +1516,13 @@
|
||||
"Bill Detail": "账单详情",
|
||||
"Change": "变更",
|
||||
"Copy invite url": "复制邀请链接",
|
||||
"Current laf Env": "当前 laf 环境",
|
||||
"Edit name": "点击修改昵称",
|
||||
"Invite Url": "邀请链接",
|
||||
"Invite url tip": "通过该链接注册的好友将永久与你绑定,其充值时你会获得一定余额奖励。\n此外,好友使用手机号注册时,你将立即获得 5 元奖励。\n奖励会发送到您的默认团队中。",
|
||||
"Laf Account Setting": "laf 账号配置",
|
||||
"Language": "语言",
|
||||
"Learn More": "查看文档",
|
||||
"Member Name": "昵称",
|
||||
"Notice": "通知",
|
||||
"Old password is error": "旧密码错误",
|
||||
@@ -1515,6 +1537,7 @@
|
||||
"Promotion rate tip": "好友充值时你将获得一定比例的余额奖励",
|
||||
"Recharge Record": "支付记录",
|
||||
"Replace": "更换",
|
||||
"Set Laf Account Failed": "设置 laf 账号异常",
|
||||
"Set OpenAI Account Failed": "设置 OpenAI 账号异常",
|
||||
"Sign Out": "登出",
|
||||
"Source": "来源",
|
||||
|
@@ -214,9 +214,6 @@ export const FlowProvider = ({
|
||||
if (source?.flowType === FlowNodeTypeEnum.classifyQuestion && !type) {
|
||||
return ModuleIOValueTypeEnum.boolean;
|
||||
}
|
||||
if (source?.flowType === FlowNodeTypeEnum.tools) {
|
||||
return ModuleIOValueTypeEnum.tools;
|
||||
}
|
||||
if (source?.flowType === FlowNodeTypeEnum.pluginInput) {
|
||||
return source?.inputs.find((input) => input.key === connect.sourceHandle)?.valueType;
|
||||
}
|
||||
|
@@ -52,6 +52,7 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
|
||||
const router = useRouter();
|
||||
const [currentParent, setCurrentParent] = useState<RenderListProps['currentParent']>();
|
||||
const [searchKey, setSearchKey] = useState('');
|
||||
const { feConfigs } = useSystemStore();
|
||||
|
||||
const {
|
||||
basicNodeTemplates,
|
||||
@@ -64,7 +65,12 @@ const ModuleTemplateList = ({ isOpen, onClose }: ModuleTemplateListProps) => {
|
||||
|
||||
const templates = useMemo(() => {
|
||||
const map = {
|
||||
[TemplateTypeEnum.basic]: basicNodeTemplates,
|
||||
[TemplateTypeEnum.basic]: basicNodeTemplates.filter((item) => {
|
||||
if (item.flowType === FlowNodeTypeEnum.lafModule && !feConfigs.lafEnv) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}),
|
||||
[TemplateTypeEnum.systemPlugin]: systemNodeTemplates,
|
||||
[TemplateTypeEnum.teamPlugin]: teamPluginNodeTemplates.filter((item) =>
|
||||
searchKey ? item.pluginType !== PluginTypeEnum.folder : true
|
||||
|
@@ -88,7 +88,7 @@ enum TabEnum {
|
||||
headers = 'headers',
|
||||
body = 'body'
|
||||
}
|
||||
type PropsArrType = {
|
||||
export type PropsArrType = {
|
||||
key: string;
|
||||
type: string;
|
||||
value: string;
|
||||
@@ -245,7 +245,7 @@ const RenderHttpMethodAndUrl = React.memo(function RenderHttpMethodAndUrl({
|
||||
);
|
||||
});
|
||||
|
||||
function RenderHttpProps({
|
||||
export function RenderHttpProps({
|
||||
moduleId,
|
||||
inputs
|
||||
}: {
|
||||
|
@@ -0,0 +1,260 @@
|
||||
import React, { useCallback, useMemo } from 'react';
|
||||
import { NodeProps } from 'reactflow';
|
||||
import NodeCard from '../render/NodeCard';
|
||||
import { FlowModuleItemType } from '@fastgpt/global/core/module/type.d';
|
||||
import Container from '../modules/Container';
|
||||
import { Box, Button, Center, Flex, useDisclosure } from '@chakra-ui/react';
|
||||
import { ModuleIOValueTypeEnum, ModuleInputKeyEnum } from '@fastgpt/global/core/module/constants';
|
||||
import { onChangeNode, useFlowProviderStore } from '../../FlowProvider';
|
||||
import { useTranslation } from 'next-i18next';
|
||||
import { getLafAppDetail } from '@/web/support/laf/api';
|
||||
import MySelect from '@fastgpt/web/components/common/MySelect';
|
||||
import { getApiSchemaByUrl } from '@/web/core/plugin/api';
|
||||
import { str2OpenApiSchema } from '@fastgpt/global/core/plugin/httpPlugin/utils';
|
||||
import { useUserStore } from '@/web/support/user/useUserStore';
|
||||
import { useSystemStore } from '@/web/common/system/useSystemStore';
|
||||
import { ChevronRightIcon } from '@chakra-ui/icons';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import dynamic from 'next/dynamic';
|
||||
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
|
||||
import { useToast } from '@fastgpt/web/hooks/useToast';
|
||||
import Divider from '../modules/Divider';
|
||||
import RenderToolInput from '../render/RenderToolInput';
|
||||
import RenderInput from '../render/RenderInput';
|
||||
import RenderOutput from '../render/RenderOutput';
|
||||
import { getErrText } from '@fastgpt/global/common/error/utils';
|
||||
|
||||
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
|
||||
|
||||
const NodeLaf = (props: NodeProps<FlowModuleItemType>) => {
|
||||
const { t } = useTranslation();
|
||||
const { toast } = useToast();
|
||||
const { feConfigs } = useSystemStore();
|
||||
const { data, selected } = props;
|
||||
const { moduleId, inputs } = data;
|
||||
|
||||
const requestUrl = inputs.find((item) => item.key === ModuleInputKeyEnum.httpReqUrl);
|
||||
|
||||
const { userInfo } = useUserStore();
|
||||
|
||||
const token = userInfo?.team.lafAccount?.token;
|
||||
const appid = userInfo?.team.lafAccount?.appid;
|
||||
|
||||
// not config laf
|
||||
if (!token || !appid) {
|
||||
return (
|
||||
<NodeCard minW={'350px'} selected={selected} {...data}>
|
||||
<ConfigLaf />
|
||||
</NodeCard>
|
||||
);
|
||||
}
|
||||
|
||||
const { data: lafData, isLoading: isLoadingFunctions } = useQuery(
|
||||
['getLafFunctionList'],
|
||||
async () => {
|
||||
// load laf app detail
|
||||
const appDetail = await getLafAppDetail(appid);
|
||||
|
||||
// load laf app functions
|
||||
const schemaUrl = `https://${appDetail?.domain.domain}/_/api-docs?token=${appDetail?.openapi_token}`;
|
||||
|
||||
const schema = await getApiSchemaByUrl(schemaUrl);
|
||||
const openApiSchema = await str2OpenApiSchema(JSON.stringify(schema));
|
||||
const filterPostSchema = openApiSchema.pathData.filter((item) => item.method === 'post');
|
||||
|
||||
return {
|
||||
lafApp: appDetail,
|
||||
lafFunctions: filterPostSchema.map((item) => ({
|
||||
...item,
|
||||
requestUrl: `https://${appDetail?.domain.domain}${item.path}`
|
||||
}))
|
||||
};
|
||||
},
|
||||
{
|
||||
onError(err) {
|
||||
toast({
|
||||
status: 'error',
|
||||
title: getErrText(err, '获取Laf函数列表失败')
|
||||
});
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
const lafFunctionSelectList = useMemo(
|
||||
() =>
|
||||
lafData?.lafFunctions.map((item) => ({
|
||||
label: item.description ? `${item.name} (${item.description})` : item.name,
|
||||
value: item.requestUrl
|
||||
})) || [],
|
||||
[lafData?.lafFunctions]
|
||||
);
|
||||
|
||||
const selectedFunction = useMemo(
|
||||
() => lafFunctionSelectList.find((item) => item.value === requestUrl?.value)?.value,
|
||||
[lafFunctionSelectList, requestUrl?.value]
|
||||
);
|
||||
|
||||
const onSyncParams = useCallback(() => {
|
||||
const lafFunction = lafData?.lafFunctions.find((item) => item.requestUrl === selectedFunction);
|
||||
|
||||
if (!lafFunction) return;
|
||||
|
||||
const bodyParams =
|
||||
lafFunction?.request?.content?.['application/json']?.schema?.properties || {};
|
||||
|
||||
const requiredParams =
|
||||
lafFunction?.request?.content?.['application/json']?.schema?.required || [];
|
||||
|
||||
const allParams = [
|
||||
...Object.keys(bodyParams).map((key) => ({
|
||||
name: key,
|
||||
desc: bodyParams[key].description,
|
||||
required: requiredParams?.includes(key) || false,
|
||||
value: `{{${key}}}`,
|
||||
type: 'string'
|
||||
}))
|
||||
].filter((item) => !inputs.find((input) => input.key === item.name));
|
||||
|
||||
// add params
|
||||
allParams.forEach((param) => {
|
||||
onChangeNode({
|
||||
moduleId,
|
||||
type: 'addInput',
|
||||
key: param.name,
|
||||
value: {
|
||||
key: param.name,
|
||||
valueType: ModuleIOValueTypeEnum.string,
|
||||
label: param.name,
|
||||
type: FlowNodeInputTypeEnum.target,
|
||||
required: param.required,
|
||||
description: param.desc || '',
|
||||
toolDescription: param.desc || '未设置参数描述',
|
||||
edit: true,
|
||||
editField: {
|
||||
key: true,
|
||||
name: true,
|
||||
description: true,
|
||||
required: true,
|
||||
dataType: true,
|
||||
inputType: true,
|
||||
isToolInput: true
|
||||
},
|
||||
connected: false
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
toast({
|
||||
status: 'success',
|
||||
title: t('common.Sync success')
|
||||
});
|
||||
}, [inputs, lafData?.lafFunctions, moduleId, selectedFunction, t, toast]);
|
||||
|
||||
return (
|
||||
<NodeCard minW={'350px'} selected={selected} {...data}>
|
||||
<Container>
|
||||
{/* select function */}
|
||||
<MySelect
|
||||
isLoading={isLoadingFunctions}
|
||||
list={lafFunctionSelectList}
|
||||
placeholder={t('core.module.laf.Select laf function')}
|
||||
onchange={(e) => {
|
||||
onChangeNode({
|
||||
moduleId,
|
||||
type: 'updateInput',
|
||||
key: ModuleInputKeyEnum.httpReqUrl,
|
||||
value: {
|
||||
...requestUrl,
|
||||
value: e
|
||||
}
|
||||
});
|
||||
}}
|
||||
value={selectedFunction}
|
||||
/>
|
||||
{/* auto set params and go to edit */}
|
||||
{!!selectedFunction && (
|
||||
<Flex justifyContent={'flex-end'} mt={2} gap={2}>
|
||||
{/* <Button variant={'whiteBase'} size={'sm'} onClick={onSyncParams}>
|
||||
{t('core.module.Laf sync params')}
|
||||
</Button> */}
|
||||
<Button
|
||||
variant={'grayBase'}
|
||||
size={'sm'}
|
||||
onClick={() => {
|
||||
const lafFunction = lafData?.lafFunctions.find(
|
||||
(item) => item.requestUrl === selectedFunction
|
||||
);
|
||||
|
||||
if (!lafFunction) return;
|
||||
const url = `${feConfigs.lafEnv}/app/${lafData?.lafApp?.appid}/function${lafFunction?.path}`;
|
||||
window.open(url, '_blank');
|
||||
}}
|
||||
>
|
||||
{t('plugin.go to laf')}
|
||||
</Button>
|
||||
</Flex>
|
||||
)}
|
||||
</Container>
|
||||
{!!selectedFunction && <RenderIO {...props} />}
|
||||
</NodeCard>
|
||||
);
|
||||
};
|
||||
export default React.memo(NodeLaf);
|
||||
|
||||
const ConfigLaf = () => {
|
||||
const { t } = useTranslation();
|
||||
const { userInfo } = useUserStore();
|
||||
const { feConfigs } = useSystemStore();
|
||||
const {
|
||||
isOpen: isOpenLafConfig,
|
||||
onOpen: onOpenLafConfig,
|
||||
onClose: onCloseLafConfig
|
||||
} = useDisclosure();
|
||||
|
||||
return !!feConfigs?.lafEnv ? (
|
||||
<Center minH={150}>
|
||||
<Button onClick={onOpenLafConfig} variant={'whitePrimary'}>
|
||||
{t('plugin.Please bind laf accout first')} <ChevronRightIcon />
|
||||
</Button>
|
||||
|
||||
{isOpenLafConfig && feConfigs?.lafEnv && (
|
||||
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLafConfig} />
|
||||
)}
|
||||
</Center>
|
||||
) : (
|
||||
<Box>系统未配置Laf环境</Box>
|
||||
);
|
||||
};
|
||||
|
||||
const RenderIO = ({ data, selected }: NodeProps<FlowModuleItemType>) => {
|
||||
const { t } = useTranslation();
|
||||
const { moduleId, inputs, outputs } = data;
|
||||
const { splitToolInputs, hasToolNode } = useFlowProviderStore();
|
||||
const { commonInputs, toolInputs } = splitToolInputs(inputs, moduleId);
|
||||
|
||||
return (
|
||||
<>
|
||||
{hasToolNode && (
|
||||
<>
|
||||
<Divider text={t('core.module.tool.Tool input')} />
|
||||
<Container>
|
||||
<RenderToolInput moduleId={moduleId} inputs={toolInputs} canEdit />
|
||||
</Container>
|
||||
</>
|
||||
)}
|
||||
<>
|
||||
<Divider text={t('common.Input')} />
|
||||
<Container>
|
||||
<Box mb={3}>自定义Body参数</Box>
|
||||
<RenderInput moduleId={moduleId} flowInputList={commonInputs} />
|
||||
</Container>
|
||||
</>
|
||||
<>
|
||||
<Divider text={t('common.Output')} />
|
||||
<Container>
|
||||
<RenderOutput moduleId={moduleId} flowOutputList={outputs} />
|
||||
</Container>
|
||||
</>
|
||||
</>
|
||||
);
|
||||
};
|
@@ -164,7 +164,7 @@ const NodeCard = (props: Props) => {
|
||||
top={'-20px'}
|
||||
right={0}
|
||||
transform={'translateX(90%)'}
|
||||
pl={'17px'}
|
||||
pl={'20px'}
|
||||
pr={'10px'}
|
||||
pb={'20px'}
|
||||
pt={'20px'}
|
||||
|
@@ -1,11 +1,25 @@
|
||||
import { FlowNodeInputItemType } from '@fastgpt/global/core/module/node/type';
|
||||
import { FlowNodeInputTypeEnum } from '@fastgpt/global/core/module/node/constant';
|
||||
import { ModuleIOValueTypeEnum } from '@fastgpt/global/core/module/constants';
|
||||
|
||||
export const defaultEditFormData: FlowNodeInputItemType = {
|
||||
valueType: 'string',
|
||||
type: FlowNodeInputTypeEnum.hidden,
|
||||
type: FlowNodeInputTypeEnum.target,
|
||||
key: '',
|
||||
label: '',
|
||||
toolDescription: '',
|
||||
required: true
|
||||
required: true,
|
||||
edit: true,
|
||||
editField: {
|
||||
key: true,
|
||||
description: true,
|
||||
dataType: true
|
||||
},
|
||||
defaultEditField: {
|
||||
label: '',
|
||||
key: '',
|
||||
description: '',
|
||||
inputType: FlowNodeInputTypeEnum.target,
|
||||
valueType: ModuleIOValueTypeEnum.string
|
||||
}
|
||||
};
|
||||
|
@@ -44,7 +44,8 @@ const nodeTypes: Record<`${FlowNodeTypeEnum}`, any> = {
|
||||
[FlowNodeTypeEnum.tools]: dynamic(() => import('./components/nodes/NodeTools')),
|
||||
[FlowNodeTypeEnum.stopTool]: (data: NodeProps<FlowModuleItemType>) => (
|
||||
<NodeSimple {...data} minW={'100px'} maxW={'300px'} />
|
||||
)
|
||||
),
|
||||
[FlowNodeTypeEnum.lafModule]: dynamic(() => import('./components/nodes/NodeLaf'))
|
||||
};
|
||||
const edgeTypes = {
|
||||
[EDGE_TYPE]: ButtonEdge
|
||||
|
185
projects/app/src/components/support/laf/LafAccountModal.tsx
Normal file
185
projects/app/src/components/support/laf/LafAccountModal.tsx
Normal file
@@ -0,0 +1,185 @@
|
||||
import React, { useCallback } from 'react';
|
||||
import { ModalBody, Box, Flex, Input, ModalFooter, Button, Link } from '@chakra-ui/react';
|
||||
import MyModal from '@fastgpt/web/components/common/MyModal';
|
||||
import { useTranslation } from 'next-i18next';
|
||||
import { useForm } from 'react-hook-form';
|
||||
import { useRequest } from '@fastgpt/web/hooks/useRequest';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import MySelect from '@fastgpt/web/components/common/MySelect';
|
||||
import { useSystemStore } from '@/web/common/system/useSystemStore';
|
||||
import { useToast } from '@fastgpt/web/hooks/useToast';
|
||||
import { putUpdateTeam } from '@/web/support/user/team/api';
|
||||
import { useUserStore } from '@/web/support/user/useUserStore';
|
||||
import type { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
|
||||
import { postLafPat2Token, getLafApplications } from '@/web/support/laf/api';
|
||||
import { getErrText } from '@fastgpt/global/common/error/utils';
|
||||
|
||||
const LafAccountModal = ({
|
||||
defaultData = {
|
||||
token: '',
|
||||
appid: ''
|
||||
},
|
||||
onClose
|
||||
}: {
|
||||
defaultData?: LafAccountType;
|
||||
onClose: () => void;
|
||||
}) => {
|
||||
const { t } = useTranslation();
|
||||
const { register, handleSubmit, setValue, getValues, watch, reset } = useForm({
|
||||
defaultValues: {
|
||||
...defaultData,
|
||||
pat: ''
|
||||
}
|
||||
});
|
||||
|
||||
const lafToken = watch('token');
|
||||
const pat = watch('pat');
|
||||
const appid = watch('appid');
|
||||
|
||||
const { feConfigs } = useSystemStore();
|
||||
const { toast } = useToast();
|
||||
const { userInfo, initUserInfo } = useUserStore();
|
||||
|
||||
const onResetForm = useCallback(() => {
|
||||
reset({
|
||||
token: '',
|
||||
appid: '',
|
||||
pat: ''
|
||||
});
|
||||
}, [reset]);
|
||||
|
||||
const { mutate: authLafPat, isLoading: isPatLoading } = useRequest({
|
||||
mutationFn: async (pat) => {
|
||||
const token = await postLafPat2Token(pat);
|
||||
setValue('token', token);
|
||||
},
|
||||
errorToast: t('plugin.Invalid Env')
|
||||
});
|
||||
|
||||
const { data: appListData = [] } = useQuery(
|
||||
['appList', lafToken],
|
||||
() => {
|
||||
return getLafApplications(lafToken);
|
||||
},
|
||||
{
|
||||
enabled: !!lafToken,
|
||||
onSuccess: (data) => {
|
||||
if (!getValues('appid') && data.length > 0) {
|
||||
setValue('appid', data[0].appid);
|
||||
}
|
||||
},
|
||||
onError: (err) => {
|
||||
onResetForm();
|
||||
toast({
|
||||
title: getErrText(err, '获取应用列表失败'),
|
||||
status: 'error'
|
||||
});
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
const { mutate: onSubmit, isLoading: isUpdating } = useRequest({
|
||||
mutationFn: async (data: LafAccountType) => {
|
||||
if (!userInfo?.team.teamId) return;
|
||||
return putUpdateTeam({
|
||||
teamId: userInfo?.team.teamId,
|
||||
lafAccount: data
|
||||
});
|
||||
},
|
||||
onSuccess() {
|
||||
initUserInfo();
|
||||
onClose();
|
||||
},
|
||||
successToast: t('common.Update Success'),
|
||||
errorToast: t('common.Update Failed')
|
||||
});
|
||||
|
||||
return (
|
||||
<MyModal isOpen iconSrc="/imgs/module/laf.png" title={t('user.Laf Account Setting')}>
|
||||
<ModalBody>
|
||||
<Box fontSize={'sm'} color={'myGray.500'}>
|
||||
<Box>{t('support.user.Laf account intro')}</Box>
|
||||
<Box textDecoration={'underline'}>
|
||||
<Link href={`https://doc.laf.run/zh/`} isExternal>
|
||||
{t('support.user.Laf account course')}
|
||||
</Link>
|
||||
</Box>
|
||||
<Box>
|
||||
<Link textDecoration={'underline'} href={`${feConfigs.lafEnv}/`} isExternal>
|
||||
{t('support.user.Go laf env')}
|
||||
</Link>
|
||||
</Box>
|
||||
</Box>
|
||||
<Flex alignItems={'center'} mt={5}>
|
||||
<Box flex={'0 0 70px'}>PAT:</Box>
|
||||
{!lafToken ? (
|
||||
<>
|
||||
<Input
|
||||
flex={'1 0 0'}
|
||||
size={'sm'}
|
||||
{...register('pat')}
|
||||
placeholder={t('plugin.Enter PAT')}
|
||||
/>
|
||||
<Button
|
||||
ml={2}
|
||||
variant={'whitePrimary'}
|
||||
isDisabled={!pat}
|
||||
onClick={() => {
|
||||
authLafPat(pat);
|
||||
}}
|
||||
isLoading={isPatLoading}
|
||||
>
|
||||
验证
|
||||
</Button>
|
||||
</>
|
||||
) : (
|
||||
<Button
|
||||
variant={'whitePrimary'}
|
||||
onClick={() => {
|
||||
onResetForm();
|
||||
putUpdateTeam({
|
||||
teamId: userInfo?.team.teamId || '',
|
||||
lafAccount: { token: '', appid: '' }
|
||||
});
|
||||
}}
|
||||
>
|
||||
已验证,点击取消绑定
|
||||
</Button>
|
||||
)}
|
||||
</Flex>
|
||||
{!!lafToken && (
|
||||
<Flex alignItems={'center'} mt={5}>
|
||||
<Box flex={'0 0 70px'}>{t('plugin.Currentapp')}</Box>
|
||||
<MySelect
|
||||
minW={'200px'}
|
||||
list={
|
||||
appListData
|
||||
.filter((app) => app.state === 'Running')
|
||||
.map((app) => ({
|
||||
label: `${app.name}`,
|
||||
value: app.appid
|
||||
})) || []
|
||||
}
|
||||
placeholder={t('plugin.App')}
|
||||
value={watch('appid')}
|
||||
onchange={(e) => {
|
||||
setValue('appid', e);
|
||||
}}
|
||||
{...(register('appid'), { required: true })}
|
||||
/>
|
||||
</Flex>
|
||||
)}
|
||||
</ModalBody>
|
||||
<ModalFooter>
|
||||
<Button mr={3} variant={'whiteBase'} onClick={onClose}>
|
||||
{t('common.Close')}
|
||||
</Button>
|
||||
<Button isLoading={isUpdating} onClick={handleSubmit((data) => onSubmit(data))}>
|
||||
{t('common.Update')}
|
||||
</Button>
|
||||
</ModalFooter>
|
||||
</MyModal>
|
||||
);
|
||||
};
|
||||
|
||||
export default LafAccountModal;
|
@@ -8,7 +8,8 @@ import {
|
||||
Input,
|
||||
Link,
|
||||
Progress,
|
||||
Grid
|
||||
Grid,
|
||||
Image
|
||||
} from '@chakra-ui/react';
|
||||
import { useForm } from 'react-hook-form';
|
||||
import { UserUpdateParams } from '@/types/user';
|
||||
@@ -41,12 +42,14 @@ import {
|
||||
} from '@/web/support/wallet/sub/constants';
|
||||
|
||||
import StandardPlanContentList from '@/components/support/wallet/StandardPlanContentList';
|
||||
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
|
||||
|
||||
const StandDetailModal = dynamic(() => import('./standardDetailModal'));
|
||||
const TeamMenu = dynamic(() => import('@/components/support/user/team/TeamMenu'));
|
||||
const PayModal = dynamic(() => import('./PayModal'));
|
||||
const UpdatePswModal = dynamic(() => import('./UpdatePswModal'));
|
||||
const OpenAIAccountModal = dynamic(() => import('./OpenAIAccountModal'));
|
||||
const LafAccountModal = dynamic(() => import('@/components/support/laf/LafAccountModal'));
|
||||
const CommunityModal = dynamic(() => import('@/components/CommunityModal'));
|
||||
|
||||
const Account = () => {
|
||||
@@ -518,7 +521,7 @@ const Other = () => {
|
||||
const { reset } = useForm<UserUpdateParams>({
|
||||
defaultValues: userInfo as UserType
|
||||
});
|
||||
|
||||
const { isOpen: isOpenLaf, onClose: onCloseLaf, onOpen: onOpenLaf } = useDisclosure();
|
||||
const { isOpen: isOpenOpenai, onClose: onCloseOpenai, onOpen: onOpenOpenai } = useDisclosure();
|
||||
const { isOpen: isOpenConcat, onClose: onCloseConcat, onOpen: onOpenConcat } = useDisclosure();
|
||||
|
||||
@@ -537,7 +540,6 @@ const Other = () => {
|
||||
},
|
||||
[reset, toast, updateUserInfo]
|
||||
);
|
||||
|
||||
return (
|
||||
<Box>
|
||||
<Grid gridGap={4} mt={3}>
|
||||
@@ -582,6 +584,32 @@ const Other = () => {
|
||||
</Box>
|
||||
</Link>
|
||||
|
||||
{feConfigs?.lafEnv && userInfo?.team.role === TeamMemberRoleEnum.owner && (
|
||||
<Flex
|
||||
bg={'white'}
|
||||
py={4}
|
||||
px={6}
|
||||
border={theme.borders.sm}
|
||||
borderWidth={'1.5px'}
|
||||
borderRadius={'md'}
|
||||
alignItems={'center'}
|
||||
cursor={'pointer'}
|
||||
userSelect={'none'}
|
||||
onClick={onOpenLaf}
|
||||
>
|
||||
<Image src="/imgs/module/laf.png" w={'18px'} alt="laf" />
|
||||
<Box ml={2} flex={1}>
|
||||
laf 账号
|
||||
</Box>
|
||||
<Box
|
||||
w={'9px'}
|
||||
h={'9px'}
|
||||
borderRadius={'50%'}
|
||||
bg={userInfo?.team.lafAccount?.token ? '#67c13b' : 'myGray.500'}
|
||||
/>
|
||||
</Flex>
|
||||
)}
|
||||
|
||||
{feConfigs?.show_openai_account && (
|
||||
<Flex
|
||||
bg={'white'}
|
||||
@@ -620,6 +648,9 @@ const Other = () => {
|
||||
)}
|
||||
</Grid>
|
||||
|
||||
{isOpenLaf && userInfo && (
|
||||
<LafAccountModal defaultData={userInfo?.team.lafAccount} onClose={onCloseLaf} />
|
||||
)}
|
||||
{isOpenOpenai && userInfo && (
|
||||
<OpenAIAccountModal
|
||||
defaultData={userInfo?.openaiAccount}
|
||||
|
@@ -2,12 +2,14 @@ import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { authCert } from '@fastgpt/service/support/permission/auth/common';
|
||||
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
|
||||
import { addHours } from 'date-fns';
|
||||
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
|
||||
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
|
||||
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
|
||||
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
|
||||
import {
|
||||
checkInvalidDatasetFiles,
|
||||
checkInvalidDatasetData,
|
||||
checkInvalidVector
|
||||
} from '@/service/common/system/cronTask';
|
||||
|
||||
let deleteImageAmount = 0;
|
||||
async function checkInvalidImg(start: Date, end: Date, limit = 50) {
|
||||
@@ -60,11 +62,12 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
|
||||
(async () => {
|
||||
try {
|
||||
console.log('执行脏数据清理任务');
|
||||
const end = addHours(new Date(), -1);
|
||||
// 360天 ~ 2小时前
|
||||
const end = addHours(new Date(), -2);
|
||||
const start = addHours(new Date(), -360 * 24);
|
||||
await checkFiles(start, end);
|
||||
await checkInvalidDatasetFiles(start, end);
|
||||
await checkInvalidImg(start, end);
|
||||
await checkInvalidCollection(start, end);
|
||||
await checkInvalidDatasetData(start, end);
|
||||
await checkInvalidVector(start, end);
|
||||
console.log('执行脏数据清理任务完毕');
|
||||
} catch (error) {
|
||||
|
@@ -1,98 +0,0 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { authCert } from '@fastgpt/service/support/permission/auth/common';
|
||||
import { MongoUsage } from '@fastgpt/service/support/wallet/usage/schema';
|
||||
import { connectionMongo } from '@fastgpt/service/common/mongo';
|
||||
import { checkFiles } from '../timerTask/dataset/checkInValidDatasetFiles';
|
||||
import { addHours } from 'date-fns';
|
||||
import { checkInvalidCollection } from '../timerTask/dataset/checkInvalidMongoCollection';
|
||||
import { checkInvalidVector } from '../timerTask/dataset/checkInvalidVector';
|
||||
import { MongoImage } from '@fastgpt/service/common/file/image/schema';
|
||||
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
|
||||
|
||||
let deleteImageAmount = 0;
|
||||
export async function checkInvalidImg(start: Date, end: Date, limit = 50) {
|
||||
const images = await MongoImage.find(
|
||||
{
|
||||
createTime: {
|
||||
$gte: start,
|
||||
$lte: end
|
||||
},
|
||||
'metadata.relatedId': { $exists: true }
|
||||
},
|
||||
'_id teamId metadata'
|
||||
);
|
||||
console.log('total images', images.length);
|
||||
let index = 0;
|
||||
|
||||
for await (const image of images) {
|
||||
try {
|
||||
// 1. 检测是否有对应的集合
|
||||
const collection = await MongoDatasetCollection.findOne(
|
||||
{
|
||||
teamId: image.teamId,
|
||||
'metadata.relatedImgId': image.metadata?.relatedId
|
||||
},
|
||||
'_id'
|
||||
);
|
||||
|
||||
if (!collection) {
|
||||
await image.deleteOne();
|
||||
deleteImageAmount++;
|
||||
}
|
||||
|
||||
index++;
|
||||
|
||||
index % 100 === 0 && console.log(index);
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`检测完成,共删除 ${deleteImageAmount} 个无效图片`);
|
||||
}
|
||||
|
||||
/* pg 中的数据搬到 mongo dataset.datas 中,并做映射 */
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
|
||||
try {
|
||||
await connectToDatabase();
|
||||
await authCert({ req, authRoot: true });
|
||||
|
||||
// 检查 usage 是否有记录
|
||||
const totalUsages = await MongoUsage.countDocuments();
|
||||
if (totalUsages === 0) {
|
||||
// 重命名 bills 集合成 usages
|
||||
await connectionMongo.connection.db.renameCollection('bills', 'usages', {
|
||||
// 强制
|
||||
dropTarget: true
|
||||
});
|
||||
}
|
||||
|
||||
(async () => {
|
||||
try {
|
||||
console.log('执行脏数据清理任务');
|
||||
const end = addHours(new Date(), -1);
|
||||
const start = addHours(new Date(), -360 * 24);
|
||||
await checkFiles(start, end);
|
||||
await checkInvalidImg(start, end);
|
||||
await checkInvalidCollection(start, end);
|
||||
await checkInvalidVector(start, end);
|
||||
console.log('执行脏数据清理任务完毕');
|
||||
} catch (error) {
|
||||
console.log('执行脏数据清理任务出错了');
|
||||
}
|
||||
})();
|
||||
|
||||
jsonRes(res, {
|
||||
message: 'success'
|
||||
});
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
|
||||
jsonRes(res, {
|
||||
code: 500,
|
||||
error
|
||||
});
|
||||
}
|
||||
}
|
@@ -4,7 +4,7 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { authFile } from '@fastgpt/service/support/permission/auth/file';
|
||||
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
|
||||
|
||||
@@ -19,7 +19,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
|
||||
const { teamId } = await authFile({ req, authToken: true, fileId });
|
||||
|
||||
const { rawText } = await readFileContent({
|
||||
const { rawText } = await readFileContentFromMongo({
|
||||
teamId,
|
||||
bucketName: BucketNameEnum.dataset,
|
||||
fileId,
|
||||
|
@@ -90,6 +90,7 @@ export async function initSystemConfig() {
|
||||
// get config from database
|
||||
const config: FastGPTConfigFileType = {
|
||||
feConfigs: {
|
||||
...fileRes?.feConfigs,
|
||||
...defaultFeConfigs,
|
||||
...(dbConfig.feConfigs || {}),
|
||||
isPlus: !!FastGPTProUrl
|
||||
|
@@ -1,7 +1,7 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
|
||||
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
|
||||
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
|
||||
@@ -36,7 +36,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
});
|
||||
|
||||
// 1. read file
|
||||
const { rawText, filename } = await readFileContent({
|
||||
const { rawText, filename } = await readFileContentFromMongo({
|
||||
teamId,
|
||||
bucketName: BucketNameEnum.dataset,
|
||||
fileId
|
||||
|
@@ -3,7 +3,7 @@ import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import {
|
||||
delFileByFileIdList,
|
||||
readFileContent
|
||||
readFileContentFromMongo
|
||||
} from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
|
||||
import { FileIdCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
|
||||
@@ -47,7 +47,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
});
|
||||
|
||||
// 1. read file
|
||||
const { rawText, filename } = await readFileContent({
|
||||
const { rawText, filename } = await readFileContentFromMongo({
|
||||
teamId,
|
||||
bucketName: BucketNameEnum.dataset,
|
||||
fileId
|
||||
|
@@ -4,7 +4,7 @@ import { connectToDatabase } from '@/service/mongo';
|
||||
import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
|
||||
import { authFile } from '@fastgpt/service/support/permission/auth/file';
|
||||
import { PostPreviewFilesChunksProps } from '@/global/core/dataset/api';
|
||||
import { readFileContent } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { readFileContentFromMongo } from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
|
||||
import { ImportDataSourceEnum } from '@fastgpt/global/core/dataset/constants';
|
||||
import { parseCsvTable2Chunks } from '@fastgpt/service/core/dataset/training/utils';
|
||||
@@ -28,7 +28,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
|
||||
const fileId = String(file._id);
|
||||
|
||||
const { rawText } = await readFileContent({
|
||||
const { rawText } = await readFileContentFromMongo({
|
||||
teamId,
|
||||
bucketName: BucketNameEnum.dataset,
|
||||
fileId,
|
||||
@@ -53,7 +53,7 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
if (type === ImportDataSourceEnum.csvTable) {
|
||||
const { file, teamId } = await authFile({ req, authToken: true, fileId: sourceId });
|
||||
const fileId = String(file._id);
|
||||
const { rawText } = await readFileContent({
|
||||
const { rawText } = await readFileContentFromMongo({
|
||||
teamId,
|
||||
bucketName: BucketNameEnum.dataset,
|
||||
fileId,
|
||||
|
69
projects/app/src/pages/api/lafApi/[...path].ts
Normal file
69
projects/app/src/pages/api/lafApi/[...path].ts
Normal file
@@ -0,0 +1,69 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { request } from 'https';
|
||||
import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';
|
||||
import url from 'url';
|
||||
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
|
||||
try {
|
||||
await connectToDatabase();
|
||||
const { path = [], ...query } = req.query as any;
|
||||
|
||||
const queryStr = new URLSearchParams(query).toString();
|
||||
const requestPath = queryStr
|
||||
? `/${path?.join('/')}?${new URLSearchParams(query).toString()}`
|
||||
: `/${path?.join('/')}`;
|
||||
|
||||
if (!requestPath) {
|
||||
throw new Error('url is empty');
|
||||
}
|
||||
|
||||
const lafEnv = global.feConfigs?.lafEnv;
|
||||
|
||||
if (!lafEnv) {
|
||||
throw new Error('lafEnv is empty');
|
||||
}
|
||||
|
||||
const parsedUrl = url.parse(lafEnv);
|
||||
delete req.headers?.cookie;
|
||||
delete req.headers?.host;
|
||||
delete req.headers?.origin;
|
||||
|
||||
const requestResult = request({
|
||||
protocol: parsedUrl.protocol,
|
||||
hostname: parsedUrl.hostname,
|
||||
port: parsedUrl.port,
|
||||
path: requestPath,
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
timeout: 30000
|
||||
});
|
||||
|
||||
req.pipe(requestResult);
|
||||
|
||||
requestResult.on('response', (response) => {
|
||||
Object.keys(response.headers).forEach((key) => {
|
||||
// @ts-ignore
|
||||
res.setHeader(key, response.headers[key]);
|
||||
});
|
||||
response.statusCode && res.writeHead(response.statusCode);
|
||||
response.pipe(res);
|
||||
});
|
||||
requestResult.on('error', (e) => {
|
||||
res.send(e);
|
||||
res.end();
|
||||
});
|
||||
} catch (error) {
|
||||
jsonRes(res, {
|
||||
code: 500,
|
||||
error
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export const config = {
|
||||
api: {
|
||||
bodyParser: false
|
||||
}
|
||||
};
|
@@ -11,7 +11,7 @@ import { MongoTeamMember } from '@fastgpt/service/support/user/team/teamMemberSc
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
|
||||
try {
|
||||
await connectToDatabase();
|
||||
const { avatar, timezone, openaiAccount } = req.body as UserUpdateParams;
|
||||
const { avatar, timezone, openaiAccount, lafAccount } = req.body as UserUpdateParams;
|
||||
|
||||
const { tmbId } = await authCert({ req, authToken: true });
|
||||
const tmb = await MongoTeamMember.findById(tmbId);
|
||||
@@ -47,7 +47,8 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
|
||||
{
|
||||
...(avatar && { avatar }),
|
||||
...(timezone && { timezone }),
|
||||
openaiAccount: openaiAccount?.key ? openaiAccount : null
|
||||
openaiAccount: openaiAccount?.key ? openaiAccount : null,
|
||||
lafAccount: lafAccount?.token ? lafAccount : null
|
||||
}
|
||||
);
|
||||
|
||||
|
@@ -1,91 +0,0 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { authCert } from '@fastgpt/service/support/permission/auth/common';
|
||||
import {
|
||||
delFileByFileIdList,
|
||||
getGFSCollection
|
||||
} from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { addLog } from '@fastgpt/service/common/system/log';
|
||||
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
|
||||
import { addHours } from 'date-fns';
|
||||
|
||||
/*
|
||||
check dataset.files data. If there is no match in dataset.collections, delete it
|
||||
可能异常情况
|
||||
1. 上传了文件,未成功创建集合
|
||||
*/
|
||||
let deleteFileAmount = 0;
|
||||
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
|
||||
try {
|
||||
const { startHour = 24, endHour = 1 } = req.body as {
|
||||
startHour?: number;
|
||||
endHour?: number;
|
||||
limit?: number;
|
||||
};
|
||||
await authCert({ req, authRoot: true });
|
||||
await connectToDatabase();
|
||||
|
||||
// start: now - maxDay, end: now - 3 day
|
||||
const start = addHours(new Date(), -startHour);
|
||||
const end = addHours(new Date(), -endHour);
|
||||
deleteFileAmount = 0;
|
||||
console.log(start, end);
|
||||
|
||||
await checkFiles(start, end);
|
||||
|
||||
jsonRes(res, {
|
||||
data: deleteFileAmount,
|
||||
message: 'success'
|
||||
});
|
||||
} catch (error) {
|
||||
addLog.error(`check valid dataset files error`, error);
|
||||
|
||||
jsonRes(res, {
|
||||
code: 500,
|
||||
error
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkFiles(start: Date, end: Date) {
|
||||
const collection = getGFSCollection('dataset');
|
||||
const where = {
|
||||
uploadDate: { $gte: start, $lte: end }
|
||||
};
|
||||
|
||||
// 1. get all file _id
|
||||
const files = await collection
|
||||
.find(where, {
|
||||
projection: {
|
||||
metadata: 1,
|
||||
_id: 1
|
||||
}
|
||||
})
|
||||
.toArray();
|
||||
console.log('total files', files.length);
|
||||
|
||||
let index = 0;
|
||||
for await (const file of files) {
|
||||
try {
|
||||
// 2. find fileId in dataset.collections
|
||||
const hasCollection = await MongoDatasetCollection.countDocuments({
|
||||
teamId: file.metadata.teamId,
|
||||
fileId: file._id
|
||||
});
|
||||
|
||||
// 3. if not found, delete file
|
||||
if (hasCollection === 0) {
|
||||
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
|
||||
console.log('delete file', file._id);
|
||||
deleteFileAmount++;
|
||||
}
|
||||
index++;
|
||||
index % 100 === 0 && console.log(index);
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
}
|
||||
console.log(`检测完成,共删除 ${deleteFileAmount} 个无效文件`);
|
||||
}
|
@@ -1,96 +0,0 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { authCert } from '@fastgpt/service/support/permission/auth/common';
|
||||
import { addLog } from '@fastgpt/service/common/system/log';
|
||||
import { deleteDatasetDataVector } from '@fastgpt/service/common/vectorStore/controller';
|
||||
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
|
||||
import { addHours } from 'date-fns';
|
||||
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
|
||||
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
|
||||
|
||||
/*
|
||||
检测无效的 Mongo 数据
|
||||
异常情况:
|
||||
1. 训练过程删除知识库,可能导致还会有新的数据插入,导致无效。
|
||||
*/
|
||||
|
||||
let deleteAmount = 0;
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
|
||||
try {
|
||||
const { startHour = 3, endHour = 1 } = req.body as { startHour?: number; endHour?: number };
|
||||
await authCert({ req, authRoot: true });
|
||||
await connectToDatabase();
|
||||
|
||||
// start: now - maxDay, end: now - endHour
|
||||
const start = addHours(new Date(), -startHour);
|
||||
const end = addHours(new Date(), -endHour);
|
||||
deleteAmount = 0;
|
||||
|
||||
await checkInvalidCollection(start, end);
|
||||
|
||||
jsonRes(res, {
|
||||
data: deleteAmount,
|
||||
message: 'success'
|
||||
});
|
||||
} catch (error) {
|
||||
addLog.error(`check Invalid user error`, error);
|
||||
|
||||
jsonRes(res, {
|
||||
code: 500,
|
||||
error
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkInvalidCollection(start: Date, end: Date) {
|
||||
// 1. 获取时间范围的所有data
|
||||
const rows = await MongoDatasetData.find(
|
||||
{
|
||||
updateTime: {
|
||||
$gte: start,
|
||||
$lte: end
|
||||
}
|
||||
},
|
||||
'_id teamId collectionId'
|
||||
).lean();
|
||||
|
||||
// 2. 合并所有的collectionId
|
||||
const map = new Map<string, { teamId: string; collectionId: string }>();
|
||||
for (const item of rows) {
|
||||
const collectionId = String(item.collectionId);
|
||||
if (!map.has(collectionId)) {
|
||||
map.set(collectionId, { teamId: item.teamId, collectionId });
|
||||
}
|
||||
}
|
||||
const list = Array.from(map.values());
|
||||
console.log('total collections', list.length);
|
||||
let index = 0;
|
||||
|
||||
for await (const item of list) {
|
||||
try {
|
||||
// 3. 查看该collection是否存在,不存在,则删除对应的数据
|
||||
const collection = await MongoDatasetCollection.findOne({ _id: item.collectionId });
|
||||
if (!collection) {
|
||||
const result = await Promise.all([
|
||||
MongoDatasetTraining.deleteMany({
|
||||
teamId: item.teamId,
|
||||
collectionId: item.collectionId
|
||||
}),
|
||||
MongoDatasetData.deleteMany({
|
||||
teamId: item.teamId,
|
||||
collectionId: item.collectionId
|
||||
}),
|
||||
deleteDatasetDataVector({
|
||||
teamId: item.teamId,
|
||||
collectionIds: [String(item.collectionId)]
|
||||
})
|
||||
]);
|
||||
console.log(result);
|
||||
console.log('collection is not found', item);
|
||||
continue;
|
||||
}
|
||||
} catch (error) {}
|
||||
console.log(++index);
|
||||
}
|
||||
}
|
@@ -1,86 +0,0 @@
|
||||
import type { NextApiRequest, NextApiResponse } from 'next';
|
||||
import { jsonRes } from '@fastgpt/service/common/response';
|
||||
import { connectToDatabase } from '@/service/mongo';
|
||||
import { authCert } from '@fastgpt/service/support/permission/auth/common';
|
||||
import { addLog } from '@fastgpt/service/common/system/log';
|
||||
import {
|
||||
deleteDatasetDataVector,
|
||||
getVectorDataByTime
|
||||
} from '@fastgpt/service/common/vectorStore/controller';
|
||||
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
|
||||
import { addHours } from 'date-fns';
|
||||
|
||||
/*
|
||||
检测无效的 Vector 数据.
|
||||
异常情况:
|
||||
1. 插入数据时,vector成功,mongo失败
|
||||
2. 更新数据,也会有插入 vector
|
||||
*/
|
||||
|
||||
let deletedVectorAmount = 0;
|
||||
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
|
||||
try {
|
||||
const { startHour = 5, endHour = 1 } = req.body as { startHour?: number; endHour?: number };
|
||||
await authCert({ req, authRoot: true });
|
||||
await connectToDatabase();
|
||||
|
||||
// start: now - maxDay, end: now - endHour
|
||||
const start = addHours(new Date(), -startHour);
|
||||
const end = addHours(new Date(), -endHour);
|
||||
deletedVectorAmount = 0;
|
||||
|
||||
await checkInvalidVector(start, end);
|
||||
|
||||
jsonRes(res, {
|
||||
data: deletedVectorAmount,
|
||||
message: 'success'
|
||||
});
|
||||
} catch (error) {
|
||||
addLog.error(`check Invalid user error`, error);
|
||||
|
||||
jsonRes(res, {
|
||||
code: 500,
|
||||
error
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkInvalidVector(start: Date, end: Date) {
|
||||
// 1. get all vector data
|
||||
const rows = await getVectorDataByTime(start, end);
|
||||
console.log('total data', rows.length);
|
||||
|
||||
let index = 0;
|
||||
|
||||
for await (const item of rows) {
|
||||
if (!item.teamId || !item.datasetId || !item.id) {
|
||||
console.log('error data', item);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
// 2. find dataset.data
|
||||
const hasData = await MongoDatasetData.countDocuments({
|
||||
teamId: item.teamId,
|
||||
datasetId: item.datasetId,
|
||||
'indexes.dataId': item.id
|
||||
});
|
||||
|
||||
// 3. if not found, delete vector
|
||||
if (hasData === 0) {
|
||||
await deleteDatasetDataVector({
|
||||
teamId: item.teamId,
|
||||
id: item.id
|
||||
});
|
||||
console.log('delete vector data', item.id);
|
||||
deletedVectorAmount++;
|
||||
}
|
||||
|
||||
index++;
|
||||
index % 100 === 0 && console.log(index);
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`检测完成,共删除 ${deletedVectorAmount} 个无效 向量 数据`);
|
||||
}
|
@@ -117,3 +117,5 @@ export const RenderUploadFiles = ({
|
||||
</>
|
||||
) : null;
|
||||
};
|
||||
|
||||
export default RenderUploadFiles;
|
||||
|
@@ -249,7 +249,7 @@ const InputDataModal = ({
|
||||
return openConfirm(onDeleteData)();
|
||||
}
|
||||
if (e === TabEnum.doc) {
|
||||
return window.open(getDocPath('/docs/course/datasetengine'), '_blank');
|
||||
return window.open(getDocPath('/docs/course/dataset_engine'), '_blank');
|
||||
}
|
||||
setCurrentTab(e);
|
||||
}}
|
||||
|
@@ -1,22 +1,61 @@
|
||||
import { setCron } from '@fastgpt/service/common/system/cron';
|
||||
import { startTrainingQueue } from '@/service/core/dataset/training/utils';
|
||||
import { clearTmpUploadFiles } from '@fastgpt/service/common/file/utils';
|
||||
import { checkInvalidDatasetFiles, checkInvalidDatasetData, checkInvalidVector } from './cronTask';
|
||||
import { checkTimerLock } from '@fastgpt/service/common/system/timerLock/utils';
|
||||
import { TimerIdEnum } from '@fastgpt/service/common/system/timerLock/constants';
|
||||
import { addHours } from 'date-fns';
|
||||
|
||||
export const startCron = () => {
|
||||
setTrainingQueueCron();
|
||||
setClearTmpUploadFilesCron();
|
||||
};
|
||||
|
||||
export const setTrainingQueueCron = () => {
|
||||
const setTrainingQueueCron = () => {
|
||||
setCron('*/1 * * * *', () => {
|
||||
startTrainingQueue();
|
||||
});
|
||||
};
|
||||
|
||||
export const setClearTmpUploadFilesCron = () => {
|
||||
clearTmpUploadFiles();
|
||||
const setClearTmpUploadFilesCron = () => {
|
||||
// Clear tmp upload files every ten minutes
|
||||
setCron('*/10 * * * *', () => {
|
||||
clearTmpUploadFiles();
|
||||
});
|
||||
};
|
||||
|
||||
const clearInvalidDataCron = () => {
|
||||
setCron('0 */1 * * *', async () => {
|
||||
if (
|
||||
await checkTimerLock({
|
||||
timerId: TimerIdEnum.checkInValidDatasetFiles,
|
||||
lockMinuted: 59
|
||||
})
|
||||
) {
|
||||
checkInvalidDatasetFiles(addHours(new Date(), 2), addHours(new Date(), 6));
|
||||
}
|
||||
});
|
||||
|
||||
setCron('10 */1 * * *', async () => {
|
||||
if (
|
||||
await checkTimerLock({
|
||||
timerId: TimerIdEnum.checkInvalidDatasetData,
|
||||
lockMinuted: 59
|
||||
})
|
||||
) {
|
||||
checkInvalidDatasetData(addHours(new Date(), 2), addHours(new Date(), 6));
|
||||
}
|
||||
});
|
||||
|
||||
setCron('30 */1 * * *', async () => {
|
||||
if (
|
||||
await checkTimerLock({
|
||||
timerId: TimerIdEnum.checkInvalidVector,
|
||||
lockMinuted: 59
|
||||
})
|
||||
) {
|
||||
checkInvalidVector(addHours(new Date(), 2), addHours(new Date(), 6));
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
export const startCron = () => {
|
||||
setTrainingQueueCron();
|
||||
setClearTmpUploadFilesCron();
|
||||
clearInvalidDataCron();
|
||||
};
|
||||
|
157
projects/app/src/service/common/system/cronTask.ts
Normal file
157
projects/app/src/service/common/system/cronTask.ts
Normal file
@@ -0,0 +1,157 @@
|
||||
import {
|
||||
delFileByFileIdList,
|
||||
getGFSCollection
|
||||
} from '@fastgpt/service/common/file/gridfs/controller';
|
||||
import { addLog } from '@fastgpt/service/common/system/log';
|
||||
import {
|
||||
deleteDatasetDataVector,
|
||||
getVectorDataByTime
|
||||
} from '@fastgpt/service/common/vectorStore/controller';
|
||||
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema';
|
||||
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
|
||||
import { MongoDatasetTraining } from '@fastgpt/service/core/dataset/training/schema';
|
||||
|
||||
/*
|
||||
check dataset.files data. If there is no match in dataset.collections, delete it
|
||||
可能异常情况
|
||||
1. 上传了文件,未成功创建集合
|
||||
*/
|
||||
export async function checkInvalidDatasetFiles(start: Date, end: Date) {
|
||||
let deleteFileAmount = 0;
|
||||
const collection = getGFSCollection('dataset');
|
||||
const where = {
|
||||
uploadDate: { $gte: start, $lte: end }
|
||||
};
|
||||
|
||||
// 1. get all file _id
|
||||
const files = await collection
|
||||
.find(where, {
|
||||
projection: {
|
||||
metadata: 1,
|
||||
_id: 1
|
||||
}
|
||||
})
|
||||
.toArray();
|
||||
addLog.info(`Clear invalid dataset files, total files: ${files.length}`);
|
||||
|
||||
let index = 0;
|
||||
for await (const file of files) {
|
||||
try {
|
||||
// 2. find fileId in dataset.collections
|
||||
const hasCollection = await MongoDatasetCollection.countDocuments({
|
||||
teamId: file.metadata.teamId,
|
||||
fileId: file._id
|
||||
});
|
||||
|
||||
// 3. if not found, delete file
|
||||
if (hasCollection === 0) {
|
||||
await delFileByFileIdList({ bucketName: 'dataset', fileIdList: [String(file._id)] });
|
||||
console.log('delete file', file._id);
|
||||
deleteFileAmount++;
|
||||
}
|
||||
index++;
|
||||
index % 100 === 0 && console.log(index);
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
}
|
||||
addLog.info(`Clear invalid dataset files finish, remove ${deleteFileAmount} files`);
|
||||
}
|
||||
|
||||
/*
|
||||
检测无效的 Mongo 数据
|
||||
异常情况:
|
||||
1. 训练过程删除知识库,可能导致还会有新的数据继续插入,导致无效。
|
||||
*/
|
||||
export async function checkInvalidDatasetData(start: Date, end: Date) {
|
||||
// 1. 获取时间范围的所有data
|
||||
const rows = await MongoDatasetData.find(
|
||||
{
|
||||
updateTime: {
|
||||
$gte: start,
|
||||
$lte: end
|
||||
}
|
||||
},
|
||||
'_id teamId collectionId'
|
||||
).lean();
|
||||
|
||||
// 2. 合并所有的collectionId
|
||||
const map = new Map<string, { teamId: string; collectionId: string }>();
|
||||
for (const item of rows) {
|
||||
const collectionId = String(item.collectionId);
|
||||
if (!map.has(collectionId)) {
|
||||
map.set(collectionId, { teamId: item.teamId, collectionId });
|
||||
}
|
||||
}
|
||||
const list = Array.from(map.values());
|
||||
addLog.info(`Clear invalid dataset data, total collections: ${list.length}`);
|
||||
let index = 0;
|
||||
|
||||
for await (const item of list) {
|
||||
try {
|
||||
// 3. 查看该collection是否存在,不存在,则删除对应的数据
|
||||
const collection = await MongoDatasetCollection.findOne({ _id: item.collectionId });
|
||||
if (!collection) {
|
||||
const result = await Promise.all([
|
||||
MongoDatasetTraining.deleteMany({
|
||||
teamId: item.teamId,
|
||||
collectionId: item.collectionId
|
||||
}),
|
||||
MongoDatasetData.deleteMany({
|
||||
teamId: item.teamId,
|
||||
collectionId: item.collectionId
|
||||
}),
|
||||
deleteDatasetDataVector({
|
||||
teamId: item.teamId,
|
||||
collectionIds: [String(item.collectionId)]
|
||||
})
|
||||
]);
|
||||
console.log(result);
|
||||
console.log('collection is not found', item);
|
||||
continue;
|
||||
}
|
||||
} catch (error) {}
|
||||
console.log(++index);
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkInvalidVector(start: Date, end: Date) {
|
||||
let deletedVectorAmount = 0;
|
||||
// 1. get all vector data
|
||||
const rows = await getVectorDataByTime(start, end);
|
||||
addLog.info(`Clear invalid vector, total vector data: ${rows.length}`);
|
||||
|
||||
let index = 0;
|
||||
|
||||
for await (const item of rows) {
|
||||
if (!item.teamId || !item.datasetId || !item.id) {
|
||||
addLog.error('error data', item);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
// 2. find dataset.data
|
||||
const hasData = await MongoDatasetData.countDocuments({
|
||||
teamId: item.teamId,
|
||||
datasetId: item.datasetId,
|
||||
'indexes.dataId': item.id
|
||||
});
|
||||
|
||||
// 3. if not found, delete vector
|
||||
if (hasData === 0) {
|
||||
await deleteDatasetDataVector({
|
||||
teamId: item.teamId,
|
||||
id: item.id
|
||||
});
|
||||
console.log('delete vector data', item.id);
|
||||
deletedVectorAmount++;
|
||||
}
|
||||
|
||||
index++;
|
||||
index % 100 === 0 && console.log(index);
|
||||
} catch (error) {
|
||||
console.log(error);
|
||||
}
|
||||
}
|
||||
|
||||
addLog.info(`Clear invalid vector finish, remove ${deletedVectorAmount} data`);
|
||||
}
|
@@ -44,8 +44,7 @@ export async function insertData2Dataset({
|
||||
indexes =
|
||||
Array.isArray(indexes) && indexes.length > 0
|
||||
? indexes.map((index) => ({
|
||||
// @ts-ignore
|
||||
...index.toObject(),
|
||||
text: index.text,
|
||||
dataId: undefined,
|
||||
defaultIndex: index.text.trim() === qaStr
|
||||
}))
|
||||
|
2
projects/app/src/types/user.d.ts
vendored
2
projects/app/src/types/user.d.ts
vendored
@@ -1,9 +1,11 @@
|
||||
import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
|
||||
import type { UserModelSchema } from '@fastgpt/global/support/user/type';
|
||||
import { LafAccountType } from '@fastgpt/global/support/user/team/type.d';
|
||||
|
||||
export interface UserUpdateParams {
|
||||
balance?: number;
|
||||
avatar?: string;
|
||||
timezone?: string;
|
||||
openaiAccount?: UserModelSchema['openaiAccount'];
|
||||
lafAccount?: LafAccountType;
|
||||
}
|
||||
|
169
projects/app/src/web/common/api/lafRequest.ts
Normal file
169
projects/app/src/web/common/api/lafRequest.ts
Normal file
@@ -0,0 +1,169 @@
|
||||
import axios, {
|
||||
Method,
|
||||
InternalAxiosRequestConfig,
|
||||
AxiosResponse,
|
||||
AxiosProgressEvent
|
||||
} from 'axios';
|
||||
import { useUserStore } from '@/web/support/user/useUserStore';
|
||||
|
||||
interface ConfigType {
|
||||
headers?: { [key: string]: string };
|
||||
timeout?: number;
|
||||
onUploadProgress?: (progressEvent: AxiosProgressEvent) => void;
|
||||
cancelToken?: AbortController;
|
||||
maxQuantity?: number;
|
||||
}
|
||||
interface ResponseDataType {
|
||||
error: string | null;
|
||||
data: any;
|
||||
}
|
||||
|
||||
const maxQuantityMap: Record<
|
||||
string,
|
||||
{
|
||||
amount: number;
|
||||
sign: AbortController;
|
||||
}
|
||||
> = {};
|
||||
|
||||
function requestStart({ url, maxQuantity }: { url: string; maxQuantity?: number }) {
|
||||
if (!maxQuantity) return;
|
||||
const item = maxQuantityMap[url];
|
||||
|
||||
if (item) {
|
||||
if (item.amount >= maxQuantity && item.sign) {
|
||||
item.sign.abort();
|
||||
delete maxQuantityMap[url];
|
||||
}
|
||||
} else {
|
||||
maxQuantityMap[url] = {
|
||||
amount: 1,
|
||||
sign: new AbortController()
|
||||
};
|
||||
}
|
||||
}
|
||||
function requestFinish({ url }: { url: string }) {
|
||||
const item = maxQuantityMap[url];
|
||||
if (item) {
|
||||
item.amount--;
|
||||
if (item.amount <= 0) {
|
||||
delete maxQuantityMap[url];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 请求开始
|
||||
*/
|
||||
function startInterceptors(config: InternalAxiosRequestConfig): InternalAxiosRequestConfig {
|
||||
if (config.headers && !config.headers.Authorization) {
|
||||
config.headers.Authorization = `Bearer ${useUserStore.getState().userInfo?.team?.lafAccount?.token || ''}`;
|
||||
}
|
||||
|
||||
return config;
|
||||
}
|
||||
|
||||
/**
|
||||
* 请求成功,检查请求头
|
||||
*/
|
||||
function responseSuccess(response: AxiosResponse<ResponseDataType>) {
|
||||
return response;
|
||||
}
|
||||
/**
|
||||
* 响应数据检查
|
||||
*/
|
||||
function checkRes(data: ResponseDataType) {
|
||||
if (data === undefined) {
|
||||
console.log('error->', data, 'data is empty');
|
||||
return Promise.reject('服务器异常');
|
||||
} else if (data.error) {
|
||||
return responseError(data.error);
|
||||
}
|
||||
|
||||
return data.data;
|
||||
}
|
||||
|
||||
/**
|
||||
* 响应错误
|
||||
*/
|
||||
function responseError(err: any) {
|
||||
console.log('error->', '请求错误', err);
|
||||
|
||||
if (!err) {
|
||||
return Promise.reject({ message: '未知错误' });
|
||||
}
|
||||
if (typeof err === 'string') {
|
||||
return Promise.reject({ message: err });
|
||||
}
|
||||
|
||||
if (err?.response?.data) {
|
||||
return Promise.reject(err?.response?.data);
|
||||
}
|
||||
return Promise.reject(err);
|
||||
}
|
||||
|
||||
/* 创建请求实例 */
|
||||
const instance = axios.create({
|
||||
timeout: 60000, // 超时时间
|
||||
headers: {
|
||||
'content-type': 'application/json'
|
||||
}
|
||||
});
|
||||
|
||||
/* 请求拦截 */
|
||||
instance.interceptors.request.use(startInterceptors, (err) => Promise.reject(err));
|
||||
/* 响应拦截 */
|
||||
instance.interceptors.response.use(responseSuccess, (err) => Promise.reject(err));
|
||||
|
||||
function request(
|
||||
url: string,
|
||||
data: any,
|
||||
{ cancelToken, maxQuantity, ...config }: ConfigType,
|
||||
method: Method
|
||||
): any {
|
||||
/* 去空 */
|
||||
for (const key in data) {
|
||||
if (data[key] === null || data[key] === undefined) {
|
||||
delete data[key];
|
||||
}
|
||||
}
|
||||
|
||||
requestStart({ url, maxQuantity });
|
||||
|
||||
return instance
|
||||
.request({
|
||||
baseURL: '/api/lafApi',
|
||||
url,
|
||||
method,
|
||||
data: ['POST', 'PUT'].includes(method) ? data : null,
|
||||
params: !['POST', 'PUT'].includes(method) ? data : null,
|
||||
signal: cancelToken?.signal,
|
||||
...config // 用户自定义配置,可以覆盖前面的配置
|
||||
})
|
||||
.then((res) => checkRes(res.data))
|
||||
.catch((err) => responseError(err))
|
||||
.finally(() => requestFinish({ url }));
|
||||
}
|
||||
|
||||
/**
|
||||
* api请求方式
|
||||
* @param {String} url
|
||||
* @param {Any} params
|
||||
* @param {Object} config
|
||||
* @returns
|
||||
*/
|
||||
export function GET<T = undefined>(url: string, params = {}, config: ConfigType = {}): Promise<T> {
|
||||
return request(url, params, config, 'GET');
|
||||
}
|
||||
|
||||
export function POST<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
|
||||
return request(url, data, config, 'POST');
|
||||
}
|
||||
|
||||
export function PUT<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
|
||||
return request(url, data, config, 'PUT');
|
||||
}
|
||||
|
||||
export function DELETE<T = undefined>(url: string, data = {}, config: ConfigType = {}): Promise<T> {
|
||||
return request(url, data, config, 'DELETE');
|
||||
}
|
34
projects/app/src/web/support/laf/api.ts
Normal file
34
projects/app/src/web/support/laf/api.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
import { GET, POST, PUT } from '@/web/common/api/lafRequest';
|
||||
|
||||
export const postLafPat2Token = (pat: string) => POST<string>(`/v1/auth/pat2token`, { pat });
|
||||
|
||||
export const getLafApplications = (token: string) =>
|
||||
GET<
|
||||
{
|
||||
appid: string;
|
||||
name: string;
|
||||
state: 'Running' | 'Failed' | 'Stopped';
|
||||
}[]
|
||||
>(
|
||||
`/v1/applications`,
|
||||
{},
|
||||
{
|
||||
headers: {
|
||||
Authorization: `Bearer ${token}`
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
export const getLafAppDetail = (appid: string) =>
|
||||
GET<{
|
||||
appid: string;
|
||||
name: string;
|
||||
openapi_token: string;
|
||||
domain: {
|
||||
_id: string;
|
||||
appid: string;
|
||||
domain: string;
|
||||
state: string;
|
||||
phase: string;
|
||||
};
|
||||
}>(`/v1/applications/${appid}`);
|
@@ -38,7 +38,7 @@
|
||||
|
||||
**镜像和端口**
|
||||
|
||||
+ 镜像名: `luanshaotong/reranker:v0.2`
|
||||
+ 镜像名: `registry.cn-hangzhou.aliyuncs.com/fastgpt/rerank:v0.2`
|
||||
+ 端口号: 6006
|
||||
|
||||
```
|
||||
|
Reference in New Issue
Block a user