mirror of
https://github.com/labring/FastGPT.git
synced 2025-07-22 04:06:18 +00:00
V4.9.11 feature (#4969)
* Feat: Images dataset collection (#4941) * New pic (#4858) * 更新数据集相关类型，添加图像文件ID和预览URL支持；优化数据集导入功能，新增图像数据集处理组件；修复部分国际化文本；更新文件上传逻辑以支持新功能。 * 与原先代码的差别 * 新增 V4.9.10 更新说明，支持 PG 设置`systemEnv.hnswMaxScanTuples`参数，优化 LLM stream 调用超时，修复全文检索多知识库排序问题。同时更新数据集索引，移除 datasetId 字段以简化查询。 * 更换成fileId_image逻辑，并增加训练队列匹配的逻辑 * 新增图片集合判断逻辑，优化预览URL生成流程，确保仅在数据集为图片集合时生成预览URL，并添加相关日志输出以便调试。 * Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages. * Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API. * Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook. * Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets. * Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process. * 图片上传到新建的 dataset_collection_images 表，逻辑跟随更改 * 修改了除了controller的其他部分问题 * 把图片数据集的逻辑整合到controller里面 * 补充i18n * 补充i18n * resolve评论：主要是上传逻辑的更改和组件复用 * 图片名称的图标显示 * 修改编译报错的命名问题 * 删除不需要的collectionid部分 * 多余文件的处理和改动一个删除按钮 * 除了loading和统一的imageId,其他都resolve掉的 * 处理图标报错 * 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId * 去除不必要文件修改 * 报错和字段修改 * 增加上传成功后删除临时文件的逻辑以及回退一些修改 * 删除path字段，将图片保存到gridfs内，并修改增删等操作的代码 * 修正编译错误 --------- Co-authored-by: archer <545436317@qq.com> * perf: image dataset * feat: insert image * perf: image icon * fix: training state --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> * fix: ts (#4948) * Thirddatasetmd (#4942) * add thirddataset.md * fix thirddataset.md * fix * delete wrong png --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: api dataset code * perf: log * add secondary.tsx (#4946) * add secondary.tsx * fix --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: multiple menu * perf: i18n * feat: parse queue (#4960) * feat: parse queue * feat: sync parse queue * fix thirddataset.md (#4962) * fix thirddataset-4.png (#4963) * feat: Dataset template import (#4934) * 模版导入部分除了文档还没写 * 修复模版导入的 build 错误 * Document production * compress pictures * Change some constants to variables --------- Co-authored-by: Archer <545436317@qq.com> * perf: template import * doc * llm pargraph * bocha tool * fix: del collection --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> Co-authored-by: dreamer6680 <1468683855@qq.com> Co-authored-by: dreamer6680 <146868355@qq.com>
This commit is contained in:
Archer
@@ -295,12 +295,15 @@ curl --location --request DELETE 'http://localhost:3000/api/core/dataset/delete?
|
||||
| --- | --- | --- |
|
||||
| datasetId | 知识库ID | ✅ |
|
||||
| parentId： | 父级ID，不填则默认为根目录 | |
|
||||
| trainingType | 数据处理方式。chunk: 按文本长度进行分割;qa: 问答对提取 | ✅ |
|
||||
| customPdfParse | PDF增强解析。true: 开启PDF增强解析;不填则默认为false | |
|
||||
| trainingType | 数据处理方式。chunk: 按文本长度进行分割;qa: 问答对提取 | ✅ |
|
||||
| chunkTriggerType | 分块条件逻辑。minSize（默认）: 大于 n 时分块;maxSize: 小于文件处理模型最大上下文时分块;forceChunk: 强制分块 | |
|
||||
| chunkTriggerMinSize | chunkTriggerType=minSize 时候填写，原文长度大于该值时候分块（默认 1000） | |
|
||||
| autoIndexes | 是否自动生成索引(仅商业版支持) | |
|
||||
| imageIndex | 是否自动生成图片索引(仅商业版支持) | |
|
||||
| chunkSettingMode | 分块参数模式。auto: 系统默认参数; custom: 手动指定参数 | |
|
||||
| chunkSplitMode | 分块拆分模式。size: 按长度拆分; char: 按字符拆分。chunkSettingMode=auto时不生效。 | |
|
||||
| chunkSplitMode | 分块拆分模式。paragraph：段落优先，再按长度分；size: 按长度拆分; char: 按字符拆分。chunkSettingMode=auto时不生效。 | |
|
||||
| paragraphChunkDeep | 最大段落深度（默认 5） | |
|
||||
| chunkSize | 分块大小，默认 1500。chunkSettingMode=auto时不生效。 | |
|
||||
| indexSize | 索引大小，默认 512，必须小于索引模型最大token。chunkSettingMode=auto时不生效。 | |
|
||||
| chunkSplitter | 自定义最高优先分割符号，除非超出文件处理最大上下文，否则不会进行进一步拆分。chunkSettingMode=auto时不生效。 | |
|
||||
@@ -428,10 +431,7 @@ data 为集合的 ID。
|
||||
"data": {
|
||||
"collectionId": "65abcfab9d1448617cba5f0d",
|
||||
"results": {
|
||||
"insertLen": 5, // 分割成多少段
|
||||
"overToken": [],
|
||||
"repeat": [],
|
||||
"error": []
|
||||
"insertLen": 5 // 分割成多少段
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -497,10 +497,7 @@ data 为集合的 ID。
|
||||
"data": {
|
||||
"collectionId": "65abd0ad9d1448617cba6031",
|
||||
"results": {
|
||||
"insertLen": 1,
|
||||
"overToken": [],
|
||||
"repeat": [],
|
||||
"error": []
|
||||
"insertLen": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -546,7 +543,7 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/collectio
|
||||
{{< tab tabName="响应示例" >}}
|
||||
{{< markdownify >}}
|
||||
|
||||
data 为集合的 ID。
|
||||
由于解析文档是异步操作，此处不会返回插入的数量。
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -556,10 +553,7 @@ data 为集合的 ID。
|
||||
"data": {
|
||||
"collectionId": "65abc044e4704bac793fbd81",
|
||||
"results": {
|
||||
"insertLen": 1,
|
||||
"overToken": [],
|
||||
"repeat": [],
|
||||
"error": []
|
||||
"insertLen": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -632,10 +626,7 @@ data 为集合的 ID。
|
||||
"data": {
|
||||
"collectionId": "65abc044e4704bac793fbd81",
|
||||
"results": {
|
||||
"insertLen": 1,
|
||||
"overToken": [],
|
||||
"repeat": [],
|
||||
"error": []
|
||||
"insertLen": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -702,10 +693,7 @@ data 为集合的 ID。
|
||||
"data": {
|
||||
"collectionId": "6646fcedfabd823cdc6de746",
|
||||
"results": {
|
||||
"insertLen": 1,
|
||||
"overToken": [],
|
||||
"repeat": [],
|
||||
"error": []
|
||||
"insertLen": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user