mirror of
https://github.com/labring/FastGPT.git
synced 2025-07-30 10:28:42 +00:00
V4.9.11 feature (#4969)
* Feat: Images dataset collection (#4941) * New pic (#4858) * 更新数据集相关类型，添加图像文件ID和预览URL支持；优化数据集导入功能，新增图像数据集处理组件；修复部分国际化文本；更新文件上传逻辑以支持新功能。 * 与原先代码的差别 * 新增 V4.9.10 更新说明，支持 PG 设置`systemEnv.hnswMaxScanTuples`参数，优化 LLM stream 调用超时，修复全文检索多知识库排序问题。同时更新数据集索引，移除 datasetId 字段以简化查询。 * 更换成fileId_image逻辑，并增加训练队列匹配的逻辑 * 新增图片集合判断逻辑，优化预览URL生成流程，确保仅在数据集为图片集合时生成预览URL，并添加相关日志输出以便调试。 * Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages. * Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API. * Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook. * Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets. * Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process. * 图片上传到新建的 dataset_collection_images 表，逻辑跟随更改 * 修改了除了controller的其他部分问题 * 把图片数据集的逻辑整合到controller里面 * 补充i18n * 补充i18n * resolve评论：主要是上传逻辑的更改和组件复用 * 图片名称的图标显示 * 修改编译报错的命名问题 * 删除不需要的collectionid部分 * 多余文件的处理和改动一个删除按钮 * 除了loading和统一的imageId,其他都resolve掉的 * 处理图标报错 * 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId * 去除不必要文件修改 * 报错和字段修改 * 增加上传成功后删除临时文件的逻辑以及回退一些修改 * 删除path字段，将图片保存到gridfs内，并修改增删等操作的代码 * 修正编译错误 --------- Co-authored-by: archer <545436317@qq.com> * perf: image dataset * feat: insert image * perf: image icon * fix: training state --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> * fix: ts (#4948) * Thirddatasetmd (#4942) * add thirddataset.md * fix thirddataset.md * fix * delete wrong png --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: api dataset code * perf: log * add secondary.tsx (#4946) * add secondary.tsx * fix --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: multiple menu * perf: i18n * feat: parse queue (#4960) * feat: parse queue * feat: sync parse queue * fix thirddataset.md (#4962) * fix thirddataset-4.png (#4963) * feat: Dataset template import (#4934) * 模版导入部分除了文档还没写 * 修复模版导入的 build 错误 * Document production * compress pictures * Change some constants to variables --------- Co-authored-by: Archer <545436317@qq.com> * perf: template import * doc * llm pargraph * bocha tool * fix: del collection --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> Co-authored-by: dreamer6680 <1468683855@qq.com> Co-authored-by: dreamer6680 <146868355@qq.com>
This commit is contained in:
Archer
208
packages/service/core/dataset/apiDataset/feishuDataset/api.ts
Normal file
208
packages/service/core/dataset/apiDataset/feishuDataset/api.ts
Normal file
@@ -0,0 +1,208 @@
|
||||
import type {
|
||||
APIFileItem,
|
||||
ApiFileReadContentResponse,
|
||||
ApiDatasetDetailResponse,
|
||||
FeishuServer
|
||||
} from '@fastgpt/global/core/dataset/apiDataset/type';
|
||||
import { type ParentIdType } from '@fastgpt/global/common/parentFolder/type';
|
||||
import axios, { type Method } from 'axios';
|
||||
import { addLog } from '../../../../common/system/log';
|
||||
|
||||
type ResponseDataType = {
|
||||
success: boolean;
|
||||
message: string;
|
||||
data: any;
|
||||
};
|
||||
|
||||
type FeishuFileListResponse = {
|
||||
files: {
|
||||
token: string;
|
||||
parent_token: string;
|
||||
name: string;
|
||||
type: string;
|
||||
modified_time: number;
|
||||
created_time: number;
|
||||
url: string;
|
||||
owner_id: string;
|
||||
}[];
|
||||
has_more: boolean;
|
||||
next_page_token: string;
|
||||
};
|
||||
|
||||
const feishuBaseUrl = process.env.FEISHU_BASE_URL || 'https://open.feishu.cn';
|
||||
|
||||
export const useFeishuDatasetRequest = ({ feishuServer }: { feishuServer: FeishuServer }) => {
|
||||
const instance = axios.create({
|
||||
baseURL: feishuBaseUrl,
|
||||
timeout: 60000
|
||||
});
|
||||
|
||||
// 添加请求拦截器
|
||||
instance.interceptors.request.use(async (config) => {
|
||||
if (!config.headers.Authorization) {
|
||||
const { data } = await axios.post<{ tenant_access_token: string }>(
|
||||
`${feishuBaseUrl}/open-apis/auth/v3/tenant_access_token/internal`,
|
||||
{
|
||||
app_id: feishuServer.appId,
|
||||
app_secret: feishuServer.appSecret
|
||||
}
|
||||
);
|
||||
|
||||
config.headers['Authorization'] = `Bearer ${data.tenant_access_token}`;
|
||||
config.headers['Content-Type'] = 'application/json; charset=utf-8';
|
||||
}
|
||||
return config;
|
||||
});
|
||||
|
||||
/**
|
||||
* 响应数据检查
|
||||
*/
|
||||
const checkRes = (data: ResponseDataType) => {
|
||||
if (data === undefined) {
|
||||
addLog.info('yuque dataset data is empty');
|
||||
return Promise.reject('服务器异常');
|
||||
}
|
||||
return data.data;
|
||||
};
|
||||
const responseError = (err: any) => {
|
||||
console.log('error->', '请求错误', err);
|
||||
|
||||
if (!err) {
|
||||
return Promise.reject({ message: '未知错误' });
|
||||
}
|
||||
if (typeof err === 'string') {
|
||||
return Promise.reject({ message: err });
|
||||
}
|
||||
if (typeof err.message === 'string') {
|
||||
return Promise.reject({ message: err.message });
|
||||
}
|
||||
if (typeof err.data === 'string') {
|
||||
return Promise.reject({ message: err.data });
|
||||
}
|
||||
if (err?.response?.data) {
|
||||
return Promise.reject(err?.response?.data);
|
||||
}
|
||||
return Promise.reject(err);
|
||||
};
|
||||
|
||||
const request = <T>(url: string, data: any, method: Method): Promise<T> => {
|
||||
/* 去空 */
|
||||
for (const key in data) {
|
||||
if (data[key] === undefined) {
|
||||
delete data[key];
|
||||
}
|
||||
}
|
||||
|
||||
return instance
|
||||
.request({
|
||||
url,
|
||||
method,
|
||||
data: ['POST', 'PUT'].includes(method) ? data : undefined,
|
||||
params: !['POST', 'PUT'].includes(method) ? data : undefined
|
||||
})
|
||||
.then((res) => checkRes(res.data))
|
||||
.catch((err) => responseError(err));
|
||||
};
|
||||
|
||||
const listFiles = async ({ parentId }: { parentId?: ParentIdType }): Promise<APIFileItem[]> => {
|
||||
const fetchFiles = async (pageToken?: string): Promise<FeishuFileListResponse['files']> => {
|
||||
const data = await request<FeishuFileListResponse>(
|
||||
`/open-apis/drive/v1/files`,
|
||||
{
|
||||
folder_token: parentId || feishuServer.folderToken,
|
||||
page_size: 200,
|
||||
page_token: pageToken
|
||||
},
|
||||
'GET'
|
||||
);
|
||||
|
||||
if (data.has_more) {
|
||||
const nextFiles = await fetchFiles(data.next_page_token);
|
||||
return [...data.files, ...nextFiles];
|
||||
}
|
||||
|
||||
return data.files;
|
||||
};
|
||||
|
||||
const allFiles = await fetchFiles();
|
||||
|
||||
return allFiles
|
||||
.filter((file) => ['folder', 'docx'].includes(file.type))
|
||||
.map((file) => ({
|
||||
id: file.token,
|
||||
parentId: file.parent_token,
|
||||
name: file.name,
|
||||
type: file.type === 'folder' ? ('folder' as const) : ('file' as const),
|
||||
hasChild: file.type === 'folder',
|
||||
updateTime: new Date(file.modified_time * 1000),
|
||||
createTime: new Date(file.created_time * 1000)
|
||||
}));
|
||||
};
|
||||
|
||||
const getFileContent = async ({
|
||||
apiFileId
|
||||
}: {
|
||||
apiFileId: string;
|
||||
}): Promise<ApiFileReadContentResponse> => {
|
||||
const [{ content }, { document }] = await Promise.all([
|
||||
request<{ content: string }>(
|
||||
`/open-apis/docx/v1/documents/${apiFileId}/raw_content`,
|
||||
{},
|
||||
'GET'
|
||||
),
|
||||
request<{ document: { title: string } }>(
|
||||
`/open-apis/docx/v1/documents/${apiFileId}`,
|
||||
{},
|
||||
'GET'
|
||||
)
|
||||
]);
|
||||
|
||||
return {
|
||||
title: document?.title,
|
||||
rawText: content
|
||||
};
|
||||
};
|
||||
|
||||
const getFilePreviewUrl = async ({ apiFileId }: { apiFileId: string }): Promise<string> => {
|
||||
const { metas } = await request<{ metas: { url: string }[] }>(
|
||||
`/open-apis/drive/v1/metas/batch_query`,
|
||||
{
|
||||
request_docs: [
|
||||
{
|
||||
doc_token: apiFileId,
|
||||
doc_type: 'docx'
|
||||
}
|
||||
],
|
||||
with_url: true
|
||||
},
|
||||
'POST'
|
||||
);
|
||||
|
||||
return metas[0].url;
|
||||
};
|
||||
|
||||
const getFileDetail = async ({
|
||||
apiFileId
|
||||
}: {
|
||||
apiFileId: string;
|
||||
}): Promise<ApiDatasetDetailResponse> => {
|
||||
const { document } = await request<{ document: { title: string } }>(
|
||||
`/open-apis/docx/v1/documents/${apiFileId}`,
|
||||
{},
|
||||
'GET'
|
||||
);
|
||||
|
||||
return {
|
||||
name: document?.title,
|
||||
parentId: null,
|
||||
id: apiFileId
|
||||
};
|
||||
};
|
||||
|
||||
return {
|
||||
getFileContent,
|
||||
listFiles,
|
||||
getFilePreviewUrl,
|
||||
getFileDetail
|
||||
};
|
||||
};
|
||||
Reference in New Issue
Block a user