4.6.7-alpha commit (#743)

Co-authored-by: Archer <545436317@qq.com>
Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>

docSite/content/docs/commercial/intro.md

@@ -44,15 +44,19 @@ FastGPT 商业版软件根据不同的部署方式，分为 3 类收费模式。
 **特有服务**
 
 {{< table "table-hover table-striped-columns" >}}
-| 部署方式 | 特有服务 | 上线时长 | 价格 |
+| 部署方式 | 特有服务 | 上线时长 | 标品价格 |
 | ---- | ---- | ---- | ---- |
 | Sealos全托管 |  1. 有效期内免费升级。<br>2. 免运维服务&数据库。 |  半天  | 3000元起/月（3个月起）<br>或<br>30000元起/年  |
 | 自有服务器-单机版 |  1. 6个版本的升级服务。 | 14天内 |  60000元/套（不限时长） | 
-| 自有服务器-Sealos版 | 1. 6个版本的升级服务。 | 14天内 | 150000元/套（不限时长）|
+| 自有服务器-高可用版 | 1. 6个版本的升级服务。 | 14天内 | 150000元/套（不限时长）|
 {{< /table >}}
 
 {{% alert icon="🤖 " context="success" %}}
-6个版本的升级服务不是指只能用 6 个版本，而是指依赖 FastGPT 团队提供的升级服务。大部分时候，建议自行升级，也不麻烦。
+- 6个版本的升级服务不是指只能用 6 个版本，而是指依赖 FastGPT 团队提供的升级服务。大部分时候，建议自行升级，也不麻烦。
+- 全托管版本适合技术人员紧缺的团队，仅需关注业务推动，无需关心服务是否正常运行。
+- 单机版和高可用版可以完全部署在自己服务器中。
+- 单机版适合中小团队对内提供服务，需要自己维护数据库备份等。
+- 高可用版适合对外提供在线服务，包含可视化监控、多副本、负载均衡、数据库自动备份等生产环境的基础设施。
 {{% /alert %}}
 
 

docSite/content/docs/development/configuration.md

@@ -277,7 +277,7 @@ weight: 708
       "maxContext": 1600,
       "maxResponse": 4000,
       "inputPrice": 0,
-      "outputPrice": 0,
+      "outputPrice": 0
     }
   ],
   "vectorModels": [ // 向量模型

docSite/content/docs/development/qa.md

@@ -25,7 +25,7 @@ OneAPI 账号的余额不足，默认 root 用户只有 200 刀，可以手动
 
 ### xxx渠道找不到
 
-OneAPI 中没有配置该模型渠道。
+OneAPI 中没有配置该模型渠道。或者是修改了配置文件中一部分的模型，但没有全部修改。
 
 ### 页面中可以正常回复，API 报错
 
@@ -35,6 +35,15 @@ OneAPI 中没有配置该模型渠道。
 
 OneAPI 的 API Key 配置错误，需要修改`OPENAI_API_KEY`环境变量，并重启容器（先 stop 然后 rm 掉，最后再 up -d 运行一次）。可以`exec`进入容器，`env`查看环境变量是否生效。
 
+### 其他模型没法进行问题分类/内容提取
+
+需要给其他模型配置`toolChoice=false`，就会默认走提示词模式。目前内置提示词仅针对了商业模型API进行测试，国内外的商业模型基本都可用。
+   
+### 页面崩溃
+
+1. 关闭翻译
+2. 检查配置文件是否正常加载，如果没有正常加载会导致缺失系统信息，在某些操作下会导致空指针。
+
 ## Docker 部署常见问题
 
 ### 如何更新？

docSite/content/docs/development/upgrading/467.md

@@ -0,0 +1,33 @@
+---
+title: 'V4.6.7（需要初始化）'
+description: 'FastGPT V4.6.7'
+icon: 'upgrade'
+draft: false
+toc: true
+weight: 829
+---
+
+## 1。执行初始化 API
+
+发起 1 个 HTTP 请求 ({{rootkey}} 替换成环境变量里的 `rootkey`，{{host}} 替换成自己域名)
+
+1. https://xxxxx/api/admin/initv464
+
+```bash
+curl --location --request POST 'https://{{host}}/api/admin/initv467' \
+--header 'rootkey: {{rootkey}}' \
+--header 'Content-Type: application/json'
+```
+
+初始化说明：
+1. 将 images 重新关联到数据集（不初始化也问题不大，就是可能会留下永久脏数据）
+
+
+## V4.6.7 更新说明
+
+1. 修改了知识库UI及新的导入交互方式。
+2. 优化知识库和对话的数据索引。
+3. 知识库 openAPI，支持通过 API 操作知识库。（文档待补充）
+4. 新增 - 输入框变量提示。输入 { 号后将会获得可用变量提示。根据社区针对高级编排的反馈，我们计划于 2 月份的版本中，优化变量内容，支持模块的局部变量以及更多全局变量写入。
+5. 修复 - API 对话时，chatId 冲突问题。
+6. 修复 - Iframe 嵌入网页可能导致的 window.onLoad 冲突。

docSite/content/docs/use-cases/datasetEngine.md

@@ -25,7 +25,9 @@ FastGPT 采用了 RAG 中的 Embedding 方案构建知识库，要使用好 Fast
 
 FastGPT 采用了 `PostgresSQL` 的 `PG Vector` 插件作为向量检索器，索引为`HNSW`。且`PostgresSQL`仅用于向量检索，`MongoDB`用于其他数据的存取。
 
-在`PostgresSQL`的表中，设置一个 `index` 字段用于存储向量，以及一个`data_id`用于在`MongoDB`中寻找对应的映射值。多个`index`可以对应一组`data_id`，也就是说，一组向量可以对应多组数据。在进行检索时，相同数据会进行合并。
+在`MongoDB`的`dataset.datas`表中，会存储向量原数据的信息，同时有一个`indexes`字段，会记录其对应的向量ID，这是一个数组，也就是说，一组向量可以对应多组数据。
+
+在`PostgresSQL`的表中，设置一个 `index` 字段用于存储向量。在检索时，会先召回向量，再根据向量的ID，去`MongoDB`中寻找原数据内容，如果对应了同一组原数据，则进行合并，向量得分取最高得分。
 
 ![](/imgs/datasetSetting1.png)
 

packages/global/common/file/icon.ts

@@ -1,9 +1,9 @@
 export const fileImgs = [
-  { suffix: 'pdf', src: '/imgs/files/pdf.svg' },
-  { suffix: 'csv', src: '/imgs/files/csv.svg' },
-  { suffix: '(doc|docs)', src: '/imgs/files/doc.svg' },
-  { suffix: 'txt', src: '/imgs/files/txt.svg' },
-  { suffix: 'md', src: '/imgs/files/markdown.svg' }
+  { suffix: 'pdf', src: 'file/fill/pdf' },
+  { suffix: 'csv', src: 'file/fill/csv' },
+  { suffix: '(doc|docs)', src: 'file/fill/doc' },
+  { suffix: 'txt', src: 'file/fill/txt' },
+  { suffix: 'md', src: 'file/fill/markdown' }
   // { suffix: '.', src: '/imgs/files/file.svg' }
 ];
 

packages/global/common/file/image/constants.ts

@@ -9,7 +9,7 @@ export enum MongoImageTypeEnum {
   teamAvatar = 'teamAvatar',
 
   chatImage = 'chatImage',
-  docImage = 'docImage'
+  collectionImage = 'collectionImage'
 }
 export const mongoImageTypeMap = {
   [MongoImageTypeEnum.systemAvatar]: {
@@ -41,8 +41,8 @@ export const mongoImageTypeMap = {
     label: 'common.file.type.chatImage',
     unique: false
   },
-  [MongoImageTypeEnum.docImage]: {
-    label: 'common.file.type.docImage',
+  [MongoImageTypeEnum.collectionImage]: {
+    label: 'common.file.type.collectionImage',
     unique: false
   }
 };

packages/global/common/file/image/type.d.ts

@@ -1,11 +1,14 @@
 import { MongoImageTypeEnum } from './constants';
 
 export type MongoImageSchemaType = {
+  _id: string;
   teamId: string;
   binary: Buffer;
   createTime: Date;
   expiredTime?: Date;
   type: `${MongoImageTypeEnum}`;
 
-  metadata?: { fileId?: string };
+  metadata?: {
+    relatedId?: string; // This id is associated with a set of images
+  };
 };

packages/global/common/string/textSplitter.ts

@@ -13,13 +13,12 @@ export const splitText2Chunks = (props: {
   chunkLen: number;
   overlapRatio?: number;
   customReg?: string[];
-  countTokens?: boolean;
 }): {
   chunks: string[];
-  tokens: number;
+  chars: number;
   overlapRatio?: number;
 } => {
-  let { text = '', chunkLen, overlapRatio = 0.2, customReg = [], countTokens = true } = props;
+  let { text = '', chunkLen, overlapRatio = 0.2, customReg = [] } = props;
   const splitMarker = 'SPLIT_HERE_SPLIT_HERE';
   const codeBlockMarker = 'CODE_BLOCK_LINE_MARKER';
   const overlapLen = Math.round(chunkLen * overlapRatio);
@@ -240,13 +239,11 @@ export const splitText2Chunks = (props: {
       mdTitle: ''
     }).map((chunk) => chunk?.replaceAll(codeBlockMarker, '\n') || ''); // restore code block
 
-    const tokens = countTokens
-      ? chunks.reduce((sum, chunk) => sum + countPromptTokens(chunk, 'system'), 0)
-      : 0;
+    const chars = chunks.reduce((sum, chunk) => sum + chunk.length, 0);
 
     return {
       chunks,
-      tokens
+      chars
     };
   } catch (err) {
     throw new Error(getErrText(err));

packages/global/common/system/types/index.d.ts

@@ -55,6 +55,8 @@ export type FastGPTFeConfigsType = {
     datasetStoreFreeSize?: number;
     datasetStorePrice?: number;
   };
+
+  uploadFileMaxSize?: number;
 };
 
 export type SystemEnvType = {

packages/global/core/app/type.d.ts

@@ -4,7 +4,7 @@ import { PermissionTypeEnum } from '../../support/permission/constant';
 import type { AIChatModuleProps, DatasetModuleProps } from '../module/node/type.d';
 import { VariableInputEnum } from '../module/constants';
 import { SelectedDatasetType } from '../module/api';
-import { DatasetSearchModeEnum } from '../dataset/constant';
+import { DatasetSearchModeEnum } from '../dataset/constants';
 
 export interface AppSchema {
   _id: string;

packages/global/core/app/utils.ts

@@ -4,7 +4,7 @@ import { ModuleOutputKeyEnum, ModuleInputKeyEnum } from '../module/constants';
 import type { FlowNodeInputItemType } from '../module/node/type.d';
 import { getGuideModule, splitGuideModule } from '../module/utils';
 import { ModuleItemType } from '../module/type.d';
-import { DatasetSearchModeEnum } from '../dataset/constant';
+import { DatasetSearchModeEnum } from '../dataset/constants';
 
 export const getDefaultAppForm = (templateId = 'fastgpt-universal'): AppSimpleEditFormType => {
   return {

packages/global/core/chat/type.d.ts

@@ -4,7 +4,7 @@ import { ChatRoleEnum, ChatSourceEnum, ChatStatusEnum } from './constants';
 import { FlowNodeTypeEnum } from '../module/node/constant';
 import { ModuleOutputKeyEnum } from '../module/constants';
 import { AppSchema } from '../app/type';
-import { DatasetSearchModeEnum } from '../dataset/constant';
+import { DatasetSearchModeEnum } from '../dataset/constants';
 
 export type ChatSchema = {
   _id: string;
@@ -92,6 +92,7 @@ export type moduleDispatchResType = {
   runningTime?: number;
   inputTokens?: number;
   outputTokens?: number;
+  charsLength?: number;
   model?: string;
   query?: string;
   contextTotalLen?: number;

packages/global/core/dataset/api.d.ts

@@ -1,5 +1,5 @@
 import { DatasetDataIndexItemType, DatasetSchemaType } from './type';
-import { TrainingModeEnum, DatasetCollectionTypeEnum } from './constant';
+import { TrainingModeEnum, DatasetCollectionTypeEnum } from './constants';
 import type { LLMModelItemType } from '../ai/model.d';
 
 /* ================= dataset ===================== */
@@ -17,27 +17,25 @@ export type DatasetUpdateBody = {
 
 /* ================= collection ===================== */
 export type DatasetCollectionChunkMetadataType = {
+  parentId?: string;
   trainingType?: `${TrainingModeEnum}`;
   chunkSize?: number;
   chunkSplitter?: string;
   qaPrompt?: string;
+  metadata?: Record<string, any>;
 };
 export type CreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & {
   datasetId: string;
-  parentId?: string;
   name: string;
   type: `${DatasetCollectionTypeEnum}`;
   fileId?: string;
   rawLink?: string;
   rawTextLength?: number;
   hashRawText?: string;
-  metadata?: Record<string, any>;
 };
 
 export type ApiCreateDatasetCollectionParams = DatasetCollectionChunkMetadataType & {
   datasetId: string;
-  parentId?: string;
-  metadata?: Record<string, any>;
 };
 export type TextCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
   name: string;
@@ -45,16 +43,24 @@ export type TextCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams
 };
 export type LinkCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
   link: string;
-  chunkSplitter?: string;
+};
+export type FileCreateDatasetCollectionParams = ApiCreateDatasetCollectionParams & {
+  name: string;
+  rawTextLength: number;
+  hashRawText: string;
+  trainingType: `${TrainingModeEnum}`;
+  chunkSize: number;
+  chunkSplitter: string;
+  qaPrompt: string;
+
+  fileMetadata?: Record<string, any>;
+  collectionMetadata?: Record<string, any>;
 };
 
 /* ================= data ===================== */
 export type PgSearchRawType = {
   id: string;
-  team_id: string;
-  tmb_id: string;
   collection_id: string;
-  data_id: string;
   score: number;
 };
 export type PushDatasetDataChunkProps = {

packages/global/core/dataset/constants.ts

@@ -6,7 +6,7 @@ export enum DatasetTypeEnum {
 }
 export const DatasetTypeMap = {
   [DatasetTypeEnum.folder]: {
-    icon: 'core/dataset/folderDataset',
+    icon: 'common/folderFill',
     label: 'core.dataset.Folder Dataset',
     collectionLabel: 'common.Folder'
   },
@@ -104,10 +104,12 @@ export enum TrainingModeEnum {
 
 export const TrainingTypeMap = {
   [TrainingModeEnum.chunk]: {
-    label: 'core.dataset.training.type chunk'
+    label: 'core.dataset.training.Chunk mode',
+    tooltip: 'core.dataset.import.Chunk Split Tip'
   },
   [TrainingModeEnum.qa]: {
-    label: 'core.dataset.training.type qa'
+    label: 'core.dataset.training.QA mode',
+    tooltip: 'core.dataset.import.QA Import Tip'
   }
 };
 
@@ -168,4 +170,8 @@ export const SearchScoreTypeMap = {
   }
 };
 
-export const FolderAvatarSrc = '/imgs/files/folder.svg';
+export const FolderIcon = 'file/fill/folder';
+export const FolderImgUrl = '/imgs/files/folder.svg';
+
+export const CustomCollectionIcon = 'common/linkBlue';
+export const LinkCollectionIcon = 'common/linkBlue';

packages/global/core/dataset/type.d.ts

@@ -8,7 +8,7 @@ import {
   DatasetTypeEnum,
   SearchScoreTypeEnum,
   TrainingModeEnum
-} from './constant';
+} from './constants';
 
 /* schema */
 export type DatasetSchemaType = {
@@ -55,6 +55,8 @@ export type DatasetCollectionSchemaType = {
   hashRawText?: string;
   metadata?: {
     webPageSelector?: string;
+    relatedImgId?: string; // The id of the associated image collections
+
     [key: string]: any;
   };
 };

packages/global/core/dataset/utils.ts

@@ -1,4 +1,4 @@
-import { TrainingModeEnum, DatasetCollectionTypeEnum, DatasetDataIndexTypeEnum } from './constant';
+import { TrainingModeEnum, DatasetCollectionTypeEnum, DatasetDataIndexTypeEnum } from './constants';
 import { getFileIcon } from '../../common/file/icon';
 import { strIsLink } from '../../common/string/tools';
 
@@ -7,18 +7,13 @@ export function getCollectionIcon(
   name = ''
 ) {
   if (type === DatasetCollectionTypeEnum.folder) {
-    return '/imgs/files/folder.svg';
+    return 'common/folderFill';
   }
   if (type === DatasetCollectionTypeEnum.link) {
-    return '/imgs/files/link.svg';
+    return 'common/linkBlue';
   }
   if (type === DatasetCollectionTypeEnum.virtual) {
-    if (name === '手动录入') {
-      return '/imgs/files/manual.svg';
-    } else if (name === '手动标注') {
-      return '/imgs/files/mark.svg';
-    }
-    return '/imgs/files/collection.svg';
+    return 'file/fill/manual';
   }
   return getFileIcon(name);
 }
@@ -30,19 +25,14 @@ export function getSourceNameIcon({
   sourceId?: string;
 }) {
   if (strIsLink(sourceId)) {
-    return '/imgs/files/link.svg';
+    return 'common/linkBlue';
   }
   const fileIcon = getFileIcon(sourceName, '');
   if (fileIcon) {
     return fileIcon;
   }
 
-  if (sourceName === '手动录入') {
-    return '/imgs/files/manual.svg';
-  } else if (sourceName === '手动标注') {
-    return '/imgs/files/mark.svg';
-  }
-  return '/imgs/files/collection.svg';
+  return 'file/fill/manual';
 }
 
 export function getDefaultIndex(props?: { q?: string; a?: string; dataId?: string }) {

packages/global/core/module/constants.ts

@@ -113,5 +113,16 @@ export enum VariableInputEnum {
   textarea = 'textarea',
   select = 'select'
 }
+export const variableMap = {
+  [VariableInputEnum.input]: {
+    icon: 'core/app/variable/input'
+  },
+  [VariableInputEnum.textarea]: {
+    icon: 'core/app/variable/textarea'
+  },
+  [VariableInputEnum.select]: {
+    icon: 'core/app/variable/select'
+  }
+};
 
 export const DYNAMIC_INPUT_KEY = 'DYNAMIC_INPUT_KEY';

packages/global/core/module/node/constant.ts

@@ -54,10 +54,9 @@ export enum FlowNodeTypeEnum {
   pluginModule = 'pluginModule',
   pluginInput = 'pluginInput',
   pluginOutput = 'pluginOutput',
-  cfr = 'cfr',
+  cfr = 'cfr'
 
   // abandon
-  variable = 'variable'
 }
 
 export const EDGE_TYPE = 'default';

packages/global/core/module/template/system/aiChat.ts

@@ -23,15 +23,15 @@ export const AiChatModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.textAnswer,
   flowType: FlowNodeTypeEnum.chatNode,
   avatar: '/imgs/module/AI.png',
-  name: 'AI 对话',
-  intro: 'AI 大模型对话',
+  name: 'core.module.template.Ai chat',
+  intro: 'core.module.template.Ai chat intro',
   showStatus: true,
   inputs: [
     Input_Template_Switch,
     {
       key: ModuleInputKeyEnum.aiModel,
       type: FlowNodeInputTypeEnum.selectChatModel,
-      label: '对话模型',
+      label: 'core.module.input.label.aiModel',
       required: true,
       valueType: ModuleIOValueTypeEnum.string,
       showTargetInApp: false,
@@ -41,42 +41,31 @@ export const AiChatModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.aiChatTemperature,
       type: FlowNodeInputTypeEnum.hidden, // Set in the pop-up window
-      label: '温度',
+      label: '',
       value: 0,
       valueType: ModuleIOValueTypeEnum.number,
       min: 0,
       max: 10,
       step: 1,
-      markList: [
-        { label: '严谨', value: 0 },
-        { label: '发散', value: 10 }
-      ],
       showTargetInApp: false,
       showTargetInPlugin: false
     },
     {
       key: ModuleInputKeyEnum.aiChatMaxToken,
       type: FlowNodeInputTypeEnum.hidden, // Set in the pop-up window
-      label: '回复上限',
+      label: '',
       value: 2000,
       valueType: ModuleIOValueTypeEnum.number,
       min: 100,
       max: 4000,
       step: 50,
-      markList: [
-        { label: '100', value: 100 },
-        {
-          label: `${4000}`,
-          value: 4000
-        }
-      ],
       showTargetInApp: false,
       showTargetInPlugin: false
     },
     {
       key: ModuleInputKeyEnum.aiChatIsResponseText,
       type: FlowNodeInputTypeEnum.hidden,
-      label: '返回AI内容',
+      label: '',
       value: true,
       valueType: ModuleIOValueTypeEnum.boolean,
       showTargetInApp: false,
@@ -85,7 +74,7 @@ export const AiChatModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.aiChatQuoteTemplate,
       type: FlowNodeInputTypeEnum.hidden,
-      label: '引用内容模板',
+      label: '',
       valueType: ModuleIOValueTypeEnum.string,
       showTargetInApp: false,
       showTargetInPlugin: false
@@ -93,7 +82,7 @@ export const AiChatModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.aiChatQuotePrompt,
       type: FlowNodeInputTypeEnum.hidden,
-      label: '引用内容提示词',
+      label: '',
       valueType: ModuleIOValueTypeEnum.string,
       showTargetInApp: false,
       showTargetInPlugin: false
@@ -110,7 +99,7 @@ export const AiChatModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.aiSystemPrompt,
       type: FlowNodeInputTypeEnum.textarea,
-      label: '系统提示词',
+      label: 'core.ai.Prompt',
       max: 300,
       valueType: ModuleIOValueTypeEnum.string,
       description: chatNodeSystemPromptTip,
@@ -122,8 +111,8 @@ export const AiChatModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.aiChatDatasetQuote,
       type: FlowNodeInputTypeEnum.target,
-      label: '引用内容',
-      description: "对象数组格式，结构：\n [{q:'问题',a:'回答'}]",
+      label: 'core.module.input.label.Quote',
+      description: 'core.module.input.description.Quote',
       valueType: ModuleIOValueTypeEnum.datasetQuote,
       showTargetInApp: true,
       showTargetInPlugin: true
@@ -134,16 +123,16 @@ export const AiChatModule: FlowModuleTemplateType = {
     Output_Template_UserChatInput,
     {
       key: ModuleOutputKeyEnum.history,
-      label: '新的上下文',
-      description: '将本次回复内容拼接上历史记录，作为新的上下文返回',
+      label: 'core.module.output.label.New context',
+      description: 'core.module.output.description.New context',
       valueType: ModuleIOValueTypeEnum.chatHistory,
       type: FlowNodeOutputTypeEnum.source,
       targets: []
     },
     {
       key: ModuleOutputKeyEnum.answerText,
-      label: 'AI回复内容',
-      description: '将在 stream 回复完毕后触发',
+      label: 'core.module.output.label.Ai response content',
+      description: 'core.module.output.description.Ai response content',
       valueType: ModuleIOValueTypeEnum.string,
       type: FlowNodeOutputTypeEnum.source,
       targets: []

packages/global/core/module/template/system/assignedAnswer.ts

@@ -9,19 +9,17 @@ export const AssignedAnswerModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.textAnswer,
   flowType: FlowNodeTypeEnum.answerNode,
   avatar: '/imgs/module/reply.png',
-  name: '指定回复',
-  intro: '该模块可以直接回复一段指定的内容。常用于引导、提示',
+  name: 'core.module.template.Assigned reply',
+  intro: 'core.module.template.Assigned reply intro',
   inputs: [
     Input_Template_Switch,
     {
       key: ModuleInputKeyEnum.answerText,
       type: FlowNodeInputTypeEnum.textarea,
       valueType: ModuleIOValueTypeEnum.any,
-      label: '回复的内容',
-      description:
-        '可以使用 \\n 来实现连续换行。\n可以通过外部模块输入实现回复，外部模块输入时会覆盖当前填写的内容。\n如传入非字符串类型数据将会自动转成字符串',
-      placeholder:
-        '可以使用 \\n 来实现连续换行。\n可以通过外部模块输入实现回复，外部模块输入时会覆盖当前填写的内容。\n如传入非字符串类型数据将会自动转成字符串',
+      label: 'core.module.input.label.Response content',
+      description: 'core.module.input.description.Response content',
+      placeholder: 'core.module.input.description.Response content',
       showTargetInApp: true,
       showTargetInPlugin: true
     }

packages/global/core/module/template/system/classifyQuestion.ts

@@ -17,12 +17,8 @@ export const ClassifyQuestionModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.functionCall,
   flowType: FlowNodeTypeEnum.classifyQuestion,
   avatar: '/imgs/module/cq.png',
-  name: '问题分类',
-  intro: `根据用户的历史记录和当前问题判断该次提问的类型。可以添加多组问题类型，下面是一个模板例子：
-类型1: 打招呼
-类型2: 关于商品“使用”问题
-类型3: 关于商品“购买”问题
-类型4: 其他问题`,
+  name: 'core.module.template.Classify question',
+  intro: `core.module.template.Classify question intro`,
   showStatus: true,
   inputs: [
     Input_Template_Switch,
@@ -30,7 +26,7 @@ export const ClassifyQuestionModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.aiModel,
       type: FlowNodeInputTypeEnum.selectCQModel,
       valueType: ModuleIOValueTypeEnum.string,
-      label: '分类模型',
+      label: 'core.module.input.label.Classify model',
       required: true,
       showTargetInApp: false,
       showTargetInPlugin: false
@@ -39,11 +35,9 @@ export const ClassifyQuestionModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.aiSystemPrompt,
       type: FlowNodeInputTypeEnum.textarea,
       valueType: ModuleIOValueTypeEnum.string,
-      label: '背景知识',
-      description:
-        '你可以添加一些特定内容的介绍，从而更好的识别用户的问题类型。这个内容通常是给模型介绍一个它不知道的内容。',
-      placeholder:
-        '例如: \n1. AIGC（人工智能生成内容）是指使用人工智能技术自动或半自动地生成数字内容，如文本、图像、音乐、视频等。\n2. AIGC技术包括但不限于自然语言处理、计算机视觉、机器学习和深度学习。这些技术可以创建新内容或修改现有内容，以满足特定的创意、教育、娱乐或信息需求。',
+      label: 'core.module.input.label.Background',
+      description: 'core.module.input.description.Background',
+      placeholder: 'core.module.input.placeholder.Classify background',
       showTargetInApp: true,
       showTargetInPlugin: true
     },

packages/global/core/module/template/system/contextExtract.ts

@@ -17,8 +17,8 @@ export const ContextExtractModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.functionCall,
   flowType: FlowNodeTypeEnum.contentExtract,
   avatar: '/imgs/module/extract.png',
-  name: '文本内容提取',
-  intro: '可从文本中提取指定的数据，例如：sql语句、搜索关键词、代码等',
+  name: 'core.module.template.Extract field',
+  intro: 'core.module.template.Extract field intro',
   showStatus: true,
   inputs: [
     Input_Template_Switch,
@@ -26,7 +26,7 @@ export const ContextExtractModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.aiModel,
       type: FlowNodeInputTypeEnum.selectExtractModel,
       valueType: ModuleIOValueTypeEnum.string,
-      label: '提取模型',
+      label: 'core.module.input.label.LLM',
       required: true,
       showTargetInApp: false,
       showTargetInPlugin: false

packages/global/core/module/template/system/datasetSearch.ts

@@ -12,22 +12,22 @@ import {
 } from '../../constants';
 import { Input_Template_Switch, Input_Template_UserChatInput } from '../input';
 import { Output_Template_Finish, Output_Template_UserChatInput } from '../output';
-import { DatasetSearchModeEnum } from '../../../dataset/constant';
+import { DatasetSearchModeEnum } from '../../../dataset/constants';
 
 export const DatasetSearchModule: FlowModuleTemplateType = {
   id: FlowNodeTypeEnum.datasetSearchNode,
   templateType: ModuleTemplateTypeEnum.functionCall,
   flowType: FlowNodeTypeEnum.datasetSearchNode,
   avatar: '/imgs/module/db.png',
-  name: '知识库搜索',
-  intro: '去知识库中搜索对应的答案。可作为 AI 对话引用参考。',
+  name: 'core.module.template.Dataset search',
+  intro: 'core.module.template.Dataset search intro',
   showStatus: true,
   inputs: [
     Input_Template_Switch,
     {
       key: ModuleInputKeyEnum.datasetSelectList,
       type: FlowNodeInputTypeEnum.selectDataset,
-      label: '关联的知识库',
+      label: 'core.module.input.label.Select dataset',
       value: [],
       valueType: ModuleIOValueTypeEnum.selectDataset,
       list: [],
@@ -38,7 +38,7 @@ export const DatasetSearchModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.datasetSimilarity,
       type: FlowNodeInputTypeEnum.hidden,
-      label: '最低相关性',
+      label: '',
       value: 0.4,
       valueType: ModuleIOValueTypeEnum.number,
       min: 0,
@@ -54,8 +54,7 @@ export const DatasetSearchModule: FlowModuleTemplateType = {
     {
       key: ModuleInputKeyEnum.datasetLimit,
       type: FlowNodeInputTypeEnum.hidden,
-      label: '引用上限',
-      description: '单次搜索最大的 Tokens 数量，中文约1字=1.7Tokens，英文约1字=1Tokens',
+      label: '',
       value: 1500,
       valueType: ModuleIOValueTypeEnum.number,
       showTargetInApp: false,
@@ -93,23 +92,22 @@ export const DatasetSearchModule: FlowModuleTemplateType = {
     Output_Template_UserChatInput,
     {
       key: ModuleOutputKeyEnum.datasetIsEmpty,
-      label: '搜索结果为空',
+      label: 'core.module.output.label.Search result empty',
       type: FlowNodeOutputTypeEnum.source,
       valueType: ModuleIOValueTypeEnum.boolean,
       targets: []
     },
     {
       key: ModuleOutputKeyEnum.datasetUnEmpty,
-      label: '搜索结果不为空',
+      label: 'core.module.output.label.Search result not empty',
       type: FlowNodeOutputTypeEnum.source,
       valueType: ModuleIOValueTypeEnum.boolean,
       targets: []
     },
     {
       key: ModuleOutputKeyEnum.datasetQuoteQA,
-      label: '引用内容',
-      description:
-        '始终返回数组，如果希望搜索结果为空时执行额外操作，需要用到上面的两个输入以及目标模块的触发器',
+      label: 'core.module.output.label.Quote',
+      description: 'core.module.output.label.Quote intro',
       type: FlowNodeOutputTypeEnum.source,
       valueType: ModuleIOValueTypeEnum.datasetQuote,
       targets: []

packages/global/core/module/template/system/http.ts

@@ -17,8 +17,8 @@ export const HttpModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.externalCall,
   flowType: FlowNodeTypeEnum.httpRequest,
   avatar: '/imgs/module/http.png',
-  name: 'HTTP模块',
-  intro: '可以发出一个 HTTP POST 请求，实现更为复杂的操作（联网搜索、数据库查询等）',
+  name: 'core.module.template.Http request',
+  intro: 'core.module.template.Http request intro',
   showStatus: true,
   inputs: [
     Input_Template_Switch,

packages/global/core/module/template/system/runApp.ts

@@ -22,8 +22,8 @@ export const RunAppModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.externalCall,
   flowType: FlowNodeTypeEnum.runApp,
   avatar: '/imgs/module/app.png',
-  name: '应用调用',
-  intro: '可以选择一个其他应用进行调用',
+  name: 'core.module.template.Running app',
+  intro: 'core.module.template.Running app intro',
   showStatus: true,
   inputs: [
     Input_Template_Switch,

packages/global/core/module/template/system/runPlugin.ts

@@ -8,7 +8,7 @@ export const RunPluginModule: FlowModuleTemplateType = {
   flowType: FlowNodeTypeEnum.pluginModule,
   avatar: '/imgs/module/custom.png',
   intro: '',
-  name: '自定义模块',
+  name: '',
   showStatus: false,
   inputs: [], // [{key:'pluginId'},...]
   outputs: []

packages/global/core/module/template/system/userGuide.ts

@@ -8,14 +8,14 @@ export const UserGuideModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.userGuide,
   flowType: FlowNodeTypeEnum.userGuide,
   avatar: '/imgs/module/userGuide.png',
-  name: '用户引导',
+  name: 'core.module.template.User guide',
   intro: userGuideTip,
   inputs: [
     {
       key: ModuleInputKeyEnum.welcomeText,
       type: FlowNodeInputTypeEnum.hidden,
       valueType: ModuleIOValueTypeEnum.string,
-      label: '开场白',
+      label: 'core.app.Welcome Text',
       showTargetInApp: false,
       showTargetInPlugin: false
     },
@@ -23,7 +23,7 @@ export const UserGuideModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.variables,
       type: FlowNodeInputTypeEnum.hidden,
       valueType: ModuleIOValueTypeEnum.any,
-      label: '对话框变量',
+      label: 'core.module.Variable',
       value: [],
       showTargetInApp: false,
       showTargetInPlugin: false
@@ -32,7 +32,7 @@ export const UserGuideModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.questionGuide,
       valueType: ModuleIOValueTypeEnum.boolean,
       type: FlowNodeInputTypeEnum.switch,
-      label: '问题引导',
+      label: '',
       showTargetInApp: false,
       showTargetInPlugin: false
     },
@@ -40,7 +40,7 @@ export const UserGuideModule: FlowModuleTemplateType = {
       key: ModuleInputKeyEnum.tts,
       type: FlowNodeInputTypeEnum.hidden,
       valueType: ModuleIOValueTypeEnum.any,
-      label: '语音播报',
+      label: '',
       showTargetInApp: false,
       showTargetInPlugin: false
     }

packages/global/core/module/template/system/userInput.ts

@@ -16,14 +16,14 @@ export const UserInputModule: FlowModuleTemplateType = {
   templateType: ModuleTemplateTypeEnum.systemInput,
   flowType: FlowNodeTypeEnum.questionInput,
   avatar: '/imgs/module/userChatInput.png',
-  name: '用户问题(入口)',
-  intro: '用户输入的内容。该模块通常作为应用的入口，用户在发送消息后会首先执行该模块。',
+  name: 'core.module.template.Chat entrance',
+  intro: 'core.module.template.Chat entrance intro',
   inputs: [
     {
       key: ModuleInputKeyEnum.userChatInput,
       type: FlowNodeInputTypeEnum.systemInput,
       valueType: ModuleIOValueTypeEnum.string,
-      label: '用户问题',
+      label: 'core.module.input.label.user question',
       showTargetInApp: false,
       showTargetInPlugin: false
     }
@@ -31,7 +31,7 @@ export const UserInputModule: FlowModuleTemplateType = {
   outputs: [
     {
       key: ModuleOutputKeyEnum.userChatInput,
-      label: '用户问题',
+      label: 'core.module.input.label.user question',
       type: FlowNodeOutputTypeEnum.source,
       valueType: ModuleIOValueTypeEnum.string,
       targets: []

packages/global/core/module/template/tip.ts

@@ -1,7 +1,4 @@
-export const chatNodeSystemPromptTip =
-  '模型固定的引导词，通过调整该内容，可以引导模型聊天方向。该内容会被固定在上下文的开头。可使用变量，例如 {{language}}';
-export const userGuideTip = '可以在对话前设置引导语，设置全局变量，设置下一步指引';
-export const welcomeTextTip =
-  '每次对话开始前，发送一个初始内容。支持标准 Markdown 语法，可使用的额外标记:\n[快捷按键]: 用户点击后可以直接发送该问题';
-export const variableTip =
-  '可以在对话开始前，要求用户填写一些内容作为本轮对话的特定变量。该模块位于开场引导之后。\n变量可以通过 {{变量key}} 的形式注入到其他模块 string 类型的输入中，例如：提示词、限定词等';
+export const chatNodeSystemPromptTip = 'core.app.tip.chatNodeSystemPromptTip';
+export const userGuideTip = 'core.app.tip.userGuideTip';
+export const welcomeTextTip = 'core.app.tip.welcomeTextTip';
+export const variableTip = 'core.app.tip.variableTip';

packages/global/core/module/utils.ts

@@ -1,5 +1,5 @@
 import { FlowNodeInputTypeEnum, FlowNodeTypeEnum } from './node/constant';
-import { ModuleIOValueTypeEnum, ModuleInputKeyEnum } from './constants';
+import { ModuleIOValueTypeEnum, ModuleInputKeyEnum, variableMap } from './constants';
 import { FlowNodeInputItemType, FlowNodeOutputItemType } from './node/type';
 import { AppTTSConfigType, ModuleItemType, VariableItemType } from './type';
 import { Input_Template_Switch } from './template/input';
@@ -94,3 +94,12 @@ export function plugin2ModuleIO(
       : []
   };
 }
+
+export const formatVariablesIcon = (
+  variables: VariableItemType[]
+): (VariableItemType & { icon: string })[] => {
+  return variables.map((item) => ({
+    ...item,
+    icon: variableMap[item.type]?.icon
+  }));
+};

packages/global/support/user/api.d.ts

@@ -3,7 +3,6 @@ import { OAuthEnum } from './constant';
 export type PostLoginProps = {
   username: string;
   password: string;
-  tmbId?: string;
 };
 
 export type OauthLoginProps = {

packages/global/support/user/type.d.ts

@@ -13,6 +13,7 @@ export type UserModelSchema = {
   createTime: number;
   timezone: string;
   status: `${UserStatusEnum}`;
+  lastLoginTmbId?: string;
   openaiAccount?: {
     key: string;
     baseUrl: string;

packages/global/support/wallet/bill/api.d.ts

@@ -3,8 +3,7 @@ import { BillListItemCountType, BillListItemType } from './type';
 
 export type CreateTrainingBillProps = {
   name: string;
-  vectorModel?: string;
-  agentModel?: string;
+  datasetId: string;
 };
 
 export type ConcatBillProps = BillListItemCountType & {

packages/global/support/wallet/bill/type.d.ts

@@ -4,9 +4,8 @@ import { BillSourceEnum } from './constants';
 export type BillListItemCountType = {
   inputTokens?: number;
   outputTokens?: number;
-  textLen?: number;
+  charsLength?: number;
   duration?: number;
-  dataLen?: number;
 
   // abandon
   tokenLen?: number;

packages/service/common/file/gridfs/controller.ts

@@ -3,9 +3,10 @@ import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
 import fsp from 'fs/promises';
 import fs from 'fs';
 import { DatasetFileSchema } from '@fastgpt/global/core/dataset/type';
-import { delImgByFileIdList } from '../image/controller';
+import { MongoFileSchema } from './schema';
 
 export function getGFSCollection(bucket: `${BucketNameEnum}`) {
+  MongoFileSchema;
   return connectionMongo.connection.db.collection(`${bucket}.files`);
 }
 export function getGridBucket(bucket: `${BucketNameEnum}`) {
@@ -21,6 +22,7 @@ export async function uploadFile({
   tmbId,
   path,
   filename,
+  contentType,
   metadata = {}
 }: {
   bucketName: `${BucketNameEnum}`;
@@ -28,6 +30,7 @@ export async function uploadFile({
   tmbId: string;
   path: string;
   filename: string;
+  contentType?: string;
   metadata?: Record<string, any>;
 }) {
   if (!path) return Promise.reject(`filePath is empty`);
@@ -44,7 +47,7 @@ export async function uploadFile({
 
   const stream = bucket.openUploadStream(filename, {
     metadata,
-    contentType: metadata?.contentType
+    contentType
   });
 
   // save to gridfs
@@ -96,40 +99,6 @@ export async function delFileByFileIdList({
     }
   }
 }
-// delete file by metadata(datasetId)
-export async function delFileByMetadata({
-  bucketName,
-  datasetId
-}: {
-  bucketName: `${BucketNameEnum}`;
-  datasetId?: string;
-}) {
-  const bucket = getGridBucket(bucketName);
-
-  const files = await bucket
-    .find(
-      {
-        ...(datasetId && { 'metadata.datasetId': datasetId })
-      },
-      {
-        projection: {
-          _id: 1
-        }
-      }
-    )
-    .toArray();
-
-  const idList = files.map((item) => String(item._id));
-
-  // delete img
-  await delImgByFileIdList(idList);
-
-  // delete file
-  await delFileByFileIdList({
-    bucketName,
-    fileIdList: idList
-  });
-}
 
 export async function getDownloadStream({
   bucketName,

packages/service/common/file/gridfs/schema.ts

@@ -0,0 +1,15 @@
+import { connectionMongo, type Model } from '../../mongo';
+const { Schema, model, models } = connectionMongo;
+
+const FileSchema = new Schema({});
+
+try {
+  FileSchema.index({ 'metadata.teamId': 1 });
+  FileSchema.index({ 'metadata.uploadDate': -1 });
+} catch (error) {
+  console.log(error);
+}
+
+export const MongoFileSchema = models['dataset.files'] || model('dataset.files', FileSchema);
+
+MongoFileSchema.syncIndexes();

packages/service/common/file/image/controller.ts

@@ -46,8 +46,8 @@ export async function readMongoImg({ id }: { id: string }) {
   return data?.binary;
 }
 
-export async function delImgByFileIdList(fileIds: string[]) {
+export async function delImgByRelatedId(relateIds: string[]) {
   return MongoImage.deleteMany({
-    'metadata.fileId': { $in: fileIds.map((item) => String(item)) }
+    'metadata.relatedId': { $in: relateIds.map((id) => String(id)) }
   });
 }

packages/service/common/file/image/schema.ts

@@ -35,6 +35,8 @@ try {
   ImageSchema.index({ expiredTime: 1 }, { expireAfterSeconds: 60 });
   ImageSchema.index({ type: 1 });
   ImageSchema.index({ teamId: 1 });
+  ImageSchema.index({ createTime: 1 });
+  ImageSchema.index({ 'metadata.relatedId': 1 });
 } catch (error) {
   console.log(error);
 }

packages/service/common/file/load/pdf.ts

@@ -1,68 +0,0 @@
-import * as pdfjs from 'pdfjs-dist/legacy/build/pdf.mjs';
-// @ts-ignore
-import('pdfjs-dist/legacy/build/pdf.worker.min.mjs');
-import { ReadFileParams } from './type';
-
-type TokenType = {
-  str: string;
-  dir: string;
-  width: number;
-  height: number;
-  transform: number[];
-  fontName: string;
-  hasEOL: boolean;
-};
-
-export const readPdfFile = async ({ path }: ReadFileParams) => {
-  const readPDFPage = async (doc: any, pageNo: number) => {
-    const page = await doc.getPage(pageNo);
-    const tokenizedText = await page.getTextContent();
-
-    const viewport = page.getViewport({ scale: 1 });
-    const pageHeight = viewport.height;
-    const headerThreshold = pageHeight * 0.95;
-    const footerThreshold = pageHeight * 0.05;
-
-    const pageTexts: TokenType[] = tokenizedText.items.filter((token: TokenType) => {
-      return (
-        !token.transform ||
-        (token.transform[5] < headerThreshold && token.transform[5] > footerThreshold)
-      );
-    });
-
-    // concat empty string 'hasEOL'
-    for (let i = 0; i < pageTexts.length; i++) {
-      const item = pageTexts[i];
-      if (item.str === '' && pageTexts[i - 1]) {
-        pageTexts[i - 1].hasEOL = item.hasEOL;
-        pageTexts.splice(i, 1);
-        i--;
-      }
-    }
-
-    page.cleanup();
-
-    return pageTexts
-      .map((token) => {
-        const paragraphEnd = token.hasEOL && /([。？！.?!\n\r]|(\r\n))$/.test(token.str);
-
-        return paragraphEnd ? `${token.str}\n` : token.str;
-      })
-      .join('');
-  };
-
-  const loadingTask = pdfjs.getDocument(path);
-  const doc = await loadingTask.promise;
-
-  const pageTextPromises = [];
-  for (let pageNo = 1; pageNo <= doc.numPages; pageNo++) {
-    pageTextPromises.push(readPDFPage(doc, pageNo));
-  }
-  const pageTexts = await Promise.all(pageTextPromises);
-
-  loadingTask.destroy();
-
-  return {
-    rawText: pageTexts.join('')
-  };
-};

packages/service/common/file/load/type.d.ts

@@ -1,18 +0,0 @@
-export type ReadFileParams = {
-  preview: boolean;
-  teamId: string;
-  path: string;
-  metadata?: Record<string, any>;
-};
-
-export type ReadFileResponse = {
-  rawText: string;
-};
-
-export type ReadFileBufferItemType = ReadFileParams & {
-  rawText: string;
-};
-
-declare global {
-  var readFileBuffers: ReadFileBufferItemType[];
-}

packages/service/common/file/load/utils.ts

@@ -1,50 +0,0 @@
-import { readPdfFile } from './pdf';
-import { readDocFle } from './word';
-import { ReadFileBufferItemType, ReadFileParams } from './type';
-
-global.readFileBuffers = global.readFileBuffers || [];
-
-const bufferMaxSize = 200;
-
-export const pushFileReadBuffer = (params: ReadFileBufferItemType) => {
-  global.readFileBuffers.push(params);
-
-  if (global.readFileBuffers.length > bufferMaxSize) {
-    global.readFileBuffers.shift();
-  }
-};
-export const getReadFileBuffer = ({ path, teamId }: ReadFileParams) =>
-  global.readFileBuffers.find((item) => item.path === path && item.teamId === teamId);
-
-export const readFileContent = async (params: ReadFileParams) => {
-  const { path } = params;
-
-  const buffer = getReadFileBuffer(params);
-
-  if (buffer) {
-    return buffer;
-  }
-
-  const extension = path?.split('.')?.pop()?.toLowerCase() || '';
-
-  const { rawText } = await (async () => {
-    switch (extension) {
-      case 'pdf':
-        return readPdfFile(params);
-      case 'docx':
-        return readDocFle(params);
-      default:
-        return Promise.reject('Only support .pdf, .docx');
-    }
-  })();
-
-  pushFileReadBuffer({
-    ...params,
-    rawText
-  });
-
-  return {
-    ...params,
-    rawText
-  };
-};

packages/service/common/file/load/word.ts

@@ -1,22 +0,0 @@
-import mammoth from 'mammoth';
-import { htmlToMarkdown } from '../../string/markdown';
-import { ReadFileParams } from './type';
-/**
- * read docx to markdown
- */
-export const readDocFle = async ({ path, metadata = {} }: ReadFileParams) => {
-  try {
-    const { value: html } = await mammoth.convertToHtml({
-      path
-    });
-
-    const md = await htmlToMarkdown(html);
-
-    return {
-      rawText: md
-    };
-  } catch (error) {
-    console.log('error doc read:', error);
-    return Promise.reject('Can not read doc file, please convert to PDF');
-  }
-};

packages/service/common/file/multer.ts

@@ -3,7 +3,6 @@ import multer from 'multer';
 import path from 'path';
 import { BucketNameEnum, bucketNameMap } from '@fastgpt/global/common/file/constants';
 import { getNanoid } from '@fastgpt/global/common/string/tools';
-import { tmpFileDirPath } from './constants';
 
 type FileType = {
   fieldname: string;
@@ -15,8 +14,6 @@ type FileType = {
   size: number;
 };
 
-const expiredTime = 30 * 60 * 1000;
-
 export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
   maxSize *= 1024 * 1024;
   class UploadModel {
@@ -31,15 +28,16 @@ export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
         // },
         filename: async (req, file, cb) => {
           const { ext } = path.parse(decodeURIComponent(file.originalname));
-          cb(null, `${Date.now() + expiredTime}-${getNanoid(32)}${ext}`);
+          cb(null, `${getNanoid(32)}${ext}`);
         }
       })
-    }).any();
+    }).single('file');
 
     async doUpload<T = Record<string, any>>(req: NextApiRequest, res: NextApiResponse) {
       return new Promise<{
-        files: FileType[];
-        metadata: T;
+        file: FileType;
+        metadata: Record<string, any>;
+        data: T;
         bucketName?: `${BucketNameEnum}`;
       }>((resolve, reject) => {
         // @ts-ignore
@@ -54,20 +52,28 @@ export const getUploadModel = ({ maxSize = 500 }: { maxSize?: number }) => {
             return reject('BucketName is invalid');
           }
 
-          resolve({
-            ...req.body,
-            files:
           // @ts-ignore
-              req.files?.map((file) => ({
+          const file = req.file as FileType;
+
+          resolve({
+            file: {
               ...file,
               originalname: decodeURIComponent(file.originalname)
-              })) || [],
+            },
+            bucketName,
             metadata: (() => {
               if (!req.body?.metadata) return {};
               try {
                 return JSON.parse(req.body.metadata);
               } catch (error) {
-                console.log(error);
+                return {};
+              }
+            })(),
+            data: (() => {
+              if (!req.body?.data) return {};
+              try {
+                return JSON.parse(req.body.data);
+              } catch (error) {
                 return {};
               }
             })()

packages/service/common/file/utils.ts

@@ -1,5 +1,4 @@
 import fs from 'fs';
-import { tmpFileDirPath } from './constants';
 
 export const removeFilesByPaths = (paths: string[]) => {
   paths.forEach((path) => {
@@ -10,24 +9,3 @@ export const removeFilesByPaths = (paths: string[]) => {
     });
   });
 };
-
-/* cron job.  check expired tmp files */
-export const checkExpiredTmpFiles = () => {
-  // get all file name
-  const files = fs.readdirSync(tmpFileDirPath).map((name) => {
-    const timestampStr = name.split('-')[0];
-    const expiredTimestamp = timestampStr ? Number(timestampStr) : 0;
-
-    return {
-      filename: name,
-      expiredTimestamp,
-      path: `${tmpFileDirPath}/${name}`
-    };
-  });
-
-  // count expiredFiles
-  const expiredFiles = files.filter((item) => item.expiredTimestamp < Date.now());
-
-  // remove expiredFiles
-  removeFilesByPaths(expiredFiles.map((item) => item.path));
-};

packages/service/common/string/cheerio.ts

@@ -64,8 +64,7 @@ export const urlsFetch = async ({
 }: UrlFetchParams): Promise<UrlFetchResponse> => {
   urlList = urlList.filter((url) => /^(http|https):\/\/[^ "]+$/.test(url));
 
-  const response = (
-    await Promise.all(
+  const response = await Promise.all(
     urlList.map(async (url) => {
       try {
         const fetchRes = await axios.get(url, {
@@ -97,8 +96,7 @@ export const urlsFetch = async ({
         };
       }
     })
-    )
-  ).filter((item) => item.content);
+  );
 
   return response;
 };

packages/service/common/vectorStore/controller.d.ts

@@ -1,21 +1,19 @@
 export type DeleteDatasetVectorProps = {
+  teamId: string;
+
   id?: string;
   datasetIds?: string[];
   collectionIds?: string[];
-
-  collectionId?: string;
-  dataIds?: string[];
+  idList?: string[];
 };
 
 export type InsertVectorProps = {
   teamId: string;
-  tmbId: string;
   datasetId: string;
   collectionId: string;
-  dataId: string;
 };
 
 export type EmbeddingRecallProps = {
-  similarity?: number;
   datasetIds: string[];
+  similarity?: number;
 };

packages/service/common/vectorStore/controller.ts

@@ -10,6 +10,7 @@ const getVectorObj = () => {
 export const initVectorStore = getVectorObj().init;
 export const deleteDatasetDataVector = getVectorObj().delete;
 export const recallFromVectorStore = getVectorObj().recall;
+export const checkVectorDataExist = getVectorObj().checkDataExist;
 export const getVectorDataByTime = getVectorObj().getVectorDataByTime;
 export const getVectorCountByTeamId = getVectorObj().getVectorCountByTeamId;
 
@@ -21,7 +22,7 @@ export const insertDatasetDataVector = async ({
   query: string;
   model: string;
 }) => {
-  const { vectors, tokens } = await getVectorsByText({
+  const { vectors, charsLength } = await getVectorsByText({
     model,
     input: query
   });
@@ -31,32 +32,27 @@ export const insertDatasetDataVector = async ({
   });
 
   return {
-    tokens,
+    charsLength,
     insertId
   };
 };
 
 export const updateDatasetDataVector = async ({
   id,
-  query,
-  model
-}: {
+  ...props
+}: InsertVectorProps & {
   id: string;
   query: string;
   model: string;
 }) => {
-  // get vector
-  const { vectors, tokens } = await getVectorsByText({
-    model,
-    input: query
+  // insert new vector
+  const { charsLength, insertId } = await insertDatasetDataVector(props);
+
+  // delete old vector
+  await deleteDatasetDataVector({
+    teamId: props.teamId,
+    id
   });
 
-  await getVectorObj().update({
-    id,
-    vectors
-  });
-
-  return {
-    tokens
-  };
+  return { charsLength, insertId };
 };

packages/service/common/vectorStore/pg/class.ts

@@ -1,20 +1,20 @@
 import {
   initPg,
   insertDatasetDataVector,
-  updateDatasetDataVector,
   deleteDatasetDataVector,
   embeddingRecall,
   getVectorDataByTime,
-  getVectorCountByTeamId
+  getVectorCountByTeamId,
+  checkDataExist
 } from './controller';
 
 export class PgVector {
   constructor() {}
   init = initPg;
   insert = insertDatasetDataVector;
-  update = updateDatasetDataVector;
   delete = deleteDatasetDataVector;
   recall = embeddingRecall;
+  checkDataExist = checkDataExist;
   getVectorCountByTeamId = getVectorCountByTeamId;
   getVectorDataByTime = getVectorDataByTime;
 }

packages/service/common/vectorStore/pg/controller.ts

@@ -4,7 +4,7 @@ import { delay } from '@fastgpt/global/common/system/utils';
 import { PgClient, connectPg } from './index';
 import { PgSearchRawType } from '@fastgpt/global/core/dataset/api';
 import { EmbeddingRecallItemType } from '../type';
-import { DeleteDatasetVectorProps, EmbeddingRecallProps } from '../controller.d';
+import { DeleteDatasetVectorProps, EmbeddingRecallProps, InsertVectorProps } from '../controller.d';
 import dayjs from 'dayjs';
 
 export async function initPg() {
@@ -16,11 +16,9 @@ export async function initPg() {
           id BIGSERIAL PRIMARY KEY,
           vector VECTOR(1536) NOT NULL,
           team_id VARCHAR(50) NOT NULL,
-          tmb_id VARCHAR(50) NOT NULL,
           dataset_id VARCHAR(50) NOT NULL,
           collection_id VARCHAR(50) NOT NULL,
-          data_id VARCHAR(50) NOT NULL,
-          createTime TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+          createtime TIMESTAMP DEFAULT CURRENT_TIMESTAMP
       );
     `);
 
@@ -34,26 +32,21 @@ export async function initPg() {
   }
 }
 
-export const insertDatasetDataVector = async (props: {
-  teamId: string;
-  tmbId: string;
-  datasetId: string;
-  collectionId: string;
-  dataId: string;
+export const insertDatasetDataVector = async (
+  props: InsertVectorProps & {
     vectors: number[][];
     retry?: number;
-}): Promise<{ insertId: string }> => {
-  const { dataId, teamId, tmbId, datasetId, collectionId, vectors, retry = 3 } = props;
+  }
+): Promise<{ insertId: string }> => {
+  const { teamId, datasetId, collectionId, vectors, retry = 3 } = props;
   try {
     const { rows } = await PgClient.insert(PgDatasetTableName, {
       values: [
         [
           { key: 'vector', value: `[${vectors[0]}]` },
           { key: 'team_id', value: String(teamId) },
-          { key: 'tmb_id', value: String(tmbId) },
           { key: 'dataset_id', value: datasetId },
-          { key: 'collection_id', value: collectionId },
-          { key: 'data_id', value: String(dataId) }
+          { key: 'collection_id', value: collectionId }
         ]
       ]
     });
@@ -72,48 +65,33 @@ export const insertDatasetDataVector = async (props: {
   }
 };
 
-export const updateDatasetDataVector = async (props: {
-  id: string;
-  vectors: number[][];
-  retry?: number;
-}): Promise<void> => {
-  const { id, vectors, retry = 2 } = props;
-  try {
-    // update pg
-    await PgClient.update(PgDatasetTableName, {
-      where: [['id', id]],
-      values: [{ key: 'vector', value: `[${vectors[0]}]` }]
-    });
-  } catch (error) {
-    if (retry <= 0) {
-      return Promise.reject(error);
-    }
-    await delay(500);
-    return updateDatasetDataVector({
-      ...props,
-      retry: retry - 1
-    });
-  }
-};
-
 export const deleteDatasetDataVector = async (
   props: DeleteDatasetVectorProps & {
     retry?: number;
   }
 ): Promise<any> => {
-  const { id, datasetIds, collectionIds, collectionId, dataIds, retry = 2 } = props;
+  const { teamId, id, datasetIds, collectionIds, idList, retry = 2 } = props;
+
+  const teamIdWhere = `team_id='${String(teamId)}' AND`;
 
   const where = await (() => {
-    if (id) return `id=${id}`;
-    if (datasetIds) return `dataset_id IN (${datasetIds.map((id) => `'${String(id)}'`).join(',')})`;
-    if (collectionIds) {
-      return `collection_id IN (${collectionIds.map((id) => `'${String(id)}'`).join(',')})`;
-    }
-    if (collectionId && dataIds) {
-      return `collection_id='${String(collectionId)}' and data_id IN (${dataIds
+    if (id) return `${teamIdWhere} id=${id}`;
+
+    if (datasetIds) {
+      return `${teamIdWhere} dataset_id IN (${datasetIds
         .map((id) => `'${String(id)}'`)
         .join(',')})`;
     }
+
+    if (collectionIds) {
+      return `${teamIdWhere} collection_id IN (${collectionIds
+        .map((id) => `'${String(id)}'`)
+        .join(',')})`;
+    }
+
+    if (idList) {
+      return `${teamIdWhere} id IN (${idList.map((id) => `'${String(id)}'`).join(',')})`;
+    }
     return Promise.reject('deleteDatasetData: no where');
   })();
 
@@ -142,13 +120,13 @@ export const embeddingRecall = async (
 ): Promise<{
   results: EmbeddingRecallItemType[];
 }> => {
-  const { vectors, limit, similarity = 0, datasetIds, retry = 2 } = props;
+  const { datasetIds, vectors, limit, similarity = 0, retry = 2 } = props;
 
   try {
     const results: any = await PgClient.query(
       `BEGIN;
         SET LOCAL hnsw.ef_search = ${global.systemEnv.pgHNSWEfSearch || 100};
-        select id, collection_id, data_id, (vector <#> '[${vectors[0]}]') * -1 AS score 
+        select id, collection_id, (vector <#> '[${vectors[0]}]') * -1 AS score 
           from ${PgDatasetTableName} 
           where dataset_id IN (${datasetIds.map((id) => `'${String(id)}'`).join(',')})
               AND vector <#> '[${vectors[0]}]' < -${similarity}
@@ -158,21 +136,10 @@ export const embeddingRecall = async (
 
     const rows = results?.[2]?.rows as PgSearchRawType[];
 
-    // concat same data_id
-    const filterRows: PgSearchRawType[] = [];
-    let set = new Set<string>();
-    for (const row of rows) {
-      if (!set.has(row.data_id)) {
-        filterRows.push(row);
-        set.add(row.data_id);
-      }
-    }
-
     return {
-      results: filterRows.map((item) => ({
+      results: rows.map((item) => ({
         id: item.id,
         collectionId: item.collection_id,
-        dataId: item.data_id,
         score: item.score
       }))
     };
@@ -184,7 +151,11 @@ export const embeddingRecall = async (
   }
 };
 
-// bill
+export const checkDataExist = async (id: string) => {
+  const { rows } = await PgClient.query(`SELECT id FROM ${PgDatasetTableName} WHERE id=${id};`);
+
+  return rows.length > 0;
+};
 export const getVectorCountByTeamId = async (teamId: string) => {
   const total = await PgClient.count(PgDatasetTableName, {
     where: [['team_id', String(teamId)]]
@@ -193,15 +164,20 @@ export const getVectorCountByTeamId = async (teamId: string) => {
   return total;
 };
 export const getVectorDataByTime = async (start: Date, end: Date) => {
-  const { rows } = await PgClient.query<{ id: string; data_id: string }>(`SELECT id, data_id
+  const { rows } = await PgClient.query<{
+    id: string;
+    team_id: string;
+    dataset_id: string;
+  }>(`SELECT id, team_id, dataset_id
   FROM ${PgDatasetTableName}
-  WHERE createTime BETWEEN '${dayjs(start).format('YYYY-MM-DD')}' AND '${dayjs(end).format(
-    'YYYY-MM-DD 23:59:59'
+  WHERE createtime BETWEEN '${dayjs(start).format('YYYY-MM-DD HH:mm:ss')}' AND '${dayjs(end).format(
+    'YYYY-MM-DD HH:mm:ss'
   )}';
   `);
 
   return rows.map((item) => ({
     id: item.id,
-    dataId: item.data_id
+    datasetId: item.dataset_id,
+    teamId: item.team_id
   }));
 };

packages/service/common/vectorStore/type.d.ts

@@ -7,6 +7,5 @@ declare global {
 export type EmbeddingRecallItemType = {
   id: string;
   collectionId: string;
-  dataId: string;
   score: number;
 };

packages/service/core/ai/embedding/index.ts

@@ -18,10 +18,9 @@ export async function getVectorsByText({
   }
 
   try {
-    // 获取 chatAPI
     const ai = getAIApi();
 
-    // 把输入的内容转成向量
+    // input text to vector
     const result = await ai.embeddings
       .create({
         model,
@@ -38,7 +37,7 @@ export async function getVectorsByText({
         }
 
         return {
-          tokens: res.usage.total_tokens || 0,
+          charsLength: input.length,
           vectors: await Promise.all(res.data.map((item) => unityDimensional(item.embedding)))
         };
       });
@@ -53,7 +52,9 @@ export async function getVectorsByText({
 
 function unityDimensional(vector: number[]) {
   if (vector.length > 1536) {
-    console.log(`当前向量维度为: ${vector.length}, 向量维度不能超过 1536, 已自动截取前 1536 维度`);
+    console.log(
+      `The current vector dimension is ${vector.length}, and the vector dimension cannot exceed 1536. The first 1536 dimensions are automatically captured`
+    );
     return vector.slice(0, 1536);
   }
   let resultVector = vector;

packages/service/core/chat/chatItemSchema.ts

@@ -11,6 +11,8 @@ import { appCollectionName } from '../app/schema';
 import { userCollectionName } from '../../support/user/schema';
 import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
 
+export const ChatItemCollectionName = 'chatitems';
+
 const ChatItemSchema = new Schema({
   teamId: {
     type: Schema.Types.ObjectId,
@@ -79,20 +81,23 @@ const ChatItemSchema = new Schema({
 });
 
 try {
-  ChatItemSchema.index({ teamId: 1 });
-  ChatItemSchema.index({ time: -1 });
-  ChatItemSchema.index({ appId: 1 });
-  ChatItemSchema.index({ chatId: 1 });
-  ChatItemSchema.index({ obj: 1 });
-  ChatItemSchema.index({ userGoodFeedback: 1 });
-  ChatItemSchema.index({ userBadFeedback: 1 });
-  ChatItemSchema.index({ customFeedbacks: 1 });
-  ChatItemSchema.index({ adminFeedback: 1 });
+  ChatItemSchema.index({ dataId: 1 }, { background: true });
+  /* delete by app; 
+     delete by chat id;
+     get chat list; 
+     get chat logs; 
+     close custom feedback; 
+  */
+  ChatItemSchema.index({ appId: 1, chatId: 1, dataId: 1 }, { background: true });
+  ChatItemSchema.index({ userGoodFeedback: 1 }, { background: true });
+  ChatItemSchema.index({ userBadFeedback: 1 }, { background: true });
+  ChatItemSchema.index({ customFeedbacks: 1 }, { background: true });
+  ChatItemSchema.index({ adminFeedback: 1 }, { background: true });
 } catch (error) {
   console.log(error);
 }
 
 export const MongoChatItem: Model<ChatItemType> =
-  models['chatItem'] || model('chatItem', ChatItemSchema);
+  models[ChatItemCollectionName] || model(ChatItemCollectionName, ChatItemSchema);
 
 MongoChatItem.syncIndexes();

packages/service/core/chat/chatSchema.ts

@@ -1,13 +1,12 @@
 import { connectionMongo, type Model } from '../../common/mongo';
 const { Schema, model, models } = connectionMongo;
 import { ChatSchema as ChatType } from '@fastgpt/global/core/chat/type.d';
-import { ChatRoleMap, ChatSourceMap } from '@fastgpt/global/core/chat/constants';
+import { ChatSourceMap } from '@fastgpt/global/core/chat/constants';
 import {
   TeamCollectionName,
   TeamMemberCollectionName
 } from '@fastgpt/global/support/user/team/constant';
 import { appCollectionName } from '../app/schema';
-import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
 
 export const chatCollectionName = 'chat';
 
@@ -48,7 +47,8 @@ const ChatSchema = new Schema({
     default: ''
   },
   top: {
-    type: Boolean
+    type: Boolean,
+    default: false
   },
   source: {
     type: String,
@@ -73,10 +73,16 @@ const ChatSchema = new Schema({
 });
 
 try {
-  ChatSchema.index({ appId: 1 });
-  ChatSchema.index({ tmbId: 1 });
-  ChatSchema.index({ shareId: 1 });
-  ChatSchema.index({ updateTime: -1 });
+  ChatSchema.index({ chatId: 1 }, { background: true });
+  // get user history
+  ChatSchema.index({ tmbId: 1, appId: 1, top: -1, updateTime: -1 }, { background: true });
+  // delete by appid; clear history; init chat; update chat; auth chat;
+  ChatSchema.index({ appId: 1, chatId: 1 }, { background: true });
+
+  // get chat logs;
+  ChatSchema.index({ teamId: 1, appId: 1, updateTime: -1 }, { background: true });
+  // get share chat history
+  ChatSchema.index({ shareId: 1, outLinkUid: 1 }, { background: true });
 } catch (error) {
   console.log(error);
 }

packages/service/core/chat/controller.ts

@@ -3,10 +3,12 @@ import { MongoChatItem } from './chatItemSchema';
 import { addLog } from '../../common/system/log';
 
 export async function getChatItems({
+  appId,
   chatId,
   limit = 30,
   field
 }: {
+  appId: string;
   chatId?: string;
   limit?: number;
   field: string;
@@ -15,7 +17,10 @@ export async function getChatItems({
     return { history: [] };
   }
 
-  const history = await MongoChatItem.find({ chatId }, field).sort({ _id: -1 }).limit(limit).lean();
+  const history = await MongoChatItem.find({ appId, chatId }, field)
+    .sort({ _id: -1 })
+    .limit(limit)
+    .lean();
 
   history.reverse();
 
@@ -23,10 +28,12 @@ export async function getChatItems({
 }
 
 export const addCustomFeedbacks = async ({
+  appId,
   chatId,
   chatItemId,
   feedbacks
 }: {
+  appId: string;
   chatId?: string;
   chatItemId?: string;
   feedbacks: string[];

packages/service/core/dataset/collection/controller.ts

@@ -1,6 +1,20 @@
-import { TrainingModeEnum, DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant';
+import {
+  TrainingModeEnum,
+  DatasetCollectionTypeEnum
+} from '@fastgpt/global/core/dataset/constants';
 import type { CreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api.d';
 import { MongoDatasetCollection } from './schema';
+import {
+  CollectionWithDatasetType,
+  DatasetCollectionSchemaType
+} from '@fastgpt/global/core/dataset/type';
+import { MongoDatasetTraining } from '../training/schema';
+import { delay } from '@fastgpt/global/common/system/utils';
+import { MongoDatasetData } from '../data/schema';
+import { delImgByRelatedId } from '../../../common/file/image/controller';
+import { deleteDatasetDataVector } from '../../../common/vectorStore/controller';
+import { delFileByFileIdList } from '../../../common/file/gridfs/controller';
+import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
 
 export async function createOneCollection({
   teamId,
@@ -85,20 +99,50 @@ export function createDefaultCollection({
   });
 }
 
-// check same collection
-export const getSameRawTextCollection = async ({
-  datasetId,
-  hashRawText
+/**
+ * delete collection and it related data
+ */
+export async function delCollectionAndRelatedSources({
+  collections
 }: {
-  datasetId: string;
-  hashRawText?: string;
-}) => {
-  if (!hashRawText) return undefined;
+  collections: (CollectionWithDatasetType | DatasetCollectionSchemaType)[];
+}) {
+  if (collections.length === 0) return;
 
-  const collection = await MongoDatasetCollection.findOne({
-    datasetId,
-    hashRawText
+  const teamId = collections[0].teamId;
+
+  if (!teamId) return Promise.reject('teamId is not exist');
+
+  const collectionIds = collections.map((item) => String(item._id));
+  const fileIdList = collections.map((item) => item?.fileId || '').filter(Boolean);
+  const relatedImageIds = collections
+    .map((item) => item?.metadata?.relatedImgId || '')
+    .filter(Boolean);
+
+  // delete training data
+  await MongoDatasetTraining.deleteMany({
+    teamId,
+    collectionId: { $in: collectionIds }
   });
 
-  return collection;
-};
+  await delay(2000);
+
+  // delete dataset.datas
+  await MongoDatasetData.deleteMany({ teamId, collectionId: { $in: collectionIds } });
+  // delete pg data
+  await deleteDatasetDataVector({ teamId, collectionIds });
+
+  // delete file and imgs
+  await Promise.all([
+    delImgByRelatedId(relatedImageIds),
+    delFileByFileIdList({
+      bucketName: BucketNameEnum.dataset,
+      fileIdList
+    })
+  ]);
+
+  // delete collections
+  await MongoDatasetCollection.deleteMany({
+    _id: { $in: collectionIds }
+  });
+}

packages/service/core/dataset/collection/schema.ts

@@ -1,7 +1,7 @@
 import { connectionMongo, type Model } from '../../../common/mongo';
 const { Schema, model, models } = connectionMongo;
 import { DatasetCollectionSchemaType } from '@fastgpt/global/core/dataset/type.d';
-import { TrainingTypeMap, DatasetCollectionTypeMap } from '@fastgpt/global/core/dataset/constant';
+import { TrainingTypeMap, DatasetCollectionTypeMap } from '@fastgpt/global/core/dataset/constants';
 import { DatasetCollectionName } from '../schema';
 import {
   TeamCollectionName,
@@ -91,11 +91,19 @@ const DatasetCollectionSchema = new Schema({
 });
 
 try {
-  DatasetCollectionSchema.index({ teamId: 1 });
-  DatasetCollectionSchema.index({ datasetId: 1 });
-  DatasetCollectionSchema.index({ teamId: 1, datasetId: 1, parentId: 1 });
-  DatasetCollectionSchema.index({ updateTime: -1 });
-  DatasetCollectionSchema.index({ hashRawText: -1 });
+  // auth file
+  DatasetCollectionSchema.index({ teamId: 1, fileId: 1 }, { background: true });
+
+  // list collection; deep find collections
+  DatasetCollectionSchema.index(
+    {
+      teamId: 1,
+      datasetId: 1,
+      parentId: 1,
+      updateTime: -1
+    },
+    { background: true }
+  );
 } catch (error) {
   console.log(error);
 }

packages/service/core/dataset/collection/utils.ts

@@ -4,16 +4,32 @@ import type { ParentTreePathItemType } from '@fastgpt/global/common/parentFolder
 import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
 import { MongoDatasetTraining } from '../training/schema';
 import { urlsFetch } from '../../../common/string/cheerio';
-import { DatasetCollectionTypeEnum, TrainingModeEnum } from '@fastgpt/global/core/dataset/constant';
+import {
+  DatasetCollectionTypeEnum,
+  TrainingModeEnum
+} from '@fastgpt/global/core/dataset/constants';
 import { hashStr } from '@fastgpt/global/common/string/tools';
 
 /**
  * get all collection by top collectionId
  */
-export async function findCollectionAndChild(id: string, fields = '_id parentId name metadata') {
+export async function findCollectionAndChild({
+  teamId,
+  datasetId,
+  collectionId,
+  fields = '_id parentId name metadata'
+}: {
+  teamId: string;
+  datasetId: string;
+  collectionId: string;
+  fields?: string;
+}) {
   async function find(id: string) {
     // find children
-    const children = await MongoDatasetCollection.find({ parentId: id }, fields);
+    const children = await MongoDatasetCollection.find(
+      { teamId, datasetId, parentId: id },
+      fields
+    ).lean();
 
     let collections = children;
 
@@ -25,8 +41,8 @@ export async function findCollectionAndChild(id: string, fields = '_id parentId
     return collections;
   }
   const [collection, childCollections] = await Promise.all([
-    MongoDatasetCollection.findById(id, fields),
-    find(id)
+    MongoDatasetCollection.findById(collectionId, fields),
+    find(collectionId)
   ]);
 
   if (!collection) {
@@ -107,8 +123,8 @@ export const getCollectionAndRawText = async ({
       });
 
       return {
-        title: result[0].title,
-        rawText: result[0].content
+        title: result[0]?.title,
+        rawText: result[0]?.content
       };
     }
 
@@ -121,7 +137,7 @@ export const getCollectionAndRawText = async ({
   })();
 
   const hashRawText = hashStr(rawText);
-  const isSameRawText = col.hashRawText === hashRawText;
+  const isSameRawText = rawText && col.hashRawText === hashRawText;
 
   return {
     collection: col,
@@ -161,8 +177,7 @@ export const reloadCollectionChunks = async ({
   // split data
   const { chunks } = splitText2Chunks({
     text: newRawText,
-    chunkLen: col.chunkSize || 512,
-    countTokens: false
+    chunkLen: col.chunkSize || 512
   });
 
   // insert to training queue

packages/service/core/dataset/controller.ts

@@ -1,24 +1,47 @@
-import { CollectionWithDatasetType } from '@fastgpt/global/core/dataset/type';
+import { CollectionWithDatasetType, DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
 import { MongoDatasetCollection } from './collection/schema';
 import { MongoDataset } from './schema';
+import { delCollectionAndRelatedSources } from './collection/controller';
 
 /* ============= dataset ========== */
 /* find all datasetId by top datasetId */
-export async function findDatasetIdTreeByTopDatasetId(
-  id: string,
-  result: string[] = []
-): Promise<string[]> {
-  let allChildrenIds = [...result];
+export async function findDatasetAndAllChildren({
+  teamId,
+  datasetId,
+  fields
+}: {
+  teamId: string;
+  datasetId: string;
+  fields?: string;
+}): Promise<DatasetSchemaType[]> {
+  const find = async (id: string) => {
+    const children = await MongoDataset.find(
+      {
+        teamId,
+        parentId: id
+      },
+      fields
+    ).lean();
 
-  // find children
-  const children = await MongoDataset.find({ parentId: id });
+    let datasets = children;
 
     for (const child of children) {
-    const grandChildrenIds = await findDatasetIdTreeByTopDatasetId(child._id, result);
-    allChildrenIds = allChildrenIds.concat(grandChildrenIds);
+      const grandChildrenIds = await find(child._id);
+      datasets = datasets.concat(grandChildrenIds);
     }
 
-  return [String(id), ...allChildrenIds];
+    return datasets;
+  };
+  const [dataset, childDatasets] = await Promise.all([
+    MongoDataset.findById(datasetId),
+    find(datasetId)
+  ]);
+
+  if (!dataset) {
+    return Promise.reject('Dataset not found');
+  }
+
+  return [dataset, ...childDatasets];
 }
 
 export async function getCollectionWithDataset(collectionId: string) {
@@ -30,3 +53,22 @@ export async function getCollectionWithDataset(collectionId: string) {
   }
   return data;
 }
+
+/* delete all data by datasetIds */
+export async function delDatasetRelevantData({ datasets }: { datasets: DatasetSchemaType[] }) {
+  if (!datasets.length) return;
+
+  const teamId = datasets[0].teamId;
+  const datasetIds = datasets.map((item) => String(item._id));
+
+  // Get _id, teamId, fileId, metadata.relatedImgId for all collections
+  const collections = await MongoDatasetCollection.find(
+    {
+      teamId,
+      datasetId: { $in: datasetIds }
+    },
+    '_id teamId fileId metadata'
+  ).lean();
+
+  await delCollectionAndRelatedSources({ collections });
+}

packages/service/core/dataset/data/controller.ts

@@ -1,87 +1,2 @@
 import { MongoDatasetData } from './schema';
-import { MongoDatasetTraining } from '../training/schema';
-import { delFileByFileIdList, delFileByMetadata } from '../../../common/file/gridfs/controller';
-import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
-import { MongoDatasetCollection } from '../collection/schema';
-import { delay } from '@fastgpt/global/common/system/utils';
-import { delImgByFileIdList } from '../../../common/file/image/controller';
 import { deleteDatasetDataVector } from '../../../common/vectorStore/controller';
-
-/* delete all data by datasetIds */
-export async function delDatasetRelevantData({ datasetIds }: { datasetIds: string[] }) {
-  datasetIds = datasetIds.map((item) => String(item));
-
-  // delete training data(There could be a training mission)
-  await MongoDatasetTraining.deleteMany({
-    datasetId: { $in: datasetIds }
-  });
-
-  await delay(2000);
-
-  // delete dataset.datas
-  await MongoDatasetData.deleteMany({ datasetId: { $in: datasetIds } });
-  // delete pg data
-  await deleteDatasetDataVector({ datasetIds });
-
-  // delete collections
-  await MongoDatasetCollection.deleteMany({
-    datasetId: { $in: datasetIds }
-  });
-
-  // delete related files
-  await Promise.all(
-    datasetIds.map((id) => delFileByMetadata({ bucketName: BucketNameEnum.dataset, datasetId: id }))
-  );
-}
-/**
- * delete all data by collectionIds
- */
-export async function delCollectionRelevantData({
-  collectionIds,
-  fileIds
-}: {
-  collectionIds: string[];
-  fileIds: string[];
-}) {
-  collectionIds = collectionIds.filter(Boolean).map((item) => String(item));
-  const filterFileIds = fileIds.filter(Boolean).map((item) => String(item));
-
-  // delete training data
-  await MongoDatasetTraining.deleteMany({
-    collectionId: { $in: collectionIds }
-  });
-
-  await delay(2000);
-
-  // delete dataset.datas
-  await MongoDatasetData.deleteMany({ collectionId: { $in: collectionIds } });
-  // delete pg data
-  await deleteDatasetDataVector({ collectionIds });
-
-  // delete collections
-  await MongoDatasetCollection.deleteMany({
-    _id: { $in: collectionIds }
-  });
-
-  // delete file and imgs
-  await Promise.all([
-    delImgByFileIdList(filterFileIds),
-    delFileByFileIdList({
-      bucketName: BucketNameEnum.dataset,
-      fileIdList: filterFileIds
-    })
-  ]);
-}
-/**
- * delete one data by mongoDataId
- */
-export async function delDatasetDataByDataId({
-  collectionId,
-  mongoDataId
-}: {
-  collectionId: string;
-  mongoDataId: string;
-}) {
-  await deleteDatasetDataVector({ collectionId, dataIds: [mongoDataId] });
-  await MongoDatasetData.findByIdAndDelete(mongoDataId);
-}

packages/service/core/dataset/data/schema.ts

@@ -10,7 +10,7 @@ import { DatasetColCollectionName } from '../collection/schema';
 import {
   DatasetDataIndexTypeEnum,
   DatasetDataIndexTypeMap
-} from '@fastgpt/global/core/dataset/constant';
+} from '@fastgpt/global/core/dataset/constants';
 
 export const DatasetDataCollectionName = 'dataset.datas';
 
@@ -71,6 +71,7 @@ const DatasetDataSchema = new Schema({
     ],
     default: []
   },
+
   updateTime: {
     type: Date,
     default: () => new Date()
@@ -85,13 +86,18 @@ const DatasetDataSchema = new Schema({
 });
 
 try {
-  DatasetDataSchema.index({ teamId: 1 });
-  DatasetDataSchema.index({ datasetId: 1 });
-  DatasetDataSchema.index({ collectionId: 1 });
-  DatasetDataSchema.index({ updateTime: -1 });
-  DatasetDataSchema.index({ collectionId: 1, q: 1, a: 1 });
+  // same data check
+  DatasetDataSchema.index({ teamId: 1, collectionId: 1, q: 1, a: 1 }, { background: true });
+  // list collection and count data; list data
+  DatasetDataSchema.index(
+    { teamId: 1, datasetId: 1, collectionId: 1, chunkIndex: 1, updateTime: -1 },
+    { background: true }
+  );
   // full text index
-  DatasetDataSchema.index({ datasetId: 1, fullTextToken: 'text' });
+  DatasetDataSchema.index({ teamId: 1, datasetId: 1, fullTextToken: 'text' }, { background: true });
+  // Recall vectors after data matching
+  DatasetDataSchema.index({ teamId: 1, datasetId: 1, 'indexes.dataId': 1 }, { background: true });
+  DatasetDataSchema.index({ updateTime: 1 }, { background: true });
 } catch (error) {
   console.log(error);
 }

packages/service/core/dataset/schema.ts

@@ -5,7 +5,7 @@ import {
   DatasetStatusEnum,
   DatasetStatusMap,
   DatasetTypeMap
-} from '@fastgpt/global/core/dataset/constant';
+} from '@fastgpt/global/core/dataset/constants';
 import {
   TeamCollectionName,
   TeamMemberCollectionName

packages/service/core/dataset/training/schema.ts

@@ -2,7 +2,7 @@
 import { connectionMongo, type Model } from '../../../common/mongo';
 const { Schema, model, models } = connectionMongo;
 import { DatasetTrainingSchemaType } from '@fastgpt/global/core/dataset/type';
-import { DatasetDataIndexTypeMap, TrainingTypeMap } from '@fastgpt/global/core/dataset/constant';
+import { DatasetDataIndexTypeMap, TrainingTypeMap } from '@fastgpt/global/core/dataset/constants';
 import { DatasetColCollectionName } from '../collection/schema';
 import { DatasetCollectionName } from '../schema';
 import {
@@ -102,11 +102,11 @@ const TrainingDataSchema = new Schema({
 });
 
 try {
-  TrainingDataSchema.index({ teamId: 1 });
+  // lock training data; delete training data
+  TrainingDataSchema.index({ teamId: 1, collectionId: 1 });
+  // get training data and sort
   TrainingDataSchema.index({ weight: -1 });
   TrainingDataSchema.index({ lockTime: 1 });
-  TrainingDataSchema.index({ datasetId: 1 });
-  TrainingDataSchema.index({ collectionId: 1 });
   TrainingDataSchema.index({ expireAt: 1 }, { expireAfterSeconds: 7 * 24 * 60 });
 } catch (error) {
   console.log(error);

packages/service/package.json

@@ -9,13 +9,11 @@
     "dayjs": "^1.11.7",
     "encoding": "^0.1.13",
     "jsonwebtoken": "^9.0.2",
-    "mammoth": "^1.6.0",
     "mongoose": "^7.0.2",
     "multer": "1.4.5-lts.1",
     "next": "13.5.2",
     "nextjs-cors": "^2.1.2",
     "node-cron": "^3.0.3",
-    "pdfjs-dist": "^4.0.269",
     "pg": "^8.10.0",
     "tunnel": "^0.0.6"
   },

packages/service/support/permission/auth/app.ts

@@ -6,7 +6,7 @@ import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
 import { parseHeaderCert } from '../controller';
 import { PermissionTypeEnum } from '@fastgpt/global/support/permission/constant';
 import { AppErrEnum } from '@fastgpt/global/common/error/code/app';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 
 // 模型使用权校验
 export async function authApp({
@@ -24,7 +24,7 @@ export async function authApp({
 > {
   const result = await parseHeaderCert(props);
   const { teamId, tmbId } = result;
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { app, isOwner, canWrite } = await (async () => {
     // get app

packages/service/support/permission/auth/dataset.ts

@@ -13,8 +13,9 @@ import {
 } from '@fastgpt/global/core/dataset/type';
 import { getFileById } from '../../../common/file/gridfs/controller';
 import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 import { CommonErrEnum } from '@fastgpt/global/common/error/code/common';
+import { MongoDatasetCollection } from '../../../core/dataset/collection/schema';
 
 export async function authDatasetByTmbId({
   teamId,
@@ -27,7 +28,7 @@ export async function authDatasetByTmbId({
   datasetId: string;
   per: AuthModeType['per'];
 }) {
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { dataset, isOwner, canWrite } = await (async () => {
     const dataset = await MongoDataset.findOne({ _id: datasetId, teamId }).lean();
@@ -107,7 +108,7 @@ export async function authDatasetCollection({
   }
 > {
   const { userId, teamId, tmbId } = await parseHeaderCert(props);
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { collection, isOwner, canWrite } = await (async () => {
     const collection = await getCollectionWithDataset(collectionId);
@@ -163,40 +164,30 @@ export async function authDatasetFile({
   }
 > {
   const { userId, teamId, tmbId } = await parseHeaderCert(props);
-  const { role } = await getTeamInfoByTmbId({ tmbId });
 
-  const file = await getFileById({ bucketName: BucketNameEnum.dataset, fileId });
+  const [file, collection] = await Promise.all([
+    getFileById({ bucketName: BucketNameEnum.dataset, fileId }),
+    MongoDatasetCollection.findOne({
+      teamId,
+      fileId
+    })
+  ]);
 
   if (!file) {
     return Promise.reject(CommonErrEnum.fileNotFound);
   }
 
-  if (file.metadata.teamId !== teamId) {
+  if (!collection) {
     return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
   }
 
-  const { dataset } = await authDataset({
+  // file role = collection role
+  try {
+    const { isOwner, canWrite } = await authDatasetCollection({
       ...props,
-    datasetId: file.metadata.datasetId,
+      collectionId: collection._id,
       per
     });
-  const isOwner =
-    role !== TeamMemberRoleEnum.visitor &&
-    (String(dataset.tmbId) === tmbId || role === TeamMemberRoleEnum.owner);
-
-  const canWrite =
-    isOwner ||
-    (role !== TeamMemberRoleEnum.visitor && dataset.permission === PermissionTypeEnum.public);
-
-  if (per === 'r' && !isOwner && dataset.permission !== PermissionTypeEnum.public) {
-    return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
-  }
-  if (per === 'w' && !canWrite) {
-    return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
-  }
-  if (per === 'owner' && !isOwner) {
-    return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
-  }
 
     return {
       userId,
@@ -206,4 +197,7 @@ export async function authDatasetFile({
       isOwner,
       canWrite
     };
+  } catch (error) {
+    return Promise.reject(DatasetErrEnum.unAuthDatasetFile);
+  }
 }

packages/service/support/permission/auth/openapi.ts

@@ -2,7 +2,7 @@ import { AuthResponseType } from '@fastgpt/global/support/permission/type';
 import { AuthModeType } from '../type';
 import { OpenApiSchema } from '@fastgpt/global/support/openapi/type';
 import { parseHeaderCert } from '../controller';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 import { MongoOpenApi } from '../../openapi/schema';
 import { OpenApiErrEnum } from '@fastgpt/global/common/error/code/openapi';
 import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
@@ -21,7 +21,7 @@ export async function authOpenApiKeyCrud({
   const result = await parseHeaderCert(props);
   const { tmbId, teamId } = result;
 
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { openapi, isOwner, canWrite } = await (async () => {
     const openapi = await MongoOpenApi.findOne({ _id: id, teamId });

packages/service/support/permission/auth/outLink.ts

@@ -9,7 +9,7 @@ import { MongoApp } from '../../../core/app/schema';
 import { OutLinkErrEnum } from '@fastgpt/global/common/error/code/outLink';
 import { PermissionTypeEnum } from '@fastgpt/global/support/permission/constant';
 import { AppErrEnum } from '@fastgpt/global/common/error/code/app';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 
 /* crud outlink permission */
 export async function authOutLinkCrud({
@@ -27,7 +27,7 @@ export async function authOutLinkCrud({
   const result = await parseHeaderCert(props);
   const { tmbId, teamId } = result;
 
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { app, outLink, isOwner, canWrite } = await (async () => {
     const outLink = await MongoOutLink.findOne({ _id: outLinkId, teamId });

packages/service/support/permission/auth/plugin.ts

@@ -1,7 +1,7 @@
 import { AuthResponseType } from '@fastgpt/global/support/permission/type';
 import { AuthModeType } from '../type';
 import { parseHeaderCert } from '../controller';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
 import { MongoPlugin } from '../../../core/plugin/schema';
 import { PluginErrEnum } from '@fastgpt/global/common/error/code/plugin';
@@ -23,7 +23,7 @@ export async function authPluginCrud({
   const result = await parseHeaderCert(props);
   const { tmbId, teamId } = result;
 
-  const { role } = await getTeamInfoByTmbId({ tmbId });
+  const { role } = await getTmbInfoByTmbId({ tmbId });
 
   const { plugin, isOwner, canWrite } = await (async () => {
     const plugin = await MongoPlugin.findOne({ _id: id, teamId });
@@ -73,7 +73,7 @@ export async function authPluginCanUse({
   }
 
   if (source === PluginSourceEnum.personal) {
-    const { role } = await getTeamInfoByTmbId({ tmbId });
+    const { role } = await getTmbInfoByTmbId({ tmbId });
     const plugin = await MongoPlugin.findOne({ _id: pluginId, teamId });
     if (!plugin) {
       return Promise.reject(PluginErrEnum.unExist);

packages/service/support/permission/auth/user.ts

@@ -3,7 +3,7 @@ import { AuthModeType } from '../type';
 import { TeamItemType } from '@fastgpt/global/support/user/team/type';
 import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
 import { parseHeaderCert } from '../controller';
-import { getTeamInfoByTmbId } from '../../user/team/controller';
+import { getTmbInfoByTmbId } from '../../user/team/controller';
 import { UserErrEnum } from '../../../../global/common/error/code/user';
 
 export async function authUserNotVisitor(props: AuthModeType): Promise<
@@ -13,7 +13,7 @@ export async function authUserNotVisitor(props: AuthModeType): Promise<
   }
 > {
   const { userId, teamId, tmbId } = await parseHeaderCert(props);
-  const team = await getTeamInfoByTmbId({ tmbId });
+  const team = await getTmbInfoByTmbId({ tmbId });
 
   if (team.role === TeamMemberRoleEnum.visitor) {
     return Promise.reject(UserErrEnum.binVisitor);
@@ -38,7 +38,7 @@ export async function authUserRole(props: AuthModeType): Promise<
   }
 > {
   const result = await parseHeaderCert(props);
-  const { role: userRole, canWrite } = await getTeamInfoByTmbId({ tmbId: result.tmbId });
+  const { role: userRole, canWrite } = await getTmbInfoByTmbId({ tmbId: result.tmbId });
 
   return {
     ...result,

packages/service/support/permission/limit/dataset.ts

@@ -14,7 +14,7 @@ export const checkDatasetLimit = async ({
   const usedSize = await getVectorCountByTeamId(teamId);
 
   if (usedSize + insertLen >= maxSize) {
-    return Promise.reject(`数据库容量已满，无法继续添加。可以在账号页面进行扩容。`);
+    return Promise.reject(`数据库容量不足，无法继续添加。可以在账号页面进行扩容。`);
   }
   return;
 };

packages/service/support/user/controller.ts

@@ -1,6 +1,6 @@
 import { UserType } from '@fastgpt/global/support/user/type';
 import { MongoUser } from './schema';
-import { getTeamInfoByTmbId, getUserDefaultTeam } from './team/controller';
+import { getTmbInfoByTmbId, getUserDefaultTeam } from './team/controller';
 import { ERROR_ENUM } from '@fastgpt/global/common/error/errorCode';
 import { UserErrEnum } from '@fastgpt/global/common/error/code/user';
 
@@ -21,16 +21,16 @@ export async function getUserDetail({
   tmbId?: string;
   userId?: string;
 }): Promise<UserType> {
-  const team = await (async () => {
+  const tmb = await (async () => {
     if (tmbId) {
-      return getTeamInfoByTmbId({ tmbId });
+      return getTmbInfoByTmbId({ tmbId });
     }
     if (userId) {
       return getUserDefaultTeam({ userId });
     }
     return Promise.reject(ERROR_ENUM.unAuthorization);
   })();
-  const user = await MongoUser.findById(team.userId);
+  const user = await MongoUser.findById(tmb.userId);
 
   if (!user) {
     return Promise.reject(ERROR_ENUM.unAuthorization);
@@ -44,7 +44,7 @@ export async function getUserDetail({
     timezone: user.timezone,
     promotionRate: user.promotionRate,
     openaiAccount: user.openaiAccount,
-    team
+    team: tmb
   };
 }
 

packages/service/support/user/schema.ts

@@ -56,9 +56,18 @@ const UserSchema = new Schema({
   timezone: {
     type: String,
     default: 'Asia/Shanghai'
+  },
+  lastLoginTmbId: {
+    type: Schema.Types.ObjectId
   }
 });
 
+try {
+  UserSchema.index({ createTime: -1 });
+} catch (error) {
+  console.log(error);
+}
+
 export const MongoUser: Model<UserModelSchema> =
   models[userCollectionName] || model(userCollectionName, UserSchema);
 MongoUser.syncIndexes();

packages/service/support/user/team/controller.ts

@@ -8,7 +8,7 @@ import {
 import { MongoTeamMember } from './teamMemberSchema';
 import { MongoTeam } from './teamSchema';
 
-async function getTeam(match: Record<string, any>): Promise<TeamItemType> {
+async function getTeamMember(match: Record<string, any>): Promise<TeamItemType> {
   const tmb = (await MongoTeamMember.findOne(match).populate('teamId')) as TeamMemberWithTeamSchema;
 
   if (!tmb) {
@@ -31,11 +31,11 @@ async function getTeam(match: Record<string, any>): Promise<TeamItemType> {
   };
 }
 
-export async function getTeamInfoByTmbId({ tmbId }: { tmbId: string }) {
+export async function getTmbInfoByTmbId({ tmbId }: { tmbId: string }) {
   if (!tmbId) {
     return Promise.reject('tmbId or userId is required');
   }
-  return getTeam({
+  return getTeamMember({
     _id: new Types.ObjectId(tmbId),
     status: notLeaveStatus
   });
@@ -45,7 +45,7 @@ export async function getUserDefaultTeam({ userId }: { userId: string }) {
   if (!userId) {
     return Promise.reject('tmbId or userId is required');
   }
-  return getTeam({
+  return getTeamMember({
     userId: new Types.ObjectId(userId),
     defaultTeam: true
   });

packages/service/support/user/team/teamSchema.ts

@@ -24,11 +24,11 @@ const TeamSchema = new Schema({
   },
   balance: {
     type: Number,
-    default: 2 * PRICE_SCALE
+    default: 0
   },
   maxSize: {
     type: Number,
-    default: 5
+    default: 3
   },
   limit: {
     lastExportDatasetTime: {
@@ -41,7 +41,7 @@ const TeamSchema = new Schema({
 });
 
 try {
-  TeamSchema.index({ lastDatasetBillTime: -1 });
+  // TeamSchema.index({ createTime: -1 });
 } catch (error) {
   console.log(error);
 }

packages/service/support/wallet/bill/controller.ts

@@ -25,15 +25,13 @@ export const createTrainingBill = async ({
       {
         moduleName: 'wallet.moduleName.index',
         model: vectorModel,
-        inputTokens: 0,
-        outputTokens: 0,
+        charsLength: 0,
         amount: 0
       },
       {
         moduleName: 'wallet.moduleName.qa',
         model: agentModel,
-        inputTokens: 0,
-        outputTokens: 0,
+        charsLength: 0,
         amount: 0
       }
     ],

packages/service/support/wallet/bill/schema.ts

@@ -52,10 +52,8 @@ const BillSchema = new Schema({
 });
 
 try {
-  BillSchema.index({ teamId: 1 });
-  BillSchema.index({ tmbId: 1 });
-  BillSchema.index({ tmbId: 1, time: 1 });
-  BillSchema.index({ time: 1 }, { expireAfterSeconds: 90 * 24 * 60 * 60 });
+  BillSchema.index({ teamId: 1, tmbId: 1, time: -1 });
+  BillSchema.index({ time: 1 }, { expireAfterSeconds: 180 * 24 * 60 * 60 });
 } catch (error) {
   console.log(error);
 }

packages/web/common/file/read/html.ts

@@ -17,5 +17,5 @@ export const readHtmlFile = async ({
     uploadImgController
   });
 
-  return { rawText: rawText };
+  return { rawText: simpleMd };
 };

packages/web/components/common/Icon/constants.ts

@@ -15,15 +15,19 @@ export const iconPaths = {
   'common/courseLight': () => import('./icons/common/courseLight.svg'),
   'common/customTitleLight': () => import('./icons/common/customTitleLight.svg'),
   'common/editor/resizer': () => import('./icons/common/editor/resizer.svg'),
+  'common/errorFill': () => import('./icons/common/errorFill.svg'),
   'common/file/move': () => import('./icons/common/file/move.svg'),
+  'common/folderFill': () => import('./icons/common/folderFill.svg'),
   'common/fullScreenLight': () => import('./icons/common/fullScreenLight.svg'),
   'common/gitFill': () => import('./icons/common/gitFill.svg'),
+  'common/gitInlight': () => import('./icons/common/gitInlight.svg'),
   'common/gitLight': () => import('./icons/common/gitLight.svg'),
   'common/googleFill': () => import('./icons/common/googleFill.svg'),
   'common/importLight': () => import('./icons/common/importLight.svg'),
   'common/inviteLight': () => import('./icons/common/inviteLight.svg'),
   'common/language/en': () => import('./icons/common/language/en.svg'),
   'common/language/zh': () => import('./icons/common/language/zh.svg'),
+  'common/linkBlue': () => import('./icons/common/linkBlue.svg'),
   'common/loading': () => import('./icons/common/loading.svg'),
   'common/navbar/pluginFill': () => import('./icons/common/navbar/pluginFill.svg'),
   'common/navbar/pluginLight': () => import('./icons/common/navbar/pluginLight.svg'),
@@ -37,10 +41,13 @@ export const iconPaths = {
   'common/retryLight': () => import('./icons/common/retryLight.svg'),
   'common/rightArrowLight': () => import('./icons/common/rightArrowLight.svg'),
   'common/routePushLight': () => import('./icons/common/routePushLight.svg'),
+  'common/saveFill': () => import('./icons/common/saveFill.svg'),
   'common/searchLight': () => import('./icons/common/searchLight.svg'),
+  'common/selectLight': () => import('./icons/common/selectLight.svg'),
   'common/settingLight': () => import('./icons/common/settingLight.svg'),
   'common/text/t': () => import('./icons/common/text/t.svg'),
   'common/tickFill': () => import('./icons/common/tickFill.svg'),
+  'common/uploadFileFill': () => import('./icons/common/uploadFileFill.svg'),
   'common/viewLight': () => import('./icons/common/viewLight.svg'),
   'common/voiceLight': () => import('./icons/common/voiceLight.svg'),
   copy: () => import('./icons/copy.svg'),
@@ -52,7 +59,12 @@ export const iconPaths = {
   'core/app/logsLight': () => import('./icons/core/app/logsLight.svg'),
   'core/app/markLight': () => import('./icons/core/app/markLight.svg'),
   'core/app/questionGuide': () => import('./icons/core/app/questionGuide.svg'),
-  'core/app/tts': () => import('./icons/core/app/tts.svg'),
+  'core/app/simpleMode/ai': () => import('./icons/core/app/simpleMode/ai.svg'),
+  'core/app/simpleMode/chat': () => import('./icons/core/app/simpleMode/chat.svg'),
+  'core/app/simpleMode/dataset': () => import('./icons/core/app/simpleMode/dataset.svg'),
+  'core/app/simpleMode/template': () => import('./icons/core/app/simpleMode/template.svg'),
+  'core/app/simpleMode/tts': () => import('./icons/core/app/simpleMode/tts.svg'),
+  'core/app/simpleMode/variable': () => import('./icons/core/app/simpleMode/variable.svg'),
   'core/app/ttsFill': () => import('./icons/core/app/ttsFill.svg'),
   'core/app/variable/input': () => import('./icons/core/app/variable/input.svg'),
   'core/app/variable/select': () => import('./icons/core/app/variable/select.svg'),
@@ -76,12 +88,15 @@ export const iconPaths = {
   'core/dataset/commonDataset': () => import('./icons/core/dataset/commonDataset.svg'),
   'core/dataset/datasetFill': () => import('./icons/core/dataset/datasetFill.svg'),
   'core/dataset/datasetLight': () => import('./icons/core/dataset/datasetLight.svg'),
-  'core/dataset/folderDataset': () => import('./icons/core/dataset/folderDataset.svg'),
+  'core/dataset/fileCollection': () => import('./icons/core/dataset/fileCollection.svg'),
   'core/dataset/fullTextRecall': () => import('./icons/core/dataset/fullTextRecall.svg'),
+  'core/dataset/manualCollection': () => import('./icons/core/dataset/manualCollection.svg'),
   'core/dataset/mixedRecall': () => import('./icons/core/dataset/mixedRecall.svg'),
   'core/dataset/modeEmbedding': () => import('./icons/core/dataset/modeEmbedding.svg'),
   'core/dataset/rerank': () => import('./icons/core/dataset/rerank.svg'),
+  'core/dataset/tableCollection': () => import('./icons/core/dataset/tableCollection.svg'),
   'core/dataset/websiteDataset': () => import('./icons/core/dataset/websiteDataset.svg'),
+  'core/modules/flowLight': () => import('./icons/core/modules/flowLight.svg'),
   'core/modules/previewLight': () => import('./icons/core/modules/previewLight.svg'),
   'core/modules/variable': () => import('./icons/core/modules/variable.svg'),
   'core/modules/welcomeText': () => import('./icons/core/modules/welcomeText.svg'),
@@ -91,6 +106,15 @@ export const iconPaths = {
   empty: () => import('./icons/empty.svg'),
   export: () => import('./icons/export.svg'),
   'file/csv': () => import('./icons/file/csv.svg'),
+  'file/fill/csv': () => import('./icons/file/fill/csv.svg'),
+  'file/fill/doc': () => import('./icons/file/fill/doc.svg'),
+  'file/fill/file': () => import('./icons/file/fill/file.svg'),
+  'file/fill/folder': () => import('./icons/file/fill/folder.svg'),
+  'file/fill/html': () => import('./icons/file/fill/html.svg'),
+  'file/fill/manual': () => import('./icons/file/fill/manual.svg'),
+  'file/fill/markdown': () => import('./icons/file/fill/markdown.svg'),
+  'file/fill/pdf': () => import('./icons/file/fill/pdf.svg'),
+  'file/fill/txt': () => import('./icons/file/fill/txt.svg'),
   'file/html': () => import('./icons/file/html.svg'),
   'file/indexImport': () => import('./icons/file/indexImport.svg'),
   'file/manualImport': () => import('./icons/file/manualImport.svg'),
@@ -103,10 +127,14 @@ export const iconPaths = {
   kbTest: () => import('./icons/kbTest.svg'),
   menu: () => import('./icons/menu.svg'),
   minus: () => import('./icons/minus.svg'),
+  'modal/edit': () => import('./icons/modal/edit.svg'),
+  'modal/manualDataset': () => import('./icons/modal/manualDataset.svg'),
+  'modal/selectSource': () => import('./icons/modal/selectSource.svg'),
   more: () => import('./icons/more.svg'),
   out: () => import('./icons/out.svg'),
   'phoneTabbar/me': () => import('./icons/phoneTabbar/me.svg'),
-  'phoneTabbar/more': () => import('./icons/phoneTabbar/more.svg'),
+  'phoneTabbar/tool': () => import('./icons/phoneTabbar/tool.svg'),
+  'phoneTabbar/toolFill': () => import('./icons/phoneTabbar/toolFill.svg'),
   save: () => import('./icons/save.svg'),
   stop: () => import('./icons/stop.svg'),
   'support/account/loginoutLight': () => import('./icons/support/account/loginoutLight.svg'),

packages/web/components/common/Icon/icons/common/closeLight.svg

@@ -1 +1,4 @@
-<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1689496771994" class="icon" viewBox="0 0 1028 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="12388" xmlns:xlink="http://www.w3.org/1999/xlink" ><path d="M646.4 512l345.6-345.6c38.4-38.4 38.4-96 0-134.4-38.4-38.4-96-38.4-134.4 0L512 377.6 166.4 32C128-6.4 70.4-6.4 32 32c-38.4 38.4-38.4 96 0 134.4L377.6 512l-345.6 345.6c-38.4 38.4-38.4 96 0 134.4 19.2 19.2 44.8 25.6 70.4 25.6s51.2-6.4 70.4-25.6L512 646.4l345.6 345.6c19.2 19.2 44.8 25.6 70.4 25.6s51.2-6.4 70.4-25.6c38.4-38.4 38.4-96 0-134.4L646.4 512z"  p-id="12389"></path></svg>
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M3.52876 3.52876C3.78911 3.26841 4.21122 3.26841 4.47157 3.52876L8.00016 7.05735L11.5288 3.52876C11.7891 3.26841 12.2112 3.26841 12.4716 3.52876C12.7319 3.78911 12.7319 4.21122 12.4716 4.47157L8.94297 8.00016L12.4716 11.5288C12.7319 11.7891 12.7319 12.2112 12.4716 12.4716C12.2112 12.7319 11.7891 12.7319 11.5288 12.4716L8.00016 8.94297L4.47157 12.4716C4.21122 12.7319 3.78911 12.7319 3.52876 12.4716C3.26841 12.2112 3.26841 11.7891 3.52876 11.5288L7.05735 8.00016L3.52876 4.47157C3.26841 4.21122 3.26841 3.78911 3.52876 3.52876Z" />
+</svg>

packages/web/components/common/Icon/icons/common/errorFill.svg

@@ -0,0 +1,6 @@
+<svg t="1705054369902" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="3174"
+    width="128" height="128">
+    <path
+        d="M512 0C229.205333 0 0 229.205333 0 512s229.205333 512 512 512 512-229.205333 512-512S794.794667 0 512 0z m0 796.458667A56.917333 56.917333 0 1 1 511.957333 682.666667 56.917333 56.917333 0 0 1 512 796.458667z m54.186667-227.797334h0.128a60.501333 60.501333 0 0 1-53.802667 55.893334c2.048 0.256 3.882667 1.152 5.973333 1.152h-11.818666c2.048 0 3.84-0.981333 5.845333-1.109334a59.093333 59.093333 0 0 1-53.162667-55.893333l-13.056-284.16a54.314667 54.314667 0 0 1 54.613334-57.045333h26.282666a52.992 52.992 0 0 1 54.186667 57.002666l-15.146667 284.16z"
+        fill="#D92D20" p-id="3175"></path>
+</svg>

packages/web/components/common/Icon/icons/common/folderFill.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 25 24" fill="none">
+    <path
+        d="M4.61611 2.88843C3.18017 2.88843 2.01611 4.05249 2.01611 5.48843V17.7115C2.01611 19.5892 3.53835 21.1115 5.41612 21.1115H19.7162C21.594 21.1115 23.1162 19.5892 23.1162 17.7115V9.15457C23.1162 7.27681 21.594 5.75457 19.7162 5.75457H13.6589L12.8388 4.17982C12.4254 3.38621 11.6049 2.88843 10.7101 2.88843H4.61611Z"
+        fill="#FDB022" />
+</svg>

packages/web/components/common/Icon/icons/common/gitInlight.svg

@@ -0,0 +1,4 @@
+<svg viewBox="0 0 25 24" fill="none" xmlns="http://www.w3.org/2000/svg">
+    <path
+        d="M13.5179 2.95716C12.3008 2.95716 11.635 3.08502 10.6322 3.30471C10.1055 2.96916 9.51659 2.67192 8.95368 2.4506C8.29526 2.19172 7.50696 1.96965 6.85824 2.03925C6.58511 2.06856 6.34952 2.20523 6.18895 2.40279C6.11526 2.48248 6.05339 2.57562 6.00739 2.68063C5.44347 3.96783 5.48842 5.25029 5.73449 6.21247C5.34326 6.74564 5.04063 7.24563 4.8355 7.79481C4.57351 8.4962 4.49805 9.20247 4.49805 10.0491C4.49805 12.2422 5.16939 13.9501 6.42439 15.1455C7.31096 15.99 8.42728 16.5198 9.65563 16.8151C9.53112 17.1739 9.47343 17.5357 9.45297 17.9085C9.44691 17.953 9.44379 17.9984 9.44379 18.0446L9.44379 18.3592C9.40123 18.3682 9.35877 18.3802 9.31665 18.3951C8.45622 18.6994 6.21396 18.59 5.07859 16.1042C4.84914 15.6018 4.25588 15.3806 3.75352 15.61C3.25115 15.8395 3.02991 16.4327 3.25936 16.9351C4.73668 20.1696 7.66374 20.7688 9.44379 20.4251L9.44379 20.9737C9.44379 21.526 9.89151 21.9737 10.4438 21.9737C10.9961 21.9737 11.4438 21.526 11.4438 20.9737L11.4438 18.2612C11.4439 17.87 11.4848 17.6313 11.5558 17.441C11.6258 17.2535 11.7505 17.0451 12.0163 16.7555C12.3897 16.3486 12.3626 15.716 11.9557 15.3425C11.8534 15.2486 11.7368 15.18 11.6136 15.1364C11.526 15.1009 11.4317 15.0774 11.3327 15.0679C9.79552 14.9211 8.60416 14.4597 7.80381 13.6973C7.02483 12.9553 6.49805 11.8157 6.49805 10.0491C6.49805 9.32024 6.56459 8.88141 6.70906 8.49464C6.85771 8.09668 7.11681 7.67798 7.62646 7.0287C7.71912 6.91066 7.78101 6.77828 7.81329 6.64123C7.8599 6.45105 7.85127 6.24526 7.77638 6.04841C7.59327 5.56715 7.47715 4.86062 7.63605 4.11538C7.80168 4.16009 7.998 4.22388 8.22184 4.31189C8.7844 4.53308 9.37842 4.85063 9.84323 5.18443C10.0026 5.29885 10.1841 5.3597 10.3665 5.37049C10.4586 5.3766 10.5531 5.36996 10.6478 5.34915L10.809 5.31367C11.9359 5.06551 12.428 4.95716 13.5179 4.95716C14.6664 4.95716 15.2211 5.06841 16.477 5.34848C16.5604 5.36708 16.6437 5.37467 16.7253 5.37222C16.9253 5.37098 17.1268 5.30992 17.3015 5.18443C17.7663 4.85063 18.3604 4.53308 18.9229 4.31189C19.1441 4.22491 19.3385 4.16159 19.5029 4.11697C19.657 4.83921 19.5505 5.50003 19.3685 5.98138C19.3487 6.03369 19.3336 6.08662 19.3229 6.13972C19.2375 6.44272 19.2962 6.78117 19.5102 7.04371C19.9324 7.56159 20.1919 7.94145 20.361 8.3625C20.5269 8.77575 20.631 9.29478 20.631 10.1064C20.631 11.7493 20.1669 12.8498 19.4307 13.5936C18.6776 14.3544 17.5199 14.8633 15.9107 15.0717C15.8939 15.0738 15.8773 15.0764 15.8608 15.0794C15.6244 15.083 15.3884 15.1701 15.2005 15.3425C14.7936 15.716 14.7665 16.3486 15.1399 16.7555C15.4057 17.0451 15.5304 17.2535 15.6004 17.441C15.6714 17.6313 15.7123 17.87 15.7124 18.2612L15.7124 20.9737C15.7124 21.526 16.1601 21.9737 16.7124 21.9737C17.2647 21.9737 17.7124 21.526 17.7124 20.9737L17.7124 18.0446C17.7124 17.9984 17.7093 17.953 17.7032 17.9085C17.6826 17.5323 17.624 17.1673 17.4972 16.8054C18.8154 16.472 19.9668 15.895 20.8521 15.0006C22.0409 13.7996 22.631 12.1471 22.631 10.1064C22.631 9.11797 22.5034 8.33085 22.217 7.61739C22.0044 7.08778 21.7185 6.63605 21.3946 6.20504C21.6562 5.22818 21.6939 3.95937 21.1196 2.64857C21.0399 2.46661 20.9125 2.32029 20.7586 2.21782C20.6236 2.12187 20.4629 2.05817 20.2865 2.03925C19.6378 1.96965 18.8495 2.19172 18.1911 2.4506C17.6268 2.67247 17.0363 2.97063 16.5087 3.30721C15.4463 3.07561 14.7386 2.95716 13.5179 2.95716Z" />
+</svg>

packages/web/components/common/Icon/icons/common/importLight.svg

@@ -1 +1,6 @@
-<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1689485689790" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="3190" xmlns:xlink="http://www.w3.org/1999/xlink" ><path d="M953.86906337 704.50762194c-19.00036972 0.29457937-34.31849726 15.90728628-34.02391788 34.90765601v87.63736424c0 38.29531883-30.34167568 69.52073264-67.75325638 69.52073264H175.00119435c-37.4115807 0-67.75325639-31.22541381-67.7532564-69.52073264V201.80791751c0-38.29531883 30.34167568-69.52073264 67.7532564-69.52073264h235.66350044l87.49007456 155.6852c7.06990501 12.37233377 20.32597692 18.70579035 33.43475913 17.23289348h320.35507094c37.4115807 0 67.75325639 31.22541381 67.75325639 69.52073264v86.90091578c0 19.2949491 15.31812754 34.907656 34.17120756 34.90765601 18.85308003 0 34.02391789-15.61270692 34.02391788-34.90765601 0-0.88373813 0-1.76747625-0.14728968-2.65121438v-84.2497014c0-76.73792734-60.68335138-139.04146527-135.50651276-139.04146528H547.20223538l-87.34278486-155.09604125c-7.80635344-13.99252034-19.00036972-17.67476253-28.86877881-17.38018316l-0.14728967-0.14728968h-256.28405678c-74.82316139 0-135.50651276 62.30353794-135.50651277 139.04146527v623.47724842c0 76.73792734 60.68335138 139.04146527 135.50651277 139.04146527H852.23917881c74.82316139 0 135.50651276-62.30353794 135.50651276-139.04146527v-83.51325297c0.1472897-0.88373813 0.1472897-1.76747625 0.14728968-2.65121439 0.1472897-9.13196064-3.38766283-17.96934191-9.86840907-24.45008817-6.33345658-6.62803595-15.02354815-10.31027814-24.15550881-10.45756784z" p-id="3191"></path><path d="M473.55739149 451.90580738c-5.44971845-5.89158752-12.96149253-9.27925033-21.06242536-9.27925032-7.95364315 0-15.61270692 3.38766283-21.06242536 9.27925032l-125.49081399 133.59174683-0.73644843 0.73644844h-0.14728971l-0.73644843 0.73644844c-11.34130596 12.51962347-11.34130596 31.66728287 0 44.18690633l125.93268305 134.32819525c5.30242877 5.89158752 12.81420285 9.13196064 20.76784599 9.27925035 7.95364315 0 15.46541722-3.24037312 20.76784598-9.13196064l0.58915876-0.73644844c11.34130596-12.51962347 11.34130596-31.66728287 0-44.18690633l-75.85418922-81.00932829h344.51057975c16.05457598 0 29.16335819-13.99252034 29.16335819-31.07812414 0-17.08560379-13.10878221-31.07812413-29.16335819-31.07812411h-343.18497256l75.70689953-80.86203861c11.63588534-12.51962347 11.63588534-32.10915193 0-44.77606508z" p-id="3192"></path></svg>
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" fill="none">
+    <path
+        d="M4.25845 0.738983C4.25845 0.606996 4.15146 0.5 4.01947 0.5C2.11725 0.5 0.575195 2.04205 0.575195 3.94428V12.0557C0.575195 13.9579 2.11725 15.5 4.01947 15.5H12.1309C14.0331 15.5 15.5752 13.9579 15.5752 12.0557V3.94428C15.5752 2.04205 14.0331 0.5 12.1309 0.5H10.4017C9.98744 0.5 9.65166 0.835786 9.65166 1.25C9.65166 1.66421 9.98744 2 10.4017 2H12.1309C13.2047 2 14.0752 2.87048 14.0752 3.94428V12.0557C14.0752 13.1295 13.2047 14 12.1309 14H4.01947C2.94568 14 2.0752 13.1295 2.0752 12.0557V3.94428C2.0752 2.87048 2.94568 2 4.01947 2C4.15146 2 4.25845 1.893 4.25845 1.76102V0.738983Z" />
+    <path
+        d="M7.38092 4.3543C7.26179 3.52006 7.01223 2.99621 6.59273 2.70009C5.59427 1.99531 4.85314 2.00002 4.74413 2.00072L4.7369 2.00075H3.9869V0.500749H4.7369C5.00097 0.500749 6.0952 0.512848 7.45775 1.47464C8.37881 2.12479 8.7247 3.15378 8.86586 4.14224C8.98099 4.94841 8.97457 5.85452 8.96865 6.68934C8.96737 6.86971 8.96612 7.04676 8.96612 7.21874C8.96612 7.70483 8.95571 8.16141 8.94035 8.57069L9.57153 7.93951C9.86442 7.64661 10.3393 7.64661 10.6322 7.93951C10.9251 8.2324 10.9251 8.70727 10.6322 9.00017L8.65543 10.9769C8.49629 11.1361 8.28342 11.2087 8.07521 11.1949C7.86701 11.2087 7.65414 11.1361 7.495 10.9769L5.51824 9.00017C5.22535 8.70727 5.22535 8.2324 5.51824 7.93951C5.81113 7.64661 6.28601 7.64661 6.5789 7.93951L7.33793 8.69854L7.42997 8.79058C7.45074 8.33162 7.46612 7.79657 7.46612 7.21874C7.46612 7.02039 7.46735 6.82532 7.46856 6.63372C7.47382 5.80363 7.47866 5.03868 7.38092 4.3543Z" />
+</svg>

packages/web/components/common/Icon/icons/common/linkBlue.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="none">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M10.0004 2.92882C11.953 0.9762 15.1188 0.9762 17.0714 2.92882C19.0241 4.88144 19.0241 8.04727 17.0714 9.99989L15.8929 11.1784C15.5675 11.5038 15.0399 11.5038 14.7144 11.1784C14.389 10.853 14.389 10.3253 14.7144 9.99989L15.8929 8.82138C17.1947 7.51963 17.1947 5.40908 15.8929 4.10733C14.5912 2.80558 12.4806 2.80559 11.1789 4.10733L10.0004 5.28584C9.67494 5.61128 9.1473 5.61128 8.82187 5.28584C8.49643 4.96041 8.49643 4.43277 8.82187 4.10733L10.0004 2.92882ZM13.5063 6.49395C13.8317 6.81939 13.8317 7.34702 13.5063 7.67246L7.67297 13.5058C7.34753 13.8312 6.81989 13.8312 6.49446 13.5058C6.16902 13.1804 6.16902 12.6527 6.49446 12.3273L12.3278 6.49395C12.6532 6.16851 13.1809 6.16851 13.5063 6.49395ZM5.28633 8.82138C5.61177 9.14681 5.61177 9.67445 5.28633 9.99989L4.10782 11.1784C2.80607 12.4801 2.80607 14.5907 4.10782 15.8924C5.40957 17.1942 7.52012 17.1942 8.82187 15.8924L10.0004 14.7139C10.3258 14.3885 10.8535 14.3885 11.1789 14.7139C11.5043 15.0394 11.5043 15.567 11.1789 15.8924L10.0004 17.071C8.04776 19.0236 4.88193 19.0236 2.92931 17.071C0.976688 15.1183 0.976689 11.9525 2.92931 9.99989L4.10782 8.82138C4.43326 8.49594 4.9609 8.49594 5.28633 8.82138Z"
+        fill="#3370FF" />
+</svg>

packages/web/components/common/Icon/icons/common/overviewLight.svg

@@ -1 +1,4 @@
-<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1689305725826" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="3372" xmlns:xlink="http://www.w3.org/1999/xlink" width="64" height="64"><path d="M512 774.4a38.4 38.4 0 0 1-38.4-38.4V304a38.4 38.4 0 0 1 76.8 0v432a38.4 38.4 0 0 1-38.4 38.4zM741.12 774.4a38.4 38.4 0 0 1-38.4-38.4V501.12a38.4 38.4 0 1 1 76.8 0v234.88a38.4 38.4 0 0 1-38.4 38.4zM282.88 774.4a38.4 38.4 0 0 1-38.4-38.4V550.4A38.4 38.4 0 0 1 320 550.4v185.6a38.4 38.4 0 0 1-37.12 38.4zM282.88 421.12a38.4 38.4 0 0 1-38.4-38.4V304a38.4 38.4 0 0 1 76.8 0V384a38.4 38.4 0 0 1-38.4 37.12z" p-id="3373"></path><path d="M869.76 140.8a76.8 76.8 0 0 1 76.8 76.8v588.8a76.8 76.8 0 0 1-76.8 76.8H154.24a76.8 76.8 0 0 1-76.8-76.8V217.6a76.8 76.8 0 0 1 76.8-76.8h715.52m0-76.8H154.24A153.6 153.6 0 0 0 0 217.6v588.8A153.6 153.6 0 0 0 154.24 960h715.52A153.6 153.6 0 0 0 1024 806.4V217.6A153.6 153.6 0 0 0 869.76 64z" p-id="3374"></path></svg>
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 18 18" fill="none">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M5.46561 0.666748H12.5344C13.2052 0.666738 13.7589 0.666729 14.2099 0.703577C14.6783 0.741848 15.1089 0.823985 15.5133 1.03006C16.1405 1.34964 16.6505 1.85957 16.97 2.48678C17.1761 2.89123 17.2583 3.32181 17.2965 3.79023C17.3334 4.24122 17.3334 4.79485 17.3334 5.46565V12.5345C17.3334 13.2053 17.3334 13.7589 17.2965 14.2099C17.2583 14.6784 17.1761 15.1089 16.97 15.5134C16.6505 16.1406 16.1405 16.6505 15.5133 16.9701C15.1089 17.1762 14.6783 17.2583 14.2099 17.2966C13.7589 17.3334 13.2053 17.3334 12.5345 17.3334H5.46558C4.79478 17.3334 4.24116 17.3334 3.79017 17.2966C3.32175 17.2583 2.89117 17.1762 2.48672 16.9701C1.85951 16.6505 1.34958 16.1406 1.03 15.5134C0.823924 15.1089 0.741787 14.6784 0.703516 14.2099C0.666668 13.7589 0.666677 13.2053 0.666687 12.5345V5.46567C0.666677 4.79486 0.666668 4.24122 0.703516 3.79023C0.741787 3.32181 0.823924 2.89123 1.03 2.48678C1.34958 1.85957 1.85951 1.34964 2.48672 1.03006C2.89117 0.823985 3.32175 0.741848 3.79017 0.703577C4.24116 0.666729 4.7948 0.666738 5.46561 0.666748ZM3.92589 2.36471C3.56054 2.39456 3.3737 2.44866 3.24337 2.51507C2.92977 2.67486 2.6748 2.92983 2.51501 3.24343C2.4486 3.37376 2.3945 3.5606 2.36465 3.92595C2.334 4.30102 2.33335 4.78627 2.33335 5.50008V12.5001C2.33335 13.2139 2.334 13.6991 2.36465 14.0742C2.3945 14.4396 2.4486 14.6264 2.51501 14.7567C2.6748 15.0703 2.92977 15.3253 3.24337 15.4851C3.3737 15.5515 3.56054 15.6056 3.92589 15.6355C4.30096 15.6661 4.7862 15.6667 5.50002 15.6667H12.5C13.2138 15.6667 13.6991 15.6661 14.0742 15.6355C14.4395 15.6056 14.6263 15.5515 14.7567 15.4851C15.0703 15.3253 15.3252 15.0703 15.485 14.7567C15.5514 14.6264 15.6055 14.4396 15.6354 14.0742C15.666 13.6991 15.6667 13.2139 15.6667 12.5001V5.50008C15.6667 4.78627 15.666 4.30102 15.6354 3.92595C15.6055 3.5606 15.5514 3.37376 15.485 3.24343C15.3252 2.92983 15.0703 2.67486 14.7567 2.51507C14.6263 2.44866 14.4395 2.39456 14.0742 2.36471C13.6991 2.33406 13.2138 2.33342 12.5 2.33342H5.50002C4.7862 2.33342 4.30096 2.33406 3.92589 2.36471ZM9.00002 4.00008C9.46026 4.00008 9.83335 4.37318 9.83335 4.83342V13.1667C9.83335 13.627 9.46026 14.0001 9.00002 14.0001C8.53978 14.0001 8.16669 13.627 8.16669 13.1667V4.83342C8.16669 4.37318 8.53978 4.00008 9.00002 4.00008ZM12.3334 7.33342C12.7936 7.33342 13.1667 7.70651 13.1667 8.16675V13.1667C13.1667 13.627 12.7936 14.0001 12.3334 14.0001C11.8731 14.0001 11.5 13.627 11.5 13.1667V8.16675C11.5 7.70651 11.8731 7.33342 12.3334 7.33342ZM5.66669 9.00008C6.12692 9.00008 6.50002 9.37318 6.50002 9.83342V13.1667C6.50002 13.627 6.12692 14.0001 5.66669 14.0001C5.20645 14.0001 4.83335 13.627 4.83335 13.1667V9.83342C4.83335 9.37318 5.20645 9.00008 5.66669 9.00008Z" />
+</svg>

packages/web/components/common/Icon/icons/common/questionLight.svg

@@ -1,4 +1,4 @@
-<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" viewBox="0 0 18 18" fill="none">
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 18 18" fill="none">
     <path fill-rule="evenodd" clip-rule="evenodd"
         d="M8.99997 2.196C5.24222 2.196 2.19596 5.24225 2.19596 9C2.19596 12.7577 5.24222 15.804 8.99997 15.804C12.7577 15.804 15.804 12.7577 15.804 9C15.804 5.24225 12.7577 2.196 8.99997 2.196ZM0.529297 9C0.529297 4.32178 4.32174 0.529331 8.99997 0.529331C13.6782 0.529331 17.4706 4.32178 17.4706 9C17.4706 13.6782 13.6782 17.4707 8.99997 17.4707C4.32174 17.4707 0.529297 13.6782 0.529297 9ZM9.18533 6.03224C8.846 5.97403 8.49702 6.0378 8.20019 6.21224C7.90337 6.38669 7.67786 6.66056 7.56361 6.98534C7.41089 7.41949 6.93512 7.64764 6.50097 7.49491C6.06681 7.34218 5.83866 6.86642 5.99139 6.43226C6.23625 5.73619 6.71956 5.14923 7.35572 4.77536C7.99188 4.40148 8.73983 4.26481 9.4671 4.38956C10.1944 4.5143 10.854 4.89241 11.3292 5.45692C11.8043 6.0213 12.0644 6.73558 12.0634 7.4733C12.063 8.67899 11.1697 9.46897 10.5467 9.88431C10.2098 10.1089 9.8788 10.2738 9.63531 10.382C9.51239 10.4367 9.4088 10.4782 9.33398 10.5067C9.2965 10.5209 9.26605 10.532 9.24377 10.54L9.21658 10.5495L9.20781 10.5525L9.20467 10.5535L9.20342 10.554C9.20317 10.554 9.20239 10.5543 8.93887 9.76373L9.20239 10.5543C8.76577 10.6998 8.29384 10.4639 8.1483 10.0273C8.00281 9.59078 8.23857 9.11902 8.67491 8.97331L8.68542 8.9696C8.69671 8.96559 8.71548 8.95878 8.74065 8.94919C8.79114 8.92996 8.86654 8.89986 8.95842 8.85902C9.14454 8.7763 9.38634 8.65481 9.62222 8.49756C10.1447 8.14925 10.3967 7.79388 10.3967 7.47253L10.3967 7.47129C10.3972 7.127 10.2759 6.79364 10.0542 6.53025C9.83245 6.26686 9.52467 6.09044 9.18533 6.03224ZM8.16663 12.8187C8.16663 12.3584 8.53973 11.9853 8.99997 11.9853H9.0076C9.46784 11.9853 9.84094 12.3584 9.84094 12.8187C9.84094 13.2789 9.46784 13.652 9.0076 13.652H8.99997C8.53973 13.652 8.16663 13.2789 8.16663 12.8187Z" />
 </svg>

packages/web/components/common/Icon/icons/common/saveFill.svg

@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M2.46341 3.92013C2.1001 4.63317 2.1001 5.56659 2.1001 7.43343V16.7668C2.1001 18.6336 2.1001 19.567 2.46341 20.2801C2.78299 20.9073 3.29292 21.4172 3.92013 21.7368C4.63317 22.1001 5.56659 22.1001 7.43343 22.1001H16.7668C18.6336 22.1001 19.567 22.1001 20.2801 21.7368C20.9073 21.4172 21.4172 20.9073 21.7368 20.2801C22.1001 19.567 22.1001 18.6336 22.1001 16.7668V9.12841C22.1001 8.58488 22.1001 8.31311 22.0387 8.05736C21.9843 7.83061 21.8945 7.61385 21.7726 7.41502C21.6352 7.19076 21.443 6.99859 21.0587 6.61426L17.5859 3.1415C17.2016 2.75716 17.0094 2.56499 16.7852 2.42756C16.5863 2.30572 16.3696 2.21594 16.1428 2.1615C15.8871 2.1001 15.6153 2.1001 15.0718 2.1001H7.43343C5.56659 2.1001 4.63317 2.1001 3.92013 2.46341C3.29292 2.78299 2.78299 3.29292 2.46341 3.92013ZM15.7762 4.34475C15.6378 4.20639 15.5687 4.13721 15.4879 4.08774C15.4163 4.04387 15.3383 4.01155 15.2567 3.99195C15.1646 3.96985 15.0668 3.96985 14.8711 3.96985H8.34351C7.89546 3.96985 7.67144 3.96985 7.50031 4.05704C7.34978 4.13374 7.2274 4.25613 7.1507 4.40666C7.06351 4.57779 7.06351 4.80181 7.06351 5.24985V6.76131C7.06351 7.20935 7.06351 7.43337 7.1507 7.6045C7.2274 7.75503 7.34978 7.87741 7.50031 7.95411C7.67144 8.04131 7.89546 8.04131 8.34351 8.04131H15.8474C16.2954 8.04131 16.5194 8.04131 16.6906 7.95411C16.8411 7.87741 16.9635 7.75503 17.0402 7.6045C17.1274 7.43337 17.1274 7.20935 17.1274 6.76131V6.22613C17.1274 6.03046 17.1274 5.93262 17.1053 5.84055C17.0857 5.75892 17.0534 5.68089 17.0095 5.60931C16.96 5.52859 16.8908 5.45941 16.7525 5.32107L15.7762 4.34475ZM7.06351 14.171C7.06351 13.7229 7.06351 13.4989 7.15071 13.3278C7.22741 13.1773 7.34979 13.0549 7.50032 12.9782C7.67145 12.891 7.89547 12.891 8.34351 12.891H15.8474C16.2954 12.891 16.5195 12.891 16.6906 12.9782C16.8411 13.0549 16.9635 13.1773 17.0402 13.3278C17.1274 13.4989 17.1274 13.7229 17.1274 14.171V18.5985C17.1274 19.0465 17.1274 19.2706 17.0402 19.4417C16.9635 19.5922 16.8411 19.7146 16.6906 19.7913C16.5195 19.8785 16.2954 19.8785 15.8474 19.8785H8.34351C7.89547 19.8785 7.67145 19.8785 7.50032 19.7913C7.34979 19.7146 7.22741 19.5922 7.15071 19.4417C7.06351 19.2706 7.06351 19.0465 7.06351 18.5985V14.171Z" />
+</svg>

packages/web/components/common/Icon/icons/common/selectLight.svg

@@ -0,0 +1,6 @@
+<svg t="1704951153054" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="8832"
+    width="128" height="128">
+    <path
+        d="M187.5 187.6l693.1 230.7L696 515.4l-76.9 40.5 61.5 61.4L885 821.4l-63.4 63.6-204.2-204.5-61.5-61.6-40.5 77-96.8 184.3L187.5 187.6M155.9 111.9c-4.5 0-9.2 0.7-13.9 2.1-23 7.4-35.4 32-28 55.6l258.4 774.5c3.1 10.8 10.6 20.8 21.4 26.7 6.6 3.6 13.6 5.3 20.6 5.3 15.7 0 30.9-8.7 38.4-23.3l119.2-227 216.4 216.8 2.5 2.2c8.4 8.4 19.6 12.6 30.8 12.6s22.4-4.2 30.7-12.6l92.2-92.5c17.1-17.1 17.1-44.4 0-61.5l-218.9-218.6 226.7-119.2c10-5.6 17.4-13.6 21.1-25.2 7.8-23-5-47.5-27.3-54.9L169 114C164.8 112.6 160.4 111.9 155.9 111.9L155.9 111.9z"
+        p-id="8833"></path>
+</svg>

packages/web/components/common/Icon/icons/common/uploadFileFill.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="33" height="32" viewBox="0 0 33 32" fill="none">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M31.0406 19.4981C31.0406 23.0747 28.2898 26.0087 24.7884 26.2999V26.3236H8.63716V26.3221C4.93579 26.2434 1.95947 23.2184 1.95947 19.4981C1.95947 16.2971 4.16297 13.6108 7.13588 12.8732C8.23514 8.73007 12.011 5.67676 16.5001 5.67676C20.9891 5.67676 24.765 8.73007 25.8642 12.8732C28.8371 13.6108 31.0406 16.2971 31.0406 19.4981ZM16.5001 23.5123C15.7637 23.5123 15.1667 22.9154 15.1667 22.179V16.932L14.4337 17.6651C13.9964 18.1024 13.2873 18.1024 12.85 17.6651C12.4127 17.2278 12.4127 16.5188 12.85 16.0814L15.511 13.4204C15.541 13.3872 15.5727 13.3556 15.6059 13.3255L15.7065 13.225L15.7082 13.2232C16.1456 12.7859 16.8546 12.7859 17.2919 13.2232L17.2937 13.225L17.3941 13.3254C17.4274 13.3555 17.4592 13.3873 17.4893 13.4206L20.1484 16.0797C20.5867 16.518 20.5867 17.2286 20.1484 17.6669C19.7101 18.1052 18.9995 18.1052 18.5612 17.6669L17.8334 16.9391V22.179C17.8334 22.9154 17.2365 23.5123 16.5001 23.5123Z"
+        fill="#3370FF" />
+</svg>

packages/web/components/common/Icon/icons/common/viewLight.svg

@@ -1 +1,4 @@
-<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1701420318127" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="13238" xmlns:xlink="http://www.w3.org/1999/xlink" width="128" height="128"><path d="M987.52 367.36c-108.16-146.56-266.88-239.36-444.16-239.36-177.28 0-335.36 92.8-444.16 239.36-48 64.64-48 160.64 0 225.28C208 739.2 366.72 832 543.36 832c177.28 0 335.36-92.8 444.16-239.36C1035.52 528 1035.52 432 987.52 367.36zM939.52 539.52C839.04 684.8 694.4 768 542.72 768c-151.68 0-296.32-83.2-396.8-228.48C121.6 505.6 121.6 455.04 145.92 420.48 246.4 275.2 391.04 192 542.72 192c151.68 0 296.32 83.2 396.8 228.48C963.2 455.04 963.2 505.6 939.52 539.52zM544 256C420.48 256 320 356.48 320 480S420.48 704 544 704 768 603.52 768 480 667.52 256 544 256zM544 640C455.68 640 384 568.32 384 480S455.68 320 544 320C632.32 320 704 391.68 704 480S632.32 640 544 640z" p-id="13239"></path></svg>
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 21 20">
+    <path
+        d="M10.7638 7.9165C11.3163 7.9165 11.8463 8.136 12.237 8.5267C12.6277 8.9174 12.8471 9.4473 12.8471 9.99984C12.8471 10.5524 12.6277 11.0823 12.237 11.473C11.8463 11.8637 11.3163 12.0832 10.7638 12.0832C10.2113 12.0832 9.68137 11.8637 9.29067 11.473C8.89997 11.0823 8.68048 10.5524 8.68048 9.99984C8.68048 9.4473 8.89997 8.9174 9.29067 8.5267C9.68137 8.136 10.2113 7.9165 10.7638 7.9165ZM10.7638 4.7915C13.8629 4.7915 16.5581 6.51198 17.9558 9.04935C18.0912 9.29504 18.1588 9.41789 18.2096 9.66644C18.243 9.83022 18.243 10.1695 18.2096 10.3332C18.1588 10.5818 18.0912 10.7046 17.9558 10.9503C16.5581 13.4877 13.8629 15.2082 10.7638 15.2082C7.66473 15.2082 4.9695 13.4877 3.5718 10.9503C3.43646 10.7046 3.36879 10.5818 3.31805 10.3332C3.28461 10.1695 3.28461 9.83022 3.31805 9.66644C3.36879 9.41789 3.43646 9.29504 3.5718 9.04935C4.9695 6.51198 7.66473 4.7915 10.7638 4.7915ZM5.32172 8.8889C5.13652 9.13408 5.04392 9.25667 4.9601 9.57335C4.90816 9.76956 4.90816 10.2301 4.9601 10.4263C5.04392 10.743 5.13652 10.8656 5.32172 11.1108C5.8227 11.774 6.44297 12.3431 7.15442 12.7868C8.23718 13.4622 9.4877 13.8202 10.7638 13.8202C12.0399 13.8202 13.2904 13.4622 14.3732 12.7868C15.0847 12.3431 15.7049 11.774 16.2059 11.1108C16.3911 10.8656 16.4837 10.743 16.5675 10.4263C16.6195 10.2301 16.6195 9.76956 16.5675 9.57335C16.4837 9.25667 16.3911 9.13408 16.2059 8.8889C15.7049 8.22567 15.0847 7.65661 14.3732 7.21285C13.2904 6.5375 12.0399 6.17948 10.7638 6.17948C9.4877 6.17948 8.23718 6.5375 7.15442 7.21285C6.44297 7.65661 5.8227 8.22567 5.32172 8.8889Z" />
+</svg>

packages/web/components/common/Icon/icons/core/app/logsLight.svg

@@ -1,14 +1,4 @@
-<?xml version="1.0" standalone="no"?>
-<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1692867122828"
-    class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="4021"
-    xmlns:xlink="http://www.w3.org/1999/xlink" width="64" height="64">
-    <path
-        d="M269.844659 81.4308h44.821057v166.626082h-44.821057zM677.140966 491.719232c52.335426 0 102.092273 19.937769 140.105639 56.13883 38.126482 36.31053 60.461599 85.284073 62.891788 137.900467 2.5056 54.276658-16.27424 106.280032-52.881549 146.431672-36.60731 40.15164-86.65972 63.643469-140.936379 66.150285-3.180653 0.147174-6.401444 0.221369-9.576016 0.221369-52.341508 0-102.102004-19.936552-140.114153-56.136398-38.126482-36.309314-60.461599-85.284073-62.891789-137.902899-2.5056-54.276658 16.27424-106.280032 52.88155-146.431672 36.60731-40.15164 86.65972-63.643469 140.936379-66.149069a208.122961 208.122961 0 0 1 9.576016-0.221369h0.008514m-0.00973-44.822274c-3.859355 0-7.746684 0.088791-11.642528 0.268805-136.951744 6.3236-242.847422 122.470346-236.525038 259.422091 6.143586 133.0559 115.942406 236.793842 247.779562 236.793842 3.859355 0 7.747901-0.088791 11.642529-0.268804 136.951744-6.322384 242.847422-122.470346 236.525037-259.422091-6.143586-133.057117-115.942406-236.798708-247.779562-236.793843z"
-        p-id="4022"></path>
-    <path
-        d="M490.264524 891.110734a272.361206 272.361206 0 0 1-32.682275-37.369937H180.453104c-20.912034 0-37.927007-17.013757-37.927007-37.92579v-590.263526c0-20.912034 17.013757-37.927007 37.927007-37.927007H732.799354c20.912034 0 37.925791 17.013757 37.925791 37.927007V441.15597a268.605238 268.605238 0 0 1 44.821057 21.463023V225.551481c0-45.70045-37.047614-82.746848-82.746848-82.746849H180.453104c-45.70045 0-82.746848 37.047614-82.746848 82.746849v590.263526c0 45.70045 37.047614 82.746848 82.746848 82.746848h317.980164a273.587248 273.587248 0 0 1-8.168744-7.451121z"
-        p-id="4023"></path>
-    <path
-        d="M770.725145 489.61623a225.243754 225.243754 0 0 1 44.821057 27.231985v-0.21407a225.182938 225.182938 0 0 0-44.821057-27.114003v0.096088zM812.590566 778.530212H646.820768V576.105667h44.821057v157.604704h120.948741zM209.55091 380.121489h498.255687v44.821057H209.55091zM600.682445 81.4308h44.821058v166.626082h-44.821058zM406.842623 712.17437H209.55091v44.821057h203.864657a272.351476 272.351476 0 0 1-6.572944-44.821057zM450.941192 546.147929H209.55091v44.821057h217.435038a268.707408 268.707408 0 0 1 23.955244-44.821057z"
-        p-id="4024"></path>
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20">
+    <path fill-rule="evenodd" clip-rule="evenodd"
+        d="M5.75002 2.14155C5.75002 1.68131 6.12312 1.30821 6.58335 1.30821C7.04359 1.30821 7.41669 1.68131 7.41669 2.14155V3.40877H12.3834V2.14155C12.3834 1.68131 12.7565 1.30821 13.2167 1.30821C13.6769 1.30821 14.05 1.68131 14.05 2.14155V3.41033C15.3274 3.41829 16.0523 3.46704 16.6264 3.76529C17.2048 4.06573 17.6764 4.53731 17.9768 5.11568C18.3334 5.80202 18.3334 6.70372 18.3334 8.5071V13.2351C18.3334 15.0385 18.3334 15.9402 17.9768 16.6265C17.6764 17.2049 17.2048 17.6764 16.6264 17.9769C15.9401 18.3334 15.0384 18.3334 13.235 18.3334H6.76502C4.96163 18.3334 4.05994 18.3334 3.3736 17.9769C2.79523 17.6764 2.32365 17.2049 2.02321 16.6265C1.66669 15.9402 1.66669 15.0385 1.66669 13.2351V8.5071C1.66669 6.70372 1.66669 5.80202 2.02321 5.11568C2.32365 4.53731 2.79523 4.06573 3.3736 3.76529C3.9171 3.48297 4.59563 3.42421 5.75002 3.41198V2.14155ZM12.3834 5.07544V5.82628C12.3834 6.28652 12.7565 6.65961 13.2167 6.65961C13.6769 6.65961 14.05 6.28652 14.05 5.82628V5.07711C14.5405 5.08022 14.9085 5.0891 15.2117 5.11428C15.6434 5.15014 15.7925 5.21023 15.8581 5.24432C16.1321 5.38663 16.3555 5.61001 16.4978 5.88397C16.5319 5.9496 16.592 6.09872 16.6278 6.53043C16.6653 6.9821 16.6667 7.57743 16.6667 8.5071V13.2351C16.6667 14.1647 16.6653 14.7601 16.6278 15.2117C16.592 15.6435 16.5319 15.7926 16.4978 15.8582C16.3555 16.1322 16.1321 16.3555 15.8581 16.4979C15.7925 16.532 15.6434 16.592 15.2117 16.6279C14.76 16.6654 14.1647 16.6667 13.235 16.6667H6.76502C5.83535 16.6667 5.24001 16.6654 4.78835 16.6279C4.35664 16.592 4.20751 16.532 4.14189 16.4979C3.86793 16.3555 3.64455 16.1322 3.50223 15.8582C3.46815 15.7926 3.40805 15.6435 3.3722 15.2117C3.33469 14.7601 3.33335 14.1647 3.33335 13.2351V8.5071C3.33335 7.57743 3.33469 6.9821 3.3722 6.53043C3.40805 6.09872 3.46815 5.9496 3.50223 5.88397C3.64455 5.61001 3.86793 5.38663 4.14189 5.24432C4.20751 5.21023 4.35664 5.15014 4.78835 5.11428C5.0489 5.09265 5.35726 5.08305 5.75002 5.07879V5.82628C5.75002 6.28652 6.12312 6.65961 6.58335 6.65961C7.04359 6.65961 7.41669 6.28652 7.41669 5.82628V5.07544H12.3834ZM14.5842 9.16667C15.0445 9.16667 15.4176 9.53976 15.4176 10C15.4176 10.4602 15.0445 10.8333 14.5842 10.8333H11.4014C10.9412 10.8333 10.5681 10.4602 10.5681 10C10.5681 9.53976 10.9412 9.16667 11.4014 9.16667H14.5842ZM15.4176 13.4352C15.4176 12.975 15.0445 12.6019 14.5842 12.6019H6.63052C6.17029 12.6019 5.79719 12.975 5.79719 13.4352C5.79719 13.8955 6.17029 14.2686 6.63052 14.2686H14.5842C15.0445 14.2686 15.4176 13.8955 15.4176 13.4352Z" />
 </svg>

packages/web/components/common/Icon/icons/core/app/questionGuide.svg

@@ -1,5 +1,5 @@
-<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
     <path fill-rule="evenodd" clip-rule="evenodd"
-        d="M10.8275 1.33325H5.17245C4.63581 1.33324 4.19289 1.33324 3.8321 1.36272C3.45737 1.39333 3.1129 1.45904 2.78934 1.6239C2.28758 1.87956 1.87963 2.28751 1.62397 2.78928C1.45911 3.11284 1.3934 3.4573 1.36278 3.83204C1.3333 4.19283 1.33331 4.63574 1.33332 5.17239L1.33328 9.42497C1.333 9.95523 1.33278 10.349 1.42418 10.6901C1.67076 11.6103 2.38955 12.3291 3.3098 12.5757C3.51478 12.6306 3.73878 12.6525 3.99998 12.6611L3.99998 13.5806C3.99995 13.7374 3.99992 13.8973 4.01182 14.0283C4.0232 14.1536 4.05333 14.3901 4.21844 14.5969C4.40843 14.8349 4.69652 14.9734 5.00106 14.973C5.26572 14.9728 5.46921 14.8486 5.57416 14.7792C5.6839 14.7066 5.80872 14.6067 5.93117 14.5087L7.53992 13.2217C7.88564 12.9451 7.98829 12.8671 8.09494 12.8126C8.20192 12.7579 8.3158 12.718 8.43349 12.6938C8.55081 12.6697 8.67974 12.6666 9.12248 12.6666H10.8275C11.3642 12.6666 11.8071 12.6666 12.1679 12.6371C12.5426 12.6065 12.8871 12.5408 13.2106 12.3759C13.7124 12.1203 14.1203 11.7123 14.376 11.2106C14.5409 10.887 14.6066 10.5425 14.6372 10.1678C14.6667 9.80701 14.6667 9.36411 14.6667 8.82747V5.17237C14.6667 4.63573 14.6667 4.19283 14.6372 3.83204C14.6066 3.4573 14.5409 3.11284 14.376 2.78928C14.1203 2.28751 13.7124 1.87956 13.2106 1.6239C12.8871 1.45904 12.5426 1.39333 12.1679 1.36272C11.8071 1.33324 11.3642 1.33324 10.8275 1.33325ZM8.99504 4.99992C8.99504 4.44763 9.44275 3.99992 9.99504 3.99992C10.5473 3.99992 10.995 4.44763 10.995 4.99992C10.995 5.5522 10.5473 5.99992 9.99504 5.99992C9.44275 5.99992 8.99504 5.5522 8.99504 4.99992ZM4.92837 7.79996C5.222 7.57974 5.63816 7.63837 5.85961 7.93051C5.90071 7.98295 5.94593 8.03229 5.99199 8.08035C6.09019 8.18282 6.23775 8.32184 6.42882 8.4608C6.81353 8.74059 7.3454 8.99996 7.99504 8.99996C8.64469 8.99996 9.17655 8.74059 9.56126 8.4608C9.75233 8.32184 9.89989 8.18282 9.99809 8.08035C10.0441 8.0323 10.0894 7.98294 10.1305 7.93051C10.3519 7.63837 10.7681 7.57974 11.0617 7.79996C11.3563 8.02087 11.416 8.43874 11.195 8.73329C11.1967 8.73112 11.1928 8.7361 11.186 8.74466C11.1697 8.7651 11.1372 8.80597 11.1261 8.81916C11.087 8.86575 11.0317 8.92884 10.9607 9.00289C10.8194 9.15043 10.6128 9.34474 10.3455 9.53912C9.81353 9.92599 9.01206 10.3333 7.99504 10.3333C6.97802 10.3333 6.17655 9.92599 5.64459 9.53912C5.37733 9.34474 5.17072 9.15043 5.02934 9.00289C4.95837 8.92884 4.90305 8.86575 4.86395 8.81916C4.84438 8.79585 4.82881 8.77659 4.81731 8.76207C4.58702 8.46455 4.61798 8.03275 4.92837 7.79996ZM5.99504 3.99992C5.44275 3.99992 4.99504 4.44763 4.99504 4.99992C4.99504 5.5522 5.44275 5.99992 5.99504 5.99992C6.54732 5.99992 6.99504 5.5522 6.99504 4.99992C6.99504 4.44763 6.54732 3.99992 5.99504 3.99992Z"
-        fill="#06AED4"></path>
+        d="M1.66675 7.97028C1.66675 6.0661 1.66675 5.11401 2.03733 4.38671C2.3633 3.74695 2.88343 3.22682 3.52318 2.90085C4.25048 2.53027 5.20257 2.53027 7.10675 2.53027H12.8934C14.7976 2.53027 15.7497 2.53027 16.477 2.90085C17.1167 3.22682 17.6369 3.74695 17.9628 4.38671C18.3334 5.11401 18.3334 6.06609 18.3334 7.97027V10.9413C18.3334 12.8455 18.3334 13.7976 17.9628 14.5249C17.6369 15.1647 17.1167 15.6848 16.477 16.0108C15.7497 16.3814 14.7976 16.3814 12.8934 16.3814H10.0744L7.42003 18.5835C6.99384 18.937 6.34598 18.7611 6.15717 18.2405L5.4777 16.3672C4.58619 16.3392 4.0053 16.2564 3.52318 16.0108C2.88343 15.6848 2.3633 15.1647 2.03733 14.5249C1.66675 13.7976 1.66675 12.8455 1.66675 10.9414V7.97028ZM6.38847 9.48176C6.98678 9.48176 7.4718 8.99673 7.4718 8.39843C7.4718 7.80012 6.98678 7.31509 6.38847 7.31509C5.79016 7.31509 5.30514 7.80012 5.30514 8.39843C5.30514 8.99673 5.79016 9.48176 6.38847 9.48176ZM14.695 8.39843C14.695 8.99673 14.21 9.48176 13.6117 9.48176C13.0134 9.48176 12.5284 8.99673 12.5284 8.39843C12.5284 7.80012 13.0134 7.31509 13.6117 7.31509C14.21 7.31509 14.695 7.80012 14.695 8.39843ZM12.7802 11.6057C13.0281 11.3091 12.9887 10.8677 12.692 10.6197C12.3954 10.3718 11.954 10.4112 11.706 10.7078C11.3434 11.1416 10.7759 11.4343 10.0383 11.4343C9.29561 11.4343 8.59499 11.0648 8.29232 10.7057C8.04317 10.4101 7.60156 10.3724 7.30595 10.6216C7.01034 10.8707 6.97268 11.3123 7.22183 11.6079C7.80834 12.3038 8.91134 12.8343 10.0383 12.8343C11.1704 12.8343 12.1389 12.3729 12.7802 11.6057Z"
+        fill="#00A9A6" />
 </svg>

packages/web/components/common/Icon/icons/core/app/simpleMode/ai.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
+    <path
+        d="M14.2733 5.04179H15.2083C16.9342 5.04179 18.3333 6.4409 18.3333 8.16676V12.3334C18.3333 14.0592 16.9342 15.4584 15.2083 15.4584H4.79163C3.06575 15.4584 1.66663 14.0592 1.66663 12.3334V8.16676C1.66663 6.4409 3.06575 5.04179 4.79163 5.04179H5.72663L5.34604 2.75844C5.26092 2.24773 5.60596 1.76469 6.11667 1.67957C6.62738 1.59445 7.11042 1.93948 7.19554 2.45019L7.61221 4.95017C7.61733 4.98083 7.62088 5.01142 7.62296 5.04179H12.377C12.379 5.01137 12.3826 4.98083 12.3877 4.95017L12.8044 2.45019C12.8895 1.93948 13.3725 1.59445 13.8833 1.67957C14.394 1.76469 14.739 2.24773 14.6539 2.75844L14.2733 5.04179ZM5.74996 9.41675V10.2501C5.74996 10.7678 6.16971 11.1876 6.68746 11.1876C7.20521 11.1876 7.62496 10.7678 7.62496 10.2501V9.41675C7.62496 8.899 7.20521 8.47926 6.68746 8.47926C6.16971 8.47926 5.74996 8.899 5.74996 9.41675ZM12.375 9.41675V10.2501C12.375 10.7678 12.7947 11.1876 13.3125 11.1876C13.8302 11.1876 14.25 10.7678 14.25 10.2501V9.41675C14.25 8.899 13.8302 8.47926 13.3125 8.47926C12.7947 8.47926 12.375 8.899 12.375 9.41675ZM5.83329 18.3333C5.31554 18.3333 4.89579 17.9136 4.89579 17.3958C4.89579 16.8781 5.31554 16.4584 5.83329 16.4584H14.1666C14.6844 16.4584 15.1041 16.8781 15.1041 17.3958C15.1041 17.9136 14.6844 18.3333 14.1666 18.3333H5.83329Z"
+        fill="#C172E7" />
+</svg>

packages/web/components/common/Icon/icons/core/app/simpleMode/chat.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
+    <path
+        d="M18.0527 6.60938C17.6133 5.56055 16.9805 4.61914 16.1758 3.81055C15.3711 3.00391 14.4316 2.36914 13.3828 1.92969C12.3105 1.47852 11.1738 1.25 9.99999 1.25H9.96093C8.77928 1.25586 7.63671 1.49023 6.56053 1.95117C5.52147 2.39648 4.58983 3.0293 3.79296 3.83594C2.99608 4.64258 2.37108 5.58008 1.93749 6.625C1.49022 7.70703 1.26366 8.85742 1.26952 10.041C1.27538 11.3965 1.5996 12.7422 2.20507 13.9453V16.9141C2.20507 17.4102 2.60741 17.8125 3.10155 17.8125H6.06639C7.26757 18.4199 8.60936 18.7441 9.96288 18.75H10.0039C11.1719 18.75 12.3027 18.5234 13.3691 18.0801C14.4121 17.6445 15.3496 17.0195 16.1523 16.2207C16.957 15.4219 17.5898 14.4883 18.0332 13.4473C18.4922 12.3691 18.7265 11.2227 18.7324 10.0391C18.7363 8.84961 18.5058 7.69531 18.0527 6.60938ZM6.10155 10.9375C5.58593 10.9375 5.166 10.5176 5.166 10C5.166 9.48242 5.58593 9.0625 6.10155 9.0625C6.61718 9.0625 7.0371 9.48242 7.0371 10C7.0371 10.5176 6.61913 10.9375 6.10155 10.9375ZM9.99999 10.9375C9.48436 10.9375 9.06444 10.5176 9.06444 10C9.06444 9.48242 9.48436 9.0625 9.99999 9.0625C10.5156 9.0625 10.9355 9.48242 10.9355 10C10.9355 10.5176 10.5156 10.9375 9.99999 10.9375ZM13.8984 10.9375C13.3828 10.9375 12.9629 10.5176 12.9629 10C12.9629 9.48242 13.3828 9.0625 13.8984 9.0625C14.414 9.0625 14.834 9.48242 14.834 10C14.834 10.5176 14.414 10.9375 13.8984 10.9375Z"
+        fill="#FF81AE" />
+</svg>

packages/web/components/common/Icon/icons/core/app/simpleMode/dataset.svg

@@ -0,0 +1,11 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
+    <path
+        d="M17.5958 5.1727V4.75464L17.5954 4.71548C17.5504 2.77983 14.167 1.66667 9.99994 1.66667C5.83264 1.66667 2.44869 2.76407 2.40405 4.71548V5.1727H2.40779C2.4053 5.20486 2.40405 5.2373 2.40405 5.27001C2.40405 6.73998 5.10661 7.93163 9.99994 7.93163C14.8933 7.93163 17.5958 6.73998 17.5958 5.27001C17.5958 5.2373 17.5946 5.20486 17.592 5.1727H17.5958Z"
+        fill="#8774EE" />
+    <path
+        d="M2.40405 7.46898V10.4345C2.40387 10.4415 2.40393 10.4484 2.40399 10.4554L2.40405 10.4659C2.40405 11.9342 5.10661 13.1244 9.99994 13.1244C14.8933 13.1244 17.5958 11.9342 17.5958 10.4659C17.5958 10.4554 17.5957 10.445 17.5954 10.4345L17.5958 7.46898H17.5436C17.1465 8.80742 14.4993 9.84075 9.99994 9.84075C5.50056 9.84075 2.85339 8.80742 2.45633 7.46898H2.40405Z"
+        fill="#8774EE" />
+    <path
+        d="M2.40405 12.3715V15.3524H2.40762C2.51854 17.0083 5.20939 18.3333 9.99994 18.3333C14.7905 18.3333 17.4813 17.0083 17.5923 15.3524H17.5958V12.3715C17.5958 13.9682 14.8933 15.2627 9.99994 15.2627C5.10661 15.2627 2.40405 13.9682 2.40405 12.3715Z"
+        fill="#8774EE" />
+</svg>

packages/web/components/common/Icon/icons/core/app/simpleMode/template.svg

@@ -0,0 +1,11 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
+    <path
+        d="M2.02994 3.4867C1.68907 4.1557 1.66801 5.01869 1.66671 6.6646H18.3332C18.3319 5.01869 18.3109 4.1557 17.97 3.4867C17.6504 2.85949 17.1405 2.34956 16.5133 2.02998C15.8002 1.66667 14.8668 1.66667 13 1.66667H6.99996C5.13312 1.66667 4.1997 1.66667 3.48666 2.02998C2.85945 2.34956 2.34952 2.85949 2.02994 3.4867Z"
+        fill="#8774EE" />
+    <path
+        d="M18.3333 8.33126H8.32478L8.32478 18.3333H13C14.8668 18.3333 15.8002 18.3333 16.5133 17.97C17.1405 17.6504 17.6504 17.1405 17.97 16.5133C18.3333 15.8003 18.3333 14.8668 18.3333 13V8.33126Z"
+        fill="#8774EE" />
+    <path
+        d="M6.65812 18.3332L6.65812 8.33126H1.66663V13C1.66663 14.8668 1.66663 15.8003 2.02994 16.5133C2.34952 17.1405 2.85945 17.6504 3.48666 17.97C4.15478 18.3104 5.0164 18.3319 6.65812 18.3332Z"
+        fill="#8774EE" />
+</svg>

packages/web/components/common/Icon/icons/core/app/simpleMode/tts.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="20" height="16" viewBox="0 0 22 18" fill="none">
+  <path fill-rule="evenodd" clip-rule="evenodd"
+    d="M4.63694 1.22278C5.02752 1.61324 5.02762 2.24641 4.63715 2.63699C2.94991 4.32474 2.00208 6.61352 2.00208 8.99999C2.00208 11.3865 2.94991 13.6752 4.63715 15.363C5.02762 15.7536 5.02752 16.3867 4.63694 16.7772C4.24636 17.1677 3.61319 17.1676 3.22273 16.777C1.16054 14.7142 0.0020752 11.9168 0.0020752 8.99999C0.0020752 6.08319 1.16054 3.2858 3.22273 1.22299C3.61319 0.832409 4.24636 0.832314 4.63694 1.22278ZM17.3629 1.22278C17.7535 0.832314 18.3867 0.832409 18.7772 1.22299C20.8393 3.2858 21.9978 6.08319 21.9978 8.99999C21.9978 11.9168 20.8393 14.7142 18.7772 16.777C18.3867 17.1676 17.7535 17.1677 17.3629 16.7772C16.9724 16.3867 16.9723 15.7536 17.3627 15.363C19.05 13.6752 19.9978 11.3865 19.9978 8.99999C19.9978 6.61352 19.05 4.32474 17.3627 2.63699C16.9723 2.24641 16.9724 1.61324 17.3629 1.22278ZM7.46744 4.04328C7.85775 4.43402 7.8574 5.06719 7.46665 5.45749C7.00177 5.92186 6.63298 6.4733 6.38135 7.08029C6.12973 7.68728 6.00022 8.33792 6.00022 8.99499C6.00022 9.65207 6.12973 10.3027 6.38135 10.9097C6.63298 11.5167 7.00177 12.0681 7.46665 12.5325C7.8574 12.9228 7.85775 13.556 7.46744 13.9467C7.07713 14.3374 6.44397 14.3378 6.05323 13.9475C5.40239 13.2974 4.88608 12.5254 4.53381 11.6756C4.18154 10.8258 4.00022 9.9149 4.00022 8.99499C4.00022 8.07508 4.18154 7.1642 4.53381 6.31441C4.88608 5.46462 5.40239 4.6926 6.05323 4.04249C6.44397 3.65219 7.07713 3.65254 7.46744 4.04328ZM14.5324 4.05328C14.9227 3.66254 15.5559 3.66219 15.9467 4.05249C16.5975 4.7026 17.1138 5.47462 17.4661 6.32441C17.8183 7.1742 17.9997 8.08509 17.9997 9.00499C17.9997 9.9249 17.8183 10.8358 17.4661 11.6856C17.1138 12.5354 16.5975 13.3074 15.9467 13.9575C15.5559 14.3478 14.9227 14.3474 14.5324 13.9567C14.1421 13.566 14.1425 12.9328 14.5332 12.5425C14.9981 12.0781 15.3669 11.5267 15.6185 10.9197C15.8701 10.3127 15.9997 9.66207 15.9997 9.00499C15.9997 8.34792 15.8701 7.69728 15.6185 7.09029C15.3669 6.4833 14.9981 5.93186 14.5332 5.46749C14.1425 5.07719 14.1421 4.44402 14.5324 4.05328ZM10.9999 7.99999C10.4477 7.99999 9.99994 8.44771 9.99994 8.99999C9.99994 9.55228 10.4477 9.99999 10.9999 9.99999C11.5522 9.99999 11.9999 9.55228 11.9999 8.99999C11.9999 8.44771 11.5522 7.99999 10.9999 7.99999ZM7.99994 8.99999C7.99994 7.34314 9.34309 5.99999 10.9999 5.99999C12.6568 5.99999 13.9999 7.34314 13.9999 8.99999C13.9999 10.6568 12.6568 12 10.9999 12C9.34309 12 7.99994 10.6568 7.99994 8.99999Z"
+    fill="#3370FF" />
+</svg>