V4.8.15 feature (#3331)

* feat: add customize toolkit (#3205) * chaoyang * fix-auth * add toolkit * add order * plugin usage * fix * delete console: * Fix: Fix fullscreen preview top positioning and improve Markdown rendering logic (#3247) * 完成任务：修复全屏预览顶部固定问题，优化 Markdown 渲染逻辑 * 有问题修改 * 问题再修改 * 修正问题 * fix: plugin standalone display issue (#3254) * 4.8.15 test (#3246) * o1 config * perf: system plugin code * 调整系统插件代码。增加html 渲染安全配置。 (#3258) * perf: base64 picker * perf: list app or dataset * perf: plugin config code * 小窗适配等问题 (#3257) * 小窗适配等问题 * git问题 * 小窗剩余问题 * feat: system plugin auth and lock version (#3265) * feat: system plugin auth and lock version * update comment * 4.8.15 test (#3267) * tmp log * perf: login direct * perf: iframe html code * remove log * fix: plugin standalone display (#3277) * refactor: 页面拆分&i18n拆分 (#3281) * refactor: account组件拆成独立页面 * script: 新增i18n json文件创建脚本 * refactor: 页面i18n拆分 * i18n: add en&hant * 4.8.15 test (#3285) * tmp log * remove log * fix: watch avatar refresh * perf: i18n code * fix(plugin): use intro instead of userguide (#3290) * Universal SSO (#3292) * tmp log * remove log * feat: common oauth * readme * perf: sso provider * remove sso code * perf: refresh plugins * feat: add api dataset (#3272) * add api-dataset * fix api-dataset * fix api dataset * fix ts * perf: create collection code (#3301) * tmp log * remove log * perf: i18n change * update version doc * feat: question guide from chatId * perf: create collection code * fix: request api * fix: request api * fix: tts auth and response type (#3303) * perf: md splitter * fix: tts auth and response type * fix: api file dataset (#3307) * perf: api dataset init (#3310) * perf: collection schema * perf: api dataset init * refactor: 团队管理独立页面 (#3302) * ui: 团队管理独立页面 * 代码优化 * fix * perf: sync collection and ui check (#3314) * perf: sync collection * remove script * perf: update api server * perf: api dataset parent * perf: team ui * perf: team 18n * update team ui * perf: ui check * perf: i18n * fix: debug variables & cronjob & system plugin callback load (#3315) * fix: debug variables & cronjob & system plugin callback load * fix type * fix * fix * fix: plugin dataset quote;perf: system variables init (#3316) * fix: plugin dataset quote * perf: system variables init * perf: node templates ui;fix: dataset import ui (#3318) * fix: dataset import ui * perf: node templates ui * perf: ui refresh * feat:套餐改名和套餐跳转配置 (#3309) * fixing:except Sidebar * 去除了多余的代码 * 修正了套餐说明的代码 * 修正了误删除的show_git代码 * 修正了名字部分等代码 * 修正了问题,遗留了其他和ui讨论不一致的部分 * 4.8.15 test (#3319) * remove log * pref: bill ui * pref: bill ui * perf: log * html渲染文档 (#3270) * html渲染文档 * 文档有点小问题 * feat: doc (#3322) * 集合重训练 (#3282) * rebaser * 一点补充 * 小问题 * 其他问题修正，删除集合保留文件的参数还没找到... * reTraining * delete uesless * 删除了一行错误代码 * 集合重训练部分 * fixing * 删除console代码 * feat: navbar item config (#3326) * perf: custom navbar code;perf: retraining code;feat: api dataset and dataset api doc (#3329) * feat: api dataset and dataset api doc * perf: retraining code * perf: custom navbar code * fix: ts (#3330) * fix: ts * fix: ts * retraining ui * perf: api collection filter * perf: retrining button --------- Co-authored-by: heheer <heheer@sealos.io> Co-authored-by: Jiangween <145003935+Jiangween@users.noreply.github.com> Co-authored-by: papapatrick <109422393+Patrickill@users.noreply.github.com>
2025-08-01 03:48:24 +00:00 · 2024-12-06 10:56:53 +08:00
parent ed95a21c96
commit 6aed7b238f
307 changed files with 7383 additions and 3981 deletions
--- a/packages/service/core/dataset/collection/controller.ts
+++ b/packages/service/core/dataset/collection/controller.ts
@@ -3,7 +3,8 @@ import type { CreateDatasetCollectionParams } from '@fastgpt/global/core/dataset
 import { MongoDatasetCollection } from './schema';
 import {
  CollectionWithDatasetType,
-  DatasetCollectionSchemaType
+  DatasetCollectionSchemaType,
+  DatasetSchemaType
 } from '@fastgpt/global/core/dataset/type';
 import { MongoDatasetTraining } from '../training/schema';
 import { MongoDatasetData } from '../data/schema';
@@ -13,7 +14,132 @@ import { delFileByFileIdList } from '../../../common/file/gridfs/controller';
 import { BucketNameEnum } from '@fastgpt/global/common/file/constants';
 import { ClientSession } from '../../../common/mongo';
 import { createOrGetCollectionTags } from './utils';
+import { rawText2Chunks } from '../read';
+import { checkDatasetLimit } from '../../../support/permission/teamLimit';
+import { predictDataLimitLength } from '../../../../global/core/dataset/utils';
+import { mongoSessionRun } from '../../../common/mongo/sessionRun';
+import { createTrainingUsage } from '../../../support/wallet/usage/controller';
+import { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
+import { getLLMModel, getVectorModel } from '../../ai/model';
+import { pushDataListToTrainingQueue } from '../training/controller';
+import { MongoImage } from '../../../common/file/image/schema';
+import { hashStr } from '@fastgpt/global/common/string/tools';

+export const createCollectionAndInsertData = async ({
+  dataset,
+  rawText,
+  relatedId,
+  createCollectionParams,
+  isQAImport = false,
+  session
+}: {
+  dataset: DatasetSchemaType;
+  rawText: string;
+  relatedId?: string;
+  createCollectionParams: CreateOneCollectionParams;
+
+  isQAImport?: boolean;
+  session?: ClientSession;
+}) => {
+  const teamId = createCollectionParams.teamId;
+  const tmbId = createCollectionParams.tmbId;
+  // Chunk split params
+  const trainingType = createCollectionParams.trainingType || TrainingModeEnum.chunk;
+  const chunkSize = createCollectionParams.chunkSize;
+  const chunkSplitter = createCollectionParams.chunkSplitter;
+  const qaPrompt = createCollectionParams.qaPrompt;
+  const usageName = createCollectionParams.name;
+
+  // 1. split chunks
+  const chunks = rawText2Chunks({
+    rawText,
+    chunkLen: chunkSize,
+    overlapRatio: trainingType === TrainingModeEnum.chunk ? 0.2 : 0,
+    customReg: chunkSplitter ? [chunkSplitter] : [],
+    isQAImport
+  });
+
+  // 2. auth limit
+  await checkDatasetLimit({
+    teamId,
+    insertLen: predictDataLimitLength(trainingType, chunks)
+  });
+
+  const fn = async (session: ClientSession) => {
+    // 3. create collection
+    const { _id: collectionId } = await createOneCollection({
+      ...createCollectionParams,
+
+      hashRawText: hashStr(rawText),
+      rawTextLength: rawText.length,
+      session
+    });
+
+    // 4. create training bill
+    const { billId } = await createTrainingUsage({
+      teamId,
+      tmbId,
+      appName: usageName,
+      billSource: UsageSourceEnum.training,
+      vectorModel: getVectorModel(dataset.vectorModel)?.name,
+      agentModel: getLLMModel(dataset.agentModel)?.name,
+      session
+    });
+
+    // 5. insert to training queue
+    const insertResults = await pushDataListToTrainingQueue({
+      teamId,
+      tmbId,
+      datasetId: dataset._id,
+      collectionId,
+      agentModel: dataset.agentModel,
+      vectorModel: dataset.vectorModel,
+      trainingMode: trainingType,
+      prompt: qaPrompt,
+      billId,
+      data: chunks.map((item, index) => ({
+        ...item,
+        chunkIndex: index
+      })),
+      session
+    });
+
+    // 6. remove related image ttl
+    if (relatedId) {
+      await MongoImage.updateMany(
+        {
+          teamId,
+          'metadata.relatedId': relatedId
+        },
+        {
+          // Remove expiredTime to avoid ttl expiration
+          $unset: {
+            expiredTime: 1
+          }
+        },
+        {
+          session
+        }
+      );
+    }
+
+    return {
+      collectionId,
+      insertResults
+    };
+  };
+
+  if (session) {
+    return fn(session);
+  }
+  return mongoSessionRun(fn);
+};
+
+export type CreateOneCollectionParams = CreateDatasetCollectionParams & {
+  teamId: string;
+  tmbId: string;
+  session?: ClientSession;
+};
 export async function createOneCollection({
  teamId,
  tmbId,
@@ -33,18 +159,15 @@ export async function createOneCollection({
  externalFileId,
  externalFileUrl,

+  apiFileId,
+
  hashRawText,
  rawTextLength,
  metadata = {},
  session,
  tags,
-  ...props
-}: CreateDatasetCollectionParams & {
-  teamId: string;
-  tmbId: string;
-  [key: string]: any;
-  session?: ClientSession;
-}) {
+  createTime
+}: CreateOneCollectionParams) {
  // Create collection tags
  const collectionTags = await createOrGetCollectionTags({ tags, teamId, datasetId, session });

@@ -52,7 +175,6 @@ export async function createOneCollection({
  const [collection] = await MongoDatasetCollection.create(
    [
      {
-        ...props,
        teamId,
        tmbId,
        parentId: parentId || null,
@@ -64,16 +186,18 @@ export async function createOneCollection({
        chunkSize,
        chunkSplitter,
        qaPrompt,
+        metadata,

-        fileId,
-        rawLink,
+        ...(fileId ? { fileId } : {}),
+        ...(rawLink ? { rawLink } : {}),
        ...(externalFileId ? { externalFileId } : {}),
-        externalFileUrl,
+        ...(externalFileUrl ? { externalFileUrl } : {}),
+        ...(apiFileId ? { apiFileId } : {}),

        rawTextLength,
        hashRawText,
-        metadata,
-        tags: collectionTags
+        tags: collectionTags,
+        createTime
      }
    ],
    { session }
@@ -116,7 +240,68 @@ export const delCollectionRelatedSource = async ({
 /**
 * delete collection and it related data
 */
-export async function delCollectionAndRelatedSources({
+export async function delCollection({
+  collections,
+  session,
+  delRelatedSource
+}: {
+  collections: (CollectionWithDatasetType | DatasetCollectionSchemaType)[];
+  session: ClientSession;
+  delRelatedSource: boolean;
+}) {
+  if (collections.length === 0) return;
+
+  const teamId = collections[0].teamId;
+
+  if (!teamId) return Promise.reject('teamId is not exist');
+
+  const datasetIds = Array.from(
+    new Set(
+      collections.map((item) => {
+        if (typeof item.datasetId === 'string') {
+          return String(item.datasetId);
+        }
+        return String(item.datasetId._id);
+      })
+    )
+  );
+  const collectionIds = collections.map((item) => String(item._id));
+
+  // delete training data
+  await MongoDatasetTraining.deleteMany({
+    teamId,
+    datasetIds: { $in: datasetIds },
+    collectionId: { $in: collectionIds }
+  });
+
+  /* file and imgs */
+  if (delRelatedSource) {
+    await delCollectionRelatedSource({ collections, session });
+  }
+
+  // delete dataset.datas
+  await MongoDatasetData.deleteMany(
+    { teamId, datasetIds: { $in: datasetIds }, collectionId: { $in: collectionIds } },
+    { session }
+  );
+
+  // delete collections
+  await MongoDatasetCollection.deleteMany(
+    {
+      teamId,
+      _id: { $in: collectionIds }
+    },
+    { session }
+  );
+
+  // no session delete: delete files, vector data
+  await deleteDatasetDataVector({ teamId, datasetIds, collectionIds });
+}
+
+/**
+ * delete delOnlyCollection
+ */
+export async function delOnlyCollection({
  collections,
  session
 }: {
@@ -148,9 +333,6 @@ export async function delCollectionAndRelatedSources({
    collectionId: { $in: collectionIds }
  });

-  /* file and imgs */
-  await delCollectionRelatedSource({ collections, session });
-
  // delete dataset.datas
  await MongoDatasetData.deleteMany(
    { teamId, datasetIds: { $in: datasetIds }, collectionId: { $in: collectionIds } },