V4.9.4 feature (#4470)

* Training status (#4424) * dataset data training state (#4311) * dataset data training state * fix * fix ts * fix * fix api format * fix * fix * perf: count training * format * fix: dataset training state (#4417) * fix * add test * fix * fix * fix test * fix test * perf: training count * count * loading status --------- Co-authored-by: heheer <heheer@sealos.io> * doc * website sync feature (#4429) * perf: introduce BullMQ for website sync (#4403) * perf: introduce BullMQ for website sync * feat: new redis module * fix: remove graceful shutdown * perf: improve UI in dataset detail - Updated the "change" icon SVG file. - Modified i18n strings. - Added new i18n string "immediate_sync". - Improved UI in dataset detail page, including button icons and background colors. * refactor: Add chunkSettings to DatasetSchema * perf: website sync ux * env template * fix: clean up website dataset when updating chunk settings (#4420) * perf: check setting updated * perf: worker currency * feat: init script for website sync refactor (#4425) * website feature doc --------- Co-authored-by: a.e. <49438478+I-Info@users.noreply.github.com> * pro migration (#4388) (#4433) * pro migration * reuse customPdfParseType Co-authored-by: gggaaallleee <91131304+gggaaallleee@users.noreply.github.com> * perf: remove loading ui * feat: config chat file expired time * Redis cache (#4436) * perf: add Redis cache for vector counting (#4432) * feat: cache * perf: get cache key --------- Co-authored-by: a.e. <49438478+I-Info@users.noreply.github.com> * perf: mobile voice input (#4437) * update:Mobile voice interaction (#4362) * Add files via upload * Add files via upload * Update ollama.md * Update ollama.md * Add files via upload * Update useSpeech.ts * Update ChatInput.tsx * Update useSpeech.ts * Update ChatInput.tsx * Update useSpeech.ts * Update constants.ts * Add files via upload * Update ChatInput.tsx * Update useSpeech.ts * Update useSpeech.ts * Update useSpeech.ts * Update ChatInput.tsx * Add files via upload * Update common.json * Update VoiceInput.tsx * Update ChatInput.tsx * Update VoiceInput.tsx * Update useSpeech.ts * Update useSpeech.ts * Update common.json * Update common.json * Update common.json * Update VoiceInput.tsx * Update VoiceInput.tsx * Update ChatInput.tsx * Update VoiceInput.tsx * Update ChatInput.tsx * Update VoiceInput.tsx * Update ChatInput.tsx * Update useSpeech.ts * Update common.json * Update chat.json * Update common.json * Update chat.json * Update common.json * Update chat.json * Update VoiceInput.tsx * Update ChatInput.tsx * Update useSpeech.ts * Update VoiceInput.tsx * speech ui * 优化语音输入组件，调整输入框显示逻辑，修复语音输入遮罩层样式，更新画布背景透明度，增强用户交互体验。 (#4435) * perf: mobil voice input --------- Co-authored-by: dreamer6680 <1468683855@qq.com> * Test completion v2 (#4438) * add v2 completions (#4364) * add v2 completions * completion config * config version * fix * frontend * doc * fix * fix: completions v2 api --------- Co-authored-by: heheer <heheer@sealos.io> * package * Test mongo log (#4443) * feat: mongodb-log (#4426) * perf: mongo log * feat: completions stop reasoner * mongo db log --------- Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com> * update doc * Update doc * fix external var ui (#4444) * action * fix: ts (#4458) * preview doc action add docs preview permission update preview action udpate action * update doc (#4460) * update preview action * update doc * remove * update * schema * update mq export;perf: redis cache (#4465) * perf: redis cache * update mq export * perf: website sync error tip * add error worker * website sync ui (#4466) * Updated the dynamic display of the voice input pop-up (#4469) * Update VoiceInput.tsx * Update VoiceInput.tsx * Update VoiceInput.tsx * fix: voice input --------- Co-authored-by: heheer <heheer@sealos.io> Co-authored-by: a.e. <49438478+I-Info@users.noreply.github.com> Co-authored-by: gggaaallleee <91131304+gggaaallleee@users.noreply.github.com> Co-authored-by: dreamer6680 <1468683855@qq.com> Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
2025-10-15 15:41:05 +00:00 · 2025-04-08 12:05:04 +08:00
parent 5839325f77
commit f642c9603b
151 changed files with 5434 additions and 1354 deletions
--- a/packages/global/core/ai/constants.ts
+++ b/packages/global/core/ai/constants.ts
@@ -1,3 +1,5 @@
+import { i18nT } from '../../../web/i18n/utils';
+
 export enum ChatCompletionRequestMessageRoleEnum {
  'System' = 'system',
  'User' = 'user',
@@ -28,3 +30,13 @@ export enum EmbeddingTypeEnm {
  query = 'query',
  db = 'db'
 }
+
+export const completionFinishReasonMap = {
+  close: i18nT('chat:completion_finish_close'),
+  stop: i18nT('chat:completion_finish_stop'),
+  length: i18nT('chat:completion_finish_length'),
+  tool_calls: i18nT('chat:completion_finish_tool_calls'),
+  content_filter: i18nT('chat:completion_finish_content_filter'),
+  function_call: i18nT('chat:completion_finish_function_call'),
+  null: i18nT('chat:completion_finish_null')
+};
--- a/packages/global/core/ai/type.d.ts
+++ b/packages/global/core/ai/type.d.ts
@@ -73,6 +73,15 @@ export type ChatCompletionMessageFunctionCall =
 export type StreamChatType = Stream<openai.Chat.Completions.ChatCompletionChunk>;
 export type UnStreamChatType = openai.Chat.Completions.ChatCompletion;

+export type CompletionFinishReason =
+  | 'close'
+  | 'stop'
+  | 'length'
+  | 'tool_calls'
+  | 'content_filter'
+  | 'function_call'
+  | null;
+
 export default openai;
 export * from 'openai';

--- a/packages/global/core/dataset/api.d.ts
+++ b/packages/global/core/dataset/api.d.ts
@@ -15,7 +15,6 @@ export type DatasetUpdateBody = {
  name?: string;
  avatar?: string;
  intro?: string;
-  status?: DatasetSchemaType['status'];

  agentModel?: string;
  vlmModel?: string;
@@ -26,6 +25,7 @@ export type DatasetUpdateBody = {
  apiServer?: DatasetSchemaType['apiServer'];
  yuqueServer?: DatasetSchemaType['yuqueServer'];
  feishuServer?: DatasetSchemaType['feishuServer'];
+  chunkSettings?: DatasetSchemaType['chunkSettings'];

  // sync schedule
  autoSync?: boolean;
@@ -141,7 +141,6 @@ export type PushDatasetDataChunkProps = {

 export type PostWebsiteSyncParams = {
  datasetId: string;
-  billId: string;
 };

 export type PushDatasetDataProps = {
--- a/packages/global/core/dataset/constants.ts
+++ b/packages/global/core/dataset/constants.ts
@@ -50,7 +50,9 @@ export const DatasetTypeMap = {

 export enum DatasetStatusEnum {
  active = 'active',
-  syncing = 'syncing'
+  syncing = 'syncing',
+  waiting = 'waiting',
+  error = 'error'
 }
 export const DatasetStatusMap = {
  [DatasetStatusEnum.active]: {
@@ -58,6 +60,12 @@ export const DatasetStatusMap = {
  },
  [DatasetStatusEnum.syncing]: {
    label: i18nT('common:core.dataset.status.syncing')
+  },
+  [DatasetStatusEnum.waiting]: {
+    label: i18nT('common:core.dataset.status.waiting')
+  },
+  [DatasetStatusEnum.error]: {
+    label: i18nT('dataset:status_error')
  }
 };

--- a/packages/global/core/dataset/type.d.ts
+++ b/packages/global/core/dataset/type.d.ts
@@ -17,6 +17,20 @@ import { SourceMemberType } from 'support/user/type';
 import { DatasetDataIndexTypeEnum } from './data/constants';
 import { ChunkSettingModeEnum } from './constants';

+export type ChunkSettingsType = {
+  trainingType: DatasetCollectionDataProcessModeEnum;
+  autoIndexes?: boolean;
+  imageIndex?: boolean;
+
+  chunkSettingMode?: ChunkSettingModeEnum;
+  chunkSplitMode?: DataChunkSplitModeEnum;
+
+  chunkSize?: number;
+  indexSize?: number;
+  chunkSplitter?: string;
+  qaPrompt?: string;
+};
+
 export type DatasetSchemaType = {
  _id: string;
  parentId?: string;
@@ -29,7 +43,6 @@ export type DatasetSchemaType = {
  name: string;
  intro: string;
  type: `${DatasetTypeEnum}`;
-  status: `${DatasetStatusEnum}`;

  vectorModel: string;
  agentModel: string;
@@ -39,14 +52,16 @@ export type DatasetSchemaType = {
    url: string;
    selector: string;
  };
+
+  chunkSettings?: ChunkSettingsType;
+
  inheritPermission: boolean;
  apiServer?: APIFileServer;
  feishuServer?: FeishuServer;
  yuqueServer?: YuqueServer;

-  autoSync?: boolean;
-
  // abandon
+  autoSync?: boolean;
  externalReadUrl?: string;
  defaultPermission?: number;
 };
@@ -163,6 +178,7 @@ export type DatasetTrainingSchemaType = {
  weight: number;
  indexes: Omit<DatasetDataIndexItemType, 'dataId'>[];
  retryCount: number;
+  errorMsg?: string;
 };

 export type CollectionWithDatasetType = DatasetCollectionSchemaType & {
@@ -192,6 +208,8 @@ export type DatasetListItemType = {
 };

 export type DatasetItemType = Omit<DatasetSchemaType, 'vectorModel' | 'agentModel' | 'vlmModel'> & {
+  status: `${DatasetStatusEnum}`;
+  errorMsg?: string;
  vectorModel: EmbeddingModelItemType;
  agentModel: LLMModelItemType;
  vlmModel?: LLMModelItemType;
@@ -216,6 +234,7 @@ export type DatasetCollectionItemType = CollectionWithDatasetType & {
  file?: DatasetFileSchema;
  permission: DatasetPermission;
  indexAmount: number;
+  errorCount?: number;
 };

 /* ================= data ===================== */
--- a/packages/global/core/workflow/runtime/constants.ts
+++ b/packages/global/core/workflow/runtime/constants.ts
@@ -5,6 +5,7 @@ export enum SseResponseEventEnum {
  answer = 'answer', // animation stream
  fastAnswer = 'fastAnswer', // direct answer text, not animation
  flowNodeStatus = 'flowNodeStatus', // update node status
+  flowNodeResponse = 'flowNodeResponse', // node response

  toolCall = 'toolCall', // tool start
  toolParams = 'toolParams', // tool params return
--- a/packages/global/core/workflow/runtime/type.d.ts
+++ b/packages/global/core/workflow/runtime/type.d.ts
@@ -22,6 +22,7 @@ import { UserSelectOptionType } from '../template/system/userSelect/type';
 import { WorkflowResponseType } from '../../../../service/core/workflow/dispatch/type';
 import { AiChatQuoteRoleType } from '../template/system/aiChat/type';
 import { LafAccountType, OpenaiAccountType } from '../../../support/user/team/type';
+import { CompletionFinishReason } from '../../ai/type';

 export type ExternalProviderType = {
  openaiAccount?: OpenaiAccountType;
@@ -59,6 +60,7 @@ export type ChatDispatchProps = {
  isToolCall?: boolean;
  workflowStreamResponse?: WorkflowResponseType;
  workflowDispatchDeep?: number;
+  version?: 'v1' | 'v2';
 };

 export type ModuleDispatchProps<T> = ChatDispatchProps & {
@@ -129,6 +131,7 @@ export type DispatchNodeResponseType = {
    obj: `${ChatRoleEnum}`;
    value: string;
  }[]; // completion context array. history will slice
+  finishReason?: CompletionFinishReason;

  // dataset search
  similarity?: number;