V4.9.11 feature (#4969)

* Feat: Images dataset collection (#4941) * New pic (#4858) * 更新数据集相关类型，添加图像文件ID和预览URL支持；优化数据集导入功能，新增图像数据集处理组件；修复部分国际化文本；更新文件上传逻辑以支持新功能。 * 与原先代码的差别 * 新增 V4.9.10 更新说明，支持 PG 设置`systemEnv.hnswMaxScanTuples`参数，优化 LLM stream 调用超时，修复全文检索多知识库排序问题。同时更新数据集索引，移除 datasetId 字段以简化查询。 * 更换成fileId_image逻辑，并增加训练队列匹配的逻辑 * 新增图片集合判断逻辑，优化预览URL生成流程，确保仅在数据集为图片集合时生成预览URL，并添加相关日志输出以便调试。 * Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages. * Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API. * Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook. * Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets. * Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process. * 图片上传到新建的 dataset_collection_images 表，逻辑跟随更改 * 修改了除了controller的其他部分问题 * 把图片数据集的逻辑整合到controller里面 * 补充i18n * 补充i18n * resolve评论：主要是上传逻辑的更改和组件复用 * 图片名称的图标显示 * 修改编译报错的命名问题 * 删除不需要的collectionid部分 * 多余文件的处理和改动一个删除按钮 * 除了loading和统一的imageId,其他都resolve掉的 * 处理图标报错 * 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId * 去除不必要文件修改 * 报错和字段修改 * 增加上传成功后删除临时文件的逻辑以及回退一些修改 * 删除path字段，将图片保存到gridfs内，并修改增删等操作的代码 * 修正编译错误 --------- Co-authored-by: archer <545436317@qq.com> * perf: image dataset * feat: insert image * perf: image icon * fix: training state --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> * fix: ts (#4948) * Thirddatasetmd (#4942) * add thirddataset.md * fix thirddataset.md * fix * delete wrong png --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: api dataset code * perf: log * add secondary.tsx (#4946) * add secondary.tsx * fix --------- Co-authored-by: dreamer6680 <146868355@qq.com> * perf: multiple menu * perf: i18n * feat: parse queue (#4960) * feat: parse queue * feat: sync parse queue * fix thirddataset.md (#4962) * fix thirddataset-4.png (#4963) * feat: Dataset template import (#4934) * 模版导入部分除了文档还没写 * 修复模版导入的 build 错误 * Document production * compress pictures * Change some constants to variables --------- Co-authored-by: Archer <545436317@qq.com> * perf: template import * doc * llm pargraph * bocha tool * fix: del collection --------- Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com> Co-authored-by: dreamer6680 <1468683855@qq.com> Co-authored-by: dreamer6680 <146868355@qq.com>
2025-07-28 09:03:53 +00:00 · 2025-06-06 14:48:44 +08:00
parent bb810a43a1
commit c30f069f2f
198 changed files with 4934 additions and 2290 deletions
--- a/packages/web/i18n/zh-Hant/dataset.json
+++ b/packages/web/i18n/zh-Hant/dataset.json
@@ -8,11 +8,11 @@
  "auto_indexes_tips": "透過大模型進行額外索引生成，提高語義豐富度，提高檢索的精度。",
  "auto_training_queue": "增強索引排隊",
  "backup_collection": "備份數據",
-  "backup_data_uploading": "備份數據上傳中: {{num}}%",
  "backup_dataset": "備份導入",
  "backup_dataset_success": "備份創建成功",
  "backup_dataset_tip": "可以將導出知識庫時，下載的 csv 文件重新導入。",
  "backup_mode": "備份導入",
+  "backup_template_invalid": "備份文件格式不正確，應該是首列為 q,a,indexes 的 csv 文件",
  "chunk_max_tokens": "分塊上限",
  "chunk_process_params": "分塊處理參數",
  "chunk_size": "分塊大小",
@@ -26,16 +26,21 @@
  "collection.training_type": "處理模式",
  "collection_data_count": "資料量",
  "collection_metadata_custom_pdf_parse": "PDF 增強解析",
+  "collection_name": "數據集名稱",
  "collection_not_support_retraining": "此集合類型不支援重新調整參數",
  "collection_not_support_sync": "該集合不支援同步",
  "collection_sync": "立即同步",
  "collection_sync_confirm_tip": "確認開始同步資料？\n系統將會拉取最新資料進行比較，如果內容不相同，則會建立一個新的集合並刪除舊的集合，請確認！",
  "collection_tags": "集合標籤",
+  "common.dataset.data.Input Error Tip": "[圖片數據集] 處理過程錯誤:",
+  "common.error.unKnow": "未知錯誤",
  "common_dataset": "通用資料集",
  "common_dataset_desc": "通過導入文件、網頁鏈接或手動錄入形式構建知識庫",
  "condition": "條件",
  "config_sync_schedule": "設定定時同步",
+  "confirm_import_images": "共 {{num}} 張圖片 | 確認創建",
  "confirm_to_rebuild_embedding_tip": "確定要為資料集切換索引嗎？\n切換索引是一個重要的操作，需要對您資料集內所有資料重新建立索引，可能需要較長時間，請確保帳號內剩餘點數充足。\n\n此外，您還需要注意修改使用此資料集的應用程式，避免與其他索引模型資料集混用。",
+  "core.dataset.Image collection": "圖片數據集",
  "core.dataset.import.Adjust parameters": "調整參數",
  "custom_data_process_params": "自訂",
  "custom_data_process_params_desc": "自訂資料處理規則",
@@ -44,8 +49,10 @@
  "data_error_amount": "{{errorAmount}} 組訓練異常",
  "data_index_image": "圖片索引",
  "data_index_num": "索引 {{index}}",
+  "data_parsing": "數據解析中",
  "data_process_params": "處理參數",
  "data_process_setting": "資料處理設定",
+  "data_uploading": "數據上傳中: {{num}}%",
  "dataset.Chunk_Number": "分塊號",
  "dataset.Completed": "完成",
  "dataset.Delete_Chunk": "刪除",
@@ -65,7 +72,9 @@
  "dataset.no_tags": "尚無標籤",
  "default_params": "預設",
  "default_params_desc": "使用系統預設的參數和規則",
+  "download_csv_template": "點擊下載 CSV 模板",
  "edit_dataset_config": "編輯知識庫設定",
+  "empty_collection": "空白數據集",
  "enhanced_indexes": "索引增強",
  "error.collectionNotFound": "找不到集合",
  "external_file": "外部檔案庫",
@@ -88,6 +97,7 @@
  "image_auto_parse": "圖片自動索引",
  "image_auto_parse_tips": "呼叫 VLM 自動標註文件裡的圖片，並生成額外的檢索索引",
  "image_training_queue": "圖片處理排隊",
+  "images_creating": "正在創建",
  "immediate_sync": "立即同步",
  "import.Auto mode Estimated Price Tips": "需呼叫文字理解模型，將消耗較多 AI 點數：{{price}} 點數 / 1K tokens",
  "import.Embedding Estimated Price Tips": "僅使用索引模型，消耗少量 AI 點數：{{price}} 點數 / 1K tokens",
@@ -102,6 +112,8 @@
  "index_size": "索引大小",
  "index_size_tips": "向量化時內容的長度，系統會自動按該大小對分塊進行進一步的分割。",
  "input_required_field_to_select_baseurl": "請先輸入必填信息",
+  "insert_images": "新增圖片",
+  "insert_images_success": "新增圖片成功，需等待訓練完成才會展示",
  "is_open_schedule": "啟用定時同步",
  "keep_image": "保留圖片",
  "loading": "加載中...",
@@ -133,6 +145,7 @@
  "process.Image_Index": "圖片索引生成",
  "process.Is_Ready": "已就緒",
  "process.Is_Ready_Count": "{{count}} 組已就緒",
+  "process.Parse_Image": "圖片解析中",
  "process.Parsing": "內容解析中",
  "process.Vectorizing": "索引向量化",
  "process.Waiting": "排隊中",
@@ -171,14 +184,20 @@
  "tag.searchOrAddTag": "搜尋或新增標籤",
  "tag.tags": "標籤",
  "tag.total_tags": "共 {{total}} 個標籤",
+  "template_dataset": "模版導入",
+  "template_file_invalid": "模板文件格式不正確，應該是首列為 q,a,indexes 的 csv 文件",
+  "template_mode": "模板導入",
  "the_knowledge_base_has_indexes_that_are_being_trained_or_being_rebuilt": "資料集有索引正在訓練或重建中",
  "total_num_files": "共 {{total}} 個文件",
  "training.Error": "{{count}} 組異常",
+  "training.Image mode": "圖片處理",
  "training.Normal": "正常",
  "training_mode": "分段模式",
  "training_ready": "{{count}} 組",
+  "upload_by_template_format": "按模版文件上傳",
  "vector_model_max_tokens_tip": "每個分塊資料，最大長度為 3000 tokens",
  "vllm_model": "圖片理解模型",
+  "vlm_model_required_warning": "需要圖片理解模型",
  "website_dataset": "網站同步",
  "website_dataset_desc": "通過爬蟲，批量爬取網頁數據構建知識庫",
  "website_info": "網站資訊",