Archer
01ff56b42b
perf: password special chars;feat: llm paragraph;perf: chunk setting params;perf: text splitter worker ( #4984 )
...
* perf: password special chars
* feat: llm paragraph;perf: chunk setting params
* perf: text splitter worker
* perf: get rawtext buffer
* fix: test
* fix: test
* doc
* min chunk size
2025-06-10 00:05:54 +08:00
Archer
c30f069f2f
V4.9.11 feature ( #4969 )
...
* Feat: Images dataset collection (#4941 )
* New pic (#4858 )
* 更新数据集相关类型，添加图像文件ID和预览URL支持；优化数据集导入功能，新增图像数据集处理组件；修复部分国际化文本；更新文件上传逻辑以支持新功能。
* 与原先代码的差别
* 新增 V4.9.10 更新说明，支持 PG 设置`systemEnv.hnswMaxScanTuples`参数，优化 LLM stream 调用超时，修复全文检索多知识库排序问题。同时更新数据集索引，移除 datasetId 字段以简化查询。
* 更换成fileId_image逻辑，并增加训练队列匹配的逻辑
* 新增图片集合判断逻辑，优化预览URL生成流程，确保仅在数据集为图片集合时生成预览URL，并添加相关日志输出以便调试。
* Refactor Docker Compose configuration to comment out exposed ports for production environments, update image versions for pgvector, fastgpt, and mcp_server, and enhance Redis service with a health check. Additionally, standardize dataset collection labels in constants and improve internationalization strings across multiple languages.
* Enhance TrainingStates component by adding internationalization support for the imageParse training mode and update defaultCounts to include imageParse mode in trainingDetail API.
* Enhance dataset import context by adding additional steps for image dataset import process and improve internationalization strings for modal buttons in the useEditTitle hook.
* Update DatasetImportContext to conditionally render MyStep component based on data source type, improving the import process for non-image datasets.
* Refactor image dataset handling by improving internationalization strings, enhancing error messages, and streamlining the preview URL generation process.
* 图片上传到新建的 dataset_collection_images 表，逻辑跟随更改
* 修改了除了controller的其他部分问题
* 把图片数据集的逻辑整合到controller里面
* 补充i18n
* 补充i18n
* resolve评论：主要是上传逻辑的更改和组件复用
* 图片名称的图标显示
* 修改编译报错的命名问题
* 删除不需要的collectionid部分
* 多余文件的处理和改动一个删除按钮
* 除了loading和统一的imageId,其他都resolve掉的
* 处理图标报错
* 复用了MyPhotoView并采用全部替换的方式将imageFileId变成imageId
* 去除不必要文件修改
* 报错和字段修改
* 增加上传成功后删除临时文件的逻辑以及回退一些修改
* 删除path字段，将图片保存到gridfs内，并修改增删等操作的代码
* 修正编译错误
---------
Co-authored-by: archer <545436317@qq.com >
* perf: image dataset
* feat: insert image
* perf: image icon
* fix: training state
---------
Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com >
* fix: ts (#4948 )
* Thirddatasetmd (#4942 )
* add thirddataset.md
* fix thirddataset.md
* fix
* delete wrong png
---------
Co-authored-by: dreamer6680 <146868355@qq.com >
* perf: api dataset code
* perf: log
* add secondary.tsx (#4946 )
* add secondary.tsx
* fix
---------
Co-authored-by: dreamer6680 <146868355@qq.com >
* perf: multiple menu
* perf: i18n
* feat: parse queue (#4960 )
* feat: parse queue
* feat: sync parse queue
* fix thirddataset.md (#4962 )
* fix thirddataset-4.png (#4963 )
* feat: Dataset template import (#4934 )
* 模版导入部分除了文档还没写
* 修复模版导入的 build 错误
* Document production
* compress pictures
* Change some constants to variables
---------
Co-authored-by: Archer <545436317@qq.com >
* perf: template import
* doc
* llm pargraph
* bocha tool
* fix: del collection
---------
Co-authored-by: Zhuangzai fa <143257420+ctrlz526@users.noreply.github.com >
Co-authored-by: dreamer6680 <1468683855@qq.com >
Co-authored-by: dreamer6680 <146868355@qq.com >
2025-06-06 14:48:44 +08:00
Archer
c25cd48e72
perf: chunk trigger and paragraph split ( #4893 )
...
* perf: chunk trigger and paragraph split
* update max size computed
* perf: i18n
* remove table
2025-05-26 18:57:22 +08:00
Archer
88bd3aaa9e
perf: backup import ( #4866 )
...
* i18n
* remove invalid code
* perf: backup import
* backup tip
* fix: indexsize invalid
2025-05-22 15:53:51 +08:00
Theresa
2d3117c5da
feat: update ESLint config with @typescript-eslint/consistent-type-imports ( #4746 )
...
* update: Add type
* fix: update import statement for NextApiRequest type
* fix: update imports to use type for LexicalEditor and EditorState
* Refactor imports to use 'import type' for type-only imports across multiple files
- Updated imports in various components and API files to use 'import type' for better clarity and to optimize TypeScript's type checking.
- Ensured consistent usage of type imports in files related to chat, dataset, workflow, and user management.
- Improved code readability and maintainability by distinguishing between value and type imports.
* refactor: remove old ESLint configuration and add new rules
- Deleted the old ESLint configuration file from the app project.
- Added a new ESLint configuration file with updated rules and settings.
- Changed imports to use type-only imports in various files for better clarity and performance.
- Updated TypeScript configuration to remove unnecessary options.
- Added an ESLint ignore file to exclude build and dependency directories from linting.
* fix: update imports to use 'import type' for type-only imports in schema files
2025-05-06 17:33:09 +08:00
Archer
fdd4e9edbd
Test parse cite and add tool call parallel ( #4737 )
...
* add quote response filter (#4727 )
* chatting
* add quote response filter
* add test
* remove comment
* perf: cite hidden
* perf: format llm response
* feat: comment
* update default chunk size
* update default chunk size
---------
Co-authored-by: heheer <heheer@sealos.io >
2025-04-30 17:43:50 +08:00
Archer
e812ad6e84
feat: chunk index independent config ( #4271 )
...
* sync collection
* remove lock
* feat: chunk index independent config
* feat: add max chunksize to split chunk function
* remove log
* update doc
* remove
* remove log
2025-03-27 10:05:31 +08:00
Archer
adf5377ebe
Add image index and pdf parse ( #3956 )
...
* feat: think tag parse
* feat: parse think tag test
* feat: pdf parse ux
* feat: doc2x parse
* perf: rewrite training mode setting
* feat: image parse queue
* perf: image index
* feat: image parse process
* feat: add init sh
* fix: ts
2025-03-06 18:28:03 +08:00
