V4.6.7-production (#759)

This commit is contained in:
Archer
2024-01-22 13:48:55 +08:00
committed by GitHub
parent 91b7d81c1a
commit aab6ee51eb
41 changed files with 1777 additions and 2113 deletions

View File

@@ -57,7 +57,7 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
- [x] 源文件引用追踪
- [x] 模块封装,实现多级复用
- [x] 混合检索 & 重排
- [ ] 自查询规划
- [ ] Tool 模块
- [ ] 嵌入 [Laf](https://github.com/labring/laf),实现在线编写 HTTP 模块
- [ ] 插件封装功能
@@ -67,10 +67,10 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
- [x] 支持知识库单独设置向量模型
- [x] 源文件存储
- [x] 支持手动输入直接分段QA 拆分导入
- [x] 支持 pdf、word、txt、md 等常用文件,支持 url 读取、CSV 批量导入
- [ ] 支持 HTML、csv、PPT、Excel 导入
- [x] 支持 pdfdocxtxthtmlmdcsv
- [x] 支持 url 读取、CSV 批量导入
- [ ] 支持 PPT、Excel 导入
- [ ] 支持文件阅读器
- [ ] 支持差异性文件同步
- [ ] 更多的数据预处理方案
`3` 应用调试能力
@@ -81,8 +81,8 @@ https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409b
- [ ] 高级编排 DeBug 模式
`4` OpenAPI 接口
- [x] completions 接口 (对齐 GPT 接口)
- [ ] 知识库 CRUD
- [x] completions 接口 (chat 模式对齐 GPT 接口)
- [x] 知识库 CRUD
- [ ] 对话 CRUD
`5` 运营能力

View File

@@ -49,48 +49,52 @@ Cloud: [fastgpt.in](https://fastgpt.in/)
## 💡 Features
1. Powerful visual workflows: Effortlessly craft AI applications
`1` Application Orchestration Features
- [x] Simple mode on deck - no need for manual arrangement
- [x] User dialogue pre-guidance
- [x] Global variables
- [x] Knowledge base search
- [x] Dialogue via multiple LLM models
- [x] Text magic - convert to structured data
- [x] Extend with HTTP
- [ ] Embed Laf for on-the-fly HTTP module crafting
- [x] Directions for the next dialogue steps
- [x] Tracking source file references
- [ ] Custom file reader
- [ ] Modules are packaged into plug-ins to achieve reuse
- [x] Offers a straightforward mode, eliminating the need for complex orchestration
- [x] Provides clear next-step instructions in dialogues
- [x] Facilitates workflow orchestration
- [x] Tracks references in source files
- [x] Encapsulates modules for enhanced reuse at multiple levels
- [x] Combines search and reordering functions
- [ ] Includes a tool module
- [ ] Integrates [Laf](https://github.com/labring/laf) for online HTTP module creation
- [ ] Plugin encapsulation capabilities
2. Extensive knowledge base preprocessing
`2` Knowledge Base Features
- [x] Reuse and mix multiple knowledge bases
- [x] Track chunk modifications and deletions
- [x] Supports manual entries, direct segmentation, and QA split imports
- [x] Supports URL fetching and batch CSV imports
- [x] Supports Set unique vector models for knowledge bases
- [x] Store original files
- [ ] File learning Agent
- [x] Allows for the mixed use of multiple databases
- [x] Keeps track of modifications and deletions in data chunks
- [x] Enables specific vector models for each knowledge base
- [x] Stores original source files
- [x] Supports direct input and segment-based QA import
- [x] Compatible with a variety of file formats: pdf, docx, txt, html, md, csv
- [x] Facilitates URL reading and bulk CSV importing
- [ ] Supports PPT and Excel file import
- [ ] Features a file reader
- [ ] Offers diverse data preprocessing options
3. Multiple effect testing channels
`3` Application Debugging Features
- [x] Single-point knowledge base search test
- [x] Feedback references and ability to modify and delete during dialogue
- [x] Complete context presentation
- [ ] Complete module intermediate value presentation
- [x] Enables targeted search testing within the knowledge base
- [x] Allows feedback, editing, and deletion during conversations
- [x] Presents the full context of interactions
- [x] Displays all intermediate values within modules
- [ ] Advanced DeBug mode for orchestration
4. OpenAPI
`4` OpenAPI Interface
- [x] completions interface (aligned with GPT interface)
- [ ] Knowledge base CRUD
- [x] The completions interface (aligned with GPT's chat mode interface)
- [x] CRUD operations for the knowledge base
- [ ] CRUD operations for conversations
5. Operational functions
`5` Operational Features
- [x] Share without requiring login
- [x] Easy embedding with Iframe
- [x] Customizable chat window embedding with features like default open, drag-and-drop
- [x] Centralizes conversation records for review and annotation
- [x] Login-free sharing window
- [x] One-click embedding with Iframe
- [ ] Unified access to dialogue records
<a href="#readme">
<img src="https://img.shields.io/badge/-Back_to_Top-7d09f1.svg" alt="#" align="right">

View File

@@ -48,7 +48,7 @@ curl --location --request POST 'https://api.fastgpt.in/api/v1/chat/completions'
{{< /markdownify >}}
{{< /tab >}}
{{< tab tabName="detail=true 响应" >}}
{{< tab tabName="参数说明" >}}
{{< markdownify >}}
{{% alert context="info" %}}
@@ -56,7 +56,7 @@ curl --location --request POST 'https://api.fastgpt.in/api/v1/chat/completions'
- chatId: string | undefined 。
-`undefined` 时(不传入),不使用 FastGpt 提供的上下文功能,完全通过传入的 messages 构建上下文。 不会将你的记录存储到数据库中,你也无法在记录汇总中查阅到。
-`非空字符串`时,意味着使用 chatId 进行对话,自动从 FastGpt 数据库取历史记录,并使用 messages 数组最后一个内容作为用户问题。请自行确保 chatId 唯一长度小于250通常可以是自己系统的对话框ID。
- messages: 结构与 [GPT接口](https://platform.openai.com/docs/api-reference/chat/object) 完全一致。
- messages: 结构与 [GPT接口](https://platform.openai.com/docs/api-reference/chat/object) chat模式一致。
- detail: 是否返回中间值(模块状态,响应的完整结果等),`stream模式`下会通过`event`进行区分,`非stream模式`结果保存在`responseData`中。
- variables: 模块变量,一个对象,会替换模块中,输入框内容里的`{{key}}`
{{% /alert %}}

File diff suppressed because it is too large Load Diff

View File

@@ -27,7 +27,8 @@ curl --location --request POST 'https://{{host}}/api/admin/initv467' \
1. 修改了知识库UI及新的导入交互方式。
2. 优化知识库和对话的数据索引。
3. 知识库 openAPI支持通过 API 操作知识库。(文档待补充)
3. 知识库 openAPI支持通过 [API 操作知识库](/docs/development/openapi/dataset)。
4. 新增 - 输入框变量提示。输入 { 号后将会获得可用变量提示。根据社区针对高级编排的反馈,我们计划于 2 月份的版本中,优化变量内容,支持模块的局部变量以及更多全局变量写入。
5. 修复 - API 对话时chatId 冲突问题
6. 修复 - Iframe 嵌入网页可能导致的 window.onLoad 冲突。
5. 优化 - 切换团队后会保存记录,下次登录时优先登录该团队
6. 修复 - API 对话时chatId 冲突问题
7. 修复 - Iframe 嵌入网页可能导致的 window.onLoad 冲突。

View File

@@ -1,5 +1,7 @@
import { replaceSensitiveLink } from '../string/tools';
export const getErrText = (err: any, def = '') => {
const msg: string = typeof err === 'string' ? err : err?.message || def || '';
msg && console.log('error =>', msg);
return msg;
return replaceSensitiveLink(msg);
};

View File

@@ -38,6 +38,12 @@ export function replaceVariable(text: string, obj: Record<string, string | numbe
return text || '';
}
/* replace sensitive link */
export const replaceSensitiveLink = (text: string) => {
const urlRegex = /(?<=https?:\/\/)[^\s]+/g;
return text.replace(urlRegex, 'xxx');
};
export const getNanoid = (size = 12) => {
return customAlphabet('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890', size)();
};

View File

@@ -4,7 +4,7 @@ export const removeFilesByPaths = (paths: string[]) => {
paths.forEach((path) => {
fs.unlink(path, (err) => {
if (err) {
console.error(err);
// console.error(err);
}
});
});

View File

@@ -3,6 +3,7 @@ import { sseResponseEventEnum } from './constant';
import { proxyError, ERROR_RESPONSE, ERROR_ENUM } from '@fastgpt/global/common/error/errorCode';
import { addLog } from '../system/log';
import { clearCookie } from '../../support/permission/controller';
import { replaceSensitiveLink } from '@fastgpt/global/common/string/tools';
export interface ResponseType<T = any> {
code: number;
@@ -52,7 +53,7 @@ export const jsonRes = <T = any>(
res.status(code).json({
code,
statusText: '',
message: message || msg,
message: replaceSensitiveLink(message || msg),
data: data !== undefined ? data : null
});
};
@@ -90,7 +91,7 @@ export const sseErrRes = (res: NextApiResponse, error: any) => {
responseWrite({
res,
event: sseResponseEventEnum.error,
data: JSON.stringify({ message: msg })
data: JSON.stringify({ message: replaceSensitiveLink(msg) })
});
};

View File

@@ -168,6 +168,10 @@ export async function parseHeaderCert({
return Promise.reject(ERROR_ENUM.unAuthorization);
})();
if (!authRoot && (!teamId || !tmbId)) {
return Promise.reject(ERROR_ENUM.unAuthorization);
}
return {
userId: String(uid),
teamId: String(teamId),

View File

@@ -1 +1,12 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1689855121257" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="3135" xmlns:xlink="http://www.w3.org/1999/xlink" width="64" height="64"><path d="M952.7 492.1c-1.4-1.8-3.1-3.4-4.8-4.9l-179-178.9c-12.5-12.5-32.9-12.5-45.4 0s-12.5 32.9 0 45.4l126 126H421.3h-0.1c-18.2 0-32.9 14.8-32.9 33s14.7 33 32.9 33c0.3 0.1 0.5 0 0.7 0h427.8l-126 126c-12.3 12.3-12.3 32.4 0 44.7l0.7 0.7c12.3 12.3 32.4 12.3 44.7 0l182-182c11.7-11.7 12.3-30.6 1.6-43z" fill="#515151" p-id="3136"></path><path d="M562.3 799c-18 0-32.7 14.7-32.7 32.7v63.8H129.2V128.7h400.4v63.1c0 18 14.7 32.7 32.7 32.7s32.7-14.7 32.7-32.7V96.3c0-3.5-0.6-6.8-1.6-10-4.2-13.3-16.6-23-31.2-23H96.6c-18 0-32.7 14.7-32.7 32.7v831.9c0 14.2 9.2 26.3 21.8 30.8 3.6 1.4 7.5 2.1 11.5 2.1h463.2c0.6 0 1.3 0.1 1.9 0.1 18 0 32.7-14.7 32.7-32.7v-96.5c0-18-14.7-32.7-32.7-32.7z" fill="#515151" p-id="3137"></path><path d="M256.8 512.7a32.9 33 0 1 0 65.8 0 32.9 33 0 1 0-65.8 0Z" fill="#515151" p-id="3138"></path></svg>
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1689855121257"
class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="3135"
xmlns:xlink="http://www.w3.org/1999/xlink" width="64" height="64">
<path
d="M952.7 492.1c-1.4-1.8-3.1-3.4-4.8-4.9l-179-178.9c-12.5-12.5-32.9-12.5-45.4 0s-12.5 32.9 0 45.4l126 126H421.3h-0.1c-18.2 0-32.9 14.8-32.9 33s14.7 33 32.9 33c0.3 0.1 0.5 0 0.7 0h427.8l-126 126c-12.3 12.3-12.3 32.4 0 44.7l0.7 0.7c12.3 12.3 32.4 12.3 44.7 0l182-182c11.7-11.7 12.3-30.6 1.6-43z"
p-id="3136"></path>
<path
d="M562.3 799c-18 0-32.7 14.7-32.7 32.7v63.8H129.2V128.7h400.4v63.1c0 18 14.7 32.7 32.7 32.7s32.7-14.7 32.7-32.7V96.3c0-3.5-0.6-6.8-1.6-10-4.2-13.3-16.6-23-31.2-23H96.6c-18 0-32.7 14.7-32.7 32.7v831.9c0 14.2 9.2 26.3 21.8 30.8 3.6 1.4 7.5 2.1 11.5 2.1h463.2c0.6 0 1.3 0.1 1.9 0.1 18 0 32.7-14.7 32.7-32.7v-96.5c0-18-14.7-32.7-32.7-32.7z"
p-id="3137"></path>
<path d="M256.8 512.7a32.9 33 0 1 0 65.8 0 32.9 33 0 1 0-65.8 0Z" p-id="3138"></path>
</svg>

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

@@ -4,8 +4,9 @@
2. 优化知识库和对话的数据索引,加快数据操作。
3. 知识库 openAPI支持通过 API 操作知识库。
4. 新增 - 输入框变量提示。输入 { 号后将会获得可用变量提示。根据社区针对高级编排的反馈,我们计划于 2 月份的版本中,优化变量内容,支持模块的局部变量以及更多全局变量写入。
5. 修复 - API 对话时chatId 冲突问题
6. 修复 - Iframe 嵌入网页可能导致的 window.onLoad 冲突。
7. [使用文档](https://doc.fastgpt.in/docs/intro/)
8. [点击查看高级编排介绍文档](https://doc.fastgpt.in/docs/workflow)
9. [点击查看商业版](https://doc.fastgpt.in/docs/commercial/)
5. 优化 - 切换团队后会保存记录,下次登录时优先登录该团队
6. 修复 - API 对话时chatId 冲突问题
7. 修复 - Iframe 嵌入网页可能导致的 window.onLoad 冲突。
8. [使用文档](https://doc.fastgpt.in/docs/intro/)
9. [点击查看高级编排介绍文档](https://doc.fastgpt.in/docs/workflow)
10. [点击查看商业版](https://doc.fastgpt.in/docs/commercial/)

View File

@@ -95,7 +95,7 @@
"Last Step": "上一步",
"Last use time": "最后使用时间",
"Load Failed": "加载失败",
"Loading": "加载中",
"Loading": "加载中...",
"Max credit": "最大金额",
"Max credit tips": "该链接最大可消耗多少金额,超出后链接将被禁止使用。-1 代表无限制。",
"More settings": "更多设置",
@@ -541,7 +541,8 @@
"success": "开始同步"
}
},
"training": {}
"training": {
}
},
"data": {
"Auxiliary Data": "辅助数据",

View File

@@ -1,7 +1,7 @@
import { useSpeech } from '@/web/common/hooks/useSpeech';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import { Box, Flex, Image, Spinner, Textarea } from '@chakra-ui/react';
import React, { useRef, useEffect, useCallback, useState } from 'react';
import React, { useRef, useEffect, useCallback, useState, useTransition } from 'react';
import { useTranslation } from 'next-i18next';
import MyTooltip from '../MyTooltip';
import MyIcon from '@fastgpt/web/components/common/Icon';
@@ -37,7 +37,7 @@ const MessageInput = ({
showFileSelector = false,
resetInputVal
}: {
onChange: (e: string) => void;
onChange?: (e: string) => void;
onSendMessage: (e: string) => void;
onStop: () => void;
isChatting: boolean;
@@ -45,6 +45,8 @@ const MessageInput = ({
TextareaDom: React.MutableRefObject<HTMLTextAreaElement | null>;
resetInputVal: (val: string) => void;
}) => {
const [, startSts] = useTransition();
const { shareId } = useRouter().query as { shareId?: string };
const {
isSpeaking,
@@ -330,17 +332,29 @@ ${images.map((img) => JSON.stringify({ src: img.src })).join('\n')}
const textarea = e.target;
textarea.style.height = textareaMinH;
textarea.style.height = `${textarea.scrollHeight}px`;
onChange(textarea.value);
startSts(() => {
onChange?.(textarea.value);
});
}}
onKeyDown={(e) => {
// enter send.(pc or iframe && enter and unPress shift)
const isEnter = e.keyCode === 13;
if (isEnter && TextareaDom.current && (e.ctrlKey || e.altKey)) {
TextareaDom.current.value += '\n';
TextareaDom.current.style.height = textareaMinH;
TextareaDom.current.style.height = `${TextareaDom.current.scrollHeight}px`;
return;
}
// 全选内容
// @ts-ignore
e.key === 'a' && e.ctrlKey && e.target?.select();
if ((isPc || window !== parent) && e.keyCode === 13 && !e.shiftKey) {
handleSend();
e.preventDefault();
}
// 全选内容
// @ts-ignore
e.key === 'a' && e.ctrlKey && e.target?.select();
}}
onPaste={(e) => {
const clipboardData = e.clipboardData;

View File

@@ -36,7 +36,7 @@ import { adaptChat2GptMessages } from '@fastgpt/global/core/chat/adapt';
import { useMarkdown } from '@/web/common/hooks/useMarkdown';
import { ModuleItemType } from '@fastgpt/global/core/module/type.d';
import { VariableInputEnum } from '@fastgpt/global/core/module/constants';
import { useForm } from 'react-hook-form';
import { UseFormReturn, useForm } from 'react-hook-form';
import type { ChatMessageItemType } from '@fastgpt/global/core/ai/type.d';
import { fileDownload } from '@/web/common/file/utils';
import { htmlTemplate } from '@/constants/common';
@@ -65,7 +65,7 @@ const SelectMarkCollection = dynamic(() => import('./SelectMarkCollection'));
import styles from './index.module.scss';
import { postQuestionGuide } from '@/web/core/ai/api';
import { splitGuideModule } from '@fastgpt/global/core/module/utils';
import type { AppTTSConfigType } from '@fastgpt/global/core/module/type.d';
import type { AppTTSConfigType, VariableItemType } from '@fastgpt/global/core/module/type.d';
import MessageInput from './MessageInput';
import { ModuleOutputKeyEnum } from '@fastgpt/global/core/module/constants';
import ChatBoxDivider from '../core/chat/Divider';
@@ -98,6 +98,15 @@ enum FeedbackTypeEnum {
hidden = 'hidden'
}
const MessageCardStyle: BoxProps = {
px: 4,
py: 3,
borderRadius: '0 8px 8px 8px',
boxShadow: '0 0 8px rgba(0,0,0,0.15)',
display: 'inline-block',
maxW: ['calc(100% - 25px)', 'calc(100% - 40px)']
};
type Props = {
feedbackType?: `${FeedbackTypeEnum}`;
showMarkIcon?: boolean; // admin mark dataset
@@ -157,7 +166,6 @@ const ChatBox = (
const isNewChatReplace = useRef(false);
const [refresh, setRefresh] = useState(false);
const [variables, setVariables] = useState<Record<string, any>>({}); // settings variable
const [chatHistory, setChatHistory] = useState<ChatSiteItemType[]>([]);
const [feedbackId, setFeedbackId] = useState<string>();
const [readFeedbackData, setReadFeedbackData] = useState<{
@@ -180,7 +188,17 @@ const ChatBox = (
);
// compute variable input is finish.
const [variableInputFinish, setVariableInputFinish] = useState(false);
const chatForm = useForm<{
variables: Record<string, any>;
}>({
defaultValues: {
variables: {}
}
});
const { setValue, watch, handleSubmit } = chatForm;
const variables = watch('variables');
const [variableInputFinish, setVariableInputFinish] = useState(false); // clicked start chat button
const variableIsFinish = useMemo(() => {
if (!variableModules || variableModules.length === 0 || chatHistory.length > 0) return true;
@@ -194,21 +212,15 @@ const ChatBox = (
return variableInputFinish;
}, [chatHistory.length, variableInputFinish, variableModules, variables]);
const { register, reset, getValues, setValue, handleSubmit } = useForm<Record<string, any>>({
defaultValues: variables
});
// 滚动到底部
const scrollToBottom = useCallback(
(behavior: 'smooth' | 'auto' = 'smooth') => {
const scrollToBottom = (behavior: 'smooth' | 'auto' = 'smooth') => {
if (!ChatBoxRef.current) return;
ChatBoxRef.current.scrollTo({
top: ChatBoxRef.current.scrollHeight,
behavior
});
},
[ChatBoxRef]
);
};
// 聊天信息生成中……获取当前滚动条位置,判断是否需要滚动到底部
const generatingScroll = useCallback(
throttle(() => {
@@ -222,7 +234,8 @@ const ChatBox = (
[]
);
// eslint-disable-next-line react-hooks/exhaustive-deps
const generatingMessage = ({ text = '', status, name }: generatingMessageProps) => {
const generatingMessage = useCallback(
({ text = '', status, name }: generatingMessageProps) => {
setChatHistory((state) =>
state.map((item, index) => {
if (index !== state.length - 1) return item;
@@ -243,7 +256,9 @@ const ChatBox = (
})
);
generatingScroll();
};
},
[generatingScroll]
);
// 重置输入内容
const resetInputVal = useCallback((val: string) => {
@@ -284,14 +299,21 @@ const ChatBox = (
}
} catch (error) {}
},
[questionGuide, scrollToBottom, shareId]
[questionGuide, shareId]
);
/**
* user confirm send prompt
*/
const sendPrompt = useCallback(
async (variables: Record<string, any> = {}, inputVal = '', history = chatHistory) => {
({
inputVal = '',
history = chatHistory
}: {
inputVal?: string;
history?: ChatSiteItemType[];
}) => {
handleSubmit(async ({ variables }) => {
if (!onStartChat) return;
if (isChatting) {
toast({
@@ -414,19 +436,20 @@ const ChatBox = (
})
);
}
})();
},
[
chatHistory,
onStartChat,
isChatting,
resetInputVal,
toast,
scrollToBottom,
generatingMessage,
createQuestionGuide,
generatingMessage,
generatingScroll,
handleSubmit,
isChatting,
isPc,
t
onStartChat,
resetInputVal,
t,
toast
]
);
@@ -444,11 +467,14 @@ const ChatBox = (
);
setChatHistory((state) => (index === 0 ? [] : state.slice(0, index)));
sendPrompt(variables, delHistory[0].value, chatHistory.slice(0, index));
sendPrompt({
inputVal: delHistory[0].value,
history: chatHistory.slice(0, index)
});
} catch (error) {}
setLoading(false);
},
[chatHistory, onDelMessage, sendPrompt, setLoading, variables]
[chatHistory, onDelMessage, sendPrompt, setLoading]
);
// delete one message
const delOneMessage = useCallback(
@@ -471,27 +497,21 @@ const ChatBox = (
defaultVal[item.key] = '';
});
reset(e || defaultVal);
setVariables(e || defaultVal);
setValue('variables', e || defaultVal);
},
resetHistory(e) {
setVariableInputFinish(!!e.length);
setChatHistory(e);
},
scrollToBottom,
sendPrompt: (question: string) => handleSubmit((item) => sendPrompt(item, question))()
sendPrompt: (question: string) => {
sendPrompt({
inputVal: question
});
}
}));
/* style start */
const MessageCardStyle: BoxProps = {
px: 4,
py: 3,
borderRadius: '0 8px 8px 8px',
boxShadow: '0 0 8px rgba(0,0,0,0.15)',
display: 'inline-block',
maxW: ['calc(100% - 25px)', 'calc(100% - 40px)']
};
const showEmpty = useMemo(
() =>
feConfigs?.show_emptyChat &&
@@ -534,14 +554,18 @@ const ChatBox = (
useEffect(() => {
const windowMessage = ({ data }: MessageEvent<{ type: 'sendPrompt'; text: string }>) => {
if (data?.type === 'sendPrompt' && data?.text) {
handleSubmit((item) => sendPrompt(item, data.text))();
sendPrompt({
inputVal: data.text
});
}
};
window.addEventListener('message', windowMessage);
eventBus.on(EventNameEnum.sendQuestion, ({ text }: { text: string }) => {
if (!text) return;
handleSubmit((data) => sendPrompt(data, text))();
sendPrompt({
inputVal: text
});
});
eventBus.on(EventNameEnum.editQuestion, ({ text }: { text: string }) => {
if (!text) return;
@@ -553,108 +577,22 @@ const ChatBox = (
eventBus.off(EventNameEnum.sendQuestion);
eventBus.off(EventNameEnum.editQuestion);
};
}, [handleSubmit, resetInputVal, sendPrompt]);
}, [resetInputVal, sendPrompt]);
return (
<Flex flexDirection={'column'} h={'100%'}>
<Script src="/js/html2pdf.bundle.min.js" strategy="lazyOnload"></Script>
{/* chat box container */}
<Box ref={ChatBoxRef} flex={'1 0 0'} h={0} w={'100%'} overflow={'overlay'} px={[4, 0]} pb={3}>
<Box id="chat-container" maxW={['100%', '92%']} h={'100%'} mx={'auto'}>
{showEmpty && <Empty />}
{!!welcomeText && (
<Box py={3}>
{/* avatar */}
<ChatAvatar src={appAvatar} type={'AI'} />
{/* message */}
<Box textAlign={'left'}>
<Card order={2} mt={2} {...MessageCardStyle} bg={'white'}>
<Markdown source={`~~~guide \n${welcomeText}`} isChatting={false} />
</Card>
</Box>
</Box>
)}
{/* variable input */}
{!!variableModules?.length && (
<Box py={3}>
{/* avatar */}
<ChatAvatar src={appAvatar} type={'AI'} />
{/* message */}
<Box textAlign={'left'}>
<Card order={2} mt={2} bg={'white'} w={'400px'} {...MessageCardStyle}>
{variableModules.map((item) => (
<Box key={item.id} mb={4}>
<VariableLabel required={item.required}>{item.label}</VariableLabel>
{item.type === VariableInputEnum.input && (
<Input
isDisabled={variableIsFinish}
bg={'myWhite.400'}
{...register(item.key, {
required: item.required
})}
/>
)}
{item.type === VariableInputEnum.textarea && (
<Textarea
isDisabled={variableIsFinish}
bg={'myWhite.400'}
{...register(item.key, {
required: item.required
})}
rows={5}
maxLength={4000}
/>
)}
{item.type === VariableInputEnum.select && (
<MySelect
width={'100%'}
isDisabled={variableIsFinish}
list={(item.enums || []).map((item) => ({
label: item.value,
value: item.value
}))}
{...register(item.key, {
required: item.required
})}
value={getValues(item.key)}
onchange={(e) => {
setValue(item.key, e);
setRefresh(!refresh);
}}
/>
)}
</Box>
))}
{!variableIsFinish && (
<Button
leftIcon={<MyIcon name={'core/chat/chatFill'} w={'16px'} />}
size={'sm'}
maxW={'100px'}
onClick={handleSubmit((data) => {
onUpdateVariable?.(data);
setVariables(data);
const onSubmitVariables = useCallback(
(data: Record<string, any>) => {
setVariableInputFinish(true);
})}
>
{t('core.chat.Start Chat')}
</Button>
)}
</Card>
</Box>
</Box>
)}
{/* chat history */}
<Box id={'history'}>
{chatHistory.map((item, index) => (
<Box key={item.dataId} py={5}>
{item.obj === 'Human' && (
onUpdateVariable?.(data);
},
[onUpdateVariable]
);
const HumanChatCard = useCallback(
({ item, index }: { item: ChatSiteItemType; index: number }) => {
return (
<>
{/* control icon */}
<Flex w={'100%'} alignItems={'center'} justifyContent={'flex-end'}>
<ChatController
<ChatControllerComponent
chat={item}
onDelete={
onDelMessage
@@ -663,7 +601,7 @@ const ChatBox = (
}
: undefined
}
onRetry={() => retryInput(index)}
onRetry={useCallback(() => retryInput(index), [index])}
/>
<ChatAvatar src={userAvatar} type={'Human'} />
</Flex>
@@ -680,13 +618,40 @@ const ChatBox = (
</Card>
</Box>
</>
);
},
[]
);
return (
<Flex flexDirection={'column'} h={'100%'}>
<Script src="/js/html2pdf.bundle.min.js" strategy="lazyOnload"></Script>
{/* chat box container */}
<Box ref={ChatBoxRef} flex={'1 0 0'} h={0} w={'100%'} overflow={'overlay'} px={[4, 0]} pb={3}>
<Box id="chat-container" maxW={['100%', '92%']} h={'100%'} mx={'auto'}>
{showEmpty && <Empty />}
{!!welcomeText && <WelcomeText appAvatar={appAvatar} welcomeText={welcomeText} />}
{/* variable input */}
{!!variableModules?.length && (
<VariableInput
appAvatar={appAvatar}
variableModules={variableModules}
variableIsFinish={variableIsFinish}
chatForm={chatForm}
onSubmitVariables={onSubmitVariables}
/>
)}
{/* chat history */}
<Box id={'history'}>
{chatHistory.map((item, index) => (
<Box key={item.dataId} py={5}>
{item.obj === 'Human' && <HumanChatCard item={item} index={index} />}
{item.obj === 'AI' && (
<>
{/* control icon */}
<Flex w={'100%'} alignItems={'center'}>
<ChatAvatar src={appAvatar} type={'AI'} />
<ChatController
{/* control icon */}
<ChatControllerComponent
ml={2}
chat={item}
setChatHistory={setChatHistory}
@@ -723,11 +688,10 @@ const ChatBox = (
}
: undefined
}
onAddUserLike={(() => {
if (feedbackType !== FeedbackTypeEnum.user || item.userBadFeedback) {
return;
}
return () => {
onAddUserLike={
feedbackType !== FeedbackTypeEnum.user || item.userBadFeedback
? undefined
: () => {
if (!item.dataId || !chatId || !appId) return;
const isGoodFeedback = !!item.userGoodFeedback;
@@ -751,8 +715,8 @@ const ChatBox = (
userGoodFeedback: isGoodFeedback ? undefined : 'yes'
});
} catch (error) {}
};
})()}
}
}
onCloseUserLike={
feedbackType === FeedbackTypeEnum.admin
? () => {
@@ -931,13 +895,12 @@ const ChatBox = (
</Box>
</Box>
{/* message input */}
{onStartChat && variableIsFinish && active ? (
{onStartChat && variableIsFinish && active && (
<MessageInput
onChange={(e) => {
setRefresh(!refresh);
}}
onSendMessage={(e) => {
handleSubmit((data) => sendPrompt(data, e))();
onSendMessage={(inputVal) => {
sendPrompt({
inputVal
});
}}
onStop={() => chatController.current?.abort('stop')}
isChatting={isChatting}
@@ -945,7 +908,7 @@ const ChatBox = (
resetInputVal={resetInputVal}
showFileSelector={showFileSelector}
/>
) : null}
)}
{/* user feedback modal */}
{!!feedbackId && chatId && appId && (
<FeedbackModal
@@ -1115,17 +1078,57 @@ export const useChatBox = () => {
};
};
function VariableLabel({
required = false,
children
const WelcomeText = React.memo(function Welcome({
appAvatar,
welcomeText
}: {
required?: boolean;
children: React.ReactNode | string;
appAvatar?: string;
welcomeText: string;
}) {
return (
<Box py={3}>
{/* avatar */}
<ChatAvatar src={appAvatar} type={'AI'} />
{/* message */}
<Box textAlign={'left'}>
<Card order={2} mt={2} {...MessageCardStyle} bg={'white'}>
<Markdown source={`~~~guide \n${welcomeText}`} isChatting={false} />
</Card>
</Box>
</Box>
);
});
const VariableInput = React.memo(function VariableInput({
appAvatar,
variableModules,
variableIsFinish,
chatForm,
onSubmitVariables
}: {
appAvatar?: string;
variableModules: VariableItemType[];
variableIsFinish: boolean;
onSubmitVariables: (e: Record<string, any>) => void;
chatForm: UseFormReturn<{
variables: Record<string, any>;
}>;
}) {
const { t } = useTranslation();
const { register, setValue, handleSubmit: handleSubmitChat, watch } = chatForm;
const variables = watch('variables');
return (
<Box py={3}>
{/* avatar */}
<ChatAvatar src={appAvatar} type={'AI'} />
{/* message */}
<Box textAlign={'left'}>
<Card order={2} mt={2} bg={'white'} w={'400px'} {...MessageCardStyle}>
{variableModules.map((item) => (
<Box key={item.id} mb={4}>
<Box as={'label'} display={'inline-block'} position={'relative'} mb={1}>
{children}
{required && (
{item.label}
{item.required && (
<Box
position={'absolute'}
top={'-2px'}
@@ -1137,8 +1140,63 @@ function VariableLabel({
</Box>
)}
</Box>
{item.type === VariableInputEnum.input && (
<Input
isDisabled={variableIsFinish}
bg={'myWhite.400'}
{...register(`variables.${item.key}`, {
required: item.required
})}
/>
)}
{item.type === VariableInputEnum.textarea && (
<Textarea
isDisabled={variableIsFinish}
bg={'myWhite.400'}
{...register(`variables.${item.key}`, {
required: item.required
})}
rows={5}
maxLength={4000}
/>
)}
{item.type === VariableInputEnum.select && (
<MySelect
width={'100%'}
isDisabled={variableIsFinish}
list={(item.enums || []).map((item) => ({
label: item.value,
value: item.value
}))}
{...register(`variables.${item.key}`, {
required: item.required
})}
value={variables[item.key]}
onchange={(e) => {
setValue(`variables.${item.key}`, e);
}}
/>
)}
</Box>
))}
{!variableIsFinish && (
<Button
leftIcon={<MyIcon name={'core/chat/chatFill'} w={'16px'} />}
size={'sm'}
maxW={'100px'}
onClick={handleSubmitChat((data) => {
onSubmitVariables(data);
})}
>
{t('core.chat.Start Chat')}
</Button>
)}
</Card>
</Box>
</Box>
);
}
});
function ChatAvatar({ src, type }: { src?: string; type: 'Human' | 'AI' }) {
const theme = useTheme();
return (
@@ -1173,7 +1231,7 @@ function Empty() {
);
}
function ChatController({
const ChatControllerComponent = React.memo(function ChatControllerComponent({
chat,
setChatHistory,
display,
@@ -1226,7 +1284,7 @@ function ChatController({
return (
<Flex {...controlContainerStyle} ml={ml} mr={mr} display={display}>
<MyTooltip label={'复制'}>
<MyTooltip label={t('common.Copy')}>
<MyIcon
{...controlIconStyle}
name={'copy'}
@@ -1246,7 +1304,7 @@ function ChatController({
/>
</MyTooltip>
)}
<MyTooltip label={'删除'}>
<MyTooltip label={t('common.Delete')}>
<MyIcon
{...controlIconStyle}
name={'delete'}
@@ -1259,7 +1317,7 @@ function ChatController({
{showVoiceIcon &&
hasAudio &&
(audioLoading ? (
<MyTooltip label={'加载中...'}>
<MyTooltip label={t('common.Loading')}>
<MyIcon {...controlIconStyle} name={'common/loading'} />
</MyTooltip>
) : audioPlaying ? (
@@ -1372,4 +1430,4 @@ function ChatController({
)}
</Flex>
);
}
});

View File

@@ -35,36 +35,79 @@ export enum CodeClassName {
img = 'img'
}
function Code({ inline, className, children }: any) {
const Markdown = ({ source, isChatting = false }: { source: string; isChatting?: boolean }) => {
const components = useMemo<any>(
() => ({
img: Image,
pre: 'div',
p: (pProps: any) => <p {...pProps} dir="auto" />,
code: Code,
a: A
}),
[]
);
const formatSource = source
.replace(/\\n/g, '\n&nbsp;')
.replace(/(http[s]?:\/\/[^\s。]+)([。,])/g, '$1 $2')
.replace(/\n*(\[QUOTE SIGN\]\(.*\))/g, '$1');
return (
<ReactMarkdown
className={`markdown ${styles.markdown}
${isChatting ? `${formatSource ? styles.waitingAnimation : styles.animation}` : ''}
`}
remarkPlugins={[RemarkMath, RemarkGfm, RemarkBreaks]}
rehypePlugins={[RehypeKatex]}
components={components}
linkTarget={'_blank'}
>
{formatSource}
</ReactMarkdown>
);
};
export default React.memo(Markdown);
const Code = React.memo(function Code(e: any) {
const { inline, className, children } = e;
const match = /language-(\w+)/.exec(className || '');
const codeType = match?.[1];
const strChildren = String(children);
const Component = useMemo(() => {
if (codeType === CodeClassName.mermaid) {
return <MermaidCodeBlock code={String(children)} />;
return <MermaidCodeBlock code={strChildren} />;
}
if (codeType === CodeClassName.guide) {
return <ChatGuide text={String(children)} />;
return <ChatGuide text={strChildren} />;
}
if (codeType === CodeClassName.questionGuide) {
return <QuestionGuide text={String(children)} />;
return <QuestionGuide text={strChildren} />;
}
if (codeType === CodeClassName.echarts) {
return <EChartsCodeBlock code={String(children)} />;
return <EChartsCodeBlock code={strChildren} />;
}
if (codeType === CodeClassName.img) {
return <ImageBlock images={String(children)} />;
return <ImageBlock images={strChildren} />;
}
return (
<CodeLight className={className} inline={inline} match={match}>
{children}
</CodeLight>
);
}
function Image({ src }: { src?: string }) {
}, [codeType, className, inline, match, strChildren]);
return Component;
});
const Image = React.memo(function Image({ src }: { src?: string }) {
return <MdImage src={src} />;
}
function A({ children, ...props }: any) {
});
const A = React.memo(function A({ children, ...props }: any) {
const { t } = useTranslation();
// empty href link
@@ -109,38 +152,4 @@ function A({ children, ...props }: any) {
}
return <Link {...props}>{children}</Link>;
}
const Markdown = ({ source, isChatting = false }: { source: string; isChatting?: boolean }) => {
const components = useMemo<any>(
() => ({
img: Image,
pre: 'div',
p: (pProps: any) => <p {...pProps} dir="auto" />,
code: Code,
a: A
}),
[]
);
const formatSource = source
.replace(/\\n/g, '\n&nbsp;')
.replace(/(http[s]?:\/\/[^\s。]+)([。,])/g, '$1 $2')
.replace(/\n*(\[QUOTE SIGN\]\(.*\))/g, '$1');
return (
<ReactMarkdown
className={`markdown ${styles.markdown}
${isChatting ? `${formatSource ? styles.waitingAnimation : styles.animation}` : ''}
`}
remarkPlugins={[RemarkMath, RemarkGfm, RemarkBreaks]}
rehypePlugins={[RehypeKatex]}
components={components}
linkTarget={'_blank'}
>
{formatSource}
</ReactMarkdown>
);
};
export default React.memo(Markdown);
});

View File

@@ -79,6 +79,8 @@ const TagTextarea = ({ defaultValues, onUpdate, ...props }: Props) => {
ref={InputRef}
variant={'unstyled'}
display={'inline-block'}
h={'24px'}
borderRadius={'none'}
w="auto"
onBlur={(e) => {
const value = e.target.value;

View File

@@ -66,7 +66,6 @@ const AIChatSettingsModal = ({
}, [getValues]);
const quoteTemplateVariables = (() => [
...pickerMenu,
{
key: 'q',
label: 'q',
@@ -91,15 +90,21 @@ const AIChatSettingsModal = ({
key: 'index',
label: t('core.dataset.search.Quote index'),
icon: 'core/app/simpleMode/variable'
}
},
...pickerMenu
])();
const quotePromptVariables = (() => [
...pickerMenu,
{
key: 'quote',
label: t('core.app.Quote templates'),
icon: 'core/app/simpleMode/variable'
}
},
{
key: 'question',
label: t('core.module.input.label.user question'),
icon: 'core/app/simpleMode/variable'
},
...pickerMenu
])();
const LabelStyles: BoxProps = {

View File

@@ -55,11 +55,13 @@ const InviteModal = ({
openConfirm(
() => onClose(),
undefined,
t('user.team.Invite Member Success Tip', {
<Box whiteSpace={'pre-wrap'}>
{t('user.team.Invite Member Success Tip', {
success: res.invite.length,
inValid: res.inValid.map((item) => item.username).join(', '),
inTeam: res.inTeam.map((item) => item.username).join(', ')
})
})}
</Box>
)();
},
errorToast: t('user.team.Invite Member Failed Tip')

View File

@@ -75,7 +75,7 @@ const TeamManageModal = ({ onClose }: { onClose: () => void }) => {
const { mutate: onSwitchTeam, isLoading: isSwitchTeam } = useRequest({
mutationFn: async (teamId: string) => {
const token = await putSwitchTeam(teamId);
setToken(token);
token && setToken(token);
return initUserInfo();
},
errorToast: t('user.team.Switch Team Failed')
@@ -286,13 +286,7 @@ const TeamManageModal = ({ onClose }: { onClose: () => void }) => {
size="sm"
borderRadius={'md'}
ml={3}
leftIcon={
<MyIcon
name={'support/account/loginoutLight'}
w={'14px'}
color={'primary.500'}
/>
}
leftIcon={<MyIcon name={'support/account/loginoutLight'} w={'14px'} />}
onClick={() => {
openLeaveConfirm(() => onLeaveTeam(userInfo?.team?.teamId))();
}}

View File

@@ -271,15 +271,18 @@ const UserInfo = () => {
)}
</Flex>
</Box>
{feConfigs?.show_pay && (
<Box mt={6} whiteSpace={'nowrap'} w={['85%', '300px']}>
<Flex alignItems={'center'}>
<Box flex={'1 0 0'} fontSize={'md'}>
{t('support.user.team.Dataset usage')}:&nbsp;{datasetUsageMap.usedSize}/
{datasetSub.maxSize}
</Box>
{userInfo?.team?.canWrite && (
<Button size={'sm'} onClick={onOpenSubDatasetModal}>
{t('support.wallet.Buy more')}
</Button>
)}
</Flex>
<Box mt={1}>
<Progress
@@ -293,6 +296,7 @@ const UserInfo = () => {
/>
</Box>
</Box>
)}
</>
)}

View File

@@ -1,85 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { uploadFile } from '@fastgpt/service/common/file/gridfs/controller';
import { getUploadModel } from '@fastgpt/service/common/file/multer';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { FileCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api';
import { removeFilesByPaths } from '@fastgpt/service/common/file/utils';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constants';
/**
* Creates the multer uploader
*/
const upload = getUploadModel({
maxSize: 500 * 1024 * 1024
});
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
let filePaths: string[] = [];
const { datasetId } = req.query as { datasetId: string };
try {
await connectToDatabase();
const { teamId, tmbId } = await authDataset({
req,
authToken: true,
authApiKey: true,
per: 'w',
datasetId
});
const { file, bucketName, data } = await upload.doUpload<FileCreateDatasetCollectionParams>(
req,
res
);
filePaths = [file.path];
if (!file || !bucketName) {
throw new Error('file is empty');
}
const { fileMetadata, collectionMetadata, ...collectionData } = data;
// upload file and create collection
const fileId = await uploadFile({
teamId,
tmbId,
bucketName,
path: file.path,
filename: file.originalname,
contentType: file.mimetype,
metadata: fileMetadata
});
// create collection
const collectionId = await createOneCollection({
...collectionData,
metadata: collectionMetadata,
teamId,
tmbId,
type: DatasetCollectionTypeEnum.file,
fileId
});
jsonRes(res, {
data: collectionId
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
removeFilesByPaths(filePaths);
}
export const config = {
api: {
bodyParser: false
}
};

View File

@@ -1,91 +0,0 @@
/*
Create one dataset collection
*/
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import type { LinkCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api.d';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import {
TrainingModeEnum,
DatasetCollectionTypeEnum
} from '@fastgpt/global/core/dataset/constants';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/limit/dataset';
import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils';
import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { getQAModel, getVectorModel } from '@/service/core/ai/model';
import { reloadCollectionChunks } from '@fastgpt/service/core/dataset/collection/utils';
import { startQueue } from '@/service/utils/tools';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const {
link,
trainingType = TrainingModeEnum.chunk,
chunkSize = 512,
chunkSplitter,
qaPrompt,
...body
} = req.body as LinkCreateDatasetCollectionParams;
const { teamId, tmbId, dataset } = await authDataset({
req,
authToken: true,
authApiKey: true,
datasetId: body.datasetId,
per: 'w'
});
// 1. check dataset limit
await checkDatasetLimit({
teamId,
freeSize: global.feConfigs?.subscription?.datasetStoreFreeSize,
insertLen: predictDataLimitLength(trainingType, new Array(10))
});
// 2. create collection
const collectionId = await createOneCollection({
...body,
name: link,
teamId,
tmbId,
type: DatasetCollectionTypeEnum.link,
trainingType,
chunkSize,
chunkSplitter,
qaPrompt,
rawLink: link
});
// 3. create bill and start sync
const { billId } = await createTrainingBill({
teamId,
tmbId,
appName: 'core.dataset.collection.Sync Collection',
billSource: BillSourceEnum.training,
vectorModel: getVectorModel(dataset.vectorModel).name,
agentModel: getQAModel(dataset.agentModel).name
});
await reloadCollectionChunks({
collectionId,
tmbId,
billId
});
startQueue();
jsonRes(res, {
data: { collectionId }
});
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -1,117 +0,0 @@
/*
Create one dataset collection
*/
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import type { TextCreateDatasetCollectionParams } from '@fastgpt/global/core/dataset/api.d';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
import {
TrainingModeEnum,
DatasetCollectionTypeEnum
} from '@fastgpt/global/core/dataset/constants';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { checkDatasetLimit } from '@fastgpt/service/support/permission/limit/dataset';
import { predictDataLimitLength } from '@fastgpt/global/core/dataset/utils';
import { pushDataToTrainingQueue } from '@/service/core/dataset/data/controller';
import { hashStr } from '@fastgpt/global/common/string/tools';
import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { getQAModel, getVectorModel } from '@/service/core/ai/model';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const {
name,
text,
trainingType = TrainingModeEnum.chunk,
chunkSize = 512,
chunkSplitter,
qaPrompt,
...body
} = req.body as TextCreateDatasetCollectionParams;
const { teamId, tmbId, dataset } = await authDataset({
req,
authToken: true,
authApiKey: true,
datasetId: body.datasetId,
per: 'w'
});
// 1. split text to chunks
const { chunks } = splitText2Chunks({
text,
chunkLen: chunkSize,
overlapRatio: trainingType === TrainingModeEnum.chunk ? 0.2 : 0,
customReg: chunkSplitter ? [chunkSplitter] : []
});
// 2. check dataset limit
await checkDatasetLimit({
teamId,
freeSize: global.feConfigs?.subscription?.datasetStoreFreeSize,
insertLen: predictDataLimitLength(trainingType, chunks)
});
// 3. create collection and training bill
const [collectionId, { billId }] = await Promise.all([
createOneCollection({
...body,
teamId,
tmbId,
type: DatasetCollectionTypeEnum.virtual,
name,
trainingType,
chunkSize,
chunkSplitter,
qaPrompt,
hashRawText: hashStr(text),
rawTextLength: text.length
}),
createTrainingBill({
teamId,
tmbId,
appName: name,
billSource: BillSourceEnum.training,
vectorModel: getVectorModel(dataset.vectorModel)?.name,
agentModel: getQAModel(dataset.agentModel)?.name
})
]);
// 4. push chunks to training queue
const insertResults = await pushDataToTrainingQueue({
teamId,
tmbId,
collectionId,
trainingMode: trainingType,
prompt: qaPrompt,
billId,
data: chunks.map((text, index) => ({
q: text,
chunkIndex: index
}))
});
jsonRes(res, {
data: { collectionId, results: insertResults }
});
} catch (err) {
jsonRes(res, {
code: 500,
error: err
});
}
}
export const config = {
api: {
bodyParser: {
sizeLimit: '10mb'
}
}
};

View File

@@ -6,6 +6,7 @@ import type { CreateDatasetParams } from '@/global/core/dataset/api.d';
import { createDefaultCollection } from '@fastgpt/service/core/dataset/collection/controller';
import { authUserNotVisitor } from '@fastgpt/service/support/permission/auth/user';
import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constants';
import { getQAModel, getVectorModel } from '@/service/core/ai/model';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
@@ -13,18 +14,18 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
const {
parentId,
name,
type,
type = DatasetTypeEnum.dataset,
avatar,
vectorModel = global.vectorModels[0].model,
agentModel = global.qaModels[0].model
} = req.body as CreateDatasetParams;
// auth
const { teamId, tmbId } = await authUserNotVisitor({ req, authToken: true });
const { teamId, tmbId } = await authUserNotVisitor({ req, authToken: true, authApiKey: true });
// check model valid
const vectorModelStore = global.vectorModels.find((item) => item.model === vectorModel);
const agentModelStore = global.qaModels.find((item) => item.model === agentModel);
const vectorModelStore = getVectorModel(vectorModel);
const agentModelStore = getQAModel(agentModel);
if (!vectorModelStore || !agentModelStore) {
throw new Error('vectorModel or qaModel is invalid');
}

View File

@@ -18,7 +18,13 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
}
// auth owner
const { teamId } = await authDataset({ req, authToken: true, datasetId, per: 'owner' });
const { teamId } = await authDataset({
req,
authToken: true,
authApiKey: true,
datasetId,
per: 'owner'
});
const datasets = await findDatasetAndAllChildren({
teamId,

View File

@@ -1,64 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { request } from '@fastgpt/service/common/api/plusRequest';
import type { Method } from 'axios';
import { setCookie } from '@fastgpt/service/support/permission/controller';
import { connectToDatabase } from '@/service/mongo';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const method = (req.method || 'POST') as Method;
const { path = [], ...query } = req.query as any;
const url = `/${path?.join('/')}?${new URLSearchParams(query).toString()}`;
if (!url) {
throw new Error('url is empty');
}
const data = req.body || query;
const repose = await request(
url,
data,
{
headers: {
...req.headers,
// @ts-ignore
rootkey: undefined
}
},
method
);
/* special response */
// response cookie
if (repose?.cookie) {
setCookie(res, repose.cookie);
return jsonRes(res, {
data: repose?.cookie
});
}
jsonRes(res, {
data: repose
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
}
export const config = {
api: {
bodyParser: {
sizeLimit: '10mb'
},
responseLimit: '10mb'
}
};

View File

@@ -0,0 +1,56 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { request } from 'http';
import { FastGPTProUrl } from '@fastgpt/service/common/system/constants';
import url from 'url';
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
try {
await connectToDatabase();
const { path = [], ...query } = req.query as any;
const requestPath = `/api/${path?.join('/')}?${new URLSearchParams(query).toString()}`;
if (!requestPath) {
throw new Error('url is empty');
}
const parsedUrl = url.parse(FastGPTProUrl);
delete req.headers?.rootkey;
const requestResult = request({
protocol: parsedUrl.protocol,
hostname: parsedUrl.hostname,
port: parsedUrl.port,
path: requestPath,
method: req.method,
headers: req.headers
});
req.pipe(requestResult);
requestResult.on('response', (response) => {
Object.keys(response.headers).forEach((key) => {
// @ts-ignore
res.setHeader(key, response.headers[key]);
});
response.statusCode && res.writeHead(response.statusCode);
response.pipe(res);
});
requestResult.on('error', (e) => {
res.send(e);
res.end();
});
} catch (error) {
jsonRes(res, {
code: 500,
error
});
}
}
export const config = {
api: {
bodyParser: false
}
};

View File

@@ -87,7 +87,7 @@ const Detail = ({ datasetId, currentTab }: { datasetId: string; currentTab: `${T
onError(err: any) {
router.replace(`/dataset/list`);
toast({
title: getErrText(err, t('common.Load Failed')),
title: t(getErrText(err, t('common.Load Failed'))),
status: 'error'
});
}

View File

@@ -46,13 +46,15 @@ import { PermissionTypeEnum } from '@fastgpt/global/support/permission/constant'
import { DatasetItemType } from '@fastgpt/global/core/dataset/type';
import ParentPaths from '@/components/common/ParentPaths';
import DatasetTypeTag from '@/components/core/dataset/DatasetTypeTag';
import { useToast } from '@/web/common/hooks/useToast';
import { getErrText } from '@fastgpt/global/common/error/utils';
const CreateModal = dynamic(() => import('./component/CreateModal'), { ssr: false });
const MoveModal = dynamic(() => import('./component/MoveModal'), { ssr: false });
const Kb = () => {
const { t } = useTranslation();
const theme = useTheme();
const { toast } = useToast();
const router = useRouter();
const { parentId } = router.query as { parentId: string };
const { setLoading } = useSystemStore();
@@ -115,9 +117,20 @@ const Kb = () => {
errorToast: t('dataset.Export Dataset Limit Error')
});
const { data, refetch, isFetching } = useQuery(['loadDataset', parentId], () => {
const { data, refetch, isFetching } = useQuery(
['loadDataset', parentId],
() => {
return Promise.all([loadDatasets(parentId), getDatasetPaths(parentId)]);
},
{
onError(err) {
toast({
status: 'error',
title: t(getErrText(err))
});
}
}
);
const paths = data?.[1] || [];

View File

@@ -106,9 +106,9 @@ const provider = ({ code, state, error }: { code: string; state: string; error?:
export async function getServerSideProps(content: any) {
return {
props: {
code: content?.query?.code,
state: content?.query?.state,
error: content?.query?.error,
code: content?.query?.code || '',
state: content?.query?.state || '',
error: content?.query?.error || '',
...(await serviceSideProps(content))
}
};

View File

@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import React, { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import { useDisclosure, Button, ModalBody, ModalFooter } from '@chakra-ui/react';
import { useTranslation } from 'next-i18next';
import MyModal from '@/components/MyModal';
@@ -35,7 +35,7 @@ export const useConfirm = (props?: {
content,
showCancel = true
} = props || {};
const [customContent, setCustomContent] = useState(content);
const [customContent, setCustomContent] = useState<string | React.ReactNode>(content);
const { isOpen, onOpen, onClose } = useDisclosure();
@@ -44,7 +44,7 @@ export const useConfirm = (props?: {
return {
openConfirm: useCallback(
(confirm?: any, cancel?: any, customContent?: string) => {
(confirm?: any, cancel?: any, customContent?: string | React.ReactNode) => {
confirmCb.current = confirm;
cancelCb.current = cancel;

View File

@@ -58,7 +58,7 @@ export const putDatasetById = (data: DatasetUpdateBody) => PUT<void>(`/core/data
export const delDatasetById = (id: string) => DELETE(`/core/dataset/delete?id=${id}`);
export const postWebsiteSync = (data: PostWebsiteSyncParams) =>
POST(`/plusApi/core/dataset/websiteSync`, data, {
POST(`/proApi/core/dataset/websiteSync`, data, {
timeout: 600000
}).catch();
@@ -76,7 +76,7 @@ export const getDatasetCollectionById = (id: string) =>
export const postDatasetCollection = (data: CreateDatasetCollectionParams) =>
POST<string>(`/core/dataset/collection/create`, data);
export const postCreateDatasetLinkCollection = (data: LinkCreateDatasetCollectionParams) =>
POST<{ collectionId: string }>(`/core/dataset/collection/create/link`, data);
POST<{ collectionId: string }>(`/proApi/core/dataset/collection/create/link`, data);
export const putDatasetCollectionById = (data: UpdateDatasetCollectionParams) =>
POST(`/core/dataset/collection/update`, data);

View File

@@ -27,7 +27,10 @@ export const fileCollectionCreate = ({
form.append('bucketName', BucketNameEnum.dataset);
form.append('file', file, encodeURIComponent(file.name));
return POST<string>(`/core/dataset/collection/create/file?datasetId=${data.datasetId}`, form, {
return POST<string>(
`/proApi/core/dataset/collection/create/emptyFile?datasetId=${data.datasetId}`,
form,
{
timeout: 480000,
onUploadProgress: (e) => {
if (!e.total) return;
@@ -38,7 +41,8 @@ export const fileCollectionCreate = ({
headers: {
'Content-Type': 'multipart/form-data; charset=utf-8'
}
});
}
);
};
export async function chunksUpload({

View File

@@ -7,8 +7,8 @@ export const getPromotionInitData = () =>
GET<{
invitedAmount: number;
earningsAmount: number;
}>('/plusApi/support/activity/promotion/getPromotionData');
}>('/proApi/support/activity/promotion/getPromotionData');
/* promotion records */
export const getPromotionRecords = (data: RequestPaging) =>
POST<PromotionRecordType>(`/plusApi/support/activity/promotion/getPromotions`, data);
POST<PromotionRecordType>(`/proApi/support/activity/promotion/getPromotions`, data);

View File

@@ -14,14 +14,14 @@ export const sendAuthCode = (data: {
username: string;
type: `${UserAuthTypeEnum}`;
googleToken: string;
}) => POST(`/plusApi/support/user/inform/sendAuthCode`, data);
}) => POST(`/proApi/support/user/inform/sendAuthCode`, data);
export const getTokenLogin = () =>
GET<UserType>('/support/user/account/tokenLogin', {}, { maxQuantity: 1 });
export const oauthLogin = (params: OauthLoginProps) =>
POST<ResLogin>('/plusApi/support/user/account/login/oauth', params);
POST<ResLogin>('/proApi/support/user/account/login/oauth', params);
export const postFastLogin = (params: FastLoginProps) =>
POST<ResLogin>('/plusApi/support/user/account/login/fastLogin', params);
POST<ResLogin>('/proApi/support/user/account/login/fastLogin', params);
export const postRegister = ({
username,
@@ -34,7 +34,7 @@ export const postRegister = ({
password: string;
inviterId?: string;
}) =>
POST<ResLogin>(`/plusApi/support/user/account/register/emailAndPhone`, {
POST<ResLogin>(`/proApi/support/user/account/register/emailAndPhone`, {
username,
code,
inviterId,
@@ -50,7 +50,7 @@ export const postFindPassword = ({
code: string;
password: string;
}) =>
POST<ResLogin>(`/plusApi/support/user/account/password/updateByCode`, {
POST<ResLogin>(`/proApi/support/user/account/password/updateByCode`, {
username,
code,
password: hashStr(password)

View File

@@ -3,7 +3,7 @@ import type { PagingData, RequestPaging } from '@/types';
import type { UserInformSchema } from '@fastgpt/global/support/user/inform/type';
export const getInforms = (data: RequestPaging) =>
POST<PagingData<UserInformSchema>>(`/plusApi/support/user/inform/list`, data);
POST<PagingData<UserInformSchema>>(`/proApi/support/user/inform/list`, data);
export const getUnreadCount = () => GET<number>(`/plusApi/support/user/inform/countUnread`);
export const readInform = (id: string) => GET(`/plusApi/support/user/inform/read`, { id });
export const getUnreadCount = () => GET<number>(`/proApi/support/user/inform/countUnread`);
export const readInform = (id: string) => GET(`/proApi/support/user/inform/read`, { id });

View File

@@ -16,29 +16,29 @@ import {
/* --------------- team ---------------- */
export const getTeamList = (status: `${TeamMemberSchema['status']}`) =>
GET<TeamItemType[]>(`/plusApi/support/user/team/list`, { status });
GET<TeamItemType[]>(`/proApi/support/user/team/list`, { status });
export const postCreateTeam = (data: CreateTeamProps) =>
POST<string>(`/plusApi/support/user/team/create`, data);
POST<string>(`/proApi/support/user/team/create`, data);
export const putUpdateTeam = (data: UpdateTeamProps) =>
PUT(`/plusApi/support/user/team/update`, data);
PUT(`/proApi/support/user/team/update`, data);
export const putSwitchTeam = (teamId: string) =>
PUT<string>(`/plusApi/support/user/team/switch`, { teamId });
PUT<string>(`/proApi/support/user/team/switch`, { teamId });
/* --------------- team member ---------------- */
export const getTeamMembers = (teamId: string) =>
GET<TeamMemberItemType[]>(`/plusApi/support/user/team/member/list`, { teamId });
GET<TeamMemberItemType[]>(`/proApi/support/user/team/member/list`, { teamId });
export const postInviteTeamMember = (data: InviteMemberProps) =>
POST<InviteMemberResponse>(`/plusApi/support/user/team/member/invite`, data);
POST<InviteMemberResponse>(`/proApi/support/user/team/member/invite`, data);
export const putUpdateMember = (data: UpdateTeamMemberProps) =>
PUT(`/plusApi/support/user/team/member/update`, data);
PUT(`/proApi/support/user/team/member/update`, data);
export const putUpdateMemberName = (name: string) =>
PUT(`/plusApi/support/user/team/member/updateName`, { name });
PUT(`/proApi/support/user/team/member/updateName`, { name });
export const delRemoveMember = (props: DelMemberProps) =>
DELETE(`/plusApi/support/user/team/member/delete`, props);
DELETE(`/proApi/support/user/team/member/delete`, props);
export const updateInviteResult = (data: UpdateInviteProps) =>
PUT('/plusApi/support/user/team/member/updateInvite', data);
PUT('/proApi/support/user/team/member/updateInvite', data);
export const delLeaveTeam = (teamId: string) =>
DELETE('/plusApi/support/user/team/member/leave', { teamId });
DELETE('/proApi/support/user/team/member/leave', { teamId });
/* team limit */
export const checkTeamExportDatasetLimit = (datasetId: string) =>

View File

@@ -4,7 +4,7 @@ import type { PagingData, RequestPaging } from '@/types';
import type { BillItemType } from '@fastgpt/global/support/wallet/bill/type';
export const getUserBills = (data: RequestPaging) =>
POST<PagingData<BillItemType>>(`/plusApi/support/wallet/bill/getBill`, data);
POST<PagingData<BillItemType>>(`/proApi/support/wallet/bill/getBill`, data);
export const postCreateTrainingBill = (data: CreateTrainingBillProps) =>
POST<string>(`/support/wallet/bill/createTrainingBill`, data);

View File

@@ -1,15 +1,15 @@
import { GET } from '@/web/common/api/request';
import type { PaySchema } from '@fastgpt/global/support/wallet/pay/type.d';
export const getPayOrders = () => GET<PaySchema[]>(`/plusApi/support/wallet/pay/getPayOrders`);
export const getPayOrders = () => GET<PaySchema[]>(`/proApi/support/wallet/pay/getPayOrders`);
export const getPayCode = (amount: number) =>
GET<{
codeUrl: string;
payId: string;
}>(`/plusApi/support/wallet/pay/getPayCode`, { amount });
}>(`/proApi/support/wallet/pay/getPayCode`, { amount });
export const checkPayResult = (payId: string) =>
GET<string>(`/plusApi/support/wallet/pay/checkPayResult`, { payId }).then((data) => {
GET<string>(`/proApi/support/wallet/pay/checkPayResult`, { payId }).then((data) => {
try {
GET('/common/system/unlockTask');
} catch (error) {}

View File

@@ -10,4 +10,4 @@ export const getTeamDatasetValidSub = () =>
}>(`/support/wallet/sub/getDatasetSub`);
export const postExpandTeamDatasetSub = (data: SubDatasetSizeParams) =>
POST('/plusApi/support/wallet/sub/datasetSize/expand', data);
POST('/proApi/support/wallet/sub/datasetSize/expand', data);