feat: Text check before synchronization (#689)

* fix: icon

* fix: web selector

* fix: web selector

* perf: link sync

* dev doc

* chomd doc

* perf: git intro

* 466 intro

* intro img

* add json editor (#5)

* team limit

* websync limit

* json editor

* text editor

* perf: search test

* change cq value type

* doc

* intro img

---------

Co-authored-by: heheer <71265218+newfish-cmyk@users.noreply.github.com>
This commit is contained in:
Archer
2024-01-04 23:19:24 +08:00
committed by GitHub
parent c2abbb579f
commit 828829011a
64 changed files with 1789 additions and 1489 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 548 KiB

After

Width:  |  Height:  |  Size: 166 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 437 KiB

After

Width:  |  Height:  |  Size: 246 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 574 KiB

After

Width:  |  Height:  |  Size: 250 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 270 KiB

After

Width:  |  Height:  |  Size: 182 KiB

View File

@@ -71,6 +71,8 @@ git clone git@github.com:<github_username>/FastGPT.git
### 5. 运行 ### 5. 运行
```bash ```bash
# 给脚本代码执行权限
chmod -R +x ./scripts/
# 代码根目录下执行,会安装根 package、projects 和 packages 内所有依赖 # 代码根目录下执行,会安装根 package、projects 和 packages 内所有依赖
pnpm i pnpm i
# 切换到应用目录 # 切换到应用目录
@@ -105,6 +107,10 @@ docker build -t dockername/fastgpt:tag --build-arg name=app --build-arg proxy=ta
1. 如果你是连接远程的数据库,先检查对应的端口是否开放。 1. 如果你是连接远程的数据库,先检查对应的端口是否开放。
2. 如果是本地运行的数据库,可尝试`host`改成`localhost``127.0.0.1` 2. 如果是本地运行的数据库,可尝试`host`改成`localhost``127.0.0.1`
### sh ./scripts/postinstall.sh 没权限
FastGPT 在`pnpm i`后会执行`postinstall`脚本,用于自动生成`ChakraUI``Type`。如果没有权限,可以先执行`chmod -R +x ./scripts/`,再执行`pnpm i`
### 加入社区 ### 加入社区
遇到困难了吗?有任何问题吗? 加入微信群与开发者和用户保持沟通。 遇到困难了吗?有任何问题吗? 加入微信群与开发者和用户保持沟通。

View File

@@ -31,6 +31,10 @@ OneAPI 中没有配置该模型渠道。
页面中是用 stream=true 模式所以API也需要设置 stream=true 来进行测试。部分模型接口(国产居多)非 Stream 的兼容有点垃圾。 页面中是用 stream=true 模式所以API也需要设置 stream=true 来进行测试。部分模型接口(国产居多)非 Stream 的兼容有点垃圾。
### Incorrect API key provided: sk-xxxx.You can find your api Key at xxx
OneAPI 的 API Key 配置错误,需要修改`OPENAI_API_KEY`环境变量,并重启容器(先 stop 然后 rm 掉,最后再 up -d 运行一次)。可以`exec`进入容器,`env`查看环境变量是否生效。
## Docker 部署常见问题 ## Docker 部署常见问题
### 如何更新? ### 如何更新?
@@ -87,3 +91,15 @@ PG 数据库没有连接上/初始化失败可以查看日志。FastGPT 会
mongo连接失败检查 mongo连接失败检查
1. mongo 服务有没有起来(有些 cpu 不支持 AVX无法用 mongo5需要换成 mongo4.x可以dockerhub找个最新的4.x修改镜像版本重新运行 1. mongo 服务有没有起来(有些 cpu 不支持 AVX无法用 mongo5需要换成 mongo4.x可以dockerhub找个最新的4.x修改镜像版本重新运行
2. 环境变量账号密码注意host和port 2. 环境变量账号密码注意host和port
## 本地开发问题
### TypeError: Cannot read properties of null (reading 'useMemo' )
用 Node18 试试,可能最新的 Node 有问题。 本地开发流程:
1. 根目录: `pnpm i`
2. 复制 `config.json` -> `config.local.json`
3. 复制 `.env.template` -> `.env.local`
4. `cd projects/app`
5. `pnpm dev`

View File

@@ -13,8 +13,11 @@ weight: 830
## V4.6.6 更新说明 ## V4.6.6 更新说明
1. 新增 - 搜索方式:分离向量语义检索,全文检索和重排,通过 RRF 进行排序合并。 1. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8)
2. 优化 - 问题分类提示词id引导。测试国产商用 api 模型(百度阿里智谱讯飞)使用 Prompt 模式均可分类 2. 新增 - Http 模块请求头支持 Json 编辑器
3. UI 优化未来将逐步替换新的UI设计。 3. 新增 - [ReRank模型部署](/docs/development/custom-models/reranker/)
4. 优化代码Icon 抽离和自动化获取 4. 新增 - 搜索方式:分离向量语义检索,全文检索和重排,通过 RRF 进行排序合并
5. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8) 5. 优化 - 问题分类提示词id引导。测试国产商用 api 模型(百度阿里智谱讯飞)使用 Prompt 模式均可分类。
6. UI 优化未来将逐步替换新的UI设计。
7. 优化代码Icon 抽离和自动化获取。
8. 修复 - 链接读取的数据集,未保存选择器,导致同步时不使用选择器。

View File

@@ -12,4 +12,5 @@ export type UrlFetchParams = {
export type UrlFetchResponse = { export type UrlFetchResponse = {
url: string; url: string;
content: string; content: string;
selector?: string;
}[]; }[];

View File

@@ -35,7 +35,7 @@ export function countPromptTokens(
const text = `${role}\n${prompt}`; const text = `${role}\n${prompt}`;
try { try {
const encodeText = enc.encode(text); const encodeText = enc.encode(text);
return encodeText.length + 3; // 补充 role 估算值 return encodeText.length + role.length; // 补充 role 估算值
} catch (error) { } catch (error) {
return text.length; return text.length;
} }

View File

@@ -44,7 +44,8 @@ export type FastGPTFeConfigsType = {
google?: string; google?: string;
}; };
limit?: { limit?: {
exportLimitMinutes?: number; exportDatasetLimitMinutes?: number;
websiteSyncLimitMinuted?: number;
}; };
scripts?: { [key: string]: string }[]; scripts?: { [key: string]: string }[];
favicon?: string; favicon?: string;

View File

@@ -73,6 +73,19 @@ export const DatasetCollectionTrainingTypeMap = {
} }
}; };
export enum DatasetCollectionSyncResultEnum {
sameRaw = 'sameRaw',
success = 'success'
}
export const DatasetCollectionSyncResultMap = {
[DatasetCollectionSyncResultEnum.sameRaw]: {
label: 'core.dataset.collection.sync.result.sameRaw'
},
[DatasetCollectionSyncResultEnum.success]: {
label: 'core.dataset.collection.sync.result.success'
}
};
/* ------------ data -------------- */ /* ------------ data -------------- */
export enum DatasetDataIndexTypeEnum { export enum DatasetDataIndexTypeEnum {
chunk = 'chunk', chunk = 'chunk',
@@ -150,16 +163,24 @@ export enum SearchScoreTypeEnum {
} }
export const SearchScoreTypeMap = { export const SearchScoreTypeMap = {
[SearchScoreTypeEnum.embedding]: { [SearchScoreTypeEnum.embedding]: {
label: 'core.dataset.search.score.embedding' label: 'core.dataset.search.score.embedding',
desc: 'core.dataset.search.score.embedding desc',
showScore: true
}, },
[SearchScoreTypeEnum.fullText]: { [SearchScoreTypeEnum.fullText]: {
label: 'core.dataset.search.score.fullText' label: 'core.dataset.search.score.fullText',
desc: 'core.dataset.search.score.fullText desc',
showScore: false
}, },
[SearchScoreTypeEnum.reRank]: { [SearchScoreTypeEnum.reRank]: {
label: 'core.dataset.search.score.reRank' label: 'core.dataset.search.score.reRank',
desc: 'core.dataset.search.score.reRank desc',
showScore: true
}, },
[SearchScoreTypeEnum.rrf]: { [SearchScoreTypeEnum.rrf]: {
label: 'core.dataset.search.score.rrf' label: 'core.dataset.search.score.rrf',
desc: 'core.dataset.search.score.rrf desc',
showScore: false
} }
}; };

View File

@@ -49,7 +49,10 @@ export type DatasetCollectionSchemaType = {
qaPrompt?: string; qaPrompt?: string;
rawTextLength?: number; rawTextLength?: number;
hashRawText?: string; hashRawText?: string;
metadata?: Record<string, any>; metadata?: {
webPageSelector?: string;
[key: string]: any;
};
}; };
export type DatasetDataIndexItemType = { export type DatasetDataIndexItemType = {

View File

@@ -7,7 +7,10 @@ export enum FlowNodeInputTypeEnum {
slider = 'slider', slider = 'slider',
target = 'target', // data input target = 'target', // data input
switch = 'switch', switch = 'switch',
// editor
textarea = 'textarea', textarea = 'textarea',
JSONEditor = 'JSONEditor',
addInputParam = 'addInputParam', // params input addInputParam = 'addInputParam', // params input

View File

@@ -55,7 +55,7 @@ export const HttpModule: FlowModuleTemplateType = {
}, },
{ {
key: ModuleInputKeyEnum.httpHeader, key: ModuleInputKeyEnum.httpHeader,
type: FlowNodeInputTypeEnum.textarea, type: FlowNodeInputTypeEnum.JSONEditor,
valueType: ModuleIOValueTypeEnum.string, valueType: ModuleIOValueTypeEnum.string,
label: 'core.module.input.label.Http Request Header', label: 'core.module.input.label.Http Request Header',
description: 'core.module.input.description.Http Request Header', description: 'core.module.input.description.Http Request Header',

View File

@@ -10,6 +10,10 @@ export type TeamSchema = {
balance: number; balance: number;
maxSize: number; maxSize: number;
lastDatasetBillTime: Date; lastDatasetBillTime: Date;
limit: {
lastExportDatasetTime: Date;
lastWebsiteSyncTime: Date;
};
}; };
export type TeamMemberSchema = { export type TeamMemberSchema = {

View File

@@ -17,10 +17,6 @@ export type UserModelSchema = {
key: string; key: string;
baseUrl: string; baseUrl: string;
}; };
limit: {
exportKbTime?: Date;
datasetMaxCount?: number;
};
}; };
export type UserType = { export type UserType = {

View File

@@ -15,7 +15,8 @@ export const cheerioToHtml = ({
// get origin url // get origin url
const originUrl = new URL(fetchUrl).origin; const originUrl = new URL(fetchUrl).origin;
const selectDom = $(selector || 'body'); const usedSelector = selector || 'body';
const selectDom = $(usedSelector);
// remove i element // remove i element
selectDom.find('i,script').remove(); selectDom.find('i,script').remove();
@@ -49,7 +50,10 @@ export const cheerioToHtml = ({
.get() .get()
.join('\n'); .join('\n');
return html; return {
html,
usedSelector
};
}; };
export const urlsFetch = async ({ export const urlsFetch = async ({
urlList, urlList,
@@ -66,25 +70,25 @@ export const urlsFetch = async ({
}); });
const $ = cheerio.load(fetchRes.data); const $ = cheerio.load(fetchRes.data);
const { html, usedSelector } = cheerioToHtml({
const md = await htmlToMarkdown( fetchUrl: url,
cheerioToHtml({ $,
fetchUrl: url, selector
$, });
selector const md = await htmlToMarkdown(html);
})
);
return { return {
url, url,
content: md content: md,
selector: usedSelector
}; };
} catch (error) { } catch (error) {
console.log(error, 'fetch error'); console.log(error, 'fetch error');
return { return {
url, url,
content: '' content: '',
selector: ''
}; };
} }
}) })

View File

@@ -21,6 +21,9 @@ export const htmlToMarkdown = (html?: string | null) =>
worker.terminate(); worker.terminate();
reject(err); reject(err);
}); });
worker.on('exit', (code) => {
console.log('html 2 md finish', code);
});
worker.postMessage(html); worker.postMessage(html);
}); });

View File

@@ -19,14 +19,16 @@ export async function createOneCollection({
qaPrompt, qaPrompt,
hashRawText, hashRawText,
rawTextLength, rawTextLength,
metadata = {} metadata = {},
}: CreateDatasetCollectionParams & { teamId: string; tmbId: string }) { ...props
}: CreateDatasetCollectionParams & { teamId: string; tmbId: string; [key: string]: any }) {
const { _id } = await MongoDatasetCollection.create({ const { _id } = await MongoDatasetCollection.create({
name, ...props,
teamId, teamId,
tmbId, tmbId,
datasetId,
parentId: parentId || null, parentId: parentId || null,
datasetId,
name,
type, type,
trainingType, trainingType,
chunkSize, chunkSize,

View File

@@ -75,6 +75,7 @@ const DatasetCollectionSchema = new Schema({
qaPrompt: { qaPrompt: {
type: String type: String
}, },
rawTextLength: { rawTextLength: {
type: Number type: Number
}, },

View File

@@ -1,11 +1,11 @@
import type { CollectionWithDatasetType } from '@fastgpt/global/core/dataset/type.d'; import type { CollectionWithDatasetType } from '@fastgpt/global/core/dataset/type.d';
import { MongoDatasetCollection } from './schema'; import { MongoDatasetCollection } from './schema';
import type { ParentTreePathItemType } from '@fastgpt/global/common/parentFolder/type.d'; import type { ParentTreePathItemType } from '@fastgpt/global/common/parentFolder/type.d';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter'; import { splitText2Chunks } from '@fastgpt/global/common/string/textSplitter';
import { MongoDatasetTraining } from '../training/schema'; import { MongoDatasetTraining } from '../training/schema';
import { urlsFetch } from '../../../common/string/cheerio'; import { urlsFetch } from '../../../common/string/cheerio';
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant'; import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant';
import { hashStr } from '@fastgpt/global/common/string/tools';
/** /**
* get all collection by top collectionId * get all collection by top collectionId
@@ -65,64 +65,114 @@ export function getCollectionUpdateTime({ name, time }: { time?: Date; name: str
return new Date(); return new Date();
} }
/* link collection start load data */ /**
export const loadingOneChunkCollection = async ({ * Get collection raw text by Collection or collectionId
*/
export const getCollectionAndRawText = async ({
collectionId, collectionId,
tmbId, collection,
billId, newRawText
rawText
}: { }: {
collectionId: string; collectionId?: string;
tmbId: string; collection?: CollectionWithDatasetType;
billId?: string; newRawText?: string;
rawText?: string;
}) => { }) => {
const collection = (await MongoDatasetCollection.findById(collectionId).populate( const col = await (async () => {
'datasetId' if (collection) return collection;
)) as CollectionWithDatasetType; if (collectionId) {
return (await MongoDatasetCollection.findById(collectionId).populate(
'datasetId'
)) as CollectionWithDatasetType;
}
if (!collection) { return null;
return Promise.reject(DatasetErrEnum.unCreateCollection); })();
if (!col) {
return Promise.reject('Collection not found');
} }
const newRawText = await (async () => { const rawText = await (async () => {
if (rawText) return rawText; if (newRawText) return newRawText;
// link // link
if (collection.type === DatasetCollectionTypeEnum.link && collection.rawLink) { if (col.type === DatasetCollectionTypeEnum.link && col.rawLink) {
// crawl new data // crawl new data
const result = await urlsFetch({ const result = await urlsFetch({
urlList: [collection.rawLink], urlList: [col.rawLink],
selector: collection.datasetId?.websiteConfig?.selector selector: col.datasetId?.websiteConfig?.selector || col?.metadata?.webPageSelector
}); });
return result[0].content; return result[0].content;
} }
// file // file
return ''; return '';
})(); })();
const hashRawText = hashStr(rawText);
const isSameRawText = col.hashRawText === hashRawText;
return {
collection: col,
rawText,
isSameRawText
};
};
/* link collection start load data */
export const reloadCollectionChunks = async ({
collectionId,
collection,
tmbId,
billId,
rawText
}: {
collectionId?: string;
collection?: CollectionWithDatasetType;
tmbId: string;
billId?: string;
rawText?: string;
}) => {
const {
rawText: newRawText,
collection: col,
isSameRawText
} = await getCollectionAndRawText({
collection,
collectionId,
newRawText: rawText
});
if (isSameRawText) return;
// split data // split data
const { chunks } = splitText2Chunks({ const { chunks } = splitText2Chunks({
text: newRawText, text: newRawText,
chunkLen: collection.chunkSize || 512, chunkLen: col.chunkSize || 512,
countTokens: false countTokens: false
}); });
// insert to training queue // insert to training queue
await MongoDatasetTraining.insertMany( await MongoDatasetTraining.insertMany(
chunks.map((item, i) => ({ chunks.map((item, i) => ({
teamId: collection.teamId, teamId: col.teamId,
tmbId, tmbId,
datasetId: collection.datasetId._id, datasetId: col.datasetId._id,
collectionId: collection._id, collectionId: col._id,
billId, billId,
mode: collection.trainingType, mode: col.trainingType,
prompt: '', prompt: '',
model: collection.datasetId.vectorModel, model: col.datasetId.vectorModel,
q: item, q: item,
a: '', a: '',
chunkIndex: i chunkIndex: i
})) }))
); );
// update raw text
await MongoDatasetCollection.findByIdAndUpdate(col._id, {
rawTextLength: newRawText.length,
hashRawText: hashStr(newRawText)
});
}; };

View File

@@ -47,15 +47,6 @@ const UserSchema = new Schema({
type: Number, type: Number,
default: 15 default: 15
}, },
limit: {
exportKbTime: {
// Every half hour
type: Date
},
datasetMaxCount: {
type: Number
}
},
openaiAccount: { openaiAccount: {
type: { type: {
key: String, key: String,

View File

@@ -32,6 +32,14 @@ const TeamSchema = new Schema({
}, },
lastDatasetBillTime: { lastDatasetBillTime: {
type: Date type: Date
},
limit: {
lastExportDatasetTime: {
type: Date
},
lastWebsiteSyncTime: {
type: Date
}
} }
}); });

View File

@@ -0,0 +1,69 @@
import { MongoTeam } from './team/teamSchema';
/* export dataset limit */
export const updateExportDatasetLimit = async (teamId: string) => {
try {
await MongoTeam.findByIdAndUpdate(teamId, {
'limit.lastExportDatasetTime': new Date()
});
} catch (error) {}
};
export const checkExportDatasetLimit = async ({
teamId,
limitMinutes = 0
}: {
teamId: string;
limitMinutes?: number;
}) => {
const limitMinutesAgo = new Date(Date.now() - limitMinutes * 60 * 1000);
// auth export times
const authTimes = await MongoTeam.findOne(
{
_id: teamId,
$or: [
{ 'limit.lastExportDatasetTime': { $exists: false } },
{ 'limit.lastExportDatasetTime': { $lte: limitMinutesAgo } }
]
},
'_id limit'
);
if (!authTimes) {
return Promise.reject(`每个团队,每 ${limitMinutes} 分钟仅可导出一次。`);
}
};
/* web sync limit */
export const updateWebSyncLimit = async (teamId: string) => {
try {
await MongoTeam.findByIdAndUpdate(teamId, {
'limit.lastWebsiteSyncTime': new Date()
});
} catch (error) {}
};
export const checkWebSyncLimit = async ({
teamId,
limitMinutes = 0
}: {
teamId: string;
limitMinutes?: number;
}) => {
const limitMinutesAgo = new Date(Date.now() - limitMinutes * 60 * 1000);
// auth export times
const authTimes = await MongoTeam.findOne(
{
_id: teamId,
$or: [
{ 'limit.lastWebsiteSyncTime': { $exists: false } },
{ 'limit.lastWebsiteSyncTime': { $lte: limitMinutesAgo } }
]
},
'_id limit'
);
if (!authTimes) {
return Promise.reject(`每个团队,每 ${limitMinutes} 分钟仅使用一次同步功能。`);
}
};

View File

@@ -15,6 +15,7 @@ export const iconPaths = {
'common/confirm/deleteTip': () => import('./icons/common/confirm/deleteTip.svg'), 'common/confirm/deleteTip': () => import('./icons/common/confirm/deleteTip.svg'),
'common/courseLight': () => import('./icons/common/courseLight.svg'), 'common/courseLight': () => import('./icons/common/courseLight.svg'),
'common/customTitleLight': () => import('./icons/common/customTitleLight.svg'), 'common/customTitleLight': () => import('./icons/common/customTitleLight.svg'),
'common/editor/resizer': () => import('./icons/common/editor/resizer.svg'),
'common/file/move': () => import('./icons/common/file/move.svg'), 'common/file/move': () => import('./icons/common/file/move.svg'),
'common/fullScreenLight': () => import('./icons/common/fullScreenLight.svg'), 'common/fullScreenLight': () => import('./icons/common/fullScreenLight.svg'),
'common/gitFill': () => import('./icons/common/gitFill.svg'), 'common/gitFill': () => import('./icons/common/gitFill.svg'),
@@ -28,10 +29,12 @@ export const iconPaths = {
'common/navbar/pluginFill': () => import('./icons/common/navbar/pluginFill.svg'), 'common/navbar/pluginFill': () => import('./icons/common/navbar/pluginFill.svg'),
'common/navbar/pluginLight': () => import('./icons/common/navbar/pluginLight.svg'), 'common/navbar/pluginLight': () => import('./icons/common/navbar/pluginLight.svg'),
'common/overviewLight': () => import('./icons/common/overviewLight.svg'), 'common/overviewLight': () => import('./icons/common/overviewLight.svg'),
'common/paramsLight': () => import('./icons/common/paramsLight.svg'),
'common/playFill': () => import('./icons/common/playFill.svg'), 'common/playFill': () => import('./icons/common/playFill.svg'),
'common/playLight': () => import('./icons/common/playLight.svg'), 'common/playLight': () => import('./icons/common/playLight.svg'),
'common/questionLight': () => import('./icons/common/questionLight.svg'), 'common/questionLight': () => import('./icons/common/questionLight.svg'),
'common/refreshLight': () => import('./icons/common/refreshLight.svg'), 'common/refreshLight': () => import('./icons/common/refreshLight.svg'),
'common/resultLight': () => import('./icons/common/resultLight.svg'),
'common/retryLight': () => import('./icons/common/retryLight.svg'), 'common/retryLight': () => import('./icons/common/retryLight.svg'),
'common/rightArrowLight': () => import('./icons/common/rightArrowLight.svg'), 'common/rightArrowLight': () => import('./icons/common/rightArrowLight.svg'),
'common/routePushLight': () => import('./icons/common/routePushLight.svg'), 'common/routePushLight': () => import('./icons/common/routePushLight.svg'),

View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1704259732773" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="4183" xmlns:xlink="http://www.w3.org/1999/xlink" width="200" height="200"><path d="M319.20128 974.56128L348.16 1003.52l655.36-655.36-28.95872-28.95872-655.36 655.36zM675.84 1003.52l327.68-327.68-28.95872-28.95872-327.68 327.68L675.84 1003.52z" fill="#000000" p-id="4184"></path></svg>

After

Width:  |  Height:  |  Size: 534 B

View File

@@ -0,0 +1,4 @@
<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
<path fill-rule="evenodd" clip-rule="evenodd"
d="M4.16663 1.66663C4.62686 1.66663 4.99996 2.03972 4.99996 2.49996V5.83329C4.99996 6.29353 4.62686 6.66663 4.16663 6.66663C3.70639 6.66663 3.33329 6.29353 3.33329 5.83329V2.49996C3.33329 2.03972 3.70639 1.66663 4.16663 1.66663ZM9.99996 1.66663C10.4602 1.66663 10.8333 2.03972 10.8333 2.49996V5.14221C11.8043 5.48541 12.5 6.41144 12.5 7.49996C12.5 8.88067 11.3807 9.99996 9.99996 9.99996C8.61925 9.99996 7.49996 8.88067 7.49996 7.49996C7.49996 6.41144 8.19563 5.48541 9.16663 5.14221V2.49996C9.16663 2.03972 9.53972 1.66663 9.99996 1.66663ZM15.8333 1.66663C16.2935 1.66663 16.6666 2.03972 16.6666 2.49996V7.49996C16.6666 7.9602 16.2935 8.33329 15.8333 8.33329C15.3731 8.33329 15 7.9602 15 7.49996V2.49996C15 2.03972 15.3731 1.66663 15.8333 1.66663ZM9.99996 6.66663C9.53972 6.66663 9.16663 7.03972 9.16663 7.49996C9.16663 7.9602 9.53972 8.33329 9.99996 8.33329C10.4602 8.33329 10.8333 7.9602 10.8333 7.49996C10.8333 7.03972 10.4602 6.66663 9.99996 6.66663ZM4.16663 9.99996C3.70639 9.99996 3.33329 10.3731 3.33329 10.8333C3.33329 11.2935 3.70639 11.6666 4.16663 11.6666C4.62686 11.6666 4.99996 11.2935 4.99996 10.8333C4.99996 10.3731 4.62686 9.99996 4.16663 9.99996ZM1.66663 10.8333C1.66663 9.45258 2.78591 8.33329 4.16663 8.33329C5.54734 8.33329 6.66663 9.45258 6.66663 10.8333C6.66663 11.9218 5.97095 12.8478 4.99996 13.191V17.5C4.99996 17.9602 4.62686 18.3333 4.16663 18.3333C3.70639 18.3333 3.33329 17.9602 3.33329 17.5L3.33329 13.191C2.3623 12.8478 1.66663 11.9218 1.66663 10.8333ZM15.8333 11.6666C15.3731 11.6666 15 12.0397 15 12.5C15 12.9602 15.3731 13.3333 15.8333 13.3333C16.2935 13.3333 16.6666 12.9602 16.6666 12.5C16.6666 12.0397 16.2935 11.6666 15.8333 11.6666ZM13.3333 12.5C13.3333 11.1192 14.4526 9.99996 15.8333 9.99996C17.214 9.99996 18.3333 11.1192 18.3333 12.5C18.3333 13.5885 17.6376 14.5145 16.6666 14.8577V17.5C16.6666 17.9602 16.2935 18.3333 15.8333 18.3333C15.3731 18.3333 15 17.9602 15 17.5V14.8577C14.029 14.5145 13.3333 13.5885 13.3333 12.5ZM9.99996 11.6666C10.4602 11.6666 10.8333 12.0397 10.8333 12.5V17.5C10.8333 17.9602 10.4602 18.3333 9.99996 18.3333C9.53972 18.3333 9.16663 17.9602 9.16663 17.5V12.5C9.16663 12.0397 9.53972 11.6666 9.99996 11.6666Z" />
</svg>

After

Width:  |  Height:  |  Size: 2.3 KiB

View File

@@ -0,0 +1,10 @@
<svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 20 20" fill="none">
<path fill-rule="evenodd" clip-rule="evenodd"
d="M14.4508 3.34193C15.5145 3.34207 16.0569 3.34752 16.4738 3.55992C16.8501 3.75166 17.1561 4.05762 17.3478 4.43395C17.5658 4.86177 17.5658 5.42182 17.5658 6.54193V15.1447C17.5658 16.2648 17.5658 16.8249 17.3478 17.2527C17.1561 17.629 16.8501 17.935 16.4738 18.1268C16.046 18.3447 15.4859 18.3447 14.3658 18.3447H5.7658C4.64569 18.3447 4.08564 18.3447 3.65781 18.1268C3.28149 17.935 2.97553 17.629 2.78378 17.2527C2.5658 16.8249 2.5658 16.2648 2.5658 15.1447V6.54193C2.5658 5.42182 2.5658 4.86177 2.78378 4.43395C2.97553 4.05762 3.28149 3.75166 3.65781 3.55992C4.07464 3.34753 4.61699 3.34207 5.68053 3.34193C5.70859 3.34193 5.73701 3.34193 5.7658 3.34193H7.35174C7.39932 2.69822 7.67081 2.1168 8.08862 1.67526C8.19058 1.56751 8.30125 1.46809 8.4195 1.37813C8.8766 1.0304 9.44704 0.823975 10.0657 0.823975C10.6843 0.823975 11.2548 1.0304 11.7119 1.37813C11.8301 1.46809 11.9408 1.56751 12.0428 1.67526C12.4606 2.1168 12.7321 2.69822 12.7796 3.34193H14.3658C14.3945 3.34193 14.4229 3.34193 14.4508 3.34193ZM14.3658 5.0086H5.7658C5.17824 5.0086 4.83781 5.00989 4.58792 5.03031C4.47782 5.0393 4.41959 5.05004 4.39536 5.05546C4.34762 5.08386 4.30773 5.12376 4.27933 5.1715C4.27391 5.19572 4.26317 5.25395 4.25418 5.36405C4.23376 5.61395 4.23246 5.95438 4.23246 6.54193V15.1447C4.23246 15.7323 4.23376 16.0727 4.25418 16.3226C4.26317 16.4327 4.27391 16.4909 4.27933 16.5152C4.30773 16.5629 4.34762 16.6028 4.39536 16.6312C4.41959 16.6366 4.47782 16.6474 4.58792 16.6564C4.83781 16.6768 5.17824 16.6781 5.7658 16.6781H14.3658C14.9533 16.6781 15.2938 16.6768 15.5437 16.6564C15.6538 16.6474 15.712 16.6366 15.7362 16.6312C15.784 16.6028 15.8239 16.5629 15.8523 16.5152C15.8577 16.491 15.8684 16.4327 15.8774 16.3226C15.8978 16.0727 15.8991 15.7323 15.8991 15.1447V6.54193C15.8991 5.95438 15.8978 5.61395 15.8774 5.36405C15.8684 5.25395 15.8577 5.19572 15.8523 5.1715C15.8239 5.12376 15.784 5.08386 15.7362 5.05546C15.712 5.05004 15.6538 5.0393 15.5437 5.03031C15.2938 5.00989 14.9533 5.0086 14.3658 5.0086ZM15.7501 5.05895L15.7489 5.05857L15.7501 5.05895ZM15.8488 16.5291L15.8492 16.5278L15.8488 16.5291ZM4.38147 16.6277L4.38274 16.6281L4.38147 16.6277ZM9.35453 2.76643C9.19102 2.91579 9.07469 3.11595 9.03053 3.34193H11.1008C11.0567 3.11595 10.9404 2.91579 10.7768 2.76643C10.5893 2.59513 10.3397 2.49064 10.0657 2.49064C9.79167 2.49064 9.54207 2.59513 9.35453 2.76643Z" />
<path
d="M6.0281 7.50999C6.0281 7.04976 6.4012 6.67666 6.86144 6.67666H13.4319C13.8921 6.67666 14.2652 7.04976 14.2652 7.50999C14.2652 7.97023 13.8921 8.34333 13.4319 8.34333H6.86144C6.4012 8.34333 6.0281 7.97023 6.0281 7.50999Z" />
<path
d="M6.0281 10.8433C6.0281 10.3831 6.4012 10.01 6.86144 10.01H13.4319C13.8921 10.01 14.2652 10.3831 14.2652 10.8433C14.2652 11.3036 13.8921 11.6767 13.4319 11.6767H6.86144C6.4012 11.6767 6.0281 11.3036 6.0281 10.8433Z" />
<path
d="M6.0281 14.1767C6.0281 13.7164 6.4012 13.3433 6.86144 13.3433H10.0657C10.5259 13.3433 10.899 13.7164 10.899 14.1767C10.899 14.6369 10.5259 15.01 10.0657 15.01H6.86144C6.4012 15.01 6.0281 14.6369 6.0281 14.1767Z" />
</svg>

After

Width:  |  Height:  |  Size: 3.2 KiB

View File

@@ -1,8 +1,6 @@
<?xml version="1.0" standalone="no"?> <svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 14 14" fill="none">
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1698125580538"
class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="8233"
xmlns:xlink="http://www.w3.org/1999/xlink">
<path <path
d="M288.633235 288.699238l446.731484 0L735.364719 421.267533l-13.261025-0.088004c-7.720845-30.709419-16.270569-52.302203-25.91216-65.759703-9.639544-13.458523-22.638603-24.233937-39.127136-32.477692-9.16166-4.03285-25.345248-6.083555-48.682769-6.083555l-51.123354 0-0.611937 336.608388c-1.394766 23.839965 1.394766 18.059308 0 33.261545-0.611937 6.783497 13.173021 20.022009 21.375843 24.733311 8.28878 4.602831 33.500998 5.692652 47.63388 5.692652l19.107173 0 0 18.735713L378.145921 735.890186 378.145921 715.88762l21.98471 0c14.394848 0 31.495318-1.176802 40.045041-6.194072 6.457062-3.533476 26.959002-1.373277 27.176966-24.25338 0.87288-31.494295 2.7486-8.723685 0-31.974225L467.352639 316.857555l-50.906413 0c-33.414017 0-57.537438 6.957459-72.5432 21.00336-21.331841 19.629059-34.767851 47.197952-40.35101 83.491553l-15.485693 0L288.633235 288.699238zM958.709483 244.030899l0 536.025183c0 98.629321-79.960123 178.608887-178.718381 178.608887L243.964896 958.664969c-98.585319 0-178.674379-79.979566-178.674379-178.608887L65.290517 244.030899c0-98.717326 80.090083-178.696892 178.674379-178.696892l536.026206 0C878.74936 65.334008 958.709483 145.314597 958.709483 244.030899zM869.413737 288.699238c0-74.003458-60.067051-134.071532-134.049019-134.071532L288.633235 154.627706c-73.981968 0-133.919059 60.067051-133.919059 134.071532l0 446.601524c0 74.091462 59.937091 134.049019 133.919059 134.049019l446.731484 0c73.981968 0 134.049019-59.957557 134.049019-134.049019L869.413737 288.699238z" d="M4.1417 4.31668V5.83334C4.1417 5.96221 4.24617 6.06668 4.37503 6.06668H5.07503C5.2039 6.06668 5.30837 5.96221 5.30837 5.83334V5.25001H6.47503V8.77626H5.80068C5.67182 8.77626 5.56735 8.88072 5.56735 9.00959V9.70959C5.56735 9.83845 5.67182 9.94292 5.80068 9.94292H8.31605C8.44492 9.94292 8.54938 9.83845 8.54938 9.70959V9.00959C8.54938 8.88072 8.44492 8.77626 8.31605 8.77626H7.6417V5.25001H8.80837V5.83334C8.80837 5.96221 8.91283 6.06668 9.0417 6.06668H9.7417C9.87057 6.06668 9.97503 5.96221 9.97503 5.83334V4.31668C9.97503 4.18781 9.87057 4.08334 9.7417 4.08334H4.37503C4.24617 4.08334 4.1417 4.18781 4.1417 4.31668Z" />
p-id="8234"></path> <path fill-rule="evenodd" clip-rule="evenodd"
d="M4.52587 1.16675H9.47405C9.9436 1.16674 10.3312 1.16674 10.6469 1.19253C10.9747 1.21932 11.2762 1.27681 11.5593 1.42107C11.9983 1.64477 12.3553 2.00173 12.579 2.44077C12.7232 2.72388 12.7807 3.02529 12.8075 3.35318C12.8333 3.66888 12.8333 4.05642 12.8333 4.52598V9.47419C12.8333 9.94375 12.8333 10.3313 12.8075 10.647C12.7807 10.9749 12.7232 11.2763 12.579 11.5594C12.3553 11.9984 11.9983 12.3554 11.5593 12.5791C11.2762 12.7234 10.9747 12.7808 10.6469 12.8076C10.3312 12.8334 9.94362 12.8334 9.47407 12.8334H4.52591C4.05633 12.8334 3.66877 12.8334 3.35306 12.8076C3.02517 12.7808 2.72376 12.7233 2.44065 12.5791C2.0016 12.3554 1.64465 11.9984 1.42094 11.5594C1.27669 11.2763 1.2192 10.9749 1.19241 10.647C1.16661 10.3313 1.16662 9.94374 1.16663 9.47417V4.52599C1.16662 4.05643 1.16661 3.66888 1.19241 3.35318C1.2192 3.02529 1.27669 2.72388 1.42094 2.44077C1.64465 2.00173 2.0016 1.64477 2.44065 1.42107C2.72376 1.27681 3.02517 1.21932 3.35306 1.19253C3.66876 1.16673 4.05631 1.16674 4.52587 1.16675ZM3.44807 2.35532C3.19232 2.37622 3.06154 2.41409 2.9703 2.46057C2.75078 2.57243 2.5723 2.7509 2.46045 2.97043C2.41397 3.06166 2.37609 3.19245 2.3552 3.44819C2.33375 3.71074 2.33329 4.05041 2.33329 4.55008V9.45008C2.33329 9.94975 2.33375 10.2894 2.3552 10.552C2.37609 10.8077 2.41397 10.9385 2.46045 11.0297C2.5723 11.2493 2.75078 11.4277 2.9703 11.5396C3.06154 11.5861 3.19232 11.6239 3.44807 11.6448C3.71062 11.6663 4.05029 11.6667 4.54996 11.6667H9.44996C9.94963 11.6667 10.2893 11.6663 10.5519 11.6448C10.8076 11.6239 10.9384 11.5861 11.0296 11.5396C11.2491 11.4277 11.4276 11.2493 11.5395 11.0297C11.586 10.9385 11.6238 10.8077 11.6447 10.552C11.6662 10.2894 11.6666 9.94975 11.6666 9.45008V4.55008C11.6666 4.05041 11.6662 3.71074 11.6447 3.44819C11.6238 3.19245 11.586 3.06166 11.5395 2.97043C11.4276 2.75091 11.2491 2.57243 11.0296 2.46058C10.9384 2.41409 10.8076 2.37622 10.5519 2.35532C10.2893 2.33387 9.94963 2.33342 9.44996 2.33342L4.54996 2.33342C4.05029 2.33342 3.71062 2.33387 3.44807 2.35532Z" />
</svg> </svg>

Before

Width:  |  Height:  |  Size: 1.8 KiB

After

Width:  |  Height:  |  Size: 2.8 KiB

View File

@@ -1 +1,4 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg t="1686557412109" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2150" xmlns:xlink="http://www.w3.org/1999/xlink" ><path d="M511.998 64C264.574 64 64 264.574 64 511.998S264.574 960 511.998 960 960 759.422 960 511.998 759.422 64 511.998 64z m353.851 597.438c-82.215 194.648-306.657 285.794-501.306 203.579S78.749 558.36 160.964 363.711 467.621 77.917 662.27 160.132c168.009 70.963 262.57 250.652 225.926 429.313a383.995 383.995 0 0 1-22.347 71.993z" p-id="2151"></path><path d="M543.311 498.639V256.121c0-17.657-14.314-31.97-31.97-31.97s-31.97 14.314-31.97 31.97v269.005l201.481 201.481c12.485 12.485 32.728 12.485 45.213 0s12.485-32.728 0-45.213L543.311 498.639z" p-id="2152"></path></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20">
<path fill-rule="evenodd" clip-rule="evenodd"
d="M10 3.08634C6.18178 3.08634 3.08647 6.18166 3.08647 9.99992C3.08647 13.8182 6.18178 16.9135 10 16.9135C13.8183 16.9135 16.9136 13.8182 16.9136 9.99992C16.9136 6.18166 13.8183 3.08634 10 3.08634ZM1.4198 9.99992C1.4198 5.26118 5.26131 1.41968 10 1.41968C14.7388 1.41968 18.5803 5.26118 18.5803 9.99992C18.5803 14.7387 14.7388 18.5802 10 18.5802C5.26131 18.5802 1.4198 14.7387 1.4198 9.99992ZM10 4.51844C10.4603 4.51844 10.8334 4.89154 10.8334 5.35178V9.48489L13.4715 10.8039C13.8831 11.0098 14.05 11.5103 13.8442 11.922C13.6383 12.3336 13.1378 12.5005 12.7261 12.2947L9.62737 10.7453C9.34505 10.6041 9.16671 10.3156 9.16671 9.99992V5.35178C9.16671 4.89154 9.53981 4.51844 10 4.51844Z" />
</svg>

Before

Width:  |  Height:  |  Size: 875 B

After

Width:  |  Height:  |  Size: 814 B

View File

@@ -0,0 +1,113 @@
import React from 'react';
import Editor from '@monaco-editor/react';
import { useCallback, useRef, useState } from 'react';
import { Box, BoxProps } from '@chakra-ui/react';
import MyIcon from '../../Icon';
type Props = Omit<BoxProps, 'onChange' | 'resize' | 'height'> & {
height?: number;
resize?: boolean;
defaultValue?: string;
value?: string;
onChange?: (e: string) => void;
};
const options = {
lineNumbers: 'off',
guides: {
indentation: false
},
automaticLayout: true,
minimap: {
enabled: false
},
scrollbar: {
verticalScrollbarSize: 4,
horizontalScrollbarSize: 8,
alwaysConsumeMouseWheel: false
},
lineNumbersMinChars: 0,
fontSize: 12,
scrollBeyondLastLine: false,
folding: false,
overviewRulerBorder: false,
tabSize: 2
};
const JSONEditor = ({ defaultValue, value, onChange, resize, ...props }: Props) => {
const [height, setHeight] = useState(props.height || 100);
const initialY = useRef(0);
const handleMouseDown = useCallback((e: React.MouseEvent) => {
initialY.current = e.clientY;
const handleMouseMove = (e: MouseEvent) => {
const deltaY = e.clientY - initialY.current;
setHeight((prevHeight) => (prevHeight + deltaY < 100 ? 100 : prevHeight + deltaY));
initialY.current = e.clientY;
};
const handleMouseUp = () => {
document.removeEventListener('mousemove', handleMouseMove);
document.removeEventListener('mouseup', handleMouseUp);
};
document.addEventListener('mousemove', handleMouseMove);
document.addEventListener('mouseup', handleMouseUp);
}, []);
return (
<Box position={'relative'}>
{resize && (
<Box
position={'absolute'}
right={'0'}
bottom={'0'}
zIndex={999}
cursor={'ns-resize'}
px={'4px'}
onMouseDown={handleMouseDown}
>
<MyIcon name={'common/editor/resizer'} width={'16px'} height={'16px'} />
</Box>
)}
<Box
borderWidth={'1px'}
borderRadius={'base'}
borderColor={'myGray.200'}
py={2}
{...props}
height={'auto'}
>
<Editor
height={height}
defaultLanguage="json"
options={options as any}
theme={'JSONEditorTheme'}
beforeMount={(monaco) => {
monaco?.editor.defineTheme('JSONEditorTheme', {
base: 'vs',
inherit: true,
rules: [],
colors: {
'editor.background': '#ffffff00',
'editorLineNumber.foreground': '#aaa',
'editorOverviewRuler.border': '#ffffff00',
'editor.lineHighlightBackground': '#F7F8FA',
'scrollbarSlider.background': '#E8EAEC',
'editorIndentGuide.activeBackground': '#ddd',
'editorIndentGuide.background': '#eee'
}
});
}}
defaultValue={defaultValue}
value={value}
onChange={(e) => onChange?.(e || '')}
/>
</Box>
</Box>
);
};
export default React.memo(JSONEditor);

View File

@@ -13,6 +13,7 @@
"@chakra-ui/system": "^2.6.1", "@chakra-ui/system": "^2.6.1",
"@emotion/react": "^11.11.1", "@emotion/react": "^11.11.1",
"@emotion/styled": "^11.11.0", "@emotion/styled": "^11.11.0",
"@monaco-editor/react": "^4.6.0",
"react": "18.2.0", "react": "18.2.0",
"react-dom": "18.2.0", "react-dom": "18.2.0",
"i18next": "^22.5.1", "i18next": "^22.5.1",

1391
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,3 +1,4 @@
packages: packages:
- 'packages/*' - 'packages/*'
- 'projects/*' - 'projects/*'
- 'scripts/icon'

View File

@@ -1,14 +1,12 @@
### Fast GPT V4.6.6 ### Fast GPT V4.6.6
1. 新增 - [问题补全模块](https://doc.fastgpt.in/docs/workflow/modules/coreferenceresolution/) 1. 新增 - Http 模块请求头支持 Json 编辑器。
2. 新增 - [文本编辑模块](https://doc.fastgpt.in/docs/workflow/modules/text_editor/) 2. 新增 - 搜索方式:分离向量语义检索,全文检索和重排,通过 RRF 进行排序合并。
3. 新增 - [判断器模块](https://doc.fastgpt.in/docs/workflow/modules/tfswitch/) 3. 新增 - [问题补全模块](https://doc.fastgpt.in/docs/workflow/modules/coreferenceresolution/)
4. 新增 - [自定义反馈模块](https://doc.fastgpt.in/docs/workflow/modules/custom_feedback/) 5. 新增 - [文本编辑模块](https://doc.fastgpt.in/docs/workflow/modules/text_editor/)
5. 新增 - 【内容提取】模块支持选择模型,以及字段枚举 6. 新增 - [判断器模块](https://doc.fastgpt.in/docs/workflow/modules/tfswitch/)
6. 优化 - docx读取兼容表格表格转markdown 7. 新增 - [自定义反馈模块](https://doc.fastgpt.in/docs/workflow/modules/custom_feedback/)
7. 优化 - 高级编排连接线交互 8. 新增 - 【内容提取】模块支持选择模型,以及字段枚举
8. 优化 - 由于 html2md 导致的 cpu密集计算阻断线程问题 9. [使用文档](https://doc.fastgpt.in/docs/intro/)
9. 修复 - 高级编排提示词提取描述 10. [点击查看高级编排介绍文档](https://doc.fastgpt.in/docs/workflow)
10. [使用文档](https://doc.fastgpt.in/docs/intro/) 11. [点击查看商业版](https://doc.fastgpt.in/docs/commercial/)
11. [点击查看高级编排介绍文档](https://doc.fastgpt.in/docs/workflow)
12. [点击查看商业版](https://doc.fastgpt.in/docs/commercial/)

View File

@@ -410,6 +410,7 @@
"Read Metadata": "Read Metadata", "Read Metadata": "Read Metadata",
"Training Type": "Training Type", "Training Type": "Training Type",
"Updatetime": "Update Time", "Updatetime": "Update Time",
"Web page selector": "Web Selector",
"metadata": "Metadata", "metadata": "Metadata",
"read source": "Read Source", "read source": "Read Source",
"source": "Source", "source": "Source",
@@ -420,6 +421,12 @@
"active": "Ready", "active": "Ready",
"syncing": "Syncing" "syncing": "Syncing"
}, },
"sync": {
"result": {
"sameRaw": "The content has not changed and no update is required.",
"success": "Start synchronization"
}
},
"training": { "training": {
"type chunk": "Chunk", "type chunk": "Chunk",
"type manual": "Manual", "type manual": "Manual",

View File

@@ -410,6 +410,7 @@
"Read Metadata": "查看元数据", "Read Metadata": "查看元数据",
"Training Type": "训练模式", "Training Type": "训练模式",
"Updatetime": "更新时间", "Updatetime": "更新时间",
"Web page selector": "网站选择器",
"metadata": "元数据", "metadata": "元数据",
"read source": "查看原始内容", "read source": "查看原始内容",
"source": "数据来源", "source": "数据来源",
@@ -420,6 +421,12 @@
"active": "已就绪", "active": "已就绪",
"syncing": "同步中" "syncing": "同步中"
}, },
"sync": {
"result": {
"sameRaw": "内容未变动,无需更新",
"success": "开始同步"
}
},
"training": { "training": {
"type chunk": "直接分段", "type chunk": "直接分段",
"type manual": "手动", "type manual": "手动",
@@ -515,9 +522,13 @@
}, },
"score": { "score": {
"embedding": "语义检索", "embedding": "语义检索",
"embedding desc": "通过计算向量之间的距离获取得分,范围为 0~1。",
"fullText": "全文检索", "fullText": "全文检索",
"fullText desc": "计算相同关键词的得分,范围为 0~无穷。",
"reRank": "结果重排", "reRank": "结果重排",
"rrf": "RRF 合并" "reRank desc": "通过 ReRank 模型计算句子之间的关联度,范围为 0~1。",
"rrf": "综合排名",
"rrf desc": "通过倒排计算的方式,合并多个检索结果。"
}, },
"search mode": "搜索模式" "search mode": "搜索模式"
}, },

View File

@@ -28,14 +28,14 @@ const NavbarPhone = ({ unread }: { unread: number }) => {
}, },
{ {
label: t('navbar.Tools'), label: t('navbar.Tools'),
icon: 'phoneTabbar/tabbarMore', icon: 'phoneTabbar/more',
link: '/tools', link: '/tools',
activeLink: ['/tools'], activeLink: ['/tools'],
unread: 0 unread: 0
}, },
{ {
label: t('navbar.Account'), label: t('navbar.Account'),
icon: 'phoneTabbar/tabbarMe', icon: 'phoneTabbar/me',
link: '/account', link: '/account',
activeLink: ['/account'], activeLink: ['/account'],
unread unread

View File

@@ -1,5 +1,5 @@
import React, { useMemo, useState } from 'react'; import React, { useMemo, useState } from 'react';
import { Box, Flex, Link, Progress, useTheme } from '@chakra-ui/react'; import { Box, Flex, Link, Progress } from '@chakra-ui/react';
import { import {
type InputDataType, type InputDataType,
RawSourceText RawSourceText
@@ -9,7 +9,6 @@ import NextLink from 'next/link';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import MyTooltip from '@/components/MyTooltip'; import MyTooltip from '@/components/MyTooltip';
import { useSystemStore } from '@/web/common/system/useSystemStore';
import dynamic from 'next/dynamic'; import dynamic from 'next/dynamic';
import MyBox from '@/components/common/MyBox'; import MyBox from '@/components/common/MyBox';
import { getDatasetDataItemById } from '@/web/core/dataset/api'; import { getDatasetDataItemById } from '@/web/core/dataset/api';
@@ -19,6 +18,36 @@ import { SearchScoreTypeEnum, SearchScoreTypeMap } from '@fastgpt/global/core/da
const InputDataModal = dynamic(() => import('@/pages/dataset/detail/components/InputDataModal')); const InputDataModal = dynamic(() => import('@/pages/dataset/detail/components/InputDataModal'));
type ScoreItemType = SearchDataResponseItemType['score'][0];
const scoreTheme: Record<
string,
{
color: string;
bg: string;
borderColor: string;
colorSchema: string;
}
> = {
'0': {
color: '#6F5DD7',
bg: '#F0EEFF',
borderColor: '#D3CAFF',
colorSchema: 'purple'
},
'1': {
color: '#9E53C1',
bg: '#FAF1FF',
borderColor: '#ECF',
colorSchema: 'pink'
},
'2': {
color: '#0884DD',
bg: '#F0FBFF',
borderColor: '#BCE7FF',
colorSchema: 'blue'
}
};
const QuoteItem = ({ const QuoteItem = ({
quoteItem, quoteItem,
canViewSource, canViewSource,
@@ -29,8 +58,6 @@ const QuoteItem = ({
linkToDataset?: boolean; linkToDataset?: boolean;
}) => { }) => {
const { t } = useTranslation(); const { t } = useTranslation();
const { isPc } = useSystemStore();
const theme = useTheme();
const [editInputData, setEditInputData] = useState<InputDataType & { collectionId: string }>(); const [editInputData, setEditInputData] = useState<InputDataType & { collectionId: string }>();
const { mutate: onclickEdit, isLoading } = useRequest({ const { mutate: onclickEdit, isLoading } = useRequest({
@@ -43,54 +70,46 @@ const QuoteItem = ({
errorToast: t('core.dataset.data.get data error') errorToast: t('core.dataset.data.get data error')
}); });
const rank = useMemo(() => {
if (quoteItem.score.length === 1) {
return quoteItem.score[0].index;
}
const rrf = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.rrf);
if (rrf) return rrf.index;
return 0;
}, [quoteItem.score]);
const score = useMemo(() => { const score = useMemo(() => {
let searchScore: number | undefined = undefined; if (!Array.isArray(quoteItem.score)) {
let text = ''; return {
primaryScore: undefined,
const reRankScore = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.reRank); secondaryScore: []
if (reRankScore) { };
searchScore = reRankScore.value;
text = t('core.dataset.search.Rerank score');
} }
const embScore = quoteItem.score?.find((item) => item.type === SearchScoreTypeEnum.embedding); // rrf -> rerank -> embedding -> fullText 优先级
if (embScore && quoteItem.score.length === 1) { let rrfScore: ScoreItemType | undefined = undefined;
searchScore = embScore.value; let reRankScore: ScoreItemType | undefined = undefined;
text = t('core.dataset.search.Embedding score'); let embeddingScore: ScoreItemType | undefined = undefined;
} let fullTextScore: ScoreItemType | undefined = undefined;
const detailScore = (() => { quoteItem.score.forEach((item) => {
if (Array.isArray(quoteItem.score)) { if (item.type === SearchScoreTypeEnum.rrf) {
return quoteItem.score rrfScore = item;
.map( } else if (item.type === SearchScoreTypeEnum.reRank) {
(item) => reRankScore = item;
`${t('core.dataset.search.Search type')}: ${t(SearchScoreTypeMap[item.type]?.label)} } else if (item.type === SearchScoreTypeEnum.embedding) {
${t('core.dataset.search.Rank')}: ${item.index + 1} embeddingScore = item;
${t('core.dataset.search.Score')}: ${item.value.toFixed(4)}` } else if (item.type === SearchScoreTypeEnum.fullText) {
) fullTextScore = item;
.join('\n----\n');
} }
return 'null'; });
})();
const primaryScore = (rrfScore ||
reRankScore ||
embeddingScore ||
fullTextScore) as unknown as ScoreItemType;
const secondaryScore = [rrfScore, reRankScore, embeddingScore, fullTextScore].filter(
// @ts-ignore
(item) => item && primaryScore && item.type !== primaryScore.type
) as unknown as ScoreItemType[];
return { return {
value: searchScore, primaryScore,
tip: t('core.dataset.Search score tip', { secondaryScore
scoreText: text ? `${text}\n` : text,
detailScore
})
}; };
}, [quoteItem.score, t]); }, [quoteItem.score]);
return ( return (
<> <>
@@ -101,86 +120,97 @@ ${t('core.dataset.search.Score')}: ${item.value.toFixed(4)}`
fontSize={'sm'} fontSize={'sm'}
whiteSpace={'pre-wrap'} whiteSpace={'pre-wrap'}
_hover={{ '& .hover-data': { display: 'flex' } }} _hover={{ '& .hover-data': { display: 'flex' } }}
h={'100%'}
display={'flex'}
flexDirection={'column'}
> >
<Flex alignItems={'flex-end'} mb={3}> <Flex alignItems={'center'} mb={3}>
{rank !== undefined && ( {score?.primaryScore && (
<MyTooltip label={t('core.dataset.search.Rank Tip')}> <MyTooltip label={t(SearchScoreTypeMap[score.primaryScore.type]?.desc)}>
<Box px={2} py={'3px'} mr={3} bg={'myGray.200'} borderRadius={'md'}> <Flex
# {rank + 1} px={'12px'}
</Box> py={'5px'}
mr={4}
borderRadius={'md'}
color={'primary.700'}
bg={'primary.50'}
borderWidth={'1px'}
borderColor={'primary.200'}
alignItems={'center'}
fontSize={'sm'}
>
<Box>#{score.primaryScore.index + 1}</Box>
<Box borderRightColor={'primary.700'} borderRightWidth={'1px'} h={'14px'} mx={2} />
<Box>
{t(SearchScoreTypeMap[score.primaryScore.type]?.label)}
{SearchScoreTypeMap[score.primaryScore.type]?.showScore
? ` ${score.primaryScore.value.toFixed(4)}`
: ''}
</Box>
</Flex>
</MyTooltip> </MyTooltip>
)} )}
{score.secondaryScore.map((item, i) => (
<RawSourceText <MyTooltip key={item.type} label={t(SearchScoreTypeMap[item.type]?.desc)}>
fontWeight={'bold'} <Box fontSize={'xs'} mr={3}>
color={'black'} <Flex alignItems={'flex-start'} lineHeight={1.2} mb={1}>
sourceName={quoteItem.sourceName} <Box
sourceId={quoteItem.sourceId} px={'5px'}
canView={canViewSource} borderWidth={'1px'}
/> borderRadius={'sm'}
<Box flex={1} /> mr={1}
{linkToDataset && ( {...(scoreTheme[i] && scoreTheme[i])}
<Link >
as={NextLink} <Box transform={'scale(0.9)'}>#{item.index + 1}</Box>
className="hover-data" </Box>
display={'none'} <Box transform={'scale(0.9)'}>
alignItems={'center'} {t(SearchScoreTypeMap[item.type]?.label)}: {item.value.toFixed(4)}
color={'primary.500'} </Box>
href={`/dataset/detail?datasetId=${quoteItem.datasetId}&currentTab=dataCard&collectionId=${quoteItem.collectionId}`} </Flex>
> <Box h={'4px'}>
{t('core.dataset.Go Dataset')} {SearchScoreTypeMap[item.type]?.showScore && (
<MyIcon name={'common/rightArrowLight'} w={'10px'} /> <Progress
</Link> value={item.value * 100}
)} h={'4px'}
w={'100%'}
size="sm"
borderRadius={'20px'}
colorScheme={scoreTheme[i]?.colorSchema}
bg="#E8EBF0"
/>
)}
</Box>
</Box>
</MyTooltip>
))}
</Flex> </Flex>
<Box color={'black'}>{quoteItem.q}</Box> <Box flex={'1 0 0'}>
<Box color={'myGray.600'}>{quoteItem.a}</Box> <Box color={'black'}>{quoteItem.q}</Box>
<Box color={'myGray.600'}>{quoteItem.a}</Box>
</Box>
{canViewSource && ( {canViewSource && (
<Flex alignItems={'center'} mt={3} gap={4} color={'myGray.500'} fontSize={'xs'}> <Flex alignItems={'center'} mt={3} gap={4} color={'myGray.500'} fontSize={'xs'}>
{isPc && (
<Flex border={theme.borders.base} px={3} borderRadius={'xs'} lineHeight={'16px'}>
ID: {quoteItem.id}
</Flex>
)}
<MyTooltip label={t('core.dataset.Quote Length')}> <MyTooltip label={t('core.dataset.Quote Length')}>
<Flex alignItems={'center'}> <Flex alignItems={'center'}>
<MyIcon name="common/text/t" w={'14px'} mr={1} color={'myGray.500'} /> <MyIcon name="common/text/t" w={'14px'} mr={1} color={'myGray.500'} />
{quoteItem.q.length + (quoteItem.a?.length || 0)} {quoteItem.q.length + (quoteItem.a?.length || 0)}
</Flex> </Flex>
</MyTooltip> </MyTooltip>
{canViewSource && score && ( <RawSourceText
<MyTooltip label={score.tip}> fontWeight={'bold'}
<Flex alignItems={'center'}> color={'black'}
<MyIcon name={'kbTest'} w={'12px'} /> sourceName={quoteItem.sourceName}
{score.value ? ( sourceId={quoteItem.sourceId}
<> canView={canViewSource}
<Progress />
mx={2}
w={['60px', '90px']}
value={score?.value * 100}
size="sm"
borderRadius={'20px'}
colorScheme="myGray"
border={theme.borders.base}
/>
<Box>{score?.value.toFixed(4)}</Box>
</>
) : (
<Box ml={1} cursor={'pointer'}>
{t('core.dataset.search.Read score')}
</Box>
)}
</Flex>
</MyTooltip>
)}
<Box flex={1} /> <Box flex={1} />
{quoteItem.id && ( {quoteItem.id && (
<MyTooltip label={t('core.dataset.data.Edit')}> <MyTooltip label={t('core.dataset.data.Edit')}>
<Box <Box
className="hover-data" className="hover-data"
display={['flex', 'none']} display={['flex', 'none']}
bg={'rgba(255,255,255,0.9)'}
alignItems={'center'} alignItems={'center'}
justifyContent={'center'} justifyContent={'center'}
boxShadow={'-10px 0 10px rgba(255,255,255,1)'} boxShadow={'-10px 0 10px rgba(255,255,255,1)'}
@@ -199,6 +229,19 @@ ${t('core.dataset.search.Score')}: ${item.value.toFixed(4)}`
</Box> </Box>
</MyTooltip> </MyTooltip>
)} )}
{linkToDataset && (
<Link
as={NextLink}
className="hover-data"
display={'none'}
alignItems={'center'}
color={'primary.500'}
href={`/dataset/detail?datasetId=${quoteItem.datasetId}&currentTab=dataCard&collectionId=${quoteItem.collectionId}`}
>
{t('core.dataset.Go Dataset')}
<MyIcon name={'common/rightArrowLight'} w={'10px'} />
</Link>
)}
</Flex> </Flex>
)} )}
</MyBox> </MyBox>

View File

@@ -94,7 +94,7 @@ const NodeCQNode = React.memo(function NodeCQNode({ data }: { data: FlowModuleIt
/> />
<SourceHandle <SourceHandle
handleKey={item.key} handleKey={item.key}
valueType={ModuleIOValueTypeEnum.string} valueType={ModuleIOValueTypeEnum.boolean}
/> />
</Box> </Box>
</Box> </Box>

View File

@@ -64,6 +64,10 @@ const RenderList: {
{ {
types: [FlowNodeInputTypeEnum.addInputParam], types: [FlowNodeInputTypeEnum.addInputParam],
Component: dynamic(() => import('./templates/AddInputParam')) Component: dynamic(() => import('./templates/AddInputParam'))
},
{
types: [FlowNodeInputTypeEnum.JSONEditor],
Component: dynamic(() => import('./templates/JsonEditor'))
} }
]; ];
const UserChatInput = dynamic(() => import('./templates/UserChatInput')); const UserChatInput = dynamic(() => import('./templates/UserChatInput'));

View File

@@ -0,0 +1,39 @@
import React, { useCallback } from 'react';
import type { RenderInputProps } from '../type';
import { onChangeNode } from '../../../../FlowProvider';
import { useTranslation } from 'next-i18next';
import JSONEditor from '@fastgpt/web/components/common/Textarea/JsonEditor';
const JsonEditor = ({ item, moduleId }: RenderInputProps) => {
const { t } = useTranslation();
const update = useCallback(
(value: string) => {
onChangeNode({
moduleId,
type: 'updateInput',
key: item.key,
value: {
...item,
value
}
});
},
[item, moduleId]
);
return (
<JSONEditor
title={t(item.label)}
bg={'myWhite.400'}
placeholder={t(item.placeholder || '')}
resize
defaultValue={item.value}
onChange={(e) => {
update(e);
}}
/>
);
};
export default React.memo(JsonEditor);

View File

@@ -50,7 +50,8 @@ const defaultFeConfigs: FastGPTFeConfigsType = {
concatMd: concatMd:
'* 项目开源地址: [FastGPT GitHub](https://github.com/labring/FastGPT)\n* 交流群: ![](https://doc.fastgpt.in/wechat-fastgpt.webp)', '* 项目开源地址: [FastGPT GitHub](https://github.com/labring/FastGPT)\n* 交流群: ![](https://doc.fastgpt.in/wechat-fastgpt.webp)',
limit: { limit: {
exportLimitMinutes: 0 exportDatasetLimitMinutes: 0,
websiteSyncLimitMinuted: 0
}, },
scripts: [], scripts: [],
favicon: '/favicon.ico' favicon: '/favicon.ico'

View File

@@ -1,73 +0,0 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { MongoUser } from '@fastgpt/service/support/user/schema';
import { addLog } from '@fastgpt/service/common/system/log';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { findDatasetIdTreeByTopDatasetId } from '@fastgpt/service/core/dataset/controller';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
let { datasetId } = req.query as {
datasetId: string;
};
if (!datasetId) {
throw new Error('缺少参数');
}
// 凭证校验
const { userId } = await authDataset({ req, authToken: true, datasetId, per: 'w' });
await limitCheck({
datasetId,
userId
});
jsonRes(res);
} catch (err) {
res.status(500);
jsonRes(res, {
code: 500,
error: err
});
}
}
export async function limitCheck({ datasetId, userId }: { datasetId: string; userId: string }) {
const exportIds = await findDatasetIdTreeByTopDatasetId(datasetId);
const limitMinutesAgo = new Date(
Date.now() - (global.feConfigs?.limit?.exportLimitMinutes || 0) * 60 * 1000
);
// auth export times
const authTimes = await MongoUser.findOne(
{
_id: userId,
$or: [
{ 'limit.exportKbTime': { $exists: false } },
{ 'limit.exportKbTime': { $lte: limitMinutesAgo } }
]
},
'_id limit'
);
if (!authTimes) {
const minutes = `${global.feConfigs?.limit?.exportLimitMinutes || 0} 分钟`;
return Promise.reject(`上次导出未到 ${minutes},每 ${minutes}仅可导出一次。`);
}
// auth max data
const total = await MongoDatasetData.countDocuments({
datasetId: { $in: exportIds }
});
addLog.info(`export datasets: ${datasetId}`, { total });
if (total > 100000) {
return Promise.reject('数据量超出 10 万,无法导出');
}
}

View File

@@ -2,14 +2,20 @@ import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response'; import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/dataset'; import { authDatasetCollection } from '@fastgpt/service/support/permission/auth/dataset';
import { loadingOneChunkCollection } from '@fastgpt/service/core/dataset/collection/utils'; import {
getCollectionAndRawText,
reloadCollectionChunks
} from '@fastgpt/service/core/dataset/collection/utils';
import { delCollectionRelevantData } from '@fastgpt/service/core/dataset/data/controller'; import { delCollectionRelevantData } from '@fastgpt/service/core/dataset/data/controller';
import { MongoDatasetCollection } from '@fastgpt/service/core/dataset/collection/schema'; import {
import { DatasetCollectionTypeEnum } from '@fastgpt/global/core/dataset/constant'; DatasetCollectionSyncResultEnum,
DatasetCollectionTypeEnum
} from '@fastgpt/global/core/dataset/constant';
import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset'; import { DatasetErrEnum } from '@fastgpt/global/common/error/code/dataset';
import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller'; import { createTrainingBill } from '@fastgpt/service/support/wallet/bill/controller';
import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants'; import { BillSourceEnum } from '@fastgpt/global/support/wallet/bill/constants';
import { getQAModel, getVectorModel } from '@/service/core/ai/model'; import { getQAModel, getVectorModel } from '@/service/core/ai/model';
import { createOneCollection } from '@fastgpt/service/core/dataset/collection/controller';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) { export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try { try {
@@ -32,6 +38,18 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
return Promise.reject(DatasetErrEnum.unLinkCollection); return Promise.reject(DatasetErrEnum.unLinkCollection);
} }
const { rawText, isSameRawText } = await getCollectionAndRawText({
collection
});
if (isSameRawText) {
return jsonRes(res, {
data: DatasetCollectionSyncResultEnum.sameRaw
});
}
/* Not the same original text, create and reload */
const vectorModelData = getVectorModel(collection.datasetId.vectorModel); const vectorModelData = getVectorModel(collection.datasetId.vectorModel);
const agentModelData = getQAModel(collection.datasetId.agentModel); const agentModelData = getQAModel(collection.datasetId.agentModel);
// create training bill // create training bill
@@ -45,26 +63,27 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
}); });
// create a collection and delete old // create a collection and delete old
const { _id } = await MongoDatasetCollection.create({ const _id = await createOneCollection({
parentId: collection.parentId,
teamId: collection.teamId, teamId: collection.teamId,
tmbId: collection.tmbId, tmbId: collection.tmbId,
parentId: collection.parentId,
datasetId: collection.datasetId._id, datasetId: collection.datasetId._id,
type: collection.type,
name: collection.name, name: collection.name,
createTime: collection.createTime, type: collection.type,
trainingType: collection.trainingType, trainingType: collection.trainingType,
chunkSize: collection.chunkSize, chunkSize: collection.chunkSize,
fileId: collection.fileId, fileId: collection.fileId,
rawLink: collection.rawLink, rawLink: collection.rawLink,
metadata: collection.metadata metadata: collection.metadata,
createTime: collection.createTime
}); });
// start load // start load
await loadingOneChunkCollection({ await reloadCollectionChunks({
collectionId: _id, collectionId: _id,
tmbId, tmbId,
billId billId,
rawText
}); });
// delete old collection // delete old collection
@@ -73,7 +92,9 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse<
fileIds: collection.fileId ? [collection.fileId] : [] fileIds: collection.fileId ? [collection.fileId] : []
}); });
jsonRes(res); jsonRes(res, {
data: DatasetCollectionSyncResultEnum.success
});
} catch (err) { } catch (err) {
jsonRes(res, { jsonRes(res, {
code: 500, code: 500,

View File

@@ -1,13 +1,15 @@
import type { NextApiRequest, NextApiResponse } from 'next'; import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes, responseWriteController } from '@fastgpt/service/common/response'; import { jsonRes, responseWriteController } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo'; import { connectToDatabase } from '@/service/mongo';
import { MongoUser } from '@fastgpt/service/support/user/schema';
import { addLog } from '@fastgpt/service/common/system/log'; import { addLog } from '@fastgpt/service/common/system/log';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset'; import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema'; import { MongoDatasetData } from '@fastgpt/service/core/dataset/data/schema';
import { findDatasetIdTreeByTopDatasetId } from '@fastgpt/service/core/dataset/controller'; import { findDatasetIdTreeByTopDatasetId } from '@fastgpt/service/core/dataset/controller';
import { limitCheck } from './checkExportLimit';
import { withNextCors } from '@fastgpt/service/common/middle/cors'; import { withNextCors } from '@fastgpt/service/common/middle/cors';
import {
checkExportDatasetLimit,
updateExportDatasetLimit
} from '@fastgpt/service/support/user/utils';
export default withNextCors(async function handler(req: NextApiRequest, res: NextApiResponse<any>) { export default withNextCors(async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try { try {
@@ -21,11 +23,11 @@ export default withNextCors(async function handler(req: NextApiRequest, res: Nex
} }
// 凭证校验 // 凭证校验
const { userId } = await authDataset({ req, authToken: true, datasetId, per: 'w' }); const { teamId } = await authDataset({ req, authToken: true, datasetId, per: 'w' });
await limitCheck({ await checkExportDatasetLimit({
userId, teamId,
datasetId limitMinutes: global.feConfigs?.limit?.exportDatasetLimitMinutes
}); });
const exportIds = await findDatasetIdTreeByTopDatasetId(datasetId); const exportIds = await findDatasetIdTreeByTopDatasetId(datasetId);
@@ -43,7 +45,9 @@ export default withNextCors(async function handler(req: NextApiRequest, res: Nex
datasetId: { $in: exportIds } datasetId: { $in: exportIds }
}, },
'q a' 'q a'
).cursor(); )
.limit(50000)
.cursor();
const write = responseWriteController({ const write = responseWriteController({
res, res,
@@ -59,12 +63,10 @@ export default withNextCors(async function handler(req: NextApiRequest, res: Nex
write(`\n"${q}","${a}"`); write(`\n"${q}","${a}"`);
}); });
cursor.on('end', async () => { cursor.on('end', () => {
cursor.close(); cursor.close();
res.end(); res.end();
await MongoUser.findByIdAndUpdate(userId, { updateExportDatasetLimit(teamId);
'limit.exportKbTime': new Date()
});
}); });
cursor.on('error', (err) => { cursor.on('error', (err) => {

View File

@@ -0,0 +1,34 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { authDataset } from '@fastgpt/service/support/permission/auth/dataset';
import { checkExportDatasetLimit } from '@fastgpt/service/support/user/utils';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
const { datasetId } = req.query as {
datasetId: string;
};
if (!datasetId) {
throw new Error('datasetId is required');
}
// 凭证校验
const { teamId } = await authDataset({ req, authToken: true, datasetId, per: 'w' });
await checkExportDatasetLimit({
teamId,
limitMinutes: global.feConfigs?.limit?.exportDatasetLimitMinutes
});
jsonRes(res);
} catch (err) {
res.status(500);
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -0,0 +1,27 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import { jsonRes } from '@fastgpt/service/common/response';
import { connectToDatabase } from '@/service/mongo';
import { checkWebSyncLimit } from '@fastgpt/service/support/user/utils';
import { authCert } from '@fastgpt/service/support/permission/auth/common';
export default async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
try {
await connectToDatabase();
// 凭证校验
const { teamId } = await authCert({ req, authToken: true });
await checkWebSyncLimit({
teamId,
limitMinutes: global.feConfigs?.limit?.websiteSyncLimitMinuted
});
jsonRes(res);
} catch (err) {
res.status(500);
jsonRes(res, {
code: 500,
error: err
});
}
}

View File

@@ -46,7 +46,8 @@ import {
DatasetCollectionTrainingModeEnum, DatasetCollectionTrainingModeEnum,
DatasetTypeEnum, DatasetTypeEnum,
DatasetTypeMap, DatasetTypeMap,
DatasetStatusEnum DatasetStatusEnum,
DatasetCollectionSyncResultMap
} from '@fastgpt/global/core/dataset/constant'; } from '@fastgpt/global/core/dataset/constant';
import { getCollectionIcon } from '@fastgpt/global/core/dataset/utils'; import { getCollectionIcon } from '@fastgpt/global/core/dataset/utils';
import EditFolderModal, { useEditFolder } from '../../component/EditFolderModal'; import EditFolderModal, { useEditFolder } from '../../component/EditFolderModal';
@@ -61,6 +62,7 @@ import { useUserStore } from '@/web/support/user/useUserStore';
import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant'; import { TeamMemberRoleEnum } from '@fastgpt/global/support/user/team/constant';
import { useDatasetStore } from '@/web/core/dataset/store/dataset'; import { useDatasetStore } from '@/web/core/dataset/store/dataset';
import { DatasetSchemaType } from '@fastgpt/global/core/dataset/type'; import { DatasetSchemaType } from '@fastgpt/global/core/dataset/type';
import { DatasetCollectionSyncResultEnum } from '../../../../../../../packages/global/core/dataset/constant';
const FileImportModal = dynamic(() => import('./Import/ImportModal'), {}); const FileImportModal = dynamic(() => import('./Import/ImportModal'), {});
const WebSiteConfigModal = dynamic(() => import('./Import/WebsiteConfig'), {}); const WebSiteConfigModal = dynamic(() => import('./Import/WebsiteConfig'), {});
@@ -246,8 +248,12 @@ const CollectionCard = () => {
mutationFn: (collectionId: string) => { mutationFn: (collectionId: string) => {
return postLinkCollectionSync(collectionId); return postLinkCollectionSync(collectionId);
}, },
onSuccess() { onSuccess(res: DatasetCollectionSyncResultEnum) {
getData(pageNum); getData(pageNum);
toast({
status: 'success',
title: t(DatasetCollectionSyncResultMap[res]?.label)
});
}, },
errorToast: t('core.dataset.error.Start Sync Failed') errorToast: t('core.dataset.error.Start Sync Failed')
}); });

View File

@@ -121,46 +121,55 @@ const DataCard = () => {
[collection?.canWrite, userInfo?.team?.role] [collection?.canWrite, userInfo?.team?.role]
); );
const metadataList = useMemo( const metadataList = useMemo(() => {
() => if (!collection) return [];
collection
const webSelector =
collection?.datasetId?.websiteConfig?.selector || collection?.metadata?.webPageSelector;
return [
{
label: t('core.dataset.collection.metadata.source'),
value: t(DatasetCollectionTypeMap[collection.type]?.name)
},
{
label: t('core.dataset.collection.metadata.source name'),
value: collection.file?.filename || collection?.rawLink || collection?.name
},
{
label: t('core.dataset.collection.metadata.source size'),
value: collection.file ? formatFileSize(collection.file.length) : '-'
},
{
label: t('core.dataset.collection.metadata.Createtime'),
value: formatTime2YMDHM(collection.createTime)
},
{
label: t('core.dataset.collection.metadata.Updatetime'),
value: formatTime2YMDHM(collection.updateTime)
},
{
label: t('core.dataset.collection.metadata.Raw text length'),
value: collection.rawTextLength ?? '-'
},
{
label: t('core.dataset.collection.metadata.Training Type'),
value: t(DatasetCollectionTrainingTypeMap[collection.trainingType]?.label)
},
{
label: t('core.dataset.collection.metadata.Chunk Size'),
value: collection.chunkSize || '-'
},
...(webSelector
? [ ? [
{ {
label: t('core.dataset.collection.metadata.source'), label: t('core.dataset.collection.metadata.Web page selector'),
value: t(DatasetCollectionTypeMap[collection.type]?.name) value: webSelector
},
{
label: t('core.dataset.collection.metadata.source name'),
value: collection.file?.filename || collection?.rawLink || collection?.name
},
{
label: t('core.dataset.collection.metadata.source size'),
value: collection.file ? formatFileSize(collection.file.length) : '-'
},
{
label: t('core.dataset.collection.metadata.Createtime'),
value: formatTime2YMDHM(collection.createTime)
},
{
label: t('core.dataset.collection.metadata.Updatetime'),
value: formatTime2YMDHM(collection.updateTime)
},
{
label: t('core.dataset.collection.metadata.Raw text length'),
value: collection.rawTextLength ?? '-'
},
{
label: t('core.dataset.collection.metadata.Training Type'),
value: t(DatasetCollectionTrainingTypeMap[collection.trainingType]?.label)
},
{
label: t('core.dataset.collection.metadata.Chunk Size'),
value: collection.chunkSize || '-'
} }
] ]
: [], : [])
[collection, t] ];
); }, [collection, t]);
return ( return (
<Box ref={BoxRef} position={'relative'} px={5} py={[1, 5]} h={'100%'} overflow={'overlay'}> <Box ref={BoxRef} position={'relative'} px={5} py={[1, 5]} h={'100%'} overflow={'overlay'}>

View File

@@ -41,6 +41,7 @@ export type FileItemType = {
type: DatasetCollectionTypeEnum.file | DatasetCollectionTypeEnum.link; type: DatasetCollectionTypeEnum.file | DatasetCollectionTypeEnum.link;
fileId?: string; fileId?: string;
rawLink?: string; rawLink?: string;
metadata?: Record<string, any>;
}; };
export interface Props extends BoxProps { export interface Props extends BoxProps {
@@ -232,7 +233,7 @@ const FileSelect = ({
// link fetch // link fetch
const onUrlFetch = useCallback( const onUrlFetch = useCallback(
(e: UrlFetchResponse) => { (e: UrlFetchResponse) => {
const result: FileItemType[] = e.map<FileItemType>(({ url, content }) => { const result: FileItemType[] = e.map<FileItemType>(({ url, content, selector }) => {
const { chunks, tokens } = splitText2Chunks({ const { chunks, tokens } = splitText2Chunks({
text: content, text: content,
chunkLen, chunkLen,
@@ -250,7 +251,10 @@ const FileSelect = ({
chunks: chunks.map((chunk) => ({ chunks: chunks.map((chunk) => ({
q: chunk, q: chunk,
a: '' a: ''
})) })),
metadata: {
webPageSelector: selector
}
}; };
}); });
onPushFiles(result); onPushFiles(result);

View File

@@ -156,19 +156,24 @@ const Provider = ({
return formatModelPrice2Read(totalTokens * inputPrice); return formatModelPrice2Read(totalTokens * inputPrice);
}, [inputPrice, mode, outputPrice, totalTokens]); }, [inputPrice, mode, outputPrice, totalTokens]);
/* start upload data */ /*
start upload data
1. create training bill
2. create collection
3. upload chunks
*/
const { mutate: onclickUpload, isLoading: uploading } = useRequest({ const { mutate: onclickUpload, isLoading: uploading } = useRequest({
mutationFn: async (props?: { prompt?: string }) => { mutationFn: async (props?: { prompt?: string }) => {
const { prompt } = props || {}; const { prompt } = props || {};
let totalInsertion = 0; let totalInsertion = 0;
for await (const file of files) { for await (const file of files) {
const chunks = file.chunks;
// create training bill // create training bill
const billId = await postCreateTrainingBill({ const billId = await postCreateTrainingBill({
name: t('dataset.collections.Create Training Data', { filename: file.filename }), name: t('dataset.collections.Create Training Data', { filename: file.filename }),
vectorModel, vectorModel,
agentModel agentModel
}); });
// create a file collection and training bill // create a file collection and training bill
const collectionId = await postDatasetCollection({ const collectionId = await postDatasetCollection({
datasetId, datasetId,
@@ -181,10 +186,12 @@ const Provider = ({
trainingType: collectionTrainingType, trainingType: collectionTrainingType,
qaPrompt: mode === TrainingModeEnum.qa ? prompt : '', qaPrompt: mode === TrainingModeEnum.qa ? prompt : '',
rawTextLength: file.rawText.length, rawTextLength: file.rawText.length,
hashRawText: hashStr(file.rawText) hashRawText: hashStr(file.rawText),
metadata: file.metadata
}); });
// upload data // upload chunks
const chunks = file.chunks;
const { insertLen } = await chunksUpload({ const { insertLen } = await chunksUpload({
collectionId, collectionId,
billId, billId,

View File

@@ -60,6 +60,7 @@ const Test = ({ datasetId }: { datasetId: string }) => {
const [inputType, setInputType] = useState<'text' | 'file'>('text'); const [inputType, setInputType] = useState<'text' | 'file'>('text');
const [datasetTestItem, setDatasetTestItem] = useState<SearchTestStoreItemType>(); const [datasetTestItem, setDatasetTestItem] = useState<SearchTestStoreItemType>();
const [refresh, setRefresh] = useState(false); const [refresh, setRefresh] = useState(false);
const [isFocus, setIsFocus] = useState(false);
const { File, onOpen } = useSelectFile({ const { File, onOpen } = useSelectFile({
fileType: '.csv', fileType: '.csv',
multiple: false multiple: false
@@ -169,7 +170,20 @@ const Test = ({ datasetId }: { datasetId: string }) => {
py={4} py={4}
borderRight={['none', theme.borders.base]} borderRight={['none', theme.borders.base]}
> >
<Box border={'2px solid'} borderColor={'primary.500'} p={3} mx={4} borderRadius={'md'}> <Box
border={'2px solid'}
p={3}
mx={4}
borderRadius={'md'}
{...(isFocus
? {
borderColor: 'primary.500',
boxShadow: '0px 0px 0px 2.4px rgba(51, 112, 255, 0.15)'
}
: {
borderColor: 'primary.300'
})}
>
{/* header */} {/* header */}
<Flex alignItems={'center'} justifyContent={'space-between'}> <Flex alignItems={'center'} justifyContent={'space-between'}>
<MySelect <MySelect
@@ -221,8 +235,12 @@ const Test = ({ datasetId }: { datasetId: string }) => {
variant={'unstyled'} variant={'unstyled'}
maxLength={datasetDetail.vectorModel.maxToken} maxLength={datasetDetail.vectorModel.maxToken}
placeholder={t('core.dataset.test.Test Text Placeholder')} placeholder={t('core.dataset.test.Test Text Placeholder')}
onFocus={() => setIsFocus(true)}
{...register('inputText', { {...register('inputText', {
required: true required: true,
onBlur: () => {
setIsFocus(false);
}
})} })}
/> />
)} )}
@@ -340,25 +358,26 @@ const TestHistories = React.memo(function TestHistories({
); );
return ( return (
<> <>
<Flex alignItems={'center'} color={'myGray.600'}> <Flex alignItems={'center'} color={'myGray.900'}>
<MyIcon mr={2} name={'history'} w={'16px'} h={'16px'} /> <MyIcon mr={2} name={'history'} w={'18px'} h={'18px'} color={'myGray.900'} />
<Box fontSize={'2xl'}>{t('core.dataset.test.test history')}</Box> <Box fontSize={'xl'}>{t('core.dataset.test.test history')}</Box>
</Flex> </Flex>
<Box mt={2}> <Box mt={2}>
<Flex py={2} fontWeight={'bold'} borderBottom={theme.borders.sm}>
<Box flex={'0 0 80px'}>{t('core.dataset.search.search mode')}</Box>
<Box flex={1}>{t('core.dataset.test.Test Text')}</Box>
<Box flex={'0 0 70px'}>{t('common.Time')}</Box>
<Box w={'14px'}></Box>
</Flex>
{testHistories.map((item) => ( {testHistories.map((item) => (
<Flex <Flex
key={item.id} key={item.id}
p={1} py={2}
px={3}
alignItems={'center'} alignItems={'center'}
borderBottom={theme.borders.base} borderColor={'borderColor.low'}
borderWidth={'1px'}
borderRadius={'md'}
_notLast={{
mb: 2
}}
_hover={{ _hover={{
bg: '#f4f4f4', borderColor: 'primary.300',
boxShadow: '1',
'& .delete': { '& .delete': {
display: 'block' display: 'block'
} }
@@ -369,7 +388,7 @@ const TestHistories = React.memo(function TestHistories({
> >
<Box flex={'0 0 80px'}> <Box flex={'0 0 80px'}>
{DatasetSearchModeMap[item.searchMode] ? ( {DatasetSearchModeMap[item.searchMode] ? (
<Flex alignItems={'center'}> <Flex alignItems={'center'} fontWeight={'500'} color={'myGray.500'}>
<MyIcon <MyIcon
name={DatasetSearchModeMap[item.searchMode].icon as any} name={DatasetSearchModeMap[item.searchMode].icon as any}
w={'12px'} w={'12px'}
@@ -381,7 +400,7 @@ const TestHistories = React.memo(function TestHistories({
'-' '-'
)} )}
</Box> </Box>
<Box flex={1} mr={2} wordBreak={'break-all'}> <Box flex={1} mr={2} wordBreak={'break-all'} fontWeight={'400'}>
{item.text} {item.text}
</Box> </Box>
<Box flex={'0 0 70px'}>{formatTimeToChatTime(item.time)}</Box> <Box flex={'0 0 70px'}>{formatTimeToChatTime(item.time)}</Box>
@@ -433,13 +452,20 @@ const TestResults = React.memo(function TestResults({
</Flex> </Flex>
) : ( ) : (
<> <>
<Box fontSize={'xl'} color={'myGray.600'}> <Flex fontSize={'xl'} color={'myGray.900'} alignItems={'center'}>
<MyIcon name={'common/paramsLight'} w={'18px'} mr={2} />
{t('core.dataset.test.Test params')} {t('core.dataset.test.Test params')}
</Box> </Flex>
<TableContainer mb={3} bg={'myGray.150'} borderRadius={'md'}> <TableContainer
mt={3}
bg={'primary.50'}
borderRadius={'lg'}
borderWidth={'1px'}
borderColor={'primary.1'}
>
<Table> <Table>
<Thead> <Thead>
<Tr> <Tr color={'myGray.600'}>
<Th>{t('core.dataset.search.search mode')}</Th> <Th>{t('core.dataset.search.search mode')}</Th>
<Th>{t('core.dataset.search.ReRank')}</Th> <Th>{t('core.dataset.search.ReRank')}</Th>
<Th>{t('core.dataset.search.Max Tokens')}</Th> <Th>{t('core.dataset.search.Max Tokens')}</Th>
@@ -447,8 +473,8 @@ const TestResults = React.memo(function TestResults({
</Tr> </Tr>
</Thead> </Thead>
<Tbody> <Tbody>
<Tr> <Tr color={'myGray.800'}>
<Td> <Td pt={0}>
<Flex alignItems={'center'}> <Flex alignItems={'center'}>
<MyIcon <MyIcon
name={DatasetSearchModeMap[datasetTestItem.searchMode]?.icon as any} name={DatasetSearchModeMap[datasetTestItem.searchMode]?.icon as any}
@@ -458,45 +484,31 @@ const TestResults = React.memo(function TestResults({
{t(DatasetSearchModeMap[datasetTestItem.searchMode]?.title)} {t(DatasetSearchModeMap[datasetTestItem.searchMode]?.title)}
</Flex> </Flex>
</Td> </Td>
<Td>{datasetTestItem.usingReRank ? '✅' : '❌'}</Td> <Td pt={0}>{datasetTestItem.usingReRank ? '✅' : '❌'}</Td>
<Td>{datasetTestItem.limit}</Td> <Td pt={0}>{datasetTestItem.limit}</Td>
<Td>{datasetTestItem.similarity}</Td> <Td pt={0}>{datasetTestItem.similarity}</Td>
</Tr> </Tr>
</Tbody> </Tbody>
</Table> </Table>
</TableContainer> </TableContainer>
<Flex alignItems={'center'}>
<Box fontSize={'xl'} color={'myGray.600'}> <Flex mt={5} mb={3} alignItems={'center'}>
<Flex fontSize={'xl'} color={'myGray.900'} alignItems={'center'}>
<MyIcon name={'common/resultLight'} w={'18px'} mr={2} />
{t('core.dataset.test.Test Result')} {t('core.dataset.test.Test Result')}
</Box> </Flex>
<MyTooltip label={t('core.dataset.test.test result tip')} forceShow> <MyTooltip label={t('core.dataset.test.test result tip')} forceShow>
<QuestionOutlineIcon mx={2} color={'myGray.600'} cursor={'pointer'} fontSize={'lg'} /> <QuestionOutlineIcon mx={2} color={'myGray.600'} cursor={'pointer'} fontSize={'lg'} />
</MyTooltip> </MyTooltip>
<Box>({datasetTestItem.duration})</Box> <Box>({datasetTestItem.duration})</Box>
</Flex> </Flex>
<Grid <Box mt={1} gap={4}>
mt={1}
gridTemplateColumns={[
'repeat(1,minmax(0, 1fr))',
'repeat(1,minmax(0, 1fr))',
'repeat(1,minmax(0, 1fr))',
'repeat(1,minmax(0, 1fr))',
'repeat(2,minmax(0, 1fr))'
]}
gridGap={4}
>
{datasetTestItem?.results.map((item, index) => ( {datasetTestItem?.results.map((item, index) => (
<Box <Box key={item.id} p={3} borderRadius={'lg'} bg={'myGray.100'} _notLast={{ mb: 2 }}>
key={item.id}
p={2}
borderRadius={'sm'}
border={theme.borders.base}
_notLast={{ mb: 2 }}
>
<QuoteItem quoteItem={item} canViewSource /> <QuoteItem quoteItem={item} canViewSource />
</Box> </Box>
))} ))}
</Grid> </Box>
</> </>
)} )}
</> </>

View File

@@ -20,9 +20,9 @@ import {
delDatasetById, delDatasetById,
getDatasetPaths, getDatasetPaths,
putDatasetById, putDatasetById,
postCreateDataset, postCreateDataset
getCheckExportLimit
} from '@/web/core/dataset/api'; } from '@/web/core/dataset/api';
import { checkTeamExportDatasetLimit } from '@/web/support/user/api';
import { useTranslation } from 'next-i18next'; import { useTranslation } from 'next-i18next';
import Avatar from '@/components/Avatar'; import Avatar from '@/components/Avatar';
import MyIcon from '@fastgpt/web/components/common/Icon'; import MyIcon from '@fastgpt/web/components/common/Icon';
@@ -99,7 +99,7 @@ const Kb = () => {
const { mutate: exportDataset } = useRequest({ const { mutate: exportDataset } = useRequest({
mutationFn: async (dataset: DatasetItemType) => { mutationFn: async (dataset: DatasetItemType) => {
setLoading(true); setLoading(true);
await getCheckExportLimit(dataset._id); await checkTeamExportDatasetLimit(dataset._id);
const a = document.createElement('a'); const a = document.createElement('a');
a.href = `/api/core/dataset/exportAll?datasetId=${dataset._id}`; a.href = `/api/core/dataset/exportAll?datasetId=${dataset._id}`;
a.download = `${dataset.name}.csv`; a.download = `${dataset.name}.csv`;

View File

@@ -68,7 +68,7 @@ export const dispatchClassifyQuestion = async (props: Props): Promise<CQResponse
}); });
return { return {
[result.key]: result.value, [result.key]: true,
[ModuleOutputKeyEnum.responseData]: { [ModuleOutputKeyEnum.responseData]: {
price: user.openaiAccount?.key ? 0 : total, price: user.openaiAccount?.key ? 0 : total,
model: modelName, model: modelName,

View File

@@ -272,8 +272,8 @@ function filterQuote({
}); });
const quoteText = const quoteText =
filterQuoteQA.length > 0 sortQuoteQAList.length > 0
? `${filterQuoteQA.map((item, index) => getValue(item, index)).join('\n')}` ? `${sortQuoteQAList.map((item, index) => getValue(item, index)).join('\n')}`
: ''; : '';
return { return {

View File

@@ -25,7 +25,10 @@ import type {
} from '@/global/core/dataset/api.d'; } from '@/global/core/dataset/api.d';
import type { PushDataResponse } from '@/global/core/api/datasetRes.d'; import type { PushDataResponse } from '@/global/core/api/datasetRes.d';
import type { DatasetCollectionItemType } from '@fastgpt/global/core/dataset/type'; import type { DatasetCollectionItemType } from '@fastgpt/global/core/dataset/type';
import { DatasetTypeEnum } from '@fastgpt/global/core/dataset/constant'; import {
DatasetCollectionSyncResultEnum,
DatasetTypeEnum
} from '@fastgpt/global/core/dataset/constant';
import type { DatasetDataItemType } from '@fastgpt/global/core/dataset/type'; import type { DatasetDataItemType } from '@fastgpt/global/core/dataset/type';
import type { DatasetCollectionsListItemType } from '@/global/core/dataset/type.d'; import type { DatasetCollectionsListItemType } from '@/global/core/dataset/type.d';
import { PagingData } from '@/types'; import { PagingData } from '@/types';
@@ -56,9 +59,6 @@ export const postWebsiteSync = (data: PostWebsiteSyncParams) =>
timeout: 600000 timeout: 600000
}).catch(); }).catch();
export const getCheckExportLimit = (datasetId: string) =>
GET(`/core/dataset/checkExportLimit`, { datasetId });
/* =========== search test ============ */ /* =========== search test ============ */
export const postSearchText = (data: SearchTestProps) => export const postSearchText = (data: SearchTestProps) =>
POST<SearchTestResponse>(`/core/dataset/searchTest`, data); POST<SearchTestResponse>(`/core/dataset/searchTest`, data);
@@ -77,7 +77,9 @@ export const putDatasetCollectionById = (data: UpdateDatasetCollectionParams) =>
export const delDatasetCollectionById = (params: { collectionId: string }) => export const delDatasetCollectionById = (params: { collectionId: string }) =>
DELETE(`/core/dataset/collection/delete`, params); DELETE(`/core/dataset/collection/delete`, params);
export const postLinkCollectionSync = (collectionId: string) => export const postLinkCollectionSync = (collectionId: string) =>
POST(`/core/dataset/collection/sync/link`, { collectionId }); POST<`${DatasetCollectionSyncResultEnum}`>(`/core/dataset/collection/sync/link`, {
collectionId
});
/* =============================== data ==================================== */ /* =============================== data ==================================== */
/* get dataset list */ /* get dataset list */

View File

@@ -13,6 +13,7 @@ import { defaultDatasetDetail } from '@/constants/dataset';
import type { DatasetUpdateBody } from '@fastgpt/global/core/dataset/api.d'; import type { DatasetUpdateBody } from '@fastgpt/global/core/dataset/api.d';
import { DatasetStatusEnum } from '@fastgpt/global/core/dataset/constant'; import { DatasetStatusEnum } from '@fastgpt/global/core/dataset/constant';
import { postCreateTrainingBill } from '@/web/support/wallet/bill/api'; import { postCreateTrainingBill } from '@/web/support/wallet/bill/api';
import { checkTeamWebSyncLimit } from '@/web/support/user/api';
type State = { type State = {
allDatasets: DatasetListItemType[]; allDatasets: DatasetListItemType[];
@@ -86,6 +87,8 @@ export const useDatasetStore = create<State>()(
}); });
}, },
async startWebsiteSync() { async startWebsiteSync() {
await checkTeamWebSyncLimit();
const [_, billId] = await Promise.all([ const [_, billId] = await Promise.all([
get().updateDataset({ get().updateDataset({
id: get().datasetDetail._id, id: get().datasetDetail._id,

View File

@@ -71,3 +71,8 @@ export const postLogin = ({ password, ...props }: PostLoginProps) =>
export const loginOut = () => GET('/support/user/account/loginout'); export const loginOut = () => GET('/support/user/account/loginout');
export const putUserInfo = (data: UserUpdateParams) => PUT('/support/user/account/update', data); export const putUserInfo = (data: UserUpdateParams) => PUT('/support/user/account/update', data);
/* team limit */
export const checkTeamExportDatasetLimit = (datasetId: string) =>
GET(`/support/user/team/limit/exportDatasetLimit`, { datasetId });
export const checkTeamWebSyncLimit = () => GET(`/support/user/team/limit/webSyncLimit`);

8
scripts/icon/index.js Normal file → Executable file
View File

@@ -33,7 +33,7 @@ app.get('/', (req, res) => {
svgPaths.forEach((filePath) => { svgPaths.forEach((filePath) => {
const name = filePath.split('.')[0]; const name = filePath.split('.')[0];
iconHtml += `<div class="item" id="${name}" onclick="onclickCopy(${name})"> iconHtml += `<div class="item" id="${name}" onclick="onclickCopy('${name}')">
<img src="/icons/${filePath}" width="30" height="30" /> <img src="/icons/${filePath}" width="30" height="30" />
<div>${name}</div> <div>${name}</div>
</div>`; </div>`;
@@ -76,16 +76,16 @@ app.get('/', (req, res) => {
</body> </body>
<script> <script>
const onclickCopy = (name) => { const onclickCopy = (name) => {
console.log(name.id) console.log(name)
try { try {
if (navigator.clipboard) { if (navigator.clipboard) {
navigator.clipboard.writeText(name.id); navigator.clipboard.writeText(name);
} else { } else {
throw new Error(''); throw new Error('');
} }
} catch (error) { } catch (error) {
const textarea = document.createElement('textarea'); const textarea = document.createElement('textarea');
textarea.value = name.id; textarea.value = name;
document.body.appendChild(textarea); document.body.appendChild(textarea);
textarea.select(); textarea.select();
document.execCommand('copy'); document.execCommand('copy');

0
scripts/icon/init.js Normal file → Executable file
View File

View File

@@ -1,613 +0,0 @@
{
"name": "icon",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "icon",
"version": "1.0.0",
"license": "ISC",
"dependencies": {
"express": "^4.18.2"
}
},
"node_modules/accepts": {
"version": "1.3.8",
"resolved": "https://registry.npmmirror.com/accepts/-/accepts-1.3.8.tgz",
"integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",
"dependencies": {
"mime-types": "~2.1.34",
"negotiator": "0.6.3"
},
"engines": {
"node": ">= 0.6"
}
},
"node_modules/array-flatten": {
"version": "1.1.1",
"resolved": "https://registry.npmmirror.com/array-flatten/-/array-flatten-1.1.1.tgz",
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg=="
},
"node_modules/body-parser": {
"version": "1.20.1",
"resolved": "https://registry.npmmirror.com/body-parser/-/body-parser-1.20.1.tgz",
"integrity": "sha512-jWi7abTbYwajOytWCQc37VulmWiRae5RyTpaCyDcS5/lMdtwSz5lOpDE67srw/HYe35f1z3fDQw+3txg7gNtWw==",
"dependencies": {
"bytes": "3.1.2",
"content-type": "~1.0.4",
"debug": "2.6.9",
"depd": "2.0.0",
"destroy": "1.2.0",
"http-errors": "2.0.0",
"iconv-lite": "0.4.24",
"on-finished": "2.4.1",
"qs": "6.11.0",
"raw-body": "2.5.1",
"type-is": "~1.6.18",
"unpipe": "1.0.0"
},
"engines": {
"node": ">= 0.8",
"npm": "1.2.8000 || >= 1.4.16"
}
},
"node_modules/bytes": {
"version": "3.1.2",
"resolved": "https://registry.npmmirror.com/bytes/-/bytes-3.1.2.tgz",
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/call-bind": {
"version": "1.0.5",
"resolved": "https://registry.npmmirror.com/call-bind/-/call-bind-1.0.5.tgz",
"integrity": "sha512-C3nQxfFZxFRVoJoGKKI8y3MOEo129NQ+FgQ08iye+Mk4zNZZGdjfs06bVTr+DBSlA66Q2VEcMki/cUCP4SercQ==",
"dependencies": {
"function-bind": "^1.1.2",
"get-intrinsic": "^1.2.1",
"set-function-length": "^1.1.1"
}
},
"node_modules/content-disposition": {
"version": "0.5.4",
"resolved": "https://registry.npmmirror.com/content-disposition/-/content-disposition-0.5.4.tgz",
"integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",
"dependencies": {
"safe-buffer": "5.2.1"
},
"engines": {
"node": ">= 0.6"
}
},
"node_modules/content-type": {
"version": "1.0.5",
"resolved": "https://registry.npmmirror.com/content-type/-/content-type-1.0.5.tgz",
"integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/cookie": {
"version": "0.5.0",
"resolved": "https://registry.npmmirror.com/cookie/-/cookie-0.5.0.tgz",
"integrity": "sha512-YZ3GUyn/o8gfKJlnlX7g7xq4gyO6OSuhGPKaaGssGB2qgDUS0gPgtTvoyZLTt9Ab6dC4hfc9dV5arkvc/OCmrw==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/cookie-signature": {
"version": "1.0.6",
"resolved": "https://registry.npmmirror.com/cookie-signature/-/cookie-signature-1.0.6.tgz",
"integrity": "sha512-QADzlaHc8icV8I7vbaJXJwod9HWYp8uCqf1xa4OfNu1T7JVxQIrUgOWtHdNDtPiywmFbiS12VjotIXLrKM3orQ=="
},
"node_modules/debug": {
"version": "2.6.9",
"resolved": "https://registry.npmmirror.com/debug/-/debug-2.6.9.tgz",
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
"dependencies": {
"ms": "2.0.0"
}
},
"node_modules/define-data-property": {
"version": "1.1.1",
"resolved": "https://registry.npmmirror.com/define-data-property/-/define-data-property-1.1.1.tgz",
"integrity": "sha512-E7uGkTzkk1d0ByLeSc6ZsFS79Axg+m1P/VsgYsxHgiuc3tFSj+MjMIwe90FC4lOAZzNBdY7kkO2P2wKdsQ1vgQ==",
"dependencies": {
"get-intrinsic": "^1.2.1",
"gopd": "^1.0.1",
"has-property-descriptors": "^1.0.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/depd": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/depd/-/depd-2.0.0.tgz",
"integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/destroy": {
"version": "1.2.0",
"resolved": "https://registry.npmmirror.com/destroy/-/destroy-1.2.0.tgz",
"integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==",
"engines": {
"node": ">= 0.8",
"npm": "1.2.8000 || >= 1.4.16"
}
},
"node_modules/ee-first": {
"version": "1.1.1",
"resolved": "https://registry.npmmirror.com/ee-first/-/ee-first-1.1.1.tgz",
"integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
},
"node_modules/encodeurl": {
"version": "1.0.2",
"resolved": "https://registry.npmmirror.com/encodeurl/-/encodeurl-1.0.2.tgz",
"integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/escape-html": {
"version": "1.0.3",
"resolved": "https://registry.npmmirror.com/escape-html/-/escape-html-1.0.3.tgz",
"integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="
},
"node_modules/etag": {
"version": "1.8.1",
"resolved": "https://registry.npmmirror.com/etag/-/etag-1.8.1.tgz",
"integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/express": {
"version": "4.18.2",
"resolved": "https://registry.npmmirror.com/express/-/express-4.18.2.tgz",
"integrity": "sha512-5/PsL6iGPdfQ/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ==",
"dependencies": {
"accepts": "~1.3.8",
"array-flatten": "1.1.1",
"body-parser": "1.20.1",
"content-disposition": "0.5.4",
"content-type": "~1.0.4",
"cookie": "0.5.0",
"cookie-signature": "1.0.6",
"debug": "2.6.9",
"depd": "2.0.0",
"encodeurl": "~1.0.2",
"escape-html": "~1.0.3",
"etag": "~1.8.1",
"finalhandler": "1.2.0",
"fresh": "0.5.2",
"http-errors": "2.0.0",
"merge-descriptors": "1.0.1",
"methods": "~1.1.2",
"on-finished": "2.4.1",
"parseurl": "~1.3.3",
"path-to-regexp": "0.1.7",
"proxy-addr": "~2.0.7",
"qs": "6.11.0",
"range-parser": "~1.2.1",
"safe-buffer": "5.2.1",
"send": "0.18.0",
"serve-static": "1.15.0",
"setprototypeof": "1.2.0",
"statuses": "2.0.1",
"type-is": "~1.6.18",
"utils-merge": "1.0.1",
"vary": "~1.1.2"
},
"engines": {
"node": ">= 0.10.0"
}
},
"node_modules/finalhandler": {
"version": "1.2.0",
"resolved": "https://registry.npmmirror.com/finalhandler/-/finalhandler-1.2.0.tgz",
"integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
"dependencies": {
"debug": "2.6.9",
"encodeurl": "~1.0.2",
"escape-html": "~1.0.3",
"on-finished": "2.4.1",
"parseurl": "~1.3.3",
"statuses": "2.0.1",
"unpipe": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/forwarded": {
"version": "0.2.0",
"resolved": "https://registry.npmmirror.com/forwarded/-/forwarded-0.2.0.tgz",
"integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/fresh": {
"version": "0.5.2",
"resolved": "https://registry.npmmirror.com/fresh/-/fresh-0.5.2.tgz",
"integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/function-bind": {
"version": "1.1.2",
"resolved": "https://registry.npmmirror.com/function-bind/-/function-bind-1.1.2.tgz",
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="
},
"node_modules/get-intrinsic": {
"version": "1.2.2",
"resolved": "https://registry.npmmirror.com/get-intrinsic/-/get-intrinsic-1.2.2.tgz",
"integrity": "sha512-0gSo4ml/0j98Y3lngkFEot/zhiCeWsbYIlZ+uZOVgzLyLaUw7wxUL+nCTP0XJvJg1AXulJRI3UJi8GsbDuxdGA==",
"dependencies": {
"function-bind": "^1.1.2",
"has-proto": "^1.0.1",
"has-symbols": "^1.0.3",
"hasown": "^2.0.0"
}
},
"node_modules/gopd": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/gopd/-/gopd-1.0.1.tgz",
"integrity": "sha512-d65bNlIadxvpb/A2abVdlqKqV563juRnZ1Wtk6s1sIR8uNsXR70xqIzVqxVf1eTqDunwT2MkczEeaezCKTZhwA==",
"dependencies": {
"get-intrinsic": "^1.1.3"
}
},
"node_modules/has-property-descriptors": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/has-property-descriptors/-/has-property-descriptors-1.0.1.tgz",
"integrity": "sha512-VsX8eaIewvas0xnvinAe9bw4WfIeODpGYikiWYLH+dma0Jw6KHYqWiWfhQlgOVK8D6PvjubK5Uc4P0iIhIcNVg==",
"dependencies": {
"get-intrinsic": "^1.2.2"
}
},
"node_modules/has-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/has-proto/-/has-proto-1.0.1.tgz",
"integrity": "sha512-7qE+iP+O+bgF9clE5+UoBFzE65mlBiVj3tKCrlNQ0Ogwm0BjpT/gK4SlLYDMybDh5I3TCTKnPPa0oMG7JDYrhg==",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/has-symbols": {
"version": "1.0.3",
"resolved": "https://registry.npmmirror.com/has-symbols/-/has-symbols-1.0.3.tgz",
"integrity": "sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A==",
"engines": {
"node": ">= 0.4"
}
},
"node_modules/hasown": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/hasown/-/hasown-2.0.0.tgz",
"integrity": "sha512-vUptKVTpIJhcczKBbgnS+RtcuYMB8+oNzPK2/Hp3hanz8JmpATdmmgLgSaadVREkDm+e2giHwY3ZRkyjSIDDFA==",
"dependencies": {
"function-bind": "^1.1.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/http-errors": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/http-errors/-/http-errors-2.0.0.tgz",
"integrity": "sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ==",
"dependencies": {
"depd": "2.0.0",
"inherits": "2.0.4",
"setprototypeof": "1.2.0",
"statuses": "2.0.1",
"toidentifier": "1.0.1"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/iconv-lite": {
"version": "0.4.24",
"resolved": "https://registry.npmmirror.com/iconv-lite/-/iconv-lite-0.4.24.tgz",
"integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
"dependencies": {
"safer-buffer": ">= 2.1.2 < 3"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmmirror.com/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
},
"node_modules/ipaddr.js": {
"version": "1.9.1",
"resolved": "https://registry.npmmirror.com/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
"integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
"engines": {
"node": ">= 0.10"
}
},
"node_modules/media-typer": {
"version": "0.3.0",
"resolved": "https://registry.npmmirror.com/media-typer/-/media-typer-0.3.0.tgz",
"integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/merge-descriptors": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
"integrity": "sha512-cCi6g3/Zr1iqQi6ySbseM1Xvooa98N0w31jzUYrXPX2xqObmFGHJ0tQ5u74H3mVh7wLouTseZyYIq39g8cNp1w=="
},
"node_modules/methods": {
"version": "1.1.2",
"resolved": "https://registry.npmmirror.com/methods/-/methods-1.1.2.tgz",
"integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/mime": {
"version": "1.6.0",
"resolved": "https://registry.npmmirror.com/mime/-/mime-1.6.0.tgz",
"integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
"bin": {
"mime": "cli.js"
},
"engines": {
"node": ">=4"
}
},
"node_modules/mime-db": {
"version": "1.52.0",
"resolved": "https://registry.npmmirror.com/mime-db/-/mime-db-1.52.0.tgz",
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/mime-types": {
"version": "2.1.35",
"resolved": "https://registry.npmmirror.com/mime-types/-/mime-types-2.1.35.tgz",
"integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
"dependencies": {
"mime-db": "1.52.0"
},
"engines": {
"node": ">= 0.6"
}
},
"node_modules/ms": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
},
"node_modules/negotiator": {
"version": "0.6.3",
"resolved": "https://registry.npmmirror.com/negotiator/-/negotiator-0.6.3.tgz",
"integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/object-inspect": {
"version": "1.13.1",
"resolved": "https://registry.npmmirror.com/object-inspect/-/object-inspect-1.13.1.tgz",
"integrity": "sha512-5qoj1RUiKOMsCCNLV1CBiPYE10sziTsnmNxkAI/rZhiD63CF7IqdFGC/XzjWjpSgLf0LxXX3bDFIh0E18f6UhQ=="
},
"node_modules/on-finished": {
"version": "2.4.1",
"resolved": "https://registry.npmmirror.com/on-finished/-/on-finished-2.4.1.tgz",
"integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
"dependencies": {
"ee-first": "1.1.1"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/parseurl": {
"version": "1.3.3",
"resolved": "https://registry.npmmirror.com/parseurl/-/parseurl-1.3.3.tgz",
"integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/path-to-regexp": {
"version": "0.1.7",
"resolved": "https://registry.npmmirror.com/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
"integrity": "sha512-5DFkuoqlv1uYQKxy8omFBeJPQcdoE07Kv2sferDCrAq1ohOU+MSDswDIbnx3YAM60qIOnYa53wBhXW0EbMonrQ=="
},
"node_modules/proxy-addr": {
"version": "2.0.7",
"resolved": "https://registry.npmmirror.com/proxy-addr/-/proxy-addr-2.0.7.tgz",
"integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
"dependencies": {
"forwarded": "0.2.0",
"ipaddr.js": "1.9.1"
},
"engines": {
"node": ">= 0.10"
}
},
"node_modules/qs": {
"version": "6.11.0",
"resolved": "https://registry.npmmirror.com/qs/-/qs-6.11.0.tgz",
"integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
"dependencies": {
"side-channel": "^1.0.4"
},
"engines": {
"node": ">=0.6"
}
},
"node_modules/range-parser": {
"version": "1.2.1",
"resolved": "https://registry.npmmirror.com/range-parser/-/range-parser-1.2.1.tgz",
"integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
"engines": {
"node": ">= 0.6"
}
},
"node_modules/raw-body": {
"version": "2.5.1",
"resolved": "https://registry.npmmirror.com/raw-body/-/raw-body-2.5.1.tgz",
"integrity": "sha512-qqJBtEyVgS0ZmPGdCFPWJ3FreoqvG4MVQln/kCgF7Olq95IbOp0/BWyMwbdtn4VTvkM8Y7khCQ2Xgk/tcrCXig==",
"dependencies": {
"bytes": "3.1.2",
"http-errors": "2.0.0",
"iconv-lite": "0.4.24",
"unpipe": "1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/safe-buffer": {
"version": "5.2.1",
"resolved": "https://registry.npmmirror.com/safe-buffer/-/safe-buffer-5.2.1.tgz",
"integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="
},
"node_modules/safer-buffer": {
"version": "2.1.2",
"resolved": "https://registry.npmmirror.com/safer-buffer/-/safer-buffer-2.1.2.tgz",
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
},
"node_modules/send": {
"version": "0.18.0",
"resolved": "https://registry.npmmirror.com/send/-/send-0.18.0.tgz",
"integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
"dependencies": {
"debug": "2.6.9",
"depd": "2.0.0",
"destroy": "1.2.0",
"encodeurl": "~1.0.2",
"escape-html": "~1.0.3",
"etag": "~1.8.1",
"fresh": "0.5.2",
"http-errors": "2.0.0",
"mime": "1.6.0",
"ms": "2.1.3",
"on-finished": "2.4.1",
"range-parser": "~1.2.1",
"statuses": "2.0.1"
},
"engines": {
"node": ">= 0.8.0"
}
},
"node_modules/send/node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmmirror.com/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
},
"node_modules/serve-static": {
"version": "1.15.0",
"resolved": "https://registry.npmmirror.com/serve-static/-/serve-static-1.15.0.tgz",
"integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
"dependencies": {
"encodeurl": "~1.0.2",
"escape-html": "~1.0.3",
"parseurl": "~1.3.3",
"send": "0.18.0"
},
"engines": {
"node": ">= 0.8.0"
}
},
"node_modules/set-function-length": {
"version": "1.1.1",
"resolved": "https://registry.npmmirror.com/set-function-length/-/set-function-length-1.1.1.tgz",
"integrity": "sha512-VoaqjbBJKiWtg4yRcKBQ7g7wnGnLV3M8oLvVWwOk2PdYY6PEFegR1vezXR0tw6fZGF9csVakIRjrJiy2veSBFQ==",
"dependencies": {
"define-data-property": "^1.1.1",
"get-intrinsic": "^1.2.1",
"gopd": "^1.0.1",
"has-property-descriptors": "^1.0.0"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/setprototypeof": {
"version": "1.2.0",
"resolved": "https://registry.npmmirror.com/setprototypeof/-/setprototypeof-1.2.0.tgz",
"integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="
},
"node_modules/side-channel": {
"version": "1.0.4",
"resolved": "https://registry.npmmirror.com/side-channel/-/side-channel-1.0.4.tgz",
"integrity": "sha512-q5XPytqFEIKHkGdiMIrY10mvLRvnQh42/+GoBlFW3b2LXLE2xxJpZFdm94we0BaoV3RwJyGqg5wS7epxTv0Zvw==",
"dependencies": {
"call-bind": "^1.0.0",
"get-intrinsic": "^1.0.2",
"object-inspect": "^1.9.0"
}
},
"node_modules/statuses": {
"version": "2.0.1",
"resolved": "https://registry.npmmirror.com/statuses/-/statuses-2.0.1.tgz",
"integrity": "sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/toidentifier": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/toidentifier/-/toidentifier-1.0.1.tgz",
"integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
"engines": {
"node": ">=0.6"
}
},
"node_modules/type-is": {
"version": "1.6.18",
"resolved": "https://registry.npmmirror.com/type-is/-/type-is-1.6.18.tgz",
"integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
"dependencies": {
"media-typer": "0.3.0",
"mime-types": "~2.1.24"
},
"engines": {
"node": ">= 0.6"
}
},
"node_modules/unpipe": {
"version": "1.0.0",
"resolved": "https://registry.npmmirror.com/unpipe/-/unpipe-1.0.0.tgz",
"integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/utils-merge": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/utils-merge/-/utils-merge-1.0.1.tgz",
"integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==",
"engines": {
"node": ">= 0.4.0"
}
},
"node_modules/vary": {
"version": "1.1.2",
"resolved": "https://registry.npmmirror.com/vary/-/vary-1.1.2.tgz",
"integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
"engines": {
"node": ">= 0.8"
}
}
}
}

0
scripts/icon/package.json Normal file → Executable file
View File

0
scripts/postinstall.sh Normal file → Executable file
View File