Commit Graph

13 Commits

Author SHA1 Message Date
Archer 9959707fb3 V4.14.9 fix issue (#6573)
* fix: session error

* fix: session error

* fix: workflow runtime and add e2b
2026-03-19 11:15:14 +08:00
Archer 6b61359516 feat(sandbox): 重构代码沙盒,支持内置函数和网络请求 (#6479)
* fix(sandbox): 重构代码沙盒,支持内置函数和网络请求 (#6462)

* feat(sandbox): 重写代码沙盒 - Bun + Hono + 统一子进程模型

- 运行时: Node.js → Bun
- HTTP 框架: NestJS + Fastify → Hono
- JS 执行: isolated-vm → Bun 子进程(与 Python 统一)
- 架构: 统一子进程模型,JS 和 Python 共享同一套执行引擎

- SubprocessRunner 基类,JS/Python 各自继承
- ProcessPool 进程池预热(SANDBOX_JS_POOL_SIZE / SANDBOX_PYTHON_POOL_SIZE)
- SystemHelper 命名空间(JS 端保留向后兼容全局函数)
- 临时文件系统隔离 + 路径遍历防护 + 磁盘配额
- 请求级资源限制(timeoutMs / memoryMB / diskMB)

- JS: 原型链冻结 + Bun API 禁用 + Function 构造器注入 safe require(模块白名单)
- Python: 宿主侧正则预检 + __import__ 拦截 + resource 限制

- 移除: @nestjs/*(6个包)、fastify、isolated-vm、node-gyp、reflect-metadata、rxjs
- 新增: hono
- 保留: tiktoken
- 新增用户可用包: lodash、dayjs、axios、moment、uuid、crypto-js、qs

- 67 个测试全部通过(单元测试 + 安全测试 + 集成测试)
- vitest 独立配置,不影响全局

* fix(sandbox): 安全加固 - 扩展 Bun API 封锁、清理 process.env、闭包封装 Python import 拦截

- JS: 扩展 Bun 危险 API 封锁列表(serve/connect/listen/udpSocket/dns/plugin/build/Transpiler)
- JS: 清理 process.env,仅保留沙箱必要变量,防止泄露敏感环境变量
- Python: 用闭包封装 _safe_import,del 掉 _original_import/_make_safe_import/_BLOCKED_MODULES
  防止用户代码恢复原始 __import__
- Dockerfile: 复制 bun.lock 并使用 --frozen-lockfile 确保构建可复现

* fix(sandbox): 将 sandbox 从 pnpm workspace 中移除,独立管理依赖

* fix(sandbox): 从全局 vitest 移除 sandbox 测试,集成测试无 SANDBOX_URL 时跳过

* ci(sandbox): 添加独立测试 workflow,仅 sandbox 代码变更时触发

* refactor(sandbox): 使用 export default 启动方式,与 sandbox_server 保持一致

* fix: sandbox security hardening & comprehensive test suite

Security fixes:
- JS: freeze Function constructor to block constructor.constructor escape
- JS: handle undefined return from main() (serialize as null)
- Python: fix http_request using from-import after __import__ interception
- Python: __import__ whitelist mode blocks exec/eval import bypasses

New tests (223 passing):
- security/escape-attacks: JS/Python escape attack vectors
- security/network-security: IP blacklist, protocol restrictions, httpRequest
- compat/legacy-js: 18 backward compatibility tests
- compat/legacy-python: 21 backward compatibility tests
- boundary: timeout, memory, disk, edge cases
- examples: common user code patterns

* feat(sandbox): env vars for all limits + rewrite README

- Network limits configurable via env: SANDBOX_MAX_REQUESTS, SANDBOX_REQUEST_TIMEOUT, SANDBOX_MAX_RESPONSE_SIZE
- Resource upper bounds configurable: SANDBOX_MAX_TIMEOUT, SANDBOX_MAX_MEMORY_MB, SANDBOX_MAX_DISK_MB
- README: architecture, API docs, env var reference, how to add JS/Python packages, security overview, built-in functions

* refactor(sandbox): extract env.ts with dotenv for typed env loading

- New env.ts: dotenv.config() + typed helpers (str/int/bool)
- config.ts re-exports env for backward compatibility
- index.ts imports env first to ensure .env loaded before anything else

* refactor(sandbox): use zod for env validation and type coercion

- Replace manual parseInt/str helpers with zod schema + coerce
- Invalid env vars now fail fast with formatted error on startup
- dotenv + zod, clean and declarative

* chore(sandbox): remove unused process pool code

- Delete pool.ts and pool.test.ts (pool was never wired into runners)
- Remove PoolConfig/PooledProcess types
- Remove pool env vars from env.ts
- Clean up README

* feat(sandbox): add concurrency limiter with semaphore

- New Semaphore utility for max concurrent subprocess control
- SANDBOX_MAX_CONCURRENCY env var (default 50)
- Excess requests queue instead of spawning unbounded processes
- Health endpoint exposes concurrency stats (current/queued/max)

* test(sandbox): add semaphore tests and expand coverage to 292 cases

- New semaphore.test.ts (11 tests): acquire/release, queuing, FIFO, stats, serial execution
- JS runner: blank code, template literals, primitive returns, more modules, unicode, partial limits
- Python runner: blank code, triple quotes, primitive returns, unicode, null vars, division errors
- JS security: process.exit, globalThis, Symbol.unscopables, Proxy, dynamic import, path traversal
- Python security: pickle/multiprocessing/threading/ctypes/signal, exec bypass, __subclasses__
- Escape attacks: type() class creation, __builtins__ tampering, getattr access
- Boundary: long vars, special JSON chars, float precision, big ints, circular refs, Promise.reject

* test(sandbox): test-master review - add 31 tests, coverage report

- base-runner.test.ts (10): BaseRunner precheck, temp dir, semaphore integration
- semaphore-race.test.ts (5): race conditions, rapid acquire/release, stress test
- coverage-gaps.test.ts (16): security coverage gaps found during review
- REVIEW-REPORT.md: full test audit report

Total: 323 passed, 0 failed

* fix(sandbox): address PR #6439 review issues

Security fixes:
- Intercept Python builtins.open(), restrict file access to sandbox tmpdir
- Remove unused pool.ts, warmup.mjs, warmup.py (security risk)
- Fix DNS rebinding TOCTOU: use resolved IP for HTTP connections
- Fix symlink path traversal: use realpath instead of normpath
- Add try/finally cleanup for __import__ hook

Robustness:
- Add __SANDBOX_RESULT__ prefix to stdout parsing, prevent user output interference
- Fix disk quota tracking: deduct old file size on overwrite
- Add __import__() pattern scanning in Python precheck

Tests:
- Fix eval+__import__ test assertion (accept both catch and fail paths)

All 323 tests passing.

* fix(sandbox): remove warmup scripts COPY from Dockerfile

* docs(sandbox): add technical design document

* feat(sandbox): configurable module allowlist/blocklist via env vars

- SANDBOX_JS_ALLOWED_MODULES: JS require whitelist (comma-separated)
- SANDBOX_PYTHON_BLOCKED_MODULES: Python import blacklist (comma-separated)
- Defaults unchanged, fully backward compatible

* fix(sandbox): 修复多个安全漏洞

1. Python HTTPS DNS rebinding: HTTPS 请求现在也使用 resolved IP 发起连接
2. Python __import__ hook 恢复漏洞: 移除 finally 块中恢复原始 __import__ 的代码
3. Python 内部变量泄露: 用户代码执行前删除 _os, _socket 等内部模块引用
4. JS process 危险 API: 禁用 process.binding/dlopen/kill/chdir 等,冻结 process.env
5. Python open() fd 绕过: 阻止通过整数文件描述符绕过路径检查
6. API 输入校验: 使用 zod schema 校验请求体,限制代码大小 1MB
7. 无认证警告: SANDBOX_TOKEN 未设置时输出生产环境警告

新增 security-fixes.test.ts 包含所有修复的回归测试

* test: consolidate security tests + add integration test suite

- Merge 6 security test files into 1 consolidated security.test.ts (109 tests)
  - JS/Python module interception (precheck + runtime)
  - JS escape attacks (prototype, constructor, Reflect, globalThis)
  - Python escape attacks (__import__ hook, exec/eval, internal vars, __subclasses__)
  - SSRF protection (private IPs, cloud metadata, file protocol)
  - File system isolation (path traversal, fd, disk quota)
  - Variable injection attacks
  - API input validation

- Add black-box integration test suite functional.test.ts (56 tests)
  - Basic operations (math, string, array, JSON, regex, Date, Promise, Map/Set)
  - Variable passing (string, number, complex objects, empty, multiple)
  - Whitelisted modules (crypto-js, moment, lodash)
  - SystemHelper/system_helper (fs, delay, strToBase64, httpRequest)
  - Error handling (syntax, runtime, undefined var, timeout)
  - Network requests (GET, POST)
  - Complex scenarios (CSV pipeline, recursion, class definition)

- Remove 34 duplicate test cases across merged files
- Total: 363 passed, 8 skipped (integration API tests need server)

* fix(sandbox): z.record() zod v4 compatibility - add key type param

* feat(sandbox): add .env.template with all config options and comments

* refactor(sandbox): remove disk write support and temp filesystem

* test(sandbox): remove all fs-related tests and add test case inventory

- Remove fs read/write tests from unit, integration, boundary, examples
- Remove path traversal, absolute path, open fd, builtins.open tests from security
- Add comprehensive test/case.md with all 344 test cases categorized
- All tests pass: 344 passed, 8 skipped, 0 failed

* feat(sandbox): add GET /sandbox/modules API to list available packages and builtins

* test(sandbox): add unit tests for GET /sandbox/modules API

* refactor(test): rewrite api.test.ts to use app.request() - no external server needed

* feat(sandbox): validate SANDBOX_TOKEN charset in env schema (ASCII printable only)

* chore(sandbox): remove DESIGN.md and package-lock.json from PR

* feat(sandbox): replace spawn-per-request with process pool architecture

- Add ProcessPool (JS) and PythonProcessPool with long-lived worker processes
- Workers communicate via stdin/stdout line-based JSON protocol
- Pool size configurable via SANDBOX_POOL_SIZE env var (default 20)
- Auto-respawn workers on crash
- Semaphore-based queueing when requests exceed pool size

Performance gains (simple functions):
- JS: 22 QPS → 1,328 QPS (60x improvement)
- Python: 14.7 QPS → 3,395 QPS (231x improvement)

- Fix import.meta.dir compatibility for vitest (Node) environments
- Export poolReady promise for test initialization
- Add benchmark scripts to test/benchmark/
- All 354 tests passing (12 test files)

* chore(sandbox): clean up unused files, update README with pool architecture

- Remove test/REVIEW-REPORT.md, test/case.md, test/benchmark.ts (obsolete)
- Rewrite README: pool architecture diagram, performance benchmarks,
  SANDBOX_POOL_SIZE config, project structure, health endpoint format

* fix(sandbox): 修复进程池超时后 worker respawn 竞态条件

根因:超时 kill worker 后,exit 事件是异步的,release() 先执行时
worker 还在列表里,死 worker 被放回 idle 池,后续请求发给死进程。

修复:
- 超时回调中先 removeWorker 再 kill,防止 release 归还死 worker
- removeWorker 返回 bool,exit 事件中避免重复 respawn
- 超时回调主动触发 spawnWorker 补充池
- release 检查 worker 是否仍在池中
- spawnWorker 完成时检查 waitQueue 直接分配

* fix: security hardening & test migration to process pool

- JS worker: harden process object (kill/chdir/env freeze/binding/dlopen)
- Python worker: stack-frame based __import__ hook to block exec/eval bypass
- Python worker: BuiltinsProxy to prevent __import__ override via builtins module
- Python worker: restricted __builtins__ dict in exec_globals (no internal refs)
- Python worker: restore __import__ before each execution
- Migrate all 9 test files from JsRunner/PythonRunner to ProcessPool/PythonProcessPool
- Configure vitest for serial execution (pool size=1, fileParallelism: false)
- Fix security test assertion for builtins tampering (success=true with escaped=false)
- All 102 security tests passing

* docs(sandbox): update README with accurate benchmark data, remove non-existent features

- Update performance table with latest benchmark results (JS 1414 QPS, Python 4247 QPS)
- Remove SANDBOX_DISK_MB/SANDBOX_MAX_DISK_MB env vars (not implemented)
- Remove SystemHelper.fs.* / system_helper.fs.* docs (not implemented in workers)
- Fix security section to match actual implementation
- Update test count to 351

* refactor(sandbox): remove legacy runner/sandbox/template code

- Delete src/runner/ (base.ts, js-runner.ts, python-runner.ts)
- Delete src/sandbox/ (js-template.ts, python-template.ts, network-config.ts)
- Delete test/unit/js-runner.test.ts, test/unit/python-runner.test.ts
- Keep src/utils/semaphore.ts (generic utility, has its own tests)
- Update README project structure and test count (297 cases)

All functionality is now in src/pool/ (process-pool architecture).
297 tests passing, 0 failures.

* test(sandbox): add process pool lifecycle/respawn/concurrency tests

- ProcessPool: init/shutdown/stats, worker crash respawn, timeout respawn,
  pool-full queuing, concurrent crash isolation
- PythonProcessPool: init/shutdown/stats, timeout respawn, queuing
- 14 new test cases, total 311 passing

* fix(sandbox): ping/pong health check, replace httpbin.org with baidu.com

- Worker health check: send actual ping message and verify pong response
  instead of only checking stdin.writable (detects stuck workers)
- JS worker.ts: handle {type:'ping'} → reply {type:'pong'}
- Python worker.py: handle {type:'ping'} → reply {type:'pong'}
- ProcessPool/PythonProcessPool: rewrite pingWorker to send ping,
  wait for pong with timeout, replace worker on failure
- Replace all httpbin.org URLs with www.baidu.com in tests
  (httpbin.org unreachable from China/Sealos Devbox)
- Add 4 new health check tests (ping/pong for JS and Python pools)
- All 318 tests passing, 0 failures

* docs: add test report (test/README.md) and update README testing section

- test/README.md: detailed report with 315 passed / 3 skipped / 0 failed
- README.md: updated test section with coverage dimensions table and link to report

* docs: add functional test cases checklist (110 cases)

* fix(sandbox): fix Dockerfile Python env and import detection

1. Dockerfile: Remove broken multi-stage Python 3.11 copy.
   - The previous approach copied python3 binary from python:3.11-alpine
     but missed libpython3.11.so.1.0, causing Python pool init failure.
   - Now uses system Python from apk and installs pip packages directly.

2. worker.py: Fix false positive import blocking for third-party packages.
   - numpy/pandas were blocked because their internal 'import os' was
     detected as user-initiated (full stack scan found user code frames).
   - Changed to check only the direct caller frame: if the import comes
     from site-packages (third-party lib internals), allow it.
   - Direct user imports of blocked modules are still properly rejected.

* fix(sandbox): block dynamic import() and restrict file system access

Security fixes found during deep review:

1. JS: Block import() dynamic imports that bypass require whitelist.
   - import('fs') could read arbitrary files on the container.
   - Added static regex check to reject code containing import().

2. Python: Restrict open() to prevent user code from reading files.
   - open('/etc/passwd') was accessible from user code.
   - Added _restricted_open() that checks caller frame: only allows
     stdlib/site-packages internal calls, blocks user code (<string>).

3. Python: Remove duplicate return statement in _safe_import.

All 315 tests pass (3 skipped).

* test(sandbox): add regression tests for import() and open() security fixes

- JS: import('fs'), import('child_process'), import('os') blocked
- JS: string containing 'import' not false-positive
- Python: open('/etc/passwd'), open('/proc/self/environ'), open('/tmp/evil.txt', 'w') blocked
- Python: numpy internal open() not affected (conditional on numpy availability)

Total: 322 passed | 3 skipped (was 315 passed)

* docs(sandbox): rewrite sandbox documentation with JS + Python coverage

- Add Python language support documentation
- Add httpRequest/http_request function docs
- Add available modules list (JS whitelist + Python safe modules)
- Add security restrictions section
- Add practical examples (data processing, date calc, webhook signing)
- Add JS/Python function name mapping table

* docs(sandbox): use SystemHelper/system_helper for built-in functions

Direct calls (countToken, delay, etc.) are deprecated (kept for compat).
All examples now use SystemHelper.xxx() / system_helper.xxx().

* docs(sandbox): Python only show named-params style as recommended

* feat(sandbox): unify Python SystemHelper API with camelCase aliases

- Add camelCase aliases to Python SystemHelper: countToken, strToBase64,
  createHmac, httpRequest (matching JS API exactly)
- Update docs to use SystemHelper uniformly for both JS and Python
- snake_case methods (count_token, etc.) still work for backward compat

* feat(sandbox): add matplotlib and increase HTTP timeout to 60s

- Add matplotlib to Python dependencies
- Increase HTTP request timeout from 10s to 60s (both JS and Python)
- Update docs accordingly

* docs(sandbox): split docs for old/new sandbox versions

- sandbox.mdx → '代码运行(旧版)' for FastGPT ≤ 4.14.7 (URL unchanged)
- sandbox-v5.mdx → '代码运行' for FastGPT ≥ 4.14.8
- Both pages cross-link to each other
- meta.json updated: sandbox-v5 listed before sandbox

* docs: rename old sandbox doc to 代码运行(弃)

* refactor(sandbox): remove SANDBOX_TIMEOUT, use SANDBOX_MAX_TIMEOUT as unified timeout

* fix(sandbox): add build dependencies for matplotlib in Dockerfile

* refactor(sandbox): migrate Python from blocklist to allowlist for module control

- Change SANDBOX_PYTHON_BLOCKED_MODULES to SANDBOX_PYTHON_ALLOWED_MODULES
- Update Python worker to use allowlist instead of blocklist
- Add comprehensive safe module list: math, json, datetime, numpy, pandas, etc.
- Improve error message: 'Module X is not in the allowlist'
- Consistent with JS allowlist approach for better security

* fix(sandbox): add _strptime to allowlist and update test assertions

- Add _strptime module (required by datetime.strptime)
- Update test assertions for Python module import errors
- All 325 tests now pass (322 passed, 3 skipped)

* fix(docs): center SVG icon in size-5 container on medium screens

* docs(sandbox): simplify built-in functions and improve module documentation

- Remove delay, countToken, strToBase64, createHmac functions (keep only httpRequest)
- Convert Python module list to table format (10 tables by category)
- Reorganize usage examples with collapsible sections (JS and Python)
- Fix icon alignment in desktop/mobile sidebar navigation
- All 325 tests passing

---------

Co-authored-by: Lobster 3 <lobster3@sandbox.dev>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>
Co-authored-by: Archer <c121914yu@gmail.com>
Co-authored-by: archer <archer@archerdeMac-mini.local>

* perf: code sandbox

* update action

* Update projects/app/src/components/core/chat/ChatContainer/ChatBox/index.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update timeout

* update memory limit function

* sandbox

* perf: process poll

* env template

* feat: code tip

* fix: code sandbox error tip

* update memory limit fn

* update memory limit fn

* fix: test

* fix: test

* fix: sandbox

---------

Co-authored-by: Archer <archer@fastgpt.io>
Co-authored-by: Lobster 3 <lobster3@sandbox.dev>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>
Co-authored-by: Archer <c121914yu@gmail.com>
Co-authored-by: archer <archer@archerdeMac-mini.local>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-28 12:36:59 +08:00
Archer 76d6234de6 V4.14.7 features (#6406)
* Agent features (#6345)

* Test agent (#6220)

* squash: compress all commits into one

* feat: plan response in ui

* response ui

* perf: agent config

* merge

* tool select ux

* perf: chat ui

* perf: agent editform

* tmp code

* feat: save chat

* Complete agent parent  (#6049)

* add role and tools filling

* add: file-upload

---------

Co-authored-by: xxyyh <2289112474@qq>

* perf: top agent code

* top agent (#6062)

Co-authored-by: xxyyh <2289112474@qq>

* fix: ts

* skill editor ui

* ui

* perf: rewrite type with zod

* skill edit ui

* skill agent (#6089)

* cp skill chat

* rebase fdf933d
 and add skill chat

* 1. skill 的 CRUD
2. skill 的信息渲染到前端界面

* solve comment

* remove chatid and chatItemId

* skill match

* perf: skill manage

* fix: ts

---------

Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: archer <545436317@qq.com>

* fix: ts

* fix: loop import

* skill tool config (#6114)

Co-authored-by: xxyyh <2289112474@qq>

* feat: load tool in agent

* skill memory (#6126)

Co-authored-by: xxyyh <2289112474@qq>

* perf: agent skill editor

* perf: helperbot ui

* agent code

* perf: context

* fix: request context

* agent usage

* perf: agent context and pause

* perf: plan response

* Test agent sigle skill (#6184)

* feat:top box fill

* prompt fix

---------

Co-authored-by: xxyyh <2289112474@qq>

* perf: agent chat ui

* Test agent new (#6219)

* have-replan

* agent

---------

Co-authored-by: xxyyh <2289112474@qq>

* fix: ts

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>

* feat: consolidate agent and MCP improvements

This commit consolidates 17 commits including:
- MCP tools enhancements and fixes
- Agent system improvements and optimizations
- Auth limit and prompt updates
- Tool response compression and error tracking
- Simple app adaptation
- Code quality improvements (TypeScript, ESLint, Zod)
- Version type migration to schema
- Remove deprecated useRequest2
- Add LLM error tracking
- Toolset ID validation fixes

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>

* fix: transform avatar copy;perf: filter invalid tool

* update llm response storage time

* fix: openapi schema

* update skill desc

* feat: cache hit data

* i18n

* lock

* chat logs support error filter & user search (#6373)

* chat log support searching by user name

* support error filter

* fix

* fix overflow

* optimize

* fix init script

* fix

* perf: get log users

* updat ecomment

* fix: ts

* fix: test

---------

Co-authored-by: archer <545436317@qq.com>

* Fix: agent  (#6376)

* Agent features (#6345)

* Test agent (#6220)

* squash: compress all commits into one

* feat: plan response in ui

* response ui

* perf: agent config

* merge

* tool select ux

* perf: chat ui

* perf: agent editform

* tmp code

* feat: save chat

* Complete agent parent  (#6049)

* add role and tools filling

* add: file-upload

---------

Co-authored-by: xxyyh <2289112474@qq>

* perf: top agent code

* top agent (#6062)

Co-authored-by: xxyyh <2289112474@qq>

* fix: ts

* skill editor ui

* ui

* perf: rewrite type with zod

* skill edit ui

* skill agent (#6089)

* cp skill chat

* rebase fdf933d
 and add skill chat

* 1. skill 的 CRUD
2. skill 的信息渲染到前端界面

* solve comment

* remove chatid and chatItemId

* skill match

* perf: skill manage

* fix: ts

---------

Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: archer <545436317@qq.com>

* fix: ts

* fix: loop import

* skill tool config (#6114)

Co-authored-by: xxyyh <2289112474@qq>

* feat: load tool in agent

* skill memory (#6126)

Co-authored-by: xxyyh <2289112474@qq>

* perf: agent skill editor

* perf: helperbot ui

* agent code

* perf: context

* fix: request context

* agent usage

* perf: agent context and pause

* perf: plan response

* Test agent sigle skill (#6184)

* feat:top box fill

* prompt fix

---------

Co-authored-by: xxyyh <2289112474@qq>

* perf: agent chat ui

* Test agent new (#6219)

* have-replan

* agent

---------

Co-authored-by: xxyyh <2289112474@qq>

* fix: ts

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>

* feat: consolidate agent and MCP improvements

This commit consolidates 17 commits including:
- MCP tools enhancements and fixes
- Agent system improvements and optimizations
- Auth limit and prompt updates
- Tool response compression and error tracking
- Simple app adaptation
- Code quality improvements (TypeScript, ESLint, Zod)
- Version type migration to schema
- Remove deprecated useRequest2
- Add LLM error tracking
- Toolset ID validation fixes

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>

* 1. 把辅助生成前端上的 system prompt 加入到上下文中
2. mcp工具的前端渲染(图标)
3. 文件读取工具和文件上传进行关联
4. 添加了辅助生成返回格式出错的重试方案
5. ask 不出现在 plan 步骤中
6. 添加了辅助生成的头像和交互 UI

* fix:read_file

* helperbot ui

* ts error

* helper ui

* delete Unused import

* perf: helper bot

* lock

---------

Co-authored-by: Archer <545436317@qq.com>
Co-authored-by: xxyyh <2289112474@qq>

* fix date variable required & model auth (#6386)

* fix date variable required & model auth

* doc

* feat: add chat id to finish callback

* fix: iphone safari shareId (#6387)

* fix: iphone safari shareId

* fix: mcp file list can't setting

* fix: reason output field

* fix: skip JSON validation for HTTP tool body with variable (#6392)

* fix: skip JSON validation for HTTP tool body with variable

* doc

* workflow fitview

* perf: selecting memory

* perf: cp api

* ui

* perf: toolcall auto adapt

* fix: catch workflow error

* fix: ts

* perf: pagination type

* remove

* ignore

* update doc

* fix: simple app tool select

* add default avatar to logs user

* perf: loading user

* select dataset ui

* rename version

* feat: add global/common test

* perf: packages/global/common test

* feat: package/global/ai,app test

* add global/chat test

* global/core test

* global/core test

* feat: packages/global all test

* perf: test

* add server api test

* perf: init shell

* perf: init4150 shell

* remove invalid code

* update doc

* remove log

* fix: chat effect

* fix: plan fake tool  (#6398)

* 1. 提示词防注入功能
2. 无工具不进入 plan,防止虚拟工具生成

* Agent-dataset

* dataset

* dataset presetInfo

* prefix

* perf: prompt

---------

Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: archer <545436317@qq.com>

* fix: review

* adapt kimi2.5 think toolcall

* feat: invoke fastgpt user info (#6403)

feat: invoke fastgpt user info

* fix: invoke fastgpt user info return orgs (#6404)

* skill and version

* retry helperbot (#6405)

Co-authored-by: xxyyh <2289112474@qq>

* update template

* remove log

* doc

* update doc

* doc

* perf: internal ip check

* adapt get paginationRecords

* tool call adapt

* fix: test

* doc

* fix: agent initial version

* adapt completions v1

* feat: instrumentation check

* rename skill

* add workflow demo mode tracks (#6407)

* chore: 统一 skills 目录命名为小写

将 .claude/Skills/ 重命名为 .claude/skills/ 以保持命名一致性。

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* add workflow demo mode tracks

* code

* optimize

* fix: improve workflowDemoTrack based on PR review

- Add comment to empty catch block for maintainability
- Add @param docs to onDemoChange clarifying nodeCount usage
- Replace silent .catch with console.debug for dev debugging
- Handle appId changes by reporting old data before re-init

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: archer <545436317@qq.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* remove repeat skill

* fix(workflow): filter out orphan edges to prevent runtime errors (#6399)

* fix(workflow): filter out orphan edges to prevent runtime errors

Runtime edges that reference non-existent nodes (orphan edges) can cause
unexpected behavior or crashes during workflow dispatch. This change adds
a pre-check to filter out such edges before execution begins, ensuring
system stability even with inconsistent graph data.

* fix(workflow): enhance orphan edge filtering with logging and tests

- Refactor: Extract logic to 'filterOrphanEdges' in utils.ts for better reusability
- Feat: Add performance monitoring (warn if >100ms) and comprehensive logging
- Feat: Support detailed edge inspection in debug mode
- Docs: Add JSDoc explaining causes of orphan edges (migration, manual edits)
- Test: Add unit tests covering edge cases and performance (1000 edges)

Addresses PR review feedback regarding logging, variable naming, and testing."

* move code

* move code

* add more unit test

---------

Co-authored-by: archer <545436317@qq.com>

* test

* perf: test

* add server/common/string test

* fix: resolve $ref references in MCP tool input schemas (#6395) (#6409)

* fix: resolve $ref references in MCP tool input schemas (#6395)

* add test code

---------

Co-authored-by: archer <545436317@qq.com>

* chore(docs): add fastgpt, fastgpt-plugin version choice guide (#6411)

* chore(doc): add fastgpt version description

* doc

* doc

---------

Co-authored-by: archer <545436317@qq.com>

* fix:dataset cite and description info (#6410)

* 1. 添加知识库引用(plan 步骤和直接知识库调用)
2. 提示词框中的@知识库工具
3. plan 中 step 的 description dataset_search 改为中文

* fix: i18n

* prompt

* prompt

---------

Co-authored-by: xxyyh <2289112474@qq>

* fix: tool call

* perf: workflow props

* fix: merge ECharts toolbox options instead of overwriting (#6269) (#6412)

* feat: integrate logtape and otel (#6400)

* fix: deps

* feat(logger): integrate logtape and otel

* wip(log): add basic infras logs

* wip(log): add request id and inject it into context

* wip(log): add basic tx logs

* wip(log): migrate

* wip(log): category

* wip(log): more sub category

* fix: type

* fix: sessionRun

* fix: export getLogger from client.ts

* chore: improve logs

* docs: update signoz and changelog

* change type

* fix: ts

* remove skill.md

* fix: lockfile specifier

* fix: test

---------

Co-authored-by: archer <545436317@qq.com>

* init log

* doc

* remove invalid log

* fix: review

* template

* replace new log

* fix: ts

* remove log

* chore: migrate all addLog to logtape

* move skill

* chore: migrate all addLog to logtape (#6417)

* update skill

* remove log

* fix: tool check

---------

Co-authored-by: YeYuheng <57035043+YYH211@users.noreply.github.com>
Co-authored-by: xxyyh <2289112474@qq>
Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: xuyafei1996 <54217479+xuyafei1996@users.noreply.github.com>
Co-authored-by: ToukoYui <2331631097@qq.com>
Co-authored-by: roy <whoeverimf5@gmail.com>
2026-02-12 16:37:50 +08:00
Archer e6c7593d95 Sandbox move and vector test (#6381)
* remove sandbox

* perf: vitest

* doc
2026-02-04 14:27:58 +08:00
Archer 64f70a41c1 feat: vector integrationTest;feat: ob quantization (#6366)
* feat(vectordb): add OceanBase HNSW quantization (HNSW_SQ/HNSW_BQ) (#6348)

Support OceanBase vector index quantization via VECTOR_VQ_LEVEL:
- 32 (default): hnsw + inner_product
- 8: hnsw_sq + inner_product (2-3x memory savings)
- 1: hnsw_bq + cosine (~15x memory savings)

HNSW_BQ requires cosine distance per OceanBase docs.
Tested on OceanBase 4.3.5.5 (BP5).

Closes #6202

* feat: add test inclusion for vectorDB tests in vitest configuration (#6358)

* feat: add test inclusion for vectorDB tests in vitest configuration

* refactor: update vectorDB README and setup for environment configuration

- Enhanced README to clarify the use of factory pattern for vectorDB integration tests.
- Updated instructions for setting up environment variables from a local file.
- Removed obsolete PG integration test file and adjusted test execution instructions.
- Improved structure explanation for shared test data and factory functions.

* perf: integrationTest

* feat: vector integration

---------

Co-authored-by: ZHANG Yixin <hi.yixinz@gmail.com>
Co-authored-by: Jingchao <alswlx@gmail.com>
2026-02-02 18:48:25 +08:00
Archer 2fea73bb68 perf: index (#6131)
* perf: index

* stop design doc

* perf: stop workflow;perf: mongo connection

* fix: ts

* mq export
2025-12-21 19:15:10 +08:00
Archer 58000324e2 feature: V4.14.3 (#5970)
* feat(marketplace): update plugin/ download count statistic (#5957)

* feat: download count

* feat: update ui

* fix: ui

* chore: update sdk verison

* chore: update .env.template

* chore: adjust

* chore: remove console.log

* chore: adjust

* Update projects/marketplace/src/pages/index.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update projects/marketplace/src/pages/index.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update projects/app/src/pages/config/tool/marketplace.tsx

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: update refresh; feat: marketplace download count per hour

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* download

* marketplace code

* fix: ui (#5963)

* feat: support dataset and files as global variables (#5961)

* json & dataset

* file

* fix file var

* fix

* fix init

* remove

* perf: file vars

* fix: file uploading errors (#5969)

* fix: file uploading errors

* fix build

* perf: fileselector ux

* feat: integrate S3 for dataset with compatibility (#5941)

* fix: text split

* remove test

* feat: integrate S3 for dataset with compatibility

* fix: delay s3 files delete timing

* fix: remove imageKeys

* fix: remove parsed images' TTL

* fix: improve codes by pr comments

---------

Co-authored-by: archer <545436317@qq.com>

* remove log

* perf: request limit

* chore: s3 migration script (#5971)

* test

* perf: s3 code

* fix: migration script (#5972)

* perf: s3 move object

* wip: fix s3 bugs (#5976)

* fix: incorrect replace origin logic (#5978)

* fix: add downloadURL (#5980)

* perf: file variable ttl & quick create dataset with temp s3 bucket (#5973)

* perf: file variable ttl & quick create dataset with temp s3 bucket

* fix

* plugin & form input variables (#5979)

* plugin & form input variables

* fix

* docs: 4143.mdx (#5981)

* doc: update 4143.mdx (#5982)

* fix form input file ttl (#5983)

* trans file type (#5986)

* trans file type

* fix

* fix: S3 script early return (#5985)

* fix: S3 script typeof

* fix: truncate large filename to fit S3 name

* perf(permission): add a schema verification for resource permission, tmbId, groupId, orgId should be set at least one of them (#5987)

* fix: version & typo (#5988)

* fix-v4.14.3 (#5991)

* fix: empty alt make replace JWT failed & incorrect image dataset preview url (#5989)

* fix: empty alt make replace JWT failed & incorrect image dataset preview url

* fix: s3 files recovery script

* fix: incorrect chat external url parsing (#5993)

---------

Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: Roy <whoeverimf5@gmail.com>
2025-11-26 20:47:28 +08:00
Archer 2ed1545eb5 V4.12.4 features (#5626)
* fix: push again, user select option button and form input radio content overflow (#5601)

* fix: push again, user select option button and form input radio content overflow

* fix: use useCallback instead of useMemo, fix unnecessary delete

* fix: Move the variable inside the component

* fix: do not pass valueLabel to MySelect

* ui

* del collection api adapt

* refactor: inherit permission (#5529)

* refactor: permission update conflict check function

* refactor(permission): app collaborator update api

* refactor(permission): support app update collaborator

* feat: support fe permission conflict check

* refactor(permission): app permission

* refactor(permission): dataset permission

* refactor(permission): team permission

* chore: fe adjust

* fix: type error

* fix: audit pagiation

* fix: tc

* chore: initv4130

* fix: app/dataset auth logic

* chore: move code

* refactor(permission): remove selfPermission

* fix: mock

* fix: test

* fix: app & dataset auth

* fix: inherit

* test(inheritPermission): test syncChildrenPermission

* prompt editor add list plugin (#5620)

* perf: search result (#5608)

* fix: table size (#5598)

* temp: list value

* backspace

* optimize code

---------

Co-authored-by: Archer <545436317@qq.com>
Co-authored-by: 伍闲犬 <whoeverimf5@gmail.com>

* fix: fe & member list (#5619)

* chore: initv4130

* fix: MemberItemCard

* fix: MemberItemCard

* chore: fe adjust & init script

* perf: test code

* doc

* fix debug variables (#5617)

* perf: search result (#5608)

* fix: table size (#5598)

* fix debug variables

* fix

---------

Co-authored-by: Archer <545436317@qq.com>
Co-authored-by: 伍闲犬 <whoeverimf5@gmail.com>

* perf: member ui

* fix: inherit bug (#5624)

* refactor(permission): remove getClbsWithInfo, which is useless

* fix: app list privateApp

* fix: get infos

* perf(fe): remove delete icon when it is disable in MemberItemCard

* fix: dataset private dataset

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Archer <545436317@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* perf: auto coupon

* chore: upgrade script & get infos avatar  (#5625)

* fix: get infos

* chore: initv4130

* feat: support WecomRobot publish, and fix AesKey can not save bug (#5526)

* feat: resolve conflicts

* fix: add param 'show_publish_wecom'

* feat: abstract out WecomCrypto type

* doc: wecom robot document

* fix: solve instability in AI output

* doc: update some pictures

* feat: remove functions from request.ts to chat.ts and toolCall.ts

* doc: wecom robot doc update

* fix

* delete unused code

* doc: update version and prompt

* feat: remove wecom crypto, delete wecom code in workflow

* feat: delete unused codes

---------

Co-authored-by: heheer <zhiyu44@qq.com>

* remove test

* rename init shell

* feat: collection page store

* reload sandbox

* pysandbox

* remove log

* chore: remove useless code (#5629)

* chore: remove useless code

* fix: checkConflict

* perf: support hidden type for RoleList

* fix: copy node

* update doc

* fix(permission): some bug (#5632)

* fix: app/dataset list

* fix: inherit bug

* perf: del app;i18n;save chat

* fix: test

* i18n

* fix: sumper overflow return OwnerRoleVal (#5633)

* remove invalid code

* fix: scroll

* fix: objectId

* update next

* update package

* object id

* mock redis

* feat: add redis append to resolve wecom stream response  (#5643)

* feat: resolve conflicts

* fix: add param 'show_publish_wecom'

* feat: abstract out WecomCrypto type

* doc: wecom robot document

* fix: solve instability in AI output

* doc: update some pictures

* feat: remove functions from request.ts to chat.ts and toolCall.ts

* doc: wecom robot doc update

* fix

* delete unused code

* doc: update version and prompt

* feat: remove wecom crypto, delete wecom code in workflow

* feat: delete unused codes

* feat: add redis append method

---------

Co-authored-by: heheer <zhiyu44@qq.com>

* cache per

* fix(test): init team sub when creating mocked user (#5646)

* fix: button is not vertically centered (#5647)

* doc

* fix: gridFs objectId (#5649)

---------

Co-authored-by: Zeng Qingwen <143274079+fishwww-ww@users.noreply.github.com>
Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: 伍闲犬 <whoeverimf5@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heheer <zhiyu44@qq.com>
2025-09-15 20:02:54 +08:00
Archer 0720bbe4da V4.9.7 feature (#4669)
* update doc

* feat: Add coupon redemption feature for team subscriptions (#4595)

* feat: Add coupon redemption feature for team subscriptions

- Introduced `TeamCouponSub` and `TeamCouponSchema` types
- Added `redeemCoupon` API endpoint
- Updated UI to include a modal for coupon redemption
- Added new icon and translations for "Redeem coupon"

* perf: remove field teamId

* perf: use dynamic import

* refactor: move to page component

* perf: coupon code

* perf: mcp server

* perf: test

* auto layout (#4634)

* fix 4.9.6 (#4631)

* fix debug quote list

* delete next text node match

* fix extract default boolean value

* export latest 100 chat items

* fix quote item ui

* doc

* fix doc

* feat: auto layout

* perf: auto layout

* fix: auto layout null

* add start node

---------

Co-authored-by: heheer <heheer@sealos.io>

* fix: share link (#4644)

* Add workflow run duration;Get audio duration (#4645)

* add duration

* get audio duration

* Custom config path (#4649)

* feat: 通过环境变量DATA_PATH获取配置文件目录 (#4622)

通过环境变量DATA_PATH获取配置文件目录,以应对不同的部署方式的多样化需求

* feat: custom configjson path

* doc

---------

Co-authored-by: John Chen <sss1991@163.com>

* 程序api调用场景下,如果大量调用带有图片或视频,产生的聊天记录会导致后台mongo数据库异常。这个修改给api客户端一个禁止生成聊天记录的选项,避免这个后果。 (#3964)

* update special chatId

* perf: vector db rename

* update operationLog (#4647)

* update operationLog

* combine operationLogMap

* solve operationI18nLogMap bug

* remoce log

* feat: Rerank usage (#4654)

* refresh concat when update (#4655)

* fix: refresh code

* perf: timer lock

* Fix operationLog (#4657)

* perf: http streamable mcp

* add alipay (#4630)

* perf: subplan ui

* perf: pay code

* hiden bank tip

* Fix: pay error (#4665)

* fix quote number (#4666)

* remove log

---------

Co-authored-by: a.e. <49438478+I-Info@users.noreply.github.com>
Co-authored-by: heheer <heheer@sealos.io>
Co-authored-by: John Chen <sss1991@163.com>
Co-authored-by: gaord <bengao168@msn.com>
Co-authored-by: gggaaallleee <91131304+gggaaallleee@users.noreply.github.com>
2025-04-26 16:17:21 +08:00
Finley Ge ab799e13cd test: concurrent test (#4548) 2025-04-16 12:05:38 +08:00
Archer 199f454b6b feat: team permission refine (#4494) (#4498)
* feat: team permission refine (#4402)

* chore: team permission extend

* feat: manage team permission

* chore: api auth

* fix: i18n

* feat: add initv493

* fix: test, org auth manager

* test: app test for refined permission

* update init sh

* fix: add/remove manage permission (#4427)

* fix: add/remove manage permission

* fix: github action fastgpt-test

* fix: mock create model

* fix: team write permission

* fix: ts

* account permission

---------

Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
2025-04-10 11:11:54 +08:00
Archer e75d81d05a V4.9.1 feature (#4206)
* fix: remove DefaultTeam (#4037)

* fix :Get application bound knowledge base information logical rewrite (#4057)

* fix :Get application bound knowledge base information logical rewrite

* fix :Get application bound knowledge base information logical rewrite

* fix :Get application bound knowledge base information logical rewrite

* fix :Get application bound knowledge base information logical rewrite

* update package

* fix: import dataset step error;perf: ai proxy avatar (#4074)

* perf: pg config params

* perf: ai proxy avatar

* fix: import dataset step error

* feat: data input ux

* perf: app dataset rewite

* fix: 文本提取不支持arrayString,arrayNumber等jsonSchema (#4079)

* update doc ;perf: model test (#4098)

* perf: extract array

* update doc

* perf: model test

* perf: model test

* perf: think tag parse (#4102)

* chat quote reader (#3912)

* init chat quote full text reader

* linked structure

* dataset data linked

* optimize code

* fix ts build

* test finish

* delete log

* fix

* fix ts

* fix ts

* remove nextId

* initial scroll

* fix

* fix

* perf: chunk read   (#4109)

* package

* perf: chunk read

* feat: api dataset support pdf parse;fix: chunk reader auth (#4117)

* feat: api dataset support pdf parse

* fix: chunk reader auth

* feat: invitation link (#3979)

* feat: invitation link schema and apis

* feat: add invitation link

* feat: member status: active, leave, forbidden

* fix: expires show hours and minutes

* feat: invalid invitation link hint

* fix: typo

* chore: fix typo & i18n

* fix

* pref: fe

* feat: add ttl index for 30-day-clean-up

* perf: invite member code (#4118)

* perf: invite member code

* fix: ts

* fix: model test channel id;fix: quote reader (#4123)

* fix: model test channel id

* fix: quote reader

* fix chat quote reader (#4125)

* perf: model test;perf: sidebar trigger (#4127)

* fix: import dataset step error;perf: ai proxy avatar (#4074)

* perf: pg config params

* perf: ai proxy avatar

* fix: import dataset step error

* feat: data input ux

* perf: app dataset rewite

* perf: model test

* perf: sidebar trigger

* lock

* update nanoid version

* fix: select component ux

* fix: ts

* fix: vitest

* remove test

* fix: prompt toolcall ui (#4139)

* load log error adapt

* fix: prompt toolcall ui

* perf: commercial function tip

* update package

* pref: copy link (#4147)

* fix(i18n): namespace (#4143)

* hiden dataset source (#4152)

* hiden dataset source

* perf: reader

* chore: move all tests into a single folder (#4160)

* fix modal close scroll (#4162)

* fix modal close scroll

* update refresh

* feat: rerank modal select and weight (#4164)

* fix loadInitData refresh (#4169)

* fix

* fix

* form input number default & api dataset max token

* feat: mix search weight (#4170)

* feat: mix search weight

* feat: svg render

* fix: avatar error remove (#4173)

* fix: avatar error remove

* fix: index

* fix: guide

* fix: auth

* update package;fix: input data model ui (#4181)

* update package

* fix: ts

* update config

* update jieba package

* add type sign

* fix: input data ui

* fix: page title refresh (#4186)

* fix: ts

* update jieba package

* fix: page title refresh

* fix: remove member length check when opening invite create modal (#4193)

* add env to check internal ip (#4187)

* fix: ts

* update jieba package

* add env to check internal ip

* package

* fix: jieba

* reset package

* update config

* fix: jieba package

* init shell

* init version

* change team reload

* update jieba package (#4200)

* update jieba package

* package

* update package

* remove invalid code

* action

* package (#4201)

* package

* update package

* remove invalid code

* package

* remove i18n tip (#4202)

* doc (#4205)

* fix: i18n (#4208)

* fix: next config (#4207)

* reset package

* i18n

* update config

* i18n

* remove log

---------

Co-authored-by: Finley Ge <32237950+FinleyGe@users.noreply.github.com>
Co-authored-by: gggaaallleee <91131304+gggaaallleee@users.noreply.github.com>
Co-authored-by: shilin <39396378+shilin66@users.noreply.github.com>
Co-authored-by: heheer <heheer@sealos.io>
2025-03-18 14:40:41 +08:00
Finley Ge bb30ca4859 chore: vitest support (#4026)
* chore: vitest

* chore: move test files

* chore: support vitest

* fix: exclude test files

* chore(ci): add test workflow

* feat: remove read env
2025-03-12 19:27:53 +08:00