mirror of
https://github.com/labring/FastGPT.git
synced 2026-03-22 01:01:49 +08:00
6b61359516
* fix(sandbox): 重构代码沙盒，支持内置函数和网络请求 (#6462) * feat(sandbox): 重写代码沙盒 - Bun + Hono + 统一子进程模型 - 运行时: Node.js → Bun - HTTP 框架: NestJS + Fastify → Hono - JS 执行: isolated-vm → Bun 子进程（与 Python 统一） - 架构: 统一子进程模型，JS 和 Python 共享同一套执行引擎 - SubprocessRunner 基类，JS/Python 各自继承 - ProcessPool 进程池预热（SANDBOX_JS_POOL_SIZE / SANDBOX_PYTHON_POOL_SIZE） - SystemHelper 命名空间（JS 端保留向后兼容全局函数） - 临时文件系统隔离 + 路径遍历防护 + 磁盘配额 - 请求级资源限制（timeoutMs / memoryMB / diskMB） - JS: 原型链冻结 + Bun API 禁用 + Function 构造器注入 safe require（模块白名单） - Python: 宿主侧正则预检 + __import__ 拦截 + resource 限制 - 移除: @nestjs/*(6个包)、fastify、isolated-vm、node-gyp、reflect-metadata、rxjs - 新增: hono - 保留: tiktoken - 新增用户可用包: lodash、dayjs、axios、moment、uuid、crypto-js、qs - 67 个测试全部通过（单元测试 + 安全测试 + 集成测试） - vitest 独立配置，不影响全局 * fix(sandbox): 安全加固 - 扩展 Bun API 封锁、清理 process.env、闭包封装 Python import 拦截 - JS: 扩展 Bun 危险 API 封锁列表（serve/connect/listen/udpSocket/dns/plugin/build/Transpiler） - JS: 清理 process.env，仅保留沙箱必要变量，防止泄露敏感环境变量 - Python: 用闭包封装 _safe_import，del 掉 _original_import/_make_safe_import/_BLOCKED_MODULES 防止用户代码恢复原始 __import__ - Dockerfile: 复制 bun.lock 并使用 --frozen-lockfile 确保构建可复现 * fix(sandbox): 将 sandbox 从 pnpm workspace 中移除，独立管理依赖 * fix(sandbox): 从全局 vitest 移除 sandbox 测试，集成测试无 SANDBOX_URL 时跳过 * ci(sandbox): 添加独立测试 workflow，仅 sandbox 代码变更时触发 * refactor(sandbox): 使用 export default 启动方式，与 sandbox_server 保持一致 * fix: sandbox security hardening & comprehensive test suite Security fixes: - JS: freeze Function constructor to block constructor.constructor escape - JS: handle undefined return from main() (serialize as null) - Python: fix http_request using from-import after __import__ interception - Python: __import__ whitelist mode blocks exec/eval import bypasses New tests (223 passing): - security/escape-attacks: JS/Python escape attack vectors - security/network-security: IP blacklist, protocol restrictions, httpRequest - compat/legacy-js: 18 backward compatibility tests - compat/legacy-python: 21 backward compatibility tests - boundary: timeout, memory, disk, edge cases - examples: common user code patterns * feat(sandbox): env vars for all limits + rewrite README - Network limits configurable via env: SANDBOX_MAX_REQUESTS, SANDBOX_REQUEST_TIMEOUT, SANDBOX_MAX_RESPONSE_SIZE - Resource upper bounds configurable: SANDBOX_MAX_TIMEOUT, SANDBOX_MAX_MEMORY_MB, SANDBOX_MAX_DISK_MB - README: architecture, API docs, env var reference, how to add JS/Python packages, security overview, built-in functions * refactor(sandbox): extract env.ts with dotenv for typed env loading - New env.ts: dotenv.config() + typed helpers (str/int/bool) - config.ts re-exports env for backward compatibility - index.ts imports env first to ensure .env loaded before anything else * refactor(sandbox): use zod for env validation and type coercion - Replace manual parseInt/str helpers with zod schema + coerce - Invalid env vars now fail fast with formatted error on startup - dotenv + zod, clean and declarative * chore(sandbox): remove unused process pool code - Delete pool.ts and pool.test.ts (pool was never wired into runners) - Remove PoolConfig/PooledProcess types - Remove pool env vars from env.ts - Clean up README * feat(sandbox): add concurrency limiter with semaphore - New Semaphore utility for max concurrent subprocess control - SANDBOX_MAX_CONCURRENCY env var (default 50) - Excess requests queue instead of spawning unbounded processes - Health endpoint exposes concurrency stats (current/queued/max) * test(sandbox): add semaphore tests and expand coverage to 292 cases - New semaphore.test.ts (11 tests): acquire/release, queuing, FIFO, stats, serial execution - JS runner: blank code, template literals, primitive returns, more modules, unicode, partial limits - Python runner: blank code, triple quotes, primitive returns, unicode, null vars, division errors - JS security: process.exit, globalThis, Symbol.unscopables, Proxy, dynamic import, path traversal - Python security: pickle/multiprocessing/threading/ctypes/signal, exec bypass, __subclasses__ - Escape attacks: type() class creation, __builtins__ tampering, getattr access - Boundary: long vars, special JSON chars, float precision, big ints, circular refs, Promise.reject * test(sandbox): test-master review - add 31 tests, coverage report - base-runner.test.ts (10): BaseRunner precheck, temp dir, semaphore integration - semaphore-race.test.ts (5): race conditions, rapid acquire/release, stress test - coverage-gaps.test.ts (16): security coverage gaps found during review - REVIEW-REPORT.md: full test audit report Total: 323 passed, 0 failed * fix(sandbox): address PR #6439 review issues Security fixes: - Intercept Python builtins.open(), restrict file access to sandbox tmpdir - Remove unused pool.ts, warmup.mjs, warmup.py (security risk) - Fix DNS rebinding TOCTOU: use resolved IP for HTTP connections - Fix symlink path traversal: use realpath instead of normpath - Add try/finally cleanup for __import__ hook Robustness: - Add __SANDBOX_RESULT__ prefix to stdout parsing, prevent user output interference - Fix disk quota tracking: deduct old file size on overwrite - Add __import__() pattern scanning in Python precheck Tests: - Fix eval+__import__ test assertion (accept both catch and fail paths) All 323 tests passing. * fix(sandbox): remove warmup scripts COPY from Dockerfile * docs(sandbox): add technical design document * feat(sandbox): configurable module allowlist/blocklist via env vars - SANDBOX_JS_ALLOWED_MODULES: JS require whitelist (comma-separated) - SANDBOX_PYTHON_BLOCKED_MODULES: Python import blacklist (comma-separated) - Defaults unchanged, fully backward compatible * fix(sandbox): 修复多个安全漏洞 1. Python HTTPS DNS rebinding: HTTPS 请求现在也使用 resolved IP 发起连接 2. Python __import__ hook 恢复漏洞: 移除 finally 块中恢复原始 __import__ 的代码 3. Python 内部变量泄露: 用户代码执行前删除 _os, _socket 等内部模块引用 4. JS process 危险 API: 禁用 process.binding/dlopen/kill/chdir 等，冻结 process.env 5. Python open() fd 绕过: 阻止通过整数文件描述符绕过路径检查 6. API 输入校验: 使用 zod schema 校验请求体，限制代码大小 1MB 7. 无认证警告: SANDBOX_TOKEN 未设置时输出生产环境警告 新增 security-fixes.test.ts 包含所有修复的回归测试 * test: consolidate security tests + add integration test suite - Merge 6 security test files into 1 consolidated security.test.ts (109 tests) - JS/Python module interception (precheck + runtime) - JS escape attacks (prototype, constructor, Reflect, globalThis) - Python escape attacks (__import__ hook, exec/eval, internal vars, __subclasses__) - SSRF protection (private IPs, cloud metadata, file protocol) - File system isolation (path traversal, fd, disk quota) - Variable injection attacks - API input validation - Add black-box integration test suite functional.test.ts (56 tests) - Basic operations (math, string, array, JSON, regex, Date, Promise, Map/Set) - Variable passing (string, number, complex objects, empty, multiple) - Whitelisted modules (crypto-js, moment, lodash) - SystemHelper/system_helper (fs, delay, strToBase64, httpRequest) - Error handling (syntax, runtime, undefined var, timeout) - Network requests (GET, POST) - Complex scenarios (CSV pipeline, recursion, class definition) - Remove 34 duplicate test cases across merged files - Total: 363 passed, 8 skipped (integration API tests need server) * fix(sandbox): z.record() zod v4 compatibility - add key type param * feat(sandbox): add .env.template with all config options and comments * refactor(sandbox): remove disk write support and temp filesystem * test(sandbox): remove all fs-related tests and add test case inventory - Remove fs read/write tests from unit, integration, boundary, examples - Remove path traversal, absolute path, open fd, builtins.open tests from security - Add comprehensive test/case.md with all 344 test cases categorized - All tests pass: 344 passed, 8 skipped, 0 failed * feat(sandbox): add GET /sandbox/modules API to list available packages and builtins * test(sandbox): add unit tests for GET /sandbox/modules API * refactor(test): rewrite api.test.ts to use app.request() - no external server needed * feat(sandbox): validate SANDBOX_TOKEN charset in env schema (ASCII printable only) * chore(sandbox): remove DESIGN.md and package-lock.json from PR * feat(sandbox): replace spawn-per-request with process pool architecture - Add ProcessPool (JS) and PythonProcessPool with long-lived worker processes - Workers communicate via stdin/stdout line-based JSON protocol - Pool size configurable via SANDBOX_POOL_SIZE env var (default 20) - Auto-respawn workers on crash - Semaphore-based queueing when requests exceed pool size Performance gains (simple functions): - JS: 22 QPS → 1,328 QPS (60x improvement) - Python: 14.7 QPS → 3,395 QPS (231x improvement) - Fix import.meta.dir compatibility for vitest (Node) environments - Export poolReady promise for test initialization - Add benchmark scripts to test/benchmark/ - All 354 tests passing (12 test files) * chore(sandbox): clean up unused files, update README with pool architecture - Remove test/REVIEW-REPORT.md, test/case.md, test/benchmark.ts (obsolete) - Rewrite README: pool architecture diagram, performance benchmarks, SANDBOX_POOL_SIZE config, project structure, health endpoint format * fix(sandbox): 修复进程池超时后 worker respawn 竞态条件 根因：超时 kill worker 后，exit 事件是异步的，release() 先执行时 worker 还在列表里，死 worker 被放回 idle 池，后续请求发给死进程。 修复： - 超时回调中先 removeWorker 再 kill，防止 release 归还死 worker - removeWorker 返回 bool，exit 事件中避免重复 respawn - 超时回调主动触发 spawnWorker 补充池 - release 检查 worker 是否仍在池中 - spawnWorker 完成时检查 waitQueue 直接分配 * fix: security hardening & test migration to process pool - JS worker: harden process object (kill/chdir/env freeze/binding/dlopen) - Python worker: stack-frame based __import__ hook to block exec/eval bypass - Python worker: BuiltinsProxy to prevent __import__ override via builtins module - Python worker: restricted __builtins__ dict in exec_globals (no internal refs) - Python worker: restore __import__ before each execution - Migrate all 9 test files from JsRunner/PythonRunner to ProcessPool/PythonProcessPool - Configure vitest for serial execution (pool size=1, fileParallelism: false) - Fix security test assertion for builtins tampering (success=true with escaped=false) - All 102 security tests passing * docs(sandbox): update README with accurate benchmark data, remove non-existent features - Update performance table with latest benchmark results (JS 1414 QPS, Python 4247 QPS) - Remove SANDBOX_DISK_MB/SANDBOX_MAX_DISK_MB env vars (not implemented) - Remove SystemHelper.fs.* / system_helper.fs.* docs (not implemented in workers) - Fix security section to match actual implementation - Update test count to 351 * refactor(sandbox): remove legacy runner/sandbox/template code - Delete src/runner/ (base.ts, js-runner.ts, python-runner.ts) - Delete src/sandbox/ (js-template.ts, python-template.ts, network-config.ts) - Delete test/unit/js-runner.test.ts, test/unit/python-runner.test.ts - Keep src/utils/semaphore.ts (generic utility, has its own tests) - Update README project structure and test count (297 cases) All functionality is now in src/pool/ (process-pool architecture). 297 tests passing, 0 failures. * test(sandbox): add process pool lifecycle/respawn/concurrency tests - ProcessPool: init/shutdown/stats, worker crash respawn, timeout respawn, pool-full queuing, concurrent crash isolation - PythonProcessPool: init/shutdown/stats, timeout respawn, queuing - 14 new test cases, total 311 passing * fix(sandbox): ping/pong health check, replace httpbin.org with baidu.com - Worker health check: send actual ping message and verify pong response instead of only checking stdin.writable (detects stuck workers) - JS worker.ts: handle {type:'ping'} → reply {type:'pong'} - Python worker.py: handle {type:'ping'} → reply {type:'pong'} - ProcessPool/PythonProcessPool: rewrite pingWorker to send ping, wait for pong with timeout, replace worker on failure - Replace all httpbin.org URLs with www.baidu.com in tests (httpbin.org unreachable from China/Sealos Devbox) - Add 4 new health check tests (ping/pong for JS and Python pools) - All 318 tests passing, 0 failures * docs: add test report (test/README.md) and update README testing section - test/README.md: detailed report with 315 passed / 3 skipped / 0 failed - README.md: updated test section with coverage dimensions table and link to report * docs: add functional test cases checklist (110 cases) * fix(sandbox): fix Dockerfile Python env and import detection 1. Dockerfile: Remove broken multi-stage Python 3.11 copy. - The previous approach copied python3 binary from python:3.11-alpine but missed libpython3.11.so.1.0, causing Python pool init failure. - Now uses system Python from apk and installs pip packages directly. 2. worker.py: Fix false positive import blocking for third-party packages. - numpy/pandas were blocked because their internal 'import os' was detected as user-initiated (full stack scan found user code frames). - Changed to check only the direct caller frame: if the import comes from site-packages (third-party lib internals), allow it. - Direct user imports of blocked modules are still properly rejected. * fix(sandbox): block dynamic import() and restrict file system access Security fixes found during deep review: 1. JS: Block import() dynamic imports that bypass require whitelist. - import('fs') could read arbitrary files on the container. - Added static regex check to reject code containing import(). 2. Python: Restrict open() to prevent user code from reading files. - open('/etc/passwd') was accessible from user code. - Added _restricted_open() that checks caller frame: only allows stdlib/site-packages internal calls, blocks user code (<string>). 3. Python: Remove duplicate return statement in _safe_import. All 315 tests pass (3 skipped). * test(sandbox): add regression tests for import() and open() security fixes - JS: import('fs'), import('child_process'), import('os') blocked - JS: string containing 'import' not false-positive - Python: open('/etc/passwd'), open('/proc/self/environ'), open('/tmp/evil.txt', 'w') blocked - Python: numpy internal open() not affected (conditional on numpy availability) Total: 322 passed | 3 skipped (was 315 passed) * docs(sandbox): rewrite sandbox documentation with JS + Python coverage - Add Python language support documentation - Add httpRequest/http_request function docs - Add available modules list (JS whitelist + Python safe modules) - Add security restrictions section - Add practical examples (data processing, date calc, webhook signing) - Add JS/Python function name mapping table * docs(sandbox): use SystemHelper/system_helper for built-in functions Direct calls (countToken, delay, etc.) are deprecated (kept for compat). All examples now use SystemHelper.xxx() / system_helper.xxx(). * docs(sandbox): Python only show named-params style as recommended * feat(sandbox): unify Python SystemHelper API with camelCase aliases - Add camelCase aliases to Python SystemHelper: countToken, strToBase64, createHmac, httpRequest (matching JS API exactly) - Update docs to use SystemHelper uniformly for both JS and Python - snake_case methods (count_token, etc.) still work for backward compat * feat(sandbox): add matplotlib and increase HTTP timeout to 60s - Add matplotlib to Python dependencies - Increase HTTP request timeout from 10s to 60s (both JS and Python) - Update docs accordingly * docs(sandbox): split docs for old/new sandbox versions - sandbox.mdx → '代码运行（旧版）' for FastGPT ≤ 4.14.7 (URL unchanged) - sandbox-v5.mdx → '代码运行' for FastGPT ≥ 4.14.8 - Both pages cross-link to each other - meta.json updated: sandbox-v5 listed before sandbox * docs: rename old sandbox doc to 代码运行（弃） * refactor(sandbox): remove SANDBOX_TIMEOUT, use SANDBOX_MAX_TIMEOUT as unified timeout * fix(sandbox): add build dependencies for matplotlib in Dockerfile * refactor(sandbox): migrate Python from blocklist to allowlist for module control - Change SANDBOX_PYTHON_BLOCKED_MODULES to SANDBOX_PYTHON_ALLOWED_MODULES - Update Python worker to use allowlist instead of blocklist - Add comprehensive safe module list: math, json, datetime, numpy, pandas, etc. - Improve error message: 'Module X is not in the allowlist' - Consistent with JS allowlist approach for better security * fix(sandbox): add _strptime to allowlist and update test assertions - Add _strptime module (required by datetime.strptime) - Update test assertions for Python module import errors - All 325 tests now pass (322 passed, 3 skipped) * fix(docs): center SVG icon in size-5 container on medium screens * docs(sandbox): simplify built-in functions and improve module documentation - Remove delay, countToken, strToBase64, createHmac functions (keep only httpRequest) - Convert Python module list to table format (10 tables by category) - Reorganize usage examples with collapsible sections (JS and Python) - Fix icon alignment in desktop/mobile sidebar navigation - All 325 tests passing --------- Co-authored-by: Lobster 3 <lobster3@sandbox.dev> Co-authored-by: OpenClaw Bot <bot@openclaw.ai> Co-authored-by: Archer <c121914yu@gmail.com> Co-authored-by: archer <archer@archerdeMac-mini.local> * perf: code sandbox * update action * Update projects/app/src/components/core/chat/ChatContainer/ChatBox/index.tsx Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update timeout * update memory limit function * sandbox * perf: process poll * env template * feat: code tip * fix: code sandbox error tip * update memory limit fn * update memory limit fn * fix: test * fix: test * fix: sandbox --------- Co-authored-by: Archer <archer@fastgpt.io> Co-authored-by: Lobster 3 <lobster3@sandbox.dev> Co-authored-by: OpenClaw Bot <bot@openclaw.ai> Co-authored-by: Archer <c121914yu@gmail.com> Co-authored-by: archer <archer@archerdeMac-mini.local> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>