 Archer
|
6b61359516
|
feat(sandbox): 重构代码沙盒，支持内置函数和网络请求 (#6479)
* fix(sandbox): 重构代码沙盒，支持内置函数和网络请求 (#6462)
* feat(sandbox): 重写代码沙盒 - Bun + Hono + 统一子进程模型
- 运行时: Node.js → Bun
- HTTP 框架: NestJS + Fastify → Hono
- JS 执行: isolated-vm → Bun 子进程（与 Python 统一）
- 架构: 统一子进程模型，JS 和 Python 共享同一套执行引擎
- SubprocessRunner 基类，JS/Python 各自继承
- ProcessPool 进程池预热（SANDBOX_JS_POOL_SIZE / SANDBOX_PYTHON_POOL_SIZE）
- SystemHelper 命名空间（JS 端保留向后兼容全局函数）
- 临时文件系统隔离 + 路径遍历防护 + 磁盘配额
- 请求级资源限制（timeoutMs / memoryMB / diskMB）
- JS: 原型链冻结 + Bun API 禁用 + Function 构造器注入 safe require（模块白名单）
- Python: 宿主侧正则预检 + __import__ 拦截 + resource 限制
- 移除: @nestjs/*(6个包)、fastify、isolated-vm、node-gyp、reflect-metadata、rxjs
- 新增: hono
- 保留: tiktoken
- 新增用户可用包: lodash、dayjs、axios、moment、uuid、crypto-js、qs
- 67 个测试全部通过（单元测试 + 安全测试 + 集成测试）
- vitest 独立配置，不影响全局
* fix(sandbox): 安全加固 - 扩展 Bun API 封锁、清理 process.env、闭包封装 Python import 拦截
- JS: 扩展 Bun 危险 API 封锁列表（serve/connect/listen/udpSocket/dns/plugin/build/Transpiler）
- JS: 清理 process.env，仅保留沙箱必要变量，防止泄露敏感环境变量
- Python: 用闭包封装 _safe_import，del 掉 _original_import/_make_safe_import/_BLOCKED_MODULES
防止用户代码恢复原始 __import__
- Dockerfile: 复制 bun.lock 并使用 --frozen-lockfile 确保构建可复现
* fix(sandbox): 将 sandbox 从 pnpm workspace 中移除，独立管理依赖
* fix(sandbox): 从全局 vitest 移除 sandbox 测试，集成测试无 SANDBOX_URL 时跳过
* ci(sandbox): 添加独立测试 workflow，仅 sandbox 代码变更时触发
* refactor(sandbox): 使用 export default 启动方式，与 sandbox_server 保持一致
* fix: sandbox security hardening & comprehensive test suite
Security fixes:
- JS: freeze Function constructor to block constructor.constructor escape
- JS: handle undefined return from main() (serialize as null)
- Python: fix http_request using from-import after __import__ interception
- Python: __import__ whitelist mode blocks exec/eval import bypasses
New tests (223 passing):
- security/escape-attacks: JS/Python escape attack vectors
- security/network-security: IP blacklist, protocol restrictions, httpRequest
- compat/legacy-js: 18 backward compatibility tests
- compat/legacy-python: 21 backward compatibility tests
- boundary: timeout, memory, disk, edge cases
- examples: common user code patterns
* feat(sandbox): env vars for all limits + rewrite README
- Network limits configurable via env: SANDBOX_MAX_REQUESTS, SANDBOX_REQUEST_TIMEOUT, SANDBOX_MAX_RESPONSE_SIZE
- Resource upper bounds configurable: SANDBOX_MAX_TIMEOUT, SANDBOX_MAX_MEMORY_MB, SANDBOX_MAX_DISK_MB
- README: architecture, API docs, env var reference, how to add JS/Python packages, security overview, built-in functions
* refactor(sandbox): extract env.ts with dotenv for typed env loading
- New env.ts: dotenv.config() + typed helpers (str/int/bool)
- config.ts re-exports env for backward compatibility
- index.ts imports env first to ensure .env loaded before anything else
* refactor(sandbox): use zod for env validation and type coercion
- Replace manual parseInt/str helpers with zod schema + coerce
- Invalid env vars now fail fast with formatted error on startup
- dotenv + zod, clean and declarative
* chore(sandbox): remove unused process pool code
- Delete pool.ts and pool.test.ts (pool was never wired into runners)
- Remove PoolConfig/PooledProcess types
- Remove pool env vars from env.ts
- Clean up README
* feat(sandbox): add concurrency limiter with semaphore
- New Semaphore utility for max concurrent subprocess control
- SANDBOX_MAX_CONCURRENCY env var (default 50)
- Excess requests queue instead of spawning unbounded processes
- Health endpoint exposes concurrency stats (current/queued/max)
* test(sandbox): add semaphore tests and expand coverage to 292 cases
- New semaphore.test.ts (11 tests): acquire/release, queuing, FIFO, stats, serial execution
- JS runner: blank code, template literals, primitive returns, more modules, unicode, partial limits
- Python runner: blank code, triple quotes, primitive returns, unicode, null vars, division errors
- JS security: process.exit, globalThis, Symbol.unscopables, Proxy, dynamic import, path traversal
- Python security: pickle/multiprocessing/threading/ctypes/signal, exec bypass, __subclasses__
- Escape attacks: type() class creation, __builtins__ tampering, getattr access
- Boundary: long vars, special JSON chars, float precision, big ints, circular refs, Promise.reject
* test(sandbox): test-master review - add 31 tests, coverage report
- base-runner.test.ts (10): BaseRunner precheck, temp dir, semaphore integration
- semaphore-race.test.ts (5): race conditions, rapid acquire/release, stress test
- coverage-gaps.test.ts (16): security coverage gaps found during review
- REVIEW-REPORT.md: full test audit report
Total: 323 passed, 0 failed
* fix(sandbox): address PR #6439 review issues
Security fixes:
- Intercept Python builtins.open(), restrict file access to sandbox tmpdir
- Remove unused pool.ts, warmup.mjs, warmup.py (security risk)
- Fix DNS rebinding TOCTOU: use resolved IP for HTTP connections
- Fix symlink path traversal: use realpath instead of normpath
- Add try/finally cleanup for __import__ hook
Robustness:
- Add __SANDBOX_RESULT__ prefix to stdout parsing, prevent user output interference
- Fix disk quota tracking: deduct old file size on overwrite
- Add __import__() pattern scanning in Python precheck
Tests:
- Fix eval+__import__ test assertion (accept both catch and fail paths)
All 323 tests passing.
* fix(sandbox): remove warmup scripts COPY from Dockerfile
* docs(sandbox): add technical design document
* feat(sandbox): configurable module allowlist/blocklist via env vars
- SANDBOX_JS_ALLOWED_MODULES: JS require whitelist (comma-separated)
- SANDBOX_PYTHON_BLOCKED_MODULES: Python import blacklist (comma-separated)
- Defaults unchanged, fully backward compatible
* fix(sandbox): 修复多个安全漏洞
1. Python HTTPS DNS rebinding: HTTPS 请求现在也使用 resolved IP 发起连接
2. Python __import__ hook 恢复漏洞: 移除 finally 块中恢复原始 __import__ 的代码
3. Python 内部变量泄露: 用户代码执行前删除 _os, _socket 等内部模块引用
4. JS process 危险 API: 禁用 process.binding/dlopen/kill/chdir 等，冻结 process.env
5. Python open() fd 绕过: 阻止通过整数文件描述符绕过路径检查
6. API 输入校验: 使用 zod schema 校验请求体，限制代码大小 1MB
7. 无认证警告: SANDBOX_TOKEN 未设置时输出生产环境警告
新增 security-fixes.test.ts 包含所有修复的回归测试
* test: consolidate security tests + add integration test suite
- Merge 6 security test files into 1 consolidated security.test.ts (109 tests)
- JS/Python module interception (precheck + runtime)
- JS escape attacks (prototype, constructor, Reflect, globalThis)
- Python escape attacks (__import__ hook, exec/eval, internal vars, __subclasses__)
- SSRF protection (private IPs, cloud metadata, file protocol)
- File system isolation (path traversal, fd, disk quota)
- Variable injection attacks
- API input validation
- Add black-box integration test suite functional.test.ts (56 tests)
- Basic operations (math, string, array, JSON, regex, Date, Promise, Map/Set)
- Variable passing (string, number, complex objects, empty, multiple)
- Whitelisted modules (crypto-js, moment, lodash)
- SystemHelper/system_helper (fs, delay, strToBase64, httpRequest)
- Error handling (syntax, runtime, undefined var, timeout)
- Network requests (GET, POST)
- Complex scenarios (CSV pipeline, recursion, class definition)
- Remove 34 duplicate test cases across merged files
- Total: 363 passed, 8 skipped (integration API tests need server)
* fix(sandbox): z.record() zod v4 compatibility - add key type param
* feat(sandbox): add .env.template with all config options and comments
* refactor(sandbox): remove disk write support and temp filesystem
* test(sandbox): remove all fs-related tests and add test case inventory
- Remove fs read/write tests from unit, integration, boundary, examples
- Remove path traversal, absolute path, open fd, builtins.open tests from security
- Add comprehensive test/case.md with all 344 test cases categorized
- All tests pass: 344 passed, 8 skipped, 0 failed
* feat(sandbox): add GET /sandbox/modules API to list available packages and builtins
* test(sandbox): add unit tests for GET /sandbox/modules API
* refactor(test): rewrite api.test.ts to use app.request() - no external server needed
* feat(sandbox): validate SANDBOX_TOKEN charset in env schema (ASCII printable only)
* chore(sandbox): remove DESIGN.md and package-lock.json from PR
* feat(sandbox): replace spawn-per-request with process pool architecture
- Add ProcessPool (JS) and PythonProcessPool with long-lived worker processes
- Workers communicate via stdin/stdout line-based JSON protocol
- Pool size configurable via SANDBOX_POOL_SIZE env var (default 20)
- Auto-respawn workers on crash
- Semaphore-based queueing when requests exceed pool size
Performance gains (simple functions):
- JS: 22 QPS → 1,328 QPS (60x improvement)
- Python: 14.7 QPS → 3,395 QPS (231x improvement)
- Fix import.meta.dir compatibility for vitest (Node) environments
- Export poolReady promise for test initialization
- Add benchmark scripts to test/benchmark/
- All 354 tests passing (12 test files)
* chore(sandbox): clean up unused files, update README with pool architecture
- Remove test/REVIEW-REPORT.md, test/case.md, test/benchmark.ts (obsolete)
- Rewrite README: pool architecture diagram, performance benchmarks,
SANDBOX_POOL_SIZE config, project structure, health endpoint format
* fix(sandbox): 修复进程池超时后 worker respawn 竞态条件
根因：超时 kill worker 后，exit 事件是异步的，release() 先执行时
worker 还在列表里，死 worker 被放回 idle 池，后续请求发给死进程。
修复：
- 超时回调中先 removeWorker 再 kill，防止 release 归还死 worker
- removeWorker 返回 bool，exit 事件中避免重复 respawn
- 超时回调主动触发 spawnWorker 补充池
- release 检查 worker 是否仍在池中
- spawnWorker 完成时检查 waitQueue 直接分配
* fix: security hardening & test migration to process pool
- JS worker: harden process object (kill/chdir/env freeze/binding/dlopen)
- Python worker: stack-frame based __import__ hook to block exec/eval bypass
- Python worker: BuiltinsProxy to prevent __import__ override via builtins module
- Python worker: restricted __builtins__ dict in exec_globals (no internal refs)
- Python worker: restore __import__ before each execution
- Migrate all 9 test files from JsRunner/PythonRunner to ProcessPool/PythonProcessPool
- Configure vitest for serial execution (pool size=1, fileParallelism: false)
- Fix security test assertion for builtins tampering (success=true with escaped=false)
- All 102 security tests passing
* docs(sandbox): update README with accurate benchmark data, remove non-existent features
- Update performance table with latest benchmark results (JS 1414 QPS, Python 4247 QPS)
- Remove SANDBOX_DISK_MB/SANDBOX_MAX_DISK_MB env vars (not implemented)
- Remove SystemHelper.fs.* / system_helper.fs.* docs (not implemented in workers)
- Fix security section to match actual implementation
- Update test count to 351
* refactor(sandbox): remove legacy runner/sandbox/template code
- Delete src/runner/ (base.ts, js-runner.ts, python-runner.ts)
- Delete src/sandbox/ (js-template.ts, python-template.ts, network-config.ts)
- Delete test/unit/js-runner.test.ts, test/unit/python-runner.test.ts
- Keep src/utils/semaphore.ts (generic utility, has its own tests)
- Update README project structure and test count (297 cases)
All functionality is now in src/pool/ (process-pool architecture).
297 tests passing, 0 failures.
* test(sandbox): add process pool lifecycle/respawn/concurrency tests
- ProcessPool: init/shutdown/stats, worker crash respawn, timeout respawn,
pool-full queuing, concurrent crash isolation
- PythonProcessPool: init/shutdown/stats, timeout respawn, queuing
- 14 new test cases, total 311 passing
* fix(sandbox): ping/pong health check, replace httpbin.org with baidu.com
- Worker health check: send actual ping message and verify pong response
instead of only checking stdin.writable (detects stuck workers)
- JS worker.ts: handle {type:'ping'} → reply {type:'pong'}
- Python worker.py: handle {type:'ping'} → reply {type:'pong'}
- ProcessPool/PythonProcessPool: rewrite pingWorker to send ping,
wait for pong with timeout, replace worker on failure
- Replace all httpbin.org URLs with www.baidu.com in tests
(httpbin.org unreachable from China/Sealos Devbox)
- Add 4 new health check tests (ping/pong for JS and Python pools)
- All 318 tests passing, 0 failures
* docs: add test report (test/README.md) and update README testing section
- test/README.md: detailed report with 315 passed / 3 skipped / 0 failed
- README.md: updated test section with coverage dimensions table and link to report
* docs: add functional test cases checklist (110 cases)
* fix(sandbox): fix Dockerfile Python env and import detection
1. Dockerfile: Remove broken multi-stage Python 3.11 copy.
- The previous approach copied python3 binary from python:3.11-alpine
but missed libpython3.11.so.1.0, causing Python pool init failure.
- Now uses system Python from apk and installs pip packages directly.
2. worker.py: Fix false positive import blocking for third-party packages.
- numpy/pandas were blocked because their internal 'import os' was
detected as user-initiated (full stack scan found user code frames).
- Changed to check only the direct caller frame: if the import comes
from site-packages (third-party lib internals), allow it.
- Direct user imports of blocked modules are still properly rejected.
* fix(sandbox): block dynamic import() and restrict file system access
Security fixes found during deep review:
1. JS: Block import() dynamic imports that bypass require whitelist.
- import('fs') could read arbitrary files on the container.
- Added static regex check to reject code containing import().
2. Python: Restrict open() to prevent user code from reading files.
- open('/etc/passwd') was accessible from user code.
- Added _restricted_open() that checks caller frame: only allows
stdlib/site-packages internal calls, blocks user code (<string>).
3. Python: Remove duplicate return statement in _safe_import.
All 315 tests pass (3 skipped).
* test(sandbox): add regression tests for import() and open() security fixes
- JS: import('fs'), import('child_process'), import('os') blocked
- JS: string containing 'import' not false-positive
- Python: open('/etc/passwd'), open('/proc/self/environ'), open('/tmp/evil.txt', 'w') blocked
- Python: numpy internal open() not affected (conditional on numpy availability)
Total: 322 passed | 3 skipped (was 315 passed)
* docs(sandbox): rewrite sandbox documentation with JS + Python coverage
- Add Python language support documentation
- Add httpRequest/http_request function docs
- Add available modules list (JS whitelist + Python safe modules)
- Add security restrictions section
- Add practical examples (data processing, date calc, webhook signing)
- Add JS/Python function name mapping table
* docs(sandbox): use SystemHelper/system_helper for built-in functions
Direct calls (countToken, delay, etc.) are deprecated (kept for compat).
All examples now use SystemHelper.xxx() / system_helper.xxx().
* docs(sandbox): Python only show named-params style as recommended
* feat(sandbox): unify Python SystemHelper API with camelCase aliases
- Add camelCase aliases to Python SystemHelper: countToken, strToBase64,
createHmac, httpRequest (matching JS API exactly)
- Update docs to use SystemHelper uniformly for both JS and Python
- snake_case methods (count_token, etc.) still work for backward compat
* feat(sandbox): add matplotlib and increase HTTP timeout to 60s
- Add matplotlib to Python dependencies
- Increase HTTP request timeout from 10s to 60s (both JS and Python)
- Update docs accordingly
* docs(sandbox): split docs for old/new sandbox versions
- sandbox.mdx → '代码运行（旧版）' for FastGPT ≤ 4.14.7 (URL unchanged)
- sandbox-v5.mdx → '代码运行' for FastGPT ≥ 4.14.8
- Both pages cross-link to each other
- meta.json updated: sandbox-v5 listed before sandbox
* docs: rename old sandbox doc to 代码运行（弃）
* refactor(sandbox): remove SANDBOX_TIMEOUT, use SANDBOX_MAX_TIMEOUT as unified timeout
* fix(sandbox): add build dependencies for matplotlib in Dockerfile
* refactor(sandbox): migrate Python from blocklist to allowlist for module control
- Change SANDBOX_PYTHON_BLOCKED_MODULES to SANDBOX_PYTHON_ALLOWED_MODULES
- Update Python worker to use allowlist instead of blocklist
- Add comprehensive safe module list: math, json, datetime, numpy, pandas, etc.
- Improve error message: 'Module X is not in the allowlist'
- Consistent with JS allowlist approach for better security
* fix(sandbox): add _strptime to allowlist and update test assertions
- Add _strptime module (required by datetime.strptime)
- Update test assertions for Python module import errors
- All 325 tests now pass (322 passed, 3 skipped)
* fix(docs): center SVG icon in size-5 container on medium screens
* docs(sandbox): simplify built-in functions and improve module documentation
- Remove delay, countToken, strToBase64, createHmac functions (keep only httpRequest)
- Convert Python module list to table format (10 tables by category)
- Reorganize usage examples with collapsible sections (JS and Python)
- Fix icon alignment in desktop/mobile sidebar navigation
- All 325 tests passing
---------
Co-authored-by: Lobster 3 <lobster3@sandbox.dev>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>
Co-authored-by: Archer <c121914yu@gmail.com>
Co-authored-by: archer <archer@archerdeMac-mini.local>
* perf: code sandbox
* update action
* Update projects/app/src/components/core/chat/ChatContainer/ChatBox/index.tsx
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* update timeout
* update memory limit function
* sandbox
* perf: process poll
* env template
* feat: code tip
* fix: code sandbox error tip
* update memory limit fn
* update memory limit fn
* fix: test
* fix: test
* fix: sandbox
---------
Co-authored-by: Archer <archer@fastgpt.io>
Co-authored-by: Lobster 3 <lobster3@sandbox.dev>
Co-authored-by: OpenClaw Bot <bot@openclaw.ai>
Co-authored-by: Archer <c121914yu@gmail.com>
Co-authored-by: archer <archer@archerdeMac-mini.local>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
2026-02-28 12:36:59 +08:00 |
|