mirror of
https://github.com/labring/FastGPT.git
synced 2025-10-13 22:56:28 +00:00
31c17999b8
- Adds a lightweight evaluation framework for app-level tracking and benchmarking. - Changes: 28 files, +1455 additions, -66 deletions. - Branch: add-evaluations -> main. - PR: https://github.com/chanzhi82020/FastGPT/pull/1 Applications built on FastGPT need repeatable, comparable benchmarks to measure regressions, track improvements, and validate releases. This initial implementation provides the primitives to define evaluation scenarios, run them against app endpoints or model components, and persist results for later analysis. I updated the PR description to emphasize that the evaluation system is targeted at FastGPT-built apps and expanded the explanation of the core pieces so reviewers understand the scope and intended use. The new description outlines the feature intent, core components, and how results are captured and aggregated for benchmarking. - Evaluation definitions - Define evaluation tasks that reference an app (app id, version, endpoint), test datasets or input cases, expected outputs (when applicable), and run configuration (parallelism, timeouts). - Support for custom metric plugins so teams can add domain-specific measures. - Runner / Executor - Executes evaluation cases against app endpoints or internal model interfaces. - Captures raw responses, response times, status codes, and any runtime errors. - Computes per-case metrics (e.g., correctness, latency) immediately after each case run. - Metrics & Aggregation - Built-in metrics: accuracy/success rate, latency (p50/p90/p99), throughput, error rate. - Aggregation produces per-run summaries and per-app historical summaries for trend analysis. - Allows combining metrics into composite scores for high-level benchmarking. - Persistence & Logging - Stores run results, input/output pairs (when needed), timestamps, environment info, and app/version metadata so runs are reproducible and auditable. - Logs are retained to facilitate debugging and root-cause analysis of regressions. - Reporting & Comparison - Produces aggregated reports suitable for CI gating, release notes, or dashboards. - Supports comparing multiple app versions or deployments side-by-side. - Extensibility & Integration - Designed to plug into CI (automated runs on PRs or releases), dashboards, and downstream analysis tools. - Easy to add new metrics, evaluators, or dataset connectors. By centering the evaluation system on FastGPT apps, teams can benchmark full application behavior (not only raw model outputs), correlate metrics with deployment configurations, and make informed release decisions. - Expand built-in metric suite (e.g., F1, BLEU/ROUGE where applicable), add dataset connectors, and provide example evaluation scenarios for sample apps. - Integrate with CI pipelines and add basic dashboarding for trend visualization. Related Issue: N/A Co-authored-by: Archer <545436317@qq.com>
FastGPT
FastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.
🎥 Comprehensive Feature Demonstration
https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409bd33f6d4
🛸 Online Use
Website: fastgpt.io
| Conversational AI Setup | Workflow Automation |
 |
 |
| Knowledge Base Setup | Integration Process |
 |
 |
💡 Features
| Features | Details |
|---|---|
| Application Orchestration Features | ✅ Offers a straightforward mode, eliminating the need for complex orchestration ✅ Provides clear next-step instructions in dialogues ✅ Facilitates workflow orchestration ✅ Tracks references in source files ✅ Encapsulates modules for enhanced reuse at multiple levels ✅ Combines search and reordering functions 🔜 Includes a tool module 🔜 Integrates Laf for online HTTP module creation 🔜 Plugin encapsulation capabilities |
| Knowledge Base Features | ✅ Allows for the mixed use of multiple databases ✅ Keeps track of modifications and deletions in data chunks ✅ Enables specific vector models for each knowledge base ✅ Stores original source files ✅ Supports direct input and segment-based QA import ✅ Compatible with a variety of file formats: pdf, docx, txt, html, md, csv ✅ Facilitates URL reading and bulk CSV importing 🔜 Supports PPT and Excel file import 🔜 Features a file reader 🔜 Offers diverse data preprocessing options |
| Application Debugging Features | ✅ Enables targeted search testing within the knowledge base ✅ Allows feedback, editing, and deletion during conversations ✅ Presents the full context of interactions ✅ Displays all intermediate values within modules 🔜 Advanced Debug mode for orchestration |
| OpenAPI Interface | ✅ The completions interface (aligned with GPT's chat mode interface) ✅ CRUD operations for the knowledge base 🔜 CRUD operations for conversation |
| Operational Features | ✅ Share without requiring login ✅ Easy embedding with Iframe ✅ Customizable chat window embedding with features like default open, drag-and-drop ✅ Centralizes conversation records for review and annotation |
👨💻 Development
Project tech stack: NextJs + TS + ChakraUI + MongoDB + PostgreSQL (PG Vector plug-in)/Milvus
-
⚡ Fast Deployment
When using Sealos services, there is no need to purchase servers or domain names. It supports high concurrency and dynamic scaling, and the database application uses the kubeblocks database, which far exceeds the simple Docker container deployment in terms of IO performance.
[](https://cloud.sealos.io/?openapp=system-fastdeploy%3FtemplateName%3Dfastgpt&uid=fnWRt09fZP)Give it a 2-4 minute wait after deployment as it sets up the database. Initially, it might be a too slow since we're using the basic settings.
💪 Related Projects
- Laf: 3-minute quick access to third-party applications
- Sealos: Rapid deployment of cluster applications
- One API: Multi-model management, supports Azure, Wenxin Yiyuan, etc.
- TuShan: Build a backend management system in 5 minutes
🤝 Third-party Ecosystem
🏘️ Community & Support
- 🌐 Visit the FastGPT website for full documentation and useful links.
- 💬 Join our Discord server is to chat with FastGPT developers and other FastGPT users. This is a good place to learn about FastGPT, ask questions, and share your experiences.
- 🐞 Create GitHub Issues for bug reports and feature requests.
👀 Others
- FastGPT FAQ
- Docker Deployment Tutorial Video
- Official Account Integration Video Tutorial
- FastGPT Knowledge Base Demo
🌱 Contributors
We welcome all forms of contributions. If you are interested in contributing code, you can check out our GitHub Issues to show us your ideas.
|
|
|
|---|---|
|
|
|
|
🌟 Star History
📄 Usage Agreement
This repository complies with the FastGPT Open Source License open source agreement.
- Direct commercial use as a backend service is allowed, but provision of SaaS services is not allowed.
- Without commercial authorization, any form of commercial service must retain relevant copyright information.
- For full details, please see FastGPT Open Source License
- Contact: Dennis@sealos.io , click to view commercial version pricing strategy