mirror of
https://github.com/labring/FastGPT.git
synced 2026-03-19 01:00:34 +08:00
747af5b517
* feat: add vector database migration solution between different vector DBs This PR implements a migration solution for transferring vector data between different vector databases (PostgreSQL, OceanBase, Milvus) as requested in #6196. Features: - Support offline (stop-the-world) migration mode - Export vectors from source DB with pagination - Import vectors to target DB with batch processing - Preserve original vector IDs when possible - Validate migration results by comparing counts - CLI script for command-line migration - Admin API endpoint for programmatic migration Files added: - packages/service/common/vectorDB/migration/ - Core migration logic - projects/app/src/pages/api/admin/migrateVector.ts - Admin API - scripts/migrateVector.ts - CLI migration script - .claude/design/vector-db-migration.md - Design document Closes #6196 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address PR review security and stability issues - Fix SQL injection vulnerabilities in PgExporter and OceanBaseExporter by using parameterized queries instead of string concatenation - Add memory leak prevention with automatic cleanup of old migration states (24h retention, hourly cleanup) - Add concurrent migration limit (max 3 simultaneous migrations) - Add transaction protection for PostgreSQL and OceanBase batch imports - Improve error handling for connection cleanup phase - Fix SQL injection in deleteBatch methods Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
FastGPT
FastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.
🎥 Comprehensive Feature Demonstration
https://github.com/labring/FastGPT/assets/15308462/7d3a38df-eb0e-4388-9250-2409bd33f6d4
🛸 Online Use
Website: fastgpt.io
| Conversational AI Setup | Workflow Automation |
 |
 |
| Knowledge Base Setup | Integration Process |
 |
 |
💡 Features
| Features | Details |
|---|---|
| Application Orchestration Features | ✅ Offers a straightforward mode, eliminating the need for complex orchestration ✅ Provides clear next-step instructions in dialogues ✅ Facilitates workflow orchestration ✅ Tracks references in source files ✅ Encapsulates modules for enhanced reuse at multiple levels ✅ Combines search and reordering functions 🔜 Includes a tool module 🔜 Integrates Laf for online HTTP module creation 🔜 Plugin encapsulation capabilities |
| Knowledge Base Features | ✅ Allows for the mixed use of multiple databases ✅ Keeps track of modifications and deletions in data chunks ✅ Enables specific vector models for each knowledge base ✅ Stores original source files ✅ Supports direct input and segment-based QA import ✅ Compatible with a variety of file formats: pdf, docx, txt, html, md, csv ✅ Facilitates URL reading and bulk CSV importing 🔜 Supports PPT and Excel file import 🔜 Features a file reader 🔜 Offers diverse data preprocessing options |
| Application Debugging Features | ✅ Enables targeted search testing within the knowledge base ✅ Allows feedback, editing, and deletion during conversations ✅ Presents the full context of interactions ✅ Displays all intermediate values within modules 🔜 Advanced Debug mode for orchestration |
| OpenAPI Interface | ✅ The completions interface (aligned with GPT's chat mode interface) ✅ CRUD operations for the knowledge base 🔜 CRUD operations for conversation |
| Operational Features | ✅ Share without requiring login ✅ Easy embedding with Iframe ✅ Customizable chat window embedding with features like default open, drag-and-drop ✅ Centralizes conversation records for review and annotation |
👨💻 Development
Project tech stack: NextJs + TS + ChakraUI + MongoDB + PostgreSQL (PG Vector plug-in)/Milvus
-
⚡ Fast Deployment
When using Sealos services, there is no need to purchase servers or domain names. It supports high concurrency and dynamic scaling, and the database application uses the kubeblocks database, which far exceeds the simple Docker container deployment in terms of IO performance.
[](https://cloud.sealos.io/?openapp=system-fastdeploy%3FtemplateName%3Dfastgpt&uid=fnWRt09fZP)Give it a 2-4 minute wait after deployment as it sets up the database. Initially, it might be a too slow since we're using the basic settings.
💪 Related Projects
- Laf: 3-minute quick access to third-party applications
- Sealos: Rapid deployment of cluster applications
- One API: Multi-model management, supports Azure, Wenxin Yiyuan, etc.
- TuShan: Build a backend management system in 5 minutes
🤝 Third-party Ecosystem
🏘️ Community & Support
- 🌐 Visit the FastGPT website for full documentation and useful links.
- 💬 Join our Discord server is to chat with FastGPT developers and other FastGPT users. This is a good place to learn about FastGPT, ask questions, and share your experiences.
- 🐞 Create GitHub Issues for bug reports and feature requests.
👀 Others
- FastGPT FAQ
- Docker Deployment Tutorial Video
- Official Account Integration Video Tutorial
- FastGPT Knowledge Base Demo
🌱 Contributors
We welcome all forms of contributions. If you are interested in contributing code, you can check out our GitHub Issues to show us your ideas.
|
|
|
|---|---|
|
|
|
|
🌟 Star History
📄 Usage Agreement
This repository complies with the FastGPT Open Source License open source agreement.
- Direct commercial use as a backend service is allowed, but provision of SaaS services is not allowed.
- Without commercial authorization, any form of commercial service must retain relevant copyright information.
- For full details, please see FastGPT Open Source License
- Contact: Dennis@sealos.io , click to view commercial version pricing strategy