Files
FastGPT/plugins/webcrawler
Theresa 2d3117c5da feat: update ESLint config with @typescript-eslint/consistent-type-imports (#4746)
* update: Add type

* fix: update import statement for NextApiRequest type

* fix: update imports to use type for LexicalEditor and EditorState

* Refactor imports to use 'import type' for type-only imports across multiple files

- Updated imports in various components and API files to use 'import type' for better clarity and to optimize TypeScript's type checking.
- Ensured consistent usage of type imports in files related to chat, dataset, workflow, and user management.
- Improved code readability and maintainability by distinguishing between value and type imports.

* refactor: remove old ESLint configuration and add new rules

- Deleted the old ESLint configuration file from the app project.
- Added a new ESLint configuration file with updated rules and settings.
- Changed imports to use type-only imports in various files for better clarity and performance.
- Updated TypeScript configuration to remove unnecessary options.
- Added an ESLint ignore file to exclude build and dependency directories from linting.

* fix: update imports to use 'import type' for type-only imports in schema files
2025-05-06 17:33:09 +08:00
..
2025-03-28 13:45:09 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00
2025-02-28 19:00:58 +08:00

webcrawler

docker版快速部署

代码版部署

  1. 按照 https://github.com/searxng/searxng-docker 的方式处理docker
  2. 参考SPIDER文件夹下的.env.example添加.env文件
  3. 进入SPIDER文件夹进行pnpm install
  4. 回到根目录运行docker compose up -d

代码版开发

  1. 将docker-compose.yml中与SPIDER相关的部分注释掉nodeapp
  2. .env文件中的URL参照注释修改
  3. 注释掉启动puppteer部分里面指定浏览器地址的代码
  4. pnpm run dev

测试样例:

Auth的Bear Token记得填,也就是.env里的ACCESS_TOKEN

读取单页面(content以HTML形式返回)

http://localhost:3000/api/read?queryUrl=<url>

返回结构


{
    "status": 200,
    "data": {
        "title": "something here",
        "content": "something here"
    }
}
{
    "status": 400,
    "error": {
        "code": "MISSING_PARAM",
        "message": "缺少必要参数: query"
    }
}

搜索(content以HTML形式返回)

http://localhost:3000/api/search?query=<something>&pageCount=5&needDetails=true&engine=baidu
{
    "status": 200,
    "data": {
        "results": [
            {
                "title": "string",
                "url": "string",
                "snippet": "string",
                "source": "string",
                "crawlStatus": "string",
                "score": 0,
                "content": "string"
            }
        ]
    }
}
{
    "status": 400,
    "error": {
        "code": "MISSING_PARAM",
        "message": "缺少必要参数: query"
    }
}