54 Commits

Author SHA1 Message Date
guqing
40f66ff665 chore: remove deprecated plugin wrapper (#161)
### What this PR does?
移除对已过时的 PluginWrapper 的引用,Halo 2.18.0 版本后将不在支持从 BasePlugin 中获取 PluginWrapper ,也不再支持依赖注入 PluginWrapper,使用 PluginContext 代替

see also https://github.com/halo-dev/halo/pull/6243 for more details

```release-note
None
```
2024-07-02 15:29:25 +00:00
longjuan
cb968de154 fix: apply natural ordering to policy and attachment listings (#160)
```release-note
None
```
传递unsorted以支持Halo 2.17版本
2024-07-02 15:27:25 +00:00
longjuan
bdd62cbe12 faet: Add file prefix search function in s3link (#153)
```release-note
在 S3关联 中支持文件名前缀搜索
```
fixes https://github.com/halo-dev/plugin-s3/issues/142
2024-06-16 12:30:45 +00:00
longjuan
2a26171fc4 Fix: Correct policy selector styling issue (#151)
```release-note
修复 S3 关联中策略选择器太短的问题
```
fixes #141
2024-05-14 02:08:33 +00:00
longjuan
6b62ce7aa4 Hide the permission of the verification interface (#147)
fixes https://github.com/halo-dev/plugin-s3/issues/146
```release-note
在角色创建中隐藏配置校验接口的权限
```
2024-05-06 10:34:54 +00:00
Ryan Wang
c60e31a033 chore: update plugin.yaml file (#138)
完善插件定义文件。

1. 将 homepage 字段指向 Halo 应用市场的地址。
2. 新增 repo 字段指向源码仓库地址。
3. 新增 issues 字段指向 GitHub issues 地址。

```release-note
None
```
2024-05-05 08:39:33 +00:00
Chenhe
7a9b0de0c6 improve usage tips in settings page (#140)
使用占位组件在设置页面展示信息并不恰当,并且此项目作为存储插件的 Example 工程易误导其他开发者。

调整为利用 FormKit `el` 特性实现。

问题背景:https://github.com/halo-dev/halo/issues/5802

**Preview - old**
<img width="373" alt="image" src="https://github.com/halo-dev/plugin-s3/assets/10266066/ac4aaede-6377-459f-a866-9bfe132906af">

**Preview - new**
<img width="373" alt="image" src="https://github.com/halo-dev/plugin-s3/assets/10266066/f884417c-045a-4396-84be-633d18ae36c9">

```release-note
优化调整提示的提示
```
2024-04-28 11:54:48 +00:00
longjuan
5e7e6620fd chore: bump version to 1.9.0 (#135)
```release-note
None
```
2024-04-22 06:08:08 +00:00
longjuan
47b6a37d0a feat: add verification function to the configuration of s3 object storage policy (#134)
```release-note
在 S3 存储策略配置中增加了验证配置的功能。
```
fixes https://github.com/halo-dev/plugin-s3/issues/132
2024-04-22 04:26:10 +00:00
longjuan
68b1a88b14 Fix original file name being incorrectly replaced (#129) 2024-04-14 20:25:05 +08:00
longjuan
f4ec56b7bc perf: optimize s3link interfaces using index mechanisms (#127)
* perf: optimize s3link list queries using index mechanisms
2024-04-03 12:10:54 +08:00
John Niang
c5d4e719a7 Simplify GitHub workflow by using reusable workflow (#130) 2024-03-26 22:11:52 +08:00
Ryan Wang
034b3f3ded chore: rename displayName to S3 对象存储 (#128)
在 Halo 完成动态数据的 i18n 之前,使用中文描述存储策略名称。

/kind improvement

<img width="821" alt="image" src="https://github.com/halo-dev/plugin-s3/assets/21301288/bc5fd32c-f7fc-43de-8bc4-cdbdd7e940d6">

```release-note
修改存储策略名称为中文显示。
```
2024-03-24 11:54:08 +00:00
longjuan
2503c6eba1 fix: Correct the formatting of Milliseconds to be a 3-digit number (#126)
```release-note
修复重命名模板中 millisecond 没有被格式化3位数的问题
```
2024-03-18 01:44:08 +00:00
John Niang
84aa7d32ba Correct release types to published instead of created 2024-03-05 22:23:26 +08:00
longjuan
73bd9e9948 chore: correcting incorrect policy template field help text (#124)
```release-note
None
```
2024-03-01 12:41:02 +00:00
longjuan
71c9784b64 chore: bump version to 1.8.0 (#123)
```release-note
None
```
2024-03-01 12:39:02 +00:00
longjuan
a16bbde9dd feat: add support for custom template in automatic renaming during upload (#115)
```release-note
上传自动重命名支持自定义模板,支持更多占位符
```

fixes #110
fixes #98
2024-02-29 13:53:38 +00:00
longjuan
9efa4b97e5 chore: remove plugin view permissions from S3 Link permission dependency (#121)
```release-note
None
```
入口已从插件设置页移除,移除依赖插件查看权限
2024-02-25 05:22:08 +00:00
longjuan
5c95a04a07 chore: updated s3 link entrance description (#120)
```release-note
None
```
2024-02-25 05:20:08 +00:00
longjuan
a63f3f4dc3 fix: missing page size selector style (#119)
```release-note
修复关联S3文件页面中 page size 下拉框样式缺失的问题
```
Before:
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/0bc89958-ce28-4be8-a6ce-6a4688c61c98)

After:
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/6099c2d0-e346-45eb-baa8-cd9e20c04b04)
2024-02-20 07:50:09 +00:00
Ryan Wang
11cbfdb9bb chore: simplify bundler config (#117)
使用 `@halo-dev/ui-plugin-bundler-kit` 简化构建配置。

/kind improvement

```release-note
None
```
2024-02-05 02:56:31 +00:00
Ryan Wang
2b0dd98575 refactor: place the entry for the S3 link page in the side menu (#116)
将 S3 关联页面放置在左侧菜单。

/kind improvement

Note: 此特性需要运行在 Halo 2.12 才生效,所以此 PR 同时修改了 requires。

<img width="1920" alt="image" src="https://github.com/halo-dev/plugin-s3/assets/21301288/a78d7837-81c2-4c50-ad19-2c40e74cbc41">

<img width="1920" alt="image" src="https://github.com/halo-dev/plugin-s3/assets/21301288/41edeed8-9ce0-40d1-82a9-1fb376a08c51">


```release-note
将 S3 关联页面放置在左侧菜单。
```
2024-02-03 15:50:55 +00:00
longjuan
8ff4acba6e perf: improve more friendly exception messages (#113)
```release-note
友好地提示异常信息
```
fixes https://github.com/halo-dev/plugin-s3/issues/105

验证方法:
1. ak/sk乱输,发生接收到403状态码(接收错误状态码)
2. endpoint网址改成不存在的,如.com改成.comaaa(未知主机)
3. endpoint端口改成没监听的(超时)
2024-01-16 14:14:12 +00:00
longjuan
82f409e349 chore: bump version to 1.7.0 (#107)
```release-note
None
```
已测试2.10.0和2.11.0无问题
2023-12-02 04:00:07 +00:00
longjuan
71d303404a chore: bold in README followed by a space (#106)
```release-note
None
```
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/c81be5ba-d575-4cb1-ba68-611bc1d1bbea)
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/399fe27b-5731-4345-89a1-fe1556912c61)

加粗后面加上空格
2023-12-01 07:56:10 +00:00
longjuan
b2cd58a7f7 chore: improve the usage and configuration guidelines in Readme (#104)
```release-note
None
```
完善使用方法和配置指南的重命名方式
2023-12-01 07:42:09 +00:00
longjuan
3fda6e0cde fix: log cannot be output and log information error (#101)
```release-note
修复无日志输出的问题
```
fixes #100 
测试上传,关联,删除,解除关联 4种操作的日志
```
---上传---
2023-11-27T22:39:22.139+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : operation: createMultipartUpload, result: CreateMultipartUploadResponse(Bucket=test-1305034426, Key=test/image-pjcl.png, UploadId=17010959629c897f8b8c1a592b0efe300de867a1ea3e891cf04b4f95c6986470a0f30195be)
2023-11-27T22:39:22.391+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : operation: uploadPart, result: UploadPartResponse(ETag="aa84eea735192b49552fa5bf3df317fa")
2023-11-27T22:39:22.547+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : operation: completeUpload, result: CompleteMultipartUploadResponse(Location=http://test-1305034426.cos.ap-guangzhou.myqcloud.com/test/image-pjcl.png, Bucket=test-1305034426, Key=test/image-pjcl.png, ETag="afdd5ce39cf49676238fcfc879664b55-1")
2023-11-27T22:39:22.616+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : operation: getMetadata, result: HeadObjectResponse(AcceptRanges=bytes, LastModified=2023-11-27T14:39:22Z, ContentLength=1420, ETag="afdd5ce39cf49676238fcfc879664b55-1", ContentType=image/png, Metadata={})
2023-11-27T22:39:22.616+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : Upload object test/image-pjcl.png to bucket test-1305034426 successfully
2023-11-27T22:39:22.626+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : Build attachment test/image-pjcl.png successfully
2023-11-27T22:39:29.915+08:00  WARN 7 --- [or-http-epoll-1] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: run.halo.s3os.S3LinkController
---关联---
2023-11-27T22:39:34.657+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : Build attachment test/34.txt successfully
---解除关联---
2023-11-27T22:39:39.937+08:00  INFO 7 --- [tReconciler-t-1] run.halo.s3os.S3OsAttachmentHandler      : Skip deleting object test/34.txt from S3.
---删除---
2023-11-27T22:39:44.290+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : operation: delete object, result: DeleteObjectResponse()
2023-11-27T22:39:44.291+08:00  INFO 7 --- [oundedElastic-5] run.halo.s3os.S3OsAttachmentHandler      : Delete object test/image-pjcl.png from bucket test-1305034426 successfully
```
2023-12-01 07:40:09 +00:00
longjuan
ac9c9b88a6 fix: list s3 file error in Halo 2.11 (#103)
```release-note
修复 Halo 2.11 中 关联S3文件 功能报错的问题
```
fixes #102
2023-11-28 02:46:10 +00:00
longjuan
77c8b21248 feat: only delete halo attachment records without deleting files in object storage (#97)
/kind feature
```release-note
提供解除 S3 关联功能(仅删除 Halo 附件记录,而不再对象存储中实际删除)
```
在附件库中选择使用本插件上传或关联的文件的更多插件,会出现 解除 S3 关联 按钮。
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/6bfd109e-3be5-4fdc-afb0-bbf841dfc8e8)
对于其他附件,不会出现该按钮
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/bc90166d-68c5-4599-b00a-5d2ddd868314)
本功能已支持权限管理。
2023-11-27 13:50:09 +00:00
longjuan
6f59d0ee32 chore: update .gitignore and remove untracked directories (#95)
移除 `src/main/resources/console/main.js` 并加入.gitignore
```release-note
None
```
2023-10-28 06:05:30 +00:00
John Niang
07127d7e54 Update gradle.properties 2023-10-27 02:18:56 -05:00
longjuan
565d3cfcaa Fix findUrlSuffix NPE when upgrading from old version (#94)
fixes https://github.com/halo-dev/plugin-s3/issues/93
```release-note
修复从旧版本升级后上传文件的NPE错误
```
/kind bug
2023-10-27 02:34:15 +00:00
longjuan
c0fb2b1017 fix: s3 link error when filename renaming enable (#91)
fixes https://github.com/halo-dev/plugin-s3/issues/90
请测试 `上传时重命名文件方式` 选项为**非**保留原文件名时,是否能正常关联s3文件
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/22ab0467-60bd-4c4c-a486-38bc922b7f37)

```release-note
当上传时文件重命名功能开启时关联S3文件错误
```
2023-10-26 06:38:14 +00:00
longjuan
c79fee9ba1 perf: improve s3link role permissions (#89)
1. 新建用户,仅赋予 s3link 权限
2. 测试 关联s3文件功能是否 正常使用
请使用 <ff7af9f0d9> 之后的版本进行测试


```release-note
完善 s3link 角色权限
```
2023-10-26 05:44:12 +00:00
longjuan
08d6ff49c8 feat: add URL suffix to files of specified type (#79)
增加网址后缀功能
![image](https://github.com/halo-dev/plugin-s3/assets/28662535/cd56ed01-70b0-44fc-ae40-110f7c2c1aea)

此功能在文件上传和关联文件均生效

Fix #77 
Fix #68 
```release-note
增加给指定类型的文件加上特定的网址后缀功能
```
2023-10-26 05:42:17 +00:00
longjuan
73112953ba feat: improve the styles of the data list filter area on the mobile devices (#87)
同步 <https://github.com/halo-dev/halo/pull/4587> 中的数据列表过滤器区域的样式改进。
```release-note
同步移动设备上数据列表过滤器区域的样式改进
```
2023-10-16 04:14:13 +00:00
Ryan Wang
84345ef349 chore: update github action (#84)
Signed-off-by: Ryan Wang <i@ryanc.cc>
2023-10-08 06:49:13 -05:00
longjuan
bc4de5fb8e fix: remove v-permission to avoid checkbox disappearing (#83)
移除v-permission以避免复选框消失
fixes https://github.com/halo-dev/plugin-s3/issues/82
```release-note
None
```
2023-10-08 03:32:16 +00:00
John Niang
c91f9981df Remove unused dependencies (#81)
This PR removes unused dependencies.

```release-note
None
```
2023-10-08 02:54:15 +00:00
John Niang
0116ae65d7 Exclude reactive-streams module for runtime classpath (#80)
Halo modified the [class loading order](https://github.com/halo-dev/halo/pull/4663) of plugins in 2.10.x to P (Plugin) D (Dependencies) A (Application), resulting in a LinkageError when the current plugin is actually run.

```java
java.lang.LinkageError: loader constraint violation: when resolving method 'reactor.core.publisher.Flux reactor.core.publisher.Mono.thenMany(org.reactivestreams.Publisher)' the class loader org.pf4j.PluginClassLoader @5da2e534 of the current class, run/halo/s3os/S3OsAttachmentHandler, and the class loader org.springframework.boot.loader.LaunchedURLClassLoader @87aac27 for the method's defining class, reactor/core/publisher/Mono, have different Class objects for the type org/reactivestreams/Publisher used in the signature (run.halo.s3os.S3OsAttachmentHandler is in unnamed module of loader org.pf4j.PluginClassLoader @5da2e534, parent loader org.springframework.boot.loader.LaunchedURLClassLoader @87aac27; reactor.core.publisher.Mono is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @87aac27, parent loader 'app')
        at run.halo.s3os.S3OsAttachmentHandler.lambda$upload$28(S3OsAttachmentHandler.java:298) ~[na:na]
        at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:85) ~[reactor-core-3.5.10.jar:3.5.10]
        at reactor.core.publisher.Mono.subscribe(Mono.java:4495) ~[reactor-core-3.5.10.jar:3.5.10]
        at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126) ~[reactor-core-3.5.10.jar:3.5.10]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.10.jar:3.5.10]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.10.jar:3.5.10]
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
```

We just remove the reactive-streams dependency that is already in Halo to resolve the problem.

Fixes https://github.com/halo-dev/halo/issues/4676

/kind bug

```release-note
修复在 Halo 2.10 中无法正常上传的问题
```
2023-10-08 02:40:16 +00:00
longjuan
0265a71c83 chore: Modify upyun compatibility and service provider document hyperlink format (#78)
```release-note
None
```
2023-09-27 08:12:14 +00:00
AcAutomaton
217f1db0de feat: Time placeholder of upload path (#74)
Fixes #69 

```release-note
文件上传路径支持使用`${year}`,`${month}`,`${day}`占位符
```
2023-09-27 08:10:17 +00:00
AcAutomaton
38f2018fb9 chore: correct typos (#76)
```release-note
None
```
2023-09-24 09:56:11 +00:00
Ryan Wang
faa1ad59bb chore: change repo organization to halo-dev (#73)
修改仓库组织为 halo-dev

/kind cleanup

```release-note
None
```
2023-09-19 03:24:21 +00:00
longjuan
f784e3789c chore: Unify the author information of plugin.yaml (#72)
```release-note
None
```
2023-09-19 03:02:20 +00:00
AcAutomaton
e2048028f6 chore: Optimize access_key description (#71)
Fixes #67 

```release-note
优化`Access Key`相关的描述,减少误解
```
2023-09-18 15:52:21 +00:00
longjuan
91a61fd1d2 feat: Associate files originally in s3 (#59)
```release-note
关联从其他渠道上传至 s3 的文件
```
【插件】->【对象存储(Amazon S3 协议)】->【关联s3文件】
![image](https://github.com/halo-sigs/plugin-s3/assets/28662535/83096e23-d362-4924-b2f0-6984683d87cf)
![image](https://github.com/halo-sigs/plugin-s3/assets/28662535/6dd74842-f9a5-4e3c-9a93-36b4b188ecf6)

目前待优化的功能:
- [ ] 列出未关联的文件时会查询多次数据库,期望优化成只查询一次,但是代码改动较大。
2023-09-11 16:04:09 +00:00
AcAutomaton
1b8bef991e feat: automatic random file renaming optional feature (#60)
图片等文件外链往往需要呈现给用户,由于大部分时候我们都是随手创建的新文件或从别处复制粘贴,文件名会如同`无标题.png`等,上传后按照默认策略也会如同`无标题-qiks.png`等,既不美观也会给用户造成困扰。此功能可以在上传文件时将文件名自动替换为UUID、日期、随机字符串等格式。
```release-note
增加自动随机重命名文件可选功能
```
2023-08-29 13:08:14 +00:00
Ryan Wang
52cfa53d7c docs: update usage documentation of readme file (#61)
更新 README 中下载和安装插件的说明。

/kind documentation

```release-note
None 
```
2023-08-12 09:02:10 +00:00
John Niang
c109bbd61f Fallback to default handler for backward compatibility (#57)
#### What type of PR is this?

/kind improvement

#### What this PR does / why we need it:

See https://github.com/halo-sigs/plugin-s3/issues/56 for more.

This PR skips permalink resolution while the object key is missing. So that the default handler will resolve permalink from annotation `storage.halo.run/external-link`

#### Which issue(s) this PR fixes:

Fixes https://github.com/halo-sigs/plugin-s3/issues/56

#### Does this PR introduce a user-facing change?

```release-note
解决导入 Halo 1.x 附件后出现“Cannot obtain object key from attachment attachment-xyz”的问题
```
2023-08-01 10:14:54 +00:00
longjuan
2320800907 chore: bump version to 1.4.1 (#53)
```release-note
None
```
2023-07-21 15:58:13 +00:00
John Niang
00537c164c Ensure non-trailing parts are of equal size (#50)
#### What type of PR is this?

/kind improvement

#### What this PR does / why we need it:

Reshape the DataBuffers into parts of the same size (5MB) except for the last part.

#### Which issue(s) this PR fixes:

Fixes https://github.com/halo-sigs/plugin-s3/issues/49

#### Does this PR introduce a user-facing change?

```release-note
保证分片上传时片段大小一致
```
2023-07-05 05:10:11 +00:00
Kevin Zhang
8be39b9898 Impove the readme (#46)
Fixes https://github.com/halo-sigs/plugin-s3/issues/43

桶填写错误:
![image](https://github.com/halo-sigs/plugin-s3/assets/34374831/d3f0855b-5e72-4c59-80c6-017715cd4ba4)

补充桶填写说明:
<img width="895" alt="image" src="https://github.com/halo-sigs/plugin-s3/assets/34374831/d8d2e288-e4e2-4788-95a6-b86d91f4a1ec">


/kind feature

```release-note
完善README.md
```
2023-07-01 08:14:12 +00:00
67 changed files with 6436 additions and 243 deletions

17
.github/workflows/cd.yaml vendored Normal file
View File

@@ -0,0 +1,17 @@
name: CD
on:
release:
types:
- published
jobs:
cd:
uses: halo-sigs/reusable-workflows/.github/workflows/plugin-cd.yaml@v1
secrets:
halo-username: ${{ secrets.HALO_USERNAME }}
halo-password: ${{ secrets.HALO_PASSWORD }}
permissions:
contents: write
with:
app-id: app-Qxhpp

13
.github/workflows/ci.yaml vendored Normal file
View File

@@ -0,0 +1,13 @@
name: CI
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
ci:
uses: halo-sigs/reusable-workflows/.github/workflows/plugin-ci.yaml@v1

View File

@@ -1,81 +0,0 @@
name: Build Plugin JAR File
on:
push:
branches: [ main ]
release:
types:
- created
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
submodules: true
- name: Set up JDK 17
uses: actions/setup-java@v2
with:
distribution: 'temurin'
cache: 'gradle'
java-version: 17
- name: Build with Gradle
run: |
# Set the version with tag name when releasing
version=${{ github.event.release.tag_name }}
version=${version#v}
sed -i "s/version=.*-SNAPSHOT$/version=$version/1" gradle.properties
./gradlew clean build -x test
- name: Archive plugin-s3 jar
uses: actions/upload-artifact@v2
with:
name: plugin-s3
path: |
build/libs/*.jar
retention-days: 1
github-release:
runs-on: ubuntu-latest
needs: build
if: github.event_name == 'release'
steps:
- name: Download plugin-s3 jar
uses: actions/download-artifact@v2
with:
name: plugin-s3
path: build/libs
- name: Get Name of Artifact
id: get_artifact
run: |
ARTIFACT_PATHNAME=$(ls build/libs/*.jar | head -n 1)
ARTIFACT_NAME=$(basename ${ARTIFACT_PATHNAME})
echo "Artifact pathname: ${ARTIFACT_PATHNAME}"
echo "Artifact name: ${ARTIFACT_NAME}"
echo "ARTIFACT_PATHNAME=${ARTIFACT_PATHNAME}" >> $GITHUB_ENV
echo "ARTIFACT_NAME=${ARTIFACT_NAME}" >> $GITHUB_ENV
echo "RELEASE_ID=${{ github.event.release.id }}" >> $GITHUB_ENV
- name: Upload a Release Asset
uses: actions/github-script@v2
if: github.event_name == 'release'
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
console.log('environment', process.versions);
const fs = require('fs').promises;
const { repo: { owner, repo }, sha } = context;
console.log({ owner, repo, sha });
const releaseId = process.env.RELEASE_ID
const artifactPathName = process.env.ARTIFACT_PATHNAME
const artifactName = process.env.ARTIFACT_NAME
console.log('Releasing', releaseId, artifactPathName, artifactName)
await github.repos.uploadReleaseAsset({
owner, repo,
release_id: releaseId,
name: artifactName,
data: await fs.readFile(artifactPathName)
});

2
.gitignore vendored
View File

@@ -70,4 +70,6 @@ application-local.yml
application-local.yaml
application-local.properties
/admin-frontend/node_modules/
/workplace/
/src/main/resources/console/

131
README.md
View File

@@ -4,14 +4,33 @@
## 使用方法
1. 在 [Releases](https://github.com/halo-sigs/plugin-s3/releases) 下载最新的 JAR 文件。
2. 在 Halo 后台的插件管理上传 JAR 文件进行安装
3. 进入后台附件管理。
4. 点击右上角的存储策略,在存储策略弹框的右上角可新建 S3 Object Storage 存储策略。
5. 创建完成之后即可在上传的时候选择新创建的 S3 Object Storage 存储策略
1. 下载,目前提供以下两个下载方式:
- GitHub Releases访问 [Releases](https://github.com/halo-dev/plugin-s3/releases) 下载 Assets 中的 JAR 文件。
- Halo 应用市场:<https://halo.run/store/apps/app-Qxhpp>
2. 安装,插件安装和更新方式可参考:<https://docs.halo.run/user-guide/plugins>
3. 配置存储策略
* 在 Halo 后台管理系统中,点击左侧“附件”导航至附件管理页面,点击右上角的存储策略。
* 在存储策略管理界面中,您可以新建和编辑 S3 Object Storage 存储策略。
4. 上传到存储策略:
* 在附件页面中点击上传按钮,选择好存储策略后上传文件即可上传到对应的对象存储中。
* 在 Halo 2.11 以上版本中可在 Halo 设置界面中设定文章附件、头像等的默认存储策略。
5. 使用“关联 S3 文件”功能:
* 在左侧侧边导航栏中,点击工具,再点击 S3 关联。
* 在此界面中,您可以浏览并选择已在对象存储中但不是通过 Halo 上传的文件,关联后会在 Halo 生成相应的附件记录。这些文件现在可以方便地在 Halo 中管理和使用。
6. 使用“解除 S3 关联”功能:
* 在附件页面中,找到由本插件管理的附件记录,点击更多操作(右侧的三个点)按钮,然后点击“解除 S3 关联”按钮。
* 此操作将仅删除 Halo 中的附件记录,而不会实际删除对象存储中的文件。如需恢复请使用“关联 S3 文件”功能。
## 配置指南
### Bucket 桶名称
一般与服务商控制台中的空间名称一致。
> 注意部分服务商 s3 空间名 ≠ 空间名称若出现“Access Denied”报错可检查 Bucket 是否正确。
>
> 可通过 S3Browser 查看桶列表,七牛云也可在“开发者平台-对象存储-空间概览-s3域名”中查看 s3 空间名。
### Endpoint 访问风格
请根据下方表格中的兼容访问风格选择,若您的服务商不在表格中,请自行查看服务商的 s3 兼容性文档或自行尝试。
@@ -33,10 +52,6 @@
与服务商自己 API 的 Access Key 和 Access Secret 相同,详情查看对应服务商的文档。
### Bucket 桶名称
与服务商的控制台中的桶名称一致。
### Region
一般留空即可。
@@ -45,24 +60,98 @@
>
> Cloudflare 需要填写均为小写字母的 `auto`。
### 上传目录
上传到对象存储的目录,前后`/`可省略,例如`/halo``halo`是等价的。
支持的占位符有:
* `${uuid-with-dash}`:带有`-`的 UUID
* `${uuid-no-dash}`:不带`-`的 UUID
* `${timestamp-sec}`秒时间戳10位时间戳
* `${timestamp-ms}`毫秒时间戳13位时间戳
* `${year}`:年份
* `${month}`:月份(两位数)
* `${day}`:日期(两位数)
* `${weekday}`星期几1-7
* `${hour}`小时24小时制两位数
* `${minute}`:分钟(两位数)
* `${second}`:秒(两位数)
* `${millisecond}`:毫秒(三位数)
* `${random-alphabetic:X}`:随机的小写英文字母,长度为`X`,例如`${random-alphabetic:5}`会生成`abcde`
* `${random-num:X}`:随机的数字,长度为`X`,例如`${random-num:5}`会生成`12345`
* `${random-alphanumeric:X}`:随机的小写英文字母和数字,长度为`X`,例如`${random-alphanumeric:5}`会生成`abc12`
> **示例**<br/>
> * `${year}/${month}/${day}/${random-alphabetic:1}`会放在`2023/12/01/a`。<br/>
> * `halo/${uuid-no-dash}`会放在`halo/123E4567E89B12D3A456426614174000`。
### 上传时重命名文件方式
* **保留原文件名:** 使用上传时的文件名。
* **自定义:** 使用`自定义文件名模板`中填写的模板,上传时替换相应占位符作后作为文件名。
* **使用 UUID** 上传时会自动重命名为随机的 UUID。
* **使用毫秒时间戳:** 上传时会自动重命名为毫秒时间戳13位时间戳
* **使使用原文件名 + 随机字母:** 上传时会自动重命名为原文件名 + 随机的小写英文字母,长度请在`随机字母长度`中设置。
* **使用日期 + 随机字母:** 上传时会自动重命名为日期 + 随机的小写英文字母,例如 `2023-12-01-abcdefgh.png`
* **使用日期时间 + 随机字母:** 上传时会自动重命名为日期时间 + 随机的小写英文字母,例如 `2023-12-01T09:30:01-abcdef.png`
* **使用随机字母:** 上传时会自动重命名为随机的小写英文字母,长度请在`随机字母长度`中设置。
### 随机字母长度
仅当`上传时重命名文件方式``使用原文件名 + 随机字母``使用日期 + 随机字母``使用日期时间 + 随机字母``使用随机字母`时出现,用于设置随机字母的长度。
### 自定义文件名模板
仅当`上传时重命名文件方式``自定义`时出现,用于设置自定义文件名模板。
支持的占位符有:
* `${origin-filename}`:原文件名
* `${uuid-with-dash}`:带有`-`的 UUID
* `${uuid-no-dash}`:不带`-`的 UUID
* `${timestamp-sec}`秒时间戳10位时间戳
* `${timestamp-ms}`毫秒时间戳13位时间戳
* `${year}`:年份
* `${month}`:月份(两位数)
* `${day}`:日期(两位数)
* `${weekday}`星期几1-7
* `${hour}`小时24小时制两位数
* `${minute}`:分钟(两位数)
* `${second}`:秒(两位数)
* `${millisecond}`:毫秒(三位数)
* `${random-alphabetic:X}`:随机的小写英文字母,长度为`X`,例如`${random-alphabetic:5}`会生成`abcde`
* `${random-num:X}`:随机的数字,长度为`X`,例如`${random-num:5}`会生成`12345`
* `${random-alphanumeric:X}`:随机的小写英文字母和数字,长度为`X`,例如`${random-alphanumeric:5}`会生成`abc12`
> **示例**<br/>
> 当原始文件名为`image.png`时<br/>
> * `${origin-filename}-${uuid-with-dash}`会生成`image-123E4567-E89B-12D3-A456-426614174000.png`。<br/>
> * `${year}-${month}-${day}T${hour}:${minute}:${second}-${random-alphanumeric:5}`会生成`2023-12-01T09:30:01-abc12.png`。<br/>
> * `${uuid-no-dash}_file_${random-alphabetic:5}`会生成`123E4567E89B12D3A456426614174000_file_abcde.png`。<br/>
> * `halo_${origin-filename}_${random-num:3}`会生成`halo_image_123.png`。
### 重复文件名处理方式
* **加随机字母数字后缀:** 如遇重名会在文件名后加上4位的随机字母数字后缀例如`image.png`会变成`image_abc1.png`
* **加随机字母后缀:** 如遇重名会在文件名后加上4位的随机字母后缀例如`image.png`会变成`image_abcd.png`
* **报错不上传** 如遇重名,会放弃上传,并在用户界面提示 Duplicate filename 错误。
## 部分对象存储服务商兼容性
|服务商|文档|兼容访问风格|兼容性|
| ----- | ---- | ----- | ----- |
|阿里云|https://help.aliyun.com/document_detail/410748.html|Virtual Hosted Style|✅|
|腾讯云|[https://cloud.tencent.com/document/product/436/41284](https://cloud.tencent.com/document/product/436/41284)|Virtual Hosted Style / <br>Path Style|✅|
|七牛云|https://developer.qiniu.com/kodo/4088/s3-access-domainname|Virtual Hosted Style / <br>Path Style|✅|
|百度云|https://cloud.baidu.com/doc/BOS/s/Fjwvyq9xo|Virtual Hosted Style / <br>Path Style|✅|
|京东云| https://docs.jdcloud.com/cn/object-storage-service/api/regions-and-endpoints |Virtual Hosted Style|✅|
|金山云|https://docs.ksyun.com/documents/6761|Virtual Hosted Style|✅|
|青云|https://docsv3.qingcloud.com/storage/object-storage/s3/intro/|Virtual Hosted Style / <br>Path Style|✅|
|网易数帆|[https://sf.163.com/help/documents/89796157866430464](https://sf.163.com/help/documents/89796157866430464)|Virtual Hosted Style|✅|
|Cloudflare|Cloudflare S3 兼容性API<br>[https://developers.cloudflare.com/r2/data-access/s3-api/](https://developers.cloudflare.com/r2/data-access/s3-api/)|Virtual Hosted Style / <br>Path Style|✅|
| Oracle Cloud |[https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)|Virtual Hosted Style / <br>Path Style|✅|
|阿里云|<https://help.aliyun.com/document_detail/410748.html>|Virtual Hosted Style|✅|
|腾讯云|<https://cloud.tencent.com/document/product/436/41284>|Virtual Hosted Style / <br>Path Style|✅|
|七牛云|<https://developer.qiniu.com/kodo/4088/s3-access-domainname>|Virtual Hosted Style / <br>Path Style|✅|
|百度云|<https://cloud.baidu.com/doc/BOS/s/xjwvyq9l4>|Virtual Hosted Style / <br>Path Style|✅|
|京东云|<https://docs.jdcloud.com/cn/object-storage-service/api/regions-and-endpoints>|Virtual Hosted Style|✅|
|金山云|<https://docs.ksyun.com/documents/6761>|Virtual Hosted Style|✅|
|青云|<https://docsv3.qingcloud.com/storage/object-storage/s3/intro/>|Virtual Hosted Style / <br>Path Style|✅|
|网易数帆|<https://sf.163.com/help/documents/89796157866430464>|Virtual Hosted Style|✅|
|Cloudflare|<https://developers.cloudflare.com/r2/data-access/s3-api/>|Virtual Hosted Style / <br>Path Style|✅|
| Oracle Cloud |<https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm>|Virtual Hosted Style / <br>Path Style|✅|
|又拍云|<https://help.upyun.com/knowledge-base/aws-s3%e5%85%bc%e5%ae%b9/>|Virtual Hosted Style / <br>Path Style|✅|
|自建minio|\-|Path Style|✅|
|华为云|文档未说明是否兼容,工单反馈不保证兼容性,实际测试可以使用|Virtual Hosted Style|❓|
|Ucloud|只支持 8MB 大小的分片,本插件暂不支持<br>[https://docs.ucloud.cn/ufile/s3/s3\_introduction](https://docs.ucloud.cn/ufile/s3/s3_introduction)|\-|❌|
|又拍云|暂不支持 s3 协议|\-|❌|
|Ucloud|只支持 8MB 大小的分片,本插件暂不支持<br><https://docs.ucloud.cn/ufile/s3/s3_introduction>|\-|❌|
## 开发环境

View File

@@ -1,42 +1,69 @@
plugins {
id "io.github.guqing.plugin-development" version "0.0.7-SNAPSHOT"
id "io.freefair.lombok" version "8.0.0-rc2"
id 'java'
id "com.github.node-gradle.node" version "5.0.0"
id "io.freefair.lombok" version "8.0.1"
id "run.halo.plugin.devtools" version "0.0.7"
}
group 'run.halo.s3os'
sourceCompatibility = JavaVersion.VERSION_17
repositories {
mavenCentral()
maven { url 'https://s01.oss.sonatype.org/content/repositories/releases' }
maven { url 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
mavenCentral()
}
jar {
enabled = true
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
from {
configurations.runtimeClasspath.collect {
it.isDirectory() ? it : zipTree(it)
}
}
maven { url 'https://repo.spring.io/milestone' }
}
dependencies {
implementation platform('run.halo.tools.platform:plugin:2.5.0-SNAPSHOT')
implementation platform('run.halo.tools.platform:plugin:2.14.0-SNAPSHOT')
compileOnly 'run.halo.app:api'
implementation platform('software.amazon.awssdk:bom:2.19.8')
implementation 'software.amazon.awssdk:s3'
implementation "javax.xml.bind:jaxb-api:2.3.1"
implementation "javax.activation:activation:1.1.1"
implementation "org.glassfish.jaxb:jaxb-runtime:2.3.3"
implementation ('software.amazon.awssdk:s3') {
exclude group: 'org.slf4j'
exclude group: 'commons-logging'
}
testImplementation 'run.halo.app:api'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'io.projectreactor:reactor-test'
}
configurations.runtimeClasspath {
exclude group: 'org.reactivestreams', module: 'reactive-streams'
}
halo {
version = '2.14.0'
}
haloPlugin {
watchDomains {
consoleSource {
files files('console/src/')
}
}
}
test {
useJUnitPlatform()
}
node {
nodeProjectDir = file("${project.projectDir}/console")
}
task buildFrontend(type: PnpmTask) {
args = ['build']
}
tasks.named('buildFrontend') {
dependsOn 'pnpmInstall'
}
build {
// build frontend before build
tasks.getByName('compileJava').dependsOn('buildFrontend')
}

12
console/.editorconfig Normal file
View File

@@ -0,0 +1,12 @@
# EditorConfig is awesome: https://EditorConfig.org
# top-most EditorConfig file
root = true
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = false
insert_final_newline = true

15
console/.eslintrc.cjs Normal file
View File

@@ -0,0 +1,15 @@
/* eslint-env node */
require("@rushstack/eslint-patch/modern-module-resolution");
module.exports = {
root: true,
extends: [
"plugin:vue/vue3-essential",
"eslint:recommended",
"@vue/eslint-config-typescript/recommended",
"@vue/eslint-config-prettier",
],
env: {
"vue/setup-compiler-macros": true,
},
};

28
console/.gitignore vendored Normal file
View File

@@ -0,0 +1,28 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
.DS_Store
dist
dist-ssr
coverage
*.local
/cypress/videos/
/cypress/screenshots/
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

6
console/env.d.ts vendored Normal file
View File

@@ -0,0 +1,6 @@
/// <reference types="vite/client" />
declare module "*.vue" {
import Vue from "vue";
export default Vue;
}

48
console/package.json Normal file
View File

@@ -0,0 +1,48 @@
{
"name": "@halo-dev/plugin-starter",
"version": "1.0.0",
"private": true,
"scripts": {
"dev": "vite build --watch",
"build": "vite build",
"preview": "vite preview --port 4173",
"test:unit": "vitest --environment jsdom",
"type-check": "vue-tsc --noEmit -p tsconfig.vitest.json --composite false",
"lint": "eslint . --ext .vue,.js,.jsx,.cjs,.mjs,.ts,.tsx,.cts,.mts --fix --ignore-path .gitignore"
},
"dependencies": {
"@halo-dev/api-client": "^2.11.0",
"@halo-dev/components": "^1.10.0",
"@halo-dev/console-shared": "^2.11.0",
"@tanstack/vue-query": "4.29.1",
"axios": "^1.4.0",
"canvas-confetti": "^1.6.0",
"path-browserify": "^1.0.1",
"vue": "^3.3.4"
},
"devDependencies": {
"@halo-dev/ui-plugin-bundler-kit": "^2.12.0",
"@iconify/json": "^2.2.18",
"@rushstack/eslint-patch": "^1.2.0",
"@types/canvas-confetti": "^1.6.0",
"@types/jsdom": "^20.0.0",
"@types/node": "^16.18.0",
"@vitejs/plugin-vue": "^3.1.2",
"@vitejs/plugin-vue-jsx": "^2.0.1",
"@vue/eslint-config-prettier": "^7.0.0",
"@vue/eslint-config-typescript": "^11.0.2",
"@vue/test-utils": "^2.2.0",
"@vue/tsconfig": "^0.1.3",
"eslint": "^8.26.0",
"eslint-plugin-vue": "^9.6.0",
"jsdom": "^19.0.0",
"npm-run-all": "^4.1.5",
"prettier": "^2.7.1",
"sass": "^1.58.0",
"typescript": "~4.7.4",
"unplugin-icons": "^0.15.2",
"vite": "^3.1.8",
"vitest": "^0.24.3",
"vue-tsc": "^1.0.9"
}
}

3768
console/pnpm-lock.yaml generated Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 5.5 KiB

View File

@@ -0,0 +1,2 @@
export * from "./s-3-link-controller";
export * from "./s-3-unlink-controller";

View File

@@ -0,0 +1,27 @@
import request from "@/utils/request";
import { S3ListResult, DeepRequired } from "../../interface";
/**
* /apis/s3os.halo.run/v1alpha1/objects/{policyName}
*/
export function getApisS3OsHaloRunV1Alpha1ObjectsByPolicyName(params: GetApisS3OsHaloRunV1Alpha1ObjectsByPolicyNameParams) {
const paramsInput = {
continuationToken: params.continuationToken,
continuationObject: params.continuationObject,
pageSize: params.pageSize,
unlinked: params.unlinked,
filePrefix: params.filePrefix,
};
return request.get<DeepRequired<S3ListResult>>(`/apis/s3os.halo.run/v1alpha1/objects/${params.policyName}`, {
params: paramsInput,
});
}
interface GetApisS3OsHaloRunV1Alpha1ObjectsByPolicyNameParams {
policyName: any;
continuationToken?: any;
continuationObject?: any;
pageSize: any;
unlinked?: any;
filePrefix?: any;
}

View File

@@ -0,0 +1,9 @@
import request from "@/utils/request";
import { DeepRequired } from "../../interface";
/**
* /apis/s3os.halo.run/v1alpha1/policies/s3
*/
export function getApisS3OsHaloRunV1Alpha1PoliciesS3() {
return request.get<DeepRequired<any>>(`/apis/s3os.halo.run/v1alpha1/policies/s3`);
}

View File

@@ -0,0 +1,3 @@
export * from "./postApisS3OsHaloRunV1Alpha1AttachmentsLink";
export * from "./getApisS3OsHaloRunV1Alpha1ObjectsByPolicyName";
export * from "./getApisS3OsHaloRunV1Alpha1PoliciesS3";

View File

@@ -0,0 +1,9 @@
import request from "@/utils/request";
import { LinkResult, DeepRequired, LinkRequest } from "../../interface";
/**
* /apis/s3os.halo.run/v1alpha1/attachments/link
*/
export function postApisS3OsHaloRunV1Alpha1AttachmentsLink(input: LinkRequest) {
return request.post<DeepRequired<LinkResult>>(`/apis/s3os.halo.run/v1alpha1/attachments/link`, input);
}

View File

@@ -0,0 +1,14 @@
import request from "@/utils/request";
import { DeepRequired } from "../../interface";
import { Attachment } from "@halo-dev/api-client";
/**
* /apis/s3os.halo.run/v1alpha1/attachments/{name}
*/
export function deleteApisS3OsHaloRunV1Alpha1AttachmentsByName(params: DeleteApisS3OsHaloRunV1Alpha1AttachmentsByNameParams) {
return request.delete<DeepRequired<Attachment>>(`/apis/s3os.halo.run/v1alpha1/attachments/${params.name}`);
}
interface DeleteApisS3OsHaloRunV1Alpha1AttachmentsByNameParams {
name: any;
}

View File

@@ -0,0 +1 @@
export * from "./deleteApisS3OsHaloRunV1Alpha1AttachmentsByName";

52
console/src/index.ts Normal file
View File

@@ -0,0 +1,52 @@
import type { Attachment } from "@halo-dev/api-client";
import { definePlugin } from "@halo-dev/console-shared";
import S3Link from "./views/S3Link.vue";
import S3Unlink from "./views/S3Unlink.vue";
import type { Ref } from "vue";
import { markRaw } from "vue";
import CarbonFolderDetailsReference from "~icons/carbon/folder-details-reference";
export default definePlugin({
components: {},
routes: [
{
parentName: "ToolsRoot",
route: {
path: "s3-link",
name: "S3Link",
component: S3Link,
meta: {
title: "S3 关联",
description: "提供将 S3 存储桶中的文件关联到 Halo 中的功能。",
searchable: true,
permissions: ["plugin:s3os:link"],
menu: {
name: "S3 关联",
icon: markRaw(CarbonFolderDetailsReference),
priority: 0,
},
},
},
},
],
extensionPoints: {
"attachment:list-item:operation:create": (attachment: Ref<Attachment>) => {
return [
{
priority: 21,
component: markRaw(S3Unlink),
permissions: ["plugin:s3os:unlink"],
props: {
attachment: attachment,
},
hidden: !(
attachment.value.metadata.annotations &&
attachment.value.metadata.annotations[
"s3os.plugin.halo.run/object-key"
]
),
},
];
},
},
});

View File

@@ -0,0 +1,4 @@
export interface LinkRequest {
objectKeys?: string[];
policyName?: string;
}

View File

@@ -0,0 +1,5 @@
import { LinkResultItem } from "../../interface";
export interface LinkResult {
items?: LinkResultItem[];
}

View File

@@ -0,0 +1,5 @@
export interface LinkResultItem {
message?: string;
objectKey?: string;
success?: boolean;
}

View File

@@ -0,0 +1,20 @@
export interface Metadata {
annotations?: MetadataAnnotations;
creationTimestamp?: string;
deletionTimestamp?: string;
finalizers?: string[];
/** The name field will be generated automatically according to the given generateName field */
generateName?: string;
labels?: MetadataLabels;
/** Metadata name */
name: string;
version?: number;
}
export interface MetadataAnnotations {
[key: string]: any;
}
export interface MetadataLabels {
[key: string]: any;
}

View File

@@ -0,0 +1,6 @@
export interface ObjectVo {
displayName?: string;
isLinked?: boolean;
key?: string;
lastModified?: string;
}

View File

@@ -0,0 +1,8 @@
import { Metadata, PolicySpec } from "../../interface";
export interface Policy {
apiVersion: string;
kind: string;
metadata: Metadata;
spec: PolicySpec;
}

View File

@@ -0,0 +1,8 @@
export interface PolicySpec {
/** Reference name of ConfigMap extension */
configMapName?: string;
/** Display name of policy */
displayName: string;
/** Reference name of PolicyTemplate */
templateName: string;
}

View File

@@ -0,0 +1,10 @@
import { ObjectVo } from "../../interface";
export interface S3ListResult {
currentContinuationObject?: string;
currentToken?: string;
hasMore?: boolean;
nextContinuationObject?: string;
nextToken?: string;
objects?: ObjectVo[];
}

View File

@@ -0,0 +1,11 @@
export * from "./apiTypes/LinkRequest";
export * from "./apiTypes/LinkResult";
export * from "./apiTypes/LinkResultItem";
export * from "./apiTypes/Metadata";
export * from "./apiTypes/ObjectVo";
export * from "./apiTypes/Policy";
export * from "./apiTypes/PolicySpec";
export * from "./apiTypes/S3ListResult";
export type Primitive = undefined | null | boolean | string | number | symbol;
export type DeepRequired<T> = T extends Primitive ? T : keyof T extends never ? T : { [K in keyof T]-?: DeepRequired<T[K]> };

View File

@@ -0,0 +1,7 @@
{
"docsUrl": "http://localhost:8090/v3/api-docs/extension-api",
"includeTags": ["s-3-link-controller", "s-3-unlink-controller"],
"excludeTags": [],
"axiosInstanceUrl": "@/utils/request",
"prefix": ""
}

View File

@@ -0,0 +1,33 @@
import axios from "axios";
import {Toast} from "@halo-dev/components";
const baseURL = import.meta.env.VITE_API_URL;
const request = axios.create({
baseURL,
withCredentials: true,
});
// 非200状态码就弹窗
request.interceptors.response.use(
(response) => {
return response;
},
async (error) => {
const errorResponse = error.response;
if (!errorResponse) {
return Promise.reject(error);
}
const { status } = errorResponse;
if (status === 400) {
Toast.error(errorResponse.data.detail);
} else if (status !== 200) {
Toast.error("status: " + status);
}
return Promise.reject(error);
}
);
request.defaults.headers.common["X-Requested-With"] = "XMLHttpRequest";
// TODO 使用halo console 中的axios https://github.com/halo-dev/halo/issues/3979
export default request;

View File

@@ -0,0 +1,472 @@
<script setup lang="ts">
import {
IconRefreshLine,
Toast,
VButton,
VCard,
VEmpty,
VEntity,
VEntityField,
VLoading,
VModal,
VPageHeader,
VSpace,
VTag,
} from "@halo-dev/components";
import CarbonFolderDetailsReference from "~icons/carbon/folder-details-reference";
import IconErrorWarning from "~icons/ri/error-warning-line";
import {computed, onMounted, ref, watch} from "vue";
import {
getApisS3OsHaloRunV1Alpha1ObjectsByPolicyName,
getApisS3OsHaloRunV1Alpha1PoliciesS3,
postApisS3OsHaloRunV1Alpha1AttachmentsLink,
} from "@/controller";
import type {ObjectVo, S3ListResult, Policy, LinkResultItem} from "@/interface";
const selectedFiles = ref<string[]>([]);
const policyName = ref<string>("");
const page = ref(1);
const size = ref(50);
const policyOptions = ref<{ label: string; value: string; attrs: any }[]>([{
label: "请选择存储策略",
value: "",
attrs: {disabled: true}
}]);
// update when fetch first page
const filePrefix = ref<string>("");
// update when user input
const filePrefixBind = ref<string>("");
const s3Objects = ref<S3ListResult>({
objects: [],
hasMore: false,
currentToken: "",
nextToken: "",
currentContinuationObject: "",
nextContinuationObject: "",
});
// view state
const isFetching = ref(false);
const isShowModal = ref(false);
const isLinking = ref(false);
const isFetchingPolicies = ref(true);
const linkTips = ref("");
const linkFailedTable = ref<LinkResultItem[]>([]);
const linkedStatusItems: { label: string; value?: boolean }[] = [
{label: "全部"},
{label: "未关联", value: true},
];
// action state
const checkedAll = ref(false);
const selectedLinkedStatusItem = ref<boolean | undefined>(linkedStatusItems[0].value);
const emptyTips = computed(() => {
if (isFetchingPolicies.value) {
return "正在加载存储策略";
} else {
if (policyOptions.value.length <= 1) {
return "没有可用的存储策略请前往【附件】添加S3存储策略";
} else {
if (!policyName.value) {
return "请在左上方选择存储策略";
} else {
return "该存储策略的 桶/文件夹 下没有文件";
}
}
}
});
const handleCheckAllChange = (e: Event) => {
const {checked} = e.target as HTMLInputElement;
if (checked) {
selectedFiles.value =
s3Objects.value.objects?.filter(file => !file.isLinked).map((file) => {
return file.key || "";
}) || [];
} else {
selectedFiles.value.length = 0;
checkedAll.value = false;
}
};
const fetchPolicies = async () => {
try {
const policiesData = await getApisS3OsHaloRunV1Alpha1PoliciesS3();
if (policiesData.status == 200) {
policyOptions.value = [{
label: "请选择存储策略",
value: "",
attrs: {disabled: true}
}];
policiesData.data.forEach((policy: Policy) => {
policyOptions.value.push({
label: policy.spec.displayName,
value: policy.metadata.name,
attrs: {}
});
});
}
} catch (error) {
console.error(error);
}
isFetchingPolicies.value = false;
};
onMounted(() => {
fetchPolicies();
});
watch(selectedFiles, (newValue) => {
checkedAll.value = s3Objects.value.objects?.filter(file => !file.isLinked)
.filter(file => !newValue.includes(file.key || "")).length == 0
&& s3Objects.value.objects?.length != 0;
});
watch(selectedLinkedStatusItem, () => {
handleFirstPage();
});
const changeNextTokenAndObject = () => {
s3Objects.value.currentToken = s3Objects.value.nextToken;
s3Objects.value.currentContinuationObject = s3Objects.value.nextContinuationObject;
s3Objects.value.nextToken = "";
s3Objects.value.nextContinuationObject = "";
};
const clearTokenAndObject = () => {
s3Objects.value.currentToken = "";
s3Objects.value.currentContinuationObject = "";
s3Objects.value.nextToken = "";
s3Objects.value.nextContinuationObject = "";
};
// filePrefix will not be updated from user input
// if you want to update filePrefix, please call `handleFirstPage`
const fetchObjects = async () => {
if (!policyName.value) {
return;
}
isFetching.value = true;
s3Objects.value.objects = [];
try {
const objectsData = await getApisS3OsHaloRunV1Alpha1ObjectsByPolicyName({
policyName: policyName.value,
pageSize: size.value,
continuationToken: s3Objects.value.currentToken,
continuationObject: s3Objects.value.currentContinuationObject,
unlinked: selectedLinkedStatusItem.value,
filePrefix: filePrefix.value
});
if (objectsData.status == 200) {
s3Objects.value = objectsData.data;
if (s3Objects.value.objects?.length == 0 && s3Objects.value.hasMore && s3Objects.value.nextToken) {
changeNextTokenAndObject();
await fetchObjects();
} else if (s3Objects.value.objects?.length == 0 && !s3Objects.value.hasMore && page.value > 1) {
page.value = 1;
clearTokenAndObject();
await fetchObjects();
Toast.warning("最后一页为空,已返回第一页");
}
}
} catch (error) {
console.error(error);
}
selectedFiles.value.length = 0;
checkedAll.value = false;
isFetching.value = false;
};
const checkSelection = (file: ObjectVo) => {
return selectedFiles.value.includes(file.key || "");
};
const handleLink = async () => {
isLinking.value = true;
isShowModal.value = true;
linkTips.value = `正在关联${selectedFiles.value.length}个文件`;
linkFailedTable.value = [];
const linkResult = await postApisS3OsHaloRunV1Alpha1AttachmentsLink({
policyName: policyName.value,
objectKeys: selectedFiles.value
});
const successCount = linkResult.data.items.filter(item => item.success).length;
const failedCount = linkResult.data.items.filter(item => !item.success).length;
linkTips.value = `关联成功${successCount}个文件,关联失败${failedCount}个文件`;
if (failedCount > 0) {
linkFailedTable.value = linkResult.data.items.filter(item => !item.success);
}
isLinking.value = false;
};
const selectOneAndLink = (file: ObjectVo) => {
selectedFiles.value = [file.key || ""];
handleLink();
};
const handleNextPage = () => {
if (!policyName.value) {
return;
}
if (s3Objects.value.hasMore) {
isFetching.value = true;
page.value += 1;
changeNextTokenAndObject();
fetchObjects();
}
};
const handleFirstPage = () => {
if (!policyName.value) {
return;
}
isFetching.value = true;
page.value = 1;
clearTokenAndObject();
filePrefix.value = filePrefixBind.value;
fetchObjects();
};
const handleModalClose = () => {
isShowModal.value = false;
fetchObjects();
};
</script>
<template>
<VPageHeader title="关联S3文件(Beta)">
<template #icon>
<CarbonFolderDetailsReference class="mr-2 self-center"/>
</template>
</VPageHeader>
<div class="m-0 md:m-4">
<VCard :body-class="['!p-0']">
<template #header>
<div class="block w-full bg-gray-50 px-4 py-3">
<div
class="relative flex flex-col flex-wrap items-start gap-4 sm:flex-row sm:items-center"
>
<div class="hidden items-center sm:flex">
<input
v-model="checkedAll"
class="h-4 w-4 rounded border-gray-300 text-indigo-600"
type="checkbox"
@change="handleCheckAllChange"
/>
</div>
<div class="flex w-full flex-1 items-center sm:w-auto">
<div
v-if="!selectedFiles.length"
class="flex flex-wrap items-center gap-2"
>
<span class="whitespace-nowrap">存储策略:</span>
<FormKit
id="policyChoose"
outer-class="!p-0"
style="min-width: 10rem;"
v-model="policyName"
name="policyName"
type="select"
:options="policyOptions"
@change="handleFirstPage"
></FormKit>
<icon-error-warning v-if="!policyName" class="text-red-500"/>
<SearchInput
v-model="filePrefixBind"
v-if="policyName"
placeholder="请输入文件名前缀搜索"
@update:modelValue="handleFirstPage"
></SearchInput>
</div>
<VSpace v-else>
<VButton type="primary" @click="handleLink">
关联
</VButton>
</VSpace>
</div>
<VSpace spacing="lg" class="flex-wrap">
<FilterCleanButton
v-if="selectedLinkedStatusItem != linkedStatusItems[0].value"
@click="selectedLinkedStatusItem = linkedStatusItems[0].value"
/>
<FilterDropdown
v-model="selectedLinkedStatusItem"
:label="$t('core.common.filters.labels.status')"
:items="linkedStatusItems"
/>
<div class="flex flex-row gap-2">
<div
class="group cursor-pointer rounded p-1 hover:bg-gray-200"
@click="fetchObjects()"
>
<IconRefreshLine
v-tooltip="$t('core.common.buttons.refresh')"
:class="{
'animate-spin text-gray-900': isFetching,
}"
class="h-4 w-4 text-gray-600 group-hover:text-gray-900"
/>
</div>
</div>
</VSpace>
</div>
</div>
</template>
<VLoading v-if="isFetching"/>
<Transition v-else-if="!s3Objects.objects?.length" appear name="fade">
<VEmpty
message="空空如也"
:title="emptyTips"
>
</VEmpty>
</Transition>
<Transition v-else appear name="fade">
<ul
class="box-border h-full w-full divide-y divide-gray-100"
role="list"
>
<li v-for="(file, index) in s3Objects.objects" :key="index">
<VEntity :is-selected="checkSelection(file)">
<template
#checkbox
>
<input
v-model="selectedFiles"
:value="file.key || ''"
class="h-4 w-4 rounded border-gray-300 text-indigo-600"
name="post-checkbox"
:disabled="file.isLinked"
type="checkbox"
/>
</template>
<template #start>
<VEntityField>
<template #description>
<AttachmentFileTypeIcon
:display-ext="false"
:file-name="file.displayName || ''"
:width="8"
:height="8"
/>
</template>
</VEntityField>
<VEntityField
:title="file.displayName || ''"
:description="file.key || ''"
/>
</template>
<template #end>
<VEntityField>
<template #description>
<VTag :theme="file.isLinked ? 'default':'primary'">
{{
file.isLinked ? '已关联' : '未关联'
}}
</VTag>
</template>
</VEntityField>
<VEntityField>
<template #description>
<VButton
:disabled="file.isLinked || false"
@click="selectOneAndLink(file)"
>
关联
</VButton>
</template>
</VEntityField>
</template>
</VEntity>
</li>
</ul>
</Transition>
<template #footer>
<div class="bg-white sm:flex sm:items-center justify-between">
<div class="inline-flex items-center gap-5">
<span class="text-xs text-gray-500 hidden md:flex"> {{ s3Objects.objects?.length }} 项数据</span>
<span class="text-xs text-gray-500 hidden md:flex">已自动过滤文件夹对象页面实际显示数量少为正常现象</span>
</div>
<div class="inline-flex items-center gap-5">
<div class="inline-flex items-center gap-2">
<VButton size="small" @click="handleFirstPage" :disabled="!policyName">返回第一页</VButton>
<span class="text-sm text-gray-500"> {{ page }} </span>
<VButton size="small" @click="handleNextPage" :disabled="!s3Objects.hasMore || isFetching || !policyName">
下一页
</VButton>
</div>
<div class="inline-flex items-center gap-2">
<select
v-model="size"
class="h-8 border outline-none rounded-base pr-10 border-solid px-2 text-gray-800 text-sm border-gray-300 page-size-select"
@change="handleFirstPage"
>
<option
v-for="(sizeOption, index) in [20, 50, 100, 200]"
:key="index"
:value="sizeOption"
>
{{ sizeOption }}
</option>
</select>
<span class="text-sm text-gray-500">/</span>
</div>
</div>
</div>
</template>
</VCard>
</div>
<VModal
:visible="isShowModal"
:fullscreen="false"
:title="'关联结果'"
:width="500"
:mount-to-body="true"
@close="handleModalClose"
>
<template #footer>
<VSpace>
<VButton
:loading="isLinking"
type="primary"
@click="handleModalClose"
>
确定
</VButton>
</VSpace>
</template>
<div class="flex flex-col">
{{ linkTips }}
<table v-if="linkFailedTable.length != 0">
<tr>
<th class="border border-black font-normal">失败对象</th>
<th class="border border-black font-normal">失败原因</th>
</tr>
<tr v-for="failedInfo in linkFailedTable" :key="failedInfo.objectKey">
<th class="border border-black font-normal">{{failedInfo.objectKey}}</th>
<th class="border border-black font-normal">{{failedInfo.message}}</th>
</tr>
</table>
</div>
</VModal>
</template>
<style lang="scss" scoped>
.page-size-select:focus {
--tw-border-opacity: 1;
border-color: rgba(var(--colors-primary),var(--tw-border-opacity));
}
</style>

View File

@@ -0,0 +1,37 @@
<script setup lang="ts">
import {deleteApisS3OsHaloRunV1Alpha1AttachmentsByName} from "@/controller";
import type {Attachment} from "@halo-dev/api-client";
import {Dialog, Toast, VDropdownDivider, VDropdownItem} from "@halo-dev/components";
import {useQueryClient} from "@tanstack/vue-query";
const props = defineProps<{
attachment: Attachment;
}>();
const queryClient = useQueryClient();
const handleUnlink = () => {
Dialog.warning({
title: "解除 S3 关联",
description: "解除关联后,附件中的记录将会被删除,而对象存储中的文件仍然保留,若需重新关联请使用“关联 S3 文件”功能。",
confirmType: "danger",
confirmText: "确定",
cancelText: "取消",
onConfirm: async () => {
try {
await deleteApisS3OsHaloRunV1Alpha1AttachmentsByName({name: props.attachment.metadata.name});
Toast.success("解除关联成功");
} catch (e) {
console.error("Failed to delete attachment", e);
} finally {
queryClient.invalidateQueries({queryKey: ["attachments"]});
}
},
});
}
</script>
<template>
<div>
<VDropdownDivider/>
<VDropdownItem type="danger" @click="handleUnlink">解除 S3 关联</VDropdownItem>
</div>
</template>

13
console/tsconfig.app.json Normal file
View File

@@ -0,0 +1,13 @@
{
"extends": "@vue/tsconfig/tsconfig.web.json",
"include": ["./env.d.ts", "./src/**/*", "./src/**/*.vue"],
"exclude": ["./src/**/__tests__/*"],
"compilerOptions": {
"composite": true,
"baseUrl": ".",
"paths": {
"@/*": ["./src/*"]
},
"types": ["unplugin-icons/types/vue"]
}
}

View File

@@ -0,0 +1,8 @@
{
"extends": "@vue/tsconfig/tsconfig.node.json",
"include": ["vite.config.*", "vitest.config.*", "cypress.config.*"],
"compilerOptions": {
"composite": true,
"types": ["node"]
}
}

14
console/tsconfig.json Normal file
View File

@@ -0,0 +1,14 @@
{
"files": [],
"references": [
{
"path": "./tsconfig.config.json"
},
{
"path": "./tsconfig.app.json"
},
{
"path": "./tsconfig.vitest.json"
}
]
}

View File

@@ -0,0 +1,9 @@
{
"extends": "./tsconfig.app.json",
"exclude": [],
"compilerOptions": {
"composite": true,
"lib": [],
"types": ["node", "jsdom"]
}
}

21
console/vite.config.ts Normal file
View File

@@ -0,0 +1,21 @@
import { fileURLToPath, URL } from "url";
import { defineConfig } from "vite";
import Vue from "@vitejs/plugin-vue";
import VueJsx from "@vitejs/plugin-vue-jsx";
import Icons from "unplugin-icons/vite";
import { HaloUIPluginBundlerKit } from "@halo-dev/ui-plugin-bundler-kit";
export default defineConfig({
plugins: [
Vue(),
VueJsx(),
Icons({ compiler: "vue3" }),
HaloUIPluginBundlerKit(),
],
resolve: {
alias: {
"@": fileURLToPath(new URL("./src", import.meta.url)),
},
},
});

View File

@@ -1 +1 @@
version=1.4.0-SNAPSHOT
version=1.9.0-SNAPSHOT

View File

@@ -1,44 +1,143 @@
package run.halo.s3os;
import static run.halo.s3os.S3OsProperties.DuplicateFilenameHandling;
import static run.halo.s3os.S3OsProperties.RandomFilenameMode;
import com.google.common.io.Files;
import org.apache.commons.lang3.RandomStringUtils;
import org.apache.commons.lang3.StringUtils;
import org.springframework.web.server.ServerWebInputException;
public final class FileNameUtils {
private FileNameUtils() {
}
public static String removeFileExtension(String filename, boolean removeAllExtensions) {
if (filename == null || filename.isEmpty()) {
return filename;
}
var extPattern = "(?<!^)[.]" + (removeAllExtensions ? ".*" : "[^.]*$");
return filename.replaceAll(extPattern, "");
/**
* Replace placeholders in filename. No duplicate handling.
*
* @param filename filename
* @param mode random filename mode
* @param randomStringLength random string length,when mode is withString or string
* @param customTemplate custom template,when mode is custom
* @return replaced filename
*/
public static String replaceFilename(String filename, RandomFilenameMode mode,
Integer randomStringLength, String customTemplate) {
var extension = Files.getFileExtension(filename);
var filenameWithoutExtension = Files.getNameWithoutExtension(filename);
var replaced = replaceFilenameByMode(filenameWithoutExtension, mode, randomStringLength,
customTemplate);
return replaced + (StringUtils.isBlank(extension) ? "" : "." + extension);
}
/**
* Append random string after file name.
* Replace placeholders in filename with duplicate handling.
* <pre>
* Case 1: halo.run -> halo-xyz.run
* Case 2: .run -> xyz.run
* Case 3: halo -> halo-xyz
* </pre>
*
* @param filename is name of file.
* @param length is for generating random string with specific length.
* @return File name with random string.
* @param filename filename
* @param mode random filename mode
* @param randomStringLength random string length,when mode is withString or string
* @param customTemplate custom template,when mode is custom
* @param handling duplicate filename handling
* @return replaced filename
*/
public static String randomFileName(String filename, int length) {
var nameWithoutExt = Files.getNameWithoutExtension(filename);
var ext = Files.getFileExtension(filename);
var random = RandomStringUtils.randomAlphabetic(length).toLowerCase();
if (StringUtils.isBlank(nameWithoutExt)) {
return random + "." + ext;
public static String replaceFilenameWithDuplicateHandling(String filename,
RandomFilenameMode mode,
Integer randomStringLength,
String customTemplate,
DuplicateFilenameHandling handling) {
var extension = Files.getFileExtension(filename);
var filenameWithoutExtension = Files.getNameWithoutExtension(filename);
var replaced =
replaceFilenameByMode(filenameWithoutExtension, mode, randomStringLength,
customTemplate);
var suffix = getDuplicateFilenameSuffix(handling);
return replaced + (StringUtils.isBlank(replaced) ? "" : "-") + suffix
+ (StringUtils.isBlank(extension) ? "" : "." + extension);
}
private static String getDuplicateFilenameSuffix(
S3OsProperties.DuplicateFilenameHandling duplicateFilenameHandling) {
if (duplicateFilenameHandling == null) {
return RandomStringUtils.randomAlphabetic(4).toLowerCase();
}
if (StringUtils.isBlank(ext)) {
return nameWithoutExt + "-" + random;
return switch (duplicateFilenameHandling) {
case randomAlphabetic -> RandomStringUtils.randomAlphabetic(4).toLowerCase();
case exception -> throw new ServerWebInputException("Duplicate filename");
// include "randomAlphanumeric" mode
default -> RandomStringUtils.randomAlphanumeric(4).toLowerCase();
};
}
private static String replaceFilenameByMode(String filenameWithoutExtension,
S3OsProperties.RandomFilenameMode mode,
Integer randomStringLength,
String customTemplate) {
if (mode == null) {
return filenameWithoutExtension;
}
return nameWithoutExt + "-" + random + "." + ext;
// default length is 8
Integer length = randomStringLength == null ? 8 : randomStringLength;
return switch (mode) {
case custom -> {
if (StringUtils.isBlank(customTemplate)) {
yield filenameWithoutExtension;
}
yield PlaceholderReplacer.replacePlaceholders(customTemplate,
filenameWithoutExtension);
}
case uuid -> PlaceholderReplacer.replacePlaceholders("${uuid-with-dash}",
filenameWithoutExtension);
case timestampMs -> PlaceholderReplacer.replacePlaceholders("${timestamp-ms}",
filenameWithoutExtension);
case dateWithString -> {
String dateWithStringTemplate =
String.format("${year}-${month}-${day}-${random-alphabetic:%d}", length);
yield PlaceholderReplacer.replacePlaceholders(dateWithStringTemplate,
filenameWithoutExtension);
}
case datetimeWithString -> {
String datetimeWithStringTemplate = String.format(
"${year}-${month}-${day}T${hour}:${minute}:${second}-${random-alphabetic:%d}",
length);
yield PlaceholderReplacer.replacePlaceholders(datetimeWithStringTemplate,
filenameWithoutExtension);
}
case withString -> {
String withStringTemplate =
String.format("${origin-filename}-${random-alphabetic:%d}", length);
yield PlaceholderReplacer.replacePlaceholders(withStringTemplate,
filenameWithoutExtension);
}
case string -> {
String stringTemplate = String.format("${random-alphabetic:%d}", length);
yield PlaceholderReplacer.replacePlaceholders(stringTemplate,
filenameWithoutExtension);
}
default ->
// include "none" mode
filenameWithoutExtension;
};
}
/**
* Extracts the file name from an Amazon S3 object key.
*
* @param objectKey The Amazon S3 object key from which to extract the file name.
* @return The extracted file name.
*/
public static String extractFileNameFromS3Key(String objectKey) {
int lastSlashIndex = objectKey.lastIndexOf("/");
if (lastSlashIndex >= 0 && lastSlashIndex < objectKey.length() - 1) {
return objectKey.substring(lastSlashIndex + 1);
}
return objectKey;
}
}

View File

@@ -0,0 +1,11 @@
package run.halo.s3os;
import lombok.experimental.UtilityClass;
@UtilityClass
public class FilePathUtils {
public static String getFilePathByPlaceholder(String filePath) {
return PlaceholderReplacer.replacePlaceholders(filePath, null);
}
}

View File

@@ -0,0 +1,12 @@
package run.halo.s3os;
import java.util.Set;
import lombok.Data;
import lombok.RequiredArgsConstructor;
@Data
@RequiredArgsConstructor
public class LinkRequest {
private String policyName;
private Set<String> objectKeys;
}

View File

@@ -0,0 +1,24 @@
package run.halo.s3os;
import lombok.AllArgsConstructor;
import lombok.Data;
import java.util.List;
@Data
@AllArgsConstructor
public class LinkResult {
private List<LinkResultItem> items;
@Data
@AllArgsConstructor
public static class LinkResultItem {
private String objectKey;
private Boolean success;
private String message;
}
}

View File

@@ -0,0 +1,184 @@
package run.halo.s3os;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import java.util.function.Function;
import lombok.experimental.UtilityClass;
import org.apache.commons.lang3.RandomStringUtils;
import org.apache.commons.lang3.StringUtils;
import org.springframework.util.PropertyPlaceholderHelper;
@UtilityClass
public class PlaceholderReplacer {
record PlaceholderFunctionInput(String[] placeholderParams,
Map<String, String> reusableParams) {
}
private static final PropertyPlaceholderHelper helper = new PropertyPlaceholderHelper("${", "}");
private static final Map<String, Function<PlaceholderFunctionInput, String>>
placeholderFunctions = new HashMap<>();
static {
initializePlaceholderFunctions();
}
private static void initializePlaceholderFunctions() {
placeholderFunctions.put("origin-filename", input -> input.reusableParams.get("filename"));
placeholderFunctions.put("uuid-with-dash", input -> generateUUIDWithDash());
placeholderFunctions.put("uuid-no-dash", input -> generateUUIDWithoutDash());
placeholderFunctions.put("timestamp-sec",
input -> currentSecondsTimestamp(input.reusableParams));
placeholderFunctions.put("timestamp-ms",
input -> currentMillisecondsTimestamp(input.reusableParams));
placeholderFunctions.put("year", input -> currentYear(input.reusableParams));
placeholderFunctions.put("month", input -> currentMonth(input.reusableParams));
placeholderFunctions.put("day", input -> currentDay(input.reusableParams));
placeholderFunctions.put("weekday", input -> currentWeekday(input.reusableParams));
placeholderFunctions.put("hour", input -> currentHour(input.reusableParams));
placeholderFunctions.put("minute", input -> currentMinute(input.reusableParams));
placeholderFunctions.put("second", input -> currentSecond(input.reusableParams));
placeholderFunctions.put("millisecond", input -> currentMillisecond(input.reusableParams));
placeholderFunctions.put("random-alphabetic",
input -> generateRandomLetter(input.placeholderParams));
placeholderFunctions.put("random-num",
input -> generateRandomNumber(input.placeholderParams));
placeholderFunctions.put("random-alphanumeric",
input -> generateRandomAlphanumeric(input.placeholderParams));
}
private static String generateRandomAlphanumeric(String[] placeholderParams) {
try {
int length = Integer.parseInt(placeholderParams[0]);
return RandomStringUtils.randomAlphanumeric(length > 0 ? length : 8).toLowerCase();
} catch (NumberFormatException | ArrayIndexOutOfBoundsException e) {
return RandomStringUtils.randomAlphanumeric(8).toLowerCase();
}
}
private static String generateRandomNumber(String[] placeholderParams) {
try {
int length = Integer.parseInt(placeholderParams[0]);
return RandomStringUtils.randomNumeric(length > 0 ? length : 8);
} catch (NumberFormatException | ArrayIndexOutOfBoundsException e) {
return RandomStringUtils.randomNumeric(8);
}
}
private static String currentMillisecond(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%03d", time.getNano() / 1000000);
}
private static String currentSecond(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%02d", time.getSecond());
}
private static String currentMinute(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%02d", time.getMinute());
}
private static String currentHour(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%02d", time.getHour());
}
private static String currentWeekday(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.valueOf(time.getDayOfWeek().getValue());
}
private static String currentDay(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%02d", time.getDayOfMonth());
}
private static String currentMonth(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.format("%02d", time.getMonthValue());
}
private static String currentYear(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.valueOf(time.getYear());
}
private static String currentMillisecondsTimestamp(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.valueOf(
time.toInstant(ZoneOffset.systemDefault().getRules().getOffset(time)).toEpochMilli());
}
private static String currentSecondsTimestamp(Map<String, String> reusableParams) {
LocalDateTime time = LocalDateTime.parse(reusableParams.get("time"));
return String.valueOf(
time.toInstant(ZoneOffset.systemDefault().getRules().getOffset(time)).getEpochSecond());
}
private static String generateRandomLetter(String[] placeholderParams) {
try {
int length = Integer.parseInt(placeholderParams[0]);
return RandomStringUtils.randomAlphabetic(length > 0 ? length : 8).toLowerCase();
} catch (NumberFormatException | ArrayIndexOutOfBoundsException e) {
return RandomStringUtils.randomAlphabetic(8).toLowerCase();
}
}
private static String generateUUIDWithoutDash() {
return UUID.randomUUID().toString().replace("-", "").toUpperCase();
}
private static String generateUUIDWithDash() {
return UUID.randomUUID().toString().toUpperCase();
}
/**
* Replace placeholders in the template with the provided filename.
*
* @param template The template to replace
* @param filename The filename without extension
* @return The replaced string
*/
public static String replacePlaceholders(String template, String filename) {
if (StringUtils.isBlank(template)) {
return filename;
}
Map<String, String> reusableParams = new HashMap<>();
reusableParams.put("filename", filename);
reusableParams.put("time", LocalDateTime.now().toString());
return helper.replacePlaceholders(template, placeholder -> getPlaceholderValue(placeholder, reusableParams));
}
private static String getPlaceholderValue(String placeholderWithParam,
Map<String, String> reusableParams) {
String[] parts = placeholderWithParam.split(":");
String placeholder = parts[0];
String[] placeholderParams;
if (parts.length > 1) {
placeholderParams = new String[parts.length - 1];
System.arraycopy(parts, 1, placeholderParams, 0, parts.length - 1);
} else {
placeholderParams = new String[0];
}
Function<PlaceholderFunctionInput, String> placeholderFunction =
placeholderFunctions.get(placeholder);
if (placeholderFunction != null) {
// Call the placeholder function with the provided map
return placeholderFunction.apply(
new PlaceholderFunctionInput(placeholderParams, reusableParams));
} else {
// If placeholder not found, return null to keep the original placeholder string
return null;
}
}
}

View File

@@ -0,0 +1,123 @@
package run.halo.s3os;
import static run.halo.s3os.S3OsAttachmentHandler.MULTIPART_MIN_PART_SIZE;
import static run.halo.s3os.S3OsAttachmentHandler.checkResult;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.Resource;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;
import org.springframework.core.io.buffer.DefaultDataBufferFactory;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import run.halo.app.infra.utils.PathUtils;
import run.halo.app.plugin.ApiVersion;
import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.utils.SdkAutoCloseable;
@ApiVersion("s3os.halo.run/v1alpha1")
@RestController
@RequiredArgsConstructor
@Slf4j
public class PolicyConfigValidationController {
private final S3OsAttachmentHandler handler;
@PostMapping("/policies/s3/validation")
public Mono<Void> validatePolicyConfig(@RequestBody S3OsProperties properties) {
var filename = "halo-s3-plugin-test-file-" + System.currentTimeMillis() + ".jpg";
var content = readImage();
return Mono.using(() -> handler.buildS3Client(properties),
client -> {
var uploadState =
new S3OsAttachmentHandler.UploadState(properties, filename, false);
return handler.checkFileExistsAndRename(uploadState, client)
// init multipart upload
.flatMap(state -> Mono.fromCallable(() -> client.createMultipartUpload(
CreateMultipartUploadRequest.builder()
.bucket(properties.getBucket())
.contentType(state.contentType)
.key(state.objectKey)
.build())))
.doOnNext((response) -> {
checkResult(response, "createMultipartUpload");
uploadState.uploadId = response.uploadId();
})
.thenMany(handler.reshape(content, MULTIPART_MIN_PART_SIZE))
// buffer to part
.windowUntil((buffer) -> {
uploadState.buffered += buffer.readableByteCount();
if (uploadState.buffered >= MULTIPART_MIN_PART_SIZE) {
uploadState.buffered = 0;
return true;
} else {
return false;
}
})
// upload part
.concatMap((window) -> window.collectList().flatMap((bufferList) -> {
var buffer = S3OsAttachmentHandler.concatBuffers(bufferList);
return handler.uploadPart(uploadState, buffer, client);
}))
.reduce(uploadState, (state, completedPart) -> {
state.completedParts.put(completedPart.partNumber(), completedPart);
return state;
})
// complete multipart upload
.flatMap((state) -> Mono.just(client.completeMultipartUpload(
CompleteMultipartUploadRequest
.builder()
.bucket(properties.getBucket())
.uploadId(state.uploadId)
.multipartUpload(CompletedMultipartUpload.builder()
.parts(state.completedParts.values())
.build())
.key(state.objectKey)
.build())
))
// get object metadata
.flatMap((response) -> {
checkResult(response, "completeUpload");
return Mono.just(client.headObject(
HeadObjectRequest.builder()
.bucket(properties.getBucket())
.key(uploadState.objectKey)
.build()
));
})
// check object metadata
.doOnNext((response) -> {
checkResult(response, "headObject");
})
// delete object
.flatMap((response) -> Mono.just(client.deleteObject(
software.amazon.awssdk.services.s3.model.DeleteObjectRequest.builder()
.bucket(properties.getBucket())
.key(uploadState.objectKey)
.build()
)))
.doOnNext((response) -> checkResult(response, "deleteObject"))
.then();
},
SdkAutoCloseable::close)
.onErrorMap(S3ExceptionHandler::map);
}
private Flux<DataBuffer> readImage() {
DefaultResourceLoader resourceLoader = new DefaultResourceLoader(this.getClass()
.getClassLoader());
String path = PathUtils.combinePath("validation.jpg");
String simplifyPath = StringUtils.cleanPath(path);
Resource resource = resourceLoader.getResource(simplifyPath);
return DataBufferUtils.read(resource, new DefaultDataBufferFactory(), 1024);
}
}

View File

@@ -0,0 +1,42 @@
package run.halo.s3os;
import lombok.experimental.UtilityClass;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.server.ServerWebInputException;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.model.S3Exception;
@Slf4j
@UtilityClass
public class S3ExceptionHandler {
/**
* Map user configuration caused S3 exception to ServerWebInputException
* @param throwable Exception
* @return ServerWebInputException or original exception
*/
public static Throwable map(Throwable throwable) {
if (throwable instanceof S3Exception s3e) {
log.error("S3Exception occurred", s3e);
return new ServerWebInputException(String.format(
"The object storage service returned an error status code %d. Please check the storage "
+ "policy configuration and make sure your account and service are working properly.",
s3e.statusCode()));
}
if (throwable instanceof SdkException sdke && sdke.getMessage() != null
&& sdke.getMessage().contains("UnknownHostException")) {
log.error("UnknownHostException occurred", sdke);
return new ServerWebInputException(
"Received an UnknownHostException, please check if the endpoint is entered correctly, "
+ "especially for any spaces before or after the endpoint.");
}
if (throwable instanceof SdkException sdke && sdke.getMessage() != null
&& sdke.getMessage().contains("Connect timed out")) {
log.error("ConnectTimeoutException occurred", sdke);
return new ServerWebInputException(
"Received a ConnectTimeoutException, please check if the endpoint is entered correctly, "
+ "and make sure your object storage service is working properly.");
}
return throwable;
}
}

View File

@@ -0,0 +1,46 @@
package run.halo.s3os;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import run.halo.app.core.extension.attachment.Policy;
import run.halo.app.plugin.ApiVersion;
@ApiVersion("s3os.halo.run/v1alpha1")
@RestController
@RequiredArgsConstructor
public class S3LinkController {
private final S3LinkService s3LinkService;
@GetMapping("/policies/s3")
public Flux<Policy> listS3Policies() {
return s3LinkService.listS3Policies();
}
@GetMapping("/objects/{policyName}")
public Mono<S3ListResult> listObjects(@PathVariable(value = "policyName") String policyName,
@RequestParam(name = "continuationToken", required = false) String continuationToken,
@RequestParam(name = "continuationObject", required = false) String continuationObject,
@RequestParam(name = "pageSize") Integer pageSize,
@RequestParam(name = "unlinked", required = false, defaultValue = "false") Boolean unlinked,
@RequestParam(name = "filePrefix", required = false) String filePrefix) {
if (unlinked) {
return s3LinkService.listObjectsUnlinked(policyName, continuationToken,
continuationObject, pageSize, filePrefix);
} else {
return s3LinkService.listObjects(policyName, continuationToken, pageSize, filePrefix);
}
}
@PostMapping("/attachments/link")
public Mono<LinkResult> addAttachmentRecord(@RequestBody LinkRequest linkRequest) {
return s3LinkService.addAttachmentRecords(linkRequest.getPolicyName(),
linkRequest.getObjectKeys());
}
}

View File

@@ -0,0 +1,19 @@
package run.halo.s3os;
import java.util.Set;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import run.halo.app.core.extension.attachment.Policy;
public interface S3LinkService {
Flux<Policy> listS3Policies();
Mono<S3ListResult> listObjects(String policyName, String continuationToken,
Integer pageSize, String filePrefix);
Mono<LinkResult> addAttachmentRecords(String policyName, Set<String> objectKeys);
Mono<S3ListResult> listObjectsUnlinked(String policyName, String continuationToken,
String continuationObject, Integer pageSize, String filePrefix);
}

View File

@@ -0,0 +1,289 @@
package run.halo.s3os;
import static run.halo.s3os.S3OsAttachmentHandler.OBJECT_KEY;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
import java.util.stream.Collectors;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.data.domain.Sort;
import org.springframework.http.HttpStatus;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.ReactiveSecurityContextHolder;
import org.springframework.security.core.context.SecurityContext;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import run.halo.app.core.extension.attachment.Attachment;
import run.halo.app.core.extension.attachment.Policy;
import run.halo.app.extension.ConfigMap;
import run.halo.app.extension.ListOptions;
import run.halo.app.extension.MetadataUtil;
import run.halo.app.extension.ReactiveExtensionClient;
import run.halo.app.extension.index.query.QueryFactory;
import run.halo.app.extension.router.selector.FieldSelector;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.S3Object;
@Service
@RequiredArgsConstructor
@Slf4j
public class S3LinkServiceImpl implements S3LinkService {
private final ReactiveExtensionClient client;
private final S3OsAttachmentHandler handler;
/**
* Map of linking file, used as a lock, key is policyName/objectKey, value is policyName/objectKey.
*/
private final Map<String, Object> linkingFile = new ConcurrentHashMap<>();
@Override
public Flux<Policy> listS3Policies() {
return client.list(Policy.class, (policy) -> "s3os".equals(
policy.getSpec().getTemplateName()), Comparator.naturalOrder());
}
@Override
public Mono<S3ListResult> listObjects(String policyName, String continuationToken,
Integer pageSize, String filePrefix) {
return client.fetch(Policy.class, policyName)
.flatMap((policy) -> {
var configMapName = policy.getSpec().getConfigMapName();
return client.fetch(ConfigMap.class, configMapName);
})
.flatMap((configMap) -> {
var properties = handler.getProperties(configMap);
var finalLocation = FilePathUtils.getFilePathByPlaceholder(properties.getLocation());
return Mono.using(() -> handler.buildS3Client(properties),
// 执行 listObjects
(s3Client) -> Mono.fromCallable(
() -> s3Client.listObjectsV2(ListObjectsV2Request.builder()
.bucket(properties.getBucket())
.prefix(buildPrefix(finalLocation, filePrefix))
.delimiter("/")
.maxKeys(pageSize)
.continuationToken(StringUtils.isNotEmpty(continuationToken)
? continuationToken : null)
.build())).subscribeOn(Schedulers.boundedElastic()),
S3Client::close)
.flatMap(listObjectsV2Response -> {
List<S3Object> contents = listObjectsV2Response.contents();
// 过滤掉目录并转换为ObjectVo
var objectVos = contents
.stream().map(S3ListResult.ObjectVo::fromS3Object)
.filter(objectVo -> !objectVo.getKey().endsWith("/"))
.collect(Collectors.toMap(S3ListResult.ObjectVo::getKey, o -> o));
// 获取已经关联的附件并标记
ListOptions listOptions = new ListOptions();
listOptions.setFieldSelector(
FieldSelector.of(QueryFactory.equal("spec.policyName", policyName)));
return client.listAll(Attachment.class, listOptions, Sort.unsorted())
.doOnNext(attachment -> {
S3ListResult.ObjectVo objectVo =
objectVos.get(attachment.getMetadata().getAnnotations()
.getOrDefault(OBJECT_KEY, ""));
if (objectVo != null) {
objectVo.setIsLinked(true);
}
})
.then()
.thenReturn(new S3ListResult(new ArrayList<>(objectVos.values()),
listObjectsV2Response.continuationToken(),
null, null,
listObjectsV2Response.nextContinuationToken(),
listObjectsV2Response.isTruncated()));
});
})
.onErrorMap(S3ExceptionHandler::map);
}
@Override
public Mono<LinkResult> addAttachmentRecords(String policyName, Set<String> objectKeys) {
return getOperableObjectKeys(objectKeys, policyName)
.flatMap(operableObjectKeys -> getExistingAttachments(objectKeys, policyName)
.flatMap(existingAttachments -> getLinkResultItems(objectKeys, operableObjectKeys,
existingAttachments, policyName)
.collectList()
.map(LinkResult::new)));
}
private Mono<Set<String>> getOperableObjectKeys(Set<String> objectKeys, String policyName) {
return Flux.fromIterable(objectKeys)
.filter(objectKey ->
linkingFile.put(policyName + "/" + objectKey, policyName + "/" + objectKey) == null)
.collect(Collectors.toSet());
}
private Mono<Set<String>> getExistingAttachments(Set<String> objectKeys,
String policyName) {
ListOptions listOptions = new ListOptions();
listOptions.setFieldSelector(
FieldSelector.of(QueryFactory.equal("spec.policyName", policyName)));
return client.listAll(Attachment.class, listOptions, Sort.unsorted())
.filter(attachment -> StringUtils.isNotBlank(
MetadataUtil.nullSafeAnnotations(attachment).get(S3OsAttachmentHandler.OBJECT_KEY))
&& objectKeys.contains(
MetadataUtil.nullSafeAnnotations(attachment).get(S3OsAttachmentHandler.OBJECT_KEY)))
.map(attachment -> MetadataUtil.nullSafeAnnotations(attachment)
.get(S3OsAttachmentHandler.OBJECT_KEY))
.collect(Collectors.toSet());
}
private Flux<LinkResult.LinkResultItem> getLinkResultItems(Set<String> objectKeys,
Set<String> operableObjectKeys,
Set<String> existingAttachments,
String policyName) {
return Flux.fromIterable(objectKeys)
.flatMap((objectKey) -> {
if (operableObjectKeys.contains(objectKey) &&
!existingAttachments.contains(objectKey)) {
return addAttachmentRecord(policyName, objectKey)
.onErrorResume((throwable) -> Mono.just(
new LinkResult.LinkResultItem(objectKey, false,
throwable.getMessage())));
} else {
return Mono.just(
new LinkResult.LinkResultItem(objectKey, false, "附件库中已存在该对象"));
}
})
.doFinally(signalType -> operableObjectKeys.forEach(
objectKey -> linkingFile.remove(policyName + "/" + objectKey)));
}
@Override
public Mono<S3ListResult> listObjectsUnlinked(String policyName, String continuationToken,
String continuationObject, Integer pageSize, String filePrefix) {
// TODO 优化成查一次数据库
return Mono.defer(() -> {
List<S3ListResult.ObjectVo> s3Objects = new ArrayList<>();
AtomicBoolean continuationObjectMatched = new AtomicBoolean(false);
AtomicReference<String> currToken = new AtomicReference<>(continuationToken);
return Flux.defer(() -> Flux.just(
new TokenState(null, currToken.get() == null ? "" : currToken.get())))
.flatMap(tokenState -> listObjects(policyName, tokenState.nextToken,
pageSize, filePrefix))
.flatMap(s3ListResult -> {
var filteredObjects = s3ListResult.getObjects();
if (!continuationObjectMatched.get()) {
// 判断s3ListResult.getObjects()里是否有continuationObject
var continuationObjectVo = s3ListResult.getObjects().stream()
.filter(objectVo -> objectVo.getKey().equals(continuationObject))
.findFirst();
if (continuationObjectVo.isPresent()) {
s3Objects.clear();
// 删除continuationObject及之前的所有对象
filteredObjects = s3ListResult.getObjects().stream()
.dropWhile(objectVo -> !objectVo.getKey()
.equals(continuationObject))
.skip(1)
.toList();
continuationObjectMatched.set(true);
}
}
filteredObjects = filteredObjects.stream()
.filter(objectVo -> !objectVo.getIsLinked())
.toList();
s3Objects.addAll(filteredObjects);
currToken.set(s3ListResult.getNextToken());
return Mono.just(new TokenState(s3ListResult.getCurrentToken(),
s3ListResult.getNextToken()));
})
.repeat()
.takeUntil(
tokenState -> tokenState.nextToken() == null || s3Objects.size() >= pageSize)
.last()
.map(tokenState -> {
var limitedObjects = s3Objects.stream().limit(pageSize).toList();
return new S3ListResult(limitedObjects, continuationToken, continuationObject,
!limitedObjects.isEmpty() ? limitedObjects.get(limitedObjects.size() - 1)
.getKey() : null, tokenState.currToken,
limitedObjects.size() == pageSize);
});
})
.onErrorMap(S3ExceptionHandler::map);
}
record TokenState(String currToken, String nextToken) {
}
public Mono<LinkResult.LinkResultItem> addAttachmentRecord(String policyName,
String objectKey) {
return authenticationConsumer(authentication -> client.fetch(Policy.class, policyName)
.flatMap((policy) -> {
var configMapName = policy.getSpec().getConfigMapName();
return client.fetch(ConfigMap.class, configMapName);
})
.flatMap(configMap -> {
var properties = handler.getProperties(configMap);
return Mono.using(() -> handler.buildS3Client(properties),
(s3Client) -> Mono.fromCallable(
() -> s3Client.headObject(
HeadObjectRequest.builder()
.bucket(properties.getBucket())
.key(objectKey)
.build()))
.subscribeOn(Schedulers.boundedElastic()),
S3Client::close)
.map(headObjectResponse -> {
var objectDetail = new S3OsAttachmentHandler.ObjectDetail(
new S3OsAttachmentHandler.UploadState(properties,
FileNameUtils.extractFileNameFromS3Key(objectKey), false),
headObjectResponse);
return handler.buildAttachment(properties, objectDetail);
})
.doOnNext(attachment -> {
var spec = attachment.getSpec();
if (spec == null) {
spec = new Attachment.AttachmentSpec();
attachment.setSpec(spec);
}
spec.setOwnerName(authentication.getName());
spec.setPolicyName(policyName);
})
.flatMap(client::create)
.thenReturn(new LinkResult.LinkResultItem(objectKey, true, null));
}))
.onErrorMap(S3ExceptionHandler::map)
.onErrorResume(throwable ->
Mono.just(new LinkResult.LinkResultItem(objectKey, false, throwable.getMessage())));
}
private <T> Mono<T> authenticationConsumer(Function<Authentication, Mono<T>> func) {
return ReactiveSecurityContextHolder.getContext()
.switchIfEmpty(Mono.error(() -> new ResponseStatusException(HttpStatus.UNAUTHORIZED,
"Authentication required.")))
.map(SecurityContext::getAuthentication)
.flatMap(func);
}
String buildPrefix(String finalLocation, String filePrefix) {
if (StringUtils.isBlank(finalLocation) && StringUtils.isBlank(filePrefix)) {
return null;
}
StringBuilder sb = new StringBuilder();
if (StringUtils.isNotBlank(finalLocation)) {
sb.append(finalLocation).append("/");
}
if (StringUtils.isNotBlank(filePrefix)) {
sb.append(filePrefix);
}
return sb.toString();
}
}

View File

@@ -0,0 +1,38 @@
package run.halo.s3os;
import lombok.AllArgsConstructor;
import lombok.Data;
import software.amazon.awssdk.services.s3.model.S3Object;
import java.time.Instant;
import java.util.List;
@Data
@AllArgsConstructor
public class S3ListResult {
private List<ObjectVo> objects;
private String currentToken;
private String currentContinuationObject;
private String nextContinuationObject;
private String nextToken;
private Boolean hasMore;
@Data
@AllArgsConstructor
public static class ObjectVo {
private String key;
private Instant lastModified;
private Boolean isLinked;
private String displayName;
public static ObjectVo fromS3Object(S3Object s3Object) {
final var key = s3Object.key();
final var displayName = key.substring(key.lastIndexOf("/") + 1);
return new ObjectVo(key, s3Object.lastModified(), false, displayName);
}
}
}

View File

@@ -14,7 +14,10 @@ import java.util.concurrent.ConcurrentHashMap;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.pf4j.Extension;
import org.reactivestreams.Publisher;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;
import org.springframework.core.io.buffer.DefaultDataBufferFactory;
import org.springframework.http.MediaType;
import org.springframework.http.MediaTypeFactory;
import org.springframework.lang.Nullable;
@@ -22,8 +25,10 @@ import org.springframework.web.server.ServerErrorException;
import org.springframework.web.server.ServerWebInputException;
import org.springframework.web.util.UriUtils;
import reactor.core.Exceptions;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import reactor.util.context.Context;
import reactor.util.retry.Retry;
import run.halo.app.core.extension.attachment.Attachment;
import run.halo.app.core.extension.attachment.Attachment.AttachmentSpec;
@@ -32,6 +37,7 @@ import run.halo.app.core.extension.attachment.Policy;
import run.halo.app.core.extension.attachment.endpoint.AttachmentHandler;
import run.halo.app.extension.ConfigMap;
import run.halo.app.extension.Metadata;
import run.halo.app.extension.MetadataUtil;
import run.halo.app.infra.utils.JsonUtils;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.awscore.presigner.SdkPresigner;
@@ -59,8 +65,14 @@ import software.amazon.awssdk.utils.SdkAutoCloseable;
@Extension
public class S3OsAttachmentHandler implements AttachmentHandler {
private static final String OBJECT_KEY = "s3os.plugin.halo.run/object-key";
private static final int MULTIPART_MIN_PART_SIZE = 5 * 1024 * 1024;
public static final String OBJECT_KEY = "s3os.plugin.halo.run/object-key";
public static final String URL_SUFFIX_ANNO_KEY = "s3os.plugin.halo.run/url-suffix";
public static final String SKIP_REMOTE_DELETION_ANNO = "s3os.plugin.halo.run/skip-remote-deletion";
public static final int MULTIPART_MIN_PART_SIZE = 5 * 1024 * 1024;
/**
* Map to store uploading file, used as a lock, key is bucket/objectKey, value is bucket/objectKey.
*/
private final Map<String, Object> uploadingFile = new ConcurrentHashMap<>();
@Override
@@ -70,7 +82,8 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
final var properties = getProperties(context.configMap());
return upload(context, properties)
.subscribeOn(Schedulers.boundedElastic())
.map(objectDetail -> this.buildAttachment(properties, objectDetail));
.map(objectDetail -> this.buildAttachment(properties, objectDetail))
.onErrorMap(S3ExceptionHandler::map);
});
}
@@ -80,6 +93,13 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
.flatMap(context -> {
var objectKey = getObjectKey(context.attachment());
if (objectKey == null) {
log.warn(
"Cannot obtain object key from attachment {}, skip deleting object from S3.",
context.attachment().getMetadata().getName());
return Mono.just(context);
} else if (MetadataUtil.nullSafeAnnotations(context.attachment())
.containsKey(SKIP_REMOTE_DELETION_ANNO)) {
log.info("Skip deleting object {} from S3.", objectKey);
return Mono.just(context);
}
var properties = getProperties(deleteContext.configMap());
@@ -97,6 +117,7 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
})
.thenReturn(context);
})
.onErrorMap(S3ExceptionHandler::map)
.map(DeleteContext::attachment);
}
@@ -133,7 +154,8 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
}
},
SdkPresigner::close)
.subscribeOn(Schedulers.boundedElastic());
.subscribeOn(Schedulers.boundedElastic())
.onErrorMap(S3ExceptionHandler::map);
}
@Override
@@ -143,11 +165,15 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
}
var objectKey = getObjectKey(attachment);
if (objectKey == null) {
return Mono.error(new IllegalArgumentException(
"Cannot obtain object key from attachment " + attachment.getMetadata().getName()));
// fallback to default handler for backward compatibility
return Mono.empty();
}
var properties = getProperties(configMap);
var objectURL = getObjectURL(properties, objectKey);
var urlSuffix = getUrlSuffixAnnotation(attachment);
if (StringUtils.isNotBlank(urlSuffix)) {
objectURL += urlSuffix;
}
return Mono.just(URI.create(objectURL));
}
@@ -160,6 +186,15 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
return annotations.get(OBJECT_KEY);
}
@Nullable
private String getUrlSuffixAnnotation(Attachment attachment) {
var annotations = attachment.getMetadata().getAnnotations();
if (annotations == null) {
return null;
}
return annotations.get(URL_SUFFIX_ANNO_KEY);
}
S3OsProperties getProperties(ConfigMap configMap) {
var settingJson = configMap.getData().getOrDefault("default", "{}");
return JsonUtils.jsonToObject(settingJson, S3OsProperties.class);
@@ -167,12 +202,19 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
Attachment buildAttachment(S3OsProperties properties, ObjectDetail objectDetail) {
String externalLink = getObjectURL(properties, objectDetail.uploadState.objectKey);
var urlSuffix = UrlUtils.findUrlSuffix(properties.getUrlSuffixes(),
objectDetail.uploadState.fileName);
var metadata = new Metadata();
metadata.setName(UUID.randomUUID().toString());
metadata.setAnnotations(new HashMap<>(
Map.of(OBJECT_KEY, objectDetail.uploadState.objectKey,
Constant.EXTERNAL_LINK_ANNO_KEY, externalLink)));
var annotations = new HashMap<>(Map.of(OBJECT_KEY, objectDetail.uploadState.objectKey));
if (StringUtils.isNotBlank(urlSuffix)) {
externalLink += urlSuffix;
annotations.put(URL_SUFFIX_ANNO_KEY, urlSuffix);
}
annotations.put(Constant.EXTERNAL_LINK_ANNO_KEY, externalLink);
metadata.setAnnotations(annotations);
var objectMetadata = objectDetail.objectMetadata();
var spec = new AttachmentSpec();
@@ -183,12 +225,11 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
var attachment = new Attachment();
attachment.setMetadata(metadata);
attachment.setSpec(spec);
log.info("Upload object {} to bucket {} successfully", objectDetail.uploadState.objectKey,
properties.getBucket());
log.info("Built attachment {} successfully", objectDetail.uploadState.objectKey);
return attachment;
}
private String getObjectURL(S3OsProperties properties, String objectKey) {
String getObjectURL(S3OsProperties properties, String objectKey) {
String objectURL;
if (StringUtils.isBlank(properties.getDomain())) {
String host;
@@ -232,10 +273,52 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
.build();
}
Flux<DataBuffer> reshape(Publisher<DataBuffer> content, int bufferSize) {
var dataBufferFactory = DefaultDataBufferFactory.sharedInstance;
return Flux.<ByteBuffer>create(sink -> {
var byteBuffer = ByteBuffer.allocate(bufferSize);
Flux.from(content)
.doOnNext(dataBuffer -> {
var count = dataBuffer.readableByteCount();
for (var i = 0; i < count; i++) {
byteBuffer.put(dataBuffer.read());
// Emit the buffer when buffer
if (!byteBuffer.hasRemaining()) {
sink.next(deepCopy(byteBuffer));
byteBuffer.clear();
}
}
})
.doOnComplete(() -> {
// Emit the last part of buffer.
if (byteBuffer.position() > 0) {
sink.next(deepCopy(byteBuffer));
}
})
.subscribe(DataBufferUtils::release, sink::error, sink::complete,
Context.of(sink.contextView()));
})
.map(dataBufferFactory::wrap)
.cast(DataBuffer.class)
.doOnDiscard(DataBuffer.class, DataBufferUtils::release);
}
ByteBuffer deepCopy(ByteBuffer src) {
src.flip();
var dest = ByteBuffer.allocate(src.limit());
dest.put(src);
src.rewind();
dest.flip();
return dest;
}
Mono<ObjectDetail> upload(UploadContext uploadContext, S3OsProperties properties) {
return Mono.using(() -> buildS3Client(properties),
client -> {
var uploadState = new UploadState(properties, uploadContext.file().filename());
var uploadState = new UploadState(properties, uploadContext.file().filename(), true);
var content = uploadContext.file().content();
return checkFileExistsAndRename(uploadState, client)
// init multipart upload
.flatMap(state -> Mono.fromCallable(() -> client.createMultipartUpload(
@@ -243,12 +326,12 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
.bucket(properties.getBucket())
.contentType(state.contentType)
.key(state.objectKey)
.build())).subscribeOn(Schedulers.boundedElastic()))
.flatMapMany((response) -> {
.build())))
.doOnNext((response) -> {
checkResult(response, "createMultipartUpload");
uploadState.uploadId = response.uploadId();
return uploadContext.file().content();
})
.thenMany(reshape(content, MULTIPART_MIN_PART_SIZE))
// buffer to part
.windowUntil((buffer) -> {
uploadState.buffered += buffer.readableByteCount();
@@ -293,6 +376,8 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
// build object detail
.map((response) -> {
checkResult(response, "getMetadata");
log.info("Uploaded object {} to bucket {} successfully",
uploadState.objectKey, properties.getBucket());
return new ObjectDetail(uploadState, response);
})
// close client
@@ -305,8 +390,8 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
SdkAutoCloseable::close);
}
private Mono<UploadState> checkFileExistsAndRename(UploadState uploadState,
S3Client s3client) {
Mono<UploadState> checkFileExistsAndRename(UploadState uploadState,
S3Client s3client) {
return Mono.defer(() -> {
// deduplication of uploading files
if (uploadingFile.put(uploadState.getUploadingMapKey(),
@@ -344,7 +429,7 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
uploadingFile.remove(uploadState.getUploadingMapKey());
uploadState.needRemoveMapKey = false;
}
uploadState.randomFileName();
uploadState.randomDuplicateFileName();
})
)
.onErrorMap(Exceptions::isRetryExhausted,
@@ -352,8 +437,8 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
}
private Mono<CompletedPart> uploadPart(UploadState uploadState, ByteBuffer buffer,
S3Client s3client) {
Mono<CompletedPart> uploadPart(UploadState uploadState, ByteBuffer buffer,
S3Client s3client) {
final int partNumber = ++uploadState.partCounter;
return Mono.just(s3client.uploadPart(UploadPartRequest.builder()
.bucket(uploadState.properties.getBucket())
@@ -372,7 +457,7 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
});
}
private static void checkResult(SdkResponse result, String operation) {
static void checkResult(SdkResponse result, String operation) {
log.info("operation: {}, result: {}", operation, result);
if (result.sdkHttpResponse() == null || !result.sdkHttpResponse().isSuccessful()) {
log.error("Failed to upload object, response: {}", result.sdkHttpResponse());
@@ -380,7 +465,7 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
}
}
private static ByteBuffer concatBuffers(List<DataBuffer> buffers) {
static ByteBuffer concatBuffers(List<DataBuffer> buffers) {
int partSize = 0;
for (DataBuffer b : buffers) {
partSize += b.readableByteCount();
@@ -420,9 +505,16 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
String objectKey;
boolean needRemoveMapKey = false;
public UploadState(S3OsProperties properties, String fileName) {
public UploadState(S3OsProperties properties, String fileName, boolean needRandomJudge) {
this.properties = properties;
this.originalFileName = fileName;
if (needRandomJudge) {
fileName =
FileNameUtils.replaceFilename(fileName, properties.getRandomFilenameMode(),
properties.getRandomStringLength(), properties.getCustomTemplate());
}
this.fileName = fileName;
this.objectKey = properties.getObjectName(fileName);
this.contentType = MediaTypeFactory.getMediaType(fileName)
@@ -433,8 +525,11 @@ public class S3OsAttachmentHandler implements AttachmentHandler {
return properties.getBucket() + "/" + objectKey;
}
public void randomFileName() {
this.fileName = FileNameUtils.randomFileName(originalFileName, 4);
public void randomDuplicateFileName() {
this.fileName = FileNameUtils.replaceFilenameWithDuplicateHandling(originalFileName,
properties.getRandomFilenameMode(),
properties.getRandomStringLength(), properties.getCustomTemplate(),
properties.getDuplicateFilenameHandling());
this.objectKey = properties.getObjectName(fileName);
}
}

View File

@@ -1,8 +1,8 @@
package run.halo.s3os;
import org.pf4j.PluginWrapper;
import org.springframework.stereotype.Component;
import run.halo.app.plugin.BasePlugin;
import run.halo.app.plugin.PluginContext;
/**
* @author johnniang
@@ -11,8 +11,8 @@ import run.halo.app.plugin.BasePlugin;
@Component
public class S3OsPlugin extends BasePlugin {
public S3OsPlugin(PluginWrapper wrapper) {
super(wrapper);
public S3OsPlugin(PluginContext pluginContext) {
super(pluginContext);
}
@Override

View File

@@ -1,8 +1,14 @@
package run.halo.s3os;
import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import org.springframework.util.StringUtils;
import java.time.LocalDate;
@Data
class S3OsProperties {
@@ -23,6 +29,14 @@ class S3OsProperties {
*/
private String location;
private RandomFilenameMode randomFilenameMode;
private String customTemplate;
private DuplicateFilenameHandling duplicateFilenameHandling;
private Integer randomStringLength = 8;
private Protocol protocol = Protocol.https;
/**
@@ -33,15 +47,34 @@ class S3OsProperties {
private String region = "Auto";
private List<urlSuffixItem> urlSuffixes;
@Data
@AllArgsConstructor
@NoArgsConstructor
public static class urlSuffixItem {
private String fileSuffix;
private String urlSuffix;
}
public enum DuplicateFilenameHandling {
randomAlphanumeric, randomAlphabetic, exception
}
public enum RandomFilenameMode {
none, custom, uuid, timestampMs, dateWithString, datetimeWithString, withString, string, random_number
}
public String getObjectName(String filename) {
var objectName = filename;
if (StringUtils.hasText(getLocation())) {
objectName = getLocation() + "/" + objectName;
var finalName = FilePathUtils.getFilePathByPlaceholder(getLocation());
if (StringUtils.hasText(finalName)) {
objectName = finalName + "/" + objectName;
}
return objectName;
}
enum Protocol {
public enum Protocol {
http, https
}
@@ -64,6 +97,16 @@ class S3OsProperties {
this.location = location;
}
public void setRandomStringLength(String randomStringLength) { // if you use Integer, it will throw Error.
try {
int length = Integer.parseInt(randomStringLength);
if (length >= 4 && length <= 16) {
this.randomStringLength = length;
}
}
catch (NumberFormatException ignored) { }
}
public void setRegion(String region) {
if (!StringUtils.hasText(region)) {
this.region = "Auto";

View File

@@ -0,0 +1,21 @@
package run.halo.s3os;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Mono;
import run.halo.app.core.extension.attachment.Attachment;
import run.halo.app.plugin.ApiVersion;
@ApiVersion("s3os.halo.run/v1alpha1")
@RestController
@RequiredArgsConstructor
public class S3UnlinkController {
private final S3UnlinkService s3UnlinkService;
@DeleteMapping("/attachments/{name}")
public Mono<Attachment> unlink(@PathVariable("name") String name) {
return s3UnlinkService.unlink(name);
}
}

View File

@@ -0,0 +1,8 @@
package run.halo.s3os;
import reactor.core.publisher.Mono;
import run.halo.app.core.extension.attachment.Attachment;
public interface S3UnlinkService {
Mono<Attachment> unlink(String name);
}

View File

@@ -0,0 +1,36 @@
package run.halo.s3os;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ServerWebInputException;
import reactor.core.publisher.Mono;
import run.halo.app.core.extension.attachment.Attachment;
import run.halo.app.core.extension.attachment.Policy;
import run.halo.app.extension.ReactiveExtensionClient;
@Service
@RequiredArgsConstructor
@Slf4j
public class S3UnlinkServiceImpl implements S3UnlinkService {
private final ReactiveExtensionClient client;
private final S3OsAttachmentHandler handler;
@Override
public Mono<Attachment> unlink(String name) {
return client.get(Attachment.class, name)
.flatMap((attachment) -> client.get(Policy.class, attachment.getSpec().getPolicyName())
.doOnNext((policy) -> {
if (!handler.shouldHandle(policy)) {
throw new ServerWebInputException(
"This attachment policy is not managed by plugin-s3.");
}
}).thenReturn(attachment))
.flatMap(attachment -> {
attachment.getMetadata().getAnnotations().put(
S3OsAttachmentHandler.SKIP_REMOTE_DELETION_ANNO, Boolean.TRUE.toString());
return client.delete(attachment);
});
}
}

View File

@@ -2,6 +2,7 @@ package run.halo.s3os;
import java.util.Arrays;
import java.util.List;
import org.apache.commons.lang3.StringUtils;
public class UrlUtils {
private static final List<String> HTTP_PREFIXES = Arrays.asList("http://", "https://");
@@ -17,4 +18,21 @@ public class UrlUtils {
}
return url;
}
public static String findUrlSuffix(List<S3OsProperties.urlSuffixItem> urlSuffixList,
String fileName) {
if (StringUtils.isBlank(fileName) || urlSuffixList == null) {
return null;
}
fileName = fileName.toLowerCase();
for (S3OsProperties.urlSuffixItem item : urlSuffixList) {
String[] fileSuffixes = item.getFileSuffix().split(",");
for (String suffix : fileSuffixes) {
if (fileName.endsWith("." + suffix.trim().toLowerCase())) {
return item.getUrlSuffix();
}
}
}
return null;
}
}

View File

@@ -3,7 +3,7 @@ kind: PolicyTemplate
metadata:
name: s3os
spec:
displayName: S3 Object Storage
displayName: S3 对象存储
settingName: s3os-policy-template-setting
---
apiVersion: v1alpha1
@@ -14,63 +14,134 @@ spec:
forms:
- group: default
formSchema:
- $formkit: text
name: bucket
label: Bucket 桶名称
validation: required
- $formkit: select
name: endpointProtocol
label: Endpoint 访问协议
options:
- label: HTTPS
value: https
- label: HTTP
value: http
validation: required
- $formkit: select
name: enablePathStyleAccess
label: Endpoint 访问风格
options:
- label: Virtual Hosted Style
- $formkit: verificationForm
action: "/apis/s3os.halo.run/v1alpha1/policies/s3/validation"
label: 对象存储验证
children:
- $formkit: text
name: bucket
label: Bucket 桶名称
validation: required
- $formkit: select
name: endpointProtocol
label: Endpoint 访问协议
options:
- label: HTTPS
value: https
- label: HTTP
value: http
validation: required
- $formkit: select
name: enablePathStyleAccess
label: Endpoint 访问风格
options:
- label: Virtual Hosted Style
value: false
- label: Path Style
value: true
value: false
- label: Path Style
value: true
value: false
validation: required
- $formkit: text
name: endpoint
label: EndPoint
placeholder: 请填写不带bucket-name的Endpoint
validation: required
help: 协议头请在上方设置,此处无需以"http://"或"https://"开头,系统会自动拼接
- $formkit: password
name: accessKey
label: Access Key
validation: required
- $formkit: password
name: accessSecret
label: Access Secret
validation: required
- $formkit: text
name: region
label: Region
placeholder: 如不填写,则默认为"Auto"
help: 若Region为Auto无法使用才需要填写对应Region
- $formkit: text
name: location
label: 上传目录
placeholder: 如不填写,则默认上传到根目录
- $formkit: select
name: protocol
label: 绑定域名协议
options:
- label: HTTPS
value: https
- label: HTTP
value: http
validation: required
- $formkit: text
name: domain
label: 绑定域名CDN域名
placeholder: 如不设置,那么将使用 Bucket + EndPoint 作为域名
help: 协议头请在上方设置,此处无需以"http://"或"https://"开头,系统会自动拼接
validation: required
- $formkit: text
name: endpoint
label: EndPoint
placeholder: 请填写不带bucket-name的Endpoint
validation: required
help: 协议头请在上方设置,此处无需以"http://"或"https://"开头,系统会自动拼接
- $formkit: password
name: accessKey
label: Access Key ID
placeholder: 存储桶用户标识(用户名)
validation: required
- $formkit: password
name: accessSecret
label: Access Key Secret
placeholder: 存储桶密钥(密码)
validation: required
- $formkit: text
name: region
label: Region
placeholder: 如不填写,则默认为"Auto"
help: 若Region为Auto无法使用才需要填写对应Region
- $formkit: text
name: location
label: 上传目录
placeholder: 如不填写,则默认上传到根目录
help: 支持的占位符请查阅https://github.com/halo-dev/plugin-s3#上传目录
- $formkit: select
name: randomFilenameMode
label: 上传时重命名文件方式
options:
- label: 保留原文件名
value: none
- label: 自定义(请在下方输入自定义模板)
value: custom
- label: 使用UUID
value: uuid
- label: 使用毫秒时间戳
value: timestampMs
- label: 使用原文件名 + 随机字母
value: withString
- label: 使用日期 + 随机字母
value: dateWithString
- label: 使用日期时间 + 随机字母
value: datetimeWithString
- label: 使用随机字母
value: string
validation: required
- $formkit: number
name: randomStringLength
key: randomStringLength
label: 随机字母长度
min: 4
max: 16
if: "$randomFilenameMode == 'dateWithString' || $randomFilenameMode == 'datetimeWithString' || $randomFilenameMode == 'withString' || $randomFilenameMode == 'string'"
help: 支持4~16位, 默认为8位
- $formkit: text
name: customTemplate
key: customTemplate
label: 自定义文件名模板
if: "$randomFilenameMode == 'custom'"
value: "${origin-filename}"
help: 支持的占位符请查阅https://github.com/halo-dev/plugin-s3#自定义文件名模板
- $formkit: select
name: duplicateFilenameHandling
label: 重复文件名处理方式
options:
- label: 加随机字母数字后缀
value: randomAlphanumeric
- label: 加随机字母后缀
value: randomAlphabetic
- label: 报错不上传
value: exception
validation: required
- $formkit: select
name: protocol
label: 绑定域名协议
options:
- label: HTTPS
value: https
- label: HTTP
value: http
validation: required
- $formkit: text
name: domain
label: 绑定域名CDN域名
placeholder: 如不设置,那么将使用 Bucket + EndPoint 作为域名
help: 协议头请在上方设置,此处无需以"http://"或"https://"开头,系统会自动拼接
- $formkit: repeater
name: urlSuffixes
label: 网址后缀
help: 用于对指定文件类型的网址添加后缀处理参数,优先级从上到下只取第一个匹配项
value: [ ]
min: 0
children:
- $formkit: text
name: fileSuffix
label: 文件后缀
placeholder: 以半角逗号分隔例如jpg,jpeg,png,gif
validation: required
- $formkit: text
name: urlSuffix
label: 网址后缀
placeholder: 例如:?imageMogr2/format/webp
validation: required

View File

@@ -0,0 +1,55 @@
apiVersion: v1alpha1
kind: "Role"
metadata:
name: role-template-s3os-link
labels:
halo.run/role-template: "true"
annotations:
rbac.authorization.halo.run/dependencies: |
[ "role-template-manage-attachments" ]
rbac.authorization.halo.run/module: "S3 Attachments Management"
rbac.authorization.halo.run/display-name: "S3 Link"
rbac.authorization.halo.run/ui-permissions: |
["plugin:s3os:link"]
rules:
- apiGroups: [ "s3os.halo.run" ]
resources: [ "policies" ]
resourceNames: [ "s3" ]
verbs: [ "get", "list" ]
- apiGroups: [ "s3os.halo.run" ]
resources: [ "objects" ]
verbs: [ "get", "list" ]
- nonResourceURLs: ["/apis/s3os.halo.run/v1alpha1/attachments/link"]
verbs: [ "create" ]
---
apiVersion: v1alpha1
kind: "Role"
metadata:
name: role-template-s3os-unlink
labels:
halo.run/role-template: "true"
annotations:
rbac.authorization.halo.run/dependencies: |
[ "role-template-manage-attachments" ]
rbac.authorization.halo.run/module: "S3 Attachments Management"
rbac.authorization.halo.run/display-name: "S3 UnLink"
rbac.authorization.halo.run/ui-permissions: |
["plugin:s3os:unlink"]
rules:
- apiGroups: [ "s3os.halo.run" ]
resources: [ "attachments" ]
verbs: [ "delete" ]
---
apiVersion: v1alpha1
kind: "Role"
metadata:
name: role-template-s3os-policy-config-validation
labels:
halo.run/role-template: "true"
halo.run/hidden: "true"
rbac.authorization.halo.run/aggregate-to-role-template-manage-configmaps: "true"
rules:
- apiGroups: ["s3os.halo.run"]
resources: ["policies/validation"]
resourceNames: ["s3"]
verbs: [ "create" ]

View File

@@ -5,10 +5,7 @@ metadata:
spec:
forms:
- group: basic
label: 基本设置
label: 使用提示
formSchema:
- $formkit: text
help: 请前往 “附件 - 存储策略” 添加策略
label: 此处不用设置,请前往 “附件 - 存储策略” 添加策略
name: text
placeholder: 此处不用设置,请前往 “附件 - 存储策略” 添加策略
- $el: p
children: 请前往 “附件 - 存储策略” 添加策略

View File

@@ -4,15 +4,16 @@ metadata:
name: PluginS3ObjectStorage
spec:
enabled: true
version: 1.4.0
requires: ">=2.0.0"
requires: ">=2.14.0"
author:
name: longjuan
website: https://github.com/longjuan
name: Halo
website: https://github.com/halo-dev
logo: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAAAXNSR0IArs4c6QAABUdJREFUeF7tmmtoXFUQx3+zMYGi/RJRULRFsL6oim2klVK0CrVqKxjoJ0Vt1bR7t6lURagiFlGpFPogydklVo1i/VB84QMfqFijpflQrI+IYj7FF+0HMbUEI82O3NNtzG6y956bvXd3IXvgksCd+c/M/8w598ycFWb5kFkePw0CGhkwyxloLIFZngCNTbCxBBpLoI4Y0A6aydNKC62kaAXGSDHCv4zQzIh0MRa3uzVdAppmMSmWoVyL0IZyWUiA3wL7gI/FcCQOMmpCQCHwDpSOCoIYQOmWLK9WgFHdr4Bu4wyOsR14uBKni3SFdxhnl+T4fCaYVcsAzbCEPNsRbpiJow46u8TwkINcCX9RNWYgr2nWIrwAzJ2BehSVAVpYJbv5y1Up8QwoBL/f1aGY5JaJ4aALVqIEqMc24EkXRxKQmSeGX8JwEyNAPR4AesMcSPD9ECdpk15GgmwkQoB2spxxPgWaEwzQBfptMdxRfQI8PgJWuniYuIyyQbLlMzH2DFCPDUAu8cDcDRwWQ1s58VgJ0E7OYZyvgAXu/lVBMiAL4iXA4z5gbxVCimpiQAxLp1OKl4AMfSj3RPWuKvJ5FkmOr0ttxUuAx9/AWVUJKKoR5QnJ8nRsBOh65tJiy9nFYJ+fwB586nUcEsN1FROgHncDG2EqWIWR/wpcUCFGsPooc6SPfyYLOS8BzbAI5RlgVQJObrXdH9iZAPb/kCkul25+jEyAejboDxJyLiMG42Orx2fAioTs+LC3iOHDSARohs0oexJxStgtPWw5jZ34IUpJS7b4kBa4BDTDTSifJBI8vC+G1ZOxdQtzGGMIOD8hm1vF2I7UxAgmwLNpn8SaHyTPjZLjWGmg6tl9YCIrYiVCuVOyvOZEQGG3fzlWB06BjZJipXTbI/OUoWmuQvgmAbs+5HIxfOlKwCFgSQKOrBNDXxCuerwH3Ba77ZPMl16GQwnQjSwkxXexO6A8JdnwDpF6tANvxG1fzNQu+LR7QEKtrFfEuNcJ6vEzcHGMJEzZdH3scgRkC6e9eOwr/Qi3iuGEK6B6PAo85yofKid40oMfV9EoR8CbENxKCjVYvNCWSg8DUXQKn8TRKDqBstOs/6AMiLelVaYUDQtO0zyPcH+YnMP7d8Vw+3Ry5TLgJeBeB2BXEX8mDwDHEUZQWzafEBNcPca2FwlrpMd+WaaM6QnI8CyKX6AkN/KsmHyfZ9voQjfKUbDPOMoCxF6TVzLKzn75JZBmOcIXlVgN0y39JGka/yOVDtOL/D5g9ssS4L9Qz5aNl0Y26KZwRAzXTBZVj8PAIjd1Z6ntYoIzuWwtoBkeK9T/ztacBUurwA1cQRODzvpugvvEcFeYaHkCHuFMRvGPwwvDQCK+P0qexZLjt9N6mmE9am+P4xnKn5LlbBew4GowzTqEF12AnGXytEuOtyaC30aKY7YRcr0zRrDg92K40hUrtCWmGfajrHUFDJHzf9tzc8na9zu1j8eCL/RKj72Zch6hBPhI6tEFbHJGnV5wSEzxjVGMrbZBlB2SJXL57kRAgQS/G7wDODcyEUKX9LC5ZOb9+wO/lV5J9+c4sIsmdkoX/v+RhzMBloRNXESeB8E+4UNtSbtHsvSXCqt/1oB2xNYc88PBiiT8uuIgTeyVLn6IqFskHomAiY0rwyUoa1BWI3YGzyu8+wPld1K8Tp5+yeL/ri90aCdXM26JuBCYN/GcOjYPIwzbv3CYJg5IF/4dQixjRgTEYrlOQBoE1MlE1MyNRgbUjPo6MdzIgDqZiJq50ciAmlFfJ4YbGVAnE1EzN2Z9BvwH8mZfUOuuxhIAAAAASUVORK5CYII=
settingName: s3os-settings
configMapName: s3os-configMap
homepage: https://github.com/halo-sigs/plugin-s3
homepage: https://www.halo.run/store/apps/app-Qxhpp
repo: https://github.com/halo-dev/plugin-s3
issues: https://github.com/halo-dev/plugin-s3/issues
displayName: "对象存储Amazon S3 协议)"
description: "提供兼容 Amazon S3 协议的对象存储策略,兼容阿里云、腾讯云、七牛云等"
license:

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

View File

@@ -0,0 +1,28 @@
package run.halo.s3os;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
class FileNameUtilsTest {
@Test
void testReplaceFilenameWithDuplicateHandling() {
// Case 1: halo.run -> halo-xyz.run
var result = FileNameUtils.replaceFilenameWithDuplicateHandling(
"halo.run", S3OsProperties.RandomFilenameMode.none, null,
null, S3OsProperties.DuplicateFilenameHandling.randomAlphanumeric);
assertTrue(result.matches("halo-[a-z0-9]{4}.run"));
// Case 2: .run -> xyz.run
result = FileNameUtils.replaceFilenameWithDuplicateHandling(
".run", S3OsProperties.RandomFilenameMode.none, null,
null, S3OsProperties.DuplicateFilenameHandling.randomAlphanumeric);
assertTrue(result.matches("[a-z0-9]{4}.run"));
// Case 3: halo -> halo-xyz
result = FileNameUtils.replaceFilenameWithDuplicateHandling(
"halo", S3OsProperties.RandomFilenameMode.none, null,
null, S3OsProperties.DuplicateFilenameHandling.randomAlphanumeric);
assertTrue(result.matches("halo-[a-z0-9]{4}"));
}
}

View File

@@ -0,0 +1,80 @@
package run.halo.s3os;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
class PlaceholderReplacerTest {
@Test
void testReplacePlaceholdersTemplateNull() {
String result1 = PlaceholderReplacer.replacePlaceholders(null, "test");
assertEquals("test", result1);
String result2 = PlaceholderReplacer.replacePlaceholders("", "test");
assertEquals("test", result2);
}
@Test
void testReplacePlaceholdersAllPlaceholder() {
String template = "${origin-filename}-${uuid-with-dash}-${uuid-no-dash}-${timestamp-sec}-" +
"${timestamp-ms}-${year}-${month}-${day}-${weekday}-${hour}-${minute}-${second}-" +
"${millisecond}-${random-alphabetic:4}-${random-num:5}-${random-alphanumeric:6}";
String result = PlaceholderReplacer.replacePlaceholders(template, "test");
String regex = "test-" +
"[A-F0-9]{8}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{12}-" +
"[A-F0-9]{32}-" +
"[0-9]{10}-" +
"[0-9]{13}-[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{1}-[0-9]{2}-[0-9]{2}-[0-9]{2}-" +
"[0-9]{3}-[a-z]{4}-[0-9]{5}-[a-z0-9]{6}";
assertTrue(result.matches(regex));
}
@Test
void testReplacePlaceholdersTimestamp() {
String template =
"${timestamp-sec}-${timestamp-sec}-${timestamp-ms}-${timestamp-ms}-${year}-${year}-" +
"${month}-${month}-${day}-${day}-${weekday}-${weekday}-${hour}-${hour}-" +
"${minute}-${minute}-${second}-${second}-${millisecond}-${millisecond}";
String result = PlaceholderReplacer.replacePlaceholders(template, "test");
String[] split = result.split("-");
for (int i = 0; i < split.length; i += 2) {
assertEquals(split[i], split[i + 1]);
}
}
@Test
void testReplacePlaceholdersRandomAlphabeticNoLength() {
String template =
"${random-alphabetic}_${random-alphabetic:}_${random-alphabetic:-1}_${random-alphabetic:0}";
String result = PlaceholderReplacer.replacePlaceholders(template, "test");
String regex = "[a-z]{8}_[a-z]{8}_[a-z]{8}_[a-z]{8}";
assertTrue(result.matches(regex));
template = "${random-num}_${random-num:}_${random-num:-1}_${random-num:0}";
result = PlaceholderReplacer.replacePlaceholders(template, "test");
regex = "[0-9]{8}_[0-9]{8}_[0-9]{8}_[0-9]{8}";
assertTrue(result.matches(regex));
template =
"${random-alphanumeric}_${random-alphanumeric:}_${random-alphanumeric:-1}_${random-alphanumeric:0}";
result = PlaceholderReplacer.replacePlaceholders(template, "test");
regex = "[a-z0-9]{8}_[a-z0-9]{8}_[a-z0-9]{8}_[a-z0-9]{8}";
assertTrue(result.matches(regex));
}
@Test
void testReplacePlaceholdersInvalid() {
String template = "file_${not-exist}_test";
String result = PlaceholderReplacer.replacePlaceholders(template, "test");
assertEquals("file_${not-exist}_test", result);
template = "file_${random-alphabetic_test";
result = PlaceholderReplacer.replacePlaceholders(template, "test");
assertEquals("file_${random-alphabetic_test", result);
template = "file_random-alphabetic}_test";
result = PlaceholderReplacer.replacePlaceholders(template, "test");
assertEquals("file_random-alphabetic}_test", result);
}
}

View File

@@ -1,12 +1,19 @@
package run.halo.s3os;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.util.List;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DefaultDataBufferFactory;
import reactor.core.publisher.Flux;
import reactor.test.StepVerifier;
import run.halo.app.core.extension.attachment.Policy;
class S3OsAttachmentHandlerTest {
@@ -33,4 +40,57 @@ class S3OsAttachmentHandlerTest {
// policy is null
assertFalse(handler.shouldHandle(null));
}
@Test
void reshapeDataBufferWithSmallerBufferSize() {
var handler = new S3OsAttachmentHandler();
var factory = DefaultDataBufferFactory.sharedInstance;
var content = Flux.<DataBuffer>fromIterable(List.of(factory.wrap("halo".getBytes())));
StepVerifier.create(handler.reshape(content, 2))
.assertNext(dataBuffer -> {
var str = dataBuffer.toString(UTF_8);
assertEquals("ha", str);
})
.assertNext(dataBuffer -> {
var str = dataBuffer.toString(UTF_8);
assertEquals("lo", str);
})
.verifyComplete();
}
@Test
void reshapeDataBufferWithBiggerBufferSize() {
var handler = new S3OsAttachmentHandler();
var factory = DefaultDataBufferFactory.sharedInstance;
var content = Flux.<DataBuffer>fromIterable(List.of(factory.wrap("halo".getBytes())));
StepVerifier.create(handler.reshape(content, 10))
.assertNext(dataBuffer -> {
var str = dataBuffer.toString(UTF_8);
assertEquals("halo", str);
})
.verifyComplete();
}
@Test
void reshapeDataBuffersWithBiggerBufferSize() {
var handler = new S3OsAttachmentHandler();
var factory = DefaultDataBufferFactory.sharedInstance;
var content = Flux.<DataBuffer>fromIterable(List.of(
factory.wrap("ha".getBytes()),
factory.wrap("lo".getBytes())
));
StepVerifier.create(handler.reshape(content, 3))
.assertNext(dataBuffer -> {
var str = dataBuffer.toString(UTF_8);
assertEquals("hal", str);
})
.assertNext(dataBuffer -> {
var str = dataBuffer.toString(UTF_8);
assertEquals("o", str);
})
.verifyComplete();
}
}

View File

@@ -1,5 +1,6 @@
package run.halo.s3os;
import java.util.List;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
@@ -12,4 +13,43 @@ class UrlUtilsTest {
assert UrlUtils.removeHttpPrefix("http://www.example.com").equals("www.example.com");
assert UrlUtils.removeHttpPrefix("https://www.example.com").equals("www.example.com");
}
@Test
public void testFindUrlSuffix() {
List<S3OsProperties.urlSuffixItem> urlSuffixList = List.of(
new S3OsProperties.urlSuffixItem("jpg,png,gif", "?imageMogr2/format/webp"),
new S3OsProperties.urlSuffixItem("pdf", "?123=123"),
new S3OsProperties.urlSuffixItem("jpg", "?456=456")
);
// 测试文件名为"example.jpg",期望匹配到"?imageMogr2/format/webp",只匹配第一个后缀
String fileName1 = "example.jpg";
String result1 = UrlUtils.findUrlSuffix(urlSuffixList, fileName1);
assertEquals("?imageMogr2/format/webp", result1);
// 测试文件名为"Document.PDF",期望匹配到"?123=123",不区分大小写
String fileName2 = "Document.PDF";
String result2 = UrlUtils.findUrlSuffix(urlSuffixList, fileName2);
assertEquals("?123=123", result2);
// 测试文件名为"unknown.txt"期望没有匹配项返回null
String fileName3 = "unknown.txt";
String result3 = UrlUtils.findUrlSuffix(urlSuffixList, fileName3);
assertNull(result3);
// 测试无后缀文件名"example"期望没有匹配项返回null
String fileName4 = "example";
String result4 = UrlUtils.findUrlSuffix(urlSuffixList, fileName4);
assertNull(result4);
// 测试空文件名期望返回null
String fileName5 = "";
String result5 = UrlUtils.findUrlSuffix(urlSuffixList, fileName5);
assertNull(result5);
// 测试urlSuffixList为null期望返回null
String fileName6 = "example";
String result6 = UrlUtils.findUrlSuffix(null, fileName6);
assertNull(result6);
}
}