KubeGems 问题排查
1. kubegems-api 与 kubegems-local 时间不同步造成请求返回 403
KubeGems 内部各模块接口调用之间均开启了 http 签名，可有效防御请求被篡改、重放攻击等手段。
什么是 HTTP 签名 ?
在 KubeGems 中，httpsign 的始终差默认为 15 分钟，如服务间时间差大于 15 分钟时会报如下错误 kubectl-api.log
2022-10-13 08:04:17.924 error middleware/httpsig.go:32 signer {"error": "httpsigs time out, origin: 1665649153, now: 2022-10-13 08:04:17.924265317 +0000 UTC m=+1221.099241986"}
2022-10-13 08:04:17.925 warn gin@v1.8.0/context.go:173 Forbidden {"method": "GET", "path": "/v1/api-resources", "code": 403, "latency": 0.001017259}
2022-10-13 08:04:24.35 error middleware/httpsig.go:32 signer {"error": "httpsigs time out, origin: 1665649160, now: 2022-10-13 08:04:24.350349102 +0000 UTC m=+1227.525325614"}
2022-10-13 08:04:24.35 warn gin@v1.8.0/context.go:173 Forbidden {"method": "GET", "path": "/internal/plugins.kubegems.io/v1beta1/namespaces/kubegems-local/Plugin", "code": 403, "latency": 0.000517216}
2022-10-13 08:04:24.445 error middleware/httpsig.go:32 signer {"error": "httpsigs time out, origin: 1665649160, now: 2022-10-13 08:04:24.445145498 +0000 UTC m=+1227.620122137"}
2022-10-13 08:04:24.446 warn gin@v1.8.0/context.go:173 Forbidden {"method": "GET", "path": "/internal/plugins.kubegems.io/v1beta1/namespaces/kubegems-local/Plugin", "code": 403, "latency": 0.001365423}
同步集群内主机的时间可解决上述问题
info
推荐在操作系统中启用 NTP 或 Chorny 始终同步服务
2. 为 kubegems 前端添加代理后，应用中心状态页面加载很慢
由于应用中心部分接口后端采用了 SSE（Server-Setnt Events） 单向连接来像页面推送消息，由于 Nginx 或其他网关默认在代理开启了 buffer 池来缓冲数据，等到 buffer 满了批量返回给客户端，这就导致 KubeGems 前端数据加载变慢。 此问题关闭 proxy buffer 即可解决。
什么是 Server-Setnt Events ?
Nginx 配置
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_buffering off;
proxy_cache off;
proxy_pass http://kubegems-dashboard:8000;
}
Nginx Ingress 配置
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.org/proxy-buffering: "false"
nginx.org/websocket-services: kubegems-dashboard
APISIX 配置
修改 apisix 配置文件，注入以下内容
http_configuration_snippet: |-
proxy_buffering off;
3. KubeGems 升级后应用部署报错 invalid session: account password has changed since token issued
KubeGems 中心服务升级后再应用部署时，页面报错account password has changed since token issued，KubeGems API 日志如下:
2022-11-09 02:09:05.716 error gin@v1.8.0/context.go:173 Error #1: rpc error: code = Unauthenticated desc = invalid session: account password has changed since token issued
此问题出现在 KubeGems v1.22 的版本升级，复现于 ArgoCD 在更新过程中的丢失了账号认证信息（ KubeGems API -> ArgoCD ）。修复此问题需重启 kubegems-api 容器加载 argocd 的最新配置即可。
4. 启用模型商店后,容器 kubegems-models-store 状态一直 Error
容器 models-store 启动日志报错如下：
2022-11-10 02:02:26.654 warn config/config.go:104 no config file found or config file format error: Config File "config" Not Found in "[/app /app/config]"
2022-11-10 02:02:26.654 info config/config.go:121 config from env: MONGO_ADDR=kubegems-models-mongodb:27017
2022-11-10 02:02:26.654 info config/config.go:121 config from env: MONGO_PASSWORD=**********
2022-11-10 02:02:26.654 info config/config.go:121 config from env: MONGO_USERNAME=root
2022-11-10 02:02:26.654 info config/config.go:121 config from env: MYSQL_ADDR=kubegems-mysql-headless:3306
2022-11-10 02:02:26.654 info config/config.go:121 config from env: MYSQL_PASSWORD=**********
2022-11-10 02:02:26.654 info config/config.go:121 config from flag: listen=:8080
2022-11-10 02:02:26.654 info config/config.go:121 config from flag: sync-addr=http://kubegems-models-sync:8080
Error: setup mongo: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: kubegems-models-mongodb:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp 10.96.165.211:27017: connect: connection refused }, ] }
setup mongo: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: kubegems-models-mongodb:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp 10.96.165.211:27017: connect: connection refused }, ] }
Error: setup mongo: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: kubegems-models-mongodb:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp 10.96.165.211:27017: connect: connection refused }, ] }
setup mongo: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: kubegems-models-mongodb:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp 10.96.165.211:27017: connect: connection refused }, ] }
由于模型商店需要向 HuggingFace 和 OpenMMLab 同步元数据，出现此错误在于 MongoDB 内无数据，在 KubeGems 管理员后台的模型商店中同步模型即可。
5. 更改集群 kubeconfig apiserver 地址后，无法创建应用
创建应用时提示:
rpc error: code = InvalidArgument desc = application destination spec for {xxxx} is invalid: unable to find destination server: there are 2 clusters with the same name: [https://xxx-apiserver1:6443 https://xxx-apiserver2:6443]"
由于 argo application 所在 project 中关联的 cluster 以 apiserver 作为唯一标识存储， 修改 apiserver 地址后,对原 cluster 进行修改 server 地址后会新建一个同名 cluster(而不是修改已有 cluster)，而导致存在两个同名 cluster。
临时解决方案： 可以将旧 cluster 删除，并手动更新所有 project 中的 server 地址至新地址（如果有）。
其它问题
如果您仍有其他问题无法解决，可以通过以下渠道联系我们：