mirror of
https://github.com/Syngnat/GoNavi.git
synced 2026-05-19 03:59:30 +08:00
* 🐛 fix(data-viewer): 修复ClickHouse尾部分页异常并增强DuckDB复杂类型兼容 - DataViewer 新增 ClickHouse 反向分页策略,修复最后页与倒数页查询失败 - DuckDB 查询失败时按列类型生成安全 SELECT,复杂类型转 VARCHAR 重试 - 分页状态统一使用 currentPage 回填,避免页码与总数推导不一致 - 增强查询异常日志与重试路径,降低大表场景卡顿与误报 * ✨ feat(frontend-driver): 驱动管理支持快速搜索并优化信息展示 - 新增搜索框,支持按 DuckDB/ClickHouse 等关键字快速定位驱动 - 显示“匹配 x / y”统计与无结果提示 - 优化头部区域排版,提升透明/暗色场景下的视觉对齐 * 🔧 fix(connection-modal): 修复多数据源URI导入解析并校正Oracle服务名校验 - 新增单主机URI解析映射,兼容 postgres/postgresql、sqlserver、redis、tdengine、dameng(dm)、kingbase、highgo、vastbase、clickhouse、oracle - 抽取 parseSingleHostUri 复用逻辑,统一 host/port/user/password/database 回填行为 - Oracle 连接新增服务名必填校验,移除“服务名为空回退用户名”的隐式逻辑 - 连接弹窗补充 Oracle 服务名输入项与 URI 示例 * 🐛 fix(query-export): 修复查询结果导出卡住并统一按数据源能力控制导出路径 - 查询结果页导出增加稳定兜底,异常时确保 loading 关闭避免持续转圈 - DataGrid 导出逻辑按数据源能力分流,优先走后端 ExportQuery 并保留结果集导出降级 - QueryEditor 传递结果导出 SQL,保证查询结果导出范围与当前结果一致 - 后端补充 ExportData/ExportQuery 关键日志,提升导出链路可观测性 * 🐛 fix(precision): 修复查询链路与分页统计的大整数精度丢失 - 代理响应数据解码改为 UseNumber,避免默认 float64 吞精度 - 统一归一化 json.Number 与超界整数,超出 JS 安全范围转字符串 - 修复 DataViewer 总数解析,超大值不再误转 Number 参与分页 - refs #142 * 🐛 fix(driver-manager): 修复驱动管理网络告警重复并强化代理引导 - 新增下载链路域名探测,区分“GitHub可达但驱动下载链路不可达” - 网络不可达场景仅保留红色强提醒,移除重复二级告警 - 强提醒增加“打开全局代理设置”入口,优先引导使用 GoNavi 全局代理 - 统一网络检测与目录说明提示图标尺寸,修复加载期视觉不一致 - refs #141 * ♻️ refactor(frontend-interaction): 统一标签拖拽与暗色主题交互实现 - 重构Tab拖拽排序实现,统一为可配置拖拽引擎 - 规范拖拽与点击事件边界,提升交互一致性 - 统一多组件暗色透明样式策略,减少硬编码色值 - 提升Redis/表格/连接面板在透明模式下的观感一致性 - refs #144 * ♻️ refactor(update-state): 重构在线更新状态流并按版本统一进度展示 - 重构更新检查与下载状态同步流程,减少前后端状态分叉 - 进度展示严格绑定 latestVersion,避免跨版本状态串用 - 优化 about 打开场景的静默检查状态回填逻辑 - 统一下载弹窗关闭/后台隐藏行为 - 保持现有安装流程并补齐目录打开能力 * 🎨 style(sidebar-log): 将SQL执行日志入口调整为悬浮胶囊样式 - 移除侧栏底部整条日志入口容器 - 新增悬浮按钮阴影/边框/透明背景并适配明暗主题 - 为树区域预留底部空间避免入口遮挡内容 * ✨ feat(redis-cluster): 支持集群模式逻辑多库隔离与 0-15 库切换 - 前端恢复 Redis 集群场景下 db0-db15 的数据库选择与展示 - 后端新增集群逻辑库命名空间前缀映射,统一 key/pattern 读写隔离 - 覆盖扫描、读取、写入、删除、重命名等核心操作的键映射规则 - 集群命令通道支持 SELECT 逻辑切库与 FLUSHDB 逻辑库清空 - refs #145 * ✨ feat(DataGrid): 大数据表虚拟滚动性能优化及UI一致性修复 - 启用动态虚拟滚动(数据量≥500行自动切换),解决万行数据表卡顿问题 - 虚拟模式下EditableCell改用div渲染,CSS选择器从元素级改为类级适配虚拟DOM - 修复虚拟模式双水平滚动条:样式化rc-virtual-list内置滚动条为胶囊外观,禁用自定义外部滚动条 - 为rc-virtual-list水平滚动条添加鼠标滚轮支持(MutationObserver + marginLeft驱动) - 修复白色主题透明模式下列名悬浮Tooltip对比度不足的问题 - 新增白色主题全局滚动条样式适配透明模式(App.css) - App.tsx主题token与组件样式优化 - refs #147 * 🔧 chore(app): 清理 App.tsx 类型告警并收敛前端壳层实现 - 清除未使用代码和冗余状态 - 替换弃用 API 以消除 IDE 提示 - 显式处理浮动 Promise 避免告警 - 保持现有更新检查和代理设置行为不变 * 🔧 fix(ci): 修复 Windows AMD64 下 DuckDB 驱动构建链路 - 将 DuckDB 工具链准备切换为优先使用 MSYS2 - 增加 gcc 和 g++ 存在性校验与版本验证 - 在 MSYS2 异常时回退 Chocolatey 安装 MinGW - 保持 Windows ARM64 跳过 DuckDB 构建与平台支持一致 * 🔧 fix(ci): 修复 Windows AMD64 下 DuckDB 驱动构建链路 - 将 DuckDB 工具链准备切换为优先使用 MSYS2 - 增加 gcc 和 g++ 存在性校验与版本验证 - 在 MSYS2 异常时回退 Chocolatey 安装 MinGW - 保持 Windows ARM64 跳过 DuckDB 构建与平台支持一致 * 🔧 fix(ci): 修复 Windows AMD64 下 DuckDB 驱动构建工具链 - 将 DuckDB 编译链从 MINGW64 切换为 MSYS2 UCRT64 - 修正 Windows AMD64 的 gcc 和 g++ 探测路径 - 增加 DuckDB 编译器版本校验步骤 * 📝 docs(contributing): 补充中英文贡献指南并统一 README 入口 - 新增英文版 CONTRIBUTING.md 作为正式贡献文档 - 新增中文版 CONTRIBUTING.zh-CN.md 作为中文贡献说明 - 调整 README 和 README.zh-CN 的贡献入口指向对应语言文档 * - feat(connection,metadata,kingbase): 增强多数据源连接能力并修复金仓/达梦/Oracle/ClickHouse兼容性问题 (#188) (#190) * feat(http-tunnel): 支持独立 HTTP 隧道连接并覆盖多数据源 refs #168 * fix(kingbase-data-grid): 修复金仓打开表卡顿并降低对象渲染开销 refs #178 * fix(kingbase-transaction): 修复金仓事务提交重复引号导致语法错误 refs #176 * fix(driver-agent): 修复老版本 Win10 升级后金仓驱动代理启动失败 refs #177 * chore(ci): 新增手动触发的 macOS 测试构建工作流 * chore(ci): 允许测试工作流在当前分支自动触发 * fix(query-editor): 修复 SQL 编辑中光标随机跳到末尾 refs #185 * feat(data-sync): 增加差异 SQL 预览能力便于审核 refs #174 * fix(clickhouse-connect): 自动识别并回退 HTTP/Native 协议连接 refs #181 * fix(oracle-metadata): 修复视图与函数加载按 schema 过滤异常 refs #155 * fix(dameng-databases): 修复显示全部库时数据库列表不完整 refs #154 * fix(connection,db-list): 统一处理空列表返回并修复达梦连接测试报错 refs #157 Co-authored-by: 辣条 <69459608+tianqijiuyun-latiao@users.noreply.github.com> * ✨ feat(release-notes): 支持自动生成 Release 更新说明并区分配置文件命名 * 🔁 chore(sync): 回灌 main 到 dev (#192) * - feat(connection,metadata,kingbase): 增强多数据源连接能力并修复金仓/达梦/Oracle/ClickHouse兼容性问题 (#188) * feat(http-tunnel): 支持独立 HTTP 隧道连接并覆盖多数据源 refs #168 * fix(kingbase-data-grid): 修复金仓打开表卡顿并降低对象渲染开销 refs #178 * fix(kingbase-transaction): 修复金仓事务提交重复引号导致语法错误 refs #176 * fix(driver-agent): 修复老版本 Win10 升级后金仓驱动代理启动失败 refs #177 * chore(ci): 新增手动触发的 macOS 测试构建工作流 * chore(ci): 允许测试工作流在当前分支自动触发 * fix(query-editor): 修复 SQL 编辑中光标随机跳到末尾 refs #185 * feat(data-sync): 增加差异 SQL 预览能力便于审核 refs #174 * fix(clickhouse-connect): 自动识别并回退 HTTP/Native 协议连接 refs #181 * fix(oracle-metadata): 修复视图与函数加载按 schema 过滤异常 refs #155 * fix(dameng-databases): 修复显示全部库时数据库列表不完整 refs #154 * fix(connection,db-list): 统一处理空列表返回并修复达梦连接测试报错 refs #157 * Release/0.5.3 (#191) --------- Co-authored-by: 辣条 <69459608+tianqijiuyun-latiao@users.noreply.github.com> Co-authored-by: Syngnat <92659908+Syngnat@users.noreply.github.com> * 🐛 fix(branch-sync): 修复 main 回灌 dev 时 mergeable 异步计算导致漏开自动合并 - 增加 mergeable 状态轮询,避免新建同步 PR 后立即返回 UNKNOWN - 在合并状态未稳定时输出中文告警与执行摘要 - 保持冲突分支、待计算分支与自动合并分支的处理路径清晰 * 🔁 chore(sync): 回灌 main 到 dev (#195) * - feat(connection,metadata,kingbase): 增强多数据源连接能力并修复金仓/达梦/Oracle/ClickHouse兼容性问题 (#188) * feat(http-tunnel): 支持独立 HTTP 隧道连接并覆盖多数据源 refs #168 * fix(kingbase-data-grid): 修复金仓打开表卡顿并降低对象渲染开销 refs #178 * fix(kingbase-transaction): 修复金仓事务提交重复引号导致语法错误 refs #176 * fix(driver-agent): 修复老版本 Win10 升级后金仓驱动代理启动失败 refs #177 * chore(ci): 新增手动触发的 macOS 测试构建工作流 * chore(ci): 允许测试工作流在当前分支自动触发 * fix(query-editor): 修复 SQL 编辑中光标随机跳到末尾 refs #185 * feat(data-sync): 增加差异 SQL 预览能力便于审核 refs #174 * fix(clickhouse-connect): 自动识别并回退 HTTP/Native 协议连接 refs #181 * fix(oracle-metadata): 修复视图与函数加载按 schema 过滤异常 refs #155 * fix(dameng-databases): 修复显示全部库时数据库列表不完整 refs #154 * fix(connection,db-list): 统一处理空列表返回并修复达梦连接测试报错 refs #157 * Release/0.5.3 (#191) * - chore(ci): 新增全平台测试包手动构建工作流 tianqijiuyun-latiao 今天 下午4:26 (#194) * feat(http-tunnel): 支持独立 HTTP 隧道连接并覆盖多数据源 refs #168 * fix(kingbase-data-grid): 修复金仓打开表卡顿并降低对象渲染开销 refs #178 * fix(kingbase-transaction): 修复金仓事务提交重复引号导致语法错误 refs #176 * fix(driver-agent): 修复老版本 Win10 升级后金仓驱动代理启动失败 refs #177 * chore(ci): 新增手动触发的 macOS 测试构建工作流 * chore(ci): 允许测试工作流在当前分支自动触发 * fix(query-editor): 修复 SQL 编辑中光标随机跳到末尾 refs #185 * feat(data-sync): 增加差异 SQL 预览能力便于审核 refs #174 * fix(clickhouse-connect): 自动识别并回退 HTTP/Native 协议连接 refs #181 * fix(oracle-metadata): 修复视图与函数加载按 schema 过滤异常 refs #155 * fix(dameng-databases): 修复显示全部库时数据库列表不完整 refs #154 * fix(connection,db-list): 统一处理空列表返回并修复达梦连接测试报错 refs #157 * fix(kingbase): 补齐主键识别并优化宽表卡顿 refs #176 refs #178 * fix(query-execution): 支持带前置注释的读查询结果识别 * chore(ci): 新增全平台测试包手动构建工作流 --------- Co-authored-by: 辣条 <69459608+tianqijiuyun-latiao@users.noreply.github.com> Co-authored-by: Syngnat <92659908+Syngnat@users.noreply.github.com> * ♻️ refactor(frontend-sync): 优化桌面交互细节并移除 main 回灌 dev 自动化 - 优化新建连接、主题设置、侧边栏工具区与 SQL 日志的界面表现 - 调整分页、筛选、透明模式与弹窗样式,统一整体交互层次 - 收口外观参数生效逻辑并补齐多组件适配 - 删除 sync-main-to-dev 工作流并同步维护者手动回灌说明 * feat: 统一筛选条件逻辑按钮宽度 (#201) * 🐛 fix(oracle-query): 修复 Oracle 表数据分页 SQL 兼容问题 refs #196 (#202) * ✨ feat(datasource): 支持 DuckDB Parquet 文件模式并优化弹窗打开链路 - 统一 DuckDB 文件库与 Parquet 文件接入能力 - 补充 URI、文件选择、只读挂载与连接缓存键处理 - 去掉数据源卡片点击前的同步驱动查询,修复打开卡顿 * ✨ feat(datasource): 支持 DuckDB Parquet 文件模式并优化弹窗打开链路 - 统一 DuckDB 文件库与 Parquet 文件接入能力 - 补充 URI、文件选择、只读挂载与连接缓存键处理 - 去掉数据源卡片点击前的同步驱动查询,修复打开卡顿 - refs #166 * 🐛 fix(dameng): 修复达梦连接成功后数据库列表为空问题 - 调整达梦数据库列表获取策略,优先回退查询当前 schema 与当前用户 - 保留可见用户与 owner 聚合逻辑,兼容低权限账号场景 - 补充前端空列表提示与后端单元测试,降低排查成本 - close #203 * ✨ feat(data-sync): 扩展跨库迁移链路并优化数据同步交互 - 统一同库同步与跨库迁移入口,补充模式区分与风险提示 - 扩展 ClickHouse 与 PG-like 双向迁移,并新增 PG-like、ClickHouse、TDengine 到 MongoDB 的迁移路由 - 完善 TDengine 目标端建表规划、回归测试与需求追踪文档 - refs #51 --------- Co-authored-by: Syngnat <yangguofeng919@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: 辣条 <69459608+tianqijiuyun-latiao@users.noreply.github.com> Co-authored-by: TSS <266256496+Zencok@users.noreply.github.com>
813 lines
22 KiB
Go
813 lines
22 KiB
Go
//go:build gonavi_full_drivers || gonavi_clickhouse_driver
|
||
|
||
package db
|
||
|
||
import (
|
||
"context"
|
||
"database/sql"
|
||
"fmt"
|
||
"net"
|
||
"net/url"
|
||
"sort"
|
||
"strconv"
|
||
"strings"
|
||
"time"
|
||
|
||
"GoNavi-Wails/internal/connection"
|
||
"GoNavi-Wails/internal/logger"
|
||
"GoNavi-Wails/internal/ssh"
|
||
"GoNavi-Wails/internal/utils"
|
||
|
||
clickhouse "github.com/ClickHouse/clickhouse-go/v2"
|
||
)
|
||
|
||
const (
|
||
defaultClickHousePort = 9000
|
||
defaultClickHouseUser = "default"
|
||
defaultClickHouseDatabase = "default"
|
||
minClickHouseReadTimeout = 5 * time.Minute
|
||
)
|
||
|
||
type ClickHouseDB struct {
|
||
conn *sql.DB
|
||
pingTimeout time.Duration
|
||
forwarder *ssh.LocalForwarder
|
||
database string
|
||
}
|
||
|
||
func normalizeClickHouseConfig(config connection.ConnectionConfig) connection.ConnectionConfig {
|
||
normalized := applyClickHouseURI(config)
|
||
if strings.TrimSpace(normalized.Host) == "" {
|
||
normalized.Host = "localhost"
|
||
}
|
||
if normalized.Port <= 0 {
|
||
normalized.Port = defaultClickHousePort
|
||
}
|
||
if strings.TrimSpace(normalized.User) == "" {
|
||
normalized.User = defaultClickHouseUser
|
||
}
|
||
if strings.TrimSpace(normalized.Database) == "" {
|
||
normalized.Database = defaultClickHouseDatabase
|
||
}
|
||
return normalized
|
||
}
|
||
|
||
func applyClickHouseURI(config connection.ConnectionConfig) connection.ConnectionConfig {
|
||
uriText := strings.TrimSpace(config.URI)
|
||
if uriText == "" {
|
||
return config
|
||
}
|
||
lowerURI := strings.ToLower(uriText)
|
||
if !strings.HasPrefix(lowerURI, "clickhouse://") {
|
||
return config
|
||
}
|
||
|
||
parsed, err := url.Parse(uriText)
|
||
if err != nil {
|
||
return config
|
||
}
|
||
|
||
if parsed.User != nil {
|
||
if strings.TrimSpace(config.User) == "" {
|
||
config.User = parsed.User.Username()
|
||
}
|
||
if pass, ok := parsed.User.Password(); ok && config.Password == "" {
|
||
config.Password = pass
|
||
}
|
||
}
|
||
|
||
if dbName := strings.TrimPrefix(strings.TrimSpace(parsed.Path), "/"); dbName != "" && strings.TrimSpace(config.Database) == "" {
|
||
config.Database = dbName
|
||
}
|
||
if strings.TrimSpace(config.Database) == "" {
|
||
if dbName := strings.TrimSpace(parsed.Query().Get("database")); dbName != "" {
|
||
config.Database = dbName
|
||
}
|
||
}
|
||
|
||
defaultPort := config.Port
|
||
if defaultPort <= 0 {
|
||
defaultPort = defaultClickHousePort
|
||
}
|
||
if strings.TrimSpace(config.Host) == "" {
|
||
host, port, ok := parseHostPortWithDefault(parsed.Host, defaultPort)
|
||
if ok {
|
||
config.Host = host
|
||
config.Port = port
|
||
}
|
||
}
|
||
if config.Port <= 0 {
|
||
config.Port = defaultPort
|
||
}
|
||
return config
|
||
}
|
||
|
||
func (c *ClickHouseDB) buildClickHouseOptions(config connection.ConnectionConfig) *clickhouse.Options {
|
||
connectTimeout := getConnectTimeout(config)
|
||
readTimeout := connectTimeout
|
||
if readTimeout < minClickHouseReadTimeout {
|
||
readTimeout = minClickHouseReadTimeout
|
||
}
|
||
protocol := detectClickHouseProtocol(config)
|
||
opts := &clickhouse.Options{
|
||
Protocol: protocol,
|
||
Addr: []string{
|
||
net.JoinHostPort(config.Host, strconv.Itoa(config.Port)),
|
||
},
|
||
Auth: clickhouse.Auth{
|
||
Database: strings.TrimSpace(config.Database),
|
||
Username: strings.TrimSpace(config.User),
|
||
Password: config.Password,
|
||
},
|
||
DialTimeout: connectTimeout,
|
||
ReadTimeout: readTimeout,
|
||
}
|
||
if tlsConfig := resolveGenericTLSConfig(config); tlsConfig != nil {
|
||
opts.TLS = tlsConfig
|
||
}
|
||
return opts
|
||
}
|
||
|
||
func detectClickHouseProtocol(config connection.ConnectionConfig) clickhouse.Protocol {
|
||
uriText := strings.ToLower(strings.TrimSpace(config.URI))
|
||
if strings.HasPrefix(uriText, "http://") || strings.HasPrefix(uriText, "https://") {
|
||
return clickhouse.HTTP
|
||
}
|
||
if config.Port == 8123 || config.Port == 8443 {
|
||
return clickhouse.HTTP
|
||
}
|
||
return clickhouse.Native
|
||
}
|
||
|
||
func isClickHouseProtocolMismatch(err error) bool {
|
||
if err == nil {
|
||
return false
|
||
}
|
||
text := strings.ToLower(strings.TrimSpace(err.Error()))
|
||
if text == "" {
|
||
return false
|
||
}
|
||
return strings.Contains(text, "unexpected packet [72]") ||
|
||
(strings.Contains(text, "unexpected packet") && strings.Contains(text, "handshake")) ||
|
||
strings.Contains(text, "http response to https client") ||
|
||
strings.Contains(text, "malformed http response")
|
||
}
|
||
|
||
func withClickHouseProtocol(config connection.ConnectionConfig, protocol clickhouse.Protocol) connection.ConnectionConfig {
|
||
next := config
|
||
switch protocol {
|
||
case clickhouse.HTTP:
|
||
if next.Port == 0 {
|
||
next.Port = 8123
|
||
}
|
||
default:
|
||
if next.Port == 0 {
|
||
next.Port = defaultClickHousePort
|
||
}
|
||
}
|
||
return next
|
||
}
|
||
|
||
func (c *ClickHouseDB) Connect(config connection.ConnectionConfig) error {
|
||
if supported, reason := DriverRuntimeSupportStatus("clickhouse"); !supported {
|
||
if strings.TrimSpace(reason) == "" {
|
||
reason = "ClickHouse 纯 Go 驱动未启用,请先在驱动管理中安装启用"
|
||
}
|
||
return fmt.Errorf("%s", reason)
|
||
}
|
||
|
||
if c.forwarder != nil {
|
||
_ = c.forwarder.Close()
|
||
c.forwarder = nil
|
||
}
|
||
if c.conn != nil {
|
||
_ = c.conn.Close()
|
||
c.conn = nil
|
||
}
|
||
|
||
runConfig := normalizeClickHouseConfig(config)
|
||
c.pingTimeout = getConnectTimeout(runConfig)
|
||
c.database = runConfig.Database
|
||
|
||
if runConfig.UseSSH {
|
||
logger.Infof("ClickHouse 使用 SSH 连接:地址=%s:%d 用户=%s", runConfig.Host, runConfig.Port, runConfig.User)
|
||
forwarder, err := ssh.GetOrCreateLocalForwarder(runConfig.SSH, runConfig.Host, runConfig.Port)
|
||
if err != nil {
|
||
return fmt.Errorf("创建 SSH 隧道失败:%w", err)
|
||
}
|
||
c.forwarder = forwarder
|
||
|
||
host, portText, err := net.SplitHostPort(forwarder.LocalAddr)
|
||
if err != nil {
|
||
return fmt.Errorf("解析本地转发地址失败:%w", err)
|
||
}
|
||
port, err := strconv.Atoi(portText)
|
||
if err != nil {
|
||
return fmt.Errorf("解析本地端口失败:%w", err)
|
||
}
|
||
|
||
runConfig.Host = host
|
||
runConfig.Port = port
|
||
runConfig.UseSSH = false
|
||
logger.Infof("ClickHouse 通过本地端口转发连接:%s -> %s:%d", forwarder.LocalAddr, config.Host, config.Port)
|
||
}
|
||
|
||
attempts := []connection.ConnectionConfig{runConfig}
|
||
if shouldTrySSLPreferredFallback(runConfig) {
|
||
attempts = append(attempts, withSSLDisabled(runConfig))
|
||
}
|
||
|
||
var failures []string
|
||
for idx, attempt := range attempts {
|
||
primaryProtocol := detectClickHouseProtocol(attempt)
|
||
protocols := []clickhouse.Protocol{primaryProtocol}
|
||
if primaryProtocol == clickhouse.Native {
|
||
protocols = append(protocols, clickhouse.HTTP)
|
||
} else {
|
||
protocols = append(protocols, clickhouse.Native)
|
||
}
|
||
|
||
for pIdx, protocol := range protocols {
|
||
protocolConfig := withClickHouseProtocol(attempt, protocol)
|
||
c.conn = clickhouse.OpenDB(c.buildClickHouseOptions(protocolConfig))
|
||
if err := c.Ping(); err != nil {
|
||
failures = append(failures, fmt.Sprintf("第%d次连接验证失败(protocol=%s): %v", idx+1, protocol.String(), err))
|
||
if c.conn != nil {
|
||
_ = c.conn.Close()
|
||
c.conn = nil
|
||
}
|
||
if pIdx == 0 && !isClickHouseProtocolMismatch(err) {
|
||
// 首次连接不是协议误配特征,避免无谓重试次协议。
|
||
break
|
||
}
|
||
continue
|
||
}
|
||
if idx > 0 {
|
||
logger.Warnf("ClickHouse SSL 优先连接失败,已回退至明文连接")
|
||
}
|
||
if pIdx > 0 {
|
||
logger.Warnf("ClickHouse 已自动切换连接协议为 %s(常见于 8123/8443 HTTP 端口)", protocol.String())
|
||
}
|
||
return nil
|
||
}
|
||
}
|
||
|
||
_ = c.Close()
|
||
return fmt.Errorf("连接建立后验证失败(可检查 ClickHouse 端口与协议是否匹配:Native=9000/9440,HTTP=8123/8443):%s", strings.Join(failures, ";"))
|
||
}
|
||
|
||
func (c *ClickHouseDB) Close() error {
|
||
if c.forwarder != nil {
|
||
if err := c.forwarder.Close(); err != nil {
|
||
logger.Warnf("关闭 ClickHouse SSH 端口转发失败:%v", err)
|
||
}
|
||
c.forwarder = nil
|
||
}
|
||
if c.conn != nil {
|
||
return c.conn.Close()
|
||
}
|
||
return nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) Ping() error {
|
||
if c.conn == nil {
|
||
return fmt.Errorf("connection not open")
|
||
}
|
||
timeout := c.pingTimeout
|
||
if timeout <= 0 {
|
||
timeout = 5 * time.Second
|
||
}
|
||
ctx, cancel := utils.ContextWithTimeout(timeout)
|
||
defer cancel()
|
||
return c.conn.PingContext(ctx)
|
||
}
|
||
|
||
func (c *ClickHouseDB) QueryContext(ctx context.Context, query string) ([]map[string]interface{}, []string, error) {
|
||
if c.conn == nil {
|
||
return nil, nil, fmt.Errorf("connection not open")
|
||
}
|
||
rows, err := c.conn.QueryContext(ctx, query)
|
||
if err != nil {
|
||
return nil, nil, err
|
||
}
|
||
defer rows.Close()
|
||
return scanRows(rows)
|
||
}
|
||
|
||
func (c *ClickHouseDB) Query(query string) ([]map[string]interface{}, []string, error) {
|
||
if c.conn == nil {
|
||
return nil, nil, fmt.Errorf("connection not open")
|
||
}
|
||
rows, err := c.conn.Query(query)
|
||
if err != nil {
|
||
return nil, nil, err
|
||
}
|
||
defer rows.Close()
|
||
return scanRows(rows)
|
||
}
|
||
|
||
func (c *ClickHouseDB) ExecContext(ctx context.Context, query string) (int64, error) {
|
||
if c.conn == nil {
|
||
return 0, fmt.Errorf("connection not open")
|
||
}
|
||
res, err := c.conn.ExecContext(ctx, query)
|
||
if err != nil {
|
||
return 0, err
|
||
}
|
||
return res.RowsAffected()
|
||
}
|
||
|
||
func (c *ClickHouseDB) Exec(query string) (int64, error) {
|
||
if c.conn == nil {
|
||
return 0, fmt.Errorf("connection not open")
|
||
}
|
||
res, err := c.conn.Exec(query)
|
||
if err != nil {
|
||
return 0, err
|
||
}
|
||
return res.RowsAffected()
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetDatabases() ([]string, error) {
|
||
data, _, err := c.Query("SELECT name FROM system.databases ORDER BY name")
|
||
if err != nil {
|
||
return nil, err
|
||
}
|
||
|
||
result := make([]string, 0, len(data))
|
||
for _, row := range data {
|
||
if val, ok := getClickHouseValueFromRow(row, "name", "database"); ok {
|
||
result = append(result, fmt.Sprintf("%v", val))
|
||
continue
|
||
}
|
||
for _, value := range row {
|
||
result = append(result, fmt.Sprintf("%v", value))
|
||
break
|
||
}
|
||
}
|
||
return result, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetTables(dbName string) ([]string, error) {
|
||
targetDB := strings.TrimSpace(dbName)
|
||
if targetDB == "" {
|
||
targetDB = strings.TrimSpace(c.database)
|
||
}
|
||
|
||
var query string
|
||
if targetDB != "" {
|
||
query = fmt.Sprintf(
|
||
"SELECT name FROM system.tables WHERE database = '%s' ORDER BY name",
|
||
escapeClickHouseSQLLiteral(targetDB),
|
||
)
|
||
} else {
|
||
query = "SELECT database, name FROM system.tables ORDER BY database, name"
|
||
}
|
||
|
||
data, _, err := c.Query(query)
|
||
if err != nil {
|
||
return nil, err
|
||
}
|
||
|
||
result := make([]string, 0, len(data))
|
||
for _, row := range data {
|
||
if targetDB != "" {
|
||
if val, ok := getClickHouseValueFromRow(row, "name", "table", "table_name"); ok {
|
||
result = append(result, fmt.Sprintf("%v", val))
|
||
continue
|
||
}
|
||
} else {
|
||
databaseValue, hasDB := getClickHouseValueFromRow(row, "database", "schema_name")
|
||
tableValue, hasTable := getClickHouseValueFromRow(row, "name", "table", "table_name")
|
||
if hasDB && hasTable {
|
||
result = append(result, fmt.Sprintf("%v.%v", databaseValue, tableValue))
|
||
continue
|
||
}
|
||
}
|
||
for _, value := range row {
|
||
result = append(result, fmt.Sprintf("%v", value))
|
||
break
|
||
}
|
||
}
|
||
return result, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetCreateStatement(dbName, tableName string) (string, error) {
|
||
database, table, err := c.resolveDatabaseAndTable(dbName, tableName)
|
||
if err != nil {
|
||
return "", err
|
||
}
|
||
|
||
query := fmt.Sprintf("SHOW CREATE TABLE %s.%s", quoteClickHouseIdentifier(database), quoteClickHouseIdentifier(table))
|
||
data, _, err := c.Query(query)
|
||
if err != nil {
|
||
return "", err
|
||
}
|
||
if len(data) == 0 {
|
||
return "", fmt.Errorf("create statement not found")
|
||
}
|
||
row := data[0]
|
||
if val, ok := getClickHouseValueFromRow(row, "statement", "create_statement", "sql", "query"); ok {
|
||
text := strings.TrimSpace(fmt.Sprintf("%v", val))
|
||
if text != "" {
|
||
return text, nil
|
||
}
|
||
}
|
||
|
||
longest := ""
|
||
for _, value := range row {
|
||
text := strings.TrimSpace(fmt.Sprintf("%v", value))
|
||
if text == "" {
|
||
continue
|
||
}
|
||
if strings.Contains(strings.ToUpper(text), "CREATE ") && len(text) > len(longest) {
|
||
longest = text
|
||
}
|
||
}
|
||
if longest != "" {
|
||
return longest, nil
|
||
}
|
||
return "", fmt.Errorf("create statement not found")
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetColumns(dbName, tableName string) ([]connection.ColumnDefinition, error) {
|
||
database, table, err := c.resolveDatabaseAndTable(dbName, tableName)
|
||
if err != nil {
|
||
return nil, err
|
||
}
|
||
|
||
query := fmt.Sprintf(`
|
||
SELECT
|
||
name,
|
||
type,
|
||
default_kind,
|
||
default_expression,
|
||
is_in_primary_key,
|
||
is_in_sorting_key,
|
||
comment
|
||
FROM system.columns
|
||
WHERE database = '%s' AND table = '%s'
|
||
ORDER BY position`,
|
||
escapeClickHouseSQLLiteral(database),
|
||
escapeClickHouseSQLLiteral(table),
|
||
)
|
||
data, _, err := c.Query(query)
|
||
if err != nil {
|
||
return nil, err
|
||
}
|
||
|
||
columns := make([]connection.ColumnDefinition, 0, len(data))
|
||
for _, row := range data {
|
||
nameValue, _ := getClickHouseValueFromRow(row, "name", "column_name")
|
||
typeValue, _ := getClickHouseValueFromRow(row, "type", "data_type")
|
||
defaultKind, _ := getClickHouseValueFromRow(row, "default_kind")
|
||
defaultExpr, hasDefault := getClickHouseValueFromRow(row, "default_expression", "column_default")
|
||
commentValue, _ := getClickHouseValueFromRow(row, "comment")
|
||
inPrimary, _ := getClickHouseValueFromRow(row, "is_in_primary_key")
|
||
inSorting, _ := getClickHouseValueFromRow(row, "is_in_sorting_key")
|
||
|
||
colType := strings.TrimSpace(fmt.Sprintf("%v", typeValue))
|
||
nullable := "NO"
|
||
if strings.HasPrefix(strings.ToLower(colType), "nullable(") {
|
||
nullable = "YES"
|
||
}
|
||
|
||
key := ""
|
||
if isClickHouseTruthy(inPrimary) {
|
||
key = "PRI"
|
||
} else if isClickHouseTruthy(inSorting) {
|
||
key = "MUL"
|
||
}
|
||
|
||
extra := ""
|
||
kindText := strings.ToUpper(strings.TrimSpace(fmt.Sprintf("%v", defaultKind)))
|
||
if kindText != "" && kindText != "DEFAULT" {
|
||
extra = kindText
|
||
}
|
||
|
||
col := connection.ColumnDefinition{
|
||
Name: strings.TrimSpace(fmt.Sprintf("%v", nameValue)),
|
||
Type: colType,
|
||
Nullable: nullable,
|
||
Key: key,
|
||
Extra: extra,
|
||
Comment: strings.TrimSpace(fmt.Sprintf("%v", commentValue)),
|
||
}
|
||
if hasDefault && defaultExpr != nil {
|
||
text := strings.TrimSpace(fmt.Sprintf("%v", defaultExpr))
|
||
if text != "" {
|
||
col.Default = &text
|
||
}
|
||
}
|
||
columns = append(columns, col)
|
||
}
|
||
return columns, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetAllColumns(dbName string) ([]connection.ColumnDefinitionWithTable, error) {
|
||
targetDB := strings.TrimSpace(dbName)
|
||
if targetDB == "" {
|
||
targetDB = strings.TrimSpace(c.database)
|
||
}
|
||
|
||
var query string
|
||
if targetDB != "" {
|
||
query = fmt.Sprintf(`
|
||
SELECT
|
||
database,
|
||
table,
|
||
name,
|
||
type
|
||
FROM system.columns
|
||
WHERE database = '%s'
|
||
ORDER BY table, position`,
|
||
escapeClickHouseSQLLiteral(targetDB),
|
||
)
|
||
} else {
|
||
query = `
|
||
SELECT
|
||
database,
|
||
table,
|
||
name,
|
||
type
|
||
FROM system.columns
|
||
WHERE database NOT IN ('system', 'information_schema', 'INFORMATION_SCHEMA')
|
||
ORDER BY database, table, position`
|
||
}
|
||
|
||
data, _, err := c.Query(query)
|
||
if err != nil {
|
||
return nil, err
|
||
}
|
||
|
||
result := make([]connection.ColumnDefinitionWithTable, 0, len(data))
|
||
for _, row := range data {
|
||
databaseValue, _ := getClickHouseValueFromRow(row, "database")
|
||
tableValue, hasTable := getClickHouseValueFromRow(row, "table", "table_name")
|
||
nameValue, hasName := getClickHouseValueFromRow(row, "name", "column_name")
|
||
typeValue, _ := getClickHouseValueFromRow(row, "type", "data_type")
|
||
if !hasTable || !hasName {
|
||
continue
|
||
}
|
||
|
||
tableName := strings.TrimSpace(fmt.Sprintf("%v", tableValue))
|
||
if targetDB == "" {
|
||
dbText := strings.TrimSpace(fmt.Sprintf("%v", databaseValue))
|
||
if dbText != "" {
|
||
tableName = dbText + "." + tableName
|
||
}
|
||
}
|
||
|
||
result = append(result, connection.ColumnDefinitionWithTable{
|
||
TableName: tableName,
|
||
Name: strings.TrimSpace(fmt.Sprintf("%v", nameValue)),
|
||
Type: strings.TrimSpace(fmt.Sprintf("%v", typeValue)),
|
||
})
|
||
}
|
||
return result, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetIndexes(dbName, tableName string) ([]connection.IndexDefinition, error) {
|
||
return []connection.IndexDefinition{}, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetForeignKeys(dbName, tableName string) ([]connection.ForeignKeyDefinition, error) {
|
||
return []connection.ForeignKeyDefinition{}, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) GetTriggers(dbName, tableName string) ([]connection.TriggerDefinition, error) {
|
||
return []connection.TriggerDefinition{}, nil
|
||
}
|
||
|
||
func (c *ClickHouseDB) resolveDatabaseAndTable(dbName, tableName string) (string, string, error) {
|
||
rawTable := strings.TrimSpace(tableName)
|
||
if rawTable == "" {
|
||
return "", "", fmt.Errorf("table name required")
|
||
}
|
||
|
||
resolvedDB := strings.TrimSpace(dbName)
|
||
resolvedTable := rawTable
|
||
if parts := strings.SplitN(rawTable, ".", 2); len(parts) == 2 {
|
||
if dbPart := normalizeClickHouseIdentifierPart(parts[0]); dbPart != "" {
|
||
resolvedDB = dbPart
|
||
}
|
||
resolvedTable = normalizeClickHouseIdentifierPart(parts[1])
|
||
} else {
|
||
resolvedTable = normalizeClickHouseIdentifierPart(rawTable)
|
||
}
|
||
|
||
if resolvedDB == "" {
|
||
resolvedDB = strings.TrimSpace(c.database)
|
||
}
|
||
if resolvedDB == "" {
|
||
resolvedDB = defaultClickHouseDatabase
|
||
}
|
||
if resolvedTable == "" {
|
||
return "", "", fmt.Errorf("table name required")
|
||
}
|
||
return resolvedDB, resolvedTable, nil
|
||
}
|
||
|
||
func normalizeClickHouseIdentifierPart(raw string) string {
|
||
text := strings.TrimSpace(raw)
|
||
if len(text) >= 2 {
|
||
first := text[0]
|
||
last := text[len(text)-1]
|
||
if (first == '`' && last == '`') || (first == '"' && last == '"') {
|
||
text = text[1 : len(text)-1]
|
||
}
|
||
}
|
||
return strings.TrimSpace(text)
|
||
}
|
||
|
||
func quoteClickHouseIdentifier(raw string) string {
|
||
return "`" + strings.ReplaceAll(strings.TrimSpace(raw), "`", "``") + "`"
|
||
}
|
||
|
||
func escapeClickHouseSQLLiteral(raw string) string {
|
||
return strings.ReplaceAll(strings.TrimSpace(raw), "'", "''")
|
||
}
|
||
|
||
func getClickHouseValueFromRow(row map[string]interface{}, keys ...string) (interface{}, bool) {
|
||
if len(row) == 0 {
|
||
return nil, false
|
||
}
|
||
for _, key := range keys {
|
||
if value, ok := row[key]; ok {
|
||
return value, true
|
||
}
|
||
}
|
||
for existingKey, value := range row {
|
||
for _, key := range keys {
|
||
if strings.EqualFold(existingKey, key) {
|
||
return value, true
|
||
}
|
||
}
|
||
}
|
||
return nil, false
|
||
}
|
||
|
||
func isClickHouseTruthy(value interface{}) bool {
|
||
switch val := value.(type) {
|
||
case bool:
|
||
return val
|
||
case int:
|
||
return val != 0
|
||
case int8:
|
||
return val != 0
|
||
case int16:
|
||
return val != 0
|
||
case int32:
|
||
return val != 0
|
||
case int64:
|
||
return val != 0
|
||
case uint:
|
||
return val != 0
|
||
case uint8:
|
||
return val != 0
|
||
case uint16:
|
||
return val != 0
|
||
case uint32:
|
||
return val != 0
|
||
case uint64:
|
||
return val != 0
|
||
case string:
|
||
normalized := strings.ToLower(strings.TrimSpace(val))
|
||
return normalized == "1" || normalized == "true" || normalized == "yes" || normalized == "y"
|
||
default:
|
||
normalized := strings.ToLower(strings.TrimSpace(fmt.Sprintf("%v", value)))
|
||
return normalized == "1" || normalized == "true" || normalized == "yes" || normalized == "y"
|
||
}
|
||
}
|
||
|
||
func (c *ClickHouseDB) ApplyChanges(tableName string, changes connection.ChangeSet) error {
|
||
if c.conn == nil {
|
||
return fmt.Errorf("connection not open")
|
||
}
|
||
|
||
database, table, err := c.resolveDatabaseAndTable(c.database, tableName)
|
||
if err != nil {
|
||
return err
|
||
}
|
||
qualifiedTable := fmt.Sprintf("%s.%s", quoteClickHouseIdentifier(database), quoteClickHouseIdentifier(table))
|
||
|
||
for _, pk := range changes.Deletes {
|
||
whereExpr := buildClickHouseWhereClause(pk)
|
||
if whereExpr == "" {
|
||
continue
|
||
}
|
||
query := fmt.Sprintf("ALTER TABLE %s DELETE WHERE %s", qualifiedTable, whereExpr)
|
||
if _, err := c.conn.Exec(query); err != nil {
|
||
return fmt.Errorf("delete error: %v; sql=%s", err, query)
|
||
}
|
||
}
|
||
|
||
for _, update := range changes.Updates {
|
||
setExpr := buildClickHouseAssignments(update.Values)
|
||
whereExpr := buildClickHouseWhereClause(update.Keys)
|
||
if setExpr == "" || whereExpr == "" {
|
||
continue
|
||
}
|
||
query := fmt.Sprintf("ALTER TABLE %s UPDATE %s WHERE %s", qualifiedTable, setExpr, whereExpr)
|
||
if _, err := c.conn.Exec(query); err != nil {
|
||
return fmt.Errorf("update error: %v; sql=%s", err, query)
|
||
}
|
||
}
|
||
|
||
for _, row := range changes.Inserts {
|
||
query, err := buildClickHouseInsertSQL(qualifiedTable, row)
|
||
if err != nil {
|
||
return err
|
||
}
|
||
if query == "" {
|
||
continue
|
||
}
|
||
if _, err := c.conn.Exec(query); err != nil {
|
||
return fmt.Errorf("insert error: %v; sql=%s", err, query)
|
||
}
|
||
}
|
||
return nil
|
||
}
|
||
|
||
func buildClickHouseInsertSQL(qualifiedTable string, row map[string]interface{}) (string, error) {
|
||
if len(row) == 0 {
|
||
return "", nil
|
||
}
|
||
cols := make([]string, 0, len(row))
|
||
for k := range row {
|
||
if strings.TrimSpace(k) == "" {
|
||
continue
|
||
}
|
||
cols = append(cols, k)
|
||
}
|
||
if len(cols) == 0 {
|
||
return "", nil
|
||
}
|
||
sort.Strings(cols)
|
||
quotedCols := make([]string, 0, len(cols))
|
||
values := make([]string, 0, len(cols))
|
||
for _, col := range cols {
|
||
quotedCols = append(quotedCols, quoteClickHouseIdentifier(col))
|
||
values = append(values, clickHouseLiteral(row[col]))
|
||
}
|
||
return fmt.Sprintf("INSERT INTO %s (%s) VALUES (%s)", qualifiedTable, strings.Join(quotedCols, ", "), strings.Join(values, ", ")), nil
|
||
}
|
||
|
||
func buildClickHouseAssignments(values map[string]interface{}) string {
|
||
if len(values) == 0 {
|
||
return ""
|
||
}
|
||
cols := make([]string, 0, len(values))
|
||
for k := range values {
|
||
if strings.TrimSpace(k) == "" {
|
||
continue
|
||
}
|
||
cols = append(cols, k)
|
||
}
|
||
sort.Strings(cols)
|
||
parts := make([]string, 0, len(cols))
|
||
for _, col := range cols {
|
||
parts = append(parts, fmt.Sprintf("%s = %s", quoteClickHouseIdentifier(col), clickHouseLiteral(values[col])))
|
||
}
|
||
return strings.Join(parts, ", ")
|
||
}
|
||
|
||
func buildClickHouseWhereClause(keys map[string]interface{}) string {
|
||
if len(keys) == 0 {
|
||
return ""
|
||
}
|
||
cols := make([]string, 0, len(keys))
|
||
for k := range keys {
|
||
if strings.TrimSpace(k) == "" {
|
||
continue
|
||
}
|
||
cols = append(cols, k)
|
||
}
|
||
sort.Strings(cols)
|
||
parts := make([]string, 0, len(cols))
|
||
for _, col := range cols {
|
||
parts = append(parts, fmt.Sprintf("%s = %s", quoteClickHouseIdentifier(col), clickHouseLiteral(keys[col])))
|
||
}
|
||
return strings.Join(parts, " AND ")
|
||
}
|
||
|
||
func clickHouseLiteral(value interface{}) string {
|
||
switch val := value.(type) {
|
||
case nil:
|
||
return "NULL"
|
||
case bool:
|
||
if val {
|
||
return "1"
|
||
}
|
||
return "0"
|
||
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64:
|
||
return fmt.Sprintf("%v", val)
|
||
case time.Time:
|
||
return fmt.Sprintf("'%s'", val.Format("2006-01-02 15:04:05"))
|
||
case []byte:
|
||
return fmt.Sprintf("'%s'", strings.ReplaceAll(string(val), "'", "''"))
|
||
default:
|
||
return fmt.Sprintf("'%s'", strings.ReplaceAll(fmt.Sprintf("%v", val), "'", "''"))
|
||
}
|
||
}
|