ChuNa7a/ChuSQL
1
0 Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

修复bug

Browse Source
This commit is contained in:
LIm1tZz7 committed 2026-10-11 17:32:12 +08:00
1 parent fc9bbc20e4
commit e87c4ee6a8
90 files changed
+3794 -418
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/README.md
+13
View File
@@ -0,0 +1,13 @@
# AI agent 协作资料
按任务读取，不将全部资料作为每次工作的上下文。
- 根目录 `AGENTS.md`：开发规则和测试入口。
- `context/progress.txt`：唯一进度、测试计数和验收来源。
- `context/known-issues.txt`：未解决限制；`design-decisions.txt`：现有设计约束。
- `context/main-roadmap.txt`：长期阶段；`side-branch-plan.txt`：近期任务及依赖。
- `skills/chusql-*/`：类型、存储、跨组件边界三个项目专用技能。
通用技能、安装锁文件及技能生成提示词仅保留在本地，不纳入版本控制。
`gate-check/` 为开发者和 CI 共用的门禁工具；`logs/` 为忽略的本地产物。
这些协作资料不参与应用构建、运行或发行打包。
.agents/context/design-decisions.txt
+99
View File
@@ -0,0 +1,99 @@
# ChuSQL 当前设计决策
进度见 progress.txt，限制见 known-issues.txt，用户操作见 docs/。
## 架构与接口
- 后续设计按领域记录于 docs/normalization.md、docs/types.md、docs/architecture.md 和 docs/release-signing.md；只读范式入口已提供；后续实施 bound IR / DOMAIN，再推进物化、递归、MVCC、compact 和并行写。各文档的设计章节定义拟议能力，现有运行语义不变。
- 范式迁移首版保留来源，拒绝 NULL、重复行与依赖不保持；发布目标须验证约束、完整重建并使用独立原子结构 WAL 组，不能复用禁止 DDL 的会话事务拼接执行。
- 发布真实性采用 TUF 元数据链与 Ed25519，初始 root 必须可信取得；离线验证不证明最新版本，操作系统原生代码签名另行接入。
- Web / CLI → server TCP → Haskell 引擎 → FFI → Rust 存储；server 独占数据目录，Haskell 库静态链接，存储为 cdylib。
- 引擎按 AST → 语义检查 → RelOp → 优化 → 执行分层；语义与优化只读 schema，表存在性以 catalog 为准，索引由存储管理。
- JSON 使用显式稳定标签，FFI 仅提供 version / open / close / request / free / last_error；Web 不本地执行 SQL。
- 兼容值与类型语义集中于 Core.Model，运行时类型与版本化编码集中于 Core.Runtime，函数注册与绑定集中于 Engine.Runtime；旧运算符保留 Engine.Builtin，错误分类集中于 Engine.Error。
- 比较与赋值兼容性分开，整数与整值浮点统一相等和哈希，SQL NULL 比较走三值逻辑；CHECK 每语句解析一次。
## 查询与优化
- 优化保持结果语义和可观察行序；相关外层列不能被投影删除，LEFT JOIN 只下推安全的左侧谓词。
- schema 携带行数、distinct、索引和直方图，缺统计时保守退回原计划。范围成本使用 16 桶等宽直方图，无直方图时按四分之一估算。
- 内连接从各起点贪心选序，仅在严格更省且行序不可观察处重排；统计不足或不安全条件时不改写。
- 表 / 列限定名在语义层按作用域及唯一候选解析，别名优先；会话将当前库限定 DDL 转为裸名，拒绝跨库目标，库名随请求传递。
- 相关子查询保留已解析外层限定名；求值绑定补回单表物理列的限定名，绑定去重与扫描投影使用同一列值，库限定引用仍校验 schema 来源。
- JOIN 条件按子查询外层引用的值缓存替换后的表达式；首次遇到行对时才求值，每个行对仍计算普通条件，保留输出顺序、空输入和错误返回。
- WAL 提交组落盘后、首次应用前进入 recovery_required；任何应用、刷盘或检查点错误都保持该状态，完整落盘与截断成功后才解除。
- FROM 是来源树，派生表必须带别名且不能引用外层，输出取裸列名，重名报错；未定型 NULL 保留独立查询推导类型，不能声明为存储类型。
- ORDER BY 优先匹配唯一输出别名，再查来源列；只将排序表达式提前计算，非排序投影保持在 LIMIT 后求值。CTE 列清单同步重命名内层排序别名。
- 派生表从透明算子与 schema 统计估算行数，缺统计或表达式可能报错时不重排；投影下推保留计算项，避免删除错误。内存存储提供实时行数及封顶去重统计。
- DECIMAL(p,s) 使用有限浮点、ties-to-even 舍入和舍入后溢出检查，要求 1 <= p <= 308、0 <= s <= p；Haskell 与 Rust 写入入口统一边界，日志记录规范化值。
- CTE 解析期展开为派生表，只接受 SELECT，不物化；多次引用重复执行，以减少实现改动面。
- 比较及 ANY / ALL 使用独立 AST 节点，量词复用三值逻辑，空集 ANY=false、ALL=true。
- 子查询在 Filter / Compute 中按外层绑定分组复用，暂不引入半连接；扫描按页分片，至少 1000 行、最多 8 线程，结果保序，safe FFI 释放执行能力。
## 会话、事务与存储
- 每连接独立会话及工作库；短锁取得库句柄后执行，每库读写门控制可见性，表锁保护页，在用库不强行删除。
- BEGIN 物化当前库快照，事务内走内存执行，COMMIT 提交相对基线差分，ROLLBACK 丢弃快照。
- COMMIT 先跳过数据未修改的表，整数 id 用 IntMap / IntSet 查找差异与冲突；保留原行顺序及重复 id 的首行查找语义，冲突检查和提交操作共享一次差分。
- 有整数 id 时按键提交及检查写写冲突，无整数 id 时整表比对；未修改表不回写，冲突保留事务并返回 serialization_failure。
- 保存点保存不可变快照引用，同名取最近一层，回退 / 释放丢弃之后的点，结束清空；事务命令由会话层执行。
- WAL 先记录操作及提交标记，再应用、刷数据与 catalog、写检查点、截断；仅重放已提交且超过检查点的记录。
- WAL 帧保留长度、LSN、类型码、表名及 JSON payload，兼容旧记录；恢复按内容、schema 和文件存在性判断幂等。
- 检查点按库 LSN，前缀截断用同目录临时文件原子替换，未提交尾部直接截短；失败保留 WAL 并进入 recovery_required。
- 检查点采用同目录临时文件替换，长度必须为 8 字节；完整 WAL 帧损坏明确报错，尾部不完整帧保留撕裂处理。刷盘或检查点失败后封闭请求，重启重放已提交组。
- distinct 在封顶前按值引用计数维护，列封顶不清除其他列状态；直方图独立累计与递减，桶重映射导致删除目标为空时扣最近非空桶以保持总量。rebuild_stats 先扫描数值范围，再按固定边界计数，消除重建过程的桶重映射偏差并保持输入顺序无关。
- 表 / 索引 DDL 进 WAL，删表不重取已移除句柄；库建删不进 WAL，删除以墓碑目录收敛。
- 页内 compact 保持槽号及索引位置，死空间达到四分之一时自动整理；无隐藏列逐页整理，有隐藏列整表重写，不跨页填空以保留行序。
## 身份、权限与引导
- 对象标识使用数据库编号与库内表编号组合；数据库编号从 system catalog 的对象高水位分配，表编号从本库高水位分配，删除不降低计数器。对象解析只读取 catalog。
- 建表编号在 WAL 组写入前分配，随建表记录提交；目录应用时推进高水位，恢复使用日志编号。只有建表组复制操作列表，普通数据提交沿用借用列表。
- 用户是可登录角色，共用名称空间；identities 保存属性、时间、稳定编号及版本，users 只存编号和口令哈希，编号不复用。
- 最高权限按身份属性判断，不按名称、不继承；默认用户 LOGIN、角色 NOLOGIN，两者 NOSUPERUSER / ENABLED。保护最后一个启用的登录管理员。
- 仅旧格式迁移按管理员参数识别初始最高权限；旧账号保留信息，角色补编号，重名拒绝；身份、凭据及删除清理用同一 Accounts WAL。
- server 判定权限，目录封装于 Server.Catalog；角色成员取向上闭包，写入拒绝环。GRANT role TO member 表示 member 继承 role 权限。
- members 保持身份成员目录；对象授权存于受保护 __system_object_acl，使用稳定对象与身份编号，保存所有者、授予者和 grant option。CONNECT / CREATE 限数据库，四项数据权限限表或单库表范围；ALL 按作用域展开。
- PUBLIC 默认只提供业务数据库 CONNECT，可逐库撤销；CREATE 建表登记创建者所有权，数据库所有权不隐含他人表的数据权限。启用角色可继承所有权，不继承 SUPERUSER。
- 授权链从独立管理来源及所有权展开固定点；撤销清理无来源下游，保留独立来源，循环不能自持。禁用保留所有权和管理来源授权，停止权限贡献并清理失效下游。
- 旧名称授权仅迁移一次到现存稳定对象；旧全局 * 固定为当时现存表，缺失对象或身份清理，旧记录无授予者时使用管理来源。
- 权限快照按字符分块存储，比较旧快照后以同一 WAL 组替换；跨页及 Unicode 不受单行上限限制。普通存储入口隐藏并拒绝访问权限快照表。
- 服务端共享公平读写锁排序查询与权限变更；事务提交复查身份、权限和稳定对象。建表前写入创建者声明，启动时依据现存对象恢复所有权或清理未完成声明。
- 口令 / 属性推进 revision，权限变更预先展开受影响身份；TCP 只标记连接失效，下一请求回 unauthorized 后关闭，避免 Windows 阻塞句柄异步中断；登记在 finally 清理。
- 系统表统一 __system_ 前缀并分层保护，内部目录按白名单访问；旧名按文件先、catalog 后迁移，崩溃后可重复收敛。
- 安装先 bootstrap：建立 system、默认 root（LOGIN / SUPERUSER / ENABLED）、18 个内置名称，再起服务；引导请求不开放 TCP。
- 清单编入引导并与解析器测试比对；system.__system_types 保存名称、基础表示、参数能力、编号。重复引导保留口令并补齐类型，冲突或身份降权失败；类型建表与写入用同组 WAL。
- sudo 认证用独立 allow_sudo_auth 属性和显式 sudo_auth_user 映射；默认关闭，只有 SUPERUSER 能管理，不继承、不隐含权限。目录管理者不得管理启用该属性的身份。
- 选择操作系统保护的随机本机凭据与 TCP 一次性 HMAC 挑战，以复用三平台既有会话协议；Unix 只信任有效 UID 0，Windows 只信任提升 Administrators 令牌。双方校验内核回环地址，不接受客户端自报 OS 用户或 SUDO_USER。
- 本机凭据只存放于系统保护的端口专属目录，原生入口校验提升身份；实例重启重新生成，现存目录拒绝覆盖。此方案要求服务端也提升运行，以保证创建凭据时的操作系统保护边界。
- 挑战绑定账号、连接及实例，60 秒过期并在每次尝试时消费；证明常量时间比对，凭据不经网络发送。成功登录复用身份版本、登录时间及连接失效机制，审计不含凭据或证明。
- sudo 属性由 Accounts WAL 持久化，旧目录缺列触发身份快照重写，旧记录缺属性默认 false，已有的错误类型明确拒绝。
## 自定义类型
- 范式检查采用阿姆斯特朗属性闭包与子集枚举，候选键声明转为决定全关系的依赖；完整覆盖声明与领域原子性分别控制可证明结论，不从样本推断。二元 BCNF 建议按违反依赖无损拆分，投影依赖之并用于验证依赖保持。
- DOMAIN 命名基础类型，列编码 domain(名称,基础类型) 保留引用，值及索引沿用底层语义。
- 各库 __system_domain_<名称> 空目录表的 base 列保存定义，复用 catalog / DDL WAL；普通 SQL 隐藏并保护目录。
- 嵌套只引用当前库已有类型，语义层从 schema 递归解析并拒绝循环，存储写锁内逐层校验；删除检查类型间及间接列依赖。
- 类型 DDL 要求管理员、事务内禁止；约束与默认值后续扩展，暂不引入自定义函数 / 运算符。
- 运行时类型使用稳定构造器编号与结构化 TypeId，逻辑类型与 PhysicalType 分离；递归值使用版本 1 标签编码，读取时校验类型、形状及载荷。
- TypedExpression / TypedPlan 仅使用 schema 绑定，SQL BoundQuery 通过既有 RelOp 执行并按输出位置验证；输出类型保留嵌套 DOMAIN 名称，基础日期、时间戳及 Blob 保留兼容类型，函数结果不继承参数领域。
- TypedExpression 列引用保存输入 schema 槽位与诊断名称；独立求值要求行顺序匹配，TypedPlan 扫描边界校验并按 schema 排序，每层投影绑定到子计划输出。SQL RelOp 在优化后绑定表达式列槽位，投影单向转为 Compute，排序/分组转为 BoundSort/BoundAggregate，存储结果按算子布局排列；子查询局部列在前、外层绑定在后。绑定算子不参与后续投影下推；独立作用域身份仍待迁移。
- SQL 局部引用使用 ColumnSlot，外层引用使用 OuterColumnSlot 的相对位置及局部布局宽度。重新绑定外层引用只查询外层布局，避免同名局部捕获；语义输入拒绝执行槽位。外层行仍扁平化，多级词法作用域/来源编号另行迁移。
- BoundQuery 执行前重新取得 schema 并比较依赖对象身份、schema 版本及列定义，依赖覆盖全部嵌套子查询及来源列的传递 DOMAIN 定义。IPC 将依赖守卫限定在本次执行环境；扫描、分片、点查及范围读均在相关库读锁内复查全部依赖后读取，跨库读锁按库名排序，持有库引用防止删除。数据、统计和无关对象不参与比较；旧目录缺失版本时仍比较列定义，已知身份或版本丢失则拒绝。守卫响应必须明确确认，旧二进制忽略字段时拒绝；未实现守卫的后端明确报错，纯内存视图可直接执行。守卫不改变既有多请求查询的数据隔离语义。
- 内建函数先解析为稳定 FunctionId 并验证签名，再执行；SQL NULL 保留三值语义。缺少能力字典的运行时类型拒绝比较、排序、分组、极值、索引及唯一键声明。
- SQL 字面量生成返回明确错误，先验证运行时值与有限浮点；复合值在 Web 使用 SQL 修改。
## 配置、构建与发行
- Web 编辑在首个业务表加载时建立快照，在提交 / 回退时结束；既有大库基准 BEGIN 269.806 ms、COMMIT 376.345 ms，不按每次单元格编辑重建整库快照。
- Web 登录分别建立普通 API / SQL 和编辑连接，页面 / 数据库标识绑定编辑事务，SQL 事务另绑定查询标签；同 Cookie 请求串行执行。
- 编辑集保留在浏览器，提交从固定保存点重放；失败回退整个编辑集，冲突回滚并重新加载，成功回执去重；系统身份修改不进入数据事务。
- system_catalog_manager 只管理普通身份，不继承、不隐含业务访问权限；存储写锁内再次拒绝特权目标及特权属性修改。
- bootstrap 维护使用独占数据目录锁，在副本中修复并验证后保留原目录备份再发布；repair 不重放 WAL、不补初始化数据，原有数据文件必须字节一致。
- listen_host 是监听配置，static 固定为静态目录，管理员名由 --user 指定；data_dir 存数据，log_files 存日志，相对路径以启动目录解释。
- resources/chusql.toml 入库并打包，UI 设置样本不入库、不进包，运行时按 resources、上一级 resources、工作目录候选查找。
- 预编译包优先，server / 存储必装，Web / CLI 可选；Windows zip、Linux/macOS tar.gz。包内模板也放 resources/，按确切产物名打包并拒绝未知工件。
- 本地解包安装不联网，管理员名同时传 bootstrap 和 server；安装冒烟隔离配置并恢复用户 PATH，卸载可保留数据 / 配置。
- 门禁先刷新 release cdylib，再并行 Rust、串行 Haskell、注释三条车道；Web 先刷新依赖，任一步失败即失败。
- 所有者运行全量门禁，agent 只测试改动模块；日志 / 临时文件在 logs/，必要时 -Full；门禁及打包强制 UTF-8 以避免 GBK 编码失败。
- CLI 终端隐藏口令，管道兼容 LF / CRLF / 无行尾，保留空格与内部回车，提示写 stderr。
- CLI 禁用逐行自动历史，只记录完整 SQL 与元命令；含 IDENTIFIED / PASSWORD（忽略大小写）的输入整体排除，避免多行口令或执行失败留下片段。解析错误也隐藏此类原文，筛选保守包含字符串和标识符中的同名文本，已有历史不自动删除。
- TCP 与 Web 使用同一解析诊断脱敏路径；Web 回归校验整个 HTTP 错误响应不含口令。内置 SQL 控制台记录结果或诊断，不将 SQL 原文写入日志，外部代理和采集需自行配置。
- BoundQuery 保存来源对象身份及列定义；DOMAIN 依赖从来源列递归展开，执行前复查。库 / 对象编号在 TableInfo 中输出十进制字符串，读取兼容旧数值，零对象编号表示无有效身份。
- 表级 schema_version 由 catalog 持久化，新表从 1 开始，旧目录缺失版本保留 0；结构变化递增，新增结构 WAL 记录保存确定目标版本，恢复不重新分配。线上使用精确十进制字符串并兼容数值输入，0 或缺失表示未知版本；IPC schema / 快照与内存 Server 保留版本，内存 DDL 同样递增并拒绝耗尽。
- 注释以 AGENTS.md 为准；原因写本文件，限制写 known-issues.txt，进度及计数只以 progress.txt 为准。
.agents/context/known-issues.txt
+54
View File
@@ -0,0 +1,54 @@
# ChuSQL 当前限制与风险
进度见 progress.txt，设计见 design-decisions.txt；仅记录未解决的限制。
## 事务、恢复与并发
- 事务是整库内存快照，不是行版本 MVCC；每会话一个事务，无嵌套事务，事务内禁止 DDL / USE。
- BEGIN / COMMIT 仍有整库快照传输与修改表比较成本，复测见 docs/performance.md。
- Web 首次加载业务表即固定编辑快照，同一登录会话只允许一个页面持有编辑事务；系统身份表只读，通过 SQL 管理。
- 页面关闭的回滚请求可能因断线未送达；此时按会话空闲 / 最长寿命过期清理，后台每分钟检查一次。编辑连接失效后必须重新登录，不能恢复旧快照。
- COMMIT 按整数 id 校验写写冲突，缺少整数 id 时整表比对及替换；冲突返回 serialization_failure，调用方回滚或重试，无自动重试或死锁检测。
- WAL 只有 redo，检查点按库 LSN 而非页 LSN，幂等恢复代价随数据量增长；库目录建删不走 WAL，删库用墓碑目录处理。
- recovery_required 期间除 Ping / SystemStatus 外请求被拒绝，需恢复后继续。
- 每库读共享、写独占，写入会阻塞同库跨表读取，尚无并行写。
- 权限变更只标记 TCP 连接失效，客户端下一次请求收到 unauthorized 后关闭，空闲连接不会立即断开；并发测试不证明性能提升。
## 查询与性能
- 基础成本及贪心连接重排，无动态规划或跨外连接重排；只在行序不可观察处重排。
- catalog 行数可能滞后，distinct 有上限；16 桶等宽直方图只统计数值并按闭区间估算，增量维护可能偏差，桶内分布仍为估算。
- 内存统计去重上限为 4096，暂不提供直方图或索引成本；派生表可从透明算子估计行数，含聚合或可能报错表达式时仍保守。投影下推保留所有计算项，不消除潜在错误。
- 按外层绑定复用子查询覆盖 Filter / Compute / JOIN 条件；JOIN 按首次遇到的绑定惰性求值，缓存仅限单次连接。无半连接或通用物化，分组求值可能更早报告同一错误。
- CTE 解析期展开，无物化、递归或写语句；只能引用前面定义的 CTE，前向引用按普通表名解析。
- 并行只覆盖页区间扫描：schema 至少 1000 行、最多 8 线程，无页内并行，统计滞后可退回顺序扫描。
- compact 不填跨页空洞，含隐藏列时仍需整表重写。
## 权限与兼容性
- 对象权限快照整体写入，鉴权会读取身份、成员和对象元数据；尚未针对大规模权限目录做性能基准。
- 对象高水位损坏或回退后无法恢复已删除对象的编号上限；旧二进制会丢弃新增目录字段，不应降级使用已升级的数据目录。
- 无列级权限、所有权转移或角色 ADMIN OPTION。
- 旧授权缺少授予来源，迁移为管理来源；旧全局 * 只绑定迁移时现存表。没有身份改名命令；已有同名 public 身份在删除前不会解析为虚拟 PUBLIC。
- 尚无 CREATEDB / CREATEROLE、继承开关、口令到期、连接上限或 Web 专门的身份属性编辑界面。
- 旧账号与角色重名会阻止迁移；迁移后的身份目录与 DOMAIN 编码不支持旧二进制。更早的 __chusql_users 无自动迁移。
- 旧配置 host 不读取，web.static_dir / web.user 被忽略，升级需按 docs/config.md 调整。
- 引导不重设已有口令、不提升已降权身份；类型定义冲突需人工解决。
## 类型与语义
- 范式检查提供 Haskell / TCP / CLI / Web 只读入口，最多 12 列；领域原子性和依赖完整性由用户声明，信息不足时不认证高阶范式。分解建议仅一轮，不保证子关系已达 BCNF；尚无自动迁移。
- DOMAIN 无自身 CHECK / NOT NULL / DEFAULT、ALTER DOMAIN、CASCADE；尚无 ENUM 或用户定义复合类型，已有内建 List / Maybe / Tuple。内置目录只登记现有语义，不能靠增加行扩展基础类型。
- DECIMAL 仍使用浮点表示，不是精确十进制存储；整数类型共用整数表示，CHAR 不补空格。
- 少量接口设置仍用字符串类型名；Haskell 值语义和 Rust 索引键分开维护。SQL 已接入 BoundQuery，旧 ColumnType / RelOp 仍是兼容桥，完整能力、转换及关系 IR 迁移待后续阶段。
- BoundQuery 在执行前比较对象身份、schema 版本和依赖列定义，IPC 的每次读取还在相关库读锁内复查全部依赖；多请求查询仍沿用既有数据隔离语义，不提供整条查询的数据快照。旧目录缺失版本时使用列定义校验，无有效对象编号时无法证明历史对象身份；旧存储二进制未确认读守卫则拒绝执行。多级外层作用域仍使用扁平行布局。
- 运行时类型尚无 Eq / Ord / Hash 等 capability 字典，不提供比较、排序、分组、极值和索引；复合字段在 Web 通过 SQL 修改，没有结构化单元格编辑器。
- 运行时协议使用版本 1，旧客户端不支持新增类型化值；Web 的 JSON 数值仍受 JavaScript 安全整数范围限制。
- 派生表输出取裸列名，重复名称报错；未定型 NULL 仅用于查询类型推导，不可声明为存储列类型。
- ORDER BY 不接受任意表达式；已支持输出列序号和 DISTINCT，复合类型去重仍要求后续 capability 字典。
## 构建、发行与安全
- 离线 recover 依赖完整可用 WAL，不能保证修复损坏数据页或已截断的历史；未知系统表缺少结构时需备份或 reset。目录计数器损坏后无法还原已删除身份编号上限。
- 旧版进程不支持数据目录独占锁；维护前必须停止所有相关进程。
- sudo 本机认证需提升身份启动 server；当前只映射 Unix root / Windows 提升 Administrators 到一个显式配置身份，无任意 OS 用户映射或自动提权。异常退出留下的端口凭据目录须在确认停止后手工清理。
- sudo 的 Windows 提升成功路径、macOS 编译运行及 Linux 完整提升 server/CLI 链路仍待目标平台实机验收；Debian/WSL 已验证原生凭据安全边界。
- Stack 工程应串行构建；旧注册库可能被复用，必要时 --force-dirty 或所有者门禁 -Full，Web 门禁已自动刷新依赖。
- 存储协议变化后必须重建实际 cdylib，开发环境须配置 PATH / LD_LIBRARY_PATH；旧命名构建工件仍可能残留。
- 真实 server 测试须先引导并能找到 bootstrap；注释检查对部分多行语法可能误报。
- GitHub 在线安装、Linux $ORIGIN 与 macOS 发布仍需实机验证；无 Release 时在线安装无法取得预编译资源。
- SHA256 只能检测损坏，无发布签名；外部采集仍可能记录 SQL 原文。旧 CLI 历史不自动清理，任意业务字符串中的敏感数据不自动识别。
.agents/context/main-roadmap.txt
+22
View File
@@ -0,0 +1,22 @@
ChuSQL 长期路线图
交付状态和验收只以 progress.txt 为准；近期顺序见 side-branch-plan.txt。
本文件描述目标，不代表现有实现或功能承诺。
目标链路：SQL → Parser / Binder → Typed IR → Planner / Optimizer → Executor → Rust Storage。
类型能力与权限保持独立；catalog 保存对象、依赖和语义，内核保留必要基础机制。
阶段
- A 类型与函数基础：稳定 TypeId / FunctionId、注册表、逻辑/物理类型分离、版本化值；验收见 progress.txt。
- B 能力与运算（2–6）：Capability、Operator、Cast、Aggregate、参数多态与约束解析。
- C 复合与自定义类型（7–12）：ADT、Record/Composite、Newtype/Domain、参数化类型、Range、用户类型。
- D 函数与效果（13–16）：用户函数、Effect、纯 Generated Column、有副作用的 Trigger。
- E 权限与类型化查询（17–19）：权限 Capability、Query a、Rewrite / Optimizer 函数化。
- F 扩展：对象体系统一后实现 Extension 的版本、依赖、安装/升级/移除；其他访问方法按真实需求推进。
贯穿约束
- 现有 ColumnType / Expr / RelOp 有真实消费者；迁移时逐步收拢，不提前删除兼容桥。
- 每阶段验证持久化、升级、依赖和恢复，不能靠重新初始化掩盖不兼容。
- 内建对象可由可信 bootstrap 元数据恢复；用户对象依赖 WAL / 备份。
- catalog 版本与 bootstrap / recovery 匹配，系统目录索引可重建。
- 不提前增加孤立扩展或绕过事务、权限及依赖管理。
.agents/context/progress.txt
+64
View File
@@ -0,0 +1,64 @@
ChuSQL 能力与进展摘要
更新：2026-10-10
本文件是状态、测试计数及验收记录的唯一来源；用户确认其等价于 project-plan.txt。
设计见 design-decisions.txt，限制见 known-issues.txt，后续支线见 side-branch-plan.txt。
## 主线能力
- SQL：CRUD、表列约束、索引、连接、聚合、排序、分页、子查询、派生表、投影别名、CTE、比较运算符、ANY / ALL、DISTINCT / ALL 与 ORDER BY 输出列序号。
- 优化：谓词 / 投影下推、常量折叠、点查 / 范围成本选择、贪心内连接重排、16 桶直方图、按外层绑定复用子查询与并行扫描。
- 存储：页、堆表、B+Tree、catalog、WAL、LSN 检查点、幂等恢复与页内 compact。
- 会话：按连接隔离工作库、快照事务、写写冲突检查、保存点和权限变更后的会话失效。
- 身份：用户与角色统一名称空间，LOGIN / SUPERUSER / ENABLED、稳定编号、角色继承、转授权、迁移及删除清理。
- 权限查询与撤销：当前登录身份 SHOW GRANTS、REVOKE GRANT OPTION FOR 与失效下游级联清理；工作区模块回归及 2026-10-09 所有者全量门禁通过。
- 范式入口：专用 TCP 请求、CLI --normalize / \normalize 与 Web 结构区只读分析；CONNECT / SELECT、事务视图、稳定编号和 schema 指纹、声明及运行预算；工作区模块回归及 2026-10-09 所有者全量门禁通过。
- 类型：CREATE / DROP / SHOW DOMAIN、库内嵌套定义、列引用、依赖 / 循环检查及持久化。
- 相关列绑定：局部 ColumnSlot 与外层 OuterColumnSlot 分离，外层使用相对槽位，重新绑定保留外层身份；同名遮蔽及多层相关引用回归通过；仍使用扁平化外层行，多级词法作用域/来源编号尚待实施。
- 绑定执行校验：执行前比较依赖表身份、schema 版本和列定义；IPC 每次扫描、分片、点查及范围读取在相关库读锁内复查全部依赖，再读取数据，跨库读锁按库名排序。依赖覆盖来源、连接条件、投影/过滤中的嵌套查询及递归 DOMAIN 定义，忽略数据/统计和无关对象变化。同名同结构重建或结构改变后恢复原定义使旧绑定失效；旧目录缺失编号或版本时保留列定义校验，已知身份或版本丢失则拒绝。守卫须有明确确认，未支持守卫的后端拒绝执行；纯内存视图保持稳定。多请求查询仍沿用既有数据隔离语义。本批待所有者新门禁验收。
- bound IR 基础：SQL 查询输出类型保留嵌套 DOMAIN 引用及其基础类型；TypedExpression 按输入 schema 槽位绑定列；SQL RelOp 优化后绑定投影、过滤、计算、聚合参数、连接条件及排序/分组键；投影转换为 Compute，排序/分组使用 BoundSort/BoundAggregate，保留重复列、稳定排序、首次分组顺序及空输入聚合行为；相关子查询绑定局部和外层布局，存储返回按布局排列；独立作用域/来源身份及约束绑定尚待实施；槽位绑定基础已通过 2026-10-09 所有者全量门禁。
- 引导与维护：初始化 system、默认 root 和 18 个内置类型；重复引导保留口令，冲突拒绝覆盖；system_catalog_manager 与 bootstrap repair / recover / reset。
- 配置与发行：listen_host、data_dir / log_files 分离，固定 static，管理员名由 --user 指定，模板统一 resources/chusql.toml；Windows 打包安装冒烟、管道口令兼容及构建 UTF-8。
## 已实现侧分支
以下四项已通过 2026-10-05 所有者全量门禁并提交；sudo 另有平台实机验收。
- Web 编辑事务：独立连接绑定页面与数据库，跨表原子提交；保存点回退保留编辑集，冲突要求重新加载；提交去重、SQL 标签隔离与断线 / 退出清理。
- sudo 认证：默认关闭的 allow_sudo_auth 与显式本机身份映射；操作系统保护凭据、回环对端校验及一次性 HMAC 挑战；不继承、不提权，撤销使既有会话失效。
- 对象授权：持久化且不复用的库 / 表编号，所有者与 grant option；DATABASE CONNECT / CREATE、表级数据权限及 PUBLIC；跨入口鉴权、来源追踪、级联撤销和事务提交复查。
- 范式检查器：基于 schema 与声明的阿姆斯特朗闭包，分析 1NF / 2NF / 3NF / BCNF；区分信息不足，输出违反依赖与一次无损分解并验证依赖保持；只读 API，最多 12 列。
## 阶段 A：类型与函数基础
- 状态：实现完成，模块测试及所有者全量门禁通过；实现提交 8ce4239，发行检查修正 0475a61，正式版 v26.10.1 发布验收进行中。
- 固定 TypeConstructorId、结构化 TypeId、TypeExpr / Kind 与内建 Type Registry；Int / Double / Bool / String / List / Maybe / Tuple。
- 独立 PhysicalType，递归复合值、kind / 参数校验、版本化类型和值编码、协议验证、catalog schema 及存储重载。
- 固定 FunctionId 与 Builtin Function Registry，length / not 的签名解析和类型化执行；SQL NULL 保持三值语义。
- schema 驱动的 TypedExpression / TypedPlan；SQL SELECT 使用 BoundQuery，函数及构造器节点携带已解析身份，输出按位置校验。
- SQL 复合列声明、LIST / MAYBE / TUPLE、嵌套构造器、INSERT / UPDATE / CHECK / 子查询接入；CLI / Web 显示及 SQL 生成。
- 兼容边界：旧 ColumnType 和 RelOp 保留，日期 / 时间戳 / Blob 明确使用 CompatibilityType；完整 capability / operator / cast / 用户 ADT / Extension 不在本阶段。
- 本批已验证：Engine 466 / Server 141 / Web 75 / CLI 17 全部通过；前端运行时值 9 条断言及事务 10 项通过，改动文件注释检查无问题，git diff --check 通过。
- 使用说明：docs/runtime-types.md；本批日志：logs/runtime-engine.log、logs/runtime-web.log、logs/runtime-web-final.log、logs/runtime-cli.log。Server 最终回归亦通过；Web 最终日志由单独重跑生成。
## 既有模块验证
- 2026-10-10 schema 版本与读守卫：Rust catalog 持久化表级 schema_version；新表为 1，旧目录缺失字段保留未知版本 0，补列、列定义变化、删列、改名与类型名迁移递增；数据及统计变化不递增，相同定义及重复删列不递增，耗尽明确返回错误。新增删列及列定义替换 WAL 记录在落盘前确定目标版本，应用与恢复使用该版本，覆盖检查点失败、多次结构变化及重复恢复；旧 WAL 载荷仍可读取。版本以精确十进制字符串透传到 TableInfo / Table、IPC schema / 快照及内存 Server，内存 DDL 同样递增；BoundQuery 比较依赖版本并为真实读取携带守卫，覆盖恢复原定义、版本丢失、校验与读取之间的 DDL、四种读路径、跨库依赖变化、写请求拒绝及旧存储缺少确认。
- 本批 Engine 511 / Server 161 / Web 80 / Rust 189，均零失败；clippy、release 动态库构建、相关文件注释检查及 git diff --check 通过。日志见 logs/schema-guard-engine.log、logs/schema-guard-server.log、logs/schema-guard-web.log、logs/schema-guard-storage.log、logs/schema-guard-build.log、logs/schema-guard-comments.log。上一批 CLI 20 / Bootstrap 9 零失败，日志见 logs/schema-bind-cli.log、logs/schema-bind-bootstrap.log，本批未重跑。尚待所有者全量门禁，未提交。
- 2026-10-10 对象绑定批次：Table / TableInfo 保留库和表身份，IPC schema / 快照及内存 Server 目录透传；精确 64 位编号采用十进制字符串输出，兼容旧数值并拒绝非法输入。查询复查对象身份与传递 DOMAIN 依赖，覆盖同结构重建、身份丢失、跨库视角和无关领域变化。
- 本批模块验证：Engine 505 / Server 160 / Web 80 / CLI 20 / Bootstrap 9，均零失败；Web 内含前端值与事务测试，17 个相关文件注释检查及 git diff --check 通过。日志见 logs/object-bind-engine.log、logs/object-bind-server.log、logs/object-bind-web.log、logs/object-bind-cli.log、logs/object-bind-bootstrap.log。本批尚待所有者全量门禁，未提交。
- 已登记最近模块结果：Engine 501 / Server 159 / Web 80 / CLI 20 / Rust 181，均通过；2026-10-09 所有者全量门禁已验证本组结果，Bootstrap 9 项及 clippy 亦通过。
- Web 前端事务 10 项测试通过；对象授权阶段 clippy、release 构建和改动文件注释检查通过。
- sudo 原生凭据已验证 Windows 非提升拒绝路径及 Debian/WSL root 安全边界；其他平台链路验收仍待完成。
- 2026-10-09 所有者全量门禁覆盖当时工作区：cargo-test / clippy、engine / server / web / cli / bootstrap、注释检查均通过；日志见 logs/cargo.log、logs/stack.log、logs/comments.log。该结果不覆盖之后的对象绑定批次。
## 后续与验收
- 2026-10-09 所有者全量门禁通过：三条车道全部 exit=0，STEPS=0,0,0,0，GATE_EXIT=0，用时 136 秒；Engine 501 / Server 159 / Web 80 / CLI 20 / Bootstrap 9，均零失败。此结果不替代生产恢复演练、持续负载及平台实机验收。
- 2026-10-05 所有者全量门禁通过：cargo / stack / comments 全部 exit=0，STEPS=0,0,0,0，GATE_EXIT=0；bootstrap 9 项通过。Windows 正式包 CLI / Web 分组件安装验证通过。
- 正式版 v26.10.1 包含从零介绍的发行说明；按用户要求仅发布 Windows x86_64 / Linux x86_64 的 zip / tar.gz 安装包及 SHA256。要求两平台安装验证及 Linux 测试通过，XDG 验证隔离已修复。最终发布状态待登记。
- sudo 的 Windows 提升成功路径、macOS 编译运行及 Linux 完整提升链路待实机验收。
- 在线安装、Linux $ORIGIN 与 macOS 发行待目标平台验收；发布签名待设计。
- 范式自动迁移、存储查询架构扩展和 DOMAIN 自身约束等后续事项见 side-branch-plan.txt 与 known-issues.txt。
.agents/context/side-branch-plan.txt
+19
View File
@@ -0,0 +1,19 @@
ChuSQL 近期任务与依赖
实现、测试及门禁状态只以 progress.txt 为准；设计见 design-decisions.txt，限制见 known-issues.txt。
下列顺序是规划，不是已完成能力；每批按实际进度选择任务。
1. 统一绑定：完成多级词法作用域、来源/列身份、写语句和约束绑定，收拢旧兼容桥；保留 nullable、错误时序、DOMAIN 及布局语义。schema 版本和读取守卫已有实现，验收见 progress.txt；旧目录身份升级仍需安全策略。
2. DOMAIN 约束：CHECK / NOT NULL / DEFAULT、ALTER DOMAIN，覆盖写入、导入及事务提交；CASCADE 依赖对象目录与原子结构事务，发布前验证旧数据和嵌套依赖。
3. 类型能力：精确 DECIMAL、复合类型 Eq / Ord / Hash，再推进 ENUM / 用户复合类型 / ADT；统一 Haskell 值与 Rust 索引语义，明确整数范围、CHAR 和数据迁移。
4. 查询：ORDER BY 表达式 → 通用物化 → 半连接 → 递归 CTE；写入型 CTE 依赖统一写入校验。保持行序、重复、NULL、错误时序及来源解析规则。
5. 结构事务：库建删、DDL、对象依赖与版本复查纳入原子发布和恢复；解决绑定失效、权限、回滚及编号高水位，不能只解除事务内 DDL 禁令。
6. 范式迁移：依赖 DOMAIN 约束、对象目录和结构事务；首版保留来源，拒绝 NULL、重复行及依赖不保持，后续再做多轮分解。
7. 存储：MVCC → 死锁检测 → 并行写；推进页 LSN、恢复、跨页整理及隐藏列重写。保留幂等、隔离、句柄生命周期、行序与资源预算，以实测决定并行扩展。
8. 其他：连接搜索/外连接变换/统计成本；列级权限、所有权转移、ADMIN OPTION、身份属性；Web 复合值编辑、大整数和多页面事务。按真实需求独立推进。
持续验收
- 新改动须等待所有者新全量门禁，通过后提交；模块测试不替代异常恢复、磁盘错误、持续负载及并发冲突演练。
- sudo、在线安装、Linux $ORIGIN、macOS 构建发行与发布签名按目标平台验证，结果登记 progress.txt。
- 升级明确账号/授权/配置迁移、禁止降级、旧历史和异常凭据清理；不承诺旧二进制兼容、损坏页自动修复或任意 OS 用户映射。
- agent 只跑改动模块规定测试；Haskell 串行，存储协议变化重建实际动态库。所有者门禁入口为 gate-check/run-gate.ps1。
.agents/skills/chusql-boundary-contracts/SKILL.md
+64
View File
@@ -0,0 +1,64 @@
---
name: chusql-boundary-contracts
description: 修改 ChuSQL Rust/Haskell FFI、客户端与 Server 协议、跨组件序列化契约或公开接口时使用；核对调用链、线程与所有权。单模块内部算法且接口未变时不加载。
---
# ChuSQL 边界契约
## Scope
负责组件间表示、调用与生命周期。类型内容使用 chusql-type-system，落盘与恢复使用 chusql-storage-integrity，通用 unsafe 检查使用 rust-c-ffi-safety。路径相对仓库根目录，按变更读相关消费者。
## Architecture
- CLI / Web 后端通过 `chusql-interface/src/ChuSQL/Interface/Link.hs` 连接 Server TCP；`Protocol.hs` 定义网络消息，`Session.hs` 提供客户端会话操作。
- `chusql-server/src/ChuSQL/Server/TCP.hs` 分派消息，`Session.hs` 管登录 / 鉴权，`Backend.hs` 调引擎。浏览器通过 `chusql-web/src/ChuSQL/Web/API.hs` 的 HTTP API 访问 Web 后端，不直接访问 Rust。
- `chusql-core/engine/src/ChuSQL/Core/Engine/Storage/IPC.hs` 将 MonadStorage 操作编码为 JSON，经 localStorageLink 调 FFI；IPC 名称不表示当前另起存储进程或 Socket 服务。
- FFI 两端是 `chusql-core/engine/src/ChuSQL/Core/Engine/Storage/FFI.hs` 与 `chusql-core/storage/src/ffi.rs`，方向 Haskell → Rust，没有回调 ABI。
- 存储 JSON 定义于 `chusql-core/model/src/ChuSQL/Core/Protocol.hs` 与 `chusql-core/storage/src/protocol.rs`；它与 Interface TCP 属于不同协议。
## ABI 与所有权
下表符号均带 chusql_storage_ 前缀；Rust usize 对 Haskell CSize、i32 对 CInt，句柄使用 Ptr ()。不直接跨语言传 Rust struct、enum、Vec 或 Haskell 对象布局。
| 符号 | C ABI 形状 | 契约 |
| --- | --- | --- |
| version | () → const char* | 静态 NUL 字符串，不释放 |
| open | const char* → Storage* | NULL 路径用默认；失败 NULL，成功由 close 归还 |
| close | Storage* → void | NULL 安全；非空句柄只关闭一次 |
| request | const Storage*, const u8*, usize, u8**, usize* → i32 | 输入借用；输出由 Rust 分配；0 / -1 |
| maintenance | const char*, const u8*, usize, u8**, usize* → i32 | 离线操作；响应 ownership 同 request |
| free | u8*, usize → void | 归还原始指针 / 长度对，不跨分配器、不重复释放 |
| last_error | usize* → const u8* | 借用线程局部错误，立即复制、不释放 |
## Invariants
- 路径为 NUL 结尾 UTF-8；请求 / 响应为显式长度 UTF-8 JSON，响应不保证 NUL 结尾，不能用 C 字符串函数读取。
- Haskell request 使用 Ptr CChar、CSize、Ptr (Ptr CChar)、Ptr CSize 并返回 CInt，maintenance 以 CString 替代句柄。改签名同步两端与实际动态库。
- 借用指针整个调用中有效，输出可写且不与输入或其他输出重叠。Rust 拒绝 NULL / 超过 isize::MAX 的请求长度，但不能验证非空指针的分配范围 / provenance；catch_unwind 不是非法指针保护。
- 成功输出是 Box<[u8]> 转出的所有权；Haskell callStorage 复制后 finally 调 free，失败清空输出。保留取消屏蔽，避免复制 / 释放窗口泄漏。
- open / request / maintenance 捕获 Rust panic，经 NULL / -1 和 last_error 返回；业务错误也可能是 ABI 成功响应中的 JSON error，两层失败都要处理。
- close / free 捕获析构 panic 但没有状态返回，当前忽略捕获结果；不能依靠析构保证提交或宣称它报告落盘失败。
- onStorageThread 在支持 bound threads 的 RTS 中绑定调用和错误读取到同一 OS 线程；last_error 缓冲只到本线程下次设置错误前有效。
- Haskell open / close / request / maintenance 是 safe ccall，version / free / last_error 是 unsafe ccall；长 I/O 不改为 unsafe。请求期间不得关闭句柄。
- IPC 的 withStorageLink 计数在途请求，closeConnection 等空闲再关闭；不绕过它更换 / 释放链路。Env 保存会话工作库，不把全局默认库当多连接状态。
- Interface TCP 是一行 UTF-8 JSON，Hello 校验 protocolVersion=1；同一 Client 的 MVar 串行化请求 / 响应。Server 每连接建 Session 并限制行长度，客户端锁不是全局数据库锁。
- SchemaColumn 字段、method / status 标签、缺省字段、错误码及 QueryResult columns / rows 的位置关系是消费者契约，不用自动派生构造器名替代稳定标签。
- 复合值标签为 __chusql_runtime_v1，由 Core.Runtime 编解码；Rust 行仍是通用 JSON。封装变化检查 CLI / Web / 存储重载，Web 数值受 JavaScript 安全整数限制。
- 原始存储透传仍经过 Server 会话鉴权；新入口核对 TCP 分派、会话策略与 Web 消费者，不绕过对象权限或业务错误。
## Engineering Decisions
优先扩展共享协议与真实调用者，不增加无用途适配层；确认无人依赖的内部接口可直接调整。新增类型化值沿 JSON 载荷传递，只有实际需要才扩展 ABI。
Rust panic、Haskell IO 失败与业务错误按边界转成明确失败；当前没有自动 ABI 兼容协商或后台回调生命周期管理。
## Validation
- ABI / ownership 变更验证 `chusql-core/storage/tests/ffi_test.rs` 的成功、错误、长度、编码和释放路径；必要时用可用 Miri，不为普通消息变更强制安装工具。
- 存储协议变化测试相关 Rust 模块并重建实际 cdylib，再串行验证受影响 Haskell 模块；rlib 或 mock 不能证明实际 FFI 可用。
- TCP 改动验证 Hello、错误响应和真实消费者，复用 `chusql-server/test/Spec.hs`、`chusql-web/test/Spec.hs`、`chusql-cli/test/Spec.hs` 中相关行为。
- 入口见 `gate-check/run-step.ps1`，测试范围按风险决定，不自动运行所有者全量门禁。
## References
Web 值展示 / SQL 生成消费者见 `chusql-web/static/core.js`；总体路径见 `docs/architecture.md`。核对协议时读两端编码 / 解码和调用者，不依赖绝对行号。
.agents/skills/chusql-storage-integrity/SKILL.md
+53
View File
@@ -0,0 +1,53 @@
---
name: chusql-storage-integrity
description: 修改 ChuSQL 存储页、索引、序列化、catalog、系统元数据、初始化、持久化事务或恢复时使用；维护实际落盘顺序与恢复约束。普通 SQL 表达式或界面任务不加载。
---
# ChuSQL 存储完整性
## Scope
负责持久化状态及其与服务端事务的关系。类型语义使用 chusql-type-system，接口生命周期使用 chusql-boundary-contracts；不照搬 PostgreSQL 内核设计。路径相对仓库根目录，只读受影响模块。
## Architecture
- `chusql-core/storage/src/runtime.rs`：Storage 持有数据目录文件锁、管理数据库；Server 管每库 catalog、WAL、表缓存和读写门。
- 数据根目录中 system/ 保存系统库，databases/<name>/ 保存业务库。每库有 catalog.json、<table>.db、内建 <table>.idx、其他 <table>.<column>.idx、wal.log 与 wal.checkpoint；诊断日志由 log_files 单独配置。
- `chusql-core/storage/src/page.rs` 管定长页与缓存刷盘；`heap.rs` 管页内槽位、JSON 行、隐藏列和 compact；`btree.rs` 管变长字节键、重复键与叶子链范围扫描。
- `chusql-core/storage/src/catalog.rs` 管 schema、索引、统计、隐藏列、对象编号 / 高水位；`protocol.rs` 定义线上 schema 和系统表常量。
- `chusql-core/storage/src/wal.rs` 管 LSN、提交组、日志扫描与检查点；runtime.rs 的 group_via_wal、recover_wal、replay_op 连接记录与应用。
- `chusql-server/src/ChuSQL/Server/Session.hs` 管会话快照、保存点、差分与冲突；`Backend.hs` 向存储提交操作。`chusql-bootstrap/` 调用 Storage::maintenance 与 maintenance_candidate 做离线维护。
## Invariants
- 表存在性以 catalog 为准，文件存在不表示已登记；不能靠打开文件创建逻辑表。语义与优化只读取 schema。
- Catalog::save 写 pending 文件、同步后替换；单文件替换不等于 catalog、页与索引已整体原子提交。
- 普通组先 append_group 落盘，再设置 recovery_required，然后应用、刷表、推进检查点、截断日志，全部成功才解除恢复状态。
- 应用 / 刷盘 / 检查点 / 截断失败必须保持恢复状态并报错。普通业务请求被阻断；Ping / SystemStatus 例外由各自分派层处理，新增入口不得绕过状态。
- 恢复仅重放检查点之后已提交组，保持 replay_op 幂等；不应用未提交组。现有 WAL 是 redo，检查点是库级 LSN，不是 undo 或页级恢复。
- 账号引导 / 变更存在独立 WAL 路径，不能假定所有写入都走普通组函数。
- 库 / 表对象和身份编号依赖持久化高水位，删除后不复用；迁移不能降低计数器，不能从现存对象推断已删除编号上限。
- 系统身份 / 凭据、角色、授权、类型与 DOMAIN 元数据保持对应关系。引导保留已有口令和降权状态，拒绝定义 / 保留文件冲突；类型目录不是可动态扩展基础类型的机制。
- 数据目录锁覆盖存储和维护生命周期；维护前停止占用进程。每库读共享 / 写独占、表句柄加锁，不假定同库跨表并行写。
- 会话事务是整库内存快照，无嵌套事务，事务内禁止 DDL / USE。提交跳过未修改表，整数 id 按键差分，否则整表比较 / 替换；对象身份和权限需复查，冲突返回 serialization_failure 并保留事务。
- 页内 compact 保留槽位编号与索引位置；隐藏列 compact 可能整表重写。布局、键编码、行序和隐藏列改动须检查真实消费者。
- 页大小、B+Tree magic / 布局、槽位小端字段、JSON 行、WAL 标签 / 帧和 catalog 字段是格式契约。改变前明确加载、迁移或拒绝策略，不以重新初始化掩盖不兼容。
## Engineering Decisions
- 优化可减少分配或重复读取，但保留验证、刷盘顺序、错误传播与可观察行序。结构变更同时核对 WAL、应用、幂等恢复与 catalog 保存。
- 数据库目录建删不走普通表 WAL，删库使用墓碑目录；不能承诺表提交组相同的崩溃原子性。
- 离线维护在候选目录操作并保留备份；repair 不改既有表数据，recover 依赖可用 WAL，reset 重建系统目录且保留业务库。失败保留原目录 / 恢复材料。
- 当前 Catalog::load 对不存在或空文件返回空目录，JSON 损坏报错；不能宣称所有损坏都会拒绝。rename_table_files 和部分析构 / 清理路径仍忽略 I/O 错误，不作为成功持久化保证。
- 尚无行版本 MVCC、页级恢复、跨页填洞、并行写或通用结构迁移事务，不据此放松校验。
## Validation
- 按影响选择 `chusql-core/storage/tests/` 的 page / heap / btree / catalog / wal / runtime 测试；说明修正核对源码即可。
- 格式 / 元数据 / WAL 改动验证重开、提交前后中断、重复恢复、缺失 / 截断 / 损坏输入与失败后状态；断言须检测实际不一致。
- 会话差分 / 冲突变更还验证 `chusql-server/test/Spec.hs`；并发压力测试不是正确性证明。
- 规定入口为 `gate-check/run-step.ps1 -Kind cargo`，动态库构建为 -Kind build；受影响 Haskell 模块按脚本命令串行构建，全量门禁遵守 AGENTS.md。
## References
恢复操作按需读 `docs/bootstrap.md`，配置见 `docs/config.md`，限制见 `.agents/context/known-issues.txt`；涉及拟议能力时仍以源码为准。
.agents/skills/chusql-type-system/SKILL.md
+48
View File
@@ -0,0 +1,48 @@
---
name: chusql-type-system
description: 修改 ChuSQL SQL 类型、值表示、赋值转换、类型推导、函数或表达式语义时使用；提供类型模块职责与一致性约束。仅调整界面样式或无关存储性能时不加载。
---
# ChuSQL 类型系统
## Scope
负责类型语义；落盘顺序交给 chusql-storage-integrity，传输契约交给 chusql-boundary-contracts。通用语言与测试规范使用已有 Skills，不重复加载。下列路径相对仓库根目录，只读本次相关模块。
## Architecture
- `chusql-core/model/src/ChuSQL/Core/Model.hs`：`ColumnType`、列约束、`Value`；集中提供 `assignable`、`comparableTypes`、`valueFits`、`coerceValue`、相等与哈希语义。
- `chusql-core/model/src/ChuSQL/Core/Runtime.hs`：`TypeExpr` / `TypeId`、`Kind`、独立 `PhysicalType`、`RuntimeValue`、注册表和版本化编码；Engine 的 `Runtime/Types.hs` 只是重新导出。
- `chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs` 与 `Syntax/AST.hs` 解析声明和构造表达式；`Semantic.hs` 根据 schema 解析 DOMAIN、推导表达式并绑定身份。
- `chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Functions.hs` 定义固定 FunctionId 与签名，现有函数 length / not；`Runtime/SQL.hs` 连接 SQL 标量和运行时值。
- `chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Query.hs` 的 BoundQuery 保存 schema、输出类型和优化后的 RelOp，执行后按位置校验。`Runtime/Binding.hs` 另提供类型化表达式与基础计划，尚未替代完整 SQL 关系代数。
- `chusql-core/model/src/ChuSQL/Core/Protocol.hs` 共享值的 JSON 转换；Rust 的 `chusql-core/storage/src/protocol.rs` 使用 `HashMap<String, serde_json::Value>` 保存行。
## Invariants
- 构造器编号固定：Int=1、Double=2、Bool=3、String=4、List=5、Maybe=6、Tuple=7。使用 resolveType 检查参数 / kind，不把显示名当身份，不再维护第二份注册表。
- CRuntime / VRuntime 身份必须匹配；递归检查列表元素、Maybe 内容与元组字段数，Double 必须有限。保留逻辑类型与物理表示的区别。
- SQL VNull 与 RMaybe Nothing 不同；CNull 是查询推导类型，不能声明为存储列。保留三值逻辑与函数 NULL 传播。
- 赋值兼容和比较兼容分别使用 Model 入口。coerceValue 负责值收敛；整值浮点转整数、字符串长度、日期格式、DECIMAL 舍入沿现有规则处理，不在 Parser / Eval / Web 重复实现转换。
- 整数与可规范化整值浮点保持相等 / 哈希一致。底层 compareValue 的实现不表示 SQL 已开放该类型的比较、排序、分组或索引。
- DOMAIN 使用 domainTableName 与目录定义解析基础类型、依赖及循环；不能拿解析器占位基础类型直接执行。系统内建类型目录也不是通过增加行即可扩展的插件注册表。
- Date / Timestamp / Blob 在 QueryType 保留 CompatibilityType；ColumnType 与 RelOp 仍有真实消费者，不能作为无用兼容层删除。
- 类型 / 值编码版本为 1；身份、类型名、封装或值结构变化可能影响 catalog、WAL、FFI、TCP 和 Web，组合相关 Skills 核对消费者。
- Rust 当前保存通用 JSON，只做局部校验（隐藏列、DOMAIN 依赖、DECIMAL 规范化），没有 Haskell validateValue 的完整等价检查；不要依赖原始存储请求完成所有 SQL 类型校验。
- 错误沿 Either String 返回；`chusql-core/engine/src/ChuSQL/Core/Engine/Error.hs` 用消息规则映射类别 / 协议码。修改错误文字须检查分类，不用默认值或 NULL 掩盖错误。
## Engineering Decisions
复用 Model / Runtime 的语义入口，在现有类型桥上做最小修改。尚无完整 capability / operator / cast 注册机制、通用显式 CAST、用户 ADT 或 Extension；DOMAIN 自身 CHECK / NOT NULL / DEFAULT 尚未实现。不要将规划当现有契约。
DECIMAL 是有限浮点表示，采用 ties-to-even 舍入与舍入后溢出检查，并非精确十进制存储。运行时复合类型尚无 SQL Eq / Ord / Hash capability 字典。
## Validation
- 语义改动验证解析 → schema 绑定 → 执行，关注 NULL、边界值、明确错误与输出位置；说明修正只需核对源码。
- 复用 `chusql-core/engine/test/RuntimeSpec.hs`、`chusql-core/engine/test/Spec.hs`，按需覆盖固定编码、错误身份和嵌套值。
- 持久化类型变更验证 Rust 写入 / 重载与实际协议消费者；Web 值展示 / SQL 生成参考 `chusql-web/test/runtime-values.mjs`。
- 测试入口见 `gate-check/run-step.ps1`，不因加载 Skill 自动执行全量门禁或追求覆盖率。
## References
按需读 `docs/runtime-types.md`；`docs/types.md` 含未来设计，必须与实现区分。以上源码模块是契约核对入口。
.gitignore
+10 -4
View File
@@ -76,11 +76,7 @@ resources/chusql.ui.settings.json
*~ *~
\#*\# \#*\#
.\#* .\#*
AGENTS.md
agent-credientials
skills-lock.json
.dsh .dsh
.agents
# 注意：.zed/tasks.json 是本仓库共享的构建任务（stack build / run / test / repl）， # 注意：.zed/tasks.json 是本仓库共享的构建任务（stack build / run / test / repl），
# 需要入库，所以这里不忽略 .zed/ # 需要入库，所以这里不忽略 .zed/
@@ -97,3 +93,13 @@ plandocs/
Thumbs.db Thumbs.db
ehthumbs.db ehthumbs.db
desktop.ini desktop.ini
learning
# 通用技能和安装状态仅供本机；只跟踪项目专用技能
/.agents/skills/*
!/.agents/skills/chusql-boundary-contracts/
!/.agents/skills/chusql-storage-integrity/
!/.agents/skills/chusql-type-system/
/.agents/context/prompt.txt
/skills-lock.json
AGENTS.md
+39
View File
@@ -0,0 +1,39 @@
# AGENTS.md — ChuSQL 开发契约
参考：
- 唯一进度与验收来源：`.agents/context/progress.txt`（等价于 project-plan.txt）。
- 限制与设计：`.agents/context/known-issues.txt`、`.agents/context/design-decisions.txt`。
## 核心约束
- 测试必须通过，禁止绕过检查。
- 不吞错误，使用明确错误返回。
- FFI 响应由 Rust 分配并通过配套 free 归还；句柄关闭不得与请求并发。
- 表存在性以 catalog 为准。
- 语义检查和优化只使用 schema。
- 索引由存储层管理，优化器只判断可点查条件。
- 优化不能改变结果语义。
- 持久化、事务、并发、类型及 FFI 改动优先维护不变量、内存所有权与错误传播；保留必要的迁移和恢复约束。
## 开发方式
- 按任务选择已安装 Skills：编码用 ponytail / karpathy-guidelines，审查与简化按需用 ponytail-review；Rust、FFI、测试及覆盖率分别使用对应项目技能。
- 类型、值、转换或表达式语义用 `chusql-type-system`；存储、序列化、系统目录、初始化或恢复用 `chusql-storage-integrity`；FFI、跨组件协议、序列化契约或外部接口用 `chusql-boundary-contracts`（均在 `.agents/skills/`）。
- 跨领域组合相关 Skills，无关任务不加载；Skills 不覆盖更高优先级的安全和工程约束。
- 简单任务直接执行；采用最小必要改动，允许有明确收益且能验证的局部结构优化，优先删除重复和无用代码。
- 测试强度与实际风险及影响范围匹配，不机械新增测试，不盲目追求覆盖率；数据库内核微改动也须验证关键行为。
- 不强制 TDD、子代理或多阶段审查；减少无关读取、重复搜索、工具调用和输出，Skills 不应增加无收益的流程。
- 文档描述当前状态；不创建整改报告、修改流水账或无用途的兼容层，不修改 Git 历史。
## 注释规范
- 文件级：外部引用语句（Haskell `import`、Rust `use`、JS `import`、HTML 的 `<link>`/`<script>`、
CSS `@import`、脚本的依赖/参数区）之后写一段注释，概括本文件的功能；没有引用语句的文件写在
文件开头。长度不超过 3 行、合计不超过 90 字。
- 函数级：每个函数（含 Haskell 的 `where`/`let` 内函数、Rust 的私有 `fn`）前用一行不超过
30 字的注释说明它做什么。
- 摘要说明做什么；仅在必要处补充不直观的不变量、所有权与安全约束，不记录历史和被删实现。
## 协作规则
- 全量门禁入口：`gate-check/run-gate.ps1`（三条车道并行，日志写仓库根的 `logs/`）。
- 门禁由项目所有者手动运行；agent 只跑改动模块的测试，拿到门禁结果后再提交。
- engine / web / storage 使用规定测试入口。
- 规定入口见 `gate-check/run-step.ps1`；Haskell 构建串行，存储协议改动须重建实际动态库。
- 改动说明只在交流中记录。
README.md
+1
View File
@@ -44,6 +44,7 @@ csql-web
## 更多参考文档 ## 更多参考文档
- [数据库原理学习与答辩文档](learnning/README.md)
- [安装与卸载](docs/install.md) - [安装与卸载](docs/install.md)
- [配置选项](docs/config.md) - [配置选项](docs/config.md)
- [命令与选项](docs/commands.md) - [命令与选项](docs/commands.md)
chusql-bootstrap/chusql-bootstrap.cabal
+4
View File
@@ -28,6 +28,10 @@ executable csql-bootstrap
, text , text
default-language: Haskell2010 default-language: Haskell2010
ghc-options: -Wall ghc-options: -Wall
if os(linux)
ld-options: -Wl,-rpath,$ORIGIN
if os(osx)
ld-options: -Wl,-rpath,@loader_path
test-suite chusql-bootstrap-test test-suite chusql-bootstrap-test
type: exitcode-stdio-1.0 type: exitcode-stdio-1.0
chusql-cli/app/Main.hs
+20 -4
View File
@@ -2,6 +2,7 @@
module Main (main) where module Main (main) where
import ChuSQL.CLI.Normalization (runNormalizationOnce)
import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat) import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat)
import ChuSQL.CLI.Password (readPasswordInput) import ChuSQL.CLI.Password (readPasswordInput)
import ChuSQL.CLI.Script (stripTerminator) import ChuSQL.CLI.Script (stripTerminator)
@@ -19,7 +20,7 @@ import ChuSQL.Interface.Sudo (SudoCredential, readSudoCredential)
import ChuSQL.Interface.TOML (resolveConfigPath) import ChuSQL.Interface.TOML (resolveConfigPath)
import Control.Exception (finally) import Control.Exception (finally)
import Control.Monad (unless) import Control.Monad (unless)
import Data.IORef (newIORef) import Data.IORef (IORef, newIORef)
import Data.Maybe (fromMaybe) import Data.Maybe (fromMaybe)
import Data.Text (Text) import Data.Text (Text)
import qualified Data.Text as T import qualified Data.Text as T
@@ -40,6 +41,8 @@ data Options = Options
, optExecute :: Maybe Text , optExecute :: Maybe Text
, optHistory :: Maybe FilePath , optHistory :: Maybe FilePath
, optConfig :: Maybe FilePath , optConfig :: Maybe FilePath
, optNormalize :: Maybe Text
, optDeclaration :: Maybe FilePath
, optSudo :: Bool , optSudo :: Bool
} }
@@ -64,16 +67,19 @@ optionsParser =
<> metavar "FORMAT" <> metavar "FORMAT"
<> value FormatTable <> value FormatTable
<> showDefaultWith (T.unpack . formatName) <> showDefaultWith (T.unpack . formatName)
<> help "Output format: table, json or csv" <> help "Output format: table (or text), json or csv"
) )
<*> optional (T.pack <$> strOption (long "execute" <> short 'e' <> metavar "SQL" <> help "Run one statement and exit")) <*> optional (T.pack <$> strOption (long "execute" <> short 'e' <> metavar "SQL" <> help "Run one statement and exit"))
<*> optional (strOption (long "history" <> metavar "FILE" <> help "SQL history file (default ~/.chusql_history)")) <*> optional (strOption (long "history" <> metavar "FILE" <> help "SQL history file (default ~/.chusql_history)"))
<*> optional (strOption (long "config" <> metavar "FILE" <> help "Config file to read (default: the fixed settings.toml under the system config directory)")) <*> optional (strOption (long "config" <> metavar "FILE" <> help "Config file to read (default: the fixed settings.toml under the system config directory)"))
<*> optional (T.pack <$> strOption (long "normalize" <> metavar "TABLE" <> help "Analyze normalization using explicit declarations"))
<*> optional (strOption (long "declaration" <> metavar "FILE" <> help "UTF-8 JSON normalization declaration"))
<*> switch (long "sudo" <> help "Authenticate using the local root or elevated administrator identity") <*> switch (long "sudo" <> help "Authenticate using the local root or elevated administrator identity")
-- | 把 --format 的取值解析成格式 -- | 把 --format 的取值解析成格式
formatReader :: String -> Either String OutputFormat formatReader :: String -> Either String OutputFormat
formatReader raw = maybe (Left "format must be table, json or csv") Right (parseFormat (T.pack raw)) formatReader "text" = Right FormatTable
formatReader raw = maybe (Left "format must be table, text, json or csv") Right (parseFormat (T.pack raw))
-- | 命令行界面与帮助 -- | 命令行界面与帮助
parserInfo :: ParserInfo Options parserInfo :: ParserInfo Options
@@ -91,6 +97,10 @@ main = do
hSetEncoding stdout utf8 hSetEncoding stdout utf8
hSetEncoding stderr utf8 hSetEncoding stderr utf8
options <- execParser parserInfo options <- execParser parserInfo
case (optNormalize options, optDeclaration options, optExecute options, optFormat options) of
(Nothing, Nothing, _, _) -> pure ()
(Just _, Just _, Nothing, format) | format /= FormatCsv -> pure ()
_ -> reportError "--normalize requires --declaration, text/json format and cannot be combined with --execute" >> exitFailure
configPath <- resolveConfigPath (optConfig options) configPath <- resolveConfigPath (optConfig options)
base <- loadWebConfigAt configPath base <- loadWebConfigAt configPath
config <- loadServerConfigAt configPath config <- loadServerConfigAt configPath
@@ -129,7 +139,13 @@ runSession options session = do
ready <- selectDatabase session (T.strip (optDatabase options)) ready <- selectDatabase session (T.strip (optDatabase options))
if not ready if not ready
then exitFailure then exitFailure
else case optExecute options of else case (optNormalize options, optDeclaration options) of
(Just table, Just file) -> runNormalizationOnce session table file (optFormat options == FormatJson) >>= \ok -> unless ok exitFailure
_ -> runSqlSession options session formatRef
-- 执行 SQL 或进入交互循环。
runSqlSession :: Options -> Session -> IORef OutputFormat -> IO ()
runSqlSession options session formatRef = case optExecute options of
Just sql -> do Just sql -> do
ok <- runStatementOnce session (optFormat options) (stripTerminator sql) ok <- runStatementOnce session (optFormat options) (stripTerminator sql)
unless ok exitFailure unless ok exitFailure
chusql-cli/app/Repl.hs
+2
View File
@@ -6,6 +6,7 @@ module Repl
, reportError , reportError
) where ) where
import ChuSQL.CLI.Normalization (runNormalizationOnce)
import ChuSQL.CLI.Format (OutputFormat, formatName, renderResult, renderRows) import ChuSQL.CLI.Format (OutputFormat, formatName, renderResult, renderRows)
import ChuSQL.CLI.History (historySettings, rememberStatement) import ChuSQL.CLI.History (historySettings, rememberStatement)
import ChuSQL.CLI.Script (Meta (..), errorHint, metaHelp, parseMeta, takeStatement) import ChuSQL.CLI.Script (Meta (..), errorHint, metaHelp, parseMeta, takeStatement)
@@ -106,6 +107,7 @@ runMeta session formatRef meta = case meta of
MetaDatabases -> listDatabases session >> pure True MetaDatabases -> listDatabases session >> pure True
MetaTables -> listTables session >> pure True MetaTables -> listTables session >> pure True
MetaDescribe name -> describeTable session name >> pure True MetaDescribe name -> describeTable session name >> pure True
MetaNormalize table file jsonOutput -> liftIO (runNormalizationOnce session table file jsonOutput) >> pure True
MetaRoles -> listRoles session >> pure True MetaRoles -> listRoles session >> pure True
MetaConnect name -> do MetaConnect name -> do
switched <- liftIO (switchDatabase session name) switched <- liftIO (switchDatabase session name)
chusql-cli/chusql-cli.cabal
+2
View File
@@ -12,6 +12,7 @@ build-type: Simple
library library
hs-source-dirs: src hs-source-dirs: src
exposed-modules: ChuSQL.CLI.Format exposed-modules: ChuSQL.CLI.Format
ChuSQL.CLI.Normalization
ChuSQL.CLI.History ChuSQL.CLI.History
ChuSQL.CLI.Password ChuSQL.CLI.Password
ChuSQL.CLI.Script ChuSQL.CLI.Script
@@ -55,6 +56,7 @@ test-suite chusql-cli-test
, directory , directory
, aeson , aeson
, bytestring , bytestring
, chusql-interface
, chusql-cli , chusql-cli
, chusql-core-model , chusql-core-model
, hspec , hspec
chusql-cli/src/ChuSQL/CLI/Normalization.hs
+75
View File
@@ -0,0 +1,75 @@
{-# LANGUAGE OverloadedStrings #-}
module ChuSQL.CLI.Normalization (normalizationArguments, loadNormalizationDeclaration, renderNormalization, runNormalizationOnce) where
import ChuSQL.Interface.Normalization
import ChuSQL.Interface.Session (Session, sessionDatabase, analyzeNormalization)
import Control.Exception (IOException, try)
import qualified Data.Aeson as A
import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy as BL
import Data.Text (Text)
import qualified Data.Text as T
import qualified Data.Text.Encoding as TE
import qualified Data.Text.IO as TIO
import System.IO (IOMode (ReadMode), stderr, withBinaryFile)
import Text.ParserCombinators.ReadP
-- CLI 范式入口：读取严格声明文件、发送专用请求并显示报告。
-- 解析表、声明路径和独立显示格式。
normalizationArguments :: Text -> Either Text (Text, FilePath, Bool)
normalizationArguments input = case readP_to_S (skipSpaces *> sepBy token skipSpaces <* skipSpaces <* eof) (T.unpack input) of
[(parts, "")] -> case parts of
[table, "--declaration", file] -> Right (T.pack table, file, False)
[table, "--declaration", file, "--format", "text"] -> Right (T.pack table, file, False)
[table, "--declaration", file, "--format", "json"] -> Right (T.pack table, file, True)
_ -> Left usage
_ -> Left usage
where
usage = "usage: \\normalize TABLE --declaration FILE [--format text|json]"
-- 保留引号内的 Windows 路径及空格。
token = quoted '"' <++ quoted '\'' <++ munch1 (\c -> c /= ' ' && c /= '\t' && c /= '\n' && c /= '"' && c /= '\'')
-- 读取不带转义解释的引号参数。
quoted quote = char quote *> munch (/= quote) <* char quote
-- 有界读取 UTF-8 声明并拒绝重复字段。
loadNormalizationDeclaration :: FilePath -> IO (Either Text NormalizationDeclaration)
loadNormalizationDeclaration path = do
loaded <- try (withBinaryFile path ReadMode (\handle -> BS.hGet handle 65537)) :: IO (Either IOException BS.ByteString)
pure $ case loaded of
Left err -> Left (T.pack (show err))
Right raw -> do
value <- decodeNormalizationJSON (BL.fromStrict raw)
case A.fromJSON value of
A.Error message -> Left (T.pack message)
A.Success declaration -> Right declaration
-- 保留报告字段并选择文本或 JSON 显示。
renderNormalization :: Bool -> A.Value -> Text
renderNormalization True = jsonText
renderNormalization False = pretty 0
-- 将 JSON 编码为 UTF-8 文本。
jsonText :: A.Value -> Text
jsonText = TE.decodeUtf8 . BL.toStrict . A.encode
-- 缩进展示完整报告且保留所有证据。
pretty :: Int -> A.Value -> Text
pretty depth (A.Object fields) = T.intercalate "\n" [indent <> K.toText key <> ":\n" <> pretty (depth + 1) value | (key, value) <- KM.toList fields]
where indent = T.replicate depth " "
pretty depth value = T.replicate depth " " <> jsonText value
-- 发送声明并让失败决定 CLI 退出码。
runNormalizationOnce :: Session -> Text -> FilePath -> Bool -> IO Bool
runNormalizationOnce session table path jsonOutput = do
declaration <- loadNormalizationDeclaration path
database <- sessionDatabase session
result <- case declaration of
Left message -> pure (Left message)
Right d -> analyzeNormalization session (NormalizationRequest 1 database table d Nothing Nothing)
case result of
Left message -> TIO.hPutStrLn stderr ("chusql: error: " <> message) >> pure False
Right analysis -> TIO.putStrLn (renderNormalization jsonOutput analysis) >> pure True
chusql-cli/src/ChuSQL/CLI/Script.hs
+4
View File
@@ -10,6 +10,7 @@ module ChuSQL.CLI.Script
, metaHelp , metaHelp
) where ) where
import ChuSQL.CLI.Normalization (normalizationArguments)
import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat) import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat)
import Data.Char (isSpace) import Data.Char (isSpace)
import Data.Text (Text) import Data.Text (Text)
@@ -23,6 +24,7 @@ data Meta
| MetaDatabases | MetaDatabases
| MetaTables | MetaTables
| MetaDescribe Text | MetaDescribe Text
| MetaNormalize Text FilePath Bool
| MetaRoles | MetaRoles
| MetaConnect Text | MetaConnect Text
| MetaSetFormat OutputFormat | MetaSetFormat OutputFormat
@@ -50,6 +52,7 @@ parseMeta raw = do
| otherwise -> MetaDescribe argument | otherwise -> MetaDescribe argument
"dr" -> MetaRoles "dr" -> MetaRoles
"roles" -> MetaRoles "roles" -> MetaRoles
"normalize" -> either (const (MetaUnknown "normalize")) (\(table, file, jsonOutput) -> MetaNormalize table file jsonOutput) (normalizationArguments argument)
"c" -> connectTo argument "c" -> connectTo argument
"connect" -> connectTo argument "connect" -> connectTo argument
"use" -> connectTo argument "use" -> connectTo argument
@@ -118,6 +121,7 @@ metaHelp =
, " \\l, \\list list databases" , " \\l, \\list list databases"
, " \\dt, \\d list tables" , " \\dt, \\d list tables"
, " \\d <table> describe a table" , " \\d <table> describe a table"
, " \\normalize TABLE --declaration FILE [--format text|json]"
, " \\dr, \\roles list roles with their grants and members" , " \\dr, \\roles list roles with their grants and members"
, " \\c <database> switch database" , " \\c <database> switch database"
, " \\f, \\format <fmt> output format: " <> T.intercalate ", " (map formatName [FormatTable, FormatJson, FormatCsv]) , " \\f, \\format <fmt> output format: " <> T.intercalate ", " (map formatName [FormatTable, FormatJson, FormatCsv])
chusql-cli/test/Spec.hs
+26
View File
@@ -2,6 +2,8 @@
module Main (main) where module Main (main) where
import ChuSQL.CLI.Normalization
import ChuSQL.Interface.Normalization
import ChuSQL.CLI.Format (OutputFormat (..), parseFormat, renderCsv, renderJson, renderTable, displayValue) import ChuSQL.CLI.Format (OutputFormat (..), parseFormat, renderCsv, renderJson, renderTable, displayValue)
import ChuSQL.CLI.History (historySettings, rememberStatement) import ChuSQL.CLI.History (historySettings, rememberStatement)
import ChuSQL.CLI.Password (readPasswordInput) import ChuSQL.CLI.Password (readPasswordInput)
@@ -31,6 +33,7 @@ main = hspec spec
-- | 全部测试用例 -- | 全部测试用例
spec :: Spec spec :: Spec
spec = do spec = do
normalizationSpec
it "renders typed compound values without losing their wire identity" $ do it "renders typed compound values without losing their wire identity" $ do
case resolveType builtinTypes (listType intType) of case resolveType builtinTypes (listType intType) of
Left err -> expectationFailure err Left err -> expectationFailure err
@@ -188,3 +191,26 @@ sampleResult =
, qrTruncated = False , qrTruncated = False
, qrDatabase = Nothing , qrDatabase = Nothing
} }
-- 范式元命令、严格文件与显示格式回归。
normalizationSpec :: Spec
normalizationSpec = describe "CLI: normalization" $ do
it "parses quoted paths without interpreting Windows backslashes" $ do
normalizationArguments "users --declaration \"C:\\My Data\\fd.json\" --format json" `shouldBe` Right ("users", "C:\\My Data\\fd.json", True)
parseMeta "\\normalize users --declaration deps.json --format text" `shouldBe` Just (MetaNormalize "users" "deps.json" False)
normalizationArguments "users --declaration deps.json --format csv" `shouldSatisfy` either (const True) (const False)
it "keeps the same report fields in text and JSON output" $ do
let report = A.object ["candidate_keys" A..= ([["id"]] :: [[T.Text]]), "conditional" A..= True, "findings" A..= [A.object ["normal_form" A..= ("bcnf" :: T.Text), "verdict" A..= ("insufficient" :: T.Text)]]]
A.eitherDecodeStrict (encodeUtf8 (renderNormalization True report)) `shouldBe` Right report
renderNormalization False report `shouldSatisfy` T.isInfixOf "candidate_keys"
renderNormalization False report `shouldSatisfy` T.isInfixOf "insufficient"
it "loads defaults and refuses duplicate JSON fields in declaration files" $ do
directory <- getTemporaryDirectory
bracket (openTempFile directory "normalization-declaration") (\(path, _) -> removeFile path) $ \(path, handle) -> do
hPutStr handle "{}"
hClose handle
loadNormalizationDeclaration path `shouldReturn` Right (NormalizationDeclaration [] [] False Nothing)
BL.writeFile path "{\"atomic_attributes\":true,\"atomic_attributes\":false}"
result <- loadNormalizationDeclaration path
result `shouldSatisfy` either (T.isInfixOf "duplicate") (const False)
chusql-core/engine/chusql-core-engine.cabal
+3
View File
@@ -28,6 +28,8 @@ library
ChuSQL.Core.Engine.Runtime.Binding ChuSQL.Core.Engine.Runtime.Binding
ChuSQL.Core.Engine.Runtime.SQL ChuSQL.Core.Engine.Runtime.SQL
ChuSQL.Core.Engine.Runtime.Query ChuSQL.Core.Engine.Runtime.Query
ChuSQL.Core.Engine.Runtime.Relation
ChuSQL.Core.Engine.Runtime.Dependencies
ChuSQL.Core.Engine.Semantic ChuSQL.Core.Engine.Semantic
ChuSQL.Core.Engine.Storage ChuSQL.Core.Engine.Storage
ChuSQL.Core.Engine.Storage.FFI ChuSQL.Core.Engine.Storage.FFI
@@ -53,6 +55,7 @@ library
if os(osx) if os(osx)
ld-options: -Wl,-rpath,@loader_path ld-options: -Wl,-rpath,@loader_path
default-language: Haskell2010 default-language: Haskell2010
default-extensions: ImportQualifiedPost
ghc-options: -Wall ghc-options: -Wall
test-suite chusql-core-engine-test test-suite chusql-core-engine-test
chusql-core/engine/src/ChuSQL/Core/Engine.hs
+39 -25
View File
@@ -3,14 +3,14 @@ module ChuSQL.Core.Engine (runStatement, runStatementM, rowsOf) where
import ChuSQL.Core.Engine.Algebra.Eval (evalCondForRowM, evalExprM) import ChuSQL.Core.Engine.Algebra.Eval (evalCondForRowM, evalExprM)
import ChuSQL.Core.Engine.Algebra.Expr (evalCondForRow, evalExpr) import ChuSQL.Core.Engine.Algebra.Expr (evalCondForRow, evalExpr)
import ChuSQL.Core.Engine.Runtime.Query (bindQuery, executeQueryM) import ChuSQL.Core.Engine.Runtime.Query (bindQuery, executeQueryM)
import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Semantic (prepare, prepareRowExpression) import ChuSQL.Core.Engine.Semantic (prepare, prepareRowExpression)
import ChuSQL.Core.Engine.Storage (MonadStorage (..)) import ChuSQL.Core.Engine.Storage (MonadStorage (..))
import ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage)) import ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage))
import ChuSQL.Core.Engine.Syntax.AST import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression) import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
import ChuSQL.Core.Model
import Data.HashSet qualified as HS
import Data.Maybe (mapMaybe) import Data.Maybe (mapMaybe)
import qualified Data.HashSet as HS
-- 引擎入口：检查语义、按语句分发执行，另有内存实现与泛型版本。 -- 引擎入口：检查语义、按语句分发执行，另有内存实现与泛型版本。
@@ -39,6 +39,10 @@ runStatementM q = do
-- | 按语句类型分发（已检查过） -- | 按语句类型分发（已检查过）
runStatementUncheckedM :: (MonadStorage m) => Database -> Statement -> m (Either String [Row]) runStatementUncheckedM :: (MonadStorage m) => Database -> Statement -> m (Either String [Row])
runStatementUncheckedM db q@SelectDistinct{} =
case bindQuery db q of
Left e -> pure (Left e)
Right bound -> executeQueryM bound
runStatementUncheckedM db q@Select{} = runStatementUncheckedM db q@Select{} =
case bindQuery db q of case bindQuery db q of
Left e -> pure (Left e) Left e -> pure (Left e)
@@ -74,7 +78,7 @@ runStatementUncheckedM db (Delete tbl mWhere) = do
result <- replaceAll tbl kept result <- replaceAll tbl kept
pure (result >> Right []) pure (result >> Right [])
where where
-- | 取行里的整数 id，取不到就报错 -- \| 取行里的整数 id，取不到就报错
rowId :: Row -> Either String Int rowId :: Row -> Either String Int
rowId r = case lookup "id" r of rowId r = case lookup "id" r of
Just (VInt k) -> Right k Just (VInt k) -> Right k
@@ -104,9 +108,15 @@ runStatementUncheckedM _ (CreateDomain name base) = do
runStatementUncheckedM _ (DropDomain name) = do runStatementUncheckedM _ (DropDomain name) = do
result <- dropTable (domainTableName name) result <- dropTable (domainTableName name)
pure (result >> Right []) pure (result >> Right [])
runStatementUncheckedM db ShowDomains = pure (Right runStatementUncheckedM db ShowDomains =
[[("domain", VStr (drop (length (domainTableName "")) name)), ("base_type", VStr (typeName (columnType base)))] pure
| (name, table) <- db, isDomainTable name, Just base <- [lookup "base" (tableCols table)]]) ( Right
[ [("domain", VStr (drop (length (domainTableName "")) name)), ("base_type", VStr (typeName (columnType base)))]
| (name, table) <- db
, isDomainTable name
, Just base <- [lookup "base" (tableCols table)]
]
)
runStatementUncheckedM _ (DropTable name) = do runStatementUncheckedM _ (DropTable name) = do
result <- dropTable name result <- dropTable name
pure (result >> Right []) pure (result >> Right [])
@@ -132,11 +142,13 @@ runStatementUncheckedM db (AlterColumnNull tbl col nullable) =
runStatementUncheckedM _ CreateUser{} = pure (Left "CREATE USER is executed by the account service") runStatementUncheckedM _ CreateUser{} = pure (Left "CREATE USER is executed by the account service")
runStatementUncheckedM _ AlterUser{} = pure (Left "ALTER USER is executed by the account service") runStatementUncheckedM _ AlterUser{} = pure (Left "ALTER USER is executed by the account service")
runStatementUncheckedM _ AlterIdentity{} = pure (Left "ALTER ROLE is executed by the account service") runStatementUncheckedM _ AlterIdentity{} = pure (Left "ALTER ROLE is executed by the account service")
runStatementUncheckedM _ ShowGrants = pure (Left "SHOW GRANTS is executed by the account service")
runStatementUncheckedM _ ShowRoles = pure (Left "SHOW ROLES is executed by the account service") runStatementUncheckedM _ ShowRoles = pure (Left "SHOW ROLES is executed by the account service")
runStatementUncheckedM _ DropUser{} = pure (Left "DROP USER is executed by the account service") runStatementUncheckedM _ DropUser{} = pure (Left "DROP USER is executed by the account service")
runStatementUncheckedM _ CreateRole{} = pure (Left "CREATE ROLE is executed by the privilege service") runStatementUncheckedM _ CreateRole{} = pure (Left "CREATE ROLE is executed by the privilege service")
runStatementUncheckedM _ DropRole{} = pure (Left "DROP ROLE is executed by the privilege service") runStatementUncheckedM _ DropRole{} = pure (Left "DROP ROLE is executed by the privilege service")
runStatementUncheckedM _ GrantPrivileges{} = pure (Left "GRANT is executed by the privilege service") runStatementUncheckedM _ GrantPrivileges{} = pure (Left "GRANT is executed by the privilege service")
runStatementUncheckedM _ RevokeGrantOption{} = pure (Left "REVOKE is executed by the privilege service")
runStatementUncheckedM _ RevokePrivileges{} = pure (Left "REVOKE is executed by the privilege service") runStatementUncheckedM _ RevokePrivileges{} = pure (Left "REVOKE is executed by the privilege service")
runStatementUncheckedM _ GrantRole{} = pure (Left "GRANT is executed by the privilege service") runStatementUncheckedM _ GrantRole{} = pure (Left "GRANT is executed by the privilege service")
runStatementUncheckedM _ RevokeRole{} = pure (Left "REVOKE is executed by the privilege service") runStatementUncheckedM _ RevokeRole{} = pure (Left "REVOKE is executed by the privilege service")
@@ -157,9 +169,9 @@ existingRows tbl table
| needsRows table = scan tbl | needsRows table = scan tbl
| otherwise = pure (Right []) | otherwise = pure (Right [])
where where
-- | 这一表有没有要判约束的列 -- \| 这一表有没有要判约束的列
needsRows t = any (constraintColumn . snd) (tableCols t) needsRows t = any (constraintColumn . snd) (tableCols t)
-- | 这一列带不带要判的约束 -- \| 这一列带不带要判的约束
constraintColumn c = columnAutoIncrement c || columnUnique c || columnPrimaryKey c constraintColumn c = columnAutoIncrement c || columnUnique c || columnPrimaryKey c
-- | 插一行：补齐缺列、给自增值、收类型、查约束 -- | 插一行：补齐缺列、给自增值、收类型、查约束
@@ -172,7 +184,7 @@ buildRows table current cols rows = do
let auto = autoColumn table let auto = autoColumn table
go auto (autoStart auto current) values [] go auto (autoStart auto current) values []
where where
-- | 逐行构造，最后统一查一遍约束 -- \| 逐行构造，最后统一查一遍约束
go _ _ [] acc = do go _ _ [] acc = do
let built = reverse acc let built = reverse acc
plan <- checkPlan table plan <- checkPlan table
@@ -189,25 +201,27 @@ buildRow :: Table -> Maybe (String, Column) -> [String] -> Int -> [Value] -> Eit
buildRow table auto cols counter vals = go (tableCols table) counter [] buildRow table auto cols counter vals = go (tableCols table) counter []
where where
provided = zip cols vals provided = zip cols vals
-- | 按列定义逐列补值 -- \| 按列定义逐列补值
go [] c acc = Right (reverse acc, c) go [] c acc = Right (reverse acc, c)
go ((name, col) : rest) c acc = do go ((name, col) : rest) c acc = do
(v, c') <- pick name col c (v, c') <- pick name col c
v' <- coerceValue (columnType col) v v' <- coerceValue (columnType col) v
go rest c' ((name, v') : acc) go rest c' ((name, v') : acc)
-- | 决定这一列取什么值 -- \| 决定这一列取什么值
pick name col c pick name col c
| Just (autoName, _) <- auto | Just (autoName, _) <- auto
, name == autoName , name == autoName
, maybe True (== VNull) (lookup name provided) = Right (VInt c, c + 1) , maybe True (== VNull) (lookup name provided) =
Right (VInt c, c + 1)
| Just v <- lookup name provided = Right (v, bump name col c v) | Just v <- lookup name provided = Right (v, bump name col c v)
| Just d <- columnDefault col = Right (d, c) | Just d <- columnDefault col = Right (d, c)
| columnNullable col = Right (VNull, c) | columnNullable col = Right (VNull, c)
| otherwise = Left ("INSERT: column " ++ name ++ " is NOT NULL and has no default") | otherwise = Left ("INSERT: column " ++ name ++ " is NOT NULL and has no default")
-- | 自增值跟着已有的值往前推 -- \| 自增值跟着已有的值往前推
bump name _ c v bump name _ c v
| Just (autoName, _) <- auto | Just (autoName, _) <- auto
, name == autoName = max (c + 1) (valueInt v + 1) , name == autoName =
max (c + 1) (valueInt v + 1)
| otherwise = c | otherwise = c
-- | 取自增值，不是整数就当 0 -- | 取自增值，不是整数就当 0
@@ -235,9 +249,9 @@ type CheckPlan = [(String, Expr)]
checkPlan :: Table -> Either String CheckPlan checkPlan :: Table -> Either String CheckPlan
checkPlan table = mapM parse (mapMaybe checked (tableCols table)) checkPlan table = mapM parse (mapMaybe checked (tableCols table))
where where
-- | 只挑出带 CHECK 文本的列 -- \| 只挑出带 CHECK 文本的列
checked (name, col) = (,) name <$> columnCheck col checked (name, col) = (,) name <$> columnCheck col
-- | 解析一列的 CHECK 文本 -- \| 解析一列的 CHECK 文本
parse (name, text) = case parseExpression text of parse (name, text) = case parseExpression text of
Left err -> Left ("CHECK on " ++ name ++ ": " ++ firstLine err) Left err -> Left ("CHECK on " ++ name ++ ": " ++ firstLine err)
Right e -> do Right e -> do
@@ -248,7 +262,7 @@ checkPlan table = mapM parse (mapMaybe checked (tableCols table))
runChecks :: CheckPlan -> Row -> Either String () runChecks :: CheckPlan -> Row -> Either String ()
runChecks plan row = mapM_ check plan runChecks plan row = mapM_ check plan
where where
-- | 跑一条 CHECK -- \| 跑一条 CHECK
check (name, e) = case evalCondForRow e row of check (name, e) = case evalCondForRow e row of
Left err -> Left ("CHECK on " ++ name ++ ": " ++ err) Left err -> Left ("CHECK on " ++ name ++ ": " ++ err)
Right True -> Right () Right True -> Right ()
@@ -262,7 +276,7 @@ firstLine = takeWhile (/= '\n')
enforceNotNull :: [(String, Column)] -> [Row] -> Either String () enforceNotNull :: [(String, Column)] -> [Row] -> Either String ()
enforceNotNull cols rows = mapM_ check cols enforceNotNull cols rows = mapM_ check cols
where where
-- | 查这一列的 NOT NULL -- \| 查这一列的 NOT NULL
check (name, col) check (name, col)
| columnNullable col = Right () | columnNullable col = Right ()
| any (\r -> lookup name r == Just VNull) rows = Left ("NOT NULL: column " ++ name ++ " cannot be null") | any (\r -> lookup name r == Just VNull) rows = Left ("NOT NULL: column " ++ name ++ " cannot be null")
@@ -273,11 +287,11 @@ enforceUnique :: [(String, Column)] -> [Row] -> Either String ()
enforceUnique cols rows = mapM_ check targets enforceUnique cols rows = mapM_ check targets
where where
targets = [(n, c) | (n, c) <- cols, columnUnique c || columnPrimaryKey c] targets = [(n, c) | (n, c) <- cols, columnUnique c || columnPrimaryKey c]
-- | 查这一列有没有重复值 -- \| 查这一列有没有重复值
check (name, _) = go name HS.empty (mapMaybe (lookup name) rows) check (name, _) = go name HS.empty (mapMaybe (lookup name) rows)
-- | 逐个值比过去 -- \| 逐个值比过去
go _ _ [] = Right () go _ _ [] = Right ()
-- | NULL 不算冲突，跳过 -- \| NULL 不算冲突，跳过
go name seen (VNull : rest) = go name seen rest go name seen (VNull : rest) = go name seen rest
go name seen (v : rest) go name seen (v : rest)
| HS.member (canonicalValue v) seen = Left ("UNIQUE: duplicate value in column " ++ name) | HS.member (canonicalValue v) seen = Left ("UNIQUE: duplicate value in column " ++ name)
@@ -317,7 +331,7 @@ setColumn col v = map (\(k, val) -> if k == col then (k, v) else (k, val))
splitByConditionM :: (MonadStorage m) => Database -> Maybe Expr -> [Row] -> m (Either String ([Row], [Row])) splitByConditionM :: (MonadStorage m) => Database -> Maybe Expr -> [Row] -> m (Either String ([Row], [Row]))
splitByConditionM db cond = go [] [] splitByConditionM db cond = go [] []
where where
-- | 逐行分流 -- \| 逐行分流
go doomed kept [] = pure (Right (reverse doomed, reverse kept)) go doomed kept [] = pure (Right (reverse doomed, reverse kept))
go doomed kept (r : rs) = do go doomed kept (r : rs) = do
hit <- case cond of hit <- case cond of
@@ -346,9 +360,9 @@ alterTable db tbl change = case lookupTable db tbl of
result <- replaceSchema tbl cols newRows result <- replaceSchema tbl cols newRows
pure (result >> Right []) pure (result >> Right [])
where where
-- | 改完再统一查一遍约束 -- \| 改完再统一查一遍约束
checkAltered (cols, rows) = do checkAltered (cols, rows) = do
let table = Table tbl cols rows Nothing let table = Table tbl cols rows Nothing Nothing Nothing
plan <- checkPlan table plan <- checkPlan table
mapM_ (runChecks plan) rows mapM_ (runChecks plan) rows
enforceNotNull cols rows enforceNotNull cols rows
@@ -387,7 +401,7 @@ retypeColumn col ty cols rows
rows' <- mapM convert rows rows' <- mapM convert rows
Right (map (\entry@(n, c) -> if n == col then (n, c{columnType = ty}) else entry) cols, rows') Right (map (\entry@(n, c) -> if n == col then (n, c{columnType = ty}) else entry) cols, rows')
where where
-- | 把这一列的老值收进新类型 -- \| 把这一列的老值收进新类型
convert row = case lookup col row of convert row = case lookup col row of
Nothing -> Right row Nothing -> Right row
Just v -> do Just v -> do
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Eval.hs
+65 -23
View File
@@ -5,7 +5,8 @@ import ChuSQL.Core.Engine.Algebra.Expr (evalCondForRow, evalExpr, hasSubquery, q
import ChuSQL.Core.Engine.Algebra.Optimize (optimize) import ChuSQL.Core.Engine.Algebra.Optimize (optimize)
import ChuSQL.Core.Engine.Algebra.Op import ChuSQL.Core.Engine.Algebra.Op
import ChuSQL.Core.Engine.Algebra.Planner (translate) import ChuSQL.Core.Engine.Algebra.Planner (translate)
import ChuSQL.Core.Engine.Algebra.Sort (sortRows) import ChuSQL.Core.Engine.Algebra.Sort (sortRows, sortBoundRows)
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
import ChuSQL.Core.Engine.Builtin (aggregateArg, builtinOfExpr, invokeBuiltin, unsupportedAggregate) import ChuSQL.Core.Engine.Builtin (aggregateArg, builtinOfExpr, invokeBuiltin, unsupportedAggregate)
import ChuSQL.Core.Model import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Storage (IndexResult (..), MonadStorage (..)) import ChuSQL.Core.Engine.Storage (IndexResult (..), MonadStorage (..))
@@ -14,6 +15,7 @@ import ChuSQL.Core.Engine.Syntax.AST (Expr (..), JoinKind (..), Subquery (..))
import Control.Monad (filterM) import Control.Monad (filterM)
import Data.List (nub) import Data.List (nub)
import qualified Data.HashMap.Strict as HM import qualified Data.HashMap.Strict as HM
import qualified Data.HashSet as HS
-- 执行：按算子树算出结果行，外层行供相关子查询引用。 -- 执行：按算子树算出结果行，外层行供相关子查询引用。
@@ -79,11 +81,30 @@ data EquiKeys = EquiKeys
-- | 认出 a = b 这种两列等值条件，并且能分清谁在左谁在右 -- | 认出 a = b 这种两列等值条件，并且能分清谁在左谁在右
equiKeys :: [String] -> [String] -> Expr -> Maybe EquiKeys equiKeys :: [String] -> [String] -> Expr -> Maybe EquiKeys
equiKeys lcols rcols (Eq (Col a) (Col b)) equiKeys lcols rcols (Eq left right)
| a /= b, a `elem` lcols, b `elem` rcols = Just (EquiKeys a b) | Just a <- columnName left, Just b <- columnName right
| a /= b, b `elem` lcols, a `elem` rcols = Just (EquiKeys b a) , a /= b, a `elem` lcols, b `elem` rcols = Just (EquiKeys a b)
| Just a <- columnName left, Just b <- columnName right
, a /= b, b `elem` lcols, a `elem` rcols = Just (EquiKeys b a)
equiKeys _ _ _ = Nothing equiKeys _ _ _ = Nothing
-- | 提取执行列的诊断名称
columnName :: Expr -> Maybe String
columnName (Col name) = Just name
columnName (ColumnSlot _ name) = Just name
columnName _ = Nothing
-- | 在存储边界按输入布局排列行
orderedRows :: [String] -> [Row] -> Either String [Row]
orderedRows columns = mapM order
where
-- | 按列布局排列一行
order row = mapM (column row) columns
-- | 缺少物理列时明确报错
column row name = case lookup name row of
Just value -> Right (name, value)
Nothing -> Left ("missing bound column: " ++ name)
-- | 连接两批行：等值键走哈希，否则嵌套循环 -- | 连接两批行：等值键走哈希，否则嵌套循环
joinRowsIn :: joinRowsIn ::
(MonadStorage m) => (MonadStorage m) =>
@@ -188,8 +209,9 @@ probeIndexJoin db outer kind mAlias t ks rcols cond l = case lookup (ekLeft ks)
Left e -> pure (Left e) Left e -> pure (Left e)
Right NoIndex -> pure (Right Nothing) Right NoIndex -> pure (Right Nothing)
Right (IndexRows rows) -> do Right (IndexRows rows) -> do
pairs <- joinRowsIn db outer kind rcols cond [l] (prefixRows mAlias rows) case orderedRows rcols (prefixRows mAlias rows) of
pure (fmap Just pairs) Left err -> pure (Left err)
Right ordered -> fmap (fmap Just) (joinRowsIn db outer kind rcols cond [l] ordered)
-- | 列与键都不是浮点时才敢用索引 -- | 列与键都不是浮点时才敢用索引
indexSafe :: Database -> String -> String -> Value -> Bool indexSafe :: Database -> String -> String -> Value -> Bool
@@ -213,20 +235,17 @@ floatValue (VFloat _) = True
floatValue _ = False floatValue _ = False
-- | 分组键的值 -- | 分组键的值
groupKey :: [String] -> Row -> Either String [Value] groupKey :: Row -> [Expr] -> Row -> Either String [Value]
groupKey keys row = mapM get keys groupKey outer keys row = mapM (\key -> evalExpr key (row ++ outer)) keys
where
-- | 取这一列的分组值
get c = maybe (Left ("unknown column: " ++ c)) Right (lookup c row)
-- | 按分组键把行分组，保留分组首次出现的顺序 -- | 按分组键把行分组，保留分组首次出现的顺序
groupRows :: [String] -> [Row] -> Either String [([Value], [Row])] groupRows :: Row -> [Expr] -> [Row] -> Either String [([Value], [Row])]
groupRows keys rows = go rows (HM.empty, []) groupRows outer keys rows = go rows (HM.empty, [])
where where
-- | 逐行塞进对应分组，并记住分组首次出现的顺序 -- | 逐行塞进对应分组，并记住分组首次出现的顺序
go [] (acc, order) = Right [(k, reverse (HM.lookupDefault [] k acc)) | k <- reverse order] go [] (acc, order) = Right [(k, reverse (HM.lookupDefault [] k acc)) | k <- reverse order]
go (r : rs) (acc, order) = do go (r : rs) (acc, order) = do
k <- groupKey keys r k <- groupKey outer keys r
let order' = if HM.member k acc then order else k : order let order' = if HM.member k acc then order else k : order
go rs (HM.insertWith (++) k [r] acc, order') go rs (HM.insertWith (++) k [r] acc, order')
@@ -249,11 +268,11 @@ oneGroup outer keys aggs (keyValues, rows) = do
Right (zip keys keyValues ++ zip (map fst aggs) values) Right (zip keys keyValues ++ zip (map fst aggs) values)
-- | 分组求值：没有分组列时，即使没有行也要出一行 -- | 分组求值：没有分组列时，即使没有行也要出一行
evalAggregate :: Row -> [String] -> [(String, Expr)] -> [Row] -> Either String [Row] evalAggregate :: Row -> [(String, Expr)] -> [(String, Expr)] -> [Row] -> Either String [Row]
evalAggregate outer keys aggs rows = do evalAggregate outer keys aggs rows = do
groups <- groupRows keys rows groups <- groupRows outer (map snd keys) rows
let top = if null keys && null groups then [([], [])] else groups let top = if null keys && null groups then [([], [])] else groups
mapM (oneGroup outer keys aggs) top mapM (oneGroup outer (map fst keys) aggs) top
-- | 绑定键：子查询引用的外层列在这一行上的取值 -- | 绑定键：子查询引用的外层列在这一行上的取值
bindingKey :: [String] -> Row -> String bindingKey :: [String] -> Row -> String
@@ -443,7 +462,9 @@ runSubqueryRows db sq outer = case translate (subqueryStatement sq) of
Left e -> pure (Left e) Left e -> pure (Left e)
Right op -> case mapM bind (subqueryRefs sq) of Right op -> case mapM bind (subqueryRefs sq) of
Left err -> pure (Left err) Left err -> pure (Left err)
Right bindings -> evalRelOpIn db (bindings ++ outer) (optimize db op) Right bindings -> case bindRelOpColumns db (map fst (bindings ++ outer)) (optimize db op) of
Left err -> pure (Left err)
Right bound -> evalRelOpIn db (bindings ++ outer) bound
where where
-- | 给外层物理列补回已解析的限定名 -- | 给外层物理列补回已解析的限定名
bind column = case bindingValue column outer of bind column = case bindingValue column outer of
@@ -481,23 +502,26 @@ evalRelOpIn db outer (Compute items op) = do
es <- boundExprs refs (r ++ outer) table es <- boundExprs refs (r ++ outer) table
mapM (\(label, e) -> (,) label <$> evalExpr e (r ++ outer)) (zip (map fst items) es) mapM (\(label, e) -> (,) label <$> evalExpr e (r ++ outer)) (zip (map fst items) es)
evalRelOpIn db outer (Aggregate keys aggs op) = do evalRelOpIn db outer (Aggregate keys aggs op) = do
rows <- evalRelOpIn db outer op
pure (rows >>= evalAggregate outer [(name, Col name) | name <- keys] aggs)
evalRelOpIn db outer (BoundAggregate keys aggs op) = do
rows <- evalRelOpIn db outer op rows <- evalRelOpIn db outer op
pure (rows >>= evalAggregate outer keys aggs) pure (rows >>= evalAggregate outer keys aggs)
evalRelOpIn db _ (Scan mAlias tbl cols) = do evalRelOpIn db _ (Scan mAlias tbl cols) = do
res <- scanTable db tbl cols res <- scanTable db tbl cols
pure (fmap (prefixRows mAlias) res) pure (res >>= orderedRows (relOpCols db (Scan mAlias tbl cols)) . prefixRows mAlias)
evalRelOpIn db outer (Lookup mAlias tbl col k) = do evalRelOpIn db outer (Lookup mAlias tbl col k) = do
res <- lookupByColumn tbl col k res <- lookupByColumn tbl col k
case res of case res of
Left e -> pure (Left e) Left e -> pure (Left e)
Right NoIndex -> scanFiltered db outer mAlias tbl (lookupCondition mAlias col k) Right NoIndex -> scanFiltered db outer mAlias tbl (lookupCondition mAlias col k)
Right (IndexRows rows) -> pure (Right (prefixRows mAlias rows)) Right (IndexRows rows) -> pure (orderedRows (relOpCols db (Lookup mAlias tbl col k)) (prefixRows mAlias rows))
evalRelOpIn db outer (Range mAlias tbl col lo hi) = do evalRelOpIn db outer (Range mAlias tbl col lo hi) = do
res <- scanRange tbl col lo hi res <- scanRange tbl col lo hi
case res of case res of
Left e -> pure (Left e) Left e -> pure (Left e)
Right Nothing -> scanFiltered db outer mAlias tbl (rangeCondition mAlias col lo hi) Right Nothing -> scanFiltered db outer mAlias tbl (rangeCondition mAlias col lo hi)
Right (Just rows) -> pure (Right (prefixRows mAlias rows)) Right (Just rows) -> pure (orderedRows (relOpCols db (Range mAlias tbl col lo hi)) (prefixRows mAlias rows))
evalRelOpIn db outer (Filter e op) = do evalRelOpIn db outer (Filter e op) = do
rows <- evalRelOpIn db outer op rows <- evalRelOpIn db outer op
case rows of case rows of
@@ -516,9 +540,15 @@ evalRelOpIn db outer (Filter e op) = do
evalRelOpIn db outer (Project cols op) = do evalRelOpIn db outer (Project cols op) = do
rows <- evalRelOpIn db outer op rows <- evalRelOpIn db outer op
pure (fmap (map (project cols)) rows) pure (fmap (map (project cols)) rows)
evalRelOpIn db outer (Distinct op) = do
rows <- evalRelOpIn db outer op
pure (distinctRows <$> rows)
evalRelOpIn db outer (Sort spec op) = do evalRelOpIn db outer (Sort spec op) = do
rows <- evalRelOpIn db outer op rows <- evalRelOpIn db outer op
pure (fmap (sortRows spec) rows) pure (fmap (sortRows spec) rows)
evalRelOpIn db outer (BoundSort spec op) = do
rows <- evalRelOpIn db outer op
pure (rows >>= sortBoundRows spec outer)
evalRelOpIn db outer (Limit n op) = do evalRelOpIn db outer (Limit n op) = do
rows <- evalRelOpIn db outer op rows <- evalRelOpIn db outer op
pure (fmap (take n) rows) pure (fmap (take n) rows)
@@ -557,8 +587,8 @@ scanFiltered ::
scanFiltered db outer mAlias tbl cond = do scanFiltered db outer mAlias tbl cond = do
res <- scanTable db tbl Nothing res <- scanTable db tbl Nothing
pure $ do pure $ do
rows <- res rows <- res >>= orderedRows (relOpCols db (Scan mAlias tbl Nothing)) . prefixRows mAlias
filterM (evalCondForRow cond . (++ outer)) (prefixRows mAlias rows) filterM (evalCondForRow cond . (++ outer)) rows
-- | 行数到阈值才值得并行分片 -- | 行数到阈值才值得并行分片
parallelRowThreshold :: Int parallelRowThreshold :: Int
@@ -574,3 +604,15 @@ scanTable db tbl cols = do
where where
-- | 统计里的行数，没有统计算 0 -- | 统计里的行数，没有统计算 0
rows = maybe 0 metaRowCount (tableMetaOf db tbl) rows = maybe 0 metaRowCount (tableMetaOf db tbl)
-- | 按输出值去重并保留首次行序
distinctRows :: [Row] -> [Row]
distinctRows = go HS.empty
where
-- | 只输出尚未遇到的值组合
go _ [] = []
go seen (row : rows)
| HS.member key seen = go seen rows
| otherwise = row : go (HS.insert key seen) rows
where
key = map (canonicalValue . snd) row
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Expr.hs
+11
View File
@@ -11,6 +11,13 @@ import ChuSQL.Core.Engine.Runtime.SQL (invokeSQLFunction, constructSQLValue)
evalExpr :: Expr -> Row -> Either String Value evalExpr :: Expr -> Row -> Either String Value
evalExpr (Col name) row = evalExpr (Col name) row =
maybe (Left ("unknown column: " ++ name)) Right (lookup name row) maybe (Left ("unknown column: " ++ name)) Right (lookup name row)
evalExpr (ColumnSlot slot name) row = case drop slot row of
(key, value) : _ | slot >= 0, key == name -> Right value
[] -> Left ("missing bound column: " ++ name)
_ -> Left ("bound column position mismatch: " ++ name)
evalExpr (OuterColumnSlot localWidth slot name) row
| localWidth < 0 || slot < 0 = Left ("bound column position mismatch: " ++ name)
| otherwise = evalExpr (ColumnSlot slot name) (drop localWidth row)
evalExpr (FunctionCall name _) _ = Left ("unbound function call: " ++ name) evalExpr (FunctionCall name _) _ = Left ("unbound function call: " ++ name)
evalExpr (Construct _ _) _ = Left "unbound runtime constructor" evalExpr (Construct _ _) _ = Left "unbound runtime constructor"
evalExpr (BoundConstruct tid arguments) row = mapM (\argument -> evalExpr argument row) arguments >>= constructSQLValue tid evalExpr (BoundConstruct tid arguments) row = mapM (\argument -> evalExpr argument row) arguments >>= constructSQLValue tid
@@ -70,6 +77,8 @@ quantifiedCompare op lhs vs q = case vs of
-- | 表达式用到了哪些列 -- | 表达式用到了哪些列
colsInExpr :: Expr -> [String] colsInExpr :: Expr -> [String]
colsInExpr (Col c) = [c] colsInExpr (Col c) = [c]
colsInExpr (ColumnSlot _ c) = [c]
colsInExpr (OuterColumnSlot _ _ c) = [c]
colsInExpr (Construct _ arguments) = concatMap colsInExpr arguments colsInExpr (Construct _ arguments) = concatMap colsInExpr arguments
colsInExpr (BoundConstruct _ arguments) = concatMap colsInExpr arguments colsInExpr (BoundConstruct _ arguments) = concatMap colsInExpr arguments
colsInExpr (FunctionCall _ arguments) = concatMap colsInExpr arguments colsInExpr (FunctionCall _ arguments) = concatMap colsInExpr arguments
@@ -247,6 +256,8 @@ bareColumns e = case e of
FunctionCall _ arguments -> concatMap bareColumns arguments FunctionCall _ arguments -> concatMap bareColumns arguments
BoundFunction _ _ arguments -> concatMap bareColumns arguments BoundFunction _ _ arguments -> concatMap bareColumns arguments
Col c -> [c] Col c -> [c]
ColumnSlot _ c -> [c]
OuterColumnSlot _ _ c -> [c]
Add a b -> both a b Add a b -> both a b
Sub a b -> both a b Sub a b -> both a b
Mul a b -> both a b Mul a b -> both a b
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/JoinOrder.hs
+1
View File
@@ -15,6 +15,7 @@ reorderJoins db = pass db False
-- | 往下带“行序是否不可观察”，可观察时不动内连接 -- | 往下带“行序是否不可观察”，可观察时不动内连接
pass :: Database -> Bool -> RelOp -> RelOp pass :: Database -> Bool -> RelOp -> RelOp
pass db free (Distinct x) = Distinct (pass db free x)
pass db _ (Sort spec x) = Sort spec (pass db True x) pass db _ (Sort spec x) = Sort spec (pass db True x)
pass db _ (Limit n x) = Limit n (pass db False x) pass db _ (Limit n x) = Limit n (pass db False x)
pass db _ (Aggregate keys aggs x) = Aggregate keys aggs (pass db False x) pass db _ (Aggregate keys aggs x) = Aggregate keys aggs (pass db False x)
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Op.hs
+10
View File
@@ -13,8 +13,11 @@ data RelOp
| Project [String] RelOp | Project [String] RelOp
| Compute [(String, Expr)] RelOp | Compute [(String, Expr)] RelOp
| Aggregate [String] [(String, Expr)] RelOp | Aggregate [String] [(String, Expr)] RelOp
| BoundAggregate [(String, Expr)] [(String, Expr)] RelOp
| Distinct RelOp
| Unit | Unit
| Sort [(String, SortDir)] RelOp | Sort [(String, SortDir)] RelOp
| BoundSort [(Expr, SortDir)] RelOp
| Limit Int RelOp | Limit Int RelOp
| Join JoinKind RelOp RelOp Expr | Join JoinKind RelOp RelOp Expr
| Derived (Maybe String) RelOp | Derived (Maybe String) RelOp
@@ -25,6 +28,7 @@ relOpCols :: Database -> RelOp -> [String]
relOpCols _ Unit = [] relOpCols _ Unit = []
relOpCols _ (Compute items _) = map fst items relOpCols _ (Compute items _) = map fst items
relOpCols _ (Aggregate keys aggs _) = keys ++ map fst aggs relOpCols _ (Aggregate keys aggs _) = keys ++ map fst aggs
relOpCols _ (BoundAggregate keys aggs _) = map fst keys ++ map fst aggs
relOpCols _ (Scan mAlias _ (Just cols)) = map (qualify mAlias) cols relOpCols _ (Scan mAlias _ (Just cols)) = map (qualify mAlias) cols
relOpCols db (Scan mAlias tbl Nothing) = relOpCols db (Scan mAlias tbl Nothing) =
case lookup tbl db of case lookup tbl db of
@@ -37,7 +41,9 @@ relOpCols db (Range mAlias tbl _ _ _) = relOpCols db (Scan mAlias tbl Nothing)
relOpCols db (Filter _ x) = relOpCols db x relOpCols db (Filter _ x) = relOpCols db x
relOpCols _ (Project ["*"] _) = ["*"] relOpCols _ (Project ["*"] _) = ["*"]
relOpCols _ (Project cols _) = cols relOpCols _ (Project cols _) = cols
relOpCols db (Distinct x) = relOpCols db x
relOpCols db (Sort _ x) = relOpCols db x relOpCols db (Sort _ x) = relOpCols db x
relOpCols db (BoundSort _ x) = relOpCols db x
relOpCols db (Limit _ x) = relOpCols db x relOpCols db (Limit _ x) = relOpCols db x
relOpCols db (Join _ l r _) = relOpCols db l ++ relOpCols db r relOpCols db (Join _ l r _) = relOpCols db l ++ relOpCols db r
relOpCols db (Derived mAlias x) = map (qualify mAlias) (relOpCols db x) relOpCols db (Derived mAlias x) = map (qualify mAlias) (relOpCols db x)
@@ -67,8 +73,12 @@ renderPlan = go 0
++ show [label ++ " <- " ++ show e | (label, e) <- aggs] ++ show [label ++ " <- " ++ show e | (label, e) <- aggs]
++ "\n" ++ "\n"
++ go (ind + 1) x ++ go (ind + 1) x
Distinct x -> pad ++ "Distinct\n" ++ go (ind + 1) x
BoundAggregate keys aggs x -> pad ++ "BoundAggregate keys=" ++ show keys
++ " aggs=" ++ show aggs ++ "\n" ++ go (ind + 1) x
Unit -> pad ++ "Unit" Unit -> pad ++ "Unit"
Sort s x -> pad ++ "Sort " ++ show s ++ "\n" ++ go (ind + 1) x Sort s x -> pad ++ "Sort " ++ show s ++ "\n" ++ go (ind + 1) x
BoundSort s x -> pad ++ "BoundSort " ++ show s ++ "\n" ++ go (ind + 1) x
Limit n x -> pad ++ "Limit " ++ show n ++ "\n" ++ go (ind + 1) x Limit n x -> pad ++ "Limit " ++ show n ++ "\n" ++ go (ind + 1) x
Join k l r c -> pad ++ kindLabel k ++ " on " ++ show c ++ "\n" ++ go (ind + 1) l ++ "\n" ++ go (ind + 1) r Join k l r c -> pad ++ kindLabel k ++ " on " ++ show c ++ "\n" ++ go (ind + 1) l ++ "\n" ++ go (ind + 1) r
Derived a x -> pad ++ "Derived " ++ show a ++ "\n" ++ go (ind + 1) x Derived a x -> pad ++ "Derived " ++ show a ++ "\n" ++ go (ind + 1) x
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Optimize.hs
+5
View File
@@ -77,6 +77,7 @@ planHasDivision (Filter p x) = hasDivision p || planHasDivision x
planHasDivision (Join _ l r c) = hasDivision c || planHasDivision l || planHasDivision r planHasDivision (Join _ l r c) = hasDivision c || planHasDivision l || planHasDivision r
planHasDivision (Compute items x) = any (hasDivision . snd) items || planHasDivision x planHasDivision (Compute items x) = any (hasDivision . snd) items || planHasDivision x
planHasDivision (Project _ x) = planHasDivision x planHasDivision (Project _ x) = planHasDivision x
planHasDivision (Distinct x) = planHasDivision x
planHasDivision (Sort _ x) = planHasDivision x planHasDivision (Sort _ x) = planHasDivision x
planHasDivision (Limit _ x) = planHasDivision x planHasDivision (Limit _ x) = planHasDivision x
planHasDivision (Derived _ x) = planHasDivision x planHasDivision (Derived _ x) = planHasDivision x
@@ -96,6 +97,8 @@ pushProjectLeaf db need leaf
-- | 把需要哪些列往下传 -- | 把需要哪些列往下传
pushProject :: Database -> [String] -> RelOp -> RelOp pushProject :: Database -> [String] -> RelOp -> RelOp
pushProject _ _ Unit = Unit pushProject _ _ Unit = Unit
pushProject _ _ op@BoundSort{} = op
pushProject _ _ op@BoundAggregate{} = op
pushProject db _ (Compute items x) = Compute items (pushProject db (concatMap (colsInExpr . snd) items) x) pushProject db _ (Compute items x) = Compute items (pushProject db (concatMap (colsInExpr . snd) items) x)
pushProject db _ (Aggregate keys aggs x) = pushProject db _ (Aggregate keys aggs x) =
Aggregate keys aggs (pushProject db (keys ++ concatMap (colsInExpr . snd) aggs) x) Aggregate keys aggs (pushProject db (keys ++ concatMap (colsInExpr . snd) aggs) x)
@@ -103,6 +106,7 @@ pushProject db _ (Project cols x) = case pushProject db cols x of
Project cols' y | cols' == cols -> Project cols y Project cols' y | cols' == cols -> Project cols y
other -> Project cols other other -> Project cols other
pushProject db need (Filter p x) = Filter p (pushProject db (need ++ colsInExpr p) x) pushProject db need (Filter p x) = Filter p (pushProject db (need ++ colsInExpr p) x)
pushProject db _ (Distinct x) = Distinct (pushProject db (relOpCols db x) x)
pushProject db need (Sort spec x) = Sort spec (pushProject db (need ++ map fst spec) x) pushProject db need (Sort spec x) = Sort spec (pushProject db (need ++ map fst spec) x)
pushProject db need (Limit n x) = Limit n (pushProject db need x) pushProject db need (Limit n x) = Limit n (pushProject db need x)
pushProject db need (Join kind l r c) pushProject db need (Join kind l r c)
@@ -250,6 +254,7 @@ optimizeRelOp db = rewriteNode db . descend
descend (Project cols x) = Project cols (optimizeRelOp db x) descend (Project cols x) = Project cols (optimizeRelOp db x)
descend (Compute items x) = Compute items (optimizeRelOp db x) descend (Compute items x) = Compute items (optimizeRelOp db x)
descend (Aggregate keys aggs x) = Aggregate keys aggs (optimizeRelOp db x) descend (Aggregate keys aggs x) = Aggregate keys aggs (optimizeRelOp db x)
descend (Distinct x) = Distinct (optimizeRelOp db x)
descend (Sort spec x) = Sort spec (optimizeRelOp db x) descend (Sort spec x) = Sort spec (optimizeRelOp db x)
descend (Limit n x) = Limit n (optimizeRelOp db x) descend (Limit n x) = Limit n (optimizeRelOp db x)
descend (Join kind l r c) = Join kind (optimizeRelOp db l) (optimizeRelOp db r) c descend (Join kind l r c) = Join kind (optimizeRelOp db l) (optimizeRelOp db r) c
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Planner.hs
+7
View File
@@ -82,6 +82,13 @@ limited orderBy mLimit op = case mLimit of
-- | 翻成：过滤 -> 分组 -> 排序 -> 截断 -> 取列 -- | 翻成：过滤 -> 分组 -> 排序 -> 截断 -> 取列
translate :: Statement -> Either String RelOp translate :: Statement -> Either String RelOp
translate q = case q of translate q = case q of
SelectDistinct inner -> do
ordering <- distinctOrdering inner
plan <- case inner of
s@Select{} -> translate s { selectOrderBy = [], selectLimit = Nothing }
s@SelectExpr{} -> translate s { selectOrderBy = [], selectLimit = Nothing }
_ -> Left "DISTINCT requires SELECT"
Right (limited ordering (selectLimit inner) (Distinct plan))
Select Select
{ selectCols = cols { selectCols = cols
, selectFrom = fromC , selectFrom = fromC
chusql-core/engine/src/ChuSQL/Core/Engine/Algebra/Sort.hs
+20 -4
View File
@@ -1,7 +1,8 @@
module ChuSQL.Core.Engine.Algebra.Sort (sortRows, compareValue) where module ChuSQL.Core.Engine.Algebra.Sort (sortRows, sortBoundRows, compareValue) where
import ChuSQL.Core.Model import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Syntax.AST import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Engine.Algebra.Expr (evalExpr)
import Data.List (sortBy) import Data.List (sortBy)
-- 排序：按多列规格排出结果行，每列可升可降。 -- 排序：按多列规格排出结果行，每列可升可降。
@@ -17,9 +18,9 @@ sortKey :: [(String, SortDir)] -> Row -> [Maybe Value]
sortKey spec row = [lookup col row | (col, _) <- spec] sortKey spec row = [lookup col row | (col, _) <- spec]
-- | 比较两个排序键 -- | 比较两个排序键
compareKeys :: [(String, SortDir)] -> [Maybe Value] -> [Maybe Value] -> Ordering compareKeys :: [SortDir] -> [Maybe Value] -> [Maybe Value] -> Ordering
compareKeys [] _ _ = EQ compareKeys [] _ _ = EQ
compareKeys ((_, dir) : rest) (a : as) (b : bs) = compareKeys (dir : rest) (a : as) (b : bs) =
case (a, b) of case (a, b) of
(Just v1, Just v2) -> (Just v1, Just v2) ->
case compareValue v1 v2 of case compareValue v1 v2 of
@@ -36,4 +37,19 @@ sortRows spec rows = map snd (sortBy cmp (map decorate rows))
-- | 取键并带上原行 -- | 取键并带上原行
decorate row = (sortKey spec row, row) decorate row = (sortKey spec row, row)
-- | 比较两个带键的行 -- | 比较两个带键的行
cmp (k1, _) (k2, _) = compareKeys spec k1 k2 cmp (k1, _) (k2, _) = compareKeys (map snd spec) k1 k2
-- | 用绑定槽位取键并稳定排序
sortBoundRows :: [(Expr, SortDir)] -> Row -> [Row] -> Either String [Row]
sortBoundRows [] _ rows = Right rows
sortBoundRows spec outer rows = do
decorated <- mapM decorate rows
Right (map snd (sortBy compareRows decorated))
where
directions = map snd spec
-- | 每行只计算一次绑定排序键
decorate row = do
keys <- mapM (\(expression, _) -> evalExpr expression (row ++ outer)) spec
Right (map Just keys, row)
-- | 比较排序键并保留同键行序
compareRows (left, _) (right, _) = compareKeys directions left right
chusql-core/engine/src/ChuSQL/Core/Engine/Builtin.hs
+22 -22
View File
@@ -1,25 +1,25 @@
module ChuSQL.Core.Engine.Builtin module ChuSQL.Core.Engine.Builtin (
( Builtin (..) Builtin (..),
, builtinNames builtinNames,
, resolveBuiltin resolveBuiltin,
, builtinName builtinName,
, invokeBuiltin invokeBuiltin,
, builtinOfExpr builtinOfExpr,
, builtinNode builtinNode,
, aggregateArg aggregateArg,
, unsupportedAggregate unsupportedAggregate,
, Operator (..) Operator (..),
, operatorOfExpr operatorOfExpr,
, operatorSymbol operatorSymbol,
, operatorOfCompare operatorOfCompare,
, compareNode compareNode,
, executeOperator executeOperator,
, threeValuedNot threeValuedNot,
, inValues inValues,
) where ) where
import ChuSQL.Core.Model (Value (..), compareValue)
import ChuSQL.Core.Engine.Syntax.AST (CompareOp (..), Expr (..)) import ChuSQL.Core.Engine.Syntax.AST (CompareOp (..), Expr (..))
import ChuSQL.Core.Model (Value (..), compareValue)
import Control.Monad (foldM) import Control.Monad (foldM)
import Data.Char (toLower) import Data.Char (toLower)
@@ -246,12 +246,12 @@ evalArith ops x y = case (asDouble x, asDouble y) of
divArith :: Arith divArith :: Arith
divArith = Arith intDiv floatDiv divArith = Arith intDiv floatDiv
where where
-- | 整数除法：除零与溢出都报错 -- \| 整数除法：除零与溢出都报错
intDiv _ 0 = Left "division by zero" intDiv _ 0 = Left "division by zero"
intDiv x y intDiv x y
| x == minBound && y == -1 = Left "integer division overflow" | x == minBound && y == -1 = Left "integer division overflow"
| otherwise = Right (x `quot` y) | otherwise = Right (x `quot` y)
-- | 浮点除法：不能除以零 -- \| 浮点除法：不能除以零
floatDiv _ 0 = Left "division by zero" floatDiv _ 0 = Left "division by zero"
floatDiv x y = Right (x / y) floatDiv x y = Right (x / y)
chusql-core/engine/src/ChuSQL/Core/Engine/Error.hs
+11 -9
View File
@@ -1,12 +1,12 @@
module ChuSQL.Core.Engine.Error module ChuSQL.Core.Engine.Error (
( ErrorCategory (..) ErrorCategory (..),
, ErrorRule (..) ErrorRule (..),
, errorRules errorRules,
, errorCategory errorCategory,
, errorCode errorCode,
, errorLike errorLike,
, accountErrorCode accountErrorCode,
) where ) where
import Data.List (isInfixOf) import Data.List (isInfixOf)
@@ -35,6 +35,8 @@ errorRules :: [ErrorRule]
errorRules = errorRules =
[ ErrorRule "no database selected" NoDatabaseError "no_database" [ ErrorRule "no database selected" NoDatabaseError "no_database"
, ErrorRule "administrator required" PermissionError "forbidden" , ErrorRule "administrator required" PermissionError "forbidden"
, ErrorRule "bound query object changed" CatalogError "serialization_failure"
, ErrorRule "bound query schema changed" CatalogError "serialization_failure"
, ErrorRule "unknown table" CatalogError "not_found" , ErrorRule "unknown table" CatalogError "not_found"
, ErrorRule "unknown domain" CatalogError "not_found" , ErrorRule "unknown domain" CatalogError "not_found"
, ErrorRule "domain is still used" CatalogError "query_error" , ErrorRule "domain is still used" CatalogError "query_error"
chusql-core/engine/src/ChuSQL/Core/Engine/Normalization.hs
+44 -18
View File
@@ -14,23 +14,27 @@ data Declaration = Declaration
, dependencies :: [Dependency] , dependencies :: [Dependency]
, dependenciesComplete :: Bool , dependenciesComplete :: Bool
, atomicAttributes :: Maybe Bool , atomicAttributes :: Maybe Bool
} deriving (Eq, Show) }
deriving (Eq, Show)
data Finding = Finding data Finding = Finding
{ normalForm :: NormalForm { normalForm :: NormalForm
, verdict :: Verdict , verdict :: Verdict
, witnesses :: [Dependency] , witnesses :: [Dependency]
, condition :: String , condition :: String
} deriving (Eq, Show) }
deriving (Eq, Show)
data Decomposition = Decomposition data Decomposition = Decomposition
{ relations :: [Attributes] { relations :: [Attributes]
, losslessJoin :: Verdict , losslessJoin :: Verdict
, dependencyPreserving :: Verdict , dependencyPreserving :: Verdict
} deriving (Eq, Show) }
deriving (Eq, Show)
data Analysis = Analysis data Analysis = Analysis
{ candidateKeys :: [Attributes] { candidateKeys :: [Attributes]
, findings :: [Finding] , findings :: [Finding]
, suggestions :: [Decomposition] , suggestions :: [Decomposition]
} deriving (Eq, Show) }
deriving (Eq, Show)
-- 规范化属性集合。 -- 规范化属性集合。
canonical :: Attributes -> Attributes canonical :: Attributes -> Attributes
@@ -45,7 +49,8 @@ attributeClosure :: [Dependency] -> Attributes -> Attributes
attributeClosure f seed = fixed (canonical seed) attributeClosure f seed = fixed (canonical seed)
where where
-- 迭代加入已蕴含的属性。 -- 迭代加入已蕴含的属性。
fixed xs = let ys = canonical (xs ++ concat [b | Dependency a b <- f, subset a xs]) fixed xs =
let ys = canonical (xs ++ concat [b | Dependency a b <- f, subset a xs])
in if xs == ys then xs else fixed ys in if xs == ys then xs else fixed ys
-- 判断声明是否蕴含依赖。 -- 判断声明是否蕴含依赖。
@@ -54,8 +59,12 @@ implies f (Dependency a b) = subset b (attributeClosure f a)
-- 枚举最小超键。 -- 枚举最小超键。
keysOf :: Attributes -> [Dependency] -> [Attributes] keysOf :: Attributes -> [Dependency] -> [Attributes]
keysOf attrs f = [x | x <- subsequences attrs, super x, keysOf attrs f =
all (not . super) [x \\ [a] | a <- x]] [ x
| x <- subsequences attrs
, super x
, all (not . super) [x \\ [a] | a <- x]
]
where where
-- 判断闭包是否覆盖关系。 -- 判断闭包是否覆盖关系。
super x = subset attrs (attributeClosure f x) super x = subset attrs (attributeClosure f x)
@@ -63,8 +72,12 @@ keysOf attrs f = [x | x <- subsequences attrs, super x,
-- 投影关系上的全部非平凡依赖。 -- 投影关系上的全部非平凡依赖。
projectDependencies :: [Dependency] -> Attributes -> [Dependency] projectDependencies :: [Dependency] -> Attributes -> [Dependency]
projectDependencies f attrs = projectDependencies f attrs =
[Dependency x [a] | x <- subsequences attrs, [ Dependency x [a]
a <- attributeClosure f x, a `elem` attrs, a `notElem` x] | x <- subsequences attrs
, a <- attributeClosure f x
, a `elem` attrs
, a `notElem` x
]
-- 验证声明并生成只读范式报告。 -- 验证声明并生成只读范式报告。
analyzeNormalization :: [(String, Column)] -> Declaration -> Either String Analysis analyzeNormalization :: [(String, Column)] -> Declaration -> Either String Analysis
@@ -90,14 +103,26 @@ analyzeNormalization schema declaration
Just False -> Violated Just False -> Violated
Nothing -> Insufficient Nothing -> Insufficient
allFDs = projectDependencies f attrs allFDs = projectDependencies f attrs
partial = [d | d@(Dependency x [a]) <- allFDs, a `notElem` prime, partial =
any (\k -> x /= k && subset x k) keys] [ d
third = [d | d@(Dependency x [a]) <- allFDs, a `notElem` prime, | d@(Dependency x [a]) <- allFDs
not (subset attrs (attributeClosure f x))] , a `notElem` prime
boyce = [d | d@(Dependency x _) <- allFDs, , any (\k -> x /= k && subset x k) keys
not (subset attrs (attributeClosure f x))] ]
results = Finding NF1 atomic [] "attributes must have atomic domains; requires an explicit declaration" : third =
[classify NF2 partial, classify NF3 third, classify BCNF boyce] [ d
| d@(Dependency x [a]) <- allFDs
, a `notElem` prime
, not (subset attrs (attributeClosure f x))
]
boyce =
[ d
| d@(Dependency x _) <- allFDs
, not (subset attrs (attributeClosure f x))
]
results =
Finding NF1 atomic [] "attributes must have atomic domains; requires an explicit declaration"
: [classify NF2 partial, classify NF3 third, classify BCNF boyce]
decompositions = [split d | d <- take 1 boyce, complete, atomic == Satisfied] decompositions = [split d | d <- take 1 boyce, complete, atomic == Satisfied]
-- 校验字段集合。 -- 校验字段集合。
valid xs = length xs == length (canonical xs) && subset xs attrs valid xs = length xs == length (canonical xs) && subset xs attrs
@@ -108,7 +133,8 @@ analyzeNormalization schema declaration
-- 结合声明完整性判定范式。 -- 结合声明完整性判定范式。
classify nf ds = Finding nf status (if complete then ds else []) (rule nf) classify nf ds = Finding nf status (if complete then ds else []) (rule nf)
where where
status | atomic == Violated = Violated status
| atomic == Violated = Violated
| atomic == Insufficient || not complete = Insufficient | atomic == Insufficient || not complete = Insufficient
| null ds = Satisfied | null ds = Satisfied
| otherwise = Violated | otherwise = Violated
chusql-core/engine/src/ChuSQL/Core/Engine/Parallel.hs
+4 -3
View File
@@ -22,6 +22,7 @@ parallelShardLimit = 8
newtype Pool = Pool (Chan (IO ())) newtype Pool = Pool (Chan (IO ()))
{-# NOINLINE poolRef #-} {-# NOINLINE poolRef #-}
-- | 进程全局的池，按容量复用 -- | 进程全局的池，按容量复用
poolRef :: MVar (Maybe (Int, Pool)) poolRef :: MVar (Maybe (Int, Pool))
poolRef = unsafePerformIO (newMVar Nothing) poolRef = unsafePerformIO (newMVar Nothing)
@@ -30,7 +31,7 @@ poolRef = unsafePerformIO (newMVar Nothing)
workerPool :: Int -> IO Pool workerPool :: Int -> IO Pool
workerPool size = modifyMVar poolRef reuse workerPool size = modifyMVar poolRef reuse
where where
-- | 复用队列并按需补充工作线程 -- \| 复用队列并按需补充工作线程
reuse current = case current of reuse current = case current of
Just (capacity, pool@(Pool jobs)) Just (capacity, pool@(Pool jobs))
| capacity >= wanted -> pure (current, pool) | capacity >= wanted -> pure (current, pool)
@@ -41,10 +42,10 @@ workerPool size = modifyMVar poolRef reuse
pool <- startPool wanted pool <- startPool wanted
pure (Just (wanted, pool), pool) pure (Just (wanted, pool), pool)
-- | 池的线程数至少一个 -- \| 池的线程数至少一个
wanted = max 1 size wanted = max 1 size
-- | 起 wanted 个常驻线程吃队列 -- \| 起 wanted 个常驻线程吃队列
startPool n = do startPool n = do
jobs <- newChan jobs <- newChan
sequence_ [forkIO (worker jobs) | _ <- [1 .. n]] sequence_ [forkIO (worker jobs) | _ <- [1 .. n]]
chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Binding.hs
+6 -6
View File
@@ -16,7 +16,7 @@ data Expression = Literal TypeExpr RuntimeValue | Reference String | Apply Strin
| TupleLiteral [Expression] deriving (Show, Eq) | TupleLiteral [Expression] deriving (Show, Eq)
-- | 保存绑定后的表达式节点 -- | 保存绑定后的表达式节点
data ExpressionNode = Constant RuntimeValue | BoundReference String | BoundApply FunctionId [TypedExpression] data ExpressionNode = Constant RuntimeValue | BoundReference Int String | BoundApply FunctionId [TypedExpression]
| BoundList [TypedExpression] | BoundNothing | BoundJust TypedExpression | BoundTuple [TypedExpression] | BoundList [TypedExpression] | BoundNothing | BoundJust TypedExpression | BoundTuple [TypedExpression]
deriving (Show, Eq) deriving (Show, Eq)
@@ -34,8 +34,8 @@ bindExpression types functions env expression = case expression of
tid <- descriptorId <$> resolveType types ty tid <- descriptorId <$> resolveType types ty
validateValue types tid value validateValue types tid value
Right (TypedExpression tid (Constant value)) Right (TypedExpression tid (Constant value))
Reference name -> case [tid | (key, tid) <- env, key == name] of Reference name -> case [(slot, tid) | (slot, (key, tid)) <- zip [0 ..] env, key == name] of
[tid] -> describeType types tid >> Right (TypedExpression tid (BoundReference name)) [(slot, tid)] -> describeType types tid >> Right (TypedExpression tid (BoundReference slot name))
[] -> Left ("unknown column: " ++ name) [] -> Left ("unknown column: " ++ name)
_ -> Left ("ambiguous column: " ++ name) _ -> Left ("ambiguous column: " ++ name)
Apply name arguments -> do Apply name arguments -> do
@@ -68,10 +68,10 @@ evaluateExpression :: TypeRegistry -> FunctionRegistry -> [(String, RuntimeValue
evaluateExpression types functions row (TypedExpression tid node) = do evaluateExpression types functions row (TypedExpression tid node) = do
value <- case node of value <- case node of
Constant v -> Right v Constant v -> Right v
BoundReference name -> case [v | (key, v) <- row, key == name] of BoundReference slot name -> case drop slot row of
[v] -> Right v (key, value) : _ | key == name -> Right value
[] -> Left ("missing bound column: " ++ name) [] -> Left ("missing bound column: " ++ name)
_ -> Left ("duplicate bound column: " ++ name) _ -> Left ("bound column position mismatch: " ++ name)
BoundApply fid arguments -> mapM evaluate arguments >>= invokeFunction types functions fid BoundApply fid arguments -> mapM evaluate arguments >>= invokeFunction types functions fid
BoundList arguments -> RList <$> mapM evaluate arguments BoundList arguments -> RList <$> mapM evaluate arguments
BoundNothing -> Right (RMaybe Nothing) BoundNothing -> Right (RMaybe Nothing)
chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Dependencies.hs
+82
View File
@@ -0,0 +1,82 @@
module ChuSQL.Core.Engine.Runtime.Dependencies (queryTables, domainDependencies) where
import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Model (Database, ColumnType (..), columnType, tableCols, domainTableName, lastTablePart)
import Data.List (nub)
-- 收集查询及嵌套表达式的表依赖，供绑定执行前校验目录。
-- | 收集全部查询来源表
queryTables :: Statement -> [String]
queryTables statement = nub (case statement of
SelectDistinct child -> queryTables child
Select _ source condition _ _ _ -> sourceTables source ++ concatMap expressionTables condition
SelectExpr items source condition _ _ _ -> sourceTables source
++ concatMap (expressionTables . snd) items ++ concatMap expressionTables condition
_ -> [])
-- | 收集来源列依赖的领域定义
domainDependencies :: Database -> [String] -> [String]
domainDependencies catalog sources = collect []
[domain | (name, table) <- catalog, name `elem` sources
, (_, column) <- tableCols table, domain <- typeDomains (columnType column)]
where
-- | 递归展开定义并防止重复访问
collect seen [] = seen
collect seen (domain : remaining) =
let definitions = [(name, table) | (name, table) <- catalog
, lastTablePart name == domainTableName domain, name `notElem` seen]
names = map fst definitions
nested = [child | (_, table) <- definitions, (_, column) <- tableCols table
, child <- typeDomains (columnType column)]
in collect (seen ++ names) (remaining ++ nested)
-- | 保留嵌套领域的每层依赖
typeDomains (CDomain name base) = name : typeDomains base
typeDomains _ = []
-- | 收集来源树与连接条件依赖
sourceTables :: FromClause -> [String]
sourceTables source = case source of
FromTable _ name -> [name]
FromSubquery _ statement -> queryTables statement
FromJoin _ left right condition -> sourceTables left ++ sourceTables right ++ expressionTables condition
FromUnit -> []
-- | 收集表达式内嵌套子查询依赖
expressionTables :: Expr -> [String]
expressionTables expression = case expression of
ScalarSub query -> queryTables (subqueryStatement query)
ExistsSub query _ -> queryTables (subqueryStatement query)
InSub value query _ -> expressionTables value ++ queryTables (subqueryStatement query)
QuantCmp _ value query _ -> expressionTables value ++ queryTables (subqueryStatement query)
_ -> concatMap expressionTables (children expression)
-- | 列出当前表达式的直接子节点
children :: Expr -> [Expr]
children expression = case expression of
FunctionCall _ arguments -> arguments
BoundFunction _ _ arguments -> arguments
Construct _ arguments -> arguments
BoundConstruct _ arguments -> arguments
InList value arguments _ -> value : arguments
Add a b -> [a, b]
Sub a b -> [a, b]
Mul a b -> [a, b]
Div a b -> [a, b]
Gt a b -> [a, b]
Lt a b -> [a, b]
Eq a b -> [a, b]
GtE a b -> [a, b]
LtE a b -> [a, b]
NotEq a b -> [a, b]
And a b -> [a, b]
Or a b -> [a, b]
Neg a -> [a]
IsNull a -> [a]
IsNotNull a -> [a]
CountOf a -> [a]
SumOf a -> [a]
AvgOf a -> [a]
MinOf a -> [a]
MaxOf a -> [a]
_ -> []
chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Query.hs
+28 -9
View File
@@ -8,22 +8,25 @@ import ChuSQL.Core.Engine.Algebra.Optimize (optimize)
import ChuSQL.Core.Engine.Algebra.Planner (translate) import ChuSQL.Core.Engine.Algebra.Planner (translate)
import ChuSQL.Core.Engine.Semantic (prepare, querySchema) import ChuSQL.Core.Engine.Semantic (prepare, querySchema)
import ChuSQL.Core.Engine.Syntax.AST (Statement) import ChuSQL.Core.Engine.Syntax.AST (Statement)
import ChuSQL.Core.Engine.Storage (MonadStorage) import ChuSQL.Core.Engine.Storage (MonadStorage (schema, withSchemaGuard))
import ChuSQL.Core.Engine.Runtime.SQL (sqlType) import ChuSQL.Core.Engine.Runtime.SQL (sqlType)
import ChuSQL.Core.Engine.Runtime.Types (TypeId) import ChuSQL.Core.Engine.Runtime.Types (TypeId)
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
import ChuSQL.Core.Engine.Runtime.Dependencies (queryTables, domainDependencies)
import ChuSQL.Core.Model import ChuSQL.Core.Model
-- SQL 查询绑定边界：保存解析后的类型、函数身份及输出 schema。 -- SQL 查询绑定边界：保存解析后的类型、函数身份及输出 schema。
-- | 保存运行时或兼容列类型 -- | 保存运行时或兼容列类型
data QueryType = RuntimeType TypeId | CompatibilityType ColumnType | NullType deriving (Show, Eq) data QueryType = RuntimeType TypeId | CompatibilityType ColumnType
| DomainType String QueryType | NullType deriving (Show, Eq)
-- | 保存仅由 schema 绑定的执行计划 -- | 保存仅由 schema 绑定的执行计划
data BoundQuery = BoundQuery Database [(String, Column)] [(String, QueryType)] RelOp data BoundQuery = BoundQuery Database [String] [(String, Column)] [(String, QueryType)] RelOp
-- | 查询绑定计划的输出类型身份 -- | 查询绑定计划的输出类型身份
queryTypes :: BoundQuery -> [(String, QueryType)] queryTypes :: BoundQuery -> [(String, QueryType)]
queryTypes (BoundQuery _ _ types _) = types queryTypes (BoundQuery _ _ _ types _) = types
-- | 绑定 SQL 查询并优化确定的计划 -- | 绑定 SQL 查询并优化确定的计划
bindQuery :: Database -> Statement -> Either String BoundQuery bindQuery :: Database -> Statement -> Either String BoundQuery
@@ -33,7 +36,10 @@ bindQuery catalog statement = do
columns <- querySchema schemaOnly resolved columns <- querySchema schemaOnly resolved
plan <- translate resolved plan <- translate resolved
types <- mapM typeOf columns types <- mapM typeOf columns
Right (BoundQuery schemaOnly columns types (optimize schemaOnly plan)) bound <- bindRelOpColumns schemaOnly [] (optimize schemaOnly plan)
let sources = queryTables resolved
dependencies = sources ++ domainDependencies schemaOnly sources
Right (BoundQuery schemaOnly dependencies columns types bound)
where where
-- | 解析输出列的运行时身份 -- | 解析输出列的运行时身份
typeOf (name, column) = do typeOf (name, column) = do
@@ -41,19 +47,32 @@ bindQuery catalog statement = do
Right (name, ty) Right (name, ty)
-- | 保留阶段外类型的兼容表示 -- | 保留阶段外类型的兼容表示
bindingType CNull = Right NullType bindingType CNull = Right NullType
bindingType (CDomain name base) = DomainType name <$> bindingType base
bindingType CDate = Right (CompatibilityType CDate) bindingType CDate = Right (CompatibilityType CDate)
bindingType CTimestamp = Right (CompatibilityType CTimestamp) bindingType CTimestamp = Right (CompatibilityType CTimestamp)
bindingType CBlob = Right (CompatibilityType CBlob) bindingType CBlob = Right (CompatibilityType CBlob)
bindingType ty@(CDomain _ base)
| typeClassOf base == TemporalClass || base == CBlob = Right (CompatibilityType ty)
bindingType ty = RuntimeType <$> sqlType ty bindingType ty = RuntimeType <$> sqlType ty
-- | 执行绑定查询并校验输出类型 -- | 执行绑定查询并校验输出类型
executeQueryM :: MonadStorage m => BoundQuery -> m (Either String [Row]) executeQueryM :: MonadStorage m => BoundQuery -> m (Either String [Row])
executeQueryM (BoundQuery catalog columns _ plan) = do executeQueryM (BoundQuery catalog dependencies columns _ plan) = do
result <- evalRelOpM catalog plan current <- schema
case current >>= validateSchema of
Left err -> pure (Left err)
Right () -> do
result <- withSchemaGuard [(name, table) | (name, table) <- catalog, name `elem` dependencies]
(evalRelOpM catalog plan)
pure (result >>= mapM validateRow) pure (result >>= mapM validateRow)
where where
-- | 拒绝绑定目录中的结构变化
validateSchema current = mapM_ (validateTable current)
[(name, table) | (name, table) <- catalog, name `elem` dependencies]
-- | 比较列定义而不比较数据或统计
validateTable current (name, expected) = case lookup name current of
Just actual | tableIdentity actual /= tableIdentity expected -> Left ("bound query object changed: " ++ name)
Just actual | tableSchemaVersion actual /= tableSchemaVersion expected -> Left ("bound query schema changed: " ++ name)
Just actual | tableCols actual == tableCols expected -> Right ()
_ -> Left ("bound query schema changed: " ++ name)
-- | 校验完整输出行而不重新解析函数 -- | 校验完整输出行而不重新解析函数
validateRow row validateRow row
| map fst row /= map fst columns = Left "bound query output schema mismatch" | map fst row /= map fst columns = Left "bound query output schema mismatch"
chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Relation.hs
+100
View File
@@ -0,0 +1,100 @@
module ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns) where
import ChuSQL.Core.Engine.Algebra.Op
import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Model (Database)
-- 优化完成后按算子输入布局绑定列槽位，外层行作为独立尾部布局。
-- | 绑定关系算子的输入列位置
bindRelOpColumns :: Database -> [String] -> RelOp -> Either String RelOp
bindRelOpColumns db outer = bind
where
-- \| 递归绑定每个算子的输入布局
bind op = case op of
Filter expression child -> Filter <$> expressionIn child expression <*> bind child
Compute items child -> Compute <$> mapM (itemIn child) items <*> bind child
Aggregate keys items child ->
BoundAggregate
<$> mapM (itemIn child) [(name, Col name) | name <- keys]
<*> mapM (itemIn child) items
<*> bind child
BoundAggregate keys items child ->
BoundAggregate
<$> mapM (itemIn child) keys
<*> mapM (itemIn child) items
<*> bind child
Join kind left right condition ->
Join kind
<$> bind left
<*> bind right
<*> bindSlots (relOpCols db left ++ relOpCols db right) outer condition
Project columns child -> do
let names = if columns == ["*"] then relOpCols db child else columns
items <- mapM (itemIn child) [(name, Col name) | name <- names]
Compute items <$> bind child
Distinct child -> Distinct <$> bind child
Sort ordering child ->
BoundSort
<$> mapM (sortIn child) [(Col name, direction) | (name, direction) <- ordering]
<*> bind child
BoundSort ordering child -> BoundSort <$> mapM (sortIn child) ordering <*> bind child
Limit count child -> Limit count <$> bind child
Derived alias child -> Derived alias <$> bind child
_ -> Right op
-- \| 使用子计划输出布局绑定表达式
expressionIn child = bindSlots (relOpCols db child) outer
-- \| 保留输出标签并绑定计算项
itemIn child (name, expression) = (,) name <$> expressionIn child expression
-- \| 绑定排序键并保留方向
sortIn child (expression, direction) = (,) <$> expressionIn child expression <*> pure direction
-- | 绑定当前表达式的列引用
bindSlots :: [String] -> [String] -> Expr -> Either String Expr
bindSlots columns outer expression = case expression of
Col name -> case slots columns name of
slot : _ -> Right (ColumnSlot slot name)
[] -> outerReference name
ColumnSlot _ name -> case slots columns name of
slot : _ -> Right (ColumnSlot slot name)
[] -> Left ("unknown local column: " ++ name)
OuterColumnSlot _ _ name -> outerReference name
FunctionCall name arguments -> FunctionCall name <$> mapM bind arguments
BoundFunction fid tid arguments -> BoundFunction fid tid <$> mapM bind arguments
Construct ty arguments -> Construct ty <$> mapM bind arguments
BoundConstruct tid arguments -> BoundConstruct tid <$> mapM bind arguments
Add a b -> binary Add a b
Sub a b -> binary Sub a b
Mul a b -> binary Mul a b
Div a b -> binary Div a b
Neg a -> Neg <$> bind a
Gt a b -> binary Gt a b
Lt a b -> binary Lt a b
Eq a b -> binary Eq a b
GtE a b -> binary GtE a b
LtE a b -> binary LtE a b
NotEq a b -> binary NotEq a b
And a b -> binary And a b
Or a b -> binary Or a b
IsNull a -> IsNull <$> bind a
IsNotNull a -> IsNotNull <$> bind a
CountOf a -> CountOf <$> bind a
SumOf a -> SumOf <$> bind a
AvgOf a -> AvgOf <$> bind a
MinOf a -> MinOf <$> bind a
MaxOf a -> MaxOf <$> bind a
InSub a query negated -> InSub <$> bind a <*> pure query <*> pure negated
InList a values negated -> InList <$> bind a <*> mapM bind values <*> pure negated
QuantCmp op a query quantifier -> QuantCmp op <$> bind a <*> pure query <*> pure quantifier
_ -> Right expression
where
-- \| 绑定当前布局的子表达式
bind = bindSlots columns outer
-- \| 查找布局中首次出现的列
slots names name = [slot | (slot, key) <- zip [0 ..] names, key == name]
-- \| 在外层布局绑定并保留作用域
outerReference name = case slots outer name of
slot : _ -> Right (OuterColumnSlot (length columns) slot name)
[] -> Left ("unknown column: " ++ name)
-- \| 绑定两个操作数
binary constructor a b = constructor <$> bind a <*> bind b
chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/SQL.hs
+10 -5
View File
@@ -1,7 +1,7 @@
module ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, invokeSQLFunction, sqlType, constructSQLValue) where module ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, invokeSQLFunction, sqlType, constructSQLValue) where
import ChuSQL.Core.Engine.Runtime.Types
import ChuSQL.Core.Engine.Runtime.Functions import ChuSQL.Core.Engine.Runtime.Functions
import ChuSQL.Core.Engine.Runtime.Types
import ChuSQL.Core.Model (ColumnType (..), Value (..)) import ChuSQL.Core.Model (ColumnType (..), Value (..))
-- SQL 现有标量表示与已解析运行时函数之间的类型边界。 -- SQL 现有标量表示与已解析运行时函数之间的类型边界。
@@ -46,16 +46,21 @@ invokeSQLFunction :: FunctionId -> TypeId -> [Value] -> Either String Value
invokeSQLFunction fid expected values = do invokeSQLFunction fid expected values = do
functions <- builtinFunctions builtinTypes functions <- builtinFunctions builtinTypes
sig <- functionSignature functions fid sig <- functionSignature functions fid
if signatureResult sig /= expected then Left "bound function result identity mismatch" if signatureResult sig /= expected
else if length values /= length (signatureArguments sig) then Left "function argument count mismatch" then Left "bound function result identity mismatch"
else
if length values /= length (signatureArguments sig)
then Left "function argument count mismatch"
else do else do
sequence_ (zipWith validateArgument (signatureArguments sig) values) sequence_ (zipWith validateArgument (signatureArguments sig) values)
if VNull `elem` values then Right VNull else do if VNull `elem` values
then Right VNull
else do
arguments <- mapM runtimeValue values arguments <- mapM runtimeValue values
result <- invokeFunction builtinTypes functions fid arguments result <- invokeFunction builtinTypes functions fid arguments
sqlValue result sqlValue result
where where
-- | 校验具体参数并保留 SQL NULL -- \| 校验具体参数并保留 SQL NULL
validateArgument _ VNull = Right () validateArgument _ VNull = Right ()
validateArgument tid value = runtimeValue value >>= validateValue builtinTypes tid validateArgument tid value = runtimeValue value >>= validateValue builtinTypes tid
chusql-core/engine/src/ChuSQL/Core/Engine/Semantic.hs
+69 -44
View File
@@ -1,12 +1,12 @@
module ChuSQL.Core.Engine.Semantic (check, prepare, prepareRowExpression, querySchema) where module ChuSQL.Core.Engine.Semantic (check, prepare, prepareRowExpression, querySchema) where
import ChuSQL.Core.Engine.Algebra.Expr (bareColumns, colsInExpr, hasAggregate) import ChuSQL.Core.Engine.Algebra.Expr (bareColumns, colsInExpr, hasAggregate)
import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
import ChuSQL.Core.Engine.Runtime.Functions (FunctionSignature (..), builtinFunctions, functionSignature) import ChuSQL.Core.Engine.Runtime.Functions (FunctionSignature (..), builtinFunctions, functionSignature)
import ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, sqlType) import ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, sqlType)
import ChuSQL.Core.Engine.Runtime.Types import ChuSQL.Core.Engine.Runtime.Types
import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
import ChuSQL.Core.Model
import Data.Char (isAlphaNum, isAscii, toLower) import Data.Char (isAlphaNum, isAscii, toLower)
import Data.List (intercalate, isInfixOf, isPrefixOf, isSuffixOf, nub) import Data.List (intercalate, isInfixOf, isPrefixOf, isSuffixOf, nub)
@@ -115,7 +115,7 @@ checkGrouping env keys items
mapM_ (checkRuntimeColumn "GROUP BY" env) keys mapM_ (checkRuntimeColumn "GROUP BY" env) keys
mapM_ okCol (nub (concatMap (bareColumns . snd) items)) mapM_ okCol (nub (concatMap (bareColumns . snd) items))
where where
-- | 这一列能不能出现在分组查询里 -- \| 这一列能不能出现在分组查询里
okCol c okCol c
| c == allColumns = Left "SELECT: * cannot be selected together with GROUP BY" | c == allColumns = Left "SELECT: * cannot be selected together with GROUP BY"
| c `elem` keys = Right () | c `elem` keys = Right ()
@@ -129,6 +129,8 @@ renderType (InferType t) = typeLabel t
-- | 推导表达式的类型（子查询也在这里整条查一遍） -- | 推导表达式的类型（子查询也在这里整条查一遍）
inferExpr :: Database -> String -> Scope -> Expr -> Either String InferredType inferExpr :: Database -> String -> Scope -> Expr -> Either String InferredType
inferExpr _ place env (Col c) = inferredColumnType . columnType . snd <$> scopeAt place env c inferExpr _ place env (Col c) = inferredColumnType . columnType . snd <$> scopeAt place env c
inferExpr _ _ _ ColumnSlot{} = Left "execution column cannot be used in semantic input"
inferExpr _ _ _ OuterColumnSlot{} = Left "execution column cannot be used in semantic input"
inferExpr _ _ _ LitNull = Right InferNull inferExpr _ _ _ LitNull = Right InferNull
inferExpr _ _ _ (LitInt _) = Right (InferType CInt) inferExpr _ _ _ (LitInt _) = Right (InferType CInt)
inferExpr _ _ _ (LitFloat _) = Right (InferType CFloat) inferExpr _ _ _ (LitFloat _) = Right (InferType CFloat)
@@ -231,18 +233,23 @@ itemType place want got
-- | 推导子查询的输出列与类型 -- | 推导子查询的输出列与类型
inferSubqueryOutput :: Database -> Scope -> Subquery -> Either String [(String, InferredType)] inferSubqueryOutput :: Database -> Scope -> Subquery -> Either String [(String, InferredType)]
inferSubqueryOutput db outerScope sq = case subqueryStatement sq of inferSubqueryOutput db outerScope sq = case subqueryStatement sq of
q@Select { selectCols = cols } -> do SelectDistinct q -> do
columns <- inferSubqueryOutput db outerScope sq{subqueryStatement = q}
mapM_ (requireOrdering "DISTINCT" . snd) columns
_ <- distinctOrdering q
Right columns
q@Select{selectCols = cols} -> do
env <- checkFrom db enclosing (selectFrom q) env <- checkFrom db enclosing (selectFrom q)
checkResolvedWith db enclosing q checkResolvedWith db enclosing q
mapM (columnTypeOf env) cols mapM (columnTypeOf env) cols
q@SelectExpr { selectItems = items } -> do q@SelectExpr{selectItems = items} -> do
env <- checkFrom db enclosing (selectFrom q) env <- checkFrom db enclosing (selectFrom q)
checkResolvedWith db enclosing q checkResolvedWith db enclosing q
mapM (\(l, e) -> (\t -> (l, t)) <$> inferExpr db "SELECT" (Scope env enclosing) e) items mapM (\(l, e) -> (\t -> (l, t)) <$> inferExpr db "SELECT" (Scope env enclosing) e) items
_ -> Left "subquery must be a SELECT" _ -> Left "subquery must be a SELECT"
where where
enclosing = scopeFlat outerScope enclosing = scopeFlat outerScope
-- | 查这一列的推导类型 -- \| 查这一列的推导类型
columnTypeOf env c = do columnTypeOf env c = do
(_, col) <- scopeAt "SELECT" (Scope env enclosing) c (_, col) <- scopeAt "SELECT" (Scope env enclosing) c
Right (c, inferredColumnType (columnType col)) Right (c, inferredColumnType (columnType col))
@@ -278,7 +285,7 @@ arithmetic db place env a b = do
mapM_ (numericOperand place (InferType want) (InferType want)) [ta, tb] mapM_ (numericOperand place (InferType want) (InferType want)) [ta, tb]
Right (arithResult ta tb) Right (arithResult ta tb)
where where
-- | 是不是浮点类型 -- \| 是不是浮点类型
isFloatSide (InferType x) = numericType x && not (integerType x) isFloatSide (InferType x) = numericType x && not (integerType x)
isFloatSide _ = False isFloatSide _ = False
@@ -312,7 +319,7 @@ boolean db place env a b = do
mapM_ (boolOperand place ta) [ta, tb] mapM_ (boolOperand place ta) [ta, tb]
Right (InferType CBool) Right (InferType CBool)
where where
-- | 挨个查操作数是不是布尔 -- \| 挨个查操作数是不是布尔
boolOperand _ _ InferNull = Right () boolOperand _ _ InferNull = Right ()
boolOperand _ _ (InferType ty) | typeClassOf ty == BooleanClass = Right () boolOperand _ _ (InferType ty) | typeClassOf ty == BooleanClass = Right ()
boolOperand p other t = Left (p ++ ": operator needs TBool on both sides, got " ++ renderType other ++ " and " ++ renderType t) boolOperand p other t = Left (p ++ ": operator needs TBool on both sides, got " ++ renderType other ++ " and " ++ renderType t)
@@ -419,13 +426,13 @@ resolveDomainTypes db statement = case statement of
AlterColumnType name col ty -> AlterColumnType name col <$> resolveDomainType ty AlterColumnType name col ty -> AlterColumnType name col <$> resolveDomainType ty
_ -> Right statement _ -> Right statement
where where
-- | 解析一列的类型 -- \| 解析一列的类型
resolveColumnType (name, col) = do resolveColumnType (name, col) = do
ty <- resolveDomainType (columnType col) ty <- resolveDomainType (columnType col)
Right (name, col {columnType = ty}) Right (name, col{columnType = ty})
-- | 从类型目录取基础类型 -- \| 从类型目录取基础类型
resolveDomainType = resolveWith [] resolveDomainType = resolveWith []
-- | 递归解析类型并拒绝循环定义 -- \| 递归解析类型并拒绝循环定义
resolveWith seen (CDomain name _) resolveWith seen (CDomain name _)
| map toLower name `elem` seen = Left ("cyclic domain definition: " ++ name) | map toLower name `elem` seen = Left ("cyclic domain definition: " ++ name)
| otherwise = do | otherwise = do
@@ -442,6 +449,7 @@ checkResolved db = checkResolvedWith db []
-- | 检查已解析的语句；outer 是相关子查询能看见的外层列 -- | 检查已解析的语句；outer 是相关子查询能看见的外层列
checkResolvedWith :: Database -> [(String, Column)] -> Statement -> Either String () checkResolvedWith :: Database -> [(String, Column)] -> Statement -> Either String ()
checkResolvedWith db outer q = case q of checkResolvedWith db outer q = case q of
SelectDistinct inner -> () <$ inferSubqueryOutput db (Scope [] outer) (Subquery (SelectDistinct inner) [])
Select Select
{ selectCols = cols { selectCols = cols
, selectFrom = fromC , selectFrom = fromC
@@ -552,6 +560,7 @@ checkResolvedWith db outer q = case q of
if length (nub (map fst attributes)) == length attributes if length (nub (map fst attributes)) == length attributes
then Right () then Right ()
else Left "duplicate identity attribute" else Left "duplicate identity attribute"
ShowGrants -> Right ()
ShowRoles -> Right () ShowRoles -> Right ()
DropUser name -> checkUserName name DropUser name -> checkUserName name
CreateRole name -> checkRoleName name CreateRole name -> checkRoleName name
@@ -560,6 +569,7 @@ checkResolvedWith db outer q = case q of
mapM_ checkPrivilege privs mapM_ checkPrivilege privs
checkGrantObject obj checkGrantObject obj
checkRoleName role checkRoleName role
RevokeGrantOption privs obj role -> checkResolvedWith db outer (RevokePrivileges privs obj role)
RevokePrivileges privs obj role -> do RevokePrivileges privs obj role -> do
mapM_ checkPrivilege privs mapM_ checkPrivilege privs
checkGrantObject obj checkGrantObject obj
@@ -577,11 +587,11 @@ checkResolvedWith db outer q = case q of
CreateDomain{} -> Right () CreateDomain{} -> Right ()
DropDomain{} -> Right () DropDomain{} -> Right ()
ShowDomains -> Right () ShowDomains -> Right ()
-- | 事务控制由会话层执行，语义层不做列检查 -- \| 事务控制由会话层执行，语义层不做列检查
BeginTransaction -> Right () BeginTransaction -> Right ()
CommitTransaction -> Right () CommitTransaction -> Right ()
RollbackTransaction -> Right () RollbackTransaction -> Right ()
-- | 保存点名字只是会话层的记号，不做列检查 -- \| 保存点名字只是会话层的记号，不做列检查
Savepoint _ -> Right () Savepoint _ -> Right ()
RollbackToSavepoint _ -> Right () RollbackToSavepoint _ -> Right ()
ReleaseSavepoint _ -> Right () ReleaseSavepoint _ -> Right ()
@@ -615,10 +625,11 @@ checkColumnDefs names cols = do
else Right () else Right ()
checkChecks names cols checkChecks names cols
where where
-- | 拒绝缺少能力实现的运行时键 -- \| 拒绝缺少能力实现的运行时键
checkRuntimeKey (_, column) checkRuntimeKey (_, column)
| typeClassOf (columnType column) == RuntimeClass | typeClassOf (columnType column) == RuntimeClass
, columnUnique column || columnPrimaryKey column = Left "runtime type key capability is not registered" , columnUnique column || columnPrimaryKey column =
Left "runtime type key capability is not registered"
| otherwise = Right () | otherwise = Right ()
-- | 校验列类型的长度和精度参数 -- | 校验列类型的长度和精度参数
@@ -649,13 +660,13 @@ checkDefault c v
checkChecks :: [String] -> [(String, Column)] -> Either String () checkChecks :: [String] -> [(String, Column)] -> Either String ()
checkChecks names = mapM_ checkOne checkChecks names = mapM_ checkOne
where where
-- | 查这一列的 CHECK -- \| 查这一列的 CHECK
checkOne (_, c) = case columnCheck c of checkOne (_, c) = case columnCheck c of
Nothing -> Right () Nothing -> Right ()
Just text -> case parseExpression text of Just text -> case parseExpression text of
Left err -> Left ("CHECK: " ++ firstLine err) Left err -> Left ("CHECK: " ++ firstLine err)
Right e -> mapM_ (known e) (colsInExpr e) Right e -> mapM_ (known e) (colsInExpr e)
-- | 这个列名在不在定义里 -- \| 这个列名在不在定义里
known _ col known _ col
| col `elem` names = Right () | col `elem` names = Right ()
| otherwise = Left ("CHECK: unknown column: " ++ col) | otherwise = Left ("CHECK: unknown column: " ++ col)
@@ -683,7 +694,7 @@ checkUserName name
| not (all allowed name) = Left "USER: account name may only contain ASCII letters, digits, dot, dash or underscore" | not (all allowed name) = Left "USER: account name may only contain ASCII letters, digits, dot, dash or underscore"
| otherwise = Right () | otherwise = Right ()
where where
-- | 账号名字符规则 -- \| 账号名字符规则
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String)) allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
-- | 建号 / 改口令：名字合法且口令不空 -- | 建号 / 改口令：名字合法且口令不空
@@ -706,7 +717,7 @@ checkRoleName name
| map toLower name `elem` privilegeNames = Left "ROLE: role name must not be a privilege keyword" | map toLower name `elem` privilegeNames = Left "ROLE: role name must not be a privilege keyword"
| otherwise = Right () | otherwise = Right ()
where where
-- | 角色名字符规则 -- \| 角色名字符规则
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String)) allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
-- | 权限关键字（小写，只用于比较） -- | 权限关键字（小写，只用于比较）
@@ -728,12 +739,13 @@ checkGrantObject obj
| not (all allowed obj) = Left "GRANT: object name may only contain ASCII letters, digits, dot, dash or underscore" | not (all allowed obj) = Left "GRANT: object name may only contain ASCII letters, digits, dot, dash or underscore"
| otherwise = Right () | otherwise = Right ()
where where
-- | 授权对象字符规则 -- \| 授权对象字符规则
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String)) allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
-- | 内置 id 列名 -- | 内置 id 列名
idColumn :: String idColumn :: String
idColumn = "id" idColumn = "id"
-- | 把列引用统一为执行期名字（子查询一起解析） -- | 把列引用统一为执行期名字（子查询一起解析）
resolveExpr :: resolveExpr ::
Database -> Database ->
@@ -783,7 +795,7 @@ resolveExpr db env outer f e = case e of
QuantCmp op a sq q -> QuantCmp op <$> resolveExpr db env outer f a <*> resolveSubquery db (env ++ outer) sq <*> pure q QuantCmp op a sq q -> QuantCmp op <$> resolveExpr db env outer f a <*> resolveSubquery db (env ++ outer) sq <*> pure q
_ -> Right e _ -> Right e
where where
-- | 递归处理两个子表达式 -- \| 递归处理两个子表达式
binary ctor a b = ctor <$> resolveExpr db env outer f a <*> resolveExpr db env outer f b binary ctor a b = ctor <$> resolveExpr db env outer f a <*> resolveExpr db env outer f b
-- | 校验具体构造器和参数逻辑类型 -- | 校验具体构造器和参数逻辑类型
@@ -800,7 +812,7 @@ inferConstruction db place env ty@(TypeApply cid parameters) arguments = do
required <- mapM (fmap descriptorId . resolveType builtinTypes) expected required <- mapM (fmap descriptorId . resolveType builtinTypes) expected
if supplied == required then Right (descriptorId descriptor) else Left "composite constructor argument type mismatch" if supplied == required then Right (descriptorId descriptor) else Left "composite constructor argument type mismatch"
where where
-- | 解析非空参数的逻辑类型身份 -- \| 解析非空参数的逻辑类型身份
argumentIdentity InferNull = Left "SQL NULL needs Maybe construction inside a runtime value" argumentIdentity InferNull = Left "SQL NULL needs Maybe construction inside a runtime value"
argumentIdentity (InferType column) = sqlType column argumentIdentity (InferType column) = sqlType column
@@ -829,7 +841,7 @@ outerRefsOf db stmt = nub [c | c <- statementColumns stmt, not (inInner c)]
inner = case checkFrom db [] (statementFrom stmt) of inner = case checkFrom db [] (statementFrom stmt) of
Right env -> env Right env -> env
Left _ -> [] Left _ -> []
-- | 这一列能不能在内层解析到 -- \| 这一列能不能在内层解析到
inInner c = case resolveColumn c inner of inInner c = case resolveColumn c inner of
Right _ -> True Right _ -> True
Left _ -> False Left _ -> False
@@ -837,13 +849,14 @@ outerRefsOf db stmt = nub [c | c <- statementColumns stmt, not (inInner c)]
-- | 一条语句里出现的所有列引用 -- | 一条语句里出现的所有列引用
statementColumns :: Statement -> [String] statementColumns :: Statement -> [String]
statementColumns stmt = case stmt of statementColumns stmt = case stmt of
SelectDistinct q -> statementColumns q
Select cols fromC mWhere groupBy orderBy _ -> Select cols fromC mWhere groupBy orderBy _ ->
cols ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy cols ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy
SelectExpr items fromC mWhere groupBy orderBy _ -> SelectExpr items fromC mWhere groupBy orderBy _ ->
concatMap (colsInExpr . snd) items ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy concatMap (colsInExpr . snd) items ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy
_ -> [] _ -> []
where where
-- | 取 FROM 里出现的列 -- \| 取 FROM 里出现的列
fromColumns FromUnit = [] fromColumns FromUnit = []
fromColumns (FromTable _ _) = [] fromColumns (FromTable _ _) = []
fromColumns (FromSubquery _ _) = [] fromColumns (FromSubquery _ _) = []
@@ -851,12 +864,14 @@ statementColumns stmt = case stmt of
-- | 取语句的 FROM 子句，没有就是 FromUnit -- | 取语句的 FROM 子句，没有就是 FromUnit
statementFrom :: Statement -> FromClause statementFrom :: Statement -> FromClause
statementFrom Select { selectFrom = fromC } = fromC statementFrom (SelectDistinct q) = statementFrom q
statementFrom SelectExpr { selectFrom = fromC } = fromC statementFrom Select{selectFrom = fromC} = fromC
statementFrom SelectExpr{selectFrom = fromC} = fromC
statementFrom _ = FromUnit statementFrom _ = FromUnit
-- | 解析子查询里的 SELECT -- | 解析子查询里的 SELECT
resolveSubqueryStatement :: Database -> [(String, Column)] -> Statement -> Either String Statement resolveSubqueryStatement :: Database -> [(String, Column)] -> Statement -> Either String Statement
resolveSubqueryStatement db outer (SelectDistinct q) = SelectDistinct <$> resolveSubqueryStatement db outer q
resolveSubqueryStatement db outer q@Select{} = resolveQuery db outer q (map (\c -> (c, Col c)) (selectCols q)) resolveSubqueryStatement db outer q@Select{} = resolveQuery db outer q (map (\c -> (c, Col c)) (selectCols q))
resolveSubqueryStatement db outer q@SelectExpr{} = resolveQuery db outer q (selectItems q) resolveSubqueryStatement db outer q@SelectExpr{} = resolveQuery db outer q (selectItems q)
resolveSubqueryStatement _ _ _ = Left "subquery must be a SELECT" resolveSubqueryStatement _ _ _ = Left "subquery must be a SELECT"
@@ -864,6 +879,7 @@ resolveSubqueryStatement _ _ _ = Left "subquery must be a SELECT"
-- | 列名解析入口：查询走连接解析，改数据的语句按本表列解析 -- | 列名解析入口：查询走连接解析，改数据的语句按本表列解析
resolveStatement :: Database -> Statement -> Either String Statement resolveStatement :: Database -> Statement -> Either String Statement
resolveStatement db q = case q of resolveStatement db q = case q of
SelectDistinct inner -> SelectDistinct <$> resolveStatement db inner
Select{} -> resolveSelect db q Select{} -> resolveSelect db q
SelectExpr{} -> resolveSelect db q SelectExpr{} -> resolveSelect db q
Update tbl assigns mWhere -> do Update tbl assigns mWhere -> do
@@ -923,34 +939,42 @@ resolveQuery :: Database -> [(String, Column)] -> Statement -> [(String, Expr)]
resolveQuery db outer q items = do resolveQuery db outer q items = do
source <- resolveSource db outer (selectFrom q) source <- resolveSource db outer (selectFrom q)
env <- checkFrom db outer source env <- checkFrom db outer source
-- | 单表无别名时把裸列名换成物理列名；带别名时表头保留别名限定名 -- \| 单表无别名时把裸列名换成物理列名；带别名时表头保留别名限定名
let physical c = case selectFrom q of let physical c = case selectFrom q of
FromTable Nothing tbl -> unqualify (Just (deriveQualifier Nothing tbl)) c FromTable Nothing tbl -> unqualify (Just (deriveQualifier Nothing tbl)) c
_ -> c _ -> c
-- | 查一列并换算成物理名 -- \| 查一列并换算成物理名
name place c name place c
| length (filter (== '.') c) == 2, not (qualifiedReference c) = Left (missingColumn place c env) | length (filter (== '.') c) == 2, not (qualifiedReference c) = Left (missingColumn place c env)
| otherwise = physical . fst <$> scopedColumnAt place env outer c | otherwise = physical . fst <$> scopedColumnAt place env outer c
-- | 库限定列只引用来源中的真实表名 -- \| 库限定列只引用来源中的真实表名
qualifiedReference c = qualifiedReferenceIn db (selectFrom q) c qualifiedReference c =
|| (any (\(key, _) -> length (filter (== '.') key) == 1 && ("." ++ key) `isSuffixOf` c) outer qualifiedReferenceIn db (selectFrom q) c
&& (not (any (elem '.' . fst) db) || any (\(table, _) -> (table ++ ".") `isPrefixOf` c) db)) || ( any (\(key, _) -> length (filter (== '.') key) == 1 && ("." ++ key) `isSuffixOf` c) outer
-- | 换算一个投影项：带别名时保留别名做输出列名 && (not (any (elem '.' . fst) db) || any (\(table, _) -> (table ++ ".") `isPrefixOf` c) db)
)
-- \| 换算一个投影项：带别名时保留别名做输出列名
output (label, Col c) = do output (label, Col c) = do
k <- name "SELECT" c k <- name "SELECT" c
pure (if label == c then (k, Col k) else (label, Col k)) pure (if label == c then (k, Col k) else (label, Col k))
output (label, e) = do output (label, e) = do
e' <- resolveExpr db env outer (name "SELECT") e e' <- resolveExpr db env outer (name "SELECT") e
pure (label, e') pure (label, e')
projected <- if items == [("*", Col "*")] projected <-
then if null env && selectFrom q == FromUnit if items == [("*", Col "*")]
then
if null env && selectFrom q == FromUnit
then Left "SELECT: * requires FROM" then Left "SELECT: * requires FROM"
else Right [(physical c, Col (physical c)) | (c, _) <- env] else Right [(physical c, Col (physical c)) | (c, _) <- env]
else mapM output items else mapM output items
condition <- mapM (resolveExpr db env outer (name "WHERE")) (selectWhere q) condition <- mapM (resolveExpr db env outer (name "WHERE")) (selectWhere q)
grouping <- mapM (name "GROUP BY") (selectGroupBy q) grouping <- mapM (name "GROUP BY") (selectGroupBy q)
-- | 优先解析唯一投影别名，再查来源列 -- \| 优先解析唯一投影别名，再查来源列
let orderName c = case [label | (label, _) <- projected, label == c, qIsExpression] of let orderName c
| "$position" `isPrefixOf` c = case reads (drop 9 c) of
[(n, "")] | n > 0, n <= length projected -> Right (fst (projected !! (n - 1)))
_ -> Left "ORDER BY: position is outside the select list"
| otherwise = case [label | (label, _) <- projected, label == c, qIsExpression] of
[] -> name "ORDER BY" c [] -> name "ORDER BY" c
[label] -> Right label [label] -> Right label
_ -> Left ("ORDER BY: ambiguous projection alias: " ++ c) _ -> Left ("ORDER BY: ambiguous projection alias: " ++ c)
@@ -994,7 +1018,7 @@ resolveFrom db outer (FromJoin kind left right cond) = do
l <- resolveFrom db outer left l <- resolveFrom db outer left
r <- resolveFrom db outer right r <- resolveFrom db outer right
env <- checkFrom db outer (FromJoin kind l r cond) env <- checkFrom db outer (FromJoin kind l r cond)
-- | 连接条件也拒绝不存在的库前缀 -- \| 连接条件也拒绝不存在的库前缀
let joinName column let joinName column
| length (filter (== '.') column) == 2, not (qualifiedReferenceIn db (FromJoin kind left right cond) column) = Left (missingColumn "ON" column env) | length (filter (== '.') column) == 2, not (qualifiedReferenceIn db (FromJoin kind left right cond) column) = Left (missingColumn "ON" column env)
| otherwise = fst <$> columnAt "ON" env column | otherwise = fst <$> columnAt "ON" env column
@@ -1003,7 +1027,8 @@ resolveFrom db outer (FromJoin kind left right cond) = do
-- | 派生表的输出列名：列取裸名，表达式保留原标签 -- | 派生表的输出列名：列取裸名，表达式保留原标签
nameDerivedColumns :: Statement -> Either String Statement nameDerivedColumns :: Statement -> Either String Statement
nameDerivedColumns q@Select { selectCols = cols, selectFrom = fromC } = do nameDerivedColumns (SelectDistinct q) = SelectDistinct <$> nameDerivedColumns q
nameDerivedColumns q@Select{selectCols = cols, selectFrom = fromC} = do
let names = map lastTablePart cols let names = map lastTablePart cols
checkDerivedNames names checkDerivedNames names
Right Right
@@ -1015,10 +1040,10 @@ nameDerivedColumns q@Select { selectCols = cols, selectFrom = fromC } = do
(selectOrderBy q) (selectOrderBy q)
(selectLimit q) (selectLimit q)
) )
nameDerivedColumns q@SelectExpr { selectItems = items } = do nameDerivedColumns q@SelectExpr{selectItems = items} = do
let names = map derivedItemName items let names = map derivedItemName items
checkDerivedNames names checkDerivedNames names
Right q { selectItems = zip names (map snd items) } Right q{selectItems = zip names (map snd items)}
nameDerivedColumns _ = Left "FROM: subquery must be a SELECT" nameDerivedColumns _ = Left "FROM: subquery must be a SELECT"
-- | 派生表里一个投影项的输出名：没写别名的列取裸名 -- | 派生表里一个投影项的输出名：没写别名的列取裸名
@@ -1032,7 +1057,7 @@ derivedItemName (label, _) = label
checkDerivedNames :: [String] -> Either String () checkDerivedNames :: [String] -> Either String ()
checkDerivedNames = go [] checkDerivedNames = go []
where where
-- | 逐个查有没有见过 -- \| 逐个查有没有见过
go _ [] = Right () go _ [] = Right ()
go seen (n : ns) go seen (n : ns)
| n `elem` seen = Left ("FROM: duplicate column name in subquery: " ++ n) | n `elem` seen = Left ("FROM: duplicate column name in subquery: " ++ n)
chusql-core/engine/src/ChuSQL/Core/Engine/Storage.hs
+11 -4
View File
@@ -13,15 +13,22 @@ data IndexResult
deriving (Show, Eq) deriving (Show, Eq)
class (Monad m) => MonadStorage m where class (Monad m) => MonadStorage m where
-- \| 在稳定存储视图中执行绑定读取
withSchemaGuard :: Database -> m (Either String a) -> m (Either String a)
withSchemaGuard _ _ = pure (Left "bound schema guards are unavailable in this storage")
-- \| 建一个库 -- \| 建一个库
createDatabase :: String -> m (Either String ()) createDatabase :: String -> m (Either String ())
createDatabase _ = pure (Left "database management is unavailable in this storage") createDatabase _ = pure (Left "database management is unavailable in this storage")
-- \| 删一个库 -- \| 删一个库
dropDatabase :: String -> m (Either String ()) dropDatabase :: String -> m (Either String ())
dropDatabase _ = pure (Left "database management is unavailable in this storage") dropDatabase _ = pure (Left "database management is unavailable in this storage")
-- \| 切当前库 -- \| 切当前库
useDatabase :: String -> m (Either String ()) useDatabase :: String -> m (Either String ())
useDatabase _ = pure (Left "database selection is unavailable in this storage") useDatabase _ = pure (Left "database selection is unavailable in this storage")
-- \| 列出现有库 -- \| 列出现有库
listDatabases :: m (Either String [String]) listDatabases :: m (Either String [String])
listDatabases = pure (Right []) listDatabases = pure (Right [])
@@ -50,7 +57,7 @@ class (Monad m) => MonadStorage m where
insertMany :: String -> [Row] -> m (Either String ()) insertMany :: String -> [Row] -> m (Either String ())
insertMany t = go insertMany t = go
where where
-- | 逐行插入，出错就停 -- \| 逐行插入，出错就停
go [] = pure (Right ()) go [] = pure (Right ())
go (r : rs) = do go (r : rs) = do
x <- insert t r x <- insert t r
@@ -66,7 +73,7 @@ class (Monad m) => MonadStorage m where
Left e -> pure (Left e) Left e -> pure (Left e)
Right rs -> replaceAll t [r | r <- rs, not (doomed r)] Right rs -> replaceAll t [r | r <- rs, not (doomed r)]
where where
-- | 这一行要不要删 -- \| 这一行要不要删
doomed r = case lookup "id" r of doomed r = case lookup "id" r of
Just (VInt k) -> k `elem` ks Just (VInt k) -> k `elem` ks
_ -> False _ -> False
@@ -114,5 +121,5 @@ class (Monad m) => MonadStorage m where
schema :: m (Either String Database) schema :: m (Either String Database)
schema = fmap (fmap (map withoutRows)) snapshot schema = fmap (fmap (map withoutRows)) snapshot
where where
-- | 表定义去掉所有行 -- \| 表定义去掉所有行
withoutRows (name, t) = (name, t {tableRows = []}) withoutRows (name, t) = (name, t{tableRows = []})
chusql-core/engine/src/ChuSQL/Core/Engine/Storage/FFI.hs
+12 -6
View File
@@ -65,7 +65,7 @@ openStorage path = onStorageThread $ mask_ $ do
| ptr == nullPtr -> Left <$> lastError | ptr == nullPtr -> Left <$> lastError
| otherwise -> pure (Right (StorageHandle ptr)) | otherwise -> pure (Right (StorageHandle ptr))
where where
-- | 没有路径就传空指针，否则转成 C 字符串 -- \| 没有路径就传空指针，否则转成 C 字符串
withCStringOrNull Nothing f = f nullPtr withCStringOrNull Nothing f = f nullPtr
withCStringOrNull (Just s) f = BS.useAsCString (TE.encodeUtf8 (T.pack s)) f withCStringOrNull (Just s) f = BS.useAsCString (TE.encodeUtf8 (T.pack s)) f
@@ -78,16 +78,22 @@ storageRequest :: StorageHandle -> BS.ByteString -> IO (Either String BS.ByteStr
storageRequest (StorageHandle handle) = callStorage (cStorageRequest handle) storageRequest (StorageHandle handle) = callStorage (cStorageRequest handle)
-- | 调用存储并复制和归还响应缓冲 -- | 调用存储并复制和归还响应缓冲
callStorage :: (Ptr CChar -> CSize -> Ptr (Ptr CChar) -> Ptr CSize -> IO CInt) callStorage ::
-> BS.ByteString -> IO (Either String BS.ByteString) (Ptr CChar -> CSize -> Ptr (Ptr CChar) -> Ptr CSize -> IO CInt) ->
callStorage call payload = onStorageThread $ mask_ $ BS.ByteString ->
IO (Either String BS.ByteString)
callStorage call payload = onStorageThread $
mask_ $
BS.useAsCStringLen payload $ \(input, size) -> BS.useAsCStringLen payload $ \(input, size) ->
alloca $ \outRef -> alloca $ \lenRef -> do alloca $ \outRef -> alloca $ \lenRef -> do
code <- call input (fromIntegral size) outRef lenRef code <- call input (fromIntegral size) outRef lenRef
if code /= 0 then Left <$> lastError else do if code /= 0
then Left <$> lastError
else do
output <- peek outRef output <- peek outRef
len <- peek lenRef len <- peek lenRef
bytes <- BS.packCStringLen (output, fromIntegral len) bytes <-
BS.packCStringLen (output, fromIntegral len)
`finally` cStorageFree output len `finally` cStorageFree output len
pure (Right bytes) pure (Right bytes)
chusql-core/engine/src/ChuSQL/Core/Engine/Storage/IPC.hs
+64 -16
View File
@@ -30,15 +30,17 @@ module ChuSQL.Core.Engine.Storage.IPC (
doListCatalog, doListCatalog,
) where ) where
import ChuSQL.Core.Model
import ChuSQL.Core.Protocol (Account (..), Request (..), Response (..), SchemaColumn (..), TableInfo (..))
import ChuSQL.Core.Engine.Parallel (parallelShardLimit, poolRun, workerPool) import ChuSQL.Core.Engine.Parallel (parallelShardLimit, poolRun, workerPool)
import ChuSQL.Core.Engine.Storage import ChuSQL.Core.Engine.Storage
import ChuSQL.Core.Engine.Storage.FFI (closeStorage, openStorage, storageRequest) import ChuSQL.Core.Engine.Storage.FFI (closeStorage, openStorage, storageRequest)
import ChuSQL.Core.Model
import ChuSQL.Core.Protocol (Account (..), Request (..), Response (..), SchemaColumn (..), TableInfo (..))
import Control.Concurrent (getNumCapabilities) import Control.Concurrent (getNumCapabilities)
import Control.Concurrent.MVar (MVar, modifyMVar_, newMVar, putMVar, readMVar, takeMVar, withMVar) import Control.Concurrent.MVar (MVar, modifyMVar_, newMVar, putMVar, readMVar, takeMVar, withMVar)
import Control.Exception (bracket, mask_) import Control.Exception (bracket, mask_)
import Data.Aeson (eitherDecodeStrict, encode) import Data.Aeson (eitherDecodeStrict, encode)
import Data.Aeson qualified as A
import Data.Aeson.KeyMap qualified as KM
import Data.ByteString qualified as BS import Data.ByteString qualified as BS
import Data.ByteString.Lazy qualified as BL import Data.ByteString.Lazy qualified as BL
import Data.Maybe (fromMaybe) import Data.Maybe (fromMaybe)
@@ -49,6 +51,7 @@ import System.IO.Unsafe (unsafePerformIO)
-- | 一条会话的存储环境：当前库名随会话走，不共享 -- | 一条会话的存储环境：当前库名随会话走，不共享
data Env = Env data Env = Env
{ envDatabase :: Maybe String { envDatabase :: Maybe String
, envSchemaGuard :: Maybe Database
} }
newtype IPCStorage a = IPCStorage newtype IPCStorage a = IPCStorage
@@ -57,11 +60,11 @@ newtype IPCStorage a = IPCStorage
-- | 按库名造一个会话环境 -- | 按库名造一个会话环境
envForDatabase :: Maybe String -> Env envForDatabase :: Maybe String -> Env
envForDatabase = Env envForDatabase name = Env name Nothing
-- | 进程默认环境：读全局库名，给测试与 CLI 用 -- | 进程默认环境：读全局库名，给测试与 CLI 用
defaultEnv :: IO Env defaultEnv :: IO Env
defaultEnv = Env <$> readMVar databaseRef defaultEnv = envForDatabase <$> readMVar databaseRef
-- | 用进程默认环境跑（兼容既有调用点） -- | 用进程默认环境跑（兼容既有调用点）
runIPCStorage :: IPCStorage a -> IO a runIPCStorage :: IPCStorage a -> IO a
@@ -86,11 +89,13 @@ data StorageLink = StorageLink
} }
{-# NOINLINE linkRef #-} {-# NOINLINE linkRef #-}
-- | 进程全局的存储链路 -- | 进程全局的存储链路
linkRef :: MVar (Maybe StorageLink) linkRef :: MVar (Maybe StorageLink)
linkRef = unsafePerformIO (newMVar Nothing) linkRef = unsafePerformIO (newMVar Nothing)
{-# NOINLINE linkUsers #-} {-# NOINLINE linkUsers #-}
-- | 在途请求计数与空闲信号 -- | 在途请求计数与空闲信号
linkUsers :: (MVar Int, MVar ()) linkUsers :: (MVar Int, MVar ())
linkUsers = unsafePerformIO $ (,) <$> newMVar 0 <*> newMVar () linkUsers = unsafePerformIO $ (,) <$> newMVar 0 <*> newMVar ()
@@ -100,7 +105,7 @@ withStorageLink :: (Maybe StorageLink -> IO a) -> IO a
withStorageLink = bracket enter leave withStorageLink = bracket enter leave
where where
(users, idle) = linkUsers (users, idle) = linkUsers
-- | 登记当前链路的在途请求 -- \| 登记当前链路的在途请求
enter = withMVar linkRef $ \current -> do enter = withMVar linkRef $ \current -> do
case current of case current of
Nothing -> pure () Nothing -> pure ()
@@ -108,7 +113,7 @@ withStorageLink = bracket enter leave
if count == 0 then takeMVar idle else pure () if count == 0 then takeMVar idle else pure ()
pure (count + 1) pure (count + 1)
pure current pure current
-- | 最后一个请求结束后发出空闲信号 -- \| 最后一个请求结束后发出空闲信号
leave Nothing = pure () leave Nothing = pure ()
leave (Just _) = mask_ $ modifyMVar_ users $ \count -> do leave (Just _) = mask_ $ modifyMVar_ users $ \count -> do
if count == 1 then putMVar idle () else pure () if count == 1 then putMVar idle () else pure ()
@@ -122,7 +127,7 @@ setStorageLink link = modifyMVar_ linkRef $ \_ -> withMVar (snd linkUsers) (cons
localStorageLink :: Maybe FilePath -> IO (Either String StorageLink) localStorageLink :: Maybe FilePath -> IO (Either String StorageLink)
localStorageLink path = fmap (fmap toLink) (openStorage path) localStorageLink path = fmap (fmap toLink) (openStorage path)
where where
-- | 把句柄包成一条链路 -- \| 把句柄包成一条链路
toLink handle = toLink handle =
StorageLink StorageLink
{ slRequest = storageRequest handle { slRequest = storageRequest handle
@@ -157,6 +162,7 @@ sendRawRequest payload = withStorageLink $ \current -> do
Just link -> slRequest link payload Just link -> slRequest link payload
{-# NOINLINE databaseRef #-} {-# NOINLINE databaseRef #-}
-- | 进程全局的当前库名 -- | 进程全局的当前库名
databaseRef :: MVar (Maybe String) databaseRef :: MVar (Maybe String)
databaseRef = unsafePerformIO (newMVar Nothing) databaseRef = unsafePerformIO (newMVar Nothing)
@@ -169,7 +175,6 @@ setDatabaseName name = modifyMVar_ databaseRef (const (pure name))
getDatabaseName :: IO (Maybe String) getDatabaseName :: IO (Maybe String)
getDatabaseName = readMVar databaseRef getDatabaseName = readMVar databaseRef
-- | 给语句类请求盖上当前库名 -- | 给语句类请求盖上当前库名
stampDatabase :: Maybe String -> Request -> Request stampDatabase :: Maybe String -> Request -> Request
stampDatabase Nothing req = req stampDatabase Nothing req = req
@@ -192,13 +197,43 @@ globalRequest _ = False
-- | 发请求并把响应翻成 Either -- | 发请求并把响应翻成 Either
ask :: Env -> String -> Request -> (Response -> Maybe a) -> IO (Either String a) ask :: Env -> String -> Request -> (Response -> Maybe a) -> IO (Either String a)
ask env what req recognize = do ask env what req recognize = do
resp <- sendRequest (stampDatabase (envDatabase env) req) resp <- case envSchemaGuard env of
Nothing -> sendRequest (stampDatabase (envDatabase env) req)
Just dependencies -> guardedRequest env dependencies req
pure $ case resp of pure $ case resp of
RespError e -> Left e RespError e -> Left e
other -> case recognize other of other -> case recognize other of
Just a -> Right a Just a -> Right a
Nothing -> Left ("unexpected response to " ++ what) Nothing -> Left ("unexpected response to " ++ what)
-- | 发送依赖守卫并要求存储确认执行
guardedRequest :: Env -> Database -> Request -> IO Response
guardedRequest env dependencies req = do
let payload = case A.toJSON (stampDatabase (envDatabase env) req) of
A.Object fields -> A.Object (KM.insert "schema_guard" (A.toJSON (map guardOf dependencies)) fields)
other -> other
result <- sendRawRequest (BL.toStrict (encode payload))
pure $ case result >>= eitherDecodeStrict of
Left err -> RespError err
Right (A.Object fields)
| KM.lookup "status" fields == Just (A.String "guarded")
, Just response <- KM.lookup "response" fields -> case A.fromJSON response of
A.Success decoded -> decoded
A.Error err -> RespError ("decode error: " ++ err)
Right value -> case A.fromJSON value of
A.Success (RespError err) -> RespError err
_ -> RespError "storage did not confirm bound schema guard"
where
-- \| 编码依赖身份、版本与完整列定义
guardOf (name, table) =
A.object
[ "table" A..= name
, "database_id" A..= fmap (\(TableIdentity database _) -> show database) (tableIdentity table)
, "object_id" A..= fmap (\(TableIdentity _ object) -> show object) (tableIdentity table)
, "schema_version" A..= fmap show (tableSchemaVersion table)
, "columns" A..= map toWire (tableCols table)
]
-- | 只认"成了"（多数写操作都这样） -- | 只认"成了"（多数写操作都这样）
okOnly :: Response -> Maybe () okOnly :: Response -> Maybe ()
okOnly RespOk = Just () okOnly RespOk = Just ()
@@ -310,18 +345,27 @@ toWire (n, c) =
} }
instance MonadStorage IPCStorage where instance MonadStorage IPCStorage where
-- \| 将依赖守卫限定于本次执行环境
withSchemaGuard dependencies action = IPCStorage $ \env ->
runIPCStorageIn action env{envSchemaGuard = Just dependencies}
-- \| 建一个库 -- \| 建一个库
createDatabase name = IPCStorage $ \env -> ask env "create_database" (ReqDatabase "create_database" name) okOnly createDatabase name = IPCStorage $ \env -> ask env "create_database" (ReqDatabase "create_database" name) okOnly
-- \| 删一个库 -- \| 删一个库
dropDatabase name = IPCStorage $ \env -> ask env "drop_database" (ReqDatabase "drop_database" name) okOnly dropDatabase name = IPCStorage $ \env -> ask env "drop_database" (ReqDatabase "drop_database" name) okOnly
-- \| 切当前库 -- \| 切当前库
useDatabase name = IPCStorage $ \env -> ask env "use_database" (ReqDatabase "use_database" name) okOnly useDatabase name = IPCStorage $ \env -> ask env "use_database" (ReqDatabase "use_database" name) okOnly
-- \| 列出现有库 -- \| 列出现有库
listDatabases = IPCStorage $ \env -> ask env "list_databases" (ReqDatabase "list_databases" "") $ \resp -> case resp of listDatabases = IPCStorage $ \env -> ask env "list_databases" (ReqDatabase "list_databases" "") $ \resp -> case resp of
RespTables names -> Just names RespTables names -> Just names
_ -> Nothing _ -> Nothing
-- \| 发 Scan -- \| 发 Scan
scan t = IPCStorage (\env -> doScan env t) scan t = IPCStorage (\env -> doScan env t)
-- \| 只取指定列 -- \| 只取指定列
scanColumns t cols = IPCStorage (\env -> doScanColumns env t cols) scanColumns t cols = IPCStorage (\env -> doScanColumns env t cols)
@@ -381,13 +425,17 @@ instance MonadStorage IPCStorage where
Left err -> pure (Left err) Left err -> pure (Left err)
Right xs -> fmap sequence (mapM (loadEntry env) xs) Right xs -> fmap sequence (mapM (loadEntry env) xs)
where where
-- | 读取表数据并保留扫描错误 -- \| 读取表数据并保留扫描错误
loadEntry env info = do loadEntry env info = do
rows <- doScan env (tiTable info) rows <- doScan env (tiTable info)
pure $ fmap (\rs -> pure $
fmap
( \rs ->
( tiTable info ( tiTable info
, Table (tiTable info) (schemaToColumns (tiColumns info)) rs (Just (metaOf info)) , Table (tiTable info) (schemaToColumns (tiColumns info)) rs (Just (metaOf info)) (tiIdentity info) (tiSchemaVersion info)
)) rows )
)
rows
-- \| 只问数据字典要结构，设了当前库只留它的表 -- \| 只问数据字典要结构，设了当前库只留它的表
schema = IPCStorage $ \env -> do schema = IPCStorage $ \env -> do
@@ -409,6 +457,6 @@ visibleTables current info
else [] else []
| otherwise = [bare info] | otherwise = [bare info]
where where
named = info {tiTable = unqualify current (tiTable info)} named = info{tiTable = unqualify current (tiTable info)}
-- | 组装成不带行的表 -- \| 组装成不带行的表
bare i = (tiTable i, Table (tiTable i) (schemaToColumns (tiColumns i)) [] (Just (metaOf i))) bare i = (tiTable i, Table (tiTable i) (schemaToColumns (tiColumns i)) [] (Just (metaOf i)) (tiIdentity i) (tiSchemaVersion i))
chusql-core/engine/src/ChuSQL/Core/Engine/Storage/Memory.hs
+26 -9
View File
@@ -1,8 +1,9 @@
module ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage)) where module ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage)) where
import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Storage import ChuSQL.Core.Engine.Storage
import qualified Data.HashSet as HS import ChuSQL.Core.Model
import Data.HashSet qualified as HS
import Data.Word (Word64)
-- 内存实现：Database 上的状态，错误通道是 Either String。 -- 内存实现：Database 上的状态，错误通道是 Either String。
@@ -26,6 +27,9 @@ instance Monad MemoryStorage where
runMemoryStorage (k a) db' runMemoryStorage (k a) db'
instance MonadStorage MemoryStorage where instance MonadStorage MemoryStorage where
-- \| 纯状态视图在求值期间不受外部修改
withSchemaGuard _ action = action
-- \| 全表扫描 -- \| 全表扫描
scan t = MemoryStorage $ \db -> do scan t = MemoryStorage $ \db -> do
tbl <- lookupTable db t tbl <- lookupTable db t
@@ -45,7 +49,7 @@ instance MonadStorage MemoryStorage where
createTable name cols = MemoryStorage $ \db -> createTable name cols = MemoryStorage $ \db ->
if any ((== name) . fst) db if any ((== name) . fst) db
then Right (Left ("table already exists: " ++ name), db) then Right (Left ("table already exists: " ++ name), db)
else Right (Right (), db ++ [(name, Table name cols [] Nothing)]) else Right (Right (), db ++ [(name, Table name cols [] Nothing Nothing (Just 1))])
-- \| 删表；不存在报错 -- \| 删表；不存在报错
dropTable name = MemoryStorage $ \db -> dropTable name = MemoryStorage $ \db ->
@@ -60,7 +64,8 @@ instance MonadStorage MemoryStorage where
Just tbl -> Just tbl ->
if col `notElem` map fst (tableCols tbl) if col `notElem` map fst (tableCols tbl)
then Right (Left ("unknown column: " ++ col), db) then Right (Left ("unknown column: " ++ col), db)
else else do
version <- nextSchemaVersion tbl
Right Right
( Right () ( Right ()
, replaceTable , replaceTable
@@ -68,6 +73,7 @@ instance MonadStorage MemoryStorage where
tbl tbl
{ tableCols = filter ((/= col) . fst) (tableCols tbl) { tableCols = filter ((/= col) . fst) (tableCols tbl)
, tableRows = map (filter ((/= col) . fst)) (tableRows tbl) , tableRows = map (filter ((/= col) . fst)) (tableRows tbl)
, tableSchemaVersion = Just version
} }
db db
) )
@@ -78,9 +84,9 @@ instance MonadStorage MemoryStorage where
-- \| 返回实时行数与去重统计的无数据结构 -- \| 返回实时行数与去重统计的无数据结构
schema = MemoryStorage $ \db -> Right (Right (map describe db), db) schema = MemoryStorage $ \db -> Right (Right (map describe db), db)
where where
-- | 将存储数据汇总为结构统计 -- \| 将存储数据汇总为结构统计
describe (name, table) = describe (name, table) =
(name, table {tableRows = [], tableMeta = Just metadata}) (name, table{tableRows = [], tableMeta = Just metadata})
where where
metadata = TableMeta (length (tableRows table)) distinct [] [] metadata = TableMeta (length (tableRows table)) distinct [] []
distinct = [columnStatistics column (tableRows table) | (column, _) <- tableCols table] distinct = [columnStatistics column (tableRows table) | (column, _) <- tableCols table]
@@ -88,13 +94,25 @@ instance MonadStorage MemoryStorage where
-- \| 改表结构：列定义与行一起换 -- \| 改表结构：列定义与行一起换
replaceSchema name cols rows = MemoryStorage $ \db -> case lookup name db of replaceSchema name cols rows = MemoryStorage $ \db -> case lookup name db of
Nothing -> Right (Left ("unknown table: " ++ name), db) Nothing -> Right (Left ("unknown table: " ++ name), db)
Just tbl -> Right (Right (), replaceTable name tbl{tableCols = cols, tableRows = rows} db) Just tbl -> do
version <-
if tableCols tbl == cols
then Right (tableSchemaVersion tbl)
else Just <$> nextSchemaVersion tbl
Right (Right (), replaceTable name tbl{tableCols = cols, tableRows = rows, tableSchemaVersion = version} db)
-- | 递增结构版本并拒绝耗尽
nextSchemaVersion :: Table -> Either String Word64
nextSchemaVersion table = case tableSchemaVersion table of
Just version | version == maxBound -> Left "catalog schema version exhausted"
Just version -> Right (version + 1)
Nothing -> Right 1
-- | 对列值去重并限制统计内存 -- | 对列值去重并限制统计内存
columnStatistics :: String -> [Row] -> (String, Int, Bool) columnStatistics :: String -> [Row] -> (String, Int, Bool)
columnStatistics column rows = go rows HS.empty columnStatistics column rows = go rows HS.empty
where where
-- | 累积规范化值，达到上限后停止 -- \| 累积规范化值，达到上限后停止
go remaining values go remaining values
| HS.size values >= 4096 = (column, 4096, True) | HS.size values >= 4096 = (column, 4096, True)
| otherwise = case remaining of | otherwise = case remaining of
@@ -105,4 +123,3 @@ columnStatistics column rows = go rows HS.empty
replaceTable :: String -> Table -> Database -> Database replaceTable :: String -> Table -> Database -> Database
replaceTable name table = replaceTable name table =
map (\(n, t) -> if n == name then (n, table) else (n, t)) map (\(n, t) -> if n == name then (n, table) else (n, t))
chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/AST.hs
+26 -1
View File
@@ -1,4 +1,4 @@
module ChuSQL.Core.Engine.Syntax.AST (Statement (..), FromClause (..), JoinKind (..), Expr (..), Subquery (..), Quantifier (..), CompareOp (..), SortDir (..), makeSelect) where module ChuSQL.Core.Engine.Syntax.AST (Statement (..), FromClause (..), JoinKind (..), Expr (..), Subquery (..), Quantifier (..), CompareOp (..), SortDir (..), makeSelect, distinctOrdering) where
import ChuSQL.Core.Model (Column (..), ColumnType, Value (..)) import ChuSQL.Core.Model (Column (..), ColumnType, Value (..))
import ChuSQL.Core.Engine.Runtime.Types (TypeId, TypeExpr, RuntimeValue) import ChuSQL.Core.Engine.Runtime.Types (TypeId, TypeExpr, RuntimeValue)
@@ -23,6 +23,7 @@ data Statement
, selectOrderBy :: [(String, SortDir)] , selectOrderBy :: [(String, SortDir)]
, selectLimit :: Maybe Int , selectLimit :: Maybe Int
} }
| SelectDistinct Statement
| Insert String [String] [[Expr]] | Insert String [String] [[Expr]]
| Delete String (Maybe Expr) | Delete String (Maybe Expr)
| Update String [(String, Expr)] (Maybe Expr) | Update String [(String, Expr)] (Maybe Expr)
@@ -42,6 +43,7 @@ data Statement
| CreateUser String String | CreateUser String String
| AlterUser String String | AlterUser String String
| AlterIdentity String [(String, Bool)] | AlterIdentity String [(String, Bool)]
| ShowGrants
| ShowRoles | ShowRoles
| DropUser String | DropUser String
-- | 新角色：权限的载体，用户靠成员资格继承它的权限 -- | 新角色：权限的载体，用户靠成员资格继承它的权限
@@ -52,6 +54,7 @@ data Statement
| GrantPrivileges [String] String String Bool | GrantPrivileges [String] String String Bool
-- | REVOKE 权限 ON 表 FROM 角色 -- | REVOKE 权限 ON 表 FROM 角色
| RevokePrivileges [String] String String | RevokePrivileges [String] String String
| RevokeGrantOption [String] String String
-- | GRANT 角色 TO 用户：给用户加角色 -- | GRANT 角色 TO 用户：给用户加角色
| GrantRole String [String] | GrantRole String [String]
-- | REVOKE <角色> FROM <用户>[, ...] -- | REVOKE <角色> FROM <用户>[, ...]
@@ -114,6 +117,8 @@ data CompareOp
-- | 条件表达式：列、字面量、算术、比较、逻辑、聚合与子查询 -- | 条件表达式：列、字面量、算术、比较、逻辑、聚合与子查询
data Expr data Expr
= Col String = Col String
| ColumnSlot Int String
| OuterColumnSlot Int Int String
| FunctionCall String [Expr] | FunctionCall String [Expr]
| BoundFunction FunctionId TypeId [Expr] | BoundFunction FunctionId TypeId [Expr]
| Construct TypeExpr [Expr] | Construct TypeExpr [Expr]
@@ -172,3 +177,23 @@ makeSelect cols tbl w =
, selectOrderBy = [] , selectOrderBy = []
, selectLimit = Nothing , selectLimit = Nothing
} }
-- | 将去重排序映射到输出列
distinctOrdering :: Statement -> Either String [(String, SortDir)]
distinctOrdering q@Select{} = resolveDistinctOrdering q
distinctOrdering q@SelectExpr{} = resolveDistinctOrdering q
distinctOrdering _ = Left "DISTINCT requires SELECT"
-- | 解析去重输出的排序列
resolveDistinctOrdering :: Statement -> Either String [(String, SortDir)]
resolveDistinctOrdering q = mapM resolve (selectOrderBy q)
where
items = case q of
Select { selectCols = cols } -> [(c, Col c) | c <- cols]
SelectExpr { selectItems = projected } -> projected
_ -> []
-- | 排序键必须出现在去重结果中
resolve (column, direction) = case [label | (label, e) <- items, label == column || e == Col column] of
[label] -> Right (label, direction)
[] -> Left "DISTINCT: ORDER BY column must appear in the select list"
_ -> Left "DISTINCT: ambiguous ORDER BY column"
chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs
+9 -4
View File
@@ -262,7 +262,7 @@ literalValue =
-- | 一个排序列 -- | 一个排序列
orderItem :: Parser (String, SortDir) orderItem :: Parser (String, SortDir)
orderItem = do orderItem = do
col <- qualifiedName col <- qualifiedName <|> (("$position" ++) . show <$> integer)
dir <- sortDir dir <- sortDir
return (col, dir) return (col, dir)
@@ -590,14 +590,15 @@ grantStatement = do
revokeStatement :: Parser Statement revokeStatement :: Parser Statement
revokeStatement = do revokeStatement = do
keyword "revoke" keyword "revoke"
optionOnly <- fromMaybe False <$> optional (True <$ (keyword "grant" *> keyword "option" *> keyword "for"))
choice choice
[ try $ do [ try $ do
privs <- sepBy1 privilegeName (symbol ",") privs <- sepBy1 privilegeName (symbol ",")
keyword "on" keyword "on"
obj <- grantObject obj <- grantObject
keyword "from" keyword "from"
RevokePrivileges privs obj <$> userName (if optionOnly then RevokeGrantOption else RevokePrivileges) privs obj <$> userName
, RevokeRole <$> userName <*> (keyword "from" *> sepBy1 userName (symbol ",")) , if optionOnly then fail "GRANT OPTION FOR requires object privileges" else RevokeRole <$> userName <*> (keyword "from" *> sepBy1 userName (symbol ","))
] ]
-- | 建表时的一列：类型后面跟若干约束 -- | 建表时的一列：类型后面跟若干约束
@@ -689,13 +690,14 @@ decimalArgs = do
selectStatement :: Parser Statement selectStatement :: Parser Statement
selectStatement = do selectStatement = do
keyword "select" keyword "select"
distinct <- fromMaybe False <$> optional ((True <$ keyword "distinct") <|> (False <$ keyword "all"))
cols <- selectList cols <- selectList
fromC <- fromMaybe FromUnit <$> optional (keyword "from" *> fromClause) fromC <- fromMaybe FromUnit <$> optional (keyword "from" *> fromClause)
mWhere <- optional (keyword "where" *> expr) mWhere <- optional (keyword "where" *> expr)
groupBy <- fromMaybe [] <$> optional groupByClause groupBy <- fromMaybe [] <$> optional groupByClause
orderBy <- fromMaybe [] <$> optional orderByClause orderBy <- fromMaybe [] <$> optional orderByClause
mLimit <- optional limitClause mLimit <- optional limitClause
pure $ if all (\(name, e) -> e == Col name) cols pure $ (if distinct then SelectDistinct else id) $ if all (\(name, e) -> e == Col name) cols
then Select (map fst cols) fromC mWhere groupBy orderBy mLimit then Select (map fst cols) fromC mWhere groupBy orderBy mLimit
else SelectExpr cols fromC mWhere groupBy orderBy mLimit else SelectExpr cols fromC mWhere groupBy orderBy mLimit
@@ -738,6 +740,7 @@ cteKey = map toLower
-- | CTE 带列清单时改本体的输出名 -- | CTE 带列清单时改本体的输出名
renameColumns :: String -> [String] -> Statement -> Either String Statement renameColumns :: String -> [String] -> Statement -> Either String Statement
renameColumns _ [] q = Right q renameColumns _ [] q = Right q
renameColumns name cols (SelectDistinct q) = SelectDistinct <$> renameColumns name cols q
renameColumns name cols q@Select { selectCols = names } = do renameColumns name cols q@Select { selectCols = names } = do
checkCteArity name cols names checkCteArity name cols names
Right Right
@@ -770,6 +773,7 @@ checkCteArity name cols out
-- | 把语句里对 CTE 的引用换成派生表 -- | 把语句里对 CTE 的引用换成派生表
expandStatement :: [(String, Statement)] -> Statement -> Either String Statement expandStatement :: [(String, Statement)] -> Statement -> Either String Statement
expandStatement env stmt = case stmt of expandStatement env stmt = case stmt of
SelectDistinct q -> SelectDistinct <$> expandStatement env q
q@Select { selectFrom = fromC, selectWhere = mWhere } -> do q@Select { selectFrom = fromC, selectWhere = mWhere } -> do
fromC' <- expandClause env fromC fromC' <- expandClause env fromC
w <- traverse (expandExpr env) mWhere w <- traverse (expandExpr env) mWhere
@@ -909,6 +913,7 @@ parseStatement input =
<|> try createUserStatement <|> try createUserStatement
<|> try alterUserStatement <|> try alterUserStatement
<|> try alterRoleStatement <|> try alterRoleStatement
<|> try (ShowGrants <$ (keyword "show" *> keyword "grants"))
<|> try (ShowRoles <$ (keyword "show" *> keyword "roles")) <|> try (ShowRoles <$ (keyword "show" *> keyword "roles"))
<|> try dropUserStatement <|> try dropUserStatement
<|> try createRoleStatement <|> try createRoleStatement
chusql-core/engine/test/RuntimeSpec.hs
+258 -4
View File
@@ -3,20 +3,38 @@ module RuntimeSpec (spec) where
import ChuSQL.Core.Engine.Runtime.Types import ChuSQL.Core.Engine.Runtime.Types
import ChuSQL.Core.Engine.Runtime.Functions import ChuSQL.Core.Engine.Runtime.Functions
import ChuSQL.Core.Engine.Runtime.Binding import ChuSQL.Core.Engine.Runtime.Binding
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
import qualified ChuSQL.Core.Engine.Algebra.Op as Rel
import ChuSQL.Core.Engine.Algebra.Expr (evalExpr)
import ChuSQL.Core.Engine.Algebra.Eval (evalRelOp)
import ChuSQL.Core.Engine.Algebra.Sort (sortBoundRows)
import ChuSQL.Core.Engine.Storage.Memory (runMemoryStorage)
import qualified ChuSQL.Core.Engine.Runtime.Query as Query import qualified ChuSQL.Core.Engine.Runtime.Query as Query
import ChuSQL.Core.Engine (rowsOf, runStatement) import ChuSQL.Core.Engine (rowsOf, runStatement)
import ChuSQL.Core.Engine.Semantic (prepare) import ChuSQL.Core.Engine.Semantic (prepare, prepareRowExpression)
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement) import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
import ChuSQL.Core.Engine.Syntax.AST (Expr (..), Statement (..)) import ChuSQL.Core.Engine.Syntax.AST (Expr (..), Statement (..), SortDir (..))
import qualified ChuSQL.Core.Model as SQL import qualified ChuSQL.Core.Model as SQL
import ChuSQL.Core.Protocol (valueToJSON, valueFromJSON) import ChuSQL.Core.Protocol (valueToJSON, valueFromJSON, TableInfo (..))
import qualified Data.Aeson as A
import Data.Word (Word64)
import Data.Aeson.Types (parseEither) import Data.Aeson.Types (parseEither)
import Data.Hashable (hash) import Data.Hashable (hash)
import Control.Monad (foldM)
import qualified Data.ByteString.Lazy.Char8 as BS import qualified Data.ByteString.Lazy.Char8 as BS
import Test.Hspec import Test.Hspec
-- 运行时类型、函数绑定、物理编码和类型化查询测试。 -- 运行时类型、函数绑定、物理编码和类型化查询测试。
-- | 执行语句建立测试目录
sqlCatalog :: [String] -> Either String SQL.Database
sqlCatalog = foldM apply []
where
-- | 更新当前测试数据库
apply db source = do
statement <- parseStatement source
fst <$> runStatement db statement
-- | 检查运行时类型基础契约 -- | 检查运行时类型基础契约
spec :: Spec spec :: Spec
spec = do spec = do
@@ -72,6 +90,35 @@ spec = do
let expression = Apply "not" [Apply "not" [Literal boolType (RBool True)]] let expression = Apply "not" [Apply "not" [Literal boolType (RBool True)]]
(bindExpression builtinTypes functions [] expression >>= evaluateExpression builtinTypes functions []) (bindExpression builtinTypes functions [] expression >>= evaluateExpression builtinTypes functions [])
`shouldBe` Right (RBool True) `shouldBe` Right (RBool True)
it "uses the bound column slot and rejects reordered expression input" $ withFunctions $ \functions ->
withType intType $ \int -> withType stringType $ \string -> do
let bound = bindExpression builtinTypes functions [("count", int), ("name", string)]
(Apply "length" [Reference "name"])
(bound >>= evaluateExpression builtinTypes functions [("count", RInt 1), ("name", RString "abc")])
`shouldBe` Right (RInt 3)
(bound >>= evaluateExpression builtinTypes functions [("name", RString "abc"), ("count", RInt 1)])
`shouldBe` Left "bound column position mismatch: name"
(bound >>= evaluateExpression builtinTypes functions [("count", RInt 1)])
`shouldBe` Left "missing bound column: name"
it "rejects ambiguous references before binding a slot" $ withFunctions $ \functions ->
withType intType $ \int ->
bindExpression builtinTypes functions [("id", int), ("id", int)] (Reference "id")
`shouldSatisfy` isFailure
it "normalizes storage order at scan boundaries before evaluating slots" $ withFunctions $ \functions -> do
let catalog = [("items", [("name", stringType), ("enabled", boolType)])]
query = Projection [("size", Apply "length" [Reference "name"])]
(Selection (Reference "enabled") (TableScan "items"))
scan _ = Right [[("enabled", RBool True), ("name", RString "abc")]]
(bindPlan builtinTypes functions catalog query >>= executePlan builtinTypes functions scan)
`shouldBe` Right [[("size", RInt 3)]]
it "rebinds slots to each projection output schema" $ withFunctions $ \functions -> do
let catalog = [("items", [("name", stringType), ("enabled", boolType)])]
source = Projection [("enabled", Reference "enabled"), ("label", Reference "name")]
(TableScan "items")
query = Projection [("size", Apply "length" [Reference "label"])] source
scan _ = Right [[("name", RString "abc"), ("enabled", RBool True)]]
(bindPlan builtinTypes functions catalog query >>= executePlan builtinTypes functions scan)
`shouldBe` Right [[("size", RInt 3)]]
it "rejects a wrong function argument type during binding" $ withFunctions $ \functions -> it "rejects a wrong function argument type during binding" $ withFunctions $ \functions ->
bindExpression builtinTypes functions [] (Apply "length" [Literal intType (RInt 1)]) `shouldSatisfy` isFailure bindExpression builtinTypes functions [] (Apply "length" [Literal intType (RInt 1)]) `shouldSatisfy` isFailure
it "rejects unknown function identities" $ withFunctions $ \functions -> it "rejects unknown function identities" $ withFunctions $ \functions ->
@@ -104,12 +151,219 @@ spec = do
(bindPlan builtinTypes functions [("t", [])] (TableScan "t") (bindPlan builtinTypes functions [("t", [])] (TableScan "t")
>>= executePlan builtinTypes functions (const (Left "storage unavailable"))) `shouldBe` Left "storage unavailable" >>= executePlan builtinTypes functions (const (Left "storage unavailable"))) `shouldBe` Left "storage unavailable"
describe "SQL runtime functions" $ do describe "SQL runtime functions" $ do
it "validates transitive DOMAIN dependencies and ignores unrelated definitions" $ do
let result = do
db <- sqlCatalog ["CREATE DOMAIN amount AS INT", "CREATE DOMAIN total AS amount"
, "CREATE DOMAIN unrelated AS TEXT", "CREATE TABLE amounts (value total)"]
statement <- parseStatement "SELECT value FROM amounts"
bound <- Query.bindQuery db statement
let change target = [(name, if name == target then table {SQL.tableCols = [("base", SQL.plainColumn SQL.CBool)]}
else table) | (name, table) <- db]
accepted <- fst <$> runMemoryStorage (Query.executeQueryM bound) (change "__system_domain_unrelated")
rejected <- fst <$> runMemoryStorage (Query.executeQueryM bound) (change "__system_domain_amount")
Right (accepted, rejected)
result `shouldBe` Right (Right [], Left "bound query schema changed: __system_domain_amount")
it "preserves exact catalog identities and rejects malformed identifiers" $ do
let identity = SQL.TableIdentity (maxBound :: Word64) (maxBound :: Word64)
info = TableInfo "items" [] 0 [] [] [] (Just identity) Nothing
decode text = A.eitherDecode (BS.pack text) :: Either String TableInfo
A.fromJSON (A.toJSON info) `shouldBe` A.Success info
fmap tiIdentity (decode "{\"table\":\"items\",\"columns\":[],\"row_count\":0}") `shouldBe` Right Nothing
fmap tiIdentity (decode "{\"table\":\"items\",\"columns\":[],\"row_count\":0,\"database_id\":0,\"object_id\":1}")
`shouldBe` Right (Just (SQL.TableIdentity 0 1))
mapM_ (\fields -> decode ("{\"table\":\"items\",\"columns\":[],\"row_count\":0," ++ fields ++ "}")
`shouldSatisfy` isFailure)
["\"database_id\":1", "\"database_id\":1,\"object_id\":-1"
, "\"database_id\":1,\"object_id\":1.5", "\"database_id\":1,\"object_id\":\"18446744073709551616\""
, "\"database_id\":1,\"object_id\":1e99999999"]
it "rejects a same-schema replacement or loss of object identity" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
let identified = [(name, table {SQL.tableIdentity = Just (SQL.TableIdentity 1 1)}) | (name, table) <- db]
statement <- parseStatement "SELECT id FROM bound_items"
bound <- Query.bindQuery identified statement
let replaced = [(name, table {SQL.tableIdentity = Just (SQL.TableIdentity 1 2)}) | (name, table) <- db]
changed <- fst <$> runMemoryStorage (Query.executeQueryM bound) replaced
lost <- fst <$> runMemoryStorage (Query.executeQueryM bound) db
Right (changed, lost)
result `shouldBe` Right (Left "bound query object changed: bound_items", Left "bound query object changed: bound_items")
it "preserves exact schema versions and rejects malformed versions" $ do
let info = TableInfo "items" [] 0 [] [] [] Nothing (Just maxBound)
decode version = A.eitherDecode (BS.pack ("{\"table\":\"items\",\"columns\":[],\"row_count\":0,\"schema_version\":" ++ version ++ "}")) :: Either String TableInfo
A.fromJSON (A.toJSON info) `shouldBe` A.Success info
fmap tiSchemaVersion (decode "0") `shouldBe` Right Nothing
fmap tiSchemaVersion (decode "7") `shouldBe` Right (Just 7)
mapM_ (\version -> decode version `shouldSatisfy` isFailure)
["-1", "1.5", "\"18446744073709551616\"", "\"+1\"", "1e99999999"]
it "invalidates a bound query after restoring its original columns" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE versioned (id INT)"]
statement <- parseStatement "SELECT id FROM versioned"
bound <- Query.bindQuery db statement
added <- parseStatement "ALTER TABLE versioned ADD COLUMN extra TEXT" >>= runStatement db
restored <- parseStatement "ALTER TABLE versioned DROP COLUMN extra" >>= runStatement (fst added)
fst <$> runMemoryStorage (Query.executeQueryM bound) (fst restored)
result `shouldBe` Right (Left "bound query schema changed: versioned")
it "rejects loss of a bound schema version" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE versioned (id INT)"]
statement <- parseStatement "SELECT id FROM versioned"
bound <- Query.bindQuery db statement
fst <$> runMemoryStorage (Query.executeQueryM bound)
[(name, table {SQL.tableSchemaVersion = Nothing}) | (name, table) <- db]
result `shouldBe` Right (Left "bound query schema changed: versioned")
it "ignores unrelated table changes but validates nested query dependencies" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE referenced (id INT)", "CREATE TABLE unrelated (id INT)"]
statement <- parseStatement "SELECT length((SELECT MIN(name) FROM (SELECT 'abc' AS name FROM referenced) d)) AS n"
bound <- Query.bindQuery db statement
unrelated <- parseStatement "ALTER TABLE unrelated ADD COLUMN extra TEXT" >>= runStatement db
accepted <- fst <$> runMemoryStorage (Query.executeQueryM bound) (fst unrelated)
referenced <- parseStatement "ALTER TABLE referenced ADD COLUMN extra TEXT" >>= runStatement db
rejected <- fst <$> runMemoryStorage (Query.executeQueryM bound) (fst referenced)
Right (accepted, rejected)
result `shouldBe` Right (Right [[("n", SQL.VNull)]], Left "bound query schema changed: referenced")
it "rejects schema drift before executing an empty bound query" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
statement <- parseStatement "SELECT id FROM bound_items"
bound <- Query.bindQuery db statement
changed <- parseStatement "ALTER TABLE bound_items ADD COLUMN extra TEXT" >>= runStatement db
fst <$> runMemoryStorage (Query.executeQueryM bound) (fst changed)
result `shouldBe` Right (Left "bound query schema changed: bound_items")
it "keeps binding valid across row and statistics changes" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
statement <- parseStatement "SELECT id FROM bound_items"
bound <- Query.bindQuery db statement
changed <- parseStatement "INSERT INTO bound_items (id) VALUES (7)" >>= runStatement db
let current = [(name, table {SQL.tableMeta = Just (SQL.TableMeta 100 [] [] [])})
| (name, table) <- fst changed]
fst <$> runMemoryStorage (Query.executeQueryM bound) current
result `shouldBe` Right (Right [[("id", SQL.VInt 7)]])
it "binds grouping keys and then sorting to the grouped output" $ do
let source = Rel.Compute [("first", LitInt 1), ("key", LitInt 2)] Rel.Unit
plan = Rel.Sort [("key", Desc)] (Rel.Aggregate ["key"] [("n", CountAll)] source)
bound = Rel.BoundSort [(ColumnSlot 0 "key", Desc)]
(Rel.BoundAggregate [("key", ColumnSlot 1 "key")] [("n", CountAll)] source)
bindRelOpColumns [] [] plan `shouldBe` Right bound
evalRelOp [] bound `shouldBe` Right [[("key", SQL.VInt 2), ("n", SQL.VInt 1)]]
evalRelOp [] (Rel.BoundAggregate [("key", ColumnSlot 0 "key")] [] source)
`shouldBe` Left "bound column position mismatch: key"
it "sorts bound NULL keys stably and rejects wrong slots" $ do
let a = [("id", SQL.VInt 1), ("key", SQL.VInt 2)]
b = [("id", SQL.VInt 2), ("key", SQL.VNull)]
c = [("id", SQL.VInt 3), ("key", SQL.VInt 2)]
sortBoundRows [(ColumnSlot 1 "key", Asc)] [] [a, b, c] `shouldBe` Right [b, a, c]
sortBoundRows [(ColumnSlot 1 "key", Desc)] [] [a, b, c] `shouldBe` Right [a, c, b]
sortBoundRows [(ColumnSlot 0 "key", Asc)] [] [a]
`shouldBe` Left "bound column position mismatch: key"
it "preserves NULL group order and empty-input global aggregates" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE grouped_slots (id INT, tag TEXT)"
, "INSERT INTO grouped_slots (id, tag) VALUES (1, 'b'), (2, NULL), (3, 'b')"]
grouped <- parseStatement "SELECT tag, COUNT(*) AS n FROM grouped_slots GROUP BY tag"
groups <- rowsOf (runStatement db grouped)
empty <- parseStatement "SELECT COUNT(*) AS n, SUM(id) AS total FROM grouped_slots WHERE id < 0"
aggregates <- rowsOf (runStatement db empty)
Right (map (map snd) groups, map (map snd) aggregates)
result `shouldBe` Right ([[SQL.VStr "b", SQL.VInt 2], [SQL.VNull, SQL.VInt 1]]
, [[SQL.VInt 0, SQL.VNull]])
it "binds SQL relation expressions to the optimized input slots" $ do
let plan = Rel.Compute [("value", Col "second")]
(Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit)
bindRelOpColumns [] [] plan `shouldBe` Right (Rel.Compute [("value", ColumnSlot 1 "second")]
(Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit))
bindRelOpColumns [] [] (Rel.Filter (Col "missing") Rel.Unit)
`shouldBe` Left "unknown column: missing"
it "checks SQL slot position instead of searching another column" $ do
evalExpr (ColumnSlot 1 "name") [("name", SQL.VStr "abc"), ("id", SQL.VInt 1)]
`shouldBe` Left "bound column position mismatch: name"
evalExpr (ColumnSlot (-1) "name") [("name", SQL.VStr "abc")]
`shouldBe` Left "bound column position mismatch: name"
it "binds projections without losing repeated outputs or empty rows" $ do
let source = Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit
bindRelOpColumns [] [] (Rel.Project ["second", "first", "second"] source)
`shouldBe` Right (Rel.Compute [("second", ColumnSlot 1 "second")
, ("first", ColumnSlot 0 "first"), ("second", ColumnSlot 1 "second")] source)
bindRelOpColumns [] [] (Rel.Project [] source)
`shouldBe` Right (Rel.Compute [] source)
bindRelOpColumns [] [] (Rel.Project ["*"] source)
`shouldBe` Right (Rel.Compute [("first", ColumnSlot 0 "first"), ("second", ColumnSlot 1 "second")] source)
it "preserves repeated SQL outputs and correlated projection values" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE slot_items (id INT)"
, "INSERT INTO slot_items (id) VALUES (1), (2)"]
repeated <- parseStatement "SELECT id, id FROM slot_items"
repeatedRows <- rowsOf (runStatement db repeated)
correlated <- parseStatement "SELECT o.id, (SELECT o.id FROM slot_items i WHERE i.id = o.id) AS echoed FROM slot_items o"
correlatedRows <- rowsOf (runStatement db correlated)
Right (map (map snd) repeatedRows, map (map snd) correlatedRows)
result `shouldBe` Right ([[SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2]]
, [[SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2]])
it "binds outer references after the local relation layout" $ do
bindRelOpColumns [] ["outer.id"] (Rel.Compute [("id", Col "outer.id")] Rel.Unit)
`shouldBe` Right (Rel.Compute [("id", OuterColumnSlot 0 0 "outer.id")] Rel.Unit)
it "preserves outer scope identity when rebinding beside a local name" $ do
let source = Rel.Compute [("id", LitInt 1)] Rel.Unit
expression = OuterColumnSlot 0 0 "id"
bound = bindRelOpColumns [] ["id"] (Rel.Compute [("outer", expression)] source)
bound `shouldBe` Right (Rel.Compute [("outer", OuterColumnSlot 1 0 "id")] source)
evalExpr (OuterColumnSlot 1 0 "id") [("id", SQL.VInt 1), ("id", SQL.VInt 9)]
`shouldBe` Right (SQL.VInt 9)
bindRelOpColumns [] [] (Rel.Compute [("outer", expression)] source)
`shouldBe` Left "unknown column: id"
prepareRowExpression [] expression `shouldBe` Left "execution column cannot be used in semantic input"
prepareRowExpression [] (ColumnSlot 0 "id") `shouldBe` Left "execution column cannot be used in semantic input"
it "keeps inner shadowing and multi-level correlated SQL references" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE scoped_items (id INT)"
, "INSERT INTO scoped_items (id) VALUES (1), (2)"]
statement <- parseStatement "SELECT o.id, (SELECT id FROM scoped_items i WHERE i.id = o.id) AS inner_id, (SELECT (SELECT o.id FROM scoped_items j WHERE j.id = i.id) FROM scoped_items i WHERE i.id = o.id) AS nested_id FROM scoped_items o"
rows <- rowsOf (runStatement db statement)
Right (map (map snd) rows)
result `shouldBe` Right [[SQL.VInt 1, SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2, SQL.VInt 2]]
it "executes SQL against reversed physical column order" $ do
let result = do
db <- sqlCatalog ["CREATE TABLE slot_items (id INT, name TEXT)"
, "INSERT INTO slot_items (id, name) VALUES (1, 'abc'), (2, 'xy')"]
let reordered = [(name, table {SQL.tableRows = map reverse (SQL.tableRows table)}) | (name, table) <- db]
statement <- parseStatement "SELECT length(name) AS size, id + 1 AS next FROM slot_items WHERE id > 0 ORDER BY size"
rowsOf (runStatement reordered statement)
fmap (map (map snd)) result `shouldBe` Right [[SQL.VInt 2, SQL.VInt 3], [SQL.VInt 3, SQL.VInt 2]]
it "binds the SQL entry point to output TypeIds" $ withType intType $ \tid -> do it "binds the SQL entry point to output TypeIds" $ withType intType $ \tid -> do
let result = parseStatement "SELECT length('abc') AS size" >>= Query.bindQuery [] let result = parseStatement "SELECT length('abc') AS size" >>= Query.bindQuery []
fmap Query.queryTypes result `shouldBe` Right [("size", Query.RuntimeType tid)] fmap Query.queryTypes result `shouldBe` Right [("size", Query.RuntimeType tid)]
it "retains compatibility for existing temporal SQL types" $ do it "retains compatibility for existing temporal SQL types" $ do
let result = parseStatement "SELECT DATE '2026-10-05' AS today" >>= Query.bindQuery [] let result = parseStatement "SELECT DATE '2026-10-05' AS today" >>= Query.bindQuery []
fmap Query.queryTypes result `shouldBe` Right [("today", Query.CompatibilityType SQL.CDate)] fmap Query.queryTypes result `shouldBe` Right [("today", Query.CompatibilityType SQL.CDate)]
it "retains nested DOMAIN identities through derived aliases and DISTINCT" $ withType intType $ \tid -> do
let result = do
db <- sqlCatalog ["CREATE DOMAIN quantity AS INT", "CREATE DOMAIN stock AS quantity"
, "CREATE TABLE inventory (qty stock)", "INSERT INTO inventory (qty) VALUES (2), (NULL)"]
statement <- parseStatement "SELECT DISTINCT d.qty AS available FROM (SELECT qty FROM inventory) d"
bound <- Query.bindQuery db statement
rows <- rowsOf (runStatement db statement)
Right (Query.queryTypes bound, map (map snd) rows)
result `shouldBe` Right ([("available", Query.DomainType "stock"
(Query.DomainType "quantity" (Query.RuntimeType tid)))], [[SQL.VInt 2], [SQL.VNull]])
it "preserves temporal and binary DOMAIN wrappers" $ do
let result = do
db <- sqlCatalog ["CREATE DOMAIN day AS DATE", "CREATE DOMAIN payload AS BLOB"
, "CREATE TABLE documents (created day, contents payload)"]
statement <- parseStatement "SELECT created, contents FROM documents"
Query.queryTypes <$> Query.bindQuery db statement
result `shouldBe` Right [("created", Query.DomainType "day" (Query.CompatibilityType SQL.CDate))
, ("contents", Query.DomainType "payload" (Query.CompatibilityType SQL.CBlob))]
it "uses base function semantics without labeling function results as DOMAIN" $ withType intType $ \tid -> do
let result = do
db <- sqlCatalog ["CREATE DOMAIN label AS VARCHAR(3)", "CREATE TABLE names (name label)"
, "INSERT INTO names (name) VALUES ('abc')"]
statement <- parseStatement "SELECT length(name) AS size FROM names"
bound <- Query.bindQuery db statement
rows <- rowsOf (runStatement db statement)
Right (Query.queryTypes bound, map (map snd) rows)
result `shouldBe` Right ([("size", Query.RuntimeType tid)], [[SQL.VInt 3]])
it "rejects composite ordering without an Ord implementation" $ it "rejects composite ordering without an Ord implementation" $
sql "SELECT LIST<Int>(1) AS xs ORDER BY xs" `shouldSatisfy` isFailure sql "SELECT LIST<Int>(1) AS xs ORDER BY xs" `shouldSatisfy` isFailure
it "rejects composite extrema without an Ord implementation" $ it "rejects composite extrema without an Ord implementation" $
@@ -201,7 +455,7 @@ sql source = do
-- | 提供函数调用测试表 -- | 提供函数调用测试表
database = [("t", SQL.Table "t" [("name", SQL.plainColumn SQL.CStr), ("enabled", SQL.plainColumn SQL.CBool)] database = [("t", SQL.Table "t" [("name", SQL.plainColumn SQL.CStr), ("enabled", SQL.plainColumn SQL.CBool)]
[[("name", SQL.VStr "ab"), ("enabled", SQL.VBool True)] [[("name", SQL.VStr "ab"), ("enabled", SQL.VBool True)]
,[("name", SQL.VStr "xyz"), ("enabled", SQL.VBool False)]] Nothing)] ,[("name", SQL.VStr "xyz"), ("enabled", SQL.VBool False)]] Nothing Nothing Nothing)]
-- | 判断结果是否明确失败 -- | 判断结果是否明确失败
isFailure :: Either a b -> Bool isFailure :: Either a b -> Bool
chusql-core/engine/test/Spec.hs
+142 -10
View File
@@ -12,6 +12,7 @@ import ChuSQL.Core.Engine.Algebra.Sort (sortRows)
import ChuSQL.Core.Engine.Builtin import ChuSQL.Core.Engine.Builtin
import ChuSQL.Core.Engine.Error import ChuSQL.Core.Engine.Error
import ChuSQL.Core.Engine import ChuSQL.Core.Engine
import qualified ChuSQL.Core.Engine.Runtime.Query as Bound
import ChuSQL.Core.Engine.Parallel (poolRun, workerPool) import ChuSQL.Core.Engine.Parallel (poolRun, workerPool)
import ChuSQL.Core.Model import ChuSQL.Core.Model
import ChuSQL.Core.Engine.Semantic (prepare) import ChuSQL.Core.Engine.Semantic (prepare)
@@ -46,7 +47,9 @@ import qualified RuntimeSpec
users :: Table users :: Table
users = users =
Table Table
{ tableName = "users" { tableSchemaVersion = Nothing
, tableIdentity = Nothing
, tableName = "users"
, tableCols = [("id", TInt), ("name", TStr), ("age", TInt)] , tableCols = [("id", TInt), ("name", TStr), ("age", TInt)]
, tableRows = , tableRows =
[ [("id", VInt 1), ("name", VStr "Alice"), ("age", VInt 25)] [ [("id", VInt 1), ("name", VStr "Alice"), ("age", VInt 25)]
@@ -60,7 +63,9 @@ users =
orders :: Table orders :: Table
orders = orders =
Table Table
{ tableName = "orders" { tableSchemaVersion = Nothing
, tableIdentity = Nothing
, tableName = "orders"
, tableCols = [("id", TInt), ("user_id", TInt), ("product", TStr)] , tableCols = [("id", TInt), ("user_id", TInt), ("product", TStr)]
, tableRows = , tableRows =
[ [("id", VInt 1), ("user_id", VInt 1), ("product", VStr "Book")] [ [("id", VInt 1), ("user_id", VInt 1), ("product", VStr "Book")]
@@ -78,7 +83,9 @@ testDB = [("users", users), ("orders", orders)]
chainA :: Table chainA :: Table
chainA = chainA =
Table Table
{ tableName = "a" { tableSchemaVersion = Nothing
, tableIdentity = Nothing
, tableName = "a"
, tableCols = [("id", TInt), ("name", TStr)] , tableCols = [("id", TInt), ("name", TStr)]
, tableRows = [[("id", VInt n), ("name", VStr ("a" ++ show n))] | n <- [1 .. 6]] , tableRows = [[("id", VInt n), ("name", VStr ("a" ++ show n))] | n <- [1 .. 6]]
, tableMeta = Just (TableMeta 6 [("id", 6, False)] [] ["id"]) , tableMeta = Just (TableMeta 6 [("id", 6, False)] [] ["id"])
@@ -88,7 +95,9 @@ chainA =
chainB :: Table chainB :: Table
chainB = chainB =
Table Table
{ tableName = "b" { tableSchemaVersion = Nothing
, tableIdentity = Nothing
, tableName = "b"
, tableCols = [("id", TInt), ("a_id", TInt)] , tableCols = [("id", TInt), ("a_id", TInt)]
, tableRows = [[("id", VInt n), ("a_id", VInt n)] | n <- [1 .. 4]] , tableRows = [[("id", VInt n), ("a_id", VInt n)] | n <- [1 .. 4]]
, tableMeta = Just (TableMeta 4 [("id", 4, False), ("a_id", 4, False)] [] ["id"]) , tableMeta = Just (TableMeta 4 [("id", 4, False), ("a_id", 4, False)] [] ["id"])
@@ -98,7 +107,9 @@ chainB =
chainC :: Table chainC :: Table
chainC = chainC =
Table Table
{ tableName = "c" { tableSchemaVersion = Nothing
, tableIdentity = Nothing
, tableName = "c"
, tableCols = [("id", TInt), ("b_id", TInt)] , tableCols = [("id", TInt), ("b_id", TInt)]
, tableRows = [[("id", VInt n), ("b_id", VInt n)] | n <- [1, 2]] , tableRows = [[("id", VInt n), ("b_id", VInt n)] | n <- [1, 2]]
, tableMeta = Just (TableMeta 2 [("id", 2, False), ("b_id", 2, False)] [] ["id"]) , tableMeta = Just (TableMeta 2 [("id", 2, False), ("b_id", 2, False)] [] ["id"])
@@ -124,7 +135,64 @@ domainSql db (sql : rest) = do
-- | 跑引擎层全部 hspec 用例 -- | 跑引擎层全部 hspec 用例
main :: IO () main :: IO ()
main = hspec $ do main = hspec $ do
describe "REVOKE GRANT OPTION FOR" $ do
it "parses object privileges and rejects membership options" $ do
parseStatement "REVOKE GRANT OPTION FOR SELECT, UPDATE ON users FROM alice"
`shouldBe` Right (RevokeGrantOption ["SELECT", "UPDATE"] "users" "alice")
parseStatement "REVOKE GRANT OPTION FOR reader FROM alice" `shouldSatisfy` either (const True) (const False)
describe "ORDER BY positions" $ do
it "resolves positions after wildcard expansion and before limiting" $ do
rowsOf (parseStatement "SELECT * FROM orders ORDER BY 2 DESC, 1 DESC LIMIT 2" >>= runStatement testDB)
`shouldBe` Right [tableRows orders !! 1, tableRows orders !! 2]
rowsOf (parseStatement "SELECT DISTINCT user_id + 1 AS k FROM orders ORDER BY 1 DESC" >>= runStatement testDB)
`shouldBe` Right [[("k", VInt 3)], [("k", VInt 2)]]
it "rejects zero and positions beyond the output" $ do
mapM_ (\sql -> (parseStatement sql >>= runStatement testDB) `shouldSatisfy` either (isInfixOf "outside the select list") (const False))
["SELECT id FROM users ORDER BY 0", "SELECT id FROM users ORDER BY 2"]
describe "SELECT DISTINCT" $ do
it "deduplicates projected values before ordering and limiting" $ do
let db = [("orders", orders { tableRows = tableRows orders ++ tableRows orders })]
rowsOf (parseStatement "SELECT DISTINCT user_id FROM orders ORDER BY user_id DESC LIMIT 2" >>= runStatement db)
`shouldBe` Right [[("user_id", VInt 2)], [("user_id", VInt 1)]]
rowsOf (parseStatement "SELECT DISTINCT user_id + 1 AS k FROM orders ORDER BY k LIMIT 2" >>= runStatement db)
`shouldBe` Right [[("k", VInt 2)], [("k", VInt 3)]]
rowsOf (parseStatement "SELECT ALL user_id FROM orders" >>= runStatement db)
`shouldBe` Right (map (filter ((== "user_id") . fst)) (tableRows orders ++ tableRows orders))
it "merges NULL and equal numeric values while retaining first rows" $ do
evalRelOp [] (Distinct Unit) `shouldBe` Right [[]]
let t = users { tableRows = [[("id", v), ("name", VStr "a"), ("age", VInt 1)] | v <- [VNull, VNull, VInt 1, VFloat 1, VInt 2]] }
evalRelOp [("users", t)] (Distinct (Project ["id"] (Scan Nothing "users" Nothing)))
`shouldBe` Right [[("id", VNull)], [("id", VInt 1)], [("id", VInt 2)]]
it "keeps all distinct columns when an outer query projects fewer" $ do
rowsOf (parseStatement "SELECT d.user_id FROM (SELECT DISTINCT user_id, product FROM orders) AS d" >>= runStatement testDB)
`shouldBe` Right [[("d.user_id", VInt 1)], [("d.user_id", VInt 2)], [("d.user_id", VInt 1)]]
rowsOf (parseStatement "WITH d(k) AS (SELECT DISTINCT user_id FROM orders) SELECT k FROM d" >>= runStatement testDB)
`shouldBe` Right [[("d.k", VInt 1)], [("d.k", VInt 2)]]
it "supports aggregates, empty inputs and output aliases" $ do
rowsOf (parseStatement "SELECT DISTINCT COUNT(*) AS n FROM orders" >>= runStatement testDB)
`shouldBe` Right [[("n", VInt 3)]]
rowsOf (parseStatement "SELECT DISTINCT user_id FROM orders WHERE user_id = 99" >>= runStatement testDB)
`shouldBe` Right []
rowsOf (parseStatement "SELECT DISTINCT user_id AS k FROM orders ORDER BY user_id" >>= runStatement testDB)
`shouldBe` Right [[("k", VInt 1)], [("k", VInt 2)]]
it "rejects composite values without equality capability" $ do
(parseStatement "SELECT DISTINCT LIST<Int>(1) AS xs" >>= runStatement [])
`shouldSatisfy` either (isInfixOf "capability is not registered") (const False)
it "evaluates correlated DISTINCT subqueries with the outer binding" $ do
rowsOf (parseStatement "SELECT u.id FROM users u WHERE u.id IN (SELECT DISTINCT o.user_id FROM orders o WHERE o.user_id = u.id)" >>= runStatement testDB)
`shouldBe` Right [[("u.id", VInt 1)], [("u.id", VInt 2)]]
it "rejects sorting by values excluded from the distinct output" $ do
let result = parseStatement "SELECT DISTINCT user_id FROM orders ORDER BY product" >>= runStatement testDB
result `shouldSatisfy` either (isInfixOf "must appear in the select list") (const False)
describe "Storage read failures" $ do describe "Storage read failures" $ do
it "rejects storage that silently ignores a bound read guard" $ do
setStorageLink StorageLink
{ slRequest = \_ -> pure (Right (BSC.pack "{\"status\":\"rows\",\"rows\":[]}"))
, slClose = pure ()
}
(runIPCStorage (withSchemaGuard [] (scan "ignored"))
`shouldReturn` Left "storage did not confirm bound schema guard") `finally` closeConnection
it "propagates catalog failures through snapshot, schema and SQL" $ do it "propagates catalog failures through snapshot, schema and SQL" $ do
setStorageLink StorageLink setStorageLink StorageLink
{ slRequest = \_ -> pure (Right (BSC.pack "{\"status\":\"error\",\"message\":\"catalog unavailable\"}")) { slRequest = \_ -> pure (Right (BSC.pack "{\"status\":\"error\",\"message\":\"catalog unavailable\"}"))
@@ -729,7 +797,7 @@ main = hspec $ do
it "rewrites >= and <= into an inclusive index range" $ do it "rewrites >= and <= into an inclusive index range" $ do
let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))] dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
indexed = dbWith (stat 10000 [("id", 10000)] ["id"]) indexed = dbWith (stat 10000 [("id", 10000)] ["id"])
low = Filter (GtE (Col "id") (LitInt 2)) (Scan Nothing "users" Nothing) low = Filter (GtE (Col "id") (LitInt 2)) (Scan Nothing "users" Nothing)
both = Filter (And (GtE (Col "id") (LitInt 1)) (LtE (Col "id") (LitInt 3))) (Scan Nothing "users" Nothing) both = Filter (And (GtE (Col "id") (LitInt 1)) (LtE (Col "id") (LitInt 3))) (Scan Nothing "users" Nothing)
@@ -1798,7 +1866,7 @@ main = hspec $ do
let buckets = replicate 8 63 ++ replicate 8 62 let buckets = replicate 8 63 ++ replicate 8 62
withHist = TableMeta 1000 [("id", 1000, False)] [("id", Histogram 0 1600 buckets)] ["id"] withHist = TableMeta 1000 [("id", 1000, False)] [("id", Histogram 0 1600 buckets)] ["id"]
plain = TableMeta 1000 [("id", 1000, False)] [] ["id"] plain = TableMeta 1000 [("id", 1000, False)] [] ["id"]
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))] dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
narrow = Filter (And (GtE (Col "id") (LitInt 10)) (LtE (Col "id") (LitInt 20))) (Scan Nothing "users" Nothing) narrow = Filter (And (GtE (Col "id") (LitInt 10)) (LtE (Col "id") (LitInt 20))) (Scan Nothing "users" Nothing)
wide = Filter (Gt (Col "id") (LitInt 0)) (Scan Nothing "users" Nothing) wide = Filter (Gt (Col "id") (LitInt 0)) (Scan Nothing "users" Nothing)
-- 直方图说这个窄范围只命中十几行：改写 -- 直方图说这个窄范围只命中十几行：改写
@@ -1884,8 +1952,8 @@ main = hspec $ do
it "picks the index only when the statistics make it cheaper" $ do it "picks the index only when the statistics make it cheaper" $ do
let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))] dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
unknown = [("users", Table "users" [("id", TInt), ("name", TStr)] [] Nothing)] unknown = [("users", Table "users" [("id", TInt), ("name", TStr)] [] Nothing Nothing Nothing)]
point = Filter (Eq (Col "id") (LitInt 1)) (Scan Nothing "users" Nothing) point = Filter (Eq (Col "id") (LitInt 1)) (Scan Nothing "users" Nothing)
range = Filter (Gt (Col "id") (LitInt 5)) (Scan Nothing "users" Nothing) range = Filter (Gt (Col "id") (LitInt 5)) (Scan Nothing "users" Nothing)
-- 索引在、点查命中一行：改写 -- 索引在、点查命中一行：改写
@@ -2332,6 +2400,51 @@ main = hspec $ do
map fst db `shouldBe` ["a", "b"] map fst db `shouldBe` ["a", "b"]
map (length . tableRows . snd) db `shouldBe` [1, 1] map (length . tableRows . snd) db `shouldBe` [1, 1]
map (map (lookup "id") . tableRows . snd) db `shouldBe` [[Just (VInt 1)], [Just (VInt 2)]] map (map (lookup "id") . tableRows . snd) db `shouldBe` [[Just (VInt 1)], [Just (VInt 2)]]
it "rejects a bound query after a same-schema persistent table recreation" $ do
withTestSession $ \srv -> withServerEnv srv $ do
runIPCStorage (createTable "rebuilt" [("id", TInt)]) `shouldReturn` Right ()
Right catalog <- runIPCStorage schema
let bound = parseStatement "SELECT id FROM rebuilt" >>= Bound.bindQuery catalog
case bound of
Left err -> expectationFailure err
Right query -> do
runIPCStorage (Bound.executeQueryM query) `shouldReturn` Right []
runIPCStorage (dropTable "rebuilt") `shouldReturn` Right ()
runIPCStorage (createTable "rebuilt" [("id", TInt)]) `shouldReturn` Right ()
changed <- runIPCStorage (Bound.executeQueryM query)
changed `shouldSatisfy` either (isInfixOf "bound query object changed") (const False)
it "preserves persistent versions in snapshots and rejects restored schema" $ do
withTestSession $ \srv -> withServerEnv srv $ do
runIPCStorage (createTable "versioned" [("id", TInt)]) `shouldReturn` Right ()
Right catalog <- runIPCStorage schema
Right contents <- runIPCStorage snapshot
fmap tableSchemaVersion (lookup "versioned" catalog) `shouldBe` Just (Just 1)
fmap tableSchemaVersion (lookup "versioned" contents) `shouldBe` Just (Just 1)
case parseStatement "SELECT id FROM versioned" >>= Bound.bindQuery catalog of
Left err -> expectationFailure err
Right query -> do
runIPCStorage (replaceSchema "versioned" [("id", TInt), ("extra", TStr)] []) `shouldReturn` Right ()
runIPCStorage (replaceSchema "versioned" [("id", TInt)] []) `shouldReturn` Right ()
changed <- runIPCStorage (Bound.executeQueryM query)
changed `shouldSatisfy` either (isInfixOf "bound query schema changed") (const False)
it "rejects a DDL race between schema validation and the actual read" $ do
withTestSession $ \srv -> withServerEnv srv $ do
runIPCStorage (createTable "raced" [("id", TInt)]) `shouldReturn` Right ()
Right catalog <- runIPCStorage schema
closeConnection
Right link <- localStorageLink (Just (dataDir srv ++ ".toml"))
setStorageLink link {slRequest = \payload -> do
if BSC.pack "schema_guard" `BS.isInfixOf` payload
then do
changed <- slRequest link (BSC.pack "{\"method\":\"replace_schema\",\"database\":\"ipctest\",\"table\":\"raced\",\"columns\":[{\"name\":\"id\",\"ty\":\"int\"},{\"name\":\"extra\",\"ty\":\"str\"}],\"rows\":[]}")
changed `shouldSatisfy` either (const False) (BS.isInfixOf (BSC.pack "ok"))
else pure ()
slRequest link payload}
case parseStatement "SELECT id FROM raced" >>= Bound.bindQuery catalog of
Left err -> expectationFailure err
Right query -> do
result <- runIPCStorage (Bound.executeQueryM query)
result `shouldSatisfy` either (isInfixOf "bound query schema changed") (const False)
it "doListTables returns the tables that exist" $ do it "doListTables returns the tables that exist" $ do
withTestSession $ \srv -> withServerEnv srv $ do withTestSession $ \srv -> withServerEnv srv $ do
@@ -2529,7 +2642,7 @@ main = hspec $ do
`shouldBe` Right (makeSelect ["*"] "t" (Just (Eq (Col "flag") (LitBool True)))) `shouldBe` Right (makeSelect ["*"] "t" (Just (Eq (Col "flag") (LitBool True))))
it "inserts and reads back a boolean column end-to-end" $ do it "inserts and reads back a boolean column end-to-end" $ do
let db = [("t", Table "t" [("id", TInt), ("flag", TBool)] [] Nothing)] let db = [("t", Table "t" [("id", TInt), ("flag", TBool)] [] Nothing Nothing Nothing)]
case runStatement db (Insert "t" ["id", "flag"] [[LitInt 1, LitBool True]]) of case runStatement db (Insert "t" ["id", "flag"] [[LitInt 1, LitBool True]]) of
Left e -> expectationFailure e Left e -> expectationFailure e
Right (db', _) -> Right (db', _) ->
@@ -3154,6 +3267,8 @@ instance Monad ProbeStorage where
ProbeStorage (\probe -> act probe >>= \value -> runProbeStorage (next value) probe) ProbeStorage (\probe -> act probe >>= \value -> runProbeStorage (next value) probe)
instance MonadStorage ProbeStorage where instance MonadStorage ProbeStorage where
-- \| 测试视图不支持并发结构变更
withSchemaGuard _ action = action
scan table = ProbeStorage $ \probe -> do scan table = ProbeStorage $ \probe -> do
recordCall probe ("scan:" ++ table) recordCall probe ("scan:" ++ table)
pure (Right (tableRows (probeTable probe))) pure (Right (tableRows (probeTable probe)))
@@ -3227,7 +3342,10 @@ planTables op = case op of
Project _ x -> planTables x Project _ x -> planTables x
Compute _ x -> planTables x Compute _ x -> planTables x
Aggregate _ _ x -> planTables x Aggregate _ _ x -> planTables x
BoundAggregate _ _ x -> planTables x
Distinct x -> planTables x
Sort _ x -> planTables x Sort _ x -> planTables x
BoundSort _ x -> planTables x
Limit _ x -> planTables x Limit _ x -> planTables x
Join _ l r _ -> planTables l ++ planTables r Join _ l r _ -> planTables l ++ planTables r
Derived _ x -> planTables x Derived _ x -> planTables x
@@ -3239,8 +3357,11 @@ leftJoinKinds (Join kind l r _) = (kind == LeftJoin) : (leftJoinKinds l ++ leftJ
leftJoinKinds (Project _ x) = leftJoinKinds x leftJoinKinds (Project _ x) = leftJoinKinds x
leftJoinKinds (Compute _ x) = leftJoinKinds x leftJoinKinds (Compute _ x) = leftJoinKinds x
leftJoinKinds (Aggregate _ _ x) = leftJoinKinds x leftJoinKinds (Aggregate _ _ x) = leftJoinKinds x
leftJoinKinds (BoundAggregate _ _ x) = leftJoinKinds x
leftJoinKinds (Filter _ x) = leftJoinKinds x leftJoinKinds (Filter _ x) = leftJoinKinds x
leftJoinKinds (Distinct x) = leftJoinKinds x
leftJoinKinds (Sort _ x) = leftJoinKinds x leftJoinKinds (Sort _ x) = leftJoinKinds x
leftJoinKinds (BoundSort _ x) = leftJoinKinds x
leftJoinKinds (Limit _ x) = leftJoinKinds x leftJoinKinds (Limit _ x) = leftJoinKinds x
leftJoinKinds (Derived _ x) = leftJoinKinds x leftJoinKinds (Derived _ x) = leftJoinKinds x
leftJoinKinds _ = [] leftJoinKinds _ = []
@@ -3251,8 +3372,11 @@ firstFilterCond (Filter p _) = Just p
firstFilterCond (Project _ x) = firstFilterCond x firstFilterCond (Project _ x) = firstFilterCond x
firstFilterCond (Compute _ x) = firstFilterCond x firstFilterCond (Compute _ x) = firstFilterCond x
firstFilterCond (Aggregate _ _ x) = firstFilterCond x firstFilterCond (Aggregate _ _ x) = firstFilterCond x
firstFilterCond (BoundAggregate _ _ x) = firstFilterCond x
firstFilterCond Unit = Nothing firstFilterCond Unit = Nothing
firstFilterCond (Distinct x) = firstFilterCond x
firstFilterCond (Sort _ x) = firstFilterCond x firstFilterCond (Sort _ x) = firstFilterCond x
firstFilterCond (BoundSort _ x) = firstFilterCond x
firstFilterCond (Limit _ x) = firstFilterCond x firstFilterCond (Limit _ x) = firstFilterCond x
firstFilterCond (Join _ l r _) = case firstFilterCond l of firstFilterCond (Join _ l r _) = case firstFilterCond l of
Just p -> Just p Just p -> Just p
@@ -3268,8 +3392,11 @@ anyFilter (Filter _ _) = True
anyFilter (Project _ x) = anyFilter x anyFilter (Project _ x) = anyFilter x
anyFilter (Compute _ x) = anyFilter x anyFilter (Compute _ x) = anyFilter x
anyFilter (Aggregate _ _ x) = anyFilter x anyFilter (Aggregate _ _ x) = anyFilter x
anyFilter (BoundAggregate _ _ x) = anyFilter x
anyFilter Unit = False anyFilter Unit = False
anyFilter (Distinct x) = anyFilter x
anyFilter (Sort _ x) = anyFilter x anyFilter (Sort _ x) = anyFilter x
anyFilter (BoundSort _ x) = anyFilter x
anyFilter (Limit _ x) = anyFilter x anyFilter (Limit _ x) = anyFilter x
anyFilter (Join _ l r _) = anyFilter l || anyFilter r anyFilter (Join _ l r _) = anyFilter l || anyFilter r
anyFilter (Scan _ _ _) = False anyFilter (Scan _ _ _) = False
@@ -3282,9 +3409,12 @@ stripProjects :: RelOp -> RelOp
stripProjects (Project _ x) = stripProjects x stripProjects (Project _ x) = stripProjects x
stripProjects (Compute items x) = Compute items (stripProjects x) stripProjects (Compute items x) = Compute items (stripProjects x)
stripProjects (Aggregate keys aggs x) = Aggregate keys aggs (stripProjects x) stripProjects (Aggregate keys aggs x) = Aggregate keys aggs (stripProjects x)
stripProjects (BoundAggregate keys aggs x) = BoundAggregate keys aggs (stripProjects x)
stripProjects Unit = Unit stripProjects Unit = Unit
stripProjects (Filter p x) = Filter p (stripProjects x) stripProjects (Filter p x) = Filter p (stripProjects x)
stripProjects (Distinct x) = Distinct (stripProjects x)
stripProjects (Sort spec x) = Sort spec (stripProjects x) stripProjects (Sort spec x) = Sort spec (stripProjects x)
stripProjects (BoundSort spec x) = BoundSort spec (stripProjects x)
stripProjects (Limit n x) = Limit n (stripProjects x) stripProjects (Limit n x) = Limit n (stripProjects x)
stripProjects (Join k l r c) = Join k (stripProjects l) (stripProjects r) c stripProjects (Join k l r c) = Join k (stripProjects l) (stripProjects r) c
stripProjects (Scan a t _) = Scan a t Nothing stripProjects (Scan a t _) = Scan a t Nothing
@@ -3379,6 +3509,8 @@ memReplaceSchema :: String -> [(String, Column)] -> [Row] -> MemoryStorage (Eith
memReplaceSchema = replaceSchema memReplaceSchema = replaceSchema
instance MonadStorage Counting where instance MonadStorage Counting where
-- \| 纯测试状态在求值期间保持稳定
withSchemaGuard _ action = action
-- \| 扫描：记一笔表名，其余走内存实现 -- \| 扫描：记一笔表名，其余走内存实现
scan t = Counting $ \db -> case runMemoryStorage (memScan t) db of scan t = Counting $ \db -> case runMemoryStorage (memScan t) db of
Left e -> Left e Left e -> Left e
chusql-core/model/src/ChuSQL/Core/Model.hs
+8 -1
View File
@@ -12,6 +12,7 @@ module ChuSQL.Core.Model
, TableMeta (..) , TableMeta (..)
, Histogram (..) , Histogram (..)
, Table (..) , Table (..)
, TableIdentity (..)
, Database , Database
, histogramOf , histogramOf
, histogramFor , histogramFor
@@ -48,6 +49,7 @@ import Data.Char (isDigit, isSpace, toLower)
import Data.Hashable (Hashable (..)) import Data.Hashable (Hashable (..))
import Data.List (isPrefixOf, isSuffixOf, stripPrefix) import Data.List (isPrefixOf, isSuffixOf, stripPrefix)
import Data.Maybe (fromMaybe, mapMaybe) import Data.Maybe (fromMaybe, mapMaybe)
import Data.Word (Word64)
import ChuSQL.Core.Runtime import ChuSQL.Core.Runtime
import qualified Data.ByteString.Lazy.Char8 as BS import qualified Data.ByteString.Lazy.Char8 as BS
@@ -556,12 +558,17 @@ data TableMeta = TableMeta
} }
deriving (Show) deriving (Show)
-- | 一张表：名字、列定义与全部行；meta 为存储层统计，拿不到是 Nothing -- | 保存持久化数据库与表编号
data TableIdentity = TableIdentity Word64 Word64 deriving (Show, Eq)
-- | 一张表：列、行、统计与独立对象身份
data Table = Table data Table = Table
{ tableName :: String { tableName :: String
, tableCols :: [(String, Column)] , tableCols :: [(String, Column)]
, tableRows :: [Row] , tableRows :: [Row]
, tableMeta :: Maybe TableMeta , tableMeta :: Maybe TableMeta
, tableIdentity :: Maybe TableIdentity
, tableSchemaVersion :: Maybe Word64
} }
deriving (Show) deriving (Show)
chusql-core/model/src/ChuSQL/Core/Protocol.hs
+29 -3
View File
@@ -15,16 +15,18 @@ module ChuSQL.Core.Protocol (
queryResultJson, queryResultJson,
) where ) where
import ChuSQL.Core.Model (Histogram (..), Row, Value (..)) import ChuSQL.Core.Model (Histogram (..), Row, Value (..), TableIdentity (..))
import ChuSQL.Core.Runtime (builtinTypes, runtimeToJSON, runtimeFromJSON) import ChuSQL.Core.Runtime (builtinTypes, runtimeToJSON, runtimeFromJSON)
import Data.Aeson (FromJSON (..), ToJSON (..), object, withObject, (.:), (.:?), (.!=), (.=)) import Data.Aeson (FromJSON (..), ToJSON (..), object, withObject, (.:), (.:?), (.!=), (.=))
import qualified Data.Aeson as A import qualified Data.Aeson as A
import qualified Data.Aeson.Key as K import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM import qualified Data.Aeson.KeyMap as KM
import Data.Aeson.Types (Parser) import Data.Aeson.Types (Parser)
import Data.Scientific (floatingOrInteger, fromFloatDigits) import Data.Scientific (floatingOrInteger, fromFloatDigits, base10Exponent)
import qualified Data.Text as T import qualified Data.Text as T
import Data.Text (Text) import Data.Text (Text)
import Data.Word (Word64)
import Text.Read (readMaybe)
-- 线上协议类型：存储层请求/响应与查询结果集。 -- 线上协议类型：存储层请求/响应与查询结果集。
@@ -255,6 +257,8 @@ data TableInfo = TableInfo
, tiIndexes :: [String] , tiIndexes :: [String]
, tiStats :: [(String, Int, Bool)] , tiStats :: [(String, Int, Bool)]
, tiHistograms :: [(String, Histogram)] , tiHistograms :: [(String, Histogram)]
, tiIdentity :: Maybe TableIdentity
, tiSchemaVersion :: Maybe Word64
} }
deriving (Show, Eq) deriving (Show, Eq)
@@ -267,6 +271,9 @@ instance ToJSON TableInfo where
, "row_count" .= tiRows info , "row_count" .= tiRows info
, "indexes" .= map (\name -> object ["column" .= name]) (tiIndexes info) , "indexes" .= map (\name -> object ["column" .= name]) (tiIndexes info)
, "stats" .= map (statJson (tiHistograms info)) (tiStats info) , "stats" .= map (statJson (tiHistograms info)) (tiStats info)
, "database_id" .= fmap (\(TableIdentity database _) -> show database) (tiIdentity info)
, "object_id" .= fmap (\(TableIdentity _ table) -> show table) (tiIdentity info)
, "schema_version" .= fmap show (tiSchemaVersion info)
] ]
where where
statJson hists (name, distinct, capped) = statJson hists (name, distinct, capped) =
@@ -276,6 +283,16 @@ instance ToJSON TableInfo where
histFields Nothing = [] histFields Nothing = []
histFields (Just h) = ["lo" .= histLow h, "hi" .= histHigh h, "hist" .= histBuckets h] histFields (Just h) = ["lo" .= histLow h, "hi" .= histHigh h, "hist" .= histBuckets h]
-- | 解码精确且有界的对象编号
parseObjectNumber :: A.Value -> Parser Word64
parseObjectNumber (A.String value)
| not (T.null value), T.length value <= 20, T.all (\c -> c >= '0' && c <= '9') value
, Just number <- (readMaybe (T.unpack value) :: Maybe Integer)
, number <= toInteger (maxBound :: Word64) = pure (fromInteger number)
parseObjectNumber value@(A.Number number)
| base10Exponent number >= -20, base10Exponent number <= 20 = A.parseJSON value
parseObjectNumber _ = fail "invalid table object identity"
-- | 线上直方图还原；区间或桶缺了就当作没有 -- | 线上直方图还原；区间或桶缺了就当作没有
histogramFromWire :: Maybe Double -> Maybe Double -> [Int] -> Maybe Histogram histogramFromWire :: Maybe Double -> Maybe Double -> [Int] -> Maybe Histogram
histogramFromWire (Just low) (Just high) counts histogramFromWire (Just low) (Just high) counts
@@ -292,9 +309,18 @@ instance FromJSON TableInfo where
stats <- o .:? "stats" .!= [] stats <- o .:? "stats" .!= []
indexColumns <- mapM (\v -> withObject "IndexWire" (.: "column") v) (indexes :: [A.Value]) indexColumns <- mapM (\v -> withObject "IndexWire" (.: "column") v) (indexes :: [A.Value])
parsed <- mapM parseStat (stats :: [A.Value]) parsed <- mapM parseStat (stats :: [A.Value])
database <- o .:? "database_id" >>= traverse parseObjectNumber
tableId <- o .:? "object_id" >>= traverse parseObjectNumber
version <- o .:? "schema_version" >>= traverse parseObjectNumber
identity <- case (database, tableId) of
(Nothing, Nothing) -> pure Nothing
(Just _, Just 0) -> pure Nothing
(Just databaseId, Just objectId) | objectId > 0 -> pure (Just (TableIdentity databaseId objectId))
_ -> fail ("incomplete table object identity: " ++ show (database, tableId))
let parsedStats = [(name, distinct, capped) | (name, distinct, capped, _) <- parsed] let parsedStats = [(name, distinct, capped) | (name, distinct, capped, _) <- parsed]
parsedHistograms = [(name, h) | (name, _, _, Just h) <- parsed] parsedHistograms = [(name, h) | (name, _, _, Just h) <- parsed]
pure (TableInfo table columns count indexColumns parsedStats parsedHistograms) pure (TableInfo table columns count indexColumns parsedStats parsedHistograms identity
(version >>= \v -> if v == 0 then Nothing else Just v))
where where
parseStat = withObject "ColumnStat" $ \o -> do parseStat = withObject "ColumnStat" $ \o -> do
name <- o .: "name" name <- o .: "name"
chusql-core/storage/src/catalog.rs
+62 -28
View File
@@ -37,6 +37,9 @@ pub struct TableSchema {
#[serde(default)] #[serde(default)]
pub object_id: u64, pub object_id: u64,
#[serde(default)] #[serde(default)]
// 0 表示旧目录尚无可验证的结构版本。
pub schema_version: u64,
#[serde(default)]
pub system: bool, pub system: bool,
pub columns: Vec<SchemaColumn>, pub columns: Vec<SchemaColumn>,
#[serde(default)] #[serde(default)]
@@ -51,7 +54,7 @@ pub struct TableSchema {
pub stats: BTreeMap<String, ColumnStat>, pub stats: BTreeMap<String, ColumnStat>,
} }
#[derive(Debug, Default, Serialize, Deserialize)] #[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct Catalog { pub struct Catalog {
#[serde(default)] #[serde(default)]
pub database_id: u64, pub database_id: u64,
@@ -134,9 +137,10 @@ impl Catalog {
} }
/// 按一行数据补列 /// 按一行数据补列
pub fn ensure_columns(&mut self, table: &str, row: &Row) { pub fn ensure_columns(&mut self, table: &str, row: &Row) -> io::Result<()> {
let entry = self.tables.entry(table.to_string()).or_default(); let entry = self.tables.entry(table.to_string()).or_default();
add_missing_columns(entry, row); add_missing_columns(entry, row)?;
Ok(())
} }
/// 建表；已存在报错 /// 建表；已存在报错
@@ -163,6 +167,7 @@ impl Catalog {
table.to_string(), table.to_string(),
TableSchema { TableSchema {
object_id, object_id,
schema_version: 1,
system: false, system: false,
columns, columns,
dropped_columns: BTreeSet::new(), dropped_columns: BTreeSet::new(),
@@ -184,15 +189,15 @@ impl Catalog {
} }
/// 记一次插入，返回列有没有变化 /// 记一次插入，返回列有没有变化
pub fn record_insert(&mut self, table: &str, row: &Row) -> bool { pub fn record_insert(&mut self, table: &str, row: &Row) -> io::Result<bool> {
let changed = { let changed = {
let entry = self.tables.entry(table.to_string()).or_default(); let entry = self.tables.entry(table.to_string()).or_default();
let changed = add_missing_columns(entry, row); let changed = add_missing_columns(entry, row)?;
entry.row_count += 1; entry.row_count += 1;
changed changed
}; };
self.count_values(table, row); self.count_values(table, row);
changed Ok(changed)
} }
/// 记一次删除：行数减少 + 更新列统计 /// 记一次删除：行数减少 + 更新列统计
@@ -230,7 +235,7 @@ impl Catalog {
} }
/// 重数一遍统计与行数 /// 重数一遍统计与行数
pub fn rebuild_stats(&mut self, table: &str, rows: &[Row]) -> bool { pub fn rebuild_stats(&mut self, table: &str, rows: &[Row]) -> io::Result<bool> {
self.seen.remove(table); self.seen.remove(table);
{ {
let entry = self.tables.entry(table.to_string()).or_default(); let entry = self.tables.entry(table.to_string()).or_default();
@@ -251,9 +256,9 @@ impl Catalog {
} }
let mut changed = false; let mut changed = false;
for row in rows { for row in rows {
changed = self.record_insert(table, row) || changed; changed = self.record_insert(table, row)? || changed;
} }
changed Ok(changed)
} }
/// 这一列第一次见到这个值就加一 /// 这一列第一次见到这个值就加一
@@ -291,16 +296,18 @@ impl Catalog {
/// 表改名到指定名字 /// 表改名到指定名字
pub fn rename_table(&mut self, from: &str, to: &str) -> io::Result<bool> { pub fn rename_table(&mut self, from: &str, to: &str) -> io::Result<bool> {
let Some(entry) = self.tables.remove(from) else { let Some(entry) = self.tables.get(from) else {
return Ok(false); return Ok(false);
}; };
if self.tables.contains_key(to) { if self.tables.contains_key(to) {
self.tables.insert(from.to_string(), entry);
return Err(io::Error::new( return Err(io::Error::new(
io::ErrorKind::AlreadyExists, io::ErrorKind::AlreadyExists,
format!("table already exists: {}", to), format!("table already exists: {}", to),
)); ));
} }
let version = next_schema_version(entry)?;
let mut entry = self.tables.remove(from).ok_or_else(|| io::Error::other("missing rename source"))?;
entry.schema_version = version;
self.seen.remove(from); self.seen.remove(from);
self.tables.insert(to.to_string(), entry); self.tables.insert(to.to_string(), entry);
Ok(true) Ok(true)
@@ -308,10 +315,21 @@ impl Catalog {
/// ALTER：整列定义换成新的，并清掉二级索引与统计 /// ALTER：整列定义换成新的，并清掉二级索引与统计
pub fn set_columns(&mut self, table: &str, columns: Vec<SchemaColumn>) -> io::Result<()> { pub fn set_columns(&mut self, table: &str, columns: Vec<SchemaColumn>) -> io::Result<()> {
self.set_columns_version(table, columns, None)
}
/// 用日志中的目标版本替换列定义
pub fn set_columns_version(&mut self, table: &str, columns: Vec<SchemaColumn>, version: Option<u64>) -> io::Result<()> {
let entry = self let entry = self
.tables .tables
.get_mut(table) .get_mut(table)
.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?; .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?;
if let Some(version) = version {
if version == 0 { return Err(io::Error::other("invalid WAL schema version")); }
entry.schema_version = version;
} else if entry.columns != columns {
entry.schema_version = next_schema_version(entry)?;
}
entry.columns = columns; entry.columns = columns;
entry.dropped_columns.clear(); entry.dropped_columns.clear();
entry.indexes.clear(); entry.indexes.clear();
@@ -321,9 +339,12 @@ impl Catalog {
} }
/// 老数据的列类型名规范化；返回有没有改动 /// 老数据的列类型名规范化；返回有没有改动
pub fn normalize_types(&mut self) -> bool { pub fn normalize_types(&mut self) -> io::Result<bool> {
let mut changed = false; let mut changed = false;
for entry in self.tables.values_mut() { for entry in self.tables.values_mut() {
if entry.columns.iter().any(|column| normalize_type(&column.ty) != column.ty) {
entry.schema_version = next_schema_version(entry)?;
}
for column in entry.columns.iter_mut() { for column in entry.columns.iter_mut() {
let fixed = normalize_type(&column.ty); let fixed = normalize_type(&column.ty);
if fixed != column.ty { if fixed != column.ty {
@@ -332,15 +353,26 @@ impl Catalog {
} }
} }
} }
changed Ok(changed)
} }
/// 删一列；之前没有也返回成功（重放幂等） /// 删一列；之前没有也返回成功（重放幂等）
pub fn remove_column(&mut self, table: &str, column: &str) -> io::Result<bool> { pub fn remove_column(&mut self, table: &str, column: &str) -> io::Result<bool> {
self.remove_column_version(table, column, None)
}
/// 用日志目标版本删除列定义
pub fn remove_column_version(&mut self, table: &str, column: &str, version: Option<u64>) -> io::Result<bool> {
let entry = self let entry = self
.tables .tables
.get_mut(table) .get_mut(table)
.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?; .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?;
if let Some(version) = version {
if version == 0 { return Err(io::Error::other("invalid WAL schema version")); }
entry.schema_version = version;
} else if entry.columns.iter().any(|c| c.name == column) {
entry.schema_version = next_schema_version(entry)?;
}
let had_column = { let had_column = {
let before = entry.columns.len(); let before = entry.columns.len();
entry.columns.retain(|c| c.name != column); entry.columns.retain(|c| c.name != column);
@@ -417,22 +449,24 @@ impl Catalog {
} }
/// 把行里"字典还不知道的列"补进来；返回有没有变化 /// 把行里"字典还不知道的列"补进来；返回有没有变化
fn add_missing_columns(entry: &mut TableSchema, row: &Row) -> bool { fn add_missing_columns(entry: &mut TableSchema, row: &Row) -> io::Result<bool> {
let mut changed = false; let additions: Vec<_> = row.iter().filter(|(name, _)|
for (k, v) in row { !entry.dropped_columns.contains(*name) && !entry.columns.iter().any(|c| &c.name == *name))
if entry.dropped_columns.contains(k) || entry.columns.iter().any(|c| &c.name == k) { .filter_map(|(name, value)| infer_type(value).map(|ty| SchemaColumn {
continue; name: name.clone(), ty, ..Default::default()
})).collect();
if additions.is_empty() {
return Ok(false);
} }
if let Some(ty) = infer_type(v) { entry.schema_version = next_schema_version(entry)?;
entry.columns.push(SchemaColumn { entry.columns.extend(additions);
name: k.clone(), Ok(true)
ty, }
..Default::default()
}); /// 递增结构版本，耗尽时拒绝变更
changed = true; fn next_schema_version(entry: &TableSchema) -> io::Result<u64> {
} entry.schema_version.checked_add(1)
} .ok_or_else(|| io::Error::other("catalog schema version exhausted"))
changed
} }
/// 由 JSON 值猜列类型 /// 由 JSON 值猜列类型
chusql-core/storage/src/ffi.rs
+18 -16
View File
@@ -74,14 +74,19 @@ pub unsafe extern "C" fn chusql_storage_close(handle: *mut Storage) {
/// 离线维护系统目录并返回 JSON /// 离线维护系统目录并返回 JSON
/// ///
/// # Safety /// # Safety
/// 输入为有效字符串；输出指针可写，响应使用 free 释放。 /// `config_path` 为 NULL 或有效 NUL 结尾字符串；请求字节在调用期间有效。
/// 输出指针可写且不与输入重叠；响应使用 free 释放。
#[unsafe(no_mangle)] #[unsafe(no_mangle)]
pub unsafe extern "C" fn chusql_storage_maintenance( pub unsafe extern "C" fn chusql_storage_maintenance(
config_path: *const c_char, request: *const u8, request_len: usize, config_path: *const c_char, request: *const u8, request_len: usize,
out: *mut *mut u8, out_len: *mut usize, out: *mut *mut u8, out_len: *mut usize,
) -> i32 { ) -> i32 {
if request.is_null() || out.is_null() || out_len.is_null() { set_error("invalid maintenance argument"); return ERR; } if !out.is_null() { unsafe { *out = ptr::null_mut(); } }
unsafe { *out = ptr::null_mut(); *out_len = 0; } if !out_len.is_null() { unsafe { *out_len = 0; } }
if request.is_null() || out.is_null() || out_len.is_null() || request_len > isize::MAX as usize {
set_error("invalid maintenance argument");
return ERR;
}
let result = catch_unwind(AssertUnwindSafe(|| { let result = catch_unwind(AssertUnwindSafe(|| {
let path = if config_path.is_null() { None } else { Some(unsafe { CStr::from_ptr(config_path) }.to_str().map_err(|e| e.to_string())?) }; let path = if config_path.is_null() { None } else { Some(unsafe { CStr::from_ptr(config_path) }.to_str().map_err(|e| e.to_string())?) };
let text = std::str::from_utf8(unsafe { slice::from_raw_parts(request, request_len) }).map_err(|e| e.to_string())?; let text = std::str::from_utf8(unsafe { slice::from_raw_parts(request, request_len) }).map_err(|e| e.to_string())?;
@@ -92,16 +97,16 @@ pub unsafe extern "C" fn chusql_storage_maintenance(
Ok(Err(err)) => { set_error(err); return ERR; } Ok(Err(err)) => { set_error(err); return ERR; }
Err(_) => { set_error("panic in system maintenance"); return ERR; } Err(_) => { set_error("panic in system maintenance"); return ERR; }
}; };
let mut buffer = response.into_bytes().into_boxed_slice(); let buffer = response.into_bytes().into_boxed_slice();
unsafe { *out = buffer.as_mut_ptr(); *out_len = buffer.len(); } unsafe { *out_len = buffer.len(); *out = Box::into_raw(buffer) as *mut u8; }
std::mem::forget(buffer);
OK OK
} }
/// 处理一条请求，响应缓冲须归还 /// 处理一条请求，响应缓冲须归还
/// ///
/// # Safety /// # Safety
/// `handle` 必须是有效句柄；`request` 指向 `request_len` 个字节；`out`/`out_len` 可写。 /// `handle` 在调用期间有效且不能并发关闭；请求字节在调用期间有效。
/// `out`/`out_len` 可写且不与输入重叠；响应使用 free 释放。
#[unsafe(no_mangle)] #[unsafe(no_mangle)]
pub unsafe extern "C" fn chusql_storage_request( pub unsafe extern "C" fn chusql_storage_request(
handle: *const Storage, handle: *const Storage,
@@ -116,7 +121,7 @@ pub unsafe extern "C" fn chusql_storage_request(
if !out_len.is_null() { if !out_len.is_null() {
unsafe { *out_len = 0 }; unsafe { *out_len = 0 };
} }
if handle.is_null() || request.is_null() || out.is_null() || out_len.is_null() { if handle.is_null() || request.is_null() || out.is_null() || out_len.is_null() || request_len > isize::MAX as usize {
set_error("invalid argument"); set_error("invalid argument");
return ERR; return ERR;
} }
@@ -139,13 +144,10 @@ pub unsafe extern "C" fn chusql_storage_request(
} }
}; };
let mut buffer = response.into_bytes().into_boxed_slice(); let buffer = response.into_bytes().into_boxed_slice();
let ptr = buffer.as_mut_ptr();
let len = buffer.len();
std::mem::forget(buffer);
unsafe { unsafe {
*out = ptr; *out_len = buffer.len();
*out_len = len; *out = Box::into_raw(buffer) as *mut u8;
} }
OK OK
} }
@@ -153,7 +155,7 @@ pub unsafe extern "C" fn chusql_storage_request(
/// 归还响应缓冲 /// 归还响应缓冲
/// ///
/// # Safety /// # Safety
/// `data`/`len` 必须是 chusql_storage_request 原样返回且未被释放过的一对值。 /// `data`/`len` 为 request 或 maintenance 原样返回且未释放的一对值。
#[unsafe(no_mangle)] #[unsafe(no_mangle)]
pub unsafe extern "C" fn chusql_storage_free(data: *mut u8, len: usize) { pub unsafe extern "C" fn chusql_storage_free(data: *mut u8, len: usize) {
if data.is_null() { if data.is_null() {
@@ -168,7 +170,7 @@ pub unsafe extern "C" fn chusql_storage_free(data: *mut u8, len: usize) {
/// 取本线程最近的失败信息 /// 取本线程最近的失败信息
/// ///
/// # Safety /// # Safety
/// `out_len` 可写，或为 NULL。 /// `out_len` 可写，或为 NULL；返回字节仅在本线程下次设置错误前有效，不能释放。
#[unsafe(no_mangle)] #[unsafe(no_mangle)]
pub unsafe extern "C" fn chusql_storage_last_error(out_len: *mut usize) -> *const u8 { pub unsafe extern "C" fn chusql_storage_last_error(out_len: *mut usize) -> *const u8 {
LAST_ERROR.with(|slot| { LAST_ERROR.with(|slot| {
chusql-core/storage/src/protocol.rs
+30
View File
@@ -283,12 +283,15 @@ fn default_id_column() -> String {
#[derive(Debug, Serialize, Deserialize)] #[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "status", rename_all = "snake_case")] #[serde(tag = "status", rename_all = "snake_case")]
pub enum Response { pub enum Response {
Guarded { response: Box<Response> },
Accounts { accounts: Vec<Account> }, Accounts { accounts: Vec<Account> },
Pong, Pong,
Rows { rows: Vec<Row> }, Rows { rows: Vec<Row> },
Tables { tables: Vec<String> }, Tables { tables: Vec<String> },
Schema { Schema {
table: String, table: String,
#[serde(default, serialize_with = "serialize_version", deserialize_with = "deserialize_version")]
schema_version: u64,
columns: Vec<SchemaColumn>, columns: Vec<SchemaColumn>,
#[serde(default)] #[serde(default)]
row_count: u64, row_count: u64,
@@ -310,6 +313,15 @@ pub enum Response {
Object { database_id: u64, object_id: Option<u64> }, Object { database_id: u64, object_id: Option<u64> },
} }
#[derive(Debug, Deserialize)]
pub struct BoundSchema {
pub table: String,
pub database_id: Option<String>,
pub object_id: Option<String>,
pub schema_version: Option<String>,
pub columns: Vec<SchemaColumn>,
}
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TableSchemaWire { pub struct TableSchemaWire {
pub table: String, pub table: String,
@@ -317,6 +329,8 @@ pub struct TableSchemaWire {
pub database_id: u64, pub database_id: u64,
#[serde(default)] #[serde(default)]
pub object_id: u64, pub object_id: u64,
#[serde(default, serialize_with = "serialize_version", deserialize_with = "deserialize_version")]
pub schema_version: u64,
pub columns: Vec<SchemaColumn>, pub columns: Vec<SchemaColumn>,
#[serde(default)] #[serde(default)]
pub row_count: u64, pub row_count: u64,
@@ -325,3 +339,19 @@ pub struct TableSchemaWire {
#[serde(default)] #[serde(default)]
pub stats: Vec<ColumnStatWire>, pub stats: Vec<ColumnStatWire>,
} }
/// 将结构版本编码成精确十进制字符串
fn serialize_version<S: serde::Serializer>(version: &u64, serializer: S) -> Result<S::Ok, S::Error> {
serializer.serialize_str(&version.to_string())
}
/// 读取精确版本并兼容旧数值载荷
fn deserialize_version<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result<u64, D::Error> {
let value = serde_json::Value::deserialize(deserializer)?;
match value {
serde_json::Value::String(s) if !s.is_empty() && s.len() <= 20 && s.bytes().all(|c| c.is_ascii_digit()) =>
s.parse().map_err(serde::de::Error::custom),
serde_json::Value::Number(n) => n.as_u64().ok_or_else(|| serde::de::Error::custom("invalid schema version")),
_ => Err(serde::de::Error::custom("invalid schema version")),
}
}
chusql-core/storage/src/runtime.rs
+115 -24
View File
@@ -236,9 +236,9 @@ impl Server {
fn apply_op(&self, op: &WalOp) -> std::io::Result<()> { fn apply_op(&self, op: &WalOp) -> std::io::Result<()> {
match op { match op {
WalOp::Commit => Ok(()), WalOp::Commit => Ok(()),
WalOp::HideColumn { table, column } => { WalOp::HideColumn { table, column, schema_version } => {
let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?; let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
c.remove_column(table, column)?; c.remove_column_version(table, column, *schema_version)?;
c.save(self.catalog_path())?; c.save(self.catalog_path())?;
drop(c); drop(c);
if let Some(handle) = self.tables.lock().map_err(|_| std::io::Error::other("table lock poisoned"))?.get(table) { if let Some(handle) = self.tables.lock().map_err(|_| std::io::Error::other("table lock poisoned"))?.get(table) {
@@ -257,7 +257,7 @@ impl Server {
})?; })?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
c.mark_compacted(table)?; c.mark_compacted(table)?;
c.rebuild_stats(table, rows); c.rebuild_stats(table, rows)?;
c.save(self.catalog_path()) c.save(self.catalog_path())
} }
WalOp::Accounts { accounts, removed, migrate_roles } => { WalOp::Accounts { accounts, removed, migrate_roles } => {
@@ -287,20 +287,20 @@ impl Server {
} }
let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?; let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
declare_system_schema(&mut c, &rows)?; declare_system_schema(&mut c, &rows)?;
c.rebuild_stats(IDENTITIES_TABLE, &identities); c.rebuild_stats(IDENTITIES_TABLE, &identities)?;
c.set_columns(IDENTITIES_TABLE, identity_columns())?; c.set_columns(IDENTITIES_TABLE, identity_columns())?;
c.rebuild_stats(IDENTITIES_TABLE, &identities); c.rebuild_stats(IDENTITIES_TABLE, &identities)?;
c.mark_system(IDENTITIES_TABLE)?; c.mark_system(IDENTITIES_TABLE)?;
c.add_index(IDENTITIES_TABLE, "user")?; c.add_index(IDENTITIES_TABLE, "user")?;
c.identity_high_water = c.identity_high_water.max(accounts.iter().map(|a| a.id).max().unwrap_or(0)); c.identity_high_water = c.identity_high_water.max(accounts.iter().map(|a| a.id).max().unwrap_or(0));
if has_roles && *migrate_roles { c.rebuild_stats(legacy_roles, &[]); } if has_roles && *migrate_roles { c.rebuild_stats(legacy_roles, &[])?; }
for (table, records) in cleaned { c.rebuild_stats(table, &records); } for (table, records) in cleaned { c.rebuild_stats(table, &records)?; }
c.save(self.catalog_path()) c.save(self.catalog_path())
} }
WalOp::Insert { table, row } => { WalOp::Insert { table, row } => {
self.with_table(table, |t| t.insert_row(row))?; self.with_table(table, |t| t.insert_row(row))?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
if c.record_insert(table, row) { if c.record_insert(table, row)? {
c.save(self.catalog_path())?; c.save(self.catalog_path())?;
} }
Ok(()) Ok(())
@@ -310,7 +310,7 @@ impl Server {
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
let mut changed = false; let mut changed = false;
for r in rows { for r in rows {
if c.record_insert(table, r) { if c.record_insert(table, r)? {
changed = true; changed = true;
} }
} }
@@ -328,17 +328,17 @@ impl Server {
WalOp::ReplaceAll { table, rows } => { WalOp::ReplaceAll { table, rows } => {
self.with_table(table, |t| t.replace_all(rows))?; self.with_table(table, |t| t.replace_all(rows))?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
c.rebuild_stats(table, rows); c.rebuild_stats(table, rows)?;
c.save(self.catalog_path()) c.save(self.catalog_path())
} }
WalOp::DropColumn { table, column, rows } => { WalOp::DropColumn { table, column, rows, schema_version } => {
self.with_table(table, |t| { self.with_table(table, |t| {
t.detach_index(column); t.detach_index(column);
t.replace_all(rows) t.replace_all(rows)
})?; })?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
c.remove_column(table, column)?; c.remove_column_version(table, column, *schema_version)?;
c.rebuild_stats(table, rows); c.rebuild_stats(table, rows)?;
c.save(self.catalog_path())?; c.save(self.catalog_path())?;
drop(c); drop(c);
let _ = std::fs::remove_file(self.index_path(table, column)); let _ = std::fs::remove_file(self.index_path(table, column));
@@ -374,7 +374,7 @@ impl Server {
} }
WalOp::CreateIndex { table, column } => self.apply_create_index(table, column), WalOp::CreateIndex { table, column } => self.apply_create_index(table, column),
WalOp::DropIndex { table, column } => self.apply_drop_index(table, column), WalOp::DropIndex { table, column } => self.apply_drop_index(table, column),
WalOp::ReplaceSchema { table, columns, rows } => { WalOp::ReplaceSchema { table, columns, rows, schema_version } => {
let old_indexes: Vec<String> = { let old_indexes: Vec<String> = {
let c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?; let c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
c.describe(table) c.describe(table)
@@ -389,8 +389,8 @@ impl Server {
t.replace_all(rows) t.replace_all(rows)
})?; })?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
c.set_columns(table, columns.clone())?; c.set_columns_version(table, columns.clone(), *schema_version)?;
c.rebuild_stats(table, rows); c.rebuild_stats(table, rows)?;
c.save(self.catalog_path())?; c.save(self.catalog_path())?;
drop(c); drop(c);
for col in old_indexes { for col in old_indexes {
@@ -483,7 +483,7 @@ impl Server {
match op { match op {
WalOp::Insert { table, row } => validate(table, std::slice::from_ref(row))?, WalOp::Insert { table, row } => validate(table, std::slice::from_ref(row))?,
WalOp::InsertBatch { table, rows } | WalOp::ReplaceAll { table, rows } => validate(table, rows)?, WalOp::InsertBatch { table, rows } | WalOp::ReplaceAll { table, rows } => validate(table, rows)?,
WalOp::HideColumn { table, column } WalOp::HideColumn { table, column, .. }
if !c.describe(table).is_some_and(|s| s.columns.iter().any(|col| &col.name == column)) => if !c.describe(table).is_some_and(|s| s.columns.iter().any(|col| &col.name == column)) =>
{ {
return Err(std::io::Error::other(format!("unknown column: {column}"))); return Err(std::io::Error::other(format!("unknown column: {column}")));
@@ -554,6 +554,29 @@ impl Server {
} }
} }
} }
if ops.iter().any(|op| matches!(op, WalOp::HideColumn { .. } | WalOp::DropColumn { .. } | WalOp::ReplaceSchema { .. })) {
let mut preview = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?.clone();
for op in ops.to_mut() {
match op {
WalOp::HideColumn { table, column, schema_version }
| WalOp::DropColumn { table, column, schema_version, .. } => {
preview.remove_column(table, column)?;
*schema_version = Some(preview.describe(table).ok_or_else(|| std::io::Error::other("missing schema version"))?.schema_version.max(1));
}
WalOp::ReplaceSchema { table, columns, schema_version, .. } => {
preview.set_columns(table, columns.clone())?;
*schema_version = Some(preview.describe(table).ok_or_else(|| std::io::Error::other("missing schema version"))?.schema_version.max(1));
}
WalOp::CreateTable { table, columns, object_id } => preview.create_table_with_id(table, columns.clone(), *object_id)?,
WalOp::DropTable { table } => preview.drop_table(table)?,
WalOp::Insert { table, row } => preview.ensure_columns(table, row)?,
WalOp::InsertBatch { table, rows } | WalOp::ReplaceAll { table, rows } => {
for row in rows { preview.ensure_columns(table, row)?; }
}
_ => {}
}
}
}
let commit_lsn = wal.append_group(&ops)?; let commit_lsn = wal.append_group(&ops)?;
self.recovery_required.store(true, Ordering::Release); self.recovery_required.store(true, Ordering::Release);
for op in ops.iter() { for op in ops.iter() {
@@ -683,9 +706,10 @@ impl Server {
}); });
Ok(!present) Ok(!present)
} }
WalOp::HideColumn { table, column } => { WalOp::HideColumn { table, column, schema_version } => {
let c = self.catalog.lock().unwrap(); let c = self.catalog.lock().unwrap();
Ok(c.describe(table).is_some_and(|s| !s.columns.iter().any(|col| &col.name == column))) Ok(c.describe(table).is_some_and(|s| !s.columns.iter().any(|col| &col.name == column)
&& schema_version.is_none_or(|version| s.schema_version == version)))
} }
WalOp::CreateTable { table, object_id, .. } => { WalOp::CreateTable { table, object_id, .. } => {
let catalog = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?; let catalog = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
@@ -739,7 +763,7 @@ impl Server {
} }
let rows = self.with_table(&table, |t| t.scan())?; let rows = self.with_table(&table, |t| t.scan())?;
let mut c = self.catalog.lock().unwrap(); let mut c = self.catalog.lock().unwrap();
c.rebuild_stats(&table, &rows); c.rebuild_stats(&table, &rows)?;
touched = true; touched = true;
} }
if touched { if touched {
@@ -791,7 +815,7 @@ impl Server {
.catalog .catalog
.lock() .lock()
.map_err(|_| std::io::Error::other("catalog lock poisoned"))?; .map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
if catalog.normalize_types() { if catalog.normalize_types()? {
catalog.save(self.catalog_path())?; catalog.save(self.catalog_path())?;
} }
Ok(()) Ok(())
@@ -954,6 +978,7 @@ impl Server {
let op = WalOp::HideColumn { let op = WalOp::HideColumn {
table: table.to_string(), table: table.to_string(),
column: column.to_string(), column: column.to_string(),
schema_version: None,
}; };
self.write_via_wal(&op) self.write_via_wal(&op)
} }
@@ -1310,6 +1335,32 @@ impl Server {
} }
} }
/// 持读门时复查绑定身份、版本和列定义
fn validate_bound_schema(&self, table: &str, expected: &crate::protocol::BoundSchema) -> std::io::Result<()> {
use std::io::Error;
if self.recovery_required.load(Ordering::Acquire) { return Err(Error::other("storage recovery required")); }
if blocked_table(table) { return Err(Error::other("reserved system table")); }
let catalog = self.catalog.lock().map_err(|_| Error::other("catalog lock poisoned"))?;
let actual = catalog.describe(table).ok_or_else(|| Error::other(format!("bound query schema changed: {}", expected.table)))?;
match (&expected.database_id, &expected.object_id) {
(Some(database), Some(object)) => {
if parse_bound_number(database)? != catalog.database_id || parse_bound_number(object)? != actual.object_id {
return Err(Error::other(format!("bound query object changed: {}", expected.table)));
}
}
(None, None) => {}
_ => return Err(Error::other("incomplete bound object identity")),
}
if let Some(version) = &expected.schema_version
&& parse_bound_number(version)? != actual.schema_version {
return Err(Error::other(format!("bound query schema changed: {}", expected.table)));
}
if expected.columns != actual.columns {
return Err(Error::other(format!("bound query schema changed: {}", expected.table)));
}
Ok(())
}
/// 持门后分发一条请求 /// 持门后分发一条请求
fn dispatch(&self, req: Request) -> Response { fn dispatch(&self, req: Request) -> Response {
if self.recovery_required.load(Ordering::Acquire) && !matches!(req, Request::Ping) { if self.recovery_required.load(Ordering::Acquire) && !matches!(req, Request::Ping) {
@@ -1558,6 +1609,7 @@ impl Server {
table: name.to_string(), table: name.to_string(),
database_id: c.database_id, database_id: c.database_id,
object_id: ts.object_id, object_id: ts.object_id,
schema_version: ts.schema_version,
columns: ts.columns.clone(), columns: ts.columns.clone(),
row_count: ts.row_count, row_count: ts.row_count,
indexes: index_wires(ts), indexes: index_wires(ts),
@@ -1571,6 +1623,7 @@ impl Server {
log_debug!(request, "describe_table table={}", table); log_debug!(request, "describe_table table={}", table);
match self.describe_table(&table) { match self.describe_table(&table) {
Ok((cols, count, indexes, stats)) => Response::Schema { Ok((cols, count, indexes, stats)) => Response::Schema {
schema_version: self.catalog.lock().unwrap().describe(&table).map_or(0, |s| s.schema_version),
table, table,
columns: cols, columns: cols,
row_count: count, row_count: count,
@@ -1713,7 +1766,7 @@ impl Server {
Request::ReplaceSchema { table, columns, rows } => { Request::ReplaceSchema { table, columns, rows } => {
log_debug!(request, "replace_schema table={} cols={}", table, columns.len()); log_debug!(request, "replace_schema table={} cols={}", table, columns.len());
let tname = table.clone(); let tname = table.clone();
let op = WalOp::ReplaceSchema { table, columns, rows }; let op = WalOp::ReplaceSchema { table, columns, rows, schema_version: None };
match self.write_via_wal(&op) { match self.write_via_wal(&op) {
Ok(()) => Response::Ok, Ok(()) => Response::Ok,
Err(e) => { Err(e) => {
@@ -1838,9 +1891,9 @@ fn identity_columns() -> Vec<SchemaColumn> {
/// 把系统表写进字典 /// 把系统表写进字典
fn declare_system_schema(catalog: &mut Catalog, rows: &[Row]) -> std::io::Result<()> { fn declare_system_schema(catalog: &mut Catalog, rows: &[Row]) -> std::io::Result<()> {
catalog.rebuild_stats(USERS_TABLE, rows); catalog.rebuild_stats(USERS_TABLE, rows)?;
catalog.set_columns(USERS_TABLE, users_columns())?; catalog.set_columns(USERS_TABLE, users_columns())?;
catalog.rebuild_stats(USERS_TABLE, rows); catalog.rebuild_stats(USERS_TABLE, rows)?;
catalog.mark_system(USERS_TABLE)?; catalog.mark_system(USERS_TABLE)?;
Ok(()) Ok(())
} }
@@ -2137,6 +2190,10 @@ impl Databases {
fn request(&self, mut value: serde_json::Value) -> std::io::Result<Response> { fn request(&self, mut value: serde_json::Value) -> std::io::Result<Response> {
use std::io::Error; use std::io::Error;
let method = value.get("method").and_then(|v| v.as_str()).unwrap_or("").to_string(); let method = value.get("method").and_then(|v| v.as_str()).unwrap_or("").to_string();
if value.get("schema_guard").is_some()
&& !matches!(method.as_str(), "scan" | "scan_shard" | "lookup_by_index" | "range_by_index") {
return Err(Error::other("schema guard requires a table read"));
}
let name = match value.get("database") { let name = match value.get("database") {
None => String::new(), None => String::new(),
Some(serde_json::Value::String(s)) => s.to_ascii_lowercase(), Some(serde_json::Value::String(s)) => s.to_ascii_lowercase(),
@@ -2258,6 +2315,8 @@ impl Databases {
selected = database.to_ascii_lowercase(); selected = database.to_ascii_lowercase();
value["table"] = serde_json::Value::String(table.to_string()); value["table"] = serde_json::Value::String(table.to_string());
} }
let guard: Option<Vec<crate::protocol::BoundSchema>> = value.get("schema_guard")
.map(|guard| serde_json::from_value(guard.clone()).map_err(Error::other)).transpose()?;
let request: Request = serde_json::from_value(value).map_err(Error::other)?; let request: Request = serde_json::from_value(value).map_err(Error::other)?;
let global = matches!(request, Request::AccountsList | Request::CatalogManage { .. } | Request::AccountCreate { .. } let global = matches!(request, Request::AccountsList | Request::CatalogManage { .. } | Request::AccountCreate { .. }
| Request::IdentityInitialize { .. } | Request::RoleCreate { .. } | Request::IdentityAlter { .. } | Request::IdentityInitialize { .. } | Request::RoleCreate { .. } | Request::IdentityAlter { .. }
@@ -2268,11 +2327,43 @@ impl Databases {
if !global && selected.is_empty() { return Err(Error::other("no database selected")); } if !global && selected.is_empty() { return Err(Error::other("no database selected")); }
let server = if global || selected == SYSTEM_DATABASE { Arc::clone(&self.system) } let server = if global || selected == SYSTEM_DATABASE { Arc::clone(&self.system) }
else { Arc::clone(named.get(&selected).ok_or_else(|| Error::other("unknown database"))?) }; else { Arc::clone(named.get(&selected).ok_or_else(|| Error::other("unknown database"))?) };
if let Some(bindings) = guard {
if !concurrent_read(&request) { return Err(Error::other("schema guard requires a table read")); }
let mut servers = std::collections::BTreeMap::from([(selected.clone(), Arc::clone(&server))]);
let mut dependencies = Vec::new();
for binding in bindings {
let (database, table) = binding.table.split_once('.').unwrap_or((&selected, &binding.table));
let database = database.to_ascii_lowercase();
if !valid_database(&database) { return Err(Error::other("invalid bound database name")); }
let dependency = if database == SYSTEM_DATABASE { Arc::clone(&self.system) }
else { Arc::clone(named.get(&database).ok_or_else(|| Error::other("bound query database changed"))?) };
servers.insert(database.clone(), dependency);
dependencies.push((database, table.to_owned(), binding));
}
if !dependencies.iter().any(|(database, table, _)| database == &selected && Some(table.as_str()) == request.table()) {
return Err(Error::other("bound read table has no schema guard"));
}
drop(named);
let _gates = servers.values().map(|server| server.gate.read()
.map_err(|_| Error::other("storage read gate poisoned"))).collect::<std::io::Result<Vec<_>>>()?;
for (database, table, binding) in &dependencies {
servers[database].validate_bound_schema(table, binding)?;
}
return Ok(Response::Guarded { response: Box::new(server.dispatch(request)) });
}
drop(named); drop(named);
Ok(server.handle_request(request)) Ok(server.handle_request(request))
} }
} }
/// 读取守卫中的精确十进制编号
fn parse_bound_number(value: &str) -> std::io::Result<u64> {
if value.is_empty() || value.len() > 20 || !value.bytes().all(|c| c.is_ascii_digit()) {
return Err(std::io::Error::other("invalid bound catalog number"));
}
value.parse().map_err(std::io::Error::other)
}
/// 只读表页的请求可与其它读并发，其余独占 /// 只读表页的请求可与其它读并发，其余独占
fn concurrent_read(req: &Request) -> bool { fn concurrent_read(req: &Request) -> bool {
matches!( matches!(
chusql-core/storage/src/wal.rs
+31 -7
View File
@@ -29,7 +29,7 @@ const OP_DROP_INDEX: u8 = 14;
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub enum WalOp { pub enum WalOp {
HideColumn { table: String, column: String }, HideColumn { table: String, column: String, schema_version: Option<u64> },
Compact { table: String, rows: Vec<Row> }, Compact { table: String, rows: Vec<Row> },
Accounts { accounts: Vec<Account>, removed: Vec<String>, migrate_roles: bool }, Accounts { accounts: Vec<Account>, removed: Vec<String>, migrate_roles: bool },
Insert { Insert {
@@ -51,11 +51,13 @@ pub enum WalOp {
DropColumn { DropColumn {
table: String, table: String,
column: String, column: String,
schema_version: Option<u64>,
rows: Vec<Row>, rows: Vec<Row>,
}, },
ReplaceSchema { ReplaceSchema {
table: String, table: String,
columns: Vec<SchemaColumn>, columns: Vec<SchemaColumn>,
schema_version: Option<u64>,
rows: Vec<Row>, rows: Vec<Row>,
}, },
CreateTable { CreateTable {
@@ -78,15 +80,26 @@ pub enum WalOp {
Commit, Commit,
} }
#[derive(Debug, Serialize, Deserialize)]
struct HideColumnPayload {
column: String,
#[serde(default)]
schema_version: Option<u64>,
}
#[derive(Debug, Serialize, Deserialize)] #[derive(Debug, Serialize, Deserialize)]
struct DropColumnPayload { struct DropColumnPayload {
column: String, column: String,
#[serde(default)]
schema_version: Option<u64>,
rows: Vec<Row>, rows: Vec<Row>,
} }
#[derive(Debug, Serialize, Deserialize)] #[derive(Debug, Serialize, Deserialize)]
struct ReplaceSchemaPayload { struct ReplaceSchemaPayload {
columns: Vec<SchemaColumn>, columns: Vec<SchemaColumn>,
#[serde(default)]
schema_version: Option<u64>,
rows: Vec<Row>, rows: Vec<Row>,
} }
@@ -379,8 +392,8 @@ fn truncate(f: &mut File) -> io::Result<()> {
/// 把操作编成字节 /// 把操作编成字节
fn encode(op: &WalOp, lsn: u64) -> io::Result<Vec<u8>> { fn encode(op: &WalOp, lsn: u64) -> io::Result<Vec<u8>> {
let (op_type, table, payload) = match op { let (op_type, table, payload) = match op {
WalOp::HideColumn { table, column } => (OP_HIDE_COLUMN, table.clone(), WalOp::HideColumn { table, column, schema_version } => (OP_HIDE_COLUMN, table.clone(),
serde_json::to_vec(column).map_err(io::Error::other)?), serde_json::to_vec(&serde_json::json!({"column":column,"schema_version":schema_version})).map_err(io::Error::other)?),
WalOp::Compact { table, rows } => { WalOp::Compact { table, rows } => {
let p = serde_json::to_vec(rows).map_err(io::Error::other)?; let p = serde_json::to_vec(rows).map_err(io::Error::other)?;
(OP_COMPACT, table.clone(), p) (OP_COMPACT, table.clone(), p)
@@ -403,17 +416,19 @@ fn encode(op: &WalOp, lsn: u64) -> io::Result<Vec<u8>> {
let p = serde_json::to_vec(rows).map_err(io::Error::other)?; let p = serde_json::to_vec(rows).map_err(io::Error::other)?;
(OP_REPLACE_ALL, table.clone(), p) (OP_REPLACE_ALL, table.clone(), p)
} }
WalOp::DropColumn { table, column, rows } => { WalOp::DropColumn { table, column, rows, schema_version } => {
let p = serde_json::to_vec(&DropColumnPayload { let p = serde_json::to_vec(&DropColumnPayload {
column: column.clone(), column: column.clone(),
schema_version: *schema_version,
rows: rows.clone(), rows: rows.clone(),
}) })
.map_err(io::Error::other)?; .map_err(io::Error::other)?;
(OP_DROP_COLUMN, table.clone(), p) (OP_DROP_COLUMN, table.clone(), p)
} }
WalOp::ReplaceSchema { table, columns, rows } => { WalOp::ReplaceSchema { table, columns, rows, schema_version } => {
let p = serde_json::to_vec(&ReplaceSchemaPayload { let p = serde_json::to_vec(&ReplaceSchemaPayload {
columns: columns.clone(), columns: columns.clone(),
schema_version: *schema_version,
rows: rows.clone(), rows: rows.clone(),
}) })
.map_err(io::Error::other)?; .map_err(io::Error::other)?;
@@ -474,9 +489,16 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
let op = match op_type { let op = match op_type {
OP_COMMIT => WalOp::Commit, OP_COMMIT => WalOp::Commit,
OP_HIDE_COLUMN => { OP_HIDE_COLUMN => {
let column = serde_json::from_slice(payload) let value: serde_json::Value = serde_json::from_slice(payload)
.map_err(|e| format!("bad hide_column payload: {e}"))?; .map_err(|e| format!("bad hide_column payload: {e}"))?;
WalOp::HideColumn { table, column } let (column, schema_version) = if let Some(column) = value.as_str() {
(column.to_string(), None)
} else {
let p: HideColumnPayload = serde_json::from_value(value)
.map_err(|e| format!("bad hide_column payload: {e}"))?;
(p.column, p.schema_version)
};
WalOp::HideColumn { table, column, schema_version }
} }
OP_COMPACT => { OP_COMPACT => {
let rows: Vec<Row> = serde_json::from_slice(payload) let rows: Vec<Row> = serde_json::from_slice(payload)
@@ -521,6 +543,7 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
WalOp::DropColumn { WalOp::DropColumn {
table, table,
column: p.column, column: p.column,
schema_version: p.schema_version,
rows: p.rows, rows: p.rows,
} }
} }
@@ -530,6 +553,7 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
WalOp::ReplaceSchema { WalOp::ReplaceSchema {
table, table,
columns: p.columns, columns: p.columns,
schema_version: p.schema_version,
rows: p.rows, rows: p.rows,
} }
} }
chusql-core/storage/tests/catalog_test.rs
+78 -16
View File
@@ -4,6 +4,68 @@ use serde_json::json;
// 数据字典测试：补列、去重、未知表。 // 数据字典测试：补列、去重、未知表。
/// 结构变化递增版本，数据及重复应用不变
#[test]
fn schema_versions_track_structure_and_survive_reload() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("catalog.json");
let mut c = Catalog::default();
c.create_table("t", vec![]).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 1);
let values = row(&[("id", json!(1)), ("name", json!("a"))]);
c.record_insert("t", &values).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 2);
c.rebuild_stats("t", &[values]).unwrap();
let columns = c.describe("t").unwrap().columns.clone();
c.set_columns("t", columns).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 2);
c.remove_column("t", "name").unwrap();
c.remove_column("t", "name").unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 3);
c.rename_table("t", "renamed").unwrap();
c.save(&path).unwrap();
let c = Catalog::load(&path).unwrap();
assert_eq!(c.describe("renamed").unwrap().schema_version, 4);
}
/// 旧目录版本未知，首个结构变更才递增
#[test]
fn legacy_schema_version_remains_unknown_until_change() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("catalog.json");
std::fs::write(&path, json!({"tables": {
"t": {"object_id": 42, "columns": []}
}}).to_string()).unwrap();
let mut c = Catalog::load(&path).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 0);
c.save(&path).unwrap();
c = Catalog::load(&path).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 0);
c.ensure_columns("t", &row(&[("id", json!(1))])).unwrap();
assert_eq!(c.describe("t").unwrap().schema_version, 1);
assert_eq!(c.describe("t").unwrap().object_id, 42);
}
/// 版本耗尽拒绝结构变更并保留原定义
#[test]
fn exhausted_schema_version_preserves_structure() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("catalog.json");
std::fs::write(&path, json!({"tables": {
"t": {"object_id": 42, "schema_version": u64::MAX,
"columns": [{"name": "id", "ty": "int"}]}
}}).to_string()).unwrap();
let mut c = Catalog::load(&path).unwrap();
assert!(c.remove_column("t", "id").is_err());
assert!(c.set_columns("t", vec![]).is_err());
assert!(c.rename_table("t", "other").is_err());
assert!(c.ensure_columns("t", &row(&[("new", json!(1))])).is_err());
assert_eq!(c.describe("t").unwrap().columns.len(), 1);
assert_eq!(c.describe("t").unwrap().schema_version, u64::MAX);
assert!(c.describe("other").is_none());
c.record_insert("t", &row(&[("id", json!(1))])).unwrap();
}
/// 表删除重建和重载均保留编号规则 /// 表删除重建和重载均保留编号规则
#[test] #[test]
fn object_ids_survive_reload_and_are_not_reused() { fn object_ids_survive_reload_and_are_not_reused() {
@@ -80,7 +142,7 @@ fn ensure_then_describe() {
let path = dir.path().join("catalog.json"); let path = dir.path().join("catalog.json");
let mut c = Catalog::load(&path).unwrap(); let mut c = Catalog::load(&path).unwrap();
c.ensure_columns("users", &row(&[("id", json!(1)), ("name", json!("Alice"))])); c.ensure_columns("users", &row(&[("id", json!(1)), ("name", json!("Alice"))])).unwrap();
c.save(&path).unwrap(); c.save(&path).unwrap();
let c2 = Catalog::load(&path).unwrap(); let c2 = Catalog::load(&path).unwrap();
@@ -102,8 +164,8 @@ fn second_insert_does_not_duplicate_columns() {
let path = dir.path().join("catalog.json"); let path = dir.path().join("catalog.json");
let mut c = Catalog::load(&path).unwrap(); let mut c = Catalog::load(&path).unwrap();
c.ensure_columns("users", &row(&[("id", json!(1))])); c.ensure_columns("users", &row(&[("id", json!(1))])).unwrap();
c.ensure_columns("users", &row(&[("id", json!(2))])); c.ensure_columns("users", &row(&[("id", json!(2))])).unwrap();
assert_eq!(c.describe("users").unwrap().columns.len(), 1); assert_eq!(c.describe("users").unwrap().columns.len(), 1);
} }
@@ -114,9 +176,9 @@ fn remove_column_drops_definition_stats_and_index() {
let path = dir.path().join("catalog.json"); let path = dir.path().join("catalog.json");
let mut c = Catalog::load(&path).unwrap(); let mut c = Catalog::load(&path).unwrap();
c.ensure_columns("t", &row(&[("id", json!(1)), ("code", json!(5))])); c.ensure_columns("t", &row(&[("id", json!(1)), ("code", json!(5))])).unwrap();
c.add_index("t", "code").unwrap(); c.add_index("t", "code").unwrap();
c.rebuild_stats("t", &[row(&[("id", json!(1)), ("code", json!(5))])]); c.rebuild_stats("t", &[row(&[("id", json!(1)), ("code", json!(5))])]).unwrap();
let removed = c.remove_column("t", "code").unwrap(); let removed = c.remove_column("t", "code").unwrap();
assert!(removed, "first remove should hit the column"); assert!(removed, "first remove should hit the column");
@@ -142,7 +204,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
let rows: Vec<Row> = (0..100) let rows: Vec<Row> = (0..100)
.map(|n| row(&[("id", json!(n)), ("name", json!(format!("u{}", n)))])) .map(|n| row(&[("id", json!(n)), ("name", json!(format!("u{}", n)))]))
.collect(); .collect();
c.rebuild_stats("t", &rows); c.rebuild_stats("t", &rows).unwrap();
let s = c.describe("t").unwrap(); let s = c.describe("t").unwrap();
let id = s.stats.get("id").unwrap(); let id = s.stats.get("id").unwrap();
@@ -155,7 +217,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
assert!(name.hist.is_empty(), "字符串列不进直方图"); assert!(name.hist.is_empty(), "字符串列不进直方图");
assert_eq!(name.lo, None); assert_eq!(name.lo, None);
c.record_insert("t", &row(&[("id", json!(1000)), ("name", json!("x"))])); c.record_insert("t", &row(&[("id", json!(1000)), ("name", json!("x"))])).unwrap();
let s = c.describe("t").unwrap(); let s = c.describe("t").unwrap();
let id = s.stats.get("id").unwrap(); let id = s.stats.get("id").unwrap();
assert_eq!(id.lo, Some(0.0)); assert_eq!(id.lo, Some(0.0));
@@ -178,7 +240,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
fn duplicate_deletion_keeps_distinct_until_last_occurrence() { fn duplicate_deletion_keeps_distinct_until_last_occurrence() {
let mut catalog = Catalog::default(); let mut catalog = Catalog::default();
let duplicate = row(&[("id", json!(1)), ("v", json!(7))]); let duplicate = row(&[("id", json!(1)), ("v", json!(7))]);
catalog.rebuild_stats("t", &[duplicate.clone(), duplicate.clone()]); catalog.rebuild_stats("t", &[duplicate.clone(), duplicate.clone()]).unwrap();
catalog.record_delete("t", std::slice::from_ref(&duplicate)); catalog.record_delete("t", std::slice::from_ref(&duplicate));
assert_eq!(catalog.describe("t").unwrap().stats["v"].distinct, 1); assert_eq!(catalog.describe("t").unwrap().stats["v"].distinct, 1);
catalog.record_delete("t", &[duplicate]); catalog.record_delete("t", &[duplicate]);
@@ -190,7 +252,7 @@ fn duplicate_deletion_keeps_distinct_until_last_occurrence() {
fn capped_distinct_keeps_histogram_and_other_columns_accurate() { fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
let mut catalog = Catalog::default(); let mut catalog = Catalog::default();
let rows: Vec<Row> = (0..5000).map(|n| row(&[("id", json!(n)), ("v", json!(n % 2))])).collect(); let rows: Vec<Row> = (0..5000).map(|n| row(&[("id", json!(n)), ("v", json!(n % 2))])).collect();
catalog.rebuild_stats("t", &rows); catalog.rebuild_stats("t", &rows).unwrap();
let schema = catalog.describe("t").unwrap(); let schema = catalog.describe("t").unwrap();
assert!(schema.stats["id"].capped); assert!(schema.stats["id"].capped);
assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 5000); assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 5000);
@@ -199,7 +261,7 @@ fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
let schema = catalog.describe("t").unwrap(); let schema = catalog.describe("t").unwrap();
assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 4000); assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 4000);
assert_eq!(schema.stats["v"].distinct, 2); assert_eq!(schema.stats["v"].distinct, 2);
catalog.rebuild_stats("t", &rows[1000..]); catalog.rebuild_stats("t", &rows[1000..]).unwrap();
assert_eq!(catalog.describe("t").unwrap().stats["id"].distinct, 4000); assert_eq!(catalog.describe("t").unwrap().stats["id"].distinct, 4000);
} }
@@ -208,16 +270,16 @@ fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
fn rebuilt_histogram_has_exact_buckets_in_any_order() { fn rebuilt_histogram_has_exact_buckets_in_any_order() {
let rows: Vec<Row> = (0..160).map(|n| row(&[("v", json!(n))])).collect(); let rows: Vec<Row> = (0..160).map(|n| row(&[("v", json!(n))])).collect();
let mut forward = Catalog::default(); let mut forward = Catalog::default();
forward.rebuild_stats("t", &rows); forward.rebuild_stats("t", &rows).unwrap();
let mut reverse = Catalog::default(); let mut reverse = Catalog::default();
let reversed: Vec<Row> = rows.iter().rev().cloned().collect(); let reversed: Vec<Row> = rows.iter().rev().cloned().collect();
reverse.rebuild_stats("t", &reversed); reverse.rebuild_stats("t", &reversed).unwrap();
let stat = &forward.describe("t").unwrap().stats["v"]; let stat = &forward.describe("t").unwrap().stats["v"];
assert_eq!(stat.hist, vec![10; 16]); assert_eq!(stat.hist, vec![10; 16]);
assert_eq!(stat.hist, reverse.describe("t").unwrap().stats["v"].hist); assert_eq!(stat.hist, reverse.describe("t").unwrap().stats["v"].hist);
forward.record_insert("t", &row(&[("v", json!(10000))])); forward.record_insert("t", &row(&[("v", json!(10000))])).unwrap();
forward.record_delete("t", &[row(&[("v", json!(10000))])]); forward.record_delete("t", &[row(&[("v", json!(10000))])]);
forward.rebuild_stats("t", &rows[80..]); forward.rebuild_stats("t", &rows[80..]).unwrap();
let repaired = &forward.describe("t").unwrap().stats["v"]; let repaired = &forward.describe("t").unwrap().stats["v"];
assert_eq!(repaired.lo, Some(80.0)); assert_eq!(repaired.lo, Some(80.0));
assert_eq!(repaired.hi, Some(159.0)); assert_eq!(repaired.hi, Some(159.0));
@@ -253,8 +315,8 @@ fn legacy_catalog_loads_and_normalizes() {
let s = c.describe("legacy").unwrap(); let s = c.describe("legacy").unwrap();
assert_eq!(s.columns.len(), 2); assert_eq!(s.columns.len(), 2);
assert!(s.columns[0].nullable, "老列默认可空"); assert!(s.columns[0].nullable, "老列默认可空");
assert!(c.normalize_types()); assert!(c.normalize_types().unwrap());
let types: Vec<&str> = c.describe("legacy").unwrap().columns.iter().map(|col| col.ty.as_str()).collect(); let types: Vec<&str> = c.describe("legacy").unwrap().columns.iter().map(|col| col.ty.as_str()).collect();
assert_eq!(types, vec!["int", "str"]); assert_eq!(types, vec!["int", "str"]);
assert!(!c.normalize_types(), "第二次没有可改的"); assert!(!c.normalize_types().unwrap(), "第二次没有可改的");
} }
chusql-core/storage/tests/ffi_test.rs
+91
View File
@@ -0,0 +1,91 @@
use std::ffi::CString;
use std::ptr;
use std::slice;
use chusql_core_storage::ffi::*;
use chusql_core_storage::runtime::Storage;
// 验证 FFI 错误输出、长度边界与响应所有权。
/// 读取当前线程的 FFI 错误
fn last_error() -> String {
let mut len = 0;
// SAFETY: 长度可写，返回缓冲在复制前没有被修改。
unsafe {
let data = chusql_storage_last_error(&mut len);
assert!(!data.is_null());
String::from_utf8(slice::from_raw_parts(data, len).to_vec()).unwrap()
}
}
/// 维护失败清空输出并拒绝超长请求
#[test]
fn maintenance_rejects_invalid_arguments_and_resets_outputs() {
let input = b"{}";
let mut out = input.as_ptr() as *mut u8;
let mut len = 99;
// SAFETY: 输出均可写；无效输入应在任何读取之前被拒绝。
unsafe {
assert_eq!(chusql_storage_maintenance(ptr::null(), ptr::null(), 0, &mut out, &mut len), -1);
assert!(out.is_null());
assert_eq!(len, 0);
len = 99;
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), input.len(), ptr::null_mut(), &mut len), -1);
assert_eq!(len, 0);
out = input.as_ptr() as *mut u8;
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), input.len(), &mut out, ptr::null_mut()), -1);
assert!(out.is_null());
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), usize::MAX, &mut out, &mut len), -1);
}
assert!(out.is_null());
assert_eq!(len, 0);
assert_eq!(last_error(), "invalid maintenance argument");
}
/// 请求验证长度和编码并归还响应
#[test]
fn request_validates_input_and_returns_owned_response() -> Result<(), Box<dyn std::error::Error>> {
let temp = tempfile::tempdir()?;
let config = temp.path().join("settings.toml");
std::fs::write(&config, format!("[storage]\ndata_dir = '{}'\nlog_files = '{}'\n[log]\nlevel = 'off'\n", temp.path().join("data").display(), temp.path().join("logs").display()))?;
let storage = Storage::open(config.to_str())?;
let input = br#"{"method":"ping"}"#;
let mut out = ptr::null_mut();
let mut len = 0;
// SAFETY: 句柄和实际请求有效；超长长度必须在构造切片前被拒绝。
unsafe {
assert_eq!(chusql_storage_request(&storage, input.as_ptr(), usize::MAX, &mut out, &mut len), -1);
assert!(out.is_null());
assert_eq!(len, 0);
assert_eq!(last_error(), "invalid argument");
assert_eq!(chusql_storage_request(&storage, [0xff].as_ptr(), 1, &mut out, &mut len), -1);
assert!(last_error().starts_with("request is not valid UTF-8:"));
assert_eq!(chusql_storage_request(&storage, input.as_ptr(), input.len(), &mut out, &mut len), 0);
assert!(!out.is_null());
let response = slice::from_raw_parts(out, len).to_vec();
chusql_storage_free(out, len);
let response: serde_json::Value = serde_json::from_slice(&response)?;
assert_eq!(response["status"], "pong");
}
drop(storage);
let config = CString::new(config.to_str().unwrap())?;
// SAFETY: 路径与输出有效；失败不产生需要释放的缓冲。
unsafe {
assert_eq!(chusql_storage_maintenance(config.as_ptr(), [0xff].as_ptr(), 1, &mut out, &mut len), -1);
}
assert!(out.is_null());
assert_eq!(len, 0);
assert!(!last_error().is_empty());
let maintenance = br#"{"mode":"reset","user":"root","password_hash":"hash","types":[{"name":"int","base_type":"int","parameterized":false}]}"#;
// SAFETY: 维护仅使用本测试临时目录；成功响应复制后由配套函数释放。
unsafe {
assert_eq!(chusql_storage_maintenance(config.as_ptr(), maintenance.as_ptr(), maintenance.len(), &mut out, &mut len), 0, "{}", last_error());
assert!(!out.is_null());
let response = slice::from_raw_parts(out, len).to_vec();
chusql_storage_free(out, len);
let response: serde_json::Value = serde_json::from_slice(&response)?;
assert_eq!(response["status"], "maintenance");
assert_eq!(response["mode"], "reset");
}
Ok(())
}
chusql-core/storage/tests/runtime_test.rs
+126 -1
View File
@@ -171,6 +171,131 @@ fn object_id_survives_committed_create_recovery() -> Result<(), Box<dyn std::err
Ok(()) Ok(())
} }
/// 读取复查跨库依赖而非只复查扫描表
#[test]
fn bound_reads_recheck_dependencies_in_other_databases() -> Result<(), Box<dyn std::error::Error>> {
let (_server, _data) = start_server();
let mut c = connect()?;
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"source","columns":[]}))?;
request_ok(&mut c, serde_json::json!({"method":"create_database","database":"dependency"}))?;
request_ok(&mut c, serde_json::json!({"method":"create_table","database":"dependency","table":"related","columns":[]}))?;
let catalog: serde_json::Value = serde_json::from_str(&send_raw(&mut c, r#"{"method":"all_catalogs"}"#))?;
let bindings: Vec<_> = catalog["schemas"].as_array().unwrap().iter().filter(|entry|
entry["table"] == "main.source" || entry["table"] == "dependency.related").map(|entry| {
let mut binding = entry.clone();
binding["database_id"] = binding["database_id"].as_u64().unwrap().to_string().into();
binding["object_id"] = binding["object_id"].as_u64().unwrap().to_string().into();
binding
}).collect();
assert_eq!(bindings.len(), 2);
let request = serde_json::json!({"method":"scan","table":"source","schema_guard":bindings});
let valid: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
assert_eq!(valid["status"], "guarded");
request_ok(&mut c, serde_json::json!({"method":"replace_schema","database":"dependency","table":"related","columns":[{"name":"new","ty":"int"}],"rows":[]}))?;
let invalid: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
assert_eq!(invalid["status"], "error");
assert!(invalid["message"].as_str().unwrap().contains("bound query schema changed: dependency.related"));
Ok(())
}
/// 每种读取都复查守卫，拒绝旧版本和写入
#[test]
fn bound_reads_validate_dependencies_and_acknowledge_guards() -> Result<(), Box<dyn std::error::Error>> {
let (_server, _data) = start_server();
let mut c = connect()?;
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"guarded","columns":[{"name":"id","ty":"int"}]}))?;
request_ok(&mut c, serde_json::json!({"method":"insert","table":"guarded","row":{"id":7}}))?;
let catalog: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"list_catalog"}"#))?;
let mut binding = catalog["schemas"][0].clone();
binding["database_id"] = binding["database_id"].as_u64().unwrap().to_string().into();
binding["object_id"] = binding["object_id"].as_u64().unwrap().to_string().into();
for mut request in [
serde_json::json!({"method":"scan","table":"guarded"}),
serde_json::json!({"method":"scan_shard","table":"guarded","shard":0,"shards":1}),
serde_json::json!({"method":"lookup_by_index","table":"guarded","column":"id","key":7}),
serde_json::json!({"method":"range_by_index","table":"guarded","column":"id","lo":null,"hi":null}),
] {
request["schema_guard"] = serde_json::json!([binding]);
let response: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
assert_eq!(response["status"], "guarded", "{response}");
assert!(response["response"]["status"] == "rows" || response["response"]["status"] == "no_index", "{response}");
}
request_ok(&mut c, serde_json::json!({"method":"replace_schema","table":"guarded","columns":[{"name":"id","ty":"int"},{"name":"extra","ty":"str"}],"rows":[{"id":7,"extra":"x"}]}))?;
let stale: serde_json::Value = serde_json::from_str(&send(&mut c, &serde_json::json!({"method":"scan","table":"guarded","schema_guard":[binding]}).to_string()))?;
assert_eq!(stale["status"], "error");
assert!(stale["message"].as_str().unwrap().contains("bound query schema changed"));
let write: serde_json::Value = serde_json::from_str(&send(&mut c, &serde_json::json!({"method":"insert","table":"guarded","row":{"id":9},"schema_guard":[binding]}).to_string()))?;
assert_eq!(write["status"], "error");
let management: serde_json::Value = serde_json::from_str(&send_raw(&mut c, &serde_json::json!({"method":"create_database","database":"blocked","schema_guard":[]}).to_string()))?;
assert_eq!(management["status"], "error");
let databases: serde_json::Value = serde_json::from_str(&send_raw(&mut c, r#"{"method":"list_databases"}"#))?;
assert!(!databases["tables"].as_array().unwrap().iter().any(|database| database == "blocked"));
let rows: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"scan","table":"guarded"}"#))?;
assert_eq!(rows["rows"].as_array().unwrap().len(), 1);
Ok(())
}
/// 描述与目录均透传精确版本字符串
#[test]
fn catalog_schema_versions_are_exact_and_validated() -> Result<(), Box<dyn std::error::Error>> {
use chusql_core_storage::protocol::TableSchemaWire;
let (_server, _data) = start_server();
let mut c = connect()?;
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"versioned","columns":[{"name":"id","ty":"int"}]}))?;
let describe: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"describe_table","table":"versioned"}"#))?;
assert_eq!(describe["schema_version"], "1");
let catalog: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"list_catalog"}"#))?;
assert_eq!(catalog["schemas"][0]["schema_version"], "1");
let wire: TableSchemaWire = serde_json::from_value(serde_json::json!({"table":"t","columns":[],"schema_version":u64::MAX.to_string()}))?;
assert_eq!(wire.schema_version, u64::MAX);
assert_eq!(serde_json::to_value(wire)?["schema_version"], u64::MAX.to_string());
for version in [serde_json::json!(-1), serde_json::json!(1.5), serde_json::json!("18446744073709551616"), serde_json::json!("+1")] {
assert!(serde_json::from_value::<TableSchemaWire>(serde_json::json!({"table":"t","columns":[],"schema_version":version})).is_err());
}
Ok(())
}
/// 检查点失败及反复恢复保留结构版本
#[test]
fn schema_versions_survive_checkpoint_failure_and_repeated_recovery() -> Result<(), Box<dyn std::error::Error>> {
use chusql_core_storage::catalog::Catalog;
use chusql_core_storage::wal::{Wal, WalOp};
let (server, data) = start_server();
let mut c = connect()?;
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"versioned","columns":[{"name":"id","ty":"int"}]}))?;
let directory = data.path().join("databases/main");
let pending = directory.join("wal.checkpoint.pending");
std::fs::create_dir(&pending)?;
let response = send(&mut c, r#"{"method":"replace_schema","table":"versioned","columns":[{"name":"id","ty":"int"},{"name":"tag","ty":"str"}],"rows":[]}"#);
assert!(response.contains("error"), "{response}");
let catalog = Catalog::load(directory.join("catalog.json"))?;
let version = catalog.describe("versioned").unwrap().schema_version;
assert_eq!(version, 2);
let wal = Wal::new(directory.join("wal.log"));
let records = wal.read_all()?.records;
assert!(records.iter().any(|(_, op)| matches!(op, WalOp::ReplaceSchema { schema_version: Some(2), .. })));
let columns = catalog.describe("versioned").unwrap().columns.clone();
drop(wal);
drop(c);
drop(server);
std::fs::remove_dir(pending)?;
for _ in 0..2 {
let wal = Wal::new(directory.join("wal.log"));
wal.append_group(&[
WalOp::ReplaceSchema { table: "versioned".into(), columns: vec![], rows: vec![], schema_version: Some(1) },
WalOp::ReplaceSchema { table: "versioned".into(), columns: columns.clone(), rows: vec![], schema_version: Some(version) },
])?;
drop(wal);
let restarted = restart_at(data.path())?;
let catalog = Catalog::load(directory.join("catalog.json"))?;
assert_eq!(catalog.describe("versioned").unwrap().schema_version, version);
assert_eq!(catalog.describe("versioned").unwrap().columns, columns);
assert_eq!(std::fs::metadata(directory.join("wal.log"))?.len(), 0);
drop(restarted);
}
Ok(())
}
/// 并发建表分配互不重复的编号 /// 并发建表分配互不重复的编号
#[test] #[test]
fn concurrent_table_creation_allocates_distinct_object_ids() -> Result<(), Box<dyn std::error::Error>> { fn concurrent_table_creation_allocates_distinct_object_ids() -> Result<(), Box<dyn std::error::Error>> {
@@ -1026,7 +1151,7 @@ fn hidden_column_wal_replays_twice_without_touching_heap() -> Result<(), Box<dyn
let mut compacted: Option<Vec<u8>> = None; let mut compacted: Option<Vec<u8>> = None;
for round in 0..2 { for round in 0..2 {
let wal = Wal::new(data.path().join("databases/main/wal.log")); let wal = Wal::new(data.path().join("databases/main/wal.log"));
let op = WalOp::HideColumn { table: "items".into(), column: "secret".into() }; let op = WalOp::HideColumn { table: "items".into(), column: "secret".into(), schema_version: None };
wal.truncate()?; wal.truncate()?;
wal.append(&op)?; wal.append(&op)?;
assert_eq!(wal.read()?, Some(op)); assert_eq!(wal.read()?, Some(op));
chusql-core/storage/tests/wal_test.rs
+28
View File
@@ -4,6 +4,33 @@ use serde_json::json;
// WAL 测试：追加、读取、清空、截断尾巴、LSN 与提交标记。 // WAL 测试：追加、读取、清空、截断尾巴、LSN 与提交标记。
/// 结构版本往返且旧隐藏列载荷可读
#[test]
fn schema_versions_round_trip_and_legacy_hide_decodes() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("wal.log");
let wal = Wal::new(&path);
let ops = [
WalOp::HideColumn { table: "t".into(), column: "name".into(), schema_version: Some(u64::MAX) },
WalOp::DropColumn { table: "t".into(), column: "name".into(), rows: vec![], schema_version: Some(3) },
WalOp::ReplaceSchema { table: "t".into(), columns: vec![], rows: vec![], schema_version: Some(4) },
];
wal.append_group(&ops).unwrap();
assert_eq!(committed_ops(&wal.read_all().unwrap().records).into_iter().map(|(_, op)| op).collect::<Vec<_>>(), ops);
drop(wal);
let mut body = 1_u64.to_le_bytes().to_vec();
body.push(8);
body.extend_from_slice(&1_u16.to_le_bytes());
body.extend_from_slice(b"t");
body.extend_from_slice(br#""name""#);
let mut frame = (body.len() as u32).to_le_bytes().to_vec();
frame.extend_from_slice(&body);
std::fs::write(&path, frame).unwrap();
assert_eq!(Wal::new(&path).read().unwrap(), Some(WalOp::HideColumn {
table: "t".into(), column: "name".into(), schema_version: None,
}));
}
/// 建表编号完整往返且兼容旧载荷 /// 建表编号完整往返且兼容旧载荷
#[test] #[test]
fn create_table_ids_round_trip_and_legacy_payloads_decode() { fn create_table_ids_round_trip_and_legacy_payloads_decode() {
@@ -58,6 +85,7 @@ fn append_and_read_round_trips() {
WalOp::DropColumn { WalOp::DropColumn {
table: "users".into(), table: "users".into(),
column: "age".into(), column: "age".into(),
schema_version: None,
rows: vec![ rows: vec![
row(&[("id", json!(1)), ("name", json!("Alice"))]), row(&[("id", json!(1)), ("name", json!("Alice"))]),
row(&[("id", json!(2)), ("name", json!("Bob"))]), row(&[("id", json!(2)), ("name", json!("Bob"))]),
chusql-interface/chusql-interface.cabal
+1
View File
@@ -16,6 +16,7 @@ library
ChuSQL.Interface.Actions ChuSQL.Interface.Actions
ChuSQL.Interface.Auth ChuSQL.Interface.Auth
ChuSQL.Interface.Config ChuSQL.Interface.Config
ChuSQL.Interface.Normalization
ChuSQL.Interface.Link ChuSQL.Interface.Link
ChuSQL.Interface.Policy ChuSQL.Interface.Policy
ChuSQL.Interface.Protocol ChuSQL.Interface.Protocol
chusql-interface/src/ChuSQL/Interface/AccountTable.hs
+1 -1
View File
@@ -43,4 +43,4 @@ accountColumnSpec name ty nullable autoIncrement primary unique =
-- | 账号表的结构（排序校验用） -- | 账号表的结构（排序校验用）
systemTableInfo :: TableInfo systemTableInfo :: TableInfo
systemTableInfo = TableInfo accountTableName accountColumns 0 [] [] [] systemTableInfo = TableInfo accountTableName accountColumns 0 [] [] [] Nothing Nothing
chusql-interface/src/ChuSQL/Interface/Link.hs
+12
View File
@@ -6,6 +6,7 @@ module ChuSQL.Interface.Link (
closeClient, closeClient,
clientLogin, clientLogin,
clientSudoLogin, clientSudoLogin,
clientNormalization,
clientQuery, clientQuery,
clientPing, clientPing,
clientCatalog, clientCatalog,
@@ -16,6 +17,8 @@ module ChuSQL.Interface.Link (
clientReloadPolicy, clientReloadPolicy,
) where ) where
import ChuSQL.Interface.Normalization (NormalizationRequest)
import qualified Data.Aeson as A
import ChuSQL.Interface.Sudo (SudoCredential, sudoProof, isLocalPeer) import ChuSQL.Interface.Sudo (SudoCredential, sudoProof, isLocalPeer)
import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo (..)) import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo (..))
import ChuSQL.Interface.Protocol ( import ChuSQL.Interface.Protocol (
@@ -244,3 +247,12 @@ describe :: IOException -> String
describe err describe err
| isEOFError err = "the server closed the connection" | isEOFError err = "the server closed the connection"
| otherwise = show err | otherwise = show err
-- 请求服务端执行只读范式分析。
clientNormalization :: Client -> NormalizationRequest -> IO (Either Text A.Value)
clientNormalization client payload = do
answer <- request client (ReqNormalization payload)
pure $ case answer of
Left message -> Left message
Right (RespNormalization value) -> Right value
Right other -> failed "normalization" other
chusql-interface/src/ChuSQL/Interface/Normalization.hs
+188
View File
@@ -0,0 +1,188 @@
{-# LANGUAGE OverloadedStrings #-}
module ChuSQL.Interface.Normalization where
import ChuSQL.Core.Protocol (SchemaColumn)
import Control.Monad (unless, when)
import Data.Aeson (FromJSON (..), ToJSON (..), object, withObject, (.:), (.:?), (.!=), (.=))
import qualified Data.Aeson as A
import qualified Data.Aeson.Key as K
import qualified Data.Aeson.KeyMap as KM
import qualified Data.Aeson.Types as AT
import qualified Data.ByteString.Lazy as BL
import Data.Char (isAlpha, isAlphaNum, isDigit, isHexDigit, isSpace)
import Data.List (nub)
import Data.Scientific (base10Exponent, toBoundedInteger)
import Data.Word (Word64)
import Data.Text (Text)
import qualified Data.Text as T
import qualified Data.Text.Encoding as TE
import Text.ParserCombinators.ReadP
-- 范式入口的版本化声明、对象身份与严格 JSON 校验。
data NormalizationDependency = NormalizationDependency [String] [String] deriving (Eq, Show)
data NormalizationDeclaration = NormalizationDeclaration
{ ndKeys :: [[String]], ndDependencies :: [NormalizationDependency]
, ndComplete :: Bool, ndAtomic :: Maybe Bool
} deriving (Eq, Show)
data ObjectIdentity = ObjectIdentity Integer Integer deriving (Eq, Show)
data NormalizationRequest = NormalizationRequest
{ nrVersion :: Int, nrDatabase :: Text, nrTable :: Text
, nrDeclaration :: NormalizationDeclaration
, nrExpectedObject :: Maybe ObjectIdentity, nrExpectedSchema :: Maybe Text
} deriving (Eq, Show)
-- 拒绝对象中未定义的字段。
onlyFields :: [Text] -> A.Object -> AT.Parser ()
onlyFields names fields = unless (all ((`elem` names) . K.toText) (KM.keys fields)) (fail "normalization: unknown field")
instance FromJSON NormalizationDependency where
-- 读取依赖两端的确切字段名。
parseJSON = withObject "dependency" $ \o -> do
onlyFields ["lhs", "rhs"] o
lhs <- o .: "lhs"
rhs <- o .: "rhs"
when (null rhs) (fail "normalization: empty dependency rhs")
pure (NormalizationDependency lhs rhs)
instance ToJSON NormalizationDependency where
-- 编码声明依赖。
toJSON (NormalizationDependency lhs rhs) = object ["lhs" .= lhs, "rhs" .= rhs]
instance FromJSON NormalizationDeclaration where
-- 读取显式声明并限制数量。
parseJSON = withObject "declaration" $ \o -> do
onlyFields ["declared_keys", "dependencies", "dependencies_complete", "atomic_attributes"] o
keys <- o .:? "declared_keys" .!= []
deps <- o .:? "dependencies" .!= []
when (length keys > 256 || length deps > 256) (fail "normalization: declaration exceeds 256 entries")
NormalizationDeclaration keys deps <$> o .:? "dependencies_complete" .!= False <*> o .:? "atomic_attributes"
instance ToJSON NormalizationDeclaration where
-- 编码全部声明标志。
toJSON d = object ["declared_keys" .= ndKeys d, "dependencies" .= ndDependencies d,
"dependencies_complete" .= ndComplete d, "atomic_attributes" .= ndAtomic d]
instance FromJSON ObjectIdentity where
-- 校验稳定对象的正整数编号。
parseJSON = withObject "object identity" $ \o -> do
onlyFields ["database_id", "object_id"] o
db <- o .: "database_id" >>= identityNumber
table <- o .: "object_id" >>= identityNumber
when (db <= 0 || table <= 0) (fail "normalization: invalid object identity")
pure (ObjectIdentity db table)
-- 兼容目录数值并保留浏览器中的完整编号。
identityNumber :: A.Value -> AT.Parser Integer
identityNumber (A.String text)
| not (T.null text) && T.length text <= 20 && T.all isDigit text = case reads (T.unpack text) of
[(value, "")] | value <= toInteger (maxBound :: Word64) -> pure value
_ -> fail "normalization: invalid identity number"
identityNumber (A.Number number)
| base10Exponent number >= -20 && base10Exponent number <= 20 =
maybe (fail "normalization: invalid identity number") (pure . toInteger) (toBoundedInteger number :: Maybe Word64)
identityNumber _ = fail "normalization: invalid identity number"
instance ToJSON ObjectIdentity where
-- 编码对象的库及表编号。
toJSON (ObjectIdentity db table) = object ["database_id" .= T.pack (show db), "object_id" .= T.pack (show table)]
instance FromJSON NormalizationRequest where
-- 校验版本、范围和预期身份。
parseJSON = withObject "normalization request" $ \o -> do
onlyFields ["api_version", "database", "table", "declaration", "expected_object", "expected_schema_fingerprint"] o
version <- o .: "api_version"
when (version /= (1 :: Int)) (fail "normalization: unsupported api_version")
db <- o .: "database"
table <- o .: "table"
unless (plainName db && case T.splitOn "." table of [name] -> plainName name; [scope, name] -> scope == db && plainName name; _ -> False)
(fail "normalization: invalid or cross-database table name")
NormalizationRequest version db table <$> o .: "declaration" <*> o .:? "expected_object" <*> o .:? "expected_schema_fingerprint"
instance ToJSON NormalizationRequest where
-- 编码专用分析请求。
toJSON r = object ["api_version" .= nrVersion r, "database" .= nrDatabase r, "table" .= nrTable r,
"declaration" .= nrDeclaration r, "expected_object" .= nrExpectedObject r, "expected_schema_fingerprint" .= nrExpectedSchema r]
-- 校验与 SQL 一致的裸标识符。
plainName :: Text -> Bool
plainName name = not (T.null name) && T.length name <= 64 && (isAlpha (T.head name) || T.head name == '_') && T.all (\c -> isAlphaNum c || c == '_') name
-- 检查大小、编码及重复 JSON 字段。
decodeNormalizationJSON :: BL.ByteString -> Either Text A.Value
decodeNormalizationJSON raw
| BL.length raw > 65536 = Left "normalization: request exceeds 64 KiB"
| otherwise = do
value <- either (Left . T.pack) Right (A.eitherDecode raw)
text <- either (const (Left "normalization: invalid UTF-8")) Right (TE.decodeUtf8' (BL.toStrict raw))
case readP_to_S (skipSpaces *> jsonNode 0 <* skipSpaces <* eof) (T.unpack text) of
[(True, "")] -> Right value
_ -> Left "normalization: duplicate JSON fields or nesting exceeds 64"
-- 遍历 JSON 结构并保留原始字段名。
jsonNode :: Int -> ReadP Bool
jsonNode depth
| depth > 64 = pfail
| otherwise = objectNode <++ arrayNode <++ (True <$ jsonString) <++ (True <$ munch1 (\c -> not (isSpace c) && c `notElem` (",]}" :: String)))
where
-- 验证对象及所有嵌套值。
objectNode = do
_ <- char '{'
skipSpaces
pairs <- sepBy pair (skipSpaces *> char ',' <* skipSpaces)
skipSpaces
_ <- char '}'
pure (length (map fst pairs) == length (nub (map fst pairs)) && all snd pairs)
-- 保留键名并验证对应值。
pair = do
key <- jsonString
skipSpaces
_ <- char ':'
skipSpaces
valid <- jsonNode (depth + 1)
pure (key, valid)
-- 验证数组的所有元素。
arrayNode = do
_ <- char '['
skipSpaces
values <- sepBy (jsonNode (depth + 1)) (skipSpaces *> char ',' <* skipSpaces)
skipSpaces
_ <- char ']'
pure (and values)
-- 解码字符串以识别转义后的相同键。
jsonString :: ReadP Text
jsonString = do
_ <- char '"'
pieces <- many piece
_ <- char '"'
case A.eitherDecodeStrict (TE.encodeUtf8 (T.pack ('"' : concat pieces ++ "\""))) of
Right value -> pure value
Left _ -> pfail
where
-- 保留 JSON 字符串转义。
piece = ((\c -> ['\\', c]) <$> (char '\\' *> get)) <++ munch1 (\c -> c /= '"' && c /= '\\')
-- 拒绝缺字段或未知标签的线上报告。
parseNormalizationReport :: A.Value -> AT.Parser A.Value
parseNormalizationReport value = withObject "normalization report" (\o -> do
version <- o .: "api_version"
conditional <- o .: "conditional"
unless (version == (1 :: Int) && conditional) (fail "normalization: invalid report version or condition")
_ <- (o .: "object" :: AT.Parser ObjectIdentity)
fingerprint <- o .: "schema_fingerprint"
unless (T.length fingerprint == 64 && T.all isHexDigit fingerprint) (fail "normalization: invalid schema fingerprint")
view <- o .: "view"
unless (view `elem` (["live_schema", "transaction_snapshot"] :: [Text])) (fail "normalization: unknown view")
_ <- (o .: "schema" :: AT.Parser [SchemaColumn])
_ <- (o .: "declaration" :: AT.Parser NormalizationDeclaration)
_ <- (o .: "candidate_keys" :: AT.Parser [[String]])
forms <- o .: "findings" >>= mapM parseFinding
unless (forms == (["1nf", "2nf", "3nf", "bcnf"] :: [Text])) (fail "normalization: invalid findings")
_ <- (o .: "suggestions" :: AT.Parser [A.Value])
_ <- (o .: "condition" :: AT.Parser Text)
pure value) value
where
-- 核对每项结论与证据的稳定形状。
parseFinding = withObject "normalization finding" $ \o -> do
form <- o .: "normal_form"
verdict <- o .: "verdict"
unless (verdict `elem` (["satisfied", "violated", "insufficient"] :: [Text])) (fail "normalization: unknown verdict")
_ <- (o .: "witnesses" :: AT.Parser [NormalizationDependency])
_ <- (o .: "condition" :: AT.Parser Text)
pure form
chusql-interface/src/ChuSQL/Interface/Protocol.hs
+12
View File
@@ -13,6 +13,8 @@ module ChuSQL.Interface.Protocol (
RoleView (..), RoleView (..),
) where ) where
import ChuSQL.Interface.Normalization
import Data.Aeson.Types (parseEither)
import ChuSQL.Core.Protocol (Account (..), QueryResult (..), TableInfo (..), queryResultJson) import ChuSQL.Core.Protocol (Account (..), QueryResult (..), TableInfo (..), queryResultJson)
import Data.Aeson (FromJSON (..), ToJSON (..), Value, object, withObject, (.:), (.:?), (.!=), (.=)) import Data.Aeson (FromJSON (..), ToJSON (..), Value, object, withObject, (.:), (.:?), (.!=), (.=))
import qualified Data.Aeson as A import qualified Data.Aeson as A
@@ -93,6 +95,7 @@ data ClientRequest
| ReqLogin Text Text | ReqLogin Text Text
| ReqSudoChallenge | ReqSudoChallenge
| ReqSudoLogin Text Text | ReqSudoLogin Text Text
| ReqNormalization NormalizationRequest
| ReqQuery Text | ReqQuery Text
| ReqPing | ReqPing
| ReqQuit | ReqQuit
@@ -141,6 +144,9 @@ decodeRequest raw = case A.eitherDecodeStrict raw of
"sudo_login" -> case (wrUser wire, wrProof wire) of "sudo_login" -> case (wrUser wire, wrProof wire) of
(Just user, Just proof) -> Right (ReqSudoLogin user proof) (Just user, Just proof) -> Right (ReqSudoLogin user proof)
_ -> Left "sudo_login needs both user and proof" _ -> Left "sudo_login needs both user and proof"
"normalization_analyze" -> do
value <- decodeNormalizationJSON (BL.fromStrict raw)
either (Left . T.pack) Right $ parseEither (withObject "normalization envelope" $ \o -> onlyFields ["method", "request"] o >> ReqNormalization <$> o .: "request") value
"query" -> maybe (Left "query needs sql") (Right . ReqQuery) (wrSql wire) "query" -> maybe (Left "query needs sql") (Right . ReqQuery) (wrSql wire)
"storage" -> maybe (Left "storage needs request") (Right . ReqStorage) (wrRequest wire) "storage" -> maybe (Left "storage needs request") (Right . ReqStorage) (wrRequest wire)
"catalog" -> Right ReqCatalog "catalog" -> Right ReqCatalog
@@ -164,6 +170,7 @@ requestJson request = case request of
ReqLogin user password -> object ["method" .= ("login" :: Text), "user" .= user, "password" .= password] ReqLogin user password -> object ["method" .= ("login" :: Text), "user" .= user, "password" .= password]
ReqSudoChallenge -> object ["method" .= ("sudo_challenge" :: Text)] ReqSudoChallenge -> object ["method" .= ("sudo_challenge" :: Text)]
ReqSudoLogin user proof -> object ["method" .= ("sudo_login" :: Text), "user" .= user, "proof" .= proof] ReqSudoLogin user proof -> object ["method" .= ("sudo_login" :: Text), "user" .= user, "proof" .= proof]
ReqNormalization payload -> object ["method" .= ("normalization_analyze" :: Text), "request" .= payload]
ReqQuery sql -> object ["method" .= ("query" :: Text), "sql" .= sql] ReqQuery sql -> object ["method" .= ("query" :: Text), "sql" .= sql]
ReqPing -> object ["method" .= ("ping" :: Text)] ReqPing -> object ["method" .= ("ping" :: Text)]
ReqQuit -> object ["method" .= ("quit" :: Text)] ReqQuit -> object ["method" .= ("quit" :: Text)]
@@ -184,6 +191,7 @@ data ServerResponse
| RespPong | RespPong
| RespBye | RespBye
| RespError Text Text | RespError Text Text
| RespNormalization Value
| RespStorage Value | RespStorage Value
| RespCatalog [TableInfo] | RespCatalog [TableInfo]
| RespDatabases [Text] | RespDatabases [Text]
@@ -205,6 +213,7 @@ responseJson response = case response of
RespPong -> object ["status" .= ("pong" :: Text)] RespPong -> object ["status" .= ("pong" :: Text)]
RespBye -> object ["status" .= ("bye" :: Text)] RespBye -> object ["status" .= ("bye" :: Text)]
RespError code message -> object ["status" .= ("error" :: Text), "code" .= code, "message" .= message] RespError code message -> object ["status" .= ("error" :: Text), "code" .= code, "message" .= message]
RespNormalization value -> object ["status" .= ("normalization" :: Text), "analysis" .= value]
RespStorage value -> object ["status" .= ("storage" :: Text), "response" .= value] RespStorage value -> object ["status" .= ("storage" :: Text), "response" .= value]
RespCatalog infos -> object ["status" .= ("catalog" :: Text), "schemas" .= infos] RespCatalog infos -> object ["status" .= ("catalog" :: Text), "schemas" .= infos]
RespDatabases names -> object ["status" .= ("databases" :: Text), "databases" .= names] RespDatabases names -> object ["status" .= ("databases" :: Text), "databases" .= names]
@@ -238,6 +247,9 @@ decodeResponse raw = case A.eitherDecode raw of
"result" -> case A.fromJSON (A.Object fields) of "result" -> case A.fromJSON (A.Object fields) of
A.Success result -> Right (RespResult result) A.Success result -> Right (RespResult result)
A.Error message -> Left (T.pack message) A.Error message -> Left (T.pack message)
"normalization" -> case KM.lookup "analysis" fields of
Nothing -> Left "normalization response is missing analysis"
Just value -> either (Left . T.pack) (Right . RespNormalization) (parseEither parseNormalizationReport value)
"catalog" -> fromField "schemas" RespCatalog fields "catalog" -> fromField "schemas" RespCatalog fields
"databases" -> fromField "databases" RespDatabases fields "databases" -> fromField "databases" RespDatabases fields
"roles" -> fromField "roles" RespRoles fields "roles" -> fromField "roles" RespRoles fields
chusql-interface/src/ChuSQL/Interface/Session.hs
+11
View File
@@ -10,15 +10,19 @@ module ChuSQL.Interface.Session (
authenticateSudoSession, authenticateSudoSession,
switchDatabase, switchDatabase,
runStatement, runStatement,
analyzeNormalization,
catalog, catalog,
databases, databases,
roleViews, roleViews,
accounts, accounts,
) where ) where
import ChuSQL.Interface.Normalization
import qualified Data.Aeson as A
import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo) import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo)
import ChuSQL.Interface.Link ( import ChuSQL.Interface.Link (
Client, Client,
clientNormalization,
clientAccounts, clientAccounts,
clientCatalog, clientCatalog,
clientDatabases, clientDatabases,
@@ -119,3 +123,10 @@ roleViews session = clientRoles (ssClient session)
-- | 账号清单（\\du 与管理页用） -- | 账号清单（\\du 与管理页用）
accounts :: Session -> IO (Either Text [Account]) accounts :: Session -> IO (Either Text [Account])
accounts session = clientAccounts (ssClient session) accounts session = clientAccounts (ssClient session)
-- 使用当前连接及工作库请求范式分析。
analyzeNormalization :: Session -> NormalizationRequest -> IO (Either Text A.Value)
analyzeNormalization session payload = do
current <- sessionDatabase session
if current /= nrDatabase payload then pure (Left "bad_request: normalization database differs from the selected database")
else clientNormalization (ssClient session) payload
chusql-server/chusql-server.cabal
+1
View File
@@ -14,6 +14,7 @@ library
exposed-modules: ChuSQL.Server.Accounts exposed-modules: ChuSQL.Server.Accounts
ChuSQL.Server.Backend ChuSQL.Server.Backend
ChuSQL.Server.Catalog ChuSQL.Server.Catalog
ChuSQL.Server.Normalization
ChuSQL.Server.Policy ChuSQL.Server.Policy
ChuSQL.Server.Privileges ChuSQL.Server.Privileges
ChuSQL.Server.ObjectACL ChuSQL.Server.ObjectACL
chusql-server/src/ChuSQL/Server/Backend.hs
+22 -10
View File
@@ -41,7 +41,7 @@ import ChuSQL.Core.Engine.Syntax.AST
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement) import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
import ChuSQL.Core.Protocol (TxnOp (..)) import ChuSQL.Core.Protocol (TxnOp (..))
import Control.Concurrent.MVar (MVar, modifyMVar, readMVar) import Control.Concurrent.MVar (MVar, modifyMVar, readMVar)
import Data.List (nub) import Data.List (nub, isPrefixOf)
import Data.Text (Text) import Data.Text (Text)
import Data.Time (getCurrentTime) import Data.Time (getCurrentTime)
import Data.Time.Format (defaultTimeLocale, formatTime) import Data.Time.Format (defaultTimeLocale, formatTime)
@@ -75,6 +75,7 @@ data Backend = Backend
-- | 按给定列名取列，固定列顺序 -- | 按给定列名取列，固定列顺序
columnsFromStatement :: Database -> Statement -> [String] columnsFromStatement :: Database -> Statement -> [String]
columnsFromStatement db stmt = case prepare db stmt of columnsFromStatement db stmt = case prepare db stmt of
Right (SelectDistinct q) -> columnsFromStatement db q
Right (Select cols _ _ _ _ _) -> cols Right (Select cols _ _ _ _ _) -> cols
Right (SelectExpr items _ _ _ _ _) -> map fst items Right (SelectExpr items _ _ _ _ _) -> map fst items
Right ShowDomains -> ["domain", "base_type"] Right ShowDomains -> ["domain", "base_type"]
@@ -173,9 +174,9 @@ memoryBackendWith trusted name ref =
Right (db', rows) Right (db', rows)
| any (\(key, _) -> hidden key && not (domainKey key)) db' -> pure (db, Left "reserved system table") | any (\(key, _) -> hidden key && not (domainKey key)) db' -> pure (db, Left "reserved system table")
| otherwise -> pure (syncMemoryObjects name (mergeScoped db db'), Right (StatementResult (colsOf (scoped db) stmt rows) rows)) | otherwise -> pure (syncMemoryObjects name (mergeScoped db db'), Right (StatementResult (colsOf (scoped db) stmt rows) rows))
, beCatalog = do , beCatalog = modifyMVar ref $ \original -> do
db <- readMVar ref let db = syncMemoryObjects name original
pure (Right (map tableInfoOf (filter (not . internalTable . fst) (scoped db)))) pure (db, Right (map tableInfoOf (filter (not . internalTable . fst) (scoped db))))
, bePing = pure True , bePing = pure True
, beAccounts = \req -> do , beAccounts = \req -> do
stamp <- currentStamp stamp <- currentStamp
@@ -187,7 +188,9 @@ memoryBackendWith trusted name ref =
, beWithDatabase = withDatabase , beWithDatabase = withDatabase
, beDatabases = pure (Right (nub [name, systemDatabaseName])) , beDatabases = pure (Right (nub [name, systemDatabaseName]))
, beStorage = memorySecurityRequest name ref , beStorage = memorySecurityRequest name ref
, beSnapshot = Right . scoped <$> readMVar ref , beSnapshot = modifyMVar ref $ \original -> do
let db = syncMemoryObjects name original
pure (db, Right (scoped db))
, beApplyTransaction = \ops -> modifyMVar ref $ \db -> case applyTransactionMemory ops (scoped db) of , beApplyTransaction = \ops -> modifyMVar ref $ \db -> case applyTransactionMemory ops (scoped db) of
Left err -> pure (db, Left err) Left err -> pure (db, Left err)
Right db' -> pure (mergeScoped db db', Right ()) Right db' -> pure (mergeScoped db db', Right ())
@@ -226,7 +229,8 @@ memoryBackendWith trusted name ref =
-- 更新内存夹具的稳定表编号。 -- 更新内存夹具的稳定表编号。
syncMemoryObjects :: String -> Database -> Database syncMemoryObjects :: String -> Database -> Database
syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id", TInt)] records Nothing) : filter ((/= registry) . fst) db syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id", TInt)] records Nothing Nothing Nothing)
: map identify (filter ((/= registry) . fst) db)
where where
registry = "__system_object_ids" registry = "__system_object_ids"
previous = [(key, identifier) | row <- maybe [] tableRows (lookup registry db), Just (VStr key) <- [lookup "name" row], Just (VInt identifier) <- [lookup "id" row]] previous = [(key, identifier) | row <- maybe [] tableRows (lookup registry db), Just (VStr key) <- [lookup "name" row], Just (VInt identifier) <- [lookup "id" row]]
@@ -234,6 +238,11 @@ syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id"
retained = [(key, identifier) | (key, identifier) <- previous, key `elem` keys] retained = [(key, identifier) | (key, identifier) <- previous, key `elem` keys]
high = maximum (0 : map snd previous) high = maximum (0 : map snd previous)
added = zip [key | key <- keys, key `notElem` map fst retained] [high + 1 ..] added = zip [key | key <- keys, key `notElem` map fst retained] [high + 1 ..]
-- | 将稳定目录编号附到表定义
identify (key, table) = case lookup (if '.' `elem` key then key else database ++ "." ++ key) (retained ++ added) of
Just identifier -> (key, table {tableIdentity = Just (TableIdentity
(if "system." `isPrefixOf` key || ('.' `notElem` key && database == "system") then 2 else 1) (fromIntegral identifier))})
Nothing -> (key, table)
records = [[("name", VStr key), ("id", VInt identifier)] | (key, identifier) <- ("__high_water", high + length added) : retained ++ added] records = [[("name", VStr key), ("id", VInt identifier)] | (key, identifier) <- ("__high_water", high + length added) : retained ++ added]
-- 执行内存夹具的权限快照协议。 -- 执行内存夹具的权限快照协议。
@@ -266,7 +275,7 @@ memorySecurityRequest database ref payload = modifyMVar ref $ \original -> do
let previous = case lookup acl db of Nothing -> Nothing; Just table -> case tableRows table of [[("payload", VStr value)]] -> Just (T.pack value); _ -> Just "invalid ACL" let previous = case lookup acl db of Nothing -> Nothing; Just table -> case tableRows table of [[("payload", VStr value)]] -> Just (T.pack value); _ -> Just "invalid ACL"
if previous /= field "expected" then Left "object ACL changed concurrently" else do if previous /= field "expected" then Left "object ACL changed concurrently" else do
value <- maybe (Left "missing ACL payload") Right (field "payload") value <- maybe (Left "missing ACL payload") Right (field "payload")
Right ((acl, Table acl [("payload", TStr)] [[("payload", VStr (T.unpack value))]] Nothing) : filter ((/= acl) . fst) db, A.object ["status" A..= ("ok" :: Text)]) Right ((acl, Table acl [("payload", TStr)] [[("payload", VStr (T.unpack value))]] Nothing Nothing Nothing) : filter ((/= acl) . fst) db, A.object ["status" A..= ("ok" :: Text)])
_ -> Left "storage is not available in this backend" _ -> Left "storage is not available in this backend"
acl = "__system_object_acl" acl = "__system_object_acl"
@@ -320,6 +329,7 @@ applyTransactionMemory ops db = foldl step (Right db) ops
-- | 语句里出现的表 -- | 语句里出现的表
statementTables :: Statement -> [Text] statementTables :: Statement -> [Text]
statementTables stmt = nub (case stmt of statementTables stmt = nub (case stmt of
SelectDistinct q -> statementTables q
Select{} -> Select{} ->
tableRefsOf (selectFrom stmt) tableRefsOf (selectFrom stmt)
++ concatMap exprTables (maybe [] (: []) (selectWhere stmt) ++ map Col (selectCols stmt ++ selectGroupBy stmt ++ map fst (selectOrderBy stmt))) ++ concatMap exprTables (maybe [] (: []) (selectWhere stmt) ++ map Col (selectCols stmt ++ selectGroupBy stmt ++ map fst (selectOrderBy stmt)))
@@ -440,6 +450,8 @@ tableInfoOf (name, tbl) =
, tiRows = length (tableRows tbl) , tiRows = length (tableRows tbl)
, tiIndexes = [] , tiIndexes = []
, tiStats = [(c, distinctOf c, False) | (c, _) <- tableCols tbl] , tiStats = [(c, distinctOf c, False) | (c, _) <- tableCols tbl]
, tiIdentity = tableIdentity tbl
, tiSchemaVersion = tableSchemaVersion tbl
, tiHistograms = [(c, h) | (c, _) <- tableCols tbl, Just h <- [histogramOf (columnValues c)]] , tiHistograms = [(c, h) | (c, _) <- tableCols tbl, Just h <- [histogramOf (columnValues c)]]
} }
where where
@@ -594,9 +606,9 @@ memoryAccounts db stamp req = do
-- | 分开保存内存身份与口令并清理授权 -- | 分开保存内存身份与口令并清理授权
storeMemoryIdentities :: Database -> [Account] -> [Account] -> Database storeMemoryIdentities :: Database -> [Account] -> [Account] -> Database
storeMemoryIdentities db previous accounts = storeMemoryIdentities db previous accounts =
(usersTable, Table usersTable [("id", TInt), ("password_hash", TStr)] credentials Nothing) : (usersTable, Table usersTable [("id", TInt), ("password_hash", TStr)] credentials Nothing Nothing Nothing) :
("__system_identities", Table "__system_identities" [("account", TStr)] identities Nothing) : ("__system_identities", Table "__system_identities" [("account", TStr)] identities Nothing Nothing Nothing) :
("__system_identity_sequence", Table "__system_identity_sequence" [("id", TInt)] [[("id", VInt highWater)]] Nothing) : ("__system_identity_sequence", Table "__system_identity_sequence" [("id", TInt)] [[("id", VInt highWater)]] Nothing Nothing Nothing) :
[(name, clean name table) | (name, table) <- db, name `notElem` [usersTable, "__system_identities", "__system_identity_sequence"]] [(name, clean name table) | (name, table) <- db, name `notElem` [usersTable, "__system_identities", "__system_identity_sequence"]]
where where
-- | 保存口令与身份编号 -- | 保存口令与身份编号
chusql-server/src/ChuSQL/Server/Normalization.hs
+100
View File
@@ -0,0 +1,100 @@
{-# LANGUAGE OverloadedStrings #-}
module ChuSQL.Server.Normalization (normalizationBudget, normalizationReport, schemaFingerprint) where
import qualified ChuSQL.Core.Engine.Normalization as N
import ChuSQL.Core.Model (Column (..), parseColumnType)
import ChuSQL.Core.Protocol (SchemaColumn (..), valueToJSON)
import ChuSQL.Interface.Normalization
import Control.Concurrent.MVar
import Control.Concurrent.QSem
import Control.Exception (bracket_, evaluate, mask)
import Crypto.Hash (Digest, SHA256, hash)
import qualified Data.Aeson as A
import Data.Aeson ((.=), object)
import qualified Data.ByteString.Lazy as BL
import Data.List (nub, sort)
import Data.Text (Text)
import qualified Data.Text as T
import System.IO.Unsafe (unsafePerformIO)
import System.Timeout (timeout)
-- 范式服务：有界执行、稳定 schema 指纹与引擎报告的线上表示。
{-# NOINLINE analysisSlots #-}
-- 保存最多八个请求和两个执行槽。
analysisSlots :: (MVar Int, QSem)
analysisSlots = unsafePerformIO ((,) <$> newMVar 0 <*> newQSem 2)
-- 限制排队、执行时间及输出大小。
normalizationBudget :: IO (Either (Text, Text) A.Value) -> IO (Either (Text, Text) A.Value)
normalizationBudget action = mask $ \restore -> do
let (queued, slots) = analysisSlots
admitted <- modifyMVar queued $ \count -> pure (if count < 8 then (count + 1, True) else (count, False))
if not admitted then pure (Left ("too_busy", "normalization queue is full")) else
bracket_ (pure ()) (modifyMVar_ queued (pure . subtract 1)) $ do
result <- restore $ timeout 5000000 $ bracket_ (waitQSem slots) (signalQSem slots) $ do
outcome <- action
case outcome of
Left err -> pure (Left err)
Right value -> do
size <- evaluate (BL.length (BL.take 64001 (A.encode value)))
pure (if size > 64000 then Left ("too_large", "normalization report exceeds transport budget") else Right value)
pure (maybe (Left ("timeout", "normalization exceeded its five second budget")) id result)
-- 编码只影响语义的列信息。
fingerprintColumn :: SchemaColumn -> A.Value
fingerprintColumn c = A.toJSON [A.toJSON (scName c), A.toJSON (scType c), A.toJSON (scNullable c),
maybe A.Null valueToJSON (scDefault c), A.toJSON (scAutoIncrement c), A.toJSON (scPrimaryKey c), A.toJSON (scUnique c), A.toJSON (scCheck c)]
-- 将稳定编号与列结构绑定到 SHA256。
schemaFingerprint :: ObjectIdentity -> [SchemaColumn] -> Text
schemaFingerprint (ObjectIdentity database table) columns = T.pack (show (hash (BL.toStrict (A.encode (A.toJSON [A.toJSON [show database, show table], A.toJSON (map fingerprintColumn columns)]))) :: Digest SHA256))
-- 绑定 schema 并编码只读分析报告。
normalizationReport :: ObjectIdentity -> [SchemaColumn] -> Text -> NormalizationRequest -> Either (Text, Text) A.Value
normalizationReport identity columns view payload = do
let fingerprint = schemaFingerprint identity columns
if maybe False (/= identity) (nrExpectedObject payload) || maybe False (/= fingerprint) (nrExpectedSchema payload)
then Left ("conflict", "normalization object or schema has changed") else Right ()
schema <- mapM column columns
let declaration = nrDeclaration payload
axioms = N.Declaration (ndKeys declaration) [N.Dependency a b | NormalizationDependency a b <- ndDependencies declaration] (ndComplete declaration) (ndAtomic declaration)
analysis <- either (Left . (,) "bad_request" . T.pack) Right (N.analyzeNormalization schema axioms)
Right (object ["api_version" .= (1 :: Int), "database" .= nrDatabase payload, "table" .= last (T.splitOn "." (nrTable payload)),
"object" .= identity, "schema_fingerprint" .= fingerprint, "view" .= view, "schema" .= columns,
"declaration" .= canonicalDeclaration declaration, "candidate_keys" .= N.candidateKeys analysis,
"findings" .= map findingJSON (N.findings analysis), "suggestions" .= map decompositionJSON (N.suggestions analysis),
"conditional" .= True, "condition" .= ("Analysis is conditional on the declarations; SQL NULL, UNIQUE and bag semantics are not proofs of classical keys or lossless data migration." :: Text)])
where
-- 还原列类型及声明约束而不读取数据。
column c = case parseColumnType (scType c) of
Nothing -> Left ("storage_error", "normalization: invalid schema type")
Just ty -> Right (scName c, Column ty (scNullable c) (scDefault c) (scAutoIncrement c) (scUnique c) (scPrimaryKey c) (scCheck c))
-- 规范化经验证的声明属性集合。
canonicalDeclaration :: NormalizationDeclaration -> NormalizationDeclaration
canonicalDeclaration d = d {ndKeys = sort (nub (map N.canonical (ndKeys d))),
ndDependencies = [NormalizationDependency a b | (a, b) <- sort (nub [(N.canonical a, N.canonical b) | NormalizationDependency a b <- ndDependencies d])]}
-- 编码稳定范式标签及证据。
findingJSON :: N.Finding -> A.Value
findingJSON f = object ["normal_form" .= formTag (N.normalForm f), "verdict" .= verdictTag (N.verdict f),
"witnesses" .= [NormalizationDependency a b | N.Dependency a b <- N.witnesses f], "condition" .= N.condition f]
-- 编码一次分解的理论结论。
decompositionJSON :: N.Decomposition -> A.Value
decompositionJSON d = object ["relations" .= N.relations d, "lossless_join" .= verdictTag (N.losslessJoin d), "dependency_preserving" .= verdictTag (N.dependencyPreserving d)]
-- 输出稳定的范式名称。
formTag :: N.NormalForm -> Text
formTag N.NF1 = "1nf"
formTag N.NF2 = "2nf"
formTag N.NF3 = "3nf"
formTag N.BCNF = "bcnf"
-- 输出稳定的结论标签。
verdictTag :: N.Verdict -> Text
verdictTag N.Satisfied = "satisfied"
verdictTag N.Violated = "violated"
verdictTag N.Insufficient = "insufficient"
chusql-server/src/ChuSQL/Server/ObjectACL.hs
+32 -8
View File
@@ -2,8 +2,8 @@
{-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE OverloadedStrings #-}
module ChuSQL.Server.ObjectACL module ChuSQL.Server.ObjectACL
( objectAuthorize, objectGrant, objectRevoke, objectViews, objectClaim ( objectAuthorize, objectGrant, objectRevoke, objectRevokeGrantOption, objectViews, objectClaim
, objectReconcile, objectAffected, objectOwner, objectConnect, objectPrepareCreate, objectUsers , objectSelfViews, objectReconcile, objectAffected, objectOwner, objectConnect, objectPrepareCreate, objectUsers
) where ) where
import ChuSQL.Core.Protocol (Account (..), TableInfo (..)) import ChuSQL.Core.Protocol (Account (..), TableInfo (..))
@@ -315,19 +315,28 @@ objectGrant backend catalog principal database names object role option = do
-- 撤销授权并收敛失去来源的下游授权。 -- 撤销授权并收敛失去来源的下游授权。
objectRevoke :: Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ()) objectRevoke :: Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
objectRevoke backend catalog principal database names object role = do objectRevoke = revokeObject False
-- 仅撤销转授权能力并清理失效来源。
objectRevokeGrantOption :: Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
objectRevokeGrantOption = revokeObject True
-- 按授予来源撤销权限或转授权能力。
revokeObject :: Bool -> Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
revokeObject optionOnly backend catalog principal database names object role = do
loaded <- load backend catalog loaded <- load backend catalog
resolved <- resolve backend database object resolved <- resolve backend database object
case (,) <$> loaded <*> resolved of case (,) <$> loaded <*> resolved of
Left err -> pure (Left err) Left err -> pure (Left err)
Right ((env, previous, acl), target) -> case (,) <$> roleId env role <*> privileges target names of Right ((env, previous, acl), target) -> case (,,) <$> roleId env role <*> privileges target names <*> (if principalIsRoot principal then Right "0" else roleId env (principalName principal)) of
Left err -> pure (Left err) Left err -> pure (Left err)
Right (recipient, items) -> do Right (recipient, items, grantor) -> do
let grantor = either (const "") id (roleId env (principalName principal)) let manager = principalIsRoot principal || owns env acl grantor target
manager = principalIsRoot principal || owns env acl grantor target
-- 选择本次可撤销的授予来源。 -- 选择本次可撤销的授予来源。
matches edge = edgeRole edge == recipient && edgeTarget edge == target && edgePrivilege edge `elem` items && (manager || edgeGrantor edge == grantor) matches edge = edgeRole edge == recipient && edgeTarget edge == target && edgePrivilege edge `elem` items && (manager || edgeGrantor edge == grantor)
next = acl{aclEdges = filter (not . matches) (aclEdges acl), aclClosed = if recipient == "public" && "connect" `elem` items then nub (targetDatabase target : aclClosed acl) else aclClosed acl} next = acl{aclEdges = if optionOnly then map removeOption (aclEdges acl) else filter (not . matches) (aclEdges acl), aclClosed = if not optionOnly && recipient == "public" && "connect" `elem` items then nub (targetDatabase target : aclClosed acl) else aclClosed acl}
-- 保留基础权限并移除匹配来源的转授权。
removeOption edge = if matches edge then edge{edgeOption = False} else edge
if recipient == "public" && "connect" `elem` items && not manager then pure (Left "administrator required") if recipient == "public" && "connect" `elem` items && not manager then pure (Left "administrator required")
else writeACL backend previous (effective env next) else writeACL backend previous (effective env next)
@@ -341,6 +350,21 @@ objectViews backend catalog role = do
let mine = [edge | edge <- aclEdges acl, edgeRole edge == identity] let mine = [edge | edge <- aclEdges acl, edgeRole edge == identity]
Right (nub [Grant role (edgePrivilege edge) (targetName (edgeTarget edge)) (any (\other -> edgeTarget other == edgeTarget edge && edgePrivilege other == edgePrivilege edge && edgeOption other) mine) | edge <- mine]) Right (nub [Grant role (edgePrivilege edge) (targetName (edgeTarget edge)) (any (\other -> edgeTarget other == edgeTarget edge && edgePrivilege other == edgePrivilege edge && edgeOption other) mine) | edge <- mine])
-- 列出当前身份实际拥有的对象权限。
objectSelfViews :: Backend -> Catalog -> Principal -> IO (Either String [Grant])
objectSelfViews backend catalog principal = do
loaded <- load backend catalog
pure $ do
(env, _, acl) <- loaded
identity <- roleId env (principalName principal)
if active env identity then Right () else Left "permission denied: disabled identity"
-- 使用当前权限闭包判断普通及转授权权限。
let permitted target privilege option = principalIsRoot principal || allowed env acl identity target privilege option
Right [Grant (principalName principal) privilege (targetName target) (permitted target privilege True)
| target <- contextTargets env
, privilege <- if targetKind target == "database" then ["connect", "create"] else ["select", "insert", "update", "delete"]
, permitted target privilege False]
-- 登记新对象的创建者为所有者。 -- 登记新对象的创建者为所有者。
objectClaim :: Backend -> Catalog -> Principal -> Text -> Text -> IO (Either String ()) objectClaim :: Backend -> Catalog -> Principal -> Text -> Text -> IO (Either String ())
objectClaim backend catalog principal database object = do objectClaim backend catalog principal database object = do
chusql-server/src/ChuSQL/Server/Privileges.hs
+10 -2
View File
@@ -2,7 +2,7 @@
module ChuSQL.Server.Privileges module ChuSQL.Server.Privileges
( Grant (..), Membership (..), PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView (..) ( Grant (..), Membership (..), PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView (..)
, affectedAccounts, authorize, authorizeTables, authorizeConnect, filterTables, listRoleViews , affectedAccounts, authorize, authorizeTables, authorizeConnect, filterTables, listRoleViews, listSelfGrants
, newPrivileges, normalizeObject, privilegeCommand, runPrivilegeCommand, claimObject, prepareObject, reconcileObjects, affectedObject , newPrivileges, normalizeObject, privilegeCommand, runPrivilegeCommand, claimObject, prepareObject, reconcileObjects, affectedObject
) where ) where
@@ -22,7 +22,7 @@ import qualified Data.Text as T
data PrivilegeError = PrivilegeError Text Text deriving (Show, Eq) data PrivilegeError = PrivilegeError Text Text deriving (Show, Eq)
data Membership = Membership { memberRole :: Text, memberUser :: Text } deriving (Show, Eq, Ord) data Membership = Membership { memberRole :: Text, memberUser :: Text } deriving (Show, Eq, Ord)
data PrivilegeCommand = CreateRoleCommand Text | DropRoleCommand Text data PrivilegeCommand = CreateRoleCommand Text | DropRoleCommand Text
| GrantPrivilegesCommand [Text] Text Text Bool | RevokePrivilegesCommand [Text] Text Text | GrantPrivilegesCommand [Text] Text Text Bool | RevokePrivilegesCommand [Text] Text Text | RevokeGrantOptionCommand [Text] Text Text
| GrantRoleCommand Text [Text] | RevokeRoleCommand Text [Text] deriving (Show, Eq) | GrantRoleCommand Text [Text] | RevokeRoleCommand Text [Text] deriving (Show, Eq)
data Privileges = Privileges { pvCatalog :: Catalog, pvBackend :: Backend } data Privileges = Privileges { pvCatalog :: Catalog, pvBackend :: Backend }
@@ -38,6 +38,7 @@ privilegeCommand statement = case statement of
CreateRole name -> Just (CreateRoleCommand (T.pack name)) CreateRole name -> Just (CreateRoleCommand (T.pack name))
DropRole name -> Just (DropRoleCommand (T.pack name)) DropRole name -> Just (DropRoleCommand (T.pack name))
GrantPrivileges items object role option -> Just (GrantPrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role) option) GrantPrivileges items object role option -> Just (GrantPrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role) option)
RevokeGrantOption items object role -> Just (RevokeGrantOptionCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role))
RevokePrivileges items object role -> Just (RevokePrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role)) RevokePrivileges items object role -> Just (RevokePrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role))
GrantRole role members -> Just (GrantRoleCommand (T.pack role) (map T.pack members)) GrantRole role members -> Just (GrantRoleCommand (T.pack role) (map T.pack members))
RevokeRole role members -> Just (RevokeRoleCommand (T.pack role) (map T.pack members)) RevokeRole role members -> Just (RevokeRoleCommand (T.pack role) (map T.pack members))
@@ -47,6 +48,7 @@ privilegeCommand statement = case statement of
runPrivilegeCommand :: Privileges -> Principal -> Text -> PrivilegeCommand -> IO (Either PrivilegeError ()) runPrivilegeCommand :: Privileges -> Principal -> Text -> PrivilegeCommand -> IO (Either PrivilegeError ())
runPrivilegeCommand service principal database command = withTables service $ case command of runPrivilegeCommand service principal database command = withTables service $ case command of
GrantPrivilegesCommand items object role option -> catalog (OA.objectGrant backend directory principal database items object role option) GrantPrivilegesCommand items object role option -> catalog (OA.objectGrant backend directory principal database items object role option)
RevokeGrantOptionCommand items object role -> catalog (OA.objectRevokeGrantOption backend directory principal database items object role)
RevokePrivilegesCommand items object role -> catalog (OA.objectRevoke backend directory principal database items object role) RevokePrivilegesCommand items object role -> catalog (OA.objectRevoke backend directory principal database items object role)
CreateRoleCommand role CreateRoleCommand role
| principalIsCatalogManager principal -> do | principalIsCatalogManager principal -> do
@@ -106,6 +108,7 @@ affectedAccounts service command = withTables service (catalog (OA.objectAffecte
CreateRoleCommand _ -> [] CreateRoleCommand _ -> []
DropRoleCommand role -> [role] DropRoleCommand role -> [role]
GrantPrivilegesCommand _ _ role _ -> [role] GrantPrivilegesCommand _ _ role _ -> [role]
RevokeGrantOptionCommand _ _ role -> [role]
RevokePrivilegesCommand _ _ role -> [role] RevokePrivilegesCommand _ _ role -> [role]
GrantRoleCommand _ members -> members GrantRoleCommand _ members -> members
RevokeRoleCommand _ members -> members RevokeRoleCommand _ members -> members
@@ -195,6 +198,7 @@ reconcileObjects service = withTables service (catalog (OA.objectReconcile (pvBa
-- 收集数据语句及子查询所需的表权限。 -- 收集数据语句及子查询所需的表权限。
requiredActions :: Statement -> Maybe [(Text, Text)] requiredActions :: Statement -> Maybe [(Text, Text)]
requiredActions statement = case statement of requiredActions statement = case statement of
SelectDistinct q -> requiredActions q
Select{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement))) Select{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement)))
SelectExpr{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement) ++ map snd (selectItems statement))) SelectExpr{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement) ++ map snd (selectItems statement)))
Insert table _ rows -> Just ((T.pack table, "insert") : selectsOf (concat rows)) Insert table _ rows -> Just ((T.pack table, "insert") : selectsOf (concat rows))
@@ -231,3 +235,7 @@ storageError message = PrivilegeError code (T.pack message)
forbidden :: PrivilegeError forbidden :: PrivilegeError
forbidden = PrivilegeError "forbidden" "administrator required" forbidden = PrivilegeError "forbidden" "administrator required"
-- 查询登录身份自己的有效权限。
listSelfGrants :: Privileges -> Principal -> IO (Either PrivilegeError [Grant])
listSelfGrants service principal = withTables service (catalog (OA.objectSelfViews (pvBackend service) (pvCatalog service) principal))
chusql-server/src/ChuSQL/Server/Session.hs
+79 -2
View File
@@ -18,6 +18,7 @@ module ChuSQL.Server.Session
, runStatement , runStatement
, runStatementCoded , runStatementCoded
, runStorageCoded , runStorageCoded
, analyzeNormalizationCoded
, catalog , catalog
, accounts , accounts
, roleViews , roleViews
@@ -27,7 +28,12 @@ module ChuSQL.Server.Session
, queryResultJson , queryResultJson
) where ) where
import ChuSQL.Core.Model (Database, Row, Table (..), Value (..)) import ChuSQL.Core.Model (Database, Row, Table (..), Value (..), Column (..), typeName)
import ChuSQL.Core.Protocol (SchemaColumn (..))
import ChuSQL.Interface.Normalization
import ChuSQL.Server.Normalization (normalizationBudget, normalizationReport)
import Data.Aeson.Types (parseEither)
import qualified Data.ByteString.Lazy as BL
import ChuSQL.Core.Protocol (Account (..), Request (..), QueryResult (..), TxnOp (..), queryResultJson) import ChuSQL.Core.Protocol (Account (..), Request (..), QueryResult (..), TxnOp (..), queryResultJson)
import ChuSQL.Core.Engine.Storage.IPC (TableInfo (..)) import ChuSQL.Core.Engine.Storage.IPC (TableInfo (..))
import qualified ChuSQL.Core.Engine as Engine import qualified ChuSQL.Core.Engine as Engine
@@ -39,7 +45,7 @@ import ChuSQL.Interface.Auth (defaultSessionPolicy, newSessionStore)
import ChuSQL.Server.Accounts import ChuSQL.Server.Accounts
import ChuSQL.Server.Backend (Backend (..), StatementResult (..), columnsFromStatement, scopeDdl) import ChuSQL.Server.Backend (Backend (..), StatementResult (..), columnsFromStatement, scopeDdl)
import ChuSQL.Server.Policy (configurePasswordPolicy) import ChuSQL.Server.Policy (configurePasswordPolicy)
import ChuSQL.Server.Privileges (PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView, affectedAccounts, affectedObject, authorize, authorizeConnect, claimObject, prepareObject, reconcileObjects, filterTables, listRoleViews, newPrivileges, privilegeCommand, runPrivilegeCommand) import ChuSQL.Server.Privileges (PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView, Grant (..), affectedAccounts, affectedObject, authorize, authorizeTables, authorizeConnect, claimObject, prepareObject, reconcileObjects, filterTables, listRoleViews, listSelfGrants, newPrivileges, privilegeCommand, runPrivilegeCommand)
import ChuSQL.Server.Security (readSecurity, writeSecurity) import ChuSQL.Server.Security (readSecurity, writeSecurity)
import Control.Exception (IOException, try) import Control.Exception (IOException, try)
import Data.Bifunctor (first) import Data.Bifunctor (first)
@@ -280,6 +286,7 @@ runStatementUnlocked session sql = case parseStatement (T.unpack sql) of
Right (Savepoint name) -> savepointTransaction session (T.pack name) Right (Savepoint name) -> savepointTransaction session (T.pack name)
Right (RollbackToSavepoint name) -> rollbackToSavepoint session (T.pack name) Right (RollbackToSavepoint name) -> rollbackToSavepoint session (T.pack name)
Right (ReleaseSavepoint name) -> releaseSavepoint session (T.pack name) Right (ReleaseSavepoint name) -> releaseSavepoint session (T.pack name)
Right ShowGrants -> showSelfGrants session
Right ShowRoles -> showIdentities session Right ShowRoles -> showIdentities session
Right statement Right statement
| Just command <- accountCommand statement -> runAccount session command | Just command <- accountCommand statement -> runAccount session command
@@ -319,6 +326,7 @@ transactionDdl statement = case statement of
-- | 事务里只有数据语句走快照 -- | 事务里只有数据语句走快照
transactionData :: Statement -> Bool transactionData :: Statement -> Bool
transactionData statement = case statement of transactionData statement = case statement of
SelectDistinct q -> transactionData q
Select{} -> True Select{} -> True
SelectExpr{} -> True SelectExpr{} -> True
Insert{} -> True Insert{} -> True
@@ -840,3 +848,72 @@ resultOf result =
-- | 按列名取单元格 -- | 按列名取单元格
cellOf row name = maybe VNull id (lookup name row) cellOf row name = maybe VNull id (lookup name row)
-- 展示当前登录身份自己的有效权限。
showSelfGrants :: Session -> IO (Either SessionError QueryResult)
showSelfGrants session = do
who <- readIORef (ssPrincipal session)
case who of
Nothing -> pure (Left (SessionError "unauthorized" "sign in first"))
Just principal -> do
grants <- listSelfGrants (ssPrivileges session) principal
pure $ case grants of
Left err -> Left (privilegeErrorOf err)
Right rows -> Right (QueryResult ["identity", "privilege", "object", "grant_option"]
[[VStr (T.unpack (grantRole grant)), VStr (T.unpack (grantPrivilege grant)), VStr (T.unpack (grantObject grant)), VBool (grantable grant)] | grant <- rows]
(length rows) False Nothing)
-- 在连接视图及权限稳定期间分析范式。
analyzeNormalizationCoded :: Session -> NormalizationRequest -> IO (Either SessionError A.Value)
analyzeNormalizationCoded session payload = do
result <- normalizationBudget $ readSecurity $ do
valid <- validatePrincipal session
current <- readIORef (ssCurrent session)
who <- readIORef (ssPrincipal session)
let table = last (T.splitOn "." (nrTable payload))
case (valid, who, A.fromJSON (A.toJSON payload) :: A.Result NormalizationRequest) of
(Left err, _, _) -> pure (Left (sessCode err, sessMessage err))
(_, Nothing, _) -> pure (Left ("unauthorized", "sign in first"))
(_, _, A.Error message) -> pure (Left ("bad_request", T.pack message))
(_, Just principal, A.Success _)
| BL.length (A.encode payload) > 65000 -> pure (Left ("too_large", "normalization request exceeds transport budget"))
| current /= nrDatabase payload -> pure (Left ("bad_request", "normalization database differs from the selected database"))
| current == "system" || "__system_" `T.isPrefixOf` table -> pure (Left ("forbidden", "normalization does not expose system objects"))
| otherwise -> do
allowed <- authorizeTables (ssPrivileges session) principal current [(table, "select")]
case allowed of
Left (PrivilegeError code message) -> pure (Left (code, message))
Right () -> normalizationView session table payload
pure (either (Left . uncurry SessionError) Right result)
-- 读取目录结构或当前连接的事务 schema。
normalizationView :: Session -> Text -> NormalizationRequest -> IO (Either (Text, Text) A.Value)
normalizationView session table payload = do
live <- snapshotObjectIds session [T.unpack table]
transaction <- readIORef (ssTransaction session)
case live >>= maybe (Left (SessionError "not_found" "unknown table")) Right . lookup (T.unpack table) of
Left err -> pure (Left (sessCode err, sessMessage err))
Right object -> case objectIdentity object of
Left message -> pure (Left ("storage_error", message))
Right identity -> case transaction of
Just tx -> case (lookup (T.unpack table) (txStaged tx), lookup (T.unpack table) (txObjects tx)) of
(Just relation, Just oldObject) | objectIdentity oldObject == Right identity ->
pure (normalizationReport identity (map normalizationColumn (tableCols relation)) "transaction_snapshot" payload)
_ -> pure (Left ("conflict", "normalization snapshot object has changed"))
Nothing -> do
backend <- currentBackend session
infos <- beCatalog backend
pure $ case infos of
Left message -> Left ("storage_error", T.pack message)
Right relations -> case [tiColumns info | info <- relations, tiTable info == T.unpack table] of
[columns] -> normalizationReport identity columns "live_schema" payload
_ -> Left ("not_found", "unknown table")
-- 从目录响应中只取稳定编号。
objectIdentity :: A.Value -> Either Text ObjectIdentity
objectIdentity = first T.pack . parseEither (A.withObject "catalog object" $ \o -> ObjectIdentity <$> o A..: "database_id" <*> o A..: "object_id")
-- 将事务 schema 转成同一线上列结构。
normalizationColumn :: (String, Column) -> SchemaColumn
normalizationColumn (name, c) = SchemaColumn name (typeName (columnType c)) (columnNullable c) (columnDefault c) (columnAutoIncrement c) (columnPrimaryKey c) (columnUnique c) (columnCheck c)
chusql-server/src/ChuSQL/Server/TCP.hs
+4 -1
View File
@@ -23,7 +23,7 @@ import ChuSQL.Interface.RateLimit (RateLimiter, rateLimitBlock, rateLimitClear,
import qualified ChuSQL.Core.Engine.Error as E import qualified ChuSQL.Core.Engine.Error as E
import ChuSQL.Core.Protocol (Account (..), Request (ReqIdentityInitialize)) import ChuSQL.Core.Protocol (Account (..), Request (ReqIdentityInitialize))
import ChuSQL.Server.Backend (Backend (..)) import ChuSQL.Server.Backend (Backend (..))
import ChuSQL.Server.Session (QueryResult (..), Session, SessionError (..), accounts, authenticateSessionCoded, authenticateSudoSessionCoded, isPlainIdentifier, catalog, databases, newSessionWith, policyOf, reloadPolicy, roleViews, runStatementCoded, runStorageCoded, sessionIsAdmin, sessionUser) import ChuSQL.Server.Session (QueryResult (..), Session, SessionError (..), accounts, authenticateSessionCoded, authenticateSudoSessionCoded, isPlainIdentifier, catalog, databases, newSessionWith, policyOf, reloadPolicy, roleViews, analyzeNormalizationCoded, runStatementCoded, runStorageCoded, sessionIsAdmin, sessionUser)
import ChuSQL.Server.Security (writeSecurity) import ChuSQL.Server.Security (writeSecurity)
import ChuSQL.Server.Privileges (newPrivileges, reconcileObjects) import ChuSQL.Server.Privileges (newPrivileges, reconcileObjects)
import Control.Concurrent (forkIO) import Control.Concurrent (forkIO)
@@ -331,6 +331,9 @@ dispatch env session entry request = case request of
ReqSudoLogin user proof -> do ReqSudoLogin user proof -> do
response <- sudoLogin env session entry user proof response <- sudoLogin env session entry user proof
pure (response, True) pure (response, True)
ReqNormalization payload -> do
result <- analyzeNormalizationCoded session payload
pure (either (\err -> RespError (sessCode err) (sessMessage err)) RespNormalization result, True)
ReqQuery sql -> do ReqQuery sql -> do
who <- sessionUser session who <- sessionUser session
response <- case who of response <- case who of
chusql-server/test/Spec.hs
+242 -9
View File
@@ -12,11 +12,14 @@ import ChuSQL.Core.Engine.Storage.IPC (
setStorageLink, setStorageLink,
) )
import ChuSQL.Core.Engine.Syntax.AST (Statement) import ChuSQL.Core.Engine.Syntax.AST (Statement)
import ChuSQL.Interface.Normalization
import ChuSQL.Server.Normalization (normalizationBudget, normalizationReport, schemaFingerprint)
import ChuSQL.Core.Protocol (SchemaColumn (..))
import Data.IORef (newIORef, readIORef, writeIORef) import Data.IORef (newIORef, readIORef, writeIORef)
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement) import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
import ChuSQL.Core.Model (Database, Histogram (..), Row, Table (..), Value (..), pattern TInt, pattern TStr) import ChuSQL.Core.Model (Database, Histogram (..), Row, Table (..), TableIdentity (..), Value (..), pattern TInt, pattern TStr)
import ChuSQL.Interface.Auth (hashPasswordWith) import ChuSQL.Interface.Auth (hashPasswordWith)
import ChuSQL.Interface.Protocol (ClientRequest (..), Grant (..), RoleView (..), ServerResponse (..), decodeRequest, encodeResponse) import ChuSQL.Interface.Protocol (ClientRequest (..), Grant (..), RoleView (..), ServerResponse (..), decodeRequest, encodeResponse, encodeRequest, decodeResponse)
import ChuSQL.Interface.RateLimit (newRateLimiter) import ChuSQL.Interface.RateLimit (newRateLimiter)
import ChuSQL.Interface.Link (connectClient, closeClient, clientSudoLogin, clientQuery) import ChuSQL.Interface.Link (connectClient, closeClient, clientSudoLogin, clientQuery)
import ChuSQL.Interface.Sudo (SudoCredential, newSudoCredential, sudoProof, verifySudoProof, validSudoChallenge, import ChuSQL.Interface.Sudo (SudoCredential, newSudoCredential, sudoProof, verifySudoProof, validSudoChallenge,
@@ -33,6 +36,7 @@ import ChuSQL.Server.Privileges (
claimObject, claimObject,
prepareObject, prepareObject,
listRoleViews, listRoleViews,
listSelfGrants,
newPrivileges, newPrivileges,
runPrivilegeCommand, runPrivilegeCommand,
) )
@@ -47,7 +51,7 @@ import ChuSQL.Server.TCP (
startServer, startServer,
isLocalPeer, isLocalPeer,
) )
import Control.Concurrent (forkIO, threadDelay) import Control.Concurrent (forkIO, forkFinally, killThread, threadDelay)
import Control.Concurrent.MVar (MVar, modifyMVar_, newEmptyMVar, newMVar, putMVar, takeMVar) import Control.Concurrent.MVar (MVar, modifyMVar_, newEmptyMVar, newMVar, putMVar, takeMVar)
import System.Timeout (timeout) import System.Timeout (timeout)
import Control.Exception (IOException, bracket, finally, try) import Control.Exception (IOException, bracket, finally, try)
@@ -64,7 +68,7 @@ import Data.Maybe (fromMaybe)
import Data.Text (Text) import Data.Text (Text)
import qualified Data.Text as T import qualified Data.Text as T
import qualified Data.Vector as V import qualified Data.Vector as V
import Data.Time.Clock (addUTCTime, getCurrentTime) import Data.Time.Clock (addUTCTime, diffUTCTime, getCurrentTime)
import Network.Socket ( import Network.Socket (
AddrInfo (..), AddrInfo (..),
SockAddr (..), SockAddr (..),
@@ -105,6 +109,7 @@ main = hspec spec
-- | 全部用例 -- | 全部用例
spec :: Spec spec :: Spec
spec = do spec = do
normalizationSpec
describe "sudo authentication" $ do describe "sudo authentication" $ do
it "defaults to disabled and rejects unconfigured TCP authentication" $ it "defaults to disabled and rejects unconfigured TCP authentication" $
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \client -> do withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \client -> do
@@ -380,14 +385,14 @@ testDb =
[ [("id", VInt i), ("name", VStr ("user" ++ show i)), ("age", VInt (20 + i))] [ [("id", VInt i), ("name", VStr ("user" ++ show i)), ("age", VInt (20 + i))]
| i <- [1 .. 5] | i <- [1 .. 5]
] ]
Nothing Nothing Nothing Nothing
) )
, ( "orders" , ( "orders"
, Table , Table
"orders" "orders"
[("id", TInt), ("user_id", TInt)] [("id", TInt), ("user_id", TInt)]
[[("id", VInt 1), ("user_id", VInt 1)]] [[("id", VInt 1), ("user_id", VInt 1)]]
Nothing Nothing Nothing Nothing
) )
] ]
@@ -1125,6 +1130,44 @@ withObjectPrivileges body = do
-- 验证对象作用域、授权来源和外部入口。 -- 验证对象作用域、授权来源和外部入口。
objectPrivilegeSpec :: Spec objectPrivilegeSpec :: Spec
objectPrivilegeSpec = describe "stable object privileges" $ do objectPrivilegeSpec = describe "stable object privileges" $ do
it "lists only the caller's effective grants including roles and PUBLIC" $ withObjectPrivileges $ \backend service root -> do
runPrivilegeCommand service root testDatabase (CreateRoleCommand "self_reader") `shouldReturn` Right ()
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "self_reader" True) `shouldReturn` Right ()
runPrivilegeCommand service root testDatabase (GrantRoleCommand "self_reader" ["alice"]) `shouldReturn` Right ()
grants <- listSelfGrants service (Ordinary testAccount)
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "select" "test.users" True))
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "connect" "database:test" False))
grants `shouldSatisfy` either (const False) (all ((== "alice") . grantRole))
runPrivilegeCommand service root testDatabase (RevokePrivilegesCommand ["select"] "users" "self_reader") `shouldReturn` Right ()
after <- listSelfGrants service (Ordinary testAccount)
after `shouldSatisfy` either (const False) (not . any ((== "test.users") . grantObject))
_ <- beAccounts backend (ReqAccountReset "alice" (testPasswordHash "Alice-Secret-1234"))
_ <- beAccounts backend (ReqAccountCreate testRootName (testPasswordHash testRootPassword))
_ <- beAccounts backend (ReqIdentityAlter testRootName Nothing (Just True) Nothing Nothing Nothing)
settings <- tempSettingsPath "self-grants"
session <- newSession backend testRootName settings
authenticateSessionCoded session "alice" "Alice-Secret-1234" `shouldReturn` Right ()
result <- mustSql session "SHOW GRANTS"
qrColumns result `shouldBe` ["identity", "privilege", "object", "grant_option"]
qrRows result `shouldSatisfy` all (\row -> take 1 row == [VStr "alice"])
rejected <- runStatementCoded session "SHOW GRANTS FOR admin"
rejected `shouldSatisfy` either (const True) (const False)
it "includes ownership permissions and excludes ungranted tables" $ withObjectPrivileges $ \backend service _ -> do
mustRun (beStatement (beWithDatabase backend "test") "CREATE TABLE personal_owned (id INT)")
claimObject service (Ordinary testAccount) testDatabase "personal_owned" `shouldReturn` Right ()
grants <- listSelfGrants service (Ordinary testAccount)
fmap (filter ((== "test.personal_owned") . grantObject)) grants `shouldBe` Right
[Grant "alice" privilege "test.personal_owned" True | privilege <- ["select", "insert", "update", "delete"]]
grants `shouldSatisfy` either (const False) (not . any ((== "test.users") . grantObject))
it "checks DISTINCT table reads and nested subqueries" $ withObjectPrivileges $ \_ service root -> do
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT age FROM users"
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT d.age FROM (SELECT DISTINCT age FROM users) d"
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" False) `shouldReturn` Right ()
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT age FROM users" `shouldReturn` Right ()
it "migrates old table grants once and ignores later legacy writes" $ withObjectPrivileges $ \backend service _ -> do it "migrates old table grants once and ignores later legacy writes" $ withObjectPrivileges $ \backend service _ -> do
let system = beWithDatabase backend "system" let system = beWithDatabase backend "system"
mustRun (beStatement system "CREATE TABLE __system_grants (role VARCHAR(64), privilege VARCHAR(16), object VARCHAR(128))") mustRun (beStatement system "CREATE TABLE __system_grants (role VARCHAR(64), privilege VARCHAR(16), object VARCHAR(128))")
@@ -1204,6 +1247,41 @@ objectPrivilegeSpec = describe "stable object privileges" $ do
authorizeSql service (Ordinary testAccount) "DROP DATABASE test" `shouldReturn` Left (PrivilegeError "forbidden" "object owner required") authorizeSql service (Ordinary testAccount) "DROP DATABASE test" `shouldReturn` Left (PrivilegeError "forbidden" "object owner required")
runPrivilegeCommand service root testDatabase (RevokePrivilegesCommand ["select"] "users" "alice") `shouldReturn` Right () runPrivilegeCommand service root testDatabase (RevokePrivilegesCommand ["select"] "users" "alice") `shouldReturn` Right ()
authorizeSql service (Ordinary testAccount) "SELECT * FROM users" `shouldReturn` Right () authorizeSql service (Ordinary testAccount) "SELECT * FROM users" `shouldReturn` Right ()
it "executes grant option revocation over TCP and keeps SELECT" $
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \admin -> do
_ <- tcpLogin admin "admin" "s3cret"
_ <- tcpQuery admin "USE test"
_ <- tcpQuery admin "CREATE USER option_reader IDENTIFIED BY 'Option-Reader-1234'"
tcpQuery admin "GRANT SELECT ON users TO option_reader WITH GRANT OPTION" >>= \reply -> at "status" reply `shouldBe` A.String "result"
tcpQuery admin "REVOKE GRANT OPTION FOR SELECT ON users FROM option_reader" >>= \reply -> at "status" reply `shouldBe` A.String "result"
withConnection handle $ \reader -> do
_ <- tcpLogin reader "option_reader" "Option-Reader-1234"
_ <- tcpQuery reader "USE test"
tcpQuery reader "SELECT DISTINCT age FROM users ORDER BY 1" >>= \reply -> at "status" reply `shouldBe` A.String "result"
tcpQuery reader "SHOW GRANTS" >>= \reply -> at "status" reply `shouldBe` A.String "result"
tcpQuery reader "GRANT SELECT ON users TO admin" >>= \reply -> at "code" reply `shouldBe` A.String "forbidden"
it "revokes grant option while preserving data access and independent sources" $ withObjectPrivileges $ \_ service root -> do
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount{accountUser = "bob"}) testDatabase (GrantPrivilegesCommand ["select"] "users" "carol" False) `shouldReturn` Right ()
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "carol" False) `shouldReturn` Right ()
affectedAccounts service (RevokeGrantOptionCommand ["select"] "users" "alice") `shouldReturn` Right ["alice", "bob", "carol"]
runPrivilegeCommand service root testDatabase (RevokeGrantOptionCommand ["select"] "users" "alice") `shouldReturn` Right ()
authorizeSql service (Ordinary testAccount) "SELECT * FROM users" `shouldReturn` Right ()
grants <- listSelfGrants service (Ordinary testAccount)
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "select" "test.users" False))
authorizeSql service (Ordinary testAccount{accountUser = "bob"}) "SELECT * FROM users"
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
authorizeSql service (Ordinary testAccount{accountUser = "carol"}) "SELECT * FROM users" `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" False)
`shouldReturn` Left (PrivilegeError "forbidden" "grant option required: SELECT ON users")
it "revokes only the caller's grant option source" $ withObjectPrivileges $ \_ service root -> do
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount) testDatabase (RevokeGrantOptionCommand ["select"] "users" "bob") `shouldReturn` Right ()
grants <- listSelfGrants service (Ordinary testAccount{accountUser = "bob"})
grants `shouldSatisfy` either (const False) (elem (Grant "bob" "select" "test.users" True))
it "cascades multihop delegations after the source grant is revoked" $ withObjectPrivileges $ \_ service root -> do it "cascades multihop delegations after the source grant is revoked" $ withObjectPrivileges $ \_ service root -> do
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right () runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right () runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
@@ -1436,6 +1514,17 @@ intValues rows = [n | row <- rows, Just (VInt n) <- [nth 0 row]]
-- | 显式事务用例 -- | 显式事务用例
transactionSpec :: Spec transactionSpec :: Spec
transactionSpec = describe "server transactions" $ do transactionSpec = describe "server transactions" $ do
it "executes DISTINCT against the connection snapshot with output columns" $ withMemorySession $ \session -> do
_ <- mustSql session "CREATE TABLE distinct_values (id INT, value INT)"
_ <- mustSql session "INSERT INTO distinct_values (id,value) VALUES (1,7),(2,7),(3,8)"
_ <- mustSql session "BEGIN"
_ <- mustSql session "UPDATE distinct_values SET value = 9 WHERE id = 3"
result <- mustSql session "SELECT DISTINCT value FROM distinct_values ORDER BY 1"
qrColumns result `shouldBe` ["value"]
qrRows result `shouldBe` [[VInt 7], [VInt 9]]
_ <- mustSql session "ROLLBACK"
pure ()
it "preserves concurrent changes in a large untouched table" $ withTwoSessions $ \first second -> do it "preserves concurrent changes in a large untouched table" $ withTwoSessions $ \first second -> do
_ <- mustSql first "CREATE TABLE untouched (id INT, value INT)" _ <- mustSql first "CREATE TABLE untouched (id INT, value INT)"
let values = T.intercalate "," [T.pack ("(" ++ show key ++ ",0)") | key <- [1 .. 5000 :: Int]] let values = T.intercalate "," [T.pack ("(" ++ show key ++ ",0)") | key <- [1 .. 5000 :: Int]]
@@ -1458,8 +1547,8 @@ transactionSpec = describe "server transactions" $ do
qrRows result `shouldBe` [[VInt 7], [VInt (-3)], [VInt 4]] qrRows result `shouldBe` [[VInt 7], [VInt (-3)], [VInt 4]]
it "filters qualified memory tables and preserves other databases on writes" $ do it "filters qualified memory tables and preserves other databases on writes" $ do
let local = Table "users" [("id", TInt)] [[("id", VInt 1)]] Nothing let local = Table "users" [("id", TInt)] [[("id", VInt 1)]] Nothing Nothing Nothing
otherTable = Table "users" [("id", TInt)] [[("id", VInt 2)]] Nothing otherTable = Table "users" [("id", TInt)] [[("id", VInt 2)]] Nothing Nothing Nothing
ref <- newMVar [("test.users", local), ("other.users", otherTable)] ref <- newMVar [("test.users", local), ("other.users", otherTable)]
let backend = memoryBackend "test" ref let backend = memoryBackend "test" ref
selected <- beStatement backend "SELECT id FROM users" selected <- beStatement backend "SELECT id FROM users"
@@ -1470,11 +1559,36 @@ transactionSpec = describe "server transactions" $ do
fmap (const ()) updated `shouldBe` Right () fmap (const ()) updated `shouldBe` Right ()
snapshot <- beSnapshot (memoryBackend "other" ref) snapshot <- beSnapshot (memoryBackend "other" ref)
fmap (map (tableRows . snd)) snapshot `shouldBe` Right [[[ ("id", VInt 2) ]]] fmap (map (tableRows . snd)) snapshot `shouldBe` Right [[[ ("id", VInt 2) ]]]
it "keeps catalog and snapshot identities stable across system views and replacements" $ do
ref <- newMVar [("test.users", Table "users" [("id", TInt)] [] Nothing Nothing Nothing)]
let backend = memoryBackend "test" ref
first <- beCatalog backend
fmap (map tiIdentity) first `shouldBe` Right [Just (TableIdentity 1 1)]
_ <- beSnapshot (beWithDatabase backend "system")
saved <- beSnapshot backend
fmap (map (tableIdentity . snd)) saved `shouldBe` Right [Just (TableIdentity 1 1)]
mustRun (beStatement backend "DROP TABLE users")
mustRun (beStatement backend "CREATE TABLE users (id INT)")
replaced <- beCatalog backend
fmap (map tiIdentity) replaced `shouldBe` Right [Just (TableIdentity 1 2)]
it "preserves memory schema versions in catalogs and snapshots" $ do
ref <- newMVar []
let backend = memoryBackend "test" ref
fmap (const ()) <$> beStatement backend "CREATE TABLE versioned (id INT)" `shouldReturn` Right ()
first <- beCatalog backend
fmap (map tiSchemaVersion) first `shouldBe` Right [Just 1]
fmap (const ()) <$> beStatement backend "ALTER TABLE versioned ADD COLUMN extra TEXT" `shouldReturn` Right ()
fmap (const ()) <$> beStatement backend "ALTER TABLE versioned DROP COLUMN extra" `shouldReturn` Right ()
current <- beCatalog backend
snapshot <- beSnapshot backend
fmap (map tiSchemaVersion) current `shouldBe` Right [Just 3]
fmap (map (tableSchemaVersion . snd)) snapshot `shouldBe` Right [Just 3]
it "rejects cross-database reads and writes including nested queries" $ withMemorySession $ \session -> do it "rejects cross-database reads and writes including nested queries" $ withMemorySession $ \session -> do
mapM_ (\query -> do mapM_ (\query -> do
result <- runStatementCoded session query result <- runStatementCoded session query
failureMessage result `shouldSatisfy` T.isInfixOf "outside the current database") failureMessage result `shouldSatisfy` T.isInfixOf "outside the current database")
[ "SELECT * FROM other.users" [ "SELECT DISTINCT age FROM other.users"
, "SELECT * FROM other.users"
, "UPDATE other.users SET age = 1" , "UPDATE other.users SET age = 1"
, "DELETE FROM other.users" , "DELETE FROM other.users"
, "INSERT INTO other.users (id) VALUES (99)" , "INSERT INTO other.users (id) VALUES (99)"
@@ -1962,3 +2076,122 @@ checkSudoDisabled admin handle credential sql = do
isRightResult :: Either a b -> Bool isRightResult :: Either a b -> Bool
isRightResult (Right _) = True isRightResult (Right _) = True
isRightResult _ = False isRightResult _ = False
-- 专用范式协议、权限、快照与预算回归。
normalizationSpec :: Spec
normalizationSpec = describe "normalization endpoint" $ do
it "round-trips versioned requests and rejects unknown and duplicate fields" $ do
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload))) `shouldBe` Right (ReqNormalization normalizationPayload)
let large = ObjectIdentity 9007199254740993 18446744073709551615
A.fromJSON (A.toJSON large) `shouldBe` A.Success large
case A.eitherDecode "{\"database_id\":1e1000000000,\"object_id\":2}" :: Either String A.Value of
Left message -> expectationFailure message
Right value -> (A.fromJSON value :: A.Result ObjectIdentity) `shouldSatisfy` isJSONFailure
report <- either (fail . show) pure (normalizationReport (ObjectIdentity 1 2) [SchemaColumn name "int" True Nothing False False False Nothing | name <- ["id", "name", "age"]] "live_schema" normalizationPayload)
let response = RespNormalization report
decodeResponse (encodeResponse response) `shouldBe` Right response
decodeResponse (encodeResponse (RespNormalization (A.object ["conditional" A..= True]))) `shouldSatisfy` either (const True) (const False)
mapM_ (\raw -> decodeNormalizationJSON raw `shouldSatisfy` either (T.isInfixOf "duplicate") (const False))
[ "{\"a\":1,\"a\":2}", "{\"a\":1,\"\\u0061\":2}", "{\"a\":[{\"b\":1,\"b\":2}]}" ]
let unknown = A.object ["api_version" A..= (1 :: Int), "database" A..= ("test" :: Text), "table" A..= ("users" :: Text), "declaration" A..= nrDeclaration normalizationPayload, "ignored" A..= True]
(A.fromJSON unknown :: A.Result NormalizationRequest) `shouldSatisfy` isJSONFailure
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrVersion = 2}))) `shouldSatisfy` either (const True) (const False)
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrTable = "other.users"}))) `shouldSatisfy` either (const True) (const False)
it "does not infer completeness or atomicity and validates declaration budgets" $ do
(A.fromJSON (A.object []) :: A.Result NormalizationDeclaration) `shouldBe` A.Success (NormalizationDeclaration [] [] False Nothing)
let many = NormalizationDeclaration (replicate 257 ["id"]) [] True (Just True)
(A.fromJSON (A.toJSON many) :: A.Result NormalizationDeclaration) `shouldSatisfy` isJSONFailure
decodeNormalizationJSON (BL.replicate 65537 32) `shouldSatisfy` either (const True) (const False)
(A.fromJSON (A.object ["dependencies" A..= [A.object ["lhs" A..= (["id"] :: [Text]), "rhs" A..= ([] :: [Text])]]]) :: A.Result NormalizationDeclaration) `shouldSatisfy` isJSONFailure
it "authorizes ordinary SELECT users and refuses anonymous or system requests" $
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \admin -> do
anonymous <- tcpTalk admin (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
at "code" anonymous `shouldBe` A.String "unauthorized"
_ <- tcpLogin admin "admin" "s3cret"
_ <- tcpQuery admin "USE test"
_ <- tcpQuery admin "CREATE USER normalization_reader IDENTIFIED BY 'Reader-Secret-1234'"
withConnection handle $ \reader -> do
_ <- tcpLogin reader "normalization_reader" "Reader-Secret-1234"
_ <- tcpQuery reader "USE test"
denied <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
at "code" denied `shouldBe` A.String "forbidden"
_ <- tcpQuery admin "GRANT SELECT ON users TO normalization_reader"
withConnection handle $ \reader -> do
_ <- tcpLogin reader "normalization_reader" "Reader-Secret-1234"
_ <- tcpQuery reader "USE test"
reply <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
at "status" reply `shouldBe` A.String "normalization"
at "conditional" (at "analysis" reply) `shouldBe` A.Bool True
length (items (at "findings" (at "analysis" reply))) `shouldBe` 4
denied <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrTable = "__system_domain_secret"})))
at "code" denied `shouldBe` A.String "forbidden"
it "uses only the caller snapshot and refuses stable object replacement" $
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \first -> withConnection handle $ \second -> do
_ <- tcpLogin first "admin" "s3cret"
_ <- tcpLogin second "admin" "s3cret"
_ <- tcpQuery first "USE test"
_ <- tcpQuery second "USE test"
_ <- tcpQuery first "CREATE TABLE normalization_snapshot (id INT, value INT)"
_ <- tcpQuery first "BEGIN"
_ <- tcpQuery second "ALTER TABLE normalization_snapshot ADD COLUMN added INT"
let payload = normalizationPayload{nrTable = "normalization_snapshot", nrDeclaration = NormalizationDeclaration [] [] False Nothing}
old <- tcpTalk first (BL.toStrict (encodeRequest (ReqNormalization payload)))
current <- tcpTalk second (BL.toStrict (encodeRequest (ReqNormalization payload)))
at "view" (at "analysis" old) `shouldBe` A.String "transaction_snapshot"
length (items (at "schema" (at "analysis" old))) `shouldBe` 2
length (items (at "schema" (at "analysis" current))) `shouldBe` 3
at "schema_fingerprint" (at "analysis" old) `shouldNotBe` at "schema_fingerprint" (at "analysis" current)
_ <- tcpQuery second "DROP TABLE normalization_snapshot"
_ <- tcpQuery second "CREATE TABLE normalization_snapshot (id INT, value INT)"
stale <- tcpTalk first (BL.toStrict (encodeRequest (ReqNormalization payload)))
at "code" stale `shouldBe` A.String "conflict"
_ <- tcpQuery first "ROLLBACK"
pure ()
it "checks expected schema fingerprints and budgets the complete output" $ do
let columns = [SchemaColumn ("c" ++ show n) "int" True Nothing False False False Nothing | n <- [0 .. 9 :: Int]]
payload = normalizationPayload{nrDeclaration = NormalizationDeclaration [] [NormalizationDependency [] ["c9"]] True (Just True)}
schemaFingerprint (ObjectIdentity 1 2) columns `shouldNotBe` schemaFingerprint (ObjectIdentity 1 3) columns
normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload{nrExpectedSchema = Just "outdated"}
`shouldBe` Left ("conflict", "normalization object or schema has changed")
result <- normalizationBudget (pure (normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload))
result `shouldSatisfy` either ((== "too_large") . fst) (const False)
it "bounds the largest schema with maximum declared dependencies" $ do
let columns = [SchemaColumn ("c" ++ show n) "int" True Nothing False False False Nothing | n <- [0 .. 11 :: Int]]
payload = normalizationPayload{nrDeclaration = NormalizationDeclaration [] (replicate 256 (NormalizationDependency [] ["c11"])) True (Just True)}
started <- getCurrentTime
result <- normalizationBudget (pure (normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload))
ended <- getCurrentTime
putStrLn ("normalization maximum input seconds: " ++ show (ended `diffUTCTime` started))
result `shouldSatisfy` either ((`elem` ["too_large", "timeout"]) . fst) (const False)
normalizationBudget (pure (Right A.Null)) `shouldReturn` Right A.Null
it "releases admission and execution slots after cancellation" $ do
started <- newEmptyMVar
blocker <- newEmptyMVar
done <- newEmptyMVar
worker <- forkFinally (normalizationBudget (putMVar started () >> takeMVar blocker >> pure (Right A.Null))) (putMVar done)
takeMVar started
killThread worker
takeMVar done >>= either (const (pure ())) (const (expectationFailure "cancelled analysis completed"))
ready <- newEmptyMVar
release <- newEmptyMVar
finished <- newEmptyMVar
replicateM_ 2 (void (forkFinally (normalizationBudget (putMVar ready () >> takeMVar release >> pure (Right A.Null))) (putMVar finished)))
mapM_ (\_ -> timeout 1000000 (takeMVar ready) >>= (`shouldBe` Just ())) [1, 2 :: Int]
replicateM_ 2 (putMVar release ())
replicateM_ 2 (takeMVar finished >>= either (expectationFailure . show) (`shouldBe` Right A.Null))
it "times out work and leaves the service usable" $ do
result <- normalizationBudget (threadDelay 6000000 >> pure (Right A.Null))
result `shouldSatisfy` either ((== "timeout") . fst) (const False)
normalizationBudget (pure (Right A.Null)) `shouldReturn` Right A.Null
-- 共用明确声明的用户表请求。
normalizationPayload :: NormalizationRequest
normalizationPayload = NormalizationRequest 1 "test" "users"
(NormalizationDeclaration [["id"]] [NormalizationDependency ["id"] ["name", "age"]] True (Just True)) Nothing Nothing
-- 判断 JSON 类型校验失败。
isJSONFailure :: A.Result a -> Bool
isJSONFailure (A.Error _) = True
isJSONFailure _ = False
chusql-web/src/ChuSQL/Web/API.hs
+28 -5
View File
@@ -72,10 +72,12 @@ import ChuSQL.Interface.Link (
closeClient, closeClient,
connectClient, connectClient,
) )
import ChuSQL.Interface.Normalization (NormalizationRequest, decodeNormalizationJSON)
import ChuSQL.Interface.Protocol (Grant (..), RoleView (..)) import ChuSQL.Interface.Protocol (Grant (..), RoleView (..))
import ChuSQL.Interface.RateLimit (RateLimiter, rateLimitBlock, rateLimitClear, rateLimitRecord, setRateLimit) import ChuSQL.Interface.RateLimit (RateLimiter, rateLimitBlock, rateLimitClear, rateLimitRecord, setRateLimit)
import ChuSQL.Interface.Session ( import ChuSQL.Interface.Session (
Session, Session,
analyzeNormalization,
authenticateSession, authenticateSession,
catalog, catalog,
databases, databases,
@@ -132,7 +134,7 @@ import qualified Data.Text.Encoding.Error as TEE
import qualified Data.Text.Lazy as TL import qualified Data.Text.Lazy as TL
import Data.Time.Clock (getCurrentTime) import Data.Time.Clock (getCurrentTime)
import qualified Data.Vector as V import qualified Data.Vector as V
import Network.HTTP.Types (Status, status204, status400, status401, status403, status404, status409, status413, status429, status500) import Network.HTTP.Types (Status, status204, status400, status401, status403, status404, status409, status413, status429, status500, status503)
import Network.Wai (Application, Request (..)) import Network.Wai (Application, Request (..))
import Text.Read (readMaybe) import Text.Read (readMaybe)
import Web.Scotty import Web.Scotty
@@ -337,6 +339,7 @@ routes env = do
delete "/api/roles/:name/grants" (withDatabase env revokeRoleH) delete "/api/roles/:name/grants" (withDatabase env revokeRoleH)
post "/api/roles/:name/members" (withDatabase env addRoleMemberH) post "/api/roles/:name/members" (withDatabase env addRoleMemberH)
delete "/api/roles/:name/members/:user" (withDatabase env removeRoleMemberH) delete "/api/roles/:name/members/:user" (withDatabase env removeRoleMemberH)
post "/api/normalization/analyze" (withDatabase env normalizationH)
post "/api/query" (withDatabase env queryH) post "/api/query" (withDatabase env queryH)
post "/api/query/rollback" (rollbackConsoleH env) post "/api/query/rollback" (rollbackConsoleH env)
post "/api/edit/begin" (beginEditH env) post "/api/edit/begin" (beginEditH env)
@@ -357,12 +360,13 @@ withDatabase env action = do
else do else do
editing <- header "X-ChuSQL-Edit" editing <- header "X-ChuSQL-Edit"
req <- request req <- request
let snapshotRead = editing /= Nothing && requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"] let snapshotRead = editing /= Nothing && ((requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"])
|| (requestMethod req == "POST" && pathInfo req == ["api", "normalization", "analyze"]))
console <- liftIO (readIORef (wsConsole ws)) console <- liftIO (readIORef (wsConsole ws))
caller <- consoleOwner caller <- consoleOwner
case console of case console of
Just (owner, database) | not snapshotRead && Just (owner, database) | not snapshotRead &&
(pathInfo req /= ["api", "query"] || caller /= owner || (not (T.null chosen) && chosen /= database)) -> (pathInfo req `notElem` [["api", "query"], ["api", "normalization", "analyze"]] || caller /= owner || (not (T.null chosen) && chosen /= database)) ->
apiError "conflict" "another SQL console owns this connection transaction; commit or rollback there first" apiError "conflict" "another SQL console owns this connection transaction; commit or rollback there first"
_ -> pure () _ -> pure ()
current <- liftIO (sessionDatabase (wsSession ws)) current <- liftIO (sessionDatabase (wsSession ws))
@@ -391,7 +395,8 @@ requireSession env = do
editHeader <- fmap TL.toStrict <$> header "X-ChuSQL-Edit" editHeader <- fmap TL.toStrict <$> header "X-ChuSQL-Edit"
req <- request req <- request
case editHeader of case editHeader of
Just editId | requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"] -> do Just editId | (requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"])
|| (requestMethod req == "POST" && pathInfo req == ["api", "normalization", "analyze"]) -> do
active <- liftIO (readIORef (wsEdit ws)) active <- liftIO (readIORef (wsEdit ws))
database <- maybe "" (T.toLower . TL.toStrict) <$> header "X-ChuSQL-Database" database <- maybe "" (T.toLower . TL.toStrict) <$> header "X-ChuSQL-Database"
case active of case active of
@@ -905,7 +910,7 @@ serverError message
(rawCode, rest) = T.breakOn ":" message (rawCode, rest) = T.breakOn ":" message
code = T.strip rawCode code = T.strip rawCode
detail = T.strip (T.drop 1 rest) detail = T.strip (T.drop 1 rest)
knownCodes = ["unauthorized", "forbidden", "not_found", "conflict", "bad_request", "no_database", "query_error", "storage_error", "too_large"] knownCodes = ["unauthorized", "forbidden", "not_found", "conflict", "bad_request", "no_database", "query_error", "storage_error", "too_large", "too_busy", "timeout"]
-- | 取错误消息里的错误码 -- | 取错误消息里的错误码
errorCodeOf :: Text -> Text errorCodeOf :: Text -> Text
@@ -920,6 +925,8 @@ statusOf code = case code of
"conflict" -> status409 "conflict" -> status409
"too_many_attempts" -> status429 "too_many_attempts" -> status429
"too_large" -> status413 "too_large" -> status413
"too_busy" -> status503
"timeout" -> status503
"storage_error" -> status500 "storage_error" -> status500
"internal" -> status500 "internal" -> status500
_ -> status400 _ -> status400
@@ -1901,3 +1908,19 @@ data QueryRequest = QueryRequest
instance FromJSON QueryRequest where instance FromJSON QueryRequest where
parseJSON = withObject "QueryRequest" $ \o -> QueryRequest <$> o .: "sql" parseJSON = withObject "QueryRequest" $ \o -> QueryRequest <$> o .: "sql"
-- 使用现有连接视图转发只读范式请求。
normalizationH :: AppEnv -> ActionM ()
normalizationH env = do
ws <- requireSession env
rawBody <- body
case decodeNormalizationJSON rawBody of
Left message | BL.length rawBody > 65536 -> reply413
| otherwise -> reply400 "bad_request" message
Right value -> case A.fromJSON value :: A.Result NormalizationRequest of
A.Error message -> reply400 "bad_request" (T.pack message)
A.Success payload -> do
result <- liftIO (analyzeNormalization (wsSession ws) payload)
case result of
Left message -> serverError message
Right report -> json report
chusql-web/static/api.js
+21 -1
View File
@@ -21,7 +21,7 @@ function hasValidRows(body) {
async function request(url, init, database) { async function request(url, init, database) {
const headers = new Headers(init && init.headers ? init.headers : undefined); const headers = new Headers(init && init.headers ? init.headers : undefined);
if (database) headers.set('X-ChuSQL-Database', database); if (database) headers.set('X-ChuSQL-Database', database);
if (editing && editing.database === database && url.startsWith('/api/tables') && (!init || init.method === 'GET')) { if (editing && editing.database === database && ((url.startsWith('/api/tables') && (!init || init.method === 'GET')) || url === '/api/normalization/analyze')) {
headers.set('X-ChuSQL-Edit', editing.id); headers.set('X-ChuSQL-Edit', editing.id);
} }
let response; let response;
@@ -326,3 +326,23 @@ export async function loadUiSettings() {
export async function saveUiSettings(settings) { export async function saveUiSettings(settings) {
await request('/api/ui-settings', jsonInit('PUT', settings)); await request('/api/ui-settings', jsonInit('PUT', settings));
} }
// 在页面已有视图上请求只读范式分析。
export async function analyzeNormalization(database, table, declaration, expected) {
const payload = { api_version: 1, database, table };
if (expected) {
payload.expected_object = expected.object;
payload.expected_schema_fingerprint = expected.schema_fingerprint;
}
const init = jsonInit('POST', payload);
if (!editing && consoleTransaction?.database === database) {
init.headers['X-ChuSQL-Console'] = consoleTransaction.console;
}
// 原始声明直接嵌入，重复字段交给服务端明确拒绝。
init.body = `${JSON.stringify(payload).slice(0, -1)},"declaration":${typeof declaration === 'string' ? declaration : JSON.stringify(declaration)}}`;
const result = await request('/api/normalization/analyze', init, database);
if (!isObject(result) || result.api_version !== 1 || !Array.isArray(result.findings) || !Array.isArray(result.schema)) {
throw new Error('服务器返回的范式报告无法识别。');
}
return result;
}
chusql-web/static/app.js
+91 -2
View File
@@ -64,6 +64,8 @@ const state = {
dockHeight: '22vh', dockHeight: '22vh',
}; };
let normalizationTarget = null;
const root = document.getElementById('root'); const root = document.getElementById('root');
// ---------------------------------------------------------------- 小工具 // ---------------------------------------------------------------- 小工具
@@ -210,6 +212,21 @@ function renderShell() {
<button type="button" class="primary" data-action="create-index">创建</button> <button type="button" class="primary" data-action="create-index">创建</button>
</footer> </footer>
</div> </div>
</dialog>
<dialog id="dlg-normalization" class="dialog">
<div class="dialog-body">
<header class="dialog-header"><h2 id="normalization-title">范式分析</h2></header>
<div class="dialog-content">
<p>只读分析，以声明为条件。PRIMARY KEY、UNIQUE 和样本不会自动成为完整业务依赖；NULL 与重复行需要另行验证。</p>
<label class="field"><span>JSON 声明（完整性 false，原子性 null 表示未声明）</span><textarea id="normalization-declaration" rows="10"></textarea></label>
<p class="alert" id="normalization-error"></p>
<div id="normalization-result"></div>
</div>
<footer class="dialog-footer">
<button type="button" data-action="close-dialog">关闭</button>
<button type="button" class="primary" data-action="analyze-normalization" disabled>分析声明</button>
</footer>
</div>
</dialog>`; </dialog>`;
renderAccount(); renderAccount();
applyLayout(); applyLayout();
@@ -282,7 +299,7 @@ function renderSidebar() {
<button class="mini danger" data-action="drop-table" data-table="${escapeHtml(table.name)}" title="删除表">✕</button> <button class="mini danger" data-action="drop-table" data-table="${escapeHtml(table.name)}" title="删除表">✕</button>
</div> </div>
${open ? `<ul class="column-list">${columns}</ul>${indexList} ${open ? `<ul class="column-list">${columns}</ul>${indexList}
<div class="tree-actions"><button class="mini" data-action="open-index-dialog" data-table="${escapeHtml(table.name)}">＋ 索引</button></div>` : ''} <div class="tree-actions">${state.database !== 'system' && !table.name.startsWith('__system_') ? `<button class="mini" data-action="open-normalization" data-table="${escapeHtml(table.name)}">范式分析</button>` : ''}<button class="mini" data-action="open-index-dialog" data-table="${escapeHtml(table.name)}">＋ 索引</button></div>` : ''}
</li>`; </li>`;
}).join(''); }).join('');
@@ -1383,7 +1400,7 @@ function startSplitterDrag(kind, event) {
// 这些动作都要求已经选中一个库（服务没有默认库） // 这些动作都要求已经选中一个库（服务没有默认库）
const DATABASE_ACTIONS = new Set([ const DATABASE_ACTIONS = new Set([
'open-table', 'demo-data', 'open-table-dialog', 'create-table', 'open-normalization', 'analyze-normalization', 'open-table', 'demo-data', 'open-table-dialog', 'create-table',
'open-index-dialog', 'create-index', 'drop-table', 'drop-column', 'drop-index', 'open-index-dialog', 'create-index', 'drop-table', 'drop-column', 'drop-index',
'drop-database', 'commit', 'rollback', 'copy-sql', 'drop-database', 'commit', 'rollback', 'copy-sql',
]); ]);
@@ -1413,7 +1430,14 @@ function onClick(event) {
} }
switch (action) { switch (action) {
case 'open-normalization':
void openNormalization(target.dataset.table);
break;
case 'analyze-normalization':
void analyzeNormalizationDeclaration();
break;
case 'logout': case 'logout':
normalizationTarget = null;
void (async () => { void (async () => {
try { try {
await api.logout(); await api.logout();
@@ -1790,3 +1814,68 @@ async function boot() {
} }
void boot(); void boot();
// 打开结构入口并绑定当前 schema 身份。
async function openNormalization(table) {
const declaration = { declared_keys: [], dependencies: [], dependencies_complete: false, atomic_attributes: null };
const target = { database: state.database, table, report: null };
normalizationTarget = target;
region('dlg-normalization').querySelector('[data-action="analyze-normalization"]').disabled = true;
region('normalization-title').textContent = `范式分析 · ${target.database}.${table}`;
region('normalization-declaration').value = JSON.stringify(declaration, null, 2);
region('normalization-result').textContent = '正在读取结构…';
region('normalization-error').textContent = '';
region('dlg-normalization').onclose = () => { normalizationTarget = null; };
region('dlg-normalization').showModal();
try {
target.report = await api.analyzeNormalization(target.database, table, declaration);
if (normalizationTarget === target) {
renderNormalization(target.report);
region('dlg-normalization').querySelector('[data-action="analyze-normalization"]').disabled = false;
}
} catch (error) {
if (normalizationTarget === target) {
region('normalization-result').textContent = '';
region('normalization-error').textContent = error.message;
}
}
}
// 提交显式声明并复核绑定的对象结构。
async function analyzeNormalizationDeclaration() {
const target = normalizationTarget;
if (!target?.report) return;
const button = region('dlg-normalization').querySelector('[data-action="analyze-normalization"]');
button.disabled = true;
region('normalization-error').textContent = '';
try {
const declaration = region('normalization-declaration').value;
const report = await api.analyzeNormalization(target.database, target.table, declaration, target.report);
if (normalizationTarget === target) {
target.report = report;
renderNormalization(report);
}
} catch (error) {
if (normalizationTarget === target) {
region('normalization-error').textContent = error.message;
if (error.code === 'conflict') {
target.report = null;
region('normalization-result').textContent = '对象或视图已变化，请关闭后重新打开范式分析。';
}
}
} finally {
button.disabled = !target.report;
}
}
// 展示条件结论、违反证据与一次分解。
function renderNormalization(report) {
const labels = { satisfied: '满足', violated: '违反', insufficient: '信息不足' };
const schema = report.schema.map((column) => `${column.name}: ${column.ty}${column.nullable ? '（可空）' : '（非空）'}`).join('，');
const findings = report.findings.map((finding) => `<li><strong>${escapeHtml(finding.normal_form.toUpperCase())}: ${escapeHtml(labels[finding.verdict] ?? finding.verdict)}</strong>
<p>${escapeHtml(finding.condition)}</p><pre>${escapeHtml(JSON.stringify(finding.witnesses, null, 2))}</pre></li>`).join('');
region('normalization-result').innerHTML = `<p>${escapeHtml(schema)}</p><p>视图：${escapeHtml(report.view)} · 结论以声明为条件</p>
<details><summary>结构与约束</summary><pre>${escapeHtml(JSON.stringify(report.schema, null, 2))}</pre></details>
<p>相对于声明的候选键：${escapeHtml(JSON.stringify(report.candidate_keys))}</p><ul>${findings}</ul>
<p>一次分解建议（无损标签不证明现有 SQL 数据）：</p><pre>${escapeHtml(JSON.stringify(report.suggestions, null, 2))}</pre>`;
}
chusql-web/test/Spec.hs
+96
View File
@@ -946,6 +946,102 @@ roleSpec env = describe "roles and grants" $ do
querySpec :: AppEnv -> Spec querySpec :: AppEnv -> Spec
querySpec env = describe "the SQL console" $ do querySpec env = describe "the SQL console" $ do
withApp (webApp env) $ do withApp (webApp env) $ do
it "analyzes explicit declarations for SELECT users and rejects stale schemas" $ do
admin <- adminHeaders
ensureDatabase admin testDatabaseName
ensureAccount admin "normalization_web_reader" "Reader-Secret-1234"
created <- testSql admin "CREATE TABLE normalization_web (id INT, course INT, name TEXT)"
expectStatusWith "normalization table" created 200
let declaration = object ["declared_keys" .= ([["id", "course"]] :: [[Text]]),
"dependencies" .= [object ["lhs" .= (["id"] :: [Text]), "rhs" .= (["name"] :: [Text])]],
"dependencies_complete" .= True, "atomic_attributes" .= True]
payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
"table" .= ("normalization_web" :: Text), "declaration" .= declaration]
unauthorized <- postAs (withDb testDatabaseName []) "/api/normalization/analyze" (encode payload)
expectStatusWith "anonymous normalization" unauthorized 401
reader <- ordinaryHeaders "normalization_web_reader" "Reader-Secret-1234"
denied <- postAs (withDb testDatabaseName reader) "/api/normalization/analyze" (encode payload)
expectStatusWith "SELECT required" denied 403
granted <- testSql admin "GRANT SELECT ON normalization_web TO normalization_web_reader"
expectStatusWith "normalization SELECT" granted 200
readerAgain <- ordinaryHeaders "normalization_web_reader" "Reader-Secret-1234"
report <- postAs (withDb testDatabaseName readerAgain) "/api/normalization/analyze" (encode payload)
expectStatusWith "conditional report" report 200
check (at "conditional" (jsonBody report) `shouldBe` A.Bool True)
check (map (asText . at "verdict") (items (at "findings" (jsonBody report))) `shouldBe` ["satisfied", "violated", "violated", "violated"])
changed <- testSql admin "ALTER TABLE normalization_web ADD COLUMN extra INT"
expectStatusWith "change normalization schema" changed 200
let stale = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
"table" .= ("normalization_web" :: Text), "declaration" .= declaration,
"expected_object" .= at "object" (jsonBody report), "expected_schema_fingerprint" .= at "schema_fingerprint" (jsonBody report)]
conflicted <- postAs (withDb testDatabaseName readerAgain) "/api/normalization/analyze" (encode stale)
expectStatusWith "schema changed" conflicted 409
it "uses a matching editing snapshot without creating a second transaction" $ do
admin <- adminHeaders
ensureDatabase admin testDatabaseName
created <- testSql admin "CREATE TABLE normalization_edit (id INT, value INT)"
expectStatusWith "normalization snapshot table" created 200
begun <- editRequest admin "/api/edit/begin" "normalization-owner" []
expectStatusWith "normalization edit begins" begun 200
changed <- testSql admin "ALTER TABLE normalization_edit ADD COLUMN extra INT"
expectStatusWith "live schema changes" changed 200
let payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
"table" .= ("normalization_edit" :: Text), "declaration" .= object []]
snapshot <- postAs (("X-ChuSQL-Edit", "normalization-owner") : withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
expectStatusWith "analysis on edit connection" snapshot 200
check (at "view" (jsonBody snapshot) `shouldBe` A.String "transaction_snapshot")
check (length (items (at "schema" (jsonBody snapshot))) `shouldBe` 2)
forged <- postAs (("X-ChuSQL-Edit", "another-page") : withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
expectStatusWith "page identity mismatch" forged 409
rolledBack <- editRequest admin "/api/edit/rollback" "normalization-owner" []
expectStatusWith "release edit snapshot" rolledBack 200
it "binds normalization to the owning SQL console transaction" $ do
admin <- adminHeaders
ensureDatabase admin testDatabaseName
created <- testSql admin "CREATE TABLE normalization_console (id INT)"
expectStatusWith "console normalization table" created 200
let owner = ("X-ChuSQL-Console", "normalization-console") : admin
payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
"table" .= ("normalization_console" :: Text), "declaration" .= object []]
begun <- testSql owner "BEGIN"
expectStatusWith "console begins" begun 200
snapshot <- postAs (withDb testDatabaseName owner) "/api/normalization/analyze" (encode payload)
expectStatusWith "owner console normalization" snapshot 200
check (at "view" (jsonBody snapshot) `shouldBe` A.String "transaction_snapshot")
denied <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
expectStatusWith "unrelated console refused" denied 409
rolledBack <- testSql owner "ROLLBACK"
expectStatusWith "console ends" rolledBack 200
it "rejects duplicate declarations and oversized normalization bodies" $ do
admin <- adminHeaders
ensureDatabase admin testDatabaseName
duplicate <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze"
"{\"api_version\":1,\"database\":\"test\",\"table\":\"users\",\"declaration\":{\"atomic_attributes\":true,\"atomic_attributes\":false}}"
expectStatusWith "duplicate declaration fields" duplicate 400
oversized <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze" (BL.replicate 65537 32)
expectStatusWith "normalization request bound" oversized 413
it "exposes DISTINCT and self grants after grant option revocation" $ do
admin <- adminHeaders
ensureDatabase admin testDatabaseName
ensureAccount admin "http_option_reader" "Reader-Secret-1234"
created <- testSql admin "CREATE TABLE http_distinct_probe (id INT, value INT)"
expectStatusWith "distinct table" created 200
inserted <- testSql admin "INSERT INTO http_distinct_probe (id,value) VALUES (1,7),(2,7),(3,8)"
expectStatusWith "distinct rows" inserted 200
granted <- testSql admin "GRANT SELECT ON http_distinct_probe TO http_option_reader WITH GRANT OPTION"
expectStatusWith "grant option" granted 200
revoked <- testSql admin "REVOKE GRANT OPTION FOR SELECT ON http_distinct_probe FROM http_option_reader"
expectStatusWith "revoke only grant option" revoked 200
reader <- ordinaryHeaders "http_option_reader" "Reader-Secret-1234"
result <- testSql reader "SELECT DISTINCT value FROM http_distinct_probe ORDER BY 1 DESC LIMIT 2"
expectStatusWith "distinct output" result 200
check (at "rows" (jsonBody result) `shouldBe` A.toJSON ([[8], [7]] :: [[Int]]))
grants <- testSql reader "SHOW GRANTS"
expectStatusWith "own grants" grants 200
check (at "columns" (jsonBody grants) `shouldBe` A.toJSON (["identity", "privilege", "object", "grant_option"] :: [Text]))
denied <- testSql reader "GRANT SELECT ON http_distinct_probe TO admin"
expectStatusWith "no delegation" denied 403
it "returns typed compound values through HTTP and TCP" $ do it "returns typed compound values through HTTP and TCP" $ do
hs <- adminHeaders hs <- adminHeaders
ensureDatabase hs testDatabaseName ensureDatabase hs testDatabaseName
chusql-web/test/transactions.mjs
+50 -1
View File
@@ -12,7 +12,7 @@ globalThis.document = { getElementById: (id) => id === 'root' ? { addEventListen
let source = await readFile(new URL('../static/app.js', import.meta.url), 'utf8'); let source = await readFile(new URL('../static/app.js', import.meta.url), 'utf8');
source = source.replaceAll("'./api.js'", JSON.stringify(new URL('../static/api.js', import.meta.url).href)) source = source.replaceAll("'./api.js'", JSON.stringify(new URL('../static/api.js', import.meta.url).href))
.replaceAll("'./core.js'", JSON.stringify(new URL('../static/core.js', import.meta.url).href)) .replaceAll("'./core.js'", JSON.stringify(new URL('../static/core.js', import.meta.url).href))
.replace('void boot();', 'export { state, commitAll, rollbackAll, stageUpdate, removeStaged, ensureEdit, runSql };'); .replace('void boot();', 'export { state, commitAll, rollbackAll, stageUpdate, removeStaged, ensureEdit, runSql, renderNormalization };');
const app = await import(`data:text/javascript;base64,${Buffer.from(source).toString('base64')}`); const app = await import(`data:text/javascript;base64,${Buffer.from(source).toString('base64')}`);
let calls; let calls;
let commitResponse; let commitResponse;
@@ -173,3 +173,52 @@ test('edits are locked while SQL releases the previous snapshot', async () => {
assert.equal(app.state.editBusy, false); assert.equal(app.state.editBusy, false);
assert.equal(app.state.edit, null); assert.equal(app.state.edit, null);
}); });
// 校验分析复用编辑视图并保留严格声明的原始字段。
test('normalization keeps snapshot ownership, exact identities and duplicate declaration fields', async () => {
await api.beginEdit('normalization-page', 'test');
const expected = { object: { database_id: '9007199254740993', object_id: '18446744073709551615' }, schema_fingerprint: 'schema' };
globalThis.fetch = async (url, init) => {
calls.push({ url, init });
return response({ api_version: 1, findings: [], schema: [] });
};
await api.analyzeNormalization('test', 'users', '{"atomic_attributes":true,"atomic_attributes":false}', expected);
const sent = calls.find((call) => call.url === '/api/normalization/analyze');
assert.equal(new Headers(sent.init.headers).get('X-ChuSQL-Edit'), 'normalization-page');
assert.equal(new Headers(sent.init.headers).get('X-ChuSQL-Database'), 'test');
assert.equal((sent.init.body.match(/"atomic_attributes"/g) ?? []).length, 2);
assert.deepEqual(JSON.parse(sent.init.body).expected_object, expected.object);
assert.equal(calls.filter((call) => call.url === '/api/edit/begin').length, 1);
api.forgetEdit();
globalThis.fetch = async (url, init) => {
calls.push({ url, init });
return response(url === '/api/query'
? { columns: [], rows: [], rowCount: 0, truncated: false, transaction: true }
: { api_version: 1, findings: [], schema: [] });
};
await api.query('BEGIN', 'test', 'normalization-console');
await api.analyzeNormalization('test', 'users', '{}', expected);
const fromConsole = calls.filter((call) => call.url === '/api/normalization/analyze').at(-1);
const begun = calls.filter((call) => call.url === '/api/query').at(-1);
assert.equal(new Headers(fromConsole.init.headers).get('X-ChuSQL-Console'), new Headers(begun.init.headers).get('X-ChuSQL-Console'));
assert.equal(new Headers(fromConsole.init.headers).get('X-ChuSQL-Edit'), null);
});
// 校验结构、违反证据和分解中的用户文字全部转义。
test('normalization escapes schema names, conditions, witnesses and suggestions', () => {
const result = { innerHTML: '' };
const original = document.getElementById;
document.getElementById = (id) => id === 'normalization-result' ? result : original(id);
try {
app.renderNormalization({ schema: [{ name: '<img src=x onerror=alert(1)>', ty: 'str', nullable: true }],
view: 'live_schema', candidate_keys: [['<script>']],
findings: [{ normal_form: 'bcnf', verdict: 'violated', condition: '<script>alert(1)</script>', witnesses: [{ lhs: ['<img>'], rhs: ['id'] }] }],
suggestions: [{ relations: [['<script>'], ['id']] }] });
assert.equal(result.innerHTML.includes('<img'), false);
assert.equal(result.innerHTML.includes('<script>'), false);
assert.equal(result.innerHTML.includes('&lt;img'), true);
assert.equal(result.innerHTML.includes('结构与约束'), true);
} finally {
document.getElementById = original;
}
});
docs/architecture.md
+1 -1
View File
@@ -117,7 +117,7 @@ TCP 连接在下一次请求时返回 unauthorized 并关闭；空闲连接不
## 查询引擎扩展设计（尚未实现） ## 查询引擎扩展设计（尚未实现）
按 bound IR → 子查询物化 / 递归 CTE 推进；DOMAIN 约束见[类型文档](types.md#domain-约束设计尚未实现)。以下设计尚未实施，当前能力见前文；实施状态以 agent-credientials/progress.txt 为准。 按 bound IR → 子查询物化 / 递归 CTE 推进；DOMAIN 约束见[类型文档](types.md#domain-约束设计尚未实现)。以下设计尚未实施，当前能力见前文；实施状态以 .agents/context/progress.txt 为准。
### bound IR ### bound IR
docs/commands.md
+2
View File
@@ -44,6 +44,8 @@ csql [options]
| `-d, --database DB` | 登录后选择数据库 | | `-d, --database DB` | 登录后选择数据库 |
| `-f, --format FORMAT` | 输出格式：`table`、`json` 或 `csv` | | `-f, --format FORMAT` | 输出格式：`table`、`json` 或 `csv` |
| `-e, --execute SQL` | 执行一条 SQL 后退出 | | `-e, --execute SQL` | 执行一条 SQL 后退出 |
| `--normalize TABLE` | 只读分析表的范式；与 `--execute` 互斥，需 `--declaration` |
| `--declaration FILE` | UTF-8 JSON 业务依赖声明；范式输出使用 `--format text` 或 `json` |
| `--history FILE` | 指定历史记录文件 | | `--history FILE` | 指定历史记录文件 |
| `--sudo` | 使用本机 root / 提升管理员身份认证；需配置映射并启用身份属性，见 [用户与角色](users-and-roles.md#本机-sudo-认证) | | `--sudo` | 使用本机 root / 提升管理员身份认证；需配置映射并启用身份属性，见 [用户与角色](users-and-roles.md#本机-sudo-认证) |
| `--config FILE` | 指定配置文件 | | `--config FILE` | 指定配置文件 |
docs/course-defense.md
+8
View File
@@ -0,0 +1,8 @@
# ChuSQL 技术总结与数据库课程答辩
完整学习与答辩材料已集中到仓库根目录的 `learnning/`：
- [分章节学习目录](../learnning/README.md)
- [答辩总览](../learnning/00-defense-overview.md)
- [关键函数导航](../learnning/17-function-atlas.md)
- [现场演示与排练](../learnning/14-demonstration.md)
docs/install.md
+22
View File
@@ -105,6 +105,28 @@ csql-web
管理员使用安装时设置的口令登录。 管理员使用安装时设置的口令登录。
## 更新
重新运行安装脚本，指定相同的安装目录和组件即可更新，也可用 `--version TAG` 选择版本：
```sh
./install.sh --component both --install-dir "$HOME/.local/share/chusql" --version <tag>
```
Linux 更新会先停止该安装目录中的 Web 和数据库进程，保留已有 `settings.toml`、数据库和管理员口令，再引导检查系统目录。修改数据位置须编辑配置并自行迁移数据；更新时不能用不同的 `--data-dir` 覆盖现有配置。安装默认重启数据库服务，Web 管理端需重新运行 `csql-web`。
Linux 安装目录中保留 `uninstall.sh`。卸载时指定安装目录，数据目录默认读取现有配置。
已构建的 Linux 包可用隔离验证脚本测试，不修改正常安装的配置或数据：
```sh
python3 scripts/test-linux-lifecycle.py releases/chusql-linux-x86_64.tar.gz
```
该验证需要 Python 3，使用真实数据库程序，覆盖安装、运行中更新、配置与 SQL 数据保留、保留数据卸载、重装和完全卸载；基础验证通过 `--no-service` 隔离 systemd 服务。
添加 `--systemd` 可验证真实用户服务的注册、启动、更新及卸载。此选项要求不存在已有的 `chusql-server.service`，并且当前用户能够启用 linger（或已由管理员启用）；临时单元链接会在测试后清理。本机 HTTP 发行站夹具用于验证在线安装流程，不依赖 GitHub 上已有 Release。
## 卸载 ## 卸载
Linux / macOS： Linux / macOS：
docs/normalization.md
+36 -11
View File
@@ -84,33 +84,58 @@ schema 类型不证明业务意义上的原子性。不满足 1NF 时更高范
建议只是一轮分解，子关系可能仍需继续分析，不承诺一次达到 BCNF。依赖不保持时需跨关系检查原约束。 建议只是一轮分解，子关系可能仍需继续分析，不承诺一次达到 BCNF。依赖不保持时需跨关系检查原约束。
## 第一阶段限制 ## 当前限制
- 最多接受 12 列，超过返回错误，避免指数枚举无界增长。 - 最多接受 12 列，超过返回错误，避免指数枚举无界增长。
- 仅提供 Haskell 只读 API。 - 提供 Haskell 只读 API、TCP 专用请求、CLI 和 Web 结构入口；不执行数据迁移。
- CLI / Web 入口和自动迁移尚未实现。 - 输入与输出有预算，超过预算返回明确错误，不截断为完整报告。
## 后续入口设计（尚未实现） ## TCP、CLI 与 Web 入口
以下 TCP、CLI 和 Web 接口尚未实现，拟议命令与端点目前不可用。实施状态以 agent-credientials/progress.txt 为准。 以下只读入口已实现。分析只读取 schema，不能据此认定 SQL 数据满足声明或直接执行分解。实施状态以 .agents/context/progress.txt 为准。
### 协议与鉴权 ### 协议与鉴权
在 `ChuSQL.Interface.Protocol` 增加专用客户端请求 `normalization_analyze`，由 server 调用 engine API；不经最高权限专用的原始 `ReqStorage` 入口。请求包含 `api_version=1`、数据库与表标识、声明，以及可选的预期对象编号和 schema 指纹。server 在现有权限读门内以 catalog 解析现存对象，检查 CONNECT 与 SELECT；系统对象拒绝访问。事务内取该连接的 schema 快照，返回视图来源和对象编号，不能混入其他连接的快照。 `ChuSQL.Interface.Protocol` 的专用客户端请求 `normalization_analyze` 由 server 调用 engine API；不经最高权限专用的原始 `ReqStorage` 入口。请求包含 `api_version=1`、数据库与表标识、声明，以及可选的预期对象编号和 schema 指纹。server 在现有权限读门内以 catalog 解析现存对象，检查 CONNECT 与 SELECT；系统对象拒绝访问。事务内取该连接的 schema 快照，返回视图来源和对象编号，不能混入其他连接的快照。
声明字段固定为 `declared_keys`、`dependencies`（`lhs` / `rhs`）、`dependencies_complete`、`atomic_attributes`。完整性缺省 false，原子性缺省 null；不把 PRIMARY KEY、UNIQUE 或样本自动当作完整业务依赖。前端可以展示实际 schema 约束供用户选择，确认后才成为声明。候选键的 nullable 与 SQL UNIQUE 语义单独提示，不自动证明经典关系模型的键。 声明字段固定为 `declared_keys`、`dependencies`（`lhs` / `rhs`）、`dependencies_complete`、`atomic_attributes`。完整性缺省 false，原子性缺省 null；不把 PRIMARY KEY、UNIQUE 或样本自动当作完整业务依赖。前端可以展示实际 schema 约束供用户选择，确认后才成为声明。候选键的 nullable 与 SQL UNIQUE 语义单独提示，不自动证明经典关系模型的键。
响应包含对象编号、schema 指纹、规范化声明、候选键、四项 findings 和一次分解建议。稳定标签使用 `1nf/2nf/3nf/bcnf`、`satisfied/violated/insufficient`；条件说明与机器字段分开。所有结论标明“以声明为条件”，无损连接标签不代表现有 SQL 数据或袋语义已经验证。 响应包含对象编号、schema 指纹、规范化声明、候选键、四项 findings 和一次分解建议。稳定标签使用 `1nf/2nf/3nf/bcnf`、`satisfied/violated/insufficient`；条件说明与机器字段分开。所有结论标明“以声明为条件”，无损连接标签不代表现有 SQL 数据或袋语义已经验证。
字段名称必须与返回 schema 的确切列名一致；对象名称使用现有标识符规则，数据库限定名必须与请求数据库一致。指纹只覆盖列名、类型和约束等语义结构，排除行数与直方图；定义确定性编码并同时绑定稳定对象编号，避免同名重建复用报告。 字段名称必须与返回 schema 的确切列名一致；对象名称使用现有标识符规则，数据库限定名必须与请求数据库一致。`schema_fingerprint` 使用 SHA256，编码为 `[库/表编号字符串数组, 列结构数组]`；每列按名称、类型、nullable、default、auto_increment、primary_key、unique、check 排列。它排除行数、索引统计与直方图，并绑定稳定对象编号，避免同名重建复用报告。对象编号在报告中使用 UInt64 范围内的正整数十进制字符串，避免浏览器数值精度损失。
拒绝未知协议版本、未知字段、重复字段、空右部、非最小声明键、空 schema 和超过 12 列。拟定请求上限 64 KiB、依赖条数 256、键声明 256；预算超限返回明确错误，不截断后给出完整报告。分析在有界工作队列运行，取消或超时释放权限读门与会话资源；输出也受传输预算约束，超限返回错误。实施时用最坏 12 列声明确定默认时间预算。 拒绝未知协议版本、未知字段、重复字段、空右部、非最小声明键、空 schema 和超过 12 列。请求 JSON 上限 64 KiB、嵌套深度上限 64；依赖与键声明分别最多 256 项。服务端最多接纳 8 个请求，最多 2 个同时执行；排队、权限门等待与计算合计最多 5 秒。完整报告超过 64000 字节即拒绝，预留协议外层开销；客户端请求也为外层协议预留空间。超限返回 too_large，队列满返回 too_busy，超时返回 timeout，不截断成完整报告。取消执行或超时均释放执行槽和权限读门；断线请求最多运行至预算到期。
请求外层形状：
```json
{"method":"normalization_analyze","request":{"api_version":1,"database":"sales","table":"orders","declaration":{"declared_keys":[["id"]],"dependencies":[],"dependencies_complete":false,"atomic_attributes":null}}}
```
成功响应的 `status` 为 `normalization`，`analysis` 保存完整报告；可选 `expected_object` 与 `expected_schema_fingerprint` 放在 `request` 中。数据库必须与该连接的工作库一致；表可使用裸名或同库限定名。没有登录返回 unauthorized，系统对象拒绝分析；不会通过另一个连接代取事务视图。
### CLI 与 Web ### CLI 与 Web
CLI 拟议元命令为 `\normalize TABLE --declaration FILE --format text|json`，通过当前登录连接和工作库请求。声明文件为 UTF-8 JSON，错误写 stderr；成功分析退出码 0，即使结论违反或信息不足；协议、权限、输入与运行错误退出非零。批处理增加独立 `--normalize` 模式，与 `--execute` 互斥，不把元命令发给 SQL 解析器。 CLI 元命令为 `\normalize TABLE --declaration FILE --format text|json`，通过当前登录连接和工作库请求。声明文件为 UTF-8 JSON，错误写 stderr；成功分析退出码 0，即使结论违反或信息不足；协议、权限、输入与运行错误退出非零。批处理使用独立 `--normalize` 模式，与 `--execute` 互斥，不把元命令发给 SQL 解析器。
Web 拟议 `POST /api/normalization/analyze`，复用登录、CSRF 和 Cookie 串行规则。入口位于表结构页：展示 schema、编辑声明、提交分析、显示证据和分解字段列表。完整性与原子性不预先勾选；没有自动执行分解按钮。编辑页分析必须携带既有页面事务标识，不能自动开启另一事务。对象或 schema 改变返回冲突并要求重新分析。 Web 使用 `POST /api/normalization/analyze`，复用登录、CSRF 和 Cookie 串行规则。入口位于资源管理器展开表后的结构区：展示 schema、编辑声明、提交分析、显示证据和分解字段列表。完整性与原子性不预先勾选；没有自动执行分解按钮。编辑页分析必须携带既有页面事务标识，不能自动开启另一事务；SQL 控制台有事务时，分析请求携带该控制台身份并使用其快照。对象或 schema 改变返回冲突并要求重新分析。
声明文件示例（不自动声明完整性或原子性）：
```json
{
"declared_keys": [["student", "course"]],
"dependencies": [{"lhs": ["student"], "rhs": ["name"]}],
"dependencies_complete": true,
"atomic_attributes": true
}
```
```sh
csql -d school --normalize enrollment --declaration dependencies.json --format json
```
交互模式：`\normalize enrollment --declaration "My Data/dependencies.json" --format text`。成功报告即使违反范式或信息不足也退出 0；输入、权限、协议或预算错误退出非零。文本输出保留与 JSON 输出相同的报告字段。Web 在资源管理器展开表后点击“范式分析”，输入声明 JSON；初始声明为空，完整性 false、原子性 null。报告给出候选键、违反证据和一次分解建议，只有分析按钮。
### 入口验收 ### 入口验收
@@ -118,7 +143,7 @@ Web 拟议 `POST /api/normalization/analyze`，复用登录、CSRF 和 Cookie
## 数据迁移设计（尚未实现） ## 数据迁移设计（尚未实现）
先实现只读入口，再在 [DOMAIN 约束](types.md#domain-约束设计尚未实现)、依赖目录和原子结构发布具备后实施迁移。 只读入口已具备；在 [DOMAIN 约束](types.md#domain-约束设计尚未实现)、依赖目录和原子结构发布具备后实施迁移。
### 规划条件 ### 规划条件
docs/release-signing.md
+1 -1
View File
@@ -1,6 +1,6 @@
# 发布签名与验证设计 # 发布签名与验证设计
本文定义尚未实现的发行安全方案。当前操作见[安装文档](install.md)，实施状态以 agent-credientials/progress.txt 为准。先实现离线验证，再接入在线安装；该工作可独立于查询和存储扩展推进。 本文定义尚未实现的发行安全方案。当前操作见[安装文档](install.md)，实施状态以 .agents/context/progress.txt 为准。先实现离线验证，再接入在线安装；该工作可独立于查询和存储扩展推进。
## 选择与信任入口 ## 选择与信任入口
docs/runtime-types.md
+23
View File
@@ -55,11 +55,34 @@ Rust 的通用行、目录和 WAL 路径保存该表示，读取时由共享模
`Runtime.Binding` 提供独立的 TypedExpression 和 TypedPlan，绑定只接受 catalog schema。 `Runtime.Binding` 提供独立的 TypedExpression 和 TypedPlan，绑定只接受 catalog schema。
表达式和计划构造器隐藏，调用者必须经过 Binder；执行使用已解析的 TypeId / FunctionId，并校验存储行。 表达式和计划构造器隐藏，调用者必须经过 Binder；执行使用已解析的 TypeId / FunctionId，并校验存储行。
该 API 覆盖单行、表扫描、投影和过滤，可构造嵌套复合值及函数调用。 该 API 覆盖单行、表扫描、投影和过滤，可构造嵌套复合值及函数调用。
列引用按当前输入 schema 的槽位绑定，列名保留用于诊断和位置校验。独立调用 `evaluateExpression`
必须按绑定时的 schema 顺序提供行；位置不符或类型不符明确报错。`executePlan` 在扫描边界
校验完整行并按 schema 排序，投影后的表达式重新绑定到子计划输出槽位。
SQL SELECT 通过 `Runtime.Query.bindQuery` 生成不可直接构造的 BoundQuery，再由 `executeQueryM` 执行。 SQL SELECT 通过 `Runtime.Query.bindQuery` 生成不可直接构造的 BoundQuery，再由 `executeQueryM` 执行。
绑定计划保存输出 schema 与类型身份；现有 RelOp 作为关系执行和优化的兼容桥，输出按位置校验，保留重复别名。 绑定计划保存输出 schema 与类型身份；现有 RelOp 作为关系执行和优化的兼容桥，输出按位置校验，保留重复别名。
类型化函数和构造器节点也进入 INSERT、UPDATE、CHECK、聚合参数和子查询。 类型化函数和构造器节点也进入 INSERT、UPDATE、CHECK、聚合参数和子查询。
日期、时间戳、Blob 等阶段 A 外的类型保留明确的 CompatibilityType。 日期、时间戳、Blob 等阶段 A 外的类型保留明确的 CompatibilityType。
`queryTypes` 用 `DomainType 名称 基础类型` 保留每一层 DOMAIN 引用，包括嵌套领域和兼容类型。
直接列引用及派生表别名保留该身份；函数结果使用函数声明的返回类型，不继承参数的 DOMAIN。
领域名称来自当前库 schema 的解析结果；定义版本与约束绑定仍待完整 bound IR 实现。
SQL 计划在优化完成后由 `Runtime.Relation` 将投影、过滤、计算、聚合参数和连接条件中的列引用
绑定为 `ColumnSlot`。扫描、点查和范围查询在存储边界按算子布局排列返回行；相关子查询
按自己的局部布局和外层行布局绑定槽位。局部引用使用 ColumnSlot，外层引用使用 OuterColumnSlot，
保存局部布局宽度与外层相对位置；重新绑定时外层引用只在外层布局中查找，防止同名局部列捕获。
执行槽位不接受语义层再次推导，语义输入必须使用未绑定列。哈希/索引连接仍使用列名识别等值键。
投影单向转换为已有 Compute 算子，使用绑定列槽位取值；保留输出顺序、重复列及空投影的行数。
排序转换为 BoundSort，分组转换为 BoundAggregate；键使用绑定槽位，列名保留为输出标签和诊断。
排序每行计算一次键并保留同键行序，分组保留首次出现顺序及空输入全局聚合行为。
该步骤仍复用 Expr/RelOp 和扁平化外层行；多级词法作用域/来源编号及 schema 版本尚未引入。
绑定后的计划用于执行；结构改写必须在绑定前完成，或重新按新布局绑定。
执行前重新取得存储 schema，比较查询依赖表的完整列定义；依赖收集覆盖来源树、连接条件、
投影和过滤表达式里的嵌套查询，并递归追踪来源列使用的 DOMAIN 定义。缺表、列定义变化或 schema
读取失败明确返回错误，行数据、统计及无关表、无关 DOMAIN 的结构变化不使绑定失效。
Table 与 TableInfo 保留 catalog 的库编号和对象编号；执行前比较身份，可识别同名同结构的对象替换。
内存 Server 目录和存储快照同样保留身份。TableInfo 输出十进制字符串编号，读取兼容数字编号，
校验无符号 64 位范围；旧响应缺少编号或原始目录对象编号为零时，以列定义校验作为兼容路径。
身份与列定义校验不能替代服务端读写门或未来的原子 schema 版本校验。
旧 ColumnType 和运算符实现尚未整体移除；统一 capability、cast 和 rewrite 将在后续阶段收拢。 旧 ColumnType 和运算符实现尚未整体移除；统一 capability、cast 和 rewrite 将在后续阶段收拢。
测试覆盖稳定编码、kind / 参数校验、共享协议、SQL 调用和复合列、实际存储重载、HTTP/TCP 与客户端显示。 测试覆盖稳定编码、kind / 参数校验、共享协议、SQL 调用和复合列、实际存储重载、HTTP/TCP 与客户端显示。
docs/syntax.md
+16 -1
View File
@@ -17,9 +17,20 @@
- `SELECT` 支持 `WHERE`、`ORDER BY`、`LIMIT`、`JOIN` / `LEFT JOIN`、`GROUP BY` 与聚合，以及标量 / `IN` / `EXISTS` 子查询和量词比较（`op ANY` / `op ALL`）。 - `SELECT` 支持 `WHERE`、`ORDER BY`、`LIMIT`、`JOIN` / `LEFT JOIN`、`GROUP BY` 与聚合，以及标量 / `IN` / `EXISTS` 子查询和量词比较（`op ANY` / `op ALL`）。
- 来源可以是派生表：`SELECT d.name FROM (SELECT name FROM users) d`。派生表必须带别名，内层不能引用外层列；输出列取裸列名（内层写 `u.id`，外层用 `d.id` 引用）。 - 来源可以是派生表：`SELECT d.name FROM (SELECT name FROM users) d`。派生表必须带别名，内层不能引用外层列；输出列取裸列名（内层写 `u.id`，外层用 `d.id` 引用）。
- 投影列可以起别名：`SELECT count(*) AS total FROM orders ORDER BY total`，别名就是结果集的列名。`ORDER BY` 优先使用唯一的投影别名，再查来源列；重复的排序别名报歧义错误，来源列可以用限定名消除同名冲突。排序暂不接受任意表达式或列序号。 - 投影列可以起别名：`SELECT count(*) AS total FROM orders ORDER BY total`，别名就是结果集的列名。`ORDER BY` 优先使用唯一的投影别名，再查来源列；重复的排序别名报歧义错误，来源列可以用限定名消除同名冲突。排序接受从 1 开始的输出列序号（例如 `ORDER BY 2 DESC`），在 `*` 展开后解析，零或越界序号报错；暂不接受任意表达式。
- 鉴权覆盖整棵来源树：查派生表时内层表也要有权限，只有外层引用的表有权限不够。 - 鉴权覆盖整棵来源树：查派生表时内层表也要有权限，只有外层引用的表有权限不够。
### DISTINCT 与 ALL
`SELECT DISTINCT` 按完整输出值组合去重，保留首次出现的行，然后执行排序和 LIMIT。NULL 在去重时视为相同值；等值整数与整值浮点按既有数值语义合并。默认 SELECT 或显式 `SELECT ALL` 保留重复行。
```sql
SELECT DISTINCT user_id FROM orders ORDER BY 1 LIMIT 10;
SELECT DISTINCT user_id + 1 AS next_id FROM orders ORDER BY next_id;
```
DISTINCT 的排序列必须出现在输出中；排序可使用输出别名、选出的来源列或输出列序号。缺少比较能力的 List / Maybe / Tuple 类型仍拒绝 DISTINCT。去重支持连接、聚合、子查询、CTE 和当前连接的事务快照。
### 比较运算符 ### 比较运算符
- 支持 `=`、`<>`（也可写 `!=`）、`>`、`>=`、`<`、`<=`；`<>` 与 `!=` 是同一个运算符。 - 支持 `=`、`<>`（也可写 `!=`）、`>`、`>=`、`<`、`<=`；`<>` 与 `!=` 是同一个运算符。
@@ -115,6 +126,8 @@ ALTER ROLE analyst NOSUPERUSER;
SHOW ROLES; SHOW ROLES;
``` ```
`SHOW GRANTS` 允许登录身份查看自己的有效对象权限，输出 `identity`、`privilege`、`object` 和 `grant_option`。结果包含直接授权、启用角色继承、PUBLIC 默认 CONNECT 和所有权隐含权限；表范围授权按现存表展开，超级用户展示全部业务对象权限。系统对象不列出，不接受 `FOR` 查询他人身份。该命令不要求选中工作库；事务中也读取当前权限目录。
`SHOW ROLES` 展示全部身份的编号、名称、登录能力、最高权限标志、启用状态及创建时间，不返回口令。`ALTER USER` 与 `ALTER ROLE` 的属性和口令修改语法通用；同一属性不能重复指定。身份管理仅允许最高权限身份执行。 `SHOW ROLES` 展示全部身份的编号、名称、登录能力、最高权限标志、启用状态及创建时间，不返回口令。`ALTER USER` 与 `ALTER ROLE` 的属性和口令修改语法通用；同一属性不能重复指定。身份管理仅允许最高权限身份执行。
`SUPERUSER` 不依赖启动参数里的管理员名称（`--user`），也不会通过角色成员关系继承。禁用登录身份会拒绝登录；禁用权限角色会停止贡献继承权限。属性修改会使受影响的既有连接失效，下一次请求返回 `unauthorized` 后关闭。最后一个启用且可以登录的最高权限身份不能删除、禁用、取消登录或取消最高权限。 `SUPERUSER` 不依赖启动参数里的管理员名称（`--user`），也不会通过角色成员关系继承。禁用登录身份会拒绝登录；禁用权限角色会停止贡献继承权限。属性修改会使受影响的既有连接失效，下一次请求返回 `unauthorized` 后关闭。最后一个启用且可以登录的最高权限身份不能删除、禁用、取消登录或取消最高权限。
@@ -152,6 +165,8 @@ REVOKE SELECT ON sales.orders FROM reviewer;
`GRANT ... WITH GRANT OPTION` 允许再转授；所有者也可授予自身对象的权限。普通授予者可撤销自己的授予，所有者及最高权限身份可撤销对象上的其他授予。撤销时收回相应 grant option，并级联清理失去合法来源的下游授权；仍有独立来源的授权保留。转授权能力随启用的角色继承，循环转授不能独立维持权限。REVOKE 不移除所有者的隐含权限。 `GRANT ... WITH GRANT OPTION` 允许再转授；所有者也可授予自身对象的权限。普通授予者可撤销自己的授予，所有者及最高权限身份可撤销对象上的其他授予。撤销时收回相应 grant option，并级联清理失去合法来源的下游授权；仍有独立来源的授权保留。转授权能力随启用的角色继承，循环转授不能独立维持权限。REVOKE 不移除所有者的隐含权限。
`REVOKE GRANT OPTION FOR SELECT ON users FROM alice` 只移除指定授权来源的转授权能力，保留 alice 的 SELECT 权限；失去唯一支持来源的下游授权会级联撤销，独立来源继续有效。最高权限身份或对象所有者可撤销全部匹配来源，普通授予者只能撤销自己的来源；所有权自身的权限不受 REVOKE 影响。
系统目录访问还受目录管理属性控制，具体能力见 [用户与角色](users-and-roles.md)。 系统目录访问还受目录管理属性控制，具体能力见 [用户与角色](users-and-roles.md)。
gate-check/README.md
+1 -1
View File
@@ -42,4 +42,4 @@ Web 步骤会先显式刷新该工程里的 model、engine、interface 与 serve
## 约定 ## 约定
- 门禁由项目所有者手动运行；agent 只跑模块级测试，不跑全量门禁。 - 门禁由项目所有者手动运行；agent 只跑模块级测试，不跑全量门禁。
- 计数、验收状态与主线进度以 `agent-credientials/progress.txt` 为准。 - 计数、验收状态与主线进度以 `.agents/context/progress.txt` 为准。
hie.yaml
+47
View File
@@ -0,0 +1,47 @@
# 为 HLS 指定各源码目录的 Stack 组件，使用现有构建配置解析本地包。
cradle:
multi:
- path: "./benchmark"
config:
cradle:
stack:
stackYaml: "./benchmark/stack.yaml"
component: "benchmark4csql:exe:benchmark4csql"
- path: "./"
config:
cradle:
stack:
stackYaml: "./chusql-cli/stack.yaml"
components:
- path: "./chusql-core/model/src"
component: "chusql-core-model:lib"
- path: "./chusql-core/engine/src"
component: "chusql-core-engine:lib"
- path: "./chusql-core/engine/test"
component: "chusql-core-engine:test:chusql-core-engine-test"
- path: "./chusql-interface/src"
component: "chusql-interface:lib"
- path: "./chusql-server/src"
component: "chusql-server:lib"
- path: "./chusql-server/app"
component: "chusql-server:exe:chusql-server"
- path: "./chusql-server/test"
component: "chusql-server:test:chusql-server-test"
- path: "./chusql-bootstrap/src"
component: "chusql-bootstrap:lib"
- path: "./chusql-bootstrap/app"
component: "chusql-bootstrap:exe:csql-bootstrap"
- path: "./chusql-bootstrap/test"
component: "chusql-bootstrap:test:chusql-bootstrap-test"
- path: "./chusql-web/src"
component: "chusql-web:lib"
- path: "./chusql-web/app"
component: "chusql-web:exe:chusql-web"
- path: "./chusql-web/test"
component: "chusql-web:test:chusql-web-test"
- path: "./chusql-cli/src"
component: "chusql-cli:lib"
- path: "./chusql-cli/app"
component: "chusql-cli:exe:csql"
- path: "./chusql-cli/test"
component: "chusql-cli:test:chusql-cli-test"
scripts/install.ps1
+1 -1
View File
@@ -169,7 +169,7 @@ if (-not $Component) {
# CHUSQL_ADMIN_PASSWORD。口令只经 stdin 传给 csql-bootstrap，不落 settings.toml。 # CHUSQL_ADMIN_PASSWORD。口令只经 stdin 传给 csql-bootstrap，不落 settings.toml。
$adminPassword = "$env:CHUSQL_ADMIN_PASSWORD" $adminPassword = "$env:CHUSQL_ADMIN_PASSWORD"
if ($Password) { $adminPassword = $Password } if ($Password) { $adminPassword = $Password }
if ($canPrompt) { if ($canPrompt -and $adminPassword.Length -eq 0) {
Write-Host '' Write-Host ''
Write-Host ('The administrator account is "{0}".' -f $RootUser) Write-Host ('The administrator account is "{0}".' -f $RootUser)
while ($true) { while ($true) {
scripts/install.sh
+60 -20
View File
@@ -28,14 +28,13 @@ usage: ./install.sh [--component web|cli|both] [options]
asks (English, y/n), command line client first: asks (English, y/n), command line client first:
command line client (csql)? default yes [Y/n] command line client (csql)? default yes [Y/n]
web front end (browser UI)? default no [y/N] web front end (browser UI)? default no [y/N]
it then asks whether to give the administrator account a it then asks for the administrator password; it goes into the
password (default no); that password goes straight into the
system catalog and is never written to settings.toml system catalog and is never written to settings.toml
--interactive ask even when there is no terminal; the answers are --interactive ask even when there is no terminal; the answers are
read from stdin (one per line) read from stdin (one per line)
--install-dir DIR where to install (default: $HOME/.local/share/chusql) --install-dir DIR where to install (default: $HOME/.local/share/chusql)
--data-dir DIR where to keep data (default: <install-dir>/data) --data-dir DIR where to keep data (default: XDG data directory)
--user NAME administrator name (default: root) --user NAME administrator name (default: root)
--keep-package do not delete the downloaded/extracted package --keep-package do not delete the downloaded/extracted package
--password VALUE password of the administrator account (required when --password VALUE password of the administrator account (required when
@@ -132,6 +131,19 @@ case "$self" in
esac esac
[ -n "$install_dir" ] || install_dir="$HOME/.local/share/chusql" [ -n "$install_dir" ] || install_dir="$HOME/.local/share/chusql"
config_file="${XDG_CONFIG_HOME:-$HOME/.config}/ChuSQL/settings.toml"
if [ "$list_versions" = 'no' ] && [ -f "$config_file" ]; then
requested_data_dir=$data_dir
data_dir=$(awk '/^[[:space:]]*\[storage\]/ { storage=1; next } /^[[:space:]]*\[/ { storage=0 } storage && /^[[:space:]]*data_dir[[:space:]]*=[[:space:]]*"[^"\\]*"[[:space:]]*(#.*)?$/ { sub(/^[^=]*=[[:space:]]*"/, ""); sub(/"[[:space:]]*(#.*)?$/, ""); print; exit }' "$config_file")
[ -n "$data_dir" ] || fail 'existing settings.toml needs an explicit double-quoted storage.data_dir without escapes'
case "$data_dir" in /*) ;; *) data_dir="$install_dir/$data_dir" ;; esac
if [ -n "$requested_data_dir" ]; then
requested_data_dir=$(CDPATH= cd -- "$requested_data_dir" && pwd -P) || fail 'requested data directory does not exist'
existing_data_dir=$(CDPATH= cd -- "$data_dir" && pwd -P) || fail 'configured data directory does not exist'
[ "$requested_data_dir" = "$existing_data_dir" ] ||
fail 'existing settings.toml: edit its data_dir instead of passing a different --data-dir'
fi
fi
[ -n "$data_dir" ] || data_dir="${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data" [ -n "$data_dir" ] || data_dir="${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data"
log_dir="${XDG_STATE_HOME:-$HOME/.local/state}/chusql/logs" log_dir="${XDG_STATE_HOME:-$HOME/.local/state}/chusql/logs"
@@ -213,7 +225,7 @@ if [ "$list_versions" = 'no' ]; then
if [ -n "$password_arg" ]; then if [ -n "$password_arg" ]; then
admin_password="$password_arg" admin_password="$password_arg"
fi fi
if [ "$can_prompt" = 'yes' ]; then if [ "$can_prompt" = 'yes' ] && [ -z "$admin_password" ]; then
echo echo
echo "The administrator account is $root_user." echo "The administrator account is $root_user."
while :; do while :; do
@@ -511,7 +523,7 @@ if [ "$from_source" = 'yes' ]; then
mkdir -p "$pack/bin" "$pack/resources" mkdir -p "$pack/bin" "$pack/resources"
cp "$storage_lib" "$pack/bin/" cp "$storage_lib" "$pack/bin/"
cp "$src/resources/settings.toml.linux" "$pack/resources/" cp "$src/resources/settings.toml.linux" "$pack/resources/"
cp "$src/scripts/install.sh" "$pack/" cp "$src/scripts/install.sh" "$src/scripts/uninstall.sh" "$pack/"
for comp in $wanted; do for comp in $wanted; do
if [ "$comp" = 'web' ]; then if [ "$comp" = 'web' ]; then
comp_dir='chusql-web'; front_name='chusql-web' comp_dir='chusql-web'; front_name='chusql-web'
@@ -626,15 +638,37 @@ if [ "$(uname -s)" = 'Linux' ] && [ "$no_service" = 'no' ]; then
command -v systemctl >/dev/null 2>&1 || fail 'systemd is required; use --no-service to opt out' command -v systemctl >/dev/null 2>&1 || fail 'systemd is required; use --no-service to opt out'
command -v loginctl >/dev/null 2>&1 || fail 'loginctl is required for boot autostart' command -v loginctl >/dev/null 2>&1 || fail 'loginctl is required for boot autostart'
systemctl --user show-environment >/dev/null || fail 'cannot connect to user systemd; run in a login session or use --no-service' systemctl --user show-environment >/dev/null || fail 'cannot connect to user systemd; run in a login session or use --no-service'
loginctl enable-linger "$(id -un)" || fail 'cannot enable boot autostart; run sudo loginctl enable-linger for this OS user and retry' os_user=$(id -un)
linger=$(loginctl show-user "$os_user" -p Linger --value) || fail 'cannot query boot autostart for this OS user'
if [ "$linger" != 'yes' ]; then
loginctl enable-linger "$os_user" || fail 'cannot enable boot autostart; run sudo loginctl enable-linger for this OS user and retry'
fi
use_service='yes' use_service='yes'
if [ -f "$service_file" ]; then if [ -f "$service_file" ]; then
systemctl --user stop chusql-server.service || fail 'cannot stop the existing service' systemctl --user stop chusql-server.service || fail 'cannot stop the existing service'
fi fi
fi fi
# 停止安装目录中的进程，避免更新运行中的文件。
stop_installed() {
for executable in chusql-web chusql-server; do
for process in /proc/[0-9]*/exe; do
target=$(readlink "$process" 2>/dev/null) || continue
[ "$target" = "$install_dir/bin/$executable" ] || continue
pid=${process#/proc/}; pid=${pid%/exe}
kill "$pid" || fail "cannot stop $executable ($pid)"
attempts=0
while [ -e "$process" ]; do
attempts=$((attempts + 1))
[ "$attempts" -lt 100 ] || fail "timed out stopping $executable ($pid)"
sleep 0.1
done
done
done
}
step 'Installing files' step 'Installing files'
mkdir -p "$install_dir/bin" "$log_dir" "$data_dir" mkdir -p "$install_dir/bin" "$log_dir" "$data_dir"
install_dir=$(CDPATH= cd -- "$install_dir" && pwd) install_dir=$(CDPATH= cd -- "$install_dir" && pwd -P)
if [ "$(uname -s)" = Linux ]; then stop_installed; fi
data_dir=$(CDPATH= cd -- "$data_dir" && pwd) data_dir=$(CDPATH= cd -- "$data_dir" && pwd)
log_dir=$(CDPATH= cd -- "$log_dir" && pwd) log_dir=$(CDPATH= cd -- "$log_dir" && pwd)
cp "$pack/bin/chusql-server" "$install_dir/bin/" cp "$pack/bin/chusql-server" "$install_dir/bin/"
@@ -657,7 +691,11 @@ if [ "$want_web" = 'yes' ]; then
chmod +x "$install_dir/csql-web.sh" chmod +x "$install_dir/csql-web.sh"
fi fi
fi fi
chmod +x "$install_dir/bin/"* 2>/dev/null || true chmod +x "$install_dir/bin/"*
if [ -f "$pack/uninstall.sh" ]; then
cp "$pack/uninstall.sh" "$install_dir/"
chmod +x "$install_dir/uninstall.sh"
fi
# ---- 写全局配置 ---- # ---- 写全局配置 ----
step 'Writing settings.toml' step 'Writing settings.toml'
@@ -666,9 +704,11 @@ config_file="$config_dir/settings.toml"
mkdir -p "$config_dir" mkdir -p "$config_dir"
config_dir=$(CDPATH= cd -- "$config_dir" && pwd) config_dir=$(CDPATH= cd -- "$config_dir" && pwd)
config_file="$config_dir/settings.toml" config_file="$config_dir/settings.toml"
sed -e "s|@DATA_DIR@|$(printf '%s' "$data_dir" | sed 's/[\&|]/\\&/g')|g" \ if [ ! -f "$config_file" ]; then
sed -e "s|@DATA_DIR@|$(printf '%s' "$data_dir" | sed 's/[\&|]/\\&/g')|g" \
-e "s|@LOG_DIR@|$(printf '%s' "$log_dir" | sed 's/[\&|]/\\&/g')|g" \ -e "s|@LOG_DIR@|$(printf '%s' "$log_dir" | sed 's/[\&|]/\\&/g')|g" \
"$pack/resources/settings.toml.linux" > "$config_file" "$pack/resources/settings.toml.linux" > "$config_file"
fi
echo " config file $config_file" echo " config file $config_file"
# ---- 引导系统目录 ---- # ---- 引导系统目录 ----
@@ -676,7 +716,7 @@ echo " config file $config_file"
step 'Initializing the administrator identity and preinstalled types' step 'Initializing the administrator identity and preinstalled types'
bootstrap="$install_dir/bin/csql-bootstrap" bootstrap="$install_dir/bin/csql-bootstrap"
[ -n "$admin_password" ] || fail 'the administrator password is required' [ -n "$admin_password" ] || fail 'the administrator password is required'
printf '%s\n' "$admin_password" | "$bootstrap" --config "$config_file" --user "$root_user" --password-stdin || ( cd "$install_dir" && printf '%s\n' "$admin_password" | "$bootstrap" --config "$config_file" --user "$root_user" --password-stdin ) ||
fail 'the bootstrap program failed; identities or preinstalled types are not ready' fail 'the bootstrap program failed; identities or preinstalled types are not ready'
# ---- 命令入口 ---- # ---- 命令入口 ----
@@ -700,17 +740,17 @@ fi
step 'Updating PATH' step 'Updating PATH'
profile="$HOME/.profile" profile="$HOME/.profile"
marker='# ChuSQL' marker='# ChuSQL'
if [ -f "$profile" ] && grep -q "^$marker" "$profile"; then if [ -f "$profile" ] && grep -q "^$marker$" "$profile"; then
echo "PATH entry already present in $profile" awk -v m="$marker" 'skip > 0 { skip--; next } $0 == m { skip=2; next } { print }' "$profile" > "$profile.chusql-install"
else mv "$profile.chusql-install" "$profile"
{ fi
{
echo '' echo ''
echo "$marker" echo "$marker"
echo "PATH=\"\$PATH:$install_dir\"" echo "PATH=\"\$PATH:$install_dir:$install_dir/bin\""
echo "export PATH" echo "export PATH"
} >> "$profile" } >> "$profile"
echo "added $install_dir to PATH in $profile (login again or: . \"$profile\")" echo "added $install_dir to PATH in $profile (login again or: . \"$profile\")"
fi
# ---- 删掉安装包 ---- # ---- 删掉安装包 ----
if [ "$keep_package" = 'no' ]; then if [ "$keep_package" = 'no' ]; then
@@ -753,7 +793,7 @@ Description=ChuSQL database server
[Service] [Service]
Type=simple Type=simple
WorkingDirectory="$(unit_quote "$install_dir")" WorkingDirectory=$(printf '%s' "$install_dir" | sed 's/%/%%/g')
ExecStart=:"$(unit_quote "$server_bin")" --config "$(unit_quote "$config_file")" --user "$(unit_quote "$root_user")" ExecStart=:"$(unit_quote "$server_bin")" --config "$(unit_quote "$config_file")" --user "$(unit_quote "$root_user")"
Restart=on-failure Restart=on-failure
RestartSec=5 RestartSec=5
@@ -778,7 +818,7 @@ elif [ "$no_start" = 'yes' ]; then
else else
step 'Starting the server' step 'Starting the server'
mkdir -p "$log_dir" mkdir -p "$log_dir"
nohup "$server_bin" --config "$config_file" --user "$root_user" > "$log_dir/server.log" 2> "$log_dir/server.err.log" & ( cd "$install_dir" && exec nohup "$server_bin" --config "$config_file" --user "$root_user" ) > "$log_dir/server.log" 2> "$log_dir/server.err.log" &
server_pid=$! server_pid=$!
sleep 1 sleep 1
if ! kill -0 "$server_pid" 2>/dev/null; then if ! kill -0 "$server_pid" 2>/dev/null; then
scripts/test-linux-lifecycle.py
+215
View File
@@ -0,0 +1,215 @@
#!/usr/bin/env python3
import json
from functools import partial
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
import os
from pathlib import Path
import shutil
import socket
import subprocess
import sys
import tempfile
import threading
import time
# 用真实发行包验证 Linux 安装、更新及卸载，数据与配置隔离在临时目录。
# 执行命令并将失败日志原样输出。
def run(args, env, cwd=None):
result = subprocess.run(args, env=env, cwd=cwd, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60)
if result.returncode:
raise RuntimeError(result.stdout)
return result.stdout
# 选择本机空闲端口。
def free_port():
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
# 登录真实服务器并执行 SQL。
def query(port, statements):
with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
with sock.makefile("rwb") as stream:
answers = []
for request in [{"method": "hello", "protocol": 1},
{"method": "login", "user": "root", "password": "Lifecycle-test-123!"}] + [
{"method": "query", "sql": sql} for sql in statements]:
stream.write(json.dumps(request).encode() + b"\n")
stream.flush()
answer = json.loads(stream.readline())
assert answer["status"] != "error", answer
answers.append(answer)
return answers[2:]
# 等待服务器可接受登录。
def wait_server(port):
deadline = time.monotonic() + 15
while time.monotonic() < deadline:
try:
query(port, [])
return
except (OSError, ValueError):
time.sleep(0.1)
raise RuntimeError("server did not become ready")
# 用真实用户 systemd 验证注册、更新启动和卸载。
def test_systemd(repo, archive, root, env):
systemctl = shutil.which("systemctl")
state = run([systemctl, "--user", "show", "chusql-server.service", "-p", "LoadState"], env)
assert "LoadState=not-found" in state, "existing chusql-server.service: refusing to change it"
runtime_unit = Path(os.environ["XDG_RUNTIME_DIR"]) / "systemd/user/chusql-server.service"
assert not runtime_unit.is_symlink(), "existing runtime service link: refusing to change it"
user = run(["id", "-un"], env).strip()
linger = run(["loginctl", "show-user", user, "-p", "Linger"], env).strip()
home = root / "service-home"
home.mkdir()
env = dict(env, HOME=str(home), XDG_CONFIG_HOME=str(home / ".config"),
XDG_STATE_HOME=str(home / ".local/state"), CHUSQL_ADMIN_PASSWORD="Lifecycle-test-123!")
install = root / "service program"
unit = home / ".config/systemd/user/chusql-server.service"
bridge = root / "service-tools"
bridge.mkdir()
wrapper = bridge / "systemctl"
wrapper.write_text(f'''#!/bin/sh
# 将隔离配置中的单元链接到用户 systemd 的运行目录。
if [ "$*" = '--user daemon-reload' ] && [ -f "$CHUSQL_TEST_UNIT" ]; then
"{systemctl}" --user link --runtime "$CHUSQL_TEST_UNIT" || exit 1
fi
exec "{systemctl}" "$@"
''')
wrapper.chmod(0o755)
env.update(PATH=str(bridge) + ":" + env["PATH"], CHUSQL_TEST_UNIT=str(unit))
installer = ["sh", str(repo / "scripts/install.sh"), "--url", archive,
"--component", "both", "--install-dir", str(install)]
try:
run(installer + ["--data-dir", str(root / "service-data"), "--no-start"], env)
config = home / ".config/ChuSQL/settings.toml"
port = free_port()
config.write_text(config.read_text().replace("port = 7777", f"port = {port}"))
run(installer, env)
wait_server(port)
run([systemctl, "--user", "is-active", "--quiet", "chusql-server.service"], env)
run(["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"], env)
assert not unit.exists() and not (root / "service-data").exists()
print("PASS real systemd registration, update/start and uninstall", flush=True)
finally:
state = run([systemctl, "--user", "show", "chusql-server.service", "-p", "LoadState"], env)
if "LoadState=loaded" in state:
run([systemctl, "--user", "stop", "chusql-server.service"], env)
user_units = Path(os.environ["HOME"]) / ".config/systemd/user"
for link in [runtime_unit, user_units / "chusql-server.service",
user_units / "default.target.wants/chusql-server.service"]:
if link.is_symlink() and link.resolve() == unit.resolve():
link.unlink()
run([systemctl, "--user", "daemon-reload"], dict(env, CHUSQL_TEST_UNIT="/nonexistent"))
if linger == "Linger=no" and run(["loginctl", "show-user", user, "-p", "Linger"], env).strip() == "Linger=yes":
run(["loginctl", "disable-linger", user], env)
# 用本机发行站验证版本查询、HTTP 下载与校验。
def test_online(repo, archive, root, env):
site = root / "release-site"
api = site / "api/v3/repos/test/ChuSQL/releases"
api.parent.mkdir(parents=True)
api.write_text(json.dumps([{"tag_name": "v-test", "prerelease": False,
"assets": [{"name": "chusql-linux-x86_64.tar.gz"}]}], indent=2))
assets = site / "test/ChuSQL/releases/download/v-test"
assets.mkdir(parents=True)
shutil.copy2(archive, assets / Path(archive).name)
shutil.copy2(archive + ".sha256", assets / (Path(archive).name + ".sha256"))
server = ThreadingHTTPServer(("127.0.0.1", 0), partial(SimpleHTTPRequestHandler, directory=str(site)))
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
install = root / "online-program"
env = dict(env, CHUSQL_GITHUB_BASE=f"http://127.0.0.1:{server.server_port}")
installer = ["sh", str(repo / "scripts/install.sh"), "--repo", "test/ChuSQL"]
try:
assert "v-test" in run(installer + ["--list-versions"], env)
output = run(installer + ["--component", "cli", "--install-dir", str(install),
"--data-dir", str(root / "online-data"), "--no-service", "--no-start"], env)
assert "checksum" in output and "(ok)" in output
assert (install / "bin/csql").exists() and not (install / "bin/chusql-web").exists()
run(["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"], env)
assert not (root / "online-data").exists()
print("PASS release API, latest version selection, HTTP download and checksum (local fixture)", flush=True)
finally:
server.shutdown()
server.server_close()
thread.join()
# 在隔离目录验证配置、SQL 数据和卸载保留选项。
def main():
if sys.platform != "linux" or len(sys.argv) not in (2, 3) or (len(sys.argv) == 3 and sys.argv[2] != "--systemd"):
raise SystemExit("usage: python3 scripts/test-linux-lifecycle.py <linux-package.tar.gz> [--systemd]")
archive = str(Path(sys.argv[1]).resolve())
repo = Path(__file__).resolve().parent.parent
(repo / "logs").mkdir(exist_ok=True)
with tempfile.TemporaryDirectory(prefix="linux-lifecycle-", dir=repo / "logs") as temporary:
root = Path(temporary)
home = root / "home"
home.mkdir()
env = dict(os.environ, HOME=str(home), XDG_CONFIG_HOME=str(home / ".config"),
XDG_DATA_HOME=str(home / ".local/share"), XDG_STATE_HOME=str(home / ".local/state"),
CHUSQL_ADMIN_PASSWORD="Lifecycle-test-123!")
env.pop("LD_LIBRARY_PATH", None)
install = root / "program"
data = install / "custom/nested/data"
config = home / ".config/ChuSQL/settings.toml"
installer = ["sh", str(repo / "scripts/install.sh"), "--url", archive,
"--install-dir", str(install), "--component", "both", "--no-service", "--no-start"]
run(installer + ["--data-dir", str(data)], env)
assert (data / "system/catalog.json").is_file()
assert (install / "uninstall.sh").is_file()
assert "@DATA_DIR@" not in config.read_text()
assert run([str(install / "bin/csql-bootstrap"), "--help"], env)
print("PASS clean installation and dynamic library lookup", flush=True)
port = free_port()
settings = config.read_text().replace("port = 7777", f"port = {port}").replace(str(data), "custom/nested/data") + "\n# preserve custom settings\n"
config.write_text(settings)
server = subprocess.Popen([str(install / "bin/chusql-server"), "--config", str(config)],
env=env, cwd=install, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
wait_server(port)
query(port, ["create database lifecycle", "use lifecycle", "create table saved (id int)",
"insert into saved (id) values (42)"])
env["CHUSQL_ADMIN_PASSWORD"] = "Different-update-password-123!"
run(installer, env)
server.wait(timeout=5)
assert config.read_text() == settings
assert (home / ".profile").read_text().count("# ChuSQL\n") == 1
server = subprocess.Popen([str(install / "bin/chusql-server"), "--config", str(config)],
env=env, cwd=install, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
wait_server(port)
result = query(port, ["use lifecycle", "select id from saved"])[-1]
assert result["rowCount"] == 1 and "42" in json.dumps(result["rows"]), result
print("PASS running update preserves configuration, credentials and SQL data", flush=True)
uninstall = ["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"]
run(uninstall + ["--keep-data", "--keep-config"], env)
server.wait(timeout=5)
assert data.is_dir() and config.read_text() == settings
assert not (install / "bin").exists()
assert "# ChuSQL" not in (home / ".profile").read_text()
print("PASS uninstall stops server and keeps nested data and configuration", flush=True)
run(installer, env)
run(uninstall, env)
assert not install.exists() and not data.exists() and not config.exists()
print("PASS reinstall and full uninstall discover configured data path", flush=True)
finally:
if server.poll() is None:
server.terminate()
server.wait(timeout=10)
test_online(repo, archive, root, env)
if len(sys.argv) == 3:
with tempfile.TemporaryDirectory(prefix="chusql-systemd-") as service_temporary:
test_systemd(repo, archive, Path(service_temporary), env)
if __name__ == "__main__":
main()
scripts/uninstall.sh
+40 -14
View File
@@ -17,7 +17,7 @@ Uninstall ChuSQL (Linux/macOS).
Usage: uninstall.sh [options] Usage: uninstall.sh [options]
--install-dir DIR program directory (default: $HOME/.local/share/chusql) --install-dir DIR program directory (default: $HOME/.local/share/chusql)
--data-dir DIR data directory (default: <install-dir>/data) --data-dir DIR data directory (default: settings.toml, then XDG data directory)
--keep-data keep the data directory, remove programs only --keep-data keep the data directory, remove programs only
--keep-config keep settings.toml --keep-config keep settings.toml
--yes do not ask (required when there is no terminal) --yes do not ask (required when there is no terminal)
@@ -36,6 +36,21 @@ fail() {
stop_by_path() { stop_by_path() {
target=$1 target=$1
[ -n "$target" ] || return 0 [ -n "$target" ] || return 0
if [ "$(uname -s)" = Linux ]; then
for process in /proc/[0-9]*/exe; do
executable=$(readlink "$process" 2>/dev/null) || continue
[ "$executable" = "$target" ] || continue
pid=${process#/proc/}; pid=${pid%/exe}
kill "$pid" || fail "cannot stop $target ($pid)"
attempts=0
while [ -e "$process" ]; do
attempts=$((attempts + 1))
[ "$attempts" -lt 100 ] || fail "timed out stopping $target ($pid)"
sleep 0.1
done
done
return 0
fi
pids=$(ps -eo pid=,args= 2>/dev/null | pids=$(ps -eo pid=,args= 2>/dev/null |
awk -v t="$target" '{ pid = $1; $1 = ""; sub(/^ +/, ""); if (index($0, t) == 1) print pid }' || true) awk -v t="$target" '{ pid = $1; $1 = ""; sub(/^ +/, ""); if (index($0, t) == 1) print pid }' || true)
[ -n "$pids" ] || return 0 [ -n "$pids" ] || return 0
@@ -73,9 +88,27 @@ while [ $# -gt 0 ]; do
done done
install_dir=${install_dir:-$HOME/.local/share/chusql} install_dir=${install_dir:-$HOME/.local/share/chusql}
data_dir=${data_dir:-${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data}
config_dir="${XDG_CONFIG_HOME:-$HOME/.config}/ChuSQL" config_dir="${XDG_CONFIG_HOME:-$HOME/.config}/ChuSQL"
config_file="$config_dir/settings.toml" config_file="$config_dir/settings.toml"
if [ -z "$data_dir" ] && [ -f "$config_file" ]; then
data_dir=$(awk '/^[[:space:]]*\[storage\]/ { storage=1; next } /^[[:space:]]*\[/ { storage=0 } storage && /^[[:space:]]*data_dir[[:space:]]*=[[:space:]]*"[^"\\]*"[[:space:]]*(#.*)?$/ { sub(/^[^=]*=[[:space:]]*"/, ""); sub(/"[[:space:]]*(#.*)?$/, ""); print; exit }' "$config_file")
[ -n "$data_dir" ] || fail 'settings.toml needs an explicit double-quoted storage.data_dir without escapes; pass --data-dir'
case "$data_dir" in /*) ;; *) data_dir="$install_dir/$data_dir" ;; esac
fi
data_dir=${data_dir:-${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data}
if [ "$(uname -s)" = Linux ]; then
install_dir=$(realpath -m -- "$install_dir")
data_dir=$(realpath -m -- "$data_dir")
else
if [ -d "$install_dir" ]; then install_dir=$(CDPATH= cd -- "$install_dir" && pwd -P); fi
if [ -d "$data_dir" ]; then data_dir=$(CDPATH= cd -- "$data_dir" && pwd -P); fi
fi
case "$install_dir" in /|"$HOME") fail 'refusing to remove the root or home directory' ;; esac
case "$data_dir" in /|"$HOME") fail 'refusing to remove the root or home data directory' ;; esac
case "$install_dir" in "$data_dir"|"$data_dir"/*) fail 'data directory must not contain the installation' ;; esac
for entry in bin static csql csql-web csql-web.sh uninstall.sh; do
case "$data_dir" in "$install_dir/$entry"|"$install_dir/$entry"/*) fail 'data directory overlaps installed program files' ;; esac
done
profile="$HOME/.profile" profile="$HOME/.profile"
if [ "$assume_yes" = 'no' ]; then if [ "$assume_yes" = 'no' ]; then
@@ -108,23 +141,16 @@ step 'Removing files'
if [ ! -d "$install_dir" ]; then if [ ! -d "$install_dir" ]; then
note "not found: $install_dir" note "not found: $install_dir"
elif [ "$keep_data" = 'yes' ]; then elif [ "$keep_data" = 'yes' ]; then
for entry in "$install_dir"/*; do for entry in "$install_dir"/bin "$install_dir"/static "$install_dir"/csql "$install_dir"/csql-web "$install_dir"/csql-web.sh "$install_dir"/uninstall.sh; do
[ -e "$entry" ] || continue [ -e "$entry" ] || continue
if [ "$entry" = "$data_dir" ]; then
note "kept data: $entry"
continue
fi
rm -rf "$entry" rm -rf "$entry"
done done
note "kept $install_dir (data only)" note "kept data: $data_dir"
else else
rm -rf "$install_dir" rm -rf "$install_dir"
case "$data_dir" in
"$install_dir" | "$install_dir"/*) ;;
*) rm -rf "$data_dir" ;;
esac
note "removed $install_dir" note "removed $install_dir"
fi fi
if [ "$keep_data" = 'no' ] && [ -d "$data_dir" ]; then rm -rf "$data_dir"; fi
step 'Removing settings.toml' step 'Removing settings.toml'
if [ "$keep_config" = 'yes' ]; then if [ "$keep_config" = 'yes' ]; then
@@ -139,11 +165,11 @@ step 'Removing the PATH entry'
marker='# ChuSQL' marker='# ChuSQL'
if [ ! -f "$profile" ]; then if [ ! -f "$profile" ]; then
note "no $profile" note "no $profile"
elif grep -q "^$marker" "$profile"; then elif grep -q "^$marker$" "$profile"; then
tmp="$profile.chusql-uninstall" tmp="$profile.chusql-uninstall"
awk -v m="$marker" ' awk -v m="$marker" '
skip > 0 { skip--; next } skip > 0 { skip--; next }
index($0, m) == 1 { skip = 2; next } $0 == m { skip = 2; next }
{ print } { print }
' "$profile" >"$tmp" && mv "$tmp" "$profile" ' "$profile" >"$tmp" && mv "$tmp" "$profile"
note "removed the $marker block from $profile" note "removed the $marker block from $profile"