修复bug
This commit is contained in:
1 parent
fc9bbc20e4
commit
e87c4ee6a8
90 files changed
+3828
-452
No files matched your search
@@ -0,0 +1,13 @@
|
||||
# AI agent 协作资料
|
||||
|
||||
按任务读取,不将全部资料作为每次工作的上下文。
|
||||
|
||||
- 根目录 `AGENTS.md`:开发规则和测试入口。
|
||||
- `context/progress.txt`:唯一进度、测试计数和验收来源。
|
||||
- `context/known-issues.txt`:未解决限制;`design-decisions.txt`:现有设计约束。
|
||||
- `context/main-roadmap.txt`:长期阶段;`side-branch-plan.txt`:近期任务及依赖。
|
||||
- `skills/chusql-*/`:类型、存储、跨组件边界三个项目专用技能。
|
||||
|
||||
通用技能、安装锁文件及技能生成提示词仅保留在本地,不纳入版本控制。
|
||||
`gate-check/` 为开发者和 CI 共用的门禁工具;`logs/` 为忽略的本地产物。
|
||||
这些协作资料不参与应用构建、运行或发行打包。
|
||||
@@ -0,0 +1,99 @@
|
||||
# ChuSQL 当前设计决策
|
||||
进度见 progress.txt,限制见 known-issues.txt,用户操作见 docs/。
|
||||
|
||||
## 架构与接口
|
||||
- 后续设计按领域记录于 docs/normalization.md、docs/types.md、docs/architecture.md 和 docs/release-signing.md;只读范式入口已提供;后续实施 bound IR / DOMAIN,再推进物化、递归、MVCC、compact 和并行写。各文档的设计章节定义拟议能力,现有运行语义不变。
|
||||
- 范式迁移首版保留来源,拒绝 NULL、重复行与依赖不保持;发布目标须验证约束、完整重建并使用独立原子结构 WAL 组,不能复用禁止 DDL 的会话事务拼接执行。
|
||||
- 发布真实性采用 TUF 元数据链与 Ed25519,初始 root 必须可信取得;离线验证不证明最新版本,操作系统原生代码签名另行接入。
|
||||
- Web / CLI → server TCP → Haskell 引擎 → FFI → Rust 存储;server 独占数据目录,Haskell 库静态链接,存储为 cdylib。
|
||||
- 引擎按 AST → 语义检查 → RelOp → 优化 → 执行分层;语义与优化只读 schema,表存在性以 catalog 为准,索引由存储管理。
|
||||
- JSON 使用显式稳定标签,FFI 仅提供 version / open / close / request / free / last_error;Web 不本地执行 SQL。
|
||||
- 兼容值与类型语义集中于 Core.Model,运行时类型与版本化编码集中于 Core.Runtime,函数注册与绑定集中于 Engine.Runtime;旧运算符保留 Engine.Builtin,错误分类集中于 Engine.Error。
|
||||
- 比较与赋值兼容性分开,整数与整值浮点统一相等和哈希,SQL NULL 比较走三值逻辑;CHECK 每语句解析一次。
|
||||
|
||||
## 查询与优化
|
||||
- 优化保持结果语义和可观察行序;相关外层列不能被投影删除,LEFT JOIN 只下推安全的左侧谓词。
|
||||
- schema 携带行数、distinct、索引和直方图,缺统计时保守退回原计划。范围成本使用 16 桶等宽直方图,无直方图时按四分之一估算。
|
||||
- 内连接从各起点贪心选序,仅在严格更省且行序不可观察处重排;统计不足或不安全条件时不改写。
|
||||
- 表 / 列限定名在语义层按作用域及唯一候选解析,别名优先;会话将当前库限定 DDL 转为裸名,拒绝跨库目标,库名随请求传递。
|
||||
- 相关子查询保留已解析外层限定名;求值绑定补回单表物理列的限定名,绑定去重与扫描投影使用同一列值,库限定引用仍校验 schema 来源。
|
||||
- JOIN 条件按子查询外层引用的值缓存替换后的表达式;首次遇到行对时才求值,每个行对仍计算普通条件,保留输出顺序、空输入和错误返回。
|
||||
- WAL 提交组落盘后、首次应用前进入 recovery_required;任何应用、刷盘或检查点错误都保持该状态,完整落盘与截断成功后才解除。
|
||||
- FROM 是来源树,派生表必须带别名且不能引用外层,输出取裸列名,重名报错;未定型 NULL 保留独立查询推导类型,不能声明为存储类型。
|
||||
- ORDER BY 优先匹配唯一输出别名,再查来源列;只将排序表达式提前计算,非排序投影保持在 LIMIT 后求值。CTE 列清单同步重命名内层排序别名。
|
||||
- 派生表从透明算子与 schema 统计估算行数,缺统计或表达式可能报错时不重排;投影下推保留计算项,避免删除错误。内存存储提供实时行数及封顶去重统计。
|
||||
- DECIMAL(p,s) 使用有限浮点、ties-to-even 舍入和舍入后溢出检查,要求 1 <= p <= 308、0 <= s <= p;Haskell 与 Rust 写入入口统一边界,日志记录规范化值。
|
||||
- CTE 解析期展开为派生表,只接受 SELECT,不物化;多次引用重复执行,以减少实现改动面。
|
||||
- 比较及 ANY / ALL 使用独立 AST 节点,量词复用三值逻辑,空集 ANY=false、ALL=true。
|
||||
- 子查询在 Filter / Compute 中按外层绑定分组复用,暂不引入半连接;扫描按页分片,至少 1000 行、最多 8 线程,结果保序,safe FFI 释放执行能力。
|
||||
|
||||
## 会话、事务与存储
|
||||
- 每连接独立会话及工作库;短锁取得库句柄后执行,每库读写门控制可见性,表锁保护页,在用库不强行删除。
|
||||
- BEGIN 物化当前库快照,事务内走内存执行,COMMIT 提交相对基线差分,ROLLBACK 丢弃快照。
|
||||
- COMMIT 先跳过数据未修改的表,整数 id 用 IntMap / IntSet 查找差异与冲突;保留原行顺序及重复 id 的首行查找语义,冲突检查和提交操作共享一次差分。
|
||||
- 有整数 id 时按键提交及检查写写冲突,无整数 id 时整表比对;未修改表不回写,冲突保留事务并返回 serialization_failure。
|
||||
- 保存点保存不可变快照引用,同名取最近一层,回退 / 释放丢弃之后的点,结束清空;事务命令由会话层执行。
|
||||
- WAL 先记录操作及提交标记,再应用、刷数据与 catalog、写检查点、截断;仅重放已提交且超过检查点的记录。
|
||||
- WAL 帧保留长度、LSN、类型码、表名及 JSON payload,兼容旧记录;恢复按内容、schema 和文件存在性判断幂等。
|
||||
- 检查点按库 LSN,前缀截断用同目录临时文件原子替换,未提交尾部直接截短;失败保留 WAL 并进入 recovery_required。
|
||||
- 检查点采用同目录临时文件替换,长度必须为 8 字节;完整 WAL 帧损坏明确报错,尾部不完整帧保留撕裂处理。刷盘或检查点失败后封闭请求,重启重放已提交组。
|
||||
- distinct 在封顶前按值引用计数维护,列封顶不清除其他列状态;直方图独立累计与递减,桶重映射导致删除目标为空时扣最近非空桶以保持总量。rebuild_stats 先扫描数值范围,再按固定边界计数,消除重建过程的桶重映射偏差并保持输入顺序无关。
|
||||
- 表 / 索引 DDL 进 WAL,删表不重取已移除句柄;库建删不进 WAL,删除以墓碑目录收敛。
|
||||
- 页内 compact 保持槽号及索引位置,死空间达到四分之一时自动整理;无隐藏列逐页整理,有隐藏列整表重写,不跨页填空以保留行序。
|
||||
|
||||
## 身份、权限与引导
|
||||
- 对象标识使用数据库编号与库内表编号组合;数据库编号从 system catalog 的对象高水位分配,表编号从本库高水位分配,删除不降低计数器。对象解析只读取 catalog。
|
||||
- 建表编号在 WAL 组写入前分配,随建表记录提交;目录应用时推进高水位,恢复使用日志编号。只有建表组复制操作列表,普通数据提交沿用借用列表。
|
||||
- 用户是可登录角色,共用名称空间;identities 保存属性、时间、稳定编号及版本,users 只存编号和口令哈希,编号不复用。
|
||||
- 最高权限按身份属性判断,不按名称、不继承;默认用户 LOGIN、角色 NOLOGIN,两者 NOSUPERUSER / ENABLED。保护最后一个启用的登录管理员。
|
||||
- 仅旧格式迁移按管理员参数识别初始最高权限;旧账号保留信息,角色补编号,重名拒绝;身份、凭据及删除清理用同一 Accounts WAL。
|
||||
- server 判定权限,目录封装于 Server.Catalog;角色成员取向上闭包,写入拒绝环。GRANT role TO member 表示 member 继承 role 权限。
|
||||
- members 保持身份成员目录;对象授权存于受保护 __system_object_acl,使用稳定对象与身份编号,保存所有者、授予者和 grant option。CONNECT / CREATE 限数据库,四项数据权限限表或单库表范围;ALL 按作用域展开。
|
||||
- PUBLIC 默认只提供业务数据库 CONNECT,可逐库撤销;CREATE 建表登记创建者所有权,数据库所有权不隐含他人表的数据权限。启用角色可继承所有权,不继承 SUPERUSER。
|
||||
- 授权链从独立管理来源及所有权展开固定点;撤销清理无来源下游,保留独立来源,循环不能自持。禁用保留所有权和管理来源授权,停止权限贡献并清理失效下游。
|
||||
- 旧名称授权仅迁移一次到现存稳定对象;旧全局 * 固定为当时现存表,缺失对象或身份清理,旧记录无授予者时使用管理来源。
|
||||
- 权限快照按字符分块存储,比较旧快照后以同一 WAL 组替换;跨页及 Unicode 不受单行上限限制。普通存储入口隐藏并拒绝访问权限快照表。
|
||||
- 服务端共享公平读写锁排序查询与权限变更;事务提交复查身份、权限和稳定对象。建表前写入创建者声明,启动时依据现存对象恢复所有权或清理未完成声明。
|
||||
- 口令 / 属性推进 revision,权限变更预先展开受影响身份;TCP 只标记连接失效,下一请求回 unauthorized 后关闭,避免 Windows 阻塞句柄异步中断;登记在 finally 清理。
|
||||
- 系统表统一 __system_ 前缀并分层保护,内部目录按白名单访问;旧名按文件先、catalog 后迁移,崩溃后可重复收敛。
|
||||
- 安装先 bootstrap:建立 system、默认 root(LOGIN / SUPERUSER / ENABLED)、18 个内置名称,再起服务;引导请求不开放 TCP。
|
||||
- 清单编入引导并与解析器测试比对;system.__system_types 保存名称、基础表示、参数能力、编号。重复引导保留口令并补齐类型,冲突或身份降权失败;类型建表与写入用同组 WAL。
|
||||
|
||||
- sudo 认证用独立 allow_sudo_auth 属性和显式 sudo_auth_user 映射;默认关闭,只有 SUPERUSER 能管理,不继承、不隐含权限。目录管理者不得管理启用该属性的身份。
|
||||
- 选择操作系统保护的随机本机凭据与 TCP 一次性 HMAC 挑战,以复用三平台既有会话协议;Unix 只信任有效 UID 0,Windows 只信任提升 Administrators 令牌。双方校验内核回环地址,不接受客户端自报 OS 用户或 SUDO_USER。
|
||||
- 本机凭据只存放于系统保护的端口专属目录,原生入口校验提升身份;实例重启重新生成,现存目录拒绝覆盖。此方案要求服务端也提升运行,以保证创建凭据时的操作系统保护边界。
|
||||
- 挑战绑定账号、连接及实例,60 秒过期并在每次尝试时消费;证明常量时间比对,凭据不经网络发送。成功登录复用身份版本、登录时间及连接失效机制,审计不含凭据或证明。
|
||||
- sudo 属性由 Accounts WAL 持久化,旧目录缺列触发身份快照重写,旧记录缺属性默认 false,已有的错误类型明确拒绝。
|
||||
|
||||
## 自定义类型
|
||||
- 范式检查采用阿姆斯特朗属性闭包与子集枚举,候选键声明转为决定全关系的依赖;完整覆盖声明与领域原子性分别控制可证明结论,不从样本推断。二元 BCNF 建议按违反依赖无损拆分,投影依赖之并用于验证依赖保持。
|
||||
- DOMAIN 命名基础类型,列编码 domain(名称,基础类型) 保留引用,值及索引沿用底层语义。
|
||||
- 各库 __system_domain_<名称> 空目录表的 base 列保存定义,复用 catalog / DDL WAL;普通 SQL 隐藏并保护目录。
|
||||
- 嵌套只引用当前库已有类型,语义层从 schema 递归解析并拒绝循环,存储写锁内逐层校验;删除检查类型间及间接列依赖。
|
||||
- 类型 DDL 要求管理员、事务内禁止;约束与默认值后续扩展,暂不引入自定义函数 / 运算符。
|
||||
- 运行时类型使用稳定构造器编号与结构化 TypeId,逻辑类型与 PhysicalType 分离;递归值使用版本 1 标签编码,读取时校验类型、形状及载荷。
|
||||
- TypedExpression / TypedPlan 仅使用 schema 绑定,SQL BoundQuery 通过既有 RelOp 执行并按输出位置验证;输出类型保留嵌套 DOMAIN 名称,基础日期、时间戳及 Blob 保留兼容类型,函数结果不继承参数领域。
|
||||
- TypedExpression 列引用保存输入 schema 槽位与诊断名称;独立求值要求行顺序匹配,TypedPlan 扫描边界校验并按 schema 排序,每层投影绑定到子计划输出。SQL RelOp 在优化后绑定表达式列槽位,投影单向转为 Compute,排序/分组转为 BoundSort/BoundAggregate,存储结果按算子布局排列;子查询局部列在前、外层绑定在后。绑定算子不参与后续投影下推;独立作用域身份仍待迁移。
|
||||
- SQL 局部引用使用 ColumnSlot,外层引用使用 OuterColumnSlot 的相对位置及局部布局宽度。重新绑定外层引用只查询外层布局,避免同名局部捕获;语义输入拒绝执行槽位。外层行仍扁平化,多级词法作用域/来源编号另行迁移。
|
||||
- BoundQuery 执行前重新取得 schema 并比较依赖对象身份、schema 版本及列定义,依赖覆盖全部嵌套子查询及来源列的传递 DOMAIN 定义。IPC 将依赖守卫限定在本次执行环境;扫描、分片、点查及范围读均在相关库读锁内复查全部依赖后读取,跨库读锁按库名排序,持有库引用防止删除。数据、统计和无关对象不参与比较;旧目录缺失版本时仍比较列定义,已知身份或版本丢失则拒绝。守卫响应必须明确确认,旧二进制忽略字段时拒绝;未实现守卫的后端明确报错,纯内存视图可直接执行。守卫不改变既有多请求查询的数据隔离语义。
|
||||
- 内建函数先解析为稳定 FunctionId 并验证签名,再执行;SQL NULL 保留三值语义。缺少能力字典的运行时类型拒绝比较、排序、分组、极值、索引及唯一键声明。
|
||||
- SQL 字面量生成返回明确错误,先验证运行时值与有限浮点;复合值在 Web 使用 SQL 修改。
|
||||
|
||||
## 配置、构建与发行
|
||||
- Web 编辑在首个业务表加载时建立快照,在提交 / 回退时结束;既有大库基准 BEGIN 269.806 ms、COMMIT 376.345 ms,不按每次单元格编辑重建整库快照。
|
||||
- Web 登录分别建立普通 API / SQL 和编辑连接,页面 / 数据库标识绑定编辑事务,SQL 事务另绑定查询标签;同 Cookie 请求串行执行。
|
||||
- 编辑集保留在浏览器,提交从固定保存点重放;失败回退整个编辑集,冲突回滚并重新加载,成功回执去重;系统身份修改不进入数据事务。
|
||||
- system_catalog_manager 只管理普通身份,不继承、不隐含业务访问权限;存储写锁内再次拒绝特权目标及特权属性修改。
|
||||
- bootstrap 维护使用独占数据目录锁,在副本中修复并验证后保留原目录备份再发布;repair 不重放 WAL、不补初始化数据,原有数据文件必须字节一致。
|
||||
- listen_host 是监听配置,static 固定为静态目录,管理员名由 --user 指定;data_dir 存数据,log_files 存日志,相对路径以启动目录解释。
|
||||
- resources/chusql.toml 入库并打包,UI 设置样本不入库、不进包,运行时按 resources、上一级 resources、工作目录候选查找。
|
||||
- 预编译包优先,server / 存储必装,Web / CLI 可选;Windows zip、Linux/macOS tar.gz。包内模板也放 resources/,按确切产物名打包并拒绝未知工件。
|
||||
- 本地解包安装不联网,管理员名同时传 bootstrap 和 server;安装冒烟隔离配置并恢复用户 PATH,卸载可保留数据 / 配置。
|
||||
- 门禁先刷新 release cdylib,再并行 Rust、串行 Haskell、注释三条车道;Web 先刷新依赖,任一步失败即失败。
|
||||
- 所有者运行全量门禁,agent 只测试改动模块;日志 / 临时文件在 logs/,必要时 -Full;门禁及打包强制 UTF-8 以避免 GBK 编码失败。
|
||||
- CLI 终端隐藏口令,管道兼容 LF / CRLF / 无行尾,保留空格与内部回车,提示写 stderr。
|
||||
- CLI 禁用逐行自动历史,只记录完整 SQL 与元命令;含 IDENTIFIED / PASSWORD(忽略大小写)的输入整体排除,避免多行口令或执行失败留下片段。解析错误也隐藏此类原文,筛选保守包含字符串和标识符中的同名文本,已有历史不自动删除。
|
||||
- TCP 与 Web 使用同一解析诊断脱敏路径;Web 回归校验整个 HTTP 错误响应不含口令。内置 SQL 控制台记录结果或诊断,不将 SQL 原文写入日志,外部代理和采集需自行配置。
|
||||
- BoundQuery 保存来源对象身份及列定义;DOMAIN 依赖从来源列递归展开,执行前复查。库 / 对象编号在 TableInfo 中输出十进制字符串,读取兼容旧数值,零对象编号表示无有效身份。
|
||||
- 表级 schema_version 由 catalog 持久化,新表从 1 开始,旧目录缺失版本保留 0;结构变化递增,新增结构 WAL 记录保存确定目标版本,恢复不重新分配。线上使用精确十进制字符串并兼容数值输入,0 或缺失表示未知版本;IPC schema / 快照与内存 Server 保留版本,内存 DDL 同样递增并拒绝耗尽。
|
||||
- 注释以 AGENTS.md 为准;原因写本文件,限制写 known-issues.txt,进度及计数只以 progress.txt 为准。
|
||||
@@ -0,0 +1,54 @@
|
||||
# ChuSQL 当前限制与风险
|
||||
进度见 progress.txt,设计见 design-decisions.txt;仅记录未解决的限制。
|
||||
|
||||
## 事务、恢复与并发
|
||||
- 事务是整库内存快照,不是行版本 MVCC;每会话一个事务,无嵌套事务,事务内禁止 DDL / USE。
|
||||
- BEGIN / COMMIT 仍有整库快照传输与修改表比较成本,复测见 docs/performance.md。
|
||||
- Web 首次加载业务表即固定编辑快照,同一登录会话只允许一个页面持有编辑事务;系统身份表只读,通过 SQL 管理。
|
||||
- 页面关闭的回滚请求可能因断线未送达;此时按会话空闲 / 最长寿命过期清理,后台每分钟检查一次。编辑连接失效后必须重新登录,不能恢复旧快照。
|
||||
- COMMIT 按整数 id 校验写写冲突,缺少整数 id 时整表比对及替换;冲突返回 serialization_failure,调用方回滚或重试,无自动重试或死锁检测。
|
||||
- WAL 只有 redo,检查点按库 LSN 而非页 LSN,幂等恢复代价随数据量增长;库目录建删不走 WAL,删库用墓碑目录处理。
|
||||
- recovery_required 期间除 Ping / SystemStatus 外请求被拒绝,需恢复后继续。
|
||||
- 每库读共享、写独占,写入会阻塞同库跨表读取,尚无并行写。
|
||||
- 权限变更只标记 TCP 连接失效,客户端下一次请求收到 unauthorized 后关闭,空闲连接不会立即断开;并发测试不证明性能提升。
|
||||
|
||||
## 查询与性能
|
||||
- 基础成本及贪心连接重排,无动态规划或跨外连接重排;只在行序不可观察处重排。
|
||||
- catalog 行数可能滞后,distinct 有上限;16 桶等宽直方图只统计数值并按闭区间估算,增量维护可能偏差,桶内分布仍为估算。
|
||||
- 内存统计去重上限为 4096,暂不提供直方图或索引成本;派生表可从透明算子估计行数,含聚合或可能报错表达式时仍保守。投影下推保留所有计算项,不消除潜在错误。
|
||||
- 按外层绑定复用子查询覆盖 Filter / Compute / JOIN 条件;JOIN 按首次遇到的绑定惰性求值,缓存仅限单次连接。无半连接或通用物化,分组求值可能更早报告同一错误。
|
||||
- CTE 解析期展开,无物化、递归或写语句;只能引用前面定义的 CTE,前向引用按普通表名解析。
|
||||
- 并行只覆盖页区间扫描:schema 至少 1000 行、最多 8 线程,无页内并行,统计滞后可退回顺序扫描。
|
||||
- compact 不填跨页空洞,含隐藏列时仍需整表重写。
|
||||
|
||||
## 权限与兼容性
|
||||
- 对象权限快照整体写入,鉴权会读取身份、成员和对象元数据;尚未针对大规模权限目录做性能基准。
|
||||
- 对象高水位损坏或回退后无法恢复已删除对象的编号上限;旧二进制会丢弃新增目录字段,不应降级使用已升级的数据目录。
|
||||
- 无列级权限、所有权转移或角色 ADMIN OPTION。
|
||||
- 旧授权缺少授予来源,迁移为管理来源;旧全局 * 只绑定迁移时现存表。没有身份改名命令;已有同名 public 身份在删除前不会解析为虚拟 PUBLIC。
|
||||
- 尚无 CREATEDB / CREATEROLE、继承开关、口令到期、连接上限或 Web 专门的身份属性编辑界面。
|
||||
- 旧账号与角色重名会阻止迁移;迁移后的身份目录与 DOMAIN 编码不支持旧二进制。更早的 __chusql_users 无自动迁移。
|
||||
- 旧配置 host 不读取,web.static_dir / web.user 被忽略,升级需按 docs/config.md 调整。
|
||||
- 引导不重设已有口令、不提升已降权身份;类型定义冲突需人工解决。
|
||||
|
||||
## 类型与语义
|
||||
- 范式检查提供 Haskell / TCP / CLI / Web 只读入口,最多 12 列;领域原子性和依赖完整性由用户声明,信息不足时不认证高阶范式。分解建议仅一轮,不保证子关系已达 BCNF;尚无自动迁移。
|
||||
- DOMAIN 无自身 CHECK / NOT NULL / DEFAULT、ALTER DOMAIN、CASCADE;尚无 ENUM 或用户定义复合类型,已有内建 List / Maybe / Tuple。内置目录只登记现有语义,不能靠增加行扩展基础类型。
|
||||
- DECIMAL 仍使用浮点表示,不是精确十进制存储;整数类型共用整数表示,CHAR 不补空格。
|
||||
- 少量接口设置仍用字符串类型名;Haskell 值语义和 Rust 索引键分开维护。SQL 已接入 BoundQuery,旧 ColumnType / RelOp 仍是兼容桥,完整能力、转换及关系 IR 迁移待后续阶段。
|
||||
- BoundQuery 在执行前比较对象身份、schema 版本和依赖列定义,IPC 的每次读取还在相关库读锁内复查全部依赖;多请求查询仍沿用既有数据隔离语义,不提供整条查询的数据快照。旧目录缺失版本时使用列定义校验,无有效对象编号时无法证明历史对象身份;旧存储二进制未确认读守卫则拒绝执行。多级外层作用域仍使用扁平行布局。
|
||||
- 运行时类型尚无 Eq / Ord / Hash 等 capability 字典,不提供比较、排序、分组、极值和索引;复合字段在 Web 通过 SQL 修改,没有结构化单元格编辑器。
|
||||
- 运行时协议使用版本 1,旧客户端不支持新增类型化值;Web 的 JSON 数值仍受 JavaScript 安全整数范围限制。
|
||||
- 派生表输出取裸列名,重复名称报错;未定型 NULL 仅用于查询类型推导,不可声明为存储列类型。
|
||||
- ORDER BY 不接受任意表达式;已支持输出列序号和 DISTINCT,复合类型去重仍要求后续 capability 字典。
|
||||
|
||||
## 构建、发行与安全
|
||||
- 离线 recover 依赖完整可用 WAL,不能保证修复损坏数据页或已截断的历史;未知系统表缺少结构时需备份或 reset。目录计数器损坏后无法还原已删除身份编号上限。
|
||||
- 旧版进程不支持数据目录独占锁;维护前必须停止所有相关进程。
|
||||
- sudo 本机认证需提升身份启动 server;当前只映射 Unix root / Windows 提升 Administrators 到一个显式配置身份,无任意 OS 用户映射或自动提权。异常退出留下的端口凭据目录须在确认停止后手工清理。
|
||||
- sudo 的 Windows 提升成功路径、macOS 编译运行及 Linux 完整提升 server/CLI 链路仍待目标平台实机验收;Debian/WSL 已验证原生凭据安全边界。
|
||||
- Stack 工程应串行构建;旧注册库可能被复用,必要时 --force-dirty 或所有者门禁 -Full,Web 门禁已自动刷新依赖。
|
||||
- 存储协议变化后必须重建实际 cdylib,开发环境须配置 PATH / LD_LIBRARY_PATH;旧命名构建工件仍可能残留。
|
||||
- 真实 server 测试须先引导并能找到 bootstrap;注释检查对部分多行语法可能误报。
|
||||
- GitHub 在线安装、Linux $ORIGIN 与 macOS 发布仍需实机验证;无 Release 时在线安装无法取得预编译资源。
|
||||
- SHA256 只能检测损坏,无发布签名;外部采集仍可能记录 SQL 原文。旧 CLI 历史不自动清理,任意业务字符串中的敏感数据不自动识别。
|
||||
@@ -0,0 +1,22 @@
|
||||
ChuSQL 长期路线图
|
||||
|
||||
交付状态和验收只以 progress.txt 为准;近期顺序见 side-branch-plan.txt。
|
||||
本文件描述目标,不代表现有实现或功能承诺。
|
||||
|
||||
目标链路:SQL → Parser / Binder → Typed IR → Planner / Optimizer → Executor → Rust Storage。
|
||||
类型能力与权限保持独立;catalog 保存对象、依赖和语义,内核保留必要基础机制。
|
||||
|
||||
阶段
|
||||
- A 类型与函数基础:稳定 TypeId / FunctionId、注册表、逻辑/物理类型分离、版本化值;验收见 progress.txt。
|
||||
- B 能力与运算(2–6):Capability、Operator、Cast、Aggregate、参数多态与约束解析。
|
||||
- C 复合与自定义类型(7–12):ADT、Record/Composite、Newtype/Domain、参数化类型、Range、用户类型。
|
||||
- D 函数与效果(13–16):用户函数、Effect、纯 Generated Column、有副作用的 Trigger。
|
||||
- E 权限与类型化查询(17–19):权限 Capability、Query a、Rewrite / Optimizer 函数化。
|
||||
- F 扩展:对象体系统一后实现 Extension 的版本、依赖、安装/升级/移除;其他访问方法按真实需求推进。
|
||||
|
||||
贯穿约束
|
||||
- 现有 ColumnType / Expr / RelOp 有真实消费者;迁移时逐步收拢,不提前删除兼容桥。
|
||||
- 每阶段验证持久化、升级、依赖和恢复,不能靠重新初始化掩盖不兼容。
|
||||
- 内建对象可由可信 bootstrap 元数据恢复;用户对象依赖 WAL / 备份。
|
||||
- catalog 版本与 bootstrap / recovery 匹配,系统目录索引可重建。
|
||||
- 不提前增加孤立扩展或绕过事务、权限及依赖管理。
|
||||
@@ -0,0 +1,64 @@
|
||||
ChuSQL 能力与进展摘要
|
||||
更新:2026-10-10
|
||||
|
||||
本文件是状态、测试计数及验收记录的唯一来源;用户确认其等价于 project-plan.txt。
|
||||
设计见 design-decisions.txt,限制见 known-issues.txt,后续支线见 side-branch-plan.txt。
|
||||
|
||||
## 主线能力
|
||||
|
||||
- SQL:CRUD、表列约束、索引、连接、聚合、排序、分页、子查询、派生表、投影别名、CTE、比较运算符、ANY / ALL、DISTINCT / ALL 与 ORDER BY 输出列序号。
|
||||
- 优化:谓词 / 投影下推、常量折叠、点查 / 范围成本选择、贪心内连接重排、16 桶直方图、按外层绑定复用子查询与并行扫描。
|
||||
- 存储:页、堆表、B+Tree、catalog、WAL、LSN 检查点、幂等恢复与页内 compact。
|
||||
- 会话:按连接隔离工作库、快照事务、写写冲突检查、保存点和权限变更后的会话失效。
|
||||
- 身份:用户与角色统一名称空间,LOGIN / SUPERUSER / ENABLED、稳定编号、角色继承、转授权、迁移及删除清理。
|
||||
- 权限查询与撤销:当前登录身份 SHOW GRANTS、REVOKE GRANT OPTION FOR 与失效下游级联清理;工作区模块回归及 2026-10-09 所有者全量门禁通过。
|
||||
- 范式入口:专用 TCP 请求、CLI --normalize / \normalize 与 Web 结构区只读分析;CONNECT / SELECT、事务视图、稳定编号和 schema 指纹、声明及运行预算;工作区模块回归及 2026-10-09 所有者全量门禁通过。
|
||||
- 类型:CREATE / DROP / SHOW DOMAIN、库内嵌套定义、列引用、依赖 / 循环检查及持久化。
|
||||
- 相关列绑定:局部 ColumnSlot 与外层 OuterColumnSlot 分离,外层使用相对槽位,重新绑定保留外层身份;同名遮蔽及多层相关引用回归通过;仍使用扁平化外层行,多级词法作用域/来源编号尚待实施。
|
||||
- 绑定执行校验:执行前比较依赖表身份、schema 版本和列定义;IPC 每次扫描、分片、点查及范围读取在相关库读锁内复查全部依赖,再读取数据,跨库读锁按库名排序。依赖覆盖来源、连接条件、投影/过滤中的嵌套查询及递归 DOMAIN 定义,忽略数据/统计和无关对象变化。同名同结构重建或结构改变后恢复原定义使旧绑定失效;旧目录缺失编号或版本时保留列定义校验,已知身份或版本丢失则拒绝。守卫须有明确确认,未支持守卫的后端拒绝执行;纯内存视图保持稳定。多请求查询仍沿用既有数据隔离语义。本批待所有者新门禁验收。
|
||||
- bound IR 基础:SQL 查询输出类型保留嵌套 DOMAIN 引用及其基础类型;TypedExpression 按输入 schema 槽位绑定列;SQL RelOp 优化后绑定投影、过滤、计算、聚合参数、连接条件及排序/分组键;投影转换为 Compute,排序/分组使用 BoundSort/BoundAggregate,保留重复列、稳定排序、首次分组顺序及空输入聚合行为;相关子查询绑定局部和外层布局,存储返回按布局排列;独立作用域/来源身份及约束绑定尚待实施;槽位绑定基础已通过 2026-10-09 所有者全量门禁。
|
||||
- 引导与维护:初始化 system、默认 root 和 18 个内置类型;重复引导保留口令,冲突拒绝覆盖;system_catalog_manager 与 bootstrap repair / recover / reset。
|
||||
- 配置与发行:listen_host、data_dir / log_files 分离,固定 static,管理员名由 --user 指定,模板统一 resources/chusql.toml;Windows 打包安装冒烟、管道口令兼容及构建 UTF-8。
|
||||
|
||||
## 已实现侧分支
|
||||
|
||||
以下四项已通过 2026-10-05 所有者全量门禁并提交;sudo 另有平台实机验收。
|
||||
|
||||
- Web 编辑事务:独立连接绑定页面与数据库,跨表原子提交;保存点回退保留编辑集,冲突要求重新加载;提交去重、SQL 标签隔离与断线 / 退出清理。
|
||||
- sudo 认证:默认关闭的 allow_sudo_auth 与显式本机身份映射;操作系统保护凭据、回环对端校验及一次性 HMAC 挑战;不继承、不提权,撤销使既有会话失效。
|
||||
- 对象授权:持久化且不复用的库 / 表编号,所有者与 grant option;DATABASE CONNECT / CREATE、表级数据权限及 PUBLIC;跨入口鉴权、来源追踪、级联撤销和事务提交复查。
|
||||
- 范式检查器:基于 schema 与声明的阿姆斯特朗闭包,分析 1NF / 2NF / 3NF / BCNF;区分信息不足,输出违反依赖与一次无损分解并验证依赖保持;只读 API,最多 12 列。
|
||||
|
||||
## 阶段 A:类型与函数基础
|
||||
|
||||
- 状态:实现完成,模块测试及所有者全量门禁通过;实现提交 8ce4239,发行检查修正 0475a61,正式版 v26.10.1 发布验收进行中。
|
||||
- 固定 TypeConstructorId、结构化 TypeId、TypeExpr / Kind 与内建 Type Registry;Int / Double / Bool / String / List / Maybe / Tuple。
|
||||
- 独立 PhysicalType,递归复合值、kind / 参数校验、版本化类型和值编码、协议验证、catalog schema 及存储重载。
|
||||
- 固定 FunctionId 与 Builtin Function Registry,length / not 的签名解析和类型化执行;SQL NULL 保持三值语义。
|
||||
- schema 驱动的 TypedExpression / TypedPlan;SQL SELECT 使用 BoundQuery,函数及构造器节点携带已解析身份,输出按位置校验。
|
||||
- SQL 复合列声明、LIST / MAYBE / TUPLE、嵌套构造器、INSERT / UPDATE / CHECK / 子查询接入;CLI / Web 显示及 SQL 生成。
|
||||
- 兼容边界:旧 ColumnType 和 RelOp 保留,日期 / 时间戳 / Blob 明确使用 CompatibilityType;完整 capability / operator / cast / 用户 ADT / Extension 不在本阶段。
|
||||
- 本批已验证:Engine 466 / Server 141 / Web 75 / CLI 17 全部通过;前端运行时值 9 条断言及事务 10 项通过,改动文件注释检查无问题,git diff --check 通过。
|
||||
- 使用说明:docs/runtime-types.md;本批日志:logs/runtime-engine.log、logs/runtime-web.log、logs/runtime-web-final.log、logs/runtime-cli.log。Server 最终回归亦通过;Web 最终日志由单独重跑生成。
|
||||
|
||||
## 既有模块验证
|
||||
|
||||
- 2026-10-10 schema 版本与读守卫:Rust catalog 持久化表级 schema_version;新表为 1,旧目录缺失字段保留未知版本 0,补列、列定义变化、删列、改名与类型名迁移递增;数据及统计变化不递增,相同定义及重复删列不递增,耗尽明确返回错误。新增删列及列定义替换 WAL 记录在落盘前确定目标版本,应用与恢复使用该版本,覆盖检查点失败、多次结构变化及重复恢复;旧 WAL 载荷仍可读取。版本以精确十进制字符串透传到 TableInfo / Table、IPC schema / 快照及内存 Server,内存 DDL 同样递增;BoundQuery 比较依赖版本并为真实读取携带守卫,覆盖恢复原定义、版本丢失、校验与读取之间的 DDL、四种读路径、跨库依赖变化、写请求拒绝及旧存储缺少确认。
|
||||
- 本批 Engine 511 / Server 161 / Web 80 / Rust 189,均零失败;clippy、release 动态库构建、相关文件注释检查及 git diff --check 通过。日志见 logs/schema-guard-engine.log、logs/schema-guard-server.log、logs/schema-guard-web.log、logs/schema-guard-storage.log、logs/schema-guard-build.log、logs/schema-guard-comments.log。上一批 CLI 20 / Bootstrap 9 零失败,日志见 logs/schema-bind-cli.log、logs/schema-bind-bootstrap.log,本批未重跑。尚待所有者全量门禁,未提交。
|
||||
|
||||
- 2026-10-10 对象绑定批次:Table / TableInfo 保留库和表身份,IPC schema / 快照及内存 Server 目录透传;精确 64 位编号采用十进制字符串输出,兼容旧数值并拒绝非法输入。查询复查对象身份与传递 DOMAIN 依赖,覆盖同结构重建、身份丢失、跨库视角和无关领域变化。
|
||||
- 本批模块验证:Engine 505 / Server 160 / Web 80 / CLI 20 / Bootstrap 9,均零失败;Web 内含前端值与事务测试,17 个相关文件注释检查及 git diff --check 通过。日志见 logs/object-bind-engine.log、logs/object-bind-server.log、logs/object-bind-web.log、logs/object-bind-cli.log、logs/object-bind-bootstrap.log。本批尚待所有者全量门禁,未提交。
|
||||
- 已登记最近模块结果:Engine 501 / Server 159 / Web 80 / CLI 20 / Rust 181,均通过;2026-10-09 所有者全量门禁已验证本组结果,Bootstrap 9 项及 clippy 亦通过。
|
||||
- Web 前端事务 10 项测试通过;对象授权阶段 clippy、release 构建和改动文件注释检查通过。
|
||||
- sudo 原生凭据已验证 Windows 非提升拒绝路径及 Debian/WSL root 安全边界;其他平台链路验收仍待完成。
|
||||
- 2026-10-09 所有者全量门禁覆盖当时工作区:cargo-test / clippy、engine / server / web / cli / bootstrap、注释检查均通过;日志见 logs/cargo.log、logs/stack.log、logs/comments.log。该结果不覆盖之后的对象绑定批次。
|
||||
|
||||
## 后续与验收
|
||||
|
||||
- 2026-10-09 所有者全量门禁通过:三条车道全部 exit=0,STEPS=0,0,0,0,GATE_EXIT=0,用时 136 秒;Engine 501 / Server 159 / Web 80 / CLI 20 / Bootstrap 9,均零失败。此结果不替代生产恢复演练、持续负载及平台实机验收。
|
||||
|
||||
- 2026-10-05 所有者全量门禁通过:cargo / stack / comments 全部 exit=0,STEPS=0,0,0,0,GATE_EXIT=0;bootstrap 9 项通过。Windows 正式包 CLI / Web 分组件安装验证通过。
|
||||
- 正式版 v26.10.1 包含从零介绍的发行说明;按用户要求仅发布 Windows x86_64 / Linux x86_64 的 zip / tar.gz 安装包及 SHA256。要求两平台安装验证及 Linux 测试通过,XDG 验证隔离已修复。最终发布状态待登记。
|
||||
- sudo 的 Windows 提升成功路径、macOS 编译运行及 Linux 完整提升链路待实机验收。
|
||||
- 在线安装、Linux $ORIGIN 与 macOS 发行待目标平台验收;发布签名待设计。
|
||||
- 范式自动迁移、存储查询架构扩展和 DOMAIN 自身约束等后续事项见 side-branch-plan.txt 与 known-issues.txt。
|
||||
@@ -0,0 +1,19 @@
|
||||
ChuSQL 近期任务与依赖
|
||||
|
||||
实现、测试及门禁状态只以 progress.txt 为准;设计见 design-decisions.txt,限制见 known-issues.txt。
|
||||
下列顺序是规划,不是已完成能力;每批按实际进度选择任务。
|
||||
|
||||
1. 统一绑定:完成多级词法作用域、来源/列身份、写语句和约束绑定,收拢旧兼容桥;保留 nullable、错误时序、DOMAIN 及布局语义。schema 版本和读取守卫已有实现,验收见 progress.txt;旧目录身份升级仍需安全策略。
|
||||
2. DOMAIN 约束:CHECK / NOT NULL / DEFAULT、ALTER DOMAIN,覆盖写入、导入及事务提交;CASCADE 依赖对象目录与原子结构事务,发布前验证旧数据和嵌套依赖。
|
||||
3. 类型能力:精确 DECIMAL、复合类型 Eq / Ord / Hash,再推进 ENUM / 用户复合类型 / ADT;统一 Haskell 值与 Rust 索引语义,明确整数范围、CHAR 和数据迁移。
|
||||
4. 查询:ORDER BY 表达式 → 通用物化 → 半连接 → 递归 CTE;写入型 CTE 依赖统一写入校验。保持行序、重复、NULL、错误时序及来源解析规则。
|
||||
5. 结构事务:库建删、DDL、对象依赖与版本复查纳入原子发布和恢复;解决绑定失效、权限、回滚及编号高水位,不能只解除事务内 DDL 禁令。
|
||||
6. 范式迁移:依赖 DOMAIN 约束、对象目录和结构事务;首版保留来源,拒绝 NULL、重复行及依赖不保持,后续再做多轮分解。
|
||||
7. 存储:MVCC → 死锁检测 → 并行写;推进页 LSN、恢复、跨页整理及隐藏列重写。保留幂等、隔离、句柄生命周期、行序与资源预算,以实测决定并行扩展。
|
||||
8. 其他:连接搜索/外连接变换/统计成本;列级权限、所有权转移、ADMIN OPTION、身份属性;Web 复合值编辑、大整数和多页面事务。按真实需求独立推进。
|
||||
|
||||
持续验收
|
||||
- 新改动须等待所有者新全量门禁,通过后提交;模块测试不替代异常恢复、磁盘错误、持续负载及并发冲突演练。
|
||||
- sudo、在线安装、Linux $ORIGIN、macOS 构建发行与发布签名按目标平台验证,结果登记 progress.txt。
|
||||
- 升级明确账号/授权/配置迁移、禁止降级、旧历史和异常凭据清理;不承诺旧二进制兼容、损坏页自动修复或任意 OS 用户映射。
|
||||
- agent 只跑改动模块规定测试;Haskell 串行,存储协议变化重建实际动态库。所有者门禁入口为 gate-check/run-gate.ps1。
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: chusql-boundary-contracts
|
||||
description: 修改 ChuSQL Rust/Haskell FFI、客户端与 Server 协议、跨组件序列化契约或公开接口时使用;核对调用链、线程与所有权。单模块内部算法且接口未变时不加载。
|
||||
---
|
||||
|
||||
# ChuSQL 边界契约
|
||||
|
||||
## Scope
|
||||
|
||||
负责组件间表示、调用与生命周期。类型内容使用 chusql-type-system,落盘与恢复使用 chusql-storage-integrity,通用 unsafe 检查使用 rust-c-ffi-safety。路径相对仓库根目录,按变更读相关消费者。
|
||||
|
||||
## Architecture
|
||||
|
||||
- CLI / Web 后端通过 `chusql-interface/src/ChuSQL/Interface/Link.hs` 连接 Server TCP;`Protocol.hs` 定义网络消息,`Session.hs` 提供客户端会话操作。
|
||||
- `chusql-server/src/ChuSQL/Server/TCP.hs` 分派消息,`Session.hs` 管登录 / 鉴权,`Backend.hs` 调引擎。浏览器通过 `chusql-web/src/ChuSQL/Web/API.hs` 的 HTTP API 访问 Web 后端,不直接访问 Rust。
|
||||
- `chusql-core/engine/src/ChuSQL/Core/Engine/Storage/IPC.hs` 将 MonadStorage 操作编码为 JSON,经 localStorageLink 调 FFI;IPC 名称不表示当前另起存储进程或 Socket 服务。
|
||||
- FFI 两端是 `chusql-core/engine/src/ChuSQL/Core/Engine/Storage/FFI.hs` 与 `chusql-core/storage/src/ffi.rs`,方向 Haskell → Rust,没有回调 ABI。
|
||||
- 存储 JSON 定义于 `chusql-core/model/src/ChuSQL/Core/Protocol.hs` 与 `chusql-core/storage/src/protocol.rs`;它与 Interface TCP 属于不同协议。
|
||||
|
||||
## ABI 与所有权
|
||||
|
||||
下表符号均带 chusql_storage_ 前缀;Rust usize 对 Haskell CSize、i32 对 CInt,句柄使用 Ptr ()。不直接跨语言传 Rust struct、enum、Vec 或 Haskell 对象布局。
|
||||
|
||||
| 符号 | C ABI 形状 | 契约 |
|
||||
| --- | --- | --- |
|
||||
| version | () → const char* | 静态 NUL 字符串,不释放 |
|
||||
| open | const char* → Storage* | NULL 路径用默认;失败 NULL,成功由 close 归还 |
|
||||
| close | Storage* → void | NULL 安全;非空句柄只关闭一次 |
|
||||
| request | const Storage*, const u8*, usize, u8**, usize* → i32 | 输入借用;输出由 Rust 分配;0 / -1 |
|
||||
| maintenance | const char*, const u8*, usize, u8**, usize* → i32 | 离线操作;响应 ownership 同 request |
|
||||
| free | u8*, usize → void | 归还原始指针 / 长度对,不跨分配器、不重复释放 |
|
||||
| last_error | usize* → const u8* | 借用线程局部错误,立即复制、不释放 |
|
||||
|
||||
## Invariants
|
||||
|
||||
- 路径为 NUL 结尾 UTF-8;请求 / 响应为显式长度 UTF-8 JSON,响应不保证 NUL 结尾,不能用 C 字符串函数读取。
|
||||
- Haskell request 使用 Ptr CChar、CSize、Ptr (Ptr CChar)、Ptr CSize 并返回 CInt,maintenance 以 CString 替代句柄。改签名同步两端与实际动态库。
|
||||
- 借用指针整个调用中有效,输出可写且不与输入或其他输出重叠。Rust 拒绝 NULL / 超过 isize::MAX 的请求长度,但不能验证非空指针的分配范围 / provenance;catch_unwind 不是非法指针保护。
|
||||
- 成功输出是 Box<[u8]> 转出的所有权;Haskell callStorage 复制后 finally 调 free,失败清空输出。保留取消屏蔽,避免复制 / 释放窗口泄漏。
|
||||
- open / request / maintenance 捕获 Rust panic,经 NULL / -1 和 last_error 返回;业务错误也可能是 ABI 成功响应中的 JSON error,两层失败都要处理。
|
||||
- close / free 捕获析构 panic 但没有状态返回,当前忽略捕获结果;不能依靠析构保证提交或宣称它报告落盘失败。
|
||||
- onStorageThread 在支持 bound threads 的 RTS 中绑定调用和错误读取到同一 OS 线程;last_error 缓冲只到本线程下次设置错误前有效。
|
||||
- Haskell open / close / request / maintenance 是 safe ccall,version / free / last_error 是 unsafe ccall;长 I/O 不改为 unsafe。请求期间不得关闭句柄。
|
||||
- IPC 的 withStorageLink 计数在途请求,closeConnection 等空闲再关闭;不绕过它更换 / 释放链路。Env 保存会话工作库,不把全局默认库当多连接状态。
|
||||
- Interface TCP 是一行 UTF-8 JSON,Hello 校验 protocolVersion=1;同一 Client 的 MVar 串行化请求 / 响应。Server 每连接建 Session 并限制行长度,客户端锁不是全局数据库锁。
|
||||
- SchemaColumn 字段、method / status 标签、缺省字段、错误码及 QueryResult columns / rows 的位置关系是消费者契约,不用自动派生构造器名替代稳定标签。
|
||||
- 复合值标签为 __chusql_runtime_v1,由 Core.Runtime 编解码;Rust 行仍是通用 JSON。封装变化检查 CLI / Web / 存储重载,Web 数值受 JavaScript 安全整数限制。
|
||||
- 原始存储透传仍经过 Server 会话鉴权;新入口核对 TCP 分派、会话策略与 Web 消费者,不绕过对象权限或业务错误。
|
||||
|
||||
## Engineering Decisions
|
||||
|
||||
优先扩展共享协议与真实调用者,不增加无用途适配层;确认无人依赖的内部接口可直接调整。新增类型化值沿 JSON 载荷传递,只有实际需要才扩展 ABI。
|
||||
Rust panic、Haskell IO 失败与业务错误按边界转成明确失败;当前没有自动 ABI 兼容协商或后台回调生命周期管理。
|
||||
|
||||
## Validation
|
||||
|
||||
- ABI / ownership 变更验证 `chusql-core/storage/tests/ffi_test.rs` 的成功、错误、长度、编码和释放路径;必要时用可用 Miri,不为普通消息变更强制安装工具。
|
||||
- 存储协议变化测试相关 Rust 模块并重建实际 cdylib,再串行验证受影响 Haskell 模块;rlib 或 mock 不能证明实际 FFI 可用。
|
||||
- TCP 改动验证 Hello、错误响应和真实消费者,复用 `chusql-server/test/Spec.hs`、`chusql-web/test/Spec.hs`、`chusql-cli/test/Spec.hs` 中相关行为。
|
||||
- 入口见 `gate-check/run-step.ps1`,测试范围按风险决定,不自动运行所有者全量门禁。
|
||||
|
||||
## References
|
||||
|
||||
Web 值展示 / SQL 生成消费者见 `chusql-web/static/core.js`;总体路径见 `docs/architecture.md`。核对协议时读两端编码 / 解码和调用者,不依赖绝对行号。
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
name: chusql-storage-integrity
|
||||
description: 修改 ChuSQL 存储页、索引、序列化、catalog、系统元数据、初始化、持久化事务或恢复时使用;维护实际落盘顺序与恢复约束。普通 SQL 表达式或界面任务不加载。
|
||||
---
|
||||
|
||||
# ChuSQL 存储完整性
|
||||
|
||||
## Scope
|
||||
|
||||
负责持久化状态及其与服务端事务的关系。类型语义使用 chusql-type-system,接口生命周期使用 chusql-boundary-contracts;不照搬 PostgreSQL 内核设计。路径相对仓库根目录,只读受影响模块。
|
||||
|
||||
## Architecture
|
||||
|
||||
- `chusql-core/storage/src/runtime.rs`:Storage 持有数据目录文件锁、管理数据库;Server 管每库 catalog、WAL、表缓存和读写门。
|
||||
- 数据根目录中 system/ 保存系统库,databases/<name>/ 保存业务库。每库有 catalog.json、<table>.db、内建 <table>.idx、其他 <table>.<column>.idx、wal.log 与 wal.checkpoint;诊断日志由 log_files 单独配置。
|
||||
- `chusql-core/storage/src/page.rs` 管定长页与缓存刷盘;`heap.rs` 管页内槽位、JSON 行、隐藏列和 compact;`btree.rs` 管变长字节键、重复键与叶子链范围扫描。
|
||||
- `chusql-core/storage/src/catalog.rs` 管 schema、索引、统计、隐藏列、对象编号 / 高水位;`protocol.rs` 定义线上 schema 和系统表常量。
|
||||
- `chusql-core/storage/src/wal.rs` 管 LSN、提交组、日志扫描与检查点;runtime.rs 的 group_via_wal、recover_wal、replay_op 连接记录与应用。
|
||||
- `chusql-server/src/ChuSQL/Server/Session.hs` 管会话快照、保存点、差分与冲突;`Backend.hs` 向存储提交操作。`chusql-bootstrap/` 调用 Storage::maintenance 与 maintenance_candidate 做离线维护。
|
||||
|
||||
## Invariants
|
||||
|
||||
- 表存在性以 catalog 为准,文件存在不表示已登记;不能靠打开文件创建逻辑表。语义与优化只读取 schema。
|
||||
- Catalog::save 写 pending 文件、同步后替换;单文件替换不等于 catalog、页与索引已整体原子提交。
|
||||
- 普通组先 append_group 落盘,再设置 recovery_required,然后应用、刷表、推进检查点、截断日志,全部成功才解除恢复状态。
|
||||
- 应用 / 刷盘 / 检查点 / 截断失败必须保持恢复状态并报错。普通业务请求被阻断;Ping / SystemStatus 例外由各自分派层处理,新增入口不得绕过状态。
|
||||
- 恢复仅重放检查点之后已提交组,保持 replay_op 幂等;不应用未提交组。现有 WAL 是 redo,检查点是库级 LSN,不是 undo 或页级恢复。
|
||||
- 账号引导 / 变更存在独立 WAL 路径,不能假定所有写入都走普通组函数。
|
||||
- 库 / 表对象和身份编号依赖持久化高水位,删除后不复用;迁移不能降低计数器,不能从现存对象推断已删除编号上限。
|
||||
- 系统身份 / 凭据、角色、授权、类型与 DOMAIN 元数据保持对应关系。引导保留已有口令和降权状态,拒绝定义 / 保留文件冲突;类型目录不是可动态扩展基础类型的机制。
|
||||
- 数据目录锁覆盖存储和维护生命周期;维护前停止占用进程。每库读共享 / 写独占、表句柄加锁,不假定同库跨表并行写。
|
||||
- 会话事务是整库内存快照,无嵌套事务,事务内禁止 DDL / USE。提交跳过未修改表,整数 id 按键差分,否则整表比较 / 替换;对象身份和权限需复查,冲突返回 serialization_failure 并保留事务。
|
||||
- 页内 compact 保留槽位编号与索引位置;隐藏列 compact 可能整表重写。布局、键编码、行序和隐藏列改动须检查真实消费者。
|
||||
- 页大小、B+Tree magic / 布局、槽位小端字段、JSON 行、WAL 标签 / 帧和 catalog 字段是格式契约。改变前明确加载、迁移或拒绝策略,不以重新初始化掩盖不兼容。
|
||||
|
||||
## Engineering Decisions
|
||||
|
||||
- 优化可减少分配或重复读取,但保留验证、刷盘顺序、错误传播与可观察行序。结构变更同时核对 WAL、应用、幂等恢复与 catalog 保存。
|
||||
- 数据库目录建删不走普通表 WAL,删库使用墓碑目录;不能承诺表提交组相同的崩溃原子性。
|
||||
- 离线维护在候选目录操作并保留备份;repair 不改既有表数据,recover 依赖可用 WAL,reset 重建系统目录且保留业务库。失败保留原目录 / 恢复材料。
|
||||
- 当前 Catalog::load 对不存在或空文件返回空目录,JSON 损坏报错;不能宣称所有损坏都会拒绝。rename_table_files 和部分析构 / 清理路径仍忽略 I/O 错误,不作为成功持久化保证。
|
||||
- 尚无行版本 MVCC、页级恢复、跨页填洞、并行写或通用结构迁移事务,不据此放松校验。
|
||||
|
||||
## Validation
|
||||
|
||||
- 按影响选择 `chusql-core/storage/tests/` 的 page / heap / btree / catalog / wal / runtime 测试;说明修正核对源码即可。
|
||||
- 格式 / 元数据 / WAL 改动验证重开、提交前后中断、重复恢复、缺失 / 截断 / 损坏输入与失败后状态;断言须检测实际不一致。
|
||||
- 会话差分 / 冲突变更还验证 `chusql-server/test/Spec.hs`;并发压力测试不是正确性证明。
|
||||
- 规定入口为 `gate-check/run-step.ps1 -Kind cargo`,动态库构建为 -Kind build;受影响 Haskell 模块按脚本命令串行构建,全量门禁遵守 AGENTS.md。
|
||||
|
||||
## References
|
||||
|
||||
恢复操作按需读 `docs/bootstrap.md`,配置见 `docs/config.md`,限制见 `.agents/context/known-issues.txt`;涉及拟议能力时仍以源码为准。
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: chusql-type-system
|
||||
description: 修改 ChuSQL SQL 类型、值表示、赋值转换、类型推导、函数或表达式语义时使用;提供类型模块职责与一致性约束。仅调整界面样式或无关存储性能时不加载。
|
||||
---
|
||||
|
||||
# ChuSQL 类型系统
|
||||
|
||||
## Scope
|
||||
|
||||
负责类型语义;落盘顺序交给 chusql-storage-integrity,传输契约交给 chusql-boundary-contracts。通用语言与测试规范使用已有 Skills,不重复加载。下列路径相对仓库根目录,只读本次相关模块。
|
||||
|
||||
## Architecture
|
||||
|
||||
- `chusql-core/model/src/ChuSQL/Core/Model.hs`:`ColumnType`、列约束、`Value`;集中提供 `assignable`、`comparableTypes`、`valueFits`、`coerceValue`、相等与哈希语义。
|
||||
- `chusql-core/model/src/ChuSQL/Core/Runtime.hs`:`TypeExpr` / `TypeId`、`Kind`、独立 `PhysicalType`、`RuntimeValue`、注册表和版本化编码;Engine 的 `Runtime/Types.hs` 只是重新导出。
|
||||
- `chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs` 与 `Syntax/AST.hs` 解析声明和构造表达式;`Semantic.hs` 根据 schema 解析 DOMAIN、推导表达式并绑定身份。
|
||||
- `chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Functions.hs` 定义固定 FunctionId 与签名,现有函数 length / not;`Runtime/SQL.hs` 连接 SQL 标量和运行时值。
|
||||
- `chusql-core/engine/src/ChuSQL/Core/Engine/Runtime/Query.hs` 的 BoundQuery 保存 schema、输出类型和优化后的 RelOp,执行后按位置校验。`Runtime/Binding.hs` 另提供类型化表达式与基础计划,尚未替代完整 SQL 关系代数。
|
||||
- `chusql-core/model/src/ChuSQL/Core/Protocol.hs` 共享值的 JSON 转换;Rust 的 `chusql-core/storage/src/protocol.rs` 使用 `HashMap<String, serde_json::Value>` 保存行。
|
||||
|
||||
## Invariants
|
||||
|
||||
- 构造器编号固定:Int=1、Double=2、Bool=3、String=4、List=5、Maybe=6、Tuple=7。使用 resolveType 检查参数 / kind,不把显示名当身份,不再维护第二份注册表。
|
||||
- CRuntime / VRuntime 身份必须匹配;递归检查列表元素、Maybe 内容与元组字段数,Double 必须有限。保留逻辑类型与物理表示的区别。
|
||||
- SQL VNull 与 RMaybe Nothing 不同;CNull 是查询推导类型,不能声明为存储列。保留三值逻辑与函数 NULL 传播。
|
||||
- 赋值兼容和比较兼容分别使用 Model 入口。coerceValue 负责值收敛;整值浮点转整数、字符串长度、日期格式、DECIMAL 舍入沿现有规则处理,不在 Parser / Eval / Web 重复实现转换。
|
||||
- 整数与可规范化整值浮点保持相等 / 哈希一致。底层 compareValue 的实现不表示 SQL 已开放该类型的比较、排序、分组或索引。
|
||||
- DOMAIN 使用 domainTableName 与目录定义解析基础类型、依赖及循环;不能拿解析器占位基础类型直接执行。系统内建类型目录也不是通过增加行即可扩展的插件注册表。
|
||||
- Date / Timestamp / Blob 在 QueryType 保留 CompatibilityType;ColumnType 与 RelOp 仍有真实消费者,不能作为无用兼容层删除。
|
||||
- 类型 / 值编码版本为 1;身份、类型名、封装或值结构变化可能影响 catalog、WAL、FFI、TCP 和 Web,组合相关 Skills 核对消费者。
|
||||
- Rust 当前保存通用 JSON,只做局部校验(隐藏列、DOMAIN 依赖、DECIMAL 规范化),没有 Haskell validateValue 的完整等价检查;不要依赖原始存储请求完成所有 SQL 类型校验。
|
||||
- 错误沿 Either String 返回;`chusql-core/engine/src/ChuSQL/Core/Engine/Error.hs` 用消息规则映射类别 / 协议码。修改错误文字须检查分类,不用默认值或 NULL 掩盖错误。
|
||||
|
||||
## Engineering Decisions
|
||||
|
||||
复用 Model / Runtime 的语义入口,在现有类型桥上做最小修改。尚无完整 capability / operator / cast 注册机制、通用显式 CAST、用户 ADT 或 Extension;DOMAIN 自身 CHECK / NOT NULL / DEFAULT 尚未实现。不要将规划当现有契约。
|
||||
DECIMAL 是有限浮点表示,采用 ties-to-even 舍入与舍入后溢出检查,并非精确十进制存储。运行时复合类型尚无 SQL Eq / Ord / Hash capability 字典。
|
||||
|
||||
## Validation
|
||||
|
||||
- 语义改动验证解析 → schema 绑定 → 执行,关注 NULL、边界值、明确错误与输出位置;说明修正只需核对源码。
|
||||
- 复用 `chusql-core/engine/test/RuntimeSpec.hs`、`chusql-core/engine/test/Spec.hs`,按需覆盖固定编码、错误身份和嵌套值。
|
||||
- 持久化类型变更验证 Rust 写入 / 重载与实际协议消费者;Web 值展示 / SQL 生成参考 `chusql-web/test/runtime-values.mjs`。
|
||||
- 测试入口见 `gate-check/run-step.ps1`,不因加载 Skill 自动执行全量门禁或追求覆盖率。
|
||||
|
||||
## References
|
||||
|
||||
按需读 `docs/runtime-types.md`;`docs/types.md` 含未来设计,必须与实现区分。以上源码模块是契约核对入口。
|
||||
+10
-4
@@ -76,11 +76,7 @@ resources/chusql.ui.settings.json
|
||||
*~
|
||||
\#*\#
|
||||
.\#*
|
||||
AGENTS.md
|
||||
agent-credientials
|
||||
skills-lock.json
|
||||
.dsh
|
||||
.agents
|
||||
# 注意:.zed/tasks.json 是本仓库共享的构建任务(stack build / run / test / repl),
|
||||
# 需要入库,所以这里不忽略 .zed/
|
||||
|
||||
@@ -97,3 +93,13 @@ plandocs/
|
||||
Thumbs.db
|
||||
ehthumbs.db
|
||||
desktop.ini
|
||||
|
||||
learning
|
||||
|
||||
# 通用技能和安装状态仅供本机;只跟踪项目专用技能
|
||||
/.agents/skills/*
|
||||
!/.agents/skills/chusql-boundary-contracts/
|
||||
!/.agents/skills/chusql-storage-integrity/
|
||||
!/.agents/skills/chusql-type-system/
|
||||
/.agents/context/prompt.txt
|
||||
/skills-lock.json
|
||||
@@ -0,0 +1,39 @@
|
||||
# AGENTS.md — ChuSQL 开发契约
|
||||
|
||||
参考:
|
||||
- 唯一进度与验收来源:`.agents/context/progress.txt`(等价于 project-plan.txt)。
|
||||
- 限制与设计:`.agents/context/known-issues.txt`、`.agents/context/design-decisions.txt`。
|
||||
|
||||
## 核心约束
|
||||
- 测试必须通过,禁止绕过检查。
|
||||
- 不吞错误,使用明确错误返回。
|
||||
- FFI 响应由 Rust 分配并通过配套 free 归还;句柄关闭不得与请求并发。
|
||||
- 表存在性以 catalog 为准。
|
||||
- 语义检查和优化只使用 schema。
|
||||
- 索引由存储层管理,优化器只判断可点查条件。
|
||||
- 优化不能改变结果语义。
|
||||
- 持久化、事务、并发、类型及 FFI 改动优先维护不变量、内存所有权与错误传播;保留必要的迁移和恢复约束。
|
||||
|
||||
## 开发方式
|
||||
- 按任务选择已安装 Skills:编码用 ponytail / karpathy-guidelines,审查与简化按需用 ponytail-review;Rust、FFI、测试及覆盖率分别使用对应项目技能。
|
||||
- 类型、值、转换或表达式语义用 `chusql-type-system`;存储、序列化、系统目录、初始化或恢复用 `chusql-storage-integrity`;FFI、跨组件协议、序列化契约或外部接口用 `chusql-boundary-contracts`(均在 `.agents/skills/`)。
|
||||
- 跨领域组合相关 Skills,无关任务不加载;Skills 不覆盖更高优先级的安全和工程约束。
|
||||
- 简单任务直接执行;采用最小必要改动,允许有明确收益且能验证的局部结构优化,优先删除重复和无用代码。
|
||||
- 测试强度与实际风险及影响范围匹配,不机械新增测试,不盲目追求覆盖率;数据库内核微改动也须验证关键行为。
|
||||
- 不强制 TDD、子代理或多阶段审查;减少无关读取、重复搜索、工具调用和输出,Skills 不应增加无收益的流程。
|
||||
- 文档描述当前状态;不创建整改报告、修改流水账或无用途的兼容层,不修改 Git 历史。
|
||||
|
||||
## 注释规范
|
||||
- 文件级:外部引用语句(Haskell `import`、Rust `use`、JS `import`、HTML 的 `<link>`/`<script>`、
|
||||
CSS `@import`、脚本的依赖/参数区)之后写一段注释,概括本文件的功能;没有引用语句的文件写在
|
||||
文件开头。长度不超过 3 行、合计不超过 90 字。
|
||||
- 函数级:每个函数(含 Haskell 的 `where`/`let` 内函数、Rust 的私有 `fn`)前用一行不超过
|
||||
30 字的注释说明它做什么。
|
||||
- 摘要说明做什么;仅在必要处补充不直观的不变量、所有权与安全约束,不记录历史和被删实现。
|
||||
|
||||
## 协作规则
|
||||
- 全量门禁入口:`gate-check/run-gate.ps1`(三条车道并行,日志写仓库根的 `logs/`)。
|
||||
- 门禁由项目所有者手动运行;agent 只跑改动模块的测试,拿到门禁结果后再提交。
|
||||
- engine / web / storage 使用规定测试入口。
|
||||
- 规定入口见 `gate-check/run-step.ps1`;Haskell 构建串行,存储协议改动须重建实际动态库。
|
||||
- 改动说明只在交流中记录。
|
||||
@@ -44,6 +44,7 @@ csql-web
|
||||
|
||||
## 更多参考文档
|
||||
|
||||
- [数据库原理学习与答辩文档](learnning/README.md)
|
||||
- [安装与卸载](docs/install.md)
|
||||
- [配置选项](docs/config.md)
|
||||
- [命令与选项](docs/commands.md)
|
||||
|
||||
@@ -28,6 +28,10 @@ executable csql-bootstrap
|
||||
, text
|
||||
default-language: Haskell2010
|
||||
ghc-options: -Wall
|
||||
if os(linux)
|
||||
ld-options: -Wl,-rpath,$ORIGIN
|
||||
if os(osx)
|
||||
ld-options: -Wl,-rpath,@loader_path
|
||||
|
||||
test-suite chusql-bootstrap-test
|
||||
type: exitcode-stdio-1.0
|
||||
|
||||
+26
-10
@@ -2,6 +2,7 @@
|
||||
|
||||
module Main (main) where
|
||||
|
||||
import ChuSQL.CLI.Normalization (runNormalizationOnce)
|
||||
import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat)
|
||||
import ChuSQL.CLI.Password (readPasswordInput)
|
||||
import ChuSQL.CLI.Script (stripTerminator)
|
||||
@@ -19,7 +20,7 @@ import ChuSQL.Interface.Sudo (SudoCredential, readSudoCredential)
|
||||
import ChuSQL.Interface.TOML (resolveConfigPath)
|
||||
import Control.Exception (finally)
|
||||
import Control.Monad (unless)
|
||||
import Data.IORef (newIORef)
|
||||
import Data.IORef (IORef, newIORef)
|
||||
import Data.Maybe (fromMaybe)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
@@ -40,6 +41,8 @@ data Options = Options
|
||||
, optExecute :: Maybe Text
|
||||
, optHistory :: Maybe FilePath
|
||||
, optConfig :: Maybe FilePath
|
||||
, optNormalize :: Maybe Text
|
||||
, optDeclaration :: Maybe FilePath
|
||||
, optSudo :: Bool
|
||||
}
|
||||
|
||||
@@ -64,16 +67,19 @@ optionsParser =
|
||||
<> metavar "FORMAT"
|
||||
<> value FormatTable
|
||||
<> showDefaultWith (T.unpack . formatName)
|
||||
<> help "Output format: table, json or csv"
|
||||
<> help "Output format: table (or text), json or csv"
|
||||
)
|
||||
<*> optional (T.pack <$> strOption (long "execute" <> short 'e' <> metavar "SQL" <> help "Run one statement and exit"))
|
||||
<*> optional (strOption (long "history" <> metavar "FILE" <> help "SQL history file (default ~/.chusql_history)"))
|
||||
<*> optional (strOption (long "config" <> metavar "FILE" <> help "Config file to read (default: the fixed settings.toml under the system config directory)"))
|
||||
<*> optional (T.pack <$> strOption (long "normalize" <> metavar "TABLE" <> help "Analyze normalization using explicit declarations"))
|
||||
<*> optional (strOption (long "declaration" <> metavar "FILE" <> help "UTF-8 JSON normalization declaration"))
|
||||
<*> switch (long "sudo" <> help "Authenticate using the local root or elevated administrator identity")
|
||||
|
||||
-- | 把 --format 的取值解析成格式
|
||||
formatReader :: String -> Either String OutputFormat
|
||||
formatReader raw = maybe (Left "format must be table, json or csv") Right (parseFormat (T.pack raw))
|
||||
formatReader "text" = Right FormatTable
|
||||
formatReader raw = maybe (Left "format must be table, text, json or csv") Right (parseFormat (T.pack raw))
|
||||
|
||||
-- | 命令行界面与帮助
|
||||
parserInfo :: ParserInfo Options
|
||||
@@ -91,6 +97,10 @@ main = do
|
||||
hSetEncoding stdout utf8
|
||||
hSetEncoding stderr utf8
|
||||
options <- execParser parserInfo
|
||||
case (optNormalize options, optDeclaration options, optExecute options, optFormat options) of
|
||||
(Nothing, Nothing, _, _) -> pure ()
|
||||
(Just _, Just _, Nothing, format) | format /= FormatCsv -> pure ()
|
||||
_ -> reportError "--normalize requires --declaration, text/json format and cannot be combined with --execute" >> exitFailure
|
||||
configPath <- resolveConfigPath (optConfig options)
|
||||
base <- loadWebConfigAt configPath
|
||||
config <- loadServerConfigAt configPath
|
||||
@@ -129,13 +139,19 @@ runSession options session = do
|
||||
ready <- selectDatabase session (T.strip (optDatabase options))
|
||||
if not ready
|
||||
then exitFailure
|
||||
else case optExecute options of
|
||||
Just sql -> do
|
||||
ok <- runStatementOnce session (optFormat options) (stripTerminator sql)
|
||||
unless ok exitFailure
|
||||
Nothing -> do
|
||||
historyFile <- resolveHistory (optHistory options)
|
||||
runRepl session formatRef historyFile
|
||||
else case (optNormalize options, optDeclaration options) of
|
||||
(Just table, Just file) -> runNormalizationOnce session table file (optFormat options == FormatJson) >>= \ok -> unless ok exitFailure
|
||||
_ -> runSqlSession options session formatRef
|
||||
|
||||
-- 执行 SQL 或进入交互循环。
|
||||
runSqlSession :: Options -> Session -> IORef OutputFormat -> IO ()
|
||||
runSqlSession options session formatRef = case optExecute options of
|
||||
Just sql -> do
|
||||
ok <- runStatementOnce session (optFormat options) (stripTerminator sql)
|
||||
unless ok exitFailure
|
||||
Nothing -> do
|
||||
historyFile <- resolveHistory (optHistory options)
|
||||
runRepl session formatRef historyFile
|
||||
|
||||
-- | -d 指定了库才切,否则保持未选库
|
||||
selectDatabase :: Session -> Text -> IO Bool
|
||||
|
||||
@@ -6,6 +6,7 @@ module Repl
|
||||
, reportError
|
||||
) where
|
||||
|
||||
import ChuSQL.CLI.Normalization (runNormalizationOnce)
|
||||
import ChuSQL.CLI.Format (OutputFormat, formatName, renderResult, renderRows)
|
||||
import ChuSQL.CLI.History (historySettings, rememberStatement)
|
||||
import ChuSQL.CLI.Script (Meta (..), errorHint, metaHelp, parseMeta, takeStatement)
|
||||
@@ -106,6 +107,7 @@ runMeta session formatRef meta = case meta of
|
||||
MetaDatabases -> listDatabases session >> pure True
|
||||
MetaTables -> listTables session >> pure True
|
||||
MetaDescribe name -> describeTable session name >> pure True
|
||||
MetaNormalize table file jsonOutput -> liftIO (runNormalizationOnce session table file jsonOutput) >> pure True
|
||||
MetaRoles -> listRoles session >> pure True
|
||||
MetaConnect name -> do
|
||||
switched <- liftIO (switchDatabase session name)
|
||||
|
||||
@@ -12,6 +12,7 @@ build-type: Simple
|
||||
library
|
||||
hs-source-dirs: src
|
||||
exposed-modules: ChuSQL.CLI.Format
|
||||
ChuSQL.CLI.Normalization
|
||||
ChuSQL.CLI.History
|
||||
ChuSQL.CLI.Password
|
||||
ChuSQL.CLI.Script
|
||||
@@ -55,6 +56,7 @@ test-suite chusql-cli-test
|
||||
, directory
|
||||
, aeson
|
||||
, bytestring
|
||||
, chusql-interface
|
||||
, chusql-cli
|
||||
, chusql-core-model
|
||||
, hspec
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module ChuSQL.CLI.Normalization (normalizationArguments, loadNormalizationDeclaration, renderNormalization, runNormalizationOnce) where
|
||||
|
||||
import ChuSQL.Interface.Normalization
|
||||
import ChuSQL.Interface.Session (Session, sessionDatabase, analyzeNormalization)
|
||||
import Control.Exception (IOException, try)
|
||||
import qualified Data.Aeson as A
|
||||
import qualified Data.Aeson.Key as K
|
||||
import qualified Data.Aeson.KeyMap as KM
|
||||
import qualified Data.ByteString as BS
|
||||
import qualified Data.ByteString.Lazy as BL
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Encoding as TE
|
||||
import qualified Data.Text.IO as TIO
|
||||
import System.IO (IOMode (ReadMode), stderr, withBinaryFile)
|
||||
import Text.ParserCombinators.ReadP
|
||||
|
||||
-- CLI 范式入口:读取严格声明文件、发送专用请求并显示报告。
|
||||
|
||||
-- 解析表、声明路径和独立显示格式。
|
||||
normalizationArguments :: Text -> Either Text (Text, FilePath, Bool)
|
||||
normalizationArguments input = case readP_to_S (skipSpaces *> sepBy token skipSpaces <* skipSpaces <* eof) (T.unpack input) of
|
||||
[(parts, "")] -> case parts of
|
||||
[table, "--declaration", file] -> Right (T.pack table, file, False)
|
||||
[table, "--declaration", file, "--format", "text"] -> Right (T.pack table, file, False)
|
||||
[table, "--declaration", file, "--format", "json"] -> Right (T.pack table, file, True)
|
||||
_ -> Left usage
|
||||
_ -> Left usage
|
||||
where
|
||||
usage = "usage: \\normalize TABLE --declaration FILE [--format text|json]"
|
||||
-- 保留引号内的 Windows 路径及空格。
|
||||
token = quoted '"' <++ quoted '\'' <++ munch1 (\c -> c /= ' ' && c /= '\t' && c /= '\n' && c /= '"' && c /= '\'')
|
||||
-- 读取不带转义解释的引号参数。
|
||||
quoted quote = char quote *> munch (/= quote) <* char quote
|
||||
|
||||
-- 有界读取 UTF-8 声明并拒绝重复字段。
|
||||
loadNormalizationDeclaration :: FilePath -> IO (Either Text NormalizationDeclaration)
|
||||
loadNormalizationDeclaration path = do
|
||||
loaded <- try (withBinaryFile path ReadMode (\handle -> BS.hGet handle 65537)) :: IO (Either IOException BS.ByteString)
|
||||
pure $ case loaded of
|
||||
Left err -> Left (T.pack (show err))
|
||||
Right raw -> do
|
||||
value <- decodeNormalizationJSON (BL.fromStrict raw)
|
||||
case A.fromJSON value of
|
||||
A.Error message -> Left (T.pack message)
|
||||
A.Success declaration -> Right declaration
|
||||
|
||||
-- 保留报告字段并选择文本或 JSON 显示。
|
||||
renderNormalization :: Bool -> A.Value -> Text
|
||||
renderNormalization True = jsonText
|
||||
renderNormalization False = pretty 0
|
||||
|
||||
-- 将 JSON 编码为 UTF-8 文本。
|
||||
jsonText :: A.Value -> Text
|
||||
jsonText = TE.decodeUtf8 . BL.toStrict . A.encode
|
||||
|
||||
-- 缩进展示完整报告且保留所有证据。
|
||||
pretty :: Int -> A.Value -> Text
|
||||
pretty depth (A.Object fields) = T.intercalate "\n" [indent <> K.toText key <> ":\n" <> pretty (depth + 1) value | (key, value) <- KM.toList fields]
|
||||
where indent = T.replicate depth " "
|
||||
pretty depth value = T.replicate depth " " <> jsonText value
|
||||
|
||||
-- 发送声明并让失败决定 CLI 退出码。
|
||||
runNormalizationOnce :: Session -> Text -> FilePath -> Bool -> IO Bool
|
||||
runNormalizationOnce session table path jsonOutput = do
|
||||
declaration <- loadNormalizationDeclaration path
|
||||
database <- sessionDatabase session
|
||||
result <- case declaration of
|
||||
Left message -> pure (Left message)
|
||||
Right d -> analyzeNormalization session (NormalizationRequest 1 database table d Nothing Nothing)
|
||||
case result of
|
||||
Left message -> TIO.hPutStrLn stderr ("chusql: error: " <> message) >> pure False
|
||||
Right analysis -> TIO.putStrLn (renderNormalization jsonOutput analysis) >> pure True
|
||||
@@ -10,6 +10,7 @@ module ChuSQL.CLI.Script
|
||||
, metaHelp
|
||||
) where
|
||||
|
||||
import ChuSQL.CLI.Normalization (normalizationArguments)
|
||||
import ChuSQL.CLI.Format (OutputFormat (..), formatName, parseFormat)
|
||||
import Data.Char (isSpace)
|
||||
import Data.Text (Text)
|
||||
@@ -23,6 +24,7 @@ data Meta
|
||||
| MetaDatabases
|
||||
| MetaTables
|
||||
| MetaDescribe Text
|
||||
| MetaNormalize Text FilePath Bool
|
||||
| MetaRoles
|
||||
| MetaConnect Text
|
||||
| MetaSetFormat OutputFormat
|
||||
@@ -50,6 +52,7 @@ parseMeta raw = do
|
||||
| otherwise -> MetaDescribe argument
|
||||
"dr" -> MetaRoles
|
||||
"roles" -> MetaRoles
|
||||
"normalize" -> either (const (MetaUnknown "normalize")) (\(table, file, jsonOutput) -> MetaNormalize table file jsonOutput) (normalizationArguments argument)
|
||||
"c" -> connectTo argument
|
||||
"connect" -> connectTo argument
|
||||
"use" -> connectTo argument
|
||||
@@ -118,6 +121,7 @@ metaHelp =
|
||||
, " \\l, \\list list databases"
|
||||
, " \\dt, \\d list tables"
|
||||
, " \\d <table> describe a table"
|
||||
, " \\normalize TABLE --declaration FILE [--format text|json]"
|
||||
, " \\dr, \\roles list roles with their grants and members"
|
||||
, " \\c <database> switch database"
|
||||
, " \\f, \\format <fmt> output format: " <> T.intercalate ", " (map formatName [FormatTable, FormatJson, FormatCsv])
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
module Main (main) where
|
||||
|
||||
import ChuSQL.CLI.Normalization
|
||||
import ChuSQL.Interface.Normalization
|
||||
import ChuSQL.CLI.Format (OutputFormat (..), parseFormat, renderCsv, renderJson, renderTable, displayValue)
|
||||
import ChuSQL.CLI.History (historySettings, rememberStatement)
|
||||
import ChuSQL.CLI.Password (readPasswordInput)
|
||||
@@ -31,6 +33,7 @@ main = hspec spec
|
||||
-- | 全部测试用例
|
||||
spec :: Spec
|
||||
spec = do
|
||||
normalizationSpec
|
||||
it "renders typed compound values without losing their wire identity" $ do
|
||||
case resolveType builtinTypes (listType intType) of
|
||||
Left err -> expectationFailure err
|
||||
@@ -188,3 +191,26 @@ sampleResult =
|
||||
, qrTruncated = False
|
||||
, qrDatabase = Nothing
|
||||
}
|
||||
|
||||
|
||||
-- 范式元命令、严格文件与显示格式回归。
|
||||
normalizationSpec :: Spec
|
||||
normalizationSpec = describe "CLI: normalization" $ do
|
||||
it "parses quoted paths without interpreting Windows backslashes" $ do
|
||||
normalizationArguments "users --declaration \"C:\\My Data\\fd.json\" --format json" `shouldBe` Right ("users", "C:\\My Data\\fd.json", True)
|
||||
parseMeta "\\normalize users --declaration deps.json --format text" `shouldBe` Just (MetaNormalize "users" "deps.json" False)
|
||||
normalizationArguments "users --declaration deps.json --format csv" `shouldSatisfy` either (const True) (const False)
|
||||
it "keeps the same report fields in text and JSON output" $ do
|
||||
let report = A.object ["candidate_keys" A..= ([["id"]] :: [[T.Text]]), "conditional" A..= True, "findings" A..= [A.object ["normal_form" A..= ("bcnf" :: T.Text), "verdict" A..= ("insufficient" :: T.Text)]]]
|
||||
A.eitherDecodeStrict (encodeUtf8 (renderNormalization True report)) `shouldBe` Right report
|
||||
renderNormalization False report `shouldSatisfy` T.isInfixOf "candidate_keys"
|
||||
renderNormalization False report `shouldSatisfy` T.isInfixOf "insufficient"
|
||||
it "loads defaults and refuses duplicate JSON fields in declaration files" $ do
|
||||
directory <- getTemporaryDirectory
|
||||
bracket (openTempFile directory "normalization-declaration") (\(path, _) -> removeFile path) $ \(path, handle) -> do
|
||||
hPutStr handle "{}"
|
||||
hClose handle
|
||||
loadNormalizationDeclaration path `shouldReturn` Right (NormalizationDeclaration [] [] False Nothing)
|
||||
BL.writeFile path "{\"atomic_attributes\":true,\"atomic_attributes\":false}"
|
||||
result <- loadNormalizationDeclaration path
|
||||
result `shouldSatisfy` either (T.isInfixOf "duplicate") (const False)
|
||||
@@ -28,6 +28,8 @@ library
|
||||
ChuSQL.Core.Engine.Runtime.Binding
|
||||
ChuSQL.Core.Engine.Runtime.SQL
|
||||
ChuSQL.Core.Engine.Runtime.Query
|
||||
ChuSQL.Core.Engine.Runtime.Relation
|
||||
ChuSQL.Core.Engine.Runtime.Dependencies
|
||||
ChuSQL.Core.Engine.Semantic
|
||||
ChuSQL.Core.Engine.Storage
|
||||
ChuSQL.Core.Engine.Storage.FFI
|
||||
@@ -53,6 +55,7 @@ library
|
||||
if os(osx)
|
||||
ld-options: -Wl,-rpath,@loader_path
|
||||
default-language: Haskell2010
|
||||
default-extensions: ImportQualifiedPost
|
||||
ghc-options: -Wall
|
||||
|
||||
test-suite chusql-core-engine-test
|
||||
|
||||
@@ -3,14 +3,14 @@ module ChuSQL.Core.Engine (runStatement, runStatementM, rowsOf) where
|
||||
import ChuSQL.Core.Engine.Algebra.Eval (evalCondForRowM, evalExprM)
|
||||
import ChuSQL.Core.Engine.Algebra.Expr (evalCondForRow, evalExpr)
|
||||
import ChuSQL.Core.Engine.Runtime.Query (bindQuery, executeQueryM)
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Semantic (prepare, prepareRowExpression)
|
||||
import ChuSQL.Core.Engine.Storage (MonadStorage (..))
|
||||
import ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage))
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
|
||||
import ChuSQL.Core.Model
|
||||
import Data.HashSet qualified as HS
|
||||
import Data.Maybe (mapMaybe)
|
||||
import qualified Data.HashSet as HS
|
||||
|
||||
-- 引擎入口:检查语义、按语句分发执行,另有内存实现与泛型版本。
|
||||
|
||||
@@ -39,6 +39,10 @@ runStatementM q = do
|
||||
|
||||
-- | 按语句类型分发(已检查过)
|
||||
runStatementUncheckedM :: (MonadStorage m) => Database -> Statement -> m (Either String [Row])
|
||||
runStatementUncheckedM db q@SelectDistinct{} =
|
||||
case bindQuery db q of
|
||||
Left e -> pure (Left e)
|
||||
Right bound -> executeQueryM bound
|
||||
runStatementUncheckedM db q@Select{} =
|
||||
case bindQuery db q of
|
||||
Left e -> pure (Left e)
|
||||
@@ -74,7 +78,7 @@ runStatementUncheckedM db (Delete tbl mWhere) = do
|
||||
result <- replaceAll tbl kept
|
||||
pure (result >> Right [])
|
||||
where
|
||||
-- | 取行里的整数 id,取不到就报错
|
||||
-- \| 取行里的整数 id,取不到就报错
|
||||
rowId :: Row -> Either String Int
|
||||
rowId r = case lookup "id" r of
|
||||
Just (VInt k) -> Right k
|
||||
@@ -104,9 +108,15 @@ runStatementUncheckedM _ (CreateDomain name base) = do
|
||||
runStatementUncheckedM _ (DropDomain name) = do
|
||||
result <- dropTable (domainTableName name)
|
||||
pure (result >> Right [])
|
||||
runStatementUncheckedM db ShowDomains = pure (Right
|
||||
[[("domain", VStr (drop (length (domainTableName "")) name)), ("base_type", VStr (typeName (columnType base)))]
|
||||
| (name, table) <- db, isDomainTable name, Just base <- [lookup "base" (tableCols table)]])
|
||||
runStatementUncheckedM db ShowDomains =
|
||||
pure
|
||||
( Right
|
||||
[ [("domain", VStr (drop (length (domainTableName "")) name)), ("base_type", VStr (typeName (columnType base)))]
|
||||
| (name, table) <- db
|
||||
, isDomainTable name
|
||||
, Just base <- [lookup "base" (tableCols table)]
|
||||
]
|
||||
)
|
||||
runStatementUncheckedM _ (DropTable name) = do
|
||||
result <- dropTable name
|
||||
pure (result >> Right [])
|
||||
@@ -132,11 +142,13 @@ runStatementUncheckedM db (AlterColumnNull tbl col nullable) =
|
||||
runStatementUncheckedM _ CreateUser{} = pure (Left "CREATE USER is executed by the account service")
|
||||
runStatementUncheckedM _ AlterUser{} = pure (Left "ALTER USER is executed by the account service")
|
||||
runStatementUncheckedM _ AlterIdentity{} = pure (Left "ALTER ROLE is executed by the account service")
|
||||
runStatementUncheckedM _ ShowGrants = pure (Left "SHOW GRANTS is executed by the account service")
|
||||
runStatementUncheckedM _ ShowRoles = pure (Left "SHOW ROLES is executed by the account service")
|
||||
runStatementUncheckedM _ DropUser{} = pure (Left "DROP USER is executed by the account service")
|
||||
runStatementUncheckedM _ CreateRole{} = pure (Left "CREATE ROLE is executed by the privilege service")
|
||||
runStatementUncheckedM _ DropRole{} = pure (Left "DROP ROLE is executed by the privilege service")
|
||||
runStatementUncheckedM _ GrantPrivileges{} = pure (Left "GRANT is executed by the privilege service")
|
||||
runStatementUncheckedM _ RevokeGrantOption{} = pure (Left "REVOKE is executed by the privilege service")
|
||||
runStatementUncheckedM _ RevokePrivileges{} = pure (Left "REVOKE is executed by the privilege service")
|
||||
runStatementUncheckedM _ GrantRole{} = pure (Left "GRANT is executed by the privilege service")
|
||||
runStatementUncheckedM _ RevokeRole{} = pure (Left "REVOKE is executed by the privilege service")
|
||||
@@ -157,9 +169,9 @@ existingRows tbl table
|
||||
| needsRows table = scan tbl
|
||||
| otherwise = pure (Right [])
|
||||
where
|
||||
-- | 这一表有没有要判约束的列
|
||||
-- \| 这一表有没有要判约束的列
|
||||
needsRows t = any (constraintColumn . snd) (tableCols t)
|
||||
-- | 这一列带不带要判的约束
|
||||
-- \| 这一列带不带要判的约束
|
||||
constraintColumn c = columnAutoIncrement c || columnUnique c || columnPrimaryKey c
|
||||
|
||||
-- | 插一行:补齐缺列、给自增值、收类型、查约束
|
||||
@@ -172,7 +184,7 @@ buildRows table current cols rows = do
|
||||
let auto = autoColumn table
|
||||
go auto (autoStart auto current) values []
|
||||
where
|
||||
-- | 逐行构造,最后统一查一遍约束
|
||||
-- \| 逐行构造,最后统一查一遍约束
|
||||
go _ _ [] acc = do
|
||||
let built = reverse acc
|
||||
plan <- checkPlan table
|
||||
@@ -189,25 +201,27 @@ buildRow :: Table -> Maybe (String, Column) -> [String] -> Int -> [Value] -> Eit
|
||||
buildRow table auto cols counter vals = go (tableCols table) counter []
|
||||
where
|
||||
provided = zip cols vals
|
||||
-- | 按列定义逐列补值
|
||||
-- \| 按列定义逐列补值
|
||||
go [] c acc = Right (reverse acc, c)
|
||||
go ((name, col) : rest) c acc = do
|
||||
(v, c') <- pick name col c
|
||||
v' <- coerceValue (columnType col) v
|
||||
go rest c' ((name, v') : acc)
|
||||
-- | 决定这一列取什么值
|
||||
-- \| 决定这一列取什么值
|
||||
pick name col c
|
||||
| Just (autoName, _) <- auto
|
||||
, name == autoName
|
||||
, maybe True (== VNull) (lookup name provided) = Right (VInt c, c + 1)
|
||||
, maybe True (== VNull) (lookup name provided) =
|
||||
Right (VInt c, c + 1)
|
||||
| Just v <- lookup name provided = Right (v, bump name col c v)
|
||||
| Just d <- columnDefault col = Right (d, c)
|
||||
| columnNullable col = Right (VNull, c)
|
||||
| otherwise = Left ("INSERT: column " ++ name ++ " is NOT NULL and has no default")
|
||||
-- | 自增值跟着已有的值往前推
|
||||
-- \| 自增值跟着已有的值往前推
|
||||
bump name _ c v
|
||||
| Just (autoName, _) <- auto
|
||||
, name == autoName = max (c + 1) (valueInt v + 1)
|
||||
, name == autoName =
|
||||
max (c + 1) (valueInt v + 1)
|
||||
| otherwise = c
|
||||
|
||||
-- | 取自增值,不是整数就当 0
|
||||
@@ -235,9 +249,9 @@ type CheckPlan = [(String, Expr)]
|
||||
checkPlan :: Table -> Either String CheckPlan
|
||||
checkPlan table = mapM parse (mapMaybe checked (tableCols table))
|
||||
where
|
||||
-- | 只挑出带 CHECK 文本的列
|
||||
-- \| 只挑出带 CHECK 文本的列
|
||||
checked (name, col) = (,) name <$> columnCheck col
|
||||
-- | 解析一列的 CHECK 文本
|
||||
-- \| 解析一列的 CHECK 文本
|
||||
parse (name, text) = case parseExpression text of
|
||||
Left err -> Left ("CHECK on " ++ name ++ ": " ++ firstLine err)
|
||||
Right e -> do
|
||||
@@ -248,7 +262,7 @@ checkPlan table = mapM parse (mapMaybe checked (tableCols table))
|
||||
runChecks :: CheckPlan -> Row -> Either String ()
|
||||
runChecks plan row = mapM_ check plan
|
||||
where
|
||||
-- | 跑一条 CHECK
|
||||
-- \| 跑一条 CHECK
|
||||
check (name, e) = case evalCondForRow e row of
|
||||
Left err -> Left ("CHECK on " ++ name ++ ": " ++ err)
|
||||
Right True -> Right ()
|
||||
@@ -262,7 +276,7 @@ firstLine = takeWhile (/= '\n')
|
||||
enforceNotNull :: [(String, Column)] -> [Row] -> Either String ()
|
||||
enforceNotNull cols rows = mapM_ check cols
|
||||
where
|
||||
-- | 查这一列的 NOT NULL
|
||||
-- \| 查这一列的 NOT NULL
|
||||
check (name, col)
|
||||
| columnNullable col = Right ()
|
||||
| any (\r -> lookup name r == Just VNull) rows = Left ("NOT NULL: column " ++ name ++ " cannot be null")
|
||||
@@ -273,11 +287,11 @@ enforceUnique :: [(String, Column)] -> [Row] -> Either String ()
|
||||
enforceUnique cols rows = mapM_ check targets
|
||||
where
|
||||
targets = [(n, c) | (n, c) <- cols, columnUnique c || columnPrimaryKey c]
|
||||
-- | 查这一列有没有重复值
|
||||
-- \| 查这一列有没有重复值
|
||||
check (name, _) = go name HS.empty (mapMaybe (lookup name) rows)
|
||||
-- | 逐个值比过去
|
||||
-- \| 逐个值比过去
|
||||
go _ _ [] = Right ()
|
||||
-- | NULL 不算冲突,跳过
|
||||
-- \| NULL 不算冲突,跳过
|
||||
go name seen (VNull : rest) = go name seen rest
|
||||
go name seen (v : rest)
|
||||
| HS.member (canonicalValue v) seen = Left ("UNIQUE: duplicate value in column " ++ name)
|
||||
@@ -317,7 +331,7 @@ setColumn col v = map (\(k, val) -> if k == col then (k, v) else (k, val))
|
||||
splitByConditionM :: (MonadStorage m) => Database -> Maybe Expr -> [Row] -> m (Either String ([Row], [Row]))
|
||||
splitByConditionM db cond = go [] []
|
||||
where
|
||||
-- | 逐行分流
|
||||
-- \| 逐行分流
|
||||
go doomed kept [] = pure (Right (reverse doomed, reverse kept))
|
||||
go doomed kept (r : rs) = do
|
||||
hit <- case cond of
|
||||
@@ -346,9 +360,9 @@ alterTable db tbl change = case lookupTable db tbl of
|
||||
result <- replaceSchema tbl cols newRows
|
||||
pure (result >> Right [])
|
||||
where
|
||||
-- | 改完再统一查一遍约束
|
||||
-- \| 改完再统一查一遍约束
|
||||
checkAltered (cols, rows) = do
|
||||
let table = Table tbl cols rows Nothing
|
||||
let table = Table tbl cols rows Nothing Nothing Nothing
|
||||
plan <- checkPlan table
|
||||
mapM_ (runChecks plan) rows
|
||||
enforceNotNull cols rows
|
||||
@@ -387,7 +401,7 @@ retypeColumn col ty cols rows
|
||||
rows' <- mapM convert rows
|
||||
Right (map (\entry@(n, c) -> if n == col then (n, c{columnType = ty}) else entry) cols, rows')
|
||||
where
|
||||
-- | 把这一列的老值收进新类型
|
||||
-- \| 把这一列的老值收进新类型
|
||||
convert row = case lookup col row of
|
||||
Nothing -> Right row
|
||||
Just v -> do
|
||||
|
||||
@@ -5,7 +5,8 @@ import ChuSQL.Core.Engine.Algebra.Expr (evalCondForRow, evalExpr, hasSubquery, q
|
||||
import ChuSQL.Core.Engine.Algebra.Optimize (optimize)
|
||||
import ChuSQL.Core.Engine.Algebra.Op
|
||||
import ChuSQL.Core.Engine.Algebra.Planner (translate)
|
||||
import ChuSQL.Core.Engine.Algebra.Sort (sortRows)
|
||||
import ChuSQL.Core.Engine.Algebra.Sort (sortRows, sortBoundRows)
|
||||
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
|
||||
import ChuSQL.Core.Engine.Builtin (aggregateArg, builtinOfExpr, invokeBuiltin, unsupportedAggregate)
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Storage (IndexResult (..), MonadStorage (..))
|
||||
@@ -14,6 +15,7 @@ import ChuSQL.Core.Engine.Syntax.AST (Expr (..), JoinKind (..), Subquery (..))
|
||||
import Control.Monad (filterM)
|
||||
import Data.List (nub)
|
||||
import qualified Data.HashMap.Strict as HM
|
||||
import qualified Data.HashSet as HS
|
||||
|
||||
-- 执行:按算子树算出结果行,外层行供相关子查询引用。
|
||||
|
||||
@@ -79,11 +81,30 @@ data EquiKeys = EquiKeys
|
||||
|
||||
-- | 认出 a = b 这种两列等值条件,并且能分清谁在左谁在右
|
||||
equiKeys :: [String] -> [String] -> Expr -> Maybe EquiKeys
|
||||
equiKeys lcols rcols (Eq (Col a) (Col b))
|
||||
| a /= b, a `elem` lcols, b `elem` rcols = Just (EquiKeys a b)
|
||||
| a /= b, b `elem` lcols, a `elem` rcols = Just (EquiKeys b a)
|
||||
equiKeys lcols rcols (Eq left right)
|
||||
| Just a <- columnName left, Just b <- columnName right
|
||||
, a /= b, a `elem` lcols, b `elem` rcols = Just (EquiKeys a b)
|
||||
| Just a <- columnName left, Just b <- columnName right
|
||||
, a /= b, b `elem` lcols, a `elem` rcols = Just (EquiKeys b a)
|
||||
equiKeys _ _ _ = Nothing
|
||||
|
||||
-- | 提取执行列的诊断名称
|
||||
columnName :: Expr -> Maybe String
|
||||
columnName (Col name) = Just name
|
||||
columnName (ColumnSlot _ name) = Just name
|
||||
columnName _ = Nothing
|
||||
|
||||
-- | 在存储边界按输入布局排列行
|
||||
orderedRows :: [String] -> [Row] -> Either String [Row]
|
||||
orderedRows columns = mapM order
|
||||
where
|
||||
-- | 按列布局排列一行
|
||||
order row = mapM (column row) columns
|
||||
-- | 缺少物理列时明确报错
|
||||
column row name = case lookup name row of
|
||||
Just value -> Right (name, value)
|
||||
Nothing -> Left ("missing bound column: " ++ name)
|
||||
|
||||
-- | 连接两批行:等值键走哈希,否则嵌套循环
|
||||
joinRowsIn ::
|
||||
(MonadStorage m) =>
|
||||
@@ -188,8 +209,9 @@ probeIndexJoin db outer kind mAlias t ks rcols cond l = case lookup (ekLeft ks)
|
||||
Left e -> pure (Left e)
|
||||
Right NoIndex -> pure (Right Nothing)
|
||||
Right (IndexRows rows) -> do
|
||||
pairs <- joinRowsIn db outer kind rcols cond [l] (prefixRows mAlias rows)
|
||||
pure (fmap Just pairs)
|
||||
case orderedRows rcols (prefixRows mAlias rows) of
|
||||
Left err -> pure (Left err)
|
||||
Right ordered -> fmap (fmap Just) (joinRowsIn db outer kind rcols cond [l] ordered)
|
||||
|
||||
-- | 列与键都不是浮点时才敢用索引
|
||||
indexSafe :: Database -> String -> String -> Value -> Bool
|
||||
@@ -213,20 +235,17 @@ floatValue (VFloat _) = True
|
||||
floatValue _ = False
|
||||
|
||||
-- | 分组键的值
|
||||
groupKey :: [String] -> Row -> Either String [Value]
|
||||
groupKey keys row = mapM get keys
|
||||
where
|
||||
-- | 取这一列的分组值
|
||||
get c = maybe (Left ("unknown column: " ++ c)) Right (lookup c row)
|
||||
groupKey :: Row -> [Expr] -> Row -> Either String [Value]
|
||||
groupKey outer keys row = mapM (\key -> evalExpr key (row ++ outer)) keys
|
||||
|
||||
-- | 按分组键把行分组,保留分组首次出现的顺序
|
||||
groupRows :: [String] -> [Row] -> Either String [([Value], [Row])]
|
||||
groupRows keys rows = go rows (HM.empty, [])
|
||||
groupRows :: Row -> [Expr] -> [Row] -> Either String [([Value], [Row])]
|
||||
groupRows outer keys rows = go rows (HM.empty, [])
|
||||
where
|
||||
-- | 逐行塞进对应分组,并记住分组首次出现的顺序
|
||||
go [] (acc, order) = Right [(k, reverse (HM.lookupDefault [] k acc)) | k <- reverse order]
|
||||
go (r : rs) (acc, order) = do
|
||||
k <- groupKey keys r
|
||||
k <- groupKey outer keys r
|
||||
let order' = if HM.member k acc then order else k : order
|
||||
go rs (HM.insertWith (++) k [r] acc, order')
|
||||
|
||||
@@ -249,11 +268,11 @@ oneGroup outer keys aggs (keyValues, rows) = do
|
||||
Right (zip keys keyValues ++ zip (map fst aggs) values)
|
||||
|
||||
-- | 分组求值:没有分组列时,即使没有行也要出一行
|
||||
evalAggregate :: Row -> [String] -> [(String, Expr)] -> [Row] -> Either String [Row]
|
||||
evalAggregate :: Row -> [(String, Expr)] -> [(String, Expr)] -> [Row] -> Either String [Row]
|
||||
evalAggregate outer keys aggs rows = do
|
||||
groups <- groupRows keys rows
|
||||
groups <- groupRows outer (map snd keys) rows
|
||||
let top = if null keys && null groups then [([], [])] else groups
|
||||
mapM (oneGroup outer keys aggs) top
|
||||
mapM (oneGroup outer (map fst keys) aggs) top
|
||||
|
||||
-- | 绑定键:子查询引用的外层列在这一行上的取值
|
||||
bindingKey :: [String] -> Row -> String
|
||||
@@ -443,7 +462,9 @@ runSubqueryRows db sq outer = case translate (subqueryStatement sq) of
|
||||
Left e -> pure (Left e)
|
||||
Right op -> case mapM bind (subqueryRefs sq) of
|
||||
Left err -> pure (Left err)
|
||||
Right bindings -> evalRelOpIn db (bindings ++ outer) (optimize db op)
|
||||
Right bindings -> case bindRelOpColumns db (map fst (bindings ++ outer)) (optimize db op) of
|
||||
Left err -> pure (Left err)
|
||||
Right bound -> evalRelOpIn db (bindings ++ outer) bound
|
||||
where
|
||||
-- | 给外层物理列补回已解析的限定名
|
||||
bind column = case bindingValue column outer of
|
||||
@@ -481,23 +502,26 @@ evalRelOpIn db outer (Compute items op) = do
|
||||
es <- boundExprs refs (r ++ outer) table
|
||||
mapM (\(label, e) -> (,) label <$> evalExpr e (r ++ outer)) (zip (map fst items) es)
|
||||
evalRelOpIn db outer (Aggregate keys aggs op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (rows >>= evalAggregate outer [(name, Col name) | name <- keys] aggs)
|
||||
evalRelOpIn db outer (BoundAggregate keys aggs op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (rows >>= evalAggregate outer keys aggs)
|
||||
evalRelOpIn db _ (Scan mAlias tbl cols) = do
|
||||
res <- scanTable db tbl cols
|
||||
pure (fmap (prefixRows mAlias) res)
|
||||
pure (res >>= orderedRows (relOpCols db (Scan mAlias tbl cols)) . prefixRows mAlias)
|
||||
evalRelOpIn db outer (Lookup mAlias tbl col k) = do
|
||||
res <- lookupByColumn tbl col k
|
||||
case res of
|
||||
Left e -> pure (Left e)
|
||||
Right NoIndex -> scanFiltered db outer mAlias tbl (lookupCondition mAlias col k)
|
||||
Right (IndexRows rows) -> pure (Right (prefixRows mAlias rows))
|
||||
Right (IndexRows rows) -> pure (orderedRows (relOpCols db (Lookup mAlias tbl col k)) (prefixRows mAlias rows))
|
||||
evalRelOpIn db outer (Range mAlias tbl col lo hi) = do
|
||||
res <- scanRange tbl col lo hi
|
||||
case res of
|
||||
Left e -> pure (Left e)
|
||||
Right Nothing -> scanFiltered db outer mAlias tbl (rangeCondition mAlias col lo hi)
|
||||
Right (Just rows) -> pure (Right (prefixRows mAlias rows))
|
||||
Right (Just rows) -> pure (orderedRows (relOpCols db (Range mAlias tbl col lo hi)) (prefixRows mAlias rows))
|
||||
evalRelOpIn db outer (Filter e op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
case rows of
|
||||
@@ -516,9 +540,15 @@ evalRelOpIn db outer (Filter e op) = do
|
||||
evalRelOpIn db outer (Project cols op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (fmap (map (project cols)) rows)
|
||||
evalRelOpIn db outer (Distinct op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (distinctRows <$> rows)
|
||||
evalRelOpIn db outer (Sort spec op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (fmap (sortRows spec) rows)
|
||||
evalRelOpIn db outer (BoundSort spec op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (rows >>= sortBoundRows spec outer)
|
||||
evalRelOpIn db outer (Limit n op) = do
|
||||
rows <- evalRelOpIn db outer op
|
||||
pure (fmap (take n) rows)
|
||||
@@ -557,8 +587,8 @@ scanFiltered ::
|
||||
scanFiltered db outer mAlias tbl cond = do
|
||||
res <- scanTable db tbl Nothing
|
||||
pure $ do
|
||||
rows <- res
|
||||
filterM (evalCondForRow cond . (++ outer)) (prefixRows mAlias rows)
|
||||
rows <- res >>= orderedRows (relOpCols db (Scan mAlias tbl Nothing)) . prefixRows mAlias
|
||||
filterM (evalCondForRow cond . (++ outer)) rows
|
||||
|
||||
-- | 行数到阈值才值得并行分片
|
||||
parallelRowThreshold :: Int
|
||||
@@ -574,3 +604,15 @@ scanTable db tbl cols = do
|
||||
where
|
||||
-- | 统计里的行数,没有统计算 0
|
||||
rows = maybe 0 metaRowCount (tableMetaOf db tbl)
|
||||
|
||||
-- | 按输出值去重并保留首次行序
|
||||
distinctRows :: [Row] -> [Row]
|
||||
distinctRows = go HS.empty
|
||||
where
|
||||
-- | 只输出尚未遇到的值组合
|
||||
go _ [] = []
|
||||
go seen (row : rows)
|
||||
| HS.member key seen = go seen rows
|
||||
| otherwise = row : go (HS.insert key seen) rows
|
||||
where
|
||||
key = map (canonicalValue . snd) row
|
||||
@@ -11,6 +11,13 @@ import ChuSQL.Core.Engine.Runtime.SQL (invokeSQLFunction, constructSQLValue)
|
||||
evalExpr :: Expr -> Row -> Either String Value
|
||||
evalExpr (Col name) row =
|
||||
maybe (Left ("unknown column: " ++ name)) Right (lookup name row)
|
||||
evalExpr (ColumnSlot slot name) row = case drop slot row of
|
||||
(key, value) : _ | slot >= 0, key == name -> Right value
|
||||
[] -> Left ("missing bound column: " ++ name)
|
||||
_ -> Left ("bound column position mismatch: " ++ name)
|
||||
evalExpr (OuterColumnSlot localWidth slot name) row
|
||||
| localWidth < 0 || slot < 0 = Left ("bound column position mismatch: " ++ name)
|
||||
| otherwise = evalExpr (ColumnSlot slot name) (drop localWidth row)
|
||||
evalExpr (FunctionCall name _) _ = Left ("unbound function call: " ++ name)
|
||||
evalExpr (Construct _ _) _ = Left "unbound runtime constructor"
|
||||
evalExpr (BoundConstruct tid arguments) row = mapM (\argument -> evalExpr argument row) arguments >>= constructSQLValue tid
|
||||
@@ -70,6 +77,8 @@ quantifiedCompare op lhs vs q = case vs of
|
||||
-- | 表达式用到了哪些列
|
||||
colsInExpr :: Expr -> [String]
|
||||
colsInExpr (Col c) = [c]
|
||||
colsInExpr (ColumnSlot _ c) = [c]
|
||||
colsInExpr (OuterColumnSlot _ _ c) = [c]
|
||||
colsInExpr (Construct _ arguments) = concatMap colsInExpr arguments
|
||||
colsInExpr (BoundConstruct _ arguments) = concatMap colsInExpr arguments
|
||||
colsInExpr (FunctionCall _ arguments) = concatMap colsInExpr arguments
|
||||
@@ -247,6 +256,8 @@ bareColumns e = case e of
|
||||
FunctionCall _ arguments -> concatMap bareColumns arguments
|
||||
BoundFunction _ _ arguments -> concatMap bareColumns arguments
|
||||
Col c -> [c]
|
||||
ColumnSlot _ c -> [c]
|
||||
OuterColumnSlot _ _ c -> [c]
|
||||
Add a b -> both a b
|
||||
Sub a b -> both a b
|
||||
Mul a b -> both a b
|
||||
|
||||
@@ -15,6 +15,7 @@ reorderJoins db = pass db False
|
||||
|
||||
-- | 往下带“行序是否不可观察”,可观察时不动内连接
|
||||
pass :: Database -> Bool -> RelOp -> RelOp
|
||||
pass db free (Distinct x) = Distinct (pass db free x)
|
||||
pass db _ (Sort spec x) = Sort spec (pass db True x)
|
||||
pass db _ (Limit n x) = Limit n (pass db False x)
|
||||
pass db _ (Aggregate keys aggs x) = Aggregate keys aggs (pass db False x)
|
||||
|
||||
@@ -13,8 +13,11 @@ data RelOp
|
||||
| Project [String] RelOp
|
||||
| Compute [(String, Expr)] RelOp
|
||||
| Aggregate [String] [(String, Expr)] RelOp
|
||||
| BoundAggregate [(String, Expr)] [(String, Expr)] RelOp
|
||||
| Distinct RelOp
|
||||
| Unit
|
||||
| Sort [(String, SortDir)] RelOp
|
||||
| BoundSort [(Expr, SortDir)] RelOp
|
||||
| Limit Int RelOp
|
||||
| Join JoinKind RelOp RelOp Expr
|
||||
| Derived (Maybe String) RelOp
|
||||
@@ -25,6 +28,7 @@ relOpCols :: Database -> RelOp -> [String]
|
||||
relOpCols _ Unit = []
|
||||
relOpCols _ (Compute items _) = map fst items
|
||||
relOpCols _ (Aggregate keys aggs _) = keys ++ map fst aggs
|
||||
relOpCols _ (BoundAggregate keys aggs _) = map fst keys ++ map fst aggs
|
||||
relOpCols _ (Scan mAlias _ (Just cols)) = map (qualify mAlias) cols
|
||||
relOpCols db (Scan mAlias tbl Nothing) =
|
||||
case lookup tbl db of
|
||||
@@ -37,7 +41,9 @@ relOpCols db (Range mAlias tbl _ _ _) = relOpCols db (Scan mAlias tbl Nothing)
|
||||
relOpCols db (Filter _ x) = relOpCols db x
|
||||
relOpCols _ (Project ["*"] _) = ["*"]
|
||||
relOpCols _ (Project cols _) = cols
|
||||
relOpCols db (Distinct x) = relOpCols db x
|
||||
relOpCols db (Sort _ x) = relOpCols db x
|
||||
relOpCols db (BoundSort _ x) = relOpCols db x
|
||||
relOpCols db (Limit _ x) = relOpCols db x
|
||||
relOpCols db (Join _ l r _) = relOpCols db l ++ relOpCols db r
|
||||
relOpCols db (Derived mAlias x) = map (qualify mAlias) (relOpCols db x)
|
||||
@@ -67,8 +73,12 @@ renderPlan = go 0
|
||||
++ show [label ++ " <- " ++ show e | (label, e) <- aggs]
|
||||
++ "\n"
|
||||
++ go (ind + 1) x
|
||||
Distinct x -> pad ++ "Distinct\n" ++ go (ind + 1) x
|
||||
BoundAggregate keys aggs x -> pad ++ "BoundAggregate keys=" ++ show keys
|
||||
++ " aggs=" ++ show aggs ++ "\n" ++ go (ind + 1) x
|
||||
Unit -> pad ++ "Unit"
|
||||
Sort s x -> pad ++ "Sort " ++ show s ++ "\n" ++ go (ind + 1) x
|
||||
BoundSort s x -> pad ++ "BoundSort " ++ show s ++ "\n" ++ go (ind + 1) x
|
||||
Limit n x -> pad ++ "Limit " ++ show n ++ "\n" ++ go (ind + 1) x
|
||||
Join k l r c -> pad ++ kindLabel k ++ " on " ++ show c ++ "\n" ++ go (ind + 1) l ++ "\n" ++ go (ind + 1) r
|
||||
Derived a x -> pad ++ "Derived " ++ show a ++ "\n" ++ go (ind + 1) x
|
||||
|
||||
@@ -77,6 +77,7 @@ planHasDivision (Filter p x) = hasDivision p || planHasDivision x
|
||||
planHasDivision (Join _ l r c) = hasDivision c || planHasDivision l || planHasDivision r
|
||||
planHasDivision (Compute items x) = any (hasDivision . snd) items || planHasDivision x
|
||||
planHasDivision (Project _ x) = planHasDivision x
|
||||
planHasDivision (Distinct x) = planHasDivision x
|
||||
planHasDivision (Sort _ x) = planHasDivision x
|
||||
planHasDivision (Limit _ x) = planHasDivision x
|
||||
planHasDivision (Derived _ x) = planHasDivision x
|
||||
@@ -96,6 +97,8 @@ pushProjectLeaf db need leaf
|
||||
-- | 把需要哪些列往下传
|
||||
pushProject :: Database -> [String] -> RelOp -> RelOp
|
||||
pushProject _ _ Unit = Unit
|
||||
pushProject _ _ op@BoundSort{} = op
|
||||
pushProject _ _ op@BoundAggregate{} = op
|
||||
pushProject db _ (Compute items x) = Compute items (pushProject db (concatMap (colsInExpr . snd) items) x)
|
||||
pushProject db _ (Aggregate keys aggs x) =
|
||||
Aggregate keys aggs (pushProject db (keys ++ concatMap (colsInExpr . snd) aggs) x)
|
||||
@@ -103,6 +106,7 @@ pushProject db _ (Project cols x) = case pushProject db cols x of
|
||||
Project cols' y | cols' == cols -> Project cols y
|
||||
other -> Project cols other
|
||||
pushProject db need (Filter p x) = Filter p (pushProject db (need ++ colsInExpr p) x)
|
||||
pushProject db _ (Distinct x) = Distinct (pushProject db (relOpCols db x) x)
|
||||
pushProject db need (Sort spec x) = Sort spec (pushProject db (need ++ map fst spec) x)
|
||||
pushProject db need (Limit n x) = Limit n (pushProject db need x)
|
||||
pushProject db need (Join kind l r c)
|
||||
@@ -250,6 +254,7 @@ optimizeRelOp db = rewriteNode db . descend
|
||||
descend (Project cols x) = Project cols (optimizeRelOp db x)
|
||||
descend (Compute items x) = Compute items (optimizeRelOp db x)
|
||||
descend (Aggregate keys aggs x) = Aggregate keys aggs (optimizeRelOp db x)
|
||||
descend (Distinct x) = Distinct (optimizeRelOp db x)
|
||||
descend (Sort spec x) = Sort spec (optimizeRelOp db x)
|
||||
descend (Limit n x) = Limit n (optimizeRelOp db x)
|
||||
descend (Join kind l r c) = Join kind (optimizeRelOp db l) (optimizeRelOp db r) c
|
||||
|
||||
@@ -82,6 +82,13 @@ limited orderBy mLimit op = case mLimit of
|
||||
-- | 翻成:过滤 -> 分组 -> 排序 -> 截断 -> 取列
|
||||
translate :: Statement -> Either String RelOp
|
||||
translate q = case q of
|
||||
SelectDistinct inner -> do
|
||||
ordering <- distinctOrdering inner
|
||||
plan <- case inner of
|
||||
s@Select{} -> translate s { selectOrderBy = [], selectLimit = Nothing }
|
||||
s@SelectExpr{} -> translate s { selectOrderBy = [], selectLimit = Nothing }
|
||||
_ -> Left "DISTINCT requires SELECT"
|
||||
Right (limited ordering (selectLimit inner) (Distinct plan))
|
||||
Select
|
||||
{ selectCols = cols
|
||||
, selectFrom = fromC
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
module ChuSQL.Core.Engine.Algebra.Sort (sortRows, compareValue) where
|
||||
module ChuSQL.Core.Engine.Algebra.Sort (sortRows, sortBoundRows, compareValue) where
|
||||
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Engine.Algebra.Expr (evalExpr)
|
||||
import Data.List (sortBy)
|
||||
|
||||
-- 排序:按多列规格排出结果行,每列可升可降。
|
||||
@@ -17,9 +18,9 @@ sortKey :: [(String, SortDir)] -> Row -> [Maybe Value]
|
||||
sortKey spec row = [lookup col row | (col, _) <- spec]
|
||||
|
||||
-- | 比较两个排序键
|
||||
compareKeys :: [(String, SortDir)] -> [Maybe Value] -> [Maybe Value] -> Ordering
|
||||
compareKeys :: [SortDir] -> [Maybe Value] -> [Maybe Value] -> Ordering
|
||||
compareKeys [] _ _ = EQ
|
||||
compareKeys ((_, dir) : rest) (a : as) (b : bs) =
|
||||
compareKeys (dir : rest) (a : as) (b : bs) =
|
||||
case (a, b) of
|
||||
(Just v1, Just v2) ->
|
||||
case compareValue v1 v2 of
|
||||
@@ -36,4 +37,19 @@ sortRows spec rows = map snd (sortBy cmp (map decorate rows))
|
||||
-- | 取键并带上原行
|
||||
decorate row = (sortKey spec row, row)
|
||||
-- | 比较两个带键的行
|
||||
cmp (k1, _) (k2, _) = compareKeys spec k1 k2
|
||||
cmp (k1, _) (k2, _) = compareKeys (map snd spec) k1 k2
|
||||
|
||||
-- | 用绑定槽位取键并稳定排序
|
||||
sortBoundRows :: [(Expr, SortDir)] -> Row -> [Row] -> Either String [Row]
|
||||
sortBoundRows [] _ rows = Right rows
|
||||
sortBoundRows spec outer rows = do
|
||||
decorated <- mapM decorate rows
|
||||
Right (map snd (sortBy compareRows decorated))
|
||||
where
|
||||
directions = map snd spec
|
||||
-- | 每行只计算一次绑定排序键
|
||||
decorate row = do
|
||||
keys <- mapM (\(expression, _) -> evalExpr expression (row ++ outer)) spec
|
||||
Right (map Just keys, row)
|
||||
-- | 比较排序键并保留同键行序
|
||||
compareRows (left, _) (right, _) = compareKeys directions left right
|
||||
@@ -1,25 +1,25 @@
|
||||
module ChuSQL.Core.Engine.Builtin
|
||||
( Builtin (..)
|
||||
, builtinNames
|
||||
, resolveBuiltin
|
||||
, builtinName
|
||||
, invokeBuiltin
|
||||
, builtinOfExpr
|
||||
, builtinNode
|
||||
, aggregateArg
|
||||
, unsupportedAggregate
|
||||
, Operator (..)
|
||||
, operatorOfExpr
|
||||
, operatorSymbol
|
||||
, operatorOfCompare
|
||||
, compareNode
|
||||
, executeOperator
|
||||
, threeValuedNot
|
||||
, inValues
|
||||
) where
|
||||
module ChuSQL.Core.Engine.Builtin (
|
||||
Builtin (..),
|
||||
builtinNames,
|
||||
resolveBuiltin,
|
||||
builtinName,
|
||||
invokeBuiltin,
|
||||
builtinOfExpr,
|
||||
builtinNode,
|
||||
aggregateArg,
|
||||
unsupportedAggregate,
|
||||
Operator (..),
|
||||
operatorOfExpr,
|
||||
operatorSymbol,
|
||||
operatorOfCompare,
|
||||
compareNode,
|
||||
executeOperator,
|
||||
threeValuedNot,
|
||||
inValues,
|
||||
) where
|
||||
|
||||
import ChuSQL.Core.Model (Value (..), compareValue)
|
||||
import ChuSQL.Core.Engine.Syntax.AST (CompareOp (..), Expr (..))
|
||||
import ChuSQL.Core.Model (Value (..), compareValue)
|
||||
import Control.Monad (foldM)
|
||||
import Data.Char (toLower)
|
||||
|
||||
@@ -246,12 +246,12 @@ evalArith ops x y = case (asDouble x, asDouble y) of
|
||||
divArith :: Arith
|
||||
divArith = Arith intDiv floatDiv
|
||||
where
|
||||
-- | 整数除法:除零与溢出都报错
|
||||
-- \| 整数除法:除零与溢出都报错
|
||||
intDiv _ 0 = Left "division by zero"
|
||||
intDiv x y
|
||||
| x == minBound && y == -1 = Left "integer division overflow"
|
||||
| otherwise = Right (x `quot` y)
|
||||
-- | 浮点除法:不能除以零
|
||||
-- \| 浮点除法:不能除以零
|
||||
floatDiv _ 0 = Left "division by zero"
|
||||
floatDiv x y = Right (x / y)
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
module ChuSQL.Core.Engine.Error
|
||||
( ErrorCategory (..)
|
||||
, ErrorRule (..)
|
||||
, errorRules
|
||||
, errorCategory
|
||||
, errorCode
|
||||
, errorLike
|
||||
, accountErrorCode
|
||||
) where
|
||||
module ChuSQL.Core.Engine.Error (
|
||||
ErrorCategory (..),
|
||||
ErrorRule (..),
|
||||
errorRules,
|
||||
errorCategory,
|
||||
errorCode,
|
||||
errorLike,
|
||||
accountErrorCode,
|
||||
) where
|
||||
|
||||
import Data.List (isInfixOf)
|
||||
|
||||
@@ -35,6 +35,8 @@ errorRules :: [ErrorRule]
|
||||
errorRules =
|
||||
[ ErrorRule "no database selected" NoDatabaseError "no_database"
|
||||
, ErrorRule "administrator required" PermissionError "forbidden"
|
||||
, ErrorRule "bound query object changed" CatalogError "serialization_failure"
|
||||
, ErrorRule "bound query schema changed" CatalogError "serialization_failure"
|
||||
, ErrorRule "unknown table" CatalogError "not_found"
|
||||
, ErrorRule "unknown domain" CatalogError "not_found"
|
||||
, ErrorRule "domain is still used" CatalogError "query_error"
|
||||
|
||||
@@ -14,23 +14,27 @@ data Declaration = Declaration
|
||||
, dependencies :: [Dependency]
|
||||
, dependenciesComplete :: Bool
|
||||
, atomicAttributes :: Maybe Bool
|
||||
} deriving (Eq, Show)
|
||||
}
|
||||
deriving (Eq, Show)
|
||||
data Finding = Finding
|
||||
{ normalForm :: NormalForm
|
||||
, verdict :: Verdict
|
||||
, witnesses :: [Dependency]
|
||||
, condition :: String
|
||||
} deriving (Eq, Show)
|
||||
}
|
||||
deriving (Eq, Show)
|
||||
data Decomposition = Decomposition
|
||||
{ relations :: [Attributes]
|
||||
, losslessJoin :: Verdict
|
||||
, dependencyPreserving :: Verdict
|
||||
} deriving (Eq, Show)
|
||||
}
|
||||
deriving (Eq, Show)
|
||||
data Analysis = Analysis
|
||||
{ candidateKeys :: [Attributes]
|
||||
, findings :: [Finding]
|
||||
, suggestions :: [Decomposition]
|
||||
} deriving (Eq, Show)
|
||||
}
|
||||
deriving (Eq, Show)
|
||||
|
||||
-- 规范化属性集合。
|
||||
canonical :: Attributes -> Attributes
|
||||
@@ -45,8 +49,9 @@ attributeClosure :: [Dependency] -> Attributes -> Attributes
|
||||
attributeClosure f seed = fixed (canonical seed)
|
||||
where
|
||||
-- 迭代加入已蕴含的属性。
|
||||
fixed xs = let ys = canonical (xs ++ concat [b | Dependency a b <- f, subset a xs])
|
||||
in if xs == ys then xs else fixed ys
|
||||
fixed xs =
|
||||
let ys = canonical (xs ++ concat [b | Dependency a b <- f, subset a xs])
|
||||
in if xs == ys then xs else fixed ys
|
||||
|
||||
-- 判断声明是否蕴含依赖。
|
||||
implies :: [Dependency] -> Dependency -> Bool
|
||||
@@ -54,8 +59,12 @@ implies f (Dependency a b) = subset b (attributeClosure f a)
|
||||
|
||||
-- 枚举最小超键。
|
||||
keysOf :: Attributes -> [Dependency] -> [Attributes]
|
||||
keysOf attrs f = [x | x <- subsequences attrs, super x,
|
||||
all (not . super) [x \\ [a] | a <- x]]
|
||||
keysOf attrs f =
|
||||
[ x
|
||||
| x <- subsequences attrs
|
||||
, super x
|
||||
, all (not . super) [x \\ [a] | a <- x]
|
||||
]
|
||||
where
|
||||
-- 判断闭包是否覆盖关系。
|
||||
super x = subset attrs (attributeClosure f x)
|
||||
@@ -63,8 +72,12 @@ keysOf attrs f = [x | x <- subsequences attrs, super x,
|
||||
-- 投影关系上的全部非平凡依赖。
|
||||
projectDependencies :: [Dependency] -> Attributes -> [Dependency]
|
||||
projectDependencies f attrs =
|
||||
[Dependency x [a] | x <- subsequences attrs,
|
||||
a <- attributeClosure f x, a `elem` attrs, a `notElem` x]
|
||||
[ Dependency x [a]
|
||||
| x <- subsequences attrs
|
||||
, a <- attributeClosure f x
|
||||
, a `elem` attrs
|
||||
, a `notElem` x
|
||||
]
|
||||
|
||||
-- 验证声明并生成只读范式报告。
|
||||
analyzeNormalization :: [(String, Column)] -> Declaration -> Either String Analysis
|
||||
@@ -90,14 +103,26 @@ analyzeNormalization schema declaration
|
||||
Just False -> Violated
|
||||
Nothing -> Insufficient
|
||||
allFDs = projectDependencies f attrs
|
||||
partial = [d | d@(Dependency x [a]) <- allFDs, a `notElem` prime,
|
||||
any (\k -> x /= k && subset x k) keys]
|
||||
third = [d | d@(Dependency x [a]) <- allFDs, a `notElem` prime,
|
||||
not (subset attrs (attributeClosure f x))]
|
||||
boyce = [d | d@(Dependency x _) <- allFDs,
|
||||
not (subset attrs (attributeClosure f x))]
|
||||
results = Finding NF1 atomic [] "attributes must have atomic domains; requires an explicit declaration" :
|
||||
[classify NF2 partial, classify NF3 third, classify BCNF boyce]
|
||||
partial =
|
||||
[ d
|
||||
| d@(Dependency x [a]) <- allFDs
|
||||
, a `notElem` prime
|
||||
, any (\k -> x /= k && subset x k) keys
|
||||
]
|
||||
third =
|
||||
[ d
|
||||
| d@(Dependency x [a]) <- allFDs
|
||||
, a `notElem` prime
|
||||
, not (subset attrs (attributeClosure f x))
|
||||
]
|
||||
boyce =
|
||||
[ d
|
||||
| d@(Dependency x _) <- allFDs
|
||||
, not (subset attrs (attributeClosure f x))
|
||||
]
|
||||
results =
|
||||
Finding NF1 atomic [] "attributes must have atomic domains; requires an explicit declaration"
|
||||
: [classify NF2 partial, classify NF3 third, classify BCNF boyce]
|
||||
decompositions = [split d | d <- take 1 boyce, complete, atomic == Satisfied]
|
||||
-- 校验字段集合。
|
||||
valid xs = length xs == length (canonical xs) && subset xs attrs
|
||||
@@ -108,10 +133,11 @@ analyzeNormalization schema declaration
|
||||
-- 结合声明完整性判定范式。
|
||||
classify nf ds = Finding nf status (if complete then ds else []) (rule nf)
|
||||
where
|
||||
status | atomic == Violated = Violated
|
||||
| atomic == Insufficient || not complete = Insufficient
|
||||
| null ds = Satisfied
|
||||
| otherwise = Violated
|
||||
status
|
||||
| atomic == Violated = Violated
|
||||
| atomic == Insufficient || not complete = Insufficient
|
||||
| null ds = Satisfied
|
||||
| otherwise = Violated
|
||||
-- 说明各范式的必要条件。
|
||||
rule NF1 = "attributes must have atomic domains"
|
||||
rule NF2 = "no nonprime attribute depends on a proper subset of any candidate key"
|
||||
|
||||
@@ -22,6 +22,7 @@ parallelShardLimit = 8
|
||||
newtype Pool = Pool (Chan (IO ()))
|
||||
|
||||
{-# NOINLINE poolRef #-}
|
||||
|
||||
-- | 进程全局的池,按容量复用
|
||||
poolRef :: MVar (Maybe (Int, Pool))
|
||||
poolRef = unsafePerformIO (newMVar Nothing)
|
||||
@@ -30,7 +31,7 @@ poolRef = unsafePerformIO (newMVar Nothing)
|
||||
workerPool :: Int -> IO Pool
|
||||
workerPool size = modifyMVar poolRef reuse
|
||||
where
|
||||
-- | 复用队列并按需补充工作线程
|
||||
-- \| 复用队列并按需补充工作线程
|
||||
reuse current = case current of
|
||||
Just (capacity, pool@(Pool jobs))
|
||||
| capacity >= wanted -> pure (current, pool)
|
||||
@@ -41,10 +42,10 @@ workerPool size = modifyMVar poolRef reuse
|
||||
pool <- startPool wanted
|
||||
pure (Just (wanted, pool), pool)
|
||||
|
||||
-- | 池的线程数至少一个
|
||||
-- \| 池的线程数至少一个
|
||||
wanted = max 1 size
|
||||
|
||||
-- | 起 wanted 个常驻线程吃队列
|
||||
-- \| 起 wanted 个常驻线程吃队列
|
||||
startPool n = do
|
||||
jobs <- newChan
|
||||
sequence_ [forkIO (worker jobs) | _ <- [1 .. n]]
|
||||
|
||||
@@ -16,7 +16,7 @@ data Expression = Literal TypeExpr RuntimeValue | Reference String | Apply Strin
|
||||
| TupleLiteral [Expression] deriving (Show, Eq)
|
||||
|
||||
-- | 保存绑定后的表达式节点
|
||||
data ExpressionNode = Constant RuntimeValue | BoundReference String | BoundApply FunctionId [TypedExpression]
|
||||
data ExpressionNode = Constant RuntimeValue | BoundReference Int String | BoundApply FunctionId [TypedExpression]
|
||||
| BoundList [TypedExpression] | BoundNothing | BoundJust TypedExpression | BoundTuple [TypedExpression]
|
||||
deriving (Show, Eq)
|
||||
|
||||
@@ -34,8 +34,8 @@ bindExpression types functions env expression = case expression of
|
||||
tid <- descriptorId <$> resolveType types ty
|
||||
validateValue types tid value
|
||||
Right (TypedExpression tid (Constant value))
|
||||
Reference name -> case [tid | (key, tid) <- env, key == name] of
|
||||
[tid] -> describeType types tid >> Right (TypedExpression tid (BoundReference name))
|
||||
Reference name -> case [(slot, tid) | (slot, (key, tid)) <- zip [0 ..] env, key == name] of
|
||||
[(slot, tid)] -> describeType types tid >> Right (TypedExpression tid (BoundReference slot name))
|
||||
[] -> Left ("unknown column: " ++ name)
|
||||
_ -> Left ("ambiguous column: " ++ name)
|
||||
Apply name arguments -> do
|
||||
@@ -68,10 +68,10 @@ evaluateExpression :: TypeRegistry -> FunctionRegistry -> [(String, RuntimeValue
|
||||
evaluateExpression types functions row (TypedExpression tid node) = do
|
||||
value <- case node of
|
||||
Constant v -> Right v
|
||||
BoundReference name -> case [v | (key, v) <- row, key == name] of
|
||||
[v] -> Right v
|
||||
BoundReference slot name -> case drop slot row of
|
||||
(key, value) : _ | key == name -> Right value
|
||||
[] -> Left ("missing bound column: " ++ name)
|
||||
_ -> Left ("duplicate bound column: " ++ name)
|
||||
_ -> Left ("bound column position mismatch: " ++ name)
|
||||
BoundApply fid arguments -> mapM evaluate arguments >>= invokeFunction types functions fid
|
||||
BoundList arguments -> RList <$> mapM evaluate arguments
|
||||
BoundNothing -> Right (RMaybe Nothing)
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
module ChuSQL.Core.Engine.Runtime.Dependencies (queryTables, domainDependencies) where
|
||||
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Model (Database, ColumnType (..), columnType, tableCols, domainTableName, lastTablePart)
|
||||
import Data.List (nub)
|
||||
|
||||
-- 收集查询及嵌套表达式的表依赖,供绑定执行前校验目录。
|
||||
|
||||
-- | 收集全部查询来源表
|
||||
queryTables :: Statement -> [String]
|
||||
queryTables statement = nub (case statement of
|
||||
SelectDistinct child -> queryTables child
|
||||
Select _ source condition _ _ _ -> sourceTables source ++ concatMap expressionTables condition
|
||||
SelectExpr items source condition _ _ _ -> sourceTables source
|
||||
++ concatMap (expressionTables . snd) items ++ concatMap expressionTables condition
|
||||
_ -> [])
|
||||
|
||||
-- | 收集来源列依赖的领域定义
|
||||
domainDependencies :: Database -> [String] -> [String]
|
||||
domainDependencies catalog sources = collect []
|
||||
[domain | (name, table) <- catalog, name `elem` sources
|
||||
, (_, column) <- tableCols table, domain <- typeDomains (columnType column)]
|
||||
where
|
||||
-- | 递归展开定义并防止重复访问
|
||||
collect seen [] = seen
|
||||
collect seen (domain : remaining) =
|
||||
let definitions = [(name, table) | (name, table) <- catalog
|
||||
, lastTablePart name == domainTableName domain, name `notElem` seen]
|
||||
names = map fst definitions
|
||||
nested = [child | (_, table) <- definitions, (_, column) <- tableCols table
|
||||
, child <- typeDomains (columnType column)]
|
||||
in collect (seen ++ names) (remaining ++ nested)
|
||||
-- | 保留嵌套领域的每层依赖
|
||||
typeDomains (CDomain name base) = name : typeDomains base
|
||||
typeDomains _ = []
|
||||
|
||||
-- | 收集来源树与连接条件依赖
|
||||
sourceTables :: FromClause -> [String]
|
||||
sourceTables source = case source of
|
||||
FromTable _ name -> [name]
|
||||
FromSubquery _ statement -> queryTables statement
|
||||
FromJoin _ left right condition -> sourceTables left ++ sourceTables right ++ expressionTables condition
|
||||
FromUnit -> []
|
||||
|
||||
-- | 收集表达式内嵌套子查询依赖
|
||||
expressionTables :: Expr -> [String]
|
||||
expressionTables expression = case expression of
|
||||
ScalarSub query -> queryTables (subqueryStatement query)
|
||||
ExistsSub query _ -> queryTables (subqueryStatement query)
|
||||
InSub value query _ -> expressionTables value ++ queryTables (subqueryStatement query)
|
||||
QuantCmp _ value query _ -> expressionTables value ++ queryTables (subqueryStatement query)
|
||||
_ -> concatMap expressionTables (children expression)
|
||||
|
||||
-- | 列出当前表达式的直接子节点
|
||||
children :: Expr -> [Expr]
|
||||
children expression = case expression of
|
||||
FunctionCall _ arguments -> arguments
|
||||
BoundFunction _ _ arguments -> arguments
|
||||
Construct _ arguments -> arguments
|
||||
BoundConstruct _ arguments -> arguments
|
||||
InList value arguments _ -> value : arguments
|
||||
Add a b -> [a, b]
|
||||
Sub a b -> [a, b]
|
||||
Mul a b -> [a, b]
|
||||
Div a b -> [a, b]
|
||||
Gt a b -> [a, b]
|
||||
Lt a b -> [a, b]
|
||||
Eq a b -> [a, b]
|
||||
GtE a b -> [a, b]
|
||||
LtE a b -> [a, b]
|
||||
NotEq a b -> [a, b]
|
||||
And a b -> [a, b]
|
||||
Or a b -> [a, b]
|
||||
Neg a -> [a]
|
||||
IsNull a -> [a]
|
||||
IsNotNull a -> [a]
|
||||
CountOf a -> [a]
|
||||
SumOf a -> [a]
|
||||
AvgOf a -> [a]
|
||||
MinOf a -> [a]
|
||||
MaxOf a -> [a]
|
||||
_ -> []
|
||||
@@ -8,22 +8,25 @@ import ChuSQL.Core.Engine.Algebra.Optimize (optimize)
|
||||
import ChuSQL.Core.Engine.Algebra.Planner (translate)
|
||||
import ChuSQL.Core.Engine.Semantic (prepare, querySchema)
|
||||
import ChuSQL.Core.Engine.Syntax.AST (Statement)
|
||||
import ChuSQL.Core.Engine.Storage (MonadStorage)
|
||||
import ChuSQL.Core.Engine.Storage (MonadStorage (schema, withSchemaGuard))
|
||||
import ChuSQL.Core.Engine.Runtime.SQL (sqlType)
|
||||
import ChuSQL.Core.Engine.Runtime.Types (TypeId)
|
||||
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
|
||||
import ChuSQL.Core.Engine.Runtime.Dependencies (queryTables, domainDependencies)
|
||||
import ChuSQL.Core.Model
|
||||
|
||||
-- SQL 查询绑定边界:保存解析后的类型、函数身份及输出 schema。
|
||||
|
||||
-- | 保存运行时或兼容列类型
|
||||
data QueryType = RuntimeType TypeId | CompatibilityType ColumnType | NullType deriving (Show, Eq)
|
||||
data QueryType = RuntimeType TypeId | CompatibilityType ColumnType
|
||||
| DomainType String QueryType | NullType deriving (Show, Eq)
|
||||
|
||||
-- | 保存仅由 schema 绑定的执行计划
|
||||
data BoundQuery = BoundQuery Database [(String, Column)] [(String, QueryType)] RelOp
|
||||
data BoundQuery = BoundQuery Database [String] [(String, Column)] [(String, QueryType)] RelOp
|
||||
|
||||
-- | 查询绑定计划的输出类型身份
|
||||
queryTypes :: BoundQuery -> [(String, QueryType)]
|
||||
queryTypes (BoundQuery _ _ types _) = types
|
||||
queryTypes (BoundQuery _ _ _ types _) = types
|
||||
|
||||
-- | 绑定 SQL 查询并优化确定的计划
|
||||
bindQuery :: Database -> Statement -> Either String BoundQuery
|
||||
@@ -33,7 +36,10 @@ bindQuery catalog statement = do
|
||||
columns <- querySchema schemaOnly resolved
|
||||
plan <- translate resolved
|
||||
types <- mapM typeOf columns
|
||||
Right (BoundQuery schemaOnly columns types (optimize schemaOnly plan))
|
||||
bound <- bindRelOpColumns schemaOnly [] (optimize schemaOnly plan)
|
||||
let sources = queryTables resolved
|
||||
dependencies = sources ++ domainDependencies schemaOnly sources
|
||||
Right (BoundQuery schemaOnly dependencies columns types bound)
|
||||
where
|
||||
-- | 解析输出列的运行时身份
|
||||
typeOf (name, column) = do
|
||||
@@ -41,19 +47,32 @@ bindQuery catalog statement = do
|
||||
Right (name, ty)
|
||||
-- | 保留阶段外类型的兼容表示
|
||||
bindingType CNull = Right NullType
|
||||
bindingType (CDomain name base) = DomainType name <$> bindingType base
|
||||
bindingType CDate = Right (CompatibilityType CDate)
|
||||
bindingType CTimestamp = Right (CompatibilityType CTimestamp)
|
||||
bindingType CBlob = Right (CompatibilityType CBlob)
|
||||
bindingType ty@(CDomain _ base)
|
||||
| typeClassOf base == TemporalClass || base == CBlob = Right (CompatibilityType ty)
|
||||
bindingType ty = RuntimeType <$> sqlType ty
|
||||
|
||||
-- | 执行绑定查询并校验输出类型
|
||||
executeQueryM :: MonadStorage m => BoundQuery -> m (Either String [Row])
|
||||
executeQueryM (BoundQuery catalog columns _ plan) = do
|
||||
result <- evalRelOpM catalog plan
|
||||
pure (result >>= mapM validateRow)
|
||||
executeQueryM (BoundQuery catalog dependencies columns _ plan) = do
|
||||
current <- schema
|
||||
case current >>= validateSchema of
|
||||
Left err -> pure (Left err)
|
||||
Right () -> do
|
||||
result <- withSchemaGuard [(name, table) | (name, table) <- catalog, name `elem` dependencies]
|
||||
(evalRelOpM catalog plan)
|
||||
pure (result >>= mapM validateRow)
|
||||
where
|
||||
-- | 拒绝绑定目录中的结构变化
|
||||
validateSchema current = mapM_ (validateTable current)
|
||||
[(name, table) | (name, table) <- catalog, name `elem` dependencies]
|
||||
-- | 比较列定义而不比较数据或统计
|
||||
validateTable current (name, expected) = case lookup name current of
|
||||
Just actual | tableIdentity actual /= tableIdentity expected -> Left ("bound query object changed: " ++ name)
|
||||
Just actual | tableSchemaVersion actual /= tableSchemaVersion expected -> Left ("bound query schema changed: " ++ name)
|
||||
Just actual | tableCols actual == tableCols expected -> Right ()
|
||||
_ -> Left ("bound query schema changed: " ++ name)
|
||||
-- | 校验完整输出行而不重新解析函数
|
||||
validateRow row
|
||||
| map fst row /= map fst columns = Left "bound query output schema mismatch"
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
module ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns) where
|
||||
|
||||
import ChuSQL.Core.Engine.Algebra.Op
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Model (Database)
|
||||
|
||||
-- 优化完成后按算子输入布局绑定列槽位,外层行作为独立尾部布局。
|
||||
|
||||
-- | 绑定关系算子的输入列位置
|
||||
bindRelOpColumns :: Database -> [String] -> RelOp -> Either String RelOp
|
||||
bindRelOpColumns db outer = bind
|
||||
where
|
||||
-- \| 递归绑定每个算子的输入布局
|
||||
bind op = case op of
|
||||
Filter expression child -> Filter <$> expressionIn child expression <*> bind child
|
||||
Compute items child -> Compute <$> mapM (itemIn child) items <*> bind child
|
||||
Aggregate keys items child ->
|
||||
BoundAggregate
|
||||
<$> mapM (itemIn child) [(name, Col name) | name <- keys]
|
||||
<*> mapM (itemIn child) items
|
||||
<*> bind child
|
||||
BoundAggregate keys items child ->
|
||||
BoundAggregate
|
||||
<$> mapM (itemIn child) keys
|
||||
<*> mapM (itemIn child) items
|
||||
<*> bind child
|
||||
Join kind left right condition ->
|
||||
Join kind
|
||||
<$> bind left
|
||||
<*> bind right
|
||||
<*> bindSlots (relOpCols db left ++ relOpCols db right) outer condition
|
||||
Project columns child -> do
|
||||
let names = if columns == ["*"] then relOpCols db child else columns
|
||||
items <- mapM (itemIn child) [(name, Col name) | name <- names]
|
||||
Compute items <$> bind child
|
||||
Distinct child -> Distinct <$> bind child
|
||||
Sort ordering child ->
|
||||
BoundSort
|
||||
<$> mapM (sortIn child) [(Col name, direction) | (name, direction) <- ordering]
|
||||
<*> bind child
|
||||
BoundSort ordering child -> BoundSort <$> mapM (sortIn child) ordering <*> bind child
|
||||
Limit count child -> Limit count <$> bind child
|
||||
Derived alias child -> Derived alias <$> bind child
|
||||
_ -> Right op
|
||||
-- \| 使用子计划输出布局绑定表达式
|
||||
expressionIn child = bindSlots (relOpCols db child) outer
|
||||
-- \| 保留输出标签并绑定计算项
|
||||
itemIn child (name, expression) = (,) name <$> expressionIn child expression
|
||||
-- \| 绑定排序键并保留方向
|
||||
sortIn child (expression, direction) = (,) <$> expressionIn child expression <*> pure direction
|
||||
|
||||
-- | 绑定当前表达式的列引用
|
||||
bindSlots :: [String] -> [String] -> Expr -> Either String Expr
|
||||
bindSlots columns outer expression = case expression of
|
||||
Col name -> case slots columns name of
|
||||
slot : _ -> Right (ColumnSlot slot name)
|
||||
[] -> outerReference name
|
||||
ColumnSlot _ name -> case slots columns name of
|
||||
slot : _ -> Right (ColumnSlot slot name)
|
||||
[] -> Left ("unknown local column: " ++ name)
|
||||
OuterColumnSlot _ _ name -> outerReference name
|
||||
FunctionCall name arguments -> FunctionCall name <$> mapM bind arguments
|
||||
BoundFunction fid tid arguments -> BoundFunction fid tid <$> mapM bind arguments
|
||||
Construct ty arguments -> Construct ty <$> mapM bind arguments
|
||||
BoundConstruct tid arguments -> BoundConstruct tid <$> mapM bind arguments
|
||||
Add a b -> binary Add a b
|
||||
Sub a b -> binary Sub a b
|
||||
Mul a b -> binary Mul a b
|
||||
Div a b -> binary Div a b
|
||||
Neg a -> Neg <$> bind a
|
||||
Gt a b -> binary Gt a b
|
||||
Lt a b -> binary Lt a b
|
||||
Eq a b -> binary Eq a b
|
||||
GtE a b -> binary GtE a b
|
||||
LtE a b -> binary LtE a b
|
||||
NotEq a b -> binary NotEq a b
|
||||
And a b -> binary And a b
|
||||
Or a b -> binary Or a b
|
||||
IsNull a -> IsNull <$> bind a
|
||||
IsNotNull a -> IsNotNull <$> bind a
|
||||
CountOf a -> CountOf <$> bind a
|
||||
SumOf a -> SumOf <$> bind a
|
||||
AvgOf a -> AvgOf <$> bind a
|
||||
MinOf a -> MinOf <$> bind a
|
||||
MaxOf a -> MaxOf <$> bind a
|
||||
InSub a query negated -> InSub <$> bind a <*> pure query <*> pure negated
|
||||
InList a values negated -> InList <$> bind a <*> mapM bind values <*> pure negated
|
||||
QuantCmp op a query quantifier -> QuantCmp op <$> bind a <*> pure query <*> pure quantifier
|
||||
_ -> Right expression
|
||||
where
|
||||
-- \| 绑定当前布局的子表达式
|
||||
bind = bindSlots columns outer
|
||||
-- \| 查找布局中首次出现的列
|
||||
slots names name = [slot | (slot, key) <- zip [0 ..] names, key == name]
|
||||
-- \| 在外层布局绑定并保留作用域
|
||||
outerReference name = case slots outer name of
|
||||
slot : _ -> Right (OuterColumnSlot (length columns) slot name)
|
||||
[] -> Left ("unknown column: " ++ name)
|
||||
-- \| 绑定两个操作数
|
||||
binary constructor a b = constructor <$> bind a <*> bind b
|
||||
@@ -1,7 +1,7 @@
|
||||
module ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, invokeSQLFunction, sqlType, constructSQLValue) where
|
||||
|
||||
import ChuSQL.Core.Engine.Runtime.Types
|
||||
import ChuSQL.Core.Engine.Runtime.Functions
|
||||
import ChuSQL.Core.Engine.Runtime.Types
|
||||
import ChuSQL.Core.Model (ColumnType (..), Value (..))
|
||||
|
||||
-- SQL 现有标量表示与已解析运行时函数之间的类型边界。
|
||||
@@ -46,16 +46,21 @@ invokeSQLFunction :: FunctionId -> TypeId -> [Value] -> Either String Value
|
||||
invokeSQLFunction fid expected values = do
|
||||
functions <- builtinFunctions builtinTypes
|
||||
sig <- functionSignature functions fid
|
||||
if signatureResult sig /= expected then Left "bound function result identity mismatch"
|
||||
else if length values /= length (signatureArguments sig) then Left "function argument count mismatch"
|
||||
else do
|
||||
sequence_ (zipWith validateArgument (signatureArguments sig) values)
|
||||
if VNull `elem` values then Right VNull else do
|
||||
arguments <- mapM runtimeValue values
|
||||
result <- invokeFunction builtinTypes functions fid arguments
|
||||
sqlValue result
|
||||
if signatureResult sig /= expected
|
||||
then Left "bound function result identity mismatch"
|
||||
else
|
||||
if length values /= length (signatureArguments sig)
|
||||
then Left "function argument count mismatch"
|
||||
else do
|
||||
sequence_ (zipWith validateArgument (signatureArguments sig) values)
|
||||
if VNull `elem` values
|
||||
then Right VNull
|
||||
else do
|
||||
arguments <- mapM runtimeValue values
|
||||
result <- invokeFunction builtinTypes functions fid arguments
|
||||
sqlValue result
|
||||
where
|
||||
-- | 校验具体参数并保留 SQL NULL
|
||||
-- \| 校验具体参数并保留 SQL NULL
|
||||
validateArgument _ VNull = Right ()
|
||||
validateArgument tid value = runtimeValue value >>= validateValue builtinTypes tid
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
module ChuSQL.Core.Engine.Semantic (check, prepare, prepareRowExpression, querySchema) where
|
||||
|
||||
import ChuSQL.Core.Engine.Algebra.Expr (bareColumns, colsInExpr, hasAggregate)
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
|
||||
import ChuSQL.Core.Engine.Runtime.Functions (FunctionSignature (..), builtinFunctions, functionSignature)
|
||||
import ChuSQL.Core.Engine.Runtime.SQL (resolveSQLFunction, sqlResultType, sqlType)
|
||||
import ChuSQL.Core.Engine.Runtime.Types
|
||||
import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseExpression)
|
||||
import ChuSQL.Core.Model
|
||||
import Data.Char (isAlphaNum, isAscii, toLower)
|
||||
import Data.List (intercalate, isInfixOf, isPrefixOf, isSuffixOf, nub)
|
||||
|
||||
@@ -115,7 +115,7 @@ checkGrouping env keys items
|
||||
mapM_ (checkRuntimeColumn "GROUP BY" env) keys
|
||||
mapM_ okCol (nub (concatMap (bareColumns . snd) items))
|
||||
where
|
||||
-- | 这一列能不能出现在分组查询里
|
||||
-- \| 这一列能不能出现在分组查询里
|
||||
okCol c
|
||||
| c == allColumns = Left "SELECT: * cannot be selected together with GROUP BY"
|
||||
| c `elem` keys = Right ()
|
||||
@@ -129,6 +129,8 @@ renderType (InferType t) = typeLabel t
|
||||
-- | 推导表达式的类型(子查询也在这里整条查一遍)
|
||||
inferExpr :: Database -> String -> Scope -> Expr -> Either String InferredType
|
||||
inferExpr _ place env (Col c) = inferredColumnType . columnType . snd <$> scopeAt place env c
|
||||
inferExpr _ _ _ ColumnSlot{} = Left "execution column cannot be used in semantic input"
|
||||
inferExpr _ _ _ OuterColumnSlot{} = Left "execution column cannot be used in semantic input"
|
||||
inferExpr _ _ _ LitNull = Right InferNull
|
||||
inferExpr _ _ _ (LitInt _) = Right (InferType CInt)
|
||||
inferExpr _ _ _ (LitFloat _) = Right (InferType CFloat)
|
||||
@@ -231,18 +233,23 @@ itemType place want got
|
||||
-- | 推导子查询的输出列与类型
|
||||
inferSubqueryOutput :: Database -> Scope -> Subquery -> Either String [(String, InferredType)]
|
||||
inferSubqueryOutput db outerScope sq = case subqueryStatement sq of
|
||||
q@Select { selectCols = cols } -> do
|
||||
SelectDistinct q -> do
|
||||
columns <- inferSubqueryOutput db outerScope sq{subqueryStatement = q}
|
||||
mapM_ (requireOrdering "DISTINCT" . snd) columns
|
||||
_ <- distinctOrdering q
|
||||
Right columns
|
||||
q@Select{selectCols = cols} -> do
|
||||
env <- checkFrom db enclosing (selectFrom q)
|
||||
checkResolvedWith db enclosing q
|
||||
mapM (columnTypeOf env) cols
|
||||
q@SelectExpr { selectItems = items } -> do
|
||||
q@SelectExpr{selectItems = items} -> do
|
||||
env <- checkFrom db enclosing (selectFrom q)
|
||||
checkResolvedWith db enclosing q
|
||||
mapM (\(l, e) -> (\t -> (l, t)) <$> inferExpr db "SELECT" (Scope env enclosing) e) items
|
||||
_ -> Left "subquery must be a SELECT"
|
||||
where
|
||||
enclosing = scopeFlat outerScope
|
||||
-- | 查这一列的推导类型
|
||||
-- \| 查这一列的推导类型
|
||||
columnTypeOf env c = do
|
||||
(_, col) <- scopeAt "SELECT" (Scope env enclosing) c
|
||||
Right (c, inferredColumnType (columnType col))
|
||||
@@ -278,7 +285,7 @@ arithmetic db place env a b = do
|
||||
mapM_ (numericOperand place (InferType want) (InferType want)) [ta, tb]
|
||||
Right (arithResult ta tb)
|
||||
where
|
||||
-- | 是不是浮点类型
|
||||
-- \| 是不是浮点类型
|
||||
isFloatSide (InferType x) = numericType x && not (integerType x)
|
||||
isFloatSide _ = False
|
||||
|
||||
@@ -312,7 +319,7 @@ boolean db place env a b = do
|
||||
mapM_ (boolOperand place ta) [ta, tb]
|
||||
Right (InferType CBool)
|
||||
where
|
||||
-- | 挨个查操作数是不是布尔
|
||||
-- \| 挨个查操作数是不是布尔
|
||||
boolOperand _ _ InferNull = Right ()
|
||||
boolOperand _ _ (InferType ty) | typeClassOf ty == BooleanClass = Right ()
|
||||
boolOperand p other t = Left (p ++ ": operator needs TBool on both sides, got " ++ renderType other ++ " and " ++ renderType t)
|
||||
@@ -419,13 +426,13 @@ resolveDomainTypes db statement = case statement of
|
||||
AlterColumnType name col ty -> AlterColumnType name col <$> resolveDomainType ty
|
||||
_ -> Right statement
|
||||
where
|
||||
-- | 解析一列的类型
|
||||
-- \| 解析一列的类型
|
||||
resolveColumnType (name, col) = do
|
||||
ty <- resolveDomainType (columnType col)
|
||||
Right (name, col {columnType = ty})
|
||||
-- | 从类型目录取基础类型
|
||||
Right (name, col{columnType = ty})
|
||||
-- \| 从类型目录取基础类型
|
||||
resolveDomainType = resolveWith []
|
||||
-- | 递归解析类型并拒绝循环定义
|
||||
-- \| 递归解析类型并拒绝循环定义
|
||||
resolveWith seen (CDomain name _)
|
||||
| map toLower name `elem` seen = Left ("cyclic domain definition: " ++ name)
|
||||
| otherwise = do
|
||||
@@ -442,6 +449,7 @@ checkResolved db = checkResolvedWith db []
|
||||
-- | 检查已解析的语句;outer 是相关子查询能看见的外层列
|
||||
checkResolvedWith :: Database -> [(String, Column)] -> Statement -> Either String ()
|
||||
checkResolvedWith db outer q = case q of
|
||||
SelectDistinct inner -> () <$ inferSubqueryOutput db (Scope [] outer) (Subquery (SelectDistinct inner) [])
|
||||
Select
|
||||
{ selectCols = cols
|
||||
, selectFrom = fromC
|
||||
@@ -552,6 +560,7 @@ checkResolvedWith db outer q = case q of
|
||||
if length (nub (map fst attributes)) == length attributes
|
||||
then Right ()
|
||||
else Left "duplicate identity attribute"
|
||||
ShowGrants -> Right ()
|
||||
ShowRoles -> Right ()
|
||||
DropUser name -> checkUserName name
|
||||
CreateRole name -> checkRoleName name
|
||||
@@ -560,6 +569,7 @@ checkResolvedWith db outer q = case q of
|
||||
mapM_ checkPrivilege privs
|
||||
checkGrantObject obj
|
||||
checkRoleName role
|
||||
RevokeGrantOption privs obj role -> checkResolvedWith db outer (RevokePrivileges privs obj role)
|
||||
RevokePrivileges privs obj role -> do
|
||||
mapM_ checkPrivilege privs
|
||||
checkGrantObject obj
|
||||
@@ -577,11 +587,11 @@ checkResolvedWith db outer q = case q of
|
||||
CreateDomain{} -> Right ()
|
||||
DropDomain{} -> Right ()
|
||||
ShowDomains -> Right ()
|
||||
-- | 事务控制由会话层执行,语义层不做列检查
|
||||
-- \| 事务控制由会话层执行,语义层不做列检查
|
||||
BeginTransaction -> Right ()
|
||||
CommitTransaction -> Right ()
|
||||
RollbackTransaction -> Right ()
|
||||
-- | 保存点名字只是会话层的记号,不做列检查
|
||||
-- \| 保存点名字只是会话层的记号,不做列检查
|
||||
Savepoint _ -> Right ()
|
||||
RollbackToSavepoint _ -> Right ()
|
||||
ReleaseSavepoint _ -> Right ()
|
||||
@@ -615,10 +625,11 @@ checkColumnDefs names cols = do
|
||||
else Right ()
|
||||
checkChecks names cols
|
||||
where
|
||||
-- | 拒绝缺少能力实现的运行时键
|
||||
-- \| 拒绝缺少能力实现的运行时键
|
||||
checkRuntimeKey (_, column)
|
||||
| typeClassOf (columnType column) == RuntimeClass
|
||||
, columnUnique column || columnPrimaryKey column = Left "runtime type key capability is not registered"
|
||||
, columnUnique column || columnPrimaryKey column =
|
||||
Left "runtime type key capability is not registered"
|
||||
| otherwise = Right ()
|
||||
|
||||
-- | 校验列类型的长度和精度参数
|
||||
@@ -649,13 +660,13 @@ checkDefault c v
|
||||
checkChecks :: [String] -> [(String, Column)] -> Either String ()
|
||||
checkChecks names = mapM_ checkOne
|
||||
where
|
||||
-- | 查这一列的 CHECK
|
||||
-- \| 查这一列的 CHECK
|
||||
checkOne (_, c) = case columnCheck c of
|
||||
Nothing -> Right ()
|
||||
Just text -> case parseExpression text of
|
||||
Left err -> Left ("CHECK: " ++ firstLine err)
|
||||
Right e -> mapM_ (known e) (colsInExpr e)
|
||||
-- | 这个列名在不在定义里
|
||||
-- \| 这个列名在不在定义里
|
||||
known _ col
|
||||
| col `elem` names = Right ()
|
||||
| otherwise = Left ("CHECK: unknown column: " ++ col)
|
||||
@@ -683,7 +694,7 @@ checkUserName name
|
||||
| not (all allowed name) = Left "USER: account name may only contain ASCII letters, digits, dot, dash or underscore"
|
||||
| otherwise = Right ()
|
||||
where
|
||||
-- | 账号名字符规则
|
||||
-- \| 账号名字符规则
|
||||
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
|
||||
|
||||
-- | 建号 / 改口令:名字合法且口令不空
|
||||
@@ -706,7 +717,7 @@ checkRoleName name
|
||||
| map toLower name `elem` privilegeNames = Left "ROLE: role name must not be a privilege keyword"
|
||||
| otherwise = Right ()
|
||||
where
|
||||
-- | 角色名字符规则
|
||||
-- \| 角色名字符规则
|
||||
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
|
||||
|
||||
-- | 权限关键字(小写,只用于比较)
|
||||
@@ -728,12 +739,13 @@ checkGrantObject obj
|
||||
| not (all allowed obj) = Left "GRANT: object name may only contain ASCII letters, digits, dot, dash or underscore"
|
||||
| otherwise = Right ()
|
||||
where
|
||||
-- | 授权对象字符规则
|
||||
-- \| 授权对象字符规则
|
||||
allowed c = isAscii c && (isAlphaNum c || c `elem` ("_.-" :: String))
|
||||
|
||||
-- | 内置 id 列名
|
||||
idColumn :: String
|
||||
idColumn = "id"
|
||||
|
||||
-- | 把列引用统一为执行期名字(子查询一起解析)
|
||||
resolveExpr ::
|
||||
Database ->
|
||||
@@ -783,7 +795,7 @@ resolveExpr db env outer f e = case e of
|
||||
QuantCmp op a sq q -> QuantCmp op <$> resolveExpr db env outer f a <*> resolveSubquery db (env ++ outer) sq <*> pure q
|
||||
_ -> Right e
|
||||
where
|
||||
-- | 递归处理两个子表达式
|
||||
-- \| 递归处理两个子表达式
|
||||
binary ctor a b = ctor <$> resolveExpr db env outer f a <*> resolveExpr db env outer f b
|
||||
|
||||
-- | 校验具体构造器和参数逻辑类型
|
||||
@@ -800,7 +812,7 @@ inferConstruction db place env ty@(TypeApply cid parameters) arguments = do
|
||||
required <- mapM (fmap descriptorId . resolveType builtinTypes) expected
|
||||
if supplied == required then Right (descriptorId descriptor) else Left "composite constructor argument type mismatch"
|
||||
where
|
||||
-- | 解析非空参数的逻辑类型身份
|
||||
-- \| 解析非空参数的逻辑类型身份
|
||||
argumentIdentity InferNull = Left "SQL NULL needs Maybe construction inside a runtime value"
|
||||
argumentIdentity (InferType column) = sqlType column
|
||||
|
||||
@@ -829,7 +841,7 @@ outerRefsOf db stmt = nub [c | c <- statementColumns stmt, not (inInner c)]
|
||||
inner = case checkFrom db [] (statementFrom stmt) of
|
||||
Right env -> env
|
||||
Left _ -> []
|
||||
-- | 这一列能不能在内层解析到
|
||||
-- \| 这一列能不能在内层解析到
|
||||
inInner c = case resolveColumn c inner of
|
||||
Right _ -> True
|
||||
Left _ -> False
|
||||
@@ -837,13 +849,14 @@ outerRefsOf db stmt = nub [c | c <- statementColumns stmt, not (inInner c)]
|
||||
-- | 一条语句里出现的所有列引用
|
||||
statementColumns :: Statement -> [String]
|
||||
statementColumns stmt = case stmt of
|
||||
SelectDistinct q -> statementColumns q
|
||||
Select cols fromC mWhere groupBy orderBy _ ->
|
||||
cols ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy
|
||||
SelectExpr items fromC mWhere groupBy orderBy _ ->
|
||||
concatMap (colsInExpr . snd) items ++ fromColumns fromC ++ concatMap colsInExpr mWhere ++ groupBy ++ map fst orderBy
|
||||
_ -> []
|
||||
where
|
||||
-- | 取 FROM 里出现的列
|
||||
-- \| 取 FROM 里出现的列
|
||||
fromColumns FromUnit = []
|
||||
fromColumns (FromTable _ _) = []
|
||||
fromColumns (FromSubquery _ _) = []
|
||||
@@ -851,12 +864,14 @@ statementColumns stmt = case stmt of
|
||||
|
||||
-- | 取语句的 FROM 子句,没有就是 FromUnit
|
||||
statementFrom :: Statement -> FromClause
|
||||
statementFrom Select { selectFrom = fromC } = fromC
|
||||
statementFrom SelectExpr { selectFrom = fromC } = fromC
|
||||
statementFrom (SelectDistinct q) = statementFrom q
|
||||
statementFrom Select{selectFrom = fromC} = fromC
|
||||
statementFrom SelectExpr{selectFrom = fromC} = fromC
|
||||
statementFrom _ = FromUnit
|
||||
|
||||
-- | 解析子查询里的 SELECT
|
||||
resolveSubqueryStatement :: Database -> [(String, Column)] -> Statement -> Either String Statement
|
||||
resolveSubqueryStatement db outer (SelectDistinct q) = SelectDistinct <$> resolveSubqueryStatement db outer q
|
||||
resolveSubqueryStatement db outer q@Select{} = resolveQuery db outer q (map (\c -> (c, Col c)) (selectCols q))
|
||||
resolveSubqueryStatement db outer q@SelectExpr{} = resolveQuery db outer q (selectItems q)
|
||||
resolveSubqueryStatement _ _ _ = Left "subquery must be a SELECT"
|
||||
@@ -864,6 +879,7 @@ resolveSubqueryStatement _ _ _ = Left "subquery must be a SELECT"
|
||||
-- | 列名解析入口:查询走连接解析,改数据的语句按本表列解析
|
||||
resolveStatement :: Database -> Statement -> Either String Statement
|
||||
resolveStatement db q = case q of
|
||||
SelectDistinct inner -> SelectDistinct <$> resolveStatement db inner
|
||||
Select{} -> resolveSelect db q
|
||||
SelectExpr{} -> resolveSelect db q
|
||||
Update tbl assigns mWhere -> do
|
||||
@@ -923,37 +939,45 @@ resolveQuery :: Database -> [(String, Column)] -> Statement -> [(String, Expr)]
|
||||
resolveQuery db outer q items = do
|
||||
source <- resolveSource db outer (selectFrom q)
|
||||
env <- checkFrom db outer source
|
||||
-- | 单表无别名时把裸列名换成物理列名;带别名时表头保留别名限定名
|
||||
-- \| 单表无别名时把裸列名换成物理列名;带别名时表头保留别名限定名
|
||||
let physical c = case selectFrom q of
|
||||
FromTable Nothing tbl -> unqualify (Just (deriveQualifier Nothing tbl)) c
|
||||
_ -> c
|
||||
-- | 查一列并换算成物理名
|
||||
-- \| 查一列并换算成物理名
|
||||
name place c
|
||||
| length (filter (== '.') c) == 2, not (qualifiedReference c) = Left (missingColumn place c env)
|
||||
| otherwise = physical . fst <$> scopedColumnAt place env outer c
|
||||
-- | 库限定列只引用来源中的真实表名
|
||||
qualifiedReference c = qualifiedReferenceIn db (selectFrom q) c
|
||||
|| (any (\(key, _) -> length (filter (== '.') key) == 1 && ("." ++ key) `isSuffixOf` c) outer
|
||||
&& (not (any (elem '.' . fst) db) || any (\(table, _) -> (table ++ ".") `isPrefixOf` c) db))
|
||||
-- | 换算一个投影项:带别名时保留别名做输出列名
|
||||
-- \| 库限定列只引用来源中的真实表名
|
||||
qualifiedReference c =
|
||||
qualifiedReferenceIn db (selectFrom q) c
|
||||
|| ( any (\(key, _) -> length (filter (== '.') key) == 1 && ("." ++ key) `isSuffixOf` c) outer
|
||||
&& (not (any (elem '.' . fst) db) || any (\(table, _) -> (table ++ ".") `isPrefixOf` c) db)
|
||||
)
|
||||
-- \| 换算一个投影项:带别名时保留别名做输出列名
|
||||
output (label, Col c) = do
|
||||
k <- name "SELECT" c
|
||||
pure (if label == c then (k, Col k) else (label, Col k))
|
||||
output (label, e) = do
|
||||
e' <- resolveExpr db env outer (name "SELECT") e
|
||||
pure (label, e')
|
||||
projected <- if items == [("*", Col "*")]
|
||||
then if null env && selectFrom q == FromUnit
|
||||
then Left "SELECT: * requires FROM"
|
||||
else Right [(physical c, Col (physical c)) | (c, _) <- env]
|
||||
else mapM output items
|
||||
projected <-
|
||||
if items == [("*", Col "*")]
|
||||
then
|
||||
if null env && selectFrom q == FromUnit
|
||||
then Left "SELECT: * requires FROM"
|
||||
else Right [(physical c, Col (physical c)) | (c, _) <- env]
|
||||
else mapM output items
|
||||
condition <- mapM (resolveExpr db env outer (name "WHERE")) (selectWhere q)
|
||||
grouping <- mapM (name "GROUP BY") (selectGroupBy q)
|
||||
-- | 优先解析唯一投影别名,再查来源列
|
||||
let orderName c = case [label | (label, _) <- projected, label == c, qIsExpression] of
|
||||
[] -> name "ORDER BY" c
|
||||
[label] -> Right label
|
||||
_ -> Left ("ORDER BY: ambiguous projection alias: " ++ c)
|
||||
-- \| 优先解析唯一投影别名,再查来源列
|
||||
let orderName c
|
||||
| "$position" `isPrefixOf` c = case reads (drop 9 c) of
|
||||
[(n, "")] | n > 0, n <= length projected -> Right (fst (projected !! (n - 1)))
|
||||
_ -> Left "ORDER BY: position is outside the select list"
|
||||
| otherwise = case [label | (label, _) <- projected, label == c, qIsExpression] of
|
||||
[] -> name "ORDER BY" c
|
||||
[label] -> Right label
|
||||
_ -> Left ("ORDER BY: ambiguous projection alias: " ++ c)
|
||||
qIsExpression = case q of SelectExpr{} -> True; _ -> False
|
||||
ordering <- mapM (\(c, d) -> (\k -> (k, d)) <$> orderName c) (selectOrderBy q)
|
||||
pure $ case q of
|
||||
@@ -994,7 +1018,7 @@ resolveFrom db outer (FromJoin kind left right cond) = do
|
||||
l <- resolveFrom db outer left
|
||||
r <- resolveFrom db outer right
|
||||
env <- checkFrom db outer (FromJoin kind l r cond)
|
||||
-- | 连接条件也拒绝不存在的库前缀
|
||||
-- \| 连接条件也拒绝不存在的库前缀
|
||||
let joinName column
|
||||
| length (filter (== '.') column) == 2, not (qualifiedReferenceIn db (FromJoin kind left right cond) column) = Left (missingColumn "ON" column env)
|
||||
| otherwise = fst <$> columnAt "ON" env column
|
||||
@@ -1003,7 +1027,8 @@ resolveFrom db outer (FromJoin kind left right cond) = do
|
||||
|
||||
-- | 派生表的输出列名:列取裸名,表达式保留原标签
|
||||
nameDerivedColumns :: Statement -> Either String Statement
|
||||
nameDerivedColumns q@Select { selectCols = cols, selectFrom = fromC } = do
|
||||
nameDerivedColumns (SelectDistinct q) = SelectDistinct <$> nameDerivedColumns q
|
||||
nameDerivedColumns q@Select{selectCols = cols, selectFrom = fromC} = do
|
||||
let names = map lastTablePart cols
|
||||
checkDerivedNames names
|
||||
Right
|
||||
@@ -1015,10 +1040,10 @@ nameDerivedColumns q@Select { selectCols = cols, selectFrom = fromC } = do
|
||||
(selectOrderBy q)
|
||||
(selectLimit q)
|
||||
)
|
||||
nameDerivedColumns q@SelectExpr { selectItems = items } = do
|
||||
nameDerivedColumns q@SelectExpr{selectItems = items} = do
|
||||
let names = map derivedItemName items
|
||||
checkDerivedNames names
|
||||
Right q { selectItems = zip names (map snd items) }
|
||||
Right q{selectItems = zip names (map snd items)}
|
||||
nameDerivedColumns _ = Left "FROM: subquery must be a SELECT"
|
||||
|
||||
-- | 派生表里一个投影项的输出名:没写别名的列取裸名
|
||||
@@ -1032,7 +1057,7 @@ derivedItemName (label, _) = label
|
||||
checkDerivedNames :: [String] -> Either String ()
|
||||
checkDerivedNames = go []
|
||||
where
|
||||
-- | 逐个查有没有见过
|
||||
-- \| 逐个查有没有见过
|
||||
go _ [] = Right ()
|
||||
go seen (n : ns)
|
||||
| n `elem` seen = Left ("FROM: duplicate column name in subquery: " ++ n)
|
||||
|
||||
@@ -13,15 +13,22 @@ data IndexResult
|
||||
deriving (Show, Eq)
|
||||
|
||||
class (Monad m) => MonadStorage m where
|
||||
-- \| 在稳定存储视图中执行绑定读取
|
||||
withSchemaGuard :: Database -> m (Either String a) -> m (Either String a)
|
||||
withSchemaGuard _ _ = pure (Left "bound schema guards are unavailable in this storage")
|
||||
|
||||
-- \| 建一个库
|
||||
createDatabase :: String -> m (Either String ())
|
||||
createDatabase _ = pure (Left "database management is unavailable in this storage")
|
||||
|
||||
-- \| 删一个库
|
||||
dropDatabase :: String -> m (Either String ())
|
||||
dropDatabase _ = pure (Left "database management is unavailable in this storage")
|
||||
|
||||
-- \| 切当前库
|
||||
useDatabase :: String -> m (Either String ())
|
||||
useDatabase _ = pure (Left "database selection is unavailable in this storage")
|
||||
|
||||
-- \| 列出现有库
|
||||
listDatabases :: m (Either String [String])
|
||||
listDatabases = pure (Right [])
|
||||
@@ -50,7 +57,7 @@ class (Monad m) => MonadStorage m where
|
||||
insertMany :: String -> [Row] -> m (Either String ())
|
||||
insertMany t = go
|
||||
where
|
||||
-- | 逐行插入,出错就停
|
||||
-- \| 逐行插入,出错就停
|
||||
go [] = pure (Right ())
|
||||
go (r : rs) = do
|
||||
x <- insert t r
|
||||
@@ -66,7 +73,7 @@ class (Monad m) => MonadStorage m where
|
||||
Left e -> pure (Left e)
|
||||
Right rs -> replaceAll t [r | r <- rs, not (doomed r)]
|
||||
where
|
||||
-- | 这一行要不要删
|
||||
-- \| 这一行要不要删
|
||||
doomed r = case lookup "id" r of
|
||||
Just (VInt k) -> k `elem` ks
|
||||
_ -> False
|
||||
@@ -114,5 +121,5 @@ class (Monad m) => MonadStorage m where
|
||||
schema :: m (Either String Database)
|
||||
schema = fmap (fmap (map withoutRows)) snapshot
|
||||
where
|
||||
-- | 表定义去掉所有行
|
||||
withoutRows (name, t) = (name, t {tableRows = []})
|
||||
-- \| 表定义去掉所有行
|
||||
withoutRows (name, t) = (name, t{tableRows = []})
|
||||
@@ -65,7 +65,7 @@ openStorage path = onStorageThread $ mask_ $ do
|
||||
| ptr == nullPtr -> Left <$> lastError
|
||||
| otherwise -> pure (Right (StorageHandle ptr))
|
||||
where
|
||||
-- | 没有路径就传空指针,否则转成 C 字符串
|
||||
-- \| 没有路径就传空指针,否则转成 C 字符串
|
||||
withCStringOrNull Nothing f = f nullPtr
|
||||
withCStringOrNull (Just s) f = BS.useAsCString (TE.encodeUtf8 (T.pack s)) f
|
||||
|
||||
@@ -78,18 +78,24 @@ storageRequest :: StorageHandle -> BS.ByteString -> IO (Either String BS.ByteStr
|
||||
storageRequest (StorageHandle handle) = callStorage (cStorageRequest handle)
|
||||
|
||||
-- | 调用存储并复制和归还响应缓冲
|
||||
callStorage :: (Ptr CChar -> CSize -> Ptr (Ptr CChar) -> Ptr CSize -> IO CInt)
|
||||
-> BS.ByteString -> IO (Either String BS.ByteString)
|
||||
callStorage call payload = onStorageThread $ mask_ $
|
||||
BS.useAsCStringLen payload $ \(input, size) ->
|
||||
alloca $ \outRef -> alloca $ \lenRef -> do
|
||||
code <- call input (fromIntegral size) outRef lenRef
|
||||
if code /= 0 then Left <$> lastError else do
|
||||
output <- peek outRef
|
||||
len <- peek lenRef
|
||||
bytes <- BS.packCStringLen (output, fromIntegral len)
|
||||
`finally` cStorageFree output len
|
||||
pure (Right bytes)
|
||||
callStorage ::
|
||||
(Ptr CChar -> CSize -> Ptr (Ptr CChar) -> Ptr CSize -> IO CInt) ->
|
||||
BS.ByteString ->
|
||||
IO (Either String BS.ByteString)
|
||||
callStorage call payload = onStorageThread $
|
||||
mask_ $
|
||||
BS.useAsCStringLen payload $ \(input, size) ->
|
||||
alloca $ \outRef -> alloca $ \lenRef -> do
|
||||
code <- call input (fromIntegral size) outRef lenRef
|
||||
if code /= 0
|
||||
then Left <$> lastError
|
||||
else do
|
||||
output <- peek outRef
|
||||
len <- peek lenRef
|
||||
bytes <-
|
||||
BS.packCStringLen (output, fromIntegral len)
|
||||
`finally` cStorageFree output len
|
||||
pure (Right bytes)
|
||||
|
||||
-- | 多线程运行时绑定存储调用线程
|
||||
onStorageThread :: IO a -> IO a
|
||||
|
||||
@@ -30,15 +30,17 @@ module ChuSQL.Core.Engine.Storage.IPC (
|
||||
doListCatalog,
|
||||
) where
|
||||
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Protocol (Account (..), Request (..), Response (..), SchemaColumn (..), TableInfo (..))
|
||||
import ChuSQL.Core.Engine.Parallel (parallelShardLimit, poolRun, workerPool)
|
||||
import ChuSQL.Core.Engine.Storage
|
||||
import ChuSQL.Core.Engine.Storage.FFI (closeStorage, openStorage, storageRequest)
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Protocol (Account (..), Request (..), Response (..), SchemaColumn (..), TableInfo (..))
|
||||
import Control.Concurrent (getNumCapabilities)
|
||||
import Control.Concurrent.MVar (MVar, modifyMVar_, newMVar, putMVar, readMVar, takeMVar, withMVar)
|
||||
import Control.Exception (bracket, mask_)
|
||||
import Data.Aeson (eitherDecodeStrict, encode)
|
||||
import Data.Aeson qualified as A
|
||||
import Data.Aeson.KeyMap qualified as KM
|
||||
import Data.ByteString qualified as BS
|
||||
import Data.ByteString.Lazy qualified as BL
|
||||
import Data.Maybe (fromMaybe)
|
||||
@@ -49,6 +51,7 @@ import System.IO.Unsafe (unsafePerformIO)
|
||||
-- | 一条会话的存储环境:当前库名随会话走,不共享
|
||||
data Env = Env
|
||||
{ envDatabase :: Maybe String
|
||||
, envSchemaGuard :: Maybe Database
|
||||
}
|
||||
|
||||
newtype IPCStorage a = IPCStorage
|
||||
@@ -57,11 +60,11 @@ newtype IPCStorage a = IPCStorage
|
||||
|
||||
-- | 按库名造一个会话环境
|
||||
envForDatabase :: Maybe String -> Env
|
||||
envForDatabase = Env
|
||||
envForDatabase name = Env name Nothing
|
||||
|
||||
-- | 进程默认环境:读全局库名,给测试与 CLI 用
|
||||
defaultEnv :: IO Env
|
||||
defaultEnv = Env <$> readMVar databaseRef
|
||||
defaultEnv = envForDatabase <$> readMVar databaseRef
|
||||
|
||||
-- | 用进程默认环境跑(兼容既有调用点)
|
||||
runIPCStorage :: IPCStorage a -> IO a
|
||||
@@ -86,11 +89,13 @@ data StorageLink = StorageLink
|
||||
}
|
||||
|
||||
{-# NOINLINE linkRef #-}
|
||||
|
||||
-- | 进程全局的存储链路
|
||||
linkRef :: MVar (Maybe StorageLink)
|
||||
linkRef = unsafePerformIO (newMVar Nothing)
|
||||
|
||||
{-# NOINLINE linkUsers #-}
|
||||
|
||||
-- | 在途请求计数与空闲信号
|
||||
linkUsers :: (MVar Int, MVar ())
|
||||
linkUsers = unsafePerformIO $ (,) <$> newMVar 0 <*> newMVar ()
|
||||
@@ -100,7 +105,7 @@ withStorageLink :: (Maybe StorageLink -> IO a) -> IO a
|
||||
withStorageLink = bracket enter leave
|
||||
where
|
||||
(users, idle) = linkUsers
|
||||
-- | 登记当前链路的在途请求
|
||||
-- \| 登记当前链路的在途请求
|
||||
enter = withMVar linkRef $ \current -> do
|
||||
case current of
|
||||
Nothing -> pure ()
|
||||
@@ -108,7 +113,7 @@ withStorageLink = bracket enter leave
|
||||
if count == 0 then takeMVar idle else pure ()
|
||||
pure (count + 1)
|
||||
pure current
|
||||
-- | 最后一个请求结束后发出空闲信号
|
||||
-- \| 最后一个请求结束后发出空闲信号
|
||||
leave Nothing = pure ()
|
||||
leave (Just _) = mask_ $ modifyMVar_ users $ \count -> do
|
||||
if count == 1 then putMVar idle () else pure ()
|
||||
@@ -122,7 +127,7 @@ setStorageLink link = modifyMVar_ linkRef $ \_ -> withMVar (snd linkUsers) (cons
|
||||
localStorageLink :: Maybe FilePath -> IO (Either String StorageLink)
|
||||
localStorageLink path = fmap (fmap toLink) (openStorage path)
|
||||
where
|
||||
-- | 把句柄包成一条链路
|
||||
-- \| 把句柄包成一条链路
|
||||
toLink handle =
|
||||
StorageLink
|
||||
{ slRequest = storageRequest handle
|
||||
@@ -157,6 +162,7 @@ sendRawRequest payload = withStorageLink $ \current -> do
|
||||
Just link -> slRequest link payload
|
||||
|
||||
{-# NOINLINE databaseRef #-}
|
||||
|
||||
-- | 进程全局的当前库名
|
||||
databaseRef :: MVar (Maybe String)
|
||||
databaseRef = unsafePerformIO (newMVar Nothing)
|
||||
@@ -169,7 +175,6 @@ setDatabaseName name = modifyMVar_ databaseRef (const (pure name))
|
||||
getDatabaseName :: IO (Maybe String)
|
||||
getDatabaseName = readMVar databaseRef
|
||||
|
||||
|
||||
-- | 给语句类请求盖上当前库名
|
||||
stampDatabase :: Maybe String -> Request -> Request
|
||||
stampDatabase Nothing req = req
|
||||
@@ -192,13 +197,43 @@ globalRequest _ = False
|
||||
-- | 发请求并把响应翻成 Either
|
||||
ask :: Env -> String -> Request -> (Response -> Maybe a) -> IO (Either String a)
|
||||
ask env what req recognize = do
|
||||
resp <- sendRequest (stampDatabase (envDatabase env) req)
|
||||
resp <- case envSchemaGuard env of
|
||||
Nothing -> sendRequest (stampDatabase (envDatabase env) req)
|
||||
Just dependencies -> guardedRequest env dependencies req
|
||||
pure $ case resp of
|
||||
RespError e -> Left e
|
||||
other -> case recognize other of
|
||||
Just a -> Right a
|
||||
Nothing -> Left ("unexpected response to " ++ what)
|
||||
|
||||
-- | 发送依赖守卫并要求存储确认执行
|
||||
guardedRequest :: Env -> Database -> Request -> IO Response
|
||||
guardedRequest env dependencies req = do
|
||||
let payload = case A.toJSON (stampDatabase (envDatabase env) req) of
|
||||
A.Object fields -> A.Object (KM.insert "schema_guard" (A.toJSON (map guardOf dependencies)) fields)
|
||||
other -> other
|
||||
result <- sendRawRequest (BL.toStrict (encode payload))
|
||||
pure $ case result >>= eitherDecodeStrict of
|
||||
Left err -> RespError err
|
||||
Right (A.Object fields)
|
||||
| KM.lookup "status" fields == Just (A.String "guarded")
|
||||
, Just response <- KM.lookup "response" fields -> case A.fromJSON response of
|
||||
A.Success decoded -> decoded
|
||||
A.Error err -> RespError ("decode error: " ++ err)
|
||||
Right value -> case A.fromJSON value of
|
||||
A.Success (RespError err) -> RespError err
|
||||
_ -> RespError "storage did not confirm bound schema guard"
|
||||
where
|
||||
-- \| 编码依赖身份、版本与完整列定义
|
||||
guardOf (name, table) =
|
||||
A.object
|
||||
[ "table" A..= name
|
||||
, "database_id" A..= fmap (\(TableIdentity database _) -> show database) (tableIdentity table)
|
||||
, "object_id" A..= fmap (\(TableIdentity _ object) -> show object) (tableIdentity table)
|
||||
, "schema_version" A..= fmap show (tableSchemaVersion table)
|
||||
, "columns" A..= map toWire (tableCols table)
|
||||
]
|
||||
|
||||
-- | 只认"成了"(多数写操作都这样)
|
||||
okOnly :: Response -> Maybe ()
|
||||
okOnly RespOk = Just ()
|
||||
@@ -310,18 +345,27 @@ toWire (n, c) =
|
||||
}
|
||||
|
||||
instance MonadStorage IPCStorage where
|
||||
-- \| 将依赖守卫限定于本次执行环境
|
||||
withSchemaGuard dependencies action = IPCStorage $ \env ->
|
||||
runIPCStorageIn action env{envSchemaGuard = Just dependencies}
|
||||
|
||||
-- \| 建一个库
|
||||
createDatabase name = IPCStorage $ \env -> ask env "create_database" (ReqDatabase "create_database" name) okOnly
|
||||
|
||||
-- \| 删一个库
|
||||
dropDatabase name = IPCStorage $ \env -> ask env "drop_database" (ReqDatabase "drop_database" name) okOnly
|
||||
|
||||
-- \| 切当前库
|
||||
useDatabase name = IPCStorage $ \env -> ask env "use_database" (ReqDatabase "use_database" name) okOnly
|
||||
|
||||
-- \| 列出现有库
|
||||
listDatabases = IPCStorage $ \env -> ask env "list_databases" (ReqDatabase "list_databases" "") $ \resp -> case resp of
|
||||
RespTables names -> Just names
|
||||
_ -> Nothing
|
||||
|
||||
-- \| 发 Scan
|
||||
scan t = IPCStorage (\env -> doScan env t)
|
||||
|
||||
-- \| 只取指定列
|
||||
scanColumns t cols = IPCStorage (\env -> doScanColumns env t cols)
|
||||
|
||||
@@ -381,13 +425,17 @@ instance MonadStorage IPCStorage where
|
||||
Left err -> pure (Left err)
|
||||
Right xs -> fmap sequence (mapM (loadEntry env) xs)
|
||||
where
|
||||
-- | 读取表数据并保留扫描错误
|
||||
-- \| 读取表数据并保留扫描错误
|
||||
loadEntry env info = do
|
||||
rows <- doScan env (tiTable info)
|
||||
pure $ fmap (\rs ->
|
||||
( tiTable info
|
||||
, Table (tiTable info) (schemaToColumns (tiColumns info)) rs (Just (metaOf info))
|
||||
)) rows
|
||||
pure $
|
||||
fmap
|
||||
( \rs ->
|
||||
( tiTable info
|
||||
, Table (tiTable info) (schemaToColumns (tiColumns info)) rs (Just (metaOf info)) (tiIdentity info) (tiSchemaVersion info)
|
||||
)
|
||||
)
|
||||
rows
|
||||
|
||||
-- \| 只问数据字典要结构,设了当前库只留它的表
|
||||
schema = IPCStorage $ \env -> do
|
||||
@@ -409,6 +457,6 @@ visibleTables current info
|
||||
else []
|
||||
| otherwise = [bare info]
|
||||
where
|
||||
named = info {tiTable = unqualify current (tiTable info)}
|
||||
-- | 组装成不带行的表
|
||||
bare i = (tiTable i, Table (tiTable i) (schemaToColumns (tiColumns i)) [] (Just (metaOf i)))
|
||||
named = info{tiTable = unqualify current (tiTable info)}
|
||||
-- \| 组装成不带行的表
|
||||
bare i = (tiTable i, Table (tiTable i) (schemaToColumns (tiColumns i)) [] (Just (metaOf i)) (tiIdentity i) (tiSchemaVersion i))
|
||||
@@ -1,8 +1,9 @@
|
||||
module ChuSQL.Core.Engine.Storage.Memory (MemoryStorage (runMemoryStorage)) where
|
||||
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Storage
|
||||
import qualified Data.HashSet as HS
|
||||
import ChuSQL.Core.Model
|
||||
import Data.HashSet qualified as HS
|
||||
import Data.Word (Word64)
|
||||
|
||||
-- 内存实现:Database 上的状态,错误通道是 Either String。
|
||||
|
||||
@@ -26,6 +27,9 @@ instance Monad MemoryStorage where
|
||||
runMemoryStorage (k a) db'
|
||||
|
||||
instance MonadStorage MemoryStorage where
|
||||
-- \| 纯状态视图在求值期间不受外部修改
|
||||
withSchemaGuard _ action = action
|
||||
|
||||
-- \| 全表扫描
|
||||
scan t = MemoryStorage $ \db -> do
|
||||
tbl <- lookupTable db t
|
||||
@@ -45,7 +49,7 @@ instance MonadStorage MemoryStorage where
|
||||
createTable name cols = MemoryStorage $ \db ->
|
||||
if any ((== name) . fst) db
|
||||
then Right (Left ("table already exists: " ++ name), db)
|
||||
else Right (Right (), db ++ [(name, Table name cols [] Nothing)])
|
||||
else Right (Right (), db ++ [(name, Table name cols [] Nothing Nothing (Just 1))])
|
||||
|
||||
-- \| 删表;不存在报错
|
||||
dropTable name = MemoryStorage $ \db ->
|
||||
@@ -60,7 +64,8 @@ instance MonadStorage MemoryStorage where
|
||||
Just tbl ->
|
||||
if col `notElem` map fst (tableCols tbl)
|
||||
then Right (Left ("unknown column: " ++ col), db)
|
||||
else
|
||||
else do
|
||||
version <- nextSchemaVersion tbl
|
||||
Right
|
||||
( Right ()
|
||||
, replaceTable
|
||||
@@ -68,6 +73,7 @@ instance MonadStorage MemoryStorage where
|
||||
tbl
|
||||
{ tableCols = filter ((/= col) . fst) (tableCols tbl)
|
||||
, tableRows = map (filter ((/= col) . fst)) (tableRows tbl)
|
||||
, tableSchemaVersion = Just version
|
||||
}
|
||||
db
|
||||
)
|
||||
@@ -78,9 +84,9 @@ instance MonadStorage MemoryStorage where
|
||||
-- \| 返回实时行数与去重统计的无数据结构
|
||||
schema = MemoryStorage $ \db -> Right (Right (map describe db), db)
|
||||
where
|
||||
-- | 将存储数据汇总为结构统计
|
||||
-- \| 将存储数据汇总为结构统计
|
||||
describe (name, table) =
|
||||
(name, table {tableRows = [], tableMeta = Just metadata})
|
||||
(name, table{tableRows = [], tableMeta = Just metadata})
|
||||
where
|
||||
metadata = TableMeta (length (tableRows table)) distinct [] []
|
||||
distinct = [columnStatistics column (tableRows table) | (column, _) <- tableCols table]
|
||||
@@ -88,13 +94,25 @@ instance MonadStorage MemoryStorage where
|
||||
-- \| 改表结构:列定义与行一起换
|
||||
replaceSchema name cols rows = MemoryStorage $ \db -> case lookup name db of
|
||||
Nothing -> Right (Left ("unknown table: " ++ name), db)
|
||||
Just tbl -> Right (Right (), replaceTable name tbl{tableCols = cols, tableRows = rows} db)
|
||||
Just tbl -> do
|
||||
version <-
|
||||
if tableCols tbl == cols
|
||||
then Right (tableSchemaVersion tbl)
|
||||
else Just <$> nextSchemaVersion tbl
|
||||
Right (Right (), replaceTable name tbl{tableCols = cols, tableRows = rows, tableSchemaVersion = version} db)
|
||||
|
||||
-- | 递增结构版本并拒绝耗尽
|
||||
nextSchemaVersion :: Table -> Either String Word64
|
||||
nextSchemaVersion table = case tableSchemaVersion table of
|
||||
Just version | version == maxBound -> Left "catalog schema version exhausted"
|
||||
Just version -> Right (version + 1)
|
||||
Nothing -> Right 1
|
||||
|
||||
-- | 对列值去重并限制统计内存
|
||||
columnStatistics :: String -> [Row] -> (String, Int, Bool)
|
||||
columnStatistics column rows = go rows HS.empty
|
||||
where
|
||||
-- | 累积规范化值,达到上限后停止
|
||||
-- \| 累积规范化值,达到上限后停止
|
||||
go remaining values
|
||||
| HS.size values >= 4096 = (column, 4096, True)
|
||||
| otherwise = case remaining of
|
||||
@@ -105,4 +123,3 @@ columnStatistics column rows = go rows HS.empty
|
||||
replaceTable :: String -> Table -> Database -> Database
|
||||
replaceTable name table =
|
||||
map (\(n, t) -> if n == name then (n, table) else (n, t))
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
module ChuSQL.Core.Engine.Syntax.AST (Statement (..), FromClause (..), JoinKind (..), Expr (..), Subquery (..), Quantifier (..), CompareOp (..), SortDir (..), makeSelect) where
|
||||
module ChuSQL.Core.Engine.Syntax.AST (Statement (..), FromClause (..), JoinKind (..), Expr (..), Subquery (..), Quantifier (..), CompareOp (..), SortDir (..), makeSelect, distinctOrdering) where
|
||||
|
||||
import ChuSQL.Core.Model (Column (..), ColumnType, Value (..))
|
||||
import ChuSQL.Core.Engine.Runtime.Types (TypeId, TypeExpr, RuntimeValue)
|
||||
@@ -23,6 +23,7 @@ data Statement
|
||||
, selectOrderBy :: [(String, SortDir)]
|
||||
, selectLimit :: Maybe Int
|
||||
}
|
||||
| SelectDistinct Statement
|
||||
| Insert String [String] [[Expr]]
|
||||
| Delete String (Maybe Expr)
|
||||
| Update String [(String, Expr)] (Maybe Expr)
|
||||
@@ -42,6 +43,7 @@ data Statement
|
||||
| CreateUser String String
|
||||
| AlterUser String String
|
||||
| AlterIdentity String [(String, Bool)]
|
||||
| ShowGrants
|
||||
| ShowRoles
|
||||
| DropUser String
|
||||
-- | 新角色:权限的载体,用户靠成员资格继承它的权限
|
||||
@@ -52,6 +54,7 @@ data Statement
|
||||
| GrantPrivileges [String] String String Bool
|
||||
-- | REVOKE 权限 ON 表 FROM 角色
|
||||
| RevokePrivileges [String] String String
|
||||
| RevokeGrantOption [String] String String
|
||||
-- | GRANT 角色 TO 用户:给用户加角色
|
||||
| GrantRole String [String]
|
||||
-- | REVOKE <角色> FROM <用户>[, ...]
|
||||
@@ -114,6 +117,8 @@ data CompareOp
|
||||
-- | 条件表达式:列、字面量、算术、比较、逻辑、聚合与子查询
|
||||
data Expr
|
||||
= Col String
|
||||
| ColumnSlot Int String
|
||||
| OuterColumnSlot Int Int String
|
||||
| FunctionCall String [Expr]
|
||||
| BoundFunction FunctionId TypeId [Expr]
|
||||
| Construct TypeExpr [Expr]
|
||||
@@ -172,3 +177,23 @@ makeSelect cols tbl w =
|
||||
, selectOrderBy = []
|
||||
, selectLimit = Nothing
|
||||
}
|
||||
|
||||
-- | 将去重排序映射到输出列
|
||||
distinctOrdering :: Statement -> Either String [(String, SortDir)]
|
||||
distinctOrdering q@Select{} = resolveDistinctOrdering q
|
||||
distinctOrdering q@SelectExpr{} = resolveDistinctOrdering q
|
||||
distinctOrdering _ = Left "DISTINCT requires SELECT"
|
||||
|
||||
-- | 解析去重输出的排序列
|
||||
resolveDistinctOrdering :: Statement -> Either String [(String, SortDir)]
|
||||
resolveDistinctOrdering q = mapM resolve (selectOrderBy q)
|
||||
where
|
||||
items = case q of
|
||||
Select { selectCols = cols } -> [(c, Col c) | c <- cols]
|
||||
SelectExpr { selectItems = projected } -> projected
|
||||
_ -> []
|
||||
-- | 排序键必须出现在去重结果中
|
||||
resolve (column, direction) = case [label | (label, e) <- items, label == column || e == Col column] of
|
||||
[label] -> Right (label, direction)
|
||||
[] -> Left "DISTINCT: ORDER BY column must appear in the select list"
|
||||
_ -> Left "DISTINCT: ambiguous ORDER BY column"
|
||||
@@ -262,7 +262,7 @@ literalValue =
|
||||
-- | 一个排序列
|
||||
orderItem :: Parser (String, SortDir)
|
||||
orderItem = do
|
||||
col <- qualifiedName
|
||||
col <- qualifiedName <|> (("$position" ++) . show <$> integer)
|
||||
dir <- sortDir
|
||||
return (col, dir)
|
||||
|
||||
@@ -590,14 +590,15 @@ grantStatement = do
|
||||
revokeStatement :: Parser Statement
|
||||
revokeStatement = do
|
||||
keyword "revoke"
|
||||
optionOnly <- fromMaybe False <$> optional (True <$ (keyword "grant" *> keyword "option" *> keyword "for"))
|
||||
choice
|
||||
[ try $ do
|
||||
privs <- sepBy1 privilegeName (symbol ",")
|
||||
keyword "on"
|
||||
obj <- grantObject
|
||||
keyword "from"
|
||||
RevokePrivileges privs obj <$> userName
|
||||
, RevokeRole <$> userName <*> (keyword "from" *> sepBy1 userName (symbol ","))
|
||||
(if optionOnly then RevokeGrantOption else RevokePrivileges) privs obj <$> userName
|
||||
, if optionOnly then fail "GRANT OPTION FOR requires object privileges" else RevokeRole <$> userName <*> (keyword "from" *> sepBy1 userName (symbol ","))
|
||||
]
|
||||
|
||||
-- | 建表时的一列:类型后面跟若干约束
|
||||
@@ -689,13 +690,14 @@ decimalArgs = do
|
||||
selectStatement :: Parser Statement
|
||||
selectStatement = do
|
||||
keyword "select"
|
||||
distinct <- fromMaybe False <$> optional ((True <$ keyword "distinct") <|> (False <$ keyword "all"))
|
||||
cols <- selectList
|
||||
fromC <- fromMaybe FromUnit <$> optional (keyword "from" *> fromClause)
|
||||
mWhere <- optional (keyword "where" *> expr)
|
||||
groupBy <- fromMaybe [] <$> optional groupByClause
|
||||
orderBy <- fromMaybe [] <$> optional orderByClause
|
||||
mLimit <- optional limitClause
|
||||
pure $ if all (\(name, e) -> e == Col name) cols
|
||||
pure $ (if distinct then SelectDistinct else id) $ if all (\(name, e) -> e == Col name) cols
|
||||
then Select (map fst cols) fromC mWhere groupBy orderBy mLimit
|
||||
else SelectExpr cols fromC mWhere groupBy orderBy mLimit
|
||||
|
||||
@@ -738,6 +740,7 @@ cteKey = map toLower
|
||||
-- | CTE 带列清单时改本体的输出名
|
||||
renameColumns :: String -> [String] -> Statement -> Either String Statement
|
||||
renameColumns _ [] q = Right q
|
||||
renameColumns name cols (SelectDistinct q) = SelectDistinct <$> renameColumns name cols q
|
||||
renameColumns name cols q@Select { selectCols = names } = do
|
||||
checkCteArity name cols names
|
||||
Right
|
||||
@@ -770,6 +773,7 @@ checkCteArity name cols out
|
||||
-- | 把语句里对 CTE 的引用换成派生表
|
||||
expandStatement :: [(String, Statement)] -> Statement -> Either String Statement
|
||||
expandStatement env stmt = case stmt of
|
||||
SelectDistinct q -> SelectDistinct <$> expandStatement env q
|
||||
q@Select { selectFrom = fromC, selectWhere = mWhere } -> do
|
||||
fromC' <- expandClause env fromC
|
||||
w <- traverse (expandExpr env) mWhere
|
||||
@@ -909,6 +913,7 @@ parseStatement input =
|
||||
<|> try createUserStatement
|
||||
<|> try alterUserStatement
|
||||
<|> try alterRoleStatement
|
||||
<|> try (ShowGrants <$ (keyword "show" *> keyword "grants"))
|
||||
<|> try (ShowRoles <$ (keyword "show" *> keyword "roles"))
|
||||
<|> try dropUserStatement
|
||||
<|> try createRoleStatement
|
||||
|
||||
@@ -3,20 +3,38 @@ module RuntimeSpec (spec) where
|
||||
import ChuSQL.Core.Engine.Runtime.Types
|
||||
import ChuSQL.Core.Engine.Runtime.Functions
|
||||
import ChuSQL.Core.Engine.Runtime.Binding
|
||||
import ChuSQL.Core.Engine.Runtime.Relation (bindRelOpColumns)
|
||||
import qualified ChuSQL.Core.Engine.Algebra.Op as Rel
|
||||
import ChuSQL.Core.Engine.Algebra.Expr (evalExpr)
|
||||
import ChuSQL.Core.Engine.Algebra.Eval (evalRelOp)
|
||||
import ChuSQL.Core.Engine.Algebra.Sort (sortBoundRows)
|
||||
import ChuSQL.Core.Engine.Storage.Memory (runMemoryStorage)
|
||||
import qualified ChuSQL.Core.Engine.Runtime.Query as Query
|
||||
import ChuSQL.Core.Engine (rowsOf, runStatement)
|
||||
import ChuSQL.Core.Engine.Semantic (prepare)
|
||||
import ChuSQL.Core.Engine.Semantic (prepare, prepareRowExpression)
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
|
||||
import ChuSQL.Core.Engine.Syntax.AST (Expr (..), Statement (..))
|
||||
import ChuSQL.Core.Engine.Syntax.AST (Expr (..), Statement (..), SortDir (..))
|
||||
import qualified ChuSQL.Core.Model as SQL
|
||||
import ChuSQL.Core.Protocol (valueToJSON, valueFromJSON)
|
||||
import ChuSQL.Core.Protocol (valueToJSON, valueFromJSON, TableInfo (..))
|
||||
import qualified Data.Aeson as A
|
||||
import Data.Word (Word64)
|
||||
import Data.Aeson.Types (parseEither)
|
||||
import Data.Hashable (hash)
|
||||
import Control.Monad (foldM)
|
||||
import qualified Data.ByteString.Lazy.Char8 as BS
|
||||
import Test.Hspec
|
||||
|
||||
-- 运行时类型、函数绑定、物理编码和类型化查询测试。
|
||||
|
||||
-- | 执行语句建立测试目录
|
||||
sqlCatalog :: [String] -> Either String SQL.Database
|
||||
sqlCatalog = foldM apply []
|
||||
where
|
||||
-- | 更新当前测试数据库
|
||||
apply db source = do
|
||||
statement <- parseStatement source
|
||||
fst <$> runStatement db statement
|
||||
|
||||
-- | 检查运行时类型基础契约
|
||||
spec :: Spec
|
||||
spec = do
|
||||
@@ -72,6 +90,35 @@ spec = do
|
||||
let expression = Apply "not" [Apply "not" [Literal boolType (RBool True)]]
|
||||
(bindExpression builtinTypes functions [] expression >>= evaluateExpression builtinTypes functions [])
|
||||
`shouldBe` Right (RBool True)
|
||||
it "uses the bound column slot and rejects reordered expression input" $ withFunctions $ \functions ->
|
||||
withType intType $ \int -> withType stringType $ \string -> do
|
||||
let bound = bindExpression builtinTypes functions [("count", int), ("name", string)]
|
||||
(Apply "length" [Reference "name"])
|
||||
(bound >>= evaluateExpression builtinTypes functions [("count", RInt 1), ("name", RString "abc")])
|
||||
`shouldBe` Right (RInt 3)
|
||||
(bound >>= evaluateExpression builtinTypes functions [("name", RString "abc"), ("count", RInt 1)])
|
||||
`shouldBe` Left "bound column position mismatch: name"
|
||||
(bound >>= evaluateExpression builtinTypes functions [("count", RInt 1)])
|
||||
`shouldBe` Left "missing bound column: name"
|
||||
it "rejects ambiguous references before binding a slot" $ withFunctions $ \functions ->
|
||||
withType intType $ \int ->
|
||||
bindExpression builtinTypes functions [("id", int), ("id", int)] (Reference "id")
|
||||
`shouldSatisfy` isFailure
|
||||
it "normalizes storage order at scan boundaries before evaluating slots" $ withFunctions $ \functions -> do
|
||||
let catalog = [("items", [("name", stringType), ("enabled", boolType)])]
|
||||
query = Projection [("size", Apply "length" [Reference "name"])]
|
||||
(Selection (Reference "enabled") (TableScan "items"))
|
||||
scan _ = Right [[("enabled", RBool True), ("name", RString "abc")]]
|
||||
(bindPlan builtinTypes functions catalog query >>= executePlan builtinTypes functions scan)
|
||||
`shouldBe` Right [[("size", RInt 3)]]
|
||||
it "rebinds slots to each projection output schema" $ withFunctions $ \functions -> do
|
||||
let catalog = [("items", [("name", stringType), ("enabled", boolType)])]
|
||||
source = Projection [("enabled", Reference "enabled"), ("label", Reference "name")]
|
||||
(TableScan "items")
|
||||
query = Projection [("size", Apply "length" [Reference "label"])] source
|
||||
scan _ = Right [[("name", RString "abc"), ("enabled", RBool True)]]
|
||||
(bindPlan builtinTypes functions catalog query >>= executePlan builtinTypes functions scan)
|
||||
`shouldBe` Right [[("size", RInt 3)]]
|
||||
it "rejects a wrong function argument type during binding" $ withFunctions $ \functions ->
|
||||
bindExpression builtinTypes functions [] (Apply "length" [Literal intType (RInt 1)]) `shouldSatisfy` isFailure
|
||||
it "rejects unknown function identities" $ withFunctions $ \functions ->
|
||||
@@ -104,12 +151,219 @@ spec = do
|
||||
(bindPlan builtinTypes functions [("t", [])] (TableScan "t")
|
||||
>>= executePlan builtinTypes functions (const (Left "storage unavailable"))) `shouldBe` Left "storage unavailable"
|
||||
describe "SQL runtime functions" $ do
|
||||
it "validates transitive DOMAIN dependencies and ignores unrelated definitions" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE DOMAIN amount AS INT", "CREATE DOMAIN total AS amount"
|
||||
, "CREATE DOMAIN unrelated AS TEXT", "CREATE TABLE amounts (value total)"]
|
||||
statement <- parseStatement "SELECT value FROM amounts"
|
||||
bound <- Query.bindQuery db statement
|
||||
let change target = [(name, if name == target then table {SQL.tableCols = [("base", SQL.plainColumn SQL.CBool)]}
|
||||
else table) | (name, table) <- db]
|
||||
accepted <- fst <$> runMemoryStorage (Query.executeQueryM bound) (change "__system_domain_unrelated")
|
||||
rejected <- fst <$> runMemoryStorage (Query.executeQueryM bound) (change "__system_domain_amount")
|
||||
Right (accepted, rejected)
|
||||
result `shouldBe` Right (Right [], Left "bound query schema changed: __system_domain_amount")
|
||||
it "preserves exact catalog identities and rejects malformed identifiers" $ do
|
||||
let identity = SQL.TableIdentity (maxBound :: Word64) (maxBound :: Word64)
|
||||
info = TableInfo "items" [] 0 [] [] [] (Just identity) Nothing
|
||||
decode text = A.eitherDecode (BS.pack text) :: Either String TableInfo
|
||||
A.fromJSON (A.toJSON info) `shouldBe` A.Success info
|
||||
fmap tiIdentity (decode "{\"table\":\"items\",\"columns\":[],\"row_count\":0}") `shouldBe` Right Nothing
|
||||
fmap tiIdentity (decode "{\"table\":\"items\",\"columns\":[],\"row_count\":0,\"database_id\":0,\"object_id\":1}")
|
||||
`shouldBe` Right (Just (SQL.TableIdentity 0 1))
|
||||
mapM_ (\fields -> decode ("{\"table\":\"items\",\"columns\":[],\"row_count\":0," ++ fields ++ "}")
|
||||
`shouldSatisfy` isFailure)
|
||||
["\"database_id\":1", "\"database_id\":1,\"object_id\":-1"
|
||||
, "\"database_id\":1,\"object_id\":1.5", "\"database_id\":1,\"object_id\":\"18446744073709551616\""
|
||||
, "\"database_id\":1,\"object_id\":1e99999999"]
|
||||
it "rejects a same-schema replacement or loss of object identity" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
|
||||
let identified = [(name, table {SQL.tableIdentity = Just (SQL.TableIdentity 1 1)}) | (name, table) <- db]
|
||||
statement <- parseStatement "SELECT id FROM bound_items"
|
||||
bound <- Query.bindQuery identified statement
|
||||
let replaced = [(name, table {SQL.tableIdentity = Just (SQL.TableIdentity 1 2)}) | (name, table) <- db]
|
||||
changed <- fst <$> runMemoryStorage (Query.executeQueryM bound) replaced
|
||||
lost <- fst <$> runMemoryStorage (Query.executeQueryM bound) db
|
||||
Right (changed, lost)
|
||||
result `shouldBe` Right (Left "bound query object changed: bound_items", Left "bound query object changed: bound_items")
|
||||
it "preserves exact schema versions and rejects malformed versions" $ do
|
||||
let info = TableInfo "items" [] 0 [] [] [] Nothing (Just maxBound)
|
||||
decode version = A.eitherDecode (BS.pack ("{\"table\":\"items\",\"columns\":[],\"row_count\":0,\"schema_version\":" ++ version ++ "}")) :: Either String TableInfo
|
||||
A.fromJSON (A.toJSON info) `shouldBe` A.Success info
|
||||
fmap tiSchemaVersion (decode "0") `shouldBe` Right Nothing
|
||||
fmap tiSchemaVersion (decode "7") `shouldBe` Right (Just 7)
|
||||
mapM_ (\version -> decode version `shouldSatisfy` isFailure)
|
||||
["-1", "1.5", "\"18446744073709551616\"", "\"+1\"", "1e99999999"]
|
||||
it "invalidates a bound query after restoring its original columns" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE versioned (id INT)"]
|
||||
statement <- parseStatement "SELECT id FROM versioned"
|
||||
bound <- Query.bindQuery db statement
|
||||
added <- parseStatement "ALTER TABLE versioned ADD COLUMN extra TEXT" >>= runStatement db
|
||||
restored <- parseStatement "ALTER TABLE versioned DROP COLUMN extra" >>= runStatement (fst added)
|
||||
fst <$> runMemoryStorage (Query.executeQueryM bound) (fst restored)
|
||||
result `shouldBe` Right (Left "bound query schema changed: versioned")
|
||||
it "rejects loss of a bound schema version" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE versioned (id INT)"]
|
||||
statement <- parseStatement "SELECT id FROM versioned"
|
||||
bound <- Query.bindQuery db statement
|
||||
fst <$> runMemoryStorage (Query.executeQueryM bound)
|
||||
[(name, table {SQL.tableSchemaVersion = Nothing}) | (name, table) <- db]
|
||||
result `shouldBe` Right (Left "bound query schema changed: versioned")
|
||||
it "ignores unrelated table changes but validates nested query dependencies" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE referenced (id INT)", "CREATE TABLE unrelated (id INT)"]
|
||||
statement <- parseStatement "SELECT length((SELECT MIN(name) FROM (SELECT 'abc' AS name FROM referenced) d)) AS n"
|
||||
bound <- Query.bindQuery db statement
|
||||
unrelated <- parseStatement "ALTER TABLE unrelated ADD COLUMN extra TEXT" >>= runStatement db
|
||||
accepted <- fst <$> runMemoryStorage (Query.executeQueryM bound) (fst unrelated)
|
||||
referenced <- parseStatement "ALTER TABLE referenced ADD COLUMN extra TEXT" >>= runStatement db
|
||||
rejected <- fst <$> runMemoryStorage (Query.executeQueryM bound) (fst referenced)
|
||||
Right (accepted, rejected)
|
||||
result `shouldBe` Right (Right [[("n", SQL.VNull)]], Left "bound query schema changed: referenced")
|
||||
it "rejects schema drift before executing an empty bound query" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
|
||||
statement <- parseStatement "SELECT id FROM bound_items"
|
||||
bound <- Query.bindQuery db statement
|
||||
changed <- parseStatement "ALTER TABLE bound_items ADD COLUMN extra TEXT" >>= runStatement db
|
||||
fst <$> runMemoryStorage (Query.executeQueryM bound) (fst changed)
|
||||
result `shouldBe` Right (Left "bound query schema changed: bound_items")
|
||||
it "keeps binding valid across row and statistics changes" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE bound_items (id INT)"]
|
||||
statement <- parseStatement "SELECT id FROM bound_items"
|
||||
bound <- Query.bindQuery db statement
|
||||
changed <- parseStatement "INSERT INTO bound_items (id) VALUES (7)" >>= runStatement db
|
||||
let current = [(name, table {SQL.tableMeta = Just (SQL.TableMeta 100 [] [] [])})
|
||||
| (name, table) <- fst changed]
|
||||
fst <$> runMemoryStorage (Query.executeQueryM bound) current
|
||||
result `shouldBe` Right (Right [[("id", SQL.VInt 7)]])
|
||||
it "binds grouping keys and then sorting to the grouped output" $ do
|
||||
let source = Rel.Compute [("first", LitInt 1), ("key", LitInt 2)] Rel.Unit
|
||||
plan = Rel.Sort [("key", Desc)] (Rel.Aggregate ["key"] [("n", CountAll)] source)
|
||||
bound = Rel.BoundSort [(ColumnSlot 0 "key", Desc)]
|
||||
(Rel.BoundAggregate [("key", ColumnSlot 1 "key")] [("n", CountAll)] source)
|
||||
bindRelOpColumns [] [] plan `shouldBe` Right bound
|
||||
evalRelOp [] bound `shouldBe` Right [[("key", SQL.VInt 2), ("n", SQL.VInt 1)]]
|
||||
evalRelOp [] (Rel.BoundAggregate [("key", ColumnSlot 0 "key")] [] source)
|
||||
`shouldBe` Left "bound column position mismatch: key"
|
||||
it "sorts bound NULL keys stably and rejects wrong slots" $ do
|
||||
let a = [("id", SQL.VInt 1), ("key", SQL.VInt 2)]
|
||||
b = [("id", SQL.VInt 2), ("key", SQL.VNull)]
|
||||
c = [("id", SQL.VInt 3), ("key", SQL.VInt 2)]
|
||||
sortBoundRows [(ColumnSlot 1 "key", Asc)] [] [a, b, c] `shouldBe` Right [b, a, c]
|
||||
sortBoundRows [(ColumnSlot 1 "key", Desc)] [] [a, b, c] `shouldBe` Right [a, c, b]
|
||||
sortBoundRows [(ColumnSlot 0 "key", Asc)] [] [a]
|
||||
`shouldBe` Left "bound column position mismatch: key"
|
||||
it "preserves NULL group order and empty-input global aggregates" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE grouped_slots (id INT, tag TEXT)"
|
||||
, "INSERT INTO grouped_slots (id, tag) VALUES (1, 'b'), (2, NULL), (3, 'b')"]
|
||||
grouped <- parseStatement "SELECT tag, COUNT(*) AS n FROM grouped_slots GROUP BY tag"
|
||||
groups <- rowsOf (runStatement db grouped)
|
||||
empty <- parseStatement "SELECT COUNT(*) AS n, SUM(id) AS total FROM grouped_slots WHERE id < 0"
|
||||
aggregates <- rowsOf (runStatement db empty)
|
||||
Right (map (map snd) groups, map (map snd) aggregates)
|
||||
result `shouldBe` Right ([[SQL.VStr "b", SQL.VInt 2], [SQL.VNull, SQL.VInt 1]]
|
||||
, [[SQL.VInt 0, SQL.VNull]])
|
||||
it "binds SQL relation expressions to the optimized input slots" $ do
|
||||
let plan = Rel.Compute [("value", Col "second")]
|
||||
(Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit)
|
||||
bindRelOpColumns [] [] plan `shouldBe` Right (Rel.Compute [("value", ColumnSlot 1 "second")]
|
||||
(Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit))
|
||||
bindRelOpColumns [] [] (Rel.Filter (Col "missing") Rel.Unit)
|
||||
`shouldBe` Left "unknown column: missing"
|
||||
it "checks SQL slot position instead of searching another column" $ do
|
||||
evalExpr (ColumnSlot 1 "name") [("name", SQL.VStr "abc"), ("id", SQL.VInt 1)]
|
||||
`shouldBe` Left "bound column position mismatch: name"
|
||||
evalExpr (ColumnSlot (-1) "name") [("name", SQL.VStr "abc")]
|
||||
`shouldBe` Left "bound column position mismatch: name"
|
||||
it "binds projections without losing repeated outputs or empty rows" $ do
|
||||
let source = Rel.Compute [("first", LitInt 1), ("second", LitInt 2)] Rel.Unit
|
||||
bindRelOpColumns [] [] (Rel.Project ["second", "first", "second"] source)
|
||||
`shouldBe` Right (Rel.Compute [("second", ColumnSlot 1 "second")
|
||||
, ("first", ColumnSlot 0 "first"), ("second", ColumnSlot 1 "second")] source)
|
||||
bindRelOpColumns [] [] (Rel.Project [] source)
|
||||
`shouldBe` Right (Rel.Compute [] source)
|
||||
bindRelOpColumns [] [] (Rel.Project ["*"] source)
|
||||
`shouldBe` Right (Rel.Compute [("first", ColumnSlot 0 "first"), ("second", ColumnSlot 1 "second")] source)
|
||||
it "preserves repeated SQL outputs and correlated projection values" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE slot_items (id INT)"
|
||||
, "INSERT INTO slot_items (id) VALUES (1), (2)"]
|
||||
repeated <- parseStatement "SELECT id, id FROM slot_items"
|
||||
repeatedRows <- rowsOf (runStatement db repeated)
|
||||
correlated <- parseStatement "SELECT o.id, (SELECT o.id FROM slot_items i WHERE i.id = o.id) AS echoed FROM slot_items o"
|
||||
correlatedRows <- rowsOf (runStatement db correlated)
|
||||
Right (map (map snd) repeatedRows, map (map snd) correlatedRows)
|
||||
result `shouldBe` Right ([[SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2]]
|
||||
, [[SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2]])
|
||||
it "binds outer references after the local relation layout" $ do
|
||||
bindRelOpColumns [] ["outer.id"] (Rel.Compute [("id", Col "outer.id")] Rel.Unit)
|
||||
`shouldBe` Right (Rel.Compute [("id", OuterColumnSlot 0 0 "outer.id")] Rel.Unit)
|
||||
it "preserves outer scope identity when rebinding beside a local name" $ do
|
||||
let source = Rel.Compute [("id", LitInt 1)] Rel.Unit
|
||||
expression = OuterColumnSlot 0 0 "id"
|
||||
bound = bindRelOpColumns [] ["id"] (Rel.Compute [("outer", expression)] source)
|
||||
bound `shouldBe` Right (Rel.Compute [("outer", OuterColumnSlot 1 0 "id")] source)
|
||||
evalExpr (OuterColumnSlot 1 0 "id") [("id", SQL.VInt 1), ("id", SQL.VInt 9)]
|
||||
`shouldBe` Right (SQL.VInt 9)
|
||||
bindRelOpColumns [] [] (Rel.Compute [("outer", expression)] source)
|
||||
`shouldBe` Left "unknown column: id"
|
||||
prepareRowExpression [] expression `shouldBe` Left "execution column cannot be used in semantic input"
|
||||
prepareRowExpression [] (ColumnSlot 0 "id") `shouldBe` Left "execution column cannot be used in semantic input"
|
||||
it "keeps inner shadowing and multi-level correlated SQL references" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE scoped_items (id INT)"
|
||||
, "INSERT INTO scoped_items (id) VALUES (1), (2)"]
|
||||
statement <- parseStatement "SELECT o.id, (SELECT id FROM scoped_items i WHERE i.id = o.id) AS inner_id, (SELECT (SELECT o.id FROM scoped_items j WHERE j.id = i.id) FROM scoped_items i WHERE i.id = o.id) AS nested_id FROM scoped_items o"
|
||||
rows <- rowsOf (runStatement db statement)
|
||||
Right (map (map snd) rows)
|
||||
result `shouldBe` Right [[SQL.VInt 1, SQL.VInt 1, SQL.VInt 1], [SQL.VInt 2, SQL.VInt 2, SQL.VInt 2]]
|
||||
it "executes SQL against reversed physical column order" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE TABLE slot_items (id INT, name TEXT)"
|
||||
, "INSERT INTO slot_items (id, name) VALUES (1, 'abc'), (2, 'xy')"]
|
||||
let reordered = [(name, table {SQL.tableRows = map reverse (SQL.tableRows table)}) | (name, table) <- db]
|
||||
statement <- parseStatement "SELECT length(name) AS size, id + 1 AS next FROM slot_items WHERE id > 0 ORDER BY size"
|
||||
rowsOf (runStatement reordered statement)
|
||||
fmap (map (map snd)) result `shouldBe` Right [[SQL.VInt 2, SQL.VInt 3], [SQL.VInt 3, SQL.VInt 2]]
|
||||
it "binds the SQL entry point to output TypeIds" $ withType intType $ \tid -> do
|
||||
let result = parseStatement "SELECT length('abc') AS size" >>= Query.bindQuery []
|
||||
fmap Query.queryTypes result `shouldBe` Right [("size", Query.RuntimeType tid)]
|
||||
it "retains compatibility for existing temporal SQL types" $ do
|
||||
let result = parseStatement "SELECT DATE '2026-10-05' AS today" >>= Query.bindQuery []
|
||||
fmap Query.queryTypes result `shouldBe` Right [("today", Query.CompatibilityType SQL.CDate)]
|
||||
it "retains nested DOMAIN identities through derived aliases and DISTINCT" $ withType intType $ \tid -> do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE DOMAIN quantity AS INT", "CREATE DOMAIN stock AS quantity"
|
||||
, "CREATE TABLE inventory (qty stock)", "INSERT INTO inventory (qty) VALUES (2), (NULL)"]
|
||||
statement <- parseStatement "SELECT DISTINCT d.qty AS available FROM (SELECT qty FROM inventory) d"
|
||||
bound <- Query.bindQuery db statement
|
||||
rows <- rowsOf (runStatement db statement)
|
||||
Right (Query.queryTypes bound, map (map snd) rows)
|
||||
result `shouldBe` Right ([("available", Query.DomainType "stock"
|
||||
(Query.DomainType "quantity" (Query.RuntimeType tid)))], [[SQL.VInt 2], [SQL.VNull]])
|
||||
it "preserves temporal and binary DOMAIN wrappers" $ do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE DOMAIN day AS DATE", "CREATE DOMAIN payload AS BLOB"
|
||||
, "CREATE TABLE documents (created day, contents payload)"]
|
||||
statement <- parseStatement "SELECT created, contents FROM documents"
|
||||
Query.queryTypes <$> Query.bindQuery db statement
|
||||
result `shouldBe` Right [("created", Query.DomainType "day" (Query.CompatibilityType SQL.CDate))
|
||||
, ("contents", Query.DomainType "payload" (Query.CompatibilityType SQL.CBlob))]
|
||||
it "uses base function semantics without labeling function results as DOMAIN" $ withType intType $ \tid -> do
|
||||
let result = do
|
||||
db <- sqlCatalog ["CREATE DOMAIN label AS VARCHAR(3)", "CREATE TABLE names (name label)"
|
||||
, "INSERT INTO names (name) VALUES ('abc')"]
|
||||
statement <- parseStatement "SELECT length(name) AS size FROM names"
|
||||
bound <- Query.bindQuery db statement
|
||||
rows <- rowsOf (runStatement db statement)
|
||||
Right (Query.queryTypes bound, map (map snd) rows)
|
||||
result `shouldBe` Right ([("size", Query.RuntimeType tid)], [[SQL.VInt 3]])
|
||||
it "rejects composite ordering without an Ord implementation" $
|
||||
sql "SELECT LIST<Int>(1) AS xs ORDER BY xs" `shouldSatisfy` isFailure
|
||||
it "rejects composite extrema without an Ord implementation" $
|
||||
@@ -201,7 +455,7 @@ sql source = do
|
||||
-- | 提供函数调用测试表
|
||||
database = [("t", SQL.Table "t" [("name", SQL.plainColumn SQL.CStr), ("enabled", SQL.plainColumn SQL.CBool)]
|
||||
[[("name", SQL.VStr "ab"), ("enabled", SQL.VBool True)]
|
||||
,[("name", SQL.VStr "xyz"), ("enabled", SQL.VBool False)]] Nothing)]
|
||||
,[("name", SQL.VStr "xyz"), ("enabled", SQL.VBool False)]] Nothing Nothing Nothing)]
|
||||
|
||||
-- | 判断结果是否明确失败
|
||||
isFailure :: Either a b -> Bool
|
||||
|
||||
+142
-10
@@ -12,6 +12,7 @@ import ChuSQL.Core.Engine.Algebra.Sort (sortRows)
|
||||
import ChuSQL.Core.Engine.Builtin
|
||||
import ChuSQL.Core.Engine.Error
|
||||
import ChuSQL.Core.Engine
|
||||
import qualified ChuSQL.Core.Engine.Runtime.Query as Bound
|
||||
import ChuSQL.Core.Engine.Parallel (poolRun, workerPool)
|
||||
import ChuSQL.Core.Model
|
||||
import ChuSQL.Core.Engine.Semantic (prepare)
|
||||
@@ -46,7 +47,9 @@ import qualified RuntimeSpec
|
||||
users :: Table
|
||||
users =
|
||||
Table
|
||||
{ tableName = "users"
|
||||
{ tableSchemaVersion = Nothing
|
||||
, tableIdentity = Nothing
|
||||
, tableName = "users"
|
||||
, tableCols = [("id", TInt), ("name", TStr), ("age", TInt)]
|
||||
, tableRows =
|
||||
[ [("id", VInt 1), ("name", VStr "Alice"), ("age", VInt 25)]
|
||||
@@ -60,7 +63,9 @@ users =
|
||||
orders :: Table
|
||||
orders =
|
||||
Table
|
||||
{ tableName = "orders"
|
||||
{ tableSchemaVersion = Nothing
|
||||
, tableIdentity = Nothing
|
||||
, tableName = "orders"
|
||||
, tableCols = [("id", TInt), ("user_id", TInt), ("product", TStr)]
|
||||
, tableRows =
|
||||
[ [("id", VInt 1), ("user_id", VInt 1), ("product", VStr "Book")]
|
||||
@@ -78,7 +83,9 @@ testDB = [("users", users), ("orders", orders)]
|
||||
chainA :: Table
|
||||
chainA =
|
||||
Table
|
||||
{ tableName = "a"
|
||||
{ tableSchemaVersion = Nothing
|
||||
, tableIdentity = Nothing
|
||||
, tableName = "a"
|
||||
, tableCols = [("id", TInt), ("name", TStr)]
|
||||
, tableRows = [[("id", VInt n), ("name", VStr ("a" ++ show n))] | n <- [1 .. 6]]
|
||||
, tableMeta = Just (TableMeta 6 [("id", 6, False)] [] ["id"])
|
||||
@@ -88,7 +95,9 @@ chainA =
|
||||
chainB :: Table
|
||||
chainB =
|
||||
Table
|
||||
{ tableName = "b"
|
||||
{ tableSchemaVersion = Nothing
|
||||
, tableIdentity = Nothing
|
||||
, tableName = "b"
|
||||
, tableCols = [("id", TInt), ("a_id", TInt)]
|
||||
, tableRows = [[("id", VInt n), ("a_id", VInt n)] | n <- [1 .. 4]]
|
||||
, tableMeta = Just (TableMeta 4 [("id", 4, False), ("a_id", 4, False)] [] ["id"])
|
||||
@@ -98,7 +107,9 @@ chainB =
|
||||
chainC :: Table
|
||||
chainC =
|
||||
Table
|
||||
{ tableName = "c"
|
||||
{ tableSchemaVersion = Nothing
|
||||
, tableIdentity = Nothing
|
||||
, tableName = "c"
|
||||
, tableCols = [("id", TInt), ("b_id", TInt)]
|
||||
, tableRows = [[("id", VInt n), ("b_id", VInt n)] | n <- [1, 2]]
|
||||
, tableMeta = Just (TableMeta 2 [("id", 2, False), ("b_id", 2, False)] [] ["id"])
|
||||
@@ -124,7 +135,64 @@ domainSql db (sql : rest) = do
|
||||
-- | 跑引擎层全部 hspec 用例
|
||||
main :: IO ()
|
||||
main = hspec $ do
|
||||
describe "REVOKE GRANT OPTION FOR" $ do
|
||||
it "parses object privileges and rejects membership options" $ do
|
||||
parseStatement "REVOKE GRANT OPTION FOR SELECT, UPDATE ON users FROM alice"
|
||||
`shouldBe` Right (RevokeGrantOption ["SELECT", "UPDATE"] "users" "alice")
|
||||
parseStatement "REVOKE GRANT OPTION FOR reader FROM alice" `shouldSatisfy` either (const True) (const False)
|
||||
describe "ORDER BY positions" $ do
|
||||
it "resolves positions after wildcard expansion and before limiting" $ do
|
||||
rowsOf (parseStatement "SELECT * FROM orders ORDER BY 2 DESC, 1 DESC LIMIT 2" >>= runStatement testDB)
|
||||
`shouldBe` Right [tableRows orders !! 1, tableRows orders !! 2]
|
||||
rowsOf (parseStatement "SELECT DISTINCT user_id + 1 AS k FROM orders ORDER BY 1 DESC" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("k", VInt 3)], [("k", VInt 2)]]
|
||||
it "rejects zero and positions beyond the output" $ do
|
||||
mapM_ (\sql -> (parseStatement sql >>= runStatement testDB) `shouldSatisfy` either (isInfixOf "outside the select list") (const False))
|
||||
["SELECT id FROM users ORDER BY 0", "SELECT id FROM users ORDER BY 2"]
|
||||
describe "SELECT DISTINCT" $ do
|
||||
it "deduplicates projected values before ordering and limiting" $ do
|
||||
let db = [("orders", orders { tableRows = tableRows orders ++ tableRows orders })]
|
||||
rowsOf (parseStatement "SELECT DISTINCT user_id FROM orders ORDER BY user_id DESC LIMIT 2" >>= runStatement db)
|
||||
`shouldBe` Right [[("user_id", VInt 2)], [("user_id", VInt 1)]]
|
||||
rowsOf (parseStatement "SELECT DISTINCT user_id + 1 AS k FROM orders ORDER BY k LIMIT 2" >>= runStatement db)
|
||||
`shouldBe` Right [[("k", VInt 2)], [("k", VInt 3)]]
|
||||
rowsOf (parseStatement "SELECT ALL user_id FROM orders" >>= runStatement db)
|
||||
`shouldBe` Right (map (filter ((== "user_id") . fst)) (tableRows orders ++ tableRows orders))
|
||||
it "merges NULL and equal numeric values while retaining first rows" $ do
|
||||
evalRelOp [] (Distinct Unit) `shouldBe` Right [[]]
|
||||
let t = users { tableRows = [[("id", v), ("name", VStr "a"), ("age", VInt 1)] | v <- [VNull, VNull, VInt 1, VFloat 1, VInt 2]] }
|
||||
evalRelOp [("users", t)] (Distinct (Project ["id"] (Scan Nothing "users" Nothing)))
|
||||
`shouldBe` Right [[("id", VNull)], [("id", VInt 1)], [("id", VInt 2)]]
|
||||
it "keeps all distinct columns when an outer query projects fewer" $ do
|
||||
rowsOf (parseStatement "SELECT d.user_id FROM (SELECT DISTINCT user_id, product FROM orders) AS d" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("d.user_id", VInt 1)], [("d.user_id", VInt 2)], [("d.user_id", VInt 1)]]
|
||||
rowsOf (parseStatement "WITH d(k) AS (SELECT DISTINCT user_id FROM orders) SELECT k FROM d" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("d.k", VInt 1)], [("d.k", VInt 2)]]
|
||||
it "supports aggregates, empty inputs and output aliases" $ do
|
||||
rowsOf (parseStatement "SELECT DISTINCT COUNT(*) AS n FROM orders" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("n", VInt 3)]]
|
||||
rowsOf (parseStatement "SELECT DISTINCT user_id FROM orders WHERE user_id = 99" >>= runStatement testDB)
|
||||
`shouldBe` Right []
|
||||
rowsOf (parseStatement "SELECT DISTINCT user_id AS k FROM orders ORDER BY user_id" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("k", VInt 1)], [("k", VInt 2)]]
|
||||
it "rejects composite values without equality capability" $ do
|
||||
(parseStatement "SELECT DISTINCT LIST<Int>(1) AS xs" >>= runStatement [])
|
||||
`shouldSatisfy` either (isInfixOf "capability is not registered") (const False)
|
||||
it "evaluates correlated DISTINCT subqueries with the outer binding" $ do
|
||||
rowsOf (parseStatement "SELECT u.id FROM users u WHERE u.id IN (SELECT DISTINCT o.user_id FROM orders o WHERE o.user_id = u.id)" >>= runStatement testDB)
|
||||
`shouldBe` Right [[("u.id", VInt 1)], [("u.id", VInt 2)]]
|
||||
it "rejects sorting by values excluded from the distinct output" $ do
|
||||
let result = parseStatement "SELECT DISTINCT user_id FROM orders ORDER BY product" >>= runStatement testDB
|
||||
result `shouldSatisfy` either (isInfixOf "must appear in the select list") (const False)
|
||||
|
||||
describe "Storage read failures" $ do
|
||||
it "rejects storage that silently ignores a bound read guard" $ do
|
||||
setStorageLink StorageLink
|
||||
{ slRequest = \_ -> pure (Right (BSC.pack "{\"status\":\"rows\",\"rows\":[]}"))
|
||||
, slClose = pure ()
|
||||
}
|
||||
(runIPCStorage (withSchemaGuard [] (scan "ignored"))
|
||||
`shouldReturn` Left "storage did not confirm bound schema guard") `finally` closeConnection
|
||||
it "propagates catalog failures through snapshot, schema and SQL" $ do
|
||||
setStorageLink StorageLink
|
||||
{ slRequest = \_ -> pure (Right (BSC.pack "{\"status\":\"error\",\"message\":\"catalog unavailable\"}"))
|
||||
@@ -729,7 +797,7 @@ main = hspec $ do
|
||||
|
||||
it "rewrites >= and <= into an inclusive index range" $ do
|
||||
let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))]
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
|
||||
indexed = dbWith (stat 10000 [("id", 10000)] ["id"])
|
||||
low = Filter (GtE (Col "id") (LitInt 2)) (Scan Nothing "users" Nothing)
|
||||
both = Filter (And (GtE (Col "id") (LitInt 1)) (LtE (Col "id") (LitInt 3))) (Scan Nothing "users" Nothing)
|
||||
@@ -1798,7 +1866,7 @@ main = hspec $ do
|
||||
let buckets = replicate 8 63 ++ replicate 8 62
|
||||
withHist = TableMeta 1000 [("id", 1000, False)] [("id", Histogram 0 1600 buckets)] ["id"]
|
||||
plain = TableMeta 1000 [("id", 1000, False)] [] ["id"]
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))]
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
|
||||
narrow = Filter (And (GtE (Col "id") (LitInt 10)) (LtE (Col "id") (LitInt 20))) (Scan Nothing "users" Nothing)
|
||||
wide = Filter (Gt (Col "id") (LitInt 0)) (Scan Nothing "users" Nothing)
|
||||
-- 直方图说这个窄范围只命中十几行:改写
|
||||
@@ -1884,8 +1952,8 @@ main = hspec $ do
|
||||
|
||||
it "picks the index only when the statistics make it cheaper" $ do
|
||||
let stat rows distinct indexes = TableMeta rows [(c, d, False) | (c, d) <- distinct] [] indexes
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m))]
|
||||
unknown = [("users", Table "users" [("id", TInt), ("name", TStr)] [] Nothing)]
|
||||
dbWith m = [("users", Table "users" [("id", TInt), ("name", TStr)] [] (Just m) Nothing Nothing)]
|
||||
unknown = [("users", Table "users" [("id", TInt), ("name", TStr)] [] Nothing Nothing Nothing)]
|
||||
point = Filter (Eq (Col "id") (LitInt 1)) (Scan Nothing "users" Nothing)
|
||||
range = Filter (Gt (Col "id") (LitInt 5)) (Scan Nothing "users" Nothing)
|
||||
-- 索引在、点查命中一行:改写
|
||||
@@ -2332,6 +2400,51 @@ main = hspec $ do
|
||||
map fst db `shouldBe` ["a", "b"]
|
||||
map (length . tableRows . snd) db `shouldBe` [1, 1]
|
||||
map (map (lookup "id") . tableRows . snd) db `shouldBe` [[Just (VInt 1)], [Just (VInt 2)]]
|
||||
it "rejects a bound query after a same-schema persistent table recreation" $ do
|
||||
withTestSession $ \srv -> withServerEnv srv $ do
|
||||
runIPCStorage (createTable "rebuilt" [("id", TInt)]) `shouldReturn` Right ()
|
||||
Right catalog <- runIPCStorage schema
|
||||
let bound = parseStatement "SELECT id FROM rebuilt" >>= Bound.bindQuery catalog
|
||||
case bound of
|
||||
Left err -> expectationFailure err
|
||||
Right query -> do
|
||||
runIPCStorage (Bound.executeQueryM query) `shouldReturn` Right []
|
||||
runIPCStorage (dropTable "rebuilt") `shouldReturn` Right ()
|
||||
runIPCStorage (createTable "rebuilt" [("id", TInt)]) `shouldReturn` Right ()
|
||||
changed <- runIPCStorage (Bound.executeQueryM query)
|
||||
changed `shouldSatisfy` either (isInfixOf "bound query object changed") (const False)
|
||||
it "preserves persistent versions in snapshots and rejects restored schema" $ do
|
||||
withTestSession $ \srv -> withServerEnv srv $ do
|
||||
runIPCStorage (createTable "versioned" [("id", TInt)]) `shouldReturn` Right ()
|
||||
Right catalog <- runIPCStorage schema
|
||||
Right contents <- runIPCStorage snapshot
|
||||
fmap tableSchemaVersion (lookup "versioned" catalog) `shouldBe` Just (Just 1)
|
||||
fmap tableSchemaVersion (lookup "versioned" contents) `shouldBe` Just (Just 1)
|
||||
case parseStatement "SELECT id FROM versioned" >>= Bound.bindQuery catalog of
|
||||
Left err -> expectationFailure err
|
||||
Right query -> do
|
||||
runIPCStorage (replaceSchema "versioned" [("id", TInt), ("extra", TStr)] []) `shouldReturn` Right ()
|
||||
runIPCStorage (replaceSchema "versioned" [("id", TInt)] []) `shouldReturn` Right ()
|
||||
changed <- runIPCStorage (Bound.executeQueryM query)
|
||||
changed `shouldSatisfy` either (isInfixOf "bound query schema changed") (const False)
|
||||
it "rejects a DDL race between schema validation and the actual read" $ do
|
||||
withTestSession $ \srv -> withServerEnv srv $ do
|
||||
runIPCStorage (createTable "raced" [("id", TInt)]) `shouldReturn` Right ()
|
||||
Right catalog <- runIPCStorage schema
|
||||
closeConnection
|
||||
Right link <- localStorageLink (Just (dataDir srv ++ ".toml"))
|
||||
setStorageLink link {slRequest = \payload -> do
|
||||
if BSC.pack "schema_guard" `BS.isInfixOf` payload
|
||||
then do
|
||||
changed <- slRequest link (BSC.pack "{\"method\":\"replace_schema\",\"database\":\"ipctest\",\"table\":\"raced\",\"columns\":[{\"name\":\"id\",\"ty\":\"int\"},{\"name\":\"extra\",\"ty\":\"str\"}],\"rows\":[]}")
|
||||
changed `shouldSatisfy` either (const False) (BS.isInfixOf (BSC.pack "ok"))
|
||||
else pure ()
|
||||
slRequest link payload}
|
||||
case parseStatement "SELECT id FROM raced" >>= Bound.bindQuery catalog of
|
||||
Left err -> expectationFailure err
|
||||
Right query -> do
|
||||
result <- runIPCStorage (Bound.executeQueryM query)
|
||||
result `shouldSatisfy` either (isInfixOf "bound query schema changed") (const False)
|
||||
|
||||
it "doListTables returns the tables that exist" $ do
|
||||
withTestSession $ \srv -> withServerEnv srv $ do
|
||||
@@ -2529,7 +2642,7 @@ main = hspec $ do
|
||||
`shouldBe` Right (makeSelect ["*"] "t" (Just (Eq (Col "flag") (LitBool True))))
|
||||
|
||||
it "inserts and reads back a boolean column end-to-end" $ do
|
||||
let db = [("t", Table "t" [("id", TInt), ("flag", TBool)] [] Nothing)]
|
||||
let db = [("t", Table "t" [("id", TInt), ("flag", TBool)] [] Nothing Nothing Nothing)]
|
||||
case runStatement db (Insert "t" ["id", "flag"] [[LitInt 1, LitBool True]]) of
|
||||
Left e -> expectationFailure e
|
||||
Right (db', _) ->
|
||||
@@ -3154,6 +3267,8 @@ instance Monad ProbeStorage where
|
||||
ProbeStorage (\probe -> act probe >>= \value -> runProbeStorage (next value) probe)
|
||||
|
||||
instance MonadStorage ProbeStorage where
|
||||
-- \| 测试视图不支持并发结构变更
|
||||
withSchemaGuard _ action = action
|
||||
scan table = ProbeStorage $ \probe -> do
|
||||
recordCall probe ("scan:" ++ table)
|
||||
pure (Right (tableRows (probeTable probe)))
|
||||
@@ -3227,7 +3342,10 @@ planTables op = case op of
|
||||
Project _ x -> planTables x
|
||||
Compute _ x -> planTables x
|
||||
Aggregate _ _ x -> planTables x
|
||||
BoundAggregate _ _ x -> planTables x
|
||||
Distinct x -> planTables x
|
||||
Sort _ x -> planTables x
|
||||
BoundSort _ x -> planTables x
|
||||
Limit _ x -> planTables x
|
||||
Join _ l r _ -> planTables l ++ planTables r
|
||||
Derived _ x -> planTables x
|
||||
@@ -3239,8 +3357,11 @@ leftJoinKinds (Join kind l r _) = (kind == LeftJoin) : (leftJoinKinds l ++ leftJ
|
||||
leftJoinKinds (Project _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Compute _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Aggregate _ _ x) = leftJoinKinds x
|
||||
leftJoinKinds (BoundAggregate _ _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Filter _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Distinct x) = leftJoinKinds x
|
||||
leftJoinKinds (Sort _ x) = leftJoinKinds x
|
||||
leftJoinKinds (BoundSort _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Limit _ x) = leftJoinKinds x
|
||||
leftJoinKinds (Derived _ x) = leftJoinKinds x
|
||||
leftJoinKinds _ = []
|
||||
@@ -3251,8 +3372,11 @@ firstFilterCond (Filter p _) = Just p
|
||||
firstFilterCond (Project _ x) = firstFilterCond x
|
||||
firstFilterCond (Compute _ x) = firstFilterCond x
|
||||
firstFilterCond (Aggregate _ _ x) = firstFilterCond x
|
||||
firstFilterCond (BoundAggregate _ _ x) = firstFilterCond x
|
||||
firstFilterCond Unit = Nothing
|
||||
firstFilterCond (Distinct x) = firstFilterCond x
|
||||
firstFilterCond (Sort _ x) = firstFilterCond x
|
||||
firstFilterCond (BoundSort _ x) = firstFilterCond x
|
||||
firstFilterCond (Limit _ x) = firstFilterCond x
|
||||
firstFilterCond (Join _ l r _) = case firstFilterCond l of
|
||||
Just p -> Just p
|
||||
@@ -3268,8 +3392,11 @@ anyFilter (Filter _ _) = True
|
||||
anyFilter (Project _ x) = anyFilter x
|
||||
anyFilter (Compute _ x) = anyFilter x
|
||||
anyFilter (Aggregate _ _ x) = anyFilter x
|
||||
anyFilter (BoundAggregate _ _ x) = anyFilter x
|
||||
anyFilter Unit = False
|
||||
anyFilter (Distinct x) = anyFilter x
|
||||
anyFilter (Sort _ x) = anyFilter x
|
||||
anyFilter (BoundSort _ x) = anyFilter x
|
||||
anyFilter (Limit _ x) = anyFilter x
|
||||
anyFilter (Join _ l r _) = anyFilter l || anyFilter r
|
||||
anyFilter (Scan _ _ _) = False
|
||||
@@ -3282,9 +3409,12 @@ stripProjects :: RelOp -> RelOp
|
||||
stripProjects (Project _ x) = stripProjects x
|
||||
stripProjects (Compute items x) = Compute items (stripProjects x)
|
||||
stripProjects (Aggregate keys aggs x) = Aggregate keys aggs (stripProjects x)
|
||||
stripProjects (BoundAggregate keys aggs x) = BoundAggregate keys aggs (stripProjects x)
|
||||
stripProjects Unit = Unit
|
||||
stripProjects (Filter p x) = Filter p (stripProjects x)
|
||||
stripProjects (Distinct x) = Distinct (stripProjects x)
|
||||
stripProjects (Sort spec x) = Sort spec (stripProjects x)
|
||||
stripProjects (BoundSort spec x) = BoundSort spec (stripProjects x)
|
||||
stripProjects (Limit n x) = Limit n (stripProjects x)
|
||||
stripProjects (Join k l r c) = Join k (stripProjects l) (stripProjects r) c
|
||||
stripProjects (Scan a t _) = Scan a t Nothing
|
||||
@@ -3379,6 +3509,8 @@ memReplaceSchema :: String -> [(String, Column)] -> [Row] -> MemoryStorage (Eith
|
||||
memReplaceSchema = replaceSchema
|
||||
|
||||
instance MonadStorage Counting where
|
||||
-- \| 纯测试状态在求值期间保持稳定
|
||||
withSchemaGuard _ action = action
|
||||
-- \| 扫描:记一笔表名,其余走内存实现
|
||||
scan t = Counting $ \db -> case runMemoryStorage (memScan t) db of
|
||||
Left e -> Left e
|
||||
|
||||
@@ -12,6 +12,7 @@ module ChuSQL.Core.Model
|
||||
, TableMeta (..)
|
||||
, Histogram (..)
|
||||
, Table (..)
|
||||
, TableIdentity (..)
|
||||
, Database
|
||||
, histogramOf
|
||||
, histogramFor
|
||||
@@ -48,6 +49,7 @@ import Data.Char (isDigit, isSpace, toLower)
|
||||
import Data.Hashable (Hashable (..))
|
||||
import Data.List (isPrefixOf, isSuffixOf, stripPrefix)
|
||||
import Data.Maybe (fromMaybe, mapMaybe)
|
||||
import Data.Word (Word64)
|
||||
import ChuSQL.Core.Runtime
|
||||
import qualified Data.ByteString.Lazy.Char8 as BS
|
||||
|
||||
@@ -556,12 +558,17 @@ data TableMeta = TableMeta
|
||||
}
|
||||
deriving (Show)
|
||||
|
||||
-- | 一张表:名字、列定义与全部行;meta 为存储层统计,拿不到是 Nothing
|
||||
-- | 保存持久化数据库与表编号
|
||||
data TableIdentity = TableIdentity Word64 Word64 deriving (Show, Eq)
|
||||
|
||||
-- | 一张表:列、行、统计与独立对象身份
|
||||
data Table = Table
|
||||
{ tableName :: String
|
||||
, tableCols :: [(String, Column)]
|
||||
, tableRows :: [Row]
|
||||
, tableMeta :: Maybe TableMeta
|
||||
, tableIdentity :: Maybe TableIdentity
|
||||
, tableSchemaVersion :: Maybe Word64
|
||||
}
|
||||
deriving (Show)
|
||||
|
||||
|
||||
@@ -15,16 +15,18 @@ module ChuSQL.Core.Protocol (
|
||||
queryResultJson,
|
||||
) where
|
||||
|
||||
import ChuSQL.Core.Model (Histogram (..), Row, Value (..))
|
||||
import ChuSQL.Core.Model (Histogram (..), Row, Value (..), TableIdentity (..))
|
||||
import ChuSQL.Core.Runtime (builtinTypes, runtimeToJSON, runtimeFromJSON)
|
||||
import Data.Aeson (FromJSON (..), ToJSON (..), object, withObject, (.:), (.:?), (.!=), (.=))
|
||||
import qualified Data.Aeson as A
|
||||
import qualified Data.Aeson.Key as K
|
||||
import qualified Data.Aeson.KeyMap as KM
|
||||
import Data.Aeson.Types (Parser)
|
||||
import Data.Scientific (floatingOrInteger, fromFloatDigits)
|
||||
import Data.Scientific (floatingOrInteger, fromFloatDigits, base10Exponent)
|
||||
import qualified Data.Text as T
|
||||
import Data.Text (Text)
|
||||
import Data.Word (Word64)
|
||||
import Text.Read (readMaybe)
|
||||
|
||||
-- 线上协议类型:存储层请求/响应与查询结果集。
|
||||
|
||||
@@ -255,6 +257,8 @@ data TableInfo = TableInfo
|
||||
, tiIndexes :: [String]
|
||||
, tiStats :: [(String, Int, Bool)]
|
||||
, tiHistograms :: [(String, Histogram)]
|
||||
, tiIdentity :: Maybe TableIdentity
|
||||
, tiSchemaVersion :: Maybe Word64
|
||||
}
|
||||
deriving (Show, Eq)
|
||||
|
||||
@@ -267,6 +271,9 @@ instance ToJSON TableInfo where
|
||||
, "row_count" .= tiRows info
|
||||
, "indexes" .= map (\name -> object ["column" .= name]) (tiIndexes info)
|
||||
, "stats" .= map (statJson (tiHistograms info)) (tiStats info)
|
||||
, "database_id" .= fmap (\(TableIdentity database _) -> show database) (tiIdentity info)
|
||||
, "object_id" .= fmap (\(TableIdentity _ table) -> show table) (tiIdentity info)
|
||||
, "schema_version" .= fmap show (tiSchemaVersion info)
|
||||
]
|
||||
where
|
||||
statJson hists (name, distinct, capped) =
|
||||
@@ -276,6 +283,16 @@ instance ToJSON TableInfo where
|
||||
histFields Nothing = []
|
||||
histFields (Just h) = ["lo" .= histLow h, "hi" .= histHigh h, "hist" .= histBuckets h]
|
||||
|
||||
-- | 解码精确且有界的对象编号
|
||||
parseObjectNumber :: A.Value -> Parser Word64
|
||||
parseObjectNumber (A.String value)
|
||||
| not (T.null value), T.length value <= 20, T.all (\c -> c >= '0' && c <= '9') value
|
||||
, Just number <- (readMaybe (T.unpack value) :: Maybe Integer)
|
||||
, number <= toInteger (maxBound :: Word64) = pure (fromInteger number)
|
||||
parseObjectNumber value@(A.Number number)
|
||||
| base10Exponent number >= -20, base10Exponent number <= 20 = A.parseJSON value
|
||||
parseObjectNumber _ = fail "invalid table object identity"
|
||||
|
||||
-- | 线上直方图还原;区间或桶缺了就当作没有
|
||||
histogramFromWire :: Maybe Double -> Maybe Double -> [Int] -> Maybe Histogram
|
||||
histogramFromWire (Just low) (Just high) counts
|
||||
@@ -292,9 +309,18 @@ instance FromJSON TableInfo where
|
||||
stats <- o .:? "stats" .!= []
|
||||
indexColumns <- mapM (\v -> withObject "IndexWire" (.: "column") v) (indexes :: [A.Value])
|
||||
parsed <- mapM parseStat (stats :: [A.Value])
|
||||
database <- o .:? "database_id" >>= traverse parseObjectNumber
|
||||
tableId <- o .:? "object_id" >>= traverse parseObjectNumber
|
||||
version <- o .:? "schema_version" >>= traverse parseObjectNumber
|
||||
identity <- case (database, tableId) of
|
||||
(Nothing, Nothing) -> pure Nothing
|
||||
(Just _, Just 0) -> pure Nothing
|
||||
(Just databaseId, Just objectId) | objectId > 0 -> pure (Just (TableIdentity databaseId objectId))
|
||||
_ -> fail ("incomplete table object identity: " ++ show (database, tableId))
|
||||
let parsedStats = [(name, distinct, capped) | (name, distinct, capped, _) <- parsed]
|
||||
parsedHistograms = [(name, h) | (name, _, _, Just h) <- parsed]
|
||||
pure (TableInfo table columns count indexColumns parsedStats parsedHistograms)
|
||||
pure (TableInfo table columns count indexColumns parsedStats parsedHistograms identity
|
||||
(version >>= \v -> if v == 0 then Nothing else Just v))
|
||||
where
|
||||
parseStat = withObject "ColumnStat" $ \o -> do
|
||||
name <- o .: "name"
|
||||
|
||||
@@ -37,6 +37,9 @@ pub struct TableSchema {
|
||||
#[serde(default)]
|
||||
pub object_id: u64,
|
||||
#[serde(default)]
|
||||
// 0 表示旧目录尚无可验证的结构版本。
|
||||
pub schema_version: u64,
|
||||
#[serde(default)]
|
||||
pub system: bool,
|
||||
pub columns: Vec<SchemaColumn>,
|
||||
#[serde(default)]
|
||||
@@ -51,7 +54,7 @@ pub struct TableSchema {
|
||||
pub stats: BTreeMap<String, ColumnStat>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, Serialize, Deserialize)]
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
pub struct Catalog {
|
||||
#[serde(default)]
|
||||
pub database_id: u64,
|
||||
@@ -134,9 +137,10 @@ impl Catalog {
|
||||
}
|
||||
|
||||
/// 按一行数据补列
|
||||
pub fn ensure_columns(&mut self, table: &str, row: &Row) {
|
||||
pub fn ensure_columns(&mut self, table: &str, row: &Row) -> io::Result<()> {
|
||||
let entry = self.tables.entry(table.to_string()).or_default();
|
||||
add_missing_columns(entry, row);
|
||||
add_missing_columns(entry, row)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 建表;已存在报错
|
||||
@@ -163,6 +167,7 @@ impl Catalog {
|
||||
table.to_string(),
|
||||
TableSchema {
|
||||
object_id,
|
||||
schema_version: 1,
|
||||
system: false,
|
||||
columns,
|
||||
dropped_columns: BTreeSet::new(),
|
||||
@@ -184,15 +189,15 @@ impl Catalog {
|
||||
}
|
||||
|
||||
/// 记一次插入,返回列有没有变化
|
||||
pub fn record_insert(&mut self, table: &str, row: &Row) -> bool {
|
||||
pub fn record_insert(&mut self, table: &str, row: &Row) -> io::Result<bool> {
|
||||
let changed = {
|
||||
let entry = self.tables.entry(table.to_string()).or_default();
|
||||
let changed = add_missing_columns(entry, row);
|
||||
let changed = add_missing_columns(entry, row)?;
|
||||
entry.row_count += 1;
|
||||
changed
|
||||
};
|
||||
self.count_values(table, row);
|
||||
changed
|
||||
Ok(changed)
|
||||
}
|
||||
|
||||
/// 记一次删除:行数减少 + 更新列统计
|
||||
@@ -230,7 +235,7 @@ impl Catalog {
|
||||
}
|
||||
|
||||
/// 重数一遍统计与行数
|
||||
pub fn rebuild_stats(&mut self, table: &str, rows: &[Row]) -> bool {
|
||||
pub fn rebuild_stats(&mut self, table: &str, rows: &[Row]) -> io::Result<bool> {
|
||||
self.seen.remove(table);
|
||||
{
|
||||
let entry = self.tables.entry(table.to_string()).or_default();
|
||||
@@ -251,9 +256,9 @@ impl Catalog {
|
||||
}
|
||||
let mut changed = false;
|
||||
for row in rows {
|
||||
changed = self.record_insert(table, row) || changed;
|
||||
changed = self.record_insert(table, row)? || changed;
|
||||
}
|
||||
changed
|
||||
Ok(changed)
|
||||
}
|
||||
|
||||
/// 这一列第一次见到这个值就加一
|
||||
@@ -291,16 +296,18 @@ impl Catalog {
|
||||
|
||||
/// 表改名到指定名字
|
||||
pub fn rename_table(&mut self, from: &str, to: &str) -> io::Result<bool> {
|
||||
let Some(entry) = self.tables.remove(from) else {
|
||||
let Some(entry) = self.tables.get(from) else {
|
||||
return Ok(false);
|
||||
};
|
||||
if self.tables.contains_key(to) {
|
||||
self.tables.insert(from.to_string(), entry);
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::AlreadyExists,
|
||||
format!("table already exists: {}", to),
|
||||
));
|
||||
}
|
||||
let version = next_schema_version(entry)?;
|
||||
let mut entry = self.tables.remove(from).ok_or_else(|| io::Error::other("missing rename source"))?;
|
||||
entry.schema_version = version;
|
||||
self.seen.remove(from);
|
||||
self.tables.insert(to.to_string(), entry);
|
||||
Ok(true)
|
||||
@@ -308,10 +315,21 @@ impl Catalog {
|
||||
|
||||
/// ALTER:整列定义换成新的,并清掉二级索引与统计
|
||||
pub fn set_columns(&mut self, table: &str, columns: Vec<SchemaColumn>) -> io::Result<()> {
|
||||
self.set_columns_version(table, columns, None)
|
||||
}
|
||||
|
||||
/// 用日志中的目标版本替换列定义
|
||||
pub fn set_columns_version(&mut self, table: &str, columns: Vec<SchemaColumn>, version: Option<u64>) -> io::Result<()> {
|
||||
let entry = self
|
||||
.tables
|
||||
.get_mut(table)
|
||||
.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?;
|
||||
if let Some(version) = version {
|
||||
if version == 0 { return Err(io::Error::other("invalid WAL schema version")); }
|
||||
entry.schema_version = version;
|
||||
} else if entry.columns != columns {
|
||||
entry.schema_version = next_schema_version(entry)?;
|
||||
}
|
||||
entry.columns = columns;
|
||||
entry.dropped_columns.clear();
|
||||
entry.indexes.clear();
|
||||
@@ -321,9 +339,12 @@ impl Catalog {
|
||||
}
|
||||
|
||||
/// 老数据的列类型名规范化;返回有没有改动
|
||||
pub fn normalize_types(&mut self) -> bool {
|
||||
pub fn normalize_types(&mut self) -> io::Result<bool> {
|
||||
let mut changed = false;
|
||||
for entry in self.tables.values_mut() {
|
||||
if entry.columns.iter().any(|column| normalize_type(&column.ty) != column.ty) {
|
||||
entry.schema_version = next_schema_version(entry)?;
|
||||
}
|
||||
for column in entry.columns.iter_mut() {
|
||||
let fixed = normalize_type(&column.ty);
|
||||
if fixed != column.ty {
|
||||
@@ -332,15 +353,26 @@ impl Catalog {
|
||||
}
|
||||
}
|
||||
}
|
||||
changed
|
||||
Ok(changed)
|
||||
}
|
||||
|
||||
/// 删一列;之前没有也返回成功(重放幂等)
|
||||
pub fn remove_column(&mut self, table: &str, column: &str) -> io::Result<bool> {
|
||||
self.remove_column_version(table, column, None)
|
||||
}
|
||||
|
||||
/// 用日志目标版本删除列定义
|
||||
pub fn remove_column_version(&mut self, table: &str, column: &str, version: Option<u64>) -> io::Result<bool> {
|
||||
let entry = self
|
||||
.tables
|
||||
.get_mut(table)
|
||||
.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, format!("unknown table: {}", table)))?;
|
||||
if let Some(version) = version {
|
||||
if version == 0 { return Err(io::Error::other("invalid WAL schema version")); }
|
||||
entry.schema_version = version;
|
||||
} else if entry.columns.iter().any(|c| c.name == column) {
|
||||
entry.schema_version = next_schema_version(entry)?;
|
||||
}
|
||||
let had_column = {
|
||||
let before = entry.columns.len();
|
||||
entry.columns.retain(|c| c.name != column);
|
||||
@@ -417,22 +449,24 @@ impl Catalog {
|
||||
}
|
||||
|
||||
/// 把行里"字典还不知道的列"补进来;返回有没有变化
|
||||
fn add_missing_columns(entry: &mut TableSchema, row: &Row) -> bool {
|
||||
let mut changed = false;
|
||||
for (k, v) in row {
|
||||
if entry.dropped_columns.contains(k) || entry.columns.iter().any(|c| &c.name == k) {
|
||||
continue;
|
||||
}
|
||||
if let Some(ty) = infer_type(v) {
|
||||
entry.columns.push(SchemaColumn {
|
||||
name: k.clone(),
|
||||
ty,
|
||||
..Default::default()
|
||||
});
|
||||
changed = true;
|
||||
}
|
||||
fn add_missing_columns(entry: &mut TableSchema, row: &Row) -> io::Result<bool> {
|
||||
let additions: Vec<_> = row.iter().filter(|(name, _)|
|
||||
!entry.dropped_columns.contains(*name) && !entry.columns.iter().any(|c| &c.name == *name))
|
||||
.filter_map(|(name, value)| infer_type(value).map(|ty| SchemaColumn {
|
||||
name: name.clone(), ty, ..Default::default()
|
||||
})).collect();
|
||||
if additions.is_empty() {
|
||||
return Ok(false);
|
||||
}
|
||||
changed
|
||||
entry.schema_version = next_schema_version(entry)?;
|
||||
entry.columns.extend(additions);
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// 递增结构版本,耗尽时拒绝变更
|
||||
fn next_schema_version(entry: &TableSchema) -> io::Result<u64> {
|
||||
entry.schema_version.checked_add(1)
|
||||
.ok_or_else(|| io::Error::other("catalog schema version exhausted"))
|
||||
}
|
||||
|
||||
/// 由 JSON 值猜列类型
|
||||
|
||||
@@ -74,14 +74,19 @@ pub unsafe extern "C" fn chusql_storage_close(handle: *mut Storage) {
|
||||
/// 离线维护系统目录并返回 JSON
|
||||
///
|
||||
/// # Safety
|
||||
/// 输入为有效字符串;输出指针可写,响应使用 free 释放。
|
||||
/// `config_path` 为 NULL 或有效 NUL 结尾字符串;请求字节在调用期间有效。
|
||||
/// 输出指针可写且不与输入重叠;响应使用 free 释放。
|
||||
#[unsafe(no_mangle)]
|
||||
pub unsafe extern "C" fn chusql_storage_maintenance(
|
||||
config_path: *const c_char, request: *const u8, request_len: usize,
|
||||
out: *mut *mut u8, out_len: *mut usize,
|
||||
) -> i32 {
|
||||
if request.is_null() || out.is_null() || out_len.is_null() { set_error("invalid maintenance argument"); return ERR; }
|
||||
unsafe { *out = ptr::null_mut(); *out_len = 0; }
|
||||
if !out.is_null() { unsafe { *out = ptr::null_mut(); } }
|
||||
if !out_len.is_null() { unsafe { *out_len = 0; } }
|
||||
if request.is_null() || out.is_null() || out_len.is_null() || request_len > isize::MAX as usize {
|
||||
set_error("invalid maintenance argument");
|
||||
return ERR;
|
||||
}
|
||||
let result = catch_unwind(AssertUnwindSafe(|| {
|
||||
let path = if config_path.is_null() { None } else { Some(unsafe { CStr::from_ptr(config_path) }.to_str().map_err(|e| e.to_string())?) };
|
||||
let text = std::str::from_utf8(unsafe { slice::from_raw_parts(request, request_len) }).map_err(|e| e.to_string())?;
|
||||
@@ -92,16 +97,16 @@ pub unsafe extern "C" fn chusql_storage_maintenance(
|
||||
Ok(Err(err)) => { set_error(err); return ERR; }
|
||||
Err(_) => { set_error("panic in system maintenance"); return ERR; }
|
||||
};
|
||||
let mut buffer = response.into_bytes().into_boxed_slice();
|
||||
unsafe { *out = buffer.as_mut_ptr(); *out_len = buffer.len(); }
|
||||
std::mem::forget(buffer);
|
||||
let buffer = response.into_bytes().into_boxed_slice();
|
||||
unsafe { *out_len = buffer.len(); *out = Box::into_raw(buffer) as *mut u8; }
|
||||
OK
|
||||
}
|
||||
|
||||
/// 处理一条请求,响应缓冲须归还
|
||||
///
|
||||
/// # Safety
|
||||
/// `handle` 必须是有效句柄;`request` 指向 `request_len` 个字节;`out`/`out_len` 可写。
|
||||
/// `handle` 在调用期间有效且不能并发关闭;请求字节在调用期间有效。
|
||||
/// `out`/`out_len` 可写且不与输入重叠;响应使用 free 释放。
|
||||
#[unsafe(no_mangle)]
|
||||
pub unsafe extern "C" fn chusql_storage_request(
|
||||
handle: *const Storage,
|
||||
@@ -116,7 +121,7 @@ pub unsafe extern "C" fn chusql_storage_request(
|
||||
if !out_len.is_null() {
|
||||
unsafe { *out_len = 0 };
|
||||
}
|
||||
if handle.is_null() || request.is_null() || out.is_null() || out_len.is_null() {
|
||||
if handle.is_null() || request.is_null() || out.is_null() || out_len.is_null() || request_len > isize::MAX as usize {
|
||||
set_error("invalid argument");
|
||||
return ERR;
|
||||
}
|
||||
@@ -139,13 +144,10 @@ pub unsafe extern "C" fn chusql_storage_request(
|
||||
}
|
||||
};
|
||||
|
||||
let mut buffer = response.into_bytes().into_boxed_slice();
|
||||
let ptr = buffer.as_mut_ptr();
|
||||
let len = buffer.len();
|
||||
std::mem::forget(buffer);
|
||||
let buffer = response.into_bytes().into_boxed_slice();
|
||||
unsafe {
|
||||
*out = ptr;
|
||||
*out_len = len;
|
||||
*out_len = buffer.len();
|
||||
*out = Box::into_raw(buffer) as *mut u8;
|
||||
}
|
||||
OK
|
||||
}
|
||||
@@ -153,7 +155,7 @@ pub unsafe extern "C" fn chusql_storage_request(
|
||||
/// 归还响应缓冲
|
||||
///
|
||||
/// # Safety
|
||||
/// `data`/`len` 必须是 chusql_storage_request 原样返回且未被释放过的一对值。
|
||||
/// `data`/`len` 为 request 或 maintenance 原样返回且未释放的一对值。
|
||||
#[unsafe(no_mangle)]
|
||||
pub unsafe extern "C" fn chusql_storage_free(data: *mut u8, len: usize) {
|
||||
if data.is_null() {
|
||||
@@ -168,7 +170,7 @@ pub unsafe extern "C" fn chusql_storage_free(data: *mut u8, len: usize) {
|
||||
/// 取本线程最近的失败信息
|
||||
///
|
||||
/// # Safety
|
||||
/// `out_len` 可写,或为 NULL。
|
||||
/// `out_len` 可写,或为 NULL;返回字节仅在本线程下次设置错误前有效,不能释放。
|
||||
#[unsafe(no_mangle)]
|
||||
pub unsafe extern "C" fn chusql_storage_last_error(out_len: *mut usize) -> *const u8 {
|
||||
LAST_ERROR.with(|slot| {
|
||||
|
||||
@@ -283,12 +283,15 @@ fn default_id_column() -> String {
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
#[serde(tag = "status", rename_all = "snake_case")]
|
||||
pub enum Response {
|
||||
Guarded { response: Box<Response> },
|
||||
Accounts { accounts: Vec<Account> },
|
||||
Pong,
|
||||
Rows { rows: Vec<Row> },
|
||||
Tables { tables: Vec<String> },
|
||||
Schema {
|
||||
table: String,
|
||||
#[serde(default, serialize_with = "serialize_version", deserialize_with = "deserialize_version")]
|
||||
schema_version: u64,
|
||||
columns: Vec<SchemaColumn>,
|
||||
#[serde(default)]
|
||||
row_count: u64,
|
||||
@@ -310,6 +313,15 @@ pub enum Response {
|
||||
Object { database_id: u64, object_id: Option<u64> },
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct BoundSchema {
|
||||
pub table: String,
|
||||
pub database_id: Option<String>,
|
||||
pub object_id: Option<String>,
|
||||
pub schema_version: Option<String>,
|
||||
pub columns: Vec<SchemaColumn>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TableSchemaWire {
|
||||
pub table: String,
|
||||
@@ -317,6 +329,8 @@ pub struct TableSchemaWire {
|
||||
pub database_id: u64,
|
||||
#[serde(default)]
|
||||
pub object_id: u64,
|
||||
#[serde(default, serialize_with = "serialize_version", deserialize_with = "deserialize_version")]
|
||||
pub schema_version: u64,
|
||||
pub columns: Vec<SchemaColumn>,
|
||||
#[serde(default)]
|
||||
pub row_count: u64,
|
||||
@@ -325,3 +339,19 @@ pub struct TableSchemaWire {
|
||||
#[serde(default)]
|
||||
pub stats: Vec<ColumnStatWire>,
|
||||
}
|
||||
|
||||
/// 将结构版本编码成精确十进制字符串
|
||||
fn serialize_version<S: serde::Serializer>(version: &u64, serializer: S) -> Result<S::Ok, S::Error> {
|
||||
serializer.serialize_str(&version.to_string())
|
||||
}
|
||||
|
||||
/// 读取精确版本并兼容旧数值载荷
|
||||
fn deserialize_version<'de, D: serde::Deserializer<'de>>(deserializer: D) -> Result<u64, D::Error> {
|
||||
let value = serde_json::Value::deserialize(deserializer)?;
|
||||
match value {
|
||||
serde_json::Value::String(s) if !s.is_empty() && s.len() <= 20 && s.bytes().all(|c| c.is_ascii_digit()) =>
|
||||
s.parse().map_err(serde::de::Error::custom),
|
||||
serde_json::Value::Number(n) => n.as_u64().ok_or_else(|| serde::de::Error::custom("invalid schema version")),
|
||||
_ => Err(serde::de::Error::custom("invalid schema version")),
|
||||
}
|
||||
}
|
||||
@@ -236,9 +236,9 @@ impl Server {
|
||||
fn apply_op(&self, op: &WalOp) -> std::io::Result<()> {
|
||||
match op {
|
||||
WalOp::Commit => Ok(()),
|
||||
WalOp::HideColumn { table, column } => {
|
||||
WalOp::HideColumn { table, column, schema_version } => {
|
||||
let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
|
||||
c.remove_column(table, column)?;
|
||||
c.remove_column_version(table, column, *schema_version)?;
|
||||
c.save(self.catalog_path())?;
|
||||
drop(c);
|
||||
if let Some(handle) = self.tables.lock().map_err(|_| std::io::Error::other("table lock poisoned"))?.get(table) {
|
||||
@@ -257,7 +257,7 @@ impl Server {
|
||||
})?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
c.mark_compacted(table)?;
|
||||
c.rebuild_stats(table, rows);
|
||||
c.rebuild_stats(table, rows)?;
|
||||
c.save(self.catalog_path())
|
||||
}
|
||||
WalOp::Accounts { accounts, removed, migrate_roles } => {
|
||||
@@ -287,20 +287,20 @@ impl Server {
|
||||
}
|
||||
let mut c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
|
||||
declare_system_schema(&mut c, &rows)?;
|
||||
c.rebuild_stats(IDENTITIES_TABLE, &identities);
|
||||
c.rebuild_stats(IDENTITIES_TABLE, &identities)?;
|
||||
c.set_columns(IDENTITIES_TABLE, identity_columns())?;
|
||||
c.rebuild_stats(IDENTITIES_TABLE, &identities);
|
||||
c.rebuild_stats(IDENTITIES_TABLE, &identities)?;
|
||||
c.mark_system(IDENTITIES_TABLE)?;
|
||||
c.add_index(IDENTITIES_TABLE, "user")?;
|
||||
c.identity_high_water = c.identity_high_water.max(accounts.iter().map(|a| a.id).max().unwrap_or(0));
|
||||
if has_roles && *migrate_roles { c.rebuild_stats(legacy_roles, &[]); }
|
||||
for (table, records) in cleaned { c.rebuild_stats(table, &records); }
|
||||
if has_roles && *migrate_roles { c.rebuild_stats(legacy_roles, &[])?; }
|
||||
for (table, records) in cleaned { c.rebuild_stats(table, &records)?; }
|
||||
c.save(self.catalog_path())
|
||||
}
|
||||
WalOp::Insert { table, row } => {
|
||||
self.with_table(table, |t| t.insert_row(row))?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
if c.record_insert(table, row) {
|
||||
if c.record_insert(table, row)? {
|
||||
c.save(self.catalog_path())?;
|
||||
}
|
||||
Ok(())
|
||||
@@ -310,7 +310,7 @@ impl Server {
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
let mut changed = false;
|
||||
for r in rows {
|
||||
if c.record_insert(table, r) {
|
||||
if c.record_insert(table, r)? {
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
@@ -328,17 +328,17 @@ impl Server {
|
||||
WalOp::ReplaceAll { table, rows } => {
|
||||
self.with_table(table, |t| t.replace_all(rows))?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
c.rebuild_stats(table, rows);
|
||||
c.rebuild_stats(table, rows)?;
|
||||
c.save(self.catalog_path())
|
||||
}
|
||||
WalOp::DropColumn { table, column, rows } => {
|
||||
WalOp::DropColumn { table, column, rows, schema_version } => {
|
||||
self.with_table(table, |t| {
|
||||
t.detach_index(column);
|
||||
t.replace_all(rows)
|
||||
})?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
c.remove_column(table, column)?;
|
||||
c.rebuild_stats(table, rows);
|
||||
c.remove_column_version(table, column, *schema_version)?;
|
||||
c.rebuild_stats(table, rows)?;
|
||||
c.save(self.catalog_path())?;
|
||||
drop(c);
|
||||
let _ = std::fs::remove_file(self.index_path(table, column));
|
||||
@@ -374,7 +374,7 @@ impl Server {
|
||||
}
|
||||
WalOp::CreateIndex { table, column } => self.apply_create_index(table, column),
|
||||
WalOp::DropIndex { table, column } => self.apply_drop_index(table, column),
|
||||
WalOp::ReplaceSchema { table, columns, rows } => {
|
||||
WalOp::ReplaceSchema { table, columns, rows, schema_version } => {
|
||||
let old_indexes: Vec<String> = {
|
||||
let c = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
|
||||
c.describe(table)
|
||||
@@ -389,8 +389,8 @@ impl Server {
|
||||
t.replace_all(rows)
|
||||
})?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
c.set_columns(table, columns.clone())?;
|
||||
c.rebuild_stats(table, rows);
|
||||
c.set_columns_version(table, columns.clone(), *schema_version)?;
|
||||
c.rebuild_stats(table, rows)?;
|
||||
c.save(self.catalog_path())?;
|
||||
drop(c);
|
||||
for col in old_indexes {
|
||||
@@ -483,7 +483,7 @@ impl Server {
|
||||
match op {
|
||||
WalOp::Insert { table, row } => validate(table, std::slice::from_ref(row))?,
|
||||
WalOp::InsertBatch { table, rows } | WalOp::ReplaceAll { table, rows } => validate(table, rows)?,
|
||||
WalOp::HideColumn { table, column }
|
||||
WalOp::HideColumn { table, column, .. }
|
||||
if !c.describe(table).is_some_and(|s| s.columns.iter().any(|col| &col.name == column)) =>
|
||||
{
|
||||
return Err(std::io::Error::other(format!("unknown column: {column}")));
|
||||
@@ -554,6 +554,29 @@ impl Server {
|
||||
}
|
||||
}
|
||||
}
|
||||
if ops.iter().any(|op| matches!(op, WalOp::HideColumn { .. } | WalOp::DropColumn { .. } | WalOp::ReplaceSchema { .. })) {
|
||||
let mut preview = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?.clone();
|
||||
for op in ops.to_mut() {
|
||||
match op {
|
||||
WalOp::HideColumn { table, column, schema_version }
|
||||
| WalOp::DropColumn { table, column, schema_version, .. } => {
|
||||
preview.remove_column(table, column)?;
|
||||
*schema_version = Some(preview.describe(table).ok_or_else(|| std::io::Error::other("missing schema version"))?.schema_version.max(1));
|
||||
}
|
||||
WalOp::ReplaceSchema { table, columns, schema_version, .. } => {
|
||||
preview.set_columns(table, columns.clone())?;
|
||||
*schema_version = Some(preview.describe(table).ok_or_else(|| std::io::Error::other("missing schema version"))?.schema_version.max(1));
|
||||
}
|
||||
WalOp::CreateTable { table, columns, object_id } => preview.create_table_with_id(table, columns.clone(), *object_id)?,
|
||||
WalOp::DropTable { table } => preview.drop_table(table)?,
|
||||
WalOp::Insert { table, row } => preview.ensure_columns(table, row)?,
|
||||
WalOp::InsertBatch { table, rows } | WalOp::ReplaceAll { table, rows } => {
|
||||
for row in rows { preview.ensure_columns(table, row)?; }
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
let commit_lsn = wal.append_group(&ops)?;
|
||||
self.recovery_required.store(true, Ordering::Release);
|
||||
for op in ops.iter() {
|
||||
@@ -683,9 +706,10 @@ impl Server {
|
||||
});
|
||||
Ok(!present)
|
||||
}
|
||||
WalOp::HideColumn { table, column } => {
|
||||
WalOp::HideColumn { table, column, schema_version } => {
|
||||
let c = self.catalog.lock().unwrap();
|
||||
Ok(c.describe(table).is_some_and(|s| !s.columns.iter().any(|col| &col.name == column)))
|
||||
Ok(c.describe(table).is_some_and(|s| !s.columns.iter().any(|col| &col.name == column)
|
||||
&& schema_version.is_none_or(|version| s.schema_version == version)))
|
||||
}
|
||||
WalOp::CreateTable { table, object_id, .. } => {
|
||||
let catalog = self.catalog.lock().map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
|
||||
@@ -739,7 +763,7 @@ impl Server {
|
||||
}
|
||||
let rows = self.with_table(&table, |t| t.scan())?;
|
||||
let mut c = self.catalog.lock().unwrap();
|
||||
c.rebuild_stats(&table, &rows);
|
||||
c.rebuild_stats(&table, &rows)?;
|
||||
touched = true;
|
||||
}
|
||||
if touched {
|
||||
@@ -791,7 +815,7 @@ impl Server {
|
||||
.catalog
|
||||
.lock()
|
||||
.map_err(|_| std::io::Error::other("catalog lock poisoned"))?;
|
||||
if catalog.normalize_types() {
|
||||
if catalog.normalize_types()? {
|
||||
catalog.save(self.catalog_path())?;
|
||||
}
|
||||
Ok(())
|
||||
@@ -954,6 +978,7 @@ impl Server {
|
||||
let op = WalOp::HideColumn {
|
||||
table: table.to_string(),
|
||||
column: column.to_string(),
|
||||
schema_version: None,
|
||||
};
|
||||
self.write_via_wal(&op)
|
||||
}
|
||||
@@ -1310,6 +1335,32 @@ impl Server {
|
||||
}
|
||||
}
|
||||
|
||||
/// 持读门时复查绑定身份、版本和列定义
|
||||
fn validate_bound_schema(&self, table: &str, expected: &crate::protocol::BoundSchema) -> std::io::Result<()> {
|
||||
use std::io::Error;
|
||||
if self.recovery_required.load(Ordering::Acquire) { return Err(Error::other("storage recovery required")); }
|
||||
if blocked_table(table) { return Err(Error::other("reserved system table")); }
|
||||
let catalog = self.catalog.lock().map_err(|_| Error::other("catalog lock poisoned"))?;
|
||||
let actual = catalog.describe(table).ok_or_else(|| Error::other(format!("bound query schema changed: {}", expected.table)))?;
|
||||
match (&expected.database_id, &expected.object_id) {
|
||||
(Some(database), Some(object)) => {
|
||||
if parse_bound_number(database)? != catalog.database_id || parse_bound_number(object)? != actual.object_id {
|
||||
return Err(Error::other(format!("bound query object changed: {}", expected.table)));
|
||||
}
|
||||
}
|
||||
(None, None) => {}
|
||||
_ => return Err(Error::other("incomplete bound object identity")),
|
||||
}
|
||||
if let Some(version) = &expected.schema_version
|
||||
&& parse_bound_number(version)? != actual.schema_version {
|
||||
return Err(Error::other(format!("bound query schema changed: {}", expected.table)));
|
||||
}
|
||||
if expected.columns != actual.columns {
|
||||
return Err(Error::other(format!("bound query schema changed: {}", expected.table)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 持门后分发一条请求
|
||||
fn dispatch(&self, req: Request) -> Response {
|
||||
if self.recovery_required.load(Ordering::Acquire) && !matches!(req, Request::Ping) {
|
||||
@@ -1558,6 +1609,7 @@ impl Server {
|
||||
table: name.to_string(),
|
||||
database_id: c.database_id,
|
||||
object_id: ts.object_id,
|
||||
schema_version: ts.schema_version,
|
||||
columns: ts.columns.clone(),
|
||||
row_count: ts.row_count,
|
||||
indexes: index_wires(ts),
|
||||
@@ -1571,6 +1623,7 @@ impl Server {
|
||||
log_debug!(request, "describe_table table={}", table);
|
||||
match self.describe_table(&table) {
|
||||
Ok((cols, count, indexes, stats)) => Response::Schema {
|
||||
schema_version: self.catalog.lock().unwrap().describe(&table).map_or(0, |s| s.schema_version),
|
||||
table,
|
||||
columns: cols,
|
||||
row_count: count,
|
||||
@@ -1713,7 +1766,7 @@ impl Server {
|
||||
Request::ReplaceSchema { table, columns, rows } => {
|
||||
log_debug!(request, "replace_schema table={} cols={}", table, columns.len());
|
||||
let tname = table.clone();
|
||||
let op = WalOp::ReplaceSchema { table, columns, rows };
|
||||
let op = WalOp::ReplaceSchema { table, columns, rows, schema_version: None };
|
||||
match self.write_via_wal(&op) {
|
||||
Ok(()) => Response::Ok,
|
||||
Err(e) => {
|
||||
@@ -1838,9 +1891,9 @@ fn identity_columns() -> Vec<SchemaColumn> {
|
||||
|
||||
/// 把系统表写进字典
|
||||
fn declare_system_schema(catalog: &mut Catalog, rows: &[Row]) -> std::io::Result<()> {
|
||||
catalog.rebuild_stats(USERS_TABLE, rows);
|
||||
catalog.rebuild_stats(USERS_TABLE, rows)?;
|
||||
catalog.set_columns(USERS_TABLE, users_columns())?;
|
||||
catalog.rebuild_stats(USERS_TABLE, rows);
|
||||
catalog.rebuild_stats(USERS_TABLE, rows)?;
|
||||
catalog.mark_system(USERS_TABLE)?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -2137,6 +2190,10 @@ impl Databases {
|
||||
fn request(&self, mut value: serde_json::Value) -> std::io::Result<Response> {
|
||||
use std::io::Error;
|
||||
let method = value.get("method").and_then(|v| v.as_str()).unwrap_or("").to_string();
|
||||
if value.get("schema_guard").is_some()
|
||||
&& !matches!(method.as_str(), "scan" | "scan_shard" | "lookup_by_index" | "range_by_index") {
|
||||
return Err(Error::other("schema guard requires a table read"));
|
||||
}
|
||||
let name = match value.get("database") {
|
||||
None => String::new(),
|
||||
Some(serde_json::Value::String(s)) => s.to_ascii_lowercase(),
|
||||
@@ -2258,6 +2315,8 @@ impl Databases {
|
||||
selected = database.to_ascii_lowercase();
|
||||
value["table"] = serde_json::Value::String(table.to_string());
|
||||
}
|
||||
let guard: Option<Vec<crate::protocol::BoundSchema>> = value.get("schema_guard")
|
||||
.map(|guard| serde_json::from_value(guard.clone()).map_err(Error::other)).transpose()?;
|
||||
let request: Request = serde_json::from_value(value).map_err(Error::other)?;
|
||||
let global = matches!(request, Request::AccountsList | Request::CatalogManage { .. } | Request::AccountCreate { .. }
|
||||
| Request::IdentityInitialize { .. } | Request::RoleCreate { .. } | Request::IdentityAlter { .. }
|
||||
@@ -2268,11 +2327,43 @@ impl Databases {
|
||||
if !global && selected.is_empty() { return Err(Error::other("no database selected")); }
|
||||
let server = if global || selected == SYSTEM_DATABASE { Arc::clone(&self.system) }
|
||||
else { Arc::clone(named.get(&selected).ok_or_else(|| Error::other("unknown database"))?) };
|
||||
if let Some(bindings) = guard {
|
||||
if !concurrent_read(&request) { return Err(Error::other("schema guard requires a table read")); }
|
||||
let mut servers = std::collections::BTreeMap::from([(selected.clone(), Arc::clone(&server))]);
|
||||
let mut dependencies = Vec::new();
|
||||
for binding in bindings {
|
||||
let (database, table) = binding.table.split_once('.').unwrap_or((&selected, &binding.table));
|
||||
let database = database.to_ascii_lowercase();
|
||||
if !valid_database(&database) { return Err(Error::other("invalid bound database name")); }
|
||||
let dependency = if database == SYSTEM_DATABASE { Arc::clone(&self.system) }
|
||||
else { Arc::clone(named.get(&database).ok_or_else(|| Error::other("bound query database changed"))?) };
|
||||
servers.insert(database.clone(), dependency);
|
||||
dependencies.push((database, table.to_owned(), binding));
|
||||
}
|
||||
if !dependencies.iter().any(|(database, table, _)| database == &selected && Some(table.as_str()) == request.table()) {
|
||||
return Err(Error::other("bound read table has no schema guard"));
|
||||
}
|
||||
drop(named);
|
||||
let _gates = servers.values().map(|server| server.gate.read()
|
||||
.map_err(|_| Error::other("storage read gate poisoned"))).collect::<std::io::Result<Vec<_>>>()?;
|
||||
for (database, table, binding) in &dependencies {
|
||||
servers[database].validate_bound_schema(table, binding)?;
|
||||
}
|
||||
return Ok(Response::Guarded { response: Box::new(server.dispatch(request)) });
|
||||
}
|
||||
drop(named);
|
||||
Ok(server.handle_request(request))
|
||||
}
|
||||
}
|
||||
|
||||
/// 读取守卫中的精确十进制编号
|
||||
fn parse_bound_number(value: &str) -> std::io::Result<u64> {
|
||||
if value.is_empty() || value.len() > 20 || !value.bytes().all(|c| c.is_ascii_digit()) {
|
||||
return Err(std::io::Error::other("invalid bound catalog number"));
|
||||
}
|
||||
value.parse().map_err(std::io::Error::other)
|
||||
}
|
||||
|
||||
/// 只读表页的请求可与其它读并发,其余独占
|
||||
fn concurrent_read(req: &Request) -> bool {
|
||||
matches!(
|
||||
|
||||
@@ -29,7 +29,7 @@ const OP_DROP_INDEX: u8 = 14;
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||
pub enum WalOp {
|
||||
HideColumn { table: String, column: String },
|
||||
HideColumn { table: String, column: String, schema_version: Option<u64> },
|
||||
Compact { table: String, rows: Vec<Row> },
|
||||
Accounts { accounts: Vec<Account>, removed: Vec<String>, migrate_roles: bool },
|
||||
Insert {
|
||||
@@ -51,11 +51,13 @@ pub enum WalOp {
|
||||
DropColumn {
|
||||
table: String,
|
||||
column: String,
|
||||
schema_version: Option<u64>,
|
||||
rows: Vec<Row>,
|
||||
},
|
||||
ReplaceSchema {
|
||||
table: String,
|
||||
columns: Vec<SchemaColumn>,
|
||||
schema_version: Option<u64>,
|
||||
rows: Vec<Row>,
|
||||
},
|
||||
CreateTable {
|
||||
@@ -78,15 +80,26 @@ pub enum WalOp {
|
||||
Commit,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct HideColumnPayload {
|
||||
column: String,
|
||||
#[serde(default)]
|
||||
schema_version: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct DropColumnPayload {
|
||||
column: String,
|
||||
#[serde(default)]
|
||||
schema_version: Option<u64>,
|
||||
rows: Vec<Row>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct ReplaceSchemaPayload {
|
||||
columns: Vec<SchemaColumn>,
|
||||
#[serde(default)]
|
||||
schema_version: Option<u64>,
|
||||
rows: Vec<Row>,
|
||||
}
|
||||
|
||||
@@ -379,8 +392,8 @@ fn truncate(f: &mut File) -> io::Result<()> {
|
||||
/// 把操作编成字节
|
||||
fn encode(op: &WalOp, lsn: u64) -> io::Result<Vec<u8>> {
|
||||
let (op_type, table, payload) = match op {
|
||||
WalOp::HideColumn { table, column } => (OP_HIDE_COLUMN, table.clone(),
|
||||
serde_json::to_vec(column).map_err(io::Error::other)?),
|
||||
WalOp::HideColumn { table, column, schema_version } => (OP_HIDE_COLUMN, table.clone(),
|
||||
serde_json::to_vec(&serde_json::json!({"column":column,"schema_version":schema_version})).map_err(io::Error::other)?),
|
||||
WalOp::Compact { table, rows } => {
|
||||
let p = serde_json::to_vec(rows).map_err(io::Error::other)?;
|
||||
(OP_COMPACT, table.clone(), p)
|
||||
@@ -403,17 +416,19 @@ fn encode(op: &WalOp, lsn: u64) -> io::Result<Vec<u8>> {
|
||||
let p = serde_json::to_vec(rows).map_err(io::Error::other)?;
|
||||
(OP_REPLACE_ALL, table.clone(), p)
|
||||
}
|
||||
WalOp::DropColumn { table, column, rows } => {
|
||||
WalOp::DropColumn { table, column, rows, schema_version } => {
|
||||
let p = serde_json::to_vec(&DropColumnPayload {
|
||||
column: column.clone(),
|
||||
schema_version: *schema_version,
|
||||
rows: rows.clone(),
|
||||
})
|
||||
.map_err(io::Error::other)?;
|
||||
(OP_DROP_COLUMN, table.clone(), p)
|
||||
}
|
||||
WalOp::ReplaceSchema { table, columns, rows } => {
|
||||
WalOp::ReplaceSchema { table, columns, rows, schema_version } => {
|
||||
let p = serde_json::to_vec(&ReplaceSchemaPayload {
|
||||
columns: columns.clone(),
|
||||
schema_version: *schema_version,
|
||||
rows: rows.clone(),
|
||||
})
|
||||
.map_err(io::Error::other)?;
|
||||
@@ -474,9 +489,16 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
|
||||
let op = match op_type {
|
||||
OP_COMMIT => WalOp::Commit,
|
||||
OP_HIDE_COLUMN => {
|
||||
let column = serde_json::from_slice(payload)
|
||||
let value: serde_json::Value = serde_json::from_slice(payload)
|
||||
.map_err(|e| format!("bad hide_column payload: {e}"))?;
|
||||
WalOp::HideColumn { table, column }
|
||||
let (column, schema_version) = if let Some(column) = value.as_str() {
|
||||
(column.to_string(), None)
|
||||
} else {
|
||||
let p: HideColumnPayload = serde_json::from_value(value)
|
||||
.map_err(|e| format!("bad hide_column payload: {e}"))?;
|
||||
(p.column, p.schema_version)
|
||||
};
|
||||
WalOp::HideColumn { table, column, schema_version }
|
||||
}
|
||||
OP_COMPACT => {
|
||||
let rows: Vec<Row> = serde_json::from_slice(payload)
|
||||
@@ -521,6 +543,7 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
|
||||
WalOp::DropColumn {
|
||||
table,
|
||||
column: p.column,
|
||||
schema_version: p.schema_version,
|
||||
rows: p.rows,
|
||||
}
|
||||
}
|
||||
@@ -530,6 +553,7 @@ fn decode_body(body: &[u8]) -> Result<(u64, WalOp), String> {
|
||||
WalOp::ReplaceSchema {
|
||||
table,
|
||||
columns: p.columns,
|
||||
schema_version: p.schema_version,
|
||||
rows: p.rows,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,68 @@ use serde_json::json;
|
||||
|
||||
// 数据字典测试:补列、去重、未知表。
|
||||
|
||||
/// 结构变化递增版本,数据及重复应用不变
|
||||
#[test]
|
||||
fn schema_versions_track_structure_and_survive_reload() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("catalog.json");
|
||||
let mut c = Catalog::default();
|
||||
c.create_table("t", vec![]).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 1);
|
||||
let values = row(&[("id", json!(1)), ("name", json!("a"))]);
|
||||
c.record_insert("t", &values).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 2);
|
||||
c.rebuild_stats("t", &[values]).unwrap();
|
||||
let columns = c.describe("t").unwrap().columns.clone();
|
||||
c.set_columns("t", columns).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 2);
|
||||
c.remove_column("t", "name").unwrap();
|
||||
c.remove_column("t", "name").unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 3);
|
||||
c.rename_table("t", "renamed").unwrap();
|
||||
c.save(&path).unwrap();
|
||||
let c = Catalog::load(&path).unwrap();
|
||||
assert_eq!(c.describe("renamed").unwrap().schema_version, 4);
|
||||
}
|
||||
|
||||
/// 旧目录版本未知,首个结构变更才递增
|
||||
#[test]
|
||||
fn legacy_schema_version_remains_unknown_until_change() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("catalog.json");
|
||||
std::fs::write(&path, json!({"tables": {
|
||||
"t": {"object_id": 42, "columns": []}
|
||||
}}).to_string()).unwrap();
|
||||
let mut c = Catalog::load(&path).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 0);
|
||||
c.save(&path).unwrap();
|
||||
c = Catalog::load(&path).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 0);
|
||||
c.ensure_columns("t", &row(&[("id", json!(1))])).unwrap();
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, 1);
|
||||
assert_eq!(c.describe("t").unwrap().object_id, 42);
|
||||
}
|
||||
|
||||
/// 版本耗尽拒绝结构变更并保留原定义
|
||||
#[test]
|
||||
fn exhausted_schema_version_preserves_structure() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("catalog.json");
|
||||
std::fs::write(&path, json!({"tables": {
|
||||
"t": {"object_id": 42, "schema_version": u64::MAX,
|
||||
"columns": [{"name": "id", "ty": "int"}]}
|
||||
}}).to_string()).unwrap();
|
||||
let mut c = Catalog::load(&path).unwrap();
|
||||
assert!(c.remove_column("t", "id").is_err());
|
||||
assert!(c.set_columns("t", vec![]).is_err());
|
||||
assert!(c.rename_table("t", "other").is_err());
|
||||
assert!(c.ensure_columns("t", &row(&[("new", json!(1))])).is_err());
|
||||
assert_eq!(c.describe("t").unwrap().columns.len(), 1);
|
||||
assert_eq!(c.describe("t").unwrap().schema_version, u64::MAX);
|
||||
assert!(c.describe("other").is_none());
|
||||
c.record_insert("t", &row(&[("id", json!(1))])).unwrap();
|
||||
}
|
||||
|
||||
/// 表删除重建和重载均保留编号规则
|
||||
#[test]
|
||||
fn object_ids_survive_reload_and_are_not_reused() {
|
||||
@@ -80,7 +142,7 @@ fn ensure_then_describe() {
|
||||
let path = dir.path().join("catalog.json");
|
||||
|
||||
let mut c = Catalog::load(&path).unwrap();
|
||||
c.ensure_columns("users", &row(&[("id", json!(1)), ("name", json!("Alice"))]));
|
||||
c.ensure_columns("users", &row(&[("id", json!(1)), ("name", json!("Alice"))])).unwrap();
|
||||
c.save(&path).unwrap();
|
||||
|
||||
let c2 = Catalog::load(&path).unwrap();
|
||||
@@ -102,8 +164,8 @@ fn second_insert_does_not_duplicate_columns() {
|
||||
let path = dir.path().join("catalog.json");
|
||||
|
||||
let mut c = Catalog::load(&path).unwrap();
|
||||
c.ensure_columns("users", &row(&[("id", json!(1))]));
|
||||
c.ensure_columns("users", &row(&[("id", json!(2))]));
|
||||
c.ensure_columns("users", &row(&[("id", json!(1))])).unwrap();
|
||||
c.ensure_columns("users", &row(&[("id", json!(2))])).unwrap();
|
||||
assert_eq!(c.describe("users").unwrap().columns.len(), 1);
|
||||
}
|
||||
|
||||
@@ -114,9 +176,9 @@ fn remove_column_drops_definition_stats_and_index() {
|
||||
let path = dir.path().join("catalog.json");
|
||||
|
||||
let mut c = Catalog::load(&path).unwrap();
|
||||
c.ensure_columns("t", &row(&[("id", json!(1)), ("code", json!(5))]));
|
||||
c.ensure_columns("t", &row(&[("id", json!(1)), ("code", json!(5))])).unwrap();
|
||||
c.add_index("t", "code").unwrap();
|
||||
c.rebuild_stats("t", &[row(&[("id", json!(1)), ("code", json!(5))])]);
|
||||
c.rebuild_stats("t", &[row(&[("id", json!(1)), ("code", json!(5))])]).unwrap();
|
||||
|
||||
let removed = c.remove_column("t", "code").unwrap();
|
||||
assert!(removed, "first remove should hit the column");
|
||||
@@ -142,7 +204,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
|
||||
let rows: Vec<Row> = (0..100)
|
||||
.map(|n| row(&[("id", json!(n)), ("name", json!(format!("u{}", n)))]))
|
||||
.collect();
|
||||
c.rebuild_stats("t", &rows);
|
||||
c.rebuild_stats("t", &rows).unwrap();
|
||||
|
||||
let s = c.describe("t").unwrap();
|
||||
let id = s.stats.get("id").unwrap();
|
||||
@@ -155,7 +217,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
|
||||
assert!(name.hist.is_empty(), "字符串列不进直方图");
|
||||
assert_eq!(name.lo, None);
|
||||
|
||||
c.record_insert("t", &row(&[("id", json!(1000)), ("name", json!("x"))]));
|
||||
c.record_insert("t", &row(&[("id", json!(1000)), ("name", json!("x"))])).unwrap();
|
||||
let s = c.describe("t").unwrap();
|
||||
let id = s.stats.get("id").unwrap();
|
||||
assert_eq!(id.lo, Some(0.0));
|
||||
@@ -178,7 +240,7 @@ fn stats_track_a_histogram_for_numeric_columns() {
|
||||
fn duplicate_deletion_keeps_distinct_until_last_occurrence() {
|
||||
let mut catalog = Catalog::default();
|
||||
let duplicate = row(&[("id", json!(1)), ("v", json!(7))]);
|
||||
catalog.rebuild_stats("t", &[duplicate.clone(), duplicate.clone()]);
|
||||
catalog.rebuild_stats("t", &[duplicate.clone(), duplicate.clone()]).unwrap();
|
||||
catalog.record_delete("t", std::slice::from_ref(&duplicate));
|
||||
assert_eq!(catalog.describe("t").unwrap().stats["v"].distinct, 1);
|
||||
catalog.record_delete("t", &[duplicate]);
|
||||
@@ -190,7 +252,7 @@ fn duplicate_deletion_keeps_distinct_until_last_occurrence() {
|
||||
fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
|
||||
let mut catalog = Catalog::default();
|
||||
let rows: Vec<Row> = (0..5000).map(|n| row(&[("id", json!(n)), ("v", json!(n % 2))])).collect();
|
||||
catalog.rebuild_stats("t", &rows);
|
||||
catalog.rebuild_stats("t", &rows).unwrap();
|
||||
let schema = catalog.describe("t").unwrap();
|
||||
assert!(schema.stats["id"].capped);
|
||||
assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 5000);
|
||||
@@ -199,7 +261,7 @@ fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
|
||||
let schema = catalog.describe("t").unwrap();
|
||||
assert_eq!(schema.stats["id"].hist.iter().sum::<u64>(), 4000);
|
||||
assert_eq!(schema.stats["v"].distinct, 2);
|
||||
catalog.rebuild_stats("t", &rows[1000..]);
|
||||
catalog.rebuild_stats("t", &rows[1000..]).unwrap();
|
||||
assert_eq!(catalog.describe("t").unwrap().stats["id"].distinct, 4000);
|
||||
}
|
||||
|
||||
@@ -208,16 +270,16 @@ fn capped_distinct_keeps_histogram_and_other_columns_accurate() {
|
||||
fn rebuilt_histogram_has_exact_buckets_in_any_order() {
|
||||
let rows: Vec<Row> = (0..160).map(|n| row(&[("v", json!(n))])).collect();
|
||||
let mut forward = Catalog::default();
|
||||
forward.rebuild_stats("t", &rows);
|
||||
forward.rebuild_stats("t", &rows).unwrap();
|
||||
let mut reverse = Catalog::default();
|
||||
let reversed: Vec<Row> = rows.iter().rev().cloned().collect();
|
||||
reverse.rebuild_stats("t", &reversed);
|
||||
reverse.rebuild_stats("t", &reversed).unwrap();
|
||||
let stat = &forward.describe("t").unwrap().stats["v"];
|
||||
assert_eq!(stat.hist, vec![10; 16]);
|
||||
assert_eq!(stat.hist, reverse.describe("t").unwrap().stats["v"].hist);
|
||||
forward.record_insert("t", &row(&[("v", json!(10000))]));
|
||||
forward.record_insert("t", &row(&[("v", json!(10000))])).unwrap();
|
||||
forward.record_delete("t", &[row(&[("v", json!(10000))])]);
|
||||
forward.rebuild_stats("t", &rows[80..]);
|
||||
forward.rebuild_stats("t", &rows[80..]).unwrap();
|
||||
let repaired = &forward.describe("t").unwrap().stats["v"];
|
||||
assert_eq!(repaired.lo, Some(80.0));
|
||||
assert_eq!(repaired.hi, Some(159.0));
|
||||
@@ -253,8 +315,8 @@ fn legacy_catalog_loads_and_normalizes() {
|
||||
let s = c.describe("legacy").unwrap();
|
||||
assert_eq!(s.columns.len(), 2);
|
||||
assert!(s.columns[0].nullable, "老列默认可空");
|
||||
assert!(c.normalize_types());
|
||||
assert!(c.normalize_types().unwrap());
|
||||
let types: Vec<&str> = c.describe("legacy").unwrap().columns.iter().map(|col| col.ty.as_str()).collect();
|
||||
assert_eq!(types, vec!["int", "str"]);
|
||||
assert!(!c.normalize_types(), "第二次没有可改的");
|
||||
assert!(!c.normalize_types().unwrap(), "第二次没有可改的");
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
use std::ffi::CString;
|
||||
use std::ptr;
|
||||
use std::slice;
|
||||
|
||||
use chusql_core_storage::ffi::*;
|
||||
use chusql_core_storage::runtime::Storage;
|
||||
|
||||
// 验证 FFI 错误输出、长度边界与响应所有权。
|
||||
|
||||
/// 读取当前线程的 FFI 错误
|
||||
fn last_error() -> String {
|
||||
let mut len = 0;
|
||||
// SAFETY: 长度可写,返回缓冲在复制前没有被修改。
|
||||
unsafe {
|
||||
let data = chusql_storage_last_error(&mut len);
|
||||
assert!(!data.is_null());
|
||||
String::from_utf8(slice::from_raw_parts(data, len).to_vec()).unwrap()
|
||||
}
|
||||
}
|
||||
|
||||
/// 维护失败清空输出并拒绝超长请求
|
||||
#[test]
|
||||
fn maintenance_rejects_invalid_arguments_and_resets_outputs() {
|
||||
let input = b"{}";
|
||||
let mut out = input.as_ptr() as *mut u8;
|
||||
let mut len = 99;
|
||||
// SAFETY: 输出均可写;无效输入应在任何读取之前被拒绝。
|
||||
unsafe {
|
||||
assert_eq!(chusql_storage_maintenance(ptr::null(), ptr::null(), 0, &mut out, &mut len), -1);
|
||||
assert!(out.is_null());
|
||||
assert_eq!(len, 0);
|
||||
len = 99;
|
||||
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), input.len(), ptr::null_mut(), &mut len), -1);
|
||||
assert_eq!(len, 0);
|
||||
out = input.as_ptr() as *mut u8;
|
||||
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), input.len(), &mut out, ptr::null_mut()), -1);
|
||||
assert!(out.is_null());
|
||||
assert_eq!(chusql_storage_maintenance(ptr::null(), input.as_ptr(), usize::MAX, &mut out, &mut len), -1);
|
||||
}
|
||||
assert!(out.is_null());
|
||||
assert_eq!(len, 0);
|
||||
assert_eq!(last_error(), "invalid maintenance argument");
|
||||
}
|
||||
|
||||
/// 请求验证长度和编码并归还响应
|
||||
#[test]
|
||||
fn request_validates_input_and_returns_owned_response() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let config = temp.path().join("settings.toml");
|
||||
std::fs::write(&config, format!("[storage]\ndata_dir = '{}'\nlog_files = '{}'\n[log]\nlevel = 'off'\n", temp.path().join("data").display(), temp.path().join("logs").display()))?;
|
||||
let storage = Storage::open(config.to_str())?;
|
||||
let input = br#"{"method":"ping"}"#;
|
||||
let mut out = ptr::null_mut();
|
||||
let mut len = 0;
|
||||
// SAFETY: 句柄和实际请求有效;超长长度必须在构造切片前被拒绝。
|
||||
unsafe {
|
||||
assert_eq!(chusql_storage_request(&storage, input.as_ptr(), usize::MAX, &mut out, &mut len), -1);
|
||||
assert!(out.is_null());
|
||||
assert_eq!(len, 0);
|
||||
assert_eq!(last_error(), "invalid argument");
|
||||
assert_eq!(chusql_storage_request(&storage, [0xff].as_ptr(), 1, &mut out, &mut len), -1);
|
||||
assert!(last_error().starts_with("request is not valid UTF-8:"));
|
||||
assert_eq!(chusql_storage_request(&storage, input.as_ptr(), input.len(), &mut out, &mut len), 0);
|
||||
assert!(!out.is_null());
|
||||
let response = slice::from_raw_parts(out, len).to_vec();
|
||||
chusql_storage_free(out, len);
|
||||
let response: serde_json::Value = serde_json::from_slice(&response)?;
|
||||
assert_eq!(response["status"], "pong");
|
||||
}
|
||||
drop(storage);
|
||||
let config = CString::new(config.to_str().unwrap())?;
|
||||
// SAFETY: 路径与输出有效;失败不产生需要释放的缓冲。
|
||||
unsafe {
|
||||
assert_eq!(chusql_storage_maintenance(config.as_ptr(), [0xff].as_ptr(), 1, &mut out, &mut len), -1);
|
||||
}
|
||||
assert!(out.is_null());
|
||||
assert_eq!(len, 0);
|
||||
assert!(!last_error().is_empty());
|
||||
let maintenance = br#"{"mode":"reset","user":"root","password_hash":"hash","types":[{"name":"int","base_type":"int","parameterized":false}]}"#;
|
||||
// SAFETY: 维护仅使用本测试临时目录;成功响应复制后由配套函数释放。
|
||||
unsafe {
|
||||
assert_eq!(chusql_storage_maintenance(config.as_ptr(), maintenance.as_ptr(), maintenance.len(), &mut out, &mut len), 0, "{}", last_error());
|
||||
assert!(!out.is_null());
|
||||
let response = slice::from_raw_parts(out, len).to_vec();
|
||||
chusql_storage_free(out, len);
|
||||
let response: serde_json::Value = serde_json::from_slice(&response)?;
|
||||
assert_eq!(response["status"], "maintenance");
|
||||
assert_eq!(response["mode"], "reset");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -171,6 +171,131 @@ fn object_id_survives_committed_create_recovery() -> Result<(), Box<dyn std::err
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 读取复查跨库依赖而非只复查扫描表
|
||||
#[test]
|
||||
fn bound_reads_recheck_dependencies_in_other_databases() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let (_server, _data) = start_server();
|
||||
let mut c = connect()?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"source","columns":[]}))?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_database","database":"dependency"}))?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_table","database":"dependency","table":"related","columns":[]}))?;
|
||||
let catalog: serde_json::Value = serde_json::from_str(&send_raw(&mut c, r#"{"method":"all_catalogs"}"#))?;
|
||||
let bindings: Vec<_> = catalog["schemas"].as_array().unwrap().iter().filter(|entry|
|
||||
entry["table"] == "main.source" || entry["table"] == "dependency.related").map(|entry| {
|
||||
let mut binding = entry.clone();
|
||||
binding["database_id"] = binding["database_id"].as_u64().unwrap().to_string().into();
|
||||
binding["object_id"] = binding["object_id"].as_u64().unwrap().to_string().into();
|
||||
binding
|
||||
}).collect();
|
||||
assert_eq!(bindings.len(), 2);
|
||||
let request = serde_json::json!({"method":"scan","table":"source","schema_guard":bindings});
|
||||
let valid: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
|
||||
assert_eq!(valid["status"], "guarded");
|
||||
request_ok(&mut c, serde_json::json!({"method":"replace_schema","database":"dependency","table":"related","columns":[{"name":"new","ty":"int"}],"rows":[]}))?;
|
||||
let invalid: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
|
||||
assert_eq!(invalid["status"], "error");
|
||||
assert!(invalid["message"].as_str().unwrap().contains("bound query schema changed: dependency.related"));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 每种读取都复查守卫,拒绝旧版本和写入
|
||||
#[test]
|
||||
fn bound_reads_validate_dependencies_and_acknowledge_guards() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let (_server, _data) = start_server();
|
||||
let mut c = connect()?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"guarded","columns":[{"name":"id","ty":"int"}]}))?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"insert","table":"guarded","row":{"id":7}}))?;
|
||||
let catalog: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"list_catalog"}"#))?;
|
||||
let mut binding = catalog["schemas"][0].clone();
|
||||
binding["database_id"] = binding["database_id"].as_u64().unwrap().to_string().into();
|
||||
binding["object_id"] = binding["object_id"].as_u64().unwrap().to_string().into();
|
||||
for mut request in [
|
||||
serde_json::json!({"method":"scan","table":"guarded"}),
|
||||
serde_json::json!({"method":"scan_shard","table":"guarded","shard":0,"shards":1}),
|
||||
serde_json::json!({"method":"lookup_by_index","table":"guarded","column":"id","key":7}),
|
||||
serde_json::json!({"method":"range_by_index","table":"guarded","column":"id","lo":null,"hi":null}),
|
||||
] {
|
||||
request["schema_guard"] = serde_json::json!([binding]);
|
||||
let response: serde_json::Value = serde_json::from_str(&send(&mut c, &request.to_string()))?;
|
||||
assert_eq!(response["status"], "guarded", "{response}");
|
||||
assert!(response["response"]["status"] == "rows" || response["response"]["status"] == "no_index", "{response}");
|
||||
}
|
||||
request_ok(&mut c, serde_json::json!({"method":"replace_schema","table":"guarded","columns":[{"name":"id","ty":"int"},{"name":"extra","ty":"str"}],"rows":[{"id":7,"extra":"x"}]}))?;
|
||||
let stale: serde_json::Value = serde_json::from_str(&send(&mut c, &serde_json::json!({"method":"scan","table":"guarded","schema_guard":[binding]}).to_string()))?;
|
||||
assert_eq!(stale["status"], "error");
|
||||
assert!(stale["message"].as_str().unwrap().contains("bound query schema changed"));
|
||||
let write: serde_json::Value = serde_json::from_str(&send(&mut c, &serde_json::json!({"method":"insert","table":"guarded","row":{"id":9},"schema_guard":[binding]}).to_string()))?;
|
||||
assert_eq!(write["status"], "error");
|
||||
let management: serde_json::Value = serde_json::from_str(&send_raw(&mut c, &serde_json::json!({"method":"create_database","database":"blocked","schema_guard":[]}).to_string()))?;
|
||||
assert_eq!(management["status"], "error");
|
||||
let databases: serde_json::Value = serde_json::from_str(&send_raw(&mut c, r#"{"method":"list_databases"}"#))?;
|
||||
assert!(!databases["tables"].as_array().unwrap().iter().any(|database| database == "blocked"));
|
||||
let rows: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"scan","table":"guarded"}"#))?;
|
||||
assert_eq!(rows["rows"].as_array().unwrap().len(), 1);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 描述与目录均透传精确版本字符串
|
||||
#[test]
|
||||
fn catalog_schema_versions_are_exact_and_validated() -> Result<(), Box<dyn std::error::Error>> {
|
||||
use chusql_core_storage::protocol::TableSchemaWire;
|
||||
let (_server, _data) = start_server();
|
||||
let mut c = connect()?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"versioned","columns":[{"name":"id","ty":"int"}]}))?;
|
||||
let describe: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"describe_table","table":"versioned"}"#))?;
|
||||
assert_eq!(describe["schema_version"], "1");
|
||||
let catalog: serde_json::Value = serde_json::from_str(&send(&mut c, r#"{"method":"list_catalog"}"#))?;
|
||||
assert_eq!(catalog["schemas"][0]["schema_version"], "1");
|
||||
let wire: TableSchemaWire = serde_json::from_value(serde_json::json!({"table":"t","columns":[],"schema_version":u64::MAX.to_string()}))?;
|
||||
assert_eq!(wire.schema_version, u64::MAX);
|
||||
assert_eq!(serde_json::to_value(wire)?["schema_version"], u64::MAX.to_string());
|
||||
for version in [serde_json::json!(-1), serde_json::json!(1.5), serde_json::json!("18446744073709551616"), serde_json::json!("+1")] {
|
||||
assert!(serde_json::from_value::<TableSchemaWire>(serde_json::json!({"table":"t","columns":[],"schema_version":version})).is_err());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 检查点失败及反复恢复保留结构版本
|
||||
#[test]
|
||||
fn schema_versions_survive_checkpoint_failure_and_repeated_recovery() -> Result<(), Box<dyn std::error::Error>> {
|
||||
use chusql_core_storage::catalog::Catalog;
|
||||
use chusql_core_storage::wal::{Wal, WalOp};
|
||||
let (server, data) = start_server();
|
||||
let mut c = connect()?;
|
||||
request_ok(&mut c, serde_json::json!({"method":"create_table","table":"versioned","columns":[{"name":"id","ty":"int"}]}))?;
|
||||
let directory = data.path().join("databases/main");
|
||||
let pending = directory.join("wal.checkpoint.pending");
|
||||
std::fs::create_dir(&pending)?;
|
||||
let response = send(&mut c, r#"{"method":"replace_schema","table":"versioned","columns":[{"name":"id","ty":"int"},{"name":"tag","ty":"str"}],"rows":[]}"#);
|
||||
assert!(response.contains("error"), "{response}");
|
||||
let catalog = Catalog::load(directory.join("catalog.json"))?;
|
||||
let version = catalog.describe("versioned").unwrap().schema_version;
|
||||
assert_eq!(version, 2);
|
||||
let wal = Wal::new(directory.join("wal.log"));
|
||||
let records = wal.read_all()?.records;
|
||||
assert!(records.iter().any(|(_, op)| matches!(op, WalOp::ReplaceSchema { schema_version: Some(2), .. })));
|
||||
let columns = catalog.describe("versioned").unwrap().columns.clone();
|
||||
drop(wal);
|
||||
drop(c);
|
||||
drop(server);
|
||||
std::fs::remove_dir(pending)?;
|
||||
for _ in 0..2 {
|
||||
let wal = Wal::new(directory.join("wal.log"));
|
||||
wal.append_group(&[
|
||||
WalOp::ReplaceSchema { table: "versioned".into(), columns: vec![], rows: vec![], schema_version: Some(1) },
|
||||
WalOp::ReplaceSchema { table: "versioned".into(), columns: columns.clone(), rows: vec![], schema_version: Some(version) },
|
||||
])?;
|
||||
drop(wal);
|
||||
let restarted = restart_at(data.path())?;
|
||||
let catalog = Catalog::load(directory.join("catalog.json"))?;
|
||||
assert_eq!(catalog.describe("versioned").unwrap().schema_version, version);
|
||||
assert_eq!(catalog.describe("versioned").unwrap().columns, columns);
|
||||
assert_eq!(std::fs::metadata(directory.join("wal.log"))?.len(), 0);
|
||||
drop(restarted);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 并发建表分配互不重复的编号
|
||||
#[test]
|
||||
fn concurrent_table_creation_allocates_distinct_object_ids() -> Result<(), Box<dyn std::error::Error>> {
|
||||
@@ -1026,7 +1151,7 @@ fn hidden_column_wal_replays_twice_without_touching_heap() -> Result<(), Box<dyn
|
||||
let mut compacted: Option<Vec<u8>> = None;
|
||||
for round in 0..2 {
|
||||
let wal = Wal::new(data.path().join("databases/main/wal.log"));
|
||||
let op = WalOp::HideColumn { table: "items".into(), column: "secret".into() };
|
||||
let op = WalOp::HideColumn { table: "items".into(), column: "secret".into(), schema_version: None };
|
||||
wal.truncate()?;
|
||||
wal.append(&op)?;
|
||||
assert_eq!(wal.read()?, Some(op));
|
||||
|
||||
@@ -4,6 +4,33 @@ use serde_json::json;
|
||||
|
||||
// WAL 测试:追加、读取、清空、截断尾巴、LSN 与提交标记。
|
||||
|
||||
/// 结构版本往返且旧隐藏列载荷可读
|
||||
#[test]
|
||||
fn schema_versions_round_trip_and_legacy_hide_decodes() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("wal.log");
|
||||
let wal = Wal::new(&path);
|
||||
let ops = [
|
||||
WalOp::HideColumn { table: "t".into(), column: "name".into(), schema_version: Some(u64::MAX) },
|
||||
WalOp::DropColumn { table: "t".into(), column: "name".into(), rows: vec![], schema_version: Some(3) },
|
||||
WalOp::ReplaceSchema { table: "t".into(), columns: vec![], rows: vec![], schema_version: Some(4) },
|
||||
];
|
||||
wal.append_group(&ops).unwrap();
|
||||
assert_eq!(committed_ops(&wal.read_all().unwrap().records).into_iter().map(|(_, op)| op).collect::<Vec<_>>(), ops);
|
||||
drop(wal);
|
||||
let mut body = 1_u64.to_le_bytes().to_vec();
|
||||
body.push(8);
|
||||
body.extend_from_slice(&1_u16.to_le_bytes());
|
||||
body.extend_from_slice(b"t");
|
||||
body.extend_from_slice(br#""name""#);
|
||||
let mut frame = (body.len() as u32).to_le_bytes().to_vec();
|
||||
frame.extend_from_slice(&body);
|
||||
std::fs::write(&path, frame).unwrap();
|
||||
assert_eq!(Wal::new(&path).read().unwrap(), Some(WalOp::HideColumn {
|
||||
table: "t".into(), column: "name".into(), schema_version: None,
|
||||
}));
|
||||
}
|
||||
|
||||
/// 建表编号完整往返且兼容旧载荷
|
||||
#[test]
|
||||
fn create_table_ids_round_trip_and_legacy_payloads_decode() {
|
||||
@@ -58,6 +85,7 @@ fn append_and_read_round_trips() {
|
||||
WalOp::DropColumn {
|
||||
table: "users".into(),
|
||||
column: "age".into(),
|
||||
schema_version: None,
|
||||
rows: vec![
|
||||
row(&[("id", json!(1)), ("name", json!("Alice"))]),
|
||||
row(&[("id", json!(2)), ("name", json!("Bob"))]),
|
||||
|
||||
@@ -16,6 +16,7 @@ library
|
||||
ChuSQL.Interface.Actions
|
||||
ChuSQL.Interface.Auth
|
||||
ChuSQL.Interface.Config
|
||||
ChuSQL.Interface.Normalization
|
||||
ChuSQL.Interface.Link
|
||||
ChuSQL.Interface.Policy
|
||||
ChuSQL.Interface.Protocol
|
||||
|
||||
@@ -43,4 +43,4 @@ accountColumnSpec name ty nullable autoIncrement primary unique =
|
||||
|
||||
-- | 账号表的结构(排序校验用)
|
||||
systemTableInfo :: TableInfo
|
||||
systemTableInfo = TableInfo accountTableName accountColumns 0 [] [] []
|
||||
systemTableInfo = TableInfo accountTableName accountColumns 0 [] [] [] Nothing Nothing
|
||||
@@ -6,6 +6,7 @@ module ChuSQL.Interface.Link (
|
||||
closeClient,
|
||||
clientLogin,
|
||||
clientSudoLogin,
|
||||
clientNormalization,
|
||||
clientQuery,
|
||||
clientPing,
|
||||
clientCatalog,
|
||||
@@ -16,6 +17,8 @@ module ChuSQL.Interface.Link (
|
||||
clientReloadPolicy,
|
||||
) where
|
||||
|
||||
import ChuSQL.Interface.Normalization (NormalizationRequest)
|
||||
import qualified Data.Aeson as A
|
||||
import ChuSQL.Interface.Sudo (SudoCredential, sudoProof, isLocalPeer)
|
||||
import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo (..))
|
||||
import ChuSQL.Interface.Protocol (
|
||||
@@ -244,3 +247,12 @@ describe :: IOException -> String
|
||||
describe err
|
||||
| isEOFError err = "the server closed the connection"
|
||||
| otherwise = show err
|
||||
|
||||
-- 请求服务端执行只读范式分析。
|
||||
clientNormalization :: Client -> NormalizationRequest -> IO (Either Text A.Value)
|
||||
clientNormalization client payload = do
|
||||
answer <- request client (ReqNormalization payload)
|
||||
pure $ case answer of
|
||||
Left message -> Left message
|
||||
Right (RespNormalization value) -> Right value
|
||||
Right other -> failed "normalization" other
|
||||
@@ -0,0 +1,188 @@
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module ChuSQL.Interface.Normalization where
|
||||
|
||||
import ChuSQL.Core.Protocol (SchemaColumn)
|
||||
import Control.Monad (unless, when)
|
||||
import Data.Aeson (FromJSON (..), ToJSON (..), object, withObject, (.:), (.:?), (.!=), (.=))
|
||||
import qualified Data.Aeson as A
|
||||
import qualified Data.Aeson.Key as K
|
||||
import qualified Data.Aeson.KeyMap as KM
|
||||
import qualified Data.Aeson.Types as AT
|
||||
import qualified Data.ByteString.Lazy as BL
|
||||
import Data.Char (isAlpha, isAlphaNum, isDigit, isHexDigit, isSpace)
|
||||
import Data.List (nub)
|
||||
import Data.Scientific (base10Exponent, toBoundedInteger)
|
||||
import Data.Word (Word64)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Text.Encoding as TE
|
||||
import Text.ParserCombinators.ReadP
|
||||
|
||||
-- 范式入口的版本化声明、对象身份与严格 JSON 校验。
|
||||
|
||||
data NormalizationDependency = NormalizationDependency [String] [String] deriving (Eq, Show)
|
||||
data NormalizationDeclaration = NormalizationDeclaration
|
||||
{ ndKeys :: [[String]], ndDependencies :: [NormalizationDependency]
|
||||
, ndComplete :: Bool, ndAtomic :: Maybe Bool
|
||||
} deriving (Eq, Show)
|
||||
data ObjectIdentity = ObjectIdentity Integer Integer deriving (Eq, Show)
|
||||
data NormalizationRequest = NormalizationRequest
|
||||
{ nrVersion :: Int, nrDatabase :: Text, nrTable :: Text
|
||||
, nrDeclaration :: NormalizationDeclaration
|
||||
, nrExpectedObject :: Maybe ObjectIdentity, nrExpectedSchema :: Maybe Text
|
||||
} deriving (Eq, Show)
|
||||
|
||||
-- 拒绝对象中未定义的字段。
|
||||
onlyFields :: [Text] -> A.Object -> AT.Parser ()
|
||||
onlyFields names fields = unless (all ((`elem` names) . K.toText) (KM.keys fields)) (fail "normalization: unknown field")
|
||||
|
||||
instance FromJSON NormalizationDependency where
|
||||
-- 读取依赖两端的确切字段名。
|
||||
parseJSON = withObject "dependency" $ \o -> do
|
||||
onlyFields ["lhs", "rhs"] o
|
||||
lhs <- o .: "lhs"
|
||||
rhs <- o .: "rhs"
|
||||
when (null rhs) (fail "normalization: empty dependency rhs")
|
||||
pure (NormalizationDependency lhs rhs)
|
||||
instance ToJSON NormalizationDependency where
|
||||
-- 编码声明依赖。
|
||||
toJSON (NormalizationDependency lhs rhs) = object ["lhs" .= lhs, "rhs" .= rhs]
|
||||
instance FromJSON NormalizationDeclaration where
|
||||
-- 读取显式声明并限制数量。
|
||||
parseJSON = withObject "declaration" $ \o -> do
|
||||
onlyFields ["declared_keys", "dependencies", "dependencies_complete", "atomic_attributes"] o
|
||||
keys <- o .:? "declared_keys" .!= []
|
||||
deps <- o .:? "dependencies" .!= []
|
||||
when (length keys > 256 || length deps > 256) (fail "normalization: declaration exceeds 256 entries")
|
||||
NormalizationDeclaration keys deps <$> o .:? "dependencies_complete" .!= False <*> o .:? "atomic_attributes"
|
||||
instance ToJSON NormalizationDeclaration where
|
||||
-- 编码全部声明标志。
|
||||
toJSON d = object ["declared_keys" .= ndKeys d, "dependencies" .= ndDependencies d,
|
||||
"dependencies_complete" .= ndComplete d, "atomic_attributes" .= ndAtomic d]
|
||||
instance FromJSON ObjectIdentity where
|
||||
-- 校验稳定对象的正整数编号。
|
||||
parseJSON = withObject "object identity" $ \o -> do
|
||||
onlyFields ["database_id", "object_id"] o
|
||||
db <- o .: "database_id" >>= identityNumber
|
||||
table <- o .: "object_id" >>= identityNumber
|
||||
when (db <= 0 || table <= 0) (fail "normalization: invalid object identity")
|
||||
pure (ObjectIdentity db table)
|
||||
-- 兼容目录数值并保留浏览器中的完整编号。
|
||||
identityNumber :: A.Value -> AT.Parser Integer
|
||||
identityNumber (A.String text)
|
||||
| not (T.null text) && T.length text <= 20 && T.all isDigit text = case reads (T.unpack text) of
|
||||
[(value, "")] | value <= toInteger (maxBound :: Word64) -> pure value
|
||||
_ -> fail "normalization: invalid identity number"
|
||||
identityNumber (A.Number number)
|
||||
| base10Exponent number >= -20 && base10Exponent number <= 20 =
|
||||
maybe (fail "normalization: invalid identity number") (pure . toInteger) (toBoundedInteger number :: Maybe Word64)
|
||||
identityNumber _ = fail "normalization: invalid identity number"
|
||||
|
||||
instance ToJSON ObjectIdentity where
|
||||
-- 编码对象的库及表编号。
|
||||
toJSON (ObjectIdentity db table) = object ["database_id" .= T.pack (show db), "object_id" .= T.pack (show table)]
|
||||
instance FromJSON NormalizationRequest where
|
||||
-- 校验版本、范围和预期身份。
|
||||
parseJSON = withObject "normalization request" $ \o -> do
|
||||
onlyFields ["api_version", "database", "table", "declaration", "expected_object", "expected_schema_fingerprint"] o
|
||||
version <- o .: "api_version"
|
||||
when (version /= (1 :: Int)) (fail "normalization: unsupported api_version")
|
||||
db <- o .: "database"
|
||||
table <- o .: "table"
|
||||
unless (plainName db && case T.splitOn "." table of [name] -> plainName name; [scope, name] -> scope == db && plainName name; _ -> False)
|
||||
(fail "normalization: invalid or cross-database table name")
|
||||
NormalizationRequest version db table <$> o .: "declaration" <*> o .:? "expected_object" <*> o .:? "expected_schema_fingerprint"
|
||||
instance ToJSON NormalizationRequest where
|
||||
-- 编码专用分析请求。
|
||||
toJSON r = object ["api_version" .= nrVersion r, "database" .= nrDatabase r, "table" .= nrTable r,
|
||||
"declaration" .= nrDeclaration r, "expected_object" .= nrExpectedObject r, "expected_schema_fingerprint" .= nrExpectedSchema r]
|
||||
|
||||
-- 校验与 SQL 一致的裸标识符。
|
||||
plainName :: Text -> Bool
|
||||
plainName name = not (T.null name) && T.length name <= 64 && (isAlpha (T.head name) || T.head name == '_') && T.all (\c -> isAlphaNum c || c == '_') name
|
||||
|
||||
-- 检查大小、编码及重复 JSON 字段。
|
||||
decodeNormalizationJSON :: BL.ByteString -> Either Text A.Value
|
||||
decodeNormalizationJSON raw
|
||||
| BL.length raw > 65536 = Left "normalization: request exceeds 64 KiB"
|
||||
| otherwise = do
|
||||
value <- either (Left . T.pack) Right (A.eitherDecode raw)
|
||||
text <- either (const (Left "normalization: invalid UTF-8")) Right (TE.decodeUtf8' (BL.toStrict raw))
|
||||
case readP_to_S (skipSpaces *> jsonNode 0 <* skipSpaces <* eof) (T.unpack text) of
|
||||
[(True, "")] -> Right value
|
||||
_ -> Left "normalization: duplicate JSON fields or nesting exceeds 64"
|
||||
|
||||
-- 遍历 JSON 结构并保留原始字段名。
|
||||
jsonNode :: Int -> ReadP Bool
|
||||
jsonNode depth
|
||||
| depth > 64 = pfail
|
||||
| otherwise = objectNode <++ arrayNode <++ (True <$ jsonString) <++ (True <$ munch1 (\c -> not (isSpace c) && c `notElem` (",]}" :: String)))
|
||||
where
|
||||
-- 验证对象及所有嵌套值。
|
||||
objectNode = do
|
||||
_ <- char '{'
|
||||
skipSpaces
|
||||
pairs <- sepBy pair (skipSpaces *> char ',' <* skipSpaces)
|
||||
skipSpaces
|
||||
_ <- char '}'
|
||||
pure (length (map fst pairs) == length (nub (map fst pairs)) && all snd pairs)
|
||||
-- 保留键名并验证对应值。
|
||||
pair = do
|
||||
key <- jsonString
|
||||
skipSpaces
|
||||
_ <- char ':'
|
||||
skipSpaces
|
||||
valid <- jsonNode (depth + 1)
|
||||
pure (key, valid)
|
||||
-- 验证数组的所有元素。
|
||||
arrayNode = do
|
||||
_ <- char '['
|
||||
skipSpaces
|
||||
values <- sepBy (jsonNode (depth + 1)) (skipSpaces *> char ',' <* skipSpaces)
|
||||
skipSpaces
|
||||
_ <- char ']'
|
||||
pure (and values)
|
||||
|
||||
-- 解码字符串以识别转义后的相同键。
|
||||
jsonString :: ReadP Text
|
||||
jsonString = do
|
||||
_ <- char '"'
|
||||
pieces <- many piece
|
||||
_ <- char '"'
|
||||
case A.eitherDecodeStrict (TE.encodeUtf8 (T.pack ('"' : concat pieces ++ "\""))) of
|
||||
Right value -> pure value
|
||||
Left _ -> pfail
|
||||
where
|
||||
-- 保留 JSON 字符串转义。
|
||||
piece = ((\c -> ['\\', c]) <$> (char '\\' *> get)) <++ munch1 (\c -> c /= '"' && c /= '\\')
|
||||
|
||||
|
||||
-- 拒绝缺字段或未知标签的线上报告。
|
||||
parseNormalizationReport :: A.Value -> AT.Parser A.Value
|
||||
parseNormalizationReport value = withObject "normalization report" (\o -> do
|
||||
version <- o .: "api_version"
|
||||
conditional <- o .: "conditional"
|
||||
unless (version == (1 :: Int) && conditional) (fail "normalization: invalid report version or condition")
|
||||
_ <- (o .: "object" :: AT.Parser ObjectIdentity)
|
||||
fingerprint <- o .: "schema_fingerprint"
|
||||
unless (T.length fingerprint == 64 && T.all isHexDigit fingerprint) (fail "normalization: invalid schema fingerprint")
|
||||
view <- o .: "view"
|
||||
unless (view `elem` (["live_schema", "transaction_snapshot"] :: [Text])) (fail "normalization: unknown view")
|
||||
_ <- (o .: "schema" :: AT.Parser [SchemaColumn])
|
||||
_ <- (o .: "declaration" :: AT.Parser NormalizationDeclaration)
|
||||
_ <- (o .: "candidate_keys" :: AT.Parser [[String]])
|
||||
forms <- o .: "findings" >>= mapM parseFinding
|
||||
unless (forms == (["1nf", "2nf", "3nf", "bcnf"] :: [Text])) (fail "normalization: invalid findings")
|
||||
_ <- (o .: "suggestions" :: AT.Parser [A.Value])
|
||||
_ <- (o .: "condition" :: AT.Parser Text)
|
||||
pure value) value
|
||||
where
|
||||
-- 核对每项结论与证据的稳定形状。
|
||||
parseFinding = withObject "normalization finding" $ \o -> do
|
||||
form <- o .: "normal_form"
|
||||
verdict <- o .: "verdict"
|
||||
unless (verdict `elem` (["satisfied", "violated", "insufficient"] :: [Text])) (fail "normalization: unknown verdict")
|
||||
_ <- (o .: "witnesses" :: AT.Parser [NormalizationDependency])
|
||||
_ <- (o .: "condition" :: AT.Parser Text)
|
||||
pure form
|
||||
@@ -13,6 +13,8 @@ module ChuSQL.Interface.Protocol (
|
||||
RoleView (..),
|
||||
) where
|
||||
|
||||
import ChuSQL.Interface.Normalization
|
||||
import Data.Aeson.Types (parseEither)
|
||||
import ChuSQL.Core.Protocol (Account (..), QueryResult (..), TableInfo (..), queryResultJson)
|
||||
import Data.Aeson (FromJSON (..), ToJSON (..), Value, object, withObject, (.:), (.:?), (.!=), (.=))
|
||||
import qualified Data.Aeson as A
|
||||
@@ -93,6 +95,7 @@ data ClientRequest
|
||||
| ReqLogin Text Text
|
||||
| ReqSudoChallenge
|
||||
| ReqSudoLogin Text Text
|
||||
| ReqNormalization NormalizationRequest
|
||||
| ReqQuery Text
|
||||
| ReqPing
|
||||
| ReqQuit
|
||||
@@ -141,6 +144,9 @@ decodeRequest raw = case A.eitherDecodeStrict raw of
|
||||
"sudo_login" -> case (wrUser wire, wrProof wire) of
|
||||
(Just user, Just proof) -> Right (ReqSudoLogin user proof)
|
||||
_ -> Left "sudo_login needs both user and proof"
|
||||
"normalization_analyze" -> do
|
||||
value <- decodeNormalizationJSON (BL.fromStrict raw)
|
||||
either (Left . T.pack) Right $ parseEither (withObject "normalization envelope" $ \o -> onlyFields ["method", "request"] o >> ReqNormalization <$> o .: "request") value
|
||||
"query" -> maybe (Left "query needs sql") (Right . ReqQuery) (wrSql wire)
|
||||
"storage" -> maybe (Left "storage needs request") (Right . ReqStorage) (wrRequest wire)
|
||||
"catalog" -> Right ReqCatalog
|
||||
@@ -164,6 +170,7 @@ requestJson request = case request of
|
||||
ReqLogin user password -> object ["method" .= ("login" :: Text), "user" .= user, "password" .= password]
|
||||
ReqSudoChallenge -> object ["method" .= ("sudo_challenge" :: Text)]
|
||||
ReqSudoLogin user proof -> object ["method" .= ("sudo_login" :: Text), "user" .= user, "proof" .= proof]
|
||||
ReqNormalization payload -> object ["method" .= ("normalization_analyze" :: Text), "request" .= payload]
|
||||
ReqQuery sql -> object ["method" .= ("query" :: Text), "sql" .= sql]
|
||||
ReqPing -> object ["method" .= ("ping" :: Text)]
|
||||
ReqQuit -> object ["method" .= ("quit" :: Text)]
|
||||
@@ -184,6 +191,7 @@ data ServerResponse
|
||||
| RespPong
|
||||
| RespBye
|
||||
| RespError Text Text
|
||||
| RespNormalization Value
|
||||
| RespStorage Value
|
||||
| RespCatalog [TableInfo]
|
||||
| RespDatabases [Text]
|
||||
@@ -205,6 +213,7 @@ responseJson response = case response of
|
||||
RespPong -> object ["status" .= ("pong" :: Text)]
|
||||
RespBye -> object ["status" .= ("bye" :: Text)]
|
||||
RespError code message -> object ["status" .= ("error" :: Text), "code" .= code, "message" .= message]
|
||||
RespNormalization value -> object ["status" .= ("normalization" :: Text), "analysis" .= value]
|
||||
RespStorage value -> object ["status" .= ("storage" :: Text), "response" .= value]
|
||||
RespCatalog infos -> object ["status" .= ("catalog" :: Text), "schemas" .= infos]
|
||||
RespDatabases names -> object ["status" .= ("databases" :: Text), "databases" .= names]
|
||||
@@ -238,6 +247,9 @@ decodeResponse raw = case A.eitherDecode raw of
|
||||
"result" -> case A.fromJSON (A.Object fields) of
|
||||
A.Success result -> Right (RespResult result)
|
||||
A.Error message -> Left (T.pack message)
|
||||
"normalization" -> case KM.lookup "analysis" fields of
|
||||
Nothing -> Left "normalization response is missing analysis"
|
||||
Just value -> either (Left . T.pack) (Right . RespNormalization) (parseEither parseNormalizationReport value)
|
||||
"catalog" -> fromField "schemas" RespCatalog fields
|
||||
"databases" -> fromField "databases" RespDatabases fields
|
||||
"roles" -> fromField "roles" RespRoles fields
|
||||
|
||||
@@ -10,15 +10,19 @@ module ChuSQL.Interface.Session (
|
||||
authenticateSudoSession,
|
||||
switchDatabase,
|
||||
runStatement,
|
||||
analyzeNormalization,
|
||||
catalog,
|
||||
databases,
|
||||
roleViews,
|
||||
accounts,
|
||||
) where
|
||||
|
||||
import ChuSQL.Interface.Normalization
|
||||
import qualified Data.Aeson as A
|
||||
import ChuSQL.Core.Protocol (Account, QueryResult (..), TableInfo)
|
||||
import ChuSQL.Interface.Link (
|
||||
Client,
|
||||
clientNormalization,
|
||||
clientAccounts,
|
||||
clientCatalog,
|
||||
clientDatabases,
|
||||
@@ -119,3 +123,10 @@ roleViews session = clientRoles (ssClient session)
|
||||
-- | 账号清单(\\du 与管理页用)
|
||||
accounts :: Session -> IO (Either Text [Account])
|
||||
accounts session = clientAccounts (ssClient session)
|
||||
|
||||
-- 使用当前连接及工作库请求范式分析。
|
||||
analyzeNormalization :: Session -> NormalizationRequest -> IO (Either Text A.Value)
|
||||
analyzeNormalization session payload = do
|
||||
current <- sessionDatabase session
|
||||
if current /= nrDatabase payload then pure (Left "bad_request: normalization database differs from the selected database")
|
||||
else clientNormalization (ssClient session) payload
|
||||
@@ -14,6 +14,7 @@ library
|
||||
exposed-modules: ChuSQL.Server.Accounts
|
||||
ChuSQL.Server.Backend
|
||||
ChuSQL.Server.Catalog
|
||||
ChuSQL.Server.Normalization
|
||||
ChuSQL.Server.Policy
|
||||
ChuSQL.Server.Privileges
|
||||
ChuSQL.Server.ObjectACL
|
||||
|
||||
@@ -41,7 +41,7 @@ import ChuSQL.Core.Engine.Syntax.AST
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
|
||||
import ChuSQL.Core.Protocol (TxnOp (..))
|
||||
import Control.Concurrent.MVar (MVar, modifyMVar, readMVar)
|
||||
import Data.List (nub)
|
||||
import Data.List (nub, isPrefixOf)
|
||||
import Data.Text (Text)
|
||||
import Data.Time (getCurrentTime)
|
||||
import Data.Time.Format (defaultTimeLocale, formatTime)
|
||||
@@ -75,6 +75,7 @@ data Backend = Backend
|
||||
-- | 按给定列名取列,固定列顺序
|
||||
columnsFromStatement :: Database -> Statement -> [String]
|
||||
columnsFromStatement db stmt = case prepare db stmt of
|
||||
Right (SelectDistinct q) -> columnsFromStatement db q
|
||||
Right (Select cols _ _ _ _ _) -> cols
|
||||
Right (SelectExpr items _ _ _ _ _) -> map fst items
|
||||
Right ShowDomains -> ["domain", "base_type"]
|
||||
@@ -173,9 +174,9 @@ memoryBackendWith trusted name ref =
|
||||
Right (db', rows)
|
||||
| any (\(key, _) -> hidden key && not (domainKey key)) db' -> pure (db, Left "reserved system table")
|
||||
| otherwise -> pure (syncMemoryObjects name (mergeScoped db db'), Right (StatementResult (colsOf (scoped db) stmt rows) rows))
|
||||
, beCatalog = do
|
||||
db <- readMVar ref
|
||||
pure (Right (map tableInfoOf (filter (not . internalTable . fst) (scoped db))))
|
||||
, beCatalog = modifyMVar ref $ \original -> do
|
||||
let db = syncMemoryObjects name original
|
||||
pure (db, Right (map tableInfoOf (filter (not . internalTable . fst) (scoped db))))
|
||||
, bePing = pure True
|
||||
, beAccounts = \req -> do
|
||||
stamp <- currentStamp
|
||||
@@ -187,7 +188,9 @@ memoryBackendWith trusted name ref =
|
||||
, beWithDatabase = withDatabase
|
||||
, beDatabases = pure (Right (nub [name, systemDatabaseName]))
|
||||
, beStorage = memorySecurityRequest name ref
|
||||
, beSnapshot = Right . scoped <$> readMVar ref
|
||||
, beSnapshot = modifyMVar ref $ \original -> do
|
||||
let db = syncMemoryObjects name original
|
||||
pure (db, Right (scoped db))
|
||||
, beApplyTransaction = \ops -> modifyMVar ref $ \db -> case applyTransactionMemory ops (scoped db) of
|
||||
Left err -> pure (db, Left err)
|
||||
Right db' -> pure (mergeScoped db db', Right ())
|
||||
@@ -226,7 +229,8 @@ memoryBackendWith trusted name ref =
|
||||
|
||||
-- 更新内存夹具的稳定表编号。
|
||||
syncMemoryObjects :: String -> Database -> Database
|
||||
syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id", TInt)] records Nothing) : filter ((/= registry) . fst) db
|
||||
syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id", TInt)] records Nothing Nothing Nothing)
|
||||
: map identify (filter ((/= registry) . fst) db)
|
||||
where
|
||||
registry = "__system_object_ids"
|
||||
previous = [(key, identifier) | row <- maybe [] tableRows (lookup registry db), Just (VStr key) <- [lookup "name" row], Just (VInt identifier) <- [lookup "id" row]]
|
||||
@@ -234,6 +238,11 @@ syncMemoryObjects database db = (registry, Table registry [("name", TStr), ("id"
|
||||
retained = [(key, identifier) | (key, identifier) <- previous, key `elem` keys]
|
||||
high = maximum (0 : map snd previous)
|
||||
added = zip [key | key <- keys, key `notElem` map fst retained] [high + 1 ..]
|
||||
-- | 将稳定目录编号附到表定义
|
||||
identify (key, table) = case lookup (if '.' `elem` key then key else database ++ "." ++ key) (retained ++ added) of
|
||||
Just identifier -> (key, table {tableIdentity = Just (TableIdentity
|
||||
(if "system." `isPrefixOf` key || ('.' `notElem` key && database == "system") then 2 else 1) (fromIntegral identifier))})
|
||||
Nothing -> (key, table)
|
||||
records = [[("name", VStr key), ("id", VInt identifier)] | (key, identifier) <- ("__high_water", high + length added) : retained ++ added]
|
||||
|
||||
-- 执行内存夹具的权限快照协议。
|
||||
@@ -266,7 +275,7 @@ memorySecurityRequest database ref payload = modifyMVar ref $ \original -> do
|
||||
let previous = case lookup acl db of Nothing -> Nothing; Just table -> case tableRows table of [[("payload", VStr value)]] -> Just (T.pack value); _ -> Just "invalid ACL"
|
||||
if previous /= field "expected" then Left "object ACL changed concurrently" else do
|
||||
value <- maybe (Left "missing ACL payload") Right (field "payload")
|
||||
Right ((acl, Table acl [("payload", TStr)] [[("payload", VStr (T.unpack value))]] Nothing) : filter ((/= acl) . fst) db, A.object ["status" A..= ("ok" :: Text)])
|
||||
Right ((acl, Table acl [("payload", TStr)] [[("payload", VStr (T.unpack value))]] Nothing Nothing Nothing) : filter ((/= acl) . fst) db, A.object ["status" A..= ("ok" :: Text)])
|
||||
_ -> Left "storage is not available in this backend"
|
||||
acl = "__system_object_acl"
|
||||
|
||||
@@ -320,6 +329,7 @@ applyTransactionMemory ops db = foldl step (Right db) ops
|
||||
-- | 语句里出现的表
|
||||
statementTables :: Statement -> [Text]
|
||||
statementTables stmt = nub (case stmt of
|
||||
SelectDistinct q -> statementTables q
|
||||
Select{} ->
|
||||
tableRefsOf (selectFrom stmt)
|
||||
++ concatMap exprTables (maybe [] (: []) (selectWhere stmt) ++ map Col (selectCols stmt ++ selectGroupBy stmt ++ map fst (selectOrderBy stmt)))
|
||||
@@ -440,6 +450,8 @@ tableInfoOf (name, tbl) =
|
||||
, tiRows = length (tableRows tbl)
|
||||
, tiIndexes = []
|
||||
, tiStats = [(c, distinctOf c, False) | (c, _) <- tableCols tbl]
|
||||
, tiIdentity = tableIdentity tbl
|
||||
, tiSchemaVersion = tableSchemaVersion tbl
|
||||
, tiHistograms = [(c, h) | (c, _) <- tableCols tbl, Just h <- [histogramOf (columnValues c)]]
|
||||
}
|
||||
where
|
||||
@@ -594,9 +606,9 @@ memoryAccounts db stamp req = do
|
||||
-- | 分开保存内存身份与口令并清理授权
|
||||
storeMemoryIdentities :: Database -> [Account] -> [Account] -> Database
|
||||
storeMemoryIdentities db previous accounts =
|
||||
(usersTable, Table usersTable [("id", TInt), ("password_hash", TStr)] credentials Nothing) :
|
||||
("__system_identities", Table "__system_identities" [("account", TStr)] identities Nothing) :
|
||||
("__system_identity_sequence", Table "__system_identity_sequence" [("id", TInt)] [[("id", VInt highWater)]] Nothing) :
|
||||
(usersTable, Table usersTable [("id", TInt), ("password_hash", TStr)] credentials Nothing Nothing Nothing) :
|
||||
("__system_identities", Table "__system_identities" [("account", TStr)] identities Nothing Nothing Nothing) :
|
||||
("__system_identity_sequence", Table "__system_identity_sequence" [("id", TInt)] [[("id", VInt highWater)]] Nothing Nothing Nothing) :
|
||||
[(name, clean name table) | (name, table) <- db, name `notElem` [usersTable, "__system_identities", "__system_identity_sequence"]]
|
||||
where
|
||||
-- | 保存口令与身份编号
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module ChuSQL.Server.Normalization (normalizationBudget, normalizationReport, schemaFingerprint) where
|
||||
|
||||
import qualified ChuSQL.Core.Engine.Normalization as N
|
||||
import ChuSQL.Core.Model (Column (..), parseColumnType)
|
||||
import ChuSQL.Core.Protocol (SchemaColumn (..), valueToJSON)
|
||||
import ChuSQL.Interface.Normalization
|
||||
import Control.Concurrent.MVar
|
||||
import Control.Concurrent.QSem
|
||||
import Control.Exception (bracket_, evaluate, mask)
|
||||
import Crypto.Hash (Digest, SHA256, hash)
|
||||
import qualified Data.Aeson as A
|
||||
import Data.Aeson ((.=), object)
|
||||
import qualified Data.ByteString.Lazy as BL
|
||||
import Data.List (nub, sort)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import System.IO.Unsafe (unsafePerformIO)
|
||||
import System.Timeout (timeout)
|
||||
|
||||
-- 范式服务:有界执行、稳定 schema 指纹与引擎报告的线上表示。
|
||||
|
||||
{-# NOINLINE analysisSlots #-}
|
||||
-- 保存最多八个请求和两个执行槽。
|
||||
analysisSlots :: (MVar Int, QSem)
|
||||
analysisSlots = unsafePerformIO ((,) <$> newMVar 0 <*> newQSem 2)
|
||||
|
||||
-- 限制排队、执行时间及输出大小。
|
||||
normalizationBudget :: IO (Either (Text, Text) A.Value) -> IO (Either (Text, Text) A.Value)
|
||||
normalizationBudget action = mask $ \restore -> do
|
||||
let (queued, slots) = analysisSlots
|
||||
admitted <- modifyMVar queued $ \count -> pure (if count < 8 then (count + 1, True) else (count, False))
|
||||
if not admitted then pure (Left ("too_busy", "normalization queue is full")) else
|
||||
bracket_ (pure ()) (modifyMVar_ queued (pure . subtract 1)) $ do
|
||||
result <- restore $ timeout 5000000 $ bracket_ (waitQSem slots) (signalQSem slots) $ do
|
||||
outcome <- action
|
||||
case outcome of
|
||||
Left err -> pure (Left err)
|
||||
Right value -> do
|
||||
size <- evaluate (BL.length (BL.take 64001 (A.encode value)))
|
||||
pure (if size > 64000 then Left ("too_large", "normalization report exceeds transport budget") else Right value)
|
||||
pure (maybe (Left ("timeout", "normalization exceeded its five second budget")) id result)
|
||||
|
||||
-- 编码只影响语义的列信息。
|
||||
fingerprintColumn :: SchemaColumn -> A.Value
|
||||
fingerprintColumn c = A.toJSON [A.toJSON (scName c), A.toJSON (scType c), A.toJSON (scNullable c),
|
||||
maybe A.Null valueToJSON (scDefault c), A.toJSON (scAutoIncrement c), A.toJSON (scPrimaryKey c), A.toJSON (scUnique c), A.toJSON (scCheck c)]
|
||||
|
||||
-- 将稳定编号与列结构绑定到 SHA256。
|
||||
schemaFingerprint :: ObjectIdentity -> [SchemaColumn] -> Text
|
||||
schemaFingerprint (ObjectIdentity database table) columns = T.pack (show (hash (BL.toStrict (A.encode (A.toJSON [A.toJSON [show database, show table], A.toJSON (map fingerprintColumn columns)]))) :: Digest SHA256))
|
||||
|
||||
-- 绑定 schema 并编码只读分析报告。
|
||||
normalizationReport :: ObjectIdentity -> [SchemaColumn] -> Text -> NormalizationRequest -> Either (Text, Text) A.Value
|
||||
normalizationReport identity columns view payload = do
|
||||
let fingerprint = schemaFingerprint identity columns
|
||||
if maybe False (/= identity) (nrExpectedObject payload) || maybe False (/= fingerprint) (nrExpectedSchema payload)
|
||||
then Left ("conflict", "normalization object or schema has changed") else Right ()
|
||||
schema <- mapM column columns
|
||||
let declaration = nrDeclaration payload
|
||||
axioms = N.Declaration (ndKeys declaration) [N.Dependency a b | NormalizationDependency a b <- ndDependencies declaration] (ndComplete declaration) (ndAtomic declaration)
|
||||
analysis <- either (Left . (,) "bad_request" . T.pack) Right (N.analyzeNormalization schema axioms)
|
||||
Right (object ["api_version" .= (1 :: Int), "database" .= nrDatabase payload, "table" .= last (T.splitOn "." (nrTable payload)),
|
||||
"object" .= identity, "schema_fingerprint" .= fingerprint, "view" .= view, "schema" .= columns,
|
||||
"declaration" .= canonicalDeclaration declaration, "candidate_keys" .= N.candidateKeys analysis,
|
||||
"findings" .= map findingJSON (N.findings analysis), "suggestions" .= map decompositionJSON (N.suggestions analysis),
|
||||
"conditional" .= True, "condition" .= ("Analysis is conditional on the declarations; SQL NULL, UNIQUE and bag semantics are not proofs of classical keys or lossless data migration." :: Text)])
|
||||
where
|
||||
-- 还原列类型及声明约束而不读取数据。
|
||||
column c = case parseColumnType (scType c) of
|
||||
Nothing -> Left ("storage_error", "normalization: invalid schema type")
|
||||
Just ty -> Right (scName c, Column ty (scNullable c) (scDefault c) (scAutoIncrement c) (scUnique c) (scPrimaryKey c) (scCheck c))
|
||||
|
||||
-- 规范化经验证的声明属性集合。
|
||||
canonicalDeclaration :: NormalizationDeclaration -> NormalizationDeclaration
|
||||
canonicalDeclaration d = d {ndKeys = sort (nub (map N.canonical (ndKeys d))),
|
||||
ndDependencies = [NormalizationDependency a b | (a, b) <- sort (nub [(N.canonical a, N.canonical b) | NormalizationDependency a b <- ndDependencies d])]}
|
||||
|
||||
-- 编码稳定范式标签及证据。
|
||||
findingJSON :: N.Finding -> A.Value
|
||||
findingJSON f = object ["normal_form" .= formTag (N.normalForm f), "verdict" .= verdictTag (N.verdict f),
|
||||
"witnesses" .= [NormalizationDependency a b | N.Dependency a b <- N.witnesses f], "condition" .= N.condition f]
|
||||
|
||||
-- 编码一次分解的理论结论。
|
||||
decompositionJSON :: N.Decomposition -> A.Value
|
||||
decompositionJSON d = object ["relations" .= N.relations d, "lossless_join" .= verdictTag (N.losslessJoin d), "dependency_preserving" .= verdictTag (N.dependencyPreserving d)]
|
||||
|
||||
-- 输出稳定的范式名称。
|
||||
formTag :: N.NormalForm -> Text
|
||||
formTag N.NF1 = "1nf"
|
||||
formTag N.NF2 = "2nf"
|
||||
formTag N.NF3 = "3nf"
|
||||
formTag N.BCNF = "bcnf"
|
||||
|
||||
-- 输出稳定的结论标签。
|
||||
verdictTag :: N.Verdict -> Text
|
||||
verdictTag N.Satisfied = "satisfied"
|
||||
verdictTag N.Violated = "violated"
|
||||
verdictTag N.Insufficient = "insufficient"
|
||||
@@ -2,8 +2,8 @@
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
|
||||
module ChuSQL.Server.ObjectACL
|
||||
( objectAuthorize, objectGrant, objectRevoke, objectViews, objectClaim
|
||||
, objectReconcile, objectAffected, objectOwner, objectConnect, objectPrepareCreate, objectUsers
|
||||
( objectAuthorize, objectGrant, objectRevoke, objectRevokeGrantOption, objectViews, objectClaim
|
||||
, objectSelfViews, objectReconcile, objectAffected, objectOwner, objectConnect, objectPrepareCreate, objectUsers
|
||||
) where
|
||||
|
||||
import ChuSQL.Core.Protocol (Account (..), TableInfo (..))
|
||||
@@ -315,19 +315,28 @@ objectGrant backend catalog principal database names object role option = do
|
||||
|
||||
-- 撤销授权并收敛失去来源的下游授权。
|
||||
objectRevoke :: Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
|
||||
objectRevoke backend catalog principal database names object role = do
|
||||
objectRevoke = revokeObject False
|
||||
|
||||
-- 仅撤销转授权能力并清理失效来源。
|
||||
objectRevokeGrantOption :: Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
|
||||
objectRevokeGrantOption = revokeObject True
|
||||
|
||||
-- 按授予来源撤销权限或转授权能力。
|
||||
revokeObject :: Bool -> Backend -> Catalog -> Principal -> Text -> [Text] -> Text -> Text -> IO (Either String ())
|
||||
revokeObject optionOnly backend catalog principal database names object role = do
|
||||
loaded <- load backend catalog
|
||||
resolved <- resolve backend database object
|
||||
case (,) <$> loaded <*> resolved of
|
||||
Left err -> pure (Left err)
|
||||
Right ((env, previous, acl), target) -> case (,) <$> roleId env role <*> privileges target names of
|
||||
Right ((env, previous, acl), target) -> case (,,) <$> roleId env role <*> privileges target names <*> (if principalIsRoot principal then Right "0" else roleId env (principalName principal)) of
|
||||
Left err -> pure (Left err)
|
||||
Right (recipient, items) -> do
|
||||
let grantor = either (const "") id (roleId env (principalName principal))
|
||||
manager = principalIsRoot principal || owns env acl grantor target
|
||||
Right (recipient, items, grantor) -> do
|
||||
let manager = principalIsRoot principal || owns env acl grantor target
|
||||
-- 选择本次可撤销的授予来源。
|
||||
matches edge = edgeRole edge == recipient && edgeTarget edge == target && edgePrivilege edge `elem` items && (manager || edgeGrantor edge == grantor)
|
||||
next = acl{aclEdges = filter (not . matches) (aclEdges acl), aclClosed = if recipient == "public" && "connect" `elem` items then nub (targetDatabase target : aclClosed acl) else aclClosed acl}
|
||||
next = acl{aclEdges = if optionOnly then map removeOption (aclEdges acl) else filter (not . matches) (aclEdges acl), aclClosed = if not optionOnly && recipient == "public" && "connect" `elem` items then nub (targetDatabase target : aclClosed acl) else aclClosed acl}
|
||||
-- 保留基础权限并移除匹配来源的转授权。
|
||||
removeOption edge = if matches edge then edge{edgeOption = False} else edge
|
||||
if recipient == "public" && "connect" `elem` items && not manager then pure (Left "administrator required")
|
||||
else writeACL backend previous (effective env next)
|
||||
|
||||
@@ -341,6 +350,21 @@ objectViews backend catalog role = do
|
||||
let mine = [edge | edge <- aclEdges acl, edgeRole edge == identity]
|
||||
Right (nub [Grant role (edgePrivilege edge) (targetName (edgeTarget edge)) (any (\other -> edgeTarget other == edgeTarget edge && edgePrivilege other == edgePrivilege edge && edgeOption other) mine) | edge <- mine])
|
||||
|
||||
-- 列出当前身份实际拥有的对象权限。
|
||||
objectSelfViews :: Backend -> Catalog -> Principal -> IO (Either String [Grant])
|
||||
objectSelfViews backend catalog principal = do
|
||||
loaded <- load backend catalog
|
||||
pure $ do
|
||||
(env, _, acl) <- loaded
|
||||
identity <- roleId env (principalName principal)
|
||||
if active env identity then Right () else Left "permission denied: disabled identity"
|
||||
-- 使用当前权限闭包判断普通及转授权权限。
|
||||
let permitted target privilege option = principalIsRoot principal || allowed env acl identity target privilege option
|
||||
Right [Grant (principalName principal) privilege (targetName target) (permitted target privilege True)
|
||||
| target <- contextTargets env
|
||||
, privilege <- if targetKind target == "database" then ["connect", "create"] else ["select", "insert", "update", "delete"]
|
||||
, permitted target privilege False]
|
||||
|
||||
-- 登记新对象的创建者为所有者。
|
||||
objectClaim :: Backend -> Catalog -> Principal -> Text -> Text -> IO (Either String ())
|
||||
objectClaim backend catalog principal database object = do
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
module ChuSQL.Server.Privileges
|
||||
( Grant (..), Membership (..), PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView (..)
|
||||
, affectedAccounts, authorize, authorizeTables, authorizeConnect, filterTables, listRoleViews
|
||||
, affectedAccounts, authorize, authorizeTables, authorizeConnect, filterTables, listRoleViews, listSelfGrants
|
||||
, newPrivileges, normalizeObject, privilegeCommand, runPrivilegeCommand, claimObject, prepareObject, reconcileObjects, affectedObject
|
||||
) where
|
||||
|
||||
@@ -22,7 +22,7 @@ import qualified Data.Text as T
|
||||
data PrivilegeError = PrivilegeError Text Text deriving (Show, Eq)
|
||||
data Membership = Membership { memberRole :: Text, memberUser :: Text } deriving (Show, Eq, Ord)
|
||||
data PrivilegeCommand = CreateRoleCommand Text | DropRoleCommand Text
|
||||
| GrantPrivilegesCommand [Text] Text Text Bool | RevokePrivilegesCommand [Text] Text Text
|
||||
| GrantPrivilegesCommand [Text] Text Text Bool | RevokePrivilegesCommand [Text] Text Text | RevokeGrantOptionCommand [Text] Text Text
|
||||
| GrantRoleCommand Text [Text] | RevokeRoleCommand Text [Text] deriving (Show, Eq)
|
||||
data Privileges = Privileges { pvCatalog :: Catalog, pvBackend :: Backend }
|
||||
|
||||
@@ -38,6 +38,7 @@ privilegeCommand statement = case statement of
|
||||
CreateRole name -> Just (CreateRoleCommand (T.pack name))
|
||||
DropRole name -> Just (DropRoleCommand (T.pack name))
|
||||
GrantPrivileges items object role option -> Just (GrantPrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role) option)
|
||||
RevokeGrantOption items object role -> Just (RevokeGrantOptionCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role))
|
||||
RevokePrivileges items object role -> Just (RevokePrivilegesCommand (map (T.toLower . T.pack) items) (T.pack object) (T.pack role))
|
||||
GrantRole role members -> Just (GrantRoleCommand (T.pack role) (map T.pack members))
|
||||
RevokeRole role members -> Just (RevokeRoleCommand (T.pack role) (map T.pack members))
|
||||
@@ -47,6 +48,7 @@ privilegeCommand statement = case statement of
|
||||
runPrivilegeCommand :: Privileges -> Principal -> Text -> PrivilegeCommand -> IO (Either PrivilegeError ())
|
||||
runPrivilegeCommand service principal database command = withTables service $ case command of
|
||||
GrantPrivilegesCommand items object role option -> catalog (OA.objectGrant backend directory principal database items object role option)
|
||||
RevokeGrantOptionCommand items object role -> catalog (OA.objectRevokeGrantOption backend directory principal database items object role)
|
||||
RevokePrivilegesCommand items object role -> catalog (OA.objectRevoke backend directory principal database items object role)
|
||||
CreateRoleCommand role
|
||||
| principalIsCatalogManager principal -> do
|
||||
@@ -106,6 +108,7 @@ affectedAccounts service command = withTables service (catalog (OA.objectAffecte
|
||||
CreateRoleCommand _ -> []
|
||||
DropRoleCommand role -> [role]
|
||||
GrantPrivilegesCommand _ _ role _ -> [role]
|
||||
RevokeGrantOptionCommand _ _ role -> [role]
|
||||
RevokePrivilegesCommand _ _ role -> [role]
|
||||
GrantRoleCommand _ members -> members
|
||||
RevokeRoleCommand _ members -> members
|
||||
@@ -195,6 +198,7 @@ reconcileObjects service = withTables service (catalog (OA.objectReconcile (pvBa
|
||||
-- 收集数据语句及子查询所需的表权限。
|
||||
requiredActions :: Statement -> Maybe [(Text, Text)]
|
||||
requiredActions statement = case statement of
|
||||
SelectDistinct q -> requiredActions q
|
||||
Select{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement)))
|
||||
SelectExpr{} -> Just (selectNeeds (selectFrom statement) (maybe [] (: []) (selectWhere statement) ++ map snd (selectItems statement)))
|
||||
Insert table _ rows -> Just ((T.pack table, "insert") : selectsOf (concat rows))
|
||||
@@ -231,3 +235,7 @@ storageError message = PrivilegeError code (T.pack message)
|
||||
|
||||
forbidden :: PrivilegeError
|
||||
forbidden = PrivilegeError "forbidden" "administrator required"
|
||||
|
||||
-- 查询登录身份自己的有效权限。
|
||||
listSelfGrants :: Privileges -> Principal -> IO (Either PrivilegeError [Grant])
|
||||
listSelfGrants service principal = withTables service (catalog (OA.objectSelfViews (pvBackend service) (pvCatalog service) principal))
|
||||
@@ -18,6 +18,7 @@ module ChuSQL.Server.Session
|
||||
, runStatement
|
||||
, runStatementCoded
|
||||
, runStorageCoded
|
||||
, analyzeNormalizationCoded
|
||||
, catalog
|
||||
, accounts
|
||||
, roleViews
|
||||
@@ -27,7 +28,12 @@ module ChuSQL.Server.Session
|
||||
, queryResultJson
|
||||
) where
|
||||
|
||||
import ChuSQL.Core.Model (Database, Row, Table (..), Value (..))
|
||||
import ChuSQL.Core.Model (Database, Row, Table (..), Value (..), Column (..), typeName)
|
||||
import ChuSQL.Core.Protocol (SchemaColumn (..))
|
||||
import ChuSQL.Interface.Normalization
|
||||
import ChuSQL.Server.Normalization (normalizationBudget, normalizationReport)
|
||||
import Data.Aeson.Types (parseEither)
|
||||
import qualified Data.ByteString.Lazy as BL
|
||||
import ChuSQL.Core.Protocol (Account (..), Request (..), QueryResult (..), TxnOp (..), queryResultJson)
|
||||
import ChuSQL.Core.Engine.Storage.IPC (TableInfo (..))
|
||||
import qualified ChuSQL.Core.Engine as Engine
|
||||
@@ -39,7 +45,7 @@ import ChuSQL.Interface.Auth (defaultSessionPolicy, newSessionStore)
|
||||
import ChuSQL.Server.Accounts
|
||||
import ChuSQL.Server.Backend (Backend (..), StatementResult (..), columnsFromStatement, scopeDdl)
|
||||
import ChuSQL.Server.Policy (configurePasswordPolicy)
|
||||
import ChuSQL.Server.Privileges (PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView, affectedAccounts, affectedObject, authorize, authorizeConnect, claimObject, prepareObject, reconcileObjects, filterTables, listRoleViews, newPrivileges, privilegeCommand, runPrivilegeCommand)
|
||||
import ChuSQL.Server.Privileges (PrivilegeCommand (..), PrivilegeError (..), Privileges, RoleView, Grant (..), affectedAccounts, affectedObject, authorize, authorizeTables, authorizeConnect, claimObject, prepareObject, reconcileObjects, filterTables, listRoleViews, listSelfGrants, newPrivileges, privilegeCommand, runPrivilegeCommand)
|
||||
import ChuSQL.Server.Security (readSecurity, writeSecurity)
|
||||
import Control.Exception (IOException, try)
|
||||
import Data.Bifunctor (first)
|
||||
@@ -280,6 +286,7 @@ runStatementUnlocked session sql = case parseStatement (T.unpack sql) of
|
||||
Right (Savepoint name) -> savepointTransaction session (T.pack name)
|
||||
Right (RollbackToSavepoint name) -> rollbackToSavepoint session (T.pack name)
|
||||
Right (ReleaseSavepoint name) -> releaseSavepoint session (T.pack name)
|
||||
Right ShowGrants -> showSelfGrants session
|
||||
Right ShowRoles -> showIdentities session
|
||||
Right statement
|
||||
| Just command <- accountCommand statement -> runAccount session command
|
||||
@@ -319,6 +326,7 @@ transactionDdl statement = case statement of
|
||||
-- | 事务里只有数据语句走快照
|
||||
transactionData :: Statement -> Bool
|
||||
transactionData statement = case statement of
|
||||
SelectDistinct q -> transactionData q
|
||||
Select{} -> True
|
||||
SelectExpr{} -> True
|
||||
Insert{} -> True
|
||||
@@ -840,3 +848,72 @@ resultOf result =
|
||||
-- | 按列名取单元格
|
||||
cellOf row name = maybe VNull id (lookup name row)
|
||||
|
||||
|
||||
-- 展示当前登录身份自己的有效权限。
|
||||
showSelfGrants :: Session -> IO (Either SessionError QueryResult)
|
||||
showSelfGrants session = do
|
||||
who <- readIORef (ssPrincipal session)
|
||||
case who of
|
||||
Nothing -> pure (Left (SessionError "unauthorized" "sign in first"))
|
||||
Just principal -> do
|
||||
grants <- listSelfGrants (ssPrivileges session) principal
|
||||
pure $ case grants of
|
||||
Left err -> Left (privilegeErrorOf err)
|
||||
Right rows -> Right (QueryResult ["identity", "privilege", "object", "grant_option"]
|
||||
[[VStr (T.unpack (grantRole grant)), VStr (T.unpack (grantPrivilege grant)), VStr (T.unpack (grantObject grant)), VBool (grantable grant)] | grant <- rows]
|
||||
(length rows) False Nothing)
|
||||
|
||||
|
||||
-- 在连接视图及权限稳定期间分析范式。
|
||||
analyzeNormalizationCoded :: Session -> NormalizationRequest -> IO (Either SessionError A.Value)
|
||||
analyzeNormalizationCoded session payload = do
|
||||
result <- normalizationBudget $ readSecurity $ do
|
||||
valid <- validatePrincipal session
|
||||
current <- readIORef (ssCurrent session)
|
||||
who <- readIORef (ssPrincipal session)
|
||||
let table = last (T.splitOn "." (nrTable payload))
|
||||
case (valid, who, A.fromJSON (A.toJSON payload) :: A.Result NormalizationRequest) of
|
||||
(Left err, _, _) -> pure (Left (sessCode err, sessMessage err))
|
||||
(_, Nothing, _) -> pure (Left ("unauthorized", "sign in first"))
|
||||
(_, _, A.Error message) -> pure (Left ("bad_request", T.pack message))
|
||||
(_, Just principal, A.Success _)
|
||||
| BL.length (A.encode payload) > 65000 -> pure (Left ("too_large", "normalization request exceeds transport budget"))
|
||||
| current /= nrDatabase payload -> pure (Left ("bad_request", "normalization database differs from the selected database"))
|
||||
| current == "system" || "__system_" `T.isPrefixOf` table -> pure (Left ("forbidden", "normalization does not expose system objects"))
|
||||
| otherwise -> do
|
||||
allowed <- authorizeTables (ssPrivileges session) principal current [(table, "select")]
|
||||
case allowed of
|
||||
Left (PrivilegeError code message) -> pure (Left (code, message))
|
||||
Right () -> normalizationView session table payload
|
||||
pure (either (Left . uncurry SessionError) Right result)
|
||||
|
||||
-- 读取目录结构或当前连接的事务 schema。
|
||||
normalizationView :: Session -> Text -> NormalizationRequest -> IO (Either (Text, Text) A.Value)
|
||||
normalizationView session table payload = do
|
||||
live <- snapshotObjectIds session [T.unpack table]
|
||||
transaction <- readIORef (ssTransaction session)
|
||||
case live >>= maybe (Left (SessionError "not_found" "unknown table")) Right . lookup (T.unpack table) of
|
||||
Left err -> pure (Left (sessCode err, sessMessage err))
|
||||
Right object -> case objectIdentity object of
|
||||
Left message -> pure (Left ("storage_error", message))
|
||||
Right identity -> case transaction of
|
||||
Just tx -> case (lookup (T.unpack table) (txStaged tx), lookup (T.unpack table) (txObjects tx)) of
|
||||
(Just relation, Just oldObject) | objectIdentity oldObject == Right identity ->
|
||||
pure (normalizationReport identity (map normalizationColumn (tableCols relation)) "transaction_snapshot" payload)
|
||||
_ -> pure (Left ("conflict", "normalization snapshot object has changed"))
|
||||
Nothing -> do
|
||||
backend <- currentBackend session
|
||||
infos <- beCatalog backend
|
||||
pure $ case infos of
|
||||
Left message -> Left ("storage_error", T.pack message)
|
||||
Right relations -> case [tiColumns info | info <- relations, tiTable info == T.unpack table] of
|
||||
[columns] -> normalizationReport identity columns "live_schema" payload
|
||||
_ -> Left ("not_found", "unknown table")
|
||||
|
||||
-- 从目录响应中只取稳定编号。
|
||||
objectIdentity :: A.Value -> Either Text ObjectIdentity
|
||||
objectIdentity = first T.pack . parseEither (A.withObject "catalog object" $ \o -> ObjectIdentity <$> o A..: "database_id" <*> o A..: "object_id")
|
||||
|
||||
-- 将事务 schema 转成同一线上列结构。
|
||||
normalizationColumn :: (String, Column) -> SchemaColumn
|
||||
normalizationColumn (name, c) = SchemaColumn name (typeName (columnType c)) (columnNullable c) (columnDefault c) (columnAutoIncrement c) (columnPrimaryKey c) (columnUnique c) (columnCheck c)
|
||||
@@ -23,7 +23,7 @@ import ChuSQL.Interface.RateLimit (RateLimiter, rateLimitBlock, rateLimitClear,
|
||||
import qualified ChuSQL.Core.Engine.Error as E
|
||||
import ChuSQL.Core.Protocol (Account (..), Request (ReqIdentityInitialize))
|
||||
import ChuSQL.Server.Backend (Backend (..))
|
||||
import ChuSQL.Server.Session (QueryResult (..), Session, SessionError (..), accounts, authenticateSessionCoded, authenticateSudoSessionCoded, isPlainIdentifier, catalog, databases, newSessionWith, policyOf, reloadPolicy, roleViews, runStatementCoded, runStorageCoded, sessionIsAdmin, sessionUser)
|
||||
import ChuSQL.Server.Session (QueryResult (..), Session, SessionError (..), accounts, authenticateSessionCoded, authenticateSudoSessionCoded, isPlainIdentifier, catalog, databases, newSessionWith, policyOf, reloadPolicy, roleViews, analyzeNormalizationCoded, runStatementCoded, runStorageCoded, sessionIsAdmin, sessionUser)
|
||||
import ChuSQL.Server.Security (writeSecurity)
|
||||
import ChuSQL.Server.Privileges (newPrivileges, reconcileObjects)
|
||||
import Control.Concurrent (forkIO)
|
||||
@@ -331,6 +331,9 @@ dispatch env session entry request = case request of
|
||||
ReqSudoLogin user proof -> do
|
||||
response <- sudoLogin env session entry user proof
|
||||
pure (response, True)
|
||||
ReqNormalization payload -> do
|
||||
result <- analyzeNormalizationCoded session payload
|
||||
pure (either (\err -> RespError (sessCode err) (sessMessage err)) RespNormalization result, True)
|
||||
ReqQuery sql -> do
|
||||
who <- sessionUser session
|
||||
response <- case who of
|
||||
|
||||
+242
-9
@@ -12,11 +12,14 @@ import ChuSQL.Core.Engine.Storage.IPC (
|
||||
setStorageLink,
|
||||
)
|
||||
import ChuSQL.Core.Engine.Syntax.AST (Statement)
|
||||
import ChuSQL.Interface.Normalization
|
||||
import ChuSQL.Server.Normalization (normalizationBudget, normalizationReport, schemaFingerprint)
|
||||
import ChuSQL.Core.Protocol (SchemaColumn (..))
|
||||
import Data.IORef (newIORef, readIORef, writeIORef)
|
||||
import ChuSQL.Core.Engine.Syntax.Parser (parseStatement)
|
||||
import ChuSQL.Core.Model (Database, Histogram (..), Row, Table (..), Value (..), pattern TInt, pattern TStr)
|
||||
import ChuSQL.Core.Model (Database, Histogram (..), Row, Table (..), TableIdentity (..), Value (..), pattern TInt, pattern TStr)
|
||||
import ChuSQL.Interface.Auth (hashPasswordWith)
|
||||
import ChuSQL.Interface.Protocol (ClientRequest (..), Grant (..), RoleView (..), ServerResponse (..), decodeRequest, encodeResponse)
|
||||
import ChuSQL.Interface.Protocol (ClientRequest (..), Grant (..), RoleView (..), ServerResponse (..), decodeRequest, encodeResponse, encodeRequest, decodeResponse)
|
||||
import ChuSQL.Interface.RateLimit (newRateLimiter)
|
||||
import ChuSQL.Interface.Link (connectClient, closeClient, clientSudoLogin, clientQuery)
|
||||
import ChuSQL.Interface.Sudo (SudoCredential, newSudoCredential, sudoProof, verifySudoProof, validSudoChallenge,
|
||||
@@ -33,6 +36,7 @@ import ChuSQL.Server.Privileges (
|
||||
claimObject,
|
||||
prepareObject,
|
||||
listRoleViews,
|
||||
listSelfGrants,
|
||||
newPrivileges,
|
||||
runPrivilegeCommand,
|
||||
)
|
||||
@@ -47,7 +51,7 @@ import ChuSQL.Server.TCP (
|
||||
startServer,
|
||||
isLocalPeer,
|
||||
)
|
||||
import Control.Concurrent (forkIO, threadDelay)
|
||||
import Control.Concurrent (forkIO, forkFinally, killThread, threadDelay)
|
||||
import Control.Concurrent.MVar (MVar, modifyMVar_, newEmptyMVar, newMVar, putMVar, takeMVar)
|
||||
import System.Timeout (timeout)
|
||||
import Control.Exception (IOException, bracket, finally, try)
|
||||
@@ -64,7 +68,7 @@ import Data.Maybe (fromMaybe)
|
||||
import Data.Text (Text)
|
||||
import qualified Data.Text as T
|
||||
import qualified Data.Vector as V
|
||||
import Data.Time.Clock (addUTCTime, getCurrentTime)
|
||||
import Data.Time.Clock (addUTCTime, diffUTCTime, getCurrentTime)
|
||||
import Network.Socket (
|
||||
AddrInfo (..),
|
||||
SockAddr (..),
|
||||
@@ -105,6 +109,7 @@ main = hspec spec
|
||||
-- | 全部用例
|
||||
spec :: Spec
|
||||
spec = do
|
||||
normalizationSpec
|
||||
describe "sudo authentication" $ do
|
||||
it "defaults to disabled and rejects unconfigured TCP authentication" $
|
||||
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \client -> do
|
||||
@@ -380,14 +385,14 @@ testDb =
|
||||
[ [("id", VInt i), ("name", VStr ("user" ++ show i)), ("age", VInt (20 + i))]
|
||||
| i <- [1 .. 5]
|
||||
]
|
||||
Nothing
|
||||
Nothing Nothing Nothing
|
||||
)
|
||||
, ( "orders"
|
||||
, Table
|
||||
"orders"
|
||||
[("id", TInt), ("user_id", TInt)]
|
||||
[[("id", VInt 1), ("user_id", VInt 1)]]
|
||||
Nothing
|
||||
Nothing Nothing Nothing
|
||||
)
|
||||
]
|
||||
|
||||
@@ -1125,6 +1130,44 @@ withObjectPrivileges body = do
|
||||
-- 验证对象作用域、授权来源和外部入口。
|
||||
objectPrivilegeSpec :: Spec
|
||||
objectPrivilegeSpec = describe "stable object privileges" $ do
|
||||
it "lists only the caller's effective grants including roles and PUBLIC" $ withObjectPrivileges $ \backend service root -> do
|
||||
runPrivilegeCommand service root testDatabase (CreateRoleCommand "self_reader") `shouldReturn` Right ()
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "self_reader" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service root testDatabase (GrantRoleCommand "self_reader" ["alice"]) `shouldReturn` Right ()
|
||||
grants <- listSelfGrants service (Ordinary testAccount)
|
||||
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "select" "test.users" True))
|
||||
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "connect" "database:test" False))
|
||||
grants `shouldSatisfy` either (const False) (all ((== "alice") . grantRole))
|
||||
runPrivilegeCommand service root testDatabase (RevokePrivilegesCommand ["select"] "users" "self_reader") `shouldReturn` Right ()
|
||||
after <- listSelfGrants service (Ordinary testAccount)
|
||||
after `shouldSatisfy` either (const False) (not . any ((== "test.users") . grantObject))
|
||||
_ <- beAccounts backend (ReqAccountReset "alice" (testPasswordHash "Alice-Secret-1234"))
|
||||
_ <- beAccounts backend (ReqAccountCreate testRootName (testPasswordHash testRootPassword))
|
||||
_ <- beAccounts backend (ReqIdentityAlter testRootName Nothing (Just True) Nothing Nothing Nothing)
|
||||
settings <- tempSettingsPath "self-grants"
|
||||
session <- newSession backend testRootName settings
|
||||
authenticateSessionCoded session "alice" "Alice-Secret-1234" `shouldReturn` Right ()
|
||||
result <- mustSql session "SHOW GRANTS"
|
||||
qrColumns result `shouldBe` ["identity", "privilege", "object", "grant_option"]
|
||||
qrRows result `shouldSatisfy` all (\row -> take 1 row == [VStr "alice"])
|
||||
rejected <- runStatementCoded session "SHOW GRANTS FOR admin"
|
||||
rejected `shouldSatisfy` either (const True) (const False)
|
||||
it "includes ownership permissions and excludes ungranted tables" $ withObjectPrivileges $ \backend service _ -> do
|
||||
mustRun (beStatement (beWithDatabase backend "test") "CREATE TABLE personal_owned (id INT)")
|
||||
claimObject service (Ordinary testAccount) testDatabase "personal_owned" `shouldReturn` Right ()
|
||||
grants <- listSelfGrants service (Ordinary testAccount)
|
||||
fmap (filter ((== "test.personal_owned") . grantObject)) grants `shouldBe` Right
|
||||
[Grant "alice" privilege "test.personal_owned" True | privilege <- ["select", "insert", "update", "delete"]]
|
||||
grants `shouldSatisfy` either (const False) (not . any ((== "test.users") . grantObject))
|
||||
|
||||
it "checks DISTINCT table reads and nested subqueries" $ withObjectPrivileges $ \_ service root -> do
|
||||
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT age FROM users"
|
||||
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
|
||||
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT d.age FROM (SELECT DISTINCT age FROM users) d"
|
||||
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" False) `shouldReturn` Right ()
|
||||
authorizeSql service (Ordinary testAccount) "SELECT DISTINCT age FROM users" `shouldReturn` Right ()
|
||||
|
||||
it "migrates old table grants once and ignores later legacy writes" $ withObjectPrivileges $ \backend service _ -> do
|
||||
let system = beWithDatabase backend "system"
|
||||
mustRun (beStatement system "CREATE TABLE __system_grants (role VARCHAR(64), privilege VARCHAR(16), object VARCHAR(128))")
|
||||
@@ -1204,6 +1247,41 @@ objectPrivilegeSpec = describe "stable object privileges" $ do
|
||||
authorizeSql service (Ordinary testAccount) "DROP DATABASE test" `shouldReturn` Left (PrivilegeError "forbidden" "object owner required")
|
||||
runPrivilegeCommand service root testDatabase (RevokePrivilegesCommand ["select"] "users" "alice") `shouldReturn` Right ()
|
||||
authorizeSql service (Ordinary testAccount) "SELECT * FROM users" `shouldReturn` Right ()
|
||||
it "executes grant option revocation over TCP and keeps SELECT" $
|
||||
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \admin -> do
|
||||
_ <- tcpLogin admin "admin" "s3cret"
|
||||
_ <- tcpQuery admin "USE test"
|
||||
_ <- tcpQuery admin "CREATE USER option_reader IDENTIFIED BY 'Option-Reader-1234'"
|
||||
tcpQuery admin "GRANT SELECT ON users TO option_reader WITH GRANT OPTION" >>= \reply -> at "status" reply `shouldBe` A.String "result"
|
||||
tcpQuery admin "REVOKE GRANT OPTION FOR SELECT ON users FROM option_reader" >>= \reply -> at "status" reply `shouldBe` A.String "result"
|
||||
withConnection handle $ \reader -> do
|
||||
_ <- tcpLogin reader "option_reader" "Option-Reader-1234"
|
||||
_ <- tcpQuery reader "USE test"
|
||||
tcpQuery reader "SELECT DISTINCT age FROM users ORDER BY 1" >>= \reply -> at "status" reply `shouldBe` A.String "result"
|
||||
tcpQuery reader "SHOW GRANTS" >>= \reply -> at "status" reply `shouldBe` A.String "result"
|
||||
tcpQuery reader "GRANT SELECT ON users TO admin" >>= \reply -> at "code" reply `shouldBe` A.String "forbidden"
|
||||
it "revokes grant option while preserving data access and independent sources" $ withObjectPrivileges $ \_ service root -> do
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount{accountUser = "bob"}) testDatabase (GrantPrivilegesCommand ["select"] "users" "carol" False) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "carol" False) `shouldReturn` Right ()
|
||||
affectedAccounts service (RevokeGrantOptionCommand ["select"] "users" "alice") `shouldReturn` Right ["alice", "bob", "carol"]
|
||||
runPrivilegeCommand service root testDatabase (RevokeGrantOptionCommand ["select"] "users" "alice") `shouldReturn` Right ()
|
||||
authorizeSql service (Ordinary testAccount) "SELECT * FROM users" `shouldReturn` Right ()
|
||||
grants <- listSelfGrants service (Ordinary testAccount)
|
||||
grants `shouldSatisfy` either (const False) (elem (Grant "alice" "select" "test.users" False))
|
||||
authorizeSql service (Ordinary testAccount{accountUser = "bob"}) "SELECT * FROM users"
|
||||
`shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON users")
|
||||
authorizeSql service (Ordinary testAccount{accountUser = "carol"}) "SELECT * FROM users" `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" False)
|
||||
`shouldReturn` Left (PrivilegeError "forbidden" "grant option required: SELECT ON users")
|
||||
it "revokes only the caller's grant option source" $ withObjectPrivileges $ \_ service root -> do
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount) testDatabase (RevokeGrantOptionCommand ["select"] "users" "bob") `shouldReturn` Right ()
|
||||
grants <- listSelfGrants service (Ordinary testAccount{accountUser = "bob"})
|
||||
grants `shouldSatisfy` either (const False) (elem (Grant "bob" "select" "test.users" True))
|
||||
it "cascades multihop delegations after the source grant is revoked" $ withObjectPrivileges $ \_ service root -> do
|
||||
runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "alice" True) `shouldReturn` Right ()
|
||||
runPrivilegeCommand service (Ordinary testAccount) testDatabase (GrantPrivilegesCommand ["select"] "users" "bob" True) `shouldReturn` Right ()
|
||||
@@ -1436,6 +1514,17 @@ intValues rows = [n | row <- rows, Just (VInt n) <- [nth 0 row]]
|
||||
-- | 显式事务用例
|
||||
transactionSpec :: Spec
|
||||
transactionSpec = describe "server transactions" $ do
|
||||
it "executes DISTINCT against the connection snapshot with output columns" $ withMemorySession $ \session -> do
|
||||
_ <- mustSql session "CREATE TABLE distinct_values (id INT, value INT)"
|
||||
_ <- mustSql session "INSERT INTO distinct_values (id,value) VALUES (1,7),(2,7),(3,8)"
|
||||
_ <- mustSql session "BEGIN"
|
||||
_ <- mustSql session "UPDATE distinct_values SET value = 9 WHERE id = 3"
|
||||
result <- mustSql session "SELECT DISTINCT value FROM distinct_values ORDER BY 1"
|
||||
qrColumns result `shouldBe` ["value"]
|
||||
qrRows result `shouldBe` [[VInt 7], [VInt 9]]
|
||||
_ <- mustSql session "ROLLBACK"
|
||||
pure ()
|
||||
|
||||
it "preserves concurrent changes in a large untouched table" $ withTwoSessions $ \first second -> do
|
||||
_ <- mustSql first "CREATE TABLE untouched (id INT, value INT)"
|
||||
let values = T.intercalate "," [T.pack ("(" ++ show key ++ ",0)") | key <- [1 .. 5000 :: Int]]
|
||||
@@ -1458,8 +1547,8 @@ transactionSpec = describe "server transactions" $ do
|
||||
qrRows result `shouldBe` [[VInt 7], [VInt (-3)], [VInt 4]]
|
||||
|
||||
it "filters qualified memory tables and preserves other databases on writes" $ do
|
||||
let local = Table "users" [("id", TInt)] [[("id", VInt 1)]] Nothing
|
||||
otherTable = Table "users" [("id", TInt)] [[("id", VInt 2)]] Nothing
|
||||
let local = Table "users" [("id", TInt)] [[("id", VInt 1)]] Nothing Nothing Nothing
|
||||
otherTable = Table "users" [("id", TInt)] [[("id", VInt 2)]] Nothing Nothing Nothing
|
||||
ref <- newMVar [("test.users", local), ("other.users", otherTable)]
|
||||
let backend = memoryBackend "test" ref
|
||||
selected <- beStatement backend "SELECT id FROM users"
|
||||
@@ -1470,11 +1559,36 @@ transactionSpec = describe "server transactions" $ do
|
||||
fmap (const ()) updated `shouldBe` Right ()
|
||||
snapshot <- beSnapshot (memoryBackend "other" ref)
|
||||
fmap (map (tableRows . snd)) snapshot `shouldBe` Right [[[ ("id", VInt 2) ]]]
|
||||
it "keeps catalog and snapshot identities stable across system views and replacements" $ do
|
||||
ref <- newMVar [("test.users", Table "users" [("id", TInt)] [] Nothing Nothing Nothing)]
|
||||
let backend = memoryBackend "test" ref
|
||||
first <- beCatalog backend
|
||||
fmap (map tiIdentity) first `shouldBe` Right [Just (TableIdentity 1 1)]
|
||||
_ <- beSnapshot (beWithDatabase backend "system")
|
||||
saved <- beSnapshot backend
|
||||
fmap (map (tableIdentity . snd)) saved `shouldBe` Right [Just (TableIdentity 1 1)]
|
||||
mustRun (beStatement backend "DROP TABLE users")
|
||||
mustRun (beStatement backend "CREATE TABLE users (id INT)")
|
||||
replaced <- beCatalog backend
|
||||
fmap (map tiIdentity) replaced `shouldBe` Right [Just (TableIdentity 1 2)]
|
||||
it "preserves memory schema versions in catalogs and snapshots" $ do
|
||||
ref <- newMVar []
|
||||
let backend = memoryBackend "test" ref
|
||||
fmap (const ()) <$> beStatement backend "CREATE TABLE versioned (id INT)" `shouldReturn` Right ()
|
||||
first <- beCatalog backend
|
||||
fmap (map tiSchemaVersion) first `shouldBe` Right [Just 1]
|
||||
fmap (const ()) <$> beStatement backend "ALTER TABLE versioned ADD COLUMN extra TEXT" `shouldReturn` Right ()
|
||||
fmap (const ()) <$> beStatement backend "ALTER TABLE versioned DROP COLUMN extra" `shouldReturn` Right ()
|
||||
current <- beCatalog backend
|
||||
snapshot <- beSnapshot backend
|
||||
fmap (map tiSchemaVersion) current `shouldBe` Right [Just 3]
|
||||
fmap (map (tableSchemaVersion . snd)) snapshot `shouldBe` Right [Just 3]
|
||||
it "rejects cross-database reads and writes including nested queries" $ withMemorySession $ \session -> do
|
||||
mapM_ (\query -> do
|
||||
result <- runStatementCoded session query
|
||||
failureMessage result `shouldSatisfy` T.isInfixOf "outside the current database")
|
||||
[ "SELECT * FROM other.users"
|
||||
[ "SELECT DISTINCT age FROM other.users"
|
||||
, "SELECT * FROM other.users"
|
||||
, "UPDATE other.users SET age = 1"
|
||||
, "DELETE FROM other.users"
|
||||
, "INSERT INTO other.users (id) VALUES (99)"
|
||||
@@ -1962,3 +2076,122 @@ checkSudoDisabled admin handle credential sql = do
|
||||
isRightResult :: Either a b -> Bool
|
||||
isRightResult (Right _) = True
|
||||
isRightResult _ = False
|
||||
|
||||
|
||||
-- 专用范式协议、权限、快照与预算回归。
|
||||
normalizationSpec :: Spec
|
||||
normalizationSpec = describe "normalization endpoint" $ do
|
||||
it "round-trips versioned requests and rejects unknown and duplicate fields" $ do
|
||||
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload))) `shouldBe` Right (ReqNormalization normalizationPayload)
|
||||
let large = ObjectIdentity 9007199254740993 18446744073709551615
|
||||
A.fromJSON (A.toJSON large) `shouldBe` A.Success large
|
||||
case A.eitherDecode "{\"database_id\":1e1000000000,\"object_id\":2}" :: Either String A.Value of
|
||||
Left message -> expectationFailure message
|
||||
Right value -> (A.fromJSON value :: A.Result ObjectIdentity) `shouldSatisfy` isJSONFailure
|
||||
|
||||
report <- either (fail . show) pure (normalizationReport (ObjectIdentity 1 2) [SchemaColumn name "int" True Nothing False False False Nothing | name <- ["id", "name", "age"]] "live_schema" normalizationPayload)
|
||||
let response = RespNormalization report
|
||||
decodeResponse (encodeResponse response) `shouldBe` Right response
|
||||
decodeResponse (encodeResponse (RespNormalization (A.object ["conditional" A..= True]))) `shouldSatisfy` either (const True) (const False)
|
||||
mapM_ (\raw -> decodeNormalizationJSON raw `shouldSatisfy` either (T.isInfixOf "duplicate") (const False))
|
||||
[ "{\"a\":1,\"a\":2}", "{\"a\":1,\"\\u0061\":2}", "{\"a\":[{\"b\":1,\"b\":2}]}" ]
|
||||
let unknown = A.object ["api_version" A..= (1 :: Int), "database" A..= ("test" :: Text), "table" A..= ("users" :: Text), "declaration" A..= nrDeclaration normalizationPayload, "ignored" A..= True]
|
||||
(A.fromJSON unknown :: A.Result NormalizationRequest) `shouldSatisfy` isJSONFailure
|
||||
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrVersion = 2}))) `shouldSatisfy` either (const True) (const False)
|
||||
decodeRequest (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrTable = "other.users"}))) `shouldSatisfy` either (const True) (const False)
|
||||
it "does not infer completeness or atomicity and validates declaration budgets" $ do
|
||||
(A.fromJSON (A.object []) :: A.Result NormalizationDeclaration) `shouldBe` A.Success (NormalizationDeclaration [] [] False Nothing)
|
||||
let many = NormalizationDeclaration (replicate 257 ["id"]) [] True (Just True)
|
||||
(A.fromJSON (A.toJSON many) :: A.Result NormalizationDeclaration) `shouldSatisfy` isJSONFailure
|
||||
decodeNormalizationJSON (BL.replicate 65537 32) `shouldSatisfy` either (const True) (const False)
|
||||
(A.fromJSON (A.object ["dependencies" A..= [A.object ["lhs" A..= (["id"] :: [Text]), "rhs" A..= ([] :: [Text])]]]) :: A.Result NormalizationDeclaration) `shouldSatisfy` isJSONFailure
|
||||
it "authorizes ordinary SELECT users and refuses anonymous or system requests" $
|
||||
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \admin -> do
|
||||
anonymous <- tcpTalk admin (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
|
||||
at "code" anonymous `shouldBe` A.String "unauthorized"
|
||||
_ <- tcpLogin admin "admin" "s3cret"
|
||||
_ <- tcpQuery admin "USE test"
|
||||
_ <- tcpQuery admin "CREATE USER normalization_reader IDENTIFIED BY 'Reader-Secret-1234'"
|
||||
withConnection handle $ \reader -> do
|
||||
_ <- tcpLogin reader "normalization_reader" "Reader-Secret-1234"
|
||||
_ <- tcpQuery reader "USE test"
|
||||
denied <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
|
||||
at "code" denied `shouldBe` A.String "forbidden"
|
||||
_ <- tcpQuery admin "GRANT SELECT ON users TO normalization_reader"
|
||||
withConnection handle $ \reader -> do
|
||||
_ <- tcpLogin reader "normalization_reader" "Reader-Secret-1234"
|
||||
_ <- tcpQuery reader "USE test"
|
||||
reply <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload)))
|
||||
at "status" reply `shouldBe` A.String "normalization"
|
||||
at "conditional" (at "analysis" reply) `shouldBe` A.Bool True
|
||||
length (items (at "findings" (at "analysis" reply))) `shouldBe` 4
|
||||
denied <- tcpTalk reader (BL.toStrict (encodeRequest (ReqNormalization normalizationPayload{nrTable = "__system_domain_secret"})))
|
||||
at "code" denied `shouldBe` A.String "forbidden"
|
||||
it "uses only the caller snapshot and refuses stable object replacement" $
|
||||
withTcpServer defaultServerConfig $ \_ handle -> withConnection handle $ \first -> withConnection handle $ \second -> do
|
||||
_ <- tcpLogin first "admin" "s3cret"
|
||||
_ <- tcpLogin second "admin" "s3cret"
|
||||
_ <- tcpQuery first "USE test"
|
||||
_ <- tcpQuery second "USE test"
|
||||
_ <- tcpQuery first "CREATE TABLE normalization_snapshot (id INT, value INT)"
|
||||
_ <- tcpQuery first "BEGIN"
|
||||
_ <- tcpQuery second "ALTER TABLE normalization_snapshot ADD COLUMN added INT"
|
||||
let payload = normalizationPayload{nrTable = "normalization_snapshot", nrDeclaration = NormalizationDeclaration [] [] False Nothing}
|
||||
old <- tcpTalk first (BL.toStrict (encodeRequest (ReqNormalization payload)))
|
||||
current <- tcpTalk second (BL.toStrict (encodeRequest (ReqNormalization payload)))
|
||||
at "view" (at "analysis" old) `shouldBe` A.String "transaction_snapshot"
|
||||
length (items (at "schema" (at "analysis" old))) `shouldBe` 2
|
||||
length (items (at "schema" (at "analysis" current))) `shouldBe` 3
|
||||
at "schema_fingerprint" (at "analysis" old) `shouldNotBe` at "schema_fingerprint" (at "analysis" current)
|
||||
_ <- tcpQuery second "DROP TABLE normalization_snapshot"
|
||||
_ <- tcpQuery second "CREATE TABLE normalization_snapshot (id INT, value INT)"
|
||||
stale <- tcpTalk first (BL.toStrict (encodeRequest (ReqNormalization payload)))
|
||||
at "code" stale `shouldBe` A.String "conflict"
|
||||
_ <- tcpQuery first "ROLLBACK"
|
||||
pure ()
|
||||
it "checks expected schema fingerprints and budgets the complete output" $ do
|
||||
let columns = [SchemaColumn ("c" ++ show n) "int" True Nothing False False False Nothing | n <- [0 .. 9 :: Int]]
|
||||
payload = normalizationPayload{nrDeclaration = NormalizationDeclaration [] [NormalizationDependency [] ["c9"]] True (Just True)}
|
||||
schemaFingerprint (ObjectIdentity 1 2) columns `shouldNotBe` schemaFingerprint (ObjectIdentity 1 3) columns
|
||||
normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload{nrExpectedSchema = Just "outdated"}
|
||||
`shouldBe` Left ("conflict", "normalization object or schema has changed")
|
||||
result <- normalizationBudget (pure (normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload))
|
||||
result `shouldSatisfy` either ((== "too_large") . fst) (const False)
|
||||
it "bounds the largest schema with maximum declared dependencies" $ do
|
||||
let columns = [SchemaColumn ("c" ++ show n) "int" True Nothing False False False Nothing | n <- [0 .. 11 :: Int]]
|
||||
payload = normalizationPayload{nrDeclaration = NormalizationDeclaration [] (replicate 256 (NormalizationDependency [] ["c11"])) True (Just True)}
|
||||
started <- getCurrentTime
|
||||
result <- normalizationBudget (pure (normalizationReport (ObjectIdentity 1 2) columns "live_schema" payload))
|
||||
ended <- getCurrentTime
|
||||
putStrLn ("normalization maximum input seconds: " ++ show (ended `diffUTCTime` started))
|
||||
result `shouldSatisfy` either ((`elem` ["too_large", "timeout"]) . fst) (const False)
|
||||
normalizationBudget (pure (Right A.Null)) `shouldReturn` Right A.Null
|
||||
it "releases admission and execution slots after cancellation" $ do
|
||||
started <- newEmptyMVar
|
||||
blocker <- newEmptyMVar
|
||||
done <- newEmptyMVar
|
||||
worker <- forkFinally (normalizationBudget (putMVar started () >> takeMVar blocker >> pure (Right A.Null))) (putMVar done)
|
||||
takeMVar started
|
||||
killThread worker
|
||||
takeMVar done >>= either (const (pure ())) (const (expectationFailure "cancelled analysis completed"))
|
||||
ready <- newEmptyMVar
|
||||
release <- newEmptyMVar
|
||||
finished <- newEmptyMVar
|
||||
replicateM_ 2 (void (forkFinally (normalizationBudget (putMVar ready () >> takeMVar release >> pure (Right A.Null))) (putMVar finished)))
|
||||
mapM_ (\_ -> timeout 1000000 (takeMVar ready) >>= (`shouldBe` Just ())) [1, 2 :: Int]
|
||||
replicateM_ 2 (putMVar release ())
|
||||
replicateM_ 2 (takeMVar finished >>= either (expectationFailure . show) (`shouldBe` Right A.Null))
|
||||
it "times out work and leaves the service usable" $ do
|
||||
result <- normalizationBudget (threadDelay 6000000 >> pure (Right A.Null))
|
||||
result `shouldSatisfy` either ((== "timeout") . fst) (const False)
|
||||
normalizationBudget (pure (Right A.Null)) `shouldReturn` Right A.Null
|
||||
|
||||
-- 共用明确声明的用户表请求。
|
||||
normalizationPayload :: NormalizationRequest
|
||||
normalizationPayload = NormalizationRequest 1 "test" "users"
|
||||
(NormalizationDeclaration [["id"]] [NormalizationDependency ["id"] ["name", "age"]] True (Just True)) Nothing Nothing
|
||||
|
||||
-- 判断 JSON 类型校验失败。
|
||||
isJSONFailure :: A.Result a -> Bool
|
||||
isJSONFailure (A.Error _) = True
|
||||
isJSONFailure _ = False
|
||||
@@ -72,10 +72,12 @@ import ChuSQL.Interface.Link (
|
||||
closeClient,
|
||||
connectClient,
|
||||
)
|
||||
import ChuSQL.Interface.Normalization (NormalizationRequest, decodeNormalizationJSON)
|
||||
import ChuSQL.Interface.Protocol (Grant (..), RoleView (..))
|
||||
import ChuSQL.Interface.RateLimit (RateLimiter, rateLimitBlock, rateLimitClear, rateLimitRecord, setRateLimit)
|
||||
import ChuSQL.Interface.Session (
|
||||
Session,
|
||||
analyzeNormalization,
|
||||
authenticateSession,
|
||||
catalog,
|
||||
databases,
|
||||
@@ -132,7 +134,7 @@ import qualified Data.Text.Encoding.Error as TEE
|
||||
import qualified Data.Text.Lazy as TL
|
||||
import Data.Time.Clock (getCurrentTime)
|
||||
import qualified Data.Vector as V
|
||||
import Network.HTTP.Types (Status, status204, status400, status401, status403, status404, status409, status413, status429, status500)
|
||||
import Network.HTTP.Types (Status, status204, status400, status401, status403, status404, status409, status413, status429, status500, status503)
|
||||
import Network.Wai (Application, Request (..))
|
||||
import Text.Read (readMaybe)
|
||||
import Web.Scotty
|
||||
@@ -337,6 +339,7 @@ routes env = do
|
||||
delete "/api/roles/:name/grants" (withDatabase env revokeRoleH)
|
||||
post "/api/roles/:name/members" (withDatabase env addRoleMemberH)
|
||||
delete "/api/roles/:name/members/:user" (withDatabase env removeRoleMemberH)
|
||||
post "/api/normalization/analyze" (withDatabase env normalizationH)
|
||||
post "/api/query" (withDatabase env queryH)
|
||||
post "/api/query/rollback" (rollbackConsoleH env)
|
||||
post "/api/edit/begin" (beginEditH env)
|
||||
@@ -357,12 +360,13 @@ withDatabase env action = do
|
||||
else do
|
||||
editing <- header "X-ChuSQL-Edit"
|
||||
req <- request
|
||||
let snapshotRead = editing /= Nothing && requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"]
|
||||
let snapshotRead = editing /= Nothing && ((requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"])
|
||||
|| (requestMethod req == "POST" && pathInfo req == ["api", "normalization", "analyze"]))
|
||||
console <- liftIO (readIORef (wsConsole ws))
|
||||
caller <- consoleOwner
|
||||
case console of
|
||||
Just (owner, database) | not snapshotRead &&
|
||||
(pathInfo req /= ["api", "query"] || caller /= owner || (not (T.null chosen) && chosen /= database)) ->
|
||||
(pathInfo req `notElem` [["api", "query"], ["api", "normalization", "analyze"]] || caller /= owner || (not (T.null chosen) && chosen /= database)) ->
|
||||
apiError "conflict" "another SQL console owns this connection transaction; commit or rollback there first"
|
||||
_ -> pure ()
|
||||
current <- liftIO (sessionDatabase (wsSession ws))
|
||||
@@ -391,7 +395,8 @@ requireSession env = do
|
||||
editHeader <- fmap TL.toStrict <$> header "X-ChuSQL-Edit"
|
||||
req <- request
|
||||
case editHeader of
|
||||
Just editId | requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"] -> do
|
||||
Just editId | (requestMethod req == "GET" && take 2 (pathInfo req) == ["api", "tables"])
|
||||
|| (requestMethod req == "POST" && pathInfo req == ["api", "normalization", "analyze"]) -> do
|
||||
active <- liftIO (readIORef (wsEdit ws))
|
||||
database <- maybe "" (T.toLower . TL.toStrict) <$> header "X-ChuSQL-Database"
|
||||
case active of
|
||||
@@ -905,7 +910,7 @@ serverError message
|
||||
(rawCode, rest) = T.breakOn ":" message
|
||||
code = T.strip rawCode
|
||||
detail = T.strip (T.drop 1 rest)
|
||||
knownCodes = ["unauthorized", "forbidden", "not_found", "conflict", "bad_request", "no_database", "query_error", "storage_error", "too_large"]
|
||||
knownCodes = ["unauthorized", "forbidden", "not_found", "conflict", "bad_request", "no_database", "query_error", "storage_error", "too_large", "too_busy", "timeout"]
|
||||
|
||||
-- | 取错误消息里的错误码
|
||||
errorCodeOf :: Text -> Text
|
||||
@@ -920,6 +925,8 @@ statusOf code = case code of
|
||||
"conflict" -> status409
|
||||
"too_many_attempts" -> status429
|
||||
"too_large" -> status413
|
||||
"too_busy" -> status503
|
||||
"timeout" -> status503
|
||||
"storage_error" -> status500
|
||||
"internal" -> status500
|
||||
_ -> status400
|
||||
@@ -1901,3 +1908,19 @@ data QueryRequest = QueryRequest
|
||||
|
||||
instance FromJSON QueryRequest where
|
||||
parseJSON = withObject "QueryRequest" $ \o -> QueryRequest <$> o .: "sql"
|
||||
|
||||
-- 使用现有连接视图转发只读范式请求。
|
||||
normalizationH :: AppEnv -> ActionM ()
|
||||
normalizationH env = do
|
||||
ws <- requireSession env
|
||||
rawBody <- body
|
||||
case decodeNormalizationJSON rawBody of
|
||||
Left message | BL.length rawBody > 65536 -> reply413
|
||||
| otherwise -> reply400 "bad_request" message
|
||||
Right value -> case A.fromJSON value :: A.Result NormalizationRequest of
|
||||
A.Error message -> reply400 "bad_request" (T.pack message)
|
||||
A.Success payload -> do
|
||||
result <- liftIO (analyzeNormalization (wsSession ws) payload)
|
||||
case result of
|
||||
Left message -> serverError message
|
||||
Right report -> json report
|
||||
@@ -21,7 +21,7 @@ function hasValidRows(body) {
|
||||
async function request(url, init, database) {
|
||||
const headers = new Headers(init && init.headers ? init.headers : undefined);
|
||||
if (database) headers.set('X-ChuSQL-Database', database);
|
||||
if (editing && editing.database === database && url.startsWith('/api/tables') && (!init || init.method === 'GET')) {
|
||||
if (editing && editing.database === database && ((url.startsWith('/api/tables') && (!init || init.method === 'GET')) || url === '/api/normalization/analyze')) {
|
||||
headers.set('X-ChuSQL-Edit', editing.id);
|
||||
}
|
||||
let response;
|
||||
@@ -326,3 +326,23 @@ export async function loadUiSettings() {
|
||||
export async function saveUiSettings(settings) {
|
||||
await request('/api/ui-settings', jsonInit('PUT', settings));
|
||||
}
|
||||
|
||||
// 在页面已有视图上请求只读范式分析。
|
||||
export async function analyzeNormalization(database, table, declaration, expected) {
|
||||
const payload = { api_version: 1, database, table };
|
||||
if (expected) {
|
||||
payload.expected_object = expected.object;
|
||||
payload.expected_schema_fingerprint = expected.schema_fingerprint;
|
||||
}
|
||||
const init = jsonInit('POST', payload);
|
||||
if (!editing && consoleTransaction?.database === database) {
|
||||
init.headers['X-ChuSQL-Console'] = consoleTransaction.console;
|
||||
}
|
||||
// 原始声明直接嵌入,重复字段交给服务端明确拒绝。
|
||||
init.body = `${JSON.stringify(payload).slice(0, -1)},"declaration":${typeof declaration === 'string' ? declaration : JSON.stringify(declaration)}}`;
|
||||
const result = await request('/api/normalization/analyze', init, database);
|
||||
if (!isObject(result) || result.api_version !== 1 || !Array.isArray(result.findings) || !Array.isArray(result.schema)) {
|
||||
throw new Error('服务器返回的范式报告无法识别。');
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -64,6 +64,8 @@ const state = {
|
||||
dockHeight: '22vh',
|
||||
};
|
||||
|
||||
let normalizationTarget = null;
|
||||
|
||||
const root = document.getElementById('root');
|
||||
|
||||
// ---------------------------------------------------------------- 小工具
|
||||
@@ -210,6 +212,21 @@ function renderShell() {
|
||||
<button type="button" class="primary" data-action="create-index">创建</button>
|
||||
</footer>
|
||||
</div>
|
||||
</dialog>
|
||||
<dialog id="dlg-normalization" class="dialog">
|
||||
<div class="dialog-body">
|
||||
<header class="dialog-header"><h2 id="normalization-title">范式分析</h2></header>
|
||||
<div class="dialog-content">
|
||||
<p>只读分析,以声明为条件。PRIMARY KEY、UNIQUE 和样本不会自动成为完整业务依赖;NULL 与重复行需要另行验证。</p>
|
||||
<label class="field"><span>JSON 声明(完整性 false,原子性 null 表示未声明)</span><textarea id="normalization-declaration" rows="10"></textarea></label>
|
||||
<p class="alert" id="normalization-error"></p>
|
||||
<div id="normalization-result"></div>
|
||||
</div>
|
||||
<footer class="dialog-footer">
|
||||
<button type="button" data-action="close-dialog">关闭</button>
|
||||
<button type="button" class="primary" data-action="analyze-normalization" disabled>分析声明</button>
|
||||
</footer>
|
||||
</div>
|
||||
</dialog>`;
|
||||
renderAccount();
|
||||
applyLayout();
|
||||
@@ -282,7 +299,7 @@ function renderSidebar() {
|
||||
<button class="mini danger" data-action="drop-table" data-table="${escapeHtml(table.name)}" title="删除表">✕</button>
|
||||
</div>
|
||||
${open ? `<ul class="column-list">${columns}</ul>${indexList}
|
||||
<div class="tree-actions"><button class="mini" data-action="open-index-dialog" data-table="${escapeHtml(table.name)}">+ 索引</button></div>` : ''}
|
||||
<div class="tree-actions">${state.database !== 'system' && !table.name.startsWith('__system_') ? `<button class="mini" data-action="open-normalization" data-table="${escapeHtml(table.name)}">范式分析</button>` : ''}<button class="mini" data-action="open-index-dialog" data-table="${escapeHtml(table.name)}">+ 索引</button></div>` : ''}
|
||||
</li>`;
|
||||
}).join('');
|
||||
|
||||
@@ -1383,7 +1400,7 @@ function startSplitterDrag(kind, event) {
|
||||
|
||||
// 这些动作都要求已经选中一个库(服务没有默认库)
|
||||
const DATABASE_ACTIONS = new Set([
|
||||
'open-table', 'demo-data', 'open-table-dialog', 'create-table',
|
||||
'open-normalization', 'analyze-normalization', 'open-table', 'demo-data', 'open-table-dialog', 'create-table',
|
||||
'open-index-dialog', 'create-index', 'drop-table', 'drop-column', 'drop-index',
|
||||
'drop-database', 'commit', 'rollback', 'copy-sql',
|
||||
]);
|
||||
@@ -1413,7 +1430,14 @@ function onClick(event) {
|
||||
}
|
||||
|
||||
switch (action) {
|
||||
case 'open-normalization':
|
||||
void openNormalization(target.dataset.table);
|
||||
break;
|
||||
case 'analyze-normalization':
|
||||
void analyzeNormalizationDeclaration();
|
||||
break;
|
||||
case 'logout':
|
||||
normalizationTarget = null;
|
||||
void (async () => {
|
||||
try {
|
||||
await api.logout();
|
||||
@@ -1790,3 +1814,68 @@ async function boot() {
|
||||
}
|
||||
|
||||
void boot();
|
||||
|
||||
// 打开结构入口并绑定当前 schema 身份。
|
||||
async function openNormalization(table) {
|
||||
const declaration = { declared_keys: [], dependencies: [], dependencies_complete: false, atomic_attributes: null };
|
||||
const target = { database: state.database, table, report: null };
|
||||
normalizationTarget = target;
|
||||
region('dlg-normalization').querySelector('[data-action="analyze-normalization"]').disabled = true;
|
||||
region('normalization-title').textContent = `范式分析 · ${target.database}.${table}`;
|
||||
region('normalization-declaration').value = JSON.stringify(declaration, null, 2);
|
||||
region('normalization-result').textContent = '正在读取结构…';
|
||||
region('normalization-error').textContent = '';
|
||||
region('dlg-normalization').onclose = () => { normalizationTarget = null; };
|
||||
region('dlg-normalization').showModal();
|
||||
try {
|
||||
target.report = await api.analyzeNormalization(target.database, table, declaration);
|
||||
if (normalizationTarget === target) {
|
||||
renderNormalization(target.report);
|
||||
region('dlg-normalization').querySelector('[data-action="analyze-normalization"]').disabled = false;
|
||||
}
|
||||
} catch (error) {
|
||||
if (normalizationTarget === target) {
|
||||
region('normalization-result').textContent = '';
|
||||
region('normalization-error').textContent = error.message;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 提交显式声明并复核绑定的对象结构。
|
||||
async function analyzeNormalizationDeclaration() {
|
||||
const target = normalizationTarget;
|
||||
if (!target?.report) return;
|
||||
const button = region('dlg-normalization').querySelector('[data-action="analyze-normalization"]');
|
||||
button.disabled = true;
|
||||
region('normalization-error').textContent = '';
|
||||
try {
|
||||
const declaration = region('normalization-declaration').value;
|
||||
const report = await api.analyzeNormalization(target.database, target.table, declaration, target.report);
|
||||
if (normalizationTarget === target) {
|
||||
target.report = report;
|
||||
renderNormalization(report);
|
||||
}
|
||||
} catch (error) {
|
||||
if (normalizationTarget === target) {
|
||||
region('normalization-error').textContent = error.message;
|
||||
if (error.code === 'conflict') {
|
||||
target.report = null;
|
||||
region('normalization-result').textContent = '对象或视图已变化,请关闭后重新打开范式分析。';
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
button.disabled = !target.report;
|
||||
}
|
||||
}
|
||||
|
||||
// 展示条件结论、违反证据与一次分解。
|
||||
function renderNormalization(report) {
|
||||
const labels = { satisfied: '满足', violated: '违反', insufficient: '信息不足' };
|
||||
const schema = report.schema.map((column) => `${column.name}: ${column.ty}${column.nullable ? '(可空)' : '(非空)'}`).join(',');
|
||||
const findings = report.findings.map((finding) => `<li><strong>${escapeHtml(finding.normal_form.toUpperCase())}: ${escapeHtml(labels[finding.verdict] ?? finding.verdict)}</strong>
|
||||
<p>${escapeHtml(finding.condition)}</p><pre>${escapeHtml(JSON.stringify(finding.witnesses, null, 2))}</pre></li>`).join('');
|
||||
region('normalization-result').innerHTML = `<p>${escapeHtml(schema)}</p><p>视图:${escapeHtml(report.view)} · 结论以声明为条件</p>
|
||||
<details><summary>结构与约束</summary><pre>${escapeHtml(JSON.stringify(report.schema, null, 2))}</pre></details>
|
||||
<p>相对于声明的候选键:${escapeHtml(JSON.stringify(report.candidate_keys))}</p><ul>${findings}</ul>
|
||||
<p>一次分解建议(无损标签不证明现有 SQL 数据):</p><pre>${escapeHtml(JSON.stringify(report.suggestions, null, 2))}</pre>`;
|
||||
}
|
||||
@@ -946,6 +946,102 @@ roleSpec env = describe "roles and grants" $ do
|
||||
querySpec :: AppEnv -> Spec
|
||||
querySpec env = describe "the SQL console" $ do
|
||||
withApp (webApp env) $ do
|
||||
it "analyzes explicit declarations for SELECT users and rejects stale schemas" $ do
|
||||
admin <- adminHeaders
|
||||
ensureDatabase admin testDatabaseName
|
||||
ensureAccount admin "normalization_web_reader" "Reader-Secret-1234"
|
||||
created <- testSql admin "CREATE TABLE normalization_web (id INT, course INT, name TEXT)"
|
||||
expectStatusWith "normalization table" created 200
|
||||
let declaration = object ["declared_keys" .= ([["id", "course"]] :: [[Text]]),
|
||||
"dependencies" .= [object ["lhs" .= (["id"] :: [Text]), "rhs" .= (["name"] :: [Text])]],
|
||||
"dependencies_complete" .= True, "atomic_attributes" .= True]
|
||||
payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
|
||||
"table" .= ("normalization_web" :: Text), "declaration" .= declaration]
|
||||
unauthorized <- postAs (withDb testDatabaseName []) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "anonymous normalization" unauthorized 401
|
||||
reader <- ordinaryHeaders "normalization_web_reader" "Reader-Secret-1234"
|
||||
denied <- postAs (withDb testDatabaseName reader) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "SELECT required" denied 403
|
||||
granted <- testSql admin "GRANT SELECT ON normalization_web TO normalization_web_reader"
|
||||
expectStatusWith "normalization SELECT" granted 200
|
||||
readerAgain <- ordinaryHeaders "normalization_web_reader" "Reader-Secret-1234"
|
||||
report <- postAs (withDb testDatabaseName readerAgain) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "conditional report" report 200
|
||||
check (at "conditional" (jsonBody report) `shouldBe` A.Bool True)
|
||||
check (map (asText . at "verdict") (items (at "findings" (jsonBody report))) `shouldBe` ["satisfied", "violated", "violated", "violated"])
|
||||
changed <- testSql admin "ALTER TABLE normalization_web ADD COLUMN extra INT"
|
||||
expectStatusWith "change normalization schema" changed 200
|
||||
let stale = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
|
||||
"table" .= ("normalization_web" :: Text), "declaration" .= declaration,
|
||||
"expected_object" .= at "object" (jsonBody report), "expected_schema_fingerprint" .= at "schema_fingerprint" (jsonBody report)]
|
||||
conflicted <- postAs (withDb testDatabaseName readerAgain) "/api/normalization/analyze" (encode stale)
|
||||
expectStatusWith "schema changed" conflicted 409
|
||||
it "uses a matching editing snapshot without creating a second transaction" $ do
|
||||
admin <- adminHeaders
|
||||
ensureDatabase admin testDatabaseName
|
||||
created <- testSql admin "CREATE TABLE normalization_edit (id INT, value INT)"
|
||||
expectStatusWith "normalization snapshot table" created 200
|
||||
begun <- editRequest admin "/api/edit/begin" "normalization-owner" []
|
||||
expectStatusWith "normalization edit begins" begun 200
|
||||
changed <- testSql admin "ALTER TABLE normalization_edit ADD COLUMN extra INT"
|
||||
expectStatusWith "live schema changes" changed 200
|
||||
let payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
|
||||
"table" .= ("normalization_edit" :: Text), "declaration" .= object []]
|
||||
snapshot <- postAs (("X-ChuSQL-Edit", "normalization-owner") : withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "analysis on edit connection" snapshot 200
|
||||
check (at "view" (jsonBody snapshot) `shouldBe` A.String "transaction_snapshot")
|
||||
check (length (items (at "schema" (jsonBody snapshot))) `shouldBe` 2)
|
||||
forged <- postAs (("X-ChuSQL-Edit", "another-page") : withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "page identity mismatch" forged 409
|
||||
rolledBack <- editRequest admin "/api/edit/rollback" "normalization-owner" []
|
||||
expectStatusWith "release edit snapshot" rolledBack 200
|
||||
it "binds normalization to the owning SQL console transaction" $ do
|
||||
admin <- adminHeaders
|
||||
ensureDatabase admin testDatabaseName
|
||||
created <- testSql admin "CREATE TABLE normalization_console (id INT)"
|
||||
expectStatusWith "console normalization table" created 200
|
||||
let owner = ("X-ChuSQL-Console", "normalization-console") : admin
|
||||
payload = object ["api_version" .= (1 :: Int), "database" .= testDatabaseName,
|
||||
"table" .= ("normalization_console" :: Text), "declaration" .= object []]
|
||||
begun <- testSql owner "BEGIN"
|
||||
expectStatusWith "console begins" begun 200
|
||||
snapshot <- postAs (withDb testDatabaseName owner) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "owner console normalization" snapshot 200
|
||||
check (at "view" (jsonBody snapshot) `shouldBe` A.String "transaction_snapshot")
|
||||
denied <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze" (encode payload)
|
||||
expectStatusWith "unrelated console refused" denied 409
|
||||
rolledBack <- testSql owner "ROLLBACK"
|
||||
expectStatusWith "console ends" rolledBack 200
|
||||
it "rejects duplicate declarations and oversized normalization bodies" $ do
|
||||
admin <- adminHeaders
|
||||
ensureDatabase admin testDatabaseName
|
||||
duplicate <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze"
|
||||
"{\"api_version\":1,\"database\":\"test\",\"table\":\"users\",\"declaration\":{\"atomic_attributes\":true,\"atomic_attributes\":false}}"
|
||||
expectStatusWith "duplicate declaration fields" duplicate 400
|
||||
oversized <- postAs (withDb testDatabaseName admin) "/api/normalization/analyze" (BL.replicate 65537 32)
|
||||
expectStatusWith "normalization request bound" oversized 413
|
||||
|
||||
it "exposes DISTINCT and self grants after grant option revocation" $ do
|
||||
admin <- adminHeaders
|
||||
ensureDatabase admin testDatabaseName
|
||||
ensureAccount admin "http_option_reader" "Reader-Secret-1234"
|
||||
created <- testSql admin "CREATE TABLE http_distinct_probe (id INT, value INT)"
|
||||
expectStatusWith "distinct table" created 200
|
||||
inserted <- testSql admin "INSERT INTO http_distinct_probe (id,value) VALUES (1,7),(2,7),(3,8)"
|
||||
expectStatusWith "distinct rows" inserted 200
|
||||
granted <- testSql admin "GRANT SELECT ON http_distinct_probe TO http_option_reader WITH GRANT OPTION"
|
||||
expectStatusWith "grant option" granted 200
|
||||
revoked <- testSql admin "REVOKE GRANT OPTION FOR SELECT ON http_distinct_probe FROM http_option_reader"
|
||||
expectStatusWith "revoke only grant option" revoked 200
|
||||
reader <- ordinaryHeaders "http_option_reader" "Reader-Secret-1234"
|
||||
result <- testSql reader "SELECT DISTINCT value FROM http_distinct_probe ORDER BY 1 DESC LIMIT 2"
|
||||
expectStatusWith "distinct output" result 200
|
||||
check (at "rows" (jsonBody result) `shouldBe` A.toJSON ([[8], [7]] :: [[Int]]))
|
||||
grants <- testSql reader "SHOW GRANTS"
|
||||
expectStatusWith "own grants" grants 200
|
||||
check (at "columns" (jsonBody grants) `shouldBe` A.toJSON (["identity", "privilege", "object", "grant_option"] :: [Text]))
|
||||
denied <- testSql reader "GRANT SELECT ON http_distinct_probe TO admin"
|
||||
expectStatusWith "no delegation" denied 403
|
||||
it "returns typed compound values through HTTP and TCP" $ do
|
||||
hs <- adminHeaders
|
||||
ensureDatabase hs testDatabaseName
|
||||
|
||||
@@ -12,7 +12,7 @@ globalThis.document = { getElementById: (id) => id === 'root' ? { addEventListen
|
||||
let source = await readFile(new URL('../static/app.js', import.meta.url), 'utf8');
|
||||
source = source.replaceAll("'./api.js'", JSON.stringify(new URL('../static/api.js', import.meta.url).href))
|
||||
.replaceAll("'./core.js'", JSON.stringify(new URL('../static/core.js', import.meta.url).href))
|
||||
.replace('void boot();', 'export { state, commitAll, rollbackAll, stageUpdate, removeStaged, ensureEdit, runSql };');
|
||||
.replace('void boot();', 'export { state, commitAll, rollbackAll, stageUpdate, removeStaged, ensureEdit, runSql, renderNormalization };');
|
||||
const app = await import(`data:text/javascript;base64,${Buffer.from(source).toString('base64')}`);
|
||||
let calls;
|
||||
let commitResponse;
|
||||
@@ -173,3 +173,52 @@ test('edits are locked while SQL releases the previous snapshot', async () => {
|
||||
assert.equal(app.state.editBusy, false);
|
||||
assert.equal(app.state.edit, null);
|
||||
});
|
||||
|
||||
// 校验分析复用编辑视图并保留严格声明的原始字段。
|
||||
test('normalization keeps snapshot ownership, exact identities and duplicate declaration fields', async () => {
|
||||
await api.beginEdit('normalization-page', 'test');
|
||||
const expected = { object: { database_id: '9007199254740993', object_id: '18446744073709551615' }, schema_fingerprint: 'schema' };
|
||||
globalThis.fetch = async (url, init) => {
|
||||
calls.push({ url, init });
|
||||
return response({ api_version: 1, findings: [], schema: [] });
|
||||
};
|
||||
await api.analyzeNormalization('test', 'users', '{"atomic_attributes":true,"atomic_attributes":false}', expected);
|
||||
const sent = calls.find((call) => call.url === '/api/normalization/analyze');
|
||||
assert.equal(new Headers(sent.init.headers).get('X-ChuSQL-Edit'), 'normalization-page');
|
||||
assert.equal(new Headers(sent.init.headers).get('X-ChuSQL-Database'), 'test');
|
||||
assert.equal((sent.init.body.match(/"atomic_attributes"/g) ?? []).length, 2);
|
||||
assert.deepEqual(JSON.parse(sent.init.body).expected_object, expected.object);
|
||||
assert.equal(calls.filter((call) => call.url === '/api/edit/begin').length, 1);
|
||||
api.forgetEdit();
|
||||
globalThis.fetch = async (url, init) => {
|
||||
calls.push({ url, init });
|
||||
return response(url === '/api/query'
|
||||
? { columns: [], rows: [], rowCount: 0, truncated: false, transaction: true }
|
||||
: { api_version: 1, findings: [], schema: [] });
|
||||
};
|
||||
await api.query('BEGIN', 'test', 'normalization-console');
|
||||
await api.analyzeNormalization('test', 'users', '{}', expected);
|
||||
const fromConsole = calls.filter((call) => call.url === '/api/normalization/analyze').at(-1);
|
||||
const begun = calls.filter((call) => call.url === '/api/query').at(-1);
|
||||
assert.equal(new Headers(fromConsole.init.headers).get('X-ChuSQL-Console'), new Headers(begun.init.headers).get('X-ChuSQL-Console'));
|
||||
assert.equal(new Headers(fromConsole.init.headers).get('X-ChuSQL-Edit'), null);
|
||||
});
|
||||
|
||||
// 校验结构、违反证据和分解中的用户文字全部转义。
|
||||
test('normalization escapes schema names, conditions, witnesses and suggestions', () => {
|
||||
const result = { innerHTML: '' };
|
||||
const original = document.getElementById;
|
||||
document.getElementById = (id) => id === 'normalization-result' ? result : original(id);
|
||||
try {
|
||||
app.renderNormalization({ schema: [{ name: '<img src=x onerror=alert(1)>', ty: 'str', nullable: true }],
|
||||
view: 'live_schema', candidate_keys: [['<script>']],
|
||||
findings: [{ normal_form: 'bcnf', verdict: 'violated', condition: '<script>alert(1)</script>', witnesses: [{ lhs: ['<img>'], rhs: ['id'] }] }],
|
||||
suggestions: [{ relations: [['<script>'], ['id']] }] });
|
||||
assert.equal(result.innerHTML.includes('<img'), false);
|
||||
assert.equal(result.innerHTML.includes('<script>'), false);
|
||||
assert.equal(result.innerHTML.includes('<img'), true);
|
||||
assert.equal(result.innerHTML.includes('结构与约束'), true);
|
||||
} finally {
|
||||
document.getElementById = original;
|
||||
}
|
||||
});
|
||||
@@ -117,7 +117,7 @@ TCP 连接在下一次请求时返回 unauthorized 并关闭;空闲连接不
|
||||
|
||||
## 查询引擎扩展设计(尚未实现)
|
||||
|
||||
按 bound IR → 子查询物化 / 递归 CTE 推进;DOMAIN 约束见[类型文档](types.md#domain-约束设计尚未实现)。以下设计尚未实施,当前能力见前文;实施状态以 agent-credientials/progress.txt 为准。
|
||||
按 bound IR → 子查询物化 / 递归 CTE 推进;DOMAIN 约束见[类型文档](types.md#domain-约束设计尚未实现)。以下设计尚未实施,当前能力见前文;实施状态以 .agents/context/progress.txt 为准。
|
||||
|
||||
### bound IR
|
||||
|
||||
|
||||
@@ -44,6 +44,8 @@ csql [options]
|
||||
| `-d, --database DB` | 登录后选择数据库 |
|
||||
| `-f, --format FORMAT` | 输出格式:`table`、`json` 或 `csv` |
|
||||
| `-e, --execute SQL` | 执行一条 SQL 后退出 |
|
||||
| `--normalize TABLE` | 只读分析表的范式;与 `--execute` 互斥,需 `--declaration` |
|
||||
| `--declaration FILE` | UTF-8 JSON 业务依赖声明;范式输出使用 `--format text` 或 `json` |
|
||||
| `--history FILE` | 指定历史记录文件 |
|
||||
| `--sudo` | 使用本机 root / 提升管理员身份认证;需配置映射并启用身份属性,见 [用户与角色](users-and-roles.md#本机-sudo-认证) |
|
||||
| `--config FILE` | 指定配置文件 |
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# ChuSQL 技术总结与数据库课程答辩
|
||||
|
||||
完整学习与答辩材料已集中到仓库根目录的 `learnning/`:
|
||||
|
||||
- [分章节学习目录](../learnning/README.md)
|
||||
- [答辩总览](../learnning/00-defense-overview.md)
|
||||
- [关键函数导航](../learnning/17-function-atlas.md)
|
||||
- [现场演示与排练](../learnning/14-demonstration.md)
|
||||
@@ -105,6 +105,28 @@ csql-web
|
||||
|
||||
管理员使用安装时设置的口令登录。
|
||||
|
||||
## 更新
|
||||
|
||||
重新运行安装脚本,指定相同的安装目录和组件即可更新,也可用 `--version TAG` 选择版本:
|
||||
|
||||
```sh
|
||||
./install.sh --component both --install-dir "$HOME/.local/share/chusql" --version <tag>
|
||||
```
|
||||
|
||||
Linux 更新会先停止该安装目录中的 Web 和数据库进程,保留已有 `settings.toml`、数据库和管理员口令,再引导检查系统目录。修改数据位置须编辑配置并自行迁移数据;更新时不能用不同的 `--data-dir` 覆盖现有配置。安装默认重启数据库服务,Web 管理端需重新运行 `csql-web`。
|
||||
|
||||
Linux 安装目录中保留 `uninstall.sh`。卸载时指定安装目录,数据目录默认读取现有配置。
|
||||
|
||||
已构建的 Linux 包可用隔离验证脚本测试,不修改正常安装的配置或数据:
|
||||
|
||||
```sh
|
||||
python3 scripts/test-linux-lifecycle.py releases/chusql-linux-x86_64.tar.gz
|
||||
```
|
||||
|
||||
该验证需要 Python 3,使用真实数据库程序,覆盖安装、运行中更新、配置与 SQL 数据保留、保留数据卸载、重装和完全卸载;基础验证通过 `--no-service` 隔离 systemd 服务。
|
||||
|
||||
添加 `--systemd` 可验证真实用户服务的注册、启动、更新及卸载。此选项要求不存在已有的 `chusql-server.service`,并且当前用户能够启用 linger(或已由管理员启用);临时单元链接会在测试后清理。本机 HTTP 发行站夹具用于验证在线安装流程,不依赖 GitHub 上已有 Release。
|
||||
|
||||
## 卸载
|
||||
|
||||
Linux / macOS:
|
||||
|
||||
+36
-11
@@ -84,33 +84,58 @@ schema 类型不证明业务意义上的原子性。不满足 1NF 时更高范
|
||||
|
||||
建议只是一轮分解,子关系可能仍需继续分析,不承诺一次达到 BCNF。依赖不保持时需跨关系检查原约束。
|
||||
|
||||
## 第一阶段限制
|
||||
## 当前限制
|
||||
|
||||
- 最多接受 12 列,超过返回错误,避免指数枚举无界增长。
|
||||
- 仅提供 Haskell 只读 API。
|
||||
- CLI / Web 入口和自动迁移尚未实现。
|
||||
- 提供 Haskell 只读 API、TCP 专用请求、CLI 和 Web 结构入口;不执行数据迁移。
|
||||
- 输入与输出有预算,超过预算返回明确错误,不截断为完整报告。
|
||||
|
||||
## 后续入口设计(尚未实现)
|
||||
## TCP、CLI 与 Web 入口
|
||||
|
||||
以下 TCP、CLI 和 Web 接口尚未实现,拟议命令与端点目前不可用。实施状态以 agent-credientials/progress.txt 为准。
|
||||
以下只读入口已实现。分析只读取 schema,不能据此认定 SQL 数据满足声明或直接执行分解。实施状态以 .agents/context/progress.txt 为准。
|
||||
|
||||
### 协议与鉴权
|
||||
|
||||
在 `ChuSQL.Interface.Protocol` 增加专用客户端请求 `normalization_analyze`,由 server 调用 engine API;不经最高权限专用的原始 `ReqStorage` 入口。请求包含 `api_version=1`、数据库与表标识、声明,以及可选的预期对象编号和 schema 指纹。server 在现有权限读门内以 catalog 解析现存对象,检查 CONNECT 与 SELECT;系统对象拒绝访问。事务内取该连接的 schema 快照,返回视图来源和对象编号,不能混入其他连接的快照。
|
||||
`ChuSQL.Interface.Protocol` 的专用客户端请求 `normalization_analyze` 由 server 调用 engine API;不经最高权限专用的原始 `ReqStorage` 入口。请求包含 `api_version=1`、数据库与表标识、声明,以及可选的预期对象编号和 schema 指纹。server 在现有权限读门内以 catalog 解析现存对象,检查 CONNECT 与 SELECT;系统对象拒绝访问。事务内取该连接的 schema 快照,返回视图来源和对象编号,不能混入其他连接的快照。
|
||||
|
||||
声明字段固定为 `declared_keys`、`dependencies`(`lhs` / `rhs`)、`dependencies_complete`、`atomic_attributes`。完整性缺省 false,原子性缺省 null;不把 PRIMARY KEY、UNIQUE 或样本自动当作完整业务依赖。前端可以展示实际 schema 约束供用户选择,确认后才成为声明。候选键的 nullable 与 SQL UNIQUE 语义单独提示,不自动证明经典关系模型的键。
|
||||
|
||||
响应包含对象编号、schema 指纹、规范化声明、候选键、四项 findings 和一次分解建议。稳定标签使用 `1nf/2nf/3nf/bcnf`、`satisfied/violated/insufficient`;条件说明与机器字段分开。所有结论标明“以声明为条件”,无损连接标签不代表现有 SQL 数据或袋语义已经验证。
|
||||
|
||||
字段名称必须与返回 schema 的确切列名一致;对象名称使用现有标识符规则,数据库限定名必须与请求数据库一致。指纹只覆盖列名、类型和约束等语义结构,排除行数与直方图;定义确定性编码并同时绑定稳定对象编号,避免同名重建复用报告。
|
||||
字段名称必须与返回 schema 的确切列名一致;对象名称使用现有标识符规则,数据库限定名必须与请求数据库一致。`schema_fingerprint` 使用 SHA256,编码为 `[库/表编号字符串数组, 列结构数组]`;每列按名称、类型、nullable、default、auto_increment、primary_key、unique、check 排列。它排除行数、索引统计与直方图,并绑定稳定对象编号,避免同名重建复用报告。对象编号在报告中使用 UInt64 范围内的正整数十进制字符串,避免浏览器数值精度损失。
|
||||
|
||||
拒绝未知协议版本、未知字段、重复字段、空右部、非最小声明键、空 schema 和超过 12 列。拟定请求上限 64 KiB、依赖条数 256、键声明 256;预算超限返回明确错误,不截断后给出完整报告。分析在有界工作队列运行,取消或超时释放权限读门与会话资源;输出也受传输预算约束,超限返回错误。实施时用最坏 12 列声明确定默认时间预算。
|
||||
拒绝未知协议版本、未知字段、重复字段、空右部、非最小声明键、空 schema 和超过 12 列。请求 JSON 上限 64 KiB、嵌套深度上限 64;依赖与键声明分别最多 256 项。服务端最多接纳 8 个请求,最多 2 个同时执行;排队、权限门等待与计算合计最多 5 秒。完整报告超过 64000 字节即拒绝,预留协议外层开销;客户端请求也为外层协议预留空间。超限返回 too_large,队列满返回 too_busy,超时返回 timeout,不截断成完整报告。取消执行或超时均释放执行槽和权限读门;断线请求最多运行至预算到期。
|
||||
|
||||
请求外层形状:
|
||||
|
||||
```json
|
||||
{"method":"normalization_analyze","request":{"api_version":1,"database":"sales","table":"orders","declaration":{"declared_keys":[["id"]],"dependencies":[],"dependencies_complete":false,"atomic_attributes":null}}}
|
||||
```
|
||||
|
||||
成功响应的 `status` 为 `normalization`,`analysis` 保存完整报告;可选 `expected_object` 与 `expected_schema_fingerprint` 放在 `request` 中。数据库必须与该连接的工作库一致;表可使用裸名或同库限定名。没有登录返回 unauthorized,系统对象拒绝分析;不会通过另一个连接代取事务视图。
|
||||
|
||||
### CLI 与 Web
|
||||
|
||||
CLI 拟议元命令为 `\normalize TABLE --declaration FILE --format text|json`,通过当前登录连接和工作库请求。声明文件为 UTF-8 JSON,错误写 stderr;成功分析退出码 0,即使结论违反或信息不足;协议、权限、输入与运行错误退出非零。批处理增加独立 `--normalize` 模式,与 `--execute` 互斥,不把元命令发给 SQL 解析器。
|
||||
CLI 元命令为 `\normalize TABLE --declaration FILE --format text|json`,通过当前登录连接和工作库请求。声明文件为 UTF-8 JSON,错误写 stderr;成功分析退出码 0,即使结论违反或信息不足;协议、权限、输入与运行错误退出非零。批处理使用独立 `--normalize` 模式,与 `--execute` 互斥,不把元命令发给 SQL 解析器。
|
||||
|
||||
Web 拟议 `POST /api/normalization/analyze`,复用登录、CSRF 和 Cookie 串行规则。入口位于表结构页:展示 schema、编辑声明、提交分析、显示证据和分解字段列表。完整性与原子性不预先勾选;没有自动执行分解按钮。编辑页分析必须携带既有页面事务标识,不能自动开启另一事务。对象或 schema 改变返回冲突并要求重新分析。
|
||||
Web 使用 `POST /api/normalization/analyze`,复用登录、CSRF 和 Cookie 串行规则。入口位于资源管理器展开表后的结构区:展示 schema、编辑声明、提交分析、显示证据和分解字段列表。完整性与原子性不预先勾选;没有自动执行分解按钮。编辑页分析必须携带既有页面事务标识,不能自动开启另一事务;SQL 控制台有事务时,分析请求携带该控制台身份并使用其快照。对象或 schema 改变返回冲突并要求重新分析。
|
||||
|
||||
声明文件示例(不自动声明完整性或原子性):
|
||||
|
||||
```json
|
||||
{
|
||||
"declared_keys": [["student", "course"]],
|
||||
"dependencies": [{"lhs": ["student"], "rhs": ["name"]}],
|
||||
"dependencies_complete": true,
|
||||
"atomic_attributes": true
|
||||
}
|
||||
```
|
||||
|
||||
```sh
|
||||
csql -d school --normalize enrollment --declaration dependencies.json --format json
|
||||
```
|
||||
|
||||
交互模式:`\normalize enrollment --declaration "My Data/dependencies.json" --format text`。成功报告即使违反范式或信息不足也退出 0;输入、权限、协议或预算错误退出非零。文本输出保留与 JSON 输出相同的报告字段。Web 在资源管理器展开表后点击“范式分析”,输入声明 JSON;初始声明为空,完整性 false、原子性 null。报告给出候选键、违反证据和一次分解建议,只有分析按钮。
|
||||
|
||||
### 入口验收
|
||||
|
||||
@@ -118,7 +143,7 @@ Web 拟议 `POST /api/normalization/analyze`,复用登录、CSRF 和 Cookie
|
||||
|
||||
## 数据迁移设计(尚未实现)
|
||||
|
||||
先实现只读入口,再在 [DOMAIN 约束](types.md#domain-约束设计尚未实现)、依赖目录和原子结构发布具备后实施迁移。
|
||||
只读入口已具备;在 [DOMAIN 约束](types.md#domain-约束设计尚未实现)、依赖目录和原子结构发布具备后实施迁移。
|
||||
|
||||
### 规划条件
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 发布签名与验证设计
|
||||
|
||||
本文定义尚未实现的发行安全方案。当前操作见[安装文档](install.md),实施状态以 agent-credientials/progress.txt 为准。先实现离线验证,再接入在线安装;该工作可独立于查询和存储扩展推进。
|
||||
本文定义尚未实现的发行安全方案。当前操作见[安装文档](install.md),实施状态以 .agents/context/progress.txt 为准。先实现离线验证,再接入在线安装;该工作可独立于查询和存储扩展推进。
|
||||
|
||||
## 选择与信任入口
|
||||
|
||||
|
||||
@@ -55,11 +55,34 @@ Rust 的通用行、目录和 WAL 路径保存该表示,读取时由共享模
|
||||
`Runtime.Binding` 提供独立的 TypedExpression 和 TypedPlan,绑定只接受 catalog schema。
|
||||
表达式和计划构造器隐藏,调用者必须经过 Binder;执行使用已解析的 TypeId / FunctionId,并校验存储行。
|
||||
该 API 覆盖单行、表扫描、投影和过滤,可构造嵌套复合值及函数调用。
|
||||
列引用按当前输入 schema 的槽位绑定,列名保留用于诊断和位置校验。独立调用 `evaluateExpression`
|
||||
必须按绑定时的 schema 顺序提供行;位置不符或类型不符明确报错。`executePlan` 在扫描边界
|
||||
校验完整行并按 schema 排序,投影后的表达式重新绑定到子计划输出槽位。
|
||||
|
||||
SQL SELECT 通过 `Runtime.Query.bindQuery` 生成不可直接构造的 BoundQuery,再由 `executeQueryM` 执行。
|
||||
绑定计划保存输出 schema 与类型身份;现有 RelOp 作为关系执行和优化的兼容桥,输出按位置校验,保留重复别名。
|
||||
类型化函数和构造器节点也进入 INSERT、UPDATE、CHECK、聚合参数和子查询。
|
||||
日期、时间戳、Blob 等阶段 A 外的类型保留明确的 CompatibilityType。
|
||||
`queryTypes` 用 `DomainType 名称 基础类型` 保留每一层 DOMAIN 引用,包括嵌套领域和兼容类型。
|
||||
直接列引用及派生表别名保留该身份;函数结果使用函数声明的返回类型,不继承参数的 DOMAIN。
|
||||
领域名称来自当前库 schema 的解析结果;定义版本与约束绑定仍待完整 bound IR 实现。
|
||||
SQL 计划在优化完成后由 `Runtime.Relation` 将投影、过滤、计算、聚合参数和连接条件中的列引用
|
||||
绑定为 `ColumnSlot`。扫描、点查和范围查询在存储边界按算子布局排列返回行;相关子查询
|
||||
按自己的局部布局和外层行布局绑定槽位。局部引用使用 ColumnSlot,外层引用使用 OuterColumnSlot,
|
||||
保存局部布局宽度与外层相对位置;重新绑定时外层引用只在外层布局中查找,防止同名局部列捕获。
|
||||
执行槽位不接受语义层再次推导,语义输入必须使用未绑定列。哈希/索引连接仍使用列名识别等值键。
|
||||
投影单向转换为已有 Compute 算子,使用绑定列槽位取值;保留输出顺序、重复列及空投影的行数。
|
||||
排序转换为 BoundSort,分组转换为 BoundAggregate;键使用绑定槽位,列名保留为输出标签和诊断。
|
||||
排序每行计算一次键并保留同键行序,分组保留首次出现顺序及空输入全局聚合行为。
|
||||
该步骤仍复用 Expr/RelOp 和扁平化外层行;多级词法作用域/来源编号及 schema 版本尚未引入。
|
||||
绑定后的计划用于执行;结构改写必须在绑定前完成,或重新按新布局绑定。
|
||||
执行前重新取得存储 schema,比较查询依赖表的完整列定义;依赖收集覆盖来源树、连接条件、
|
||||
投影和过滤表达式里的嵌套查询,并递归追踪来源列使用的 DOMAIN 定义。缺表、列定义变化或 schema
|
||||
读取失败明确返回错误,行数据、统计及无关表、无关 DOMAIN 的结构变化不使绑定失效。
|
||||
Table 与 TableInfo 保留 catalog 的库编号和对象编号;执行前比较身份,可识别同名同结构的对象替换。
|
||||
内存 Server 目录和存储快照同样保留身份。TableInfo 输出十进制字符串编号,读取兼容数字编号,
|
||||
校验无符号 64 位范围;旧响应缺少编号或原始目录对象编号为零时,以列定义校验作为兼容路径。
|
||||
身份与列定义校验不能替代服务端读写门或未来的原子 schema 版本校验。
|
||||
旧 ColumnType 和运算符实现尚未整体移除;统一 capability、cast 和 rewrite 将在后续阶段收拢。
|
||||
|
||||
测试覆盖稳定编码、kind / 参数校验、共享协议、SQL 调用和复合列、实际存储重载、HTTP/TCP 与客户端显示。
|
||||
+16
-1
@@ -17,9 +17,20 @@
|
||||
|
||||
- `SELECT` 支持 `WHERE`、`ORDER BY`、`LIMIT`、`JOIN` / `LEFT JOIN`、`GROUP BY` 与聚合,以及标量 / `IN` / `EXISTS` 子查询和量词比较(`op ANY` / `op ALL`)。
|
||||
- 来源可以是派生表:`SELECT d.name FROM (SELECT name FROM users) d`。派生表必须带别名,内层不能引用外层列;输出列取裸列名(内层写 `u.id`,外层用 `d.id` 引用)。
|
||||
- 投影列可以起别名:`SELECT count(*) AS total FROM orders ORDER BY total`,别名就是结果集的列名。`ORDER BY` 优先使用唯一的投影别名,再查来源列;重复的排序别名报歧义错误,来源列可以用限定名消除同名冲突。排序暂不接受任意表达式或列序号。
|
||||
- 投影列可以起别名:`SELECT count(*) AS total FROM orders ORDER BY total`,别名就是结果集的列名。`ORDER BY` 优先使用唯一的投影别名,再查来源列;重复的排序别名报歧义错误,来源列可以用限定名消除同名冲突。排序接受从 1 开始的输出列序号(例如 `ORDER BY 2 DESC`),在 `*` 展开后解析,零或越界序号报错;暂不接受任意表达式。
|
||||
- 鉴权覆盖整棵来源树:查派生表时内层表也要有权限,只有外层引用的表有权限不够。
|
||||
|
||||
### DISTINCT 与 ALL
|
||||
|
||||
`SELECT DISTINCT` 按完整输出值组合去重,保留首次出现的行,然后执行排序和 LIMIT。NULL 在去重时视为相同值;等值整数与整值浮点按既有数值语义合并。默认 SELECT 或显式 `SELECT ALL` 保留重复行。
|
||||
|
||||
```sql
|
||||
SELECT DISTINCT user_id FROM orders ORDER BY 1 LIMIT 10;
|
||||
SELECT DISTINCT user_id + 1 AS next_id FROM orders ORDER BY next_id;
|
||||
```
|
||||
|
||||
DISTINCT 的排序列必须出现在输出中;排序可使用输出别名、选出的来源列或输出列序号。缺少比较能力的 List / Maybe / Tuple 类型仍拒绝 DISTINCT。去重支持连接、聚合、子查询、CTE 和当前连接的事务快照。
|
||||
|
||||
### 比较运算符
|
||||
|
||||
- 支持 `=`、`<>`(也可写 `!=`)、`>`、`>=`、`<`、`<=`;`<>` 与 `!=` 是同一个运算符。
|
||||
@@ -115,6 +126,8 @@ ALTER ROLE analyst NOSUPERUSER;
|
||||
SHOW ROLES;
|
||||
```
|
||||
|
||||
`SHOW GRANTS` 允许登录身份查看自己的有效对象权限,输出 `identity`、`privilege`、`object` 和 `grant_option`。结果包含直接授权、启用角色继承、PUBLIC 默认 CONNECT 和所有权隐含权限;表范围授权按现存表展开,超级用户展示全部业务对象权限。系统对象不列出,不接受 `FOR` 查询他人身份。该命令不要求选中工作库;事务中也读取当前权限目录。
|
||||
|
||||
`SHOW ROLES` 展示全部身份的编号、名称、登录能力、最高权限标志、启用状态及创建时间,不返回口令。`ALTER USER` 与 `ALTER ROLE` 的属性和口令修改语法通用;同一属性不能重复指定。身份管理仅允许最高权限身份执行。
|
||||
|
||||
`SUPERUSER` 不依赖启动参数里的管理员名称(`--user`),也不会通过角色成员关系继承。禁用登录身份会拒绝登录;禁用权限角色会停止贡献继承权限。属性修改会使受影响的既有连接失效,下一次请求返回 `unauthorized` 后关闭。最后一个启用且可以登录的最高权限身份不能删除、禁用、取消登录或取消最高权限。
|
||||
@@ -152,6 +165,8 @@ REVOKE SELECT ON sales.orders FROM reviewer;
|
||||
|
||||
`GRANT ... WITH GRANT OPTION` 允许再转授;所有者也可授予自身对象的权限。普通授予者可撤销自己的授予,所有者及最高权限身份可撤销对象上的其他授予。撤销时收回相应 grant option,并级联清理失去合法来源的下游授权;仍有独立来源的授权保留。转授权能力随启用的角色继承,循环转授不能独立维持权限。REVOKE 不移除所有者的隐含权限。
|
||||
|
||||
`REVOKE GRANT OPTION FOR SELECT ON users FROM alice` 只移除指定授权来源的转授权能力,保留 alice 的 SELECT 权限;失去唯一支持来源的下游授权会级联撤销,独立来源继续有效。最高权限身份或对象所有者可撤销全部匹配来源,普通授予者只能撤销自己的来源;所有权自身的权限不受 REVOKE 影响。
|
||||
|
||||
系统目录访问还受目录管理属性控制,具体能力见 [用户与角色](users-and-roles.md)。
|
||||
|
||||
|
||||
|
||||
@@ -42,4 +42,4 @@ Web 步骤会先显式刷新该工程里的 model、engine、interface 与 serve
|
||||
## 约定
|
||||
|
||||
- 门禁由项目所有者手动运行;agent 只跑模块级测试,不跑全量门禁。
|
||||
- 计数、验收状态与主线进度以 `agent-credientials/progress.txt` 为准。
|
||||
- 计数、验收状态与主线进度以 `.agents/context/progress.txt` 为准。
|
||||
@@ -0,0 +1,47 @@
|
||||
# 为 HLS 指定各源码目录的 Stack 组件,使用现有构建配置解析本地包。
|
||||
cradle:
|
||||
multi:
|
||||
- path: "./benchmark"
|
||||
config:
|
||||
cradle:
|
||||
stack:
|
||||
stackYaml: "./benchmark/stack.yaml"
|
||||
component: "benchmark4csql:exe:benchmark4csql"
|
||||
- path: "./"
|
||||
config:
|
||||
cradle:
|
||||
stack:
|
||||
stackYaml: "./chusql-cli/stack.yaml"
|
||||
components:
|
||||
- path: "./chusql-core/model/src"
|
||||
component: "chusql-core-model:lib"
|
||||
- path: "./chusql-core/engine/src"
|
||||
component: "chusql-core-engine:lib"
|
||||
- path: "./chusql-core/engine/test"
|
||||
component: "chusql-core-engine:test:chusql-core-engine-test"
|
||||
- path: "./chusql-interface/src"
|
||||
component: "chusql-interface:lib"
|
||||
- path: "./chusql-server/src"
|
||||
component: "chusql-server:lib"
|
||||
- path: "./chusql-server/app"
|
||||
component: "chusql-server:exe:chusql-server"
|
||||
- path: "./chusql-server/test"
|
||||
component: "chusql-server:test:chusql-server-test"
|
||||
- path: "./chusql-bootstrap/src"
|
||||
component: "chusql-bootstrap:lib"
|
||||
- path: "./chusql-bootstrap/app"
|
||||
component: "chusql-bootstrap:exe:csql-bootstrap"
|
||||
- path: "./chusql-bootstrap/test"
|
||||
component: "chusql-bootstrap:test:chusql-bootstrap-test"
|
||||
- path: "./chusql-web/src"
|
||||
component: "chusql-web:lib"
|
||||
- path: "./chusql-web/app"
|
||||
component: "chusql-web:exe:chusql-web"
|
||||
- path: "./chusql-web/test"
|
||||
component: "chusql-web:test:chusql-web-test"
|
||||
- path: "./chusql-cli/src"
|
||||
component: "chusql-cli:lib"
|
||||
- path: "./chusql-cli/app"
|
||||
component: "chusql-cli:exe:csql"
|
||||
- path: "./chusql-cli/test"
|
||||
component: "chusql-cli:test:chusql-cli-test"
|
||||
+1
-1
@@ -169,7 +169,7 @@ if (-not $Component) {
|
||||
# CHUSQL_ADMIN_PASSWORD。口令只经 stdin 传给 csql-bootstrap,不落 settings.toml。
|
||||
$adminPassword = "$env:CHUSQL_ADMIN_PASSWORD"
|
||||
if ($Password) { $adminPassword = $Password }
|
||||
if ($canPrompt) {
|
||||
if ($canPrompt -and $adminPassword.Length -eq 0) {
|
||||
Write-Host ''
|
||||
Write-Host ('The administrator account is "{0}".' -f $RootUser)
|
||||
while ($true) {
|
||||
|
||||
+64
-24
@@ -28,14 +28,13 @@ usage: ./install.sh [--component web|cli|both] [options]
|
||||
asks (English, y/n), command line client first:
|
||||
command line client (csql)? default yes [Y/n]
|
||||
web front end (browser UI)? default no [y/N]
|
||||
it then asks whether to give the administrator account a
|
||||
password (default no); that password goes straight into the
|
||||
it then asks for the administrator password; it goes into the
|
||||
system catalog and is never written to settings.toml
|
||||
|
||||
--interactive ask even when there is no terminal; the answers are
|
||||
read from stdin (one per line)
|
||||
--install-dir DIR where to install (default: $HOME/.local/share/chusql)
|
||||
--data-dir DIR where to keep data (default: <install-dir>/data)
|
||||
--data-dir DIR where to keep data (default: XDG data directory)
|
||||
--user NAME administrator name (default: root)
|
||||
--keep-package do not delete the downloaded/extracted package
|
||||
--password VALUE password of the administrator account (required when
|
||||
@@ -132,6 +131,19 @@ case "$self" in
|
||||
esac
|
||||
|
||||
[ -n "$install_dir" ] || install_dir="$HOME/.local/share/chusql"
|
||||
config_file="${XDG_CONFIG_HOME:-$HOME/.config}/ChuSQL/settings.toml"
|
||||
if [ "$list_versions" = 'no' ] && [ -f "$config_file" ]; then
|
||||
requested_data_dir=$data_dir
|
||||
data_dir=$(awk '/^[[:space:]]*\[storage\]/ { storage=1; next } /^[[:space:]]*\[/ { storage=0 } storage && /^[[:space:]]*data_dir[[:space:]]*=[[:space:]]*"[^"\\]*"[[:space:]]*(#.*)?$/ { sub(/^[^=]*=[[:space:]]*"/, ""); sub(/"[[:space:]]*(#.*)?$/, ""); print; exit }' "$config_file")
|
||||
[ -n "$data_dir" ] || fail 'existing settings.toml needs an explicit double-quoted storage.data_dir without escapes'
|
||||
case "$data_dir" in /*) ;; *) data_dir="$install_dir/$data_dir" ;; esac
|
||||
if [ -n "$requested_data_dir" ]; then
|
||||
requested_data_dir=$(CDPATH= cd -- "$requested_data_dir" && pwd -P) || fail 'requested data directory does not exist'
|
||||
existing_data_dir=$(CDPATH= cd -- "$data_dir" && pwd -P) || fail 'configured data directory does not exist'
|
||||
[ "$requested_data_dir" = "$existing_data_dir" ] ||
|
||||
fail 'existing settings.toml: edit its data_dir instead of passing a different --data-dir'
|
||||
fi
|
||||
fi
|
||||
[ -n "$data_dir" ] || data_dir="${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data"
|
||||
log_dir="${XDG_STATE_HOME:-$HOME/.local/state}/chusql/logs"
|
||||
|
||||
@@ -213,7 +225,7 @@ if [ "$list_versions" = 'no' ]; then
|
||||
if [ -n "$password_arg" ]; then
|
||||
admin_password="$password_arg"
|
||||
fi
|
||||
if [ "$can_prompt" = 'yes' ]; then
|
||||
if [ "$can_prompt" = 'yes' ] && [ -z "$admin_password" ]; then
|
||||
echo
|
||||
echo "The administrator account is $root_user."
|
||||
while :; do
|
||||
@@ -511,7 +523,7 @@ if [ "$from_source" = 'yes' ]; then
|
||||
mkdir -p "$pack/bin" "$pack/resources"
|
||||
cp "$storage_lib" "$pack/bin/"
|
||||
cp "$src/resources/settings.toml.linux" "$pack/resources/"
|
||||
cp "$src/scripts/install.sh" "$pack/"
|
||||
cp "$src/scripts/install.sh" "$src/scripts/uninstall.sh" "$pack/"
|
||||
for comp in $wanted; do
|
||||
if [ "$comp" = 'web' ]; then
|
||||
comp_dir='chusql-web'; front_name='chusql-web'
|
||||
@@ -626,15 +638,37 @@ if [ "$(uname -s)" = 'Linux' ] && [ "$no_service" = 'no' ]; then
|
||||
command -v systemctl >/dev/null 2>&1 || fail 'systemd is required; use --no-service to opt out'
|
||||
command -v loginctl >/dev/null 2>&1 || fail 'loginctl is required for boot autostart'
|
||||
systemctl --user show-environment >/dev/null || fail 'cannot connect to user systemd; run in a login session or use --no-service'
|
||||
loginctl enable-linger "$(id -un)" || fail 'cannot enable boot autostart; run sudo loginctl enable-linger for this OS user and retry'
|
||||
os_user=$(id -un)
|
||||
linger=$(loginctl show-user "$os_user" -p Linger --value) || fail 'cannot query boot autostart for this OS user'
|
||||
if [ "$linger" != 'yes' ]; then
|
||||
loginctl enable-linger "$os_user" || fail 'cannot enable boot autostart; run sudo loginctl enable-linger for this OS user and retry'
|
||||
fi
|
||||
use_service='yes'
|
||||
if [ -f "$service_file" ]; then
|
||||
systemctl --user stop chusql-server.service || fail 'cannot stop the existing service'
|
||||
fi
|
||||
fi
|
||||
# 停止安装目录中的进程,避免更新运行中的文件。
|
||||
stop_installed() {
|
||||
for executable in chusql-web chusql-server; do
|
||||
for process in /proc/[0-9]*/exe; do
|
||||
target=$(readlink "$process" 2>/dev/null) || continue
|
||||
[ "$target" = "$install_dir/bin/$executable" ] || continue
|
||||
pid=${process#/proc/}; pid=${pid%/exe}
|
||||
kill "$pid" || fail "cannot stop $executable ($pid)"
|
||||
attempts=0
|
||||
while [ -e "$process" ]; do
|
||||
attempts=$((attempts + 1))
|
||||
[ "$attempts" -lt 100 ] || fail "timed out stopping $executable ($pid)"
|
||||
sleep 0.1
|
||||
done
|
||||
done
|
||||
done
|
||||
}
|
||||
step 'Installing files'
|
||||
mkdir -p "$install_dir/bin" "$log_dir" "$data_dir"
|
||||
install_dir=$(CDPATH= cd -- "$install_dir" && pwd)
|
||||
install_dir=$(CDPATH= cd -- "$install_dir" && pwd -P)
|
||||
if [ "$(uname -s)" = Linux ]; then stop_installed; fi
|
||||
data_dir=$(CDPATH= cd -- "$data_dir" && pwd)
|
||||
log_dir=$(CDPATH= cd -- "$log_dir" && pwd)
|
||||
cp "$pack/bin/chusql-server" "$install_dir/bin/"
|
||||
@@ -657,7 +691,11 @@ if [ "$want_web" = 'yes' ]; then
|
||||
chmod +x "$install_dir/csql-web.sh"
|
||||
fi
|
||||
fi
|
||||
chmod +x "$install_dir/bin/"* 2>/dev/null || true
|
||||
chmod +x "$install_dir/bin/"*
|
||||
if [ -f "$pack/uninstall.sh" ]; then
|
||||
cp "$pack/uninstall.sh" "$install_dir/"
|
||||
chmod +x "$install_dir/uninstall.sh"
|
||||
fi
|
||||
|
||||
# ---- 写全局配置 ----
|
||||
step 'Writing settings.toml'
|
||||
@@ -666,9 +704,11 @@ config_file="$config_dir/settings.toml"
|
||||
mkdir -p "$config_dir"
|
||||
config_dir=$(CDPATH= cd -- "$config_dir" && pwd)
|
||||
config_file="$config_dir/settings.toml"
|
||||
sed -e "s|@DATA_DIR@|$(printf '%s' "$data_dir" | sed 's/[\&|]/\\&/g')|g" \
|
||||
-e "s|@LOG_DIR@|$(printf '%s' "$log_dir" | sed 's/[\&|]/\\&/g')|g" \
|
||||
"$pack/resources/settings.toml.linux" > "$config_file"
|
||||
if [ ! -f "$config_file" ]; then
|
||||
sed -e "s|@DATA_DIR@|$(printf '%s' "$data_dir" | sed 's/[\&|]/\\&/g')|g" \
|
||||
-e "s|@LOG_DIR@|$(printf '%s' "$log_dir" | sed 's/[\&|]/\\&/g')|g" \
|
||||
"$pack/resources/settings.toml.linux" > "$config_file"
|
||||
fi
|
||||
echo " config file $config_file"
|
||||
|
||||
# ---- 引导系统目录 ----
|
||||
@@ -676,7 +716,7 @@ echo " config file $config_file"
|
||||
step 'Initializing the administrator identity and preinstalled types'
|
||||
bootstrap="$install_dir/bin/csql-bootstrap"
|
||||
[ -n "$admin_password" ] || fail 'the administrator password is required'
|
||||
printf '%s\n' "$admin_password" | "$bootstrap" --config "$config_file" --user "$root_user" --password-stdin ||
|
||||
( cd "$install_dir" && printf '%s\n' "$admin_password" | "$bootstrap" --config "$config_file" --user "$root_user" --password-stdin ) ||
|
||||
fail 'the bootstrap program failed; identities or preinstalled types are not ready'
|
||||
|
||||
# ---- 命令入口 ----
|
||||
@@ -700,17 +740,17 @@ fi
|
||||
step 'Updating PATH'
|
||||
profile="$HOME/.profile"
|
||||
marker='# ChuSQL'
|
||||
if [ -f "$profile" ] && grep -q "^$marker" "$profile"; then
|
||||
echo "PATH entry already present in $profile"
|
||||
else
|
||||
{
|
||||
echo ''
|
||||
echo "$marker"
|
||||
echo "PATH=\"\$PATH:$install_dir\""
|
||||
echo "export PATH"
|
||||
} >> "$profile"
|
||||
echo "added $install_dir to PATH in $profile (login again or: . \"$profile\")"
|
||||
if [ -f "$profile" ] && grep -q "^$marker$" "$profile"; then
|
||||
awk -v m="$marker" 'skip > 0 { skip--; next } $0 == m { skip=2; next } { print }' "$profile" > "$profile.chusql-install"
|
||||
mv "$profile.chusql-install" "$profile"
|
||||
fi
|
||||
{
|
||||
echo ''
|
||||
echo "$marker"
|
||||
echo "PATH=\"\$PATH:$install_dir:$install_dir/bin\""
|
||||
echo "export PATH"
|
||||
} >> "$profile"
|
||||
echo "added $install_dir to PATH in $profile (login again or: . \"$profile\")"
|
||||
|
||||
# ---- 删掉安装包 ----
|
||||
if [ "$keep_package" = 'no' ]; then
|
||||
@@ -753,7 +793,7 @@ Description=ChuSQL database server
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory="$(unit_quote "$install_dir")"
|
||||
WorkingDirectory=$(printf '%s' "$install_dir" | sed 's/%/%%/g')
|
||||
ExecStart=:"$(unit_quote "$server_bin")" --config "$(unit_quote "$config_file")" --user "$(unit_quote "$root_user")"
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
@@ -778,7 +818,7 @@ elif [ "$no_start" = 'yes' ]; then
|
||||
else
|
||||
step 'Starting the server'
|
||||
mkdir -p "$log_dir"
|
||||
nohup "$server_bin" --config "$config_file" --user "$root_user" > "$log_dir/server.log" 2> "$log_dir/server.err.log" &
|
||||
( cd "$install_dir" && exec nohup "$server_bin" --config "$config_file" --user "$root_user" ) > "$log_dir/server.log" 2> "$log_dir/server.err.log" &
|
||||
server_pid=$!
|
||||
sleep 1
|
||||
if ! kill -0 "$server_pid" 2>/dev/null; then
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
from functools import partial
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
|
||||
# 用真实发行包验证 Linux 安装、更新及卸载,数据与配置隔离在临时目录。
|
||||
|
||||
|
||||
# 执行命令并将失败日志原样输出。
|
||||
def run(args, env, cwd=None):
|
||||
result = subprocess.run(args, env=env, cwd=cwd, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60)
|
||||
if result.returncode:
|
||||
raise RuntimeError(result.stdout)
|
||||
return result.stdout
|
||||
|
||||
|
||||
# 选择本机空闲端口。
|
||||
def free_port():
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
# 登录真实服务器并执行 SQL。
|
||||
def query(port, statements):
|
||||
with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
|
||||
with sock.makefile("rwb") as stream:
|
||||
answers = []
|
||||
for request in [{"method": "hello", "protocol": 1},
|
||||
{"method": "login", "user": "root", "password": "Lifecycle-test-123!"}] + [
|
||||
{"method": "query", "sql": sql} for sql in statements]:
|
||||
stream.write(json.dumps(request).encode() + b"\n")
|
||||
stream.flush()
|
||||
answer = json.loads(stream.readline())
|
||||
assert answer["status"] != "error", answer
|
||||
answers.append(answer)
|
||||
return answers[2:]
|
||||
|
||||
|
||||
# 等待服务器可接受登录。
|
||||
def wait_server(port):
|
||||
deadline = time.monotonic() + 15
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
query(port, [])
|
||||
return
|
||||
except (OSError, ValueError):
|
||||
time.sleep(0.1)
|
||||
raise RuntimeError("server did not become ready")
|
||||
|
||||
|
||||
# 用真实用户 systemd 验证注册、更新启动和卸载。
|
||||
def test_systemd(repo, archive, root, env):
|
||||
systemctl = shutil.which("systemctl")
|
||||
state = run([systemctl, "--user", "show", "chusql-server.service", "-p", "LoadState"], env)
|
||||
assert "LoadState=not-found" in state, "existing chusql-server.service: refusing to change it"
|
||||
runtime_unit = Path(os.environ["XDG_RUNTIME_DIR"]) / "systemd/user/chusql-server.service"
|
||||
assert not runtime_unit.is_symlink(), "existing runtime service link: refusing to change it"
|
||||
user = run(["id", "-un"], env).strip()
|
||||
linger = run(["loginctl", "show-user", user, "-p", "Linger"], env).strip()
|
||||
home = root / "service-home"
|
||||
home.mkdir()
|
||||
env = dict(env, HOME=str(home), XDG_CONFIG_HOME=str(home / ".config"),
|
||||
XDG_STATE_HOME=str(home / ".local/state"), CHUSQL_ADMIN_PASSWORD="Lifecycle-test-123!")
|
||||
install = root / "service program"
|
||||
unit = home / ".config/systemd/user/chusql-server.service"
|
||||
bridge = root / "service-tools"
|
||||
bridge.mkdir()
|
||||
wrapper = bridge / "systemctl"
|
||||
wrapper.write_text(f'''#!/bin/sh
|
||||
# 将隔离配置中的单元链接到用户 systemd 的运行目录。
|
||||
if [ "$*" = '--user daemon-reload' ] && [ -f "$CHUSQL_TEST_UNIT" ]; then
|
||||
"{systemctl}" --user link --runtime "$CHUSQL_TEST_UNIT" || exit 1
|
||||
fi
|
||||
exec "{systemctl}" "$@"
|
||||
''')
|
||||
wrapper.chmod(0o755)
|
||||
env.update(PATH=str(bridge) + ":" + env["PATH"], CHUSQL_TEST_UNIT=str(unit))
|
||||
installer = ["sh", str(repo / "scripts/install.sh"), "--url", archive,
|
||||
"--component", "both", "--install-dir", str(install)]
|
||||
try:
|
||||
run(installer + ["--data-dir", str(root / "service-data"), "--no-start"], env)
|
||||
config = home / ".config/ChuSQL/settings.toml"
|
||||
port = free_port()
|
||||
config.write_text(config.read_text().replace("port = 7777", f"port = {port}"))
|
||||
run(installer, env)
|
||||
wait_server(port)
|
||||
run([systemctl, "--user", "is-active", "--quiet", "chusql-server.service"], env)
|
||||
run(["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"], env)
|
||||
assert not unit.exists() and not (root / "service-data").exists()
|
||||
print("PASS real systemd registration, update/start and uninstall", flush=True)
|
||||
finally:
|
||||
state = run([systemctl, "--user", "show", "chusql-server.service", "-p", "LoadState"], env)
|
||||
if "LoadState=loaded" in state:
|
||||
run([systemctl, "--user", "stop", "chusql-server.service"], env)
|
||||
user_units = Path(os.environ["HOME"]) / ".config/systemd/user"
|
||||
for link in [runtime_unit, user_units / "chusql-server.service",
|
||||
user_units / "default.target.wants/chusql-server.service"]:
|
||||
if link.is_symlink() and link.resolve() == unit.resolve():
|
||||
link.unlink()
|
||||
run([systemctl, "--user", "daemon-reload"], dict(env, CHUSQL_TEST_UNIT="/nonexistent"))
|
||||
if linger == "Linger=no" and run(["loginctl", "show-user", user, "-p", "Linger"], env).strip() == "Linger=yes":
|
||||
run(["loginctl", "disable-linger", user], env)
|
||||
|
||||
|
||||
# 用本机发行站验证版本查询、HTTP 下载与校验。
|
||||
def test_online(repo, archive, root, env):
|
||||
site = root / "release-site"
|
||||
api = site / "api/v3/repos/test/ChuSQL/releases"
|
||||
api.parent.mkdir(parents=True)
|
||||
api.write_text(json.dumps([{"tag_name": "v-test", "prerelease": False,
|
||||
"assets": [{"name": "chusql-linux-x86_64.tar.gz"}]}], indent=2))
|
||||
assets = site / "test/ChuSQL/releases/download/v-test"
|
||||
assets.mkdir(parents=True)
|
||||
shutil.copy2(archive, assets / Path(archive).name)
|
||||
shutil.copy2(archive + ".sha256", assets / (Path(archive).name + ".sha256"))
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), partial(SimpleHTTPRequestHandler, directory=str(site)))
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
install = root / "online-program"
|
||||
env = dict(env, CHUSQL_GITHUB_BASE=f"http://127.0.0.1:{server.server_port}")
|
||||
installer = ["sh", str(repo / "scripts/install.sh"), "--repo", "test/ChuSQL"]
|
||||
try:
|
||||
assert "v-test" in run(installer + ["--list-versions"], env)
|
||||
output = run(installer + ["--component", "cli", "--install-dir", str(install),
|
||||
"--data-dir", str(root / "online-data"), "--no-service", "--no-start"], env)
|
||||
assert "checksum" in output and "(ok)" in output
|
||||
assert (install / "bin/csql").exists() and not (install / "bin/chusql-web").exists()
|
||||
run(["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"], env)
|
||||
assert not (root / "online-data").exists()
|
||||
print("PASS release API, latest version selection, HTTP download and checksum (local fixture)", flush=True)
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join()
|
||||
|
||||
|
||||
# 在隔离目录验证配置、SQL 数据和卸载保留选项。
|
||||
def main():
|
||||
if sys.platform != "linux" or len(sys.argv) not in (2, 3) or (len(sys.argv) == 3 and sys.argv[2] != "--systemd"):
|
||||
raise SystemExit("usage: python3 scripts/test-linux-lifecycle.py <linux-package.tar.gz> [--systemd]")
|
||||
archive = str(Path(sys.argv[1]).resolve())
|
||||
repo = Path(__file__).resolve().parent.parent
|
||||
(repo / "logs").mkdir(exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="linux-lifecycle-", dir=repo / "logs") as temporary:
|
||||
root = Path(temporary)
|
||||
home = root / "home"
|
||||
home.mkdir()
|
||||
env = dict(os.environ, HOME=str(home), XDG_CONFIG_HOME=str(home / ".config"),
|
||||
XDG_DATA_HOME=str(home / ".local/share"), XDG_STATE_HOME=str(home / ".local/state"),
|
||||
CHUSQL_ADMIN_PASSWORD="Lifecycle-test-123!")
|
||||
env.pop("LD_LIBRARY_PATH", None)
|
||||
install = root / "program"
|
||||
data = install / "custom/nested/data"
|
||||
config = home / ".config/ChuSQL/settings.toml"
|
||||
installer = ["sh", str(repo / "scripts/install.sh"), "--url", archive,
|
||||
"--install-dir", str(install), "--component", "both", "--no-service", "--no-start"]
|
||||
run(installer + ["--data-dir", str(data)], env)
|
||||
assert (data / "system/catalog.json").is_file()
|
||||
assert (install / "uninstall.sh").is_file()
|
||||
assert "@DATA_DIR@" not in config.read_text()
|
||||
assert run([str(install / "bin/csql-bootstrap"), "--help"], env)
|
||||
print("PASS clean installation and dynamic library lookup", flush=True)
|
||||
port = free_port()
|
||||
settings = config.read_text().replace("port = 7777", f"port = {port}").replace(str(data), "custom/nested/data") + "\n# preserve custom settings\n"
|
||||
config.write_text(settings)
|
||||
server = subprocess.Popen([str(install / "bin/chusql-server"), "--config", str(config)],
|
||||
env=env, cwd=install, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
wait_server(port)
|
||||
query(port, ["create database lifecycle", "use lifecycle", "create table saved (id int)",
|
||||
"insert into saved (id) values (42)"])
|
||||
env["CHUSQL_ADMIN_PASSWORD"] = "Different-update-password-123!"
|
||||
run(installer, env)
|
||||
server.wait(timeout=5)
|
||||
assert config.read_text() == settings
|
||||
assert (home / ".profile").read_text().count("# ChuSQL\n") == 1
|
||||
server = subprocess.Popen([str(install / "bin/chusql-server"), "--config", str(config)],
|
||||
env=env, cwd=install, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
wait_server(port)
|
||||
result = query(port, ["use lifecycle", "select id from saved"])[-1]
|
||||
assert result["rowCount"] == 1 and "42" in json.dumps(result["rows"]), result
|
||||
print("PASS running update preserves configuration, credentials and SQL data", flush=True)
|
||||
uninstall = ["sh", str(install / "uninstall.sh"), "--install-dir", str(install), "--yes"]
|
||||
run(uninstall + ["--keep-data", "--keep-config"], env)
|
||||
server.wait(timeout=5)
|
||||
assert data.is_dir() and config.read_text() == settings
|
||||
assert not (install / "bin").exists()
|
||||
assert "# ChuSQL" not in (home / ".profile").read_text()
|
||||
print("PASS uninstall stops server and keeps nested data and configuration", flush=True)
|
||||
run(installer, env)
|
||||
run(uninstall, env)
|
||||
assert not install.exists() and not data.exists() and not config.exists()
|
||||
print("PASS reinstall and full uninstall discover configured data path", flush=True)
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.terminate()
|
||||
server.wait(timeout=10)
|
||||
test_online(repo, archive, root, env)
|
||||
if len(sys.argv) == 3:
|
||||
with tempfile.TemporaryDirectory(prefix="chusql-systemd-") as service_temporary:
|
||||
test_systemd(repo, archive, Path(service_temporary), env)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+40
-14
@@ -17,7 +17,7 @@ Uninstall ChuSQL (Linux/macOS).
|
||||
Usage: uninstall.sh [options]
|
||||
|
||||
--install-dir DIR program directory (default: $HOME/.local/share/chusql)
|
||||
--data-dir DIR data directory (default: <install-dir>/data)
|
||||
--data-dir DIR data directory (default: settings.toml, then XDG data directory)
|
||||
--keep-data keep the data directory, remove programs only
|
||||
--keep-config keep settings.toml
|
||||
--yes do not ask (required when there is no terminal)
|
||||
@@ -36,6 +36,21 @@ fail() {
|
||||
stop_by_path() {
|
||||
target=$1
|
||||
[ -n "$target" ] || return 0
|
||||
if [ "$(uname -s)" = Linux ]; then
|
||||
for process in /proc/[0-9]*/exe; do
|
||||
executable=$(readlink "$process" 2>/dev/null) || continue
|
||||
[ "$executable" = "$target" ] || continue
|
||||
pid=${process#/proc/}; pid=${pid%/exe}
|
||||
kill "$pid" || fail "cannot stop $target ($pid)"
|
||||
attempts=0
|
||||
while [ -e "$process" ]; do
|
||||
attempts=$((attempts + 1))
|
||||
[ "$attempts" -lt 100 ] || fail "timed out stopping $target ($pid)"
|
||||
sleep 0.1
|
||||
done
|
||||
done
|
||||
return 0
|
||||
fi
|
||||
pids=$(ps -eo pid=,args= 2>/dev/null |
|
||||
awk -v t="$target" '{ pid = $1; $1 = ""; sub(/^ +/, ""); if (index($0, t) == 1) print pid }' || true)
|
||||
[ -n "$pids" ] || return 0
|
||||
@@ -73,9 +88,27 @@ while [ $# -gt 0 ]; do
|
||||
done
|
||||
|
||||
install_dir=${install_dir:-$HOME/.local/share/chusql}
|
||||
data_dir=${data_dir:-${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data}
|
||||
config_dir="${XDG_CONFIG_HOME:-$HOME/.config}/ChuSQL"
|
||||
config_file="$config_dir/settings.toml"
|
||||
if [ -z "$data_dir" ] && [ -f "$config_file" ]; then
|
||||
data_dir=$(awk '/^[[:space:]]*\[storage\]/ { storage=1; next } /^[[:space:]]*\[/ { storage=0 } storage && /^[[:space:]]*data_dir[[:space:]]*=[[:space:]]*"[^"\\]*"[[:space:]]*(#.*)?$/ { sub(/^[^=]*=[[:space:]]*"/, ""); sub(/"[[:space:]]*(#.*)?$/, ""); print; exit }' "$config_file")
|
||||
[ -n "$data_dir" ] || fail 'settings.toml needs an explicit double-quoted storage.data_dir without escapes; pass --data-dir'
|
||||
case "$data_dir" in /*) ;; *) data_dir="$install_dir/$data_dir" ;; esac
|
||||
fi
|
||||
data_dir=${data_dir:-${XDG_DATA_HOME:-$HOME/.local/share}/chusql/data}
|
||||
if [ "$(uname -s)" = Linux ]; then
|
||||
install_dir=$(realpath -m -- "$install_dir")
|
||||
data_dir=$(realpath -m -- "$data_dir")
|
||||
else
|
||||
if [ -d "$install_dir" ]; then install_dir=$(CDPATH= cd -- "$install_dir" && pwd -P); fi
|
||||
if [ -d "$data_dir" ]; then data_dir=$(CDPATH= cd -- "$data_dir" && pwd -P); fi
|
||||
fi
|
||||
case "$install_dir" in /|"$HOME") fail 'refusing to remove the root or home directory' ;; esac
|
||||
case "$data_dir" in /|"$HOME") fail 'refusing to remove the root or home data directory' ;; esac
|
||||
case "$install_dir" in "$data_dir"|"$data_dir"/*) fail 'data directory must not contain the installation' ;; esac
|
||||
for entry in bin static csql csql-web csql-web.sh uninstall.sh; do
|
||||
case "$data_dir" in "$install_dir/$entry"|"$install_dir/$entry"/*) fail 'data directory overlaps installed program files' ;; esac
|
||||
done
|
||||
profile="$HOME/.profile"
|
||||
|
||||
if [ "$assume_yes" = 'no' ]; then
|
||||
@@ -108,23 +141,16 @@ step 'Removing files'
|
||||
if [ ! -d "$install_dir" ]; then
|
||||
note "not found: $install_dir"
|
||||
elif [ "$keep_data" = 'yes' ]; then
|
||||
for entry in "$install_dir"/*; do
|
||||
for entry in "$install_dir"/bin "$install_dir"/static "$install_dir"/csql "$install_dir"/csql-web "$install_dir"/csql-web.sh "$install_dir"/uninstall.sh; do
|
||||
[ -e "$entry" ] || continue
|
||||
if [ "$entry" = "$data_dir" ]; then
|
||||
note "kept data: $entry"
|
||||
continue
|
||||
fi
|
||||
rm -rf "$entry"
|
||||
done
|
||||
note "kept $install_dir (data only)"
|
||||
note "kept data: $data_dir"
|
||||
else
|
||||
rm -rf "$install_dir"
|
||||
case "$data_dir" in
|
||||
"$install_dir" | "$install_dir"/*) ;;
|
||||
*) rm -rf "$data_dir" ;;
|
||||
esac
|
||||
note "removed $install_dir"
|
||||
fi
|
||||
if [ "$keep_data" = 'no' ] && [ -d "$data_dir" ]; then rm -rf "$data_dir"; fi
|
||||
|
||||
step 'Removing settings.toml'
|
||||
if [ "$keep_config" = 'yes' ]; then
|
||||
@@ -139,11 +165,11 @@ step 'Removing the PATH entry'
|
||||
marker='# ChuSQL'
|
||||
if [ ! -f "$profile" ]; then
|
||||
note "no $profile"
|
||||
elif grep -q "^$marker" "$profile"; then
|
||||
elif grep -q "^$marker$" "$profile"; then
|
||||
tmp="$profile.chusql-uninstall"
|
||||
awk -v m="$marker" '
|
||||
skip > 0 { skip--; next }
|
||||
index($0, m) == 1 { skip = 2; next }
|
||||
$0 == m { skip = 2; next }
|
||||
{ print }
|
||||
' "$profile" >"$tmp" && mv "$tmp" "$profile"
|
||||
note "removed the $marker block from $profile"
|
||||
|
||||
Reference in new issue
Block a user