diff --git a/AGENTS.md b/AGENTS.md index bc0aee8..5967b12 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,5 +28,5 @@ - 改动说明只在交流中记录。 ## 当前重点 -- 主线已完成:高级优化器 → 并行查询 → M1 会话隔离 → M2 快照事务 → M3 WAL 与崩溃恢复 → M4 表结构进 WAL → M5 系统表统一命名 → M6 检查点按 LSN 部分截断 → M7 事务写写冲突检测 → M8 事务保存点 → M9 角色继承 → M10 转授权 → M11 FROM 派生表与投影别名。 +- 主线已完成:高级优化器 → 并行查询 → M1 会话隔离 → M2 快照事务 → M3 WAL 与崩溃恢复 → M4 表结构进 WAL → M5 系统表统一命名 → M6 检查点按 LSN 部分截断 → M7 事务写写冲突检测 → M8 事务保存点 → M9 角色继承 → M10 转授权 → M11 FROM 派生表与投影别名 → M12 `WITH` / CTE。 - 进度与计数见 `agent_credientials/project-plan.txt`,限制与风险见 `agent_credientials/known-issues.txt`。 diff --git a/chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs b/chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs index ddf2693..96d67a2 100644 --- a/chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs +++ b/chusql-core/engine/src/ChuSQL/Core/Engine/Syntax/Parser.hs @@ -171,7 +171,7 @@ inPredicate = do -- | 括号里的子查询;对外层列的引用留给语义检查填 subquerySelect :: Parser Subquery -subquerySelect = Subquery <$> selectStatement <*> pure [] +subquerySelect = Subquery <$> queryStatement <*> pure [] -- | 聚合调用:名称与节点都由内置函数表给出 aggregateCall :: Parser Expr @@ -278,6 +278,7 @@ reservedWords = , "auto_increment" , "type" , "group" + , "with" ] -- | 别名(不许用保留字) @@ -307,7 +308,7 @@ fromSource = fromSubquerySource <|> (uncurry FromTable <$> tableRef) -- | 派生表:括号里的一条 SELECT,必须有别名 fromSubquerySource :: Parser FromClause fromSubquerySource = do - stmt <- between (symbol "(") (symbol ")") selectStatement + stmt <- between (symbol "(") (symbol ")") queryStatement mAlias <- optional ((keyword "as" *> aliasName) <|> aliasName) case mAlias of Nothing -> fail "FROM subquery needs an alias" @@ -598,6 +599,123 @@ selectStatement = do then Select (map fst cols) fromC mWhere groupBy orderBy mLimit else SelectExpr cols fromC mWhere groupBy orderBy mLimit +-- | 读 WITH:展开每个 CTE 再读主查询 +withStatement :: Parser Statement +withStatement = do + keyword "with" + defs <- sepBy1 cteDef (symbol ",") + main <- selectStatement + either fail return (expandWith defs main) + +-- | 一个 CTE:名字、可选列清单与查询 +cteDef :: Parser (String, [String], Statement) +cteDef = do + name <- identifier + cols <- fromMaybe [] <$> optional (between (symbol "(") (symbol ")") (sepBy1 identifier (symbol ","))) + keyword "as" + body <- between (symbol "(") (symbol ")") queryStatement + return (name, cols, body) + +-- | 查询:允许 WITH 开头 +queryStatement :: Parser Statement +queryStatement = withStatement <|> selectStatement + +-- | 按顺序把每个 CTE 展开成派生表 +expandWith :: [(String, [String], Statement)] -> Statement -> Either String Statement +expandWith defs main = go [] defs + where + -- | 前面的 CTE 已进环境,接着展开本体 + go env [] = expandStatement env main + go env ((name, cols, body) : rest) = do + body' <- expandStatement env body + named <- renameColumns name cols body' + go (env ++ [(cteKey name, named)]) rest + +-- | CTE 名不分大小写 +cteKey :: String -> String +cteKey = map toLower + +-- | CTE 带列清单时改本体的输出名 +renameColumns :: String -> [String] -> Statement -> Either String Statement +renameColumns _ [] q = Right q +renameColumns name cols q@Select { selectCols = names } = do + checkCteArity name cols names + Right + ( SelectExpr + [(c, Col n) | (c, n) <- zip cols names] + (selectFrom q) + (selectWhere q) + (selectGroupBy q) + (selectOrderBy q) + (selectLimit q) + ) +renameColumns name cols q@SelectExpr { selectItems = items } = do + checkCteArity name cols (map snd items) + Right q { selectItems = zip cols (map snd items) } +renameColumns _ _ _ = Left "WITH: a CTE must be a SELECT" + +-- | 列清单个数必须和本体输出一致 +checkCteArity :: String -> [String] -> [a] -> Either String () +checkCteArity name cols out + | length cols == length out = Right () + | otherwise = Left ("WITH: column list of " ++ name ++ " does not match its query") + +-- | 把语句里对 CTE 的引用换成派生表 +expandStatement :: [(String, Statement)] -> Statement -> Either String Statement +expandStatement env q = case q of + q@Select { selectFrom = fromC, selectWhere = mWhere } -> do + fromC' <- expandClause env fromC + w <- traverse (expandExpr env) mWhere + Right q { selectFrom = fromC', selectWhere = w } + q@SelectExpr { selectFrom = fromC, selectItems = items, selectWhere = mWhere } -> do + fromC' <- expandClause env fromC + items' <- traverse (\(label, e) -> fmap ((,) label) (expandExpr env e)) items + w <- traverse (expandExpr env) mWhere + Right q { selectFrom = fromC', selectItems = items', selectWhere = w } + _ -> Right q + +-- | 来源树里的 CTE 引用换成派生表 +expandClause :: [(String, Statement)] -> FromClause -> Either String FromClause +expandClause env fromC = case fromC of + FromTable mAlias tbl + | Just body <- lookup (cteKey tbl) env -> Right (FromSubquery (Just (fromMaybe tbl mAlias)) body) + | otherwise -> Right fromC + FromSubquery mAlias stmt -> FromSubquery mAlias <$> expandStatement env stmt + FromJoin kind left right cond -> + FromJoin kind <$> expandClause env left <*> expandClause env right <*> expandExpr env cond + FromUnit -> Right FromUnit + +-- | 表达式里的子查询也要展开 +expandExpr :: [(String, Statement)] -> Expr -> Either String Expr +expandExpr env e = case e of + Add a b -> Add <$> go a <*> go b + Sub a b -> Sub <$> go a <*> go b + Mul a b -> Mul <$> go a <*> go b + Div a b -> Div <$> go a <*> go b + Gt a b -> Gt <$> go a <*> go b + Lt a b -> Lt <$> go a <*> go b + Eq a b -> Eq <$> go a <*> go b + And a b -> And <$> go a <*> go b + Or a b -> Or <$> go a <*> go b + Neg a -> Neg <$> go a + IsNull a -> IsNull <$> go a + IsNotNull a -> IsNotNull <$> go a + CountOf a -> CountOf <$> go a + SumOf a -> SumOf <$> go a + AvgOf a -> AvgOf <$> go a + MinOf a -> MinOf <$> go a + MaxOf a -> MaxOf <$> go a + ScalarSub sq -> ScalarSub <$> goSub sq + InSub x sq neg -> (\x' sq' -> InSub x' sq' neg) <$> go x <*> goSub sq + InList x es neg -> (\x' es' -> InList x' es' neg) <$> go x <*> traverse go es + ExistsSub sq neg -> (\sq' -> ExistsSub sq' neg) <$> goSub sq + _ -> Right e + where + -- | 继续看子表达式 + go = expandExpr env + -- | 继续看子查询里的语句 + goSub sq = (\stmt -> sq { subqueryStatement = stmt }) <$> expandStatement env (subqueryStatement sq) + -- | 读 INSERT,可一次插多行 insertStatement :: Parser Statement insertStatement = do @@ -660,7 +778,8 @@ parseStatement input = where -- | 按关键字分派到各语句解析器 statementP = - transactionStatement + withStatement + <|> transactionStatement <|> selectStatement <|> try (CreateDatabase <$> (keyword "create" *> keyword "database" *> identifier)) <|> try (DropDatabase <$> (keyword "drop" *> keyword "database" *> identifier)) diff --git a/chusql-core/engine/test/Spec.hs b/chusql-core/engine/test/Spec.hs index 275238d..99ef9d9 100644 --- a/chusql-core/engine/test/Spec.hs +++ b/chusql-core/engine/test/Spec.hs @@ -459,6 +459,65 @@ main = hspec $ do sql "SELECT d.name FROM users u JOIN (SELECT name FROM users WHERE id = u.id) d ON u.name = d.name" `shouldSatisfy` isLeft + describe "ChuSQL.Core.Engine (WITH)" $ do + -- sql:在默认库上跑 SQL 取结果行 + let sql input = parseStatement input >>= rowsOf . runStatement testDB + -- values:只取结果里的值 + values input = fmap (map (map snd)) (sql input) + it "parses a CTE into a derived table" $ do + parseStatement "WITH t AS (SELECT name FROM users) SELECT t.name FROM t" + `shouldBe` Right + ( Select + { selectCols = ["t.name"] + , selectFrom = + FromSubquery + (Just "t") + (Select ["name"] (FromTable Nothing "users") Nothing [] [] Nothing) + , selectWhere = Nothing + , selectGroupBy = [] + , selectOrderBy = [] + , selectLimit = Nothing + } + ) + + it "runs a CTE as a source" $ do + values "WITH t AS (SELECT name FROM users) SELECT t.name FROM t" + `shouldBe` Right [[VStr "Alice"], [VStr "Bob"], [VStr "Carol"]] + + it "refers to an earlier CTE inside the next one" $ do + values "WITH young AS (SELECT name FROM users WHERE age < 20), picked AS (SELECT y.name FROM young y) SELECT p.name FROM picked p" + `shouldBe` Right [[VStr "Bob"]] + + it "renames the output columns of a CTE" $ do + values "WITH t(who) AS (SELECT name FROM users) SELECT t.who FROM t ORDER BY t.who" + `shouldBe` Right [[VStr "Alice"], [VStr "Bob"], [VStr "Carol"]] + + it "takes a bare column from a CTE" $ do + values "WITH t(who) AS (SELECT name FROM users) SELECT who FROM t ORDER BY who" + `shouldBe` Right [[VStr "Alice"], [VStr "Bob"], [VStr "Carol"]] + + it "runs a CTE inside an IN subquery" $ do + values "WITH grown AS (SELECT id FROM users WHERE age > 20) SELECT name FROM users WHERE id IN (SELECT id FROM grown) ORDER BY name" + `shouldBe` Right [[VStr "Alice"], [VStr "Carol"]] + + it "joins a CTE with itself" $ do + values "WITH t AS (SELECT id, name FROM users) SELECT a.name FROM t a JOIN t b ON a.id = b.id ORDER BY a.name" + `shouldBe` Right [[VStr "Alice"], [VStr "Bob"], [VStr "Carol"]] + + it "keeps a CTE inside a derived table" $ do + values "SELECT d.name FROM (WITH t AS (SELECT name FROM users) SELECT t.name FROM t) d" + `shouldBe` Right [[VStr "Alice"], [VStr "Bob"], [VStr "Carol"]] + + it "rejects a column list that does not match" $ do + sql "WITH t(a, b) AS (SELECT name FROM users) SELECT t.a FROM t" `shouldSatisfy` isLeft + + it "reports an unknown table for a forward reference" $ do + sql "WITH a AS (SELECT b.name FROM b), b AS (SELECT name FROM users) SELECT a.name FROM a" + `shouldSatisfy` isLeft + + it "reports an unknown table for a name that is not a CTE" $ do + sql "WITH t AS (SELECT name FROM users) SELECT x.name FROM x" `shouldSatisfy` isLeft + describe "ChuSQL.Core.Engine (SELECT)" $ do -- sql:在默认库上跑 SQL 取结果行 let sql input = parseStatement input >>= rowsOf . runStatement testDB diff --git a/chusql-server/test/Spec.hs b/chusql-server/test/Spec.hs index fa69888..15e6a99 100644 --- a/chusql-server/test/Spec.hs +++ b/chusql-server/test/Spec.hs @@ -652,6 +652,15 @@ privilegeSpec = describe "server privileges" $ do authorizeSql service (Ordinary testAccount) "SELECT d.name FROM (SELECT product FROM orders) d" `shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON orders") + it "requires the privilege on the table inside a CTE" $ withPrivileges $ \service root -> do + _ <- runPrivilegeCommand service root testDatabase (CreateRoleCommand "reader") + _ <- runPrivilegeCommand service root testDatabase (GrantPrivilegesCommand ["select"] "users" "reader" False) + _ <- runPrivilegeCommand service root testDatabase (GrantRoleCommand "reader" ["alice"]) + authorizeSql service (Ordinary testAccount) "WITH d AS (SELECT name FROM users) SELECT d.name FROM d" + `shouldReturn` Right () + authorizeSql service (Ordinary testAccount) "WITH d AS (SELECT product FROM orders) SELECT d.product FROM d" + `shouldReturn` Left (PrivilegeError "forbidden" "permission denied: SELECT ON orders") + it "keeps the role tables on the real storage" $ withIpcStorage $ do base <- ipcBackend service <- newPrivileges base diff --git a/docs/architecture.md b/docs/architecture.md index 93b8d2c..eb2d008 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -49,6 +49,8 @@ chusql-server 语句里的库前缀和表限定符由引擎在语义层解析:存储层只接受精确的表名键,因此 `FROM sales.orders` 与选定 `sales` 后的 `FROM orders` 指向同一张表。FROM 的来源是一棵树:单表、派生表(括号里的子查询,必须带别名)与连接共用同一个 `FromClause`,所以 `JOIN` 两侧都能是派生表。解析派生表时先把内层语句在空作用域里解析成一张“表”——内层不能引用外层列,输出列统一改成裸列名(内层 `u.id` 在外层用 `d.id` 引用),重名直接报错;执行侧对应一个 `Derived` 算子,把内层计划的结果按别名加前缀。派生表不参与连接顺序重排(叶子的行数未知),投影也不下推进去,但内层计划照常做谓词下推与索引改写。投影项的 `AS` 别名与 `SELECT` 的输出列名一起在语义层解析,派生表与最外层查询都按别名暴露列名。 +`WITH` 不新增语法树节点:解析器在读到一个 `WITH` 时就把每个 CTE 的名字与语句记进一张环境表,遇到 `FROM` 里出现的 CTE 名(不分大小写,只认前面已定义的)就换成 M11 的派生表节点,主语句与 CTE 内部都能再写 `WITH`。这样展开出来的语句对语义层、执行器与鉴权完全等价于手写派生表:CTE 用列清单改名走派生表的投影别名,来源树与权限覆盖不必另做一套。代价是 CTE 不做物化(引用两次就执行两次)、不支持递归,且前向引用会当成普通表名去查。 + 显式事务在会话层实现:`BEGIN` 把当前数据库整库读成内存快照,事务内的语句都在快照上执行;`SAVEPOINT` 把当前快照压进会话里的保存点栈,`ROLLBACK TO` 退回该点并保留它,`RELEASE` 丢掉该点与其后的保存点;`COMMIT` 先把快照、基线与库里现值按行 `id` 逐键比对,检测到写写冲突就报 `serialization_failure` 并保留事务;没有冲突才把差分用一条 `apply_transaction` 请求交给存储层应用;`ROLLBACK` 直接丢弃快照。因此隔离级别是快照读加原子提交:事务里看不到别的事务未提交或后来提交的改动,提交对外一次性可见,冲突由调用方回滚或重试。 WAL 负责崩溃恢复:每条记录带自增 LSN,一组写操作与一条提交标记一次落盘;每组提交在数据和字典都刷盘之后把 `wal.checkpoint` 推进到这条提交的 LSN,含义是「到此为止的改动都已落盘」,再按这个边界丢掉日志前缀——所以日志里通常只有当组在飞的记录。进程重启时只重放检查点之后的记录,按标记分组取已提交的组、丢弃没有标记的整组,重放前逐条做幂等检查作为第二道保险,结束后再把检查点推到最后一条提交。LSN 从检查点与文件尾里较大的那个续号,跨重启单调,`SystemStatus` 应答里的 `last_lsn` 报告该检查点 LSN。建表、删表、建索引、删索引与数据改动走同一条 WAL,恢复时一并重放。 diff --git a/docs/commands.md b/docs/commands.md index f61da16..180a18a 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -52,6 +52,14 @@ csql -f json -e "SELECT count(*) FROM orders" - 投影列可以起别名:`SELECT count(*) AS total FROM orders`,别名就是结果集的列名。`ORDER BY` 仍按来源列解析,要引用派生表的列得写限定名(如 `ORDER BY d.total`)。 - 鉴权覆盖整棵来源树:查派生表时内层表也要有权限,只有外层引用的表有权限不够。 +公共表表达式: + +- `WITH 名字 AS (SELECT …), 另一个 AS (SELECT …) SELECT … FROM 名字` 先定义再引用,名字可以带列清单(`WITH t(who) AS (SELECT name FROM users) …`)。 +- `WITH` 在解析期就展开成派生表,所以引用处只能当普通表用:没有别名时用 CTE 名当限定名(`WITH t AS (…) SELECT t.name FROM t`)。 +- 名字不分大小写;只能引用前面已经定义的 CTE,向前引用会当成普通表名去查(有同名表就用表,没有就报 `unknown table`)。 +- 不支持递归 CTE,主语句与每个 CTE 都只能是 `SELECT`;同一个 CTE 被引用两次会各执行一次(不做物化)。 +- 鉴权同样覆盖 CTE 里的表:`WITH d AS (SELECT name FROM orders) …` 要求对 `orders` 有权限。 + 事务: - `BEGIN`(或 `START TRANSACTION`)开始显式事务,`COMMIT` 提交,`ROLLBACK` 回滚。