introduce CTE + RECURSIVE

[valq7711]

db.define_table(
    'org',
    Field('name'),
    Field('boss', 'reference org')
)

# define CTE
works_for = db(org.name == 'Alice').cte(
    'works_for',
    org.id,
    org.name.with_alias('top_boss'),  # i.e. Alice is top_boss
    org.name,
    Expression(db, '1', type = 'integer').with_alias('xdepth')
)

# add recursive part (also there is .union_all)
works_for.union(
    db((org.boss == works_for.id) & (org.id != org.boss)).nested_select(
        org.id,
        works_for.top_boss,
        org.name,
        (works_for.xdepth + 1).with_alias('xdepth')
    )
)


print(db(works_for.xdepth <= 4)._select(works_for.ALL))
>>>
WITH RECURSIVE 
    works_for(id, top_boss, name, xdepth) AS (
        SELECT
            "org"."id", 
            "org"."name" AS top_boss,
            "org"."name",
            (1) AS xdepth FROM "org"
        WHERE 
            ("org"."name" = 'Alice') 
        UNION SELECT
            "org"."id",
            "works_for"."top_boss",
            "org"."name", 
            ("works_for"."xdepth" + 1) AS xdepth 
        FROM
            "org", 
            "works_for" 
        WHERE 
            (("org"."boss" = "works_for"."id") 
            AND ("org"."id" <> "org"."boss"))
    )
SELECT
    "works_for"."id", 
    "works_for"."top_boss", 
    "works_for"."name",
    "works_for"."xdepth"
FROM 
    "works_for"
WHERE 
    ("works_for"."xdepth" <= 4);

[valq7711]

@mdipierro
any feedback on the concept?

[valq7711]

@mdipierro
Hello!

[mdipierro]

I am sorry. Been swamped. Your PR are particularly complex. I will review asap.

[mdipierro]

I love this but I have some requests:

1. document the simplest usage case as well a more complex usage case
2. provide unit test for both cases
3. Can we simplify the UI with a single API without loosing generality? You have given more thought to this then mine so perhaps the answer maybe no.

[valq7711]

1, 2 - yes of course, I just needed your approval of the general concept
3 - ? Is it about RestAPI?

[mdipierro]

3. Not specifically. Just because I am afraid people will not use this feature if too complex.

…

On Sun, Apr 25, 2021, 11:11 valq7711 ***@***.***> wrote: 1, 2 - yes of course, I just needed your approval of the general concept 3 - ? Is it about RestAPI? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#656 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHLZT5YQXADGL2IV3XY3ELTKRLLNANCNFSM42KYKYQA> .

[valq7711]

Those people who need recursive select will definitely use it. We have simplest syntax!!!, just compare
https://docs.peewee-orm.com/en/latest/peewee/querying.html#recursive-ctes
https://gist.github.com/cairabbit/d64fccf7cf2abe180e69c843706f46c7

due to self-reference, single call can be look like this (it is easy to implement)

works_for = db(org.name == 'Alice').cte(
    'works_for',
    org.id,
    org.name.with_alias('top_boss'),  # i.e. Alice is top_boss
    org.name,
    Expression(db, '1', type = 'integer').with_alias('xdepth')
).union( lambda works_for:  \
    db((org.boss == works_for.id) & (org.id != org.boss)).nested_select(
        org.id,
        works_for.top_boss,
        org.name,
        (works_for.xdepth + 1).with_alias('xdepth')
    )
)

[valq7711]

Also, CTE seems to be better than nested select
https://www.alisa-in.tech/post/2019-10-02-ctes/#:~:text=A%20Common%20Table%20Expression%20(aka,as%20part%20of%20a%20query.&text=A%20subquery%20is%20a%20nested,used%20again%20within%20that%20query.

[valq7711]

cte can be used to denormalize/flatterize

user = db(db.auth_user.id == db.profile.user).cte(
        'user',
        db.auth_user.ALL,
        db.profile.id.with_alias('profile_id'),
        db.profile.image
)

>>> db(user.username.startswith('v'))._select(user.ALL))

WITH 
    user(id, username, email, password, first_name, last_name, sso_id, action_token, last_password_change, profile_id, image) 
    AS (SELECT 
            "auth_user"."id", 
            "auth_user"."username", 
            "auth_user"."email", 
            "auth_user"."password", 
            "auth_user"."first_name", 
            "auth_user"."last_name", 
            "auth_user"."sso_id", 
            "auth_user"."action_token", 
            "auth_user"."last_password_change", 
            "profile"."id" AS profile_id,
            "profile"."image" 
        FROM "auth_user", "profile" 
        WHERE ("auth_user"."id" = "profile"."user")
    )
    
    SELECT 
        "user"."id",
        "user"."username", 
        "user"."email", 
        "user"."password", 
        "user"."first_name", 
        "user"."last_name", 
        "user"."sso_id", 
        "user"."action_token", 
        "user"."last_password_change", 
        "user"."profile_id"
        "user"."image"
    FROM "user" 
    WHERE ("user"."username" LIKE 'v%' ESCAPE '\');"

[remcoboerma]

Thank you so much for CTE support. That will make a few things way easier and with less database queries.