Get best of the week

Arbre - un seul AST pour tout gouverner

Bonjour, je m'appelle Dmitry Karlovsky et je ... dissèque sur les vélos ... hors route ... contre le vent ... en montée ... en ski. Et aujourd'hui, je vous invite à rouler avec moi le long et à travers les formats de données texte et à concevoir ensemble le format idéal.


J'en ai déjà parlé il y a 5 ans, ce qui a conduit à un débat houleux, qui a entraîné de petits changements de syntaxe. Par conséquent, permettez-moi de vous dire à partir de zéro ce que c'est actuellement.


 \ 
 \PiterJS #47
 2020-05-20

Il s'agit d'une version texte étendue des performances éponymes de PiterJS # 47 . Vous pouvez le lire sous forme d'article , l' ouvrir dans l'interface de présentation ou regarder une vidéo .


Plan


  • Analyser les formats de données de texte populaires
  • À partir de zéro, développez un nouveau format sans défauts
  • Montrer des exemples d'application du nouveau format

Les formats


Nous comparerons 5 formats.


Format
XML
Json
Yaml
Tol
Arbre

Les trois premiers n'ont pas entendu que les sourds. Mais les deux derniers pour beaucoup sont des chevaux noirs. Eh bien, rien, aujourd'hui je vais les éclairer.


XML


XML — , " ". , , -.


<!DOCTYPE svg
    PUBLIC "-//W3C//DTD SVG 1.1//EN"
    "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"
>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg version="1.1" xmlns="http://www.w3.org/2000/svg">
    <circle r="30" cx="50" cy="50" fill="orange" />
</svg>

JSON


XML — JSON.


{
    "name": "example",
    "version": "1.0.0",
    "description": "example package",
    "main": "index.js",
    "repository": "https://example.org",
    "author": "anonymous",
    "license": "MIT"
}

, , , , .


YAML


- YAML JSON.


Date: 2001-11-23 15:03:17 -5
User: ed
Fatal:
  Unknown variable "bar"
Where:
  file: TopClass.py
  line: 23
  code: |
    x = MoreObject("345\n")

.


TOML


TOML . , , .


[servers]

[servers.alpha]
ip = "10.0.0.1"
dc = "eqdc10"

[servers.beta]
ip = "10.0.0.2"
dc = "eqdc10"

, , INI-, JSON. .


Tree


, , Tree, .


spoiler


. .


  • ?
  • ?

, .


XML


XML .


  • NodeList
  • Element Node (<br/>)
  • Attribute Node (tabindex="1")
  • Text Node (Hello, World!)
  • CDATA Node (<![CDATA[ ... ]]>)
  • Processing Instruction Node (<? ... ?>)
  • Comment Node (<!-- ... -->)
  • Document Node
  • Document Type Node (<!DOCTYPE html>)

XML


, : , . , XML , . , : .


<panel>
    <head> ?</head>
    <body>
        <button></button>
        <button></button>
    </body>
</panel>

panel — , body — , . , .


XML


, XML , .


<xsl:stylesheet
    version="1.0"
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:template match="/">
        <html>
            <head>
                <link rel="stylesheet" href="web.css" />
            </head>
            <body>
                <xsl:apply-templates select="*" />
            </body>
        </html>
    </xsl:template>

</xsl:stylesheet>

— , .


JSON


JSON , . .


  • Null
  • Boolean
  • Number
  • String
  • Array
  • Dictionary

JSON


, . , . , .


{
    "foo": 777,
    "bar": 666
}

?


[
    [ "foo" , 777 ],
    [ "bar" , 666 ]
]

. - .


JSON


JSON , - , . MongoDB, , JSON .


{
    "$or": [
        {
            "sex": "female",
            "age": { "$gt": 16 },
        },
        {
            "hobby": {
                "$regex": "\\b(?:java|type)script\\b"
            }
        }
    ]
}

, OR AND . , "", "" " ". , , JSON , "$regexp", .


YAML


YAML JSON. .


  • !!null
  • !!bool
  • !!int
  • !!float
  • !!str
  • !!timestamp
  • !!seq
  • !!map
  • Anchor & Alias
  • Document
  • TypeTags

YAML


YAML , .


--- !!omap
- foo: 777
- bar: 666

" -" OrderedMap ( ).


TOML


TOML JSON, . , , , .


  • Boolean
  • Integer
  • Float
  • String
  • DateTime
  • Array
  • Dictionary

, JSON.


Tree


, . . , , . . .


  • Struct Node
  • Data Node

, . , .



, . , , , .


XMLJSONYAMLTOMLTree
+-+-+
90302109010

YAML. . , . , JSON , 30 .


, , .



: , , , , .


. .


XMLJSONYAMLTOMLTree
--+++

XML


XML , . , , , , .


, !
 ?
       ?

<message>
    <greeting>
        , <a href="http://example.org/user/alice"></a>!
    </greeting>
    <body>
        <s> ?</s><br/>
             
        <time datetime="1979-10-14T12:00:00.001-04:00"></time>
        ?
    </body>
</message>

JSON


XML , JSON, , . , .


{ "greetings": ", !\n ?\n      ?\n" }


. YAML .


XMLJSONYAMLTOMLTree
++-++

YAML


a: true # boolean
b: tru  # string
c: (-:  # error
d: :-)  # string

YAML .



— . . .



XML


XML — , .


foo > 0 && foo < 10

- , , .


`foo &gt; 0 &amp;&amp; foo &lt; 10`

JSON


JSON, . - VSCode, , JSON , .


/"[\s\S]*"/

, . , - .


"\"[\\s\\S]*\""

YAML


YAML , .


  • 5
  • 4

, YAML .


Tree

— . . , , , .



. .



XML


<users>
    <user>
        <name>Alice</name>
        <age>20</age>
    </user>
</users>

XML, , .


<!-- 13% less -->
<users><user><name>Alice</name><age>20</age></user></users>

JSON


{
    "users": [
        {
            "name": "Alice",
            "age": 20
        }
    ]
}

JSON , — .


// 30% less
{"users":[{"name":"Alice","age":20}]}

Tree

— , .



XMLJSONYAMLTOMLTree
195%140%125%110%100%
170%101%---

.


, , , , . , , , .



— , , .


  • ?
  • 2 4 ?
  • ?
  • ?
  • /?
  • //?

, . , , , . , - .



, .


, JSON, , . . . .


serialization:    foo\bar    =>  "foo\\bar"

parsing:         "foo\\bar"  =>   foo\bar

, , . , .



. , JSON, , - , , . , . , . .


XMLJSONYAMLTOMLTree
+---+
----+
----+

XML- , — . , , . - , .


, , . . , AST .



, , , . , . , . .


XMLJSONYAMLTOMLTree
--+++

, . , , — . , , , — .


, . , XML SAX , , : - , , - . JSON . , .


, . , , . , . .


Tree


, , .



tree-


, "house". ?


house

.


tree-


, ?


house
roof
wall
door
window
floor

.


tree-


?


house
    roof
    wall
    door
    window
    floor

. Python — , .


tree-


.


house
    roof
    wall
        door
        window
            glass
    floor


, - - .


street
    house
        wall
            door
            window

, .


street house wall
    window
    door

.



, - , - .


\  \(^_^)/

. . , , , . — , — .



? : .


\
    \ ‍
    \     ‍
    \          ‍

.



, . , .


user
    name \Jin
    age \35
    hobby
        \kendo 
        \dance 
        \role play 
            default

, . 2 4 .



, . , .


XMLXHTML, SVG, XSLT, ...
JSONJSON Schema, json:api, ...
YAMLyaml.org/type
TOML-
Treexml.tree, json.tree, view.tree, ...

— , .


tree.


grammar.tree


grammar.tree — . , tree.


tree .is .optional .list_of line

line .is .sequence
    .optional indent
    .optional nodes
    new_line

nodes .is .sequence
    .optional .list_of struct
    .optional data
    .with_delimiter space

struct .is .list_of .byte
    .except special

data .is .sequence
    data_prefix
    .optional .list_of .byte
        .except new_line

special .is .any_of
    new_line
    data_prefix
    indent
    space

new_line .is .byte \0A
indent .is .list_of .byte \09
data_prefix .is .byte \5C
space .is .list_of .byte \20

, , .


: tree — , — , . .


grammar.tree vs EBNF


grammar.tree , , , — , , , .


tree .is .optional .list_of line

line .is .sequence
    .optional indent
    .optional nodes
    new_line

nodes .is .sequence
    .optional .list_of struct
    .optional data
    .with_delimiter space

tree = { line };

line = [ indent ],
    [ nodes ],
    new_line;

nodes = data |
    struct,
    { space , struct },
    [ space , data ];

xml.tree vs XML


xml.tree — XML tree . XML. , XML xml.tree.


! doctype html
html
    meta @ charset \utf-8
    link
        @ href \web.css
        @ rel \stylesheet
    script @ src \web.js
    body
        h1 \Procter & Gamble

<!doctype html>
<html>

    <meta charset="utf-8" />
    <link href="web.css" rel="stylesheet" />
    <script src="web.js"></script>

    <body>
        <h1>Procter &amp; Gamble</div>
    </body>

</html>

IDE, XML xml.tree , XML. XML , , , markdown.


json.tree vs JSON


json.tree — json.


* user *
    name \Jin
    age 35
    hobby /
        \kendo 
        \dance 
    home \C:\users\jin\

{
    "user": {
        "name": "Jin",
        "age": 35,
        "hobby": [
            "kendo ",
            "dance ",
        ],
        "home": "C:\\users\\jin\\"
    }
}

2 — .


json.tree


, XML Tree , . , json, tree . , , . ?


*
    # \If disabled will be used platform specific delimiters
    # \CRLN on windows and LN on others
    unix_delimiters true

tree , .


{
    "unix_delimiters#1": "If disabled will be used platform specific delimiters",
    "unix_delimiters#2": "CRLN on windows and LN on others",
    "unix_delimiters": true,
}

JSON , - .


view.tree vs TypeScript


view.tree — $mol.


$my_details $mol_view
    sub /
        <= Pager $mol_paginator
            value?val <=> page?val 0

, . , view.tree json.tree , , JSON .


TypeScript . , .


class $my_details extends $mol_view {

    sub() { return [ this.Pager() ] }

    @ $mol_mem Pager() {
        const Pager = new $mol_paginator
        Pager.value = val => this.page( val )
        return Pager
    }

    @ $mol_mem page( val = 0 ) {
        return val
    }

}

API


, API .


API
XMLXHTML, SVG, XSLT, ...DOM, SAX, AST
JSONJSON Schema, json:api, ...Native, AST
YAMLyaml.org/typeNative, AST
TOML-Native, AST
Treexml.tree, json.tree, ...AST

XML, , DOM, SAX. , , , . JSON , — . . . , API.


JSON AST


JSON ASTExplorer.


{
  "user": {}
}

{
    "type" : "Object",
    "children" : [
        {
            "type" : "Property",
            "key" : {
                "type": "Identifier",
                "value": "user"
            }
            "value": {
                "type": "Object",
                "children": []
            }
        }
    ]
}

, AST . JSON AST. .


AST Tree


tree .


user
    name \Jin
    age 35
    hobby
        \kendo 
        \dance 
        \role play

AST.


user
    name \Jin
    age 35
    hobby
        \kendo 
        \dance 
        \role play

, - . . , , , tree — AST.


Tree


TypeScript .


interface $mol_tree2 {
    type: string
    value: string
    kids: $mol_tree2[]
    span: $mol_span
}

Span — .


interface $mol_span {
    uri: string
    row: number
    col: number
    length: number
}

Tree


, . , , span . .


interface $mol_tree2 {
    struct : ( type , kids )=> $mol_tree2
    data : ( value , kids )=> $mol_tree2
    list : ( kids )=> $mol_tree2
    clone : ( kids )=> $mol_tree2
}

Tree


, , — , .


const config_path = './config.tree'
const config_text = fs.readFileSync( config_path )
const config = $mol_tree2.fromString( config_text , config_path )
//  server auth
//      login \root
//      password \qwerty

const password = config.select( 'server' , 'auth' , 'password' , '' )

if( !auth( password.text() ) ) {
    // AuthError: Wrong password
    // \default
    // ./config.tree#5:3-11
    throw password.error( 'Wrong password' , AuthError )
}

Tree


— , "auth" "credentials". :


//  server credentials
//      login \root
//      password \qwerty
const new_config = config.list(
    input.hack({

        'auth' : ( tree , context )=> [
            tree.struct( 'credentials' , tree.hack( context ) ),
        ] ,

        '' : ( tree , context )=> [
            tree.clone( tree.hack( context ) ),
        ] ,

    })
)
fs.writeFileSync( config_path , new_config )

, tree , , AST.




, . , .




, .



XMLJSONYAMLTOMLTree
195%140%125%110%100%
90302109010
++-++
--+++
----+
----+
--+++
+-+-+
+++--


, tree .


  • LISP-
  • AST

sql.tree —


MongoDB? SQL:


select
    from $users
    fetch
        @name
        @phone
        @photo *
            @uri
            @width
            @height
    where or
        and
            @sex = female
            @age > 16
        @hobby ~ \\b(?:java|type)script\b

— , SQL. , "", "" " ". , tree, .


select
    from $users
    fetch *
    where @hobby ~ 
        word-edge
        or
            \java
            \type
        \script
        word-edge

domain.tree —


. .


hyoo_api_person
    descr \  
    inherit hyoo_api_entity
    field
        id
            descr \  
            example \person=jin
            key unique
            type text
            edit author
        avatar
            descr \  
            type list hyoo_api_image
            edit author
        mail
            descr \ 
            type set hyoo_api_mail

API, ACL, .



. — , . , , , — .



access.log.tree —


, , ?


193.34.12.132 - - [2011-10-20T12:46:08+04:00] GET /nin-jin/slides/edit/master/t
ree/readme.md HTTP/1.1 200 4435
193.34.12.132 - - [2011-10-20T12:46:09+04:00] GET /nin-jin/slides/edit/master/t
ree/readme.html HTTP/1.1 404 4435

access
    ip \193.34.12.132
    time \2011-10-20T12:46:08+04:00
    method \GET
    uri \/nin-jin/slides/edit/master/tree/readme.md
    protocol \HTTP/1.1
    response \200
    size \4435

tree-tools — CLI


, . , , , .


> cat access.log.tree | pick ip time method uri | table

\193.34.12.132  2011-10-20T12:46:08+04:00   GET /index.html
\193.34.12.132  2011-10-20T12:46:10+04:00   GET /index.css
\193.34.12.132  2011-10-20T12:46:20+04:00   GET /index.js

> cat access.log.tree | filter time >= 2019-09 | pick ip uri | table

\193.34.12.132  /index.html
\193.34.12.132  /index.css
\193.34.12.132  /index.js

, - . , - . , , .


tree


tree, , . JSON XML, — , . , - , .


> git log

commit
    message \$mol_style: TS@3.9 compatibility
    sha \b1a8f07c839604d0d34430a186246f0c1f71e628
    date \2020-05-15T23:24:32+0300
    author \nin-jin <sairi-na-tenshi@ya.ru>
commit
    message \$mol_regexp: concurent parse ability
    sha \be1abfa50542728dd5c156517ea31f469e7fb4d4
    date \2020-05-15T23:03:30+0300
    author \nin-jin <nin-jin@ya.ru>

> git log | pick date message | table

\2020-05-15T23:24:32+0300   $mol_style: TS@3.9 compatibility
\2020-05-15T23:03:30+0300   $mol_regexp: concurent parse ability

WAT


WebAssembly — , . , s-expressions.


(func $fact (param $x i64) (result i64)
    (if $x (result i64) 
      (i64.eqz
        (local.get $x)) 
      (then
        (i64.const 1))
      (else
        (i64.mul
          (local.get $x)
          (call $fact      
            (i64.sub
              (local.get $x)
              (i64.const 1)))))))

. , .


wasm.tree —


wasm.tree .


func
    name $fact
    param $x i64
    result i64
    body switch
        test i64.eqz local.get $x
        then i64.const 1
        else i64.mul
            local.get $x
            call $fact i64.sub
                local.get $x
                64.const 1

- bin.tree, .


00
61
73
6d
01
00
00
00
.
.
.

- - — WAT2.0. WebAssembly — .


jack.tree — LISP


. -, . , , .


jack
    import wasm
    tree func $fact
        > $x #8
        < #8 switch
            test is-zero $x
            then #8 1
            else mul
                $x
                $fact sub
                    $x
                    #8 1

, AST , wasm-. , , , , tree , . .


$mol_jack


LLVM


wasm , , .


compile pipelines:

                jack.tree => wasm.tree =============> bin.tree
                jack.tree => wasm.tree => arm.tree => bin.tree
any-dsl.tree => jack.tree => wasm.tree => arm.tree => bin.tree

, , , .


optimization midlewares:

jack.tree => jack.tree
wasm.tree => wasm.tree
arm.tree => arm.tree

, , , . AST tree .


, , LLVM.


AST


, , . Tree — AST. , TypeScript .


code =(P)=> loader =(P)=> compiler =(SP)=> bundler =(SP)=> terser =(S)=> bundle

P - Parse
S - Serialize

AST, , . AST, .


code =(P)=> loader =====> compiler ======> bundler ======> terser =(S)=> bundle

( ), tree AST , .


,


. . , , . , .


More articles:


All Articles