Get best of the week

Lena数据分析架构框架简介

哈Ha！我将谈论我正在开发的体系结构框架。


体系结构决定了程序的最一般结构及其组件之间的交互。Lena作为框架实现了用于数据分析的特定体系结构（在下文中有更多有关此方面的信息），并为用户提供了可能有用的类和功能（考虑到此体系结构）。


Lena用流行的Python语言编写，并且可以使用Python 2、3和PyPy版本。它是根据免费Apache许可（版本2）在此处发布的。目前，它仍在开发中，但是本手册中描述的功能已经在使用，测试（整个框架的总覆盖率约为90％），并且不太可能更改。莉娜（Lena）出现在中微子物理学实验数据分析中，以西伯利亚大河而得名。



通常，在大中型项目中会出现体系结构问题。如果您正在考虑使用此框架，那么这里是其任务和优点的简要概述。


从编程的角度来看：


  • 模块化，参与度低。可以轻松添加，替换或重用算法。
  • ( ). . PyPy " ".
  • . . .
  • . , . . .
  • , .

, Python, , .


:


  • ( ).
  • (, , ).
  • . , , .
  • .

(tutorial) – Lena. , , , , . . .



Lena





. , , . .


, . Lena , , . , , .



Lena




Lena — . .


Lena . , :


>>> from __future__ import print_function
>>> from lena.core import Sequence
>>> s = Sequence(
...     lambda i: pow(-1, i) * (2 * i + 1),
... )
>>> results = s.run([0, 1, 2, 3])
>>> for res in results:
...     print(res)
1 -3 5 -7

Lena Python 2 3, print. .


Sequence . run. ( ).


, for.


. - , - . Source:


from lena.core import Sequence, Source
from lena.flow import CountFrom, ISlice

s = Sequence(
    lambda i: pow(-1, i) * (2 * i + 1),
)
spi = Source(
    CountFrom(0),
    s,
    ISlice(10**6),
    lambda x: 4./x,
    Sum(),
)
results = list(spi())
# [3.1415916535897743]

Source __call__, . : , .


CountFrom — , . , ¹. CountFrom ( ). CountFrom — start ( ) step ( 1).


Source ( ) (callable) run. Sequence.


. s Source. , s s.


Sequence , Sequence. Sequence Source, (flow).


: Sequence Source , LenaTypeError ( TypeError Python).

Lena — LenaException. ( , ).

, - . ISlice. ISlice CountFrom islice count itertools Python. ISlice start, stop[, step], ( ) step ( step , ).


, .


.




. run, flow:



class Sum():
    def run(self, flow):
        s = 0
        for val in flow:
            s += val
        yield s

, return, yield. Yield — Python, .


— Python.


>>> results = s.run([0, 1, 2, 3])

Sequence run . , , , . , . ( ) :


>>> for res in results:
...     print(res)

:


  • . . , , , . , .
  • . -. , , .

Python yield. Lena. run, . , , , , - .


(yield) . (flow) . , (value).




Lena . — , .


Lena , . Jinja . Lena , . LaTeX:


% histogram_1d.tex
\documentclass{standalone}
\usepackage{tikz}
\usepackage{pgfplots}
\pgfplotsset{compat=1.15}

\begin{document}
\begin{tikzpicture}
\begin{axis}[]
\addplot [
    const plot,
]
table [col sep=comma, header=false] {\VAR{ output.filepath }};
\end{axis}
\end{tikzpicture}
\end{document}

TikZ , : \VAR{ output.filepath }. \VAR{ var } var . , . output.filepath .


:


\BLOCK{ set var = variable if variable else '' }
\begin{tikzpicture}
\begin{axis}[
    \BLOCK{ if var.latex_name }
        xlabel = { $\VAR{ var.latex_name }$
        \BLOCK{ if var.unit }
            [$\mathrm{\VAR{ var.unit }}$]
        \BLOCK{ endif }
        },
    \BLOCK{ endif }
]
...

variable, var . latex_name unit (), x. , x [m] E [keV] . , , .


Jinja . , . Jinja² .


Jinja LaTeX, Lena ³: \BLOCK \VAR .


— Python . Flow Lena (data, context). dataflow, . , Lena. . , :


class ReadData():
    """Read data from CSV files."""

    def run(self, flow):
        """Read filenames from flow and yield vectors.

        If vector component could not be cast to float,
        *ValueError* is raised.
        """
        for filename in flow:
            with open(filename, "r") as fil:
                for line in fil:
                    vec = [float(coord)
                           for coord in line.split(',')]
                    # (data, context) pair
                    yield (vec, {"data": {"filename": filename}})

flow . data ( ). filename data["filename"] data.filename.


-, HTML LaTeX , , . , . — , - ( ).


Lena. , .


, , . , , .




. x.


docs/examples/tutorial .

main.py


from __future__ import print_function

import os

from lena.core import Sequence, Source
from lena.math import mesh
from lena.output import HistToCSV, Writer, LaTeXToPDF, PDFToPNG
from lena.output import MakeFilename, RenderLaTeX
from lena.structures import Histogram

from read_data import ReadData

def main():
    data_file = os.path.join("..", "data", "normal_3d.csv")
    s = Sequence(
        ReadData(),
        lambda dt: (dt[0][0], dt[1]),
        Histogram(mesh((-10, 10), 10)),
        HistToCSV(),
        MakeFilename("x"),
        Writer("output"),
        RenderLaTeX("histogram_1d.tex"),
        Writer("output"),
        LaTeXToPDF(),
        PDFToPNG(),
    )
    results = s.run([data_file])
    print(list(results))

if __name__ == "__main__":
    main()

, output/, :


$ python main.py
pdflatex -halt-on-error -interaction batchmode -output-directory output output/x.tex
pdftoppm output/x.pdf output/x -png -singlefile
[(‘output/x.png’, {‘output’: {‘filetype’: ‘png’}, ‘data’: {‘filename’: ‘../data/normal_3d.csv’}, ‘histogram’: {‘ranges’: [(-10, 10)], ‘dim’: 1, ‘nbins’: [10]}})]

LaTeXToPDF pdflatex, PDFToPNG pdftoppm. , LaTeX , output/x.tex ( ).


— , (run) . , , ( , ). , ( ) output/x.png.


. s ( ). ReadData (data, context), lambda , ( (data, context)).


lambda , . , .


x Histogram, (edges), (mesh) -10 10 .


, , CSV (, ). ( pdflatex) , .


MakeFilename context["output"]. Context.output.filename — ( : csv, pdf ..). , x.


Writer . . , "output".


csv, LaTeX histogram_1d.tex , pdf png. , RenderLaTeX , .


: , . Lena, .




:


from lena.context import Context
from lena.flow import Cache, End, Print

s = Sequence(
    Print(),
    ReadData(),
    # Print(),
    ISlice(1000),
    lambda val: val[0][0], # data.x
    Histogram(mesh((-10, 10), 10)),
    Context(),
    Cache("x_hist.pkl"),
    # End(),
    HistToCSV(),
    # ...
)

Print , . , , Print . print .


ISlice, , , . , , , .


Context — , , , . Context , , ( , ). .


Cache . — , . , Cache , , . , . Cache pickle, Python ( ). (, , ), Cache. Cache, , .


End . , Cache ( End), HistToCSV . End , .




Lena , . , , . , .


(callable) . , , . , .


. — . , .


. Sequence , . Source Sequence, .


Sequence__call__(value) run(flow) ( )s.run(flow)
Source__call__() ( ), Sequences()

, , , . .



  1. End. :


    class End(object):
    """Stop sequence here."""

    def run(self, flow):
        """Exhaust all preceding flow and stop iteration."""
        for val in flow:
            pass
        raise StopIteration()

    main.py . ,


    Traceback (most recent call last):
File “main.py”, line 46, in <module>
main()
File “main.py”, line 42, in main
results = s.run([data_file])
File “lena/core/sequence.py”, line 70, in run
flow = elem.run(flow)
File “main.py”, line 24, in run
raise StopIteration()
StopIteration

    , , , . , StopIteration . ?


  2. , . , .


  3. Count , . , . ? , .


  4. , .


    " - ",- . " CSV, , , ,… , , code bloat ( )."


    ? ?


  5. ** Sum . , , .


    Sum , ? ? .



练习的答案在手册末尾给出。


脚注


1.将来可能会添加此功能。
2. Jinja文档
3. 这里和这里都建议使用Jinja进行LaTeX布局，模板的语法取自原始文章。


备择方案


Ruffus是用于科学和生物信息学的Python 计算管道。它通过写入和读取文件来连接程序组件。

More articles:


All Articles