Get best of the week

How to learn development in Python: a new video course of Yandex

Last fall, the first Backend Development School was held at Yandex's Moscow office. We shot the lessons on video and today we are pleased to share the full video course of the School on Habré. It will allow you to learn industrial development in Python. The authors of the lectures are experienced developers at Yandex. Each video has links to examples and useful materials.

To learn the course, you need to know the basics of Python and understand how applications are deployed on servers. We expect that you can make queries to databases and know how web applications are created, at least at the initial level.

 1. CPython device
 2. Object-oriented programming
 3. Testing
 4. Databases
 5. Databases: models, migrations, testing
 6. Architecture
 7. Infrastructure
 8. Algorithms
 9. Debug, logging, profiling
 10. Asynchronous programming. Lecture one
 11. Asynchronous programming. Lecture two
 12. Asynchronous programming. Third lecture

1. CPython device - Egor Ovcharenko


Let's talk about why and what we write in Python. We will discuss the structure of the interpreter and dictionaries, memory management and typing. You will learn how generators work and how exceptions work.


Presentation: yadi.sk/i/dcNx5Sgix4axOA

References


The interpreter as a whole:
docs.python.org/3/reference/executionmodel.html
github.com/python/cpython
leanpub.com/insidethepythonvirtualmachine/read

Memory Management:
arctrix.com/nas/python/gc
rushter.com/blog/python -memory-managment
instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172
stackify.com/python-garbage-collection

Exceptions:
bugs.python.org/issue17611

2. Object-oriented programming - Valery Lisay


Let's talk about OOP and its implementation in Python. Consider topics and concepts such as decorators, descriptors, and metaclasses.


Presentation: yadi.sk/i/f-UpCHCsnxqf9Q

References


habr.com/ru/post/141411
docs.python.org/3/howto/descriptor.html
habr.com/ru/post/145835
ibm.com/developerworks/ru/library/l-pymeta
docs.python.org/ 2.5 / ref / slots.html

3. Testing - Maria Zelenova


Let's talk about what software testing is, what tests are and why write them. Let's talk about libraries for testing Python code: unittest, pytest, doctest. We will find out how they differ, and look at simple test examples. We will touch on the topic of continuous integration: what is it, how is it used in development.


Presentation: yadi.sk/i/g6nd4aORJyrPMQ

References


docs.python.org/3/library/unittest.html
docs.pytest.org/en/latest
docs.python.org/3/library/doctest.html

4. Databases - Tatyana Denisova


Let's talk about what data is, what databases are and how they differ. You will learn what features of working with the database the developer needs to keep in mind. We will discuss how to characterize, structure and store data based on the current features of the system and its future scaling.

Thanks to the knowledge gained at the lecture, the developer will be able to understand which of the topics mentioned needs to be delved into to solve a specific problem, and in case of bugs, to determine whether working with the database is the source of the problem. And if so, which way to dig.


Presentation: yadi.sk/i/Uf5U_xwt5qGBIQ

5. Databases: models, migrations, testing - Alexander Vasin


This is a lecture on practical work with relational databases using PostgreSQL as an example. Let's talk about how to choose an RDBMS, what the infrastructure looks like in production, compare the synchronous and asynchronous PostgreSQL drivers. We will find out how the database driver is arranged. We will discuss the effective work with data, named and unnamed cursors, the use of transactions, RETURNING and UPSERT.

You will learn how to make a fault-tolerant and scalable application, what Query Builder, ORM are and when to use them (using SQLAlchemy as an example), how to write database migrations (using Alembic as an example), and why and how to test them.


Presentation: yadi.sk/i/DqYmAbrPu6en2g
Examples: github.com/alvassin/alembic-quickstart

6. Architecture - Oleg Ermakov


Let's consider three parts of designing new system functionality:

- API for client-server interaction;
- code-level design patterns (justification for the need for decomposition with reference to Martin Fowler);
- architecture of interservice interaction.


Presentation: yadi.sk/d/PivB-ZJ0UJGjYQ

References


HTTP protocol specification:
tools.ietf.org/html/rfc2616
tools.ietf.org/html/rfc5789

15 trivial facts about working with the HTTP protocol:
habr.com/ru/company/yandex/blog/265569

Trainee Vasya and his story about idempotency API:
habr.com/en/company/yandex/blog/442762

Patterns of Enterprise Application Architecture: martinfowler.com/eaaCatalog/index.html
Microservices Patterns: manning.com/books/microservices-patterns
Design Patterns: litres.ru / elizabet-robson / head-first-patterny-proektirovaniya-39123671
Domain-Driven Design: Tackling Complexity in the Heart of Software: amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215
Optimizing the Netflix API: medium.com/netflix-techblog/optimizing-the-netflix-api-5c9ac715cf19

7. Infrastructure - Dmitry Orlov


Let's discuss the infrastructure around the Python program: Python Package Index repository (PyPI), virtualenv tool, deployment process, Ansible configuration management system, virtualization and containerization.


Presentation: yadi.sk/i/9eOXdelTpXkoEQ

References


Python Package: docs.python.org/3.8/distutils/setupscript.html
virtualenv: docs.python.org/3.8/library/venv.html
Software Deployment: en.wikipedia.org/wiki/Software_deployment
Ansible: docs.ansible.com /ansible/latest/user_guide/quickstart.html
Virtualization: en.wikipedia.org/wiki/Hardware_virtualization
Containerization: en.wikipedia.org/wiki/OS-level_virtualization

8. Algorithms - Ilya Volkov


Let's talk about data structures, computational and amortized complexity.

Using an example of a simple list of tasks and its evolution with gradual improvements, we consider approaches to the tasks of writing the server side of web applications. We will review approaches and common techniques for optimizing tasks and code. In the final part of the lecture, we will consider the process of passing technical interviews in IT companies.


Presentation: yadi.sk/i/hBbOjd4SqMlw5g

Links


Implementation of lists in Python: docs.python.org/3.7/faq/design.html#how-are-lists-implemented-in-cpython
Analysis of implementation of lists in Python on Habr: habr.com/en/post/273045
Analysis of algorithm B -Tree on Habré: habr.com/en/post/114154
Analysis of the LRU algorithm: habr.com/en/post/136758
Work of LRU cache in Redis: redis.io/topics/lru-cache The
list of Redis commands with computational difficulties - it is possible Guess what happens under the hood: redis.io/commands
Galactic algorithms: en.wikipedia.org/wiki/Galactic_algorithm
Example of an interview task from a lecture: leetcode.com/problems/trapping-rain-water

Introduction to complexity theory on Habré:habr.com/en/post/196560 A
digest of services for programming practice: tproger.ru/digest/competitive-programming-practice
An example of a service with a more or less simple set of tasks for exploring new languages: exercism.io

9. Debug, logging, profiling - Yuri Shikanov


At the beginning of the lecture you will find an introduction to basic knowledge of operating systems (in particular, Linux). Then we will consider memory management, processes, multitasking, IPC, files, system calls. We’ll show you how to work with the Pdb debugger built into Python. Let's talk about logging in general and about the logging library in Python. We learn how to deal with insufficient program performance and how to find a bottleneck in which the most processor or memory resources are spent.


Presentation: yadi.sk/d/BCky8YkLcbVeUA
Linux System Programming book: oreilly.com/library/view/linux-system-programming/9781449341527

10. Asynchronous programming. Lecture One - Edward Beetle


Consider what the problem of synchronous applications is and what can be done about it.

We will discuss what happens from the OS point of view during an HTTP request, how to process several requests at the same time, and what are the advantages and disadvantages of processes and threads in web servers. You will learn about the features of threads in Python. Let's discuss non-blocking I / O: how to handle multiple requests in a single thread.

Let's talk about what event-loop is and why it is needed. We will touch on the choice between a synchronous and asynchronous solution.


Presentation: yadi.sk/i/OhqXMEOKzNlK6g

References


A Web Crawler With asyncio Coroutines: aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html
David Beazley: Generators: The Final Frontier: youtube.com/watch?v=D1twn9kLmYg

11. Asynchronous programming. Lecture Two - Alexander Vasin


This lecture is a short overview tour of asyncio: the library, interface, and standard for asynchronous programming in Python.

We will look at the high-level asyncio interface (coroutine, task, future and methods for working with them), the low-level interface (event loop, policies), as well as the asynchronous Python interfaces (context managers, iterators, generators, comprehensions). We will tell you why aiohttp is needed, how to write web applications on it. You will learn what middleware is, how aiohttp allows you to serialize data, and how to perform asynchronous tasks in the background.


Presentation: disk.yandex.ru/i/4gyVoP1DM9enxQ

References


Your own the async the Build (by David Beazley): youtu.be/Y4Gt3Xjd7G8
Asyncio today and tomorrow (Yuri Selivanov): youtu.be/3rSAtD2gKQE
the PEP 492: coroutines with the async and the await syntax: python.org/dev/peps/pep-0492
the PEP 530: Asynchronous Comprehensions: python.org/dev/peps/pep-0530

12. Asynchronous programming. Lecture Three - Dmitry Orlov


When developing asynchronous applications with Python and asyncio, sometimes there are problems associated with managing background tasks, blocking stdout, retry policy, and blocking operations. You will learn how the Edadil service team solves these problems with aiomisc, the basic concepts of this open source project, the base classes, ready-made services, decorators, work with streams and much more.


Presentation: yadi.sk/i/SAJnWVQNfdHV0w aiomisc
Documentation: pypi.org/project/aiomisc

More articles:


All Articles