🤙🏽 💿 🤟🏻 Specify it. Yandex Report 🥢 🤸🏻 👩🏿‍🍳

A good API specification helps customers use it. A few months ago at Big Pytup, Yandex developer Alexander Bryazginbryazginnnmade a presentation on what the REST API specification is based on as an example of OpenAPI + Swagger and why such a bundle is needed. From the synopsis, you can find out how we screwed up the automatic generation of the specification in the ready-made service, which libraries were useful to us and what kind of tuning is around the OpenAPI specification.

- Hello everyone, my name is Alexander. I would like to talk to you about the specifications.

Specifically, I would like to touch on several topics. First of all, what are specifications using the example of REST API specifications. Further - as we fastened generation of the specification in our service. And in the end, on the sweet side - what tools, tuning, to use the results of our specification generation, using an example of such a specification as OpenAPI and Swagger UI.

REST API Specification

What are specifications and what do I mean by this in my report?

First of all, it is Interface Description Language, a language for describing interfaces, some description that corresponds to a specific format. It can be parsed automatically, because it has a strict format and such specifics that it does not depend on the language itself in which the interface is implemented. That is, let's say you have a Python web server, for the Interface Description Language it does not matter.

There are a lot of languages for specifying specifications. These are specifications for REST, RPC, GraphQL, etc. Today we will dwell on the REST specification.

Openapi

Let's talk about specifications using the OpenAPI example.

First, let's understand what OpenAPI is. This is just the same language to describe the specifications. By itself, it is a file - in the format of YAML, JSON or XML, whoever likes it more - with a description of the interface, handles-endpoints and schemes.

Swagger

When we talk about OpenAPI, we can not say about Swagger.

Swagger is essentially an OpenAPI UI that allows you to beautifully visualize your specification.

In Swagger, as in OpenAPI, there is a description of the endpoints that are in your specification.

There is a description of authorization in your API.

There is a description of the models or circuits with which you manipulate in your API - for example, accept it for input or return it for output.

There is a description of examples and expected requests and responses from your API.

And additionally in your Swagger, due to the fact that it is a UI and it is running in your browser, you can in real time execute requests to your API right here and now.

^{_{Link from the slide}}

With all this, you can play around on a test bench prepared by the Swagger team. This is petstore.swagger.io. There is an example specification, raised by Swagger. You can go to the real backend and touch. Recommend.

So, we say: OpenAPI, Swagger, specification. But why all this is needed for a person who writes, say, a backend?

First of all, specifications are often used during the design phase of our API. That is, normal people, having come to the task of writing a backend, start by describing what they want to implement. That is, they begin to design their API, a list of endpoints that they want to implement, a list of schemes to be manipulated with, parameters, etc. In this case, the specifications and tools around the specifications are very convenient.

In addition, there is a popular case. For the sake of what we dragged the specification to us in service? This is the documentation. The availability of specifications gives you such a good opportunity to describe your API and share this information with someone: either with the front-end providers in your team, or with external clients who want to integrate with you.

Also, due to the fact that the specification is written in a certain strict format, it can be automatically processed, that is, engaged in introspection of the API. And on top you can catch a lot of tools that can make your life easier. I’ll tell you more specifically and in more detail later.

How we prepare OpenAPI

We decided why we needed all this. Let's talk about how we screwed up the generation of specifications in our service. First, a little context.

When we came to the conclusion that we were going to cut the specification, we didn’t follow the normal path when we first wrote the project, in the sense of a prototype with a specification. We already raised some backend, it was already flying with us, and we wanted to share the specification as a way of documenting.

And our backend was suddenly in Python. We used Falcon as a framework, marshmallow as a serialization and deserialization tool, and webargs as a validation tool.

Using the simplest clipping from petstore that I mentioned earlier, we’ll imagine how our service might look at the stage when we wanted to generate a specification.

In our service, there would be a simplified pair of schemes: the type of our pet, that is, the category, and, in fact, the animals themselves.

That is, we would describe several schemes on Marshmallow. Further, we would prepare a few pens in order, for example, to register and receive our favorites.

And, in fact, they would describe the application itself, which would raise the endpoint for creating and receiving pets.

This is how it would look together. Fairly simple service.

Let's think about what options we have for creating an OpenAPI config. The simplest and most naive option, as I said earlier, is just a file. Why don’t we make this file with our hands? We know what input data we have, we have endpoints. It seems obvious, just take and write.

But first of all, we are programmers. We do not like to do anything with our hands. We want something standard to be generated for us. Secondly, if we supported everything by hand, we would certainly have to support API changes, as well as a file that lies somewhere else. Like any service, our product is volatile, and so is the specification.

Therefore, we have come to the conclusion that we want to generate an OpenAPI config. Given our technology stack, we found a library like apispec . Apispec is a library from manufacturers, that is, the authors of Marshmallow and Webargs. Together they make up a large ecosystem.

Let's see how you can use apispec on our stack to teach it how to generate a specification for us.

We had a resource and a specific handler for the get method of this resource. He has got some kind of body.

To teach apispec, to let him know what we are doing in our pen, we add something to the docstring. We add a description there that is understandable to man. In our case, this is an example of answers: sometimes it happens, our pen answers with two hundred, and in the body of our answer there is JSON with a scheme, Pet data model.

Using the example of a post-request, we describe: in addition to the fact that there is a response 201 in which we return Pet, there is also the expected body of the request that comes to us. And inside this request, we expect JSON, and expect that this JSON will be in Pet format.

We added docstrings, then we need to learn how to generate a specification. To do this, we write some kind of script, a red button, and teach apispec to generate a specification for us. We declare an apispec object, from which we describe some metadata of our service, and then just write the file, the generated specification in the YAML format.

Let's see what was generated for this huge application in this file.

First of all, in this file there will be a description of the version in which the specification was generated. This is important so that everyone who interprets this specification understands which format to interpret. Specifications vary from version to version.

Further there is a description of our service. This title, version, perhaps there will be some description, if you want it, and then a list of servers that you can go to, pull the pens, see and touch our backend.

In addition, the data schemes themselves that we manipulate sprout there, with a description of the types, with examples of filling in this data and some rules. Perhaps these are some extreme values, perhaps the binding parameters. Here you see a list of required attributes.

In addition to the schemes, the description of endpoints themselves grows there. In this case, the description of each method sprouts.

Using the get method, we get a description of what we do with the get method.

And exactly the description that we put in the docstrings of the corresponding endpoint sprouted on the post. That is, we have already received some kind of file.

What shall we do with it now? What is the use? First of all, you, as the owners of the backend, can already give all this potential customers who will go to your pens, give this specification. And this will help the guys integrate. That is, the file that we generated does not make you gouge out your eyes. It is quite readable. It can be seen through the eyes and somehow parsed.

But this is also difficult. Let's see what other tools are there to simplify your life and what kind of tuning is around the generated specification.

Buns

And then we come to the next stage, where we will talk about what is a useful toolkit around specifications using the OpenAPI example.

^{_{Link from the slide}}

The first tool I would like to talk about is Swagger-UI. Hereinafter, I will provide links to either a web resource where you can poke the corresponding tool, in this case SwaggerHub, or to the command to launch the corresponding document using Docker.

We all love Docker a lot. Docker helps you not to put JavaScript or any Java on the local. Just take it, run it with one command, and everything works for you.

So, Swagger-UI is, in fact, a utility that allows you to beautifully display your specification. As I said earlier, these are running locally using Docker, possibly Swagger.

You can view or share on the locale by running the specification in Swagger on your hardware. And further send all your potential users to this specification. She is already much more understandable, beautiful. Directly in it, you can do query execution.

^{_{Link from the slide}}

In addition to Swagger-UI, there is a convenient Swagger-editor tool. You can use it on the web, you can pick it up on a locale or on any machine. It allows you to change the specification in real time and see its display right here. That is, it is a very convenient tool at the stage of, say, designing an API, when you do not have a specification generated, and you want to somehow prank or describe your backend without having a real or generated backend behind.

Next up is the wonderful Prism tool. I would like to dwell on it in more detail. Prism is a utility from Spotlight. It allows you, with the specification, to raise the mock-server, which will work according to the specification that you fed him.

That is, in this case, we see the launch of Prism on top of the specification, which I stole just from under the Swagger store. We see that Prism found all the endpoints that are specified in the specification, and said that they honestly bother them.

Let's try to feel what Prism can do.

First of all, we find the pen, try to pull it and suddenly we see that Prism tells us: sorry, 401 error. We go into the specification and see that in fact this pen is hidden behind authorization. The security section is clearly described there. In it, we see that the authorization method is OAuth, and we understand: in fact, Prism expected some kind of authorization data from us.

Let's try to transfer authorization data to him according to the format that he is waiting for. It is clear that the token is somehow airy, for Prism it does not matter what it is in essence. The format is important to him. That is, if he is waiting for OAuth, he is waiting for authorization data in a specific format.

We jerk with Authorization and see the 400th error already. We look at what we have in the specification. In the specification, we see that we did not pass some required attributes. Let's pass them on.

The magic attribute is status. It is, in fact, given according to the ENUM specification. If we transferred a status that is not included in this ENUM, we would have got a four-dot.

Here we passed the valid status and we see that the backend responded with a completely valid xml with the data. The data in this xml is now only returned from the example specified in the specification. That is, in the specification for each field there is such a section as example, where you can specify an example of data that can be returned. Here Prism just brought them back.

But besides the fact that you can pull the pen and expect that there will be an honest two-hundred, there is an opportunity to say Prism - return me a specific response code. Usually in the specification we describe the different behavior of our pen, and with valid data they do not always correspond to two hundred. They can look, for example, for the presence of IDs in the database, to see that such an ID does not exist.

For this case, we declare a certain code, say, the 400th or 422nd, to anyone as you like. That is, with a valid entry, there is not always a valid exit. Therefore, in the specification, we describe various answer options. And we can clearly say when jerking the pen Prism: this time I'm waiting for you to answer me, say, the 404th error. Let's say this is a case when I pull the handle with an ID, but there is no such ID.

To summarize, what are the features in general, in addition to those that I have already mentioned? Again, this is a mock server that you can pick up. He will respond according to the specification, validate authorization requests. It will also validate on the data that you sent to it.

In addition to validating the request, it will generate responses. In the simplest case, as I said, it generates answers by example. There is a second option - when you ask for it explicitly: please generate dynamic answers. A dynamic strategy means that, depending on the type, say, int, it will generate thousands, billions, or something else. Or for the string, a strange and incomprehensible hash.

Also, when I said on the first run that you can generate data, we asked ourselves: it would be cool if Prism integrated with faker. Those who used the Python faker probably know how cool it is when you want to lock some data or emulate data in the database.

So, Prism is written in JavaScript. JavaScript also has Faker.js, and Prism has automatic integration with faker. You can explicitly specify in your specification the types of faker data that your pen will give. That is, OpenAPI supports a plug-in system that allows you to expand your specification so that OpenAPI and the parser itself do not care. But if there are plugins that parse your specification, they can use some additional fields.

So, Prism offers you to use the optional X-faker plugin field. Very comfortably.

Here, under the asterisk, I indicated that callbacks can be described in OpenAPI. This is the interaction scheme when you register a callback to a certain pen and wait for that after that you will receive a specific callback to the url you registered. So, Prism and it knows how to get wet.

From interesting: Prism can be raised not only in mock mode, but also in Proxy mode. You just put this Proxy in front of your real backend, and all requests that go through Prism and come back, Prism will validate to specification.

If some data - say, input or output, request or response - will diverge, he will write this in the log. He does this quite transparently, it does not affect the real behavior. In general, the documentation says that it can be easily raised in production. Honestly, I have not tried it myself. Surely there is some kind of overhead, but I still can’t say about it. You can feel, try and tell.

Further, through what I have already said, it is possible to force certain requests. That is, come to the mock server and say: I want a specific example or type of response. We already talked about the type of answer. Also in the OpenAPI specification it is possible to specify several answer options and explicitly name them.

So, you can come and say Prism: this time I want a specific example, next time I want another example.

^{_{Link from the slide}}

About Prism talked. Now about Postman. I love him so much. So, Postman has out of the box integration with OpenAPI. You can literally import the OpenAPI specification with just two clicks of buttons. And he will create a collection of requests according to your specification.

At the same time, he will pre-prepare all query parameters, all body parameters, all environment and authorization. You just have to finish the data with some real IDs or something else, authorization tokens. Very comfortably.

We pass from Postman further. We talked about Prism, it has functionality - query validation. In fact, there is a separate utility that allows you to mercilessly suck your really running backend and see if it really works according to the specification.

Dredd parses the specification. He takes, pulls all the pens and looks at what has returned to him. In general, dredd can be treated as an infrastructure for writing tests, because all its requests can be supplemented with its hooks. That is, from the box, you can start dredd just as is, as I launched it here.

Or you can run dredd by passing it a set of hooks that actually work in the ideology of a regular test. There is something that rises to the whole set of tests, there are hooks that run before the test, after the test, the whole infrastructure.

^{_{Link from the slide}}

Next I want to talk in more detail about such a tool from Swagger itself as Swagger-codegen. In fact, it is a bit like a Swiss knife. To begin with, what kind of thoughts did the guys have when they wrote this instrument.

Usually, when someone needs to integrate with a backend, you rarely describe an http request right here, at a certain stage, in business logic. Most often you encapsulate work with a specific backend in some entity, in the client.

So, the guys came up with the idea that, having a specification, we can quite clearly describe and predict what the client would look like for this backend. And the guys made a customer generator.

Let's try to start client generation according to the petstore specification. We will see that this generator was generated by the Swagger client itself, in which there is the Python client itself. Here in the launch line you can clearly see that I'm generating a client in Python. In fact, it supports a huge number of languages. What is important here? Front-runners will like JavaScript, hipsters will like Go. Everything you want.

So, in addition to the client in Python, he generated a couple of docs and test magic folders. We will talk about them later in more detail. And a lot of sugar so that you can wrap this client in a ready-made library. There is gitignore, there are CI files, let's say for Travis, there is sugar for git. There are requirements, setup.py and tox.ini. That is, everything that helps you simply take, commit and send this client is already in the form of a library that can be reused.

Let's take a closer look at what he generated in the client.

In the client, he generated, in addition to some auxiliary files, two very important directories api and models. In models, we have those data models that we manipulate. Each model is simply a Python class that is inherited from object with __init__, which takes all kinds of parameters for this scheme, and getters and setters that change the state of the object.

In addition to the model, entities such as PetApi also appeared there - these are just client wrappers over the endpoints group, which OpenAPI groups either by tags or by endpoints.

And this client has all the pens that you can twitch, transmitting real data. Then they will fly away to some backend.

What is implemented from this generated client? From the interesting - the contractual approach. Let's talk more about him.

The contractual approach to programming suggests that when you implement the interaction between the client and the server, you actually always lay down on certain rules, on contracts, whether it is data that the client gives to the server, or data that the server gives to the client.

So, the contract approach recommends you to check your data and your interaction for compliance with the contract every time in real time. If this contract is a scheme of the data that we transmit, and does not meet our expectations, is not satisfied, then you need to say this, show the error right here and now. Do not go to the real backend, do not wait for any four-dot, but say right now.

Or another case: we went to the backend, received some data, but they do not correspond to what we were waiting for. Suppose some fields that we expected to see filled come empty. We’ll talk about it here and now, and not when we leave somewhere on the call stack somewhere in the forest and don’t find out why something has fallen off.

So, Swagger-codegen generates clients and models according to the contract approach. It allows you to say in real time: yes, I want the client in runtime to check all the data for compliance with the contracts. And if the contract is not satisfied, he will immediately say so.

We generated some kind of pit-client with a contractual approach, but besides that, Swagger-codegen generated documentation stubs for us. In fact, they are quite simple, but they can all be twisted for each model.

Also Swagger-codegen generated some kind of test stubs. Everything so that you can roll out the generated code with minimal effort and run it with tests, with Continuous Integration, in the form of a library with a git, with all the bells and whistles.

Let us briefly go through it again. We looked at just one case - a way to generate clients for the API. From the important} contractual approach. I'd like to say: when I generated a client for a real petstore according to their specifications and really went to their backend, then suddenly this contractual approach caught the invalid data that was returned by petstore.

At first I thought - they are a little strange. They themselves generated the specification, and the backend does not work correctly for them. But it seems to me that they did it on purpose so that we could see the power of the contract approach. There was not much data, and they were clearly generated.

Also in customer generation and in another generation, you can use your own templates. That is, if you like to generate a client in a certain format, you can save and prepare your template and generate clients the way you want. If you go around and generate integrations every day, you might like it a lot.

You can also configure these clients and write the import of your models, rather than those generated by Swagger-codegen.

In addition to generating clients for the API, you can feed the specification to the generator and generate the stubs of the server itself. It is very strange. That is, you generate some kind of backend that supposedly works according to the specification. It is clear that there is no business logic there. But perhaps it will be convenient as some kind of scaffolding, that is, preparing a draft with which you can already do something. I have not used it, but I recommend to take a look - maybe you will find something useful for yourself.

Among other things, it allows you to generate documentation in HTML and Confluence format if someone is using it. The examples I wanted to show for OpenAPI also ended there.

^{_{All of these tools are available at the two links below on the slide: openapi.tools andgithub.com/APIs-guru/awesome-openapi3}}

I want to show the tuning groups around the OpenAPI specification. In fact, there are a lot of tools. These are just groups, that is, types of tools that you can use.

Of the important here is a converter from one specification to another. That is, you can generate the Blueprint API or whatever you like from the OpenAPI specification, and vice versa. There is a library for mock, for documentation. That is, everything that I talked about will be reflected in these groups.

There is a link that you can follow too. Be sure to go look.

^{_{Link from the slide}}

Besides the tools around OpenAPI, there are tools around Swagger. There are SwaggerHub and Swagger Inspector. They are highlighted in blue because it is a web-based toolkit. You can go directly there and as a service use the SwaggerHub and Swagger Inspector, which are actually a superposition of the following libraries: Swagger-editor, Swagger-codegen, Swagger-UI. All that we just discussed.

All libraries are yellow; they are open source, as we saw. There is on GitHub, is in the form of Docker. Use.

Conclusion

What did we discuss today? REST API specification using OpenAPI and Swagger as examples. I want to emphasize that OpenAPI is just one way to describe the specification. A lot of them. From interesting look still Blueprint API. Maybe someone else will like something.

We also watched Swagger. In essence, this, as we have already said, is just a beautiful UI around the specification, which allows you to look and explore it in a convenient way. In fact, these tools are also many, ranging from the popular ReDoc and Sphinx, and ending, in fact, Markdown. That is, you can generate Markdown from OpenAPI, and it will be displayed beautifully in all GitHub, GitLab - wherever you want.

We also looked at an example on our technology stack how we now generate the specification. But as you noticed, this is quite complicated and inconvenient, because we, in fact, duplicate business logic and docstrings.

^{_{Links from the slide: FastAPI , SAFRS , rororo}}

From interesting, I advise you to look at FastAPI. Styopa will tell you more about this today . FastAPI helps you generate a specification out of the box. You there can not think of anything at all. Just write the code and get the finished specification.

There are two more frameworks: SAFRS and rororo. The names, of course, are magical. They work a little on a different model, do not make you not think about the specification and get it just as a side effect. On the contrary, you have a specification that drives handlers. This is, roughly speaking, specification driven development. They do not have as many stars as FastAPI, but it seems to me that they deserve attention, take a look.

And finally, we discussed what kind of utility there are around the specification using the examples of OpenAPI and Swagger. For all the points I make links, look for sure, there are a lot of interesting things:

- OpenAPI / Swagger
swagger.io/docs/specification/about

- Example on falcon + marshmallow + apispec
github.com/marshmallow-code/apispec
github.com/tiangolo/fastapi

- OpenAPI / Swagger
buns
openapi.tools swagger.io/tools swagger.io/tools/open-source/open-source-integrations github.com/APIs-guru/awesome-openapi3

What do I have was the message of the report? Guys, writing specifications is not some kind of bureaucracy. And this is not as difficult as it might seem at first glance, but rather simple and provides many useful tools, opens up great opportunities for you.

Write the specifications. Recommend. Thank you so much.

All code and links are available in the presentation .