Get best of the week

Cloud Games: Stress Test 5 Cloud Game Services With Bad Internet

image

About a year ago, I published the article “Cloud Games: First-Hand Assessment of the Services Capabilities for Playing on Weak PCs” . It analyzed the pros and cons of different services for cloud games on weak PCs. During the game, I tested each service and shared my overall impression as a result.

In the comments to those and other similar articles, readers often shared their impressions of different gaming services. Often there were opposing opinions about the same thing. Someone’s everything is perfect, but someone can’t play because of lags and friezes. Then I had the idea to evaluate the quality of work of these services in different conditions - from ideal to terrible. We are talking about the quality of networks, because far from always the user can boast a fast and hassle-free communication channel, right? In general, under the cut is the evaluation of services with a simulation of different network performance.

What is the problem at all?


As mentioned above - as a connection. Or rather - in the loss of packets during the game. The higher the loss, the more problems a gamer has, the less satisfied he is with the game. But rarely, anyone has an ideal communication channel, such as optical fiber to the device, and with dedicated, and not rummaged over all the inhabitants of the apartment building Internet.

For reference, at a connection speed of 25 Mbit / s, 40-50 data packets are needed to transfer 1 frame / frame. The more packets are lost, the less quality the picture becomes, and the more noticeable the lags and friezes. In severe cases, it becomes simply impossible to play.

Naturally, the cloud service itself can in no way affect the width and stability of the user's channel (although it would be great, of course). But then you can provide different ways to level communication problems. What services cope with the problem best of all - see below.

What exactly are we comparing?


Normal PC (Intel i3-8100, GTX 1060 6 GB, 8GB RAM), GeForce Now (its Russian version of GFN with servers in Moscow), Loudplay , Vortex , Playkey , Stadia . On all services except Stadia, we study the quality of the game in The Witcher. Google Stadia did not have this game at the time of writing, so I had to test another - Odyssey.

What are the conditions and methods of testing?


Testing from Moscow. Provider - MGTS, tariff of 500 Mbps, cable connection, not WiFi. We take the default graphics quality settings in services, the resolution is FullHD.

Using the Clumsy program, we simulate network problems, namely, packet loss of various types and sizes.

Uniform single losses. This is when only 1 packet is lost and losses are distributed more or less evenly. So, a uniform loss of 10% means that for every 100 packets every 10th is lost, but always only 1 packet. The problem usually manifests itself with distortions (shielding) on ​​the channel from the client to the server.

We test uniform losses of 5%, 10%, 25%.

Uneven Mass Losseswhen at one point 40-70 packets in a row are immediately lost. Such losses most often occur due to problems with network equipment (routers, etc.) at the user or at the provider. May be associated with a buffer overflow of network equipment on the user-server communication line. WiFi with thick walls can also cause such losses. Wireless network congestion due to the presence of a large number of devices is another reason that is very typical for offices and apartment buildings.

We test uneven losses of 0.01%, 0.1%, 0.5%.

Below I analyze all these cases and apply video comparison for clarity. And at the end of the article I give a link to raw, unmounted video gameplay from all services and cases - there you can look at artifacts and technical information in more detail (in all services except Stadia, data from the technical console are recorded; I didn’t find these from Stadia).

Go!


Below are 7 stress test scenarios and time-stamped videos (the video is the same, for convenience, viewing starts at the right moment in each paragraph). At the very end of the post are the source videos for each of the services. A good friend helped me make the video, thanks to him for that!

Scenario number 1. Ideal conditions. Zero network loss


Everything, as it should be in an ideal world. There are no communication problems, no gaps, no interference, your access point is the light of the Internet. In such greenhouse conditions, almost all test participants show themselves worthy.


PC

In each scenario, we took frames from a game on a PC as a reference. It is clear that the network quality does not affect it at all, the game runs on the PC locally. The presence of these frames answers the question “is there a difference when playing in the cloud compared to playing on your PC”. In ideal conditions, in our case, it is not felt in most services. Below we will not write anything about PCs, just remember that it is.

GeForce Now

Everything is fine, the picture is clear, the process is going smoothly, without friezes.

Vortex

Vortex spoils our ideal world. He immediately started having problems - the picture is worse than everyone else, plus the “brakes” are clearly visible. A possible problem is that the game servers are located far from Moscow, plus the hardware on the game servers seems to be weaker and FullHD does not export well. In all tests, Vortex performed poorly. If someone has a positive experience playing with Vortex - write in the comments, share where you played from and how well everything turned out.

Playkey

Everything is fine, just like on the local PC. Visible problems like friezes, lags, etc. no.

Loudplay

Service shows an excellent picture, there are no visible problems.

Stadia

The game service from Google works fine despite the fact that it does not have servers in the Russian Federation, and indeed Stadia does not officially work in Russia. However, all is well. It’s a pity, of course, that Stadia didn’t have a “Witcher” at the time of the game, but what to do, they took the Odyssey — also demanding, also about a man who cuts people and animals.

Scenario number 2. Uniform loss 5%


In this test, out of 100 packets, approximately every 20th is lost. Let me remind you that for rendering one frame you need 40-50 packets.


GeForce Now

Nvidia’s service is doing well, no problem. The picture is slightly more blurred than that of Playkey, but The Witcher is still playable.

Vortex Things

got worse here. Why - it is not entirely incomprehensible, most likely, redundancy is not provided for or it is minimal. Redundancy is error-correcting encoding of transmitted data (FEC - Forward Error Correction). This technology recovers data with partial loss due to network problems. You can implement and configure it in different ways, and judging by the results, the creators of Vortex did not succeed in this. Even with miserable uniform losses, it will not work. In subsequent tests, Vortex simply “died.”

Playkey

All is well, there is not much difference with ideal conditions. Perhaps it helps that the company's servers are located including in Moscow, from where the tests were conducted. Well, and perhaps the aforementioned redundancy is better tuned.

Loudplay

Service has sharply become unplayable, despite the relatively small packet loss. What can be wrong? I will assume that Loudplay works with the TCP protocol. In this case, while there is no confirmation of receipt of the package, other packages are not sent, the system awaits confirmation of delivery. Accordingly, if a package is lost, there will be no confirmation of its delivery, new packages will not be sent, the picture will stand up, the end of the story.

But if you use UDP, then you won’t need to confirm the receipt of the packet. As far as you can tell, all other services except Loudplay use the UDP protocol. If this is not the case, please correct me in the comments.

Stadia

Everything is playable. Sometimes a picture is pixelated, there are minimal response delays. Perhaps it imperfectly works out error-correcting coding, hence minor artifacts when playable in the whole stream.

Scenario number 3. Uniform loss 10%


We lose every 10th packet per hundred. This is a challenge for services. To effectively deal with such losses, you need technologists to recover and / or resend the lost data.


GeForce Now

Geforce has small drawbacks as a video stream. As far as one can judge, GFN responds to network problems by trying to reduce them. The service reduces the bitrate, that is, the number of bits for data transfer. So he is trying to reduce the load on a network of insufficient quality, in his opinion, and maintain a stable connection. And there really are no questions about stability, but here the quality of the video suffers significantly. We see significant pixelation of the picture. Well, since modeling assumes a constant loss of 10% of packets, a decrease in bitrate does not really help, the situation does not come back to normal.

In real life, the picture will most likely not be stably bad, but floating. Losses increased - the image was blurred; losses were reduced - the image returned to normal and so on. Gaming experience is not good, of course.

Playkey

No special problems. Probably, the algorithm detects problems on the network, determines the level of losses and focuses more on redundancy, rather than reducing bitrate. It turns out that with 10% uniform losses, the picture quality remains practically unchanged, the user is unlikely to notice such losses.

Loudplay

Inoperative, it just did not start. In the course of further tests, the situation repeated. As far as one can judge, this service does not adapt in any way to network problems. Perhaps the TCP protocol is to blame. The slightest loss paralyzes the service completely. Not very practical for real life, of course.

Vortex is

also a big problem. You can’t play in such conditions, although there is still a picture and the character continues to run, however, in jerks. I think it’s all about the same poorly implemented or absent redundancy. Packages are often lost and are not restored in any way. As a result, image quality degrades to an unplayable level.

Stadia

Unfortunately, everything is bad here. There is a flow break, due to which events on the screen occur jerkily, it is extremely difficult to play. It can be assumed that the problem appeared, as in the case of Vortex, due to minimal redundancy or its absence. I consulted with a couple of acquaintances who are “in the subject”, they said that Stadia, most likely, is waiting for the full assembly of the frame. Unlike GFN, it does not try to save the situation by a total decrease in bitrate. As a result, there are no artifacts, but friezes and lags appear (GFN, on the contrary, has fewer friezes / lags, but due to the low bitrate the picture is completely unattractive).

Other services, too, do not seem to wait for the complete assembly of the frame, replacing the missing part with a fragment of the old frame. This is a good solution, in most cases the user will not notice a catch (30+ frames are replaced per second), although artifacts can sometimes occur.

Scenario number 4. Uniform loss 25%


Every fourth packet is lost. It’s getting worse and worse . In general, with such a “leaky” connection, normal play in the cloud is hardly possible. Although some comparison participants are coping, albeit not perfectly.


GFN The

problems are already quite noticeable. The image is pixelated and blurred. You can still play, but this is not at all what GFN offered at the very beginning. And definitely not how to play beautiful games. Beauty is no longer appreciated.

Playkey

Gameplay is going well. There is smoothness, although the picture suffers a little. By the way, at the top left are the numbers of how many lost packets are being recovered. As you can see, 96% of packets are restored.

Loudplay

did not start.

Vortex

You can’t play even with a very strong desire, friezes (stopping the image, resuming the video stream from a new fragment) are even more noticeable.

Stadia

The service is almost unplayable. The reasons have already been mentioned above. Waiting for the assembly of the frame, redundancy is minimal, with such losses it is not enough.

Scenario number 5. Uneven losses of 0.01%.


For 10,000 packets, 40-70 packets in a row are lost once. That is, we lose about 1 out of 200 frames. It happens that the buffer of a network device is full and all new packets are simply discarded (dropped) until the buffer is freed. All participants in the comparison, except Loudplay, worked out such losses to one degree or another.


GFN The

picture has lost a little quality, it became a bit unclear, but everything is quite playable.

Playkey

Everything is very good. The picture is smooth, the image is good. You can play without problems.

Loudplay

The first few seconds the picture was, the hero even ran. But the connection to the server was almost immediately lost. Oh, that TCP protocol. The first loss cut the service down.

Vortex

Common problems observed. Friezes, lags and that’s all. It would be very difficult to play under such conditions.

Stadia

Playable. Small drawdowns are noticeable, the picture is sometimes pixelated.

Scenario number 6. Uneven loss 0.1%


For 10,000 packets, 40-70 packets in a row are lost 10 times. It turns out that we are losing 10 out of 200 frames.

I must say right away that noticeable problems appeared in most services. For example, a picture twitches, so redundancy does not help here. That is, there is a positive effect in the case of using redundancy technology, but it is small.

The fact is that the reaction time to the actions of the user and the game itself is limited, the video stream must be continuous. It is impossible to restore the flow to acceptable quality in spite of any efforts of the services.

Artifacts appear (an attempt to compensate for the loss of packets, there is not enough data) and jerks of the image.


GFN

The picture quality has fallen noticeably, the bitrate is clearly reduced, and very significantly.

Playkey

copes better - probably because redundancy is well tuned, plus the bitrate algorithm considers the losses not very high and does not turn the picture into a pixelated mess.

Loudplay

did not start.

Vortex

Launched, but with terrible picture quality. Jerks and drawdowns are very noticeable. It is hardly possible to play under such conditions.

Stadia Jerks are

clearly visible, this is a clear indicator that redundancy is not enough. The picture freezes, then other frames appear, there is a break in the video stream. You can play, in principle, if there is a great desire and a clinical tendency to self-torture.

Scenario number 7. Uneven loss of 0.5%


For 10,000 packets, 40-70 packets in a row are lost 50 times. We are losing 50 frames out of 200.

The situation is of the “uniform pi *** c” class. Your router sparks, the provider has an accident, the wires are bitten by mice, but you still want to play in the cloud. Which service should you choose?


GFN

It's already very difficult to play, if at all possible - the bitrate is very low. Frames are lost, instead of a normal picture we see "soap". Frames are not restored - there is not enough information to recover. If GFN has a recovery at all. The way the service aggressively tries to save the situation with bitrate raises doubts about its willingness to work with redundancy.

Playkey

There is a distortion of the frame, the image twitches, that is, the elements of individual frames are repeated. It is seen that most of the “broken” frame was restored from pieces of the previous one. That is, in the new frames there are parts of the old frames. But the image is more or less clear. You can control it, but in dynamic scenes, for example, in a fight, where you need a good reaction - it's difficult.

Loudplay

Did not start.

Vortex

started, but it would be better not to start - you can’t play it.

Stadia

Service in such conditions is unplayable. The reasons are the need to wait for the assembly of the frame and poor redundancy.

Who is the winner?


The rating, of course, is subjective. You can argue in the comments. Well, the first place, of course, is for the local PC. Due to the fact that cloud services are extremely sensitive to network quality, and the quality is rather unstable in the real world, your own gaming PC remains out of competition. But if for some reason it is not there, then see the rating.

  1. Local pc. Expected.
  2. Playkey
  3. GeForce Now
  4. Google stadia
  5. Vortex
  6. Loudplay

As a conclusion, let me remind you again that it plays a major role in cloud gaming in terms of resistance to problems on the network:
  • . UDP. , Loudplay TCP, . .
  • (FEC — Forward Error Correction, ). . , .
  • . , . — .
  • How post-processing is configured. If problems arise, the frames are either reset, or restored, or completed with fragments of old frames.
  • The proximity of servers to a gamer and the power of iron also significantly affect the quality of the game, but this is also true for an ideal network. If the ping to the servers is too high, even on an ideal network you won’t play comfortably. We did not experiment with ping in this study.

As promised, here is a link to raw videos from various services in all cases .

More articles:


All Articles