Hello again. In anticipation of the start of the course "Java Developer" prepared a translation of another small material.
Recently, one of the Plumbr APM users had a strange problem with an emergency stop of the docker-container with code 137. The configuration was simple with several nested containers and virtual machines, similar to a nested doll:- own iron server with Ubuntu;
- many docker containers with Ubuntu inside;
- Java Virtual Machine inside docker containers.
During the investigation of the problem, we found Docker documentation on this topic. It became clear that the reason was either a manual stop of the container, or a lack of memory and the subsequent intervention of oomkiller (Out-Of-Memory Killer).We look at syslog and see that, indeed, oomkiller was called :[138805.608851] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[138805.608887] [<ffffffff8116d20e>] oom_kill_process+0x24e/0x3b0
[138805.608916] Task in /docker/264b771811d88f4dbd3249a54261f224a69ebffed6c7f76c7aa3bc83b3aaaf71 killed as a result of limit of /docker/264b771811d88f4dbd3249a54261f224a69ebffed6c7f76c7aa3bc83b3aaaf71
[138805.608902] [<ffffffff8116da84>] pagefault_out_of_memory+0x14/0x90
[138805.608918] memory: usage 3140120kB, limit 3145728kB, failcnt 616038
[138805.608940] memory+swap: usage 6291456kB, limit 6291456kB, failcnt 2837
[138805.609043] Memory cgroup out of memory: Kill process 20611 (java) score 1068 or sacrifice child
As you can see, the java process reached a limit of 3145728 KB (this is about 3 GB), which caused the container to stop. This is rather strange, since docker itself was launched with a limit of 4 GB (in the file docker-compose).You probably know that the JVM also imposes its limitations on memory usage. Although docker itself was set to a 4 GB limit, the JVM was launched with the option Xmx=3GB. This can be even more confusing, but keep in mind that the JVM can use more memory than what -Xmx indicated (see the article on analyzing memory usage in the JVM ).Until we understand what is happening. Docker should allow the use of 4 GB. So why did OOMkiller work on 3 GB? A further search for information led us to the fact that there is another limitation of memory in the OS, which was deployed on hardware.Say thanks to cgroups (control groups). cgroups is the Linux kernel engine for restricting, controlling, and accounting for the use of resources by process groups. Compared to other solutions (team niceor /etc/security/limits.conf), cgroups offer more flexibility, since they can work with (under) sets of processes.In our situation, cgroups limited memory usage to 3 GB (via memory.limit_in_bytes). We have some progress!A study of GC memory and events using Plumbr showed that most of the time the JVM used around 700 MB. The exception was only immediately before the stop, when there was a surge in memory allocation. He was followed by a long pause of GC. So it seems the following happened:- Java code running inside the JVM is trying to get a lot of memory.
- The JVM, having verified that the Xmx limit of 3 GB is still far away, asks to allocate memory for the operating system.
- Docker also checks and sees that its 4 GB limit has not been reached either.
- The OS kernel checks the cgroup limit of 3 GB and kills the container.
- The JVM stops with the container before it can process its own
OutOfMemoryError.
Understanding this, we have configured all the limitations of 2.5 GB for docker and 1.5 for java. After that, the JVM could handle OutOfMemoryErrorand throw an OutOfMemoryError exception. This allowed Plumbr to do its magic - to get a memory snapshot with the corresponding stack dumps and show that there was one query to the database, which in a certain situation tried to load almost the entire database.findings
Even in such a simple situation, there were three memory limitations:- JVM via parameter
-Xmx - Docker through options in file
docker-compose - Operating system through parameter
memory.limit_in_bytes cgroups
Thus, when you encounter an OOM killer, you should pay attention to all the memory limitations involved.Another conclusion for docker developers. It seems that it makes no sense to allow such “nested dolls” to run in which the memory limit of the enclosed container is higher than the cgroups limit. A simple check of this when starting the container with the appropriate warning can save hundreds of hours of debugging for your users.That's all. We are waiting for you on the course .