Tale of how to make a time machine for a database and accidentally write an exploit

Good day, Habr.

Have you ever wondered how to change the time inside the database? Easy? Well, in some cases, yes, it’s easy - the linux command is date and the point is in the hat. And if you need to change the time only inside one instance of the database if there are several of them on the server? And for a single database process? AND? Uh, that's it, my friend, that's the whole point.Someone will say that this is another sur, not related to reality, which is periodically laid out on Habré. But no, the task is quite real and is dictated by production necessity - code testing. Although I agree, the test case can be quite exotic - check how the code behaves for a certain date in the future. In this article, I will examine in detail how this task was solved, and at the same time capture a little the process of organizing test and dev stands for the Oracle base. Ahead of a long reading, get comfortable and ask for a cat.

Background


Let's start with a short introduction to show why this is necessary. As already announced, we write tests when implementing edits in the database. The system under which these tests are done was developed at the beginning (or maybe a little before the start) of the zero ones, so all business logic is inside the database and written in the form of stored procedures in the pl / sql language. And yes, it brings us pain and suffering. But this is legacy, and you have to live with it. In the code and the tabular model, it is possible to specify how the parameters inside the system evolve over time, in other words, set the activity from which date and to what date they can be applied. What to go far - the recent change in the VAT rate is a vivid example of this. And so that such changes in the system can be checked in advance,a database with such changes needs to be transferred to a certain date in the future, the code parameters in the tables will become active at the "current moment". And due to the specifics of the supported system, you cannot use mock tests that would simply change the return value of the current system date in the language when the test session starts.

So, we determined why, then we need to determine how the goal is achieved. To do this, I will make a small retrospective of the options for building test benches for developers and how each test session started.

Stone Age


Once upon a time, when the trees were small, and the mainframes were large, there was only 1 server for development and it was also conducting tests. And in principle, all this was enough for everyone ( 640K is enough for everyone! )

Cons: for the task of changing the time, it was necessary to involve many related departments - system administrators (doing the time change on the subd server from root), DBMS administrators (doing the database restart), programmers ( it was necessary to notify that a time change would occur, because part of the code stopped working, for example, web tokens previously issued for calling api methods ceased to be valid and this could come as a surprise), testers (testing itself) ... When you return the time to the present everything was repeated in reverse order.

Middle Ages


Over time, the number of developers in the department grew and at some point 1 server ceased to be enough. Mainly due to the fact that different developers want to change the same pl / sql package and conduct testing for it (even without changing the time). More and more indignation was heard: “How long! Enough tolerating this! Factories to workers, land to peasants! Every programmer has a database! ” However, if you have a few terabytes of product database, and 50-100 developers, then honestly in this form, the requirement is not very real. And still everyone wants the test and dev base not to lag very much behind the sales, both in structure and in the data inside the tables. So there was a separate server for testing, let's call it pre-production. It was built from 2 identical servers,where the sale was made to restore the database from RMAN bucks and it took about 2-2.5 days. After recovery, the database made anonymization of personal and other important data and the load from the test applications was applied to this server (as well as the programmers themselves always worked with the recently restored server). The work with the required server was ensured using the cluster ip-resource supported through corosync (pacemaker). While everyone is working with the active server, on the 2nd node, the database recovery starts again and after 2-3 days they again change places.The work with the required server was ensured using the cluster ip-resource supported through corosync (pacemaker). While everyone is working with the active server, on the 2nd node, the database recovery starts again and after 2-3 days they again change places.The work with the required server was ensured using the cluster ip-resource supported through corosync (pacemaker). While everyone is working with the active server, on the 2nd node, the database recovery starts again and after 2-3 days they again change places.

Of the obvious disadvantages: you need 2 servers and 2 times more resources (mainly disk) than prod.

Pros: time change operation and testing - it can be performed on the 2nd server, on the main server at this time developers live and go about their business. Server change occurs only when the database is ready, and downtime of the test environment is minimal.

The era of scientific and technological progress


When we switched to the 11g Release 2 database, we read about an interesting technology that Oracle provides under the name CloneDB. The bottom line is that the product database back-ups (there is a directly bit copy of the product data files) are stored on a special server, which then publishes this set of data files via DNFS (direct NFS) to basically any number of servers, and you don’t need to have one on the server the same volume of disks, because the Copy-On-Write approach is implemented: the database uses a network share with data files from the back-up server for reading data in tables, and changes are written to local data files on the dev server itself. Periodically, “zeroing the deadlines” is done for the server so that the local data files do not grow very much and the place does not end. When updating the server, data is also depersonalized in the tables,in this case, all table updates fall into local data files and those tables are read from the local server, all other tables are read over the network.

Cons: there are still 2 servers (to ensure smooth updates with minimal downtime for consumers), but now the volume of disks is greatly reduced. To store bucks on an nfs ball, you need 1 more server in size + - as a prod, but the update execution time itself is reduced (especially when using incremental bucks). Networking with an nfs ball noticeably slows down IO read operations. To use the CloneDB technology, the base must be an Enterprise Edition; in our case, we had to carry out the upgrade procedure on test bases each time. Fortunately, test databases are exempted from Oracle licensing policies.

Pros: the operation to restore a base from a bakup takes less than 1 day (I don’t remember the exact time).

Change of time: no major changes. Although by this time scripts had already been made to change the time on the server and restart the database in order to do this without attracting the attention of the orderlies of the administrators.

Era of New History


In order to save disk space even more and make data reading offline, we decided to implement our CloneDB version (with flashback and snapshots) using a file system with compression. During the preliminary tests, the choice fell on ZFS, although there is no official support for it in the Linux kernel (quote from the article) For comparison, we also looked at BTRFS (b-tree fs), which Oracle is promoting, but the compression ratio was less with the same CPU and RAM consumption in the tests. To enable ZFS support on RHEL5, its own kernel based on UEK (unbreakable enterprise kernel) was built, and on newer axes and kernels you can simply use the ready-made UEK kernel. The implementation of such a test base is also based on the COW mechanism, but at the level of file system snapshots. 2 disk devices are supplied to the server, on one, the zfs pool is made, where through RMAN an additional standby database is made from the sale, and since we use compression, the partition takes up less than production.
The system is installed on the second disk device and the rest is necessary for the server and the database itself to work, for example, partitions for undo and temp. At any time, you can make a snapshot from the zfs pool, which then opens as a separate database. Creating a snapshot takes a couple of seconds. It's magic! And such databases can be tilted in principle quite a lot, if only the server had enough RAM for all instances and the zfs pool size itself (for storing changes in data files during depersonalization and during the life cycle of the database clone). The main time for updating the test base is the operation of data depersonalization, but it also fits in 15-20 minutes. There is significant acceleration.

Cons: on the server you can’t change the time simply by translating the system time, because then all database instances running on this server will fall into this time at once. A solution to this problem has been found and will be described in the appropriate section. Looking ahead, I’ll say that it allows you to change the time inside only 1 instance of the database ( per instance time change approach) without affecting the rest on the same server. And the time on the server itself does not change either. This eliminates the need for a root script to change the time on the server. Also at this stage, time change automation for instances via Jenkins CI is implemented and users (relatively speaking development teams) who own their booth are given rights to the jobs through which they themselves can change the time, update the booth to the current state with sales, make snapshots and restoration (rollback) of the base to the previously created snapshot.

Era of Recent History


With the advent of Oracle 12c, a new technology appeared - pluggable databases and, as a result, container databases (cdb). With this technology, within one physical instance, several “virtual” databases can be made that share a common memory area of ​​the instance. Pros: you can save memory for the server (and increase the overall performance of our database, because all the memory that was occupied before, for example, 5 different instances, can be shared for all deployed pdb containers inside cdb, and they will only use it when they really need it, and not as it was in the previous phase, when each instance “blocked” the memory allocated to it for itself and when the activity of one of the clones was low, the memory was not used effectively, in other words, it was idle).The data files of different pdb still lie in the zfs pool, and when deploying clones, they use the same zfs snapshot mech. At this stage, we came close enough to the ability to give almost every developer their own database. Changing the time at this stage does not require a restart of the database and works very accurately only for those processes that need a time change; all other users working with this database are not affected in any way.

Minus: you cannot use the per instance time change approach from the previous phase, because we have one instance now. However, a solution for this case was found. And it was precisely this that served as the impetus for writing this article. Looking ahead, I will say that it is a time change per process approach i.e. in each database process, you can set your own unique time in general.

In this case, a typical testing session immediately after connecting to the database sets the right time at the beginning of its work, conducts tests and returns the time back at the end. Returning the time is necessary for one simple reason: some Oracle database processes do not end when the database client disconnects from the server, these are server processes called shared servers, which, unlike dedicated processes, run when the database server starts and live almost indefinitely (in ideal picture of the world). If you leave the time changed in such a server process, then another connection that will be served in this process will receive the wrong time.

In our system, shared servers are used a lot, because up to 11g there was practically no adequate solution for our system to withstand high load (in 11g DRCP appeared - database resident connection pooling). And here's why - in sub there is a limit on the total number of server processes that it can create in both dedicated and shared mode. Dedicated processes are generated more slowly than the database can issue an already ready shared process from the pool of shared processes, which means that when new connections are constantly arriving (especially if the process does some other slow operations), the total number of processes will increase. When the limit of sessions / processes is reached, the database ceases to service new connections and collapse occurs.The transition to the use of a pool of shared processes allowed us to reduce the number of new processes on the server when connecting.

That’s where the review of the technologies for building test databases is completed, and we can finally begin to implement the time-change algorithms for the database itself.

The fake per instance approach


How to change the time inside the database?

The first thing that came to mind was to create in a scheme that contains all the business logic code, its own function, which overlaps the language functions that work with time (sysdate, current_date, etc.) and, under certain conditions, begins to give other values, for example, you could set values ​​through the session context at the beginning of the test run. It didn’t work out, the built-in language functions did not overlap with the user ones.

Then, light virtualization systems (Vserver, OpenVZ) and containerization via docker were tested. It doesn’t work either, they use the same kernel as the host system, which means they use the same system timer values. Falling out again.

And here I am not afraid to come to the rescue of this word, a brilliant invention of the Linux world - redefinition / interception of functions at the stage of dynamic loading of shared objects. It is known to many as tricks with LD_PRELOAD. In the environment variable LD_PRELOAD, you can specify the library that will be loaded before all the others that the process needs, and if this library contains characters with the same name as for example in the standard libc, which will be loaded later, then the symbol import table for the application will look like a function provides our replacement module. And that’s exactly what the libfaketime project library doeswhich we began to use in order to start the database at a different time separately from the system one. The library misses calls that concern working with the system timer and getting the system time and date. To control how much time moves relative to the current server date or from what point in time the time should go inside the process - everything is controlled by environment variables that must be set together with LD_PRELOAD. To implement the time change, we implemented a job on the Jenkins server, which enters the database server and restarts the DBMS either with or without environment variables set for libfaketime.

An example algorithm for starting a database with a substitution time:

export LD_PRELOAD=/usr/local/lib/faketime/libfaketime.so
export FAKETIME="+1d"
export FAKETIME_NO_CACHE=1

$ORACLE_HOME/bin/sqlplus @/home/oracle/scripts/restart_db.sql

And if you think up that everything worked right away, then you are deeply mistaken. Because, as it turned out, validates those libraries that are loaded into the process when the DBMS starts. And in the alertlog, he begins to resent the noticed forgery, while the base does not start. Now I don’t remember exactly how to get rid of it, there is some parameter that can disable the execution of sanity-checks at startup.

The fake per process approach


The general idea of ​​changing the time only within 1 process remained the same - use libfaketime. We start the database with a library preloaded into it, but set a zero time offset at startup, which is then propagated to all DBMS processes. And then, inside the test session, set the environment variable for this process only. Pff, business something.

However, for those who are familiar with the pl / sql language, the whole doom of this idea is immediately clear. Because the language is very limited and basically suitable for high-level tasks. No system programming can be implemented there. Although some low-level operations (for example, working with a network, working with files) are present in the form of pre-installed system dbms / utl packages. For the entire time I worked with Oracle, I did reverse engineering of pre-installed packages several times, the code of some of them is hidden from the eyes of strangers (they are called wrapped). If you are forbidden to watch something, then the temptation to find out how it is arranged inside only increases. But often, even after the anvrapper, there is not always something to see, because the functions of such packages are implemented as c interface to so-libraries on disk.
In total, we approached one candidate for implementation - technology with external procedures .
The library designed in a special way can export methods, which then the Oracle database can call via pl / sql. Seems promising. Only once I met this in Advanced plsql courses, so I remembered very remotely how to cook it. And it means it is necessary to read documentation. I read it - and immediately became depressed. Because the loading of such a custom so-library goes in a separate agent process through a database listener, and communication with this agent goes through dlink. So our idea cried to set an environment variable inside the database process itself. And this is all done for security reasons.

A picture from the documentation that shows how it works:



The type of the so / dll library is not so important, but for some reason the picture is only for Windows.

Perhaps someone noticed here another 1 potential opportunity. Yes, yes, this is Java. Oracle allows you to write stored procedure code not only in plsql, but also in java, which nevertheless are exported the same way as plsql methods. Periodically, I did this, so there should not be a problem with this. But then another pitfall was hidden. Java works with a copy of the environment, and allows you to only get the environment variables that the JVM process had at startup. The built-in JVM inherits the environment variables of the database process, but that’s all. I saw tips on the Internet how to change the readonly map through reflection, but what's the point, because it's still just a copy. That is, the woman was again left with nothing.

However, Java is not only valuable fur. Using it, you can spawn processes from within a database process. Although all unsafe operations must be resolved separately through the java grants mechanism, which are done using the dbms_java package. From inside the plsql code, you can get the process pid of the current server process in which the code is running, using the system views v $ session and v $ process. Further we can spawn some child process from our session to do something with this pid. To get started, I simply deduced all the environment variables that are inside the database process (to test the hypothesis)

#!/bin/sh

pid=$1

awk 'BEGIN {RS="\0"; ORS="\n"} $0' "/proc/$pid/environ"

Well deduced, and then what. It’s still impossible to change the variables in the environ file, this is the data that was transferred to the process when it started and they are read only.

I searched the Internet on stackoverflow "How to change an environment variable in another process." Most of the answers were that it was impossible, but there was one answer that described this opportunity as a substandard and dirty hack. And that answer was Albert Einstein gdb. The debugger can hook onto any process knowing its pid and execute any function / procedure in it that exists in it as a publicly exported symbol, for example, from some library. In libc, there are functions for working with environment variables, and libc is loaded into any process of the Oracle database (and practically any program on linux).

This is how the environment variable is set in a foreign process (you need to call it from root because of the used ptrace):

#!/bin/sh

pid=$1
env_name=$2
env_val="$3"

out=`gdb -q -batch -ex "attach $pid" -ex 'call (int) setenv("'$env_name'", "'"$env_val"'", 1)' -ex "detach" 2>&1`


Also, to see the environment variables inside the gdb process is also suitable. As mentioned earlier, the environ file from / proc / pid / shows only the variables that existed at the start of the process. And if the process created something in the course of its work, then this can only be seen through the debugger:
#!/bin/sh

pid=$1
var_name=$2

var_value=`gdb -q -batch -ex "attach $pid" -ex 'call (char*) getenv("'$var_name'")' -ex 'detach' | egrep '^\$1 ='`

if [ "$var_value" == '$1 = 0x0' ]
then
  # variable empty or does not exist
  echo -n
else
  # gdb returns $1 = hex_value "string value"
  var_hex=`echo "$var_value" | awk '{print $3}'`
  var_value=`echo "$var_value" | sed -r -e 's/^\$1 = '$var_hex' //;s/^"//;s/"$//'`
  
  echo -n "$var_value"
fi


So, the solution is already in our pocket - through java we spawn the debugger process, which goes to the process that generated it and sets the desired environment variable for it and then ends (the Moor has done his job - the Moor can leave). But there was a feeling that it was some kind of crutch. I wanted something more elegant. It would be somehow all the same to force the database process itself to set environment variables without external assault.

An egg in a duck, a duck in a hare ...


And then someone comes to the rescue, yes, you guessed it right, again Java, namely JNI (java native interface). JNI allows you to call native C methods inside the JVM. The code is issued in a special way in the form of a shared object of the library, which the JVM then loads, while the methods that were in the library map to the java methods inside the class declared with the native modifier.

Well, ok, we are writing a class (in fact, this is only a workpiece):

public class Posix {

    private static native int setenv(String key, String value, boolean overwrite);

    private static native String getenv(String key);
    
    public static void stub() 
    {
        
    }
}

After that, compile it and get the generated h-file of the future library:

#  
javac Posix.java

#   Posix.h        JNI
javah Posix

Having received the header file, we write the body for each method:

#include <stdlib.h>
#include "Posix.h"

JNIEXPORT jint JNICALL Java_Posix_setenv(JNIEnv *env, jclass cls, jstring key, jstring value, jboolean overwrite)
{
    char* k = (char *) (*env)->GetStringUTFChars(env, key, NULL);
    char* v = (char *) (*env)->GetStringUTFChars(env, value, NULL);

    int err = setenv(k, v, overwrite);

    (*env)->ReleaseStringUTFChars(env, key, k);
    (*env)->ReleaseStringUTFChars(env, value, v);

    return err;
}

JNIEXPORT jstring JNICALL Java_Posix_getenv(JNIEnv *env, jclass cls, jstring key)
{
    char* k = (char *) (*env)->GetStringUTFChars(env, key, NULL);
    char* v = getenv(k);

    return (*env)->NewStringUTF(env, v);
}

and compile the library

gcc -I"$JAVA_HOME/include" -I"$JAVA_HOME/include/linux" -fPIC Posix.c -shared -o libPosix.so -Wl,-soname -Wl,--no-whole-archive

strip libPosix.so

In order for Java to load the native library, it must be found by the system ld according to all Linux rules. Additionally, Java has a set of properties that contain the paths where the library searches take place. The easiest way to work inside Oracle is to put our library in $ ORACLE_HOME / lib.

And after we have created the library, we need to compile the class inside the database and publish it as a plsql package. There are 2 options for creating Java classes inside the database:

  • load binary class-file via loadjava utility
  • compile class code from source using sqlplus

We will use the second method, although they are basically equal. For the first case, it was necessary to immediately write all the class code in stage 1, when we received a stub class for the h-file.

To create a class in subd, a special syntax is used:

CREATE OR REPLACE AND RESOLVE JAVA SOURCE NAMED "Posix" AS
...
...
/

When the class is created, it needs to be published as plsql methods, and here again the special syntax:

procedure set_env(var_name varchar2, var_value varchar2)
is
language java name 'Posix.set_env(java.lang.String, java.lang.String)';

When you try to call potentially unsafe methods inside Java, an execution is raised that says that no java grant has been issued for the user. Loading native methods is another unsafe operation, because we inject extraneous code directly into the database process (the same exploit that was announced in the header).

But since the database is test, we give a grant without any concern connecting from sys:

begin
dbms_java.grant_permission( 'SYSTEM', 'SYS:java.lang.RuntimePermission', 'loadLibrary.Posix', '');
commit;
end;
/

The system username is the one where I compiled the java code and plsql wrapper package.
It is important to note that when loading a library through a call to System.loadLibrary, we omit the lib prefix and the so extension (as described in the documentation) and do not pass any path where to look. There is a similar System.load method that can only load a library using an absolute path.

And then 2 unpleasant surprise awaits us - I landed in the next rabbit hole of Oracle. When issuing a grant, an error occurs with a rather foggy message:

ORA-29532: Java call terminated by uncaught Java exception: java.lang.SecurityException: policy table update

The problem is googled on the Internet and leads to My Oracle Support (aka Metalink). Because According to the Oracle rules, publishing articles from a metalink is not allowed in open sources, I’ll only mention the document number - 259471.1 (those who have access will be able to read for themselves).

The essence of the problem is that Oracle will not let us just allow the loading of suspicious third-party code into our process. Which is logical.

But since the base is test and we are confident in our code, we allow the download without special fears.
Fuh, misadventures are all over.

It's alive, alive


With bated breath, I decided to try to breathe life into my Frankenstein.
We start the database with the preloaded libfaketime and 0 offset.
Connect to the database and make a call to the code that simply displays the time before and after changing the environment variable:

begin
dbms_output.enable(100000);
dbms_java.set_output(100000);
dbms_output.put_line('old time: '||to_char(sysdate, 'dd.mm.yyyy hh24:mi:ss'));
system.posix.set_env('FAKETIME','+1d');
dbms_output.put_line('new time: '||to_char(sysdate, 'dd.mm.yyyy hh24:mi:ss'));
end;
/


It works, damn it! Honestly, I was expecting some more surprises, such as ORA-600 errors. However, the alert had the whole number and the code continued to work.
It is important to note that if the connection to the database is done as dedicated, then after the connection is completed, the process will be destroyed and there will be no trace. But if we use shared connections, then in this case a ready-made process is allocated from the server pool, we change the time in it through environment variables and when it is disconnected it will remain changed inside the process. And when then another database session falls into the same server process, it will receive the wrong time to its considerable surprise. Therefore, at the end of the test session, it is better to always return the time back to zero offset.

Conclusion


I hope the story was interesting (and maybe even useful to someone).

Source codes are all available on Github .

The libfaketime documentation too .

How do you do testing? And how do you create dev and test databases in a company?

Bonus for those who read to the end


All Articles