On July 23, 2013, the source code for the Second Reality demo (1993) was published . Like many, I was eager to look at the insides of the demo that has inspired us so much over the years.I expected to see monolithic chaos from assembler, but instead of it, to my surprise, I discovered a complex architecture that gracefully combines several languages. I've never seen such a code before, perfectly representing two integral aspects of developing a demo:As usual, I formed an article for my notes: I hope this saves someone a few hours, and perhaps inspires others to read more source code and become more experienced engineers.Part 1: Introduction
Demo
Before embarking on the code, I’ll give a link to capture the legendary demo in HD video (Michael Hut). Today, this is the only way to fully evaluate the demo without graphical glitches (even DOSBox cannot launch it correctly).First contact with code
The source code is posted on GitHub. Just enter one command git:git clone git@github.com:mtuomi/SecondReality.git
At first, the content is confusing: 32 folders and a mysterious U2.EXEone that does not start in DosBox.The demo had the working title “Unreal 2” (the first “Unreal” was the previous Future Crew demo, released for the first Assembly in 1992). And only during the development process the name was changed to “Second Reality”. This explains the file name “U2.EXE”, but not why the file does not work ...If you run CLOC , we will get interesting metrics: -------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Assembly 99 3029 1947 33350
C++ 121 1977 915 24551
C/C++ Header 8 86 240 654
make 17 159 25 294
DOS Batch 71 3 1 253
-------------------------------------------------------------------------------
SUM: 316 5254 3128 59102
-------------------------------------------------------------------------------
- «» 50% .
- Doom.
- It has seventeen makefiles. Why not just one?
Launch demo
It's hard to figure it out, but the released demo can be launched in DosBox: you need to rename it U2.EXEand run it from the right place.When I learned about the inner workings of the code, it began to look very logical: CD MAIN
MOVE U2.EXE DATA / SECOND.EXE
CD DATA
SECOND.EXEAnd voila!Architecture
In the 90s, demos were mostly distributed on floppy disks. After unpacking, it was necessary to install two large files: SECOND.EXEand REALITY.FC: . <DIR> 08/08/2013 16:40
.. <DIR> 08/01/2013 16:40
FCINFO10 TXT 48,462 04-10-1993 11:48
FILE_ID DIZ 378 04-10-1993 11:30
README 1ST 4.222 04-10-1993 12:59
REALITY FC 992.188 07-10-1993 12:59
SECOND EXE 1,451,093 07-10-1993 13:35
5 Files (s) 2.496,343 Bytes.
2 Dir (s) 262,111,744 Bytes free.Based on my experience in game development, I always expect that the whole picture should look like this:SECOND.EXE: engine with all the effects in an executable file.REALITY.FC: Assets (music, sound effects, images) in a proprietary / encrypted format a la WADDoom.
But after reading it, MAIN/PACK.CI found that I was mistaken: the Second Reality engine is just a Loader and an Interrupt server (called DIS). Each scene (also called “PART”) demo is a fully functional DOS executable. Each part is loaded by the Loader loader and launched one after another. Parts are stored in encrypted form at the end SECOND.EXE:REALITY.FC contains two musical compositions played during the demo (for filling obfuscation added filling and marker at the beginning).SECOND.EXE contains bootloader and Demo Interrupt Server (DIS).- After the end
SECOND.EXE, 32 parts (PART) of the demo are added as executable DOS files (encrypted).
Such an architecture provides many advantages:- : PART ,
_start (450 ). - EXE
SECOND.EXE -. - : Loader DIS 20 . DOS .
- : PART PART .
- / : , PART ( ), : EXE , .
- Any language can be used for PART programming: in the code we find C, Assembly ... and Pascal.
Recommended Reading
The three pillars for understanding Second Reality source code are VGA, assembler, and PC architecture (PIC and PIT programming). Here are some incredibly useful links:Part 2: Second Reality Engine
As discussed in Part 1, the foundation of Second Reality consists of:- The bootloader as a DOS executable.
- Memory manager (simple stack pool)
- Demo Interrupt Server (DIS).
In this part I will give recommendations to programmers who want to read the engine and bootloader (DIS will be discussed in the next part).Engine code
The engine code is 100% ASM, but it is very well written and fairly well documented:In pseudo-code it can be written like this: exemus db 'STARTMUS.EXE',0
exe0 db 'START.EXE',0
...
exe23 db 'ENDSCRL.EXE',0
start:
cli ; Disable all interrupts
mov ah,4ah ; Deallocate all memory
call checkall ; Check for 570,000 bytes of mem, 386 CPU and VGA
call file_getexepath
call dis_setint ; Install Demo Interrupt Server on Interrupt 0fch
call file_initpacking ; Check exe signature (no tempering) !
call file_setint ; Replace DOS routines (only OPENFILE, SEEK and READ) on Interrupt 021h
call flushkbd ; Flush the keyboard buffer
call checkcmdline ; check/process commandline
;======== Here we go! ========
call vmode_init ; Init VGA (not necessarly Mode13h or ModeX), each PARTs had its own resolution
mov si,OFFSET exe0
call executehigh ; loaded to high in memory. Used for loading music loaders and stuff.
call _zinit ; Start music
call restartmus
mov si,OFFSET exe1 ;Parameter for partexecute: Offset to exec name
call partexecute
; Execute all parts until exe23
call fademusic
;======== And Done! (or fatal exit) ========
fatalexit:
mov cs:notextmode,0
call vmode_deinit
All steps are pretty easy to read:- Set DIS interrupt server as interrupt
0fch. - Replacing DOS system calls by interruption
021h(for more details, see the section "Dev and Prod Modes" ). - Download music to a sound card via EMS memory.
- Running music.
- Performing each part of the demo.
- Done!
Details of the procedures execute: execute:
cld
call openfile ; Open the DOS executable for this PART
call loadexe ; loads the specified exe file to memory, does relocations and creates psp
call closefile
call runexe ;runs the exe file loaded previously with loadexe.
; returns after exe executed, and frees the memory
; it uses.
Memory manager
There were many legends that Second Reality uses a complex memory manager via MMU; there were no traces in the engine. The memory management is actually transferred to DOS: the engine starts by freeing up all RAM and then distributing it on request . The only tricky trick is the ability to allocate RAM from the end of the heap: it is done using the malloc DOS return value when too much RAM is requested .Part 3: DIS
Demo Interrupt Server (DIS) provides a wide range of services for each PART: from data exchange between different PART to synchronization with VGA.DIS Services
At run-time, PART, the DIS server provides services to it. A list of features can be found at DIS/DIS.H.The most important services:- Exchange between different PART (
dis_msgarea): DIS provides three buffers of 64 bytes each so that PART can receive parameters from the loader of the previous PART. - Emulation Copper (
dis_setcopper): Amiga Copper simulator that allows you to perform operations that are switched by the state of VGA. - Dev / Prod (
dis_indemo) mode : allows the PART to know that it is running in DEV mode (which means it must initialize the video) or launched from the bootloader in PROD mode. - VGA Frame Count (
_dis_getmframe) - Waiting for VGA backstop (
dis_waitb).
Demo Interrupt Server Code
The DIS source code is also 100% ASM ... and quite well commented:How it works
DIS is set as an interrupt handler for programmatic int 0fch. The great thing about it is that it can run internally SECOND.EXEwhen the demo is running, or as a resident program ( TSR ) in Dev mode. This flexibility allows you to individually test different PART demos during development: // Let's pretend we are a FC developer and want to start the STAR part directly.
C: \> CD DDSTARS
C: \ DDSTARS> K
ERROR: DIS not loaded.
// Oops, the PART could not find the DIS at int 0fch.
C: \ DDSTARS> CD .. \ DIS
C: \ DIS> DIS
Demo Int Server (DIS) V1.0 Copyright (C) 1993 The Future Crew
BETA VERSION - Compiled: 07/26/93 03:15:53
Installed (int fc).
NOTE: This DIS server doesn't support copper or music synchronization!
// DIS is installed, let's try again.
C: \ DIS> CD ../DDSTARS
C: \ DDSTARS> KAnd voila!Copper
"Copper" is the coprocessor that the developers of the demo for Amiga loved. It was part of the Original Chip Set and allowed you to execute a programmable stream of commands synchronized with video equipment. There was no such coprocessor on the PC, and Future Crew had to write a Copper simulator running inside DIS.The FC team used the 8254-PIT and 8259-PIC PC hardware chipsets to simulate Copper. She created a system synchronized with the VGA frequency , capable of starting procedures in three places of the vertical backward beam :- Place 0: after turning on the display (approximately on scan line 25)
- Place 1: immediately after the reverse beam sweep (IT IS POSSIBLE TO AVOID IT POSSIBLE)
- Place 2: in the reverse beam of the scan beam
How this is done can be read in MAIN/COPPER.ASM(and see in the diagram below):- The timer of the 8254 chip is configured to trigger IRQ0 with the desired frequency.
- The 8h interrupt handler (which is called by the 8259 PIC after receiving IRQ0) is replaced with a procedure here
intti8.
Note: DIS frame count service is actually provided by copper simulator.Part 4: Dev and Prod Modes
Reading the source code of Second Reality, you are most struck by how much attention the team paid to the seamless switch from DEV to PROD.Development Mode
In Development mode, each component of the demo was a separate executable file.- DIS was loaded into the resident TSR and accessed through an interrupt
0cfh. - The bootloader caused a DOS interrupt
21hto open, read, search, and close files.
This DEV configuration has the following advantages:- Each coder and artist could work on the executable file and test it separately, without affecting the rest of the team.
- The full demo at any time could be tested using a small one
SECOND.EXE(without adding all the EXEs to the end). The executable of each PART was loaded using a DOS interrupt 021hfrom a separate file.
Production (demo mode)
In Production mode, the small SECOND.EXE(containing the bootloader), DIS, and parts of the demo as separate EXEs were combined into one thick one SECOND.EXE.- Access to DIS was still done via interrupt
0fch. - The DOS 21h interrupt API was patched by its own Future Crew routines, which open files from the end of a large file
SECOND.EXE.
This PROD configuration has an advantage in terms of load time and protection against reverse engineering ... but most importantly, from the point of view of programming or loading PART, NOTHING changes when switching from DEV to PROD.Part 5: Separate PART
Each of the Second Reality visual effects is a fully functional DOS executable. They are called PART and all of them 23. Such an architectural solution allowed for rapid prototyping, parallel development (since FC most likely did not have version control tools) and free choice of languages ​​(ASM, C, and even Pascal are found in the source).Separate PART
A list of all PART / EXEs can be found in the engine source code: U2.ASM . Here is a more convenient short description of all 23 parts (with the location of the source code, although the names can be very confusing):It seems that each developer had his own specialization, which could be shared in one part. This is especially noticeable in the first scene with scrolling, ships and explosions (Alkutekstit). Although this looks like one continuous effect, in fact it is three executable files written by three different people:Assets
Image Assets ( .LBM) are generated using Deluxe Paint , an extremely popular bitmap editor in the 90s. Interestingly, they are converted to an array of bytes and compiled inside PART. As a result of this, the exe file also downloads all assets. In addition, this complicates reverse engineering.Among the cool asset sets are the famous CITY and SHIP from the latest 3D scene:Internal unit PART
Since they were all compiled into DOS executables, in PART, any language could be used:As for memory usage, I read a lot about MMU on Wikipedia and other websites ... but in fact, each part could use anything, because after execution it was completely unloaded from memory.When working with VGA, each part used its own set of tricks and worked in its resolution. In all of them, not Mode 13h and not ModeX were used, but rather a modified mode 13h mode with its own resolution. The SCRIPT file often mentions 320x200 and 320x400.Unfortunately, when analyzing PART, reading the source code becomes a daunting task: the quality of the code and comments drops dramatically. Perhaps this happened because of the rush or because each PART worked on its own developer (that is, there was no "real" need for comments or code comprehensibility), but the result was something completely confusing:Sophisticated algorithms are not only difficult to understand even the names of variables ( a, b, co[]...). The code would be much more readable if the developers left us hints in the release notes. As a result, I did not devote much time to studying each part; The exception was the 3D engine responsible for U2A.EXE and U2E.EXE.3D engine Second Reality
Anyway, I decided to study in detail the 3D engine, which was used in two parts: U2A.EXEand U2E.EXE.The source code is C with assembler-optimized procedures (especially Gouro fill and shade):- CITY.C (main code).
- VISU.C (library
visu.lib). - AVID.ASM (optimized assembler video (cleaning, screen copying, etc.)).
- ADRAW.ASM (drawing objects and truncating).
- ACALC.ASM (matrices and fast sin / cos calculations).
The architecture of these components is quite remarkable: the library VISUperforms all complex tasks, for example, loading assets: 3DS objects, materials and flows (camera and ship movements).The engine sorts the objects that need to be drawn, and renders them using the artist’s algorithm. This leads to a large amount of redrawing, but since VGA latches allow you to record 4 pixels at the same time, it's not so bad.An interesting fact: the engine performs transformations in an “old-school” way: instead of using common homogeneous 4x4 matrices, it uses 3 * 3 rotation matrices and a displacement vector.Here is a summary in pseudo code: main(){
scenem=readfile(tmpname);
scene0=readfile(tmpname);
for(f=-1,c=1;c<d;c++){
sprintf(tmpname,"%s.%03i",scene,e);
co[c].o=vis_loadobject(tmpname);
}
vid_init(1);
vid_setpal(cp);
for(;;){
vid_switch();
_asm mov bx,1 _asm int 0fch
vid_clear();
for(;;){}
vid_cameraangle(fov);
for(a=1;ac<conum;a++) if(co[a].on)
calc_applyrmatrix(o->r,&cam);
for(a=0;ac<ordernum;a++)
for(b=a-1;b>=0 && dis>co[order[b]].dist;b--)
for(a=0;ac<ordernum;a++)
vis_drawobject(o);
}
}
return(0);
}
Ports for modern systems
After the release of this article, many developers began to port Second Reality to modern systems. Claudio Matsuoka set about creating sr-port , a C port for Linux and OpenGL ES 2.0, which so far looks pretty impressive. Nick Kovacs did a great job on PART PLZ, porting it to C (now it is part of the sr-port source code), as well as to javascript :