A study of one vague behavior

Introduction

• indefinite behavior may not seem dangerous enough due to the abstractness of the possible consequences;
• it’s not always clear where the line is.

Read Standard

6.6.3 The return statement
...
Flowing off the end of a function is equivalent to a return with no value; this results in undefined
behavior in a value-returning function.
...

Reaching the end of a function is equivalent to return without a return value; for a function whose return value is provided, this leads to undefined behavior.

9.6.3 The return statement
...
Flowing off the end of a constructor, a destructor, or a function with a cv void return type is equivalent to a return with no operand. Otherwise, flowing off the end of a function other than main (6.8.3.1) results in undefined behavior.
...

Reaching the end of a constructor, destructor, or function with a void return value (possibly with const and volatile qualifiers) is equivalent to return without a return value. For all other functions, this leads to undefined behavior (except for the main function).

• its execution should end with a return statement with an instance of the appropriate type;
• otherwise, vague behavior;
• undefined behavior does not start from the moment the function is called and not from the moment the returned value is used, but from the moment the function is not completed properly;
• if the function contains both correct and incorrect execution paths - undefined behavior will occur only on incorrect paths;
• the undefined behavior in question does not affect the execution of instructions contained in the body of the function.

Example 1 - bool

#include <iostream>

bool bad() {};

int main()
{
    std::cout << bad();

    return 0;
}

#include <iostream>
#include <stdlib.h>

bool bad()
{
    if (rand() == 0) {
        return true;
    }
}

int main()
{
    std::cout << bad();

    return 0;
}

Linux x86-x64 Clang 10.0.0, -O0

UD2—Undefined Instruction
Generates an invalid opcode exception. This instruction is provided for software testing to explicitly generate an invalid opcode exception. The opcode for this instruction is reserved for this purpose.
Other than raising the invalid opcode exception, this instruction has no effect on processor state or memory.

Even though it is the execution of the UD2 instruction that causes the invalid opcode exception, the instruction pointer saved by delivery of the exception references the UD2 instruction (and not the following instruction).

This instruction’s operation is the same in non-64-bit modes and 64-bit mode.

Linux x86-x64 Clang 10.0.0, -O1, -O2

Linux x86-x64 gcc 9.3, -O0

Linux x86-x64 gcc 9.3, -O1, -O2, -O3

macOs X Apple clang version 11.0.0, -O0

macOs X Apple clang version 11.0.0, -O1, -O2

Windows MSVC 2019 16.5.4, Advanced Example, / Od

Windows MSVC 2019 16.5.4 Complicated Example, / O1, / O2

• copies one byte from ebx from memory located at [rsp + 30h];
• if rand () returned zero, copy the unit from ecx to ebx (line 11);
• copies the same value to dl (more precisely, its least significant byte) (line 13);
• calls the output function in stream, which outputs the dl value (line 14).

The conclusion of example 1

Example 1a - Managing Undefined Behavior

Linux x86-x64 gcc 9.3, -O0

#include <iostream>

bool bad() {}

bool goodTrue()
{
    return rand();
}

bool goodFalse()
{
    return !goodTrue();
}

unsigned char goodChar(unsigned char ch)
{
    return ch;
}

int main()
{
    goodTrue();
    std::cout << bad() << std::endl;

    goodChar(85);
    std::cout << bad() << std::endl;

    goodFalse();
    std::cout << bad() << std::endl;

    goodChar(240);
    std::cout << bad() << std::endl;

    return 0;
}

1
85
0
240

Windows MSVC 2019 16.5.4, / Od

#include <iostream>
#include <stdlib.h>

void control(unsigned char value)
{
    uint32_t count = 0;
    srand(0);
    while ((rand() & 0xff) != value) {
        ++count;
    }

    srand(0);
    for (uint32_t i = 0; i < count; ++i) {
        rand();
    }
}

bool bad()
{
    if (rand() == 0) {
        return true;
    }
}

int main()
{
    control(1);
    std::cout << bad() << std::endl;

    control(85);
    std::cout << bad() << std::endl;

    control(0);
    std::cout << bad() << std::endl;

    control(240);
    std::cout << bad() << std::endl;

    return 0;
}

1
85
0
240

Windows MSVC 2019 16.5.4, / O1, / O2

#include <iostream>
#include <stdlib.h>

bool bad()
{
  if (rand() == 0) {
    return true;
  }
}

int main()
{
  volatile unsigned char ch = 1;
  std::cout << bad() << std::endl;

  ch = 85;
  std::cout << bad() << std::endl;

  ch = 0;
  std::cout << bad() << std::endl;

  ch = 240;
  std::cout << bad() << std::endl;

  return 0;
}

1
85
0
240

macOs X Apple clang version 11.0.0, -O0

#include <iostream>

bool bad() {}

void putToStack(uint8_t value)
{
    uint8_t memory[1]{value};
}

int main()
{
    putToStack(20);
    std::cout << bad() << std::endl;

    putToStack(55);
    std::cout << bad() << std::endl;

    putToStack(0xfe);
    std::cout << bad() << std::endl;

    putToStack(11);
    std::cout << bad() << std::endl;

    return 0;
}

0
1
0
1

The conclusion of example 1a

Example 1b - broken bool

Windows MSVC 2019 16.5.4, / Od

#include <iostream>
#include <stdlib.h>
#include <set>
#include <unordered_set>

bool bad()
{
    if (rand() == 0) {
        return true;
    }
}

int main()
{
    bool badBool1 = bad();
    bool badBool2 = bad();

    std::cout << "badBool1: " << badBool1 << std::endl;
    std::cout << "badBool2: " << badBool2 << std::endl;

    if (badBool1) {
      std::cout << "if (badBool1): true" << std::endl;
    } else {
      std::cout << "if (badBool1): false" << std::endl;
    }
    if (!badBool1) {
      std::cout << "if (!badBool1): true" << std::endl;
    } else {
      std::cout << "if (!badBool1): false" << std::endl;
    }

    std::cout << "(badBool1 == true || badBool1 == false || badBool1 == badBool2): "
              << std::boolalpha << (badBool1 == true || badBool1 == false || badBool1 == badBool2)
              << std::endl;
    std::cout << "std::set<bool>{badBool1, badBool2, true, false}.size(): "
              << std::set<bool>{badBool1, badBool2, true, false}.size()
              << std::endl;
    std::cout << "std::unordered_set<bool>{badBool1, badBool2, true, false}.size(): "
              << std::unordered_set<bool>{badBool1, badBool2, true, false}.size()
              << std::endl;

    return 0;
}

badBool1: 41
badBool2: 35
if (badBool1): true
if (! badBool1): false
(badBool1 == true || badBool1 == false || badBool1 == badBool2): false
std :: set <bool> {badBool1, badBool2 , true, false} .size (): 4
std :: unordered_set <bool> {badBool1, badBool2, true, false} .size (): 4

• comparison operators for boolean values;
• hash function of boolean value.

Windows MSVC 2019 16.5.4, / O1, / O2

#include <iostream>
#include <stdlib.h>
#include <set>
#include <unordered_set>

bool bad()
{
  if (rand() == 0) {
    return true;
  }
}

int main()
{
  volatile unsigned char ch = 213;
  bool badBool1 = bad();
  ch = 137;
  bool badBool2 = bad();

  std::cout << "badBool1: " << badBool1 << std::endl;
  std::cout << "badBool2: " << badBool2 << std::endl;

  if (badBool1) {
    std::cout << "if (badBool1): true" << std::endl;
  }
  else {
    std::cout << "if (badBool1): false" << std::endl;
  }
  if (!badBool1) {
    std::cout << "if (!badBool1): true" << std::endl;
  }
  else {
    std::cout << "if (!badBool1): false" << std::endl;
  }

  std::cout << "(badBool1 == true || badBool1 == false || badBool1 == badBool2): "
    << std::boolalpha << (badBool1 == true || badBool1 == false || badBool1 == badBool2)
    << std::endl;
  std::cout << "std::set<bool>{badBool1, badBool2, true, false}.size(): "
    << std::set<bool>{badBool1, badBool2, true, false}.size()
    << std::endl;
  std::cout << "std::unordered_set<bool>{badBool1, badBool2, true, false}.size(): "
    << std::unordered_set<bool>{badBool1, badBool2, true, false}.size()
    << std::endl;

  return 0;
}

badBool1: 213
badBool2: 137
if (badBool1): true
if (! badBool1): false
(badBool1 == true || badBool1 == false || badBool1 == badBool2): false
std :: set <bool> {badBool1, badBool2 , true, false} .size (): 4
std :: unordered_set <bool> {badBool1, badBool2, true, false} .size (): 4

Linux x86-x64 gcc 9.3, -O0

#include <iostream>
#include <stdlib.h>
#include <set>
#include <unordered_set>

bool bad()
{
}

unsigned char goodChar(unsigned char ch)
{
  return ch;
}

int main()
{
  goodChar(213);
  bool badBool1 = bad();

  goodChar(137);
  bool badBool2 = bad();

  std::cout << "badBool1: " << badBool1 << std::endl;
  std::cout << "badBool2: " << badBool2 << std::endl;

  if (badBool1) {
    std::cout << "if (badBool1): true" << std::endl;
  }
  else {
    std::cout << "if (badBool1): false" << std::endl;
  }
  if (!badBool1) {
    std::cout << "if (!badBool1): true" << std::endl;
  }
  else {
    std::cout << "if (!badBool1): false" << std::endl;
  }

  std::cout << "(badBool1 == true || badBool1 == false || badBool1 == badBool2): "
    << std::boolalpha << (badBool1 == true || badBool1 == false || badBool1 == badBool2)
    << std::endl;
  std::cout << "std::set<bool>{badBool1, badBool2, true, false}.size(): "
    << std::set<bool>{badBool1, badBool2, true, false}.size()
    << std::endl;
  std::cout << "std::unordered_set<bool>{badBool1, badBool2, true, false}.size(): "
    << std::unordered_set<bool>{badBool1, badBool2, true, false}.size()
    << std::endl;

  return 0;
}

badBool1: 213
badBool2: 137
if (badBool1): true
if (! badBool1): true
(badBool1 == true || badBool1 == false || badBool1 == badBool2): false
std :: set <bool> {badBool1, badBool2 , true, false} .size (): 4
std :: unordered_set <bool> {badBool1, badBool2, true, false} .size (): 4

The conclusion of example 1b

Example 2 - struct

#include <iostream>
#include <stdlib.h>

struct Test
{
    Test(uint64_t v)
        : value(v)
    {
        std::cout << "Test::Test(" << v << ")" << std::endl;
    }
    ~Test()
    {
        std::cout << "Test::~Test()" << std::endl;
    }

    uint64_t value;
};

Test bad(int v)
{
    if (v == 0) {
        return {42};
    } else if (v == 1) {
        return {142};
    }
}

int main()
{
    const auto rnd = rand();
    std::cout << "rnd: " << rnd << std::endl;

    std::cout << bad(rnd).value << std::endl;

    return 0;
}

• in some cases, execution continued without calling the constructor - in this case, the object was in some random state;
• in other cases, a constructor call was not provided for on the execution path, which is rather strange.

Linux x86-x64 Clang 10.0.0, -O1, -O2

• if the second condition is true - you need to call the constructor Test with argument 142;
• if the second condition is not true, the function will exit without returning a value, which means undefined behavior in which the compiler can do anything. Including - call the same constructor with the same argument;
• verification is superfluous; the Test constructor with argument 142 can be called without checking the condition.

Test bad(int v)
{
    if (v == 0) {
        return {42};
    } else if (v == rand()) {
        return {142};
    }
}

#include <iostream>
#include <stdlib.h>

struct Test
{
    Test(uint64_t v)
        : value(v)
    {
        std::cout << "Test::Test(" << v << ")" << std::endl;
    }
    ~Test()
    {
        std::cout << "Test::~Test()" << std::endl;
    }

    uint64_t value;
};

Test bad(int v)
{
    if (v == 0) {
        return {42};
    } else if (v == rand()) {
        return {142};
    }
}

int main()
{
    const auto rnd = rand();
    std::cout << "rnd: " << rnd << std::endl;

    std::cout << bad(rnd).value << std::endl;

    return 0;
}

Windows MSVC 2019 16.5.4, / Od / RTCs

• filling in a variable whose address is passed through the rcx register (here we see RVO in action);
• returning the address of this variable through the rax register.

• in eax, the value 0xCCCCCCCC is entered, which we saw in the Access violation message (line 9) (note that it is only 4 bytes, while in the AccessViolation message the address consists of 8 bytes);
• the rep stos command is called, executing 0xC cycles of writing the contents of eax to memory starting from the address rdi (line 10). These are 48 bytes - exactly as much as is allocated on the stack in line 6;
• on the correct execution paths, the value from rsp + 40h is entered in rax (lines 23, 36);
• the value of the rcx register (through which main () passed the destination address) is pushed onto the stack at rsp + 8 (line 4);
• rdi is pushed onto the stack, which reduces rsp by 8 (line 5);
• 30h bytes are allocated on the stack by decreasing rsp (line 6).

• zeros in the upper part of the address - there could be any garbage;
• the address accidentally turned out to be incorrect.

Example 2a

Windows MSVC 2019 16.5.4, / Od / RTCs

#include <iostream>
#include <stdlib.h>

struct Test
{
    Test(uint64_t v)
        : value(v)
    {
        std::cout << "Test::Test(" << v << ")" << std::endl;
    }
    ~Test()
    {
        std::cout << "Test::~Test()" << std::endl;
    }

    uint64_t value;
};

Test bad(int v)
{
  std::cout << "rnd: " << v << std::endl;
  
  if (v == 0) {
        return {42};
    } else if (v == 1) {
        return {142};
    }
}

int main()
{
    const auto rnd = rand();

    std::cout << bad(rnd).value << std::endl;

    return 0;
}

rnd: 41
8791039331928
Test :: ~ Test ()

Conclusion

• collect about 10 different manifestations of indefinite behavior;
• learn in detail exactly how these options will be executed.