At the other 2 posts about SGX, you got an idea about what it is and what it is good for.

#### And we barley saw any code!

This post is all about real SGX code demos.
By the time we conducted this project, SGX was a new concept and lacks a big community that provide working code samples!

# Ocalls

System calls like `printf()`, `fopen()`, `time()`
cannot be invoked from inside the Enclave, since the OS (Operating System) is considered as untrusted from the Enclave’s point-of-view!

To workaround this lack of trust, we can still use system calls, only if we define them as untrusted, thus we give the Enclave heads-up to take precautions prior executing the system call.

### How do we declare them as untrusted?!

#### The EDL file!

It is the exact tool for defining which methods are Untrusted (i.e. functions which are not defined inside an Enclave, however, they are used in one), as well as defining which functions are trusted, such functions are defined inside the current Enclave ,
and you can think of them as the API which the Enclave provides the Application which imports that Enclave.

Another important feature which the EDL provides; you can import other trusted functions from other Enclaves as well!

### Awesome! Let’s code some OCALLS!!

#### Right on!

Since the Ocalls are basically untrusted functions, we will go forward and define them inside the Application:

``` void Ocall_printf(char* str) {
printf("[Ocall prinf] - %s\n", str);
}
void Ocall_time(char* outTime, size_t len) {
time_t seconds = time(NULL);
char buf[26] = {'\0'};
ctime_s(buf, 26, &seconds);
memcpy(outTime, buf, 26);
}

void Ocall_open(const char* filename, unsigned int* fd, size_t len) {
FILE* _fd;
fopen_s(_fd, filename, "a+");
*fd = (unsigned int)_fd;
}

void Ocall_write(char* data, unsigned int* fd, size_t len) {
FILE* _fd = (FILE*)(*fd);
fwrite(data, sizeof(char), strlen(data), _fd);
}

void Ocall_close(unsigned int* fd) {
fclose((FILE*)(*fd));
}
```

You are looking at 5 Ocalls! That is a LOT of Ocalls!

it’s getting interesting, you’re saying, eh?
You bet it is!!

We’re just getting started though!

Our aim right now is to use those 5 Ocalls above inside an Enclave inorder to
log to a file the current time, as well as some debugging messages, using `Ocall_printf()`

First, we will start by creating an Enclave!

```/*
* App.cpp - untrusted code.
*/

#include "stdafx.h" // /lt-- contains all the required headers

const long long ITERATIONS = 1000;

/*
* Creates an Enclave using sgx_create_enclave()
*
* returns the id of the created Enclave,
* via the output arg.
*/
sgx_status_t createEnclave(sgx_enclave_id_t *eid) {
sgx_status_t		ret   = SGX_SUCCESS;
sgx_launch_token_t	token = {0};
int					updated = 0;
ret = sgx_create_enclave(ENCLAVE_FILE,
SGX_DEBUG_FLAG, &\token, &\updated, eid, NULL);
return ret;
}

/*
* main()
* calls the Trusted functions of the Enclave.
*/
int _tmain(int argc, _TCHAR* argv[]) {
sgx_enclave_id_t eid;
sgx_status_t res = createEnclave(&\eid);
if (res != SGX_SUCCESS) {
printf("App: error-, failed to create enclave.\n");
return -1;
}
for(long long i = 0; i \< ITERATIONS; ++i) {
OcallFunctions_Enclave(eid);
}
. . .
return 0;
}
```

### Awesome! Now we have an Enclave with the an ID stored in `eid`, ready to be launched!

It feels just about the right time to fill in the EDL file:

```/*
* Enclave_ocalls.edl
*/
enclave {
trusted {
/* functions defined inside this block are the API-
* which the Enclave defines for the Applications (i.e. untrusted programs)
* that import it.
*
* In this example, we will call OcallFunctions_Enclave() from the main() of App.cpp
*/
public void OcallFunctions_Enclave();
};

untrusted {
void Ocall_printf([in, string] char* str);
void Ocall_time([out, size=len]char* outTime, size_t len);
void Ocall_open([in, string] const char* filename,
[out, size=len]unsigned int* fd, size_t len);
void Ocall_write([in, string] char* data,
[in, size=len]unsigned int* fd, size_t len);
void Ocall_close([in] unsigned int* fd);
};
};
```

Notice that the functions in the EDL file (both the trusted and untrusted) that take a pointer as an argument, must be
attached to an attribute, for example:
`void Ocall_time([out, size=len]char* outTime, size_t len);` Here the attribute is `[out, size=len]`.
We defined `Ocall_time` to return the current time in a char buffer,
therefore we’ve written `out` in the attribute to indicate that this argument is an output.
Moreover, we must define the length of the buffer beforehand and then pass it to the function as `size_t len`, so we write `size=len`.

Also, `void Ocall_open([in, string] const char* filename, ...`, takes as an input the name of the file as a `char*`.
Since it is a NULL termination string (i.e. ending in `'\\0'`), it is suffecient to write `string` inside the attribute.

Next, we are going to implement the trusted Enclave function (`OcallFunctions_Enclave()`, remember?!)

```#include "Enclave_ocalls_t.h"
#include "sgx_trts.h"
#include <\stdlib.h\>
#include <\string.h\>
void OcallFunctions_Enclave() {
Ocall_printf("[Enclave] - printing message..");
char local_time[26] = {'\0'};
Ocall_time((local_time), 26);
Ocall_printf( local_time);
unsigned int fd = 0;
Ocall_open("logging.txt", &\fd, sizeof(unsigned int*));
Ocall_write(local_time, &\fd, sizeof(unsigned int*));
Ocall_close(&\fd);
}
```

Great! looks like a pretty straight-forward function:

• calculates current time
• printes it to `stdout`
• opens a file
• writes the time to that file
• closes the file!

### So far so good.

Let’s see the rest of the `main()` which we couldn’t care less to continue before 🙂

```/*
* main()
* calls the Trusted functions of the Enclave.
*/
int _tmain(int argc, _TCHAR* argv[]) {
sgx_enclave_id_t eid;
sgx_status_t res = createEnclave(&\eid);
if (res != SGX_SUCCESS) {
printf("App: error-, failed to create enclave.\n");
return -1;
}
for(long long i = 0; i <\ ITERATIONS; ++i) {
OcallFunctions_Enclave(eid);
}
for(long long i = 0; i <\ ITERATIONS; ++i) {
OcallFunctions_Enclave(eid);
}
return 0;
}
```

## Intel Vtune

We used Intel Vtune software to measure the performance of the Ocall program above
and compare it with the same functions running in a non-Enclave environment.

Here what it looks like:

 Enclave No Enclave Elapsed Time [sec] 4.067 3.465 Instructions Retired 26,870,200,000 22,181,600,000 CPI Rate 0.83 0.948 Cache Bound (Hit Rate) 0.141 0.128 Load 8,055,024,165 7,470,022,410 Store 3,859,057,885 3,640,054,600

Note that the overhead in this test is not only due to switching environments between the untrusted and trusted code.
Since each iteration in the App, performs multiple Ocalls per call to the Enclave’s function,
thus we are switching environments between the trusted code (Enclave) and the Ocalls,
multiple times in each iteration.

Also the Enclave seems to perform more Load/Store instructions, comparing to the non enclave test,
because when swapping out a page from the DRAM to the disk, it has to be encrypted,
since it is an untrusted storing area, and the Enclave’s contents has to be confidential.

The cache hit ratio is better at the Enclave environment.

# Recursion functions inside Enclave

Through recursion, we can get a clear picture for the behavior of memory inside the Enclave and on a regular environment,
because, with each recursion call, the program must store the previous ‘snapshot’ of the function
so it could restore its stack when the recursion regrets back.

### Fibonacci and Factorial

Probably the classic examples when it comes to recurssion, and both are quit intersting, since they cove both ends of the
time complexity spectrum, linear and exponential, whereas Fibonacci runs in O(2^{n})\$ and Factorial in $O(n)$.

### Let’s dive right to the App.cpp code!

```#include "stdafx.h"
#include "sgx_urts.h"
#include "Enclave_recursive_func_u.h"
#define ENCLAVE_FILE _T("Enclave_recursive_func.signed.dll")
#define NO_ENCLAVE
#define FIBONACCI
//#define FACTORIAL
const long long ITERATIONS = 1000000;
#ifdef ENCLAVE
sgx_status_t createEnclave(sgx_enclave_id_t *eid) {
sgx_status_t        ret   = SGX_SUCCESS;
sgx_launch_token_t    token = {0};
int                    updated = 0;
ret = sgx_create_enclave(ENCLAVE_FILE, SGX_DEBUG_FLAG,
\&amp;amp;amp;token, \&amp;amp;amp;updated, eid, NULL);
return ret;
}
#endif

//#ifdef FACTORIAL
const int FACT_CONST = 25;
static int aux_factorial (int num) {
if(num == 1) return num;
return num * aux_factorial(num-1);
}
int factorial_NoEnclave() {
return aux_factorial(FACT_CONST);
}

const int FIB_CONST= 10;
int fibonacci_NoEnclave(int num) {
if(num == 1 || num == 0) return num;
else if(num &amp;amp;amp;amp;amp;amp;amp;amp;lt; 0) return -1;
return fibonacci_NoEnclave(num-1) +
fibonacci_NoEnclave(num-2);
}

int _tmain(int argc, _TCHAR* argv[])
{
int* res = new int;
*res = 0;
#ifdef ENCLAVE
sgx_enclave_id_t    eid;
if (createEnclave(\&amp;amp;amp;eid) != SGX_SUCCESS) {
printf("App: error-, failed to create enclave.\n");
return -1;
}
for(long long i(0); i /lt; ITERATIONS; ++i) {
fibonacci_Enclave(eid, (int*)res, sizeof(int));
//factorial_Enclave(eid, (int*)res, sizeof(int));
}
printf("res = %d\n", *res);
#endif

#ifdef NO_ENCLAVE
printf("[NoEnclave]");
for(long long i(0); i \&amp;amp;lt; ITERATIONS; ++i) {
#ifdef FIBONACCI
fibonacci_NoEnclave(FIB_CONST);
#endif

#ifdef FACTORIAL
factorial_NoEnclave();
#endif
}
#endif
return 0;
}
```

The most scary thing in the code above is the use of ` #ifdef / #endif `

### No worries though!

since we are comparing Enclave and non-Enclave environments,
say, we want to switch from Enclave to non-Enclave, then we comment out the `#define ENCLAVE`
and replace it with `#define NO_ENCLAVE`.

The code of the Enclave functions’ for the Fibonacci and Factorial look exactly the same in both environments!
which is the essence of the comparason we’ll see now that we got from Vtune –

### Fibonacci

 Enclave No Enclave Elapsed Time [sec] 2.934 2.233 Instructions Retired 6,092,800,000 6,143,800,00 CPI Rate 1.868 1.395

As we have already seen running inside an enclave takes much longer time: $\dfrac {3.601}{3.141}=1.146$ .
More instructions are executed with the enclave implementation so the CPI is lower. The frequency is the same.

### Factorial

 Enclave No Enclave Elapsed Time [sec] 2.775 0.463 Instructions Retired 3,648,200,000 1,557,200,00 CPI Rate 2.888 1.158

Here the gap is huge: $\dfrac {2.775} {0.463} = 5.993$
so even though the number of instructions executed with the enclave implementation is higher, the CPI of the enclave implementation is still bigger.

As we know the complexity of Fibonacci is exponential while it is linear for factorial.
We can see that while running without enclave,
factorial is running much faster than Fibonacci: $\dfrac {2.775} {0.463} = 5.994$,
running in enclave shrinks the difference because the overhead of running inside an enclave is not negligible.

This is among the most interesting features we tested in SGX.
Although you cannot create threads inside the Enclave itself, rather outside, in the untrusted App,
each thread would still have its private entry to the Enclave, which creates a virtual simulation of creating

Each thread has to own its own TCS.

### Number of TCS is set in the config file of the Enclave!

the config file above looks like so (check line 6 bellow):

```<EnclaveConfiguration>
<ProdID\>0</ProdID>
<ISVSVN&amp;amp;gt;0</ISVSVN>
<HeapMaxSize>0x100000</HeapMaxSize >
<StackMaxSize>0x40000</StackMaxSize>
<TCSNum>100</TCSNum>
<TCSPolicy>1</TCSPolicy>
<DisableDebug>0</DisableDebug>
<MiscSelect>0&amp;amp;lt;/MiscSelect>
</EnclaveConfiguration>
```

Notice the field of the TCSNum field which is by default is initialized to one!
Hence, we had to manually change it to 100, since in our example code below,
we are producing 100 threads (50 per writers and 50 for readers).

Now that we have a steady ground, let’s go a head and implement the

Following is the API that the Enclave supply the App with:

``` /* trusted - Enclave_MultiThreading.cpp */
void Enclave_Init() {
Ocall_printf("   [Enclave] - Initialized.\n", NULL);
}

int* tmp = (int*) malloc(sizeof(int));
}

void Enclave_Write(int* thread_id, size_t len) {
writers_lock();
int* tmp = (int*) malloc(sizeof(int));
Ocall_printf(WRITE_IN, tmp);
writers_unlock();
Ocall_printf(WRITE_OUT, tmp);
}
```

And here are the main driver (i.e. the static funcitons) of the API above,
as well as the global variables we needed:

```/*
*
*/
#include <stdlib.h>
#include <string.h>
#include <time.h  >
#include "sgx_trts.h"

#define WRITE_IN  "write_in"
#define WRITE_OUT "write_out"

static int number_of_writers = 0;

number_of_writers = 0;
}

while(number_of_writers > 0) {
&global_lock);
}
int* tmp = (int*) malloc(sizeof(int));
}

}
}

static void writers_lock() {
while(number_of_writers < 0 || 				  number_of_readers > 0) {
&global_lock);
}
number_of_writers++;
}

static void writers_unlock() {
number_of_writers--;
if(number_of_writers == 0) {
}
}
```

Meanwhile, in the `App.cpp` we used `std::thread` to create the threads:

``` /* un-trusted - App.cpp */
static sgx_status_t createEnclave(sgx_enclave_id_t *eid) {
sgx_status_t		ret   = SGX_SUCCESS;
sgx_launch_token_t	token = {0};
int					updated = 0;
ret = sgx_create_enclave(ENCLAVE_FILE, SGX_DEBUG_FLAG,
&token, &updated, eid, NULL);
return ret;
}

&(data.id), sizeof(int));
}

static void App_write(App_Api_Struct& data) {
Enclave_Write(data.eid, &(data.id), sizeof(int));
}

int _tmain(int argc, _TCHAR* argv[])
{
sgx_enclave_id_t eid;
sgx_status_t res = createEnclave(&eid);
if (res != SGX_SUCCESS) {
printf("[APP] Error - failed to create enclave.\n");
return -1;
} else {
printf("[APP] Enclave created!\n");
}
Enclave_Init(eid);
for(int i(0); i <= NUMBER_OF_THREADS; ++i) {
}

thr.join();
}
return 0;
}
```

## Results

In order to insure that the code above meets the requirements of concurrency as well as synchronization,
we conducted two test cases:

• All the locking mechanism on both readers and writers are applied as in the pseudo code
• Locking mechanism is omitted from both reading function and writing.

The goal is to see that without the locking mechanism we can get a scenario
where multiple writers are in the shared resource as well as there might be
readers along with writers in the resource as well, which is a prohibited situation,
according to the definition of the problem.
Also we want to show that there could be multiple readers in the shared critical section,
however, there ought to be no writers inside.

### Results of the code with the locking mechanism

In this section we are going to explain how much the Enclave costs, in terms of creation time and energy.
For this purpos we used the software Intel Power Gaget 3.0

That Software measures each $100[ms]$ various parameters and logs them to a ` .csv ` file.

Among the important parameters of the logged file, are ` IA_Power_0 (Watt)`
` Cumulative IA Energy_0 (Joule)`

1.  As we observed the power graph of the CPU when running a program of SGX, we get a peak.
Thus, in the logged .csv file, we looked for a sudden increase in the values,
and a sudden decrease which indicates that the CPU has done processing the program.
2. Then, we consider the accumolated energy measurements and subtract both section to get the exact
energy which the CPU invested throughout the execution of the SGX program.
3.  We do the same process in 2) for the program without Enclave.
4.  the overhead of all the Enclave starting from entering till exiting multiplied by the number of iteration is
the subtraction of both results in 2) and 3)
5. Afterwards, we divide the total energy calculated in 4)
and divide it by the number of iterations whichh the App program performs,
which gives a precise estimation per Enclave overhead.

#### Results from four tests

 Test Name Enclave [Joule] No Enclave [Joule] Empty Function 8.893 1.37 Library Calls 207.384 99.493 Fibonacci 53.127 42.662 Factorial 45.249 16.298 Ocalls 73.308 71.682

 Test Name Enclave Overhead[sec] Enclave Overhead [Joule] Empty Function 0.000602 0.001626 Library Calls 0.000002374 0.000007523 Fibonacci 0.000007198 0.000107891 Factorial 0.000002615 0.000010465 Ocalls 0.000003077 0.000028951

All the measurements above are of the same order, except for the Ocall test,
and that is due to te fact that the same overhead is been consumed multiple
time at the same iteration, as we explained before.

# Breach No. 1 – Leaking info from the data segment

I am not going to dwell on the theoritical interpretation of the attack, rather on what we have
coded to simulate it on SGX!

We used Microsoft Visual Studio 2012 for creating the code in this project.
There is the flag of `/GS` in the compiler that insures all the functions used in the program are safe
, as well as guards against buffer overflows in the stack.
Thus, we did not have to bother bypassing its regulations since our aim right now
is pointed toward the data segment and not the stack.

Since the data segment does not contain any return addresses,
and even it is determined during compilation time,
as well as the huge space address that isolates this segment from the stack segment,
there is no interest in protecting this area.

However, since the data section could contain key variables which lead to important calculations and returned values from various functions in the program, make it a valuable target for us to try and corrupt its contents, and see how SGX and even the compiler could restrict our attack.

Thus we are going to examine the SGX vulnerability to attacks targeting the data
memory segment of the process, and see how critical impact it will hae on the outcome of the program.

### Buffer overflow inside Intel SGX Enclave

We will be demonstrating the buffer overrun in SGX program inside the data section,
so we can show that memory of the Enclave can be vulnerable.

Our program will demonstrate how a notorious user can take advantage of that vulnerability

The motive for using this approach, as explained before,
is that we can achieve it without the use of system calls,
as well as without the use of unsafe library functions like `gets()`,
which reads input from the user (i.e. console) to a buffer, without prior knowledge of the input’s length.
As a workaround for avoiding the use of `gets()`,
and hence avoiding error messages from the compiler, we defined our very own safe implementation of this function.

```void getStr(char buffer[]) {
char c;
int idx(0);
do {
c = getchar();
buffer[idx++] = c;
} while(c != '\n');
buffer[--idx] = '\0';
}
```

The App (i.e. un-trusted code) contains two main entry points:

• `Authenticate` – This is where the App receives the credentials from the user to ensure their validity in the system, and only then it could allow the user to interact with the Enclave’s functions to exchange data.
• `toUser` – Un-trusted code where the user is permitted to send requests to the Enclave’s functions.

The Authentication is part of the untrusted code which invokes an `ECALL`
which is responsible for checking if the user is registered properly by the username and password :

```static bool App_Authenticate(sgx_enclave_id_t eid, char* buf_user,
char* buf_pass) {
int is_valid = FAIL;
Enclave_Authenticate(eid, buf_user, buf_pass, &is_valid);
if(is_valid == SUCCESS && strcmp(buf_user, "guy") == 0) {
return true;
} else if(is_valid == SUCCESS &amp;amp;amp;amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;amp;amp;amp; strcmp(buf_user, "bassam") == 0) {
return true;
} else {
printf("INVALID - try again\n\n");
return false;
}
}

int main() {
.  .  .
do {
printf("> ");
getStr(buf_user);
getStr(buf_pass);
printf("\n");
} while(App_Authenticate(eid, buf_user, buf_pass) == false);
.  .  .
return 0;
}
```

The following code snippet demonstrates how the user could
leak data out of the data segment by passing large length of the array which gets it filled with classified sealed data.

```void toUser(int gradesArr[], int len) {
for(int i = 0; i < len ; i++) {
if(i <= 2) {
} else {
return;
}
}
}
int main() {
. . .

int* arr = (int*) malloc(num_of_grades * sizeof(int));
if(arr == NULL) {
return -1;
}
size_t len = num_of_grades * sizeof(int);

.  .  .

return 0;
}
```

The function above is under the absolute control of the user. As we can see,
it receives an integer array and a length, where the array contains some of the data
which the user is permit to request – grades, as well as leaked data
due to large length of the inputted number which caused the array to be filled with data from the adjacent memory sections.

To understand better how data is leaked, we will have a look at the data segment of the Enclave:

```// sample_enclave.cpp
#include "sample_enclave_t.h"
#include <string.h>

#define PASS_LENGTH  4

int grades_Guy[3] = {100, 99, 50};
char CORRECT_PASS_GUY[] = "123";
int grades_Bassam[3] = {76, 88, 28};
char CORRECT_PASS_BASSAM[] = "asd";

static const int FAIL = 0;
static const int SUCCESS = 1;

char buffer[PASS_LENGTH ] = {'\0'};
int pass = 0;
```

Look at the green-highlighted data.
Notice that the char buffer `CORRECT_PASS_GUY`
falls right after the integer array `grades_Guy` such that if we
over-read from `CORRECT_PASS_GUY` then we end up reading from `grades_Guy`!
And that is due to the lack of protection in the data segment as we previously explained.
Thus, if we try to read one, two, or three grades, then we will get a valid result without
causing any leakage, because we are still within the boundary of the grades array – of a length of three.
However, if we chose to read four or more grades, then we are going to read the char
buffer of the password, which is located right after the grades array!
Notice that since the password is of length three chars plus the null terminator of `‘0\’`,
the total password is four bytes in size which is equivalent to exactly one integer,
thus, it would fit perfectly within an integer array cell.
Below are the screen shots of the two major test cases described above:

This slideshow requires JavaScript.

# Breach No. 2 – Data Segment Buffer Overrun

In this example, we are going to change the behaviour of the Enclave function,
as a result of overrunning a flag value.
At the Enclave code, we put a global variable of int as well as a global char buffer.
Since the global variables sit at the Data segment of the process’s memory, hence,
this section is determined at compilation, as it is permanent throughout the execution.

The trusted Enclave’s function receives from the App a string as it copies it to its memory domain to run few checks on it.
Since we chose the above string to be larger than the Enclave’s global buffer could handle, surprisingly,
it overruns the adjacent memory location, which contains another global integer flag,
resulting in changing its value and thus jeopardising the logical flow of the function which ends up giving wrong results.
Before looking at the code, we want to sketch the big picture behind the idea of the breach in the figure bellow –

We will start by the Enclave function that does the check on the Authentication,
and see how we can bypass the password check:
Here is the function itself:

```void Enclave_Authenticate (char* user, char* password, int* res) {
if( strcmp(user, "guy") == 0  &&
strcmp(buffer, CORRECT_PASS_GUY) == 0 ) {
pass = 1;
} else if (strcmp(user, "bassam") == 0   &&
strcmp(buffer, CORRECT_PASS_BASSAM) == 0) {
pass = 1;
} else {
memcpy(res, &FAIL, sizeof(FAIL));
}
if (pass != 0) {
memcpy(res, &SUCCESS, sizeof(SUCCESS));
} else {
memcpy(res, &FAIL, sizeof(FAIL));
}
}
```

The attack starts at `strncpy`, where we copy the input string to the global buffer of the Enclave.
Notice that the pass variable come right after the buffer array,
as seen in the data segment code above, thus, if we overwrite to buffer – via `strncpy`,
then we can overwrite the current value of pass, which will impact the `if – else` logic flow afterwards,
since we can determine the output of the function `Enclave_Authenticate`,
since if we overwrite pass, then its value will be different than zero which will return `SUCCESS`