Getting-Started.md

# How to reproduce the results

### rev.ng installation

You will need a copy of the *rev.ng* framework. To get *rev.ng* up an running you'll need *orchestra*, a tool which automatically compiles all the software which composes *rev.ng*, and which also manages the runs of the SPECint 2006 benchmarks.

You can retrieve the *orchestra* repository at https://rev.ng/gitlab/revng-bar-2019/orchestra.git

Please keep in mind that this version of *orchestra* is the one that has been used for producing the results presented in the paper. Since then, some heavy changes have been made to *orchestra*, so if you are currently using a more recent version of *orchestra* consider that these instructions may not be valid anymore.

You'll also need a copy of the [SPECint 2006 benchmarks](https://www.spec.org/cpu2006/).

First of all, for building *rev.ng* you need some dependencies. For a debian based system, you can easily install them with:

```
apt-get install --no-install-recommends \
  aufs-tools \
  build-essential \
  ca-certificates \
  cmake \
  curl \
  gawk \
  git \
  libglib2.0-dev \
  m4 \
  pkg-config \
  python \
  python-pyelftools \
  python3 \
  python3-cffi \
  python3-pygraphviz \
  python3-setuptools \
  sed \
  texinfo \
  valgrind \
  zlib1g-dev
```

You'll also need to install `git-lfs`:

```
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
apt-get install --no-install-recommends git-lfs
```

Now you can proceed with the cloning of the *orchestra* repository:

```
git clone https://rev.ng/gitlab/revng-bar-2019/orchestra.git
```

Then you'll need to build *rev.ng*. To do this, first move into the `orchestra` folder, and checkout the branch which contains all the tools and configurations for running the SPEC benchmarks:

```
git checkout feature/spec
```

Now you'll need to export some environment variables, which are used to prepare the build environment:

```
export BINARY_COMPONENTS=""
```

This tells *orchestra* to compile all the needed components from scratch.

```
export QEMU_DEFAULT_BUILD="qemu-release"
```

This tells *orchestra* to use the optimized version of QEMU during the translation of the binaries.

```
export LIBC_DEFAULT_CONFIG="gc-o2"
```

This tells *orchestra* to compile the `libc` library using optimizations.

```
export DEFAULT_TOOLCHAINS="x86-64"
```

This tells *orchestra* to compile only the toolchains for the `x86-64` architecture, without building all the other toolchains.


At this point, you can build *rev.ng* using *orchestra* with the command:

```
make install-revamb
```

### Installing SPEC

For what concerns the SPECint benchmark installation, you can provide, with the `SPEC_ARCHIVE_URL` environment variable, an address where to retrieve an archive with the content of the SPEC DVD. The archive should be named `spec2006_int_fp.tar.gz` (or you can specify a different archive name using the `SPEC_ARCHIVE_NAME` environment variable).

Alternatively you can directly place the archive in the `source-archives` folder inside the `orchestra` folder.

The archive structure should be made in such way a that, at the root level, there is a folder called `spec2006_int_fp/`, containing the content of the SPEC DVD. Basically, the archive structure should be like this:
```
user@machine:$ tar taf spec2006_int_fp.tar.gz

spec2006_int_fp/benchspec
spec2006_int_fp/bin
spec2006_int_fp/config
spec2006_int_fp/cshrc
spec2006_int_fp/Docs
spec2006_int_fp/Docs.txt
spec2006_int_fp/install_archives
spec2006_int_fp/install.bat
spec2006_int_fp/install.sh
spec2006_int_fp/LICENSE
spec2006_int_fp/LICENSE.txt
spec2006_int_fp/MANIFEST
spec2006_int_fp/README
spec2006_int_fp/README.txt
spec2006_int_fp/redistributable_sources
spec2006_int_fp/result
spec2006_int_fp/Revisions
spec2006_int_fp/shrc
spec2006_int_fp/shrc.bat
spec2006_int_fp/tools
spec2006_int_fp/uninstall.sh
spec2006_int_fp/version.txt
...
```

### Running the benchmarks

At this point, we are ready to run the benchmarks. *orchestra* already provides various targets which are designed for this purpose.
For example, the command:

```
make toolchain/x86-64/spec/native/make-gc-o2-static
```

is responsible of building the binaries that we will use for benchmarking the performances of the native binaries.

To actually run the benchmarks, you can type:

```
make toolchain/x86-64/spec/native/use-gc-o2-static
```

The results of the run will be stored inside the folder `build/toolchain/x86-64/spec/native/use-gc-o2-static/result` (the path is relative to the `orchestra` folder) in an uncompressed form. A bundle of the results obtained will also be saved in `root/x86_64-gentoo-linux-musl`.

By default, the benchmarks will execute three runs of the `ref` size of the benchmarks (which have been used to generate the data included in the paper). You can customize the SPEC invocation using the `SPEC_FLAGS` environment variable of *orchestra* (the default value is `--iterations=3 --loose --size ref int`)

The targets seen above are the ones used for benchmarking the native binaries, and there are other 3 couples of targets for building and running the benchmarks, which are the targets for QEMU, binaries translated with *rev.ng*, and binaries translated and to which function isolation has been applied.

In particular, the possible targets are:

```
native
qemu
revamb
revambisolated
```

For example, if we want to run the benchmarks for the binaries translated with function isolation on, we need to run:

```
make toolchain/x86-64/spec/revambisolated/make-gc-o2-static
make toolchain/x86-64/spec/revambisolated/use-gc-o2-static
```

These targets will take care of building the binaries, translating them with *rev.ng* (and applying function isolation), and running the SPEC benchmarks using the translated binaries.

If you need to re-do a benchmark run, you can easily remove the installed benchmarks with the `clean` target, for example the commands:

```
make clean-toolchain/x86-64/spec/revambisolated/make-gc-o2-static
make clean-toolchain/x86-64/spec/revambisolated/use-gc-o2-static
```

remove the benchmarks of the binaries translated with function isolation.

### Configure and build with *orchestra*

*orchestra* provides two useful targets to inspect the available targets and configurations. The first target is `make help-components`, which prints all the targets that *orchestra* can build.

The `make help-variables` command instead, prints all the environment variables that can be configured and which will affect the building of the components (such as the environment used above to build QEMU in release mode instead of debug mode).

### SPEC integration in *orchestra*

The integration of the SPEC benchmarks with the *rev.ng* framework is done by the `revcc` script (which you can find in the `revamb/scripts` folder), which is in charge of invoking the compilation, lifting, function isolation and recompilation of the binaries that we want to translate using *rev.ng*.

In the `support/spec-config` folder, you can find the files used by SPEC as config files. Taking as an example the file `x86_64-gentoo-linux-musl-revambisolated.cfg`, you can see the invocation of `revcc` with the `-O2` and `-i` parameters, which are respectively in charge of instructing *rev.ng* to apply optimizations to the lifted module, and to apply function isolation before recompilation. The remaining parameters (the ones that follows `--`), represent the compiler and the parameters that will be used for compiling the original binaries (before lifting them).