====== syzkaller with Ubuntu host and FreeBSD guest ======

syzkaller is a fuzzer that test system calls. Therefore, it starts one or more VMs in order to test system calls on the OS (target) inside the VM.

syzkaller consists of three main components. syz-manager runs on the host. It starts the VMs and connects to them via SSH. syz-fuzzer runs inside a VM. It maintains one or more syz-executor, syz-executor actually executes the system call on the target.

To be able to start a VM, syzkaller uses a virtualization software. syzkaller best supports qemu.

The first task is to create VM with target OS. The second one is to build syzkaller (requires go) and the third and last one is to create a config in order to run syzkaller.

My environment:
  * Host: Ubuntu 18.04.1 LTS (amd64) 
  * Target: FreeBSD-13.0-CURRENT-amd64-20190110
  * QEMU emulator version 2.11.1

===== Create FreeBSD VM =====

==== Create VM and install FreeBSD ====

create directory and qemu hard disk image
  mkdir -p ~/qemu/freebsd
  cd ~/qemu/freebsd
  sudo qemu-img create freebsd.img 16G

start vm and install FreeBSD (download iso from freebsd.org). Make sure FreeBSD source files are copied to /usr/src during installation.
  sudo qemu-system-x86_64 -enable-kvm -hda freebsd.img -m 2G -net user,hostfwd=tcp::10021-:22 -net nic -curses -cdrom FreeBSD.iso -boot d

after installation let vm print to console -> add in /boot/loader.conf
  console="comconsole"

start vm without graphic and with network
  sudo qemu-system-x86_64 -hda freebsd.img -enable-kvm -m 2G -net user,hostfwd=tcp::10021-:22 -net nic -nographic

  * -net user let the vm uses a user mode network stack. Since hostfwd=tcp::10021-:22 is added, a connection to localhost:10021 (on host) is forwarded to the vm.
<code>
     guest (10.0.2.15)  <------>  Firewall/DHCP server <-----> Internet
                           |          (10.0.2.2)
                           |
                           ---->  DNS server (10.0.2.3)
                           |
                           ---->  SMB server (10.0.2.4)</code>
  * -nographic prints the console output of the vm directly into the console on the host

==== Enable KCOV support for FreeBSD ====

By now, KCOV support is not enabled by default. Create a own kernel config file in the VM (/usr/src should contain the FreeBSD source files).
  cd /usr/src/sys/amd64/conf
  cp GENERIC KCOV
  
In file KCOV, uncomment the lines
  options        COVERAGE
  options        KCOV

Build and install FreeBSD kernel as root, and restart
  cd /usr/src
  make -j4 kernel KERNCONF=KCOV
  shutdown -r now

==== Allow SSH root access to FreeBSD ====

syzkaller needs root access to the OS in the VM via SSH.

create ssh keys on the host
  ssh-keygen -f freebsd_id_rsa

add public key of the host (freebsd_id_rsa.pub) into /root/.ssh/authorized_keys in the vm

allow ssh as root in vm. In /etc/ssh/sshd_config set
  #PermitRootLogin no
to
  PermitRootLogin yes
after that, restart sshd
  sudo service sshd restart

Test ssh from host to vm
  ssh -i ~/qemu/freebsd/freebsd_id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhost
This must work without entering a password.

===== Build syzkaller =====

syzkaller uses Go -> Install Go

  wget https://storage.googleapis.com/golang/go1.10.4.linux-amd64.tar.gz
  tar -xf go1.10.4.linux-amd64.tar.gz
  mv go goroot
  export GOROOT=`pwd`/goroot
  export PATH=$GOROOT/bin:$PATH
  mkdir gopath
  export GOPATH=`pwd`/gopath
  
get syzkaller
  go get -u -d github.com/google/syzkaller/...
  
build syz-manager, syz-fuzzer on host
  cd gopath/src/github.com/google/syzkaller/
  mkdir workdir
  make manager fuzzer execprog TARGETOS=freebsd
  
note the used GitRevision, for me it was 
  GitRevision=5b11ac2c91d99eababa8f04c7e2f9484c0763b53

build syz-executor on target (inside the VM)
  * create folder in VM

  mkdir -p ~/syzkaller/executor

  * use scp on host to copy source files from host to VM

  scp -P 10021 ~/gopath/github.com/google/syzkaller/executor/* localhost:~/src/syzkaller/executor/

  * build syz-executor in vm using the GitRevision as noted before.

  cd ~/syzkaller
  c++ -o syz-executor executor/executor.cc -pthread -Wall -Wframe-larger-than=8192 -Wparentheses -Werror -O2 -m64 -static -DGOOS_freebsd=1 -DGOARCH_amd64=1 -DGIT_REVISION=\"5b11ac2c91d99eababa8f04c7e2f9484c0763b53\"

  * use scp to copy built executable from VM to host.

  scp -P 10021 localhost:~/syzkaller/syz-executor ~/gopath/src/github.com/google/syzkaller/bin/freebsd_amd64/

===== Run syzkaller =====

To present code coverage, syzkaller needs FreeBSD source and object files. Start the VM with qemu and use scp on host to copy source and object files from target to host.
  mkdir -p ~/freebsd/usr
  scp -r -o "StrictHostKeyChecking no" -P 10021 localhost:/usr/src ~/freebsd/usr/
  scp -r -o "StrictHostKeyChecking no" -P 10021 localhost:/usr/obj ~/freebsd/usr/

create syz-manager config file (on host) ~/gopath/src/github.com/google/syzkaller/freebsd.cfg
  {
    "target": "freebsd/amd64",
    "http": "127.0.0.1:56741",
    "workdir": "/home/msvoelker/gopath/src/github.com/google/syzkaller/workdir",
    "image": "/home/msvoelker/qemu/freebsd/freebsd.img",
    "sshkey": "/home/msvoelker/qemu/freebsd/freebsd_id_rsa",
    "syzkaller": "/home/msvoelker/gopath/src/github.com/google/syzkaller",
    "kernel_obj": "/home/msvoelker/freebsd/usr/obj/usr/src/amd64.amd64/sys/KCOV",
    "kernel_src": "/home/msvoelker/freebsd",
    "enable_syscalls": [
      "socket$inet_sctp",
      "bind$inet",
      "listen",
      "connect$inet",
      "accept$inet",
      "read",
      "write",
      "close"
    ],
    "procs": 1,
    "type": "qemu",
    "cover": true,
    "vm": {
      "count": 16,
      "cpu": 1,
      "mem": 1024
    }
  }

start syz-manager on host
  cd ~/gopath/src/github.com/google/syzkaller
  sudo bin/syz-manager -config freebsd.cfg
  
For a status page, open http://localhost:56741 in a browser.
===== Links =====

github repository for syzkaller (contains documentation): https://github.com/google/syzkaller