sync_up: Discussion, add rest of points from thread

[libreriscv.git] / HDL_workflow / microwatt.mdwn

# Tutorial for setting up Microwatt chroot and running simulations

Useful Links (External):

* <https://codeconstruct.com.au/docs/microwatt-orangecrab/>
* <https://shenki.github.io/boot-linux-on-microwatt/>
* <https://github.com/gregdavill/OrangeCrab-test-sw>
* [Verilator docs, commands](https://verilator.org/guide/latest/exe_verilator.html)
* [Verilator runtime command documentation](https://verilator.org/guide/latest/exe_sim.html)
* Tutorials for how to work with verilator:
[part1](https://www.itsembedded.com/dhd/verilator_1/),
[part2](https://www.itsembedded.com/dhd/verilator_2/)

Useful links (Libre-SOC):

* Libre-SOC page covering our workflow: [[HDL_workflow]]
* Devscripts Libre-SOC page: [[devscripts]]
* Original Microwatt Libre-SOC page: [[microwatt]]
* [Libre-SOC Microwatt repo branch](https://git.libre-soc.org/?p=microwatt.git;a=tree;hb=refs/heads/verilator_trace)
* [Libre-SOC devscripts repo](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree)

Other Tutorials (Libre-SOC):

* First steps for working with PowerISA instructions Libre-SOC page:
[[/docs/firststeps]]

## Video Tutorial

[43min tutorial](https://youtu.be/02LCl3ang8g) was made and uploaded to
Youtube, covering some of the material you'll find on this page.

## Development environment scripts

If you haven't already, clone Libre-SOC's development environment setup scripts.
These are bash scripts, and greatly simplify the time it takes to create a:

- Stable environment
- With all software and libraries at specific versions
(which are known to work).

These attributes are absolutely critical, and no support will be
provided, unless you use these scripts to setup a development environment. This
helps us fix any bugs in the scripts, and make sure everyone runs on the same
page.

    $ git clone https://git.libre-soc.org/git/dev-env-setup.git


Do *look through* the
[code](https://git.libre-soc.org/?p=dev-env-setup.git;a=tree) before running
any of those scripts.
This is your own legal responsibility (to not run
arbitrary code off of the internet) and we take no responsibility or accept
any liability whatsoever for your use or misuse of them.

It is expected for you to use Debian for the host OS (anything else
is unsupported: many contributors have repeatedly gotten into trouble by not
following this advice), while all the chroots - which are developed
very specifically for "reproducible builds" - run Debian 10 (Buster).

## Setting up chroot

Scripts we will be using for the setup are:

* `mk-deb-chroot`, `cp-scripts-to-chroot` for chroot setup
* `install-hdl-apt-reqs`, `verilator-install`, `hdl-tools-yosys` for working
with Microwatt

(*Current limitation for `mk-deb-chroot`, is that you must be the first user on
the host machine, having user ID 1000.*)

Commands to run in terminal to setup a new chroot environment for microwatt
simulations.

    $ cd dev-env-setup
    $ sudo bash
    # ./mk-deb-chroot microwatt
    # ./cp-scripts-to-chroot microwatt
    # exit
    $ schroot -c microwatt
    (microwatt):$ cd dev-env-setup
    (microwatt):$ sudo bash
    (microwatt):# ./install-hdl-apt-reqs
    (microwatt):# ./verilator-install
    (microwatt):# ./hdl-tools-yosys
    (microwatt):# exit
    (microwatt):$ cd ~/src/
    (microwatt):$ git clone https://git.libre-soc.org/git/microwatt.git
    (microwatt):$ cd microwatt
    (microwatt):$ git checkout verilator_trace

Make sure verilator binaries in $PATH:

    (microwatt):$ export PATH=/usr/local/verilator/bin:$PATH
    (microwatt):$ export GHDLSYNTH=ghdl

(GHDLSYNTH needs to be redefined because the Makefile has default `ghdl.so`,
but somewhere else '.so' gets appended. You may see the following error if you
don't redefine:
`ERROR: Can't load module
./ghdl.so':/usr/local/bin/../share/yosys/plugins/**ghdl.so.so**`)
[IRC](https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47)

## Compiling the verilator sim for Microwatt

* [Libre-SOC Microwatt repo branch, Makefile](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;hb=refs/heads/verilator_trace)

Verilator creates a fairly fast simulation by converting the HDL design to C++,
and then compiling a binary which the user runs.

To compile the verilator simulation, first set verilator as the target for the
makefile:

    (microwatt):$ export FPGA_TARGET=verilator

Before compiling, you can change the `THREADS` variable in the makefile, which
will allow the compiled verilator simulation binary to use more than 1 thread
(*make sure to check how many CPU threads you have before changing this!*)

To compile the verilator simulation binary, call make with the
`microwatt-verilator` rule.

    (microwatt):$ make microwatt-verilator

## Compiling hello world code

We need some code to actually run on the core, so start with the 'hello world'.
Instructions assume you're still in the microwatt directory.

    (microwatt):$ cd hello_world
    (microwatt):$ make

A `hello_world.bin` should be generated (the final binary to be loaded), as
well as an
[.elf file](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format), and
.hex (representing the binary data as hex text strings).

To view the symbol table (useful to see where various sections of the binary
begin):

    (microwatt):$ powerpc64le-linux-gnu-objdump -h hello_world.elf
    (microwatt):$ powerpc64le-linux-gnu-objdump -x hello_world.elf

`-h` shows just the section headers, `-x` shows all headers.

And to view the disassembly (great for learning about the PowerISA instructions,
and for associating the binary hex with actual instructions), you can view the
automatically generated `hello_world.as` file.

Command to generate the disassembly:

    (microwatt):$ powerpc64le-linux-gnu-objdump -D hello_world.elf

For more information about `objdump` (common utility, not just for PowerISA),
see the manual pages.

    (microwatt):$ man powerpc64le-linux-gnu-objdump

The binary is ready to go, now it can be loaded into the simulation.

## Simulation

### Command line args

To find out the `microwatt-verilator` arguments, you can check with `-h` arg:

    (microwatt):$ ./microwatt-verilator -h

Some of the arguments are explained in further sections.

### Running

Run the `microwatt-verilator` binary, with `hello_world/hello_world.bin` as an
argument:

    (microwatt):$ time ./microwatt-verilator hello_world/hello_world.bin

`time` is a utility you can use to measure how long it takes to run the sim.

A pretty ASCII art of a lightbulb should be printed, and then the user can type
any characters, which will be echoed back. To end the simulation press Ctrl+C.

If no characters are appearing after about 20 seconds, stop the simulation,
as there might be other issues.

Single-threaded verilator sim binary, on a 2nd gen intel i5 (sandybridge)
takes 53 seconds to print the ASCII lightbulb.

On another dev's machine, ASUS KGPE D16, this takes just over a minute.

(*You'll find that uart printout is one of the longer parts of the simulation
in general.*)

## Analysing results after simulation

The following files will be generated during the sim:

- `bram.dump` - Shows the PC address and instruction being executed. If the sim
hangs without any printing, view this file, as the processor may have hit an
exception etc. Grows in size as the sim runs.

- `bram.snapshot.[NUMBER]`, `verilator.save.[NUMBER]` - Snapshot files of the
contents of bram and verilator model respectively. Can be used to resume the
simulation. The number on the end corresponds to the tick time (i.e.
`bram.snapshot.1999990`/`verilator.save.1999990`). First the verilator model is
loaded, and then the bram contents are loaded. See lines `#65-108` and
`#189-195` of the
[microwatt-verilator.cpp file](https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=refs/heads/verilator_trace).
Pass the tick number on the end of the filename with the '-s' flag:

    (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -s 1999990

You'll get a message like this:

    loading hello_world/hello_world.bin at 0x0 size 0x1888
    loading bram.snapshot.1999990 at 0x0 size 0x10000000
    restored at 1999990

These snapshots are generated at intervals of every 2,000,000 ticks.

- `microwatt-verilator.vcd` - GTKWave waveform file, allowing you to look at
processor signals and transitions during simulation.
Pass `-d` flag to `microwatt-verilator` binary:

    (microwatt):$ ./microwatt-verilator hello_world/hello_world.bin -d

**NOTE**: Trace dumping will generate a large VCD file (about 6GB for the hello
world example)!

If you want GTKWave to load it faster, convert to fst first:

    (microwatt):$ vcd2fst --vcdname=microwatt-verilator.vcd --fstname=microwatt-verilator.fst
    (microwatt):$ gtkwave microwatt-verilator.fst

Fst files are orders-of-magnitude smaller (about 20MB vs 6GB), but are specific
to the GTKWave tool.

## Micropython

The Microwatt repo comes with a pre-compiled
[micropython binary](https://git.libre-soc.org/?p=microwatt.git;a=tree;f=micropython;h=18fa078c8145bdaa75667a0ab04eb0b261245665;hb=refs/heads/verilator_trace)
(version 1.12), which you can try out after confirming 'hello world' works.
Bear in mind, not all features of python will be available. Such as
floating-point numbers.

For micropython to work, you'll need to increase the RAM size in the makefile.
Go to the microwatt-verilator makefile, and comment out the following lines:

    MEMORY_SIZE=8192
    RAM_INIT_FILE=hello_world/hello_world.hex

And uncomment the following:

    MEMORY_SIZE=393216
    RAM_INIT_FILE=micropython/firmware.hex

This will increase the RAM size from 8KiB to 384KiB. The `RAM_INIT_FILE` in
these examples isn't doing anything, however good practice to follow.

Clean up generated files, and recompile:

    (microwatt):$ make clean
    (microwatt):$ make microwatt-verilator

Once the binary has been built, run the same way as before, but point to the
micropython firmware binary:

    (microwatt):$ microwatt-verilator micropython/firmware.bin

On the same system as above, with 1 thread, it took 49 seconds to get to the
micropython shell.

## Verilator runtime commands
A few examples:

    # Show the version of verilator being used
    (microwatt):$ ./microwatt-verilator +verilator+version

## Building `microwatt-verilator` using the Libre-SOC core

In the Makefile, you need to set `EXTERNAL_CORE` to true, and copy the
generated core from soc repo to microwatt. *(If you use a separate chroot to
generate Libre-SOC cores, then you'll need to copy from that chroot to
microwatt chroot from host.*

        cd /path/to/soc
        make microwatt_external_core
        cp external_core_top.v /path/to/microwatt

Then compile verilator sim binary as before:

        cd ~/src/microwatt/
        export FPGA_TARGET=verilator
        export GHDLSYNTH=ghdl
        make microwatt-verilator

## Running Linux kernel

To run Linux on Microwatt, you'll need two binaries:

- The `sdram_init.bin`, which is easy to compile (no additional software
required).

- The `dtbImage.microwatt` device tree Linux kernel. This can be compiled (see
below), or a copy can be downloaded from: <https://ftp.libre-soc.org/dtbImage.microwatt>.

Like with the micropython example, you'll need to increase the RAM size in the
makefile, and recompile the microwatt-verilator binary.

Uncomment the following:

    MEMORY_SIZE=536870912

Which will change the RAM size to 512KiB.

As there is no `dtbImage.microwatt.hex`, you can leave `RAM_INIT_FILE` unchanged.

### Building the kernel - TODO:

*(Please don't build the kernel yourself, until you've tested with the existing kernel linked
above!)*

On a POWER9 there is no need to install gcc-powerpc64le-linux-gnu, 
you can omit CROSS_COMPILE and ARCH in this case

        apt install gcc-powerpc64le-linux-gnu
        apt install flex bison lz4
        git clone -b microwatt-5.7 https://git.kernel.org/pub/scm/linux/kernel/git/joel/microwatt.git
        cd microwatt
        wget https://ftp.libre-soc.org/microwatt-linux-5.7.patch
        patch -p1 < microwatt-linux-5.7.patch
        wget https://ftp.libre-soc.org/rootfs.cpio
        CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt microwatt_defconfig
        CROSS_COMPILE="ccache powerpc64le-linux-gnu-" ARCH=powerpc make -j8 O=microwatt
        
This will produce a file 
        microwatt/arch/powerpc/boot/dtbImage.microwatt

### Building `sdram_init.bin`
This needs gcc-powerpc64le-linux-gnu (already included in the setup step) if
cross compilation is used. It is assumed you're already in `~/src/microwatt/`
directory.

    (microwatt):$ cd litedram/gen-src/sdram_init/
    (microwatt):$ make

The resulting binary will be in the `obj/` directory.

### Running the simulation

Make sure to return back to `src/microwatt/`.

    (microwatt):$ cd ~/src/microwatt/
    (microwatt):$ cp microwatt/arch/powerpc/boot/dtbImage.microwatt
    (microwatt):$ ./microwatt-verilator sdram_init.bin dtbImage.microwatt

This will take some time...

### Sysconn information

TODO WIP integrate from <https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html>

Sysconn is a module which includes information about the SoC, and the info is
printed at the start of the simulation.

### Time benchmarks

`microwatt-verilator` was compiled with 3 threads for faster simulation.

- Time to finish printing Sysconn info: about 1min
- Time to allocate bytes to kernel: ?
- Time to login prompt: about 1 hour
- Time to user shell: ?

### TODO: buildroot

* https://github.com/shenki/buildroot/commits/microwatt
* https://codeconstruct.com.au/docs/microwatt-orangecrab/

## FPGA Development - TODO: Need checking
### Building the bitstring for OrangeCrab

        cd microwatt
        export FPGA_TARGET=ORANGE-CRAB
        export GHDLSYNTH=ghdl
        make microwatt.bit
  
### flashing the bitstring to the OrangeCrab

        make prog # this will run OpenOCD

### Notes for ulx3s

notes for how to compile for ulx3s

    git clone https://github.com/kost/fujprog
    (follow build procedure shown in fujprog README)
    git clone https://git.libre-soc.org/git/microwatt.git
    git checkout -b verilator_trace
    export FPGA_TARGET=ulx3s
    make microwatt.svf
    fujprog microwatt.svf


### Notes for nextpnr-xilinx

superceded: see page [[nextpnr-xilinx]] and devscript 
<https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=nextpnr-xilinx-install;hb=HEAD>

for compiling nextpnr-xilinx and making it useable for nmigen
to compile for the digilent arty-a7-100t, requires a little
futzing around, using the symbiflow version of prjxray-db
instead of the one recommended as a submodule

    git clone https://github.com/gatecat/nextpnr-xilinx
    cd nextpnr-xilinx
    git checkout cd8b15db6ff5c1a7f10a9e
    git submodule init
    git submodule update
    cd xilinx/external
    mv prjxray-db prjxray-db-no
    git clone https://github.com/SymbiFlow/prjxray-db
    cd prjxray-db
    git checkout 0a0addedd73e7
    cp ./artix7/xc7a100t/*.json \
       ./artix7/xc7a100tcsg324-1
    cd ../../..
    cmake -DARCH=xilinx .
    make
    make install
    python3 xilinx/python/bbaexport.py --device xc7a100tcsg324-1 --bba xilinx/xc7a100t.bba
    ./bbasm --l xilinx/xc7a100t.bba xilinx/xc7a100t.bin
    mkdir -p /usr/share/nextpnr/xilinx-chipdb
    cp xilinx/*.bin /usr/share/nextpnr/xilinx-chipdb
    cp -aux xilinx/external/prjxray-db /usr/share/nextpnr

# Additional Useful Info for UART <-> USB Serial Interface Through OrangeCrab's Built-in USB Interface

This uses OrangeCrab's built-in USB interface, rather than needing a
separate USB-serial adapter. see the following for further details:

* <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058800570>
* <https://github.com/antonblanchard/microwatt/pull/347#issuecomment-1058834790>

# running orangecrab-examples before flashing microwatt

See <https://github.com/orangecrab-fpga/orangecrab-hardware/blob/main/contrib/10-orangecrab.rules>

If the OrangeCrab is running in DFU mode, lsusb will show:

    1209:5af0 Generic OrangeCrab r0.2 DFU Bootloader v3.1-6-g62e92e2
    
OrangeCrab has two DFU devices:

    Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=1, name="0x00100000 RISC-V Firmware", serial="UNKNOWN"
    Found DFU: [1209:5af0] ver=0101, devnum=22, cfg=1, intf=0, path="1-4.2", alt=0, name="0x00080000 Bitstream", serial="UNKNOWN"
    
Then clone and patch orangecrab-examples:

        git clone https://github.com/orangecrab-fpga/orangecrab-examples
        patch -p1 < orangecrab-examples.diff
        
To build and flash the example:

        pushd orangecrab-examples/nmigen
        python3 blink.py
        popd
        sudo dfu-util -D orangecrab-examples/nmigen/build/top.bit -a 0