sync_up: Discussion, add rest of points from thread

[libreriscv.git] / docs / firststeps.mdwn

[[!toc ]]

---

# Introduction

This tutorial intends to shed some light at the first steps for a newcomer.
Note that this tutorial is a work in progress; feel free to update it.
This tutorial assumes that the environment and and repositories are set.
The information on environment can be found at
[[HDL workflow]] page, and
an easy set of commands is at the [[HDL_workflow/devscripts]] page

In this tutorial, we will perform these steps:

0. Checking we're all ready and set.
1. Running the first test and observing its results.
2. Updating the test so that it enables/disables some functions.

Note that this tutorial is ISA-centric, since the idea of adding it
was born by the times there were several newcomer-oriented tasks around
the decoder. Many of key concepts, however, are not ISA-specific, so
there is a hope that this guide can be helpful for other tasks.

# Checking we're all ready and set

Once we established the environment and cloned openpower-isa repository,
we can run the first test we'll use as an example to observe.
Since we're starting from decoder, we will work with openpower-isa
repository. If you followed the HDL guidelines strictly, you can chroot
into libresoc environment and work with the repository immediately,
by changing the directory to `${HOME}/src/openpower-isa`.

    schroot -c libresoc /bin/bash
    cd "${HOME}/src/openpower-isa"

If for some reason `openpower-isa` repository is not yet in your `src`
subdirectory, you need to clone the relevant repository. The recommended
way to do it is to use
[hdl-dev-repos](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-dev-repos;hb=HEAD)
script.

The environment is quite newcomer-friendly: the test we intend to dissect
is implemented in pure Python, so there is no need to build anything before
running the test. All we need is a good ol' Python; however, before proceeding,
make sure that Python is at least of version 3.7.2.

    python3 --version

If the version is less than 3.7.2, consider re-visiting the
[HDL workflow](https://libre-soc.org/HDL_workflow/) page and checking that all
preparations are done correctly. Again, the preferred way is using simple
scripts to keep the life easy. In this case, the script of interest is
[install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD).

# Running the very first test

Once we have the repository and the needed Python version, we can launch
our first test suite (assuming we're already inside openpower-isa
repository):

    python3 src/openpower/decoder/isa/test_caller.py > /tmp/log

The test takes a while to complete; once it's ready, you should observe that
it exits with success, and all tests from suite report no errors. In case
some tests report failures, consider raising a question in #libre-soc on
irc.libera.chat, or in
[mailing list](https://lists.libre-soc.org/mailman/listinfo/libre-soc-dev).
If you choose the latter, again, consider visiting
[HDL workflow](https://libre-soc.org/HDL_workflow/) page, to be aware
of mailing lists etiquette and preferences. Note, however, that some tests
can be skipped; this is acceptable.

If everything works fine, that's a good sign; let's proceed to the first
routine we'd like to dissect.

# Dissecting the test of interest

The script we launched contained several tests; amongst others, it checks
that addition instruction works correctly. Since we're going to look through
this instruction, let's for this time disable other tests so that we would
run only test related to addition.
Let's edit the test_caller.py script and mark all tests except addition one
as those which must be skipped. This goal can be achieved by two ways.

1. The first option is to rename all methods in `DecoderTestCase` class
so that the names start with prefix other than `test_`.
2. Alternatively, and this is the recommended way, tests can be temporarily
disabled via @unittest.skip decorator applied to the method. This method
is mentioned in HDL workflow as well (section 10.4).

Regardless of the option considered, `test_add`, which we're looking at,
should be kept intact. Once all tests but `test_add` are renamed or skipped,
re-run the test:

    python3 src/openpower/decoder/isa/test_caller.py > /tmp/log

This time the suite should complete much faster. Let's observe the log and
look for a line `call add add`. This line comes from
`src/openpower/decoder/isa/caller.py`, and gives us an important information:
we're calling an `add` instruction, and its assembly mnemonic is `add` as well.

So far so good; we dropped other tests, and now look at the test for `add`
instruction. Now we're ready to check how the instruction behaves.

# A quick look at ADD instruction test

Let's return to the test and the logs. What the test for `add` instruction does?
For reader's convenience, the overall `test_add` code is duplicated here:

```
    def test_add(self):
        lst = ["add 1, 3, 2"]
        initial_regs = [0] * 32
        initial_regs[3] = 0x1234
        initial_regs[2] = 0x4321
        with Program(lst, bigendian=False) as program:
            sim = self.run_tst_program(program, initial_regs)
            self.assertEqual(sim.gpr(1), SelectableInt(0x5555, 64))
```

What do we see here? First of all, we have an assembly listing, consisting of
exactly one instruction, `"add 1, 3, 2"`. Then we establish the initial values
for registers we're going to work with. After that, we instantiate the program
with the assembly listing, execute the program, and compare the state of the
first general-purpose register (aka GPR) with some predefined value (64-bit
integer with value 0x5555).

Now let's turn to logs to see how they correspond to what we see.
The lines of interest are `reading reg RA 3 0` and `reading reg RB 2 0`.
These, unsurprisingly, are two registers (`3` and `2`) which act as
input parameters to `add` instruction; the result is then placed as
an output into register `1`.

Note that the initial values for registers `3` and `2` are `0x1234` and `0x4321`
respectively, and this matches to the input parameters in the logs:

```
inputs [SelectableInt(value=0x1234, bits=64), SelectableInt(value=0x4321, bits=64)]
```

The simulator performs the actual computation, obtaining the result, and then
updates the general-purpose register we used as an output parameter:

```
results (SelectableInt(value=0x5555, bits=64),)
writing gpr 1 SelectableInt(value=0x5555, bits=64) 0
```

In the end, we see that our assertion indeed passes:

```
__eq__ SelectableInt(value=0x5555, bits=64) SelectableInt(value=0x5555, bits=64)
```

You can play around the test, e.g. modify the input/output registers (there are
32 GPRs, so there's a plethora of combinations possible). In the next chapter,
we're going to take a deeper look and cover some bits of implementation.

# Diving into the instruction execution

One of interesting aspects we saw in the previous chapters is that whenever
the test executes (or, more precisely, simulates) some instructions, there's
some logic beneath these actions. Let's take a look at execution flow; for
now, we won't dive into that many details, but rather take a quick look.
We already saw that there are some logs coming from the
`src/openpower/decoder/isa/caller.py` script; but how do we end up there?
By the time of writing, all tests inside `test_caller.py` use internal
method called `run_tst_program`. This method, in turn, calls `run_tst`,
and this is the place where the magic happens. In `process` nested function,
we actually simulate that our instructions are executed one-by-one, literally
calling `yield from simulator.execute_one()`. And this method inside the
simulator instance belongs to
[`src/openpower/decoder/isa/caller.py`](https://git.libre-soc.org/?p=openpower-isa.git;a=blob;f=src/openpower/decoder/isa/caller.py;hb=HEAD)
script.
Inside `execute_one` method, the most crucial part is
`yield from self.call(opname)` call; and, if we take a look inside of the
`call` method, we will see that, aside of the aforementioned log
(`log("call", ins_name, asmop)`), this function also takes care of the rest
of the magic. This includes a lot of manipulations before and after executing
instruction, but the crucial part is quite simple:

```
        # execute actual instruction here (finally)
        log("inputs", inputs)
        results = info.func(self, *inputs)
        log("results", results)
```

The part of the most interest is `info.func` call; we'll take a look at it
in the next chapter.

# Markdown enters the scene

If we investigate the `info.func` object, we'll discover that, in case of our
`add` instruction, it is actually `fixedarith.op_add` function. Where does it
come from?

This function is defined inside `fixedarith.py`, which is generated from the
corresponding Markdown file, `fixedarith.mdwn`. On the first glance, one can
find an idea to generate the code from the documentation somewhat surprising;
however, this is perfectly reasonable, since we try to stay as close to the
documentation as possible. Some Markdown files, like `fixedarith.mdwn`,
contain the pseudocode in exactly the same form as it is present in the ISA
docs. Let's take a look at `fixedarith.mdwn` and find our instruction.

As we can see in `# Add` section, there are four `add` variants:

* add RT,RA,RB (OE=0 Rc=0)
* add.  RT,RA,RB (OE=0 Rc=1)
* addo RT,RA,RB (OE=1 Rc=0)
* addo.  RT,RA,RB (OE=1 Rc=1)

All variants are covered in the relevant OpenPOWER ISA. As of time of writing,
the most recent edition is 3.0C, and it is available here:

https://ftp.libre-soc.org/PowerISA_public.v3.0C.pdf

The instructions of our interest can be found exactly where we'd expect them
to be found, at chapter `3.3: Fixed-Point Facility Instructions`. We can see
here the instruction encoding, as well as its form (XO) and all information
we already found in `fixedarith.mdwn` file.

All these details are available in some form as part of `info` variable we
observed before (at `caller.py` file). The `info` variable is, actually,
a simple instance of named tuple; the overall structure of this named tuple
is left as an exercise for the reader.

# Modifying the pseudocode

We won't dive into all gory details of the decoder, at least for now. What
is important for us at this stage is, how can we affect the generated code?
As we saw recently, the markdown file is somehow converted into the Python
code; but how is the conversion done?

This is done by the script called pywriter. Again, we omit the exact details,
since this script clearly deserves its own documentation; the only crucial
information that pywriter script uses PLY (hint: Python-Lex-Yacc) in order
to get the work done, and, thanks to PLY magic, it is able to convert ISA
pseudocode into something more Pythonic.

For illustrative purposes, let's modify the pseudocode so that, instead of
addition, it performs a subtraction. Let's find the pseudocode for `add`
instructions inside `fixedarith.mdwn`...

```
Pseudo-code:

    RT <- (RA) + (RB)
```

and modify it to:


```
Pseudo-code:

    RT <- (RA) - (RB)
```

OK, we changed the pseudocode inside the Markdown, but it hasn't yet changed
a word inside the corresponding Python file (`fixedarith.py`). If we attempt
to re-run the unit test, nothing happens, since Python code is kept intact.

In order to force the source code re-generation, `pywriter noall fixedarith`
command is used. Once `pywriter` completes its task, we can re-run the test,
and it should **fail**. The logs show a different result.

Work out by hand what 0x1234 - 0x4321 in 64-bit unsigned arithmetic is, and
change the assert in the unit test to match. Re-run: the test should now
**pass**. These manipulations, again, are left as an exercise for the reader.

This chapter concludes the First Steps tutorial. We'll take a deeper look
at pseudocode generation when we will dive into an example of modifying
BCD instructions.