mention page number of vgbbd
[libreriscv.git] / HDL_workflow.mdwn
1 [[!toc ]]
2
3 ---
4
5 # HDL workflow
6
7 This section describes the workflow and some best practices for developing
8 the Libre-SOC hardware. We use nmigen, yosys and symbiyosys, and this
9 page is intended not just to help you get set up, it is intended to
10 help advise you of some tricks and practices that will help you become
11 effective team contributors.
12
13 It is particularly important to bear in mind that we are not just
14 "developing code", here: we are creating a "lasting legacy educational
15 resource" for other people to learn from, and for businesses and students
16 alike to be able to use, learn from and augment for their own purposes.
17
18 It is also important to appreciate and respect that we are funded under
19 NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full
20 transparency, readability, documentation, effective team communication
21 and formal mathematical proofs for all code at all levels is therefore
22 paramount.
23
24 Therefore, we need not only to be "self-sufficient" (absolutely
25 under no circumstances critically reliant on somebody else's servers
26 **or protocols**) we also need to ensure that everything (including
27 all communication such as the mailing list archives) are recorded,
28 replicable, and accessible in perpetuity. Use of slack or a "forum"
29 either actively prevents or makes that much harder.
30
31 # Collaboration resources
32
33 The main message here: **use the right tool for the right job**.
34
35 * mailing list: general communication and discussion.
36 * irc channel #libre-soc on irc.libera.chat: real(ish)-time communication.
37 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
38 * ikiwiki: document store, information store, and (editable) main website
39 * git repositories: code stores (**not binary or auto-generated output store**)
40 * ftp server (<https://ftp.libre-soc.org/>): large (temporary,
41 auto-generated) file store.
42
43 Note also the lack of a "forum" in the above list. this is very
44 deliberate. forums are a serious distraction when it comes to technical
45 heavily goal-orientated development. recent internet users may enjoy
46 looking up the "AOL metoo postings" meme.
47
48 Note also the complete lack of "social platforms". if we wanted to tell
49 everybody how much better each of us are than anyone else in the team,
50 how many times we made a commit (look at me, look at me, i'm so clever),
51 and how many times we went to the bathroom, we would have installed a
52 social media based project "management" system.
53
54 ## Main contact method: mailing list
55
56 To respect the transparency requirements, conversations need to be
57 public and archived (i.e not skype, not telegram, not discord,
58 and anyone seriously suggesting slack will be thrown to the
59 lions). Therefore we have a mailing list. Everything goes through
60 there. <https://lists.libre-soc.org/mailman/listinfo/libre-soc-dev>
61 therefore please do google "mailing list etiquette" and at the very
62 minimum look up and understand the following:
63
64 * This is a technical mailing list with complex topics. Top posting
65 is completely inappropriate. Don't do it unless you have mitigating
66 circumstances, and even then please apologise and explain ("hello sorry
67 using phone at airport flight soon, v. quick reply: ....")
68 * Always trim context but do not cut excessively to the point where people
69 cannot follow the discussion. Especially do not cut the attribution
70 ("On monday xxx wrote") of something that you are actually replying
71 to.
72 * Use inline replies i.e. reply at the point in the relevant part of
73 the conversation, as if you were actually having a conversation.
74 * Follow standard IETF reply formatting, using ">" for cascaded
75 indentation of other people's replies. If using gmail, please: SWITCH
76 OFF RICH TEXT EDITING.
77 * Please for god's sake do not use "my replies are in a different
78 colour". Only old and highly regarded people still using AOL are allowed
79 to get away with that (such as Mitch).
80 * Start a new topic with a relevant subject line. If an existing
81 discussion changes direction, change the subject line to reflect the
82 new topic (or start a new conversation entirely, without using the
83 "reply" button)
84 * DMARC is a pain on the neck. Try to avoid GPG signed messages. sigh.
85 * Don't send massive attachments. Put them online (no, not on facebook or
86 google drive or anywhere else that demands privacy violations) and provide
87 the link. Which should not require any kind of login to access. ask the
88 listadmin if you don't have anywhere suitable: FTP access can be arranged.
89
90 ### Actionable items from mailing list
91
92 If discussions result in any actionable items, it is important not to
93 lose track of them. Create a bugreport, find the discussion in the
94 archives <https://lists.libre-soc.org/pipermail/libre-soc-dev/>,
95 and put the link actually in the bugtracker as one of the comments.
96
97 At some point in any discussion, the sudden realisation may dawn on one
98 or more people that this is an "actionable" discussion. at that point
99 it may become better to use <https://bugs.libre-soc.org/>
100 itself to continue the discussion rather than to keep on dropping copies
101 of links into the bugtracker. The bugtracker sends copies of comments
102 *to* the list however this is 'one-way' (note from lkcl: because this
103 involves running an automated perl script from email, on every email,
104 on the server, that is a high security risk, and i'm not doing it. sorry.)
105
106 ### Mailing list != editable document store
107
108 Also, please do not use the mailing list as an "information or document
109 store or poor-man's editor" **including not sending large images**.
110 We have the wiki for that. Edit a page and
111 tell people what you did (summarise rather than drop the entire contents
112 at the list) and include the link to the page.
113
114 Or, if it is more appropriate, commit a document (or source code)
115 into the relevant git repository then look up the link in the gitweb
116 source tree browser and post that (in the bugtracker or mailing list)
117 See <https://git.libre-soc.org/>
118
119 ### gmail "spam"ifying the list
120
121 See <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
122
123 Basically it is possible to select any message from the list, create a
124 "filter" (under "More"), and, on the 2nd dialog box, click the "never
125 send this to Spam" option.
126
127 ## Bugtracker
128
129 * **LibreSOC bug/task process**: [[libresoc_bug_process]]
130
131 bugzilla. old and highly effective. sign up in the usual way. any
132 problems, ask on the list.
133
134 Please do not ask for the project to be transferred to github or other
135 proprietary nonfree service "because it's soooo convenient", as the
136 lions are getting wind and gout from overfeeding on that one.
137
138 one.
139
140 ## ikiwiki
141
142 Runs the main libre-soc.org site (including this page). effective,
143 stunningly light on resources, and uses a git repository not a database.
144 That means it can be edited offline.
145
146 Usual deal: register an account and you can start editing and contributing
147 straight away.
148
149 Hint: to create a new page, find a suitable page that would link to it,
150 first, then put the link in of the page you want to create, as if the
151 page already exists. Save that page, and you will find a question mark
152 next to the new link you created. click that link, and it will fire up a
153 "create new page" editor.
154
155 Wiki pages are formatted in [[markdown|ikiwiki/markdown]] syntax.
156
157 Hint again: the wiki is backed by a git repository. Don't go overboard
158 but at the same time do not be afraid that you might "damage" or "lose"
159 pages. Although it would be a minor pain, the pages can always be
160 reverted or edited by the sysadmins to restore things if you get in a tiz.
161
162 Assistance in creating a much better theme greatly appreciated. e.g.
163 <http://www.math.cmu.edu/~gautam/sj/blog/20140720-ikiwiki-navbar.html>
164
165 ## git
166
167 We use git. More on this below. We also use
168 [gitolite3](https://gitolite.com/gitolite/) running on a dedicated server.
169 again, it is extremely effective and low resource utilisation. Reminder:
170 lions are involved if github is mentioned.
171
172 [gitweb](https://git.wiki.kernel.org/index.php/Gitweb) is provided which
173 does a decent job. <https://git.libre-soc.org/>
174
175 [Git](https://en.wikipedia.org/wiki/Git) does version control, ie it
176 tracks changes to files so that previous versions can be got back or
177 compared.
178
179 Checklist page [[HDL_workflow/git_checklist]]
180
181 ## ftp server
182
183 <https://ftp.libre-soc.org/> is available for storing large files
184 that do not belong in a git repository, if we have (or ever need)
185 any. Images (etc.) if small and appropriate should go into the
186 wiki, however .tgz archives (etc.) and, at some point, binaries,
187 should be on the ftp server.
188
189 Ask on the list if you have a file that belongs on the ftp server.
190
191 ## server
192
193 As an aside: all this is "old school" and run on a single core 512MB
194 VM with only a 20GB HDD allocation. it costs only 8 GBP per month from
195 mythic-beasts and means that the project is in no way dependent on anyone
196 else - not microsoft, not google, not facebook, not amazon.
197
198 We tried [gitlab](https://about.gitlab.com/). it didn't go well. please
199 don't ask to replace the above extremely resource-efficient services
200 with it.
201
202 # Hardware
203
204 RAM is the biggest requirement. Minimum 16GB, the more the better (32
205 or 64GB starts to reach "acceptable" levels. Disk space is not hugely
206 critical: 256GB SSD should be more than adequate. Simulations and
207 FPGA compilations however are where raw processing power is a must.
208 High end Graphics Cards are nonessential.
209
210 What is particularly useful is to have hi-res screens (curved is
211 *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs
212 going "prism" through long term use), and to have several of them: the
213 more the better. Either a DisplayLink UD160A (or more modern variant)
214 or simply using a second machine (lower spec hardware because it will
215 run editors) is really effective.
216
217 Also it is really recommended to have a UHD monitor (4k - 3840x2160),
218 or at least 2560x1200. If given a choice, 4:3 aspect ratio is better
219 than 16:9 particularly when using several of them. However, caveat
220 (details below): please when editing do not assume that everyone will
221 have access to such high resolution screens.
222
223 # Operating System
224
225 First install and become familiar with
226 [Debian](https://www.debian.org/) ([Ubuntu](https://ubuntu.com/)
227 if you absolutely
228 must) for standardisation cross-team and so that toolchain installation
229 is greatly simplified. yosys in particular warns that trying to use
230 Windows, BSD or MacOS will get you into a world of pain.
231
232 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly
233 sufficient (alongside wicd-gtk for network management). Other more
234 complex desktops can be used however may consume greater resources.
235
236 # editors and editing
237
238 Whilst this is often a personal choice, the fact that many editors are
239 GUI based and run full-screen with the entire right hand side *and* middle
240 *and* the majority of the left side of the hi-res screen entirely unused
241 and bereft of text leaves experienced developers both amused and puzzled.
242
243 At the point where such full-screen users commit code with line lengths
244 well over 160 characters, that amusement quickly evaporates.
245
246 Where the problems occur with full-screen editor usage is when a project
247 is split into dozens if not hundreds of small files (as this one is). At
248 that point it becomes pretty much essential to have as many as six to
249 eight files open *and on-screen* at once, without overlaps i.e. not in
250 hidden tabs, next to at least two if not three additional free and clear
251 terminals into which commands are regularly and routinely typed (make,
252 git commit, nosetests3 etc). Illustrated with the following 3840x2160
253 screenshot (click to view full image), where *every one* of those 80x70
254 xterm windows is *relevant to the task at hand*.
255
256 [[!img 2020-01-24_11-56.png size=640x ]]
257
258 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
259 additionally requiring a mouse click, can save a huge amount of cumulative
260 development time here, switching between editor terminal(s) and the
261 command terminals).
262
263 Once this becomes necessary, it it turn implies that having greater
264 than 80 chars per line - and running editors full-screen - is a severe
265 hinderance to an essential *and highly effective* workflow technique.
266
267 Additionally, care should be taken to respect that not everyone will have
268 200+ column editor windows and the eyesight of a hawk. They may only have
269 a 1280 x 800 laptop which barely fits two 80x53 xterms side by side.
270 Consequently, having excessively long functions is also a hindrance to
271 others, as such developers with limited screen resources would need to
272 continuously page-up and page-down to read the code even of a single
273 function, in full.
274
275 This helps explain in part, below, why compliance with
276 [pep8](https://pep8.org/) is enforced, including its 80 character limit.
277 In short: not everyone has the same "modern" GUI workflow or has access
278 to the same computing resources as you, so please do respect that.
279
280 More on this concept is
281 [here](https://www.linuxjournal.com/content/line-length-limits).
282 Note *very pointedly* that Linus Torvalds *specifically* states that
283 he does not want Linux kernel development to become the exclusive
284 domain of the "wealthy". That means **no** to assumptions about
285 access to ultra-high resolution screens.
286
287 # Software prerequisites<a name="software-prerequisites"></a>
288
289 **Please make sure if you install manually that you install dependencies
290 in strict order. Failing to adhere to this will result in pip3 downloading
291 unauthorised older software versions. See
292 <http://lists.libre-soc.org/pipermail/libre-soc-dev/2021-September/003666.html>**
293
294 Whilst many resources online advocate "`sudo`" in front of all root-level
295 commands below, this quickly becomes tiresome. run "`sudo bash`", get a
296 root prompt, and save yourself some typing.
297
298 * sudo bash
299 * apt-get install vim exuberant-ctags
300 * apt-get install build-essential
301 * apt-get install git python3.7 python3.7-dev python3-nose
302 * apt-get install graphviz xdot gtkwave
303 * apt-get install python3-venv
304 * apt-get install python-virtualenv # this is an alternative to python3-venv
305 * apt-get install tcl-dev libreadline-dev bison flex libffi-dev iverilog
306 * return to user prompt (ctrl-d)
307
308 (The above assumes that you are running Debian.)
309
310 This will get you python3 and other tools that are
311 needed. [graphviz](https://graphviz.org/) is essential
312 for showing the interconnections between cells, and
313 [gtkwave](http://gtkwave.sourceforge.net/) is essential for debugging.
314
315 If you would like to save yourself a lot more typing, check out the
316 [dev-env-setup](https://git.libre-soc.org/?p=dev-env-setup.git;a=summary)
317 repository, examine the scripts there and use them to automate much of
318 the process below.
319
320 If you would like just to install only the apt dependencies use
321 [install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD) instead.
322
323 This page gives more details and a step by step process : [[HDL_workflow/devscripts]]
324
325 ## git
326
327 Look up good tutorials on how to use git effectively. There are so many
328 it is hard to recommend one. This is however essential. If you are not
329 comfortable with git, and you let things stay that way, it will seriously
330 impede development progress.
331
332 If working all day you should expect to be making at least two commits per
333 hour, so should become familiar with it very quickly. If you are *not*
334 doing around 2 commits per hour, something is wrong and you should read
335 the workflow instructions below more carefully, and also ask for advice
336 on the mailing list.
337
338 Worth noting: *this project does not use branches*. All code is committed
339 to master and we *require* that it be either zero-impact additions or that
340 relevant unit tests pass 100%. This ensures that people's work does not
341 get "lost" or isolated and out of touch due to major branch diversion,
342 and that people communicate and coordinate with each other.
343
344 This is not a hard rule: under special cirmstances branches can be useful.
345 They should not however be considered "routine".
346
347 For guidance on when branches are appropriate,
348 see [[HDL_workflow/libresoc_bug_process]].
349
350 For advice on commit messages see the Coding section further down on this page.
351
352 ## yosys
353
354 Follow the source code (git clone) instructions here, do **not** use
355 the "stable" version (do not download the tarball):
356 <https://github.com/YosysHQ/yosys>
357
358 Or, alternatively, use the
359 [hdl-tools-yosys](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-tools-yosys;hb=HEAD)
360 script (which also installs symbiyosys and its dependencies)
361
362 Do not try to use a fixed revision of yosys (currently 0.9), nmigen is
363 evolving and frequently interacts with yosys.
364
365 [Yosys](https://github.com/YosysHQ/yosys is a framework for Verilog RTL.
366 [Verilog](https://en.wikipedia.org/wiki/Verilog) is a hardware description
367 language.
368 RTL [Register Transfer
369 Level](https://en.wikipedia.org/wiki/Register-transfer_level)
370 models how data moves between
371 [registers](https://en.wikipedia.org/wiki/Hardware_register).
372
373 ## symbiyosys
374
375 To install follow the [instructions
376 here](https://symbiyosys.readthedocs.io/en/latest/install.html)
377 Once done look at [A simple BMC
378 example](https://symbiyosys.readthedocs.io/en/latest/quickstart.html)
379
380 You do not have to install all of those (avy, boolector can be left
381 out if desired) however the more that are installed the more effective
382 the formal proof scripts will be (less resource utilisation in certain
383 circumstances).
384
385 [SymbiYosys](https://symbiyosys.readthedocs.io/en/latest/) (sby) is a
386 front-end driver program for Yosys-based formal hardware verification
387 flows.
388
389 ## nmigen (TM)
390
391 *nmigen is a registered trademark of M-Labs <https://uspto.report/TM/88980893>*
392
393 **PLEASE NOTE: it is critical to install nmigen as the first dependency
394 prior to installing any further python-based Libre-SOC HDL repositories.
395 If "pip3 list" shows that nmigen has been auto-installed please remove it**
396
397 [nmigen](https://m-labs.hk/gateware/nmigen/) may be installed as follows:
398
399 * mkdir ~/src
400 * cd !$
401 * git clone https://gitlab.com/nmigen/nmigen.git
402 * cd nmigen
403 * sudo bash
404 * python3 setup.py develop
405 * ctrl-d
406
407 Testing can then be carried out with "python3 setup.py test"
408
409 nmigen is a Python toolbox for building complex digital hardware.
410
411 ## Softfloat and sfpy
412
413 These are a test suite dependency for the
414 [ieee754fpu](https://www.gaisler.com/index.php/products/ipcores/ieee754fpu)
415 library, and will be changed in the future to use Jacob's
416 [simple-soft-float](https://crates.io/crates/simple-soft-float) library.
417 In the meantime, sfpy can be built as follows:
418
419 git clone --recursive https://github.com/billzorn/sfpy.git
420 cd sfpy
421 git apply /path/to/ieee754fpu/sfpy.patch
422 cd SoftPosit
423 git apply ../softposit_sfpy_build.patch
424 git apply /path/to/ieee754fpu/SoftPosit.patch
425 cd ../berkely-softfloat-3
426 # Note: Do not apply the patch included in sfpy for berkely-softfloat,
427 # it contains the same changes as this one
428 git apply /path/to/ieee754fpu/berkeley-softfloat.patch
429 cd ..
430
431 # prepare a virtual environment for building
432 python3 -m venv .env
433
434 # or, if you prefer the old way:
435 # virtualenv -p python3 .env
436
437 # install dependencies
438 source .env/bin/activate
439 pip3 install --upgrade -r requirements.txt
440
441 # build
442 make lib -j$(nproc)
443 make cython
444 make inplace -j$(nproc)
445 make wheel
446
447 # install
448 deactivate # deactivates venv, optional
449 pip3 install dist/sfpy*.whl
450
451 You can test your installation by doing the following:
452
453 python3
454 >>> from sfpy import Posit8
455 >>> Posit8(1.3)
456
457 It should print out `Posit8(1.3125)`
458
459 ## qemu, cross-compilers, gdb
460
461 As we are doing POWER ISA, POWER ISA compilers, toolchains and
462 emulators are required.
463 Again, if you want to save yourself some typing, use the dev scripts.
464 [install-hdl-apt-reqs](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD)
465 script will install the qemu;
466 [ppc64-gdb-gcc](https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=ppc64-gdb-gcc;hb=HEAD)
467 script will install the toolchain and the corresponding debugger.
468 The steps are provided below only for reference; when in doubt,
469 consider checking and running the scripts.
470
471 Install powerpc64 gcc:
472
473 apt-get install gcc-8-powerpc64-linux-gnu
474
475 Install qemu:
476
477 apt-get install qemu-system-ppc
478
479 Install gdb from source. Obtain the required tarball matching
480 the version of gcc (8.3) from here <https://ftp.gnu.org/gnu/gdb/>,
481 unpack it, then:
482
483 cd gdb-8.3 (or other location)
484 mkdir build
485 cd build
486 ../configure --srcdir=.. --host=x86_64-linux --target=powerpc64-linux-gnu
487 make -j$(nproc)
488 make install
489
490 [gdb](https://en.wikipedia.org/wiki/GNU_Debugger) lets you debug running
491 programs. [qemu](https://www.qemu.org/) emulates processors, you can
492 run programs under qemu.
493
494 ## power-instruction-analyzer (pia)
495
496 We have a custom tool built in Rust by programmerjake to help analyze
497 the OpenPower instructions' execution on *actual* hardware.
498
499 Install Rust:
500
501 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
502
503 Make sure we have the correct and up-to-date rust compiler (rustc & cargo):
504
505 rustup default stable
506 rustup update
507
508 Install the Python extension from git source by doing the following:
509
510 git clone https://salsa.debian.org/Kazan-team/power-instruction-analyzer.git pia
511 cd pia
512 ./libre-soc-install.sh
513
514 ## Chips4Makers JTAG
515
516 As this is an actual ASIC, we do not rely on an FPGA's JTAG TAP
517 interface, instead require a full complete independent implementation
518 of JTAG. Staf Verhaegen has one, with a full test suite, and it is
519 superb and well-written. The Libre-SOC version includes DMI (Debug
520 Memory Interface):
521
522 git clone https://git.libre-soc.org/git/c4m-jtag.git/
523 cd c4m-jtag
524 python3 setup.py develop
525
526 Included is an IDCODE tap point, Wishbone Master (for direct memory read
527 and write, fully independent of the core), IOPad redirection and testing,
528 and general purpose shift register capability for any custom use.
529
530 We added a DMI to JTAG bridge in LibreSOC which is
531 directly connected to the core, to access registers and
532 to be able to start and stop the core and change the PC.
533 In combination with the JTAG Wishbone interface the test
534 [ASIC](https://en.wikipedia.org/wiki/Application-specific_integrated_circuit)
535 can have a bootloader uploaded directly into onboard
536 [SRAM](https://en.wikipedia.org/wiki/Static_random-access_memory) and
537 execution begun.
538
539 [Chips4Makers](https://chips4makers.io/) make it possible for makers
540 and hobbyists to make their own open source chips.
541
542 [JTAG](https://en.wikipedia.org/wiki/JTAG) (Joint Test Action Group) is
543 an industry standard for verifying designs and testing printed circuit
544 boards after manufacture.
545
546 The [Wishbone
547 bus](https://en.wikipedia.org/wiki/Wishbone_%28computer_bus%29) is an open
548 source hardware computer bus intended to let the parts of an integrated
549 circuit communicate with each other.
550
551 ## Coriolis2
552 See [[HDL_workflow/coriolis2]] page, for those people doing layout work.
553
554 ## Nextpnr
555
556 A portable FPGA place and route tool.
557
558 See [[HDL_workflow/nextpnr]] page for installation instructions of nextpnr with ECP5 support for Lattice FPGA ECP5 series. Also see
559 [[HDL_workflow/ECP5_FPGA]] for connecting up to JTAG with a ULX3S
560 and the Lattice VERSA_ECP5.
561
562 ## Nextpnr-xilinx
563
564 An open source place and route framework for Xilinx FPGAs using Project Xray. We will use it for Xilinx 7-series FPGAs like Artix-7.
565
566 One of the ways to get Arty A7 100t Digilent FPGA board working.
567
568 See [[HDL_workflow/nextpnr-xilinx]] for installation instructions and dependencies.
569
570
571 ## Verilator
572
573 The fastest Verilog and SystemVerilog simulator. It compiles Verilog to C++ or SystemC.
574
575 Advise use only v4.106 at the moment.
576
577 See [[HDL_workflow/verilator]] page for installation instructions.
578
579 ## GHDL
580
581 GHDL is a shorthand for G Hardware Design Language. It is a VHDL analyzer, compiler, simulator and (experimental) synthesizer that can process (nearly) any VHDL design.
582
583 VHDL is an acronym for Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (HDL), which is a programming language used to describe a logic circuit by function, data flow behavior, or structure.
584
585 Unlike some other simulators, GHDL is a compiler: it directly translates a VHDL file to machine code, without using an intermediary language such as C or C++. Therefore, the compiled code should be faster and the analysis time should be shorter than with a compiler using an intermediary language.
586
587 GHDL aims at implementing VHDL as defined by IEEE 1076. It supports the 1987, 1993 and 2002 revisions and, partially, 2008. PSL is also partially supported.
588
589 See [[HDL_workflow/ghdl]] page for installation instructions.
590
591 ## Icarus Verilog
592
593 Icarus Verilog is a Verilog simulation and synthesis tool. It operates as a compiler, compiling source code written in Verilog (IEEE-1364) into some target format.
594
595 See [[HDL_workflow/iverilog]] page for installation instructions.
596
597 ## Cocotb
598
599 cocotb is a COroutine based COsimulation TestBench environment for verifying VHDL and SystemVerilog RTL using Python.
600
601 See [[HDL_workflow/cocotb]] page for installation instructions.
602
603 ## Symbiflow
604
605 A fully open source toolchain for the development of FPGAs. Currently it targets Xilinx 7-series, Lattice iCE40 and ECP5, Quicklogic EOS S3.
606
607 One way to get the Arty A7 100t Digilent FPGA board working.
608
609 See [[HDL_workflow/symbiflow]] for installation instructions
610 and dependencies.
611
612 ## FPGA/Board Boot-Loaders-Programmers
613
614 Open source FPGA/Board boot-loaders and programmers for ULX3S, ECP5 and
615 OrangeCrab.
616
617 Currently these programs dfu-util, openFPGALoader, ujprog, fujprog,
618 xc3sprog and ecpprog are going to be used.
619
620 See [[HDL_workflow/fpga-boot-loaders-progs]] for installation instructions and dependencies.
621
622 ## ls2 peripheral fabric
623
624 [[HDL_workflow/ls2]]
625
626 # Registering for git repository access<a name="gitolite3_access"></a>
627
628 After going through the onboarding process and having agreed to take
629 responsibility for certain tasks, ask on the mailing list for git
630 repository access, sending in a public key (`id_rsa.pub`). If you do
631 not have one then generate it with `ssh-keygen -t rsa`. You will find it
632 in `~/.ssh`
633
634 NEVER SEND ANYONE THE PRIVATE KEY. By contrast the public key, on
635 account of being public, is perfectly fine to make... err... public.
636
637 Create a file `~/.ssh/config` with the following lines:
638
639 Host git.libre-soc.org
640 Port 922
641
642 Test that you have access with this command:
643
644 ssh -v -p922 gitolite3@git.libre-soc.org
645
646 Please note: **DO NOT TYPE A PASSWORD** - the server gets hit by a lot of
647 port-scanning, and detection of password failures are used to instantly
648 ban IP addresses.
649
650 Wait for the Project Admin to confirm that the ssh key has been added
651 to the required repositories. Once confirmed, you can clone any of the
652 repos at https://git.libre-soc.org/:
653
654 git clone gitolite3@git.libre-soc.org:REPONAME.git
655
656 Alternatively, the .ssh/config can be skipped and this used:
657
658 git clone ssh://gitolite3@git.libre-soc.org:922/REPONAME.git
659
660 Note: **DO NOT ATTEMPT TO LOG IN TO THE SERVER WITH A PERSONAL ACCOUNT**.
661 fail2ban is running and, due to repeated persistent port-scanning spammers
662 is set up to instantly ban any unauthorised ssh access for up to two weeks.
663 This keeps log file sizes down on the server (which is resource-constrained).
664 If you are wondering why this is done, it's a *lot* of port-scans.
665
666 Therefore, *only* ssh in to server with the gitolite3 account, *only*
667 on port 922, and *only* once the systems administrator has given you
668 the all-clear that the ssh key has been added.
669
670 # git configuration
671
672 Although there are methods online which describe how (and why) these
673 settings are normally done, honestly it is simpler and easier to open
674 ~/.gitconfig and add them by hand.
675
676 core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
677 files they don't become a pain. pull.rebase is something that is greatly
678 preferred for this project because it avoids the mess of "multiple
679 extra merge git tree entries", and branch.autosetuprebase=always will,
680 if you want it, always ensure that a new git checkout is set up with rebase.
681
682 [core]
683 autocrlf = input
684 [push]
685 default = simple
686 [pull]
687 rebase = true
688 [branch]
689 autosetuprebase = always
690
691 # Checking out the HDL repositories
692
693 Before running the following, install the
694 dependencies. This is easiest done with this script
695 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=install-hdl-apt-reqs;hb=HEAD>
696
697 **It is critically important to install these in STRICT order, otherwise
698 pip3 interferes and performs unauthorised downloads without informing
699 you of what it is doing**.
700
701 * mkdir ~/src
702 * cd !$
703 * git clone https://gitlab.com/nmigen/nmigen
704 * git clone https://gitlab.com/nmigen/nmigen-boards
705 * git clone https://gitlab.com/nmigen/nmigen-soc
706 * git clone https://gitlab.com/nmigen/nmigen-stdio
707 * git clone gitolite3@git.libre-soc.org:c4m-jtag.git
708 * git clone gitolite3@git.libre-soc.org:nmutil.git
709 * git clone gitolite3@git.libre-soc.org:openpower-isa.git
710 * git clone gitolite3@git.libre-soc.org:ieee754fpu.git
711 * git clone gitolite3@git.libre-soc.org:soc.git
712
713 In each of these directories, **in the order listed**, track down the
714 `setup.py` file, then, as root (`sudo bash`), run the following:
715
716 * python3 setup.py develop
717
718 The reason for using "develop" mode is that the code may be edited
719 in-place yet still imported "globally". There are variants on this theme
720 for multi-user machine use however it is often just easier to get your
721 own machine these days.
722
723 The reason for the order is because soc depends on ieee754fpu, and
724 ieee754fpu depends on nmutil. If you do not follow the listed order
725 pip3 will go off and download an arbitrary version without your
726 consent.
727
728 If "`python3 setup.py install`" is used it is a pain: edit, then
729 install. edit, then install. It gets extremely tedious, hence why
730 "develop" was created.
731
732 If you prefer you can use this script instead: of course you checked it
733 in advance and accept full responsibility.
734 <https://git.libre-soc.org/?p=dev-env-setup.git;a=blob;f=hdl-dev-repos;hb=HEAD>
735
736 # Development Rules
737
738 Team communication:
739
740 * new members, add yourself to the [[about_us]] page and create yourself
741 a home page using someone else's page as a template.
742 * communicate on the mailing list or the bugtracker an intent to take
743 responsibility for a particular task.
744 * assign yourself as the bug's owner
745 * *keep in touch* about what you are doing, and why you are doing it.
746 * edit your home page regularly, particularly to track tasks so that
747 they can be paid by NLNet.
748 * if you cannot do something that you have taken responsibility for,
749 then unless it is a dire personal emergency please say so, on-list. we
750 won't mind. we'll help sort it out.
751
752 Regarding the above it is important that you read, understand, and agree
753 to the [[charter]] because the charter is about ensuring that we operate
754 as an effective organisation. It's *not* about "setting rules and meting
755 out punishment".
756
757 ## Coding
758
759 for actual code development
760
761 ### Copyright Notices
762
763 **All code must have copyright and grant notices (where work was done
764 under budget).**
765
766 * [Example from soc.git repo](https://git.libre-soc.org/?p=soc.git;a=blob;f=src/soc/fu/div/experiment/goldschmidt_div_sqrt.py;h=3f7c2480742d6913859461da120099385f99d18a;hb=HEAD)
767
768 Breakdown of the header in the above example:
769
770 - Code was worked on by Jacob Lifshay during 2022.
771 - Work was done under LibreSOC's Crypto Router
772 [grant](https://libre-soc.org/nlnet_2021_crypto_router/) submitted to NLnet.
773 NLnet grant code is `2021-02-052`.
774 - The NLnet grant was under the
775 [NLnet Assure fund](https://nlnet.nl/assure).
776 - Financial support for NGI Assure comes from European Commission's
777 [Next Generation Internet](https://ngi.eu/) Programme,
778 grant agreement no. 957073.
779
780 Template:
781
782 ```
783 # SPDX-License-Identifier: LGPL-3-or-later
784 # Copyright 202X [Name] [email]
785 #
786 # Funded by NLnet [Programme Name] Programme [202X-YY-ZZZ], [NLnet URL] part
787 # of [EU Programme Name] 202X EU Programme [Programme Number].
788 ```
789
790 ### Plan unit tests
791
792 * plan in advance to write not just code but a full test suite for
793 that code. **this is not optional**. large python projects that do not
794 have unit tests **FAIL** (see separate section below).
795 * Prioritise writing formal proofs and a single clear unit test that is more
796 like a "worked example".
797 We receive NLNet funds for writing formal proofs, plus they
798 cover corner cases and take far less time to write
799
800 ### Commit tested or zero-dependent code
801
802 * only commit code that has been tested (or is presently unused). other
803 people will be depending on you, so do take care not to screw up.
804 not least because, as it says in the [[charter]] it will be your
805 responsibility to fix. that said, do not feel intimidated: ask for help
806 and advice, and you'll get it straight away.
807
808 ### Commit often
809
810 * commit often. several times a day, and "git push" it. this is
811 collaboration. if something is left even overnight uncommitted and not
812 pushed so that other people can see it, it is a red flag.
813 * if you find
814 yourself thinking "i'll commit it when it's finished" or "i don't want to
815 commit something that people might criticise" *this is not collaboration*,
816 it is making yourself a bottleneck. pair-programming is supposed to help
817 avoid this kind of thing however pair-programming is difficult to organise
818 for remote collaborative libre projects (suggestions welcomed here)
819
820 ### Enable editor auto-detection of file changes by external programs
821
822 This is important. "`git pull`" will merge in changes. If you then
823 arbitrarily save a file without re-loading it, you risk destroying
824 other people's work.
825
826 You can avoid damaging the repositories by following some simple procedures:
827
828 run appropriate unit tests
829 git pull
830 run appropriate unit tests again (checks other people's work)
831 git diff # and actually read and review the output
832 git status # check for any missing files
833 git commit # with appropriate arguments and message
834 git push # always always always do this
835
836 ### Absolutely no auto-generated output
837
838 * **do not commit autogenerated output**. write a shell script and commit
839 that, or add a `Makefile` to run the command that generates the output, but
840 **do not** add the actual output of **any** command to the repository.
841 ever. this is really important. even if it is a human-readable file
842 rather than a binary object file.
843 * it is very common to add PDFs (the result of running `latex2pdf`) or
844 configure.in (the result of running `automake`), they are an absolute
845 nuisance and interfere hugely with git diffs, as well as waste hard
846 disk space *and* network bandwidth. don't do it.
847 * do not add multi-megabyte or multi-gigabyte "test data".
848 use shell scripts and commit that, which automatically downloads the
849 "test data" from a well-known known-good reliable location instead.
850
851 ### Write commands that do tasks and commit those
852
853 * if the command needed to create any given autogenerated output is not
854 currently in the list of known project dependencies, first consult on
855 the list if it is okay to make that command become a hard dependency of
856 the project (hint: java, node.js php and .NET commands may cause delays
857 in response time due to other list participants laughing hysterically),
858 and after a decision is made, document the dependency and how its source
859 code is obtained and built (hence why it has to be discussed carefully)
860 * if you find yourself repeating commands regularly, chances are high
861 that someone else will need to run them, too. clearly this includes
862 yourself, therefore, to make everyone's lives easier including your own,
863 put them into a `.sh` shell script (and/or a `Makefile`), commit them to
864 the repository and document them at the very minimum in the README,
865 INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
866 ask on the mailing list for advice.
867
868 ### Keep commits single-purpose
869
870 * edit files making minimal *single purpose* modifications (even if
871 it involves multiple files. Good extreme example: globally changing
872 a function name across an entire codebase is one purpose, one commit,
873 yet hundreds of files. miss out one of those files, requiring multiple
874 commits, and it actually becomes a nuisance).
875
876 ### Run unit tests prior to commits
877
878 * prior to committing make sure that relevant unit tests pass, or that
879 the change is a zero-impact addition (no unit tests fail at the minimum)
880
881 ### Do not break existing code
882
883 * keep working code working **at all times**. find ways to ensure that
884 this is the case. examples include writing alternative classes that
885 replace existing functionality and adding runtime options to select
886 between old and new code.
887
888 ### Small commits with relevant commit message
889
890 * commit no more than around 5 to 10 lines at a time, with a CLEAR message
891 (no "added this" or "changed that").
892 * if as you write you find that the commit message involves a *list* of
893 changes or the word "and", then STOP. do not proceed: it is a "red flag"
894 that the commit has not been properly broken down into separate-purpose
895 commits. ask for advice on-list on how to proceed.
896
897 ### *Git commit message format*
898
899 * Additional articles on commit messages
900 [here](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
901 and
902 [here](https://github.com/torvalds/subsurface-for-dirk/blob/master/README.md#contributing)
903
904 LibreSOC message format based on description given in
905 [bug #1126#c40](https://bugs.libre-soc.org/show_bug.cgi?id=1126#c40):
906
907 1. Every commit MUST start with a short title, up to 50 characters.
908 2. The commit title MUST contain either subsystem, or a file path,
909 or a subsystem/path, or a subsystem/subsubsystem combination, which got
910 modified or introduced, and a short summary. These parts must be separated
911 by the colon.
912 3. A good rule is to imagine that the short message begins with
913 "if this patch is applied, it will". For example, a good title is
914 "X: update Y", not "updated Y in X".
915 4. After the title, there must be an empty line, which documents the
916 changes. The limit is 72 characters per line.
917 5. The long description can be omitted if the short description provides
918 enough information or if the commit itself is simple enough.
919
920 Example:
921
922 ```
923 subsystem/file.py: document usage
924
925 Here goes the long description, which explains everything. First of all,
926 we stick to limit of 72 characters. Then, perhaps, we'd like to explain
927 the rationale in more details.
928 ```
929
930 It is suggested to stick to common sense whenever choosing subsystem names
931 or files or long descriptions.
932
933 Primary concerns are:
934
935 1. short titles
936 2. short summaries
937 3. wording for the first line
938
939 The rest is up for the committers.
940
941 ### Exceptions to small commit: atomic single purpose commit
942
943 * if it is essential to commit large amounts of code, ensure that it
944 is **not** in use **anywhere** by any other code. then make a *small*
945 (single purpose) followup commit which actually puts that code into use.
946
947 This last rule is kinda flexible, because if you add the code *and* add
948 the unit test *and* added it into the main code *and* ran all relevant
949 unit tests on all cascade-impacted areas by that change, that's perfectly
950 fine too. however if it is the end of a day, and you need to stop and
951 do not have time to run the necessary unit tests, do *not* commit the
952 change which integrates untested code: just commit the new code (only)
953 and follow up the next day *after* running the full relevant unit tests.
954
955 ### Why such strict rules?
956
957 The reason for all the above is because python is a dynamically typed
958 language. make one tiny change at the base level of the class hierarchy
959 and the effect may be disastrous.
960
961 It is therefore worth reiterating: make absolutely certain that you *only*
962 commit working code or zero-impact code.
963
964 Therefore, if you are absolutely certain that a new addition (new file,
965 new class, new function) is not going to have any side-effects, committing
966 it (a large amount of code) is perfectly fine.
967
968 As a general rule, however, do not use this an an excuse to write code
969 first then write unit tests as an afterthought. write *less* code *in
970 conjunction* with its (more basic) unit tests, instead. then, folliw up with
971 additions and improvements.
972
973 The reason for separating out commits to single purpose only becomes
974 obvious (and regretted if not followed) when, months later, a mistake
975 has to be tracked down and reverted. if the commit does not have an
976 easy-to-find message, it cannot even be located, and once found, if the
977 commit confuses several unrelated changes, not only the diff is larger
978 than it should be, the reversion process becomes extremely painful.
979
980 ### PHP-style python format-strings
981
982 As the name suggests, "PHP-style" is not given as a compliment.
983 Format-strings - `f"{variable} {pythoncodefragment}" are a nightmare
984 to read. The lesson from PHP, Zope and Plone: when code is embedded,
985 the purpose of the formatting - the separation of the format from
986 the data to be placed in it - is merged, and consequently become
987 unreadable.
988
989 By contrast, let us imagine a situation where 12 variables need to
990 be inserted into a string, four of which are the same variablename:
991
992 x = "%s %s %s %s %s %s %s %s %s %s %s %s" % (var1, var2, var3,
993 var3, var4, var2,
994 var1, var9, var1,
995 var3, var4, var1)
996
997 This is just as unreadable, but for different reasons. Here it *is*
998 useful to do this as:
999
1000 x = f"{var1} {var2} {var3}" \
1001 ...
1002 f"{var3} {var4} {var1}"
1003
1004 As a general rule, though, format-specifiers should be strongly
1005 avoided, given that they mix even variable-names directly inside
1006 a string.
1007
1008 This additionally gives text editors (and online web syntax
1009 highlighters) the opportunity to colour syntax-highlight the
1010 ASCII string (the format) from the variables to be inserted *into*
1011 that format. gitweb for example (used by this project) cannot
1012 highlight string-formatted code.
1013
1014 It turns out that colour is processed by the **opposite** hemisphere
1015 of the brain from written language. Thus, colour-syntax-highlighting
1016 is not just a "nice-to-have", it's **vital** for easier and faster
1017 identification of context and an aid to rapid understanding.
1018
1019 Anything that interferes with that - such as python format-strings -
1020 has to take a back seat, regardless of its perceived benefits.
1021
1022 **If you absolutely must** use python-format-strings, **only** do
1023 so by restricting to variables. Create temporary variables if you
1024 have to.
1025
1026 y = '/'.join(a_list)
1027 x = f"{y}"
1028
1029 ### PEP8 format
1030
1031 * all code needs to conform to pep8. use either pep8checker or better
1032 run autopep8. however whenever committing whitespace changes, *make a
1033 separate commit* with a commit message "whitespace" or "autopep8 cleanup".
1034 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if
1035 you think you need greater than 80 chars, it *fundamentally* indicates
1036 poor code design. split the code down further into smaller classes
1037 and functions.
1038
1039 ### Docstring checker
1040
1041 * TBD there is a docstring checker. at the minimum make sure to have
1042 an SPD license header, module header docstring, class docstring and
1043 function docstrings on at least non-obvious functions.
1044
1045 ### Clear code commenting and docstrings
1046
1047 * make liberal but not excessive use of comments. describe a group of
1048 lines of code, with terse but useful comments describing the purpose,
1049 documenting any side-effects, and anything that could trip you or other
1050 developers up. unusual coding techniques should *definitely* contain
1051 a warning.
1052
1053 ### Only one class per module (ish)
1054
1055 * unless they are very closely related, only have one module (one class)
1056 per file. a file only 25 lines long including imports and docstrings
1057 is perfectly fine however don't force yourself. again, if unsure,
1058 ask on-list.
1059
1060 ### File and Directory hierarchy
1061
1062 * *keep files short and simple*. see below as to why
1063 * create a decent directory hierarchy but do not go mad. ask for advice
1064 if unsure
1065
1066 ### No import star!
1067
1068 * please do not use "from module import \*". it is extremely bad practice,
1069 causes unnecessary resource utilisation, makes code readability and
1070 tracking extremely difficult, and results in unintended side-effects.
1071
1072 Example: often you want to find the code from which a class was imported.
1073 nirmally you go to the top of the file, check the imports, and you know
1074 exactly which file has the class because of the import path. by using
1075 wildcards, you have absolutely *no clue* which wildcard imported which
1076 class or classes.
1077
1078 Example: sometimes you may accidentally have duplicate code maintained
1079 in two or more places. editing one of them you find, puzzlingly, that
1080 the code behaves in some files with the old behaviour, but in others it
1081 works. after a massive amount of investigation, you find that the working
1082 files happen to have a wildcard import of the newer accidental duplicate
1083 class **after** the wildcard import of the older class with exactly the
1084 same name. if you had used explicit imports, you would have spotted
1085 the double import of the class from two separate locations, immediately.
1086
1087 Really. don't. use. wildcards.
1088
1089 More about this here:
1090
1091 * <https://www.asmeurer.com/removestar/>
1092 * <https://rules.sonarsource.com/python/RSPEC-2208>
1093
1094 ### Keep file and variables short but clear
1095
1096 * try to keep both filenames and variable names short but not ridiculously
1097 obtuse. an interesting compromise on imports is "from ridiculousfilename
1098 import longsillyname as lsn", and to assign variables as well: "comb =
1099 m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
1100 that can reduce code indentation by 6 characters without reducing clarity.
1101
1102 Additionally, use comments just above an obtuse variable in order to
1103 help explain what it is for. In combination with keeping the the module
1104 itself short, other readers will not need to scroll back several pages
1105 in order to understand the code.
1106
1107 Yes it is tempting to actually use the variables as
1108 self-explanatory-comments and generally this can be extremely good
1109 practice. the problem comes when the variable is so long that a function
1110 with several parameters csn no longer fit on a single line, and takes
1111 up five to ten lines rather than one or two. at that point, the length
1112 of the code is adversely affected and thus so is readability by forcing
1113 readers to scroll through reams of pages.
1114
1115 It is a tricky balance: basically use your common sense, or just ask
1116 someone else, "can you understand this code?"
1117
1118 ### Reasons for code structure
1119
1120 Regarding code structure: we decided to go with small modules that are
1121 both easy to analyse, as well as fit onto a single page and be readable
1122 when displayed as a visual graph on a full UHD monitor. this is done
1123 as follows:
1124
1125 * using the capability of nmigen (TODO crossref to example) output the
1126 module to a yosys ilang (.il) file
1127 * in a separate terminal window, run yosys
1128 * at the yosys prompt type "read_ilang modulename.il"
1129 * type "show top" and a graphviz window should appear. note that typing
1130 show, then space, then pressing the tab key twice will give a full list
1131 of submodules (one of which will be "top")
1132
1133 You can now fullsize the graphviz window and scroll around. if it looks
1134 reasonably obvious at 100% zoom, i.e the connections can be clearly
1135 related in your mind back to the actual code (by matching the graph names
1136 against signals and modules in the original nmigen code) and the words are
1137 not tiny when zoomed out, and connections are not total incomprehensible
1138 spaghetti, then congratulations, you have well-designed code. If not,
1139 then this indicates a need to split the code further into submodules
1140 and do a bit more work.
1141
1142 The reasons for doing a proper modularisation job are several-fold:
1143
1144 * firstly, we will not be doing a full automated layout-and-hope
1145 using alliance/coriolis2, we will be doing leaf-node thru tree node
1146 half-automated half-manual layout, finally getting to the floorplan,
1147 then revising and iteratively adjusting.
1148 * secondly, examining modules at the gate level (or close to it) is just
1149 good practice. poor design creeps in by *not* knowing what the tools
1150 are actually doing (word to experienced developers: yes, we know that
1151 the yosys graph != final netlist).
1152 * thirdly, unit testing, particularly formal proofs, is far easier on
1153 small sections of code, and complete in a reasonable time.
1154
1155 ## Special warning / alert to vim users!
1156
1157 Some time around the beginning of 2019 some bright spark decided that
1158 an "auto-recommend-completion-of-stuff" option would be a nice, shiny
1159 idea to enable by default from that point onwards.
1160
1161 This incredibly annoying "feature" results in tabs (or spaces) being
1162 inserted "on your behalf" when you press return on one line, for your
1163 "convenience" of not needing to type lots of spaces/tabs just to get
1164 to the same indentation level.
1165
1166 Of course, this "feature", if you press return on one line in edit
1167 mode and then press "escape", leaves a bundle-of-joy extraneous
1168 whitespace **exactly** where you don't want it, and didn't ask for it,
1169 pooped all over your file.
1170
1171 Therefore, *please*: **before** running "git commit", get into the
1172 habit of always running "git diff", and at the very minimum
1173 speed-skim the entire diff, looking for tell-tale "red squares"
1174 (these show up under bash diff colour-syntax-highlighting) that
1175 inform you that, without your knowledge or consent, vim has
1176 "helpfully" inserted extraneous whitespace.
1177
1178 Remove them **before** git committing because they are not part
1179 of the actual desired code-modifications, and committing them
1180 is a major and constant distraction for reviewers about actual
1181 important things like "the code that actually *usefully* was
1182 modified for that commit"
1183
1184 This has the useful side-effect of ensuring that, right before
1185 the commit, you've got the actual diff right in front of you
1186 in the xterm window, on which you can base the "commit message".
1187
1188 ## Unit tests
1189
1190 For further reading, see the wikipedia page on
1191 [Test-driven Development](https://en.wikipedia.org/wiki/Test-driven_development)
1192
1193 This deserves its own special section. It is extremely important to
1194 appreciate that without unit tests, python projects are simply unviable.
1195 Python itself has over 25,000 individual tests.
1196
1197 This can be quite overwhelming to a beginner developer, especially one
1198 used to writing scripts of only 100 lines in length.
1199
1200 Thanks to Samuel Falvo we learned that writing unit tests as a formal
1201 proof is not only shorter, it's also far more readable and also, if
1202 written properly, provides 100% coverage of corner-cases that would
1203 otherwise be overlooked or require tens to hundreds of thousands of
1204 tests to be run.
1205
1206 No this is not a joke or even remotely hypothetical, this is an actual
1207 real-world problem.
1208
1209 The ieee754fpu requires several hundreds of thousands of tests to be
1210 run (currently needing several days to run them all), and even then we
1211 cannot be absolutely certain that all possible combinations of input have
1212 been tested. With 2^128 permutations to try with 2 64 bit FP numbers
1213 it is simply impossible to even try.
1214
1215 This is where formal proofs come into play.
1216
1217 Samuel illustrated to us that "ordinary" unit tests can then be written
1218 to *augment* the formal ones, serving the purpose of illustrating how
1219 to use the module, more than anything.
1220
1221 However it is appreciated that writing formal proofs is a bit of a
1222 black art. This is where team collaboration particularly kicks in,
1223 so if you need help, ask on the mailing list.
1224
1225 ## Don't comment out unit tests: add them first (as failures) and fix code later
1226
1227 Unit tests serve an additional critical purpose of keeping track of code
1228 that needs to be written. In many cases, you write the unit test *first*,
1229 despite knowing full well that the code doesn't even exist or is completely
1230 broken. The unit test then serves as a constant and important reminder
1231 to actually fix (or write) the code.
1232
1233 Therefore, *do not* comment out unit tests just because they "don't work".
1234 If you absolutely must stop a unit test from running, **do not delete it**.
1235 Simply mark it with an appropriate
1236 ["skip" decorator](https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures),
1237 preferably with a link to a URL in the [bugtracker](https://bugs.libre-soc.org/)
1238 with further details as to why the unit test should not be run.
1239
1240 ## Liskov Substitution Principle
1241
1242 * [Wikipedia entry](https://en.wikipedia.org/wiki/Liskov_substitution_principle)
1243 * [Stackoverflow answer on LSP](https://stackoverflow.com/questions/56860/what-is-an-example-of-the-liskov-substitution-principle)
1244 * [Article on LSP](https://reflectoring.io/lsp-explained/)
1245
1246 Copying from Wikipedia:
1247
1248 ```
1249 The Liskov substitution principle (LSP) is a particular definition of a
1250 subtyping relation, called strong behavioral subtyping, that was initially
1251 introduced by Barbara Liskov in a 1987 conference keynote address titled
1252 Data abstraction and hierarchy. It is based on the concept of
1253 "substitutability" - a principle in object-oriented programming stating
1254 that an object (such as a class) may be replaced by a sub-object (such as
1255 a class that extends the first class) without breaking the program.
1256 It is a semantic rather than merely syntactic relation, because it intends
1257 to guarantee semantic interoperability of types in a hierarchy, object
1258 types in particular.
1259 ```
1260
1261 To paraphrase: an *original object/class may be replaced with another object*
1262 (whose class extends the first class) *without breaking* the program.
1263
1264 Python is a programming language which makes using LSP pretty straightforward.
1265
1266 In LibreSOC, we aim to follow this principle whenever possible and (bearing
1267 time and budget constraints).
1268
1269 *(Luke, please include some examples from LibreSOC source here)*
1270
1271 ## Principle of Least Astonishment/Surprise (POLA)
1272
1273 * Example shown by Luke on
1274 [comment #33 bug #1039](https://bugs.libre-soc.org/show_bug.cgi?id=1039#c33)
1275 * [Wikipedia entry](https://en.wikipedia.org/wiki/Principle_of_least_astonishment)
1276 * Example answer on
1277 [software eng. stack exchange](https://softwareengineering.stackexchange.com/questions/187457/what-is-the-principle-of-least-astonishment)
1278
1279 Wikipedia mentions that the origin of the term "Principle of Least
1280 Astonishment" (or Surprise) comes from a PL/I programming language bulletin
1281 board from 1957:
1282
1283 ```
1284 For those parts of the system which cannot be adjusted to the peculiarities
1285 of the user, the designers of a systems programming language should obey
1286 the "Law of Least Astonishment." In short, this law states that every construct
1287 in the system should behave exactly as its syntax suggests. Widely accepted
1288 conventions should be followed whenever possible, and exceptions to previously
1289 established rules of the language should be minimal.
1290 ```
1291
1292 If a method name has a prefix `test_` it should be a unit test (or some other
1293 test which is there to check the functionality of a given feature).
1294
1295 A method/function (or attributes/variables, etc.) *shouldn't* be given a name
1296 which is wildly different from the what the user would expect it to do.
1297
1298 *Andrey*: One example which may meet this rule is `TestIssuer`,
1299 which is the FSM-based nMigen HDL CPU core used by LibreSOC (so far).
1300 It is so called because it is used to issue unit tests (while also
1301 being synthesisable). This name was however was initially confusing
1302 to me, because my background is in hardware not software engineering.
1303
1304
1305 # Task management guidelines
1306
1307 * New guide for RfP submission (in-progress):
1308 [[HDL_workflow/rfp_submission_guide]]
1309
1310 (This section needs to be compared with [[HDL_workflow/libresoc_bug_process]])
1311
1312 1. Create the task in appropriate "Product" section with appropriate
1313 "Component" section. Most code tasks generally use "Libre-SOC's
1314 first SOC".
1315 2. Fill in "Depends on" and "Blocks" section whenever appropriate.
1316 Also add as many related ("See Also") links to other bugreports
1317 as possible. bugreports are never isolated.
1318 3. Choose the correct task for a budget allocation. Usually the parent
1319 task is used.
1320 4. Choose the correct NLnet milestone. The best practice is to check
1321 the parent task for a correct milestone.
1322 5. Assign the budget to the task in `"USER=SUM"` form, where "USER"
1323 corresponds to your username and "SUM" corresponds to the actual
1324 budget in EUR. There may be multiple users.
1325 6. When the task is completed, you can begin writing an RFP.
1326 **DO NOT submit it without explicit authorisation and review**.
1327 Leave out your bank and personal address details if you prefer
1328 when sending to the Team Manager for review.
1329 7. Once the RFP is written, notify the Team Manager and obtain their
1330 explicit approval to send it.
1331 8. Once approval is received and the RFP sent, update the `"USER=SUM"`
1332 field to include the submitted date:
1333 `"USER={amount=SUM, submitted=SDATE}"`. The SDATE is entered in
1334 `YYYY-MM-DD` form.
1335 9. Once the task is paid, again notify the Team Manager (IRC is fine),
1336 and update `"USER={amount=SUM, submitted=SDATE}"`
1337 to `"USER={amount=SUM, submitted=SDATE, paid=PDATE}"`. The PDATE is
1338 entered in `YYYY-MM-DD` form, too.
1339
1340 Throughout all of this you should be using budget-sync to check the
1341 database consistency
1342 <https://git.libre-soc.org/?p=utils.git;a=blob;f=README.txt;hb=HEAD>
1343
1344 [[!img bugzilla_RFP_fields.jpg size=640x ]]
1345
1346 # TODO Tutorials
1347
1348 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
1349
1350 * Robert Baruch's nmigen tutorials look really good:
1351 <https://github.com/RobertBaruch/nmigen-tutorial>
1352 * Although a verilog example this is very useful to do
1353 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
1354 * This tutorial looks pretty good and will get you started
1355 <https://web.archive.org/web/20210123052724/http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install>
1356 and walks not just through simulation, it takes you through using
1357 gtkwave as well.
1358 * There exist several nmigen examples which are also executable
1359 <https://gitlab.com/nmigen/nmigen/tree/master/examples/> exactly as
1360 described in the above tutorial (python3 filename.py -h)
1361 * More nmigen tutorials at [[learning_nmigen]]