(no commit message)
[libreriscv.git] / HDL_workflow.mdwn
1 # HDL workflow
2
3 This section describes the workflow and some best practices for developing
4 the Libre-SOC hardware. We use nmigen, yosys and symbiyosys, and this
5 page is intended not just to help you get set up, it is intended to
6 help advise you of some tricks and practices that will help you become
7 effective team contributors.
8
9 It is particularly important to bear in mind that we are not just
10 "developing code", here: we are creating a "lasting legacy educational
11 resource" for other people to learn from, and for businesses and students
12 alike to be able to use, learn from and augment for their own purposes.
13
14 It is also important to appreciate and respect that we are funded under
15 NLNet's Privacy and Enhanced Trust Programme <http://nlnet.nl/PET>. Full
16 transparency, readability, documentation, effective team communication
17 and formal mathematical proofs for all code at all levels is therefore
18 paramount.
19
20 # Collaboration resources
21
22 The main message here: **use the right tool for the right job**.
23
24 * mailing list: general communication and discussion.
25 * bugtracker: task-orientated, goal-orientated *focussed* discussion.
26 * ikiwiki: document store, information store, and (editable) main website
27 * git repositories: code stores (**not binary or auto-generated output store**)
28 * ftp server (<http://ftp.libre-riscv.org>): large file store.
29
30 we will add an IRC channel at some point when there are enough people
31 to warrant having one.
32
33 note also the lack of a "forum" in the above list. this is very deliberate. forums are a serious distraction when it comes to technical heavily goal-orientated development. recent internet users may enjoy looking up the "AOL metoo postings" meme.
34
35 note also the complete lack of "social platforms". if we wanted to tell everybody how much better each of us are than anyone else in the team, how many times we made a commit (look at me, look at me, i'm so clever), and how many times we went to the bathroom, we would have installed a social media based project "management" system.
36
37 ## Main contact method: mailing list
38
39 To respect the transparency requirements, conversations need to be
40 public and archived (i.e not skype, not telegram, not discord,
41 and anyone seriously suggesting slack will be thrown to the
42 lions). Therefore we have a mailing list. Everything goes through
43 there. <http://lists.libre-riscv.org/mailman/listinfo/libre-riscv-dev>
44 therefore please do google "mailing list etiquette" and at the very
45 minimum look up and understand the following:
46
47 * This is a technical mailing list with complex topics. Top posting
48 is completely inappropriate. Don't do it unless you have mitigating
49 circumstances, and even then please apologise and explain ("hello sorry
50 using phone at airport flight soon, v. quick reply: ....")
51 * Always trim context but do not cut excessively to the point where people
52 cannot follow the discussion. Especially do not cut the attribution
53 ("On monday xxx wrote") of something that you are actually replying
54 to.
55 * Use inline replies i.e. reply at the point in the relevant part of
56 the conversation, as if you were actually having a conversation.
57 * Follow standard IETF reply formatting, using ">" for cascaded
58 indentation of other people's replies. If using gmail, please: SWITCH
59 OFF RICH TEXT EDITING.
60 * Please for god's sake do not use "my replies are in a different
61 colour". Only old and highly regarded people still using AOL are allowed
62 to get away with that (such as Mitch).
63 * Start a new topic with a relevant subject line. If an existing
64 discussion changes direction, change the subject line to reflect the
65 new topic (or start a new conversation entirely, without using the
66 "reply" button)
67 * DMARC is a pain on the neck. Try to avoid GPG signed messages. sigh.
68 * Don't send massive attachments. Put them online (no, not on facebook or
69 google drive or anywhere else that demands privacy violations) and provide
70 the link. Which should not require any kind of login to access. ask the
71 listadmin if you don't have anywhere suitable: FTP access can be arranged.
72
73 ### Actionable items from mailing list
74
75 If discussions result in any actionable items, it is important not to
76 lose track of them. Create a bugreport, find the discussion in the
77 archives <http://lists.libre-riscv.org/pipermail/libre-riscv-dev/>,
78 and put the link actually in the bugtracker as one of the comments.
79
80 At some point in any discussion, the sudden realisation may dawn on one
81 or more people that this is an "actionable" discussion. at that point
82 it may become better to use <http://bugs.libre-riscv.org>
83 itself to continue the discussion rather than to keep on dropping copies
84 of links into the bugtracker. The bugtracker sends copies of comments
85 *to* the list however this is 'one-way' (note from lkcl: because this
86 involves running an automated perl script from email, on every email,
87 on the server, that is a high security risk, and i'm not doing it. sorry.)
88
89 ### Mailing list != editable document store
90
91 Also, please do not use the mailing list as an "information or document
92 store or poor-man's editor". We have the wiki for that. Edit a page and
93 tell people what you did (summarise rather than drop the entire contents
94 at the list) and include the link to the page.
95
96 Or, if it is more appropriate, commit a document (or source code)
97 into the relevant git repository then look up the link in the gitweb
98 source tree browser and post that (in the bugtracker or mailing list)
99 See <http://git.libre-riscv.org>
100
101 ### gmail "spam"ifying the list
102
103 see <https://blog.kittycooper.com/2014/05/keeping-my-mailing-list-emails-out-of-gmails-spam-folder/>
104
105 basically it is possible to select any message from the list, create a "filter" (under "More"),
106 and, on the 2nd dialog box, click the "never send this to Spam" option.
107
108 ## Bugtracker
109
110 bugzilla. old and highly effective. sign up in the usual way. any
111 problems, ask on the list.
112
113 Please do not ask for the project to be transferred to github or other
114 proprietary nonfree service "because it's soooo convenient", as the
115 lions are getting wind and gout from overfeeding on that one.
116
117 ## ikiwiki
118
119 Runs the main libre-riscv.org site (including this page). effective,
120 stunningly light on resources, and uses a git repository not a database.
121 That means it can be edited offline.
122
123 Usual deal: register an account and you can start editing and contributing
124 straight away.
125
126 Hint: to create a new page, find a suitable page that would link to it,
127 first, then put the link in of the page you want to create, as if the
128 page already exists. Save that page, and you will find a questionmark
129 next to the new link you created. click that link, and it will fire up a
130 "create new page" editor.
131
132 Hint again: the wiki is backed by a git repository. Don't go overboard
133 but at the same time do not be afraid that you might "damage" or "lose"
134 pages. Although it would be a minor pain, the pages can always be
135 reverted or edited by the sysadmins to restore things if you get in a tiz.
136
137 Assistance in creating a much better theme greatly appreciated. e.g.
138 <http://www.math.cmu.edu/~gautam/sj/blog/20140720-ikiwiki-navbar.html>
139
140 ## git
141
142 we use git. more on this below. we also use gitolite3 running on a
143 dedicated server. again, it is extremely effective and low resource
144 utilisation. reminder: lions are involved if github is mentioned.
145
146 gitweb is provided which does a decent job. <http://git.libre-riscv.org>
147
148 ## ftp server
149
150 <http://ftp.libre-riscv.org> is available for storing large files
151 that do not belong in a git repository, if we have (or ever need)
152 any. images (etc.) if small and appropriate should go into the
153 wiki, however .tgz archives (etc.) and, at some point, binaries,
154 should be on the ftp server.
155
156 ask on the list if you have a file that belongs on the ftp server.
157
158 ## server
159
160 as an aside: all this is "old school" and run on a single core 512MB
161 VM with only a 20GB HDD allocation. it costs only 8 GBP per month from
162 mythic-beasts and means that the project is in no way dependent on anyone
163 else - not microsoft, not google, not facebook, not amazon.
164
165 we tried gitlab. it didn't go well. please don't ask to replace the
166 above extremely resource-efficient services with it.
167
168 # Hardware
169
170 RAM is the biggest requirement. Minimum 16GB, the more the better (32
171 or 64GB starts to reach "acceptable" levels. Disk space is not hugely
172 critical: 256GB SSD should be more than adequate. Simulations and
173 FPGA compilations however are where raw processing power is a must.
174 High end Graphics Cards are nonessential.
175
176 What is particularly useful is to have hi-res screens (curved is
177 *strongly* recommended if the LCD is over 24in wide, to avoid eyeballs
178 going "prism" through longterm use), and to have several of them: the
179 more the better. Either a DisplayLink UD160A (or more modern variant)
180 or simply using a second machine (lower spec hardware because it will
181 run editors) is really effective.
182
183 Also it is really recommended to have a UHD monitor (4k - 3840x2160),
184 or at least 2560x1200. If given a choice, 4:3 aspect ratio is better
185 than 16:9 particularly when using several of them. However, caveat
186 (details below): please when editing do not assume that everyone will
187 have access to such high resolution screens.
188
189 # Operating System
190
191 First install and become familiar with Debian (ubuntu if you absolutely
192 must) for standardisation cross-team and so that toolchain installation
193 is greatly simplified. yosys in particular warns that trying to use
194 Windows, BSD or MacOS will get you into a world of pain.
195
196 Only a basic GUI desktop is necessary: fvwm2, xfce4, lxde are perfectly
197 sufficient (alongside wicd-gtk for network management). Other more
198 complex desktops can be used however may consume greater resources.
199
200 # editors and editing
201
202 Whilst this is often a personal choice, the fact that many editors are
203 GUI based and run fullscreen with the entire right hand side *and* middle
204 *and* the majority of the left side of the hi-res screen entirely unused
205 and bereft of text leaves experienced developers both amused and puzzled.
206
207 At the point where such fullscreen users commit code with line lengths
208 well over 160 characters, that amusement quickly evaporates.
209
210 Where the problems occur with fullscreen editor usage is when a project
211 is split into dozens if not hundreds of small files (as this one is). At
212 that point it becomes pretty much essential to have as many as six to
213 eight files open *and on-screen* at once, without overlaps i.e. not in
214 hidden tabs, next to at least two if not three additional free and clear
215 terminals into which commands are regularly and routinely typed (make,
216 git commit, nosetests3 etc). Illustrated with the following 3840x2160
217 screenshot (click to view full image), where *every one* of those 80x70
218 xterm windows is *relevant to the task at hand*.
219
220 [[!img 2020-01-24_11-56.png size=640x ]]
221
222 (hint/tip: fvwm2 set up with "mouse-over to raise focus, rather than
223 additionally requiring a mouseclick, can save a huge amount of cumulative
224 development time here, switching between editor terminal(s) and the
225 command terminals).
226
227 Once this becomes necessary, it it turn implies that having greater
228 than 80 chars per line - and running editors fullscreen - is a severe
229 hindance to an essential *and highly effective* workflow technique.
230
231 Additionally, care should be taken to respect that not everyone will have
232 200+ column editor windows and the eyesight of a hawk. They may only have
233 a 1280 x 800 laptop which barely fits two 80x53 xterms side by side.
234 Consequently, having excessively long functions is also a hindrance to
235 others, as such developers with limited screen resources would need to
236 continuously page-up and page-down to read the code even of a single
237 function, in full.
238
239 This helps explain in part, below, why compliance with pep8 is enforced,
240 including its 80 character limit. In short: not everyone has the same
241 "modern" GUI workflow or has access to the same computing resources as
242 you, so please do respect that.
243
244 # Software prerequisites
245
246 Whilst many resources online advocate "sudo" in front of all root-level
247 commands below, this quickly becomes tiresome. run "sudo bash", get a
248 root prompt, and save yourself some typing.
249
250 * sudo bash
251 * apt-get install vim exuberant-ctags
252 * apt-get install build-essential
253 * apt-get install git python3.7 python3.7-dev python-nosetest3
254 * apt-get install graphviz xdot gtkwave
255 * return to user prompt (ctrl-d)
256
257 This will get you python3 and other tools that are needed. graphviz is
258 essential for showing the interconnections between cells, and gtkwave
259 is essential for debugging.
260
261 ## git
262
263 Look up good tutorials on how to use git effectively. There are so many
264 it is hard to recommend one. This is however essential. If you are not
265 comfortable with git, and you let things stay that way, it will seriously
266 impede development progress.
267
268 If working all day you should expect to be making at least two commits per
269 hour, so should become familiar with it very quickly. If you are *not*
270 doing around 2 commits per hour, something is wrong and you should read
271 the workflow instructions below more carefully, and also ask for advice
272 on the mailing list.
273
274 Worth noting: *this project does not use branches*. All code is committed
275 to master and we *require* that it be either zero-impact additions or that
276 relevant unit tests pass 100%. This ensures that people's work does not
277 get "lost" or isolated and out of touch due to major branch diversion,
278 and that people communicate and coordinate with each other.
279
280 ## yosys
281
282 Follow the source code (git clone) instructions here:
283 <http://www.clifford.at/yosys/download.html>
284
285 Do not try to use a fixed revision (currently 0.9), nmigen is evolving
286 and frequently interacts with yosys
287
288 ## symbiyosys
289
290 Follow the instructions here:
291 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#installing>
292
293 You do not have to install all of those (avy, boolector can be left
294 out if desired) however the more that are installed the more effective
295 the formal proof scripts will be (less resource utilisation in certain
296 circumstances).
297
298 ## nmigen
299
300 nmigen may be installed as follows:
301
302 * mkdir ~/src
303 * cd !$
304 * git clone https://github.com/m-labs/nmigen.git
305 * cd nmigen
306 * sudo bash
307 * python3 setup.py develop
308 * ctrl-d
309
310 testing can then be carried out with "python3 setup.py test"
311
312 ## Softfloat and sfpy
313
314 These are a test suite dependency for the ieee754fpu library, and
315 will be changed in the future to use Jacob's algorithmic numeric
316 library. In the meantime, sfpy can be built as follows:
317
318 git clone --recursive https://github.com/billzorn/sfpy.git
319 cd sfpy
320 cd SoftPosit
321 git apply ../softposit_sfpy_build.patch
322 git apply /path/to/ieee754fpu/SoftPosit.patch
323 cd ../berkely-softfloat-3
324 # Note: Do not apply the patch included in sfpy for berkely-softfloat,
325 # it contains the same changes as this one
326 git apply /path/to/ieee754fpu/berkeley-softfloat.patch
327 cd ..
328
329 # install dependencies
330 python3 -m venv .env
331 . .env/bin/activate
332 pip3 install --upgrade -r requirements.txt
333
334 # build
335 make lib -j8
336 make cython
337 make inplace -j8
338 make wheel
339
340 # install
341 deactivate # deactivates venv, optional
342 pip3 install dist/sfpy*.whl
343
344 You can test your installation by doing the following:
345
346 python3
347 >>> from sfpy import *
348 >>> Posit8(1.3)
349
350 It should print out `Posit8(1.3125)`
351
352 ## Coriolis2
353
354 See [[coriolis2]] page, for those people doing layout work.
355
356 # Registering for git repository access
357
358 After going through the onboarding process and having agreed to take
359 responsibility for certain tasks, ask on the mailing list for git
360 repository access, sending in a public key (id_rsa.pub). If you do
361 not have one then generate it with ssh-keygen -t rsa. You will find it
362 in ~/.ssh
363
364 NEVER SEND ANYONE THE PRIVATE KEY. By contrast the public key, on
365 account of being public, is perfectly fine to make... err... public.
366
367 Create a file ~/.ssh/config with the following lines:
368
369 Host git.libre-riscv.org
370 Port 922
371
372 Wait for the Project Admin to confirm that the ssh key has been added
373 to the required repositories. Once confirmed, you can clone any of the
374 repos at http://git.libre-riscv.org:
375
376 git clone gitolite3@git.libre-riscv.org:REPONAME.git
377
378 Alternatively, the .ssh/config can be skipped and this used:
379
380 git clone ssh://gitolite3@git.libre-riscv.org:922/REPONAME.git
381
382 # git configuration
383
384 Although there are methods online which describe how (and why) these
385 settings are normally done, honestly it is simpler and easier to open
386 ~/.gitconfig and add them by hand.
387
388 core.autocrlf is a good idea to ensure that anyone adding DOS-formatted
389 files they don't become a pain. pull.rebase is something that is greatly
390 preferred for this project because it avoids the mess of "multiple
391 extra merge git tree entries", and branch.autosetuprebase=always will,
392 if you want it, always ensure that a new git checkout is set up with rebase.
393
394 [core]
395 autocrlf = input
396 [push]
397 default = simple
398 [pull]
399 rebase = true
400 [branch]
401 autosetuprebase = always
402
403 # Checking out the HDL repositories
404
405 * mkdir ~/src
406 * cd !$
407 * git clone gitolite3@git.libre-riscv.org:nmutil.git
408 * git clone gitolite3@git.libre-riscv.org:ieee754fpu.git
409 * git clone gitolite3@git.libre-riscv.org:soc.git
410
411 In each of these directories, in the order listed, track down the
412 setup.py file, then, as root (sudo bash), run the following:
413
414 * python3 setup.py develop
415
416 The reason for using "develop" mode is that the code may be edited
417 in-place yet still imported "globally". There are variants on this theme
418 for multi-user machine use however it is often just easier to get your
419 own machine these days.
420
421 The reason for the order is because soc depends on ieee754fpu, and
422 ieee754fpu depends on nmutil
423
424 If "python3 setup.py install" is used it is a pain: edit, then
425 install. edit, then install. It gets extremely tedious, hence why
426 "develop" was created.
427
428 # Development Rules
429
430 team communication:
431
432 * new members, add yourself to the [[about_us]] page and create yourself a home page using someone else's page as a template.
433 * communicate on the mailing list or the bugtracker an intent to take
434 responsibility for a particular task.
435 * assign yourself as the bug's owner
436 * *keep in touch* about what you are doing, and why you are doing it.
437 * edit your home page regularly, particularly to track tasks so that they can be paid by NLNet.
438 * if you cannot do something that you have taken responsibility for,
439 then unless it is a dire personal emergency please say so, on-list. we
440 won't mind. we'll help sort it out.
441
442 regarding the above it is important that you read, understand, and agree
443 to the [[charter]] because the charter is about ensuring that we operate
444 as an effective organisation. It's *not* about "setting rules and meting
445 out punishment".
446
447 ## Coding
448
449 for actual code development
450
451 ### Plan unit tests
452
453 * plan in advance to write not just code but a full test suite for
454 that code. **this is not optional**. large python projects that do not
455 have unit tests **FAIL** (see separate section below).
456 * Prioritise writing formal proofs and a single clear unit test that is more like a "worked example".
457 We receive NLNet funds for writing formal proofs, plus they
458 cover corner cases and take far less time to write
459
460 ### Commit tested or zero-dependent code
461
462 * only commit code that has been tested (or is presently unused). other
463 people will be depending on you, so do take care not to screw up.
464 not least because, as it says in the [[charter]] it will be your
465 responsibility to fix. that said, do not feel intimidated: ask for help
466 and advice, and you'll get it straight away.
467
468 ### Commit often
469
470 * commit often. several times a day, and "git push" it. this is
471 collaboration. if something is left even overnight uncommitted and not
472 pushed so that other people can see it, it is a red flag. if you find
473 yourself thinking "i'll commit it when it's finished" or "i don't want to
474 commit something that people might criticise" *this is not collaboration*,
475 it is making yourself a bottleneck. pair-programming is supposed to help
476 avoid this kind of thing however pair-programming is difficult to organise
477 for remote collaborative libre projects (suggestions welcomed here)
478
479 ### Absolutely no auto-generated output
480
481 * **do not commit autogenerated output**. write a shell script and commit
482 that, or add a Makefile to run the command that generates the output, but
483 **do not** add the actual output of **any** command to the repository.
484 ever. this is really important. even if it is a human-readable file
485 rather than a binary object file.
486 it is very common to add pdfs (the result of running latex2pdf) or configure.in (the result of running automake), they are an absolute nuisance and interfere hugely with git diffs, as well as waste hard disk space *and* network bandwidth. don't do it.
487
488 ### Write commands that do tasks and commit those
489
490 * if the command needed to create any given autogenerated output is not
491 currently in the list of known project dependencies, first consult on
492 the list if it is okay to make that command become a hard dependency of
493 the project (hint: java, node.js php and .NET commands may cause delays
494 in response time due to other list participants laughing hysterically),
495 and after a decision is made, document the dependency and how its source
496 code is obtained and built (hence why it has to be discussed carefully)
497 * if you find yourself repeating commands regularly, chances are high
498 that someone else will need to run them, too. clearly this includes
499 yourself, therefore, to make everyone's lives easier including your own,
500 put them into a .sh shell script (and/or a Makefile), commit them to
501 the repository and document them at the very minimum in the README,
502 INSTALL.txt or somewhere in a docs folder as appropriate. if unsure,
503 ask on the mailing list for advice.
504
505 ### Keep commits single-purpose
506
507 * edit files making minimal *single purpose* modifications (even if
508 it involves multiple files. Good extreme example: globally changing
509 a function name across an entire codebase is one purpose, one commit,
510 yet hundreds of files. miss out one of those files, requiring multiple
511 commits, and it actually becomes a nuisance).
512
513 ### Run unit tests prior to commits
514
515 * prior to committing make sure that relevant unit tests pass, or that
516 the change is a zero-impact addition (no unit tests fail at the minimum)
517
518 ### Do not break existing code
519
520 * keep working code working **at all times**. find ways to ensure that this is the case. examples include writing alternative classes that replace existing functionality and adding runtime optiobs to select between old and new code.
521
522 ### Small commits with relevant commit message
523
524 * commit no more than around 5 to 10 lines at a time, with a CLEAR message
525 (no "added this" or "changed that").
526 * if as you write you find that the commit message involves a *list* of
527 changes or the word "and", then STOP. do not proceed: it is a "red flag"
528 that the commit has not been properly broken down into separate-purpose
529 commits. ask for advice on-list on how to proceed.
530
531 ### Exceptions to small commit: atomic single purpose commit
532
533 * if it is essential to commit large amounts of code, ensure that it
534 is **not** in use **anywhere** by any other code. then make a *small*
535 (single purpose) followup commit which actually puts that code into use.
536
537 this last rule is kinda flexible, because if you add the code *and* add
538 the unit test *and* added it into the main code *and* ran all relevant
539 unit tests on all cascade-impacted areas by that change, that's perfectly
540 fine too. however if it is the end of a day, and you need to stop and
541 do not have time to run the necessary unit tests, do *not* commit the
542 change which integrates untested code: just commit the new code (only)
543 and follow up the next day *after* running the full relevant unit tests.
544
545 ### Why such strict rules?
546
547 the reason for all the above is because python is a weakly typed language.
548 make one tiny change at the base level of the class hierarchy and the
549 effect may be disastrous.
550
551 it is therefore worth reiterating: make absolutely certain that you *only*
552 commit working code or zero-impact code.
553
554 therefore, if you are absolutely certain that a new addition (new file,
555 new class, new function) is not going to have any side-effects, committing
556 it (a large amount of code) is perfectly fine.
557
558 as a general rule, however, do not use this an an excuse to write code
559 first then write unit tests as an afterthought. write *less* code *in
560 conjunction* with its (more basic) unit tests, instead. then, folliw up with
561 additions and improvements.
562
563 the reason for separating out commits to single purpose only becomes
564 obvious (and regretted if not followed) when, months later, a mistake
565 has to be tracked down and reverted. if the commit does not have an
566 easy-to-find message, it cannot even be located, and once found, if the
567 commit confuses several unrelated changes, not only the diff is larger
568 than it should be, the reversion process becomes extremely painful.
569
570 ### PEP8 format
571
572 * all code needs to conform to pep8. use either pep8checker or better
573 run autopep8. however whenever committing whitespace changes, *make a
574 separate commit* with a commit message "whitespace" or "autopep8 cleanup".
575 * pep8 REQUIRES no more than 80 chars per line. this is non-negotiable. if
576 you think you need greater than 80 chars, it *fundamentally* indicates
577 poor code design. split the code down further into smaller classes
578 and functions.
579
580 ### Docstring checker
581
582 * TBD there is a docstring checker. at the minimum make sure to have
583 an SPD license header, module header docstring, class docstring and
584 function docstrings on at least non-obvious functions.
585
586 ### Clear code commenting and docstrings
587
588 * make liberal but not excessive use of comments. describe a group of
589 lines of code, with terse but useful comments describing the purpose,
590 documenting any side-effects, and anything that could trip you or other
591 developers up. unusual coding techniques should *definitely* contain
592 a warning.
593
594 ### Only one class per module (ish)
595
596 * unless they are very closely related, only have one module (one class)
597 per file. a file only 25 lines long including imports and docstrings
598 is perfectly fine however don't force yourself. again, if unsure,
599 ask on-list.
600
601 ### File and Directory hierarchy
602
603 * *keep files short and simple*. see below as to why
604 * create a decent directory hierarchy but do not go mad. ask for advice
605 if unsure
606
607 ### No import star!
608
609 * please do not use "from module import \*". it is extremely bad practice,
610 causes unnecessary resource utilisation, makes code readability and
611 tracking extremely difficult, and results in unintended side-effects.
612
613 ### Keep file and variables short but clear
614
615 * try to keep both filenames and variable names short but not ridiculously
616 obtuse. an interesting compromise on imports is "from ridiculousfilename
617 import longsillyname as lsn", and to assign variables as well: "comb =
618 m.d.comb" followed by multiple "comb += nmigen_stmt" lines is a good trick
619 that can reduce code indentation by 6 characters without reducing clarity.
620
621 ### Reasons for code structure
622
623 regarding code structure: we decided to go with small modules that are
624 both easy to analyse, as well as fit onto a single page and be readable
625 when displayed as a visual graph on a full UHD monitor. this is done
626 as follows:
627
628 * using the capability of nmigen (TODO crossref to example) output the
629 module to a yosys ilang (.il) file
630 * in a separate terminal window, run yosys
631 * at the yosys prompt type "read_ilang modulename.il"
632 * type "show top" and a graphviz window should appear. note that typing
633 show, then space, then pressing the tab key twice will give a full list
634 of submodules (one of which will be "top")
635
636 you can now fullsize the graphviz window and scroll around. if it looks
637 reasonably obvious at 100% zoom, i.e the connections can be clearly
638 related in your mind back to the actual code (by matching the graph names
639 against signals and modules in the original nmigen code) and the words are
640 not tiny when zoomed out, and connections are not total incomprehensible
641 spaghetti, then congratulations, you have well-designed code. If not,
642 then this indicates a need to split the code further into submodules
643 and do a bit more work.
644
645 The reasons for doing a proper modularisation job are several-fold:
646
647 * firstly, we will not be doing a full automated layout-and-hope
648 using alliance/coriolis2, we will be doing leaf-node thru tree node
649 half-automated half-manual layout, finally getting to the floorplan,
650 then revising and iteratively adjusting.
651 * secondly, examining modules at the gate level (or close to it) is just
652 good practice. poor design creeps in by *not* knowing what the tools
653 are actually doing (word to experienced developers: yes, we know that
654 the yosys graph != final netlist).
655 * thirdly, unit testing, particularly formal proofs, is far easier on
656 small sections of code, and complete in a reasonable time.
657
658 ## Special warning / alert to vim users!
659
660 Some time around the beginning of 2019 some bright spark decided that
661 an "auto-recommend-completion-of-stuff" option would be a nice, shiny
662 idea to enable by default from that point onwards.
663
664 This incredibly annoying "feature" results in tabs (or spaces) being
665 inserted "on your behalf" when you press return on one line, for your
666 "convenience" of not needing to type lots of spaces/tabs just to get
667 to the same indentation level.
668
669 Of course, this "feature", if you press return on one line in edit
670 mode and then press "escape", leaves a bundle-of-joy extraneous
671 whitespace **exactly** where you don't want it, and didn't ask for it,
672 pooped all over your file.
673
674 Therefore, *please*: **before** running "git commit", get into the
675 habit of always running "git diff", and at the very minimum
676 speed-skim the entire diff, looking for tell-tale "red squares"
677 (these show up under bash diff colour-syntax-highlighting) that
678 inform you that, without your knowledge or consent, vim has
679 "helpfully" inserted extraneous whitespace.
680
681 Remove them **before** git committing because they are not part
682 of the actual desired code-modifications, and committing them
683 is a major and constant distraction for reviewers about actual
684 important things like "the code that actually *usefully* was
685 modified for that commit"
686
687 This has the useful side-effect of ensuring that, right before
688 the commit, you've got the actual diff right in front of you
689 in the xterm window, on which you can base the "commit message".
690
691 ## Unit tests
692
693 This deserves its own special section. It is extremely important to
694 appreciate that without unit tests, python projects are simply unviable.
695 Python itself has over 25,000 individual tests.
696
697 This can be quite overwhelming to a beginner developer, especially one
698 used to writing scripts of only 100 lines in length.
699
700 Thanks to Samuel Falvo we learned that writing unit tests as a formal
701 proof is not only shorter, it's also far more readable and also, if
702 written properly, provides 100% coverage of corner-cases that would
703 otherwise be overlooked or require tens to hundreds of thousands of
704 tests to be run.
705
706 No this is not a joke or even remotely hypothetical, this is an actual
707 real-world problem.
708
709 The ieee754fpu requires several hundreds of thousands of tests to be
710 run (currently needing several days to run them all), and even then we
711 cannot be absolutely certain that all possible combinations of input have
712 been tested. With 2^128 permutations to try with 2 64 bit FP numbers
713 it is simply impossible to even try.
714
715 This is where formal proofs come into play.
716
717 Samuel illustrated to us that "ordinary" unit tests can then be written
718 to *augment* the formal ones, serving the purpose of illustrating how
719 to use the module, more than anything.
720
721 However it is appreciated that writing formal proofs is a bit of a
722 black art. This is where team collaboration particularly kicks in,
723 so if you need help, ask on the mailing list.
724
725 # TODO Tutorials
726
727 Find appropriate tutorials for nmigen and yosys, as well as symbiyosys.
728
729 * Although a verilog example this is very useful to do
730 <https://symbiyosys.readthedocs.io/en/latest/quickstart.html#first-step-a-simple-bmc-example>
731 * This tutorial looks pretty good and will get you started
732 <http://blog.lambdaconcept.com/doku.php?id=nmigen:nmigen_install> and
733 walks not just through simulation, it takes you through using gtkwave
734 as well.
735 * There exist several nmigen examples which are also executable
736 <https://github.com/m-labs/nmigen/tree/master/examples/> exactly as
737 described in the above tutorial (python3 filename.py -h)
738 * Robert Baruch's nmigen tutorials look really good:
739 <https://github.com/RobertBaruch/nmigen-tutorial>