src/add/singlepipe.py

   1 """ Pipeline and BufferedHandshake implementation, conforming to the same API.
   2     For multi-input and multi-output variants, see multipipe.
   3
   4     eq:
   5     --
   6
   7     a strategically very important function that is identical in function
   8     to nmigen's Signal.eq function, except it may take objects, or a list
   9     of objects, or a tuple of objects, and where objects may also be
  10     Records.
  11
  12     Stage API:
  13     ---------
  14
  15     stage requires compliance with a strict API that may be
  16     implemented in several means, including as a static class.
  17     the methods of a stage instance must be as follows:
  18
  19     * ispec() - Input data format specification
  20                 returns an object or a list or tuple of objects, or
  21                 a Record, each object having an "eq" function which
  22                 takes responsibility for copying by assignment all
  23                 sub-objects
  24     * ospec() - Output data format specification
  25                 requirements as for ospec
  26     * process(m, i) - Processes an ispec-formatted object
  27                 returns a combinatorial block of a result that
  28                 may be assigned to the output, by way of the "eq"
  29                 function
  30     * setup(m, i) - Optional function for setting up submodules
  31                 may be used for more complex stages, to link
  32                 the input (i) to submodules.  must take responsibility
  33                 for adding those submodules to the module (m).
  34                 the submodules must be combinatorial blocks and
  35                 must have their inputs and output linked combinatorially.
  36
  37     Both StageCls (for use with non-static classes) and Stage (for use
  38     by static classes) are abstract classes from which, for convenience
  39     and as a courtesy to other developers, anything conforming to the
  40     Stage API may *choose* to derive.
  41
  42     StageChain:
  43     ----------
  44
  45     A useful combinatorial wrapper around stages that chains them together
  46     and then presents a Stage-API-conformant interface.  By presenting
  47     the same API as the stages it wraps, it can clearly be used recursively.
  48
  49     RecordBasedStage:
  50     ----------------
  51
  52     A convenience class that takes an input shape, output shape, a
  53     "processing" function and an optional "setup" function.  Honestly
  54     though, there's not much more effort to just... create a class
  55     that returns a couple of Records (see ExampleAddRecordStage in
  56     examples).
  57
  58     PassThroughStage:
  59     ----------------
  60
  61     A convenience class that takes a single function as a parameter,
  62     that is chain-called to create the exact same input and output spec.
  63     It has a process() function that simply returns its input.
  64
  65     Instances of this class are completely redundant if handed to
  66     StageChain, however when passed to UnbufferedPipeline they
  67     can be used to introduce a single clock delay.
  68
  69     ControlBase:
  70     -----------
  71
  72     The base class for pipelines.  Contains previous and next ready/valid/data.
  73     Also has an extremely useful "connect" function that can be used to
  74     connect a chain of pipelines and present the exact same prev/next
  75     ready/valid/data API.
  76
  77     UnbufferedPipeline:
  78     ------------------
  79
  80     A simple stalling clock-synchronised pipeline that has no buffering
  81     (unlike BufferedHandshake).  Data flows on *every* clock cycle when
  82     the conditions are right (this is nominally when the input is valid
  83     and the output is ready).
  84
  85     A stall anywhere along the line will result in a stall back-propagating
  86     down the entire chain.  The BufferedHandshake by contrast will buffer
  87     incoming data, allowing previous stages one clock cycle's grace before
  88     also having to stall.
  89
  90     An advantage of the UnbufferedPipeline over the Buffered one is
  91     that the amount of logic needed (number of gates) is greatly
  92     reduced (no second set of buffers basically)
  93
  94     The disadvantage of the UnbufferedPipeline is that the valid/ready
  95     logic, if chained together, is *combinatorial*, resulting in
  96     progressively larger gate delay.
  97
  98     PassThroughHandshake:
  99     ------------------
 100
 101     A Control class that introduces a single clock delay, passing its
 102     data through unaltered.  Unlike RegisterPipeline (which relies
 103     on UnbufferedPipeline and PassThroughStage) it handles ready/valid
 104     itself.
 105
 106     RegisterPipeline:
 107     ----------------
 108
 109     A convenience class that, because UnbufferedPipeline introduces a single
 110     clock delay, when its stage is a PassThroughStage, it results in a Pipeline
 111     stage that, duh, delays its (unmodified) input by one clock cycle.
 112
 113     BufferedHandshake:
 114     ----------------
 115
 116     nmigen implementation of buffered pipeline stage, based on zipcpu:
 117     https://zipcpu.com/blog/2017/08/14/strategies-for-pipelining.html
 118
 119     this module requires quite a bit of thought to understand how it works
 120     (and why it is needed in the first place).  reading the above is
 121     *strongly* recommended.
 122
 123     unlike john dawson's IEEE754 FPU STB/ACK signalling, which requires
 124     the STB / ACK signals to raise and lower (on separate clocks) before
 125     data may proceeed (thus only allowing one piece of data to proceed
 126     on *ALTERNATE* cycles), the signalling here is a true pipeline
 127     where data will flow on *every* clock when the conditions are right.
 128
 129     input acceptance conditions are when:
 130         * incoming previous-stage strobe (p.i_valid) is HIGH
 131         * outgoing previous-stage ready   (p.o_ready) is LOW
 132
 133     output transmission conditions are when:
 134         * outgoing next-stage strobe (n.o_valid) is HIGH
 135         * outgoing next-stage ready   (n.i_ready) is LOW
 136
 137     the tricky bit is when the input has valid data and the output is not
 138     ready to accept it.  if it wasn't for the clock synchronisation, it
 139     would be possible to tell the input "hey don't send that data, we're
 140     not ready".  unfortunately, it's not possible to "change the past":
 141     the previous stage *has no choice* but to pass on its data.
 142
 143     therefore, the incoming data *must* be accepted - and stored: that
 144     is the responsibility / contract that this stage *must* accept.
 145     on the same clock, it's possible to tell the input that it must
 146     not send any more data.  this is the "stall" condition.
 147
 148     we now effectively have *two* possible pieces of data to "choose" from:
 149     the buffered data, and the incoming data.  the decision as to which
 150     to process and output is based on whether we are in "stall" or not.
 151     i.e. when the next stage is no longer ready, the output comes from
 152     the buffer if a stall had previously occurred, otherwise it comes
 153     direct from processing the input.
 154
 155     this allows us to respect a synchronous "travelling STB" with what
 156     dan calls a "buffered handshake".
 157
 158     it's quite a complex state machine!
 159
 160     SimpleHandshake
 161     ---------------
 162
 163     Synchronised pipeline, Based on:
 164     https://github.com/ZipCPU/dbgbus/blob/master/hexbus/rtl/hbdeword.v
 165 """
 166
 167 from nmigen import Signal, Cat, Const, Mux, Module, Value
 168 from nmigen.cli import verilog, rtlil
 169 from nmigen.lib.fifo import SyncFIFO, SyncFIFOBuffered
 170 from nmigen.hdl.ast import ArrayProxy
 171 from nmigen.hdl.rec import Record, Layout
 172
 173 from abc import ABCMeta, abstractmethod
 174 from collections.abc import Sequence
 175 from queue import Queue
 176
 177
 178 class RecordObject(Record):
 179     def __init__(self, layout=None, name=None):
 180         Record.__init__(self, layout=layout or [], name=None)
 181
 182     def __setattr__(self, k, v):
 183         if k in dir(Record) or "fields" not in self.__dict__:
 184             return object.__setattr__(self, k, v)
 185         self.fields[k] = v
 186         if isinstance(v, Record):
 187             newlayout = {k: (k, v.layout)}
 188         else:
 189             newlayout = {k: (k, v.shape())}
 190         self.layout.fields.update(newlayout)
 191
 192
 193
 194 class PrevControl:
 195     """ contains signals that come *from* the previous stage (both in and out)
 196         * i_valid: previous stage indicating all incoming data is valid.
 197                    may be a multi-bit signal, where all bits are required
 198                    to be asserted to indicate "valid".
 199         * o_ready: output to next stage indicating readiness to accept data
 200         * i_data : an input - added by the user of this class
 201     """
 202
 203     def __init__(self, i_width=1, stage_ctl=False):
 204         self.stage_ctl = stage_ctl
 205         self.i_valid = Signal(i_width, name="p_i_valid") # prev   >>in  self
 206         self._o_ready = Signal(name="p_o_ready") # prev   <<out self
 207         self.i_data = None # XXX MUST BE ADDED BY USER
 208         if stage_ctl:
 209             self.s_o_ready = Signal(name="p_s_o_rdy") # prev   <<out self
 210
 211     @property
 212     def o_ready(self):
 213         """ public-facing API: indicates (externally) that stage is ready
 214         """
 215         if self.stage_ctl:
 216             return self.s_o_ready # set dynamically by stage
 217         return self._o_ready      # return this when not under dynamic control
 218
 219     def _connect_in(self, prev, direct=False, fn=None):
 220         """ internal helper function to connect stage to an input source.
 221             do not use to connect stage-to-stage!
 222         """
 223         i_valid = prev.i_valid if direct else prev.i_valid_test
 224         i_data = fn(prev.i_data) if fn is not None else prev.i_data
 225         return [self.i_valid.eq(i_valid),
 226                 prev.o_ready.eq(self.o_ready),
 227                 eq(self.i_data, i_data),
 228                ]
 229
 230     @property
 231     def i_valid_test(self):
 232         vlen = len(self.i_valid)
 233         if vlen > 1:
 234             # multi-bit case: valid only when i_valid is all 1s
 235             all1s = Const(-1, (len(self.i_valid), False))
 236             i_valid = (self.i_valid == all1s)
 237         else:
 238             # single-bit i_valid case
 239             i_valid = self.i_valid
 240
 241         # when stage indicates not ready, incoming data
 242         # must "appear" to be not ready too
 243         if self.stage_ctl:
 244             i_valid = i_valid & self.s_o_ready
 245
 246         return i_valid
 247
 248
 249 class NextControl:
 250     """ contains the signals that go *to* the next stage (both in and out)
 251         * o_valid: output indicating to next stage that data is valid
 252         * i_ready: input from next stage indicating that it can accept data
 253         * o_data : an output - added by the user of this class
 254     """
 255     def __init__(self, stage_ctl=False):
 256         self.stage_ctl = stage_ctl
 257         self.o_valid = Signal(name="n_o_valid") # self out>>  next
 258         self.i_ready = Signal(name="n_i_ready") # self <<in   next
 259         self.o_data = None # XXX MUST BE ADDED BY USER
 260         #if self.stage_ctl:
 261         self.d_valid = Signal(reset=1) # INTERNAL (data valid)
 262
 263     @property
 264     def i_ready_test(self):
 265         if self.stage_ctl:
 266             return self.i_ready & self.d_valid
 267         return self.i_ready
 268
 269     def connect_to_next(self, nxt):
 270         """ helper function to connect to the next stage data/valid/ready.
 271             data/valid is passed *TO* nxt, and ready comes *IN* from nxt.
 272             use this when connecting stage-to-stage
 273         """
 274         return [nxt.i_valid.eq(self.o_valid),
 275                 self.i_ready.eq(nxt.o_ready),
 276                 eq(nxt.i_data, self.o_data),
 277                ]
 278
 279     def _connect_out(self, nxt, direct=False, fn=None):
 280         """ internal helper function to connect stage to an output source.
 281             do not use to connect stage-to-stage!
 282         """
 283         i_ready = nxt.i_ready if direct else nxt.i_ready_test
 284         o_data = fn(nxt.o_data) if fn is not None else nxt.o_data
 285         return [nxt.o_valid.eq(self.o_valid),
 286                 self.i_ready.eq(i_ready),
 287                 eq(o_data, self.o_data),
 288                ]
 289
 290
 291 class Visitor:
 292     """ a helper routine which identifies if it is being passed a list
 293         (or tuple) of objects, or signals, or Records, and calls
 294         a visitor function.
 295
 296         the visiting fn is called when an object is identified.
 297
 298         Record is a special (unusual, recursive) case, where the input may be
 299         specified as a dictionary (which may contain further dictionaries,
 300         recursively), where the field names of the dictionary must match
 301         the Record's field spec.  Alternatively, an object with the same
 302         member names as the Record may be assigned: it does not have to
 303         *be* a Record.
 304
 305         ArrayProxy is also special-cased, it's a bit messy: whilst ArrayProxy
 306         has an eq function, the object being assigned to it (e.g. a python
 307         object) might not.  despite the *input* having an eq function,
 308         that doesn't help us, because it's the *ArrayProxy* that's being
 309         assigned to.  so.... we cheat.  use the ports() function of the
 310         python object, enumerate them, find out the list of Signals that way,
 311         and assign them.
 312     """
 313     def visit(self, o, i, act):
 314         if isinstance(o, dict):
 315             return self.dict_visit(o, i, act)
 316
 317         res = act.prepare()
 318         if not isinstance(o, Sequence):
 319             o, i = [o], [i]
 320         for (ao, ai) in zip(o, i):
 321             #print ("visit", fn, ao, ai)
 322             if isinstance(ao, Record):
 323                 rres = self.record_visit(ao, ai, act)
 324             elif isinstance(ao, ArrayProxy) and not isinstance(ai, Value):
 325                 rres = self.arrayproxy_visit(ao, ai, act)
 326             else:
 327                 rres = act.fn(ao, ai)
 328             res += rres
 329         return res
 330
 331     def dict_visit(self, o, i, act):
 332         res = act.prepare()
 333         for (k, v) in o.items():
 334             print ("d-eq", v, i[k])
 335             res.append(act.fn(v, i[k]))
 336         return res
 337
 338     def record_visit(self, ao, ai, act):
 339         res = act.prepare()
 340         for idx, (field_name, field_shape, _) in enumerate(ao.layout):
 341             if isinstance(field_shape, Layout):
 342                 val = ai.fields
 343             else:
 344                 val = ai
 345             if hasattr(val, field_name): # check for attribute
 346                 val = getattr(val, field_name)
 347             else:
 348                 val = val[field_name] # dictionary-style specification
 349             val = self.visit(ao.fields[field_name], val, act)
 350             if isinstance(val, Sequence):
 351                 res += val
 352             else:
 353                 res.append(val)
 354         return res
 355
 356     def arrayproxy_visit(self, ao, ai, act):
 357         res = act.prepare()
 358         for p in ai.ports():
 359             op = getattr(ao, p.name)
 360             #print (op, p, p.name)
 361             res.append(fn(op, p))
 362         return res
 363
 364
 365 class Eq(Visitor):
 366     def __init__(self):
 367         self.res = []
 368     def prepare(self):
 369         return []
 370     def fn(self, o, i):
 371         rres = o.eq(i)
 372         if not isinstance(rres, Sequence):
 373             rres = [rres]
 374         return rres
 375     def __call__(self, o, i):
 376         return self.visit(o, i, self)
 377
 378
 379 def eq(o, i):
 380     """ makes signals equal: a helper routine which identifies if it is being
 381         passed a list (or tuple) of objects, or signals, or Records, and calls
 382         the objects' eq function.
 383     """
 384     return Eq()(o, i)
 385
 386
 387 def flatten(i):
 388     """ flattens a compound structure recursively using Cat
 389     """
 390     if not isinstance(i, Sequence):
 391         i = [i]
 392     res = []
 393     for ai in i:
 394         print ("flatten", ai)
 395         if isinstance(ai, Record):
 396             print ("record", list(ai.layout))
 397             rres = []
 398             for idx, (field_name, field_shape, _) in enumerate(ai.layout):
 399                 if isinstance(field_shape, Layout):
 400                     val = ai.fields
 401                 else:
 402                     val = ai
 403                 if hasattr(val, field_name): # check for attribute
 404                     val = getattr(val, field_name)
 405                 else:
 406                     val = val[field_name] # dictionary-style specification
 407                 print ("recidx", idx, field_name, field_shape, val)
 408                 val = flatten(val)
 409                 print ("recidx flat", idx, val)
 410                 if isinstance(val, Sequence):
 411                     rres += val
 412                 else:
 413                     rres.append(val)
 414
 415         elif isinstance(ai, ArrayProxy) and not isinstance(ai, Value):
 416             rres = []
 417             for p in ai.ports():
 418                 op = getattr(ai, p.name)
 419                 #print (op, p, p.name)
 420                 rres.append(flatten(p))
 421         else:
 422             rres = ai
 423         if not isinstance(rres, Sequence):
 424             rres = [rres]
 425         res += rres
 426         print ("flatten res", res)
 427     return Cat(*res)
 428
 429
 430
 431 class StageCls(metaclass=ABCMeta):
 432     """ Class-based "Stage" API.  requires instantiation (after derivation)
 433
 434         see "Stage API" above..  Note: python does *not* require derivation
 435         from this class.  All that is required is that the pipelines *have*
 436         the functions listed in this class.  Derivation from this class
 437         is therefore merely a "courtesy" to maintainers.
 438     """
 439     @abstractmethod
 440     def ispec(self): pass       # REQUIRED
 441     @abstractmethod
 442     def ospec(self): pass       # REQUIRED
 443     #@abstractmethod
 444     #def setup(self, m, i): pass # OPTIONAL
 445     @abstractmethod
 446     def process(self, i): pass  # REQUIRED
 447
 448
 449 class Stage(metaclass=ABCMeta):
 450     """ Static "Stage" API.  does not require instantiation (after derivation)
 451
 452         see "Stage API" above.  Note: python does *not* require derivation
 453         from this class.  All that is required is that the pipelines *have*
 454         the functions listed in this class.  Derivation from this class
 455         is therefore merely a "courtesy" to maintainers.
 456     """
 457     @staticmethod
 458     @abstractmethod
 459     def ispec(): pass
 460
 461     @staticmethod
 462     @abstractmethod
 463     def ospec(): pass
 464
 465     #@staticmethod
 466     #@abstractmethod
 467     #def setup(m, i): pass
 468
 469     @staticmethod
 470     @abstractmethod
 471     def process(i): pass
 472
 473
 474 class RecordBasedStage(Stage):
 475     """ convenience class which provides a Records-based layout.
 476         honestly it's a lot easier just to create a direct Records-based
 477         class (see ExampleAddRecordStage)
 478     """
 479     def __init__(self, in_shape, out_shape, processfn, setupfn=None):
 480         self.in_shape = in_shape
 481         self.out_shape = out_shape
 482         self.__process = processfn
 483         self.__setup = setupfn
 484     def ispec(self): return Record(self.in_shape)
 485     def ospec(self): return Record(self.out_shape)
 486     def process(seif, i): return self.__process(i)
 487     def setup(seif, m, i): return self.__setup(m, i)
 488
 489
 490 class StageChain(StageCls):
 491     """ pass in a list of stages, and they will automatically be
 492         chained together via their input and output specs into a
 493         combinatorial chain.
 494
 495         the end result basically conforms to the exact same Stage API.
 496
 497         * input to this class will be the input of the first stage
 498         * output of first stage goes into input of second
 499         * output of second goes into input into third (etc. etc.)
 500         * the output of this class will be the output of the last stage
 501     """
 502     def __init__(self, chain, specallocate=False):
 503         self.chain = chain
 504         self.specallocate = specallocate
 505
 506     def ispec(self):
 507         return self.chain[0].ispec()
 508
 509     def ospec(self):
 510         return self.chain[-1].ospec()
 511
 512     def _specallocate_setup(self, m, i):
 513         for (idx, c) in enumerate(self.chain):
 514             if hasattr(c, "setup"):
 515                 c.setup(m, i)               # stage may have some module stuff
 516             o = self.chain[idx].ospec()     # last assignment survives
 517             m.d.comb += eq(o, c.process(i)) # process input into "o"
 518             if idx == len(self.chain)-1:
 519                 break
 520             i = self.chain[idx+1].ispec()   # new input on next loop
 521             m.d.comb += eq(i, o)            # assign to next input
 522         return o                            # last loop is the output
 523
 524     def _noallocate_setup(self, m, i):
 525         for (idx, c) in enumerate(self.chain):
 526             if hasattr(c, "setup"):
 527                 c.setup(m, i)               # stage may have some module stuff
 528             i = o = c.process(i)            # store input into "o"
 529         return o                            # last loop is the output
 530
 531     def setup(self, m, i):
 532         if self.specallocate:
 533             self.o = self._specallocate_setup(m, i)
 534         else:
 535             self.o = self._noallocate_setup(m, i)
 536
 537     def process(self, i):
 538         return self.o # conform to Stage API: return last-loop output
 539
 540
 541 class ControlBase:
 542     """ Common functions for Pipeline API
 543     """
 544     def __init__(self, stage=None, in_multi=None, stage_ctl=False):
 545         """ Base class containing ready/valid/data to previous and next stages
 546
 547             * p: contains ready/valid to the previous stage
 548             * n: contains ready/valid to the next stage
 549
 550             Except when calling Controlbase.connect(), user must also:
 551             * add i_data member to PrevControl (p) and
 552             * add o_data member to NextControl (n)
 553         """
 554         self.stage = stage
 555
 556         # set up input and output IO ACK (prev/next ready/valid)
 557         self.p = PrevControl(in_multi, stage_ctl)
 558         self.n = NextControl(stage_ctl)
 559
 560         # set up the input and output data
 561         if stage is not None:
 562             self.p.i_data = stage.ispec() # input type
 563             self.n.o_data = stage.ospec()
 564
 565     def connect_to_next(self, nxt):
 566         """ helper function to connect to the next stage data/valid/ready.
 567         """
 568         return self.n.connect_to_next(nxt.p)
 569
 570     def _connect_in(self, prev):
 571         """ internal helper function to connect stage to an input source.
 572             do not use to connect stage-to-stage!
 573         """
 574         return self.p._connect_in(prev.p)
 575
 576     def _connect_out(self, nxt):
 577         """ internal helper function to connect stage to an output source.
 578             do not use to connect stage-to-stage!
 579         """
 580         return self.n._connect_out(nxt.n)
 581
 582     def connect(self, pipechain):
 583         """ connects a chain (list) of Pipeline instances together and
 584             links them to this ControlBase instance:
 585
 586                       in <----> self <---> out
 587                        |                   ^
 588                        v                   |
 589                     [pipe1, pipe2, pipe3, pipe4]
 590                        |    ^  |    ^  |     ^
 591                        v    |  v    |  v     |
 592                      out---in out--in out---in
 593
 594             Also takes care of allocating i_data/o_data, by looking up
 595             the data spec for each end of the pipechain.  i.e It is NOT
 596             necessary to allocate self.p.i_data or self.n.o_data manually:
 597             this is handled AUTOMATICALLY, here.
 598
 599             Basically this function is the direct equivalent of StageChain,
 600             except that unlike StageChain, the Pipeline logic is followed.
 601
 602             Just as StageChain presents an object that conforms to the
 603             Stage API from a list of objects that also conform to the
 604             Stage API, an object that calls this Pipeline connect function
 605             has the exact same pipeline API as the list of pipline objects
 606             it is called with.
 607
 608             Thus it becomes possible to build up larger chains recursively.
 609             More complex chains (multi-input, multi-output) will have to be
 610             done manually.
 611         """
 612         eqs = [] # collated list of assignment statements
 613
 614         # connect inter-chain
 615         for i in range(len(pipechain)-1):
 616             pipe1 = pipechain[i]
 617             pipe2 = pipechain[i+1]
 618             eqs += pipe1.connect_to_next(pipe2)
 619
 620         # connect front of chain to ourselves
 621         front = pipechain[0]
 622         self.p.i_data = front.stage.ispec()
 623         eqs += front._connect_in(self)
 624
 625         # connect end of chain to ourselves
 626         end = pipechain[-1]
 627         self.n.o_data = end.stage.ospec()
 628         eqs += end._connect_out(self)
 629
 630         return eqs
 631
 632     def _postprocess(self, i): # XXX DISABLED
 633         return i # RETURNS INPUT
 634         if hasattr(self.stage, "postprocess"):
 635             return self.stage.postprocess(i)
 636         return i
 637
 638     def set_input(self, i):
 639         """ helper function to set the input data
 640         """
 641         return eq(self.p.i_data, i)
 642
 643     def ports(self):
 644         res = [self.p.i_valid, self.n.i_ready,
 645                 self.n.o_valid, self.p.o_ready,
 646                ]
 647         if hasattr(self.p.i_data, "ports"):
 648             res += self.p.i_data.ports()
 649         else:
 650             res += self.p.i_data
 651         if hasattr(self.n.o_data, "ports"):
 652             res += self.n.o_data.ports()
 653         else:
 654             res += self.n.o_data
 655         return res
 656
 657     def _elaborate(self, platform):
 658         """ handles case where stage has dynamic ready/valid functions
 659         """
 660         m = Module()
 661
 662         if self.stage is not None and hasattr(self.stage, "setup"):
 663             self.stage.setup(m, self.p.i_data)
 664
 665         if not self.p.stage_ctl:
 666             return m
 667
 668         # intercept the previous (outgoing) "ready", combine with stage ready
 669         m.d.comb += self.p.s_o_ready.eq(self.p._o_ready & self.stage.d_ready)
 670
 671         # intercept the next (incoming) "ready" and combine it with data valid
 672         sdv = self.stage.d_valid(self.n.i_ready)
 673         m.d.comb += self.n.d_valid.eq(self.n.i_ready & sdv)
 674
 675         return m
 676
 677
 678 class BufferedHandshake(ControlBase):
 679     """ buffered pipeline stage.  data and strobe signals travel in sync.
 680         if ever the input is ready and the output is not, processed data
 681         is shunted in a temporary register.
 682
 683         Argument: stage.  see Stage API above
 684
 685         stage-1   p.i_valid >>in   stage   n.o_valid out>>   stage+1
 686         stage-1   p.o_ready <<out  stage   n.i_ready <<in    stage+1
 687         stage-1   p.i_data  >>in   stage   n.o_data  out>>   stage+1
 688                               |             |
 689                             process --->----^
 690                               |             |
 691                               +-- r_data ->-+
 692
 693         input data p.i_data is read (only), is processed and goes into an
 694         intermediate result store [process()].  this is updated combinatorially.
 695
 696         in a non-stall condition, the intermediate result will go into the
 697         output (update_output).  however if ever there is a stall, it goes
 698         into r_data instead [update_buffer()].
 699
 700         when the non-stall condition is released, r_data is the first
 701         to be transferred to the output [flush_buffer()], and the stall
 702         condition cleared.
 703
 704         on the next cycle (as long as stall is not raised again) the
 705         input may begin to be processed and transferred directly to output.
 706     """
 707
 708     def elaborate(self, platform):
 709         self.m = ControlBase._elaborate(self, platform)
 710
 711         result = self.stage.ospec()
 712         r_data = self.stage.ospec()
 713
 714         # establish some combinatorial temporaries
 715         o_n_validn = Signal(reset_less=True)
 716         n_i_ready = Signal(reset_less=True, name="n_i_rdy_data")
 717         nir_por = Signal(reset_less=True)
 718         nir_por_n = Signal(reset_less=True)
 719         p_i_valid = Signal(reset_less=True)
 720         nir_novn = Signal(reset_less=True)
 721         nirn_novn = Signal(reset_less=True)
 722         por_pivn = Signal(reset_less=True)
 723         npnn = Signal(reset_less=True)
 724         self.m.d.comb += [p_i_valid.eq(self.p.i_valid_test),
 725                      o_n_validn.eq(~self.n.o_valid),
 726                      n_i_ready.eq(self.n.i_ready_test),
 727                      nir_por.eq(n_i_ready & self.p._o_ready),
 728                      nir_por_n.eq(n_i_ready & ~self.p._o_ready),
 729                      nir_novn.eq(n_i_ready | o_n_validn),
 730                      nirn_novn.eq(~n_i_ready & o_n_validn),
 731                      npnn.eq(nir_por | nirn_novn),
 732                      por_pivn.eq(self.p._o_ready & ~p_i_valid)
 733         ]
 734
 735         # store result of processing in combinatorial temporary
 736         self.m.d.comb += eq(result, self.stage.process(self.p.i_data))
 737
 738         # if not in stall condition, update the temporary register
 739         with self.m.If(self.p.o_ready): # not stalled
 740             self.m.d.sync += eq(r_data, result) # update buffer
 741
 742         # data pass-through conditions
 743         with self.m.If(npnn):
 744             o_data = self._postprocess(result)
 745             self.m.d.sync += [self.n.o_valid.eq(p_i_valid), # valid if p_valid
 746                               eq(self.n.o_data, o_data),    # update output
 747                              ]
 748         # buffer flush conditions (NOTE: can override data passthru conditions)
 749         with self.m.If(nir_por_n): # not stalled
 750             # Flush the [already processed] buffer to the output port.
 751             o_data = self._postprocess(r_data)
 752             self.m.d.sync += [self.n.o_valid.eq(1),  # reg empty
 753                               eq(self.n.o_data, o_data), # flush buffer
 754                              ]
 755         # output ready conditions
 756         self.m.d.sync += self.p._o_ready.eq(nir_novn | por_pivn)
 757
 758         return self.m
 759
 760
 761 class SimpleHandshake(ControlBase):
 762     """ simple handshake control.  data and strobe signals travel in sync.
 763         implements the protocol used by Wishbone and AXI4.
 764
 765         Argument: stage.  see Stage API above
 766
 767         stage-1   p.i_valid >>in   stage   n.o_valid out>>   stage+1
 768         stage-1   p.o_ready <<out  stage   n.i_ready <<in    stage+1
 769         stage-1   p.i_data  >>in   stage   n.o_data  out>>   stage+1
 770                               |             |
 771                               +--process->--^
 772         Truth Table
 773
 774         Inputs   Temporary  Output Data
 775         -------  ---------- -----  ----
 776         P P N N  PiV& ~NiR&  N P
 777         i o i o  PoR  NoV    o o
 778         V R R V              V R
 779
 780         -------   -    -     - -
 781         0 0 0 0   0    0    >0 0    reg
 782         0 0 0 1   0    1    >1 0    reg
 783         0 0 1 0   0    0     0 1    process(i_data)
 784         0 0 1 1   0    0     0 1    process(i_data)
 785         -------   -    -     - -
 786         0 1 0 0   0    0    >0 0    reg
 787         0 1 0 1   0    1    >1 0    reg
 788         0 1 1 0   0    0     0 1    process(i_data)
 789         0 1 1 1   0    0     0 1    process(i_data)
 790         -------   -    -     - -
 791         1 0 0 0   0    0    >0 0    reg
 792         1 0 0 1   0    1    >1 0    reg
 793         1 0 1 0   0    0     0 1    process(i_data)
 794         1 0 1 1   0    0     0 1    process(i_data)
 795         -------   -    -     - -
 796         1 1 0 0   1    0     1 0    process(i_data)
 797         1 1 0 1   1    1     1 0    process(i_data)
 798         1 1 1 0   1    0     1 1    process(i_data)
 799         1 1 1 1   1    0     1 1    process(i_data)
 800         -------   -    -     - -
 801     """
 802
 803     def elaborate(self, platform):
 804         self.m = m = ControlBase._elaborate(self, platform)
 805
 806         r_busy = Signal()
 807         result = self.stage.ospec()
 808
 809         # establish some combinatorial temporaries
 810         n_i_ready = Signal(reset_less=True, name="n_i_rdy_data")
 811         p_i_valid_p_o_ready = Signal(reset_less=True)
 812         p_i_valid = Signal(reset_less=True)
 813         m.d.comb += [p_i_valid.eq(self.p.i_valid_test),
 814                      n_i_ready.eq(self.n.i_ready_test),
 815                      p_i_valid_p_o_ready.eq(p_i_valid & self.p.o_ready),
 816         ]
 817
 818         # store result of processing in combinatorial temporary
 819         m.d.comb += eq(result, self.stage.process(self.p.i_data))
 820
 821         # previous valid and ready
 822         with m.If(p_i_valid_p_o_ready):
 823             o_data = self._postprocess(result)
 824             m.d.sync += [r_busy.eq(1),      # output valid
 825                          eq(self.n.o_data, o_data), # update output
 826                         ]
 827         # previous invalid or not ready, however next is accepting
 828         with m.Elif(n_i_ready):
 829             o_data = self._postprocess(result)
 830             m.d.sync += [eq(self.n.o_data, o_data)]
 831             # TODO: could still send data here (if there was any)
 832             #m.d.sync += self.n.o_valid.eq(0) # ...so set output invalid
 833             m.d.sync += r_busy.eq(0) # ...so set output invalid
 834
 835         m.d.comb += self.n.o_valid.eq(r_busy)
 836         # if next is ready, so is previous
 837         m.d.comb += self.p._o_ready.eq(n_i_ready)
 838
 839         return self.m
 840
 841
 842 class UnbufferedPipeline(ControlBase):
 843     """ A simple pipeline stage with single-clock synchronisation
 844         and two-way valid/ready synchronised signalling.
 845
 846         Note that a stall in one stage will result in the entire pipeline
 847         chain stalling.
 848
 849         Also that unlike BufferedHandshake, the valid/ready signalling does NOT
 850         travel synchronously with the data: the valid/ready signalling
 851         combines in a *combinatorial* fashion.  Therefore, a long pipeline
 852         chain will lengthen propagation delays.
 853
 854         Argument: stage.  see Stage API, above
 855
 856         stage-1   p.i_valid >>in   stage   n.o_valid out>>   stage+1
 857         stage-1   p.o_ready <<out  stage   n.i_ready <<in    stage+1
 858         stage-1   p.i_data  >>in   stage   n.o_data  out>>   stage+1
 859                               |             |
 860                             r_data        result
 861                               |             |
 862                               +--process ->-+
 863
 864         Attributes:
 865         -----------
 866         p.i_data : StageInput, shaped according to ispec
 867             The pipeline input
 868         p.o_data : StageOutput, shaped according to ospec
 869             The pipeline output
 870         r_data : input_shape according to ispec
 871             A temporary (buffered) copy of a prior (valid) input.
 872             This is HELD if the output is not ready.  It is updated
 873             SYNCHRONOUSLY.
 874         result: output_shape according to ospec
 875             The output of the combinatorial logic.  it is updated
 876             COMBINATORIALLY (no clock dependence).
 877
 878         Truth Table
 879
 880         Inputs  Temp  Output  Data
 881         -------   -   -----   ----
 882         P P N N ~NiR&  N P
 883         i o i o  NoV   o o
 884         V R R V        V R
 885
 886         -------   -    - -
 887         0 0 0 0   0    0 1    reg
 888         0 0 0 1   1    1 0    reg
 889         0 0 1 0   0    0 1    reg
 890         0 0 1 1   0    0 1    reg
 891         -------   -    - -
 892         0 1 0 0   0    0 1    reg
 893         0 1 0 1   1    1 0    reg
 894         0 1 1 0   0    0 1    reg
 895         0 1 1 1   0    0 1    reg
 896         -------   -    - -
 897         1 0 0 0   0    1 1    reg
 898         1 0 0 1   1    1 0    reg
 899         1 0 1 0   0    1 1    reg
 900         1 0 1 1   0    1 1    reg
 901         -------   -    - -
 902         1 1 0 0   0    1 1    process(i_data)
 903         1 1 0 1   1    1 0    process(i_data)
 904         1 1 1 0   0    1 1    process(i_data)
 905         1 1 1 1   0    1 1    process(i_data)
 906         -------   -    - -
 907
 908         Note: PoR is *NOT* involved in the above decision-making.
 909     """
 910
 911     def elaborate(self, platform):
 912         self.m = m = ControlBase._elaborate(self, platform)
 913
 914         data_valid = Signal() # is data valid or not
 915         r_data = self.stage.ospec() # output type
 916
 917         # some temporaries
 918         p_i_valid = Signal(reset_less=True)
 919         pv = Signal(reset_less=True)
 920         buf_full = Signal(reset_less=True)
 921         m.d.comb += p_i_valid.eq(self.p.i_valid_test)
 922         m.d.comb += pv.eq(self.p.i_valid & self.p.o_ready)
 923         m.d.comb += buf_full.eq(~self.n.i_ready_test & data_valid)
 924
 925         m.d.comb += self.n.o_valid.eq(data_valid)
 926         m.d.comb += self.p._o_ready.eq(~data_valid | self.n.i_ready_test)
 927         m.d.sync += data_valid.eq(p_i_valid | buf_full)
 928
 929         with m.If(pv):
 930             m.d.sync += eq(r_data, self.stage.process(self.p.i_data))
 931         o_data = self._postprocess(r_data)
 932         m.d.comb += eq(self.n.o_data, o_data)
 933
 934         return self.m
 935
 936 class UnbufferedPipeline2(ControlBase):
 937     """ A simple pipeline stage with single-clock synchronisation
 938         and two-way valid/ready synchronised signalling.
 939
 940         Note that a stall in one stage will result in the entire pipeline
 941         chain stalling.
 942
 943         Also that unlike BufferedHandshake, the valid/ready signalling does NOT
 944         travel synchronously with the data: the valid/ready signalling
 945         combines in a *combinatorial* fashion.  Therefore, a long pipeline
 946         chain will lengthen propagation delays.
 947
 948         Argument: stage.  see Stage API, above
 949
 950         stage-1   p.i_valid >>in   stage   n.o_valid out>>   stage+1
 951         stage-1   p.o_ready <<out  stage   n.i_ready <<in    stage+1
 952         stage-1   p.i_data  >>in   stage   n.o_data  out>>   stage+1
 953                               |             |    |
 954                               +- process-> buf <-+
 955         Attributes:
 956         -----------
 957         p.i_data : StageInput, shaped according to ispec
 958             The pipeline input
 959         p.o_data : StageOutput, shaped according to ospec
 960             The pipeline output
 961         buf : output_shape according to ospec
 962             A temporary (buffered) copy of a valid output
 963             This is HELD if the output is not ready.  It is updated
 964             SYNCHRONOUSLY.
 965
 966         Inputs  Temp  Output Data
 967         -------   -   -----
 968         P P N N ~NiR&  N P   (buf_full)
 969         i o i o  NoV   o o
 970         V R R V        V R
 971
 972         -------   -    - -
 973         0 0 0 0   0    0 1   process(i_data)
 974         0 0 0 1   1    1 0   reg (odata, unchanged)
 975         0 0 1 0   0    0 1   process(i_data)
 976         0 0 1 1   0    0 1   process(i_data)
 977         -------   -    - -
 978         0 1 0 0   0    0 1   process(i_data)
 979         0 1 0 1   1    1 0   reg (odata, unchanged)
 980         0 1 1 0   0    0 1   process(i_data)
 981         0 1 1 1   0    0 1   process(i_data)
 982         -------   -    - -
 983         1 0 0 0   0    1 1   process(i_data)
 984         1 0 0 1   1    1 0   reg (odata, unchanged)
 985         1 0 1 0   0    1 1   process(i_data)
 986         1 0 1 1   0    1 1   process(i_data)
 987         -------   -    - -
 988         1 1 0 0   0    1 1   process(i_data)
 989         1 1 0 1   1    1 0   reg (odata, unchanged)
 990         1 1 1 0   0    1 1   process(i_data)
 991         1 1 1 1   0    1 1   process(i_data)
 992         -------   -    - -
 993
 994         Note: PoR is *NOT* involved in the above decision-making.
 995     """
 996
 997     def elaborate(self, platform):
 998         self.m = m = ControlBase._elaborate(self, platform)
 999
1000         buf_full = Signal() # is data valid or not
1001         buf = self.stage.ospec() # output type
1002
1003         # some temporaries
1004         p_i_valid = Signal(reset_less=True)
1005         m.d.comb += p_i_valid.eq(self.p.i_valid_test)
1006
1007         m.d.comb += self.n.o_valid.eq(buf_full | p_i_valid)
1008         m.d.comb += self.p._o_ready.eq(~buf_full)
1009         m.d.sync += buf_full.eq(~self.n.i_ready_test & self.n.o_valid)
1010
1011         o_data = Mux(buf_full, buf, self.stage.process(self.p.i_data))
1012         o_data = self._postprocess(o_data)
1013         m.d.comb += eq(self.n.o_data, o_data)
1014         m.d.sync += eq(buf, self.n.o_data)
1015
1016         return self.m
1017
1018
1019 class PassThroughStage(StageCls):
1020     """ a pass-through stage which has its input data spec equal to its output,
1021         and "passes through" its data from input to output.
1022     """
1023     def __init__(self, iospecfn):
1024         self.iospecfn = iospecfn
1025     def ispec(self): return self.iospecfn()
1026     def ospec(self): return self.iospecfn()
1027     def process(self, i): return i
1028
1029
1030 class PassThroughHandshake(ControlBase):
1031     """ A control block that delays by one clock cycle.
1032
1033         Inputs   Temporary          Output Data
1034         -------  ------------------  ----- ----
1035         P P N N  PiV& PiV| NiR| pvr   N P  (pvr)
1036         i o i o  PoR  ~PoR ~NoV       o o
1037         V R R V                       V R
1038
1039         -------   -    -    -   -     - -
1040         0 0 0 0   0    1    1   0     1 1   odata (unchanged)
1041         0 0 0 1   0    1    0   0     1 0   odata (unchanged)
1042         0 0 1 0   0    1    1   0     1 1   odata (unchanged)
1043         0 0 1 1   0    1    1   0     1 1   odata (unchanged)
1044         -------   -    -    -   -     - -
1045         0 1 0 0   0    0    1   0     0 1   odata (unchanged)
1046         0 1 0 1   0    0    0   0     0 0   odata (unchanged)
1047         0 1 1 0   0    0    1   0     0 1   odata (unchanged)
1048         0 1 1 1   0    0    1   0     0 1   odata (unchanged)
1049         -------   -    -    -   -     - -
1050         1 0 0 0   0    1    1   1     1 1   process(in)
1051         1 0 0 1   0    1    0   0     1 0   odata (unchanged)
1052         1 0 1 0   0    1    1   1     1 1   process(in)
1053         1 0 1 1   0    1    1   1     1 1   process(in)
1054         -------   -    -    -   -     - -
1055         1 1 0 0   1    1    1   1     1 1   process(in)
1056         1 1 0 1   1    1    0   0     1 0   odata (unchanged)
1057         1 1 1 0   1    1    1   1     1 1   process(in)
1058         1 1 1 1   1    1    1   1     1 1   process(in)
1059         -------   -    -    -   -     - -
1060
1061     """
1062
1063     def elaborate(self, platform):
1064         self.m = m = ControlBase._elaborate(self, platform)
1065
1066         r_data = self.stage.ospec() # output type
1067
1068         # temporaries
1069         p_i_valid = Signal(reset_less=True)
1070         pvr = Signal(reset_less=True)
1071         m.d.comb += p_i_valid.eq(self.p.i_valid_test)
1072         m.d.comb += pvr.eq(p_i_valid & self.p.o_ready)
1073
1074         m.d.comb += self.p.o_ready.eq(~self.n.o_valid |  self.n.i_ready_test)
1075         m.d.sync += self.n.o_valid.eq(p_i_valid       | ~self.p.o_ready)
1076
1077         odata = Mux(pvr, self.stage.process(self.p.i_data), r_data)
1078         m.d.sync += eq(r_data, odata)
1079         r_data = self._postprocess(r_data)
1080         m.d.comb += eq(self.n.o_data, r_data)
1081
1082         return m
1083
1084
1085 class RegisterPipeline(UnbufferedPipeline):
1086     """ A pipeline stage that delays by one clock cycle, creating a
1087         sync'd latch out of o_data and o_valid as an indirect byproduct
1088         of using PassThroughStage
1089     """
1090     def __init__(self, iospecfn):
1091         UnbufferedPipeline.__init__(self, PassThroughStage(iospecfn))
1092
1093
1094 class FIFOControl(ControlBase):
1095     """ FIFO Control.  Uses SyncFIFO to store data, coincidentally
1096         happens to have same valid/ready signalling as Stage API.
1097
1098         i_data -> fifo.din -> FIFO -> fifo.dout -> o_data
1099     """
1100
1101     def __init__(self, depth, stage, in_multi=None, stage_ctl=False,
1102                                      fwft=True, buffered=False, pipe=False):
1103         """ FIFO Control
1104
1105             * depth: number of entries in the FIFO
1106             * stage: data processing block
1107             * fwft : first word fall-thru mode (non-fwft introduces delay)
1108             * buffered: use buffered FIFO (introduces extra cycle delay)
1109
1110             NOTE 1: FPGAs may have trouble with the defaults for SyncFIFO
1111                     (fwft=True, buffered=False)
1112
1113             NOTE 2: i_data *must* have a shape function.  it can therefore
1114                     be a Signal, or a Record, or a RecordObject.
1115
1116             data is processed (and located) as follows:
1117
1118             self.p  self.stage temp    fn temp  fn  temp  fp   self.n
1119             i_data->process()->result->flatten->din.FIFO.dout->flatten(o_data)
1120
1121             yes, really: flatten produces a Cat() which can be assigned to.
1122             this is how the FIFO gets de-flattened without needing a de-flatten
1123             function
1124         """
1125
1126         assert not (fwft and buffered), "buffered cannot do fwft"
1127         if buffered:
1128             depth += 1
1129         self.fwft = fwft
1130         self.buffered = buffered
1131         self.pipe = pipe
1132         self.fdepth = depth
1133         ControlBase.__init__(self, stage, in_multi, stage_ctl)
1134
1135     def elaborate(self, platform):
1136         self.m = m = ControlBase._elaborate(self, platform)
1137
1138         # make a FIFO with a signal of equal width to the o_data.
1139         (fwidth, _) = self.n.o_data.shape()
1140         if self.buffered:
1141             fifo = SyncFIFOBuffered(fwidth, self.fdepth)
1142         else:
1143             fifo = Queue(fwidth, self.fdepth, fwft=self.fwft, pipe=self.pipe)
1144         m.submodules.fifo = fifo
1145
1146         # store result of processing in combinatorial temporary
1147         result = self.stage.ospec()
1148         m.d.comb += eq(result, self.stage.process(self.p.i_data))
1149
1150         # connect previous rdy/valid/data - do flatten on i_data
1151         # NOTE: cannot do the PrevControl-looking trick because
1152         # of need to process the data.  shaaaame....
1153         m.d.comb += [fifo.we.eq(self.p.i_valid_test),
1154                      self.p.o_ready.eq(fifo.writable),
1155                      eq(fifo.din, flatten(result)),
1156                    ]
1157
1158         # connect next rdy/valid/data - do flatten on o_data
1159         connections = [self.n.o_valid.eq(fifo.readable),
1160                      fifo.re.eq(self.n.i_ready_test),
1161                    ]
1162         if self.fwft or self.buffered:
1163             m.d.comb += connections
1164         else:
1165             m.d.sync += connections # unbuffered fwft mode needs sync
1166         o_data = flatten(self.n.o_data).eq(fifo.dout)
1167         o_data = self._postprocess(o_data)
1168         m.d.comb += o_data
1169
1170         return m
1171
1172
1173 # aka "RegStage".
1174 class UnbufferedPipeline(FIFOControl):
1175     def __init__(self, stage, in_multi=None, stage_ctl=False):
1176         FIFOControl.__init__(self, 1, stage, in_multi, stage_ctl,
1177                                    fwft=True, pipe=False)
1178
1179 # aka "BreakReadyStage" XXX had to set fwft=True to get it to work
1180 class PassThroughHandshake(FIFOControl):
1181     def __init__(self, stage, in_multi=None, stage_ctl=False):
1182         FIFOControl.__init__(self, 1, stage, in_multi, stage_ctl,
1183                                    fwft=True, pipe=True)
1184
1185 # this is *probably* BufferedHandshake, although test #997 now succeeds.
1186 class BufferedHandshake(FIFOControl):
1187     def __init__(self, stage, in_multi=None, stage_ctl=False):
1188         FIFOControl.__init__(self, 2, stage, in_multi, stage_ctl,
1189                                    fwft=True, pipe=False)
1190
1191
1192 # this is *probably* SimpleHandshake (note: memory cell size=0)
1193 class SimpleHandshake(FIFOControl):
1194     def __init__(self, stage, in_multi=None, stage_ctl=False):
1195         FIFOControl.__init__(self, 0, stage, in_multi, stage_ctl,
1196                                    fwft=True, pipe=False)