(no commit message)
[libreriscv.git] / vpu.mdwn
1 # Video Processing and Acceleration
2
3 Here's some preliminary planning for the video part. Top-level headings.
4 Any actual work will need to know the ISA.
5
6 Decoding is prioritized over encoding. In general, faster decoding
7 speeds up a part of encoding.
8
9 Codecs to target:
10
11 * Audio: MP3, AC3, Vorbis, Opus
12 * Video: MPEG1/2, MPEG4 ASP (xvid), H.264, H.265, VP8, VP9, AV1
13
14 YUV-RGB conversion for the most common formats (there may be a hw block for this, if so skip):
15
16 * rgb/bgr24,
17 * rgbx/bgrx/xrgb/xbgr32,
18 * nv12,
19 * nv21
20
21 Most of these are DCT-based, meaning the instructions will likely
22 benefit image formats like JPEG as well. However image formats are not
23 a part of this project, though MJPEG could be considered a video codec?
24 Should we include it and thus speed up JPEG decoding?
25
26 Audio was not explicitly mentioned in the proposal, but fast audio
27 decoding is necessary for smooth video playback. Otherwise a 300MHz
28 chip may not even play Opus in realtime.
29
30 The entropy part of codecs similarly has overlap with general
31 compression.
32
33 Any speedups should really be compared to C (what gcc generates), not a
34 hand-made simple asm implementation. Both for no use in trying to match
35 the compiler for simple loops, and for potential new instructions
36 possibly needing entirely different approaches. We'll also need some
37 guideline on what parallelism to target in the simulator (how many ops
38 we can assume will be completed at once).
39
40 The software part will be ffmpeg and binutils forks until the standards
41 are set and hw exists; then submission; then other projects
42 (gstreamer, lib*).
43
44 # Links
45
46 * Cocotb co-simulation of JPEG <https://github.com/chiggs/oc_jpegencode/>
47 * Top level bugreport: <http://bugs.libre-riscv.org/show_bug.cgi?id=137>
48 * [[nlnet_2019_video]]