Symphony system

Symphony is a system for evaluating the usage of CHERIoT Ibex core as a microcontroller for embedded, IoT and Operational Technology applications that require the use of a root of trust The system is an integration of a Sonata system (containing a CHERIoT Ibex core) with OpenTitan Earl Grey. It is designed for use on FPGA and specifically targets the NewAE CW340.

The system is at its architectural definition and specification stage. RTL development has yet to begin. In the meantime, please read the architecture specfication.

Symphony is part of the Sunburst Project funded by UKRI / DSbD.

Contributing

Installing Nix

The Nix package manager is used to create reproducible builds and consistent development environments. Follow the instructions on the zero to nix site to install the Nix package manager. If you've downloaded nix through another method, make sure the experimental features "flakes" and "nix-command" are enabled.

Building Documentation

The documentation uses mdBook see the installation guide for further details on installation.

Once mdBook is installed the documentation can be built and viewed with:

mdbook serve --open

Code Quality

To check the python code run:

nix run .#lint_python

To automatically fix some lints and formats use:

nix run .#lint_python -- fix

To automatically format Nix files use:

nix fmt

License

Unless otherwise noted, everything in the repository is covered by the Apache License, Version 2.0. See the LICENSE file for more information on licensing.

Symphony architecture specification

Symphony brings together OpenTitan Earl Grey (an open silicon root of trust) and the Sonata CHERIoT Ibex system. Earl Grey is production quality (including extensive DV) and security hardened. Symphony leverages the functionality and quality level of Earl Grey. It is intended for applications that want to utilize CHERIoT but also require the security functionality and guarantees that can be obtained with a root of trust.

Unlike Earl Grey, Symphony will not be brought to production quality within the scope of the Sunburst project. In particular whilst due consideration will be given towards security hardening it will not receive the same level of scrutiny and validation as Earl Grey.

The Symphony architecture comprises a number of components:

• FPGA configuration architecture: What changes do we make to Sonata, what changes to we make to OpenTitan Earl Grey and how do we connect the two together? It defines everything we are using a hardware description language for.
• Software architecture: How do we compose Earl Grey and Sonata together? How do we enable Sonata to use peripherals and services provided by Earl Grey?

FPGA configuration architecture

The target board for the Symphony system is the NewAE CW340.

Sonata system

A cut down version of the Sonata system will be used. Most of the peripherals are not required as the Earl Grey peripherals be used via peripheral donation instead (see Software architecture). The follow parts of Sonata will be included:

• CHERIoT Ibex
• CHERIoT debug module
• DMA
• UART
• GPIO
• SRAM

OpenTitan Earl Grey

Modifications to Earl Grey will be kept to an absolute minimum. Avoiding changes minimizes the risk of introducing new bugs or security issues. The main modification will be to the two crossbars to enable the connection of the bridge interface.

Bridge interface

The bridge interface connects the Sonata system to Earl Grey. Sonata communicates with the bridge interface exclusively via a TileLink port. Earl Grey can raise an interrupt in the Sonata system via the bridge interface and has control over the clock, reset, power enable and boot vector of the Sonata system. On power up, the Sonata system will be inactive with its startup controlled by Earl Grey via the bridge interface.

Software architecture

CHERIoT Secure Boot

The boot of the CHERIoT Ibex core in Sonata is controlled entirely by Earl Grey. Sonata does not have its own embedded flash or SPI peripheral. This means all accesses to non-volatile storage from Sonata must be mediated by Earl Grey. Earl Grey provides a secure boot flow to Sonata to enable this.

Peripheral Donation

Earl Grey has a number of useful peripherals that Sonata may wish to use. The peripheral donation mechanism allows Sonata to access these via the bridge interface.

Bridge Interface

The bridge interface enables the Sonata system to communicate with Earl Grey via Tile Link transactions. It maintains a security boundary so all accesses into the Earl Grey Tile Link fabric are strictly controlled and the security properties of Earl Grey are not violated. At its core the bridge interface is a Tile Link crossbar with a number of peripherals connected to it. This crossbar is connected into the Earl Grey Tile Link fabric and is directly visible in the Ibex core address space. The Sonata system has no direct access to the Tile Link crossbar or the bridge interface peripherals.

All transactions from the Sonata system go through the 'Earl Grey access port' which provides address translation services which are under the control of Earl Grey. For details see Earl Grey Access Port.

To enable communication between Sonata and Earl Grey a mailbox and two banks of SRAM are attached to the bridge interface crossbar.

Bridge Interface Components

• Interrupt Bridge - Sends interrupts from Earl Grey to Sonata.
• Sonata System Control - Gives control over Sonata clock, reset, power and boot to Earl Grey.
• Shared SRAM - Two banks of SRAM that can be accessed in parallel along with a write monitor that can notify Earl Grey of writes to monitored locations.
• Mailbox - For passing small messages between Earl Grey and Sonata.
• Earl Grey Access Port - Provides access to the Earl Grey Tile Link interconnect via an address translation and protection mechanism controlled by Earl Grey.
• Bridge Crossbar - Connects the bridge interface components together and provides a connection into the Earl Grey Tile Link fabric.

Interrupt Bridge

Passes interrupts lines from Earl Grey through to Sonata. Earl Grey controls which interrupt lines are sent.

Theory of Operations

A fixed subset of Earl Grey interrupt lines are connected into the interrupt bridge. There is an enable bit for each interrupt. When the interrupt is enabled it is sent through to Sonata. The pass through uses a simple masking scheme. When enabling an interrupt through the bridge if an interrupt is raised on the Earl Grey side it will be instantly raised on the Sonata side. Similarly when disabling an interrupt through the bridge whilst it is raised it will be instanatly dropped on the Sonata side.

When Earl Grey is donating a peripheral to Sonata and enabling the relevant interrupts on the bridge it is recommended interrupts are disabled at the peripheral to avoid Sonata receiving interrupts during the donation process. The expectation is Sonata will enable the interrupts it is interested in when the peripheral donation is complete. Similarly interrupts should be disabled at the peripheral when Sonata is releasing the donated peripheral back to Earl Grey.

Registers

Enable Register

Read-write

One bit per interrupt enable. Each register has 32 enable bits. The bit index within this register of an enable is Bridge ID % 32. The index of the register of an enable is Bridge ID / 32.

Register Offsets

Interrupt Mapping Table

This table provides a mapping from the Earl Grey PLIC IRQ IDs to the Interrupt Bridge IDs

Sonata System Control

This block controls the clock, reset, power and boot vector of Sonata. On power up Sonata will be held in reset with the power and clock disabled. Earl Grey must turn it on and boot it with the Sonata System Control block.

Theory of Operation

Each of the clock, reset and power is under the control of Earl Grey. Direct access is given to the clock and power enables for the Sonata System. It is the software's reponsibility to obey any specific sequencing requirements (e.g. a minimum period between power enable and clock enable). Reset is also under direct Earl Grey control. To reset, software must first assert the reset then deassert the reset. The system starts with the reset asserted. The block handles all clock and reset domain crossing requirements.

Registers

Control

Read-write

pwr_en and clk_en control the power and clock enables respectively. reset asserts and deasserts the reset.

An 8-bit encoding is used for each field which is given below. For the reset the Enabled encoding asserts the reset and the Disabled encoding deasserts the reset. Values that don't match either of these encodings have the same effect as Disabled. It is not recommended you use the other values for Disabled as the recommended encoding has been chosen to have a hamming distance of 4 from the Enabled encoding.

Boot Vector

Read-write

Specifies the address (in Sonata address space) that Sonata boots from.

Register Offsets

Shared SRAM

The shared SRAM consists of two separate 4 KiB banks. The banks can be accessed in parallel, so Sonata can access one the same cycle Earl Grey accesses the other.

A write monitor allows Earl Grey to be notified on writes to the SRAM.

Theory of Operation

Two Tile Link ports connect the shared SRAM to the bridge interface crossbar. Both Tile Link ports can access both banks. If both ports present an access in the same cycle they can both proceed if they're targetting different SRAM banks. If they target the same SRAM bank static arbitration is used giving Earl Grey the priority.

The write monitor can monitor one 32-bit region per bank. This means there are two write monitors in total, each with its own address to monitor and interrupt.

Registers

Monitor Address

Read-write

addr gives the address to monitor within a particular SRAM bank. This is an aligned address and as such the bottom 2 bits are not required. When the valid bit is set any write within that 32-bit region triggers the interrupt.

Interrupt Status

Read-write

The intr_0 and intr_1 fields indicate whether the interrupt for the write monitor of bank 0 and 1 monitor have been triggered, respectively. Writing 0 to these fields acknowledges and clears the interrupt.

Interrupt Enable

Read-write

The intr_en_0 and intr_en_1 fields control the enable for interrupts 0 and 1. If an interrupt is disabled but has been triggered (the corresponding intr_0 or intr_1 is set) then enabling the interrupt will cause it be immediately raised with Earl Grey.

Register Offsets

Mailbox

The mailbox provides small (32-bit) message passing between Sonata and Earl Grey. Interrupts are used for message notifications.

Theory of Operation

There are two mailboxes, one for each direction that can hold a single message each. When a message is posted in a mailbox an interrupt is raised to Earl Grey or Sonata (depending on which mailbox has the message) to notify of the incoming message.

There is no fixed format for messages, it is entirely software defined. Where a message does not fit into 32 bits the SRAM banks can be used to carry more data. Hardware does nothing to assist in this, software defined messages should include a field for an offset into an SRAM bank that can be used as an address.

Registers

Message

Read-write

Register that holds a message, it is either in a full or empty state. Each message register has a fixed sender and recepient. The mailbox is aware if it is the sender or recepient that is accessing a register (via the Tile Link ID) and has different behaviours for each.

When in an empty state a write from the sender will switch to the full state and raise an interrupt to the recipient.

When in a full state a read from the recipient will return the message written and clear the interrupt.

Any write from the recipient is ignored and any read from the sender returns 0. Writes from the sender are ignored in the full state and reads from the recipient return 0 in the empty state.

Status

Read-only, writes are ignored.

Indicates empty/full status of each message register. full 0 is the status of message 0 and full 1 is the status of message 1.

Interrupt Enable

Read-write

The intr_en field control the interrupt enable for either Sonata or Earl Grey (there is a separate enable register for each). If an interrupt is disabled but has been triggered (the corresponding message register is full) then enabling the interrupt will cause it be immediately raised.

Writes from Sonata or Earl Grey to the other's interrupt enable register are ignored. Reads from Sonata or Earl Grey to the other's interrupt enable register always return 0.

Register Offsets

Earl Grey Access Port

All Tile Link transactions from Sonata go through the access port. It is the only way Sonata can access Earl Grey. Transactions go through address translation and undergo permission checks. Earl Grey controls the translations and permissions. This means that access to Earl Grey is only provided to Sonata if the Earl Grey firmware permits it.

Theory of operation

The access port is intended to be used in a similar way to demand paging. When a transaction from Sonata is for an address that isn't currently mapped in the available address translation regions, Earl Grey receives an interrupt. If the Earl Grey firmware wishes to allow the transaction, it can setup an appropriate mapping and instruct the access port to let it go through. If it doesn't want to allow the transaction, it will instruct the access port to deny the transaction and produce a bus error. On the Sonata side this happens transparently. The transaction is held in the access port (where there's no existing matching address translation region) and only sends a response back to Sonata once Earl Grey has signaled whether or not the accept the transaction. As this is handled by software in Earl Grey it introduces substantial latency whenever Sonata accesses a region the Earl Grey firmware allows it to access but is not currently mapped in. To help ameliorate this multiple address translation regions are provided. The Earl Grey firmware will aim to setup the regions so what it anticipates being the most regularly used and latency critical ones can be left mapped in using a subset for dynamic remapping as required.

Four address translation regions are provided. Each region is controlled by two registers, a region register and a translation register. The region register specifies the address region in the Sonata address map that will get translated. Regions must all be power of two sized and naturally aligned. The NAPOT (naturally aligned power of two) scheme used by PMP (Physical Memory Protection) in the RISC-V architecture is used to specify a region. This is explained in the register definitions below.

Additionally there are three fixed regions that have immutable address however their access permissions are controlled by Earl Grey. These are used to provide access to the two SRAM banks and the mailbox. When accessing one of these fixed regions if access is disabled Earl Grey will receive an interrupt identically to accesses outside of the fixed regions that don't match a region.

The translation register specifies the base address to translate to in the Earl Grey address map along with permissions. When an incoming memory transaction matches a region it is split into the region address (R) and the offset (O). R is replaced with the translation address (T) leaving O untouched to produce the Earl Grey address. This is illustrated below. The bottom 3 bits of the translation register (which aren't needed for an address as the minimum region size is 8 bytes) provide read/write/execute permissions.

If an incoming Sonata transaction matches a region and passes permissions checks as described above it is passed through to Earl Grey. If it does not match a region it is held in the access port (no response is returned initially) and an interrupt is signaled to Earl Grey. The transaction remains held until Earl Grey releases it by writing to the decision register. Whilst a transaction is held, the access port will not accept any other transactions.

Either the transaction is rejected or accepted. An accepted transaction goes through the region matching and address translation again (exactly as if it was a fresh transaction). When accepting a transaction, it is Earl Grey's responsibility to ensure an appropriate region has been setup to allow the transaction. If this is not done correctly Earl Grey will receive another interrupt and the transaction will be held awaiting a decision once again. A rejected transaction results in an error response being sent back to Sonata.

If a transaction matches a region but fails the permission checks it is immediately rejected with an error response sent back to Sonata.

Two registers, last error address and last error info are visible in the Sonata address map. When an error response is seen from the access port by Sonata it can consult these registers to find the rejection reason.

The access port is visible in two separate places in the Sonata address space.

1. The Earl Grey Access Region - A 1 GiB aligned region, all accesses here are sent to the access port for region matching, permission checks and translation.
2. The Sonata Register Region - A small region for accessing the Sonata visible registers (see details below).

Fixed Regions

The fixed regions are at a constant offset within the Earl Grey Access Region in the Sonata address space. They translate to fixed addresses inside of the Earl Grey address space.

Earl Grey Visible Registers

Region

Read-write

There are 4 region registers (one for each supported region). Both the base address and size of the region are encoded in the region_addr field using the same NAPOT encoding scheme used by PMP addresses in the RISC-V privileged architecture (see Table 3.11 in the privileged architecture version 20211203). The minimum size of a region is 8 bytes, all region sizes must be a power of two and their base address aligned to their size. The location of the first 0 in region_addr sets the size, where it occurs in bit position 0, it is an 8-byte region, in bit position 1 it is a 16-byte region and so on. The remaining bits then provide the base address, this is illustrated in the table below for the first 3 region sizes.

For example a region_addr of 0x2000ffff encodes a region of size 512 kiB (first 0 in bit position 16, (2^16) * 8 == 512k), with a base address of 0x80000000.

Where regions overlap and an incoming address matches multiple regions the one with the lowest index is chosen.

When matching an incoming transaction against a region the address if the full address, it is not effected by where the Earl Grey access region is situated in the Sonata address map. For example if the Earl Grey access region is mapped into the 1 GiB region starting at 0x40000000 an transaction for address 0x40080000 will be matched against regions using that address unaltered, not the address 0x00080000 (the offset of the address within the access region). Configured regions always take priority over the fixed regions.

Translation

Read-write

There are 4 translation registers (one for each supported region). The translation register provides the base address of the region an allowed transaction is mapped into. Only the top N bits of translation_addr are relevant with N dependent on the size of the region. Where the region is 8-bytes the top 29 bits are relevant, with 16-bytes the top 28 and so on. When an access passes the permission checks the top N bits of the incoming Sonata address are replaced with translation_addr to produce the Earl Grey address.

The three permission bits r, w and x indicate whether read, write and execute are permitted in the region respectively. All combinations of permission bits are allowed. Where all three permission bits are cleared the region is disabled and ignored when matching incoming transactions to the region register.

The 3 fixed regions have Translation registers but only the permission bits are writable, the translation_addr field will ignore writes and return the fixed base address for the fixed region.

Pending Transaction Address

Read-only, writes are ignored.

When a transaction is being held by the access port this register gives its address (in Sonata address space). It is guaranteed the value in this register remains constant when valid is set in the Pending Transaction Access Register until a write to the Decision Register occurs.

Pending Transaction Access

Read-only, writes are ignored.

When a transaction is being held by the access port this register provide information about what it wants to do. valid indicates there is a transaction being held. One of r, w or x will be set when valid is set, indicating the transaction is a read, a write or an execute access respectively. It is guaranteed the value in this register remains constant when valid is set until a write to the Decision Register occurs.

Decision

Write-only, reads always return 0.

When a transaction is being held by the access port writing to this register tells the access port what to do with it. Either it is accepted or rejected. If accepted the access is replayed against the region matching (it is the software's responsibility to setup the region and translation registers such that an accepted access will pass). If rejected the transaction is dropped by the access port and an error response sent in return to Sonata.

Writing the following values to cmd will either accept or reject the transaction. A write of another value or a write whilst there is no pending transaction (valid in the Pending Transaction Access Register is unset) has no effect.

These values have been chosen so they have a Hamming distance of 4 between them and also aren't a simple inversion of one another.

Sonata Visible Registers

Last Error Address

Read-only, writes are ignored.

When a transaction has produced an error response the address of that transaction is held in this register. It is guaranteed to remain constant until the next transaction enters the access port.

Last Error Info

Read-only, writes are ignored.

When a transaction has produced an error response this register provides additional information about that error.

r, w, and x indicate if the transaction tried to perform a read, write or execute access respectively.

status indicates the status of the last transaction, it can take one of the following values:

The value of this register is guaranteed to remain constant until the next transaction enters the access port.

Register Offsets

Earl Grey Visible Registers

Sonata Visible Registers

Bridge Crossbar

The crossbar connects all of the blocks on the bridge interface with Earl Grey. It has an additional Tile Link host port for another device or crossbar to connect to. The base Symphony system doesn't connect anything to it, it is an optional feature for users who wish to expand the base Symphony system.

Theory of Operation

The crossbar is constructed in a similar way to those in Earl Grey (see the Earl Grey TL-UL Specification). It connects to two hosts, the Sonata system (via the access port) and Earl Grey. It links separately into the high-speed domain and peripheral domain cross bars in Earl Grey. In each of these crossbars it can only transact with a subset of the connected devices. This is enforced through the physical connectivity of the crossbars, there is no way to route a transaction from the bridge crossbar to the excluded devices on the Earl Grey crossbars.

The crossbar and all of the bridge interface device are in the Earl Grey high speed domain.

Reachable Earl Grey Devices

The following devices in Earl Grey can receive transactions from the bridge crossbar

• SPI Host 0, 1
• USB
• UART 0 - 3
• I2C 0 - 2
• Flash (read-only access, no access to flash control configuration registers)

CHERIoT Secure Boot

Note the flow here is provided as an example of a potential secure boot flow. It has not received the same level of scrutiny as the Earl Grey secure boot flow and would require further refinement for production use-cases.

Earl Grey itself boots using the existing Open Titan secure boot process. This boots into firmware developed for Symphony that controls the CHERIoT secure boot process.

The CHERIoT secure boot is separated into two stages:

Stage 1 - CHERIoT Boot ROM Load

Earl Grey loads and verifies a small 'boot ROM' from it's internal flash and copies it into the shared SRAM connected to the bridge interface (giving a maximum size of 8 KiB for the boot ROM). Provided verification was successful, Sonata's clock and power are enabled and it is brought out of reset. The boot vector points to the code in the shared SRAM and Earl Grey enables read and execute for the SRAM. From Sonata's point of view the initial boot stage is identical to booting from a fixed ROM at a fixed address.

Stage 2 -- CHERIoT Boot ROM Execute

The boot ROM performs the initial Sonata system setup and loads the main Sonata firmware image. It uses security services provided by Earl Grey to verify an image stored in Earl Grey's internal flash (using external flash instead would be possible but the initial Symphony system won't have support for it). Earl Grey maps the firmware image into Sonata's address space and Sonata jumps to it (if verification succeeds). The Earl Grey firmware only maps the firmware image if the verification has succeeded. Should there be an injected fault causing Sonata to believe verification has succeeded when it has failed Sonata won't be able to execute the code it incorrectly jumps to. For Earl Grey existing security hardening (in both software and hardware) mitigates fault injection attacks that could cause it to incorrectly map the firmware image into the Sonata address space following verification failure.

The firmware loads further things as it deems necessary, it could copy code in Sonata's SRAM for direct execution or just continue executing from Earl Grey's flash.

Execute from External Flash

Support for executing out of external flash is not included in the scope of Symphony however it would be possible to add. Earl Grey contains a quad speed serial peripheral interface (QSPI) controller that could be used to access an external flash, however it does not contain a controller that can translate Tile Link transactions commands to access the flash via QSPI. Instead software must generate the required commands and copy data from the QSPI's received data to an appropriate memory. Sonata could simply read from external flash using the QSPI controller this way but everything read would need to be placed in SRAM.

To avoid needing a copy to SRAM a suitable controller would be needed for the QSPI controller so TileLink transactions can produce suitable flash reads. There are a few possibilities of how this could be used:

• With the controller coupled to an SPI host block provided as an entirely new block, connected to the external device port on the bridge crossbar.
• Earl Grey could be further modified so one of its SPI hosts has the controller.
• The controller itself could be part of Sonata and send appropriate TileLink transactions to an existing Earl Grey SPI host that will send the commands to read the flash.