dramforever

a row of my life

VisionFive JTAG adventures, Part 1: JH7100 GPIO

2022-06-27

Background

The research that went into this article began as a simple technical question: How do I connect to the JTAG debug port on the VisionFive?

When luojia asked this very question on the support forum, the only responses were, surprisingly, discouragement. However, for any low-level software development, such a debug port is pretty much the only way to sanely, well, debug anything at all. Even though printf debugging was technically possible, it is still worth it to see if we can reach it over JTAG, given that the board is supposed to have JTAG connectors.

VisionFive

The development board in question, VisionFive, is a RISC-V single-board computer from StarFive. At its core is a StarFive JH7100 SoC, with two SiFive U74 cores and one E24 core.

(As of writing, the version of VisionFive currently available is also known as ‘VisionFive V1’, though even official documentation often omits the version number. The name ‘VisionFive’ in this article consistently refers to VisionFive V1, in case confusion arises in the future.)

The same processor core is also found on the HiFive Unmatched, on which the main SoC has four U74 cores and one S7 core. Unmatched had more RAM and better peripherals, and was made in the form factor of a regular PC motherboard. VisionFive is, instead, clearly intended for a slightly lower end market, or for those who prefers a palm-sized Raspberry Pi lookalike.

JTAG on the VisionFive, or not

For our purposes, the one main difference between the Unmatched and the VisionFive is how the JTAG port is connected. On the Unmatched, an FT2322H adapter on-board means that the Micro-USB port gives access to both the UART port and the JTAG port, readily usable with riscv-openocd.

On the VisionFive, however, JTAG access seems… elusive. Nowhere in the documentation is the JTAG port on-board mentioned. On drawings of the board, the words JTAG are written next to the PMIC, which is, at least to me, nonsensical, as there are no notable ports to be found there.

One of the VisionFive drawings
A photo of the VisionFive board

Next to the color-coded (nice!) 40-pin header though, one can find seven plated holes, one of them having a square outline. Browsing through the schematic reveals that this is indeed where the JTAG port is connected.

The JTAG port

Problem solved, right? Solder up some header pins or build a pogo-pin rig, and just connect it up to your workstation with an FT2322H, and we have JTAG.

Or so we thought. Unfortunately, this JTAG ports does not seem to respond at all to any input. It seems as if this port isn’t connected at all.

Finding the JTAG port

Chasing through labels on the schematics reveals that the JTAG_* through-holes connect to a level shifter, which presents the SoC with 1.8 V signals instead of 3.3 V ones. They then connect to pads on the SoC, mysteriously named GPIOxx/U74_JTAG_*.

JTAG connections to the SOC

So are these GPIO or JTAG? Thankfully the pad connections have labels with positions, so we can look them up on the datasheet. For example, A25 is described as…

A25     GPIO[0]     IO      function IO share with GPIO

‘Function IO Share’ is the title of section 11 in the datasheet. One register, named IO_PADSHARE_SEL, has one of 7 valid values, 0 through 6, is a global configuration controlling the functions two-hundred-odd pads PAD_FUNC_SHARE[141:0] and PAD_GPIO[63:0]. Each of the configurations is called a ‘signal group’, and the groups themselves are called ‘Function 0’ through ‘Function 7’. A giant table then follows, showing each pad’s function under each signal group.

The pads PAD_GPIO[4:0] would be the connections we found earlier, and in Function 0, they. Since Function 0 is supposed to be the default value on reset, this means that we should see a JTAG port there, right?

At this point, the only thing I could think of is connecting to JTAG while holding down the reset button. However, since I had neither a JTAG adapter nor a VisionFive board, all I did was tell luojia about it, and moved on to finish my finals.

Digging deeper into the GPIO multiplexer

The single document that made the system ‘click’ for me is the devicetree documentation for starfive,jh7100-pinctrl. This node can be found in jh7100.dtsi, which is included in jh7100-starfive-visionfive-v1.dts:

gpio: pinctrl@11910000 {
    compatible = "starfive,jh7100-pinctrl";
    reg = <0x0 0x11910000 0x0 0x10000>,
          <0x0 0x11858000 0x0 0x1000>;
    reg-names = "gpio", "padctl";
    /* <snip> */
};

The address ranges mentioned here correspond to these rows in the datasheet:

SYSCTRL-IOPAD_CTRL  0x00_1185_8000  0x00_1185_BFFF  16KB
GPIO                0x00_1191_0000  0x00_1191_FFFF  64KB

These registers control the functions and states of PAD_GPIO[63:0] and PAD_FUNC_SHARE[141:0]. The following diagram is included in the devicetree bindings documentation:

                          Signal group 0, 1, ... or 6
                                 ___|___
                                |       |
    LCD output -----------------|       |
    CMOS Camera interface ------|       |--- PAD_GPIO[0]
    Ethernet PHY interface -----|  MUX  |--- PAD_GPIO[1]
      ...                       |       |      ...
                                |       |--- PAD_GPIO[63]
     -------- GPIO0 ------------|       |
    |  -------|-- GPIO1 --------|       |--- PAD_FUNC_SHARE[0]
    | |       |   |             |       |--- PAD_FUNC_SHARE[1]
    | |       |   |  ...        |       |       ...
    | |       |   |             |       |--- PAD_FUNC_SHARE[141]
    | |  -----|---|-- GPIO63 ---|       |
    | | |     |   |   |          -------
    UART0     UART1 --

These pads on the package, or ‘function IO share with GPIO’ as listed in the datasheet, are connected to the internal signals through a two-stage multiplexer:

First, as mentioned, the IO_PADSHARE_SEL register selects one of 7 ‘signal groups’. This is, curiously, a global setting, meaning that it affects all of the ‘function share’ pads at once. A huge table in the datasheet describes the function of each such pad’s function in each signal group. For example, the row in the table for PAG_GPIO[0] reads: (Reproduced here vertically for convenience)

Interface               GPIO
IO Name                 PAD_GPIO[0]
Function 0 (Default)    U74_JTAG_TDO
Function 1              GPIO0
Function 2              X2C_TX_DATA3
Function 3              LCD_DATA4
Function 4              X2C_TX_DATA3
Function 5              PLL_RFSLIP[0]
Function 6              MIPITX_MPOSV[0]

Note that GPIOn has no direct correspondence to PAD_GPIO[n]. For example, PAD_FUNC_SHARE[0] can also be connected to GPIO0:

Interface               ChipLink
IO Name                 PAD_FUNC_SHARE[0]
Function 0 (Default)    X2C_TX_CLK
Function 1              LCD_CLK
Function 2              CM_CLK
Function 3              X2C_TX_CLK
Function 4              GPIO0
Function 5              GPIO0
Function 6              GPIO0

Instead, GPIOn are internal signals further multiplexed into internal inputs and outputs by the ‘GPIO FMUX’. Each of the GPIOn signals also has configurable pull up/down and Schmitt triggers, though not all options are available for all I/O pads.

After that, there are three connections to be made: output, output enable, and input. These are all configured from the destination side, so:

In addition, the input value from each GPIO may be read from MMIO registers GPIODIN_0 and GPIODIN_1, and it’s also possible to configure them to fire interrupts.

This allows quite a flexible usage of the GPIO internal signals. They can be selected to work with the 1.8 V or 3.3 V I/O pads, and can either be fully controlled by software with interrupt support, or connected to one of the internal I2C/I2S/SDIO/… controllers.

Curiously, in Function 0, the default mode for IO_PADSHARE_SEL, none of the GPIOn signals are connected to the I/O pads. Instead, some of the internal signals are directly connected to I/O pads. Moreover, in a few of the cases, the internal signals PAD_GPIO[n] connects to in Function 0 are conveniently also by default connected to the same-numbered internal GPIOn. For example, most relevant to our original use cases, in Function 0 these connections are made:

PAD_GPIO[0]         U74_JTAG_TDO
PAD_GPIO[1]         U74_JTAG_TCK
PAD_GPIO[2]         U74_JTAG_TDI
PAD_GPIO[3]         U74_JTAG_TMS
PAD_GPIO[4]         U74_JTAG_TRSTN

At the same time, the default GPIO FMUX configuration for these JTAG signals are:

CPU_JTAG_TDO        GPIO0
CPU_JTAG_TCK        GPIO1
CPU_JTAG_TDI        GPIO2
CPU_JTAG_TMS        GPIO3
CPU_JTAG_TRSTN      GPIO4

(It seems that CPU_JTAG_* are synonymous with U74_JTAG_*.)

Finding the JTAG port, take two

It seems that Function 0 is intended for booting and initialization, with many of the internal functions available on I/O pads right away without configuration. However given the lack of, well, actual general purpose input/output, there is no chance Linux runs in Function 0.

We already see our problem: The seven through-holes going to PAD_GPIO[4:0] are JTAG in Function 0 but might not be when another ‘Function’ is selected. This means that at some time after booting, IO_PADSHARE_SEL is set from 0 to some other value, and these JTAG-appearing through-holes would no longer be JTAG.

Which value is it then? Curiously, the example listed in the devicetree documentation has this property:

starfive,signal-group = <6>;

Which would suggest that Linux selects Function 6 at initialization, though the actual jh7100.dtsi did not have this property. The documentation indicates that in case the property is not specified, the IO_PADSHARE_SEL register is left unchanged:

starfive,signal-group:
  description: |
    Select one of the 7 signal groups. If this property is not set it
    defaults to the configuration already chosen by the earlier boot stages.
  $ref: /schemas/types.yaml#/definitions/uint32
  enum: [0, 1, 2, 3, 4, 5, 6]

After a short chat with luojia, who tried connecting to JTAG without a microSD card unsuccessfully, it is apparent that some earlier boot stage sets IO_PADSHARE_SEL. It did not take long crawling through the code provided by StarFive to find this particular line in secondBoot:

_SET_SYSCON_REG_register104_SCFG_io_padshare_sel(6);

For those not familiar with secondBoot, it is one of the first stages of bootloaders on the VisionFive, second to only the internal ROM.

VisionFive boot flow

All of the ‘Firmware’ stages run from an on-board QSPI flash, without requiring a microSD card present. This means that early on in the boot sequence, IO_PADSHARE_SEL is switched from 0 to 6, disabling the JTAG through-holes. Searching through the other files in this repository also reveals the undocumented address of IO_PADSHARE_SEL:

0x118581a0

There is still a way to confirm IO_PADSHARE_SEL. If you hold down the ‘Boot mode’ button while powering the board up, instead of following the normal boot process, a prompt appears on the ‘debug’ serial console port at 9600 8n1, running off internal ROM, where you can read and write arbitrary physical memory. At this point, NickCao helped me out by connecting to this ‘recovery console’, and reading IO_PADSHARE_SEL:

# rh 0x118581a0

 Read Half : 0x0000

Reading the same address in the U-Boot shell gives 6, confirming much of what we had seen.

Connecting to JTAG, for the first time

But wait, if IO_PADSHARE_SEL is 0 when running in internal ROM, does this mean JTAG is available on the seven through-holes? Should we be able to connect to the debug modules in this state?

I asked Icenowy, the only person I know of with both a VisionFive and some JTAG adapter handy to help out. I told them to power up the board with ‘Boot mode’ button held down, and then try connecting to JTAG. They came back with what was, at the time of writing, the first screenshot of GDB-over-OpenOCD connected to the JH7100, at least from what I could find on the Internet.

Connected to GDB, yay!

Among the addresses found in the registers are:

Finally, we have JTAG access to the VisionFive.

Finding the JTAG port, take three

Even though technically we’re connected to the debug module, running the SoC entirely in Function 0 isn’t really an option as the board doesn’t seem to be configured this way. Once we switch to Function 6 though, our connection would be cut off.

Where do the JTAG signals now go go? As mentioned earlier, the JTAG signals are mapped by default to GPIO0 through GPIO4. Since we’re now in Function 6, these GPIO signals correspond to I/O pads… (looks at datasheet) PAD_FUNC_SHARE[4:0], which are 3.3 V and connected on the schematic to nets confusingly named GPIO0 through GPIO4.

Nets GPIOn on the schematic

These nets are connected to the 40-pin Raspberry Pi compatible header at pins… Wait what?

Nets GPIOn connected to the Raspberry Pi header

For some reason, these pins are labelled on the schematic with JTAG signal names. On any other documentation, such as StarFive’s GPIO Header Guide, they are only referred to as GPIO0, etc. and nowhere is JTAG mentioned.

A straightforward test of connecting the JTAG adapter while the system is up and running showed that these five pins… do not respond to the JTAG adapter.

Confusingly, Icenowy found out that for a certain period during the boot process, they could connect through those pins on the 40-pin header. Before this period, JTAG is found on the seven through-holes, and after that, it just seems to… disappear. Perhaps somewhere in the boot process another piece of code configured the second-stage GPIO FMUX and disconnected the JTAG signals. It makes sense because it frees up five pins on the header for actual GPIO.

Finding the JTAG port, take four

Around the same time when I asked NickCao to confirm the value of IO_PADSHARE_SEL in U-Boot, I also asked them to check the GPIO FMUX configuration, which showed that JTAG signals are connected correctly to GPIO0 through GPIO4. Therefore, whatever changed the configuration must have come after. Linux it is.

As mentioned before, the devicetree node starfive,jh7100-pinctrl manages both layers of GPIO multiplexing. The driver itself can be found in drivers/pinctrl/pinctrl-starfive.c.

Initially, I had assumed that the driver in StarFive’s fork of Linux was identical to that found in mainline Linux 5.18, though I would soon be proven wrong. As I was browsing through the dts files hoping to gain some insight on why the JTAG signals were gone, a section in jh7100-starfive-visionfive-v1.dts caught my eye.

&gpio {
    /* don't reset gpio mux for serial console and reset gpio */
    starfive,keep-gpiomux = <13 14 63>;
};

There was no mention of starfive,keep-gpiomux in mainline Linux’s version of pinctrl-starfive, but these two lines in StarFive’s version seemed relevant.

if (!keepmux)
    starfive_pinmux_reset(sfp);

keepmux is a module option defined as:

static bool keepmux;
module_param(keepmux, bool, 0644);
MODULE_PARM_DESC(keepmux, "Keep pinmux settings from previous boot stage");

starfive_pinmux_reset disables the inputs and outputs associated with every GPIOn signal, unless n was mentioned in starfive,keep-gpiomux.

The next step was clear: Boot with the kernel command line option pinctrl-starfive.keepmux=1, or modify jh7100-starfive-visionfive-v1.dts and add the JTAG signals to starfive,keep-gpiomux:

    starfive,keep-gpiomux = <0 1 2 3 4 13 14 63>;

Connecting to VisionFive with OpenOCD

At this point I borrowed the board from NickCao, and used a Raspberry Pi as an adapter to connect to it, over the JTAG pins on the 40-pin connector. Two TAPs can be found on the JTAG port, the first of which connects to the E24 core, and the second connects to the two U74 cores. This was my OpenOCD configuration file:

adapter driver linuxgpiod

linuxgpiod gpiochip 0
linuxgpiod jtag_nums 11 25 10 9
linuxgpiod trst_num 7

reset_config trst_only

transport select jtag

jtag newtap e24 cpu -irlen 5 -expected-id 0x200005fd
jtag newtap u74 cpu -irlen 5 -expected-id 0x200003fd

target create e24.cpu0 riscv -chain-position e24.cpu -coreid 0
target create u74.cpu0 riscv -chain-position u74.cpu -coreid 0 -rtos hwthread
target create u74.cpu1 riscv -chain-position u74.cpu -coreid 1
target smp u74.cpu0 u74.cpu1

init

Before Linux boots, I was able to connect to all of these cores, and read some information off of them. For example, reading CSRs from the U74 cores:

Reading some CSRs over GDB

As expected, this is a SiFive (mvendorid = 0x489) 7-series (marchid = (1 << 63) | 7) core, version 19.05 (mimpid = 0x20190531). Seen from the ISA implemented, namely RV64GC with Supervisor/User, this was certainly a U74 core.

After Linux starts, however, the E24 core seems to start misbehaving. OpenOCD starts generating error messages like:

Warn : target e24.cpu0 examination failed
Error: [e24.cpu0] DMI operation didn't complete in 2 seconds. The target is either really slow or broken. You could increase the timeout with riscv set_command_timeout_sec.

It seems that the E24 core had been disconnected or disabled. In any case, it was not responding. OpenOCD was confused by this lack of response and debugging on the U74 was also affected:

GDB errors due to E24 not responding

Commenting out the e24.cpu0 line in the config would ignore the E24 core and work around this issue.

# target create e24.cpu0 riscv -chain-position e24.cpu -coreid 0

What is going on with the E24 core? Seeing that the main star of the show, the dual U74 cores, have already been ‘conquered’, it’s probably an appropriate time to take a break. This article has already been filled with too many details, so we will look at the E24 ‘microcontroller’ core next time, as a side quest, in a future post.

(By the way: It seems that beta/sample versions of the now cancelled BeagleV Starlight board has a very similar JTAG configuration. If anyone still has one of these boards, it would be extremely interesting to try it out.)

To be continued…

References

I made a post to StarFive’s RVSpace forum with a how-to of connecting to VisionFive’s JTAG port, hoping it would help those looking to do low-level work on the board: https://forum.rvspace.org/t/connecting-to-visionfive-s-jtag-port-a-short-guide/514.

Acknowledgements

I would like to thank these people for helping out during the research: