many "undefined reference to" errors while building Nios V software
Hi, I created a Nios V based software, where VGA.c, VGA.h and main.c are included, after building the project, it indicated few "undefined reference to" errors, as shown in figure below. However, they are all defined in VGA.c, VGA.h as shown in figure, I don't know why this error got. Anyone can share any advice? Thanks in advance.
Nios® V Processor Installation and Hello World Execution, Part 1
Note: This article is an English translation of this Japanese article. Please refer to the original article for updates. Introduction This article targets those who are new to FPGA, covering the overview, installation, hardware construction, and execution of Hello World for the Nios® V/m embedded processor. Those who already have experience with FPGA may skim through certain parts like the installation. The Operating Systems used for operational verification are as follows: Windows 11 Enterprise Version: 22H2, Build: 22621.2715 Ubuntu 22.04.3 LTS (Jammy Jellyfish), Kernel version: 6.2.0-36-generic The work will primarily be conducted on Linux, and explanations will include actual logs. However, when commands differ significantly on Windows, they will be explained as necessary. Below are reference materials for the Nios® V/m embedded processor: Nios® V Embedded Processor Design Handbook Nios® V Processor Reference Manual NiosV tool setup for Eclipse CDT and OpenOCD 1. Overview of the Nios® V/m Embedded Processor 1.1 Instruction Set The Nios® V/m embedded processor supports the RV32IA instruction set. RV32I denotes a 32-bit integer instruction set, and the symbols following it indicate extensions. The main extensions are listed below. Extension Description M Integer Multiplication and Division A Atomic Instructions F Single-Precision Floating-Point D Double-Precision Floating-Point Q Quad-Precision Floating-Point C Compressed Instructions Therefore, R32IA supports 32-bit integer instructions and atomic instructions. The processor has 32 32-bit wide integer registers, implemented using block memories like M20K. As indicated by the extension symbols, floating-point registers (f0-f31, fcsr) are not implemented. For more details, please refer to RISC-V Unprivileged Spec v.20191213. 1.2 Configuration The basic configuration is shown below. Item Details General Purpose Register file 32-bit width x 32 integer general-purpose register file Arithmetic Logic Unit (ALU) Performs arithmetic, comparison, logical, and shift operations. Also used for generating addresses for loads and stores Control and Status Registers (CSR) Control registers Exception Controller Exception processing controller Interrupt Controller Interrupt controller, level-triggered Instruction Bus Instruction bus Data Bus Data bus RISC-V based Debug Module Debug module Here are the main differences from Nios® II: Item Nios® V/m Embedded Processor Nios® II Bus Arm* ABMA* AXI Avalon-MM Number of Interrupt Terminals 16 32 Debug Open OCD (On-Chip Debugger) compliant Proprietary specification 1.3 Instruction, Data Bus While Nios® II used the Avalon-MM bus, the Nios® V/m Processor has switched to Arm* ABMA* AXI. This change allows for exceptions to be triggered in the Nios® V/m Processor when the response from AXI is not OK. The types of AXI responses are as follows: RESP[1:0] Response Description 2'b00 OKAY Normal access OK 2'b01 EXOKAY Exclusive access OK 2'b10 SLVERR Slave returned an error 2'b11 DECERR No slave at the specified address For more details, please refer to Read and Write Response Structure. 1.4 Pipeline The Nios® V/m embedded processor is a 5-stage pipeline CPU. An overview of each stage is provided below. Stage Denotation Function Instruction Fetch F • PC+4 calculation<br>• Next instruction fetch<br>• Pre-decode for register file read Instruction Decode D • Decode the instruction<br>• Register file read data available<br>• Hazard resolution and data forwarding Instruction Execute E • ALU operations<br>• Memory address calculation<br>• Branch resolution<br>• CSR read/write Memory M • Memory and multicycle operations<br>• Register file write<br>• Branch redirection Write Back W • Facilitates data dependency resolution by providing General Purpose Register value. 1.5 Exceptions The exceptions currently defined are as follows: Exception Description Instruction Address Misaligned Exception is issued when the Program Counter (PC) is not aligned on a 32-bit boundary during the instruction fetch stage (i.e., the lower 2 bits of the PC are not zero). Instruction Access Fault Exception is issued if the AXI response during instruction read is not OKAY. Illegal Instruction Exception is issued if an unimplemented or undefined instruction is detected during instruction decode, or if an unimplemented or undefined CSR access is detected during the execute stage. Breakpoint Exception is issued at the decode stage when a break instruction is executed. Load Address Misaligned<br>Store/AMO Address Misaligned Exception is issued during the memory access stage if the load/store instruction's address is not aligned with the access size. Load Access Fault<br>Store/AMO Access Fault Exception is issued if the AXI bus returns a response other than OKAY during memory access. Environment Call from U-mode Exception is issued when ECALL instruction is executed in U-mode. Environment Call from S-mode Exception is issued when ECALL instruction is executed in S-mode. Environment Call from M-mode Exception is issued when ECALL instruction is executed in M-mode. Instruction Page Fault<br>Load Page Fault<br>Store/AMO Page Fault Exception is issued when accessing a virtual address that does not correspond to a physical address, or when access rights are insufficient. 1.6 CSR Registers The registers mapped in the CSR area are shown below. For more details on the registers, please refer to RISC-V Privileged Spec v.20190608. Register Name Description mvendorid Machine Vendor ID Register marchid Machine Architecture ID Register mimpid Machine Implementation ID Register mhartid Hart ID Register mstatus Machine Status Register mtvec Machine Trap-Vector Base-Address Register mip Machine Interrupt-pending Register mie Machine Interrupt-enable Register mcounteren Machine Counter-enable Register mepc Machine Exception Program Counter mcause Machine Cause Register The registers mapped in the memory-mapped area are shown below. Register Name Description Offset mtimecmp Machine Time Register 0x00 mtime Machine Time Register 0x08 1.7 Supported Devices The FPGA devices currently supported are as follows: Intel® Quartus® Prime Edition Device Family Pro Intel® Cyclone® 10 device Pro Intel® Arria® 10 device Pro Intel® Stratix® 10 device Pro Intel Agilex® 7 device Standard Intel® MAX® 10 Standard Intel® Cyclone® IV E Standard Intel® Cyclone® IV GX Standard Intel® Cyclone® V Standard Intel® Cyclone® 10 LP Standard Intel® Arria® 10 Standard Intel® Arria® II GX Standard Intel® Arria® II GZ Standard Intel® Arria® V Standard Intel® Arria® V GZ Standard Intel® Stratix® IV Standard Intel® Stratix® V 2. Obtaining a License To use the Nios® V/m embedded processor, you need to acquire a free license. This section explains the procedure for obtaining a license. Additionally, we will also obtain a license for Questa*-Intel® FPGA Edition Software, used as a simulator. 2.1 Sign In Click on the Self Service Licensing Center. A Sign In screen like the one below will appear, so please sign in with your Intel account. If you do not have an Intel account, please create one. 2.2 Obtaining a Nios® V/m Embedded Processor License After signing in, the following screen will appear. Select Intel® FPGA IP IP-NIOSVM, enter "1" in "# of Seats", and click "Get License" at the bottom right. 2.3 Registering a New Computer The following screen will appear. If you are obtaining a license for the first time, select "Create a New Computer". If you have already registered, select "Assign an Existing Computer" and click the "Generate" button at the bottom right. This time, we will register a new computer, creating an ID using the Ethernet MAC ID. 2.4 Entering New Computer Information The following screen will appear, where you set up the computer you are using. Enter the necessary items below and click "Generate License". Item Description Computer Name Enter any name License Type Select FIXED Computer Name Select NIC ID Primary Computer ID Enter the PC's Ethernet MAC ID 2.5 Checking the Nios® V/m Embedded Processor License File Once generated, the following screen will appear, and the license file will be sent to the registered email address, so please check the file. 2.6 Obtaining a Questa*-Intel® FPGA Edition Software License Follow the same procedure to obtain a license for Questa*-Intel® FPGA Edition Software. If you already have a license, proceed to the next chapter. As before, click on Self Service Licensing Center. Select Questa*-Intel® FPGA Starter software Edition SW-QUESTA, enter "1" in "# of Seats", and click "Get License" at the bottom right. 2.7 Adding to an Existing Computer If the following screen appears, specify the PC you registered earlier by selecting "Assign an Existing Computer", enter the MAC ID, and press the "Generate" button at the bottom right. 2.8 Checking the Questa*-Intel® FPGA Edition Software License File Once generated, the following screen will appear, and the license file will be sent to the registered email address. With this, the process of obtaining the license is complete. 3. Installation This chapter explains how to download and install Intel® Quartus® Prime Software, and how to set up environment variables. 3.1 Installing Intel® Quartus® Prime Pro Edition 23.3 3.1.1 Download First, click on the FPGA Software Download Center and go to the page below. Follow the instructions for your OS to click on the corresponding version of Intel® Quartus® Prime Software. This explanation will cover installation on Linux. Clicking will bring up the following page. Verify that the version is 23.3, then scroll down. Download the Intel® Quartus® Prime Pro Edition Installer (RUN) from the page below. For Windows, download the Intel® Quartus® Prime Pro Edition Installer (EXE). Once downloaded, enter the following commands in the download directory to launch the installer: $ chmod +x *.run $ ./qinst-linux-23.3-104.run For Windows, double-click the downloaded qint-windows-23.3-104.exe. When the installer launches, select: Intel® Quartus® Prime Pro Edition Part 1 (includes Nios II EDS) Intel® Quartus® Prime Pro Edition Part 2 Questa*-Intel® FPGA and Starter Editions Intel® Arria® 10 device support Ashling RiscFree IDE for Intel® FPGAs Check "Agree to Intel License Agreement" and click Download. If you wish to change the download or installation location, do so on this screen. For Windows, ensure the following are checked under After-install actions: Launch USB Blaster driver installation Launch USB blaster II driver installation Install JTAG Server as a Windows service During download & installation on Windows, the Device Driver Installation Wizard will open for Intel® FPGA Download Cable II (formerly USB Blaster II) and Intel® FPGA Download Cable (formerly USB Blaster), which are necessary for FPGA configuration and debugging the Nios® V/m embedded processor. Please install them. For Linux, follow "1.6. Installing the Intel FPGA Download Cable II Driver on Linux Systems" in the Intel® FPGA Download Cable II User Guide. Install the Intel® FPGA Download Cable (formerly USB Blaster) driver. Click Next. Click Finish. Install the Intel® FPGA Download Cable II (formerly USB Blaster II) driver. Click Next. Click Finish. This is the screen after completing the Quartus installation. With this, the installation of Intel® Quartus® Prime Pro Edition is complete. 3.2 Setting up Environment Variables 3.2.1 Linux Execute the following to set paths to tools and specify the license file. #!/bin/bash export QUARTUS_ROOTDIR="$HOME/intelFPGA_pro/23.3/quartus" export QSYS_ROOTDIR="$QUARTUS_ROOTDIR/../qsys/bin" export SOPC_KIT_NIOS2="${QUARTUS_ROOTDIR}/../nios2eds" # Tools QUARTUS="$QUARTUS_ROOTDIR/bin64" QSYS="$QSYS_ROOTDIR" NIOSV="$QUARTUS_ROOTDIR/../niosv/bin" NIOS2EDS="$QUARTUS_ROOTDIR/../nios2eds/bin" MAKE="$QUARTUS_ROOTDIR/../riscfree/build_tools/bin" CMAKE="$QUARTUS_ROOTDIR/../riscfree/build_tools/cmake/bin" GCC="$QUARTUS_ROOTDIR/../riscfree/toolchain/riscv32-unknown-elf/bin" if [ -d "$QUARTUS_ROOTDIR/../questa_fe" ]; then QSIM="$QUARTUS_ROOTDIR/../questa_fe/bin" elif [ -d "$QUARTUS_ROOTDIR/../questa_fse" ]; then QSIM="$QUARTUS_ROOTDIR/../questa_fse/bin" else echo "Questa not found" fi export PATH="$QUARTUS:$QSYS:$NIOSV:$NIOS2EDS:$MAKE:$CMAKE:$GCC:$QSIM:$PATH" # License files export LM_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat" export MGLS_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat" Regarding the license, $HOME/license/1-V4ICFB_License.dat is specified. If you have multiple license files, separate them with ":" in the following two lines: export LM_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat" export MGLS_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat" For example: # Example export LM_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat:$HOME/license/XXX_License.dat" export MGLS_LICENSE_FILE="$HOME/license/1-V4ICFB_License.dat:$HOME/license/XXX_License.dat" 3.2.2 Windows For Windows, use the Nios V Command Shell, so setting the tool paths is not necessary. Only specify the license. Press the Windows key on your keyboard and type "env". Then select "Edit the system environment variables". The following dialog opens, click on "Environment Variables". In the User Environment Variables settings, as shown in the red frame, specify LM_LICENSE_FILE and MGLS_LICENSE_FILE to the license file. If there are multiple files, separate them with ";". After restarting, the setup is complete. 4. Hardware Construction 4.1 Design Sample & Preparation Here, we construct a design using the Nios® V/m embedded processor. Please download the Intel® Arria® 10 FPGA - Nios® V/m Processor based Hello World version 23.3 from the FPGA Design Store. The file name is top.par. Once the download is complete, copy top.par to your work directory. If your work directory is $HOME/ws, then execute: cp $HOME/Downloads/top.par $HOME/ws For Windows, if your work directory is C:\ws, then drag and drop to copy the file. Next, set up the environment variables in Terminal for Linux or launch Nios V Command Shell (Quartus Prime Pro 23.3) from the Start menu in Windows. Move to $HOME/ws and execute the following: quartus top.par & When Quartus launches, it will ask for the location to expand the Example Design. Specify your preferred work directory and click OK. From now on, we will proceed assuming the Example Design is expanded in $HOME/ws/top_project for Linux and C:/ws/top_project for Windows. All terminal commands should be executed from $HOME/ws/top_project or C:/ws/top_project. As you can see in $HOME/ws/top_project/hw/top.v, this design uses only one 50MHz clock input, making it easy to port to other FPGAs. For more details, see 4.5 Porting to Other FPGAs. For the reset, we use an IP that controls signals from JTAG called ISSP (In-System Source & Probe) and perform the reset with toggle_issp.tcl. For more details, see 5.1.6 CPU Reset by ISSP (In-System Source & Probe). File Description readme.txt README file hw/top.qpf Quartus Project file hw/top.qsf Quartus Settings file hw/top.sdc Synopsys Design Constraints file hw/top.v Design TOP RTL file scripts/create_qsys.tcl Platform Designer creation script scripts/toggle_issp.tcl Reset execution script sw/app_hal/hello.c Main file for HAL Hello World sw/app_ucosii/hello_ucosii.c Main file for uC/OS-II Hello World sw/app_freertos/hello_freertos.c Main file for FreeRTOS™ Hello World As a premise, the Example Design just downloaded and expanded is in a ready-to-test state, meaning all files and binaries required for running the Nios® V/m embedded processor are already included. The FPGA configuration file is at $HOME/ws/top_project/ready_to_test/top.sof, and the ELF files for the Nios® V/m embedded processor application are at $HOME/ws/top_project/ready_to_test/app_hal.elf, $HOME/ws/top_project/ready_to_test/app_ucosii.elf, $HOME/ws/top_project/ready_to_test/app_freertos.elf. Use these for testing on actual hardware. The following will explain the steps for compiling the hardware design and building the application for the Nios® V/m embedded processor. 4.2 Checking the Platform Designer Design Let's check the Platform Designer design in the GUI. There are two ways to launch Platform Designer: one is to launch it from the Intel® Quartus® Prime software, and the other is to launch it via command from the Terminal. 4.2.1 Launching Platform Designer from Intel® Quartus® Prime Software Execute the following in the Terminal to launch Quartus: quartus & For Windows, you can also launch Quartus from the Start menu by selecting Quartus Prime Pro Edition 23.3.0.104 -> Quartus (Quartus Prime Pro 23.3). From the File menu, select "Open Project..." and specify $HOME/ws/top_project/hw/top.qpf, then click "Open". From the Tools menu, select "Platform Designer". In the "Open System" dialog that appears, click the red frame and specify the $HOME/ws/top_project/hw/sys.qsys file. Click Open. 4.2.2 Launching Platform Designer from Terminal Move to $HOME/ws/top_project and enter the following command: qsys-edit --quartus-project=hw/top.qpf hw/sys.qsys 4.2.3 Platform Designer Design Here, we'll check the design configuration and settings for each IP. The following is the overall design configuration: 4.2.3.1 Clock This is the configuration screen for the clock input IP. If the input clock differs, change this value. If you change it, also modify the clock period in top.sdc (period is specified in ns). For IP details, please refer to 7.1.1. Clock Bridge Intel® FPGA IP. 4.2.3.2 Reset This is the configuration screen for the reset input IP. For IP details, refer to 6.6.3. Reset Bridge. 4.2.3.3 Nios® V/m Embedded Processor This is the configuration screen for the Nios® V/m embedded processor IP. For details on each parameter, refer to the following: 2.1.1.2.1. Debug Tab 2.1.1.2.2. Use Reset Request Tab 2.1.1.2.3. Vectors Tab 2.1.1.2.4. CPU Architecture 2.1.1.2.5. ECC Tab 4.2.3.4 Internal RAM This is the configuration screen for the internal RAM IP. For details on each parameter, refer to the following: 25.2.1. Memory Type 25.2.2. Size 25.2.3. Read Latency 25.2.4. ROM/RAM Memory Protection 25.2.5. ECC Parameter 4.2.3.5 JTAG UART This is the configuration screen for the JTAG UART IP. For details on each parameter, refer to the following: 12.3.1.1. Write FIFO Settings 12.3.1.2. Read FIFO Settings Once you have confirmed the settings, please close Platform Designer. 4.3 Compiling the Design Next, execute the following command in the Terminal to start compiling the design: quartus_sh --flow compile hw/top Output: Info: ******************************************************************* Info: Running Quartus Prime Shell ... [Output Truncated for Brevity] ... Info: Quartus Prime Shell was successful. 0 errors, 35 warnings Info: Peak virtual memory: 1034 megabytes Info: Processing ended: Mon Nov 27 17:18:51 2023 Info: Elapsed time: 00:02:25 Info: System process ID: 64166 If you look at line 10 in $HOME/ws/top_project/hw/top.qsf: set_global_assignment -name PROJECT_OUTPUT_DIRECTORY output_files This indicates where the design compilation outputs are stored, and the .sof file is generated in $HOME/ws/top_project/hw/output_files. 4.4 Configuring the FPGA Execute the next command to configure the FPGA. Note that the command slightly differs between Linux and Windows. Linux: quartus_pgm -c 1 -m JTAG -o p\;hw/output_files/top.sof@1 Output: Info: ******************************************************************* Info: Running Quartus Prime Programmer ... [Output Truncated for Brevity] ... Info: Quartus Prime Programmer was successful. 0 errors, 0 warnings Info: Peak virtual memory: 1917 megabytes Info: Processing ended: Mon Nov 27 17:20:32 2023 Info: Elapsed time: 00:00:13 Info: System process ID: 67118 Windows: quartus_pgm -c 1 -m JTAG -o p;hw/output_files/top.sof@1 Output: Info: ******************************************************************* Info: Running Quartus Prime Programmer ... [Output Truncated for Brevity] ... Info: Quartus Prime Programmer was successful. 0 errors, 0 warnings Info: Peak virtual memory: 1296 megabytes Info: Processing ended: Tue Nov 28 23:35:04 2023 Info: Elapsed time: 00:00:17 Info: System process ID: 26872 4.5 Porting to Other FPGAs If you are porting this design to other FPGA boards, the main changes will be the device and clock pins. From the Assignments menu, select "Device..." and in the dialog that appears, specify the device you are using. Once the device is changed, Quartus will display "IP update required" in red at the top left. Click "Launch IP Upgrade Tool...". In the dialog that appears, click "Auto Upgrade" to update the IP. Once the IP update is complete and the Status column shows "Success," you can close the dialog. From the Processing menu, select “Start Compilation..." to re-run the compilation. You may encounter errors in the Fitter stage if you haven't set up the clock pin, but you can ignore these for now. From the Assignments menu, select "Pin Planner..." and in the dialog that appears, update the location of the clock pin. Also, update the I/O Standard accordingly. Finally, from the Processing menu, select “Start Compilation..." again and ensure that the compilation runs without errors this time. References Nios® V processor Boot, Simulation, and Debug Edition Nios® V Embedded Processor Design Handbook Nios® V Processor Reference Manual NiosV tool setup for Eclipse CDT and OpenOCD RISC-V Unprivileged Spec v.20191213 RISC-V Privileged Spec v.20190608 RISC-V External Debug Support v.0.13.2 AMBA AXI and ACE Protocol Specification AXI3, AXI4, and AXI4-Lite ACE and ACE-Lite Notices & Disclaimers Intel technologies may require enabled hardware, software, or service activation. No product or component can be absolutely secure. Your costs and results may vary. © Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from the course of performance, course of dealing, or usage in trade. Nios is a trademark of Intel Corporation or its subsidiaries.The error message is as follows: "nios2-bsp-update-settings : command not found"
I downloaded the A10_DP_RX_FMC_PRO.par file and compiled it with Quartus Pro 19.2. AN 900: Intel® Arria 10 DisplayPort 8K RX-only Design Link: https://www.intel.com/content/www/us/en/docs/programmable/683799/current/compiling-the-design.html After opening the altera Nios2 Command Shell, I proceeded as in document an900. When I run source build_sw.sh in the command shell, the following error occurs. The error message is as follows: "nios2-bsp-update-settings : command not found" I send you a screenshot of the file directory contents and the execution result. I have a Quartus Pro license. I know that nios2 does not require a license. Is this error related to the nios2 licnse? How to solve this error ? ThanksIs it possible to run NIOSV through SDRAM ?
Hello, I am developing a project where NIOS II needs to replaced with NIOSV processor. When my design was having NIOS II as CPU, I was running the software code through External SDRAM on the Cyclone IV E Kit(EP4CE40F23C8). It's working fine with NIOS II. Able to run the software program. Now, when I replaced the NIOS II processor with the NIOSV processor, I am not getting the expected results. I am trying to print a Simple "Hello World" code through the NIOSV processor with the same configuration of design. Almost all BSP settings are same for NIOSV processor as of the NIOS II. Still not getting the results. When running the NIOSV from On-Chip-Ram, everything works as expected, but when I try to run the NIOSV through external SDRAM, the code does not gets executed as expected. I want to inquire that if we can run the NIOSV processor through external SDRAM or not ? Please note that I am not talking here about booting the NIOSV. I am simply running the software image through RiscFree IDE software on the external SDRAM of the board. Regards, Himanshu
Nios V uC/TCP IP Failed
Hi support, I am learning the simple socket server design on Cyclone 10 GX. For unknown reason, the initialization process stopped with the error code (2010), It's the TSE with DP83848. The signals on MDIO and MDC is working. Therefore I would like to know how to debug this issue. Thank you. BRs, Johnson
AXI4 DMA Master Error
Hi expert, I am building SoC system that includes components as follows: + CPU + onchip_memory_2_0 + onchip_memory_2_1 (source address) + onchip_memory_2_2 (destination address) + IP (Active Ascon) has AXI4 DMAC As program runs at simulation mode, read burst request runs well, but write burst always encounters problem (awready is always "0" logic level), so DMAC always wait slave, leading my system to be stuck. Could you clarify it to me? Thanks you so much. Attached imageAbout OpenOCD
Note: This article is an English translation of this Japanese article. Please refer to the original article for updates. Introduction In this “Intel® FPGA Advent Calendar 2021”, various articles have highlighted the RISC-V version of the Soft CPU Core, known as the Nios® V processor (hereinafter referred to as Nios V), which is now included in Intel® Quartus® Prime Pro Edition 21.3 (hereafter QuartusPro v21.3). Along with this introduction, OpenOCD has been implemented as the debugging mechanism for Nios V. It raises the question: Could OpenOCD be used for JTAG debugging with USB Blaster, not just for Nios V but for other CPUs as well? This article explores that possibility. The experiment conducted focused on the CPU (Arm Cortex-A9) of the Cyclone® V SoC FPGA1. This section includes the OpenOCD configuration file for connecting to the Cyclone V SoC, along with gdb debugger scripts for downloading and executing code, and a TCL script for telnet connection. What is OpenOCD? OpenOCD, short for Open On-Chip Debugger, is an "Open Source software aimed at providing debugging, in-system programming, and boundary-scan for embedded devices."2 It incorporates features of a gdb server and a telnet server, supporting a variety of debug probes, thereby enabling debugging on numerous embedded devices. The basic usage involves running OpenOCD as a server in the background, acting as an intermediary to facilitate connections between client software (like gdb or telnet clients) and the debug probe. The version of OpenOCD included in QuartusPro v21.3 supports USB Blaster, allowing JTAG debugging via gdb debugger, targeting not just Nios V but potentially other CPUs as well, such as Arm Cortex-A9. This section delves into the investigation and testing of this capability. For setting up OpenOCD, on Windows, the installation of QuartusPro v21.3 is required. On Linux, installing only the Programmer of QuartusPro v21.3 suffices for using OpenOCD. No additional installation of the Nios V toolchain is necessary. Furthermore, OpenOCD is available for use free of charge, without the need for a paid license. Overall Structure This section outlines the tested configuration. Hardware: The debugger PC and the FPGA board are connected via JTAG (actually using an OnBoard USB Blaster, so a USB cable). The PC can be either Windows or Linux. PC Software: OpenOCD is kept running, with the gdb server accessing the USB Blaster being active. The gdb debugger for Arm CPU is connected to the OpenOCD gdb server to facilitate debugging, which includes code downloading and execution. Board Used for Testing The Terasic DE10-Nano kit, equipped with the Cyclone V SoC, was used for the experiment. Launching OpenOCD To start OpenOCD, the Nios V Command Shell from Quartus v21.3 is used. Initially, it's necessary to verify whether the USB Blaster of the DE10-Nano kit is correctly recognized by the PC as a JTAG device and if the HPS and FPGA are detected. This involves connecting the J13 of the DE10-nano to the PC using a mini-USB cable without an SD card inserted, and then powering on the DE10-nano. Following this, the jtagconfig command is executed. [niosv-shell] C:\intelFPGA_pro\21.3> jtagconfig 1) DE-SoC [USB-1] 4BA00477 SOCVHPS 02D020DD 5CSEBA6(.|ES)/5CSEMA6/.. If this display appears, it is normal. It confirms the JTAG ID of the HPS containing the Arm CPU as 4BA00477 (the same as in Arria10's HPS). Creating OpenOCD Configuration File To use OpenOCD, a configuration file detailing the target environment is required. The format adheres to TCL. Initially, reference files within the installation folder of QuartusPro v21.3 are sought. Within the folder named <install_folder>/quartus/bin64/oocd/openocd/scripts/board there is a file named altera_arria10__aji_client.cfg. It appears to be a configuration file for connecting to the HPS of Arria10 SoC, as inferred from its name and comments. Using this as a reference, a successful connection to the target Cyclone V SoC was established. The configuration file is as follows, named cyclone5_client.cfg. ############################################################### # Point: Specify aji_client as the Driver adapter driver aji_client ############################################################### # Chip Name: Can be changed to any name you prefer (let's keep it as cvsoc) set _CHIPNAME cvsoc ############################################################### # JTAG ID of HPS TAP (Test Access Port)/DAP (Debug Access Port) set _DAP_TAPID 0x4ba00477 ############################################################### # Declare HPS DAP as the JTAG TAP (Test Access Port) jtag newtap $_CHIPNAME cpu -irlen 4 -expected-id $_DAP_TAPID ############################################################### # Declare that the HPS TAP is an ARM DAP (Debug Access Port). # Do this when ARMv6-M, ARMv7, ARMv8 are the targets dap create $_CHIPNAME.dap -chain-position $_CHIPNAME.cpu ############################################################### # The target to be operated by the gdb debugger will be core0 of the HPS DAP target create $_CHIPNAME.cpu.0 cortex_a -dap $_CHIPNAME.dap -coreid 0 -dbgbase 0x80110000 After several trials and errors, the file was ultimately reduced to only the essential parts from the original Arria10 file. One addition, in the last line of target create…, is -dbgbase 0x80110003. Launching and Verifying OpenOCD The Nios V Command Shell of v21.3 is started. In this Shell, after changing the directory to where the cyclone5_client.cfg file was saved, execute openocd -f cyclone5_client.cfg. Subsequently, the following log appears, and then the log display stops. [niosv-shell] C:\de10nano\OpenOCD_download> openocd -f cyclone5_client.cfg Open On-Chip Debugger 0.11.0-R21.3 Licensed under GNU GPL v2 ...[omitted for brevity]... Info : Listening on port 4444 for telnet connections ...[omitted for brevity]... Info : At present, The first hardware cable will be used [1 cable(s) detected] Info : Cable 1: device_name=(null), hw_name=DE-SoC, server=(null), port=USB-1, chain_id=0000019c3ca88620, persistent_id=1, chain_type=1, features=2048, server_version_info=Version 21.1.0 Build 842 10/21/2021 SJ Standard Edition Info : TAP position 0 (4BA00477) has 0 SLD nodes Info : TAP position 1 (2D020DD) has 0 SLD nodes Info : Discovered 2 TAP devices Info : Detected device (tap_position=0) device_id=4ba00477, instruction_length=4, features=0, device_name=SOCVHPS Info : Found a ARM device at tap_position 0. Currently assume it is JTAG-DP capable Info : Detected device (tap_position=1) device_id=02d020dd, instruction_length=10, features=4, device_name=5CSEBA6(.|ES)/5CSEMA6/.. Info : Found an Intel device at tap_position 1.Currently assuming it is SLD Hub Info : This adapter doesn't support configurable speed Info : JTAG tap: cvsoc.cpu tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd), part: 0xba00, ver: 0x4) Info : JTAG tap: auto0.tap tap/device found: 0x02d020dd (mfg: 0x06e (Altera), part: 0x2d02, ver: 0x0) Warn : AUTO auto0.tap - use "jtag newtap auto0 tap -irlen 10 -expected-id 0x02d020dd" ...[omitted for brevity]... Info : starting gdb server for cvsoc.cpu.0 on 3333 Info : Listening on port 3333 for gdb connections From the first half of the log, it's evident that the telnet server has started, waiting for connections on port 4444. The latter half, beginning with "Info: JTAG tap:", indicates that the device ID has been correctly recognized. Furthermore, from the last two lines, it can be seen that the gdb server has been successfully launched and is waiting for connections on port 3333, confirming a successful connection to the Cyclone V HPS gdb server. By the way, the "Warn: AUTO auto0.tap…" message that appears in between indicates that the FPGA's JTAG TAP definition was omitted in the cyclone5_client.cfg file, leading the tool to automatically detect it. The JTAG instruction length and other details are also auto-detected. Connecting to the gdb Debugger Now that OpenOCD has been successfully launched, the next step is to connect it with the gdb debugger. Starting and Verifying gdb The gdb debugger is initiated and connected to the OpenOCD gdb server. Here, the Arm gdb included with SoCEDS v18.1 standard edition is used (generally, any gdb supporting the target CPU should work, but there are concerns that if an ELF file is read by gdb, the symbol table might not be loaded correctly unless the gdb version matches the gcc toolchain that created the ELF). Several reasons for using v18.1 include: The sample HW design for DE10-nano compiles without issues (up to v18.1, the sample containing FrameReader Video IP can be compiled). There's no need to download & install Cygwin or WSL separately, among other reasons4. The SoCEDS command shell is launched, and from there, the arm-altera-eabi-gdb for Arm is started. It's crucial to include the option for connecting to OpenOCD's gdb server on port 3333 at this stage. arm-altera-eabi-gdb -ex 'target extended-remote localhost:3333' $ arm-altera-eabi-gdb -ex 'target extended-remote localhost:3333' GNU gdb (Sourcery CodeBench Lite 2016.11-88) 7.11.1.20160608-git Copyright (C) 2016 Free Software Foundation, Inc. ...[omitted for brevity]... Remote debugging using localhost:3333 warning: No executable has been specified and target does not support determining executable automatically. Try using the "file" command. 0x00002fa8 in ?? () (gdb) The gdb server is successfully connected on localhost:3333, and the CPU is stopped at address 0x00002fa8. This also adds a few lines of messages to the Nios V command shell that is running OpenOCD (examples are omitted here). Next, to see if the state of the Arm CPU is visible, the info reg command is used to check. (gdb) info reg r0 0x0 0 r1 0x0 0 r2 0xfffffee0 4294967008 r3 0xff704000 4285546496 r4 0x77857713 2005235475 r5 0xffd02000 4291829760 r6 0x76 118 r7 0xfffff014 4294963220 r8 0xfffff014 4294963220 r9 0x76 118 r10 0xffd02000 4291829760 r11 0xffd02000 4291829760 r12 0xa0002041 2684362817 sp 0xfffffed0 0xfffffed0 lr 0x3104 0x3104 pc 0x2fa8 0x2fa8 cpsr 0x600001d3 1610613203 (gdb) All register contents are successfully verified. Trying the print command for system memory access is also explored. Here, the crc register of the System Manager, mapped at address 0xffd080f0, is accessed. (gdb) print/x *0xffd080f0 $1 = 0xe763552a (gdb) As per the Cyclone V SoC documentation, the value is at its initial state after a cold reset, which is 0xE763552A. This confirms that all basic gdb commands are functioning correctly! To exit gdb, use the 'q' command. Also, terminate OpenOCD by inputting Ctrl-C. Download and Run Script with gdb Now, let's proceed to the final goal of this article: downloading and running code using gdb scripts. Below are some script examples, tailored to the file format and version of the preloader. Basically, converting the contents of the scripts (*.ds) provided for Arm DS (or Arm DS-5) to gdb format will suffice. ELF Format Preloader (v2013.01) First, a script for downloading and executing an ELF format file of the preloader version 2013.01. For a long time (until SoCEDS v18.1), the preloader (u-boot) code included with SoCEDS was of version 2013.01. The script example for downloading and executing the resulting ELF format file (spl/u-boot-spl) on the target device (DE10-Nano's Cyclone V SoC) is as follows (created as download_preloader_elf.gdb): set confirm off set pagination off restore symbol-file -readnow thbreak spl_boot_device jump _start #Stop watchdog timer #permodrst Reg , reset watch dog timer #set $permodrst = (int *)0xffd05014 #set *$permodrst = (*$permodrst) | (1<<6) #set *$permodrst = (*$permodrst) & ~(1<<6) [Explanation of the Script] The key point here is setting a breakpoint at the function spl_boot_device. This function checks where the preloader will load the next Application image from (SD card or QSPI or NAND). When this point is reached, all necessary initializations for booting are completed, and the debugger stops here (further execution will result in an error as no SD card is inserted). In most cases, the next step would be to download and debug the Application. The final three lines (set ...), which are commented out, are meant to stop the WatchDog timer started by the preloader. They toggle the reset bit of the WatchDogTimer0 in the reset manager to stop WatchDogTimer0. If additional tasks, like running an Application, are to be performed afterward, the WatchDog timer, if left running, might cause a reset due to a timeout (leading to an unexpected reset). To prevent this, uncomment these last three lines. This script is executed using the -x option with gdb. arm-altera-eabi-gdb -ex 'target extended-remote localhost:3333' -x download_preloader_elf.gdb Important: Before running this gdb command, please restart the DE10-Nano board5 and restart OpenOCD. $ arm-altera-eabi-gdb -ex 'target extended-remote localhost:3333' -x download_preloader.gdb GNU gdb (Sourcery CodeBench Lite 2016.11-88) 7.11.1.20160608-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> ...[omitted for brevity]... Remote debugging using localhost:3333 warning: No executable has been specified and target does not support determining executable automatically. Try using the "file" command. 0x00002fa4 in ?? () Restoring section .text (0xffff0000 to 0xffff6968) Restoring section .rodata (0xffff6968 to 0xffff81f0) Restoring section .data (0xffff81f0 to 0xffff90c4) Hardware assisted breakpoint 1 at 0xffff1342 cvsoc.cpu.0 rev 0, partnum c09, arch f, variant 3, implementor 41 Temporary breakpoint 1, 0xffff1342 in spl_boot_device () (gdb) When executing this command, if you have the DE10-Nano's serial terminal open, the following display occurs, confirming that the preloader is executed: U-Boot SPL 2013.01.01 (Dec 14 2021 - 13:59:40) BOARD : Altera SOCFPGA Cyclone V Board ...[omitted for brevity]... INFO : Watchdog enabled SDRAM: Initializing MMR registers SDRAM: Calibrating PHY SEQ.C: Preparing to start memory calibration SEQ.C: CALIBRATION PASSED SDRAM: 1024 MiB ELF Format Preloader (v2021.04 and Later) Preloader, based on the u-boot code, has undergone changes since version 2021.04 to include device tree files. Consequently, the debugger download script has also needed to be modified. Below is the updated script: set confirm off set pagination off restore symbol-file -readnow restore u-boot-spl.dtb binary &__bss_end thbreak spl_boot_device jump _start #Stop watchdog timer #permodrst Reg , reset watch dog timer #set $permodrst = (int *)0xffd05014 #set *$permodrst = (*$permodrst) | (1<<6) #set *$permodrst = (*$permodrst) & ~(1<<6) Explanation of the Script The new preloader (u-boot) requires loading both the ELF file and the device tree (.dtb) into memory. The line restore u-boot-spl.dtb binary &__bss_end loads the device tree. __bss_end is the end address of the preloader code, and the .dtb is placed immediately after it in memory, as per the specifications of this version of preloader (u-boot). Binary Format Preloader-mkpimage.bin This script is used to execute the preloader using the binary format file preloader-mkpimage.bin, intended for writing to a boot SD card. (This file comes included with the sample design DE10_NANO_SoC_GHRD folder provided by Terasic, useful for those who want a quick operational check). The script can also launch the preloader binary file u-boot-splx4.sfp, created for newer versions (later than 2013.xx) of boot media. set confirm off set pagination off restore binary 0xffff0000 0x0 0xefff watch *0xff704004 jump *0xffff004c #Stop watchdog timer #permodrst Reg , reset watch dog timer #set $permodrst = (int *)0xffd05014 #set *$permodrst = (*$permodrst) | (1<<6) #set *$permodrst = (*$permodrst) & ~(1<<6) Explanation of the Script The key point is specifying the memory offset address (0xffff0000) for writing the binary, the start (0x0), and the end position (0xefff) of the binary file in the restore command. The memory offset address (0xffff0000) is the starting address of the HPS OnChipMemory. The start of the binary file is from the beginning (0x0), and the end is the maximum size of the preloader code, 0xefff. A watch point (triggered on specified memory access) is used to stop the preloader since it's a binary code without symbol information. The watch point is set to halt the CPU when the SD card controller is accessed for reading/writing. 0xff704004 is the address of a register in the SD card controller. The execution start address is 0xffff004c, as specified by the binary format and bootrom specifications. Downloading and Running Preloader/Baremetal App As a more practical example, this script loads and executes the preloader, followed by a sample Baremetal application. set confirm off set pagination off restore <u-boot-splのpath> symbol-file -readnow <u-boot-splのpath> thbreak spl_boot_device jump _start #Stop watchdog timer #permodrst Reg , reset watch dog timer set $permodrst = (int *)0xffd05014 set *$permodrst = (*$permodrst) | (1<<6) set *$permodrst = (*$permodrst) & ~(1<<6) # load and run Application restore <application elf path> symbol-file -readnow <application elf path> jump _start Explanation of the Script: This script adds commands to download and execute an application ELF file (.axf) to the previously introduced script for downloading the preloader. After executing the preloader, a process to stop the WatchDog Timer is enabled. Ideally, this process to stop the WatchDog Timer should be performed at the beginning of the application. Since this process is not included in the Software Examples provided with SoCEDS, it's incorporated here. Caution When Building the Software Example Included with SoCEDS: This time, I tried compiling and verifying the operation of the Software Example included with SoCEDS v18.1, specifically the Altera-SoCFPGA-HardwareLib-16550-CV-GNU (/embedded/examples/software/Altera-SoCFPGA-HardwareLib-16550-CV-GNU). Some points to note: the original sample code is intended to be executed using an Arm debugger and is set up to display printf output on the debugger's console (using a feature called semi-hosting). With the gdb debugger, this feature needs to be turned off. For this purpose, line 46 of the Makefile was changed to LINKER_SCRIPT := cycloneV-dk-ram.ld. Telnet Connection Now, let's try connecting to OpenOCD using telnet. With a telnet client, one can directly execute commands supported by OpenOCD6. Additionally, a TCL interpreter called Jim-Tcl runs as the command-line interface, enabling the execution of TCL scripts. First, ensure that OpenOCD is running. By default, OpenOCD's telnet server waits for connections on port 4444. The following image shows an example using Tera Term as the Telnet client. To start, configure the settings as shown in the figure: Host: localhost, Service: Telnet, TCP port#: 4444, and click "OK". After a few seconds, if you see text output from OpenOCD and a prompt for command input, the connection is successful. Open On-Chip Debugger > Simultaneously, the following log should be added to the Shell window where OpenOCD is running. Info : accepting 'telnet' connection on tcp/4444 Next, here's an example of executing three OpenOCD commands related to CPU debugging (halt, reg, mdw). > # Halt CPU > halt cvsoc.cpu.0 rev 0, partnum c09, arch f, variant 3, implementor 41 target halted in ARM state due to debug-request, current mode: Supervisor cpsr: 0x600f0153 pc: 0x00101fac MMU: disabled, D-Cache: disabled, I-Cache: enabled > > > # print registers > reg ===== ARM registers (0) r0 (/32): 0x3fffdfa4 (dirty) (1) r1 (/32): 0x3fffd300 (dirty) (2) r2 (/32): 0x00000200 (3) r3 (/32): 0x00000000 (4) r4 (/32): 0x00000000 (5) r5 (/32): 0x00000000 (6) r6 (/32): 0x00000076 (7) r7 (/32): 0xfffff014 (8) r8 (/32): 0xffffffff (9) r9 (/32): 0x00000005 (10) r10 (/32): 0xffd02000 (11) r11 (/32): 0x3fffd2ec (12) r12 (/32): 0xffffffff (13) sp_usr (/32) (14) lr_usr (/32) (15) pc (/32): 0x00101fac (16) r8_fiq (/32) (17) r9_fiq (/32) ...[omitted for brevity]... (73) d29 (/64) (74) d30 (/64 > > # Read a memory mapped register > mdw 0xffd05014 0xffd05014: 01b6c035 Additionally, let's check other OpenOCD commands such as ```scan_chain``` and ```dap info```. > scan_chain TapName Enabled IdCode Expected IrLen IrCap IrMask -- ------------------- -------- ---------- ---------- ----- ----- ------ 0 cvsoc.cpu Y 0x4ba00477 0x4ba00477 4 0x01 0x03 1 auto0.tap Y 0x02d020dd 0x00000000 10 0x01 0x03 > dap info DAP transaction stalled (WAIT) - slowing down AP ID register 0x24770002 Type is MEM-AP APB MEM-AP BASE 0x80000000 ROM table in legacy format Component base address 0x80000000 Peripheral ID 0x00000ee001 Designer is 0x0ee, Altera Part is 0x1, Unrecognized Component class is 0x1, ROM table MEMTYPE system memory not present: dedicated debug bus ROMTABLE[0x0] = 0x1003 Component base address 0x80001000 Peripheral ID 0x04001bb961 Designer is 0x4bb, ARM Ltd Part is 0x961, CoreSight TMC (Trace Memory Controller) Component class is 0x9, CoreSight component Type is 0x32, Trace Link, FIFO, buffer ...[omitted for brevity]... [L01] ROMTABLE[0x20] = 0x0 [L01] End of ROM table ROMTABLE[0x24] = 0x0 End of ROM table > The dap info command displays a list of the contents of the Arm CoreSight DAP's ROM table. Downloading and Running the Preloader/Baremetal App A TCL script has been written to download and run both the preloader and a baremetal application. As noted in the file's comments, unlike with gdb, one needs to determine the symbol-to-address mapping in advance7. Once saved as 'download_preloader_app.tcl', it can be executed from the OpenOCD prompt in the telnet terminal using source download_preloader_app.tcl. Make sure to adjust the parts enclosed in […] to fit your environment. Note that if a relative path is specified for the file path, it should be relative to the current working directory of the Shell where OpenOCD is launched. Similarly to gdb, before executing through the telnet connection, restart the DE10-Nano board5 and restart OpenOCD. halt ################################################################### # Download the Preloader # Specify using a relative path from the pwd of the shell where OpenOCD was launched. load_image [path to preloader's elf file] ################################################################### # Setting the Start Address for Execution reg pc 0xffff0000 ################################################################### # Set a breakpoint at the spl_boot_device function in the preloader # Example of how to find this address: # $ arm-altera-eabi-readelf.exe -s [path to preloader elf file] | grep spl_boot_device # bp [address of spl_boot_device function] 2 hw ################################################################### # Execute the preloader resume wait_halt ################################################################### # Reset the WatchDog Timer set permodrst 0xffd05014 set permodrst_val "0x"[lindex [mdw $permodrst] 1] mww $permodrst [expr ($permodrst_val)|(1<<6)] mww $permodrst $permodrst_val ################################################################### # Load the Baremetal Application load_image [path to application elf] ################################################################### # How to find the start address of the application # (for SOCEDS bundled Altera-SoCFPGA-HardwareLib-16550-CV-GNU) # arm-altera-eabi-readelf.exe -s [path to application elf] | grep _start reg pc [start address of application] ################################################################### # Clear Thumb2 Mode # cpsr bit-5 : Thumb execution state bit - to be cleared set cpsr_val [lindex [reg cpsr] 2] reg cpsr [expr ($cpsr_val)&(~(1<<5)&0xffffffff)] ################################################################### # Execute the application resume Summary It has been confirmed that with OpenOCD included in QuartusPro v21.3, code download and execution to the Arm Cortex-A9 CPU of Cyclone V SoC is possible via gdb and telnet connections. It's now feasible to perform JTAG Debugging for Intel SoC FPGAs, which previously required a paid license, with free tools. Of course, paid debuggers excel in terms of code download speed and GUI environment8. While "official documentation support is still anticipated," it is hoped that when the support documents are released, there will be improvements in usability and enhanced features. Postscript Although the name OpenOCD was somewhat familiar, it had not been used until now. However, the opportunity arose with the inclusion of precompiled binaries in the tool, prompting an early trial. First impressions are: JTAG debugging with gdb is convenient and straightforward. Telnet connection looks promising for various operations around JTAG. It also seems convenient when combined with Tera Term's macros. The source code is also available on GitHub (see References), which could be interesting for enthusiasts. That's the general feeling. Thank you. (Comments and feedback are welcome!) References Open On-Chip Debugger official site Link to intel/aji_openocd on GitHub Link to intel/libaji_client on GitHub Installation and Hello World execution for Nios® V processor Boot, simulation, and debugging for Nios® V processor Attempted porting of an existing NiosR II/e system to NiosR V/m Notices & Disclaimers Intel technologies may require enabled hardware, software, or service activation. No product or component can be absolutely secure. Your costs and results may vary. © Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from the course of performance, course of dealing, or usage in trade. While QuartusPro v21.3 does not support this device, it was challenged due to the desire to see it operate on a reasonably priced FPGA. ↩︎ From the official website's What is OpenOCD section. ↩︎ Even without this description, if the FPGA is not configured, it will operate by automatically detecting the dbgbase address at startup. The address detected automatically was displayed as Detected core 0 dbgbase: 80110000 when the debug option (-d) was used. On the other hand, if an IP (SLD Node) for JTAG is configured in the FPGA, automatic detection does not work well, and without this option, a startup error occurs. ↩︎ With recent versions of Windows, the make of the preloader may result in an error. One countermeasure is to set /usr/bin at the beginning of the PATH environment variable with export PATH=/usr/bin:$PATH. ↩︎ Since a CPU reset command has not been found, manual resetting of the target is currently used as a substitute. ↩︎ ↩︎ For detailed commands, refer to the OpenOCD User’s Guide. Not all commands are implemented. ↩︎ If anyone knows a better way, please share in the comments m(–)m ↩︎ Combining with VSCode or Eclipse might be possible for self-assembly. Using VSCode+gdb+OpenOCD for ARM development can be a reference. ↩︎
