Example designs for communicating two Altera boards

--- Quote Start ---

I am looking for example designs or tutorials showing how to communicate two Altera boards using simple protocols. The purpose is to see how the two devices interact by transmitting data back and forth and to check the transmission somehow. I am using two Altera Transceiver Signal Integrity Development Kit, Stratix IV GT (EP4S100G2F40I1) boards.

Please help to advice if anyone knows such tutorials and examples. Also, any suggestions are highly appreciated.

--- Quote End ---

What are you looking to test? The transceivers? At what data rate? What have you tried so far, eg., do you know how to configure the ALTGX and ALGX_RECONFIG components?

What do you have in addition to the GT boards? Do you have break out cables? A synthesizer to provide a common clock between the boards?

Cheers,

Dave

Many thanks to Dave and please find my answers as following:

1-What are you looking to test? The transceivers?

Yes, the main purpose is to to test transceivers, in particular with the Altera PHY IP cores. But besides these, I want to know how to create a design for two boards to communicate one another using some simpler protocol than the PHY IP cores. The data rate can be any not greater than 11.3Gbps

2-What have you tried so far?

I have tried to test XCVR with On-chip debugging design examples provided by Altera. Then I created similar designs with Lowlatency and Custom PHY for various data rates using Qsys. However, I would like to have a 10Gbase-r PHY design and I could not create this in Qsys. I don't know if it is possible to create such design by using available components in Qsys.

3-Do you know how to configure the ALTGX and ALGX_RECONFIG components?

I have general idea about the ALTGX and ALTGX_RECONFIG but have not ever used these as I want to try designing with Qsys first.

4-What do you have in addition to the GT boards? Do you have break out cables? A synthesizer to provide a common clock between the boards?

I have SMA cables for physical connection between the two boards. For a synthesizer to provide the common clock, if you can make this clearer?

Thanks.

--- Quote Start ---

1-What are you looking to test? The transceivers?

Yes, the main purpose is to to test transceivers, in particular with the Altera PHY IP cores. But besides these, I want to know how to create a design for two boards to communicate one another using some simpler protocol than the PHY IP cores. The data rate can be any not greater than 11.3Gbps

--- Quote End ---

Unfortunately, there are lots of protocols, and lots of reasons for selecting a specific one. What is your final application? Are you going to be having FPGAs communicating with FPGAs, or do you need to interface with a 10G switch, or a Serial RapidIO switch, or SATA drives, or ... etc etc?

--- Quote Start ---

2-What have you tried so far?

I have tried to test XCVR with On-chip debugging design examples provided by Altera. Then I created similar designs with Lowlatency and Custom PHY for various data rates using Qsys. However, I would like to have a 10Gbase-r PHY design and I could not create this in Qsys. I don't know if it is possible to create such design by using available components in Qsys.

--- Quote End ---

What have you managed to test? Have you only tried these in hardware, or have you simulated them in Modelsim?

--- Quote Start ---

3-Do you know how to configure the ALTGX and ALGX_RECONFIG components?

I have general idea about the ALTGX and ALTGX_RECONFIG but have not ever used these as I want to try designing with Qsys first.

--- Quote End ---

You're doing things in reverse. Qsys is a convenience for connecting components. For a successful system design, you need to understand how these components work. For that understanding, you should start by configuring an ALTGX, an ALTGX_RECONFIG, create a reset controller, and then get them working in simulation and hardware. Use SignalTap II to trace that the hardware does what the simulation does (it doesn't, but the handbook tells you the difference - the key is being able to see and understand the differences). Since all other components are based on these two components, this will give you a fundamental understanding of how the interfaces work.

--- Quote Start ---

4-What do you have in addition to the GT boards? Do you have break out cables? A synthesizer to provide a common clock between the boards?

I have SMA cables for physical connection between the two boards. For a synthesizer to provide the common clock, if you can make this clearer?

--- Quote End ---

If one board uses a 156.25MHz reference that is actually a few kHz high, and the other board uses a reference that is a few kHz low, then one board will send data to the other faster than ideal. The clock-and-data recovery (CDR) in the receiver will track the higher frequency, however, if you use a FIFO to cross clock domains between the recovered clock and the local clock, you will eventually get a FIFO overrun (or underrun in the case of a slower transmitter). Protocols like 10G take care of this by having protocol codes that can be added or deleted. If you are implementing FPGA-to-FPGA communications, then the system becomes simpler if you use a synchronous reference. You can implement this reference using an external synthesizer and send a copy of the signal to each board, or you can use one of your boards as the clock source and the other can receive a clock from the first, and use it for clocking its receiver logic.

To understand what I am talking about, configure an ALTGX transmitter with one reference frequency (156.25MHz + something), and an ALTGX receiver with another frequency (156.25MHz - something), and simulate it using Modelsim.

If you don't know how to use Modelsim, its time to learn :)

Ask questions, and I'll try to help.

Cheers,

Dave

Thanks Dave very much for your time and very detailed explanation.

Let me describe briefly my final application. I am going to build a 10Gbps Ethernet Passive Optical Network (EPON) system where one FPGA will play a role as a center office device (OLT) and several other FPGAs will be customer premises (ONU). The boards will interface SFP+ modules. The design will consist of a physical layer protocol (probably by using Altera 10GBASE-R IP core) and my own implementation of MAC protocol (probably by modifying Altera 10GbE MAC IP core). The MAC for OLT device is different from that for ONUs.

Therefore, I am trying to have physical layer first. I studied the Altera XCVR PHY IP Core User Guide and XCVR Architecture in Stratix IV manual. Then I tested the provided On-chip XCVR debugging examples in harware (not yet in Modelsim). However, 10GBASE-R is provided by Altera without a similar example for testing in hardware, so I tried to create that in Qsys with Avalon ultilities. With some knowledge from reading specs and Qsys, I though/hoped that I would create a complete 10GBASE-R design, then simulate it in Modelsim as well as test it on harware.

I hoped that this top-down approach would help me understand the overview of my design more quickly. Moreover, as the PHY layer is provided as IP core, I do not really want to spend much time for this. I just want to know how to test this in hardware and know how to interface this with the other components in my final design. The major part of my project is to design the customized MAC sublayer. That is why I am doing things in reverse.

I don't know if I am right when choosing this approach. Please give me some advice or/and suggest me better way(s) to reach my target application. For sure, to have a better understanding, I will learn to designs by configuring the ALTGX, RECONFIG,...like you recommended and learn to simulate in Modelsim. Also, I will learn to work with the SignalTap II and synthesizer soon.

Again, thank you very much and look forward to your further advice.

--- Quote Start ---

I am going to build a 10Gbps Ethernet Passive Optical Network (EPON) system where one FPGA will play a role as a center office device (OLT) and several other FPGAs will be customer premises (ONU). The boards will interface SFP+ modules. The design will consist of a physical layer protocol (probably by using Altera 10GBASE-R IP core) and my own implementation of MAC protocol (probably by modifying Altera 10GbE MAC IP core). The MAC for OLT device is different from that for ONUs.

--- Quote End ---

Ok, thanks for the description.

--- Quote Start ---

Therefore, I am trying to have physical layer first. I studied the Altera XCVR PHY IP Core User Guide and XCVR Architecture in Stratix IV manual. Then I tested the provided On-chip XCVR debugging examples in harware (not yet in Modelsim). However, 10GBASE-R is provided by Altera without a similar example for testing in hardware, so I tried to create that in Qsys with Avalon ultilities. With some knowledge from reading specs and Qsys, I though/hoped that I would create a complete 10GBASE-R design, then simulate it in Modelsim as well as test it on harware.

--- Quote End ---

My recommendation would be to take the XCVR debug example, which works, and simulate it. That'll allow you to probe a working design.

I assume this is the 10GBASE-R IP you are looking at:

http://www.altera.com/products/ip/iup/ethernet/m-alt-10gbase-r-pcs.html

Note that 64/66B encoding is not supported in the Stratix IV hard-IP PCS, so it gets implemented in the fabric. You should synthesize to see how much logic the PCS layer is using. Given that the Stratix IV GT might not be a good use of resources, have you considered using a GX device with an external PHY?

--- Quote Start ---

I hoped that this top-down approach would help me understand the overview of my design more quickly.

--- Quote End ---

It might, if you also know how to simulate Qsys and create custom components. However, if you're introducing lots of new concepts, then you might actually be making the design more complicated.

--- Quote Start ---

Moreover, as the PHY layer is provided as IP core, I do not really want to spend much time for this. I just want to know how to test this in hardware and know how to interface this with the other components in my final design. The major part of my project is to design the customized MAC sublayer. That is why I am doing things in reverse.

--- Quote End ---

What you really should be doing then is getting the PHY working in the simulator so that you can simulate while working on your custom logic.

Cheers,

Dave

Dear Dave,

Once again, thank you very much for your very useful advice.

--- Quote Start ---

My recommendation would be to take the XCVR debug example, which works, and simulate it. That'll allow you to probe a working design.

--- Quote End ---

Yes, this is my plan. However, there is not available example for 10GBASE-R, and I am trying to create generator/checker component with XGMII data width instead of using the existing Avalon ST gen/checker. So, I need to write the component in verilog and then add them to Qsys library. Once I have those, I can create the complete design that includes the 10GBASE-R, the generator/checker. The JTAG to Avalon Master Bridge has a master interface to control the three components so that I can run design on devices and use XCVR Toolkit to test the signal quality. If you have any suggestion/comment about this intention?

--- Quote Start ---

I assume this is the 10GBASE-R IP you are looking at:

http://www.altera.com/products/ip/iup/ethernet/m-alt-10gbase-r-pcs.html

--- Quote End ---

Yes, exactly the document I have studied. I also refer to a hardware demonstration in http://www.alterawiki.com/wiki/10g_ethernet_and_10g_base_r_phy_interoperability_hardware_demonstration_design

Actually, this works with different board and I changed the pin assignment for my device. But I got some errors during recompilation. I will fix this next days. Do you suggest other materials or design examples?

--- Quote Start ---

You should synthesize to see how much logic the PCS layer is using. Given that the Stratix IV GT might not be a good use of resources, have you considered using a GX device with an external PHY?

--- Quote End ---

Frankly, I have not considered this. Can you put some more details as it looks very important. Regarding to the resources of my boards and devices, I also wonder that in case I need memory to store data for board-to-board communication application, which kind of memory I can use? As far as I understand, both the on-chip memory (TriMatrix memory includes 640-bit MLABs, 9-Kbit M9K, and 144-Kbit M144K blocks) and on-board memory (64-MB synchronous flash) are quite limited. Is it possible for my board to use external memory and if yes, how to do so as I do not see this infos in the reference manual. Can you please help to confirm my understanding and give me some explanation also?

Unfortunately, I cannot change my devices (I am not the one who decides). I need to do everything with these boards.

And how can I check the amount of resource used for PCS only? As I know the compilation report provides the summary for whole design rather than each sublayer.

--- Quote Start ---

What you really should be doing then is getting the PHY working in the simulator so that you can simulate while working on your custom logic.

--- Quote End ---

I will try to do this next days. I see the importance of this step. Thank you very much for this timely advice.

Best,

PLMT

--- Quote Start ---

Yes, this is my plan. However, there is not available example for 10GBASE-R, and I am trying to create generator/checker component with XGMII data width instead of using the existing Avalon ST gen/checker. So, I need to write the component in verilog and then add them to Qsys library.

--- Quote End ---

That sounds like a good plan.

--- Quote Start ---

Once I have those, I can create the complete design that includes the 10GBASE-R, the generator/checker. The JTAG to Avalon Master Bridge has a master interface to control the three components so that I can run design on devices and use XCVR Toolkit to test the signal quality. If you have any suggestion/comment about this intention?

--- Quote End ---

Here's a tutorial that shows you how to use the JTAG-to-Avalon-MM master component and Avalon-MM BFM components:

http://www.alterawiki.com/wiki/using_the_usb-blaster_as_an_sopc/qsys_avalon-mm_master_tutorial

--- Quote Start ---

>> Use of an external PHY

Frankly, I have not considered this. Can you put some more details as it looks very important.

--- Quote End ---

If your design fits in a Cyclone or Arria series device that has transceivers that operate to 3.125Gbps with a XAUI interface that uses 8/10B encoding, then you can interface to an external PHY (I think Vitesse has some) that will decode-and-reencode the data as 10.3125Gbps SFP+ (with 64/66B encoding). A lower-cost FPGA plus external PHY may be more cost effective than using a high-end FPGA.

--- Quote Start ---

Regarding to the resources of my boards and devices, I also wonder that in case I need memory to store data for board-to-board communication application, which kind of memory I can use? As far as I understand, both the on-chip memory (TriMatrix memory includes 640-bit MLABs, 9-Kbit M9K, and 144-Kbit M144K blocks) and on-board memory (64-MB synchronous flash) are quite limited. Is it possible for my board to use external memory and if yes, how to do so as I do not see this infos in the reference manual. Can you please help to confirm my understanding and give me some explanation also?

--- Quote End ---

Only you can answer this question. It depends on what you have to store. If you are only testing, then you can use a PRBS generator to create a lot of random data.

--- Quote Start ---

Unfortunately, I cannot change my devices (I am not the one who decides). I need to do everything with these boards.

--- Quote End ---

You can develop with the boards you have, however, someone should see whether this is appropriate for the final application. It will not change the logic you develop, since that MAC logic is probably identical.

--- Quote Start ---

And how can I check the amount of resource used for PCS only? As I know the compilation report provides the summary for whole design rather than each sublayer.

--- Quote End ---

The hierarchy window can be used to show how much each block in the design uses. You just need to realize its there - use the horizontal scroll bar and you'll see it displayed after place-and-route.

Cheers,

Dave

Dear Dave,

Please find my comments below.

--- Quote Start ---

Here's a tutorial that shows you how to use the JTAG-to-Avalon-MM master component and Avalon-MM BFM components:

http://www.alterawiki.com/wiki/using_the_usb-blaster_as_an_sopc/qsys_avalon-mm_master_tutorial

--- Quote End ---

Thank you very much. I find it very useful and will try to study soon. I've just looked through the design and it has one On-chip SRAM. I am not sure if this works in Qsys. Because these days, I learnt to create designs for On-chip memory (both On-chip FIFO and RAM memory cores), but always the generation process hangs with the message "Starting classic module elaboration" in the step of "creating HDL design files for senthesis". I am using Quartus 11.0 and 11.1 and the error occurs whenever the design has On-chip memory core. If you can help to check this or give me a clue to fix the error. Sorry for putting this here:)

--- Quote Start ---

If your design fits in a Cyclone or Arria series device that has transceivers that operate to 3.125Gbps with a XAUI interface that uses 8/10B encoding, then you can interface to an external PHY (I think Vitesse has some) that will decode-and-reencode the data as 10.3125Gbps SFP+ (with 64/66B encoding). A lower-cost FPGA plus external PHY may be more cost effective than using a high-end FPGA.

--- Quote End ---

Maybe I will understand this later. For now, I am quite confused. By external PHY, do you mean something like the "Dual XAUI to SFP+ HSMC" board for connecting SFP+ modules in case of Stratix IV GX? As fas as I know, people has tested 10Gbps Ethernet with Stratix IV GT (but in 100G SIV GT dev board) in http://www.alterawiki.com/wiki/10g_ethernet_and_10g_base_r_phy_interoperability_hardware_demonstration_design. Acording to the block diagram, they connect directly to SFP+ Optical module without an external board or PHY. If you can help to correct my understanding and clarify?

--- Quote Start ---

It depends on what you have to store. If you are only testing, then you can use a PRBS generator to create a lot of random data.

--- Quote End ---

Testing with PRBS generator is what I want now. But, later on, I need to read data in terms of multimedia file like a movie file from host PC, store in some on-chip memory or buffer, then transmit in packets to other boards. So, I am wondering how should I do that and what is the most suitable solution. I wait for your advice also:)

Thanks indeed for your time and explanation. I learnt a lot through this conversation.

PLMT.

--- Quote Start ---

always the generation process hangs with the message "Starting classic module elaboration" in the step of "creating HDL design files for senthesis". I am using Quartus 11.0 and 11.1 and the error occurs whenever the design has On-chip memory core. If you can help to check this or give me a clue to fix the error. Sorry for putting this here:)

--- Quote End ---

Try 11.1sp1. The tutorial was tested under Windows XP and 7, Linux Centos 6.2 and Ubuntu 11.10. There were no problems with that version of Quartus.

Cheers,

Dave

--- Quote Start ---

By external PHY, do you mean something like the "Dual XAUI to SFP+ HSMC" board for connecting SFP+ modules in case of Stratix IV GX?

--- Quote End ---

The external PHY is the chip on that HSMC board. If you look at the price of a GX FPGA plus the external PHY, you'll find that it is less than the price of a GT FPGA. However, the board layout will then need more chips, so you need to figure out what is the cost effective solution for your final design.

--- Quote Start ---

As fas as I know, people has tested 10Gbps Ethernet with Stratix IV GT (but in 100G SIV GT dev board) in http://www.alterawiki.com/wiki/10g_ethernet_and_10g_base_r_phy_interoperability_hardware_demonstration_design. Acording to the block diagram, they connect directly to SFP+ Optical module without an external board or PHY. If you can help to correct my understanding and clarify?

--- Quote End ---

The GT boards have 10Gbps transceivers, so they do not need the external PHY.

--- Quote Start ---

Testing with PRBS generator is what I want now. But, later on, I need to read data in terms of multimedia file like a movie file from host PC, store in some on-chip memory or buffer, then transmit in packets to other boards. So, I am wondering how should I do that and what is the most suitable solution. I wait for your advice also:)

--- Quote End ---

Transferring data over PCIe would be suitable. However, since you purchased the GT signal integrity kit, you do not have that option.

http://www.altera.com/products/devkits/altera/kit-stratix-iv-gt-si.html

The kit does have a 1000Mbit PHY, so that is about all you can use.

The other option would be to build a board that interfaces to the FCI Airmax VS connectors, but you'd need to check the hardware design to see if the PCIe hard-IP was available on those transceivers. If it was, then you could build a x1 PCIe breakout cable, and then connect to the board using one of the adapters from OneStopSystems:

http://www.onestopsystems.com/components.php

Cheers,

Dave