Image Processing Project on the DE2 board

It looks like your projects needs a lot of software work.

The only hardware seems to be the JPEG compression.

If you don't have strong real time requirements this also can be done in software.

Probably the NIOS allows to better implement all this controller stuff.

Hope this helps

--- Quote Start ---

1. Capture an image

2. Display the Image on the LCD

3. Store the Image in an SD card

4. Convert the image into a JPEG format.

5. Have a user interface to control these functions.

6. The user interface should be implemented on top of a controller

7. The controller should load the JPEG algorithm bit stream into the FPGA to configure it at runtime from a memory (like the SD card).

--- Quote End ---

1. I don't have a DE2 board with me, so I don't know exactly how to do this. But, you can take a look at the CMOS imager specs and write your HDL based on it. It's basically just handshaking between the FPGA and this CMOS imager, and getting all the data correctly.

2/3. Again, take a look at the LCD's specs. For the SD card, you need to implement the SDIO interface in the FPGA. Think there are a few free implementations on this out there...

4. Haven't done JPEG, sorry...

5. Not sure if the LCD panel is also a touchscreen? If so, you can write HDL that renders the UI graphics to the touchscreen. Just superimpose the UI on top of the image that's currently on the screen. Or, you can play around with alpha-blending.

6. What is this controller for? I am guessing that this controller receives user inputs and processes those inputs. If so, for a simple touchscreen controller, just detect the coordinates of a user "touch". Be sure to add at least a few adjacent pixels to be certain it is the user who "touched" the screen, rather than some bug or dust.

7. I am thinking that you should put the JPEG compression algorithm in the HDL, which means it becomes active as soon as your machine becomes active. But if you really need to load this algorithm everytime you do a JPEG compression, then you can write your own scan chain (see below).

--- Quote Start ---

In reconfiguring the Cyclone II, does the whole chip need to be altered? or can only a portion of it alloted for JPEG compression be changed? I ask this because I plan to implement the controller in the FPGA chip instead of the NIOS II embedded controller.

--- Quote End ---

No, you don't need to reconfigure the whole chip. You can try the "partial reconfiguration" feature released by Altera a while ago, though I've never tried it yet. Or, what I'd do is write my own scan chain and have it hooked to an interface to receive data. The interface can be USB/Ethernet/Wi-Fi or whatever. For example, say you've made changes to your JPEG compression algorithm and you want your FPGA to reload the new algorithm, then you can scan in the new algorithm into your FPGA using the said interface + scan chain. The scan chain could be just some shift registers, or controls to internal/external memory. What the scan chain does is just receive data from the interface, and stores the data accordingly (usually in shift registers, but can be other internal/external memory as well). This would update your old algorithm with the new data given by the user. You could also implement your own protocols for storage/memory control or error detection in between the interface controller and the scan chain (if you wish).

--- Quote Start ---

What is this controller for? I am guessing that this controller receives user inputs and processes those inputs.

--- Quote End ---

The controller would be for:

1. Yes, handling commands such as "capture image", "store to sd card", "display on lcd", compress to <compression type>". For the compression type I would implement JPEG but the project leaves room for future developments of other compression types.

2. It would also serve as the configuring device during run time. The idea is when the "compress <>" option is chosen, the corresponding synthesized HDL code of the compression would be loaded to the portion of the FPGA to be reconfigured. The reconfiguring bit stream (compression HDL code) would come from a memory (SD Card for example) and loaded by the controller (NIOS or FPGA implementation) to the FPGA compression portion.

--- Quote Start ---

7. I am thinking that you should put the JPEG compression algorithm in the HDL, which means it becomes active as soon as your machine becomes active.

--- Quote End ---

I will implement the compression in Verilog but I need the capability to load it at run time so that future compression algorithms could be added during run time (as users need them).

--- Quote Start ---

But if you really need to load this algorithm everytime you do a JPEG compression, then you can write your own scan chain (see below).

Or, what I'd do is write my own scan chain and have it hooked to an interface to receive data.

you can scan in the new algorithm into your FPGA using the said interface + scan chain. The scan chain could be just some shift registers, or controls to internal/external memory.

--- Quote End ---

From what I understand about the scan chain, it looks like the concept of using a controller to load the compression algorithm bit stream to reconfigure the fpga. Am I correct?

--- Quote Start ---

The controller would be for:

1. Yes, handling commands such as "capture image", "store to sd card", "display on lcd", compress to <compression type>". For the compression type I would implement JPEG but the project leaves room for future developments of other compression types.

2. It would also serve as the configuring device during run time. The idea is when the "compress <>" option is chosen, the corresponding synthesized HDL code of the compression would be loaded to the portion of the FPGA to be reconfigured. The reconfiguring bit stream (compression HDL code) would come from a memory (SD Card for example) and loaded by the controller (NIOS or FPGA implementation) to the FPGA compression portion.

--- Quote End ---

okay, my guess was something like that too... :) You can write a controller in HDL to do those things.

--- Quote Start ---

I will implement the compression in Verilog but I need the capability to load it at run time so that future compression algorithms could be added during run time (as users need them).

--- Quote End ---

you very well need the shift register / scan chain logic then...

--- Quote Start ---

From what I understand about the scan chain, it looks like the concept of using a controller to load the compression algorithm bit stream to reconfigure the fpga. Am I correct?

--- Quote End ---

Yes, you're right. It probably would be just some shift registers, or some kind of matrix memory controlled by some logic. You might even like to write your HDL such that it uses the internal RAM (such as M4K, etc.) of the device. Well, you can write your own flavour using your own taste... :)

--- Quote Start ---

okay, my guess was something like that too... :) You can write a controller in HDL to do those things.

--- Quote End ---

So I can implement the controller in a portion of the FPGA chip OR I can use the NIOS right? If I go for the first one then I would have to retain the circuitry of the controller while reconfiguring the other portion for the image processing. If I go for the NIOS then I would have to implement an interface to the FGPA. Which do you suggest?:)

--- Quote Start ---

you very well need the shift register / scan chain logic then...

Yes, you're right. It probably would be just some shift registers, or some kind of matrix memory controlled by some logic. You might even like to write your HDL such that it uses the internal RAM (such as M4K, etc.) of the device. Well, you can write your own flavour using your own taste... :)

--- Quote End ---

Either choice of controller implementation I would have to use scan chain logic right? What would the memory be used for? I get the concept of scanning in bit stream to configure but I don't get why use a memory/shift register/RAM. Can't I use the controller to read the configuration bits from an external memory like an SD card and just load it to the FPGA? Is the memory you're referring to a memory that is holding the reconfiguration bits of the compression algorithm?

Oh yeah! regarding the reconfiguration of the FGPA, my HDL code would be compiled into a bit stream that would configure the chip right? I am in the assumption that I would always need the Quartus software to configure the chip. If that's the case then how can the chip be changed without the software?:)

--- Quote Start ---

So I can implement the controller in a portion of the FPGA chip OR I can use the NIOS right? If I go for the first one then I would have to retain the circuitry of the controller while reconfiguring the other portion for the image processing. If I go for the NIOS then I would have to implement an interface to the FGPA. Which do you suggest?:)

--- Quote End ---

Yes, you can implement the controller in a portion of the FPGA. Actually, even if you use NIOS, it will also be in a portion of the FPGA (Nios will also get compiled to use the FPGA's logic resources). Either way, you have to implement an interface to the FPGA.

Think of it very simply, it's just your controller + interface. Controller is a block in FPGA (yes, you need to retain this circuitry while reconfiguring the other portion), and interface logic is another block in the same FPGA. Your controller implementation can be of your own choice, you can write your own HDL, or use Nios. Either way, it's still your controller. You have to hook this controller (Nios or HDL) up to an interface, maybe RS232/USB/etc. Again, it's Controller + Interface. :) Of course, in Nios, you can use any of the interfaces available, and build your Controller + Interface totally in Nios. It's really your own preference. I personally would like everything in VHDL.

--- Quote Start ---

Either choice of controller implementation I would have to use scan chain logic right? What would the memory be used for? I get the concept of scanning in bit stream to configure but I don't get why use a memory/shift register/RAM. Can't I use the controller to read the configuration bits from an external memory like an SD card and just load it to the FPGA? Is the memory you're referring to a memory that is holding the reconfiguration bits of the compression algorithm?

--- Quote End ---

Yes, you are right about the concept of scanning in the bitstream. And yes, the bitstream can be drawn from external memory like an SD card. The contents of your bitstream can be your new JPEG compression algorithm, or whatever. Or, you could write a generic JPEG compressor, and only change some parameters during runtime. So, in this case, your bitstream would be just the new parameters to scan into your algorithm. Whatever it is, it deals with some kind of memory. You need to scan in the bitstream (e.g. from your SD card), and store it somewhere (RAM). Now, it's up to you to define which memory to use (could be shift registers, M4K, external RAM, etc.). It's really up to you. Well, at least you got the concept right already :)

--- Quote Start ---

Oh yeah! regarding the reconfiguration of the FGPA, my HDL code would be compiled into a bit stream that would configure the chip right? I am in the assumption that I would always need the Quartus software to configure the chip. If that's the case then how can the chip be changed without the software?:)

--- Quote End ---

No, you don't *always* need Quartus to configure the chip, but it's good to start with that first. Later on, you can always use a configuration device (such as EPCS), or configuration controller (like a MAX-II CPLD) to do the auto-configuration during power up of your board. Here, you don't need to hook your board up to Quartus.

Regarding your question on how to "change the chip" configuration without Quartus, actually all you need to do is to scan in your new bitstream to your FPGA board, and you don't need Quartus (or even JTAG) to do the scanning for you. If you've implemented the RS232 interface (and of course, hooked it up to your bitstream controller) for example, then you can even connect your computer to your FPGA board via your RS232 port, and use some tool (or write your own) to do the bitstream scanning. Of course, you might as well use other interfaces such as Ethernet/USB/etc.

--- Quote Start ---

Actually, even if you use NIOS, it will also be in a portion of the FPGA (Nios will also get compiled to use the FPGA's logic resources). Either way, you have to implement an interface to the FPGA.

--- Quote End ---

Okay, now I think I have the wrong idea about the NIOS embedded processor. I thought it was a separate chip that can be used as a general purpose microcontroller. I double checked and there's no chip like that in the board. What then is NIOS? :))

And what do you mean by "interface to the FPGA"?

--- Quote Start ---

Think of it very simply, it's just your controller + interface. Controller is a block in FPGA (yes, you need to retain this circuitry while reconfiguring the other portion), and interface logic is another block in the same FPGA.

--- Quote End ---

Is the interface you mention here the same with the interface in the next quote (RS232/USB/etc.)? If not, what is the interface logic here for?

--- Quote Start ---

Your controller implementation can be of your own choice, you can write your own HDL, or use Nios. Either way, it's still your controller. You have to hook this controller (Nios or HDL) up to an interface, maybe RS232/USB/etc. Again, it's Controller + Interface. :) Of course, in Nios, you can use any of the interfaces available, and build your Controller + Interface totally in Nios. It's really your own preference. I personally would like everything in VHDL.

--- Quote End ---

If HDL code is to be compiled and loaded in the FPGA, how then is the implementation done in NIOS?

--- Quote Start ---

You need to scan in the bitstream (e.g. from your SD card), and store it somewhere (RAM). Now, it's up to you to define which memory to use (could be shift registers, M4K, external RAM, etc.).

--- Quote End ---

When the bit stream is scanned why does it have to be stored somewhere? Can't it be loaded to the FPGA immediately? What is the other memory for?

--- Quote Start ---

Later on, you can always use a configuration device (such as EPCS), or configuration controller (like a MAX-II CPLD) to do the auto-configuration during power up of your board. Here, you don't need to hook your board up to Quartus.

--- Quote End ---

Doesn't the controller I'm planning to implement (with HDL or NIOS) handles the reconfiguring? Here, the "configuration controller" pertains to upon power up, not during run time?

Can this configuration controller be used as the reconfiguration controller during run time to load in the new scanned bit stream instead of the one discussed above?

Please bear with me:) Thank you very much.

--- Quote Start ---

And what do you mean by "interface to the FPGA"?

Is the interface you mention here the same with the interface in the next quote (RS232/USB/etc.)? If not, what is the interface logic here for?

--- Quote End ---

Yes, I meant the same interface as in the next quote.

--- Quote Start ---

If HDL code is to be compiled and loaded in the FPGA, how then is the implementation done in NIOS?

--- Quote End ---

The "implementation" in the FPGA should as much as possible be similar (HDL and Nios), it is just the source files that are different. For HDL, you would use VHDL or Verilog, or other HDL formats, while for Nios, you would use C for the source code. The resulting RTL from both should be similar (though not always so - I still prefer HDL to Nios when improving area/efficiency).

--- Quote Start ---

When the bit stream is scanned why does it have to be stored somewhere? Can't it be loaded to the FPGA immediately? What is the other memory for?

--- Quote End ---

You can use the same external SD memory to do your RAW-to-JPEG manipulation, JPEG compression, etc., and use something like a temporary file for the manipulation of bits. But I personally would want to manipulate bits in a RAM (could be on-chip RAM like M4K/M8K, or external RAM) because it performs faster. So it depends on how fast you want your application to work.

--- Quote Start ---

Doesn't the controller I'm planning to implement (with HDL or NIOS) handles the reconfiguring? Here, the "configuration controller" pertains to upon power up, not during run time?

--- Quote End ---

No, this is a different controller. Your controller does not reconfigure the whole FPGA the way Quartus configures the FPGA. The built-in JTAG controller in the FPGA does the configuring. Now, you want to implement your own controller that configures a part of your whole design, so you need to write a separate HDL for it (your HDL, again, is not the same as the built-in JTAG configuration controller in the FPGA).

--- Quote Start ---

Can this configuration controller be used as the reconfiguration controller during run time to load in the new scanned bit stream instead of the one discussed above?

--- Quote End ---

Yes. But only to load the new bitstream for your JPEG compression/or whatever other function. You can't use this to configure the whole FPGA (i.e. you can't load a SOF/POF with this).

--- Quote Start ---

Please bear with me:) Thank you very much.

--- Quote End ---

No problem.

--- Quote Start ---

You can use the same external SD memory to do your RAW-to-JPEG manipulation, JPEG compression, etc., and use something like a temporary file for the manipulation of bits. But I personally would want to manipulate bits in a RAM (could be on-chip RAM like M4K/M8K, or external RAM) because it performs faster. So it depends on how fast you want your application to work.

--- Quote End ---

I think the memory here is for the image data itself, the data from the camera module? So, I have to implement 2 memories. (Correct me if I'm wrong):

1. One to hold the image data from the camera - this is the ram you were referring to?

2. One to hold the Algorithm bit stream to be loaded to the FPGA for it to be reconfigured to do the processing - I'll use an SD card for this.

So, I'm thinking the process will go like this: (Assuming that the image data from the camera is stored in a memory (RAM))

a. Choose compression option

b. The bit stream from the SD Card is loaded to the FGPA

c. The FPGA is reconfigured to do the compression

d. The RAW data from the RAM is fed through the FPGA (I don't know if this is what really happens)

e. The processed data is stored in the RAM as a compressed image data.

If I'm correct, what part of this process does the scan chain happen?

I got confused with the memory part because you pointed out that from the SD card, the bit stream must be loaded to an internal memory. I'm assuming now that the bit stream you were referring to here is the image data from the camera not the JPEG algorithm (HDL) bit stream for configuring the FPGA.

I know that a controller must be implemented to handle control and interface with the FPGA, memory, camera, etc. Also, to load the configuration bits from the SD to reconfigure the FPGA. However, from the quote below, the controller I mention here does not do this function (reconfiguring). So what do I do to reconfigure? Write THE controller to access the JTAG?

--- Quote Start ---

No, this is a different controller. Your controller does not reconfigure the whole FPGA the way Quartus configures the FPGA. The built-in JTAG controller in the FPGA does the configuring. Now, you want to implement your own controller that configures a part of your whole design, so you need to write a separate HDL for it (your HDL, again, is not the same as the built-in JTAG configuration controller in the FPGA).

--- Quote End ---

--- Quote Start ---

I think the memory here is for the image data itself, the data from the camera module? So, I have to implement 2 memories. (Correct me if I'm wrong):

1. One to hold the image data from the camera - this is the ram you were referring to?

2. One to hold the Algorithm bit stream to be loaded to the FPGA for it to be reconfigured to do the processing - I'll use an SD card for this.

--- Quote End ---

The memory can be for image data, or also your compression algorithm. I'm not sure how you are going to load the algorithm into the FPGA as a bitstream - is it going to be the whole algorithm, or just parameters to the algorithm? I would actually write the algorithm in HDL (means this is part of your SOF), and only load the parameters as a bitstream. However, even with this I would load the bitstream into a RAM (best if internal RAM), so the algorithm doesn't need to access the external SD card all the time at every loop. Imagine, if your algorithm is running, and you remove the SD card, your whole system is going to fail. So, I'd always load the bitstream to (internal or external) RAM.

--- Quote Start ---

So, I'm thinking the process will go like this: (Assuming that the image data from the camera is stored in a memory (RAM))

a. Choose compression option

b. The bit stream from the SD Card is loaded to the FGPA

c. The FPGA is reconfigured to do the compression

d. The RAW data from the RAM is fed through the FPGA (I don't know if this is what really happens)

e. The processed data is stored in the RAM as a compressed image data.

If I'm correct, what part of this process does the scan chain happen?

--- Quote End ---

You're almost there. This process will start after you have already configured your FPGA using Quartus. The scan chain bitstream scanning will happen in (b), after your FPGA is already configured by Quartus. (c) is also related to (b), where after scanning in the bitstream, your FPGA automatically already gets configured with the new compression algorithm. For (d), usually RAW comes from the image sensor, but u will want to store it once in RAM before you do further processing (such as JPEG compression). The processing should work on the contents of the RAM, and not modify contents in the SD card (if for example, u want to load an image from SD instead of sensor and do image processing on it). For your JPEG compression, it is also better to have the compression parameters stored in RAM, so that your algorithm can use it directly from RAM instead of rely on SD card (again, SD card may be removed anytime during runtime).

--- Quote Start ---

So what do I do to reconfigure? Write THE controller to access the JTAG?

--- Quote End ---

Reconfigure the FPGA using the normal way, Quartus programmer via USB JTAG. Your design should contain another controller to scan in the bitstream via another interface (maybe another USB port, or others).