Easy Eye: SiLabs 8051 + Camera

by Craig Sullender on May 24, 2011

Image capture and 60fps color object targeting with an 8-bit microprocessor and a CMOS camera.

This is a simple demo to illustrate a larger idea: computer vision with limited resources enables emerging applications for consumer products.

The goal is to prove the ability of the F360 to perform machine vision tasks. The C8051F360 syncs to video from a digital camera, reads pixel values, makes decisions about objects in the scene, communicates by serial port, and drives displays or servos by GPIO.

Microcontroller: SiliconLabs C8051F360
Camera: Omnivision OV7720

The serial port connection to the host PC is not shown, it wasn’t connected in this photo. The VGA connector in the image is for a related project.

The Silicon Labs C8051F360 runs at 100Mips, which makes it a perfect mate for a camera running at 60fps, 640×480 resolution. The data rate for demosaiced RGB is approx. 48MHz. The F360 takes two cycles per port read, or 50MHz max. The 24MHz clk from the camera is input to the F360, and the F360 internal PLL converts the 24MHz clk to a 96MHz system clock.

Download project files here.

 

_____________

 

System:

Camera -> Microcontroller -> PC

The F360 communicates captured pixels and target data to the host PC via a virtual com port supplied by Silicon Labs.

All signals pass through the 8051.

Code in C and Assembly.

Due to loop overhead, any pixel can be read from the camera, but not every pixel in sequence.

It takes a minute to transfer an image over the serial port, but the video motion targets tracking the red toy car are transferred at full speed for the full frame rate, 60 times a second.

 

_____________

 

Circuit:

The Omnvision OV7720 eval. board is wired directly to the F360 port pins.

Connections between camera and microcontroller:

F360 input P1.7-P1.0 – Camera Data [7:0]
F360 input P0.4 – Camera Hsync
F360 input P0.5 – Camera Vsync
F360 input P0.6 – Camera Clk
F360 Gnd – Camera Gnd

 

_____________

 

Sampling the camera image with the F360:

 

The F360 cache had to be flushed before horizontal and vertical sync routines or the results were erratic.

 

F360 synced to the camera, every pixel in its place.

 

_____________

 

Filtering for a color window:

 

Image from pixels captured by the F360.

 

Pixel values filtered so that only red pixels are captured.

 

_____________

 

Computer Vision:

 

Camera view. The F360 will perform a very simple grouping of red pixels to fit a target box on the remote controlled red car.

 

The 8051 filters the RGB input and marks the red object at 60fps. Only the target coordinates are sent to the host PC. The coordinates are updated at 60fps, so the motion of the target box tracking the car is full speed and is seen on the host PC. The image above is misleading as the moving targets are superimposed on the static red pixels.

It takes a minute to transfer an image over the serial port, but the video motion targets tracking the red toy car are transferred at full speed for the full frame rate, 60 times a second. The first part of the video below shows the full-speed targets from the F360 and the last half of the video shows an image being transferred slowly to the host PC.

Advantages of directly connecting a CMOS digital camera to a microcontroller:

  • Two components
  • Easy to program
  • Quick to prototype
  • Low power

Disadvantages of directly connecting a CMOS digital camera to a microcontroller:

  • Processor is fully occupied while sampling pixels for the object
  • Cannot sample every pixel in sequence
  • Very limited number of colors/shapes/objects can be detected (e.g. only one color detected in this project)

If a microcontroller like the F360 had a pixel-processing peripheral, you would get the best of both worlds – easy, low-cost, low-power vision with high resolution, high frame rate, multiple object targeting.

Download project files here.

Share:
---
---

{ 18 comments }

Paul Jurczak May 25, 2011 at 7:40 pm

Hi Craig,

Have a look at STMicro STM32 F2 series microcontrollers. They have camera interface with DMA data transfer (STM32F207/217), so MCU can do image processing while acquiring pixel data. MCU itself is much more capable than 8-bit 8051 and has up to 128KB embedded SRAM. Lower end models cost below $10 at quantity, so they should qualify as low-cost.

Paul

Craig Sullender May 25, 2011 at 8:05 pm

Thanks! I had not heard of that part. I will check it out.

I am designing for a $2 (BOM) vision system including processor. Some of the methods I use on other projects include no frame buffers and no random-order memory accesses, so no DMA needed. Check out the ChipSight slides.

Paul Jurczak May 25, 2011 at 8:26 pm

$2 BOM is very tough. You have to compromise on everything. $20 BOM may be almost equally applicable, but with much less compromise. Note that most smart cams are at $500+ range. Even simple color segmentation solutions like CMUCam3 cost almost $200. There is practically nothing in sub $100 range.

I have large amount of data on this subject and some on and off interest in making my own low cost smart cam.

Craig Sullender May 25, 2011 at 8:46 pm

Excellent comments. We should talk so I can learn about your research. Or you could post about it on this site. Every week I do a post called “Semi-Conscious:…” about chips and other developments for the consumer vision market. Or I just complain about the semiconductor companies and machine vision architectures.

Now there will be something available in the sub-$10 range. I am careful about trade-offs. ChipSight is high performance for low-resource systems.

Embedded vision developers no longer need to compromise performance for cost.

Paul Jurczak May 26, 2011 at 2:09 am

My “hidden” agenda is to nudge you in direction of a little bit more capable solution, away from the cheapest one. Here are the arguments:

1. Simple color segmentation (color tracking) solutions were available since early 1990s and their real world applications were very limited (robotics contests and toys). CMUCam clone can be build for less than $50 or it can be purchased for $100+. I think that limited functionality, not the price was a reason these cameras weren’t a huge success.

2. Making a smart cam (image sensor with external MCU) for $2 is not realistic. Even, if you think in 100K pcs. category, it is not very likely. For 1K pcs. range, achieving $10 unit cost is going to difficult.

3. Most importantly, the fundamental problem with computer vision is not lack of inexpensive hardware, it is lack of “smart” software. Most vision software is ad-hock, special purpose and very fragile. Therefore having decent amount of computational power on board is crucial to enable “smart” software.

I hope this didn’t decrease you enthusiasm. Feel free to contact me directly or we can continue this discussion here. Maybe someone will find it helpful.

Craig Sullender May 26, 2011 at 4:35 am

This is good, others might be thinking the same way, as you said.

I’ve probably confused the discussion by talking about ChipSight under this post about the SiLabs project.

For 1. in your comment above, I am getting requests for help with products where cost and functionality are the decisive factors–typically applications I never thought of. My ideas for applications are limited, but the applications others think up are limitless. That’s why I stick to really simple demos of basic functionality, and otherwise focus on making a platform for vision applications.

For 2., you get that I am doing something unlikely and unrealistic. To have embedded vision take off, that will be worth it. $10 (BOM), already worked out for today’s parts in low volume. $2? Definitely possible.

For 3., exactly, I’m with you. Some vision applications need very smart software. Why spend 90% of the processor and system resources chewing through pixels? Separating pixel processing from the application allows developers to size the processor to the application instead of to the pixel data rate. Providing a standardizable front-end helps developers build products in a more methodical way across different types of applications.

Back to what you were saying about making a more capable solution. What are the most important capabilities to include? And what parts or products are in this general area? You mentioned the STMicro STM32 and the CMUCam so far.

I appreciate your comments here. Few people have taken a very close look at small, general purpose vision systems. The difficult nature of this area, coupled with the existing ad-hoc solutions, have created an opportunity.

Paul Jurczak May 26, 2011 at 7:49 pm

I didn’t notice that ChipSight and SiLabs are two different projects. That puts my doubts about your low BOM costs in different light.

Here is my take on cost/functionality classes of smart cams:

1. CMUCam/ChipSight class: no full frame storage; MPU barely able to keep up with incoming pixel stream (usually software loop to read data from image sensor); only simple, dramatically reducing image data algorithms possible (e.g. color segmentation with fixed number of classes); slow interface to host system (RS232).

2. Moore’s Law evolution from class 1: multiple frames can be stored (external memory interface available); dedicated hardware handles image data stream (image sensor interface with DMA as in STMicro STM32 F-2 or whole dedicated I/O MPU as in NXP LPC4300); MPU fast enough to go beyond simplest image processing (100MHz+ 32-bit); fast interface to host system allowing to stream full frames in real time (USB2.0 high speed, Ethernet).

3. Almost a real deal: poor cousin of a full PC or a PC from the past; 128MB+ RAM, 400MHz+ 32-bit CPU; dedicated hardware camera interface (ISI or USB); runs Liunx and large image processing packages like OpenCV; Boardcon MINI100 would be a good example here (it is priced below $40 in qty).

4. Real deal: basically a full blown PC or equivalent with USB camera(s); capable system based on mini-ITX motherboard can be put together for about $150. You would pay more for higher speed or reduced power or size.

Paul Jurczak May 27, 2011 at 6:59 am

Of course, the above list would not reach its full entertainment potential without mentioning the newest vaporware – Raspberry Pi. At $25 a pop, it will make a quite attractive smart cam.

Tom May 26, 2011 at 9:29 am

Hello Craig!

Great work, very impressive!
I’ll be looking forward to your advances with this project!

Would you mind sharing where you got the OV7720 eval board from?
I’ve been looking for such a board but haven’t been able to find and purchase one yet.

Thanks a lot & keep up the work!
Tom

Craig Sullender May 26, 2011 at 1:53 pm

Thanks!

Here is where I found the eval board:
http://www.ovt.com/products/sensor.php?id=79
but now I don’t see links to it. I had to go through sales anyway.

The difficulty with getting good new cameras (with data sheets and eval boards that convert to your prototype) is one of the things driving me to build a camera module. Subscribe to this blog via RSS or Twitter and I’ll keep you posted on developments.

I liked the OV7720 because its video signal timing is right for driving a VGA monitor. 60fps VGA.

Paul Jurczak May 26, 2011 at 6:30 pm

Tom,

OmniVision eval boards are available from Arrow for about $200 each. There are much less expensive alternatives available. Go to eBay.com and search for the following items: 280421730740, 280476233787, 150589389820, 200592386128 and many more.

Paul

Craig Sullender May 27, 2011 at 6:25 pm

I started a “parts” page to collect information on consumer vision products.

Steve Battazzo May 27, 2011 at 7:41 am

Cool. I am kind of curious about transferring the whole image to PC when the processor can’t grab all the pixels in sequence.

Taking a quick look at your source code this is the impression I get:
It looks like you’re really grabbing one pixel for each command that says “get me a pixel”, right? So after the hsync, you set a counter that counts for a certain amount of time, and since the CPU is running at a multiple of the camera clock, you have a window of a couple clock cycles that the data you want is available on the input port.

Then, each pixel is actually sampled on a separate frame from the camera?
Am I interpreting this properly?

Interesting solution as far as getting the most you can out of very limited resources, but what happens when the image is not still for the whole minute it takes to grab the image?

I suppose in this application, the point is not to grab frames, but to analyze a small region of interest on board your MCU anyway.

Craig Sullender May 27, 2011 at 12:37 pm

Yes to everything. In the last part of the video you can see the F360 sending pixels to the PC in vertical lines. The serial com is added to the delay, not just the processor loop. The timing for serial transfer is right for grabbing a pixel, sending it to the PC, and waiting for the pixel directly under it. This approach gives you one vertical line per frame. So it takes over a minute to send the image, but motion targets are full speed.

tmk June 11, 2011 at 7:38 am

Craig, where did you get the OV7720 on a nice breakout board like that?

Can’t seem to find one..

Also it turns out the USB cam i have uses the same chipset. funny

-tmk

Craig Sullender June 11, 2011 at 3:09 pm

Here’s a copy of comments to a similar question:

Where I found the eval board:
http://www.ovt.com/products/sensor.php?id=79
but now I don’t see links to it. I had to go through sales anyway.

The difficulty with getting good new cameras (with data sheets and eval boards that convert to your prototype) is one of the things driving me to build a camera module. Subscribe to this blog via RSS or Twitter and I’ll keep you posted on developments.

I liked the OV7720 because its video signal timing is right for driving a VGA monitor. 60fps VGA.

Commenter Paul said:
“OmniVision eval boards are available from Arrow for about $200 each. There are much less expensive alternatives available. Go to eBay.com and search for the following items: 280421730740, 280476233787, 150589389820, 200592386128 and many more.”

I followed his eBay items and this camera from Hong Kong Electronics looks like the best way to go:
http://chipsight.com/parts-from-different-sources/

Ivin October 14, 2011 at 5:23 am

I am wondering if we could use these http://www.silabs.com/applications/industrial/Pages/EnergyHarvesting.aspx to be the micro controller to the camera which eliminates the battery.

Craig Sullender October 15, 2011 at 7:53 pm

Hi Ivin. Yes you could use some type of energy harvesting, but the cameras are low-power, not micro-power. You could turn the camera off (standby) most of the time, store up energy from the energy harvesting source, then take a picture.

Comments on this entry are closed.

{ 5 trackbacks }

Previous post:

Next post: