Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

DZone's Guide to

EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

·
Free Resource

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.

Live

To run the application you need to install the Silverlight 4 runtime. At this time there are only beta developer runtimes for Windows and Mac available.

I recommend the non-embedded version of the application.

 

The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture devices. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight in the context menu and select the new "Webcam / Mic" tab to set them.

Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.

The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works

The base Silverlight 4 webcam usage code was covered in my last blog post.

The new eventhandler for the "Save Snapshot" button:

private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
if (saveFileDlg.ShowDialog().Value)
{
using (Stream dstStream = saveFileDlg.OpenFile())
{
SaveSnapshot(dstStream);
}
}
}

The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The SaveSnapshot method including comments:

// Render Rectangle manually into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(ViewportHost, null);

// Init buffer in FluxJpeg format
int w = bmp.PixelWidth;
int h = bmp.PixelHeight;
byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
pixelsForJpeg[0] = new byte[w, h];
pixelsForJpeg[1] = new byte[w, h];
pixelsForJpeg[2] = new byte[w, h];

// Copy WriteableBitmap data into buffer for FluxJpeg
int[] pixels = bmp.Pixels;
int i = 0;
for (int x = 0; x < w; x++)
{
for (int y = 0; y < h; y++)
{
int color = pixels[i++];
// Swap x and y coordinates to cheaply rotate the image 90° clockwise
pixelsForJpeg[0][y, x] = (byte)(color >> 16); // R
pixelsForJpeg[1][y, x] = (byte)(color >> 8); // G
pixelsForJpeg[2][y, x] = (byte)(color); // B
}
}

using (MemoryStream memStream = new MemoryStream())
{
//Encode Image as JPEG using the FluxJpeg library
ColorModel cm = new ColorModel { colorspace = ColorSpace.RGB };
FluxJpeg.Core.Image jpegImage = new FluxJpeg.Core.Image(cm, pixelsForJpeg);
JpegEncoder encoder = new JpegEncoder(jpegImage, 95, memStream);
encoder.Encode();

// Seek to begin of stream and write the encoded bytes to the FileSteram
memStream.Seek(0, SeekOrigin.Begin);
// Use the new .Net 4 CopyTo method :)
memStream.CopyTo(dstStream);
}

The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it and used the new .Net 4 Stream.CopyTo method.

Why not CaptureSource.AsyncCaptureImage?

You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.

  1. The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.
  2. It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.
  3. It seems that it doesn't always work or there's a bug in the beta version or in the FJCore library. Whatever it might be, I wasn't able to use it in combination with the FJCore encoder. I've tested it on two different machines and the output image was always a 0 byte file. 

Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here

The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.

To make this idea usable for a Silverlight video chat only the rendering, the JPEG encoder and the transfer method  need to be fast enough for real time streaming.

Please leave a comment what you think about it.

Source code

Download the Visual Studio 2010 solution here.

 

Topics:

Published at DZone with permission of Rene Schulte. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}