copying nokia’s photobeamer with signalr and wp8

Some time ago Nokia launched a pretty impressive, almost magical app called “Photobeamer” (exclusive for the Nokia Lumia range). This is one of those apps that look very impressive and magical at first but when you start to think about it it’s not really that hard to figure out how this works. I thought about it and decided to hack together a copy of this.

If you have no clue what Photobeamer is all about, check out this video from Pocketnow

Disclaimer

Small remark about this video before we get started. The Nokia Photobeamer caches every image for about 30 days, meaning that if you select the same picture again it appears almost instantly, that’s what happens in this video. We will get nowhere near that performance since my concept here doesn’t do any caching. Also, they are streaming the image to the server (hence the fact that the picture is blurry at first and sharpens up later) we will just send over the entire image at once.

Puzzle pieces

All right, let’s get started. We need a few pieces to complete this puzzle. We need a Windows Phone app, we’ll need a service that takes care of sending the right picture to the right client and we’ll need a web client to display the QR code and the picture.

This schematic shows the pieces and their roles

The steps are:

  • webclient generates unique ID (guid)
  • webclient registers a group on the server with that ID
  • webclients uses ID to generate QR code
  • user launches Windows Phone app
  • user selects picture
  • user scans QR code to get the generated ID
  • phone app joins the group with the ID
  • phone app deserializes the picture
  • phone app sends the picture to the other group members
  • webclient receives picture, serializes it again and shows it

Lots of steps but it’s easier to build as you might think. We’ll start with the service as that’s the glue to hold everything together.

SignalR

Since we need some form of real-time communication I’ve decided to use SignalR. A snippet from the SignalR page that describes its functionality (http://signalr.net/):

ASP.NET SignalR is a new library for ASP.NET developers that makes it incredibly simple to add real-time web functionality to your applications. What is "real-time web" functionality? It's the ability to have your server-side code push content to the connected clients as it happens, in real-time.

You may have heard of WebSockets, a new HTML5 API that enables bi-directional communication between the browser and server. SignalR will use WebSockets under the covers when it's available, and gracefully fallback to other techniques and technologies when it isn't, while your application code stays the same.

All we need to get started is an empty web project and a nuget package use the package manager or the GUI

Code Snippet
  1. Install-Package Microsoft.AspNet.SignalR -Pre

SignalR, at the time of writing, is in Release Candidate, so we need to include prerelease versions in Nuget or you won’t find signalR at all. (although it’s prerelease, SignalR is working really well so no worries)

Before we start building the service we need to add some method calls in Global.asax.cs in the Application_Start method.

Code Snippet
  1. void Application_Start(object sender, EventArgs e)
  2. {
  3.     // Code that runs on application startup
  4.     AuthConfig.RegisterOpenAuth();
  5.  
  6.     GlobalHost.HubPipeline.EnableAutoRejoiningGroups();
  7.  
  8.     // Register the default hubs route: ~/signalr/hubs
  9.     RouteTable.Routes.MapHubs();
  10. }

The MapHubs() makes sure that the service calls are rerouted to the correct functions. The EnableAutoRejoiningGroups() is something that we need to do in this RC2 version but it will go away in RTM, see my StackOverflow question for more information.

SignalR uses the concept of Hubs, a hub is a class that contains all the service calls. Add a class called ImgHub to a folder called Hubs in the empty web project.

Code Snippet
  1. publicclassImgHub : Hub
  2. {
  3.     publicvoid Create(string guid)
  4.     {
  5.         Groups.Add(Context.ConnectionId, guid);
  6.     }
  7.     
  8.     publicvoid ShareImage(byte[] image, string guid)
  9.     {
  10.         Clients.OthersInGroup(guid).ReceiveImage(image);
  11.     }
  12.  
  13.     publicvoid Leave(string guid)
  14.     {
  15.         Groups.Remove(Context.ConnectionId, guid);
  16.     }
  17. }

That’s really not a lot of code for a service. First, every hub in a SignalR project needs to inherit from Microsoft.AspNet.SignalR.Hub. Next to that, every public method we declare in this class will be callable from the clients. First method is Create(), this takes in the unique ID that will be generated by the webclient in a minute. That ID is used to create a group. SignalR tries to add the connectionId from the client to a group with the guid as groupname. If that group doesn’t exist yet it will create it. The second method is ShareImage. This will take in the deserialized image (as a byte array) and that same guid again. Now we need to send that byte array to other clients so we call Clients.OthersInGroup, other options are Clients.All, Clients.Groups, Clients.Others, Clients.Caller, Clients.AllExcept. Plenty of choices but OthersInGroup suits our needs the best. The OthersInGroup returns a dynamic object, we can attach anything we want here. We want to call ReceiveImage() on the client, that method isn’t declared anywhere but since the object is dynamic the compiler won’t give us any trouble. And finally there’s a Leave() method allowing us to leave a group should it be needed.

The SignalR project can be hosted on any webhost that supports .net, for this demo I’ve used Windows Azure Websites.

Webclient

1/3th done, the second part is the webclient. This is a normal website using asp.net webforms. It’s responsible for generating the ID, passing it to SignalR to create the group, get a QR code and receive the image.

First thing we need is the project, second is a SignalR client and the SignalR Javascript library (yes we’ll be needing Javascript and no I’m not proud of this…)

First some HTML code, the Default.aspx page is nothing fancy, it will only show the QR code.

Code Snippet
  1. <body>
  2.     <formid="form1"method="post"action="ImagePage.aspx">
  3.         <inputtype="hidden"id="hiddenByteArray"name="hiddenByteArray"/>
  4.         <div>
  5.             <imgsrc="/imagehandler.ashx"style="text-align: center"/>
  6.         </div>
  7.     </form>
  8. </body>

Notice the imagehandler.ashx reference? That’s a generic handler that will take care of generating the QR code and passing it into this img tag. Generic handler is a file type that you can add through Visual Studio, here’s the code for imagehandler.ashx.

Code Snippet
  1. publicclassImageHandler : IHttpHandler, IRequiresSessionState
  2. {
  3.     publicvoid ProcessRequest(HttpContext context)
  4.     {
  5.         var guid = HttpContext.Current.Session["InstanceGuid"];
  6.  
  7.         string url = string.Format(@"http://api.qrserver.com/v1/create-qr-code/?size=300x300&data={0}", guid);
  8.  
  9.         WebClient client = newWebClient();
  10.         var bytes = client.DownloadData(newUri(url));
  11.  
  12.         context.Response.OutputStream.Write(bytes, 0, bytes.Length);
  13.         context.Response.ContentType = "image/JPEG";
  14.     }
  15.  
  16.     publicbool IsReusable
  17.     {
  18.         get
  19.         {
  20.             returnfalse;
  21.         }
  22.     }
  23. }

We’re getting our guid from the session object (how it got there, we’ll see in a minute) that’s why we need the IRequiresSessionState interface. The ProcessRequest method we get from IHttpHandler and will get executed when the handler is called. So we first get the guid from the session, then we’ll build a url to qpi.qrserver.com, an api that takes in a value and generates a QR code from that value. We use WebClient to get the data from that url, the byte array that we receive from it will be our generated QR code, we write it to the outputstream of the page’s context and set the type to be an image/JPEG. There should be some error handling here if you’re using this for production code (sometimes the qrserver API takes to long and times out).

Next, we’ll have a look at Default.aspx.cs.

Code Snippet
  1. protectedvoid Page_Load(object sender, EventArgs e)
  2. {
  3.     var guid = Guid.NewGuid();
  4.     HttpContext.Current.Session["InstanceGuid"] = guid.ToString();
  5. }

Not much going on, we generate a new Guid and set it to the current session object.

Now, let’s have a look at connection the webclient to the signalR server and getting ready to receive the image.

Warning: the next part contains javascript code…

In Default.aspx we add this script

Code Snippet
  1. <scriptsrc="Scripts/jquery-1.7.1.min.js"></script>
  2. <scriptsrc="Scripts/jquery.signalR-1.0.0-rc2.min.js"></script>
  3. <scriptsrc="http://pbclone.azurewebsites.net/signalr/hubs/"type="text/javascript"></script>
  4. <scripttype="text/javascript">
  5.     $(function () {
  6.         $.connection.hub.url = 'http://pbclone.azurewebsites.net/signalr';
  7.         //$.connection.hub.url = 'http://localhost:4341/signalr';
  8.         // Proxy created on the fly
  9.         var mainHub = $.connection.imgHub;
  10.         var guid = '<%=HttpContext.Current.Session["InstanceGuid"] %>';
  11.  
  12.         // Declare a function on the hub so the server can invoke it
  13.         mainHub.client.receiveImage = function (imageArray) {
  14.             //window.location = "/ImagePage.aspx?arr=" + imageArray;
  15.             $('#hiddenByteArray').val(imageArray);
  16.             $('#form1').submit();
  17.         };
  18.        
  19.         // Start the connection
  20.         $.connection.hub.start().done(function () {
  21.             mainHub.server.create(guid);
  22.         });
  23.     });
  24. </script>

First, we need to add jquery and the SignalR javascript library to our page. The third included script comes from wherever you host your SignalR service. The /signalr/hubs/ will be a proxy in javascript that contains the methods in the hub, allowing us to use them from our clients (try browsing to that url and have a look inside the javascript). $.connection comes from the SignalR javascript library, we set the correct url and get the correct hub. We’ll also use a bit of inline asp.net to get the guid from the session. Remember in the SignalR part that we called ReceiveImage on the dynamic object? Line 13 is where we declare a callback handler on that method call. We set the received value, which will be a byte array, to a hidden field and POST the form. Those handlers need to be set before we call the start() method on the hub. <yourhub>.client is where all your client side callbacks are registered. <yourhub>.server is where all server side methods can be called, those methods are loaded from the /signalr/hubs/ proxy. On line 20 we start the connection to the hub, once we’re connected we’ll call the create method and pass the guid in to create and join the group.

We’ll need a second page in this webclient to actually show the image. Only one element on the page, an img element.

Code Snippet
  1. <body>
  2.     <formid="form1"runat="server">
  3.     <div>
  4.         <imgsrc="/ByteArrayHandler.ashx"style="text-align: center"/>
  5.     </div>
  6.     </form>
  7. </body>

The img element uses a second generic handler that will take care of deserializing the byte array back into an image.

Code Snippet
  1. publicclassByteArrayHandler : IHttpHandler, IRequiresSessionState
  2. {
  3.  
  4.     publicvoid ProcessRequest(HttpContext context)
  5.     {
  6.         string base64String = HttpContext.Current.Session["ByteArray"].ToString();
  7.         byte[] convertedFromBase64 = Convert.FromBase64String(base64String);
  8.  
  9.         context.Response.OutputStream.Write(convertedFromBase64, 0, convertedFromBase64.Length);
  10.         context.Response.ContentType = "image/JPEG";
  11.     }
  12.  
  13.     publicbool IsReusable
  14.     {
  15.         get
  16.         {
  17.             returnfalse;
  18.         }
  19.     }
  20. }

We’ll get the byte array from the session object. SignalR encodes arrays with Base64 encoding so we need to decode that first, once that’s done we can just write the byte array into the outputstream as a JPEG, just like we did with the QR code. Onto the ImagePage.aspx.cs to see how the byte array goes into the session object

Code Snippet
  1. protectedvoid Page_Load(object sender, EventArgs e)
  2. {
  3.     NameValueCollection postedValues = Request.Form;
  4.  
  5.     var base64String = postedValues["hiddenByteArray"];
  6.     HttpContext.Current.Session["ByteArray"] = base64String;
  7. }

We get our POST values from Request.Form, look for the hidden field called hiddenByteArray and place its value in the session.

We now have our service and one client complete, all that’s left is building a Windows Phone application that connects to that same group on that same service and send the picture over.

Windows Phone client

For this we’ll need a Windows Phone 8 project as SignalR has no official support for Windows Phone 7. Make sure that you have the latest version of Nuget installed or it won’t find the correct SignalR assembly for Windows Phone 8 (thanks David Fowler for pointing this out).

Add the SignalR.Client assembly to the project. Here’s the XAML for the MainPage.

Code Snippet
  1. <StackPanel x:Name="TitlePanel" Grid.Row="0" Margin="12,17,0,28">
  2.     <TextBlock Text="MY APPLICATION" Style="{StaticResource PhoneTextNormalStyle}" Margin="12,0"/>
  3.     <TextBlock x:Name="StatusText" Text="Not connected" Margin="9,-7,0,0" Style="{StaticResource PhoneTextTitle1Style}"/>
  4. </StackPanel>
  5.  
  6. <!--ContentPanel - place additional content here-->
  7. <Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0">
  8.     <Rectangle x:Name="_previewRect"
  9.        Margin="0"
  10.        Height="800"
  11.        Width="600"
  12.        HorizontalAlignment="Center"
  13.        VerticalAlignment="Center">
  14.         <Rectangle.Fill>
  15.             <VideoBrush x:Name="_previewVideo">
  16.                 <VideoBrush.RelativeTransform>
  17.                       <CompositeTransform
  18.                 x:Name="_previewTransform" CenterX=".5" CenterY=".5" />
  19.                 </VideoBrush.RelativeTransform>
  20.             </VideoBrush>
  21.         </Rectangle.Fill>
  22.     </Rectangle>
  23.     <ListBox Margin="10" x:Name="_matchesList" FontSize="30" FontWeight="ExtraBold" />
  24. </Grid>

I use one of the default textboxes in the page’s header for a connection status. In the content grid we place a rectangle filled with a VideoBrush, this will be used to scan the QR code, Let’s have a look at the code for the WP app.

First we declare some fields

Code Snippet
  1. privatestring _guid;
  2. privatePhotoChooserTask _photoChooserTask;
  3. privateIHubProxy _mainHub;
  4. privateHubConnection _connection;
  5. privatereadonlyDispatcherTimer _timer;
  6. privatePhotoCameraLuminanceSource _luminance;
  7. privateQRCodeReader _reader;
  8. privatePhotoCamera _photoCamera;
  9. privateStream _imgStream;

I’ll explain these when we encounter them. Now for the page’s constructor

 

Code Snippet
  1. public MainPage()
  2. {
  3.     InitializeComponent();
  4.  
  5.     _photoCamera = newPhotoCamera();
  6.     _photoCamera.Initialized += OnPhotoCameraInitialized;
  7.     _previewVideo.SetSource(_photoCamera);
  8.     CameraButtons.ShutterKeyHalfPressed += (sender, args) => _photoCamera.Focus();
  9.  
  10.     _photoChooserTask = newPhotoChooserTask();
  11.     _photoChooserTask.Completed += photoChooserTask_Completed;
  12.     _photoChooserTask.Show();
  13.  
  14.     _timer = newDispatcherTimer
  15.                  {
  16.                      Interval = TimeSpan.FromMilliseconds(250)
  17.                  };
  18.  
  19.     _timer.Tick += (o, arg) => ScanPreviewBuffer();
  20. }

First we initialize a PhotoCamera instance. _previewVideo is the VideoBrush set in the rectangle, we set its source to the PhotoCamera instance. On line 8 we state that when the hardware camera button is half pressed we focus the camera, just a small helper for when the app has troubles reading the QR code. The next part is calling the PhotoChooserTask, this task gives us access to the albums and pictures on the device. We’ll need a timer as well, every timer tick we’ll check if the camera preview window contains a QR code. We’ve declared some event handlers in this constructor, let’s go over them one by one. We’ll start with the OnPhotoCameraInitialized.

Code Snippet
  1. privatevoid OnPhotoCameraInitialized(object sender, CameraOperationCompletedEventArgs e)
  2. {
  3.     int width = Convert.ToInt32(_photoCamera.PreviewResolution.Width);
  4.     int height = Convert.ToInt32(_photoCamera.PreviewResolution.Height);
  5.     _photoCamera.FlashMode = FlashMode.Off;
  6.  
  7.     _luminance = newPhotoCameraLuminanceSource(width, height);
  8.     _reader = newQRCodeReader();
  9.  
  10.     Dispatcher.BeginInvoke(() =>
  11.     {
  12.         _previewTransform.Rotation = _photoCamera.Orientation;
  13.     });
  14. }

We get the resolution’s width and height and turn the flash off. The PhotoCameraLuminanceSource on line 7 is a custom class that will provide us with a previewbuffer that we can fill. Here’s the class

Code Snippet
  1. publicclassPhotoCameraLuminanceSource : LuminanceSource
  2. {
  3.     publicbyte[] PreviewBufferY { get; privateset; }
  4.  
  5.  
  6.     public PhotoCameraLuminanceSource(int width, int height)
  7.         : base(width, height)
  8.     {
  9.         PreviewBufferY = newbyte[width * height];
  10.     }
  11.  
  12.  
  13.     publicoverridesbyte[] Matrix
  14.     {
  15.         get { return (sbyte[])(Array)PreviewBufferY; }
  16.     }
  17.  
  18.  
  19.     publicoverridesbyte[] getRow(int y, sbyte[] row)
  20.     {
  21.         if (row == null || row.Length < Width)
  22.         {
  23.             row = newsbyte[Width];
  24.         }
  25.  
  26.  
  27.         for (int i = 0; i < Height; i++)
  28.             row[i] = (sbyte)PreviewBufferY[i * Width + y];
  29.  
  30.  
  31.         return row;
  32.     }
  33. }

The class inherits from LuminanceSource which comes from the Google zxing project. Zxing is a library to decode QR and barcode images, it has a .NET port on codeplex (http://zxingnet.codeplex.com/) that port is what I use in this project and that’s where the LuminanceSource comes from. That’s also where the QRCodeReader class lives.

Next event handler that we attached in the constructor is the photoChooserTask_Completed

Code Snippet
  1. void photoChooserTask_Completed(object sender, PhotoResult e)
  2. {
  3.     if (e.TaskResult == TaskResult.OK)
  4.     {
  5.         _imgStream = e.ChosenPhoto;
  6.  
  7.         _timer.Start();
  8.     }
  9. }

If the task returns succesfully we’ll set the chosen photo, which arrives here as a stream, to the _imgStream field and we start the timer. Now on every timer tick (every 250 milliseconds in this example) we will scan the previewbuffer for QR codes.

Code Snippet
  1. privatevoid ScanPreviewBuffer()
  2. {
  3.     if (_guid != null)
  4.     {
  5.         _timer.Stop();
  6.  
  7.         SendImage();
  8.     }
  9.  
  10.     try
  11.     {
  12.         _photoCamera.GetPreviewBufferY(_luminance.PreviewBufferY);
  13.         var binarizer = newHybridBinarizer(_luminance);
  14.         var binBitmap = newBinaryBitmap(binarizer);
  15.         var result = _reader.decode(binBitmap);
  16.  
  17.         Dispatcher.BeginInvoke(() =>
  18.                                    {
  19.                                        _guid = result.Text;
  20.                                    });
  21.     }
  22.     catch
  23.     {
  24.  
  25.     }
  26. }

First thing we do here is checking if we already have a guid, if we do we stop the timer and send the image to the SignalR service. If _guid is still null we’ll get the previewbuffer and try to decode it, if there’s no QR code in the previewbuffer it will throw an exception, hence the empty catch block. When we can decode  we’ll go to the SendImage() method.

Code Snippet
  1. privateasyncvoid SendImage()
  2. {
  3.     if (_connection == null || _connection.State != ConnectionState.Connected)
  4.     {
  5.         await SetupSignalRConnection();
  6.     }
  7.  
  8.     if (_connection.State == ConnectionState.Connected || _connection.State == ConnectionState.Reconnecting)
  9.     {
  10.         MemoryStream s = newMemoryStream();
  11.         _imgStream.CopyTo(s);
  12.         _mainHub.Invoke("ShareImage", newobject[]{s.ToArray(), _guid});
  13.     }
  14.     else
  15.     {
  16.         MessageBox.Show("not connected");
  17.     }
  18. }

If there’s no active connection, we’ll call the SetupSignalRConnection() method. if there is and we are connected we copy the imagestream into a MemoryStream and we invoke the ShareImage() method on the SignalR server, passing in the memorystream, converted into a byte array, and the guid we got from the qr code.

Now for the connection to the server.

Code Snippet
  1. privateasyncTask SetupSignalRConnection()
  2. {
  3.     _connection = newHubConnection("http://pbclone.azurewebsites.net/");
  4.     _connection.StateChanged += ConnectionOnStateChanged;
  5.     _mainHub = _connection.CreateHubProxy("imghub");
  6.  
  7.     await _connection.Start();
  8.  
  9.     _mainHub.Invoke("Create", _guid);
  10. }

We instantiate a new HubConnection with the url to the service as parameter. We generate a proxy for the hub we want to use and we call the start() method on the connection. It will connect to the hub, get the javascript proxy and translate it into a .net proxy. We then invoke the Create() method and pass in the guid so that our Windows Phone client joins the same group as the web client. The ConnectionOnStateChanged event handler is only used to update the textblock on the page to show whether or not we’re connected.

Code Snippet
  1. privatevoid ConnectionOnStateChanged(StateChange stateChange)
  2. {
  3.     switch (stateChange.NewState)
  4.     {
  5.         caseConnectionState.Connecting:
  6.             StatusText.Text = "Connecting...";
  7.             break;
  8.         caseConnectionState.Connected:
  9.             Dispatcher.BeginInvoke(() => StatusText.Text = "Connected");
  10.             break;
  11.         caseConnectionState.Reconnecting:
  12.             Dispatcher.BeginInvoke(() => StatusText.Text = "Reconnecting...");
  13.             break;
  14.         caseConnectionState.Disconnected:
  15.             Dispatcher.BeginInvoke(() => StatusText.Text = "Disconnected");
  16.             break;
  17.     }
  18. }

And that’s it, we can now start sending images to any device with a browser and an internet connection.

Conclusion

In this post I’ve tried to demystify the magic from Nokia’s Photobeamer app. It is a really cool app but when you take it apart, it’s not that magical. Like a lot of impressive looking tech it just combines a bunch of existing techs into something no one else thought of.

The code in this article is all my own, I have no clue how the Nokia app actually works and what technology they are using, I’m only mimicking their behavior.

Update: I've uploaded the project to Github https://github.com/NicoVermeir/photobeamerclone

This is an imported post. It was imported from my old blog using an automated tool and may contain formatting errors and/or broken images.

Leave a Comment