Nico's digital footprint

I grew up in the nineties, that makes me awesome by default

WP8, Leap Motion and a glue called Sockets

by Nico

I’m one of the lucky few who got their hands on a Leap Motion developer device (check out the video above if you’ve never heard of the Leap Motion). It’s a pretty cool device that gives you motion tracking not unlike Kinect but limited to finger and hand movement instead of complete body tracking.

I had the device and needed something to play with and what’s cooler then combining a cool gadget with an awesome smartphone? So I decided to build a small proof-of-concept that would capture finger movement in a WPF application and translate that movement to a moving ellipse in a Windows Phone application.

The Leap Motion’s documentation got me to a moving ellipse in a WPF application pretty fast thanks to the samples and documentation found on their developer portal. It took me a bit more time to get the position of the ellipse send to the phone, I wanted to use Sockets for this (SignalR would be way easier but I didn’t want an extra service running, now I have peer to peer communication).

Here’s how I did it

WPF and the Leap Motion

First, the WPF project. This project will be the socket server and the app that captures the Leap Motion’s output. The application only has one page with this as XAML

Code Snippet
  1. <Window x:Class="LeapWpPoc.MainWindow"
  2.         xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  3.         xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  4.         Title="MainWindow"
  5.         Width="525"
  6.         Height="350">
  7.     <Canvas x:Name="TheCanvas">
  8.         <Ellipse x:Name="TheEllipse"
  9.                  Canvas.Left="211"
  10.                  Canvas.Top="118"
  11.                  Width="70"
  12.                  Height="70"
  13.                  Fill="#FFFF0C00"
  14.                  Stroke="Black" />
  15.         <TextBlock x:Name="TextBlockStatus"
  16.                    Canvas.Left="10"
  17.                    Canvas.Top="10"
  18.                    Foreground="Red"
  19.                    Text="Not connected"
  20.                    TextWrapping="Wrap" />
  21.     </Canvas>
  22. </Window>

Let’s have a look at how to interact with the Leap Motion first.

First thing you need when working with the Leap Motion (apart from the actual device that is) is a reference to LeapCSharp.NET4.0.dll however, you’ll also need Leap.dll and LeapCSharp.dll. Now I find this a bit dirty and I sincerely hope that the Leap Motion team will find a way to fix this but those two libraries aren’t referencable in our project but they need to be present in the application’s directory or it won’t work. What I did is add them as an existing item to the project, set their build action to Content and Copy if newer, at least this way they will always get copied into the build directory of the application.

Next, you’ll needs a class that inherits from Listener (Listener is part of the Leap Motion SDK). Listener is a base class that provides a bunch of virtual methods like OnExit, OnDisconnect, OnConnect and OnFrame. Feel free to override those to add some logging or logic but I’m only using the OnFrame method here. Here’s my Listener class

Code Snippet
  1. class PocListener : Listener
  2. {
  3.     public event EventHandler<FrameDetectedEventArgs> FrameDetected;
  4.  
  5.     public override void OnFrame (Controller controller)
  6.     {
  7.         // Get the most recent frame and report some basic information
  8.         Frame frame = controller.Frame ();
  9.  
  10.         if (FrameDetected != null)
  11.         {
  12.             FrameDetected(this, new FrameDetectedEventArgs(frame));
  13.         }
  14.     }
  15. }

The OnFrame method fires constantly, passing all the movement information, if any, in a frame. If my listener implementation detects a frame the FrameDetected event will fire, passing in the detected frame as an eventarg, FrameDetectedEventArgs is a very basic class that only passes the frame data to whoever is listening to the event.

Code Snippet
  1. public class FrameDetectedEventArgs : EventArgs
  2. {
  3.     public Frame Frame { get; set; }
  4.  
  5.     public FrameDetectedEventArgs(Frame frame)
  6.     {
  7.         Frame = frame;
  8.     }
  9. }

Notice that the OnFrame method needs a Controller as parameter? Controller is the class from the Leap Motion API that talks to the device. I create a new instance of Controller and my Listener in the MainWindow constructor.

Code Snippet
  1. private Frame _previousFrame;
  2. private Frame _currenFrame;
  3. private readonly PocListener _listener;
  4. private readonly Controller _controller;
  5. private readonly SocketFactory _socketFactory;
  6.  
  7. public MainWindow()
  8. {
  9.     InitializeComponent();
  10.  
  11.     _listener = new PocListener();
  12.     _controller = new Controller();
  13.  
  14.     // Have the sample listener receive events from the controller
  15.     _controller.AddListener(_listener);
  16.  
  17.     _listener.FrameDetected += ListenerOnFrameDetected;
  18. }

This is the constructor and some private fields, I need to keep track of both the current frame and the previous one to detect if there’s any change in the position of the hand or fingers. I also have an instance of PocListener (my own Listener class) and of Controller, both get instantiated in the constructor.Next I need to register my Listener in the Controller, that’s done on line 14 and then finally I attach an event handler to the FrameDetected event.

The event handler of the FrameDetected event will be responsible of checking if hands and fingers are detected.

Code Snippet
  1. private void ListenerOnFrameDetected(object sender, FrameDetectedEventArgs frameDetectedEventArgs)
  2. {
  3.     _currenFrame = frameDetectedEventArgs.Frame;
  4.  
  5.     if (!_currenFrame.Hands.Empty)
  6.     {
  7.         // Get the first hand
  8.         Hand hand = _currenFrame.Hands[0];
  9.  
  10.         // Check if the hand has any fingers
  11.         FingerList fingers = hand.Fingers;
  12.         if (!fingers.Empty)
  13.         {
  14.             if (_previousFrame == null) _previousFrame = _currenFrame;
  15.             //check if the current frame is different from the last frame
  16.             if (_currenFrame != _previousFrame)
  17.             {
  18.                 //we only need one finger so we'll take the first one that's detected
  19.                 Finger finger = fingers[0];
  20.  
  21.                 float x = _previousFrame.Fingers[0].TipPosition.x - finger.TipPosition.x;
  22.                 float y = _previousFrame.Fingers[0].TipPosition.y - finger.TipPosition.y;
  23.  
  24.                 //update the sphere's position
  25.                 Dispatcher.BeginInvoke((Action)(() =>
  26.                                                 {
  27.                                                     Canvas.SetTop(TheEllipse, Canvas.GetTop(TheEllipse) + y);
  28.                                                     Canvas.SetLeft(TheEllipse, Canvas.GetLeft(TheEllipse) - x);
  29.                                                 }));
  30.  
  31.                 //set this frame as the previous one and get ready to receive a new frame
  32.                 _previousFrame = _currenFrame;
  33.             }
  34.         }
  35.     }
  36. }

First we check if the previous frame has the same information as the current frame, if it does there’s no need to update the ellipse’s position.

If a hand and fingers are detected I select the first detected finger, because that’s currently the only one I’m interested in. Since all the Leap Motion actions are happening on a separate thread I need to invoke the UI thread to update the ellipse’s position, that’s what’s happening on line 25. The position of a UIElement in a Canvas in XAML is set through attached properties (Canvas.Left=”177” for example) to set these in code we use Canvas.SetLeft. To determine the new location of the ellipse I take the X and Y positions of the first finger in the previous frame and substract the X and Y of the first finger in the current frame. The new Y value gets added to the Canvas.Top of the ellipse and the new X value gets subtracted  from the Canvas.Left value. Last but not least I set the previousframe to the currentframe.

That’s all the code you need to get an ellipse moving in a canvas with the Leap Motion. So part 1 is a great success. Now onto the bigger challenge, getting a similar ellipse to move in a Windows Phone project.

Windows Phone and sockets

Implementing sockets to make devices talk to each other over the wire can be challenging but it’s also very rewarding and just plain fun once it works. I haven’t worked with sockets before so I had a real blast trying to figure this out and I went through the roof once that ellipse started moving on the Windows Phone emulator Glimlach. Before we dive into the sockets, let’s have a quick look at the MainPage.

Code Snippet
  1. <Canvas x:Name="TheCanvas">
  2.     <Ellipse x:Name="TheEllipse"
  3.              Canvas.Left="302"
  4.              Canvas.Top="186"
  5.              Width="70"
  6.              Height="70"
  7.              Fill="#FFFF0C00"
  8.              Stroke="Black" />
  9.     <TextBlock x:Name="TextBlockStatus"
  10.                Canvas.Left="10"
  11.                Canvas.Top="10"
  12.                Foreground="Red"
  13.                Text="Not connected"
  14.                TextWrapping="Wrap" />
  15. </Canvas>

Exactly the same controls as in the WPF project, nothing spectacular here. Let’s have a look at the code behind the MainPage.

Code Snippet
  1. public partial class MainPage
  2. {
  3.     private SocketClient _client;
  4.     const int Port = 8000;
  5.     private const string IpAddress = "10.16.79.70";
  6.  
  7.     // Constructor
  8.     public MainPage()
  9.     {
  10.         InitializeComponent();
  11.         _client = new SocketClient();
  12.  
  13.         _client.OnConnected += (sender, args) => Dispatcher.BeginInvoke(() =>
  14.             {
  15.                 TextBlockStatus.Text = "Connected";
  16.                 TextBlockStatus.Foreground = new SolidColorBrush(Colors.Green);
  17.             });
  18.         _client.OnMessageReceived += (sender, args) => Dispatcher.BeginInvoke(() => MoveBall(args.Response));
  19.         _client.Connect(IpAddress, Port);
  20.  
  21.         _client.Receive();
  22.     }
  23.  
  24.     private void MoveBall(string response)
  25.     {
  26.         response = response.Replace(',', '.');
  27.         var coordinates = response.Split(';');
  28.  
  29.         float y = float.Parse(coordinates[0]);
  30.         float x = float.Parse(coordinates[1]);
  31.  
  32.         Canvas.SetTop(TheEllipse, y);
  33.         Canvas.SetLeft(TheEllipse, x);
  34.     }
  35. }

First some fields, don’t worry about the SocketClient class, we’ll get to that in a minute. Some constants holding the IP address and the port of the server (don’t forget to change this IP address to the address of your own pc!)

In the constructor we instantiate the SocketClient instance and handle its two events. The OnConnected event is going to change the text of the TextBlock to “Connected” and the OnMessageReceived event handler will move the ellipse.

The MoveBall() method will do the actual moving. The message that we will receive will be a string that has the Y and X of the ellipse in the WPF application seperated by a semicolon. The problem I had here was that instead of a dot to separate the decimals .NET had changed it into a comma (which is the default decimal sign in Belgium) so I need to change that back. I then split up the string using the semicolon as split character, parse the values into a float and set the ellipse to its new position. Let’s get serious and dive into the socket stuff now.

Sockets provide us with TCP and UDP communication, next to a whole bunch of other features. This app will use a TCP connection between WPF and a Windows Phone application. Since we’re currently looking at the Windows Phone app, let’s continue there. I’ve added a class called SocketClient that takes care of connection a socket, sending and receiving messages. Let’s start with the fields and constructor.

Code Snippet
  1. // Cached Socket object that will be used by each call for the lifetime of this class
  2. Socket _socket;
  3.  
  4. // Signaling object used to notify when an asynchronous operation is completed
  5. static ManualResetEvent _clientDone;
  6.  
  7. // Define a timeout in milliseconds for each asynchronous call. If a response is not received within this
  8. // timeout period, the call is aborted.
  9. const int TimeoutMilliseconds = 5000;
  10.  
  11. // The maximum size of the data buffer to use with the asynchronous socket methods
  12. const int MaxBufferSize = 2048;
  13.  
  14. public event EventHandler<MessageReceivedEventArgs> OnMessageReceived;
  15. public event EventHandler OnConnected;
  16.  
  17. public SocketClient()
  18. {
  19.     _clientDone = new ManualResetEvent(false);
  20. }

First field is the actual socket that we’ll be using, second one is a reset event that we can use to make a thread block while waiting for an event to fire. The timeout is the max amount of time that the reset event will block a thread. The buffersize is the size of the buffer for socket messages. The two events are fired when the app receives a socket message or when a socket connects.

The most important function of the SocketClient class that I’m building here is to connect two sockets.

Code Snippet
  1. public string Connect(string hostName, int portNumber)
  2. {
  3.     string result = string.Empty;
  4.  
  5.     // Create DnsEndPoint. The hostName and port are passed in to this method.
  6.     DnsEndPoint hostEntry = new DnsEndPoint(hostName, portNumber);
  7.  
  8.     // Create a stream-based, TCP socket using the InterNetwork Address Family.
  9.     _socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
  10.  
  11.     // Create a SocketAsyncEventArgs object to be used in the connection request
  12.     SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs {RemoteEndPoint = hostEntry};
  13.  
  14.     socketEventArg.Completed += (sender, args) =>
  15.                                         {
  16.                                             // Retrieve the result of this request
  17.                                             result = args.SocketError.ToString();
  18.  
  19.                                             // Signal that the request is complete, unblocking the UI thread
  20.                                             _clientDone.Set();
  21.  
  22.                                             if (OnConnected == null) return;
  23.  
  24.                                             OnConnected(this, new EventArgs());
  25.                                         };
  26.  
  27.     // Sets the state of the event to nonsignaled, causing threads to block
  28.     _clientDone.Reset();
  29.  
  30.     // Make an asynchronous Connect request over the socket
  31.     _socket.ConnectAsync(socketEventArg);
  32.     
  33.     // Block the UI thread for a maximum of TIMEOUT_MILLISECONDS milliseconds.
  34.     // If no response comes back within this time then proceed
  35.     _clientDone.WaitOne(TimeoutMilliseconds);
  36.  
  37.     return result;
  38. }

The connect method takes in two parameters, the hostname (or IP address) and the portnumber of the socket we want to connect to. Based on these two parameters we create a DnsEndPoint that will be passed to the server socket as being the remote endpoint so that the server socket knows where to send his messages. Next we instantiate the Socket, stating that it’s using an internal IPv4 network, a stream socket and the TCP protocol. Now that we have a socket and an endpoint we’ll need some eventargs, those args will be passed onto the server. Once the connection succeeds (or fails) the Completed event on the SocketAsyncEventArgs will fire, in that event handler we’ll trigger the OnConnected event. With that in place we call the Reset() method on the ManualResetEvent to set the event to nonsignaled, call the ConnectAsync() method on the Socket and pass in the eventargs and finally blocking the thread for a certain time to allow the socket time to connect.

So now that our client can connect to a socket, let’s build something to send messages to the connected socket. From the Windows Phone 8 version it looks like this

Code Snippet
  1. public void Send(string data)
  2. {
  3.     data = data + "<EOF>";
  4.  
  5.     if (_socket != null)
  6.     {
  7.         SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs
  8.             {
  9.                 RemoteEndPoint = _socket.RemoteEndPoint,
  10.                 UserToken = null
  11.             };
  12.  
  13.         socketEventArg.Completed += (sender, args) =>
  14.                                         {
  15.                                             _clientDone.Set();
  16.                                         };
  17.  
  18.         // Add the data to be sent into the buffer
  19.         byte[] payload = Encoding.UTF8.GetBytes(data);
  20.         socketEventArg.SetBuffer(payload, 0, payload.Length);
  21.  
  22.         // Sets the state of the event to nonsignaled, causing threads to block
  23.         _clientDone.Reset();
  24.  
  25.         // Make an asynchronous Send request over the socket
  26.         _socket.SendAsync(socketEventArg);
  27.  
  28.         // Block the UI thread for a maximum of TIMEOUT_MILLISECONDS milliseconds.
  29.         // If no response comes back within this time then proceed
  30.         _clientDone.WaitOne(TimeoutMilliseconds);
  31.     }
  32. }

We get in the message as a parameter, at the end of the message I add “<EOF>” just to make sure that I only get the part that I need at the server side and that the message has been delivered in full. If at this point the socket is null then it isn’t connected yet. If it isn’t we once again instantiate SocketAsyncEventArgs. The data gets serialized into a byte array and set as a buffer in the eventargs. Set the ManualResetEvent to nonsignaled, start sending the message over the socket async and block the thread.

And last but not least there’s the code to receive messages on the socket.

Code Snippet
  1. public void Receive()
  2. {
  3.     if (_socket != null)
  4.     {
  5.         SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs
  6.             {
  7.                 RemoteEndPoint = _socket.RemoteEndPoint
  8.             };
  9.  
  10.         // Setup the buffer to receive the data
  11.         socketEventArg.SetBuffer(new Byte[MaxBufferSize], 0, MaxBufferSize);
  12.  
  13.         socketEventArg.Completed += (sender, args) =>
  14.             {
  15.                 if (args.SocketError == SocketError.Success)
  16.                 {
  17.                     // Retrieve the data from the buffer
  18.                     string response = Encoding.UTF8.GetString(args.Buffer, args.Offset, args.BytesTransferred);
  19.                     response = response.Trim('\0');
  20.  
  21.                     if (response.Contains("<EOF>"))
  22.                     {
  23.                         response = response.Substring(0, response.IndexOf("<EOF>"));
  24.                         if (OnMessageReceived != null)
  25.                             OnMessageReceived(this, new MessageReceivedEventArgs(response));
  26.                     }
  27.  
  28.                     Receive();
  29.                 }
  30.  
  31.                 _clientDone.Set();
  32.             };
  33.  
  34.         // Sets the state of the event to nonsignaled, causing threads to block
  35.         _clientDone.Reset();
  36.  
  37.         // Make an asynchronous Receive request over the socket
  38.         _socket.ReceiveAsync(socketEventArg);
  39.  
  40.         // Block the UI thread for a maximum of TIMEOUT_MILLISECONDS milliseconds.
  41.         // If no response comes back within this time then proceed
  42.         _clientDone.WaitOne(TimeoutMilliseconds);
  43.     }
  44. }

So the same story with the null check on the socket and creating of the SocketEventArgs. Once the Completed event fires we’ll check if the receive was successful  and if the received message contains “<EOF>”, if it does we take everything before <EOF> and pass it in the MessageReceivedEventArgs that go with the MessageReceived event.

Code Snippet
  1. public class MessageReceivedEventArgs : EventArgs
  2. {
  3.     public string Response { get; set; }
  4.  
  5.     public MessageReceivedEventArgs(string response)
  6.     {
  7.         Response = response;
  8.     }
  9. }

Once everthing is handled we call the Receive method again so it’s ready to start receiving the next message.

And that’s basically all the logic for a socket connection on the Windows Phone side of things. The only thing left to do in the app is change the MainPage’s constructor to initialize the SocketClient and add some fields.

Code Snippet
  1. private SocketClient _client;
  2. const int Port = 8000;
  3. private const string IpAddress = "10.16.79.70";
  4.  
  5. // Constructor
  6. public MainPage()
  7. {
  8.     InitializeComponent();
  9.     _client = new SocketClient();
  10.  
  11.     _client.OnConnected += (sender, args) => Dispatcher.BeginInvoke(() =>
  12.         {
  13.             TextBlockStatus.Text = "Connected";
  14.             TextBlockStatus.Foreground = new SolidColorBrush(Colors.Green);
  15.         });
  16.     _client.OnMessageReceived += (sender, args) => Dispatcher.BeginInvoke(() => MoveBall(args.Response));
  17.     _client.Connect(IpAddress, Port);
  18.  
  19.     _client.Receive();
  20. }

The constructor instantiates the SocketClient, attaches an handler to the OnConnected event to update the textbox with the connection status and handle the OnMessageReceived event to update the position of the ellipse. Don’t forget to update the IP address to the one from your own pc!

Back to WPF!

Now that our Windows Phone app is ready, it’s time to implement the socket server. As mentioned before, the WPF app that gets input from the Leap Motion will serve as socket server.

I’ve added a class to the WPF project called SocketFactory, it serves the same function as the SocketClient class in the Windows Phone project but from a server point of view. The way to build and use a socket in full blown .net 4.5 differs a bit from how we did it in Windows Phone. First we’ll need a state class, this contains a socket, the buffersize, a byte array to function as the buffer and a stringbuilder to recompose the message.

Code Snippet
  1. // State object for reading client data asynchronously
  2. public class StateObject
  3. {
  4.     // Client  socket.
  5.     public Socket WorkSocket = null;
  6.     // Size of receive buffer.
  7.     public const int BufferSize = 1024;
  8.     // Receive buffer.
  9.     public byte[] Buffer = new byte[BufferSize];
  10.     // Received data string.
  11.     public StringBuilder Sb = new StringBuilder();
  12. }

Next, we’ll need some fields in the SocketFactory class

Code Snippet
  1. // Thread signal.
  2. public static ManualResetEvent AllDone = new ManualResetEvent(false);
  3. private StateObject _state;
  4. public event EventHandler OnConnected;

And here’s the function to receive a connection request

Code Snippet
  1. public void Start()
  2. {
  3.     IPHostEntry ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
  4.     IPAddress ipAddress = ipHostInfo.AddressList[3];
  5.     IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 8000);
  6.  
  7.     Socket listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
  8.  
  9.     // Bind the socket to the local endpoint and listen for incoming connections.
  10.     try
  11.     {
  12.         listener.Bind(localEndPoint);
  13.         listener.Listen(100);
  14.  
  15.         while (true)
  16.         {
  17.             // Set the event to nonsignaled state.
  18.             AllDone.Reset();
  19.  
  20.             // Start an asynchronous socket to listen for connections.
  21.             Console.WriteLine("Waiting for a connection...");
  22.             listener.BeginAccept(AcceptCallback, listener);
  23.  
  24.             // Wait until a connection is made before continuing.
  25.             AllDone.WaitOne();
  26.  
  27.             if (OnConnected == null) return;
  28.  
  29.             OnConnected(this, new EventArgs());
  30.         }
  31.  
  32.     }
  33.     catch (Exception e)
  34.     {
  35.         Console.WriteLine(e.ToString());
  36.     }
  37. }

We declare a socket called listener, pass it the same parameters as we did for Windows Phone, stating that it’s local network, streaming and using the TCP protocol. We need to bind this socket to an IPEndPoint, to create an IPEndPoint we need an IPAddress and a port. Lines 3 and 4 are used to getting the computer’s IP address from its hostname. Once we have that we can bind the socket to the endpoint. Make sure that the port you set here is the same port you try to connect to in the Windows Phone app. Once the socket connects the callback fires, in the callback we start receiving data from the connected client.

Code Snippet
  1. public void AcceptCallback(IAsyncResult ar)
  2. {
  3.     // Signal the main thread to continue.
  4.     AllDone.Set();
  5.  
  6.     // Get the socket that handles the client request.
  7.     Socket listener = (Socket)ar.AsyncState;
  8.     Socket handler = listener.EndAccept(ar);
  9.  
  10.     // Create the state object.
  11.     _state = new StateObject { WorkSocket = handler };
  12.  
  13.     handler.BeginReceive(_state.Buffer, 0, StateObject.BufferSize, 0, ReadCallback, _state);
  14. }

Once data is received, the ReadCallback fires

Code Snippet
  1. public void ReadCallback(IAsyncResult ar)
  2. {
  3.     StateObject state = (StateObject)ar.AsyncState;
  4.     Socket handler = state.WorkSocket;
  5.  
  6.     int bytesRead = handler.EndReceive(ar);
  7.  
  8.     if (bytesRead > 0)
  9.     {
  10.         // There  might be more data, so store the data received so far.
  11.         state.Sb.Append(Encoding.ASCII.GetString(
  12.             state.Buffer, 0, bytesRead));
  13.  
  14.         // Check for end-of-file tag. If it is not there, read
  15.         // more data.
  16.         string content = state.Sb.ToString();
  17.         if (content.IndexOf("<EOF>") > -1)
  18.         {
  19.             Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
  20.                 content.Length, content);
  21.         }
  22.         else
  23.         {
  24.             // Not all data received. Get more.
  25.             handler.BeginReceive(state.Buffer, 0, StateObject.BufferSize, 0, ReadCallback, state);
  26.         }
  27.     }
  28. }

We can get the stateobject and socket from the IAsyncResult and we start reading until we encounter “<EOF>”, once everything is received we write it out in the output window. Now in this example the server won’t actually be receiving any messages, I’ve just put in this method in case you want to enhance it or have some use for the code.

What is important in this sample is the Send method, this will send a message over the connected socket to the client.

Code Snippet
  1. public void Send(string data)
  2. {
  3.     Send(_state.WorkSocket, data);
  4. }
  5.  
  6. private void Send(Socket handler, string data)
  7. {
  8.     data = data + "<EOF>";
  9.     // Convert the string data to byte data using ASCII encoding.
  10.     byte[] byteData = Encoding.ASCII.GetBytes(data);
  11.  
  12.     // Begin sending the data to the remote device.
  13.     handler.BeginSend(byteData, 0, byteData.Length, 0, SendCallback, handler);
  14. }

I’ve split up the Send into two methods, the public one just receives the message we want to send and passes it trough to the private one, also passing in the socket to use. This way the application itself doesn’t need to worry about selecting a socket, let the socketFactory deal with that. The Send method adds the end of file part to the message, serializes it into a byte array and sends it over the socket. Once the send is complete, the callback will fire.

Code Snippet
  1. private void SendCallback(IAsyncResult ar)
  2. {
  3.     try
  4.     {
  5.         // Retrieve the socket from the state object.
  6.         Socket handler = (Socket)ar.AsyncState;
  7.  
  8.         handler.EndSend(ar);
  9.     }
  10.     catch (Exception e)
  11.     {
  12.         Console.WriteLine(e.ToString());
  13.     }
  14. }

I’m not really doing anything in the callback, but this would be the perfect place to check for successful delivery and maybe notifying the UI thread to show a confirmation or something similar.

Now that our socket infrastructure is in place and we have a client updating a UI based on the received messages, it’s time to finish this sample by letting the server send the coordinates of the ellipse. In the MainWindow add a field for the socket factory. (line 5 is the extra field)

Code Snippet
  1. private Frame _previousFrame;
  2. private Frame _currenFrame;
  3. private readonly PocListener _listener;
  4. private readonly Controller _controller;
  5. private readonly SocketFactory _socketFactory;

With the new field in place, replace (or update) the constructor to this

Code Snippet
  1. public MainWindow()
  2. {
  3.     InitializeComponent();
  4.  
  5.     _socketFactory = new SocketFactory();
  6.     _socketFactory.OnConnected += (sender, args) => Dispatcher.BeginInvoke((Action) (() =>
  7.         {
  8.             TextBlockStatus.Text = "Connected";
  9.             TextBlockStatus.Foreground = new SolidColorBrush(Colors.Green);
  10.         }));
  11.  
  12.     Task.Run(() => _socketFactory.Start());
  13.  
  14.     _listener = new PocListener();
  15.     _controller = new Controller();
  16.  
  17.     // Have the sample listener receive events from the controller
  18.     _controller.AddListener(_listener);
  19.  
  20.     _listener.FrameDetected += ListenerOnFrameDetected;
  21.  
  22.     Timer timer = new Timer(200);
  23.     timer.Elapsed += (sender, args) => Dispatcher.BeginInvoke((Action)(() =>
  24.         _socketFactory.Send(string.Format("{0};{1}", Canvas.GetTop(TheEllipse), Canvas.GetLeft(TheEllipse)))));
  25.  
  26.     timer.Start();
  27. }

We instantiate the SocketFactory, attach an handler for the OnConnected event to update the UI. The SocketFactory’s Start method is queued on the thread pool to run async, this way the UI thread will remain responsive. The Leap Motion get’s initialized and we start a timer. This timer will make sure that every 200 milliseconds the coordinates of the ellipse are send over the socket to the client. The sending of the message needs to be done on the UI thread because we need the ellipse’s coordinates, and that ellipse lives on the UI thread.

Now, why use a timer? In the first version of this sample a message was send every time the ellipse moved, this resulted in really really really poor performance, messages were being send faster than they arrived causing all kinds of weird behavior. Sending it every 200 milliseconds makes it move quite well.

Everything is in place now, so run it and move the ellipse around with the Leap Motion!

Conclusion

In this post I’ve explained my adventure of connecting the Leap Motion to a Windows Phone application by using sockets. While not the easiest thing to set up or use, sockets are a really powerful way of communicating between applications no matter what platform they’re on (as long as that platform supports sockets).

The Leap Motion is a great device. It’s small, light, has a very small footprint on your system and is just plain fun to mess around with. I could’ve used anything for a socket example but making something move on screen, on two devices at the same time by just moving your hand has something magically. The Leap Motion is definitely on my list of awesome gadgets.

The code for this post can be found on my Skydrive


Tags:

NUI | LeapMotion | .Net | Devices | WP8 | WPF

blog comments powered by Disqus
  Log in

About the author

Hi,

My name is Nico, I’m an MVP Windows Platform Development living in Belgium.
I’m currently employed as a .NET consultant at RealDolmen, one of Belgium’s leading IT single source providers.

I'm also founding member and board member of the Belgian Metro App Developer Network, a user group focussed on Windows 8 and Windows Phone development. If you're in Belgium feel free to drop by if we're doing an event. http://www.madn.be

Since June 2012 I'm a proud member of Microsoft's Extended Experts Team Belgium. And in February 2013 I became a member of DZone's Most Valuable Bloggers family.

In 2013 I became a book author and wrote "Windows 8 app projects, XAML & C# edition".

In 2014 I received the MVP award for the very first time.

I hope to get feedback from my readers either through comments, mail (nico_vermeir@hotmail.com), twitter, facebook, …

 

mvp

 

mvp

 

 

MeetLogo

 

MVBLogo

mybook

 

Month List