Steve Spencer's Blog

Blogging on Azure Stuff

Handling A Topic Dead Letter Queue in Windows Azure Service Bus

Whilst working on a project in which we we using the Topics on Windows Azure Service Bus, we noticed that our subscription queues (when viewed from the Windows Azure Management portal) didn’t seem to be empty even though our subscription queue processing code was working correctly. On closer inspection we found that our subscription queue was empty and the numbers in the management portal against the subscription were messages that had automatically faulted and had been moved into the Dead Letter queue.

The deadletter queue is a separate queue that allows messages that fail to be processed to be stored and analysed. The address of the deadletter queue is slightly different from your subscription queue and is the form:

YourTopic/Subscriptions/YourSubscription/ $DeadLetterQueue

for a subscription and

YourQueue/$DeadLetterQueue for a queue

Luckily you don’t have to remember this as there are helpful methods to retrieve the address for you:

SubscriptionClient.FormatDeadLetterPath(subscriptionClient.TopicPath, messagesSubscription.Name);

To create a subscription to the deadletter queue you need to append /$DeadLetterQueue to the subscription name when you create the subscription client

Once you have this address you can connect to the dead letter queue in the same way you would connect to the subscription queue. Once a deadletter brokered message is received the properties of the message should contain error information highlighting why it has failed. The message should also contain the message body from the original message. By default the subscription will move a faulty message to the dead letter queue after 10 attempts to deliver. You can also move the message yourself and put in sensible data in the properties if it fails to be processed by calling the DeadLetter method on the BrokeredMessage. The DeadLetter method allows you to pass in your own data to explain why the message has failed.

The DeadLetter can be deleted in the same was as a normal message by calling the Complete() method on the received dead letter message

Here is an example of retrieving a dead lettered message from a subscription queue

var baseAddress = Properties.Settings.Default.ServiceBusNamespace;
var issuerName = Properties.Settings.Default.ServiceBusUser;
var issuerKey = Properties.Settings.Default.ServiceBusKey;

Uri namespaceAddress = ServiceBusEnvironment.CreateServiceUri("sb", baseAddress, string.Empty);

this.namespaceManager = new NamespaceManager(namespaceAddress, 
                           TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
this.messagingFactory = MessagingFactory.Create(namespaceAddress, 
                           TokenProvider.CreateSharedSecretTokenProvider(issuerName, issuerKey));
var topic = this.namespaceManager.GetTopic(Properties.Settings.Default.TopicName);
if (topic != null)
{

    if (!namespaceManager.SubscriptionExists(topic.Path, 
                                  Properties.Settings.Default.SubscriptionName))
    {
        messagesSubscription = this.namespaceManager.CreateSubscription(topic.Path, 
                                            Properties.Settings.Default.SubscriptionName);
    }
    else
    {
        messagesSubscription = namespaceManager.GetSubscription(topic.Path, 
                                            Properties.Settings.Default.SubscriptionName);
    }
}
if (messagesSubscription != null)
{
    SubscriptionClient subscriptionClient = this.messagingFactory.CreateSubscriptionClient(
                                            messagesSubscription.TopicPath,
                                            messagesSubscription.Name, ReceiveMode.PeekLock);

   // Get the Dead Letter queue path for this subscription
    var dlQueueName = SubscriptionClient.FormatDeadLetterPath(subscriptionClient.TopicPath,
                                             messagesSubscription.Name);

   // Create a subscription client to the deadletter queue
    SubscriptionClient deadletterSubscriptionClient = messagingFactory.CreateSubscriptionClient(
                                           subscriptionClient.TopicPath, 
                                            messagesSubscription.Name + "/$DeadLetterQueue");

    // Get the dead letter message
    BrokeredMessage dl = deadletterSubscriptionClient.Receive(new TimeSpan(0, 0, 300));

   // get the properties
    StringBuilder sb = new StringBuilder();
    sb.AppendLine(string.Format("Enqueue Time {0}", dl.EnqueuedTimeUtc));
    foreach (var props in dl.Properties)
    {
        sb.AppendLine(string.Format("{0}:{1}", props.Key, props.Value));
    }
    dl.Complete();
}

MVP Cloud OS Community Week

Black Marble are participating in the Microsoft MVP Cloud OS Community Week. During the week commencing 9th September there will be daily events held at Cardinal Place, Victoria, London. Richard and I along with other MVPs will be participating in the event on Friday 13th Sept which is titled “Enables Modern Business Applications”. The sessions will include developing services for modern apps, case studies, ALM and much more. Further details and registration can be found at: http://mvpcloudosweek-eorg.eventbrite.co.uk/.

Registration for this specific day can be found at: http://enablemodernapps-es2.eventbrite.co.uk/?rank=1

Pricing Changes to Windows Azure

Scott Guthrie made an announcement on his blog about changes to the way Windows Azure is priced

http://weblogs.asp.net/scottgu/archive/2013/06/03/windows-azure-announcing-major-improvements-for-dev-test-in-the-cloud.aspx

The main two changes that will affect most people are

  1. Per minute Billing
  2. No charge for turned off VMs

Prior to this announcement you would be charge for Windows Azure usage in hour blocks of time which meant that if you used 5 minutes of compute time or 55 minutes of compute time you were charged for the full hour, similarly if you used 1 hour 5 minutes you were charged for 2 hours. This change now means that you will be charged for 5 minutes, 55 minutes and 1 hour 5 minutes, matching your actual usage, This doesn’t affect you if you have a fixed usage but for those who take advantage of the easy scalability of Windows Azure this could have a significant impact on your costs.

By far the biggest cost saving is the ability to turn off a VM and not be charged for its usage. Prior to this change you would need to undeploy the solution in order to not be charged and just turning it off would still incur charges, This allows for systems that are in staging, dev/test scenarios, training systems etc to be left deployed and configured but turned off without being charged. Saving both time and money. I have a number of customers where this change alone should half their monthly bills as they have systems running that are not needed all the time but can not be undeployed because of the time taken to deploy and configure them. The ability to turn off VMs will provide both customers and developers with flexibility whilst reducing costs. Always a good thing.

Windows Azure and SignalR with Gadgeteer

I’ve been playing with Gadgeteer (http://www.netmf.com/gadgeteer/) for a while now and I am a big fan of the simple way we can build embedded hardware applications with high functionality. We have a proof of concept device that includes a Colour touch screen, RFID reader and an Ethernet connections. This device is capable of connecting to a web api REST service which we have hosted in Windows Azure and we can use this service to retrieve data from our service depending upon the RFID code that is read. This works well but there are times when we would like to notify the device when something has changed. SignalR seems to be the right technology for this as it removes the need to write polling code in your application.

Gadgeteer uses the .Net Micro framework which is a cut down .Net framework and doesn’t support the ASP.NET SignalR libraries. As we can use web api using the micro framework using the WebRequest classes,  I wondered what was involved to get SignalR working on my Gadgeteer device.

The first problem was to work out the protocol used by SignalR and after a short while trawling the web for details of the protocol I gave up and got my old friend fiddler out to see what was really happening.

After creating a SignalR service I connected my working example to the signalR hub running on my local IIS..

The first thing that pleased me was that the protocol looked fairly simple. It starts with a negotiate which is used to return a token which is needed for the actual connection.

GET /signalr/negotiate?_=1369908593886

Which returns some JSON:

{"Url":"/signalr","ConnectionToken":"xyxljdMWO9CZbAfoGRLxNu54GLHm7YBaSe5Ctv6RseIJpQPRJIquHQKF4heV4B_C2PbVab7OA2_8KA-AoowOEeWCqKljKr4pNSxuyxI0tLIZXqTFpeO7OrZJ4KSx12a30","ConnectionId":"9dbc33c2-0d5e-458f-9ca6-68e3f8ff423e","KeepAliveTimeout":20.0,"DisconnectTimeout":30.0,"TryWebSockets":true,"WebSocketServerUrl":null,"ProtocolVersion":"1.2"}

I used this JSON to pull out the connection id and connection token. This was the first tricky part with the .Net Micro framework. There is not the same support for JSON serialisation you get with the full framework plus the string functions are limited as well. For this I used basic string functions using Substring and IndexOf as follows:

int index = negJson.IndexOf("\""+token+"\":\"");
if (index != -1)
{
    // Extracts the exact JSON value for then name represented by token
    int startindex = index + token.Length + 4;
    int endindex = negJson.IndexOf("\"", startindex);
    if (endindex != -1)
    {
        int length = endindex - startindex;
        stringToExtract = negJson.Substring(startindex, length);
    }
}

With the correct token received Fiddler led me to the actual connection of signalR:

GET /signalr/connect?transport=webSockets&connectionToken=yourtoken&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D&tid=2 HTTP/1.1

Looking at this I could determine that I needed to pass in the token I retrieved from negotiate, the transport type and the name of the hub I want to connect to. After a bit of investigating I used the transport of longPolling.

Now as I think I understood the protocol, I tried to implement it in SignalR. The first issue that arose was what to send with the negotiate call. I figured that this was some sort of id of the client that is trying to connect so I decided to use the current tick count. This seemed to work and I guess that as long as my devices don’t connect at exactly the same time then Signal R would work. I’ve had no problems so far with this.

Upon connecting to the hub I needed to create a separate thread to handle signalR so that the main device wouldn't stop running whilst the connection to the SignalR hub was waiting for a response. Once a response is received the response returns with a block of JSON data appropriate to the SignalR message being received. This needs to be decoded and passed onto the application. You then need to reconnect back to the SignalR hub. The period between receiving data and then reconnecting back to the hub needs to be small. Whilst the message is being processed it cannot receive any more message and may miss some data. I retrieve the response stream and then pass the processing of the stream to a separate thread so that I can reconnect to the hub as fast as possible.

This is not a full implementation of SignalR on the .Net Micro-framework but it is the implementation of a simple client and can be used fairly successfully on the Gadgeteer device. I still need to do a little more work to try to speed up the connections as it is possible to miss some data.

The SignalR hub is hosted on a Windows Azure website along side the web api service which allows both web, Windows 8 and Gadgeteer applications to work side by side.

Gadgeteer has opened up another avenue for development and helps us to provide more variety of devices in a solution

Global Windows Azure Bootcamp

The Global Windows Azure Bootcamp is a one day deep dive session to get you developing on Windows Azure. This is a global event organised by the community for the community. There will be a number of events held at locations across the globe. This event will happen on April 27th, 2013 and Black Marble will be hosting one of the UK events at our offices near Bradford.

Join me and my fellow MVPs Robert Hogg and Andy Westgarth and sign up here to attend this event at Black Marble.

For further information visit the the Global Windows Azure Bootcamp website

Imagine Cup NE Hackathon

We’ve just come back from the Imagine Cup NE hackathon in Newcastle. We had a great time with all the students from Universities and Colleges around the NE of England. Rik has written a good blog post about it. As always I am impressed by the young talent there and there was a number of good ideas. This year the Imagine Cup world finals are being held in St. Petersburg in Russia with three main competitions:

  1. Games
  2. Innovation
  3. World Citizenship

There were entries in all three categories. The two day hack consisted of a series of talk/presentations from MVPs and other industry experts who also provided support and guidance for the teams throughout the 2 days.

At the end of the two days the teams had 10 minutes to present their ideas and be “grilled” in a Q & A session. All teams equipped themselves well.

Helping out with the Imagine Cup not only helps the new generation of software developer, it also inspires us to be better at what we do. There’s some good young talent that are going to give us a good run for our money in a few years time. We’d better be ready for them Smile

Gadgeteer, Ethernet and Windows Azure

I was having problems getting my Gadgeteer ethernet card initialised and running. I wanted to set it up to use DHCP but I never got an IP address assigned. I am using a GHI Electronics J11D ethernet card and browsing for examples seemed to pull up a lot of code but none of it seemed to work or the code didn’t seem to match what the libraries were providing. I eventually found the solution.

// Wire up the event handler to notify when the ip address has been assigned 
// and the port is ready to use
ethernet_J11D.Interface.NetworkAddressChanged += new
   NetworkInterfaceExtension.NetworkAddressChangedEventHandler(
Interface_NetworkAddressChanged); // Open the ethernet port ethernet_J11D.Interface.Open(); // Assign the network stack to the ethernet card if (!ethernet_J11D.Interface.IsActivated) { NetworkInterfaceExtension.AssignNetworkingStackTo(ethernet_J11D.Interface); } // Turn on DHCP and Dynamic DNS ethernet_J11D.Interface.NetworkInterface.EnableDhcp(); ethernet_J11D.Interface.NetworkInterface.EnableDynamicDns();

It was the line (NetworkInterfaceExtension.AssignNetworkingStackTo(ethernet_J11D.Interface); ) that was the issue, once that was in everything worked fine.

I can now connect to my Windows Azure Websites hosted web api/signalR service.

The code for this is fairly standard and once I got the connection it worked well. The code below shows you how to call the web api service from Gadgeteer. This method works for both GET (read) and PUT (update) requests.

private string CallWebservice(string fn, bool put, string data)
{

string responseFromServer ;
try
{
    // Create a request for the URL. 
    WebRequest request = WebRequest.Create(url + fn);

    // set a timeout of a nice big value - 10 minutes
    request.Timeout = 600000;
    if (put)
    {
        request.Method = "PUT";
        System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
        byte[] arr = encoding.GetBytes(data);
        request.ContentType = "application/json";
        request.ContentLength = arr.Length;
        Stream requestStream = request.GetRequestStream();
        requestStream.Write(arr, 0, arr.Length);
        requestStream.Close(); 

    }

    // Get the response.
    WebResponse response = request.GetResponse();

    // Get the stream containing content returned by the server.
    Stream dataStream = response.GetResponseStream();

    // Open the stream using a StreamReader for easy access.
    StreamReader reader = new StreamReader(dataStream);

    // Read the content.
    responseFromServer = reader.ReadToEnd();

    // Tidy up
    reader.Close();
    response.Close();
}
catch (Exception ex)
{
    Debug.Print(ex.Message);
}

return responseFromServer ;
}

 

 

Windows Azure Websites and Custom Domains

I’ve had a number of people ask me about adding a custom domain to a Windows Azure Website and it is a straight forward task providing you have access to edit your DNS records. the first thing to note is that custom domains can’t be configured on the Free instances but on Share and Reserved you get an option to configure the domain.

image

Clicking this option brings up the next screen

image

when you enter the domain name you want to use. You need to fill this in otherwise you will not redirect to your site.

You now need to configure you DNS. I set an CName record up for “www” which I pointed to my website. You need to make sure that the website you configure in your DNS and the domain name you configure in the website match up.

When I change the website back to the free instance I then get 404 errors

Windows Azure Websites, Web API and SignalR

One of our projects involves a web service that implements both SignalR and Web API and we were looking at the quickest and most cost effective way to get it deployed so that one of our customers could run a Windows 8 application as a demo away from the office. The application works well internally as we have the service deployed on one of our servers on IIS. The options we were considering were:

  1. Package the application up in an install package, ship this to our customer and then provide them with instructions and support to allow them to deploy and configure their application
  2. Deploy it on one of our servers and then publish the service through our firewall
  3. Deploy as a Cloud service in Windows Azure
  4. Deploy as a website in Windows Azure

We considered the fact that the first option would probably take us a fair amount of time to make a deployment package, test it and provide enough documentation and support to allow our customer to deploy it on their servers. The other 3 options involved us doing a smaller amount of work, but at least we could get everything working well before shipping the demo out. Option 2 would mean using our internal resources for something that would not be used that often but we would not necessarily know whether and when it was being used so the resources would need to be kept running limiting our capacity internally. Windows Azure was a good fit for this application and the choice was really between setting up a cloud service or use a web site, I guess we could have set up a virtual machine hosted in Windows Azure, but this was a bit excessive just for a simple web service. The two remaining options were to set up a cloud service by creating a web role in deploying to Windows Azure or to use Websites. The cloud service would involve more work for us as we would need to change the project to add in the cloud service project and web role and then do a full PaaS deploy to Windows Azure. This would then utilise a whole virtual machine (although we would have used an Extra Small instance), but the web sites seem a sensible option especially as we already have a number of them available for free. How easy was this going to be and will both Web API and SignalR work with Windows Azure Websites, especially as we were using preview software. I was surprised about how easy this was to deploy and I’ll walk through the process we went through.

Step 1: Make sure that the service runs locally,

Step 2: Our service uses Code First Entity Framework using a local SQL server. Create a database using Windows Azure SQL Server via the Windows Azure Management portal (https://manage.windowsazure.com), the copy the ADO.NET connection string.

image

Paste this into your web config file of the web api service. You will need to make sure that the Windows Azure SQL Server firewall has your public IP address configured and you will need to make sure that your firewall will allow connections through port 1433. Now run your application and make sure that you can connect to the Windows Azure SQL database. As we are using Code First Entity Framework, the database tables were created for me so I didn’t need to do any database deployment. The only issue I had with this approach was that I had to create the database first in Windows Azure.

Step 3: With our service running locally but with the database in Windows Azure we are now ready to deploy to the cloud. In the Windows Azure Management portal, click the “New” button

image

The “Quick Create”, enter the url you want to use and click “Create Web Site”

image

Step 4: We now need to deploy our service. In the Azure management portal, navigate to the web site you just created and click “Download Publishing Profile”. Save this to your computer.

image

In Visual Studio 2012, open your web api project, right click on the project in Solution Explorer and click publish.

image

This will display the publish dialog.

image

Click the import button and navigate to the folder where the publish profile was saved. This should then allow you to complete the wizard

image

Click Next and check to make sure the correct connection string is displayed, click Next then Publish. This should then start to upload your web api project to the Windows Azure Website. The deploy should be relatively quick and no where near the time it takes to deploy a cloud service. When completed, your deployed website should start in the browser and you can carry out whatever tests you need.

Step 5: With your website deployed you should just need to change the url of your service in the Window 8 application.

This whole process took less than 10 minutes to setup and deploy. One of the nice features of using websites is that changes are quick to deploy.

We had a number of issues to get this all working fully:

  1. As I mentioned earlier we had to ensure that the database was created before the EF code would create the correct tables
  2. When we first ran the Windows 8 application we were getting an error each time we tried to use SignalR. We received an “Incompatible protocol version”. This was because I installed the latest SignalR libraries on the server side code but the client was using an older version. You need to make sure that both the client and server are using the same version of SignalR
  3. We also had an issue when deployed to Windows Azure where it looked like the SignalR hubs were not being created correctly. It looked like the hub creation was hanging and not returning. This is a known issue that has been fixed but not yet deployed to Azure. There is a work around which is to configure SignalR to use long polling (https://github.com/SignalR/SignalR/issues/510). We did that with the following code:
   1: hubConnection = new HubConnection(App.SignalRUrl);            
   2: proxy = App.hubConnection.CreateHubProxy("statushub");
   3: App.hubConnection.Start(new LongPollingTransport()).Wait();

Windows Azure Web Sites is not just for web sites, using it also for services can make a lot of sense as the scaling model will allow a lot of flexibility and can provide a cost effective way to host your services, especially if they are not heavily loaded at the start. They are also easy and fast to deploy which is always a bonus Smile

Windows Azure Queues vs Service Bus Queues

If you have been wondering why you would use Windows Azure Queues that are part of the Storage Service or the queues that are part of the service bus then the following MSDN article will give you full details.

http://msdn.microsoft.com/en-us/library/hh767287(VS.103).aspx

Recent changes in pricing make the choice even harder. There are two specific areas I like that makes the service bus queue a better offering than the storage queues:

  1. Long connection timeout
  2. Topics

The long connection timeout means that I don't have to keep polling the service bus queue for messages. I can make a connection for say 20 minutes and then when a message is added to the queue my application immediately returns the data, which you then process and then reconnect to the queue to get the next message. After 20 minutes without a message the connection closes in the same way it does when a message is received except that the message is null. You then just reconnect again for another 20 minute. The makes your application a more event driven application rather than a polling application and it should be more responsive. You can make multiple connections to the queue this way and load balance in the same way as you would when polling queues.

The following code shows how you can connect with a long poll.

 1: NamespaceManager namespaceManager;
 2: MessagingFactory messagingFactory;
 3: Uri namespaceAddress = ServiceBusEnvironment.CreateServiceUri("sb", "yournamespace", string.Empty);
 4:  
 5: namespaceManager = new NamespaceManager(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider("yourIssuerName", "yourIssuerKey"));
 6: messagingFactory = MessagingFactory.Create(namespaceAddress, TokenProvider.CreateSharedSecretTokenProvider("yourIssuerName", "yourIssuerKey"));
 7:  
 8: WaitTimeInMinutes = 20;
 9:  
 10: // check to see if the queue exists. If not then create it
 11: if (!namespaceManager.QueueExists(queueName))
 12: {
 13:     namespaceManager.CreateQueue(queueName);
 14: }
 15:  
 16: QueueClient queueClient = messagingFactory.CreateQueueClient(queueName, ReceiveMode.PeekLock);
 17:  
 18: queueClient.BeginReceive(new TimeSpan(0, WaitTimeInMinutes, 0), this.ReceiveCompleted, messageCount);

When a message is received or the 20 minute timeout expires then the ReceiveCompleted delegate is called and a check is made to see if the message is not null before processing it.Once processed another long poll connecting is made and the process repeats. The beauty of this method is that you don’t have to manage timers or separate threads manage the queue.

Topics are private queues that are subscribed to by consumers and each private queue will receive a copy of the messages put into the original queue and are all manages individually. Topics can also apply filters to the message so that they only receive messages that they are interested in.

Further details of Service bus topics and queues

http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/