Steve Spencer's Blog

Blogging on Azure Stuff

Custom ASP.NET MVC app running in a Container on Service Fabric

In an earlier post, I talked about how to create a Docker container on Windows that housed a custom ASP.Net MVC app. What I want to show now is how you can get this container running in Service Fabric.

I created 3 identical virtual machines all capable of running Docker as in my earlier post. Now I needed to make my three VMs into a Service fabric cluster. These two posts explain how:

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-get-started

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-standalone-deployment-preparation

My 3 VMs are called sf0, sf1 & sf2 and I needed to  put these into my cluster config. I picked the ClusterConfig.Unsecure.MultiMachine config file that comes with the Service Fabric files and changed it to include my 3 VMs, so my nodes look like this:

"nodes": [
{
      "nodeName": "sf0",
      "iPAddress": "sf0",
      "nodeTypeRef": "NodeType0",
      "faultDomain": "fd:/dc1/r0",
      "upgradeDomain": "UD0"
},
{
      "nodeName": "sf1",
      "iPAddress": "sf1",
      "nodeTypeRef": "NodeType0",
      "faultDomain": "fd:/dc2/r0",
      "upgradeDomain": "UD1"
},
{
      "nodeName": "sf2",
      "iPAddress": "sf2",
      "nodeTypeRef": "NodeType0",
      "faultDomain": "fd:/dc3/r0",
      "upgradeDomain": "UD2"
}
],

I then remoted onto one of the machines and ran the following PowerShell:

.\TestConfiguration.ps1 -ClusterConfigFilePath .\ClusterConfig.json

This will check all the machines in the ClusterConfig.json file to see if they are configured correctly and report any errors. I got the following error:

Machine 'sf2' is not reachable on port 445. Check connectivity/open ports. Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 192.168.1.222:445

This meant I needed to open the correct firewall ports on my VM. I got this error for all the machines in the cluster. Once I fixed this and reran the PowerShell, the tests passed which meant I could install Service Fabric on each of the machines as follows:

.\CreateServiceFabricCluster.ps1 -ClusterConfigFilePath .\ClusterConfig.json –AcceptEULA

When this completes successfully you should see something like this:

Your cluster is successfully created! You can connect and manage your cluster using Microsoft Azure Service Fabric Explorer or Powershell. To connect through Powershell, run 'Connect-ServiceFabricCluster

I could connect to Service Fabric Explorer using: http://sf0:19080

Now I have my cluster running I needed to create a Service Fabric App and deploy it to the cluster. Make sure that you have installed the Service Fabric SDK, then run Visual Studio. Create a new Service Fabric project. When the project is created, right click on the services node, Select Add->New Service Fabric Service

clip_image001[4]

Then pick Guest Container and enter the name in your Docker Hub repository where your Docker image resides.

clip_image001

This will add in the necessary files to your service fabric project. If you remember from my earlier post, the website was hosted on port 8000 of the container. We need to tell service fabric about this and also we may want to map this to a different port.

If you open the containers ServiceManifest file

clip_image001[8]

Add an endpoint with the endpoint you want Service Fabric to use to publish the website out

clip_image001[10]

In this example I’m using the same port. If you want to map the port to a different one then changes this to something else e.g. If I wanted to use http://sf0:8080 as the website then I would change the Service Manifest to this:

image

You also need to tell service fabric about the Container port that is published. This is done in the application manifest file:

image

This is set to 8000 as that is the port exposed by the Docker container

Now deploy your application to service fabric. It may take a while to initialise your container as it will need to be downloaded from Docker Hub before it will run. Once it is running you should see it as Ready in the Service Fabric Explorer

image

Error updating SSL certificates in Azure App Services

I was asked to update the SSL certificates on a website that was hosted in Azure Web Apps. No problem I thought.

Go to the Azure Portal,

Select the website you want to update.

When the blade appears scroll down the left panel and select SSL Certificates

image

image

 

remove the binding, by clicking … at the end of the binding row and select Delete

image

Now remove the certificate by clicking .. at the end of the certificate row and select Delete

This is where I got an error

image

It took a short while to resolve this.

I tried a few things like restarting the site & checking the staging slot, but I still got the error. Finally, I checked other sites in the same app service plan and I had the same certificate used for another Web App (both using the same domain url). Once I removed the binding from that site, I could delete the certificate and upload a new one. I had to then add the new bindings to both sites.

Custom ASP.NET MVC app running in a Windows Container

With the introduction of Windows Containers on  Window Server 2016 and the ability to run containers in Service Fabric I thought it was time to investigate Windows Containers and I wanted to know how to build one that will run a web site using IIS.

As I’m new to containers, although I’ve done a very similar exercise with Docker on Linux, I decided to follow the Windows Quick Start Guide. I hit a few problems early on so I’ve put the steps I followed here:

After opening a PowerShell window as administrator I ran the following commands:

Install-Module -Name DockerMsftProvider -Repository PSGallery –Force – Ran OK
Install-Package -Name docker -ProviderName DockerMsftProvider – Had an error
WARNING: Cannot verify the file SHA256. Deleting the file.
WARNING: C:\Users\ADMINI~1.DEV\AppData\Local\Temp\DockerMsftProvider\Docker-1-12-2-cs2-ws-beta.zip does not exist
Install-Package : Cannot find path 'C:\Users\ADMINI~1.DEV\AppData\Local\Temp\DockerMsftProvider\Docker-1-12-2-cs2-ws-beta.zip' because it does not exist.

 

Not sure what was causing this to fail but I followed the instructions to manually install (from https://github.com/OneGet/MicrosoftDockerProvider/issues/15)

Start-BitsTransfer -Source https://dockermsft.blob.core.windows.net/dockercontainer/docker-1-12-2-cs2-ws-beta.zip -Destination /docker.zip
Get-FileHash -Path /docker.zip -Algorithm SHA256
mkdir C:\Users\Administrator\AppData\Local\Temp\DockerMsftProvider\
cp .\docker.zip C:\Users\Administrator\AppData\Local\Temp\DockerMsftProvider\
cd C:\Users\Administrator\AppData\Local\Temp\DockerMsftProvider\
cp .\docker.zip Docker-1-12-2-cs2-ws-beta.zip
Install-Package -Name docker -ProviderName DockerMsftProvider -Verbose Restart-Computer –Force

After Rebooting I tried to download and run a sample container

docker run microsoft/dotnet-samples:dotnetapp-nanoserver

but I got the following error

docker : C:\Program Files\Docker\docker.exe: error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.25/containers/create: open //./pipe/docker_engine: The

system cannot find the file specified..

At line:1 char:1

+ docker run microsoft/dotnet-samples:dotnetapp-nanoserver

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : NotSpecified: (C:\Program File...ile specified..:String) [], RemoteException

+ FullyQualifiedErrorId : NativeCommandError

It turns out that the docker service wasn’t running after the reboot, so open services.msc and find the docker service to start it.

Running the same command again will download the image from docker hub, create a container from it and then run it. As this is a visual container it runs once and then stops.

clip_image001

Every time I do docker run microsoft/dotnet-samples:dotnetapp-nanoserver it creates a new container. What I want to do is to run one that is stopped and view the output on the screen.

For this I needed to start the container I had already created. If you run

docker –ls –a

This will list all the containers that are both running and stopped and you can see from the image below that I had run docker run a number of time. Each time it tried to download the image (which was already downloaded) and then create a new container from it.

clip_image001[6]

docker container start -a 12d382ae0bd6 (-a attached STDOUT so you can see the output)

clip_image001[8]

Now I know how to create and start containers I wanted to build one of my own. This is easier than I first though as there a lots of base templates stored on docker hub and git hub.

https://docs.microsoft.com/en-us/virtualization/windowscontainers/samples#Application-Frameworks

https://hub.docker.com/r/microsoft/

I picked one on docker hub that has IIS and ASP.Net installed already, so all I needed to do after was to add my own website and configure IIS correctly. Using docker pull,

docker pull microsoft/aspnet (see https://hub.docker.com/r/microsoft/aspnet/)

This retrieves the template from Docker Hub and I want to use that template to install my ASP.Net MVC site and configure IIS to serve the pages on port 8000. Following the instructions here (https://docs.microsoft.com/en-us/dotnet/articles/framework/docker/aspnetmvc). I published my MVC site and copied the publish folder to my docker machine. Then I needed to create a Dockerfile recipe to instruct docker what to install in my image. So I created a folder that contained the Dockerfile and also the published website as below

clip_image001[10]

The contents of the Dockerfile are:

# The FROM instruction specifies the base image. You are
# extending the microsoft/aspnet image.
FROM microsoft/aspnet
# Next, this Dockerfile creates a directory for your application
RUN mkdir C:\sdsweb
# configure the new site in IIS.
RUN powershell -NoProfile -Command \
Import-module IISAdministration; \
New-IISSite -Name "sdsweb" -PhysicalPath C:\sdsweb -BindingInformation "*:8000:"
# This instruction tells the container to listen on port 8000.
EXPOSE 8000
# The final instruction copies the site you published earlier into the container.
ADD sdswebsource/ /sdsweb

Now I need to run this to create the image

In PowerShell, I changed directory to the folder containing my Dockerfile, then ran

docker build -t sdsweb .

This has created an image and we need to now get this running as a container

clip_image001[12]

using docker run again

docker run -d -p 8000:8000 --name sdsweb sdsweb

clip_image001[14]

My container is now running and I should be able to view the web pages in my browser on port 8000, but I need to know the IP address first

docker inspect -f "{{ .NetworkSettings.Networks.nat.IPAddress }}" sdsweb

clip_image001[16]

Now I can browser to http://172.17.97.235:8000

clip_image001[18]

I changed the default web page to show the machine name serving the pages under Getting Started. Listing the containers will show the container ID and this matches the machine name displayed on the web page

image

That’s it running in a container. There are a couple more things I’d like to do before I’ve finished. The first is to make sure that when my Windows Server restarts, then my sdsweb container also starts. At the moment it will not start as I didn’t add a restart parameter when I called docker run. Adding –restart always will cause the container to restart when windows restarts.

docker run -d -p 8000:8000 --name sdsweb --restart always sdsweb

The final thing I want to do is to be able to share this image so I’d like to push it up to docker hub

docker login - enter username and password

docker push recneps/sdsweb

clip_image001[20]

Then to use it on another machine

docker pull recneps/sdsweb

clip_image002

In my next post I am going to look at how I can create a container that can be hosted in Service Fabric

Diagnosing Azure Logic App Faults

In my previous post I showed you how to create a flat file decoding Logic App. When I first tested this I used URL generated when I first saved the Logic App to create a post command using the compose feature of fiddler.

image

The body of the message contained a single cvs row that I wanted to debug and I was hoping to see the response containing the xml representation of the csv file.

When I submitted the request, fiddler displayed an 502 error – Bad Gateway

image

This wasn’t very helpful and I needed to understand what had gone wrong.

Navigating to the Logic App in the Azure Portal I noticed that the Overview blade showed the runs for the Logic App

image

Clicking the failed run opened a new blade in the portal which showed my Logic App and it failing at the Flat File Decoding stage

image

Clicking the Flat File Decode caused it to expand and show me a more detailed (and useful) error

image

The error told me that the Flat File Decode couldn’t find the CR/LF at the end of the line. As I’d put a single line of text and no CR/LF the decoding failed.

I repeated the POST, this time with the CR/LF at the end of the body and the Logic App worked correctly and the correct xml representation of the csv was displayed in the body of the response.

Processing a flat file with Azure Logic Apps

A lot of companies require the transfer of files in order to transact business and there is always a need to translate these files from one format to another. Logic Apps provides a straight forward way to build serverless components that provide the integration points into your systems. This post is going to look at Logic apps enterprise integration to convert a multi-record CSV file into and XML format. Most of the understanding for this came from the following post:

https://seroter.wordpress.com/2016/09/09/trying-out-standard-and-enterprise-templates-in-azure-logic-apps/

Logic Apps can be created in Visual Studio or directly in the Azure Portal using the browser. Navigate to the azure portal https://portal.azure.com click the plus button at the top of the right hand column, then Web + Mobile then Logic App

clip_image002

Complete the form and click Create

clip_image003

This will take a short while to complete. Once complete you can select the logic app from you resource list to start to use it.

If you look at my recent list

clip_image005

You can see the logic app I’ve just created but you will also see my previous logic app and you will also notice that there is also an integration account and an Azure function. These are both required in order to create the necessary schemas and maps required to translate the CSV file to XML.

The integration account stores the schemas and maps and the Azure function provides some code that is used to translate the CSV to XML.

An integration account is created the same way as a logic app. The easiest way is to click on the plus symbol and then search for integration

clip_image007

Click on Integration Account then Create

clip_image009

Complete the form

clip_image010

Then Create. Once created you can start to add your schemas and maps

clip_image012

You will now need to jump into Visual Studio to create your maps and schemas. You will need to install the Logic Apps Integration Tools for Visual Studio

You will need to create a schema for the CSV file and a schema for the XML file. These two blog posts walk you through creating a flat file schema for a CSV file and also a positional file

I created the following two schemas

clip_image014

clip_image016

Once you have create the two schemas you will need to create a map which allows you to map the fields from one schema to the fields in the other schema.

clip_image018

In order to upload the map you will need to build the project in Visual Studio to build the xslt file.

The schemas and map file project can be found in my repository on GitHub

To upload the files to the integration account, go back to the Azure portal where you previously selected the integration account, click Schemas then Add

clip_image020

Complete the form, select the schema file from your Visual Studio project and click OK. Repeat this for both schema files. You do the same thing for the map file. You will need to navigate to your bin/Debug (or Release) folder to find the xslt file that was built. Your integration account should now show your schemas and maps as uploaded

clip_image021

There’s one more thing to do before you can create your logic app. In order to process the transformation some code is required in an Azure Function. This is standard code and can be created by clicking the highlighted link on this page. Note: If you haven’t used Azure Functions before then you will also need to click the other link first.

clip_image023

clip_image025

This creates you a function with the necessary code required to perform the transformation

clip_image027

You are now ready to start your logic app. Click on the Logic App you created earlier. This will display a page where you can select a template with which to create your app.

clip_image029

Close this down as you need to link your integration account to your logic app.

Click on Settings, then Integration Account and pick the integration Account where you previously uploaded the Schemas and Map files. Save this and return to the logic app template screen.

Select VETER Pipeline

clip_image031

Then “Use This Template”. This is the basis for your transformation logic. All you need to do now is to complete each box.

clip_image032

clip_image034

In Flat File Decoding & XML Validation, pick the CSV schema

clip_image035

In the transform XML

clip_image036

Select the function container, the function and the map file

All we need to do now is to return the transformed xml in the response message. Click “Add an Action” on Transform XML and search for Response.

clip_image038

Pick the Transformed XML content as the body of the response. Click save and the URL for the logic app will be populated in the Request flow

clip_image039

We now have a Request that takes the CSV in the body and it returns the XML transform in the body of the response. You can test this using a tool like PostMan or Fiddler to send in the request to the request URL above.

There is also a test CSV file in my repository on GitHub which can be used to test this.

My next post covers how I diagnosed a fault with this Logic App

TFS Release Manager, Remote PowerShell & errorcode 0x80090322

I’m using Release Manager in Visual Studio Team Services (i.e. the one in the cloud) to deploy to my On Premises backend servers. Release Manager does this by using an agent in the environment within which you want to deploy. You can deploy and configure the agent through Release Manager and instructions are at https://blogs.msdn.microsoft.com/visualstudioalm/2016/04/05/deploy-artifacts-from-onprem-tfs-server-with-release-management-service/

However this requires remote PowerShell configuring on the target machine to work correctly. This you may think is really easy to do.

Run PowerShell as admin then type

Winrm quickconfig

Once configured I had to allow the machine with the agent on to access the target machine

Set-Item wsman:localhost\client\trustedhosts *.<domain name>

And also set up to allow for remote scripts using:

Set-ExecutionPolicy RemoteSigned

Testing this I used:

Enter-PSSession <SERVER NAME>

On all my local VMs this worked fine but as soon as I tried it on my UAT and Production servers I got a generic error which listed a lot of possible problems:

Connecting to remote server <SERVER NAME> failed with the following error message: WinRM cannot process the request.
The following error with errorcode 0x80090322 occurred while using Negotiate authentication: An unknown security error occurred.
Possible causes are:
-The user name or password specified are invalid.
-Kerberos is used when no authentication method and no user name are specified.
-Kerberos accepts domain user names, but not local user names.
-The Service Principal Name (SPN) for the remote computer name and port does not exist.
-The client and remote computers are in different domains and there is no trust between the two domains.
After checking for the above issues, try the following:
-Check the Event Viewer for events related to authentication.
-Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.
Note that computers in the TrustedHosts list might not be authenticated.
-For more information about WinRM configuration, run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.

Searching for the error returned a lot of similar results including:

Deleting SPNs

https://social.technet.microsoft.com/Forums/windows/en-US/a4c5c787-ea65-4150-8d16-2a19c569a589/enterpssession-winrm-cannot-process-the-request-kerberos-authentication-error-0x80090322?forum=winserverpowershell

A conflict between ports

http://sharepoint-community.net/profiles/blogs/powershell-remoting-error

Sites to help with troubleshooting

https://blogs.technet.microsoft.com/jonjor/2009/01/09/winrm-windows-remote-management-troubleshooting/

None of them fixed the problem. I managed to get Remote Powershell to work by setting the SPN to specific ports but this then broke Reporting Services so I reverted my changes.

You know you are struggling when all the searches you do yield results you have already read, but in one web site that didn’t appear to be relevant I found this little nugget of information

Remote PowerShell requires port 80 to be available on the Default Web Site”

https://blogs.technet.microsoft.com/exchange/2010/02/04/troubleshooting-exchange-2010-management-tools-startup-issues/

Looking at my web server there was no default website and nothing using port 80 so I added one and remote PowerShell started to work!

I can now deploy from the cloud on to my back end servers without opening any firewall portsJ

Migrating Azure WebJobs to Azure Service Fabric

As part of a proof of concept for Azure Service Fabric one of the challenges was to migrate backend services from a variety of different places. I had a number of services running as Azure Webjobs on the same platform as my web site. The WebJobs were hosted as triggered services meaning that they were using the WebJobs SDK and this has the advantage that the WebJob will run as a console application outside of the Azure Web site it is currently hosted in.

Azure Service Fabric has the capability to run any Windows application that can be run from a command line as a guest executable. This means that I could host my WebJob in Service Fabric as a guest executable.

Once I had Visual Studio setup with the Service Fabric SDK & Tools. It was relatively straight forward to add the WebJob.

As an example, my WebJob is triggered when a message is placed onto an Azure Storage Queue and it then passes the message into an Azure Service Bus Topic. The WebJob project was added to my Service Fabric application

clip_image001

To add this as a Guest Executable, right click on your service node in the Service Fabric application and select “New Service Fabric Service”

clip_image002

When the “New Service Fabric Service” dialog appears, select “Guest Executable”

clip_image004

Click Browse and select your WebJob executable folder. The WebJob executable should now appear in the Program drop down. Select this, change the service name and click OK.

This should add your WebJob as a guest executable to your application package root

clip_image005

Once deployed to a Service Fabric cluster, your WebJob should run as normal. If you leave the connection string settings the same as they are in the WebJob then your diagnostic traces will appear in the same blob container as they are now.

Service Fabric: Resolving External Service Address

I am using Azure Service Fabric to host my application but I’ve deployed it on premises using a 3 machine cluster (running version Microsoft.Azure.ServiceFabric.WindowsServer.5.1.156.9590). It was easy to deploy and I only needed to run PowerShell on 1 of the nodes to configure up all 3. I followed the instructions here.

From Visual Studio I deployed my application which consisted of a number of stateless services and a WCF service. When everything is running in the cluster it all works fine but I wanted to access the WCF service from outside of the cluster. The first issue was that the actual address of the service is not known but you can see the address if you look at the Service Fabric Explorer of the cluster. Navigating through to the application on one of the nodes returns the url of the service e.g.

net.tcp://192.168.56.122:8081/4f341989-ec72-4cd5-8778-6e11e01dc727/968d5932-935a-4773-b83b-fa99f59d9073-131148669247596041

You don’t want to use this url directly as it could change depending upon the configuration of your cluster and the health of each of the nodes. Service Fabric provides a mechanism for discovering the address of the end point using the Service Resolver. If you are running in the cluster then you can use the default resolver and this will return the url of the end point which you can connect to. However, when you are outside of the cluster you need to tell the resolver where to look for the cluster.

Again if you look at the Service Fabric Explorer you can find out the ports used in the cluster e.g.

<ClientConnectionEndpoint Port="19000" />
<LeaseDriverEndpoint Port="9026" />
<ClusterConnectionEndpoint Port="19001" />
<HttpGatewayEndpoint Port="19080" Protocol="http" />
<ServiceConnectionEndpoint Port="9027" />
<ApplicationEndpoints StartPort="20001" EndPort="20031" />
<EphemeralEndpoints StartPort="20032" EndPort="20062" />

The example here shows how to connect to the resolver in an Azure hosted environment.

ServicePartitionResolver resolver = new  ServicePartitionResolver("mycluster.cloudapp.azure.com:19000", "mycluster.cloudapp.azure.com:19001");

This example provides a list of endpoints to try on both ports 19000 & 19001. Mapping this to my environment I used the ip address of the node on which I ran the PowerShell which also is the node that displays the Service Fabric Explorer. I also needed to know the application name in order for the resolver to find the end point I was after. The code below is part of a console application that attempts to call a WCF service from outside of the cluster. I’ve highlighted the service name and addresses I’ve used

string uri = "fabric:/ServiceFabricApp/FileStoreServiceStateless";
Binding binding = WcfUtility.CreateTcpClientBinding();
// Create a partition resolver
var serviceResolver = new ServicePartitionResolver(new string[] { "192.168.56.122:19000" , "192.168.56.122:19001" });
 
// create a  WcfCommunicationClientFactory object.
var clientFactory = new WcfCommunicationClientFactory
                    (clientBinding: binding, servicePartitionResolver: serviceResolver);
 
var client = new ServicePartitionClient>(
                    clientFactory,
                    new Uri(uri), partitionKey: Microsoft.ServiceFabric.Services.Client.ServicePartitionKey.Singleton);
 
var result = client.InvokeWithRetry(svc => svc.Channel.GetDocuments("Document", "1000848776", null));
However, when the code ran it always locked up on the call to InvokeWithRetry. On further investigation by calling ResolveAsyc first, I determined that may application was locking up when trying to resolve the address of the service. This took me a long time to figure out what was wrong and I tried a number of different addresses and ports with no luck. It was only after trying to run the code here, which lists all the services in a cluster, in the Visual Studio debugger that things started to work. This was confusing because I’d already tried loads of different things. The only difference was that the Development Service Fabric was running. I then ran my console app and no lock up occurred. Turning off the development service fabric and the console app locked up again. I moved the console app on to another computer that didn’t have the development service fabric installed and everything worked fine.

The good thing about this is that everything seems to be working and and I’ve learnt more about service fabric now Smile

How to emulate Azure Service Bus Topic Subscription Filtering in RabbitMQ

When creating a subscription to an Azure Service Bus Topic you can add a filter which will determine which messages to send to the subscription based upon the properties of the message.

image

This is done by passing a SqlFilter to the Create Subscription method

e.g.

if (!_NamespaceManager.SubscriptionExists(topic, subscription))
{
    if (!String.IsNullOrEmpty(filter))
    {
        SqlFilter strFilter = new SqlFilter(filter);
        await _NamespaceManager.CreateSubscriptionAsync(topic, subscription, strFilter);
        bSuccess = true;
    }
    else
    {
        await _NamespaceManager.CreateSubscriptionAsync(topic, subscription);
        bSuccess = true;
    }
}

Where strFilter is a string representing the properties that you want to filter on e.g.

// Create a "LowMessages" filtered subscription.

SqlFilter lowMessagesFilter = new SqlFilter("MessageNumber <= 3");

namespaceManager.CreateSubscription("TestTopic","LowMessages",lowMessagesFilter);

Applying properties to messages makes it easier to configure multiple subscribers to sets of messages rather than having multiple subscribers that receive all the messages, providing you with a flexible approach to building your messaging applications.

Subscriptions are effectively individual queues that each subscriber uses to hold the messages that a relevant to the subscriptions

When a message is pushed onto a Topic the Service Bus will look at all the subscriptions for the Topic and determine which messages are relevant to the subscription. If it is relevant then the subscription will receive the message into its queue. If there are no subscriptions capable of receiving the message then the message will be lost unless the topic is configured to throw an exception when there are no subscriptions to receive the message.

This approach is useful if most of the message data is stored in the properties (which are subject to a size limit of 64KB) and the body content is serialised to the same object (or the body object types are known).

Receiving messages on a Service Bus Subscription is as follows:

MessagingFactory messageFactory = MessagingFactory.CreateFromConnectionString(_ConnectionString);
SubscriptionClient client = messageFactory.CreateSubscriptionClient(topic, subscription);
message = await client.ReceiveAsync(new TimeSpan(0, 5, 0));
if (message != null)
{
    properties = message.Properties;
    body = message.GetBody<MyCustomBodyData>();
    if (processMessage != null)
    {
        // do some work
    }
    message.Complete();
}

Over the past few months I have been looking at RabbitMQ and trying to apply my Service Bus knowledge, as well as looking at the differences. Routing messages based upon the message properties rather than a routing key defined in the message is still applicable in the RabbitMQ world and RabbitMQ is configurable enough to work in this way. RabbitMQ requires more configuration than Service Bus but there is a mechanism called Header Exchange which can be used to route messages based upon the properties of the message.

The first thing to do is to create the exchange, then assign a queue to it based upon a set of filter criteria. I’ve been creating my exchanges with an alternate exchange to allow me to receive message that are not handled in a default queue. The code to create the exchange and queue that subscribes to messages where the ClientId property is “Client1” and the FileType property is “transaction”.

// Create Header Exchange with alternate-exchange

IDictionary<String, Object> args4 = new Dictionary<String, Object>();

args4.Add("alternate-exchange", alternateExchangeNameForHeaderExchange);

channel.ExchangeDeclare(HeaderExchangeName, "headers", true, false, args4);

channel.ExchangeDeclare(alternateExchangeNameForHeaderExchange, "fanout");

//Queue for Header Exchange Client1 & transaction

Dictionary<string, object> bindingArgs = new Dictionary<string, object>();

bindingArgs.Add("x-match", "all"); //any or all

bindingArgs.Add("ClientId", "Client1");

bindingArgs.Add("FileType", "transaction");

channel.QueueDeclare(HeaderQueueName, true, false, false, args5);

channel.QueueBind(HeaderQueueName, HeaderExchangeName, "", bindingArgs);

//queue for Header Exchange alternate exchange (all other)

channel.QueueDeclare(unroutedMessagesQueueNameForHeaderExchange, true, false, false, null);

channel.QueueBind(unroutedMessagesQueueNameForHeaderExchange, alternateExchangeNameForHeaderExchange, "");

This will setup the exchange and queue in RabbitMQ and now you can send a message to the exchange with the correct properties as follows:

IBasicProperties properties = channel.CreateBasicProperties();
properties.Headers = new Dictionary<string, object>();
properties.Headers.Add("ClientId", "Client1");
properties.Headers.Add("FileType", "transaction");


string routingkey = "header.key";
var message = "Hello World";
var body = Encoding.UTF8.GetBytes(message);

channel.BasicPublish(exchange: TopicName,
                                routingKey: routingkey,
                                basicProperties: properties,
                                body: body);

Receiving messages from the queue is as follows:

var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
    var body = ea.Body;
    var message = Encoding.UTF8.GetString(body);
    var routingKey = ea.RoutingKey;
    Byte[] FileTypeBytes = (Byte[])ea.BasicProperties.Headers["FileType"];
    Byte[] ClientIDBytes = (Byte[])ea.BasicProperties.Headers["ClientId"];
    string FileType = System.Text.Encoding.ASCII.GetString(FileTypeBytes);
    string ClientID = System.Text.Encoding.ASCII.GetString(ClientIDBytes);
    Console.WriteLine(" [x] Received '{0}':'{1}' [{2}] [{3}]",
                        routingKey,
                        message,
                        ClientID,
                        FileType);
    EventingBasicConsumer c = model as EventingBasicConsumer;
    if (c != null)
    {
        c.Model.BasicAck(ea.DeliveryTag, false);
        Console.WriteLine(" [x] Received {0} rk {1} ex {2} ct {3}", message, ea.RoutingKey, ea.Exchange, ea.ConsumerTag);
    }
};
channel.BasicConsume(queue: queueProcessorBaseName + textBox1.Text,
                        noAck: false,
                        consumer: consumer);

Again an out of the box feature for Service Bus can also be implemented in RabbitMQ but it is much simpler to use in Service Bus. The use of properties to route messages offers a much more flexible approach but does require that the body of the messages are either not used or are understood by each consumer. Service Bus offers more flexibility as the query string can contain a variety of operators whereas RabbitMQ matches all or some of the header values and not a range.