Steve Spencer's Blog

Blogging on Azure Stuff

Testing connectivity to your backend service with Hybrid Connection in App Service

I have a backend service that is connected to my web site which is hosted in Azure App Service using a Hybrid Connection. When setting up a Hybrid Connection it is useful to be able to test connectivity to your backend service. I’ve previously posted a video to show you to access the Kudu control panel so that you can look at the files in the hosted site. We’ll use the Kudu control panel to also test connectivity, only this time we’ll use the PowerShell debug console.

In Azure Portal click on your app service and go to Advanced Tools and select Go

image

This will open the Kudu console in a new tab.

Click on Debug console the PowerShell

image

This opens the debug console in PowerShell and allows your to run PowerShell commands on the App Service. As you are running on the App Service you will have access to the backend service that is connected via the web service.

image

My service has a Get endpoint that I can call to test connectivity. There are a number of commands we could run but I’ll use Invoke-WebRequest.

For this I am using www.bing.com but you use the url of your backend service that you have configured in your Hybrid Connection.

Invoke-WebRequest -Uri http://www.bing.com

image

This returns the following error

The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer's first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

The error says what you need to do. Add UseBasicParsing to the command

Invoke-WebRequest -Uri http://www.bing.com –UseBasicParsing

image

We now get the error:

Win32 internal error "The handle is invalid" 0x6 occurred while reading the console output buffer

To fix this we need to tell PowerShell to silently continue by issuing the following command:

$progressPreference = "silentlyContinue"

Then call your web request

image

I’ve now got the output from the web request with a status code of 200 showing I’ve got connectivity.

Using Graph API to automate Azure AD

In my previous posts I discussed how you can manage access to applications (part 1) using Azure AD and also how you can add users users from outside of your organisation (part 2). Now we will look at how you can automate this using Graph API.

“The Microsoft Graph API offers a single endpoint, https://graph.microsoft.com, to provide access to rich, people-centric data and insights exposed as resources of Microsoft 365 services. You can use REST APIs or SDKs to access the endpoint and build apps that support scenarios spanning across productivity, collaboration, education, security, identity, access, device management, and much more.” - https://docs.microsoft.com/en-us/graph/overview

From the overview you can see that Graph API covers a large area of Microsoft 365 services. One of the services it covers is Azure AD. What I’ll show you today is how to invite users and then add/remove them to/from groups using Graph API.

There are two ways to access Graph API. A user centric approach (Delegated) that requires a user account and an application centric approach that uses an application key and secret. Accessing Azure AD for user invite and group management utilises the application centric approach. In order to get an application id and secret you will need to create an application in Azure AD. The first post in the series talks about how to create an App Registration.

Once you have created your application, there are a couple of bits of information you require in order to get started. These are the tenantId and clientId. These can be found in the Azure portal. Navigate to your App Registration and the details can be found in the Overview blade.

image

If you hover over each of the Guids a copy icon appears to allow you to easily copy these values.

Next you will need a key generating. For this you click on the Certificates and secrets blade.

image

Then click “New client secret” and populate the form and click “Add”

image

Your key will now appear.

image

Make sure you copy this as it is not visible again once you navigate away and you will need to generate a new one.

image

We are now ready to start looking at Graph API. There is good documentation about each of the functions in Graph API including the permissions required to access and code samples in a variety of languages. If we look at the list User function:

https://docs.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http

image

You can see the permissions needed to access this function. As we are using an Application permission type we need to set one of the permissions: User.Read.All, User.ReadWrite.All, Directory.Read.All or Directory.ReadWrite.All.

You can set the permissions required by going to your App Registration and clicking on the “API permissions”

image

The application by default requires a user login that can read their own user profile. We need to add some additional permissions to allow our application to list the users in AD.Click on “Add permission”

This shows the list of built-in API’s that you can access. We are only looking at Microsoft Graph today

image

Click “Microsoft graph”

image

Then “Application permission” and scroll to the User section

image

To list users we need the User.Read.All permission, but we’ll also add the User.Invite.All so that we can invite B2B users. click “Add permissions”.

image

Although you have added the permissions you cannot currently access the Graph API as you will need to Grant admin consent in first. If we had  added a Delegated permission then the user could try an access the Graph API but Admin consent would be required to stop anyone from accessing certain features. This can be done in a workflow with selected Admins being notified of access. Before the use can access an Administrator would need to approve each access. This process will not work for our application as it is an unattended application using the application permission type. We can however grant access to this application user by clicking “Grant admin consent …” button and clicking Yes to the message box that pops up.

image

Clicking the button adds admin consent to all permissions. If you want to remove it from any, click the ellipsis (…) at the end and click “Revoke admin consent”

image

You can also remove permissions from this menu.

Your user is now ready to go. I’m using the C# SDK and this is available as a nuget package

Once the nuget package is installed. You will need to create an instance of the Graph API client:

ConfidentialClientApplicationOptions _applicationOptions = new ConfidentialClientApplicationOptions
{
     ClientId = ConfigurationManager.AppSettings["ClientId"],
     TenantId = ConfigurationManager.AppSettings["TenantId"],
     ClientSecret = ConfigurationManager.AppSettings["AppSecret"]
};


// Build a client application.
IConfidentialClientApplication confidentialClientApplication = ConfidentialClientApplicationBuilder
                 .CreateWithApplicationOptions(_applicationOptions)
             .Build();


// Create an authentication provider by passing in a client application and graph scopes.
ClientCredentialProvider authProvider = new ClientCredentialProvider(confidentialClientApplication);


// Create a new instance of GraphServiceClient with the authentication provider.
GraphServiceClient graphClient = new GraphServiceClient(authProvider);

You will need the ClientId, TenentId and Secret you copied earlier. Looking at the Graph API documentation there are example of how to use each of the functions.

image

We want to see if a user existing in our AAD before we invite them, so we will use the filter option as above.

var user = (await graphClient.Users
                 .Request(options)
                 .Filter($"mail eq '{testUserEmail}'")
                 .GetAsync()).FirstOrDefault();

Console.WriteLine($"{testUserEmail} {user != null} {user?.Id} [{user?.DisplayName}] [{user?.Mail}]");

If user is null then is does not exist in your AzureAD tenant. Assuming that this is an external user then you will need to invite the user to be able to access your application. I created a method for this:

private async Task<Invitation> InviteUser(IGraphServiceClient graphClient, string displayName, string emailAddress, string redirectUrl, bool wantCustomEmaiMessage, string emailMessage)
{
     // Needs: User.InviteAll

    var invite = await graphClient.Invitations
                     .Request().AddAsync(new Invitation
                     {
                         InvitedUserDisplayName = displayName,
                         InvitedUserEmailAddress = emailAddress,
                         SendInvitationMessage = wantCustomEmaiMessage,
                         InviteRedirectUrl = redirectUrl,
                         InvitedUserMessageInfo = wantCustomEmaiMessage ? new InvitedUserMessageInfo
                         {
                             CustomizedMessageBody = emailMessage,
                         } : null
                     });

    return invite;
}

Now you’ve just invited a B2B user into your Azure AD tenant. At the moment they do not have access to anything as you’ve not assigned them to any application. The Graph API for assigning users to applications uses the delegated permissions model which means you need to use an actual user account. The Graph API with the application permission model does not support adding users to applications. In order to use the same application client you used for inviting users, you could assign a group to your application and then use the Graph API to add/remove users to/from that group.

Adding/removing a user to/from a group requires one of the following permissions: GroupMember.ReadWrite.All, Group.ReadWrite.All and Directory.ReadWrite.All. This is set in the same way as for the user permissions in the App Registration/Api permission section mentioned earlier. Admin consent will also need to be granted for these permissions.

The code to add & remove users is below:

// find group
var groupFound = (await graphClient.Groups
                                         .Request()
                                         .Filter($"displayName eq '{groupName}'")
                                         .Expand("members")
                                         .GetAsync()).FirstOrDefault();

Console.WriteLine($"{groupName} {groupFound != null } [{groupFound?.Id}] [{groupFound?.DisplayName}] [{groupFound?.Members?.Count}]");

if (groupFound != null)
{
     // check is the user is already in the group
     var user = (from u in groupFound.Members
                 where u.Id == user.Id
                 select u).FirstOrDefault();
     Console.WriteLine($"user Found {user != null}");

    if (user != null)
     {
         Console.WriteLine($"removing user {user.Id}");
         // remove from group
         await graphClient.Groups[groupFound.Id].Members[user.Id].Reference
                                     .Request()
                                     .DeleteAsync();

    }
     else
     {
         Console.WriteLine($"adding user {user.Id}");
         // add to group
         await graphClient.Groups[groupFound.Id].Members.References
             .Request()
             .AddAsync(new DirectoryObject
             {
                 Id = user.Id
             });
     }
}

In the code above I wanted the Graph API to return me the list of users in the group. By default you do not see this data when retrieving group information. Adding the Expand method tells Graph API to extend the query and return the additional data. This is something to bear in mind when using Graph API. Just because the data is null does not mean that there is no data, you might need to expand the data set returned.

I hope you found this a useful introduction to Graph API, I will be posting more on Azure AD in the future including more on Graph API.

Managing Application Access with Azure AD–Part 2

In my previous post I showed you how to set up an application in Azure AD and allow Azure AD users to access it. In this post I will show how you can give access to these applications to users outside of your organisation using B2B (Business to Business) as guest users.

B2B is a feature of Azure AD that allows you to easily add two types of user to your applications.

  1. Users who are part of another Azure AD tenant
  2. Users who are not.

If your new user is part of another Azure AD tenant, then when we add them as a guest user to you application and they will use the credentials provided by their own organisation. This means they do not have to remember a new username and password when they want to access your application. It is also useful as they will be managed by their own organisation so you will not be responsible for resetting their passwords for example. Another advantage of using their own Azure AD credentials is that they will lose the ability to sign in to your application when their accounts are disabled or removed from your customer’s tenant. They will however still exist as a guest user in your application but they will no longer be able to sign in.

If your new user is not part of another Azure AD tenant, then they will automatically have a Microsoft account created for them. They will also be prompted to enter a new password. Again this is not managed by you but by Microsoft this time, so password resets are handled by a link provided by them.

To assign a guest user to your application you will need to invited them to use your application. They will then receive an invitation via email that they will need to redeem in order to access your application.

So, go back to the Azure AD blade of the Azure portal and click on Users:

Home ) Default Directory > Users All users 
a 
Users I All users 
Default Direct'bqj - Azure Active Director,' 
All users 
Deleted users 
Password reset 
+ New guest user 
New user 
Search users 
Name 
T Bulk create Bulk invite 
Add filters 
user name 
Bulk delete 
Download users 
user type 
Refresh 
p Reset passwo 
Source

The click on “New guest user”

Home > Default Directory > Users I All users > New user 
New user 
Default 
C) Got feedback? 
o 
Create user 
Create a new user in your organization. 
This user will have a user name like 
alice@sdspencergmail.onmicrosoft.com. 
I want to create users in bulk 
Invite user 
Invite a new guest user to collaborate With 
your organization. The user will be emailed 
an invitation they can accept in order to 
begin collaborating. 
I want to invite guest users in bulk 
Help me decide 
Identity 
Name C) 
Email address* @ 
First name 
Last name 
Personal message 
Invite 
Example: •Chris Green' 
xample: chris@contoso.com

Fill in the form and enter your own personal message and click  “Invite”. You need to enter a valid email address otherwise the user will not be able to receive the invite, as seen below:

image

The text highlighted inside the red box was the custom message I entered in the invitation process. It is possible to change the branding of this email but it is an Azure AD premium feature.

The invite process proves that the user has access to the mail box linked to the email address used. Also, if they are using their organisations Azure AD email address then they must also sign in with their own username an password so you can be confident that they user is who they say they are. This example shows the flow when a user is part of another Azure AD tenant. If the user is not part of another tenant then there will be additional screens for setting up their new Microsoft account and password.

When the user clicks the Accept invitation link they will be redirected to a consent page which is asking for permissions to read their user profile from their Azure AD tenant.

image

Accepting the permissions then will redirect the user to the application portal where the user can access the applications they have been assigned. As we have not allocated any applications to this user yet, they will not see anything,

image

To assign applications to the users, go back to the Azure AD blade in teh Azure portal and click on Users then click on the one you have just added to view their profile:

image

You can see, in this example, in the red box that this is a Guest user who has accepted the invitation.

Click on applications in the left hand menu bar you will see that there are none assigned. To assign this user to an application, navigate back to the Azure AD main blade and click Enterprise applications, then select the application you wish to assign this user to.

image

Click “Assign users and groups”, then Add User

image

Click “None Selected” then search for your new user, select them and click Select.

image

Now click Assign

image

The new users is now assigned. Go back to the  Application screen the user viewed after they signed in and refresh the page

image

The assigned application should now be visible and clicking the application will redirect the user to that applications web site.

Using Azure AD it is easy to now invite users to user your applications and when they are part of another Azure AD tenant, Azure AD takes all the pain out of federating with these new users tenants. Hopefully you have found that this is straight forward and this will have opened up access to your applications in a controlled way. My next post will look at how we can automate this using Graph API.

Introduction to Azure Role Based Access Control (RBAC)

Up until fairly recently I have been managing  access to a number of Azure subscriptions but as I’ve been working for smaller organisations the number of people who needed access was fairly small and easy to manage. It also meant that I generally gave the users Owner or Contributor access to the subscriptions as we were all managing everything so we needed the access at that level. Now I work for a large organisation there is a greater need to  limit access to certain areas of Azure and giving subscription wide access is limited to a few key administrators. This means that I need to look at the minimum access that is required for each of the users who need access to the resources. First I’d like to talk about the scope within which permissions can be set within Azure. For most of the scenarios I’ve worked in I have visibility of a single subscription. For organisations  with a large number of subscriptions there is a further level of scope, Management group, which I won’t be discussing.

image

Permissions can be set at the Subscription, Resource group or the individual resource scope.Depending upon the level of access your user requires there are three basic levels which you can use

  • Owner
  • Contributor
  • Reader

Owner gives the user full access to everything within the scope and can also assign roles to other users.

Contributor gives the user full access to everything within the scope except they are not able to assign roles to other users

Reader give the user access to view the resources within the scope but they are not able to change anything or assign roles.

So assigning the user the Owner role at the Subscription level, then the user can manage all resources within the subscription and assign roles to users. A user can be assigned multiple roles and Azure RBAC is additive so if a user was assigned Contributor at the subscription scope but only Reader on one of the resource groups, the Contributor role would override the reader role. It is also possible to have Deny role assignments. Where a user is Denied permissions on a specific role. Deny assignments take precedence over role assignments.

These roles plus the variety of scopes give some flexibility of access but it is still a large surface area of access that is provided. Azure offers a large number of finer grained roles to allow users to be given specific permissions to specific services. There are a large number of built in assignments as can be seen here: https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles

These finer grained roles allow you to set specific permissions on a specific user within a specific scope. For example if  you wanted to give a user access to a blob store to upload files via the Azure portal there are two permissions that can be set: Reader and Data Access and

Storage Blob Data Contributor. If you assign these two roles to a user in the storage account, then the user is able to login to the Azure portal and navigate to the storage account and access the blob store.

To do this, navigate to the storage account within which you want to assign a role and click the access control item

image

The click “Add role assignment”

image

In the role drop down pick “Storage Blob Data Contributor”, select the user you want to assign the role to and click save. Repeat this for the Reader and Data Access role. Your user now has access only to blob storage and has no access elsewhere in the resource group or subscription. I could have done the same thing by selecting the resource group and Access control and adding these roles there. This would have give the user access to all blob stores within the resource group.

Another example is that you may want to give someone access to your app service so that they can configure and deploy. So navigate to your App Service and click “Access control”, then select the role “Website Contributor”. See https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#website-contributor for more details. This lets you manage the selected website but not app service plans and no other web sites. If you want to manage other app services then you could add the same role at the resource group level.

Managing Application Access with Azure AD – Part 1

In my next series of blog post I want to talk about how to manage access to applications using Azure AD.

I’ve been looking at how I can set up access to my web based applications and I want to be able to:

  1. Have a single sign on with multiple applications
  2. Allow some users access to only some of the applications
  3. Be able to give access to users outside of my organisation
  4. Be able to control access via code

Part 1 will cover setting my applications up and then restricting access to the applications via Azure AD.

In order to test this I needed to have a number of applications that I could use. I used this example:

https://github.com/AzureADQuickStarts/AppModelv2-WebApp-OpenIDConnect-DotNet

It allows me to login and see my claims. I deployed this into two different app services so I could navigate to them separately. I’m not going to talk about the code on the web side apart from the bits you need to configure up the sample. This series of blogs are more about how to setup Azure AD and the path I went through to my end goal of configuring up users programmatically.

In order to integrate with Azure AD we need to set up each of the applications. This will provide us with an ID with which we can  use to configure each of the applications.

In Azure Portal navigate to Azure Active Directory, or search for it in the search bar

C portal-azure.com/#home 
Microsoft Azure 
p Search resources, services, and docs (G./) 
Azure services 
Create a 
resource 
Azure Active 
Directory 
SQL databases 
Azure AD 
Privileged. 
App 
registrations    C portal-azure.com/#home 
p activd 
Microsoft Azure 
Services 
Azure s 
Azure AD Privileged Identity Management 
-+ Activity log 
Azure Active Directory 
reso HDlnsight clusters 
e Monitor

Home > Default Directory I Overview 
O Default Directory I Overviev 
Azure Active Directory 
p Search (Ctrl 4/) 
O Overview 
Getting started 
Diagnose and solve problems 
Manage 
users 
Organizational relationships 
Roles and administrators 
Enterprise applications 
Devices 
App registrations 
Identity Governance    Home > Default Directory App registrations 
Default Directory I App registrations 
Active 
Search (Ctrl *

In the menu bar on the left select App Registrations –> New registration and complete the form:

Home ) Default Directory App registrations ) Register an application 
Register an application 
-k Name 
The user-facing display name for this application (this can be changed later). 
My New app 
Supported account types 
Who can use this application or access this API? 
@ Accounts in this organizational directory only (Default Directory only - Single tenant) 
O Accounts in any organizational directory (Any Azure AD directory - Multitenant) 
O Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) 
Help me choose... 
Redirect URI (optional) 
We'll return the authentication response to this URI after successfully authenticating the user. Providing this now is optional and it can be 
changed later, but a value is required for most authentication scenarios. 
web 
v http /mynewapp.azurewebsites.net 
gy proceeding, you agree to the Microsoft Platform Policies 
Register

I've picked single tenant as I want to invite users using B2B. Now click Register

You need to copy the ID's needed for your web app:

Delete Endpoints 
O 
Got a second? We would love your feedback on Microsoft identity platform (previously Azure AD for developer). * 
Display name 
Application (client) ID 
Directory (tenant) ID 
Object ID 
My New app 
Supported account types 
: My organization only 
Redirect URIS 
: I web, O public client 
Application ID URI 
: Add an Application ID URI 
Managed application in My New app

Copy the Client ID and Tenant ID. Repeat this process for the next app. I've created two apps as I wanted to test limiting access to a single app and deny access to the second if the users has not been invited to it or added manually.

Now add these to the web.config in the sample app. There will be two settings for ClientId and Tenant. Make sure that the redirect url matches the url of the application you registered and redeploy. Repeat this for the second application.

If you navigate to the web apps and try and login, you may get an error as we haven't setup any users, although any users currently in your Azure AD should be able to login.

To give users access to your app. Go back to Azure Active Directory and this time select Enterprise Applications and click on the app you just created.

Home 
) Default Directory > Enterprise applications All applications > My New app I Overview 
My New app I Overview 
Enterp•ise Application 
rvlew 
Diagnose and solve problems 
Manage 
Properties 
Owners 
Users and groups 
Provlston•ng 
Application proxy 
Splf_+ruire 
Properties 
O 
Name 
MN 
My New app 
Application ID 
Object ID Q) 
Getting Started

Click Users and groups

My New app I Users and groups 
Enterprse Applicat& 
+ Add user 
Edit Remove p Update Credentials 
Overview 
O 
The application will appear on the Access Panel for assigned users. 
Diagnose and solve problems 
irst 100 shown, to search all users & groups, enter a display nami 
Manage 
Display Name 
Properties 
NO application assignments found 
Owners 
Users and groups 
provisioning

Click Add user

Home > Default Directory > Enterprise applications I All applications > My New app Users and groups > Add Assignment 
Add Assignment 
Default 
Groups are not available for assignment due to your Active Directory plan level. 
Users 
None Selected 
Select Role 
Default Access

Click None Selected, pick users from the list and click Select. These users have now been given access to your application. However, as I mentioned earlier all users who are part of your Azure AD currently are able to login to your web app, we need to now configure the app so that only assigned users can access it.

Click Properties in your enterprise application and set User Assignment required to yes and click Save. (repeat this for your other application)

Home Default Directory 
My New app I 
Enterprise Applicati'H 
Overview 
> Enterprise applications I All applications > My New app I Properties 
Properties 
Save Discard Delete 
Enabled for users to sign-in? O 
Diagnose and solve problems 
Manage 
properties 
Owners 
Users and groups 
provisioning 
Application proxy 
Self-service 
Security 
Conditional Access 
permissions 
Token encryption 
Name * (D 
Homepage URL G) 
Logo @ 
Application ID O 
Object ID O 
User assignment required? C) 
Visible to users? O 
My New app 
MN 
Select a file

Now only users who are assigned to your application can login. You can test this now. Go to the first application url and login with one of the users you assigned. Then go to the second app (you shouldn't have assigned any users just yet.) and login. This time you will get an error.

You can now assign users to the second application and the error should go away when you attempt to login.

We’ve now set up our applications in Azure AD and limited access to each application. In my next post I’ll show you how you can then add users from outside of your organisation to these applications.

Exporting Logs from Application Insights using Continuous Export

This is the fifth post of my series of posts about Log Analytics and Application Insights. The previous post talked about adding custom logging to you code using Application Insights. Now you’ve got your logging into Application Insights you can run log analytics queries and build dash boards, alerts etc. Sometime though you want to use this data in other systems and it would be useful if you could export the data and use it else where. This post will show you how you can regularly export the data from Application Insights into Azure Storage. Once it is in Storage it can easily be moved into other systems or used else where such as PowerBI. This can be achieved using the Continuous Export feature of Application Insights

To enable Continuous Export, login to the Azure management portal and navigate to Application Insights. Click on the instance you want Continuous Export enabled. The scroll down the options on the left until you find the Configure section and click on Continuous Export

image

To use Continuous Export you will need to configure a storage account. Click Add:

image

Click Data types to export:

image

I was only interested in the logs I emitted from my custom logging, so selected Custom Event, Exception and Trace then clicked OK

image

Next pick a storage location. Make sure you use one where your Application Insights instance is located otherwise you will be charged egress fees to move the data to a new datacentre.

image

You can now pick an existing storage account or create a new one. Upon selecting a storage account you can pick an existing  container or create a new one.Once a blob container is selected click OK.

Continuous Export is now configured. You will not see anything in the storage container until the next set of logs are sent to Application Insights.

My log analytics query shows the following logs have been generated:

image

If you look at the continuous Export configuration page you will see that the last updated date has changed.

image

Now look in blob storage. You should see a folder that is<ApplicationInsights_ServiceName>_<ApplicationInsights_InstrumentationKey

image

Click through and you will see a number of folders. One for each of the logs that I enabled when setting up Continuous Export. Click through one of them and you will see a folder for the date and then a folder for the hour of the logs. Then a file containing the logs for that hour.

image

You will get a row of json data for each row output for the log query. Note the whole logs emitted will be in each of the folders.

e.g.

{
     "event": [
         {
             "name": "Some Important Work Completed",
             "count": 1
         }
     ],
     "internal": {
         "data": {
             "id": "a guid is here",
             "documentVersion": "1.61"
         }
     },
     "context": {
         "data": {
             "eventTime": "2020-03-22T18:12:30.2553417Z",
             "isSynthetic": false,
             "samplingRate": 100.0
         },
         "cloud": {},
         "device": {
             "type": "PC",
             "roleInstance": "yourcomputer",
             "screenResolution": {}
         },
         "session": {
             "isFirst": false
         },
         "operation": {},
         "location": {
             "clientip": "0.0.0.0",
             "continent": "Europe",
             "country": "United Kingdom",
             "province": "Nottinghamshire",
             "city": "Nottingham"
         },
         "custom": {
             "dimensions": [
                 {
                     "CustomerID": "4df16004-2f1b-48c0-87d3-c1251a5db3f6"
                 },
                 {
                     "OrderID": "5440d1cf-5d06-4b0e-bffb-fad522af4ad1"
                 },
                 {
                     "InvoiceID": "a7d5a8fb-2a2e-4697-8ab4-f7bf8b8dbe18"
                 }
             ]
         }
     }
}

As the data is now out of Application Insights you can move it where ever you need it. You will also need to manage the blob storage data too otherwise you will end up with the logs stored in two places and the storage costs will be doubled.

One example of subsequent usage is exporting the data to Event Hub. As the data is in blob storage you can use a function with a blob trigger to read the blob in a row at a time and publish the data onto Event Hub:

[FunctionName("ContinuousExport")]
public static async void Run([BlobTrigger("logs/{name}", Connection = "ConitnuousExportBlobSetting")]Stream myBlob, string name,
     [EventHub("logging", Connection = "EventHubConnectionAppSetting")] IAsyncCollector<string> outputEvents,TraceWriter log)
{
     log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
     StreamReader textReader = new StreamReader(myBlob);
     while (!textReader.EndOfStream)
     {
         string line = textReader.ReadLine();
         log.Info(line);
         await outputEvents.AddAsync(line);
     }
}

Note: This is an example, so will need additional code to make sure that you don’t exceed the Event Hub maximum message size

So with Continuous Export you can extract your log data from Application insights and move it to other systems for processing.

Processing data from IoT Hub in Azure Functions

If you have been following my previous posts (Part 1, part 2, part 3) you will know that I’m using an ESP 8266 to send data to the Azure IoT hub. This post will show you how to receive that data and store it in Azure Storage and also show how you can also forward the data onto the Azure Service Bus.

I’m going to use Visual Studio and C# to write my function. If you are unfamiliar with Azure functions you can setup bindings to a variety of Azure resources. These bindings make it easy to interface without needing to write a lot of boiler plate code. These bindings allow your function to be triggered when something happens on the resource or also use the output bindings to write data to these resources. For example, there are bindings for Blob and Table storage, Service bus, Timers etc. We’re interested in the IoT hub binding. The IoT hub trigger will be fired when an event is sent to the underlying Event hub. You can also use an output binding to put messages into the IoT hub event stream. We’re going to use the Table storage and Service bus output bindings.

To get started you need to create a new Function project in Visual Studio.

image

Select IoT hub trigger and browse to a storage account you wish to use (for logging) plus add in the setting name you want to use to store the IoT hub connection string.

image

This will generate your empty function with you preconfigured IoT hub trigger.

You need to add your IoT hub connection string to your setting file. Open local.settings.json and add in a new line below the AzureWebjobs settings with the same name you entered in the dialog. ConnectionStringSetting in my example.Your connection string can be found in the Azure Portal.

Navigate to your IoT hub, then click Shared Access Policies

image

Select the user you want to use to access the IoT hub and click the copy icon next to the primary key connection string.

image

You can run this in the Visual Studio debugger and when messages are sent to your IoT hub you should see a log appearing in the output window.

What I want to do is to receive the temperature and humidity readings from my ESP 8266 and store the data in Azure storage so that we can process it later.

For that I need to use the Table storage output binding. Add the binding attribute to your function below the FunctionName binding.

[return: Table("MyTable", Connection = "StorageConnectionAppSetting")]

Again, you will need to add the storage setting into your config file. Find your storage account in the Azure portal, click Access keys then copy the key1 connection string and paste it in your config file

image

To use Azure Storage Output binding you will need to create a class that represents the columns in you table.

image

I included a device id so that I can identify which device the reading we associated to. You will need to change the return type of your function to be TempHumidityIoTTableEntity then add the code to extract the data from the message.

Firstly, I changed the python code in my ESP8266 to send the data as json so we can process it easier. I’ve also added a message identifier so that we can send different messages from the ESP8266 and be able to process them differently.

sensor.measure()

dataDict = {'partitionKey': 'r',

      'rowkey':'recneptiot'+str(utime.ticks_ms()),

      'message':'temphumidity',

      'temperature':str(sensor.temperature()),

      'humidity': str(sensor.humidity())}

mqtt.publish(sendTopic,ujson.dumps(dataDict),True)

That means we can serialise the Iot Hub message into something we can easily access. So the whole function is below:

[FunctionName("Function1")]
[return: Table("yourtablename", Connection = "StorageConnectionAppSetting")]
public static TempHumidityIoTTableEntity Run([IoTHubTrigger("messages/events", Connection = "ConnectionStringSetting")]EventData message, TraceWriter log)
{
     var messageAsJson = Encoding.UTF8.GetString(message.GetBytes());
     log.Info($"C# IoT Hub trigger function processed a message: {messageAsJson}");

    var data = JsonConvert.DeserializeObject<Dictionary<string, string>>(messageAsJson);

    var deviceid = message.SystemProperties["iothub-connection-device-id"];

    return new TempHumidityIoTTableEntity
     {
         PartitionKey = deviceid.ToString(),
         RowKey = $"{deviceid}{message.EnqueuedTimeUtc.Ticks}",
         DeviceId = deviceid.ToString(),
         Humidity = data.ContainsKey("humidity") ? data["humidity"] : "",
         Temperature = data.ContainsKey("temperature") ? data["temperature"] : "",
         DateMeasured = message.EnqueuedTimeUtc.ToString("O")
     };

}

Providing your config is correct you should be able to run this in the Visual Studio debugger and view your data in Table Storage:

image

I mentioned at the start that I wanted to pass some messages onto the Azure Service bus. For example we may want to do something if the humidity goes above 60 percent. In this example we could add a HighHumidity message to service bus for some other service or function to respond to. We’ll send the message as a json string so that we can action it later in a different service. You can easily add a Service Bus output binding to your function. However, this binding documentation shows it as another return value. There is an alternative binging that allows you to set a message string out parameter with the message contents. This can be used as follows:

    [FunctionName("Function1")]
     [return: Table("yourtablename", Connection = "StorageConnectionAppSetting")]
     public static TempHumidityIoTTableEntity Run([IoTHubTrigger("messages/events", Connection = "ConnectionStringSetting")]EventData message,
         [ServiceBus("yourQueueOrTopicName", Connection = "ServiceBusConnectionSetting", EntityType = EntityType.Topic)]out string queueMessage,
         TraceWriter log)
     {
         var messageAsJson = Encoding.UTF8.GetString(message.GetBytes());
         log.Info($"C# IoT Hub trigger function processed a message: {messageAsJson}");

        var data = JsonConvert.DeserializeObject<Dictionary<string, string>>(messageAsJson);

        var deviceid = message.SystemProperties["iothub-connection-device-id"];

        queueMessage = null;
         if (data.ContainsKey("humidity"))
         {
             int humidity = int.Parse(data["humidity"]);

            if (humidity > 60)
             {
                 Dictionary<string, string> overHumidityThresholdMessage = new Dictionary<string, string>
                 {      
                     { "deviceId",deviceid.ToString()},
                     { "humidity", humidity.ToString()},
                     {"message", "HighHumidityThreshold" }
                 };
                 queueMessage = JsonConvert.SerializeObject(overHumidityThresholdMessage);
             }
         }

        return new TempHumidityIoTTableEntity
         {
             PartitionKey = deviceid.ToString(),
             RowKey = $"{deviceid}{message.EnqueuedTimeUtc.Ticks}",
             DeviceId = deviceid.ToString(),
             Humidity = data.ContainsKey("humidity") ? data["humidity"] : "",
             Temperature = data.ContainsKey("temperature") ? data["temperature"] : "",
             DateMeasured = message.EnqueuedTimeUtc.ToString("O")
         };

    }
}

We now have a function that reads the device temperature and humidity reading into table storage and then sends a message to a Service Bus Topic if the temperature goes above a threshold value.

Generating your IoT Hub Shared Access Signature for your ESP 8266 using Azure Functions

In my last 2 posts I showed how you can connect your ESP 8266 to the IoT hub to receive messages from the hub and also to send messages. One of the issue I had was generating the Shared Access Signature (SAS) which is required to connect to the IoT hub. I was unable to generate this on the device so I decided to use Azure Functions. The code required is straight forward and can be found here: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-security#security-tokens

To create an Azure Function, go to the Azure management portal click the menu icon in the top left and select “Create a Resource”

image

Search for “Function”

image

and select “Function App” and click Create

image

Complete the form

image

And click Review and Create to accept the defaults or click next and work through the wizard if you want to change from the default values.

image

Click create to kick of the deployment of your new Azure Function. Once the deployment is complete navigate to the Function by clicking “Go To Resource”. You now need to create your function.

Click the + sign next to “Functions”. I used the In-portal editor as it was the easiest to use at the time as I already had most of the code copied from the site mentioned above.

image

Click In-Portal, then Continue and choose the Webhook + API template and click Create

image

Your function is now ready for editing. It will have some default code in there to give you an idea how to start

image


We’re going to use the previous SAS code in here and modify it to accept a json payload with the parameters you need for the SAS to be created.

The json we’ll use is as follows:

{
     "resourceUri":"[Your IOT Hub Name].azure-devices.net/devices/[Your DeviceId]",
     "expiryInSeconds":86400,
     "key":"[SAS Key from IoT hub]"
}

You can get you SAS key from the IoT hub in the Azure Portal in the devices section. Click on the device

image

Then copy the Primary or Secondary key.

Back to the function. In the editor Paste the following code:

C# function

#r "Newtonsoft.Json"

using System;

using System.Net;

using Microsoft.AspNetCore.Mvc;

using Microsoft.Extensions.Primitives;

using Newtonsoft.Json;

using System.Globalization;

using System.Net.Http;

using System.Security.Cryptography;

using System.Text;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)

{

     log.LogInformation("C# HTTP trigger function processed a request.");

     string token = "";

     try

     {

          string requestBody = await new StreamReader(req.Body).ReadToEndAsync();

          dynamic data = JsonConvert.DeserializeObject(requestBody);

          int expiryInSeconds = (int)data?.expiryInSeconds;

          string resourceUri = data?.resourceUri;

          string key = data?.key;

          string policyName = data?.policyName;

          TimeSpan fromEpochStart = DateTime.UtcNow - new DateTime(1970, 1, 1);

          string expiry = Convert.ToString((int)fromEpochStart.TotalSeconds + expiryInSeconds);

          string stringToSign = WebUtility.UrlEncode(resourceUri) + "\n" + expiry;

          HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(key));

          string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

          token = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}", WebUtility.UrlEncode(resourceUri), WebUtility.UrlEncode(signature), expiry);

          if (!String.IsNullOrEmpty(policyName))

          {

               token += "&skn=" + policyName;

          }

     }

     catch(Exception ex)

     {

          return (ActionResult)new OkObjectResult($"{ex.Message}");

     }

     return (ActionResult)new OkObjectResult($"{token}");

}

Click Save and Run and make sure that there are no compilation errors. To use the function you need to post the json payload to the following address:

https://[your Function Name].azurewebsites.net/api/HttpTrigger1?code=[your function access key]

To retrieve your function access key, click Manage and copy your key from the Function Keys section

image

We’re now ready to use this in micropython on your ESP 8266. I created a function to retrieve the SAS

def getsas(hubname, deviceid, key):

    import urequests

    import ujson

    dict = {}

    dict["resourceUri"] = hubname+'.azure-devices.net/devices/'+deviceid

    dict["key"] = key

    dict["expiryInSeconds"]=86400

    payload = ujson.dumps(dict)

    response = urequests.post('https://[your function name].azurewebsites.net/api/HttpTrigger1?code=[your function access key]', data=payload)

    return response.text

In my connectMQTT() function from the first post I replaced the hard coded SAS string with a call to the getsas function. The function returns a SAS which is valid for 24 hours so you will need to retrieve a new SAS once 24 hours has elapsed.


I can now run my ESP 8266 code without modifying it to give it a new SAS each time I want to use it. I always forgot and wondered why it never worked the next time I used it. I can now both send and receive data from/to the ESP 8266 and also generate a SAS to access the IoT hub. The next step is to use the data received by the hub in an application and send action messages back to the ESP 8266 if changes are made. I look forward to letting you know how I got on with that in a future post.

Sending data from the ESP 8266 to the Azure IoT hub using MQTT and MicroPython

In my previous post I showed you how to connect your ESP 8266 to the Azure IoT hub and be able to receive messages from the IoT hub to turn on a LED. In this post I'll show you how to send data to the IoT hub. For this I need to use a sensor that I will read at regular intervals and then send the data back to the IoT hub. I picked a temperature and humidity sensor I had from the kit of sensors I bought

image

This sensor is compatible with the DHT MicroPython library. I order to connect to the IoT hub use the same connect code that is in my previous post. The difference with sending is you need a end point for MQTT to send you temperature and humidity data to. The topic to send to is as follows:

devices/<your deviceId>/messages/events/

So using the same device id as in the last post then my send topic would be devices/esp8266/messages/events/

To send a message to the IoT hub use the publish method. This needs the topic plus the message you want to send. I concatenated the temperature and humidity and separated them with a comma for simplicity

import dht

import time

sensor = dht.DHT11(machine.Pin(16))

mqtt=connectMQTT()

sendTopic = 'devices/<your deviceId>/messages/events/'

while True:

    sensor.measure()

    mqtt.publish(sendTopic,str(sensor.temperature())+','+str(sensor.humidity()),True)

    time.sleep(1)

The code above is all that is required to read the sensor every second and send the data to the IoT hub.

In Visual Studio Code with the Azure IoT Hub Toolkit extension installed, you can monitor the messages that are sent to your IoT hub. In the devices view, right click on the device that has sent the data and select “Start Monitoring Built-in Event Endpoint”

v NO FOLDER OPENED 
You have not yet opened 
Open Fol 
> OUTLINE 
v AZURE IOT HUB 
v o recnepsiotu)l 
> Modules 
> Interfaces (Preview) 
Send D2C Message to 10T Hub 
Send C2D Message to Device 
Invoke Device Direct Method 
Edit Device Twin 
Start Monitoring Built-in Event Endpoint 
Start Receiving C2D Message 
Generate Code 
Generate SAS Token for Device 
Get Device Info 
Copy Device Connection 
Delete Device 
> Distributed Tracing Setting (Preview) 
> Endpoints 
"body": "23 54 
"applicationPro 
"mqtt-retain" 
[ 10THubFbni tor]

This then displays the messages that are received by your IoT hub in the output window

PROBLEMS 
OUTPUT DEBUG coNSOLE 
Azure IOT Hub Toolkit 
[10THub"bnitor] Created partition receiver [1] for consumerGroup [$Defau1t] 
[10THub"bnitor] [9:12:39 PM] Message received from [recnepsiotoøl] : 
"body": "23 54 
"applicationproperties 
"mqtt-retain": "true 
[10THubFbnitor] [9:14:28 PM] Message received from [recnepsiotoøl] : 
"body": "23,54 
"applicationproperties 
"mqtt-retain": "true

You can see in the body of the received message the temperature and humidity values that were sent.

I still need to sort out generating the Shared Access Signature and also programmatically access the data I send to the IoT hub. I hope to have blog posts for these soon.

Connecting the ESP 8266 to Azure IoT Hub using MQTT and MicroPython

Recently  was introduced to the ESP 8266 processor which is a low cost IoT device with built in Wi-Fi, costing around £3 - £4 for a development board. The thing that interested me (apart from price) was the device is Arduino compatible and will also run MicroPython. The version I purchased from Amazon was the NodeMcu variant with built in power and serial port via a microUsb port, so it makes an ideal board to start with as there are no additional components required.

clip_image001

This board however did not have MicroPython installed and that required a firmware change. The instructions were fairly straight forward and I followed this tutorial.

After installing MicroPython you can connect to the device using a terminal emulator via the USB serial port. Check in Device Manager to find the COM port number and the default baud rate is 115200. I used the Arduino Serial Monitor tool. In the terminal emulator you can press enter and you should get back the python REPL prompt. If not then you have the COM port or Baud rate wrong.

image

You can write you python directly into here but its easier to write the python in you PC then run it on the device. For this I use ampy

In Command Prompt install ampy using:

pip install adafruit-ampy

This allows you to connect to your device. Close the terminal emulator to free up the COM port then type the following to list the files on your device:

ampy --port COM4 --baud 115200 ls

The MicroPython Quick Ref will summarise how to access the GPIO ports etc but in order to connect to the IoT hub you will need to configure the Wi-Fi on the device. This can be done using the network module.

So create a new text file on your PC and write the code to connect to your Wi-Fi. To test this you can use ampy to run the python on the device:

ampy --port COM4 --baud 115200 run networking.py

Its a good idea to use print statements to help debug as once the run has complete the output will be reflected back in your Command Prompt.

Now you are connected to Wi-Fi we can start to look at connecting to the IoT hub. I am assuming that you already have your IoT hub set up. We now need to configure you new device. Navigate to the IoT hub in your Azure Portal. In Explorers click IoT Devices, then New

image

Enter your device id, the name your device will be known as. All your devices need a name that is unique to your IoT hub. Then click Save. This will auto generate the keys needed to generate the shared access signature needed to access the IoT hub later.

image

Once created you may need to click refresh in the devices list to see you new device. Click the device and copy the primary key, you will ned this for later to generate the Shared Access Signature used in the connection string. In order to generate a new Shared Access Token you can use Visual Studio Code with the Azure IoT Hub Toolkit extension installed. This puts a list of devices and endpoints in the explorer view and allows you to create a new Shared Access Token. find your device in the Devices list, Right click and select Generate SAS Token For Device

image

You will be prompted to enter the number of hours the token is valid for and the new SAS token will appear in the output window:

image

SharedAccessSignature sr=[your iothub name].azure-devices.net%2Fdevices%2Fesp8266&sig=bSpX6UMM5hdUKXHfTagZF7cNKDwKnp7I3Oi9LWTZpXI%3D&se=1574590568

The shared access signature is made up of the full address of your device, a time stamp indicating how long the signature is valid for and the whole thing is signed. You can take this an use it to test your access to IoT hub, so make sure you make the time long enough to allow you to test. The ESP8266 doesn't have a clock that can be used to generate the correct time so you will need to create the SAS off board. I’m going to use an Azure function with the code here to generate it.

Back to Python now. In order to connect to the IoT hub you will need to use the MQTT protocol. MicroPython uses umqtt.simple.

There are a few things required before you can connect.

Firstly the Shared Access Signature that you created above.

Next you will need to get the DigiCert Baltimore Root certificate that Iot Hub uses for SSL. This can be found here. Copy the text from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----, including both the Begin and End lines. Remove the quotes and replace the \r\n with real new line in your text editor then save the file as something like baltimore.cer.

Next you will need a ClientId. For IoT hub the ClientId is the name of your device in IoT Hub. In this example it is esp8266

Next you will new a Username. For IoT hub, this is the full cname of your IoT Hub with your client id and a version. e.g. [your iothub name].azure-devices.net/esp8266//?api-version=2018-06-30

The following code should allow you to connect to the IoT Hub:

def connectMQTT():
     from umqtt.simple import MQTTClient

    CERT_PATH = "baltimore.cer"
     print('getting cert')
     with open(CERT_PATH, 'r') as f:
         cert = f.read()
     print('got cert')
     sslparams = {'cert':cert}

   CLIENT_ID='esp8266'
     Username='yourIotHub.azure-devices.net/esp8266/?api-version=2018-06-30'
     Password='SharedAccessSignature sr=yourIotHub.azure-devices.net%2Fdevices%2Fesp8266&sig=bSpX6UMM5hdUKXHfTagZF7cNKDwKnp7I3Oi9LWTZpXI%3D&se=1574590568'

   

    mqtt=MQTTClient(client_id=CLIENT_ID,server='yourIotHub.azure-devices.net',port=8883,user=Username,password=Password, keepalive=4000, ssl=True, ssl_params=sslparams)


     mqtt.set_callback(lightLed)
     mqtt.connect(False)

    mqtt.subscribe('devices/esp8266/messages/devicebound/#')
     flashled(4,0.1, blueled)


    return mqtt

set_callback requires a function which will be called when there is a device message sent from the IoT Hub. Mine just turns a Led on or off

def lightLed(topic, msg):
     if msg== b'on':
         statusled.on()
     else:
         statusled.off()

connect(False) means that the topic this device subscribes to will persist after the device disconnects.

I’ve also configured the device to connect to its bound topics so that any message sent to the device will call the callback function.

Now we need to have a process loop so that we can receive the messages. The ESP8266 does not seem to run async code so we need to call the wait_msq function to get any message back from the IoT hub

mqtt=connectMQTT()
print('connected...')
while True:
     mqtt.wait_msg()

save your python as networking.py (and make sure that all the code you wrote initially to connect to Wi-Fi is included) then run ampy again:

ampy --port COM4 --baud 115200 run networking.py

Your device should run now. I’ve used the Led flash to show me progress for connecting to Wi-Fi then connecting to IoT Hub and also through to receiving a message. There is a blue LED on the board which I’ve been using as well as a standard LED which is turned on/off based upon the device message received from the IoT Hub. The blue LED is GPIO 2.

In order to send a message from the IoT hub to your device then you can do this from the Azure Portal in the devices view. Click on the device then click Message To Device

image

Enter the Message Body (on or off) and click Send Message

image

Alternatively you can do this in Visual Studio Code by right clicking the device and selecting Send C2D Message To Device and enter the message in the box that pops up

image

In my example the Led lights when I enter on and turns off when I enter off. ampy is likely to timeout during this process, but that’s ok as the board will still be running. As we’ve put the message retrieval inside a loop then the board will continue to run. To stop it running you will need to reset the board by pressing the reset button.

My next step is to sort out automatically generating the Shared Access Signature  and then I’ll look at sending data to the IoT Hub

Steve Spencer's Blog | Blogging on Azure Stuff

Steve Spencer's Blog

Blogging on Azure Stuff

Azure SDK 1.4 Released

The Azure SDK has been released and is available here

The changes are as follows:

  • Resolved an issue that caused full IIS fail when the web.config file was set to read-only.
  • Resolved an issue that caused full IIS packages to double in size when packaged.
  • Resolved an issue that caused a full IIS web role to recycle when the diagnostics store was full.
  • Resolved an IIS log file permission Issue which caused diagnostics to be unable to transfer IIS logs to Windows Azure storage.
  • Resolved an issue preventing csupload to run on x86 platforms.
  • User errors in the web.config are now more easily diagnosable.
  • Enhancements to improve the stability and robustness of Remote Desktop to Windows Azure Roles.

One of the changes fixes the issue I blogged in CommunicationObjectFaultedException after checking an Azure project in to TFS

User Interface Design Principles

Following on from a previous post on UI design (Clicks Cost Money) I thought it was time to write some more.

Designing a software system with user interaction in mind is a challenging task. We are now in a world where the user interface is no longer just mouse clicks on buttons, menus and other controls. Touch screen devices in mobile phones, notebook and desktop computers, tablet devices and multi-touch devices (e.g. Microsoft’s Surface) all lead to a more complex real world user interface where the User Interface designers need to combine an obvious and functional user interface (UI), with sexy graphics and an immersive experience. It is often the need to make the system look good that becomes the dominant factor in UI design and it becomes more important to place things in an aesthetically pleasing way rather than a way that makes the system easy to use. It is a balancing act that the designers and developers need to perform. If you left the UI design to a software developer you will probably end up with a very functional design but not necessarily usable or pretty. However, leaving the designs totally to the designer will lend itself to a sexy looking design but not necessarily functional. You need to combine these roles somehow, either by educating each of the roles to do the other’s job (this is often not a practical approach as the developers aren’t designers and the designers aren’t developers) or by ensuring that there is a high level of collaboration between your developers and designers. In order to ensure that the collaboration works there are a number of guidelines and principles that both sides need to appreciate in order to create a good UI design.

· Visual Structure

· Understandable actions

· Conceptual Model

· Learning

Visual Structure provides the clues to the user about how something works. Understandable actions uses previous learning, existing knowledge and feedback to help the user understand what the system does. The two categories combined will define how well the system works. User maps their ideas of systems they already know to the way your system behaves and create their own conceptual model of the system. Learning is an important process the user must go through in order to make best use of the system. A good user interface should allow the user to experiment with the system and apply their existing knowledge in order to learn how it works.

Visual Structure

When designing user interfaces putting some structure around the information you are trying to convey, will make it easier for the user to understand the system. Formatting the presentation of data is the simplest form of visual structure for example formatting a telephone number as 441274841205 is a lot harder to read than +44(01274)841-205. Also formatting verbose text in a readable way will help the user to easily get to the information they require for example:

Flight FR123 Dublin to Manchester Dep 10:35 Arr 11:20

is harder to read than

Flight:    

         FR123

         

         Dublin -> Manchester

Depart    

         10:35

Arrive:    

         11:20

There are 3 principle areas that help with Visual Structure

· Affordance

· Constraints

· Mapping

Affordance is what is obvious about an object in the way it is used. For example, a button is for pressing and you would expect something to happen when a button is pressed. With a switch you would expect there to be two different states (normally on or off). Getting affordance wrong can lead to confusion in users. How often have you walked up to a door which has a handle which you have pulled to open on to find that it needs to be pushed? These doors often have the words “Push” and “Pull” next to the handle to prompt you into carrying out the correct action. You will still get it wrong from time to time and pull when you should push. If you went to a door that had a flat metal plate on the door and no handle you would instinctively push the door. If when walking to the door from the other side you saw a nice big handle to grab hold of you would pull it. This is where affordance has worked and prompts are not needed to help the user. We need to make our user interfaces obvious to the users especially when the traditional “Window” is now being replaced with more graphical non-windows style user interfaces on smart phones and tablets.

Constraints provide the user with a visual guidance of the limits that they can place on the object for example a scroll bar is constrained within a guide rail so that you can understand the limits of scrolling and also roughly the relative position you are in. Pick lists give the user a limited set of choices they can make rather than allowing the user to pick anything.

Mappings provide clues to how the controls map to the physical object. In a car you will often get a picture of the vehicle with an indication of which door is not shut by showing the specific door in the picture as open. This guides the user directly to the problem. Cooker hob controls can often benefit from mapping. Generally in a 4 ring hob you would have 4 controls and to make the hob look nice and not waste too much space the hob. Most of the time these controls bear no relation to the actual ring that it controls for example:

image image

image image 

Sometimes there does appear to be a relationship. The example below shows that the buttons are grouped left and right but we still do not know which is for the front or the back, but it is a start.

image

A better example is as follows:

image

Here the three rings map onto the position of the controls and it becomes easier to understand without relying on the images on the controls to help you work out which to use. Using mapping to indicate what the control is going to work on, helps the user to pick the correct control for the action they want to do, without having to spend a lot of time thinking.

Understandable Actions

People will use their experiences of existing similar systems to help them to learn about how a new system works. The more similar things there are the easier it will be for a user to learn how to use your system. They will also look at the systems response to make decisions about how the system functions and these feedback mechanisms are often the way a user determines whether your system is good or not. Understandable actions can be broken down into 3 categories:

· Causality

· Transfer Effects

· Population Stereotypes

Causality is about how a system behaves once a user has performed an action. The thing that happens immediately after the user does something is deemed to have been caused by the action. For example when a sound is played once a button is pressed then the user ties the action of pressing the button with the response of playing the sound. Feedback is therefore important. When the user carries out an action they expect something to happen straight after. If it doesn’t then they think is has not worked so they will do it again. If the response to the action is not going to be immediate then the user needs to be given some other feedback immediately for example a busy spinning icon to indicate that the user’s action has been recognised and that something will happen later. The user can get both positive and negative causality from the system. If the system crashes when the user presses a button then they assume that it was the action of pressing the button that caused the crash.

Transfer Effects is where people transfer their previous learning on to the current situation. They take the actions they currently perform on one object and attempt to perform them on similar objects on your application. There can be both positive and negative transfer affects. Positive transfer is when the same perceived behaviour happens on objects within your system that happen on other system. Negative transfer occurs when the actions in your system do not match the perceived behaviour of the action in an alternative system. For example and artist might mix colours by overlaying paints. As the paints allow colour from lower layers to appear the artists can get the exact colour they want. They transfer this knowledge to a computer based painting system, only to find that the colours just overlay on each other and it is only the top colour that is visible and not the mixed colours. To the artist this is not a good result and they are not happy with the system. If the paint application allowed the artist to mix the paints in the same way they do in the real word then they are more likely to want to use the system.

When designing the user interface you must consider the user population that will be interacting with your system and what they are familiar with. This does not necessarily mean country or cultural differences but what are the users used to. For example if you are building an on screen number pad for users who are used to texting using a mobile phone then you would set up the number pad as if it was a mobile phone. If however the users were accountants and used to using calculators then the number pad would match that of a calculator.

Conceptual Model

The principles mentioned previously will help the user to build a picture of how the application works in their minds. It is this picture that we call a conceptual model. The conceptual model is usually based upon something similar that the user knows about. This allows the user to predict the effect of their actions. This is generally difficult to achieve because the designer’s conceptual model may be different to the user’s. The designer also has the difficulty of designing the system so that his own conceptual model is replicated in the system image. If the system image does not make the model clear and consistent, users will develop the wrong conceptual model.

image

Learning

Once you have built a wonderful new user interface and you have applied the principles of user interface design, does this mean you have a good user interface? That depends. It depends upon how easily the user can understand your system and how quickly they can learn how to use it. The final list of principles which will help the user to learn how your system can be used and help them to be confident about it are:

· Be consistent

· Context

· Undo or Cancel

· Good Error handling

Be consistent in the user interface. Make object that look the same perform similarly. Ensure that if you do one action in one place that the same action in a different place does a similar thing. For example single clicking on an object selects the object and double clicking actions the object. If these were swapped around depending upon which application was run the users would be confused and frustrated really easily. Consistency also means taking advantage of the standard. Why reinvent the wheel. If someone has built a UI feature that is used by others then why be different. Being consistent will give the user confidence to try to work out how your system works. Users will take their previous experience and apply it to the current application to see if it works. A good example of this is the two finger resize on Microsoft’s Surface, Windows Phone 7 and IPhone. Each uses the same mechanism to resize which make it easy for the user to interact as they already know how to do things. A web page link in a browser is underlined and often a different colour to the normal text. It is amazing how many web pages attempt to make the link “pretty” by making it different. Having underlined text of a different colour makes it easy for the user to distinguish the links from standard text. When a designer changes the colour and underlines specific text then the user expects to click on it. Similarly if there is no underlining or colour change then how does the user know to click the link?

Context is important because it determines what effect the action the user performs will have, this is also important when linked with consistency. The same action performed to different objects should have the same response, but the response should be appropriate to the context that the user thinks they are in. For example, on Microsoft’s Surface (or other multi touch environment) a user may use a two finger gesture to resize and object. If the two finger gesture is done on a photograph then the user would expect the picture to resize. If the same action was carried out on the map that is underneath the picture, then you would expect the map to resize. Context also help the user “forget” information as they can carry out an action in one context that requires data entry and then once the action has completed the user does not need to think about that data as they are no longer in a context where the data is important. It is therefore necessary to make it obvious to the user that they have finished in one context and moved into another. In a windows environment this is done through the use of modal dialog boxes which appear in front of windows and allow data entry. Once the user presses the OK button the data is saved and the window disappears and then goes back to whatever they were doing before. Context is also a useful technique to show the user what actions can be carried out on the object. We see this often by the greying out of icons and menu items when different objects are selected. Context is important but it also needs to be consistent.

Users learn through trial and error. The users want the confidence to know that if they try something they won’t be stuck doing some really bad action without a way of getting out of it. Make sure that there are cancel buttons and undo mechanisms on actions where the outcome of the action might not be desirable.

Good error handling is also important especially on data entry, make sure that the data is validated and suitable error messages appear so that the user understands what they are expected to enter. If possible make the choices easy for the user and provide fixed data entry using check boxes, radio buttons and lists.

Using these principles can be difficult to get right as the design of the user interface depends upon the experience of the user and what their existing knowledge is. We develop Microsoft surface applications for users who are unfamiliar with the technology so what do we do with UI design. We could design the system for users who are familiar with that style of UI but then we will alienate the new users. Does this mean we need to adapt the UI to provide the consistent user interface that an experienced user requires but also to provide hooks and visual guides for novice users. One example we have here is the two finger resize. How does a novice user know how to resize using two fingers if they haven’t done this before? Do we put together a user interface that has some sort of resize control on the objects we want to resize for example display an image of a control that affords the idea of twisting so that the user can try and twist it and see what happens? We can also provide the two finger resize for the experienced user. Creating a good user interface that meets the expectations of new and existing users is difficult but the principles mentioned here are the same regardless of which style of UI you are using. User interfaces need to look nice, sexy and modern but they also need to be functional and easy to use. Make sure you get the balance right.

Resources

“The Design of Everyday things” – Donald Norman

“Don't Make Me Think!: A Common Sense Approach to Web Usability” - Steve Krug

The Microsoft Surface Design Guidelines - http://msdn.microsoft.com/en-us/library/ff318692(v=Surface.10).aspx

“Human Computer Interaction” – Jenny Preece et al

What is classed as a Storage transaction for billing purposes in Windows Azure?

The simple answer is that each REST call made to the Azure Storage Service is counted as a single transaction. This means that each time you query your table or check the size of a queue or upload a blob you will call the Azure Storage REST api and it will be classed as a transaction. It also means that if you are doing a full table query and you start to get continuation tokens you will get multiple transactions.

For a fuller description see the following blog post:

http://blogs.msdn.com/b/windowsazurestorage/archive/2010/07/09/understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx

Add your own Claims to your ADFS Provider

Following on from my previous blog on “Creating your own identity provider …” The following changes can be made to add in your own claims.

Firstly in the App_Data\CustomSecurityTokenService.cs file of your identity provider web site I changed the following code

outputIdentity.Claims.Add( new Claim( System.IdentityModel.Claims.ClaimTypes.Name, principal.Identity.Name ) );
if (principal.Identity.Name.Equals("Steve") == true)
{
outputIdentity.Claims.Add(new Claim(ClaimTypes.Role, "Administrator"));

outputIdentity.Claims.Add(new Claim("http://schemas.BlackMarble/Identity/Claims/Business",
"Black Marble"));

}
else
{
outputIdentity.Claims.Add(new Claim(ClaimTypes.Role, "User"));
}

 

 
The first parameter of the Claim constructor needs to be in the format of a namespace and I added this one up as it was an internal name we are using.
The second parameter of the Claim constructor is the value you want to pass through.
 
Next go to the appfabric portal and add in the following rule to your STS provider. You need to make sure that the schema string you have in your code matches the Input Claim Type you added in your rule.
 
 
 
image
Now you should be passing through the Business claim to your website. To get access to the claim use the following code:
 
using System.Threading;
using Microsoft.IdentityModel.Claims;
IClaimsPrincipal principal = (IClaimsPrincipal)Thread.CurrentPrincipal;
var business = "";
foreach (Claim claim in principal.Identities[0].Claims)
{
if (claim.ClaimType.Equals("http://schemas.BlackMarble/Identity/Claims/Business"))
{
business = claim.Value;
break;
}
}

if (!String.IsNullOrEmpty(business))
{
// we have a claim value for School so lets display it
BusinessLabel.Text = business;
}
else
{
BusinessLabel.Text = "No business claim found";
}
Again, note that the claim type namespace is the same as you specified previously.

The following claims are passed through to my website:

image

Denial of Service and Windows Azure

A question I regularly get asked is whether there is any support in Azure for preventing Denial of Service (DOS) attacks or at least reducing the impact of the DOS attack. In a white paper written by Microsoft called “Security Best Practices For Developing Windows Azure Applications“ in June 2010 the following statement was made:

“Windows Azure’s load balancing will partially mitigate Denial of Service attacks from the Internet and internal networks. This mitigation is done in conjunction with the developer defining an appropriate Service Definition VM instance count scale-out. On the Internet, Windows Azure VMs are only accessible through public Virtual IP Addresses (VIPs). VIP traffic is routed through Windows Azure’s load-balancing infrastructure. Windows Azure monitors and detects internally initiated Denial of Service attacks and removes offending VMs/accounts from the network. As a further protection, the root host OS that controls guest VMs in the cloud is not directly addressable internally by other tenants on the Windows Azure network and the root host OS is not externally addressable.”

The following table shows the types of protection that are available (or planned)

Threat

Layer where mitigation is implemented

Nature of mitigation provided

(if specific to Windows Azure)

Application/Service-layer mitigation required

Is this issue higher risk or more complex in cloud deployments?

Denial of Service

       

Denial of Service attacks via network bandwidth saturation (packet flooding)

Platform

Load balancing & throttling in network infrastructure

None Required

No

Identification of botnets and malicious network traffic

Infrastructure

Windows Azure Live Services monitors and investigates

None Required

Yes

Deep packet inspection for network attacks with known signatures

Platform (Not yet implemented)

 

None Required

Yes

Flooding of Web Role local storage or blob/table storage

Platform

Quotas, ACLs, Reduced privilege execution and flood monitoring protection

None Required

Yes

Request flooding at the customer code/app level

Web Role (Needs coding)

 

Implement application-level request throttling if necessary

No

My understanding is that there is support within Azure to stop someone from launching DOS attacks within Azure itself. It is difficult to stop a DOS attack on your site but there are things that can be done to minimise their impact. Bandwidth throttling and Load balancing are built in to the platform, but the developer has to code around issues at an application level for example implementing a back off policy when trying to access resources that are affected by DOS.

Windows Azure AppFabric CTP February release now available

Microsoft released on Thursday the latest CTP for Windows Azure AppFabric. Details of the release can be found here, in the AppFabric Team blog and in Wade Wegner’s blog

 

The CTP contains changes to the Caching service and the AppFabric portal is now in Silverlight. The changes are as follows:

  • New Silverlight-based LABS portal, bringing consistency with the production Windows Azure portal.
  • Ability to select either a 128MB or 256MB cache size.
  • Ability to dynamically upgrade or downgrade your cache size.
  • Improved diagnostics with client side tracing and client request tracking capabilities.
  • Overall performance improvements.

You can access the CTP by signing in to the AppFabric labs at http://portal.appfabriclabs.com/

Retrieving Role Instance Count in Azure From a Different Role

When using the following code from a worker role the trace information shows that there are one worker role instance and zero web role instances

public override void Run()
{
while (true)
{
Thread.Sleep(10000);
Trace.WriteLine(string.Format("WorkerRole Instances {0}",
RoleEnvironment.Roles["WorkerRole1"].Instances.Count),
"Information");

Trace.WriteLine(string.Format("WebRole Instances {0}",
RoleEnvironment.Roles["WebRole1"].Instances.Count),
"Information");
}
}

This is because an internal endpoint is required on the role in order for the role environment to be able to retrieve the instance count. So add a new end point to the webrole and set it as internal. Running the code again, then shows both roles with 1 instance running.

See the Role.Instances MSDN topic:

http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.serviceruntime.role.instances.aspx

Windows Azure Training Kit Update

The Windows Azure Training Kit January update is available at:

http://msdn.microsoft.com/en-us/windowsazure/wazplatformtrainingcourse.aspx

The January 2011 update of the training kit includes the following updates:

  • [New demo script] Windows Azure Connect
  • [New demo script] Web and Worker Role Enhancements
  • [New demo script] Windows Azure Virtual Machine Roles
  • [New demo script] Rafiki
  • [New lab] Windows Phone 7 and The Cloud
  • [Improved] Visual Studio code snippets installation
  • [Fixes] Several bug fixes in demos and labs

Migrating an Existing Web Application to Windows Azure

Moving to Windows Azure is not as difficult as you might think. Using a basic MVC 2.0 web application created in Visual Studio 2010, the Azure tools within Visual Studio enable you to migrate the web side really easily.

Load your MVC 2.0 application into Visual Studio and right click on the solution in Solution Explorer and add a new project to this solution.

image

Select Windows Azure Project and enter a new project name.

When the next dialog appears, do not select any role and press OK.

 

image

In the Windows Azure project rigt click on the roles folder and select Add->Web Role Project in solution

image

 

Select the MVC project and press OK

image

You MVC web application should now be ready for deployment to azure. Hit F5 and make sure that the solution builds and runs in development fabric.

For most people this is not the end. There are a number of issues that you may need to resolve for example file access, database access, security and on premise integration. There are solutions for each of these and it depends upon what you are trying to achieve.

With data access, if you are currently using a SQL database and providing the database is not too complicated then it should just port across my running the same SQL scripts on your SQL azure database. You will then just need to change your connection string to point to SQL Azure rather than whatever database you are currently using.  To see the issues the TFS team had converting to the cloud see the PDC video

Information about setting up security using the Access Control Service can be seen in my earlier blog

File access can be done using Azure Drives if there is a lot of file access, but it may be easier to store the file as a single blob or migrate the data you are storing to able storage. I depends upon what you are using the files for.

Connectivity to on premise applications could be done a number of ways by using the service bus or Azure connect. Again it depends on what you are trying to connect to and how “real time” the connection needs to be. I will be putting together a comparison of the methods as soon as I get some free time.

CommunicationObjectFaultedException after checking an Azure project in to TFS

I suddenly started to get a CommunicationObjectFaultedException after I checked my azure code in to TFS

image

I could get it working by editing the web.config file manually, but it didn’t seem to matter what I actually changed!! It was the act of editing the web.config file that made it writable and it could therefore be written to by the development fabric. When comparing the files it looks like the machine key section is changed. Further investigation pointed me in the direction of the changes made in the Azure SDK 1.3 to support full IIS. During deployment “automatic configuration [of the machine key] occurs at the site-level, overriding any user-supplied value”. When the file is read-only the error occurs. Making the file writable fixes the problem.

The following links explain:

http://blogs.msdn.com/b/windowsazure/archive/2010/12/08/specifying-machine-keys-with-windows-azure-sdk-1-3.aspx

http://msdn.microsoft.com/en-us/library/gg494981.aspx

10 March 2011 : Update – Issue now fixed in Azure SDK 1.4. See Azure SDK 1.4 Released