Steve Spencer's Blog

Recently I’ve been looking at how we can load test one of our services so that we are able to understand the load our partners can put onto our systems before we start to have any issues. We used the Cloud-based Load Testing (CLT) service of Visual Studio Team Services (VSTS). I created a short video showing you how to easily setup a url based load test. The next stage of our load testing was to load test the service that our partners provide, Their service required IP whitelisting to connect which meant the CLT service would not be able to connect. Luckily for us the CLT service allows you to deploy agents into your own infrastructure  to carry out the load testing and they are controlled by the same CLT service that we used to load test our own service. This blog post will show you how to install the agent and configure the load test for this scenario.

The load test agent is installed using a PowerShell script which can be obtained from here. Open the PowerShell as administrator and don’t forget to unblock the script

To enable you to run the script and to configure the agents to talk to the CLT service, you will need to create a PAT token in VSTS.

Login and click on your user icon, then select Security

Select Personal Access Tokens, then Add

Fill in the form and click Create Token at the bottom of the page

This creates your token and this is the only time you will be able to access the token

You need to make sure that you copy it now as it you will not be able to access it after you have left the page,

Gong back to PowerShell, run the following command

.\ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName StevesVSTS -PATToken 37abawavsmsgj6hpakltwhdjt4jrqsmup2jx62hlcxbju2l2tbja -ConfigureAgent -AgentGroupName StevesInternalTest

This will take a few minutes to run. If you didn’t run PowerShell as an Administrator you might see errors. When it has run the VSTSLoadAgentService should be installed and running

Now need to configure a load test. Following on from my video, you can create a url test

I ran a test web app in Visual Studio using localhost as the address

When the test has been created select the Settings tab, click “Use self-provisioned agents” and select your agent from the list and add the number of agents you want to use. You could install the agents on a number of machines in your environment using the same script, then you will be able to add more than 1 agent if required. As I only installed the one agent I can only select 1. You can see how many agents the CTL service can see

If there are less than you think you will need to check to make sure the service was installed without error and that it is running 

Save the load test and run it.

Whilst the test was running the performance meters in Visual Studio showed that the web page was being loaded.

When the test is complete you should see that it has not cost you any VUM (Virtual User Minutes) as the load tests are running on your own agents

The Cloud Load Test service allows you to load test both publicly accessible and private websites and services. As long as the servers running the load test agents have outbound access to the internet for HTTPS then we are able to load test private sites and services and the load test does not cost anything apart from the cost of the infrastructure that the agents are running on locally.

I’m using Release Manager in Visual Studio Team Services (i.e. the one in the cloud) to deploy to my On Premises backend servers. Release Manager does this by using an agent in the environment within which you want to deploy. You can deploy and configure the agent through Release Manager and instructions are at https://blogs.msdn.microsoft.com/visualstudioalm/2016/04/05/deploy-artifacts-from-onprem-tfs-server-with-release-management-service/

However this requires remote PowerShell configuring on the target machine to work correctly. This you may think is really easy to do.

Run PowerShell as admin then type

Winrm quickconfig

Once configured I had to allow the machine with the agent on to access the target machine

Set-Item wsman:localhost\client\trustedhosts *.<domain name>

And also set up to allow for remote scripts using:

Set-ExecutionPolicy RemoteSigned

Testing this I used:

Enter-PSSession <SERVER NAME>

On all my local VMs this worked fine but as soon as I tried it on my UAT and Production servers I got a generic error which listed a lot of possible problems:

Connecting to remote server <SERVER NAME> failed with the following error message: WinRM cannot process the request.

The following error with errorcode 0x80090322 occurred while using Negotiate authentication: An unknown security error occurred.

Possible causes are:

-The user name or password specified are invalid.

-Kerberos is used when no authentication method and no user name are specified.

-Kerberos accepts domain user names, but not local user names.

-The Service Principal Name (SPN) for the remote computer name and port does not exist.

-The client and remote computers are in different domains and there is no trust between the two domains.

After checking for the above issues, try the following:

-Check the Event Viewer for events related to authentication.

-Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.

Note that computers in the TrustedHosts list might not be authenticated.

-For more information about WinRM configuration, run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.

Searching for the error returned a lot of similar results including:

Deleting SPNs

https://social.technet.microsoft.com/Forums/windows/en-US/a4c5c787-ea65-4150-8d16-2a19c569a589/enterpssession-winrm-cannot-process-the-request-kerberos-authentication-error-0x80090322?forum=winserverpowershell

A conflict between ports

http://sharepoint-community.net/profiles/blogs/powershell-remoting-error

Sites to help with troubleshooting

https://blogs.technet.microsoft.com/jonjor/2009/01/09/winrm-windows-remote-management-troubleshooting/

None of them fixed the problem. I managed to get Remote Powershell to work by setting the SPN to specific ports but this then broke Reporting Services so I reverted my changes.

You know you are struggling when all the searches you do yield results you have already read, but in one web site that didn’t appear to be relevant I found this little nugget of information

“Remote PowerShell requires port 80 to be available on the Default Web Site”

https://blogs.technet.microsoft.com/exchange/2010/02/04/troubleshooting-exchange-2010-management-tools-startup-issues/

Looking at my web server there was no default website and nothing using port 80 so I added one and remote PowerShell started to work!

I can now deploy from the cloud on to my back end servers without opening any firewall portsJ

TFS Release Manager and Octopus Deploy, both support the concept of approval steps, but why do you need both pre-release and post-release approval steps. When I first started to look at the automated release tools such as TFS release manager I could understand the reason behind the pre-release approval. This step is your quality gate and adds some controls into your process. When creating your release pipeline you will setup a number of environments (e.g. Test, UAT, Pre production, Production) and at each stage you can make a different set of people responsible for allowing the deployment onto each environment.

When the developers have finished their new piece of code and it has been tested in their own environments, they will want to get it onto the servers so that the testers can test it in a more formal way. Currently this may involve the developer talking to the testers and the developers hanging around whilst the testers finish what they are doing so that they can free up the servers ready for the deploy. The developers will then deploy the software to the test environment, hopefully using some method of automation. With release manager, the developers can kick of an automated build and when it is completed it can then automatically create a release ready for deployment. This release can be configured to automatically deploy to the target environment.

The developer can force the software on to the test environment without the tester being ready. The testers for example may be finishing off testing a previous release and need some time before they are ready to accept the software. They may also have a set of criteria that need to be met before they will accept the software onto their environment.

Adding a pre-release approval step that allows the test team to “Accept” the release gives control to the test team and allows them to accept and indeed reject a release. This “Pause” the process allows the testers to check that all the developer quality gates have been met and therefore allow them push back to the developers if they are not happy. As the deployments can be automated, the testers can use the approval process to also control when the new software is to be deployed into their environment, allowing them to complete their current set of tests first. It also frees up the developer, so that they are not hanging around waiting to deploy. Similarly, moving on to UAT, Pre-Prod or Production, a pre-release approval step can be configured with different approvers who then become the gate keepers for each environment.

A pre-release approval step makes a lot of sense and provides order and control to a process and remove a lot of user error from the process.

So what about a post-release approval step, why would you need one? It wasn’t until I started to use TFS Release manager to automatically deploy my applications to Azure Websites where the need to have pre-release approval process became clear. Once I had released my software onto the test environment, I needed a mechanism to allow the testers to be able to reject a release if it failed testing for whatever reason. The post release approval step allowed them to have this power. By adding both a pre and post release approval step for each environment allowed the environment owner to accept the release into the environment when they are ready for it and when they are satisfied that the developers have done their jobs correctly. They can also control when it is ready to move to the next stage in the process. If after completing testing the software is ready to release to UAT then the tester can approve the release which pushes it to the next environment. If the tester is not happy with the release then they can reject it and the release does not move forwards. The tester can comment on the reason for rejection and the release will show red for failure on the dash board. Adding pre and post approval steps to each environment moves the control of software releases onto each environment to a group of people who are responsible for what happens on each.

Using these approval steps can also act as a sanity check to ensure that software releases do not accidentally get pushed onto an environment if someone kicks of the wrong build for example.

I’ve created a release pipeline for my applications which use pre and post approval steps for releases to Test and UAT, I don’t’ have a pre-production environment, but production utilises the staging slots feature of Azure Websites to allow me to deploy the release to staging prior to actually going live. The production environment only has a pre-release approval step, but as it is only going to staging, there is an additional safe guard to allow the coordinated live release when the business is ready.

Both Pre and Post release approval steps provide a useful feature to put the control of the release with the teams that are responsible for each environment. The outcome of each approval process can be visible, which also highlights if and when there are issues with the quality of the software being released.

I’m currently evaluating Release Manager in Visual Studio Team Services and I am using it to deploy website to Azure Web Apps. I recently tried to deploy an Asp.Net MVC 4 application and ran into some issues.

I’ve created a build that packages and zips up my web application which runs successfully.I’ve linked a Release pipeline to this build and I can deploy to my test Azure site without any errors, but when I try and run the web application I get the following error:

Could not load file or assembly 'System.Web.Mvc, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

I’m using Visual Studio 2013 with MVC as a nuget package.Looking at the properties of System.Web.Mvc I can see that it is set to Copy Local = True

I tried a few different things to try to get the assembly to be copied like redoing the nuget install and eventually I toggled the Copy Local to False, saved the project file and then set it back to true. When I looked at the diff of the project file I found an additional property

This seems to fix the build. When I checked this in and rebuild, System.Web.Mvc now appears in the zip file. The build was then release to Azure and the web app worked correctly.