Steve Spencer's Blog

Blogging on Azure Stuff

TFS Release Manager, Remote PowerShell & errorcode 0x80090322

I’m using Release Manager in Visual Studio Team Services (i.e. the one in the cloud) to deploy to my On Premises backend servers. Release Manager does this by using an agent in the environment within which you want to deploy. You can deploy and configure the agent through Release Manager and instructions are at https://blogs.msdn.microsoft.com/visualstudioalm/2016/04/05/deploy-artifacts-from-onprem-tfs-server-with-release-management-service/

However this requires remote PowerShell configuring on the target machine to work correctly. This you may think is really easy to do.

Run PowerShell as admin then type

Winrm quickconfig

Once configured I had to allow the machine with the agent on to access the target machine

Set-Item wsman:localhost\client\trustedhosts *.<domain name>

And also set up to allow for remote scripts using:

Set-ExecutionPolicy RemoteSigned

Testing this I used:

Enter-PSSession <SERVER NAME>

On all my local VMs this worked fine but as soon as I tried it on my UAT and Production servers I got a generic error which listed a lot of possible problems:

Connecting to remote server <SERVER NAME> failed with the following error message: WinRM cannot process the request.
The following error with errorcode 0x80090322 occurred while using Negotiate authentication: An unknown security error occurred.
Possible causes are:
-The user name or password specified are invalid.
-Kerberos is used when no authentication method and no user name are specified.
-Kerberos accepts domain user names, but not local user names.
-The Service Principal Name (SPN) for the remote computer name and port does not exist.
-The client and remote computers are in different domains and there is no trust between the two domains.
After checking for the above issues, try the following:
-Check the Event Viewer for events related to authentication.
-Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.
Note that computers in the TrustedHosts list might not be authenticated.
-For more information about WinRM configuration, run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.

Searching for the error returned a lot of similar results including:

Deleting SPNs

https://social.technet.microsoft.com/Forums/windows/en-US/a4c5c787-ea65-4150-8d16-2a19c569a589/enterpssession-winrm-cannot-process-the-request-kerberos-authentication-error-0x80090322?forum=winserverpowershell

A conflict between ports

http://sharepoint-community.net/profiles/blogs/powershell-remoting-error

Sites to help with troubleshooting

https://blogs.technet.microsoft.com/jonjor/2009/01/09/winrm-windows-remote-management-troubleshooting/

None of them fixed the problem. I managed to get Remote Powershell to work by setting the SPN to specific ports but this then broke Reporting Services so I reverted my changes.

You know you are struggling when all the searches you do yield results you have already read, but in one web site that didn’t appear to be relevant I found this little nugget of information

Remote PowerShell requires port 80 to be available on the Default Web Site”

https://blogs.technet.microsoft.com/exchange/2010/02/04/troubleshooting-exchange-2010-management-tools-startup-issues/

Looking at my web server there was no default website and nothing using port 80 so I added one and remote PowerShell started to work!

I can now deploy from the cloud on to my back end servers without opening any firewall portsJ

Why do I need Pre and Post Approval Steps in my Release Pipeline?

TFS Release Manager and Octopus Deploy, both support the concept of approval steps, but why do you need both pre-release and post-release approval steps. When I first started to look at the automated release tools such as TFS release manager I could understand the reason behind the pre-release approval. This step is your quality gate and adds some controls into your process. When creating your release pipeline you will setup a number of environments (e.g. Test, UAT, Pre production, Production) and at each stage you can make a different set of people responsible for allowing the deployment onto each environment.

When the developers have finished their new piece of code and it has been tested in their own environments, they will want to get it onto the servers so that the testers can test it in a more formal way. Currently this may involve the developer talking to the testers and the developers hanging around whilst the testers finish what they are doing so that they can free up the servers ready for the deploy. The developers will then deploy the software to the test environment, hopefully using some method of automation. With release manager, the developers can kick of an automated build and when it is completed it can then automatically create a release ready for deployment. This release can be configured to automatically deploy to the target environment.

The developer can force the software on to the test environment without the tester being ready. The testers for example may be finishing off testing a previous release and need some time before they are ready to accept the software. They may also have a set of criteria that need to be met before they will accept the software onto their environment.

Adding a pre-release approval step that allows the test team to “Accept” the release gives control to the test team and allows them to accept and indeed reject a release. This “Pause” the process allows the testers to check that all the developer quality gates have been met and therefore allow them push back to the developers if they are not happy. As the deployments can be automated, the testers can use the approval process to also control when the new software is to be deployed into their environment, allowing them to complete their current set of tests first. It also frees up the developer, so that they are not hanging around waiting to deploy. Similarly, moving on to UAT, Pre-Prod or Production, a pre-release approval step can be configured with different approvers who then become the gate keepers for each environment.

A pre-release approval step makes a lot of sense and provides order and control to a process and remove a lot of user error from the process.

So what about a post-release approval step, why would you need one? It wasn’t until I started to use TFS Release manager to automatically deploy my applications to Azure Websites where the need to have pre-release approval process became clear. Once I had released my software onto the test environment, I needed a mechanism to allow the testers to be able to reject a release if it failed testing for whatever reason. The post release approval step allowed them to have this power. By adding both a pre and post release approval step for each environment allowed the environment owner to accept the release into the environment when they are ready for it and when they are satisfied that the developers have done their jobs correctly. They can also control when it is ready to move to the next stage in the process. If after completing testing the software is ready to release to UAT then the tester can approve the release which pushes it to the next environment. If the tester is not happy with the release then they can reject it and the release does not move forwards. The tester can comment on the reason for rejection and the release will show red for failure on the dash board. Adding pre and post approval steps to each environment moves the control of software releases onto each environment to a group of people who are responsible for what happens on each.

Using these approval steps can also act as a sanity check to ensure that software releases do not accidentally get pushed onto an environment if someone kicks of the wrong build for example.

I’ve created a release pipeline for my applications which use pre and post approval steps for releases to Test and UAT, I don’t’ have a pre-production environment, but production utilises the staging slots feature of Azure Websites to allow me to deploy the release to staging prior to actually going live. The production environment only has a pre-release approval step, but as it is only going to staging, there is an additional safe guard to allow the coordinated live release when the business is ready.

Both Pre and Post release approval steps provide a useful feature to put the control of the release with the teams that are responsible for each environment. The outcome of each approval process can be visible, which also highlights if and when there are issues with the quality of the software being released.

System.Web.Mvc not found after deploying to Azure Web Apps using Release Manager

I’m currently evaluating Release Manager in Visual Studio Team Services and I am using it to deploy website to Azure Web Apps. I recently tried to deploy an Asp.Net MVC 4 application and ran into some issues.

I’ve created a build that packages and zips up my web application which runs successfully.I’ve linked a Release pipeline to this build and I can deploy to my test Azure site without any errors, but when I try and run the web application I get the following error:

Could not load file or assembly 'System.Web.Mvc, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

image

I’m using Visual Studio 2013 with MVC as a nuget package.Looking at the properties of System.Web.Mvc I can see that it is set to Copy Local = True

image

I tried a few different things to try to get the assembly to be copied like redoing the nuget install and eventually I toggled the Copy Local to False, saved the project file and then set it back to true. When I looked at the diff of the project file I found an additional property

image

This seems to fix the build. When I checked this in and rebuild, System.Web.Mvc now appears in the zip file. The build was then release to Azure and the web app worked correctly.