Case Study – Upgrading Sitecore 9.0.1 to 9.3 on an enterprise site

A few months ago a client asked if I could upgrade their Sitecore 9.0 Update 1 real estate to the latest version 9.3. Sitecore upgrades always require a fair amount of due diligence, planning and attention to detail in execution. Unfortunately talk to anyone who has carried out a Sitecore upgrade before (especially one that skips intermediate releases) and they will tell you the process is often not as simple as it sounds. If you or your company have the budget I would definitely recommend employing someone who has planned and executed an upgrade before, as experience can count for a lot of time saved in planning, efficiency and debugging issues later on. 

In this blog I won’t be discussing every step necessary to upgrade Sitecore but rather pointing out problem areas and things to consider before you start. The Sitecore upgrade guide is very comprehensive and I heartily recommend reading it thoroughly beforehand, as well as keeping it close to hand during the upgrade.

As always in every upgrade I offered my advice and expertise, but the client had the final say over how they wanted the solution to look/behave and the way the platform was to be distributed. Sounds fair enough to me as I won’t be supporting it ?.

Obviously a lot of this post will contain information pertaining to the client’s individual circumstances and setup, but you may find a useful nugget or two to help with your own upgrade ?

The planning stage

Scope of the upgrade

The Solution Architect’s preferred approach for this upgrade was as follows:

• Employ Docker to host the Sitecore ancilliary services in Linux containers.
• Move the Sitecore platform out from custom Nuget packages and employ a custom Powershell script to install it.
• Deprioritise any issues that go beyond the ability to run up the site and load test it.
• Minimal impact approach – upgrade as few components as possible in the first iteration. If it doesn’t need to change then don’t change it. Because of the size of the solution there are a large number of opportunities to improve the codebase but due to the lack of resources, this would prolong the project and possibly lose focus given I am only the only developer on the project.
• Keep in the in-webroot approach as is – Again, change as little as possible to reach a working solution. Since the solution works inside the hosting folder keep this approach for now – the merits and drawbacks of that approach are a debate for another day.
• Time and resources allocated to the upgrade project are for 1.5 people’s time, if that.
• On going development from 10 other teams means the work will need to be integrated in source control with care.

Success criteria

• Make it as easy as possible for the 50 or so developers working day to day on the solution, to continue their work seamlessly
• Improve the laborious process new developers to the team have to endure when setting up the solution using a set of document manual steps.
• Ensure the site was at least as performant on version 9.3 as it was pre-upgrade on 9.0.1.
• Gather some metrics to gauge if possible whether 9.3 could allow analytics data collection to be switched on – currently due to the vast amount of traffic on the production site, the Sitecore 9.0.1 site struggles to run with analytics turned on.

The existing Sitecore real estate

This Sitecore solution in question was fairly large and established and ran the Visual Studio project inside the webroot. I’m not fully aware of the historical reasons for this, but I’m speculating that they preferred changes made to be reflected instantly in the website. I believe they also disliked the fact that the publish approach can leave a legacy of old files hanging about e.g. renamed assemblies or configuration files. Obviously this is solvable but the availability of Sitecore SMEs was limited and as in many businesses, most of the teams’ efforts were focused on revenue generating tasks, sometimes at the expense of Sitecore best practice.

For the current 9.0 setup the client used custom Nuget packages to distribute the Sitecore platform – that is, the Sitecore DLLs and files all wrapped up in Nuget packages. A Powershell script inside the Nuget package was employed to drop files in the correct location on installation. However these Powershell scripts are no longer executed on a Nuget install with PackageReference and are not supported going forward.The Solution Architect’s plan therefore was to reference Sitecore’s own set of Nuget packages going forward.

The proposed platform provisioning and developer experience

The proposal from the SA was for developers to deploy the Sitecore platform into their working folder via a custom Powershell script and check out the Sitecore solution from source control, allowing them to continue to work on the solution in the same manner they were accustomed to.

The Sitecore Experience Platform has a vast number of logical roles now in version 9.3 and as part of the upgrade the SA prepared the required Sitecore ancilliary services in Linux Docker containers including the following:

• xConnect Collection
• xConnect Collection Search
• xConnect Search Indexer
• xDB Processing
• xDB Reporting
• Marketing Automation Engine
• Marketing Automation Operations
• Marketing Automation Reporting
• Reference Data
• Solr

These ancillary containers would be installed on Developer’s machine by Powershell scripts as part of a workstation installation script.

Taking this approach was a step on the journey toward bringing the developers’ workstations closer to the production real estate and makes it much easier to setup a new developer workstation. This would provide a repeatable set of steps that the developers could use to deck their local installations and get back to a solid starting point. A point sorely lacking for years with the current manual set up of the Sitecore installation detailed in a list of workstation set up steps.

Encapsulating the Sitecore services in Linux containers was fantastic though I had my misgivings about the approach to platforming developer machines primarily due to the fact in my experience a Visual studio Clean command will clear out the bin folder of not just solution binaries but the platform binaries as well. This would require a redeploy of the Sitecore binaries. Gitignore also needs to be set perfectly to ignore all the platform files. I would have preferred to use Docker containers for Sitecore itself but unfortunately there was some resistance from the client to this approach and Windows/Linux containers may not have played nicely together.

My preferred approach is as per Sitecore’s guidance which is for developers to publish their changes to the site as and when necessary. This avoids unnecessary app pool recycles when saving code files in the solution. It saves time spent waiting around waiting for Sitecore to reload all its configuration. And when you multiply this time up by 50 developers, this can be a lot of wasted time! This approach often also more closely resembles the CI CD environment which means you can fail faster while still in development by spotting any potential issues earlier. Keeping separation between the files of your solution and the Sitecore platform files also helps to make upgrades easier by maintaining a clear boundary between the installation and the solution files.

I was surprised moving to the Publish outside of the webroot approach was not made a higher priority before now as anecdotally, the amount of complaining I heard from talking to developers about Sitecore’s slow startup time meant a lot of time was being wasted on the shop floor in app pool restarts. However in my experience in larger firms there’s all to often a higher priority / not enough time / not enough money to address such concerns…

The upgrade

So after establishing a plan of action it was time to crack on with the upgrade!

Undoing anomalies

As anyone that has done a Sitecore upgrade before can tell you, there are often many unexpected time sinks that crop up when you get stuck into the detail. To help mitigate this I dug around looking for any unwanted anomalies in the solution or pain points that might crop up later. One such anomaly that cropped up (probably as a result of running the solution in the webroot) was that the platform Sitecore.config file had been checked into source control at some point in the past and had been modified which is obviously a no-no. So I set about comparing this to the stock config file and moving any alterations out to patch files. The Sitecore.config file could then be deleted from source control and remain as part of the Platform installation

Upgrading the .NET Framework version

The first step was to upgrade the .NET Framework to support Sitecore 9.3. The new version supports .NET 4.7 upward as shown in the Compatibility table so I decided to go with .NET Framework 4.8. However with almost 50 projects in the solution this was tedious prospect.

Step up the Target Framework Migrator. This little gem takes the time and hassle out of upgrading each project by hand by going into the project properties and selecting the new version. I then went through and carried out a global find and replace on any Assembly bindings making reference to .NET 4.6.2.

Support packages

This client’s way of dealing with Sitecore Support patches was to wrap them up into custom Nuget packages and install them in the main Sitecore web project. Since 9.3 included all of these legacy fixes, they were now obsolete and we could just safely remove these from the project.

Installing the new Sitecore 9.3 Nuget packages

By now attempting to build the solutions resulted in a few errors ? :

Since Sitecore no longer publish the NoReferences nuget packages –   if you install the 9.3 packages you will get a lot of dependencies installed. Now with some projects in your solution e.g. a Foundation project, you may not want or need to bring in all and sundry. The way around this is to use the Dependency Behaviour option of the Nuget Package Manager. If we set the dependency behaviour to “IgnoreDependencies” and install the package, the dependencies will not be installed.
The Package Manager Console command to use is:

Install-Package -Id <package name> -IgnoreDependencies -ProjectName <project name>

Then it is a case of installing the packages you need, until your compiler errors start to diminish. ?

Dependent Nuget package version upgrades

Some of the Nuget packages in use in the solution were fairly old and the Sitecore 9.3 binaries were compiled against later versions than we were currently using. Therefore many of these needed to be upgraded to be at least the same version. Cases in point were:

• Castle Windsor which we upgraded to version 4.0.0
• Castle Core upgraded to 4.2.1
• Microsoft.Aspnet.MVC upgraded to 5.2.4

Again I needed to globally search the assembly bindings here to make sure no bindings were redirecting to old versions of these binaries.

Breaking API changes

Something to be aware of when upgrading Sitecore is that there are often breaking changes to public APIs even between point releases which can make the upgrade a challenge. Usually these methods are marked as Obsolete in the previous version which should flag a compiler warning, however this doesn’t help you much if you are skipping point releases during an upgrade.
A few breaking changes I experienced to the Sitecore API that have occurred since version 9.01:

• The Sitecore.Jobs.Job and Sitecore.Jobs.JobOptions classes have been removed in Sitecore 9.2. In their place you now have BaseJob and BaseJobOptions abstractions with DefaultJob and DefaultJobOptions concrete classes
• We had some custom code that wrapped the static class Sitecore.Job.JobManager for unit testing purposes. This code required updating to accommodate the the changes to BaseJob and BaseJobOptions above to match the new method signatures. This will probably be a common problem across the board as Sitecore refactors more and more of the codebase as they introduce more abstractions in favour of static classes. Ideally we would rewrite the unit testing classes to use the abstractions if possible at this point, but we don’t always have the luxury of time or budget to do so. 
• Many obsolete methods have been removed such as the static Sitecore.Caching.CacheManager.FindCacheByName(string name) and Sitecore.Eventing.EventManager.QueueEvent<TEvent>() methods. This meant I ended up having to rewrite someone else’s code which inevitably didn’t have any unit tests. Not fun. ?
• The Sitecore.Pipelines.HttpRequest.HttpRequestArgs.Context property has been removed in favour of HttpRequestArgs.HttpContext
• The Solr ServiceBaseAddress configuration setting has been removed as of version 9.0 Update 2 and is now set as a connection string. Therefore any patch file present to override the setting ContentSearch.Solr.ServiceBaseAddress in  Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config can be deleted. The new connectionstring can be added to your connectionstring.config file like so: 

<add name="solr.search" connectionString="https://localhost:8994/solr" />

Pipeline processor adjustments

If you have custom pipeline processors that override some of the defaults in Sitecore you might be faced with some compiler errors; Since some of the built in base classes no longer have parameterless constructors you may have to call the base class with parameters from your custom processor classes or your code will no longer compile.
In addition you may get the following error at runtime and you will need to adjust the configuration:

Could not create instance of type: MyProcessor. No matching constructor was found.

Some of the custom processors that were patched into pipelines such as <mvc.getPageItem> and <mvc.renderRendering> now require an extra attribute of resolve=”true” adding to the XML config to allow the dependency injection mechanism in Sitecore to work in the base classes: 
e.g. For a custom method for Generating Cache keys the config patching had to change from:

<mvc.renderRendering>
  <processor type="MyProcessor, MyDLL"patch:instead="processor[@type='Sitecore.Mvc.Pipelines.Response.RenderRendering.GenerateCacheKey, Sitecore.Mvc']"/>
</mvc.renderRendering>

to:

<mvc.renderRendering>
  <processor type="MyProcessor, MyDLL"patch:instead="processor[@type='Sitecore.Mvc.Pipelines.Response.RenderRendering.GenerateCacheKey, Sitecore.Mvc']" resolve="true" />
</mvc.renderRendering>

Get the solution compiling

My next aim was to get the solution compiling with the new DLLs. I find this is a tedious but necessary cycle of compiling the project, and watching the the error count slowly reduce as you tick off problem by problem. When this step is complete you are well on your way and your solution may run to some extent but  don’t get cracking out the champagne yet theres plenty left to do.

Caching changes

There has been a change in version 9.3 which automatically clears the rendering cache for sites which use it on publish, and does not require additional configuration as long as cacheHtml=true is set on your site in the Sites configuration. The publish:end event handler is configured like this in a standard installation:

<event name="publish:end">
  <handler type="Sitecore.Publishing.SmartHtmlCacheClearer, Sitecore.Kernel" method="ClearCache" resolve="true" />
  ...
</event>

If you don’t want the cache for a site to be cleared when you publish you can add the preventHtmlCacheClear attribute to the site definition like this:

<site name="custom_website" cacheHtml="true" preventHtmlCacheClear="true" … />

LinkManager changes

There have been changes in 9.3 as to how the LinkManager operates with  Sitecore.Links.UrlOptions becoming obsolete and new configuration added. I wrote about this but since then Volodymyr Hil has summed up the changes far better than I could have in his blog here.

Upgrading Glass Mapper

If you’re using Glass like this client does, it will require an update to version 5. Glass has many changes in v5 including a new way of accessing content from Sitecore using the new IMvcContext, IRequestContext and IWebFormsContext abstractions.I had to make changes in this solution around IsLazy attributes on model classes as Glass lazy loads by default now.

I won’t detail all the changes as they are well documented:

• There is guidance from Mike on upgrading from version 4 here.  
• Shu Jackson has also summarised the changes very nicely here. 
• Thomas Paterno has also posted his findings from such an upgrade as is definitely worth a read before you begin.

What I will say is if time and budget is against you like it was for me being the only developer assigned to this upgrade for a few weeks, you can in fact stick with the old ISitecoreContext class. It will still work, despite being made obsolete in favour of the new contexts such as IMvcContext. Obviously this is not recommended as it will almost certainly be removed in the next version and you are just kicking the can down the road. But in my case the time or budget did not allow me to rewrite every class in the system to use the new abstractions.

The main problem I encountered after upgrading Glass was a bunch of run time errors due to a lack of virtual property modifiers in model classes. It appears that in old versions of Glass Mapper, model properties still map correctly even without the virtual modifier, but not any longer. This only manifests itself when the model is hydrated (ie. at runtime) so its tricky to locate if you have a lot of models. Which this client does! I found as many as I could by carrying out all the common journeys on the site. I’m sure there is a clever reg ex you could use to do a global search in Visual Studio. Alternatively you could harness an automated tool to send an HTTP request to every page of the site looking for 500 response errors.

Upgrading Unicorn

Unicorn required upgrading as 4.1.1 is the minimum version that supports Sitecore 9.3. This was very easy to upgrade as it formed part of a Serialisation Foundation module.

Analyse Web.config for changes

It is a good idea when upgrading to compare the current web.config to a standard OOTB web config for the current version. So I did this for v9.01 and noted that there were many customisations present. I then compared new 9.3 web.config with the 9.0.1 web.config and brought across the necessary changes.e.g. in the 9.1 release a change was made to the <authentication> node for the Identity Server changes, and so that needed to be carefully merged in as:

<authentication mode="None"></authentication>

Upgrade the Databases

I recommend taking a copy of your current Sitecore databases and running the appropriate SQL scripts on the databases as specified in the installation guide. This is usually one of the least problematic steps of an upgrade. But don’t get complacent yet ?

Install the Sitecore upgrade package

Once the site will build and you can run it up without error you can install the content update package provided in the platform installation files. In my case this failed to complete without error as the base templates had been customised.

This is the kind of problem that is hard to anticipate in any Sitecore upgrade and is a reason contingency time needs to be allocated for unforeseen events and hidden issues. So at this point I had to branch off and investigate these templates and find out what has been changed. Of course there was no documentation present on why these changes had been made so it required deducing carefully whether the modifications were still needed.

Checking the assembly versions

Sitecore provides an Assembly list (See Release information -> Assembly list) of all the Sitecore binaries and their respective versions. I compared each binary to the Sitecore Assembly list to ensure the versions were correct.
There is a small mistake in this document – it contains 2 rows for PDFSharp dlls that have an erroneous “resource” string in the filename so just ignore that part:

Run it up!

With any luck, by now your solution will be working enough to navigate around the system and note down any problems that need resolving.

Check the Error logs

Once the site is up and running it’s time to delve straight into the Sitecore logs to clear up any errors that are being logged. If you’re not using it already Sitecore Log Analyzer is an old but gold tool for analysing the Sitecore error logs.

Solr Indexing Issue #1 – Missing index errors

If you’re not using SIF to install Solr (which you most likely won’t on a larger team – we’re using a Docker container here, remember) then the standard Sitecore config files in v9.3 will leave you with missing index errors in the log due to their configuration out of the box for the following indexes:

• sitecore_testing_index
• sitecore_suggested_test_index

The core name OOTB for sitecore_testing_index is set as follows:

<param desc="core">Sitecore-sitecore_testing_index<param>

The core name OOTB for sitecore_suggested_text_index is as follows:

<param desc="core">Sitecore-sitecore_suggested_test_index</param>

These names do not match the core names in SOLR so I had to create a patch for this to use the index id attribute as the core name as per the other indexes:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">  
  <sitecore role:require="Standalone or ContentManagement" search:require="Solr">    
    <contentSearch>      
      <configuration>        
        <indexes>          
          <!-- The OOTB Sitecore config has the incorrect core name set for the sitecore_testing_index and sitecore_suggested_test_index -->          
          <index id="sitecore_testing_index">            
            <param desc="core">$(id)</param>          
          </index>          
          <index id="sitecore_suggested_test_index">            
            <param desc="core">$(id)</param>          
          </index>        
        </indexes>      
      </configuration>    
    </contentSearch>  
  </sitecore>
</configuration>

This eliminated the index errors as the names now matched the SOLR cores.

Solr indexing issue #2 – ComputedIndexField error

SOLR was failing to index some items as it was generating an error when trying to return a URL as the result of a Computed Field:

207140 15:41:35 WARN  Could not compute value for ComputedIndexField: item_url Exception: System.NullReferenceExceptionMessage: Object reference not set to an instance of an object.Source: Sitecore.Kernel   at Sitecore.Links.LinkProvider.GetDefaultUrlBuilderOptions()   at CustomLinkProvider.GetItemUrl(Item item)

The Client were using a Link Provider switcher which chooses a custom LinkProvider class at run time to generate URLs for Sitecore items. This is a very common approach used by many users of Sitecore (SXA has made this much easier). Many implementations generate URLs differently depending on the context site. This one generated different URLs based on the item’s location within the content tree. 
As discussed earlier there have been changes to the LinkManager and the LinkProvider base class has changed since 9.0.1 internally. The LinkProvider.GetItemUrl method now uses an instance of the Sitecore.Links.UrlBuilders.ItemUrlBuilder class to build the URL. Internally the LinkProvider class has a new public Initialise method which is used to create an instance of the ItemUrlBuilder populated with the configuration from the <links><itemUrlBuilder> node in the Sitecore config. This new Intialize() method usually gets called by Sitecore via the LinkManager but not in our case. Unfortunately in this computed field the code was manually instantiating a copy of one of the custom provider implementations. This means we get errors when instantiating it in such a way.
There are 2 approaches to resolve this:

A light touch approach is to simply initalise the LinkProvider (lame but the code works):

var customLinkProvider = new CustomLinkProvider();
customLinkProvider.Initialize("customConfig",newNameValueCollection());
return customLinkProvider.GetItemUrl(item);

Or go via the standard LinkManager which does initialise the ItemUrlBuilder correctly:

LinkManager.GetItemUrl(item);

After a change like this I always ensure I compare the generated URLs generate pre and post upgrade to ensure they match.

Solr Indexing issue #3 – Hexadecimal value is an invalid character
Another series of errors encountered in the Crawling log when reindexing the master core were:

Exception: System.ArgumentException
Message: '.', hexadecimal value 0x00, is an invalid character.
Message: '', hexadecimal value 0x01, is an invalid character
Message: '', hexadecimal value 0x08, is an invalid character.
Message: '', hexadecimal value 0x1D, is an invalid character.
Message: '', hexadecimal value 0x1F, is an invalid character.

There is some detailed discussion about this issue on the Stack Overflow post here: https://sitecore.stackexchange.com/questions/18832/wildly-inconsistent-index-data-after-rebuilds

Using this stack exchange post as a guide – I ran the following SQL to identify problem records: 

SELECT * FROM (
    SELECT ItemId, FieldId , Value FROM [dbo].[SharedFields] 
    UNION
    SELECT ItemId, FieldId , Value FROM [dbo].[UnversionedFields]
    UNION
    SELECT ItemId, FieldId , Value FROM [dbo].[VersionedFields]
  ) A
WHERE Value Like '%' + CHAR(0x00) + '%'
WHERE Value Like '%' + CHAR(0x01) + '%'
OR Value Like '%' + CHAR(0x08) + '%'
OR Value Like '%' + CHAR(0x29) + '%'                                                                           
OR Value Like '%' + CHAR(0x31) + '%'

However that brought back over half a million records.

I tried disabling the SOLR indexing for some file types by removing the <extension>pdf</extension> and <extension>doc</extension> inclusions inside the <mediaindexing> configuration in Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config but still experienced the errors when indexing.

Ultimately Sitecore Support recommended patching out all the <extension> inclusions which resolved the error.

And finally…

Unfortunately I didn’t get to the see this project through to completion due to Covid-19 bringing the contract to an end but hopefully there are one or two things in here that can help you should you be about to attempt a similar upgrade. As always with Sitecore YMMV, so do your own due diligence.

Finally to leave you with a couple of tips:

• Have a bottle of red wine close to hand
• Keep the Sitecore upgrade guide close to hand – multiscreen setups are very useful here! 

• Occasionally close Visual Studio and reopen it to get rid of transient errors where the IDE gets itself in a bit of a twist.
• Sitecore Slack is a useful resource to have one hand, feel free to message me @sitecorium if you get stuck, I’ll do my best to help.

Good luck!