From time to time I have to do some Solr configuration for a client and tend to experience a couple of obscure errors that I stumble into, fix, and then proceed to forget about until the next time. Well it’s time to write these tidbits down for future reference….
Often you will complete your new Solr core setup and want to do an initial build on the index via the Index Manager in the Sitecore Control Panel. Occasionally on triggering a rebuild, the Indexing Manager will proudly report that “The search indexes have been rebuilt.” despite the result below saying something like:
There are large performance gains to be had in Sitecore by employing judicious use of caching. As Sitecore developers we should try and be mindful, that performance should be a consideration from the start when building a component. Caching is an important part of ensuring we build performant controls, yet to some caching is a complicated black art best avoided. This needn’t be the case – it is actually very straightforward to take advantage of Sitecore’s built in caching options.
What is the HTML cache?
The HTML Cache in Sitecore, (sometimes referred to as the output or web cache) is a mechanism that stores the rendered output of a component in a key based table. Every time Sitecore renders a component on the page, it checks the HTML cache to ascertain if the cached output is present. If it is, it is retrieved and rendered immediately otherwise it goes through the normal process of running the rendering pipelines and constructing the output. If you’re not employing caching for your site, you’re leaving a lot of performance on the table.
How do I use it?
The HTML Cache is employed by selecting appropriate caching options on the Rendering itself or on the Presentation details for a component.
To enable caching select the Cacheable checkbox. We must then decide how we want the cache to be created. This will vary depending on the type of component and it’s use case. Varying the cache by a given parameter means we would like Sitecore to create an individual cache key for every variation of that parameter. e.g. if we Vary by Data[source] we are telling Sitecore want to store a cached version of the component output for every different data source it encounters when rendering.
The available options that come with Sitecore are:
Clear on Index Update: Indicates that the cache should be cleared when the index is updated – if your code depends on an index, selecting this will help prevent any race conditions occurring between HTML caching and index update operations.
Vary By Data: Generate a cached version per data source
Vary By Device: Generate a cached version per device
Vary By Login: Generate a cached version for authenticated users and one for unauthenticated users
Vary By Parm: Generate a cached version per rendering parameter
Vary By Query String: Generate a cached version for each unique combination of querystring parameter
Vary by User: Generate a cached version per authenticated user
How does it work?
A component is rendered by Sitecore and if the caching options are set, a cache key is constructed. The name of the key corresponds to the caching options selected. See below for the format of the cache keys from the Sitecore documentation:
Vary by Data –The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#data:[Path of the Datasource Item]
Vary by Device -The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#dev:Responsive
Vary by Login – The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#login:False
Vary by Parameters – The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#parm:[Key=Value]
Vary by Query String –The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#qs:[request.QueryString]
Vary by User – The cache key is: controller::[Controller]#[Controller Action]#lang:[Language Culture Code]#user:[user]
Once this key has been generated, the rendered output is stored against this key. The cache key is checked the next time a control is rendered (by a pipeline in the Sitecore.MVC DLL – Sitecore.Mvc.Pipelines.Response.RenderRendering.RenderFromCache). If it is present, the stored output is rendered rather than executing the relatively slow process of constructing the model and rendering the output again.
Note: Bear in mind if a cached rendering contains placeholders which contain other components or statically bound renderings when these are also cached with the parent.
Can I override a rendering’s default cache settings?
Default caching options can be overridden by changing the settings on the presentation details for a component.
How can I check if my rendering is cached?
There is no built in tool with Sitecore that allows you to easily inspect the cache keys that have been constructed in the HTML cache. To do this you need to write your own inspection page or tool. Alternatively you could install Sitecore Rocks – the tool that integrates with Visual Studio which allows you to view the cache keys. To do this Manage your site from within the Visual Studio Sitecore Explorer and select the caches tab. Scroll down to “yoursitename[html]” and double click it. The Explore cache window should appear and you should then be able to see the cache keys currently in the cache e.g.:
Cache tuning is a fine art and you will want to measure how your changes are affecting the rendering performance. To do this there are a few options. The first is to use the Sitecore debugger which is accessed through the Experience Editor -> Experience Tab -> Other button -> Debug or alterntively add “?sc_debug=1” to the current page URL in a browser.This will add very useful detailed information below the page such as a profile of the full rendering pipeline detailing timings, cache hits and item reads as well as tracing information showing each step Sitecore pipeline executed when rendering out the page. Analysing the timings here and looking at the Hotspots sections can help you target problem renderings and ensure you optimise the slowest renderings first.Another alternative is a tool like JMeter to run load tests against a URL and measuring the before and after results.
This just scratches the surface of Sitecore’s caching capabilities, but hopefully it will have given a small insight on how to get started with caching your Sitecore renderings.
Being a contractor I tend to move “jobs” more frequently than if I were a permanent member of staff. I therefore get to work with a huge range of people across the development spectrum. At most clients I tend to meet at least one person I would call an all star. These people are exceptionally gifted at their job, whether that entails architecture, development, business knowledge or other areas. I sometimes day dream about what it would be like to cherry pick these outstanding people from each client and build out our own consultancy. Released of company bureaucracy, lazy colleagues, obstructive management and other drags, the sheer amount of work they could produce with their talents when combined would be staggering. Of course the practicalities of doing so would rarely work; Some people are happy working for their employers year on year with no stress and would not relish the prospect of additional risk. Some contractors would get bored and want to push boundaries on their own. Some people enjoy being the go to person in a company. And unfortunately I’ve only ever met one “salesman” (my first real boss who is all but retired nowadays) with such a deep command of the technical and the selling side of business, that winning clients’ business would be far easier said than done. Still it’s nice to dream…
Despite working with Sitecore in depth for a couple of years I had never had the opportunity to delve into the Marketing and Analytics parts of the XP. The perfect opportunity came up recently when the client upgraded their Sitecore 6.6 installation to 8.2. The Analytics functions were not displaying data through the Apps on the Launchpad (Experience Analytics, Experience Profile etc) so I set to task.
When dealing with Sitecore in general it pays dividends to take a patient, analytical approach. Sitecore produces a lot of documentation on how your installation should be configured, however take this with a big pinch of salt; The documentation tends to fall down slightly when you move into the CM/CD arena. Setting descriptions are often brief with minimal examples so you will more than often need to turn to the community to help. Stack Overflow and Slack can be invaluable tools picking up where the Sitecore docs left off.
To briefly explain the production environment I was dealing with, the client had:
One CM server also doing processing and reporting
6 load balanced CD servers
Sitecore 8.2 Update 4 (170614)
To diagnose issues with analytics it helps to have an understanding of the fundamentals of how it works. In simple terms:
A user browses the site
Data about the user’s habits and preferences are stored and information that might identify them is stored in session until the user has ended their browsing session
This Analytics data is then stored in the Collection database, that is, the MongoDB database specified in your Sitecore connection strings.
The Sitecore Processing server (the CM server in our case) uses Aggregators to then curate the Collection data and write it to the Reporting database at predefined intervals. The default interval is 15 seconds but this can be adjusted in the configuration. The sitecore_analytics_index is also updated.
The Reporting service then queries the reporting database to display data in applets such as Experience Analytics
The first issue to investigate were the errors being reported in the Analytics interface
Problem 1: Errors in Experience Analytics interface
In our case the following files were enabled on CM which should be disabled according to the documentation from Sitecore for a CM+processing server:
The accompanying note from the Sitecore documentation (below) took a few reads through before it was clear in my mind!
Note All Remote.Client.config files are disabled by default. You should enable these files if you have a reporting server in another location. Disable these files on a content delivery server and enable them on a content management server.
Advice from Sitecore
Since our config is set via Octopus deploy, it was a case of correcting any mismatches in the config enabled/disabled states inside the package deployment steps.
Problem 2: No data was appearing at all in the Experience Analytics
There can be many reasons why no data is being displayed but there are a few common avenues you can use for investigating Analytics / Reporting issues:
Enable detailed logging and check the logs
Often the first port of call in diagnosing issues is to check the Sitecore logs. Log viewers such as the Sitecore Instance Manager (SIM) Dynamic Log Viewer or the Log Analyzer tool make light work of analysing any errors. To enable debug level logging for analytics change the configuration setting as per below:
Check the Sitecore configuration
With so many working parts and the multiple ways in which your production system can be deployed, its important to get the Sitecore configuration correct. The official documents for configuration at the time of writing are located below:
Each document provides a link to a spreadsheet detailing the list of all the configuration files that should be enabled or disabled for each version of Sitecore 8.2 e.g. Update 1 – Update 6 at the time of writing. Since in our case we have one CM server doing the processing and reporting we use the CM+Processing columns and the Content Delivery columns.
This in itself is confusing as their is no mention of reporting in that column. Sitecore don’t provide an explanation of exactly what each column means which is a cause for confusion so I am left to infer the following:
Content Delivery – Each customer facing CD server
Content Management – A Content Management server only
Processing – A Processing server only
CM + Processing – A combined CM, Processing AND Reporting server (Reporting is not specified in the column)
Reporting – A Reporting server only
It is important to get this right or it can create spurious errors that are hard to track down. However as we will read later, sometimes Sitecore can get this information wrong!
Shorten the feedback loop
To aid testing and shorten the time it takes to analyse and debug Analytics issues you can create a quick and dirty aspx page that will abandon the session. Drop the following code into a new file with the aspx extension and drop it temporarily into the site. e.g.
Navigating to this page when testing should flush the data from the current session into Mongo DB. The alternative is to reduce the session timeout in Web.config to a lower value to avoid waiting around for long period of time during testing:
Now the user’s session has ended, data should be written to the Interactions collection in MongoDB. To check this we can use a tool such as RoboMongo to connect to the Mongo instance in question and browse the data. By issuing a query we can check that site interactions are getting:
In our case the interactions returned confirm that data are indeed being written correctly. If there are no interactions with a datetime of around the time you killed the session, it suggests this part of the process is not working. With the collection database checked it’s time to move on to the next step in the process: the SQL Server Reporting Database.
Checking the Reporting DB
By connecting to the Reporting database in SQL Server using the analytics connection string, you can check whether the data are being aggregated by runing a query:
SELECT TOP 1000 [SegmentRecordId], [Visits], [Value], [Conversions], [TimeOnSite], [Pageviews], [SegmentId], [Date], [DimensionKey] FROM [<AnalyticsDatabaseName>].[dbo].[ReportDataView] ORDER BY [Date] DESC
In our case again the data itself was being aggregated correctly to the SQL database but if yours isn’t try the steps detailed here.
Experience Analytics Caching
After the aggregation process completes, the data should be available in Experience Analytics. However in my experience the page is often cached which means closing the browser window and logging into Sitecore again to refresh the data. To combat this the cache can be disabled by altering the cache settings in Sitecore.ExperienceAnalytics.WebAPI.config as this Stack overflow post by Jacob Neilsen details.
This was enough to get our instance of Experience Analytics displaying data.
Problem 3: Experience Profile not logging contacts or Empty
This issue of no Contacts appearing in the Experience Profile is quite often accompanied by many errors in the log such as:
ERROR Error during aggregation.Exception: System.NullReferenceExceptionMessage: Object reference not set to an instance of an object.
ERROR General error when submitting contact.Exception: System.NotImplementedExceptionMessage: The current data adapter provider does not support renumbering interactions.Source: Sitecore.Analytics.DataAccess at Sitecore.Analytics.DataAccess.DataAdapterProvider.RenumberInteractions(ID contactId) at Sitecore.Analytics.Data.ContactRepository.ReconcileContact(Contact contact, LeaseOwner owner, TimeSpan duration) at Sitecore.Analytics.Tracking.ContactManager.SubmitContact(Contact contact, ContactSaveOptions options, Contact& realContact)
Checking the Sitecore documentation for configuration maintains that the following files should be disabled for Content Management and enabled for Content Delivery:
However I had proof that following this advice and disabling the above files on CM prevented Contacts being created and in fact no Experience Analytics data appeared. Enabling the files caused the contacts to start getting logged!
In the past I’ve not put much onus on visiting user groups. I once visited a .NET user group for the first time when I lived in Sussex and the experience was heavily tarnished by trying to interact with developers possessing zero social skills or even the ability to converse in basic terms. After talking to a few guys there it felt like talking to a brick wall, they seemed more interested in the free pizza than interacting with the community. After receiving one word answers time and again I quickly got bored, gained very little from the experience and was put off for years. However in the past year I decided to give it another try – I visited a few Sitecore user groups and found them much more useful. Everyone there has been approachable and friendly with few egos despite the breadth of talent that attends. You also tend to see the same faces over and over again and people you’ve worked with before. Maybe its something to do with the small community feel of Sitecore which still remains a fairly specialised skill compared with the vastness of .NET. Traditionally Sitecore documentation is one of the company’s weaker points and lags behind so it helps to see how people are using it in the wild and how they solve real world issues. Meeting the community also helps with networking, and staying up to date with the rapid pace of Sitecore.
To find your nearest User Group, Google is often your best bet, but you can also use the dedicated channel on Sitecore Slack or the Meetup.com website host regular Sitecore events.
Having worked with Sitecore for a couple of years, thrown in at the deep end and self taught – I decided it was time to do the certification and get a firm grounding in the software from scratch. Sitecore 8.2 was the latest version at the time, though I realised later that if I’d waited a month, the Sitecore 9 training would have been available! :doh As of late 2017 there were four eLearning options
Sitecore Developer Foundations
Sitecore Experience Solution Developer
Sitecore Context Marketing Fundamentals
Sitecore Web Experience Management
The first one is free and it made sense for me to take advantage. According to Sitecore: “The free Developer Foundations course teaches students how to create data infrastructure, build editable pages, and publish content. During the course, you will build a basic website using the powerful and easy-to-use Sitecore XP tools. Students also build simple MVC components using a text editor and minimal coding.“ After completing this short course, which is pretty rudimentary, covering templates, publishing and presentation it was time to shell out for the Sitecore Experience Solution Developer course, which at the time of writing was $1500. Although I would heartily recommend the training as it does cover a range of topics, obviously it can never furnish you with a command of the whole of the Sitecore ecosystem – the system is far too broad to condense down. Rather see it as a stepping stone to gaining further knowledge and exploring the system. There is no substitute for working on a Sitecore project at a client or partner.
My own experience with the training was mixed. Although it filled in a lot of gaps where I was hazy, especially on the marketing features, the training materials were littered with errors and mistakes which made reading hard going especially when the example wasn’t working despite following the instructions to the letter.
Despite the problems it was good to see references to real world concerns like the use of Glass, Content Search API, IoC and Helix patterns in the training. The approach to the training is down to the individual. As with any form of study people learn in different ways and what is effective for one person will not work for another. To get the most out of the training I took the following approach – YMMV:
Made my own typed notes throughout each chapter condensing it down into a text file
Wrote down every question and answer at the end of each chapter for revision later
Carried out every practical workbook exercise locally including the optional ones
Despite the course notes saying it will take approximately twelve hours to complete, the above took me around 2 weeks of evenings studying – so probably around 30 hours. I’m not sure if their estimates includes doing the workbook exercises because if it does I find it very optimistic! The exam consists of 70 multiple choice questions but the time period of 2 hours to complete it is more than enough. Although this approach is not necessary to pass the exam bit does give you a better grasp of the system overall. And since in any form of life I can’t approach something half-arsed I had to do it this way 😛 The pay off was a good resulting grade but it was still annoying to get some questions wrong and even more annoying to not find out which ones! I believe the pass mark is 80%, which does allow you to get 14 questions wrong, however you could probably pass the exam without training if you’ve been using Sitecore for any time at all.
As with a lot of closed-book exams it seems more of a test of memory than competency; you would rarely carry out Sitecore work without access to the Internet or other resources. However there is something to be said IMO for committing the basics to memory so you’re not overly dependent on Google. It also means employers/clients can feel comfortable that in taking you on as a certified developer, they can expect a certain level of knowledge for their money.
Here’s looking forward to taking the Sitecore 9 exam just as v10 is released!