Escapades and exploits in the world of Sitecore and .NET

Despite working with Sitecore in depth for a couple of years I had never had the opportunity to delve into the Marketing and Analytics parts of the XP. The perfect opportunity came up recently when the client upgraded their Sitecore 6.6 installation to 8.2. The Analytics functions were not displaying data through the Apps on the Launchpad (Experience Analytics, Experience Profile etc) so I set to task.

When dealing with Sitecore in general it pays dividends to take a patient, analytical approach. Sitecore produces a lot of documentation on how your installation should be configured, however take this with a big pinch of salt; The documentation tends to fall down slightly when you move into the CM/CD arena. Setting descriptions are often brief with minimal examples so you will more than often need to turn to the community to help. Stack Overflow and Slack can be invaluable tools picking up where the Sitecore docs left off.

To briefly explain the production environment I was dealing with, the client had:

• One CM server also doing processing and reporting
• 6 load balanced CD servers
• Sitecore 8.2 Update 4 (170614)

To diagnose issues with analytics it helps to have an understanding of the fundamentals of how it works. In simple terms:

• A user browses the site
• Data about the user’s habits and preferences are stored and information that might identify them is stored in session until the user has ended their browsing session
• This Analytics data is then stored in the Collection database, that is, the MongoDB database specified in your Sitecore connection strings. 
• The Sitecore Processing server (the CM server in our case) uses Aggregators to then curate the Collection data and write it to the Reporting database at predefined intervals. The default interval is 15 seconds but this can be adjusted in the configuration. The sitecore_analytics_index is also updated.
• The Reporting service then queries the reporting database to display data in applets such as Experience Analytics

The first issue to investigate were the errors being reported in the Analytics interface

Problem 1: Errors in Experience Analytics interface

In our case the following files were enabled on CM which should be disabled according to the documentation from Sitecore for a CM+processing server:

Config file name	Type	Search Provider Used	Content Delivery (CD)	Content Management (CM)	Processing	CM + Processing	Reporting
Sitecore.Xdb.Remote.Client.config.disabled	config		Disable	Enable	Disable	Disable	Disable
Sitecore.Xdb.Remote.Client.MarketingAssets.config.disabled	config		Disable	Enable	Disable	Disable	Disable

The accompanying note from the Sitecore documentation (below) took a few reads through before it was clear in my mind!

Note All Remote.Client.config files are disabled by default. You should enable these files if you have a reporting server in another location. Disable these files on a content delivery server and enable them on a content management server. 

Advice from Sitecore

Since our config is set via Octopus deploy, it was a case of correcting any mismatches in the config enabled/disabled states inside the package deployment steps.

Problem Solved!

Problem 2: No data was appearing at all in the Experience Analytics 

There can be many reasons why no data is being displayed but there are a few common avenues you can use for investigating Analytics / Reporting issues:

Enable detailed logging and check the logs

Often the first port of call in diagnosing issues is to check the Sitecore logs. Log viewers such as the Sitecore Instance Manager (SIM) Dynamic Log Viewer or the Log Analyzer tool make light work of analysing any errors. To enable debug level logging for analytics change the configuration setting as per below:

<setting name="Analytics.LogLevel" value="Debug" />

Check the Browser Console for Errors

Occasionally Javascript errors can appear in the browser developer tools that help point you in the right direction.  

Check the Sitecore configuration

With so many working parts and the multiple ways in which your production system can be deployed, its important to get the Sitecore configuration correct. The official documents for configuration at the time of writing are located below:

• Configuring a CD server https://doc.sitecore.net/sitecore_experience_platform/82/setting_up_and_maintaining/xdb/configuring_servers/configure_a_content_delivery_server
• Configuring a CM server https://doc.sitecore.net/sitecore_experience_platform/82/setting_up_and_maintaining/xdb/configuring_servers/configure_a_content_management_server

Each document provides a link to a spreadsheet detailing the list of all the configuration files that should be enabled or disabled for each version of Sitecore 8.2 e.g. Update 1 – Update 6 at the time of writing. Since in our case we have one CM server doing the processing and reporting we use the CM+Processing columns and the Content Delivery columns. 

This in itself is confusing as their is no mention of reporting in that column. Sitecore don’t provide an explanation of exactly what each column means which is a cause for confusion so I am left to infer the following:

• Content Delivery – Each customer facing CD server
• Content Management – A Content Management server only 
• Processing – A Processing server only
• CM + Processing – A combined CM, Processing AND Reporting server (Reporting is not specified in the column)
• Reporting – A Reporting server only

It is important to get this right or it can create spurious errors that are hard to track down. However as we will read later, sometimes Sitecore can get this information wrong!

Shorten the feedback loop

To aid testing and shorten the time it takes to analyse and debug Analytics issues you can create a quick and dirty aspx page that will abandon the session. Drop the following code into a new file with the aspx extension and drop it temporarily into the site. e.g.

<% Session.Abandon();%>
<html>
<head></head>
<body>
<h1>Session killed to death</h1>
</body>
</html>

Navigating to this page when testing should flush the data from the current session into Mongo DB. The alternative is to reduce the session timeout in Web.config to a lower value to avoid waiting around for long period of time during testing:

<sessionState mode="InProc" cookieless="false" timeout="1">

Checking the MongoDB database

Now the user’s session has ended, data should be written to the Interactions collection in MongoDB. To check this we can use a tool such as RoboMongo to connect to the Mongo instance in question and browse the data.
By issuing a query we can check that site interactions are getting:

 logged:db.getCollection('Interactions').find({}).sort({'StartDateTime': -1})

In our case the interactions returned confirm that data are indeed being written correctly. If there are no interactions with a datetime of around the time you killed the session, it suggests this part of the process is not working. With the collection database checked it’s time to move on to the next step in the process: the SQL Server Reporting Database.

Checking the Reporting DB

By connecting to the Reporting database in SQL Server using the analytics connection string, you can check whether the data are being aggregated by runing a query:

SELECT TOP 1000 [SegmentRecordId],
                [Visits],
                [Value],
                [Conversions],
                [TimeOnSite],
                [Pageviews],
                [SegmentId],
                [Date],
                [DimensionKey]  
FROM [<AnalyticsDatabaseName>].[dbo].[ReportDataView]
ORDER BY [Date] DESC

In our case again the data itself was being aggregated correctly to the SQL database but if yours isn’t try the steps detailed here.

Experience Analytics Caching

After the aggregation process completes, the data should be available in Experience Analytics. However in my experience the page is often cached which means closing the browser window and logging into Sitecore again to refresh the data. To combat this the cache can be disabled by altering the cache settings in Sitecore.ExperienceAnalytics.WebAPI.config as this Stack overflow post by Jacob Neilsen details.

Rebuild the reporting database

A final last ditch option, if you’re still experiencing issues is to rebuild the Reporting database via the relevant Sitecore admin page: /sitecore/admin/RebuildReportingDB.aspx

This was enough to get our instance of Experience Analytics displaying data.

Problem 3: Experience Profile not logging contacts or Empty 

This issue of no Contacts appearing in the Experience Profile is quite often accompanied by many errors in the log such as:

ERROR Error during aggregation.Exception: System.NullReferenceExceptionMessage: Object reference not set to an instance of an object.

ERROR General error when submitting contact.Exception: System.NotImplementedExceptionMessage: The current data adapter provider does not support renumbering interactions.Source: Sitecore.Analytics.DataAccess   at Sitecore.Analytics.DataAccess.DataAdapterProvider.RenumberInteractions(ID contactId)   at Sitecore.Analytics.Data.ContactRepository.ReconcileContact(Contact contact, LeaseOwner owner, TimeSpan duration)   at Sitecore.Analytics.Tracking.ContactManager.SubmitContact(Contact contact, ContactSaveOptions options, Contact& realContact)

Checking the Sitecore documentation for configuration maintains that the following files should be disabled for Content Management and enabled for Content Delivery:

• Sitecore.Analytics.Tracking.Aggregation.config
• Sitecore.Analytics.Tracking.Database.config
• Sitecore.Analytics.Tracking.RobotDetection.config

However I had proof that following this advice and disabling the above files on CM prevented Contacts being created and in fact no Experience Analytics data appeared. Enabling the files caused the contacts to start getting logged!

Kam Figy to the rescue! 

An answer on Stack Exchange from the man himself indicates that the configuration documentation is, in fact incorrect, and the files need to be enabled.

Moral of the story: Don’t disable these files on a CM if you’re identifying contacts.

Kam Figy

Happily this solved our issue and the client could start making use of Sitecore 8.2’s great Analytics features.