Blog

Two ways to import an XML file with .Net Core or .Net Framework

21 November 2019

It's always the simple stuff you forget how you do. For years I've mainly been working with JSON files, so when faced with that task of reading an XML file my brain went "I can do that" followed by "actually how did I used to do that?".

So here's two different methods. They work on .Net Core and theoretically .Net Framework (my project is .Net Core and haven't checked that they do actually work on framework).

My examples are using an XML in the following format:

1<?xml version="1.0" encoding="utf-8"?>
2<jobs>
3    <job>
4        <company>Construction Co</company>
5        <sector>Construction</sector>
6        <salary>£50,000 - £60,000</salary>
7        <active>true</active>
8        <title>Recruitment Consultant - Construction Management</title>
9    </job>
10    <job>
11        <company>Medical Co</company>
12        <sector>Healthcare</sector>
13        <salary>£60,000 - £70,000</salary>
14        <active>false</active>
15        <title>Junior Doctor</title>
16    </job>
17</jobs>

Method 1: Reading an XML file as a dynamic object

The first method is to load the XML file into a dynamic object. This is cheating slightly by first using Json Convert to convert the XML document into a JSON string and then deserializing that into a dynamic object.

1using Newtonsoft.Json;
2using System;
3using System.Collections.Generic;
4using System.Dynamic;
5using System.IO;
6using System.Text;
7using System.Xml;
8using System.Xml.Linq;
9
10namespace XMLExportExample
11{
12    class Program
13    {
14        static void Main(string[] args)
15        {
16            string jobsxml = "<?xml version=\"1.0\" encoding=\"utf-8\"?><jobs>    <job><company>Construction Co</company><sector>Construction</sector><salary>£50,000 - £60,000</salary><active>true</active><title>Recruitment Consultant - Construction Management</title></job><job><company>Medical Co</company><sector>Healthcare</sector><salary>£60,000 - £70,000</salary><active>false</active><title>Junior Doctor</title></job></jobs>";
17
18            byte[] byteArray = Encoding.UTF8.GetBytes(jobsxml);
19            MemoryStream stream = new MemoryStream(byteArray);
20            XDocument xdoc = XDocument.Load(stream);
21
22            string jsonText = JsonConvert.SerializeXNode(xdoc);
23            dynamic dyn = JsonConvert.DeserializeObject<ExpandoObject>(jsonText);
24
25            foreach (dynamic job in dyn.jobs.job)
26            {
27                string company;
28                if (IsPropertyExist(job, "company"))
29                    company = job.company;
30
31                string sector;
32                if (IsPropertyExist(job, "sector"))
33                    company = job.sector;
34
35                string salary;
36                if (IsPropertyExist(job, "salary"))
37                    company = job.salary;
38
39                string active;
40                if (IsPropertyExist(job, "active"))
41                    company = job.active;
42
43                string title;
44                if (IsPropertyExist(job, "title"))
45                    company = job.title;
46
47                // A property that doesn't exist
48                string foop;
49                if (IsPropertyExist(job, "foop"))
50                    foop = job.foop;
51            }
52
53            Console.ReadLine();
54        }
55
56        public static bool IsPropertyExist(dynamic settings, string name)
57        {
58            if (settings is ExpandoObject)
59                return ((IDictionary<string, object>)settings).ContainsKey(name);
60
61            return settings.GetType().GetProperty(name) != null;
62        }
63    }
64}
65

A foreach loop then goes through each of the jobs, and a helper function IsPropertyExist checks for the existence of a value before trying to read it.

Method 2: Deserializing with XmlSerializer

My second approach is to turn the XML file into classes and then deserialize the XML file into it.

This approch requires more code, but most of it can be auto generated by visual studio for us, and we end up with strongly typed objects.

Creating the XML classes from XML

To create the classes for the XML structure:

1. Create a new class file and remove the class that gets created. i.e. Your just left with this

1using System;
2using System.Collections.Generic;
3using System.Text;
4
5namespace XMLExportExample
6{
7
8}

2. Copy the content of the XML file to your clipboard
3. Select the position in the file you want to the classes to go and then go to Edit > Paste Special > Paste XML as Classes

If your using my XML you will now have a class file that looks like this:

1using System;
2using System.Collections.Generic;
3using System.Text;
4
5namespace XMLExportExample
6{
7
8    // NOTE: Generated code may require at least .NET Framework 4.5 or .NET Core/Standard 2.0.
9    /// <remarks/>
10    [System.SerializableAttribute()]
11    [System.ComponentModel.DesignerCategoryAttribute("code")]
12    [System.Xml.Serialization.XmlTypeAttribute(AnonymousType = true)]
13    [System.Xml.Serialization.XmlRootAttribute(Namespace = "", IsNullable = false)]
14    public partial class jobs
15    {
16
17        private jobsJob[] jobField;
18
19        /// <remarks/>
20        [System.Xml.Serialization.XmlElementAttribute("job")]
21        public jobsJob[] job
22        {
23            get
24            {
25                return this.jobField;
26            }
27            set
28            {
29                this.jobField = value;
30            }
31        }
32    }
33
34    /// <remarks/>
35    [System.SerializableAttribute()]
36    [System.ComponentModel.DesignerCategoryAttribute("code")]
37    [System.Xml.Serialization.XmlTypeAttribute(AnonymousType = true)]
38    public partial class jobsJob
39    {
40
41        private string companyField;
42
43        private string sectorField;
44
45        private string salaryField;
46
47        private bool activeField;
48
49        private string titleField;
50
51        /// <remarks/>
52        public string company
53        {
54            get
55            {
56                return this.companyField;
57            }
58            set
59            {
60                this.companyField = value;
61            }
62        }
63
64        /// <remarks/>
65        public string sector
66        {
67            get
68            {
69                return this.sectorField;
70            }
71            set
72            {
73                this.sectorField = value;
74            }
75        }
76
77        /// <remarks/>
78        public string salary
79        {
80            get
81            {
82                return this.salaryField;
83            }
84            set
85            {
86                this.salaryField = value;
87            }
88        }
89
90        /// <remarks/>
91        public bool active
92        {
93            get
94            {
95                return this.activeField;
96            }
97            set
98            {
99                this.activeField = value;
100            }
101        }
102
103        /// <remarks/>
104        public string title
105        {
106            get
107            {
108                return this.titleField;
109            }
110            set
111            {
112                this.titleField = value;
113            }
114        }
115    }
116
117}

Notice that the active field was even picked up as being a bool.

Doing the Deserialization

To do the deserialization, first create an instance of XmlSerializer for the type of the object we want to deserialize too. In my case this is jobs.

1            var s = new System.Xml.Serialization.XmlSerializer(typeof(jobs));

Then call Deserialize passing in a XML Reader. I'm creating and XML reader on the stream I used in the dynamic example.

1            jobs o = (jobs)s.Deserialize(XmlReader.Create(stream));

The complete file now looks like this:

1using System;
2using System.IO;
3using System.Text;
4using System.Xml;
5
6namespace XMLExportExample
7{
8    class Program
9    {
10        static void Main(string[] args)
11        {
12            string jobsxml = "<?xml version=\"1.0\" encoding=\"utf-8\"?><jobs>    <job><company>Construction Co</company><sector>Construction</sector><salary>£50,000 - £60,000</salary><active>true</active><title>Recruitment Consultant - Construction Management</title></job><job><company>Medical Co</company><sector>Healthcare</sector><salary>£60,000 - £70,000</salary><active>false</active><title>Junior Doctor</title></job></jobs>";
13
14            byte[] byteArray = Encoding.UTF8.GetBytes(jobsxml);
15            MemoryStream stream = new MemoryStream(byteArray);
16
17            var s = new System.Xml.Serialization.XmlSerializer(typeof(jobs));
18            jobs o = (jobs)s.Deserialize(XmlReader.Create(stream));
19
20            Console.ReadLine();
21        }
22    }
23}

And thats it. Any missing nodes in your XML will just be blank rather than causing an error.

How to create a shared access signature for a specific folder in Azure Storage

13 November 2019

As we move into a serverless wold where devlopment work is done less on generic multipurpose servers that require patching both to the OS and to the application installed on them, things become easier and in some cases cheaper.

For instance I am currently working on a project that needs to send and recieve XML files from different sources and with no other requirements for a server, Azure storage seemed the perfect fit. As it's just a storage service you pay for what you need and nothing more. All the patching and maintencance of API's is provided by Microsoft and we have an instant place to store out files. Ideal!

However new technology always comes with new challenges. My first issue came with the plan on how we would be getting files in and out with partners. This isn't the first time I've worked with Azure File Storage, but it is the first time I did it without an opps person setting things up. Previously we used FTP, which while dated works with a lot of applications. Like a headphone jack it's not the most glamorus of connection, but it works, everyone knows how it works and everyone has things that work with it. However it transpires that despite Azure storage being 10 years old and reciving requests for FTP from the start, Microsoft have decided to go the same route as Apple with the headphone jack and not have it. Instead the only option for an integration is REST. As it transpires the opps people I had worked with in the past when faced with this issue had just put a VM infront of it, which kind of defeats the point of using Azure storage in the first place!

So we're going with REST and Microsoft provide quite a straightforward REST API all good so far, but how do we limit access? Well there's a guide to Using the Azure Storage REST API which contains a section on creating an authorisation header. It's long and overly complex, but does point you in the direction that to do this you need a Shared Access Signature. The other option is an access key, but this is something you should never give away to a third party.

Shared Access Signature

After a bit more digging through the documentation (and just clicking the thing that sounded right in the potal) I found this documentation on creating an Account SAS which sounded like what I wanted (it wasn't, but it's close).

With a shared access signature you can say what kind of service should be allowed, what permissions they should have, IP address's, start and end dates. All awesome things.

Once I had this I could then use the REST API, but there was a problem. I could access every folder in the storage account and there was no way to stop this! For integrating with 2 third partys they would both be able to access each others stuff, and our own private stuff.

There is also no way to revoke the SAS once it's been generated other than refreshing the access keys which would affect everyone.

Folder Level Shared Access Signature

After a bit more research I found what I was looking for. How to create a shared access signature at a folder or item level and how to link it to a policy.

The first thing you need is Azure Storage Explorer. Once your set up with this you will be able to view all your storage accounts.

From here you are able to browse to the folder you want to share right click it and choose Manage Access Policies.

This will open a dialoge to manage the policies for this specific object rather than the account.

Here you can set all the same permissions as you could for a signature at an account level but now for a specific object and against a policy rather than an actual signature, meaning the policy can be updated in the future with no change to the signature.

Better still you can remove the policy which will then invalidate any signature using it.

For the actual signature key right click the same folder and click Get Shared Access Signature.

Then in the dialoge select the policy from the drop down rather than spcifying the individual permissions.

Click create and you can copy the keys.

You now have an access key that is limited to a specific folder rather than the entire account.

This is only possible to do though one of the code/scripting interfaces. e.g. PowerShell or the storage explorer. The azure portal will only let you get signatures at an account level.

Debugging Sitecore 9 Analytics Issues

11 October 2019

With Sitecore 9 the way analytics is recorded and processed changed. Rather than everything being done by the IIS application, the architecture changed to include:

A second IIS application called xconnect
A windows services called Sitecore XConnect Search Indexer

As well as this Mongo was replaced by:

SQL Server databases
A SOLR XDB Core

Sitecore 9 also introduced data being secure in transit as well as at rest which means all traffic is encrypted using certificates.

While all these peices can be considered good, it does also create more points of failure that becomes harder to debug. So after spending a decent amount of time debugging why no analytic reports were loading and then why no data was appearing in the reports, I've made a checklist to go through.

Debugging Sitecore Analytics Checklist

1. XConnect Connection String

The main Sitecore application has connection strings for where it will find the XConnect service. They will look like this:

1 <add name="xconnect.collection" connectionString="https://mysite.xconnect" />
2  <add name="xconnect.collection.certificate" connectionString="StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=EF7F38B623E6664359110F2C6EB6DA00D567950F" />

One gives the path to the XConnect service and the other gives the path to find the certificate.

Firstly check that the XConnect path is correct and that you can access it and secondly check that the thumbprint corresponds with a certificate in the certificate store.

You can see what certificates are on your machine using this PowerShell script:

1Get-ChildItem -Path "cert:\LocalMachine\Root" | Format-Table Subject, FriendlyName, Thumbprint
2Get-ChildItem -Path "cert:\LocalMachine\My" | Format-Table Subject, FriendlyName, Thumbprint
3Get-ChildItem -Path "cert:\CurrentUser\My" | Format-Table Subject, FriendlyName, Thumbprint

2. Check certificate expiry date

Having a certificate is a good start, but it could still have expired.

Find the certificate in the certificate store by hitting start > manage computer certificates and find it in one of the folders.

Check the expiration date. If it's in the past then it's not going to work. This is common because the installation script for Sitecore 9 will set the expiration date to a year after install by default.

If it has expired you will need a new cert, you can create this by using the same script that you used to install sitcore origionally (Just the certificate bit).

1#Switch to correct vesion of SIF
2Remove-Module -Name SitecoreInstallFramework
3Import-Module -Name SitecoreInstallFramework -RequiredVersion 1.2.1
4
5#define parameters
6$prefix = "SitePrefix"
7$PSScriptRoot = "C:\resourcefiles9.0"
8$XConnectCollectionService = "$prefix.xconnect"
9$sitecoreSiteName = "$prefix.sc"
10
11#install client certificate for xconnect
12$certParams = @{
13 Path = "$PSScriptRoot\xconnect-createcert.json"
14 CertificateName = "$prefix.xconnect_client"
15 RootCertFileName = "SIF121Root"
16}

3. Check security permissons on the certificate

If you have a certificate and it's still valid then it could be that the app pool the site is running in doesn't have access to read the certificate.

To check this:

Right click the certificate
All Tasks
Manage Private Keys
Check that the app pool user for your site is listed in the list of users and that it has read permission. If it's not there add it using the name
IIS APPPOOL\app pool name

4. Check License Files

Partner licenses only last for a year so if your using one of those it may have expired.

We're all used to checking the license file in the Sitecore application but XConnect has a license too.

These will be in:
sitename.xconnect\App_data\
sitename.xconnect\App_data\jobs\continuous\IndexWorker\App_data
sitename.xconnect\App_data\jobs\continuous\AutomationEngine\App_Data

Only the first will be used when your viewing the site, but it's worth knowing about the others to, incase you ever run a job manually.

5. Check manual rebuild of indexes

You can trigger a manual rebuild of the xDB index by following these instructions:

https://doc.sitecore.com/developers/90/sitecore-experience-platform/en/rebuild-the-xdb-index-in-solr.html

Remember in point 4 that it has it's own license file. It also has it's own connection strings.

6. Check XConnect site works in a browser

If you open XConnect in a browser you should recieve no certificate errors and a timestamp saying how long XConnect had been running for.

7. Check certificates are in the right store

This stack overflow post was a big help for me (https://stackoverflow.com/questions/26247462/http-error-403-16-client-certificate-trust-issue ). I was at the point where everything seemed right, but moving the certificates as shown here got it to the point of the analytics reports loading.

Windows 2012 introduced stricter certificate store validations. According to KB 2795828: Lync Server 2013 Front-End service cannot start in Windows Server 2012, the Trusted Root Certification Authorities (i.e. Root) store can only have certificates that are self-signed. If that store contains non-self-signed certificates, client certificate authentication under IIS returns with a 403.16 error code.

To solve the problem, you have to remove all non-self-signed certificates from the root store. This PowerShell command will identify non-self-signed certificates:

1Get-Childitem cert:\LocalMachine\root -Recurse | 
2    Where-Object {$_.Issuer -ne $_.Subject}

In my situation, we moved these non-self-signed certificates into the Intermediate Certification Authorities (i.e. CA) store:

1Get-Childitem cert:\LocalMachine\root -Recurse | 
2    Where-Object {$_.Issuer -ne $_.Subject} | 
3    Move-Item -Destination Cert:\LocalMachine\CA

Checklist for data not going into Analytics

If you've got to the point of the analytics reports working, but not showing any data, this is my checklist for making sure data goes in. In my case I was trying to log site searches as per my article from a few years ago https://himynameistim.com/2017/09/13/populating-the-internal-search-report-in-sitecore/ there wern't any errors, but no data ever showed.

1. Enable Analytics Debugging

In Sitecore.Analytics.Tracking.config there is a setting to set the analytics logging level to debug. You will also need to set the log level on log4net root in Sitecore.config to debug.

2. Disable Robot Detection

In my case Sitecore thought I was a robot. Changing these settings will disable that:

1<setting name="Analytics.AutoDetectBots" set:value="false" />
2<setting name="Analytics.Robots.IgnoreRobots" set:value="false" />

3. Test analytics is tracking something

One of the hardest parts about analytics is it's not instant. The initial tracking only goes into the DB at the end of the users session and that's only for collection. It won't appear in the reports until processing has happened. So to speed this up:

Create a page called kill.aspx as follows. This will end the users session and trigger the data to be fed into the DB.

1<%@ Page language="c#" %>
2
3<!DOCTYPE html>
4<html>
5  <head>
6
7  </head>
8  <body>
9
10<div>Session Abandoned</div>
11<% Session.Abandon(); %>
12
13  </body>
14</html>

Next do something on the site that will cause some tracking to get added to the DB. In my case it was the search. Then go to kill.aspx to force session abondon.

Check the logs. You should see something like this...

125016 11:19:13 DEBUG [Analytics]: The CommitSession pipeline, ProcessSubscriptions is skipped - there is no subscriptions for location id: 4ebd0208-8328-5d69-8c44-ec50939c0967

Check the DB an entry should have gone into a shard db for [xdb_collection].[Interactions] table

To speed up processing, restart the main sitecore application.