Many initiatives must import information. I’ve discovered it to be a painful course of each time. Every import is customized and requires customized code. There are components of the method which can be the identical every time (connecting with supply information, studying information, creating a brand new merchandise, avoiding duplicates, updating current gadgets). I needed to discover a strategy to make this course of simpler and fewer painful whereas reusing as a lot code as potential. I did loads of analysis and got here throughout Sitecore Knowledge Alternate Framework (DEF). I used to be shocked to study it has been round since model 8.2! I’ve by no means heard of it! After performing some preliminary analysis, I made a decision it sounded precisely like what I used to be searching for. It’s supported by Sitecore and has a pipeline structure, so I ought to be capable to make it do precisely what I want. And I can package deal up the bottom and share with others to hurry up future imports. Ultimately, I made a decision it was not for me. However let me share my journey so you may resolve for your self.
Getting Began with DEF
I began by following Sitecore’s tutorial walkthrough. The tutorial is properly written. I used to be shortly in a position to set up the instrument inside a docker container by making just a few adjustments to my docker-compose.xml and .env information. It took me a few days to get right through the tutorial. There are a lot of ideas to know and naming conventions to know. I might say the training curve is kind of excessive. On the finish of the tutorial, I used to be in a position to import information from a csv file and create gadgets in Sitecore! Mapping the supply information to Sitecore gadgets was simple and versatile. I used to be very excited concerning the prospect, and I used to be prepared to maneuver on!
Taking my First Steps
The primary change I needed to make was to have the ability to learn from a JSON file as an alternative of a csv. This was simple to perform with both system.textual content.json or newtonsoft.json and a customized mannequin class. As an alternative of studying positions from an array, I needed to learn properties of an object. I modified my Worth Accessor from an Array Worth Accessor to a Property Worth Accessor. I modified the Examples.DataExchange.Suppliers.FileSystem.ReadTextFileStepProcessor class to course of the json information as an alternative of strains of a csv.
var textual content = File.ReadAllText(settings.Path);
var json = JsonConvert.DeserializeObject<Response>(textual content);
foreach(Particular person p in json.outcomes)
{
yield return p;
}
The second change I made was to learn information from an api as an alternative of a textual content file. This was simple to do with WebRequest. I might have come again later and altered this to a dependency injected HttpClient.
var webRequest = WebRequest.Create(uri);
utilizing (var webResponse = webRequest.GetResponse())
{
utilizing (var stream = new StreamReader(webResponse.GetResponseStream()))
{
textual content = stream.ReadToEnd();
}
}
Going Past the Fundamentals
As I began to implement the particular wants of my importer, I began to battle with DEF.
My first purpose was to have the ability to delete gadgets in Sitecore that have been not a part of the supply information. I needed to learn or write some property so I might know which gadgets I may delete.
My first thought was to learn the beginning time of the pipeline batch merchandise and set because the created or final up to date time of the newly created gadgets. I didn’t discover a straightforward strategy to entry this merchandise in the course of the pipeline execution. My second thought was to put in writing a guid to every merchandise. However I couldn’t discover a strategy to change the guid on every run of the pipeline batch. I additionally thought of deleting gadgets created inside a sure period of time from the final pipeline batch run utilizing powershell. However I needed to maintain the method inside DEF and there can be no assure how briskly the importer would run to know which gadgets have been modified this run.
I settled on including boolean worth to the merchandise’s information template to point if it imported on the final run. I added a pipeline to reset the sphere to false on any current gadgets. I added a subject to the worth accessor set that used a continuing worth reader to set the sphere to true whereas importing the info (both creating new gadgets or updating current gadgets). I added a second pipeline to delete any gadgets the place the sphere was set to false. I used to be shocked that deleting a Sitecore merchandise was not a part of DEF. I ended up writting a customized pipeline step processor (https://github.com/ericsanner/SitecoreContentImporter).
DEF Pipeline Batch exhibiting three pipeline steps
DEF Pipelines exhibiting every step within the course of
Then Issues Received Attention-grabbing
My subsequent purpose was to map a droplink subject. The information would come from the supply as textual content. I needed to lookup the associated merchandise and retailer the guid on the goal merchandise. I couldn’t discover an out of the field part that matched these standards. The documentation talked about while you would wish to prolong the default elements however didn’t give any examples of the way to do it. I scoured the online with out a lot success. I decompiled the DEF dlls and created a customized worth accessor. This required a Worth Accessor Converter and a Worth Reader. The plan was to make use of the worth reader on the supply information, discover a matching merchandise within the content material tree, and write the guid worth to the goal. I fought with this for a lot of days on and off. I had issues getting the pipeline to name my customized worth accessor. When it did get known as, it will not use my customized learn technique. I attempted going the opposite approach utilizing a customized worth author earlier than writing the worth to the goal with equally unsuccessful outcomes. I made a decision to maneuver on to the following downside and are available again to this later.
The following purpose was to course of baby information. Every merchandise within the supply information had related places. I wanted to interrupt the principle pipeline to kick off a brand new pipeline to iterate the places. I spent a day looking for an answer, all of the whereas considering I might have the identical downside of mapping the associated location to the present merchandise by way of guid (or guids if an merchandise had a number of places). The issue was worse but as a result of the I needed to create Sitecore gadgets for the state and nation and retailer the guids on the placement merchandise. I put this on the again burner as properly.
At this level, I used to be fairly annoyed, I virtually gave up, however I moved on to my subsequent purpose. If I may get to work, I might be capable to return to the opposite gadgets. I needed to skip gadgets from the supply information if sure fields have been empty. This appeared like a straightforward win. The worth mappings help mapping guidelines, worth transformers and have an possibility for “mapping set fails if this mapping fails”. I created a mapping rule to test for an empty worth. I checked the field to fail the mapping set. I received a brand new Sitecore merchandise with an empty worth on the sphere. Wait. What? I decompiled some extra dlls and located that the mapping guidelines don’t cease the pipeline or trigger it to abort processing the present merchandise. The brand new merchandise is already created by the point it applies the mapping set. The pipeline continues and writes any out there information to the merchandise. That isn’t what I might have anticipated based mostly on the label on the checkbox.
And Then I Gave Up
I had three main duties left and I nonetheless needed to publish the brand new/up to date gadgets and set the pipeline batch to run on a schedule. I used to be already previous my deadline of needing to finish this work. I couldn’t see the way to make this simple for others to make use of sooner or later. I made a decision to chop my losses, transfer on and go in a brand new route. In a earlier mission, we wrote a customized importer in C#. That took perpetually too. A colleague beneficial powershell and supplied some examples to get me going shortly. Their code was arduous to observe, and I nonetheless needed one thing that was extra reusable.
I did find yourself utilizing powershell. I created a library of features that can hopefully make future imports simpler. Essentially the most attention-grabbing piece was the replace perform. I move an merchandise and a hashtable of the important thing worth pairs to replace. It makes use of variable substitute to replace the merchandise based mostly on the keys within the hashtable.
Operate Replace-SitecoreItem {
$merchandise.Modifying.BeginEdit()
foreach($key in $updates.GetEnumerator())
{
if($merchandise.($key.Title) -ne $null -and $merchandise.($key.Title) -ne $key.Worth)
{
$merchandise.($key.Title) = $key.Worth
}
}
$itemModified = $merchandise.Modifying.EndEdit()
}
This makes it quick and straightforward to create the mapping out of your supply information to the Sitecore merchandise. It is extremely simple for the following particular person to learn and replace as properly.
$merchandise = Get-NewOrExistingSitecoreItem $itemRoot $itemTemplateId $itemName
$updates = @{}
$updates.Add(“__Display title”, $job.title)
$updates.Add(“Title”, $job.title)
$updates.Add(“OpenGraphTitle”, $job.title)
$updates.Add(“OpenGraphDescription”, $briefDescription)
$updates.Add(“MetaDescription”, $briefDescription)
$updates.Add(“NavigationTitle”, $job.title)
$updates.Add(“Content material”, $job.description)
$updates.Add(“ImportedOnLastRun”, “1”)
Replace-SitecoreItem $merchandise $updates
I used to be in a position to save the script to the Script Library (/sitecore/system/Modules/PowerShell/Script Library) and shortly create a schedule activity to execute my script each day.
Conclusion
I actually needed to love Knowledge Alternate Framework. Sitecore’s tutorial walkthrough is properly written. It provides you the fundamentals however leaves quite a bit so that you can uncover by yourself. I had days of analysis decompiling Sitecore dlls and looking out on-line, adopted by a day of really feel good success, then days of frustration as I tackled the following step within the course of. After three weeks, I threw within the towel and wrote the importer in Powershell. You could find my full powershell script on my github https://github.com/ericsanner/SitecoreContentImporter. Let me know when you discover it helpful or when you’ve got any recommendations on issues I may add based mostly on an import you’ve carried out prior to now.
