I have run into a number of clients that want to create a system to syndicate content to multiple sites. There are several ways to set a system like that up, perhaps using the Migrate or Deploy modules. But it’s also possible to do this using Services and Views on the content source and Feeds on the consuming sites.
The development team at Highlights for Children is diving into Drupal in a big way, and Lullabot has been helping them. They are building a suite of Drupal sites that need a way to consume content from a central Drupal source. The idea of solving this problem using Services and Feeds was a good fit for their requirements, so we worked through the details of how to get this accomplished. Highlights wanted to share the recipe for doing this with the community, and I pulled the details together into this article. We made the example a little more generic, so it would apply to other situations, and focused on ways that you can accomplish this without code. The result is a general recipe that we hope will be widely useful, rather than an exact description of the way they ended up using these tools.
Those who aren’t trying to solve this particular problem may still find it interesting to see an example of how to configure and use Services, Views, and Feeds together. Services and Feeds are two very powerful and interesting modules, but sometimes they can be confusing to configure, and a step-by-step illustration may make it more clear.
Terminology
In this recipe we have two different Drupal sites. One of them contains content that can be shared (we call that the ‘Source’) and the other needs to consume that content (we call this the ‘Destination’). The Source will share its content using Services and Services Views. The Destination will consume the content using Feeds.
Modules needed on the Source
- CTools (http://drupal.org/project/ctools)
- Services and Rest Server (http://drupal.org/project/services)
- Services Views (http://drupal.org/project/services_views)
- Views (http://drupal.org/project/views)
Modules needed on the Destination
- Feeds (http://drupal.org/project/feeds)
- Feeds xPath Parser (http://drupal.org/project/feeds_xpathparser)
- CTools (http://drupal.org/project/ctools)
- Job Scheduler (http://drupal.org/project/job_scheduler)
- Views (http://drupal.org/project/views)
- Feeds Tamper (http://drupal.org/project/feeds_tamper)
Prepare the Source Content
First, create the content type(s) and content on the source. If the content needs to have references to other content, use the EntityReference module to create the reference fields (http://drupal.org/project/entityreference).
Set up Services on the Source
Enable the following modules on the source site:
- CTools (http://drupal.org/project/ctools)
- Services and Rest Server (http://drupal.org/project/services)
- Services Views (http://drupal.org/project/services_views)
- Views (http://drupal.org/project/views)
The Rest Server requires that you add spyc.php to sites/all/modules/services/server/lib. You can get that file from http://code.google.com/p/spyc/.
Create a view of the content you want to syndicate. Instead of creating a page display, create a ‘Services’ display. On the Services display, use the style option to create an unformatted list of fields. Then add each field that you want to move to the Destination to the list of fields in the display.
When editing the field, set up a custom value key with the name you want to use for this value in the xml field, keeping in mind that you want to be sure that each value is unique in the feed.
As you add the fields, you can see in the preview an array of the values that will be displayed in the XML. Many fields in the field API use the entity id to retrieve their values, so you will only see that value in this array. Don’t worry, it will expose the right field value in the XML.
Give this view a path. This will be the services path for this view.
Next go to admin/structure/services and create a new service. Give it a name, use the REST server, and give it a ‘Path to endpoint’ of ‘rest’.
After it has been created you will see a link next to it to ‘Edit Resources’. That will bring up a screen like the following that shows the possible resources for this services. You will see a resource for ‘Views’ and for the path you created in the view. Check both of them.
You can see if this is working by navigating to that path, like http://example.com/rest/articles. You should see something like the following:
You can see the output as xml by going to http://example.com/rest/articles.xml, as json by going to http://example.com/rest/articles.json, etc.
Once you have confirmed that your XML service is working correctly, you can switch to the destination.
Prepare the Destination
Next, create the content type(s) on the destination. If the content needs to have references to other content, use the EntityReference module to create the reference fields (http://drupal.org/project/entityreference). We’re going to use Feeds to populate the content from the XML we just created on the Source.
Set up Feeds
Enable the following modules on the Destination:
- Feeds (http://drupal.org/project/feeds)
- Feeds xPath Parser (http://drupal.org/project/feeds_xpathparser)
- CTools (http://drupal.org/project/ctools)
- Job Scheduler (http://drupal.org/project/job_scheduler)
- Views (http://drupal.org/project/views)
Create a new content type for the feed (this is not the same as the content type we will use for the nodes that the feed will create). It only needs a title, no body, so you can remove the body field.
Go to admin/structure/feeds and click the link to ‘Add importer’. Give the item a name and description.
Once the importer is created, you will see that it can be edited.
Edit the importer and you will see that you can set up various components. We will attach it to the Feed content type we just created, use the HTTP Fetcher to retrieve it from the XML link we just created on the Source, use the xPath XML Parser to deconstruct the values and move them into right fields, and the Node Processor to create nodes from the results.
Create the node mapping before you configure the xPath parser. This is somewhat confusing. First set up the Node processor to create nodes of whatever content type you want to contain the imported values, in this example that is ‘Article’.
Once you have done that, use the Mapping link to identify what will go into each field in that content type. When using xPath parser, we are going to populate each value with an xPath value. So we need to create a mapping item for each target field that gets its value from xPath. This may look odd, but it is correct. You will end up with a list of fields that all have the same source, the xPath parser. It looks like the following screenshot:
The other tricky part of the configuration is the xPath parser, especially if you’ve never used xPath. xPath is a standard for locating values in a XML file. If you’re not familiar with xPath, there is reference material at http://w3schools.com/xPath.
We need to identify an xPath identifier for the ‘Context’, which is the place in the XML that represents an individual node, and then one for each of the fields within that node. The configuration would up looking something like the following:
Import the Content
We’re finally ready! Create a new Feed node. Input the services path we created above. Be sure to append ‘.xml’ to the path so the parser knows it is dealing with XML.
Once the feed node has been created, you will see an ‘Import’ tab on it. Click on that to actually import the data. We enabled Views so we can also see a ‘Log’ tab on the feed node. That tab shows us a view of the log messages that were created when trying to import the feed.
References, A Special Problem
Moving content that has references in it from one site to another creates a special problem. The reference field on the source site is pointing to the nid of the referenced material, but it is using the nid from the source site. When you move all this content to another site, it will no longer have the same nid. Therefore, references to that material need to be updated to contain the right nid for that item on the destination.
For example. You might have a node 70 on the source that has a reference to node 89 on the source. When you move these two pieces of content to another site, the item that was node 70 on the source might become node 650, and the item that was node 89 might become node 777. The destination reference to node 89 needs to be updated so it now points to node 777.
We’re using Feeds to pull in the content from the Source site. Each field on our destination site is managed using a handler that understands where its value belongs on that particular type of field. We need the Entityreference handler to be smart enough to figure out which value is actually needed on the Destination.
To do that we currently need an EntityReference patch, from http://drupal.org/node/1616680. Once that is applied, EntityReference will analyze each value that is passed into it, look through the Feeds tables to see if the referenced node has already been created. If it has, it will swap in the nid of the node that was created from that value. If it can’t find information that tells it that node has already been created, it will wipe out the value, because a reference to the wrong or a non-existing node would not work correctly.
This process will work best if the referenced material is a different content type. That makes it possible to pull the referenced material before the content that references it, so that all the referenced material exists, to keep the reference links working.
If that is not possible, the alternative is to run the migration twice. By the second pass all the nodes will exist and the links can be created. For this second pass to work correctly, you have to choose the option to ‘Update existing nodes’ in the Node processor settings.
Images, Another Special Problem
Another problem with using Feeds and Services to do this trying to find a way to get images to import correctly from the central server. If images are available at publicly available urls, the following method will work.
In the view of the image field, set the field up to ‘Display download path instead of file storage URI ‘. This will create XML that displays the image like ‘public://field/image/imagefield_O0ivdK.png’. That isn’t quite what we need to access it from the Destination. So on the Destination we need to add one more module to our mix, the Feeds Tamper module (http://drupal.org/project/feeds_tamper).
Feeds Tamper lets us massage our Feed values before trying to do something with them. In this case we want to replace ‘public://’ in the image path with the actual path to the image on the Source, like ‘http://example.com/sites/default/files/’.
Once Feeds Tamper is enabled, you will see a ‘Tamper’ tab on the Feeds importer. It will show us a list of all the fields that have been defined. Select the field that contains the image field, and then choose to use a Regex to replace the value. It will look something like the following:
Now when you import nodes that have images, the images will be retrieved from the full image url and be properly transformed into images on the Destination site.
Conclusion
This should get you started on creating a simple, no-code, content syndication system. I want to thank the team from Highlights for Children for their willingness to share this information with the Drupal community. As noted above, there are other ways to solve this problem, in particular by using the Migrate or Deploy modules, but this is the approach that seemed to fit their needs the best.
If you haven’t used these modules before and have been wondering how they can work together, you may want to try this recipe out.