Why and how we migrated comments to Disqus

When we started working on the latest relaunch of Lullabot.com, we wanted to decouple the front end from the back end. While evaluating how to migrate comments, we realized the following:

• User navigation at Lullabot.com is anonymous so Drupal had nothing to do with comments apart from storing them in the database.
• Even though we were using Mollom, we were still getting spam comments.

After evaluating a few third party commenting tools, we chose Disqus for its moderation and auto-spam detection system. This article details how we exported  a large quantity of Drupal comments into a Disqus account. The process was tedious and it required us to do some debugging and polishing in order to make things easier for everyone. For those reasons, it's worth documenting  the steps we took.

Here is the list of steps we will follow:

1. Create an account at Disqus.
2. Find a way to export Drupal comments for Disqus.
3. Install and configure Disqus Migrate module.
4. Export comments.
5. Upload comments to Disqus.
6. Verify the setup.

1. Creating an account at Disqus

In order to embed Disqus threads on our website, we need a Disqus account and a Disqus site. Once we have created an account we can open https://disqus.com/admin/create and register our site:

Next, at the General tab, we need to set the following options:

At the top of this form we see that our site's shortname is lullabot. We will use this parameter later to identify ourselves against Disqus. Once we are done with this form, we will move to the Advanced tab, where we can specify which domains are allowed to load our site's comments:

This setting will save you a lot of headaches as it prevents Disqus from rendering  the Discussion Widget if the current hostname does not match. This in turn precludes developers or testers from creating new threads from development domains such as dev.lullabot.com or lullabot.local, or even worse: posting test comments that would appear later in the live site. Instead, they will see the following message:

Tip: If you need to test the Disqus Discussion Widget locally, you can edit your Hosts file in order to fake the trusted domain.

2. Finding a way to export Drupal comments for Disqus

Disqus can import comments into an XML file following the WXR (WordPress eXtended RSS) format. Here is the Disqus interface to upload the file:

What we need is a tool in Drupal to export Lullabot.com comments into a single WXR file. By looking at the Disqus module description, exporting comments was implemented for the Drupal 6 version, but never got completed for Drupal 7:

By searching further I found a sandbox called Disqus Migrate, which is a port of the Drupal 6 sub-module for Drupal 7. The module's description did not look promising: a sandbox project  marked as Unsupported, encouraging us not to use it with real accounts, and other warnings: 

Suprisingly, the module proved to be a solid project that simply needed some polish. In fact, during the process, I submitted a few patches to the module's issue queue and promoted it to a full project.

3. Installing and configuring the Disqus Migrate module

Installing sandbox projects is not among best practices in Drupal but at Lullabot sometimes we discover sandbox projects that are mature enough to become full projects. In these cases, we help module maintainers fix existing bugs and then we encourage them to promote the sandbox to a full project. 

I glanced through the module's source code at Drupal.org and my first impression was good. I did not see any security holes and the code was easy to understand. Next, I downloaded and installed the module in my local instance  of Lullabot.com by entering the following commands:

git clone --branch 7.x-1.x http://git.drupal.org/sandbox/dwkitchen/1483518.git sites/all/modules/sandbox/disqus_migrate
drush en disqus_migrate

Disqus Migrate leverages the Disqus Drupal module by adding a tab called Migrate at admin/config/services/disqus. At the top of this administration form, we must set the shortname, which we saw before at the Disqus admin panel. If we are going to use Drupal to render the Disqus widget, then we need a set of keys to connect with the Disqus API. Once you have registered your application there, you can enter the keys at the form like so:

Unless we are exporting comments from the production environment (which we don't recommend), we need to override the base URL so each thread will have the trusted domain. Here is the field where we can set this:

4. Exporting comments

Now we are ready to export comments. My local environment has a recent copy of the production environment's database. Disqus Migrate adds a task at admin/content/comment where we can export comments:

After confirming, we get all of the comments in an XML file:

Here is how the XML file looks:

In the above XML file, there is a <comment> entry for each comment which belongs to a thread. Each thread has a unique identifier which the Disqus Migrate module sets to the node path, such as node/30. Now we are ready to upload comments to Disqus.

5. Uploading comments to Disqus

Here is a screenshot of the Import tool in the Disqus administration. We have selected the XML file and after clicking Upload and import, the file will be uploaded and processed by Disqus.

That's all, after a few minutes we should see comments at the Comments tab:

If you have issues uploading comments to Disqus, then have a look at these troubleshooting tips. In our case, we used xmllint to discover bugs in the the XML file and then we submitted a few patches to the Disqus Migrate's issue queue to fix them.

6. Verifying the setup and wrapping up

If you are using Drupal to render the frontend and you have filled out the administration form that we saw at Installing and configuring Disqus Migrate module, Disqus comments should load automatically for your site. Lullabot.com has a decoupled frontend powered by React.js so we wrote a React component which, given a thread title and identifier, renders the Disqus widget with its comments. You can see it in action below this article. Try it out and leave a comment with your thoughts.

Image credit: https://nyibelieve.wordpress.com/2010/06/05/new-york-i-love-you