Results tagged “software”

Now, I’ve done it. I’ve relaunched my blog using a design I’ve been building on for a few months. (I don’t have much time for such things, so it takes some time to build anything out for this site.) Anyway, there’s more bits and bobbles to tweak out, but the basics are now there. It’s quite an update since I haven’t really dinked with either the overall design or platform running my web site for a few years. The big technical change has been that I am now using Movable Type rather than Drupal to serve my blogs. I’m going to start by explaining why and then I’ll delve into how I made the move.

Why Movable Type?

The main reason for the switch from Drupal to Movable Type is because I wanted to try something new. I’ve been on Drupal for a long time and am familiar with it. I’m also likely to maintain that familiarity for quite sometime since we use Drupal extensively at work. To paraphrase The Matrix Revolutions, “You do not truly know an application until you fight it.” In this case, I’m running Movable Type and getting a feel and we’ll see how long it lasts. In the past, I’ve had similar short runs of Blosxom, WordPress, and a few experiments in creating my very own brand of blogging software for kicks and giggles. Drupal has outlasted them all. If I decided I don’t like this experiment, I may go back.

Movable Type is written primarily in Perl. I like Perl and think it beats the pants off of PHP as far as languages go. Drupal is written in PHP. I have written some plugins for Drupal in PHP, but I’d rather be writing Perl. So, if I’m going to dink around with something at home for fun, I figure it ought to be in something I enjoy.

By using Movable Type, I have been able to vastly improve the performance of my blogs. I host on Dreamhost, which is great for the price, but performance is an issue from time to time since it is a shared hosting solution. Movable Type publishes most of my pages as static HTML (the ultimate page caching system!) and then updates them as needed when new posts or comments are added. My web site is much zippier now since it doesn’t have to what for PHP (or even Perl CGI/FastCGI) to load for every page.

Movable Type does blogging with a lot of subtlety and nuance. If you look at how everything works, there are a lot of teeny details that it does just right. Drupal does general content management pretty well, but blogging is done just alright. For example, Movable Type provides Atom and RSS feeds out of the box, Drupal provides just RSS unless you add a module on. Movable Type provides RDF and other summaries as part of every post to make it easier for bots to understand my site. The widgets (similar to Drupal’s blocks) are geared towards blogging and seem to make more sense overall. It’s a tool tailored to the specific job rather than a tool tailored to a larger job. I’m not at all knocking Drupal. But even a really great Swiss Army knife isn’t the best screwdriver, it just has a pretty decent one available.

Those are the main reasons for moving to Movable Type. Now, on to how to do it.

Migration from Drupal to Movable Type

Most of the information I was able to find on moving away from one platform to another was, unsurprisingly, Movable Type to Drupal. I found some folks looking or information on going the other way, but nothing definite. I’m going to explain the process I used in hopes that someone else might find it useful if they decide to make a similar move.

Database Conversion

The first and most complicated problem is moving the data. Movable Type helped to simplify this process by providing a standardized import tool. If you go into the back-end MT interface and look on the home page, there’s a link in the right sidebar labeled “Import Content”. If you follow that you can see a form that lets you import data in Movable Type import format or WordPress Import Format. Since I couldn’t find any information on the WordPress format and since I didn’t use WordPress, I left that alone.

The documentation for the Movable Type format is readily available here. That’s not actually the whole story, though. I discovered, by experimenting with the data export function, that there are some additional fields, such as “TAGS,” which allow you to import tags and such as well. Getting the Drupal data out was relatively easy since this is a very simple text format.

I have a little trick I do whenever I need to perform a one-time script on a Drupal instance. The trick is that I write a PHP script that does whatever I need, go into Drupal to create a new page, paste the code into that page, and then switch the input format to PHP. Then, instead of creating the page, I click “Preview,” which executes the script and does whatever I want. I wrote such a script for this occasion.

Here it is:

<?php
header('Content-type: text/plain');
$nids = db_query("select distinct n.nid from {node} n inner join {term_node} t on t.nid = n.nid where t.tid = 360 or t.tid = 362");
while ($node = node_load(db_fetch_object($nids))) {
  print "AUTHOR: ".($node->name=="Andrew Sterling Hanenkamp"?"zostay":$node->name)."\n";
  print "TITLE: ".$node->title."\n";
  print "STATUS: ".($node->status?"Publish":"Draft")."\n";
  print "ALLOW COMMENTS: ".($node->comment>0)."\n";
  print "CONVERT BREAKS: 1\n";
  print "ALLOW PINGS: ".($node->comment>0)."\n";
  print "DATE: ".strftime("%m/%d/%Y %I:%M:%S %p", $node->created)."\n";
  $tag = array();
  foreach ($node->category as $category) {
    $tag[] = $category->title;
  }
  print "TAGS: ".implode(",", $tag)."\n";
  print "-----\n";
  print "BODY:\n";
  print $node->body."\n";
  print "-----\n";
  print "KEYWORDS:\n";
  print "/node/".$node->nid.",".url('node/'.$node->nid)."\n";
  $comments = db_query("select * from {comments} c where c.nid = %d", $node->nid);
  while ($comment = db_fetch_object($comments)) {
    print "-----\n";
    print "COMMENT:\n";
    print "AUTHOR: ".$comment->name."\n";
    print "EMAIL: ".$comment->mail."\n";
    print "IP: ".$comment->hostname."\n";
    print "URL: ".$comment->homepage."\n";
    print "DATE: ".strftime("%m/%d/%Y %I:%M:%S %p", $comment->timestamp)."\n";
    print "SUBJECT: ".$comment->subject."\n";
    print $comment->comment."\n";
  }
  print "--------\n";
}
exit;
?>

This script takes all the nodes belonging to the categories with IDs 360 or 362 and outputs the files in the Movable Type export format. If you change the first query to use a different set of terms or remove them you can export everything or some different subset. I did this because I had 4 web sites served from one Drupal database and the terms determined which site they appeared on. I then condensed two sites together, which is which site this is.

The export script applies all terms as the tags. The script could be customized to narrow that down for your site, if needed.

The script had two basic flaws, which I fixed after the export and reimport (rather than doing everything over again). First, the “SUBJECT:” line from the comments didn’t look as good as I wanted. I’d recommend changing that to “Subject:” or removing it all together. I used the Search and Replace feature of Movable Type to correct it. The script also added “————” at the end of some of the comments, which I also removed using Search and Replace.

The other nice thing I did here (at least for me) is that I added the main “node/123” URL and also the main URL alias to the keywords. Which I take advantage of next.

Fixing up the URLs

Drupal provides URLs as just the identifier into the node table, like “node/123”. I had used some modules (i.e., “pathauto” and “URL Alias”) to provide nicer URLs that looked like “2008/03/30/drupal-to-movable-type”. Movable Type provides URLs as “2008/03/drupaltomovable_type”. This works well to allow me to provide some redirects to the new URLs. To do this, I’ve imported the URLs as keywords (as mentioned above) and then added the following lines to my .htaccess files in my new blog sites:

RewriteEngine on
RewriteRule ^(node/\d+)       /mt/mt-search.cgi?search=$1 [R,L]
RewriteRule ^(\d+/\d+/\d+/.*) /mt/mt-search.cgi?search=$1 [R,L]

Now, whenever someone hits one of the old links, they are redirected to a search, which will find the post they were supposed to hit. This worked out pretty well.

Moving the RSS Feeds

In the process of moving to the new server, my RSS feeds changed names. I had five different feeds. One for each site and then one that summarized all of them. I’ve decided to start using FeedBurner to host my feeds. So, after I switched to Movable Type and setup my feeds, I then setup more rules in the .htaccess file to redirect anyone who might be reading my feeds.

RewriteRule ^rss.xml            http://feeds.feedburner.com/TildeSterling [R=301,L]
RewriteRule ^frontpage/feed     http://feeds.feedburner.com/AndrewSterlingHanenkamp [R=301,L]
RewriteRule ^contentment/feed   http://feeds.feedburner.com/Contentment [R=301,L]
RewriteRule ^openscripture/feed http://feeds.feedburner.com/AndrewSterlingHanenkamp [R=301,L]
RewriteRule ^gabe/feed          http://feeds.feedburner.com/GabrielScottHanenkamp [R=301,L]

Now, all my feeds are permanently redirected to their new locations at FeedBurner.

The Design

The rest was the design work. I’m not going into detail here. Just look at it. I did it. That’s me and my shoddy Gimp/Photoshop work.

That’s It

That’s pretty much it. I’ve got some other stuff I’m planning, but that will probably take months more to get there.

This is the list of all tags I've used in my blog to those point (this includes or will include tags used in content aggregated from other sites as well).

Categories

$container_id = 181;
$sql =
"SELECT d.nid, d.title, c.description, ".
" MAX(n.created) AS updated, ".
" COUNT(*) AS count, ".
" SUM(2.5/LOG(0.25*((UNIX_TIMESTAMP()-n.created)/2592000)+1.5)-1) AS score ".
"FROM {node} d ".
" INNER JOIN {category} c ON c.cid = d.nid ".
" INNER JOIN {category_node} cn ON cn.cid = c.cid ".
" INNER JOIN {node} n ON n.nid = cn.nid ".
"WHERE c.cnid = %d AND n.status = 1 ".
"GROUP BY d.nid, d.title, c.description ".
"ORDER BY score DESC";

$count_sql = "SELECT COUNT(*) FROM {category} c WHERE c.cnid = $container_id";

$result = pager_query($sql, 30, 0, $count_sql, $container_id);

while ($category = db_fetch_object($result)) {
$items[] = array(
l($category->title, 'node/'. $category->nid,
array('title' => $category->description)),
$category->count,
round($category->score, 1),
t('%time ago', array('%time' => format_interval(time() - $category->updated, 3))),
);
}

print theme('table', array('tag', 'count', 'score', 'last update'), $items);
print theme('pager');
?>

How it works

The list is sorted in order of popularity using an inverse-logarithm scoring method I developed myself. Basically, the most recently used terms get a very high score that tapers off to a relatively low score once the post is a year old or older. Each use of a term is cumulative so the more commonly used terms will appear higher on the list even if they haven't been used as recently as others. This list will sort itself as time goes on according to whatever my latest posts are about.

Originally, when I used the built-in taxonomy module of Drupal, I used this SQL query to do the calculation:

SELECT d.tid, d.name, d.description, 
    MAX(n.created) AS updated, 
    COUNT(*) AS count, 
    SUM(2.5/LOG(0.25*((UNIX_TIMESTAMP() 
        - n.created)/2592000)+1.5)-1) AS score 
FROM {term_data} d 
    INNER JOIN term_node USING (tid) 
    INNER JOIN node n USING (nid) 
WHERE d.vid = ? 
    AND n.status = 1 
GROUP BY d.tid, d.name, d.description 
ORDER BY score DESC

However, since I recently switched to the Category module, I updated the SQL to reflect this with:

SELECT d.nid, d.title, c.description,
       MAX(n.created) AS updated,
       COUNT(*) AS count,
       SUM(2.5/LOG(0.25*((UNIX_TIMESTAMP()-n.created)/2592000)+1.5)-1) AS score
FROM {node} d
     INNER JOIN {category} c ON c.cid = d.nid
     INNER JOIN {category_node} cn ON cn.cid = c.cid
     INNER JOIN {node} n ON n.nid = cn.nid
WHERE c.cnid = ? AND n.status = 1
GROUP BY d.nid, d.title, c.description
ORDER BY score DESC

Here's the function describing the math involved. I developed this function for calculating the score by arbitrarily modifying the log curve to suit my needs.

In the equation, s represents the final score, n is the number of terms and i is the iterator over the terms. The m_i represents the number of months since the creation of the post the ith use of the term belongs to. Since Drupal stores time in seconds since the epoch (i.e., January 1, 1970), the current time is calculated by subtracting the node's creation time from the current time and then dividing by 2,592,000, which is the number of seconds in a month.

If you view a curve plot for an individual iteration, you would note that when the delta (m_i) is 0, the score for that term will be around 5.2. When the delta is 12, the score is about 1.