After over a year of development, Drupal 4.7 is finally out the door. Woo! I started poking around in the Drupal world during the 4.5 days, and I saw (from a distance) the release of 4.6. As I've learned more about the system and become more involved in the community, I've realized what a significant milestone 4.7 really is. A few of the improvements that have really excited me include:

• Much improved installation process for third-party modules (copy the files over, enable them in the control panel, and they install themselves)
• A tremendously powerful API for creating and rendering forms, the meat-and-potatos of the web app world. This part of the 4.7 upgrade is one of the reasons it took much longer than expected, but the wait is worth it.
• PHPTemplate, a powerful and robust theming engine, is now part of the Drupal core
• Numerous performance improvements in areas like caching, path aliasing, and so on

Work on 4.8 is just beginning, and the general consensus in the development community is that it should focus on polishing and refining the Drupal user experience, particular for administration tasks. 4.7 provides a host of new tools we can use to that end, and the next several months should be very exciting.

Congratulations, everyone!

One of the topics that comes up occasionally on Drupal.org is the idea of comments as nodes. For those who haven't dug into the system's innards, almost all content on a Drupal site is stored as a 'node.' Different kinds of nodes (images, quotations, blog entries, weblinks, etc.) can have additional data fields and different presentaiton, but they all get to leverage the same core infrastructure of the node api. Comments, on the other hand, are a special kind of content that sits in its own separate table. Adding features to the commenting system is, by nature, a lot more difficult and only really possible with Drupal 4.7's enhanced form building API.

A good case can be made for keeping comments separate. On sites with huge numbers of comments, it speeds up queries for the code nodes table. It's also relatively join-friendly, as making comments into a type of node would still require an auxiliary 'comment_data' table, or somesuch. On really high-traffic sites, that's a pain and a half: pulling data from the node table and joining to it is already one of the things Drupal does most often, and anything that slows it down unecessarily is bad.

That said, though, I think that it could still be a noble goal. Certain core functions (say, file attachments, saving edits in revisions, and so on) would come 'for free' if nodes were used for commenting. Three specific cases come to mind where wthis would be useful. The first is the 'microfiction' content type I created a while back for Drupal 4.6. The name was a misnomer -- in reality it was a microfiction contest -- the comments on the node were the actual stories. I would've much rather seen each response be treated as an actual node. Another is Something Awful style photoshop contests. Instead of simply posting images inline in comments, each comment could be an actual image node -- automatically added to the appropriate gallery. And finally, the forum module that works well but gets so many complaints. With comments as nodes, the ability to split a discussion thread -- taking one comment and its children and promoting it to a full-fledged new topic -- would be rather more straightforward. Promoting great comments to the front page -- just as one would a node -- would also be simple.

If the system were flexible enough to utilize different kinds of nodes as comments, things like photoshop image contests, audio remixing contests, etc would all be much easier. In addition to turning comments on or off for a given node type, one could choose between the 'default' comment type, images, audio, etc. A while back, javanaut's interesting relativity module supported something kind of like that. Administrators could configure parent/child relationships between different kinds of nodes, even enforcing cardinality in some cases. That might be going a bit far, but it would be cool -- very cool -- to support that sort of relationship in commenting.

Naturally, ideas like this get little interest with Drupal 4.7's release around the corner and much bug-hunting still going on. But the groundwork that 4.7 lays make this feature a lot more feasible. The Forms API makes 'remixing' content creation forms a piece of cake, for example. The more I think about it, the higher this rises on my list of personal projects for 2006.

About six months ago I decided that I really, really wanted a centralized API for voting and content-rating systems in Drupal.

After a few rounds of trial-and-error revisioning, VotingAPI was born. It's been encouraging to see people pick it up and utilize it to create a variety of new tools for Drupal sites.

Additional development has been sponsored by Code0range, and while there have been a few delays due to my schedule, the results are exciting. They needed a 4.7 upgrade of the Node Moderation module, which allows users to collaboratively decide what content should be put on the front page. The new version uses VotingAPI for its backend data storage. In addition, they're sponsoring the completion of a new more flexible voting system that allows administrators to moderate different node types, comments, and so on indepdendently. That part of the project has driven a lot of enhancements to the underlying VotingAPI itself.

For each individual VotingAPI-based project, the benefits are marginal. There's a shared API, so some operations (like casting votes and retrieving them) don't have to be written by the module author. The real advantage comes as the shared data is leveraged by presentation and workflow modules. Integration with Earl Miles' essential Views module is in place, though there's still work to be done to make it cleaner and easier for admins to customize. The Voting Actions module also ties it into the Actions API. Vote results can now drive anything from node promotion, emails, etc.

Working with the various developers who've been using VotingAPI has definitely been an educational process. Things I thought were unecessary turned out to be anything but, and stuff I thought would be essential for the API have been overkill in other areas. For the most part, though, I think it's going well. The next steps are laid out pretty clearly, I think.

1. Integrate Voting Actions directly into the APIThere's already an issue in the queue for this change, and some of the preliminary coding has already been done. The API needs to allow modules to expose arbitrarily complex sets of filtering criteria, and actions to take if the criteria are met. (For example, promoting images to the front page if enough people like them, but archiving them in a 'cult hits' gallery if only a few people REALLY love them.) There needs to be a user interfae for advanced admins to tweak these criteria and action sets, as well as a convenient way of saving, importing, and exporting them. Basically, it needs a UI that's less CRUD and more like the Views UI.
2. Improve the Views integrationSpeaking of Views, VotingAPI already supports basic integration. A few key patches to views have already gone in to enable custom tabs on node screens, listing the votes each user has cast on a piece of content. Creating complex listings of voted content, though, is still difficult -- especially if a user has multiple voting modules installed and operating on different kinds of content side by side. The reasons it's tricky are wrapped up in the innards of VotingAPI's internal data storage and Views' way or representing filter criteria, but when the problem is solved it should get much easier to create things like 'top rated nodes' lists, lists of votes that a particular user has cast, and so on.
3. More hooks for displayRight now, most of the intelligence in VotingAPI revolves around calculation. It has default ways of aggregating data and saving it, and modules can override them as necessary. With Views module doing more heavy lifting for data display, though, it's necessary for modules to override how vote data and vote results are displayed as well. There's preliminary support for this in the SimpleVote module, and some more complex support in the under-development CustomVote code.
4. Better documentationNo, scratch that -- SOME documentation. Earl Miles has done a bang-up job documenting the complexities of Views module in his new series of handbook pages. The very simple uses of VotingAPI are straightforward, but once you get into he subtleties of multiple criteria, custom calculations, etc, it can get confusing.
5. More work on separating Presentation, Calculations, and ActionsThis has already received some attention at a thread on Drupal.org but it'll be important for the future. A 'voting construction kit' that lets users mix and match UI widgets (five star voting? a slider?) with custom calculations (the median of all votes weighted by the age of the vote?) with custom presentation (Bar graph of vote values? Chart of votes over time?) and so on would be awesome. It's hard to combine that sort of flexibility with an easy to use UI, though -- some features will always have to be the domain of experienced customization freaks or PHP programmers.
6. Keep it fast Working on the CustomVote module, I realized how easy it is to churn the database if I'm not careful when doing vote aggregation, etc. Thinking through effective caching mechanisms -- especially when viewing pages that display tabular voting results -- is essential. The API itself is pretty clean as far as the DB is concerned, but additional support for caching wouldn't hurt.

Whew. I wasn't expecting to write THAT much, but it feels good to get it down and on record. Hopefully additional updates over the next couple of weeks will have good news for progress on these points.

As Drupal has grown in popularity, more and more people have started using it for consulting work, and more people are beginning to evaluate Drupal-based business models. Since Drupal is released under the GPL software license, all derivitive software (including plugin modules) is also covered under the GPL. There's a lot of confusion over what's kosher and what's not. I've spent the last couple of days chatting with some of the core Drupal developers and reading up on the GPL license. I think I've figured out the basics -- My words are not canon, and anyone is free to correct me if I am factually incorrect...

The following are automatically released under the GPL license:

1. Any files checked into the Drupal CVS repository
2. Any Drupal modules, even if they are not checked into the CVS
3. CODE from Drupal themes (ie, the page.tpl.php, template.php, and so on)

As long as they are not checked into the Drupal CVS repositiry, the following are NOT automatically released under the GPL license:

1. CSS files, image files, media files, and any other non-PHP files
2. Programs (including PHP code) that do not directly interface with Drupal. (For example, the TinyMCE text editor project is a separate program, with a Drupal module that 'connects' it to Drupal. The module is automatically GPL'd, but the TinyMCE project is not.)

Now, what does it mean to say that something is 'released under the GPL license?' There are a lot of lawyer-ish terms to throw around, but after much discussion, it seems to boil down to this:

1. You can charge a client for your work that is released under the GPL.
2. You can host GPL-licensed stuff on your web site and charge people for downloading it.
3. You cannot control what someone ELSE does with software released under the GPL once they have it. (This is because anyone who has a copy of GPL-covered software has the right to modify it or give it away to anyone else.).
4. You CAN control, via licenses or agreements or what not, stuff that isn't covered by GPL. (images, css layouts, media, and 'helper' programs like TinyMCE that don't directly connect to Drupal). This distinction is important.

What it boils down to is this. If you want to make money off of your work with Drupal, there are a couple of options:

1. Charge consulting clients for the time it takes you to do the work
2. Petition the community for 'sponsors' if you think a project would be valuable, but can't afford to donate the time (this is sometimes called a reverse auction)
3. Sell your Drupal modules and themes. After people purchase them, they can give away *anything that's covered by the GPL.* You, though, don't have to give technical support or accept feature requests from those who didn't pay.
4. Write your custom software as an external program (in PHP or some other language) that can be used without Drupal. Then write a simple module that 'connects' it to Drupal. The module is GPL'd, but people can't distribute the separate program without your permission.
5. For themes, the basic template files and php files will be GPL'd. Anyone who has your theme can hand those files to their friends willy-nilly, but they must still obtain your permission to redistribute the images, media, and CSS files that make your theme distinctive.

I may be wrong -- if I've gotten the facts wrong (not 'what you would like to be different,' just the facts), please correct me.

Well, it certainly has been an interesting weekend. After some turkey, some excellent casserole, and a relaxing several days with Catherine, I hammered out a new revision of the VotingAPI module that matches the brainstorming from an earlier post. The idea is NOT to provide a UI for voting, or to provide an infrastructure for 'comparing' one piece of content to another. Rather, it's an attempt to unify how various voting systems record and tabulate votes so they can be combined in cleaner ways.

Next steps include converting latestgreatest.module over to it, and looking at the feasibility of converting other voting modules over to it. I'm pretty sure that the aggregate retrieval functions will need a lot of work to streamline them, but so I'm happy with where it's at so far.

So, I mentioned in my previously posted Drupal To-Do list that I wanted to whip up an Amazon 'reading list' module for single and multi-user blogs. Thanks to the holiday weekend, I put some time into it and a rough initial version is now available in the Drupal.org repository.

I want to get a few things cleaned up before I put an official project page up for it in particular, the theming leaves something to be desired and I want more options for configuring and displaying the 'states' a given book can be in. It also requires a patched version of amazon.module for Drupal 4.7... something I figure most people aren't using at the moment.

Still... progress!

Well, my previous post outlined a laundry list of stuff I'd like to do in Drupal. Last night, while waiting for a data import to finish at work, I started thinking again about the schema that's necessary for voting to work cleanly. Originally, the idea was to include all sorts of cool aggregation and averaging features for free, so that modules could drop a vote in the box and get back the 'average rating' for a node with no effort.

After talking things over with the authors of various voting modules, though, I think that might be a mistake. There are too many ways to figure out what a vote 'means', and too many possibly conflicting methods for producing a final valuation, to centralize that process. An example is a poetry critique site in which every conceptual 'vote' is really three discrete votes on meter, imagery, and originality. Or a commentary site where each vote isn't a 1-5 quality rating, but a thumbs up or thumbs down rating with an implicit 'reason'.

How to map the 'simple' voting scenerios to the same API as the complicated multicriteria ones? How, oh how? What we really need is a way of identifying not only what is being voted on, who is voting, and what the value of the vote is... but also what kind of vote it is. With that in mind, I'm working on the following:

#
# Table structure for table 'votingapi'
#

CREATE TABLE votes (
voting_id int(10) unsigned NOT NULL auto_increment,
content_type varchar(20) default NULL,
content_id int(10) unsigned default NULL,
vote int(10) unsigned default NULL,
vote_type varchar(128) default NULL,
uid int(10) unsigned default '0',
timestamp int(11) default NULL,
hostname varchar(128) default '0',
PRIMARY KEY (voting_id)
) TYPE=MyISAM;

As with the present votingapi implementation, votes will be cast by calling a simple votingapi_cast_vote() function with content information, and the vote_type parameter. The votingapi_get_votes() function will pull back votes data grouped by vote_type and grouped by aggregation function. Instead of a single 'vote' property, a nested set of arrays will contain the slice-and-dice data necessary for proper display.

In addition, it allows multi-criteria votes to be stored as discrete votes but retrieved as a group. In the poetry site above, for example, three actual votes would be cast. The vote_types would be set to 'poetry-meter', 'poetry-imagery', and 'poetry-originality', respectively. On retrieval, the raw vote objects for a given piece of content would be pulled from the DB, and assembled into a vote object. Individual vote_types would become keys. In addition, a votingapi hook would allow other modules to add their own custom calculations into the mix (for example, a median vote value). Some examples:

$content-type = 'node-' . $node->type;
$content-id = $node->nid;
$votes = votingapi_get_votes($content-type, $content-id);

// a particular user's rating of the poem's meter
// inserted by votingapi automatically, from the raw
// entry in the votes table
$votes[$uid]['poetry-meter']

// the median rating of the poetry's imagery
// calculated by poetry.module
$votes['median']['poetry-imagery']

// the highest rating cast for the poem's originality
// calculated by poetry.module
$votes['highest']['poetry-originality']

// the total number of users who voted to promote
// the poem to the front page, calculated by queue.module
$votes['sum']['promote']

There are still some hitches to work out, but I'm pondering and I think it has some potential. Code to come shortly.

I've spent the last couple of months working seriously with Drupal, pushing through a few pet patches and trying to help out with others that came down the pipe. I'm already looking forward to 2006, though, and the projects I'd like to tackle. In no particular order, my list of Stuff I Want To Implement Or See Implemented In Drupal.

Generalized voting infrastructure

I've already put a bit of work into this, but I'm hoping to hammer out ideas with the developers of other node voting modules before moving much more. There seems to be consensus around the idea that we need flexible multicriteria voting, but I'm trying to figure out what the data footprint would be in a system with lots of nodes, lots of votes, and multiple modules all trying to implement different kinds of voting data in the same tablespace. Perhaps the answer is that it's impossible -- like the node security system currently in place, only ONE voting implementation could be active at a given time.

Reading List module

This one would let every user on a Drupal site maintain a list of Amazon ASINs in various states: Want, Waiting To Read, Reading, Finished, Discarded. They'd be able to keep notes on their thoughts on the book in its current state ("Really looking forward to starting this one, as I've enjoyed previous books by the author..." and so on) or create a full Amazon node as a review upon completion. User-specific or site-wide reading lists, and "currently reading" sidebar blocks, would be easy to generate from this data. It's a must-have for quite a few bloggers. The excellent Amazon Tools module already handles a lot of the backend work for this.

Relationship management system

Druapl's built-in taxonomy system handles classification of information pretty well, but it manages implicit relationships between pieces of content, not explicit ones, and that's very important distinction. Image.module's gallery functions and the inherent limitations of Forum.module demonstrate the drawbacks, I think. It's a testament to Taxonomy's strengths that it can be made to do those things, but solid relationship management infrastructure would be much, much better. I don't think I'm the person to design this system (OK, I KNOW I'm not), so it's exciting to see other smarter people tackling the problem.

Sparklines Filter

Sparklines are a cool concept in data presentation. They imbed a tiny, tiny micro-graph of small data set right in the text that you're writing. The idea is to take a small set of points and (United States yearly unemployment over the past decade, for example, is just 10 numbers) and turn the set into an instantly grokable icon. Ideally, they should be no taller than the line of text in which they're imbedded. There's already a PHP library for generating them, and someone discussed the possbility of Druapl integration months ago. Clean syntax for generating and imbedding a Sparkline is the real need -- perhaps something like [barchart:1.3,2.1,3,1.9] for a simple example? I don't know. There's enough flexibility in the Sparklines library that exposing all the options could result in option overload.

Comic.module rewrite

Ah, Comic.module. The first project I really tackled for Drupal. The idea was great -- create a module for posting and navigating strips in a webcomic. The implementation, sadly, was lacking. I broke more coding conventions than I followed and never implemented the features I really wanted: clean organization of issues, storylines, and so on. Once a generalized relationship API solidifies, I'd like to use it as the basis of Comic Module 2.0. Support for multiple comic strips per drupal site would be a nice bonus.

Node-based comments

This one's controversial in the Drupal community, mostly because comments have been around in their current form for quite a while. Changing the system raises important questions about performance and compatability. I think the benefits of a well-implemented shift would be considerable, though. All features available to nodes -- revisions, file attachments, path aliasing, etc -- would be immediately available. It'd be a huge project, but it's soooooo tempting.

Microfiction.module rewrite

Eons ago, when I started http://www.ferretshock.com, I ran a weekly microfiction contest. All entries had to be exactly 100 words long. Unfortunately, that meant people had to hand-count their entries before posting them on a PHPBB message board. When I started reimplimenting things in Drupal recently, I decided to add that validation automatically. This, of course, led to the realization that no hooks existed in comment.module for validation. Which led to my pushing a core patch. Which led to my getting involved in the first wave of the great FormAPI migration. Years from now, old men will sings songs of that adventure. Anyhow.

The new version of Microfiction would be generalized as a 'contest' module, with optional rules for each type of contest. Photo contests would require an image file attachment. Writing contests would have variable wordcount requirements. Caption-the-photo contests would hide only allow a single line of text to be entered. And so on. A ratings system and 'winner hilighting' would be built in.

Timeline.module

This one's a bit of a holy grail. Way back in the day, I set up http://www.havana-mod.com as a place to organize thoughts on the graphic novel and RPG setting I was trying to assemble. What I really needed was a soup of interconnected data -- a Who's Who list, an encyclopedia, and a timeline combined into a big soup of data. I have some ideas about how this would be implemented, but it too relies on the generalized relationship module that I mentioned above.

Holy Cow, That's A Long List

Yikes. it's a little unsettling looking at how much work is involved in those ideas. Still, it's a gameplan for my personal hacking in 2006... hopefully I'll be able to accomplish some of it.

Those who know me know that I am official consumer of the kool-aid when it comes to Drupal. It is, after all, the best CMS on the market. There have been quite a few threads on the Drupal site, though, discussing the dearth of good themes available for folks who want a swanky looking site. Those who dive in and try to build their own without considerable experience tend to run screaming. Or, if they persevere, end up with a slightly modified version of one of the core themes like Grey_Box. (For this blog, I just grabbed one of the more attractive themes and haven't bothered customizing it. Rolling my own theme is on the to-do list, but what a loooong list it is...) Anyhow, I spent some time looking around at some of the highly visible "showcase" themes for blogging software like WordPress and Movable Type.

What I started to notice was a trend: the themes that got lots of attention, the ones that rise above the crowd, fall into two rough categories. Either they're highly customized for a particular approach (a photo-blog organized by photographer, for example), and thus unsuitable for general use, or they focus on flexibility and compatability with third-party plugins. In the world of Movable Type and WordPress, it's a foregone conclusion that connecting your blog to your de.licio.us favorites or adding a 'quote of the day' in the sidebar will involve wading deep into templating languages and perhaps Perl or PHP.

One of the snappier WordPress themes, K2, boasts features like "compatability with multiple authors" and "Showing the latest comments." A lot of work has gone into making K2 flexible enough to deal with more than one or two configuration scenerios, and the work shows.

What really struck me, though, was that any Drupal theme released without that sort of flexibility would be sent back to the drawing board as broken. New modules are released almost daily, shoving new content types and new presentation paradigms and new tools and options into the Drupal framework. The Drupal themes one can download from the main site take them in stride thanks to the tremendously flexible PHPTemplate system that (almost) everyone now uses.

I've thrown together quickie themes for Drupal that make all sorts of assumptions (we'll never had sidebars, we don't need to display comments, only Module X and Module Y will ever be installed...) and it's as easy as any other system. Easier, in some cases. In about twenty minutes , I whipped up a very rough 'compatability theme' that lets a Drupal site use CSS skins originally designed for Movable Type blogs. But making a theme that's visually stunning, cleanly designed, AND flexible enough to handle anything J. Random User installs into their Drupal build can be a pretty daunting task.

I suppose this post is both a defense of Drupal's relative deart of themes and a challenge. Should we simply resign ourselves to the fact that Drupal installs will either use simple CSS skins, or elaborate site-specific layouts? How can we, the Drupal die-hards, figure out ways to make the 'middle ground' more approachable?

A couple of you might have noticed that there's now a sidebar of google ads. I'm tinkering around with a few of the drupal modules that handle google integration, and wanted to see how that one works out. I might take them down eventually, but for now they don't seem too intrusive. If you're annoyed by them, logging in on this site with a livejournal or blogger username will make them disappear -- only anonymous visitors see them.