Why, of course...
Asking a craftsman what his tools are capable of can be a bit deceptive. The answer is likely to be very impressive -- and not terribly useful. Why? Oftentimes, the unspoken question is really "What can I, a relative newcomer, hope to accomplish with the set of tools that you are familiar with?" Unless you're willing to spent a large chunk of time becoming just as proficient with those tools, the answer to that question is often dramatically different.
When it comes to assessing the capabilities of a particular piece of software, this problem comes up quite a bit. A computer program, or a web app, or (deep breath, now) a CMS is rarely as straightforward as a hammer or a table saw or a chisel. So the insights of resident experts is often essential when evaluating the capabilities of a product. We've all met folks who make their favorite app jump up, roll over, make coffee, and cure cancer. But just because someone who spends 8 hours a day in MS Word can use it as a spreadsheet doesn't mean that it should be used that way. More importantly, it doesn't mean that a person who doesn't spend 8 hours a day immersed in that app would be comfortable doing the same thing.
I see this up close every day when working with Drupal. Don't get me wrong -- I love Drupal, and the folks I hang out with and work with can make it do some incredibly crazy things. Drupal's flexible APIs make this problem particularly seductive. In many web apps, adding major new functionality requires patching the core package, threatening forward compatibility. In Drupal, it's almost always possible to leverage intimate knowledge of the APIs to create a clean plugin module that alters the system's behavior. Matt Westgate of Lullabot once made a Drupal site that completely bypassed the entire 'node' system (one of Drupal's core conceptual building blocks, about as central as 'paragraphs' in a word processor). Steven Wittens recently spent a long weekend writing the Drupal-powered web app ComicJuice. A long weekend. He came up with the idea, lost a night of sleep hacking, and finished it up within a few days.
Tools like CCK, Views, and Workflow make it possible for beginners to build really complex and powerful systems in a very short amount of time. There's no denying it. But once you wander off the beaten path of 'what another developer built a friendly UI for,' it will always stop being simple. For the hardcore devs who spend their time immersed in Drupal every day, that distinction is less visible. We're in there building those tools, and in most cases making them user friendly is actually more work. From that perspective, writing a quick hook_form_alter() implementation or whipping up a quick CCK formatter is 'totally simple.'
Is there an easy solution to this problem? I'm not sure, to be honest. It's certainly not Drupal specific. I've seen the same thing in other OSS communities (and obviously in proprietary software) where the 'checked off feature on the comparison table' doesn't necessarily mean that anyone without a PhD can actually USE a particular piece of functionality. It's possible to use Wordpress as a full-fledged content management system. It's possible to build a turn-based strategy game in Drupal. It's possible to run a community collaboration site with Movable Type, and a personal blog in Typo3. But just because the experts in those fields can do it doesn't mean that the newcomers can, too. And it definitely doesn't mean that the newcomers will be able to build those systems in ways that will scale well or grow in the future.
What's the solution to this problem? I'm not sure, to be honest. More self-awareness? Less defensiveness? A one-to-five level difficulty rating that we can assign to answers? Often, it's hard to judge how experienced someone is when asking a question.
Does anyone have any thoughts on this?
Is this a solvable problem?
Is this a solvable problem? In specific cases yes, but more generally, this type of problem is a moving target. Drupal gets better, people do more, and thus it's a never-ending cycle of development and documentation.
Recently, I tried to write some Apple Script which need some sort of extra library. I tried to steamroll the task in 15 mins like any other Mac feature I'm learning. I've scripted a few languages, so surely I can write this simple app. I failed badly. Now, Person A dismisses Apple as having "usability issues" (which, relatively speaking, is absurd), Person B says I am a fool to think I can do what I was trying to do in 15 mins.
I prefer Person B's take on things.
So, to try and draw this together, I work to make Drupal easy and accessible - but that doesn't mean I have sympathy for people who demand to understand the bowels of Drupal in 2 days.
Here's one suggestion
Much of Drupal's complexity stems from the vast library of modules. Some are wonderful works of art, with solid programming and good support. Other modules are less comprehensive, lacking in documentation, and with questionable code.
What Drupal needs desperately is a module ranking system, similar to Joomla's excellent extension library, where users can rate modules and write reviews on the module's strengths and weaknesses.
In that way, the community will apply subtle pressure on developers with a system that will reward good work and thoughtful follow through.
The whole concept of open source is to get the community involved in the future development of software. What better way to accomplish this than through a rating and review system?
re: one suggestion
I think the idea of a module rating system is a great one.
I literally just spent about 8 hours trying to get a module to work -- essentially, do what it seemed to say it would do in the documentation -- only to find out it did not. Searching the forums, I found others had run into this too with this particular module. Lesson to me.
It would have been great to see rankings and stars on that module before I started ... potentially warning me off.
Don't get me wrong. I love drupal -- and I appreciate all the hard work of both the core developers and those who work on contributed modules. Maybe someday I'll know enough to contribute a module too!
However, this kind of usability improvement would help a lot for people like myself ... non-web designers who are getting into drupal to accomplish a specific task.
This IS important.
Right now, probably the best metric available is, 'How many issues are sitting in the issue queue for a module, and how quickly do they turn around?' It's not as easy as a star rating, but it does help.
There have been a number of attempts to get voting into the drupal.org project system. During the 4.7 release cycle I even added support for five-star multiple-criteria ratings to project.module, though it was never committed to the core project. A frequent objection/concern is that people are notoriously bad at rating things reliably. One person may give a module five stars because it sounds really cool, while another gives it one star because he didn't understand how to configure it.
Give people space to write 'reviews' of a module and the reviews devolve into an ad-hoc issue queue for those who don't know how to post correct bug reports. I don't mean to be snarky, just reflecting on some of the things that have frequently been encountered.
I'm leaning more and more towards a simple five star scale that's weighted by release -- heaviest weight given to ratings on the current release version.
Well, this smells of a lack of confidence in group intelligence
Not aiming this at you, but at the individuals who would object to a rating system. The entire concept of open source is predicated on the idea that group intelligence overcomes the occasional misbehavior of a few select individuals. So what if there is an occasional unjust review? The well-thought-out reviews will hold much greater weight anyway, and drown out the misbehaviors.
This is also why I'm such a big advocate of a voting system that truly works in Drupal, one that would allow us to develop a Karma rating for members. That way, the Drupal users who earn respect from their peers with their opinions, or contributions, would over time, earn greater weight in the community.
When developers tend to fall back on the age-old "excuse" that users can't be "trusted" to express their opinions, then the whole community falls apart. Fact is, most users of a software package are quite well qualified to rate that package on productivity and ease of use.
Any democratic process will have its black sheep, but humans have a natural ability to ignore the anomalies and absorb the intelligence of the masses. Just ask the founder of Wikipedia about this....
Misunderstandings
I think we've had a couple of misunderstandings, here. The issue is not one of trust in individual users, but rather a lack of trust in specific technical mechanisms for gathering user input. The question, 'What are we measuring?' is an important one, especially in areas like image handling where the field is crowded with modules that do different things in different ways.
Well thought out reviews are definitely good. And I don't want to suggest that I am opposed to them -- or that any developer I know of has objected to them on principle. Even scathing commentary is useful in many cases. Even reviews, though, are only useful if you have a way of judging the perspective of the writer. Are they a guru? A newcomer who's just stumbling through their first site setup? The problems mentioned in my original post all come back into play when reading the reviews.
Given those issues, with limited developer hours (a problem that is only exacerbated as the community of site-implementors grows faster than the community of code-writers), decisions have to be made. Is a 'review system' and an amazon-style infrastructure so much more useful than, say, the existing forums or the existing issue/bug tracking queue that it merits spending extended amounts of time implementing? In the past, the consensus has been 'no.' In part because the developers qualified to make those kinds of changes to the Drupal.org infrastructure had even more pressing things to work on.
Trusting users is one thing -- trusting software to properly capture the correct information is another. That's really the fundamental challenge of any opinion/value voting system. :-) Properly mapping and weighting karma, for example, is a deliriously tough problem domain. Everyone knows how THEY would do it, but almost everyone's perfect vision is different. Ultimately, the most efficient karma-mapping system we have is the use of unique usernames: our human wetware handles the +1 and -1 associations that go with someone we recognize, for better or worse. ;-)
Obviously, that's an oversimplification. And, yes, I agree that a simple rating mechanism would help sort out the obviously trash modules from the ones worth looking at. But that information won't solve the problems originally mentioned.
The simple question to ask, I suppose, is....
Are we better off with a ranking and review system, or not?
While any ranking system has its problems, the alternative (lack of one) may have more.
The module system sure needs something. If not a rating system, then what?
rating by release
Jeff, I like the rating-by-release idea. It would suck to get a lot of popularity for a new/unstable module, and then have no-one look at it ever again based on the early ratings.
Yeah, I pondered that a lot...
...after watching the TinyMCE debacle over the last month or two. It helps both the 'new and unstable, but maturing' modules -- and also helps people spot older and more mature modules that have recently suffered reliability drops.
User ratings can be made reliable ("enough").
Ref.
We can probably work out a questionaire at drupal.org:
- a user competence test - that anyone who wants to take part in rating modules have to pass in order to be given those privileges. The questions in that test will be aimed at making sure the person is "skilled enough" to make reviews, and could also contain a statement with examples of what to be aware of when rating.
...the solution
Hi Jeff,
To solve this problem, someone (or even a team of people with well organized and defined reviewing capabilities would be required to investigate a couple of dozen (or more) CMS's, 2 or 3 dozen development platforms, half a dozen languages, operating systems, (IDE's - from your podcast) etc etc and choose the exact correct fit for the experience, knowledge and intelligence of who will implement and support this solution.
For 99.9% of cases the review task is far too expensive (in terms of money, time and resources) for the actual task being solved.
My words of wisdom ... "if you only have a hammer, every problem starts to look like a nail ... which is ok as long as it works".
Technologies should be reviewed reasonably frequently to see if there are better hammers around.
People who know technologies such as Ruby on Rails (or Django and Turbo Gears for the Python fans!) might be able to produce complex websites faster in their preferred technology then gurus can in Drupal. You wouldn't know this of course until you became a guru!
If this problem (in general) is ever solved will we all be driving 1 of 3 types of cars(small/medium/large)?
I doubt it, I guess it comes down to "personal taste" ;)
(which means we'll always have dozens of choices and never really know if we choose the right one!)
Regards,
Ray Smith
http://RaymondSmith.com
True, dat...
This is something that we get a lot in the #drupal channel. Someone will come into the room, interested in migration information, and will ask if there are any hardcore drupal experts that are also knee-deep in Joomla! knowledge. Those folks are probably out there, but generally they're not hanging out in #drupal (or #joomla, for that matter).
I try to stay up to speed on what's going on with a variety of projects (the 'pop up for air and see if someone's made a better hammer' step in your comment). But it's tough, especially because achieving mastery in a NEW system requires spending a nontrivial stretch of time in the 'unproductive and frustrating' trough.
This issue is a bit off topic.....
But it relates back to the idea of a rating system. One of the big issues with Drupal and its developers centers around the identity and goals of the Drupal community.
Just exactly what are the goals of the Drupal community? Do we want to have a powerful CMS framework designed specifically for established web designers, or do we want to have an easy-to-use CMS system for untrained Internet newbie?
This message seems to get confused quite a bit. When Drupal developers brag about ease of use, or all the new Drupal members, or users of Drupal websites, it appears as if they're tailoring Drupal to become the next Wordpress.
But when those same developers complain about the lack of contribution of new members, or distrust and complain about unqualified people who might "rate" their work incorrectly, you begin to see the dichotomy that is developing.
If Drupal core developers truly want to attract the unwashed masses to their software, then they need to learn how to respect and listen to the viewpoints of said unwashed masses.
If Drupal programmers would prefer to develop a system by developers, FOR developers, then say so up front and try your best to run off the unqualified masses, so they don't waste their time.
I guess the simple question is: What does Drupal want to be?
Excellent question
I think it's obvious that most people in the Drupal community would say, 'No!' to that. It's not that the niche isn't there, or that there is any kind of antagonism. It's that tailoring the actual Drupal core system to that market would likely make it much LESS useful for those who need to use it for something more. It's certainly possible to make a site WITH it that's easy to use, but that's something different, I think.
I don't necessarily see this as conflicting with the desire to see it grow and become more user and developer friendly, but you're right in observing that the 'turnkey solution versus platform/framework' divide is one that probably needs more attention.
Lack of contribution isn't necessarily a problem; more accurately, it's the problem of managing the support, training, and so on for a larger pool of users when the pool of contributors grows much slower. It's a basic problem not restricted to Drupal. :)
I hope that my comments above weren't misunderstood. The issue is not whether individuals are 'qualified' to rate a module, or whether they will do something 'incorrectly' or whether they are 'trusted'. It's a matter of providing tools that actually capture the information that's useful and necessary. I may rate a module 5 based on its powerful APIs, but a user who wants something that's plug-and-play may give it a zero for an unfriendly UI and messy HTML output. On the flip side, they may find a module that provides a specific set of features absolutely essential, while I gripe about the scalability of the code. Before rating can help, we need to figure out WHAT we're rating.
At least, that was the dilemma I attempted to explain earlier. I've come around to the idea that just giving people a five-star Amazon-style rating widget would probably help in some cases. Seven hundred five-star ratings says SOMETHING, no matter what the details are.
My apologies
I wasn't directing my comments at you, more so to the Drupal community at large for some of the issues I've noticed over my four months of struggling to learn the Drupal way. So no, I don't think there was a misunderstanding, just a misdirection on my part.
I guess I trust rating systems for the very reason you mentioned earlier. I trust the "wetware" of the review readers to filter out the nonsensical reviews from the truly useful ones. What a review system does, is put the information up front and center, at the point of decision on a module: the module page itself.
That's much less labor intensive than filtering through scores of forum posts in order to try and evaluate a module.
Drupal is becoming a victim of its own success. As it grows more powerful and easy to use, hoards of new visitors will increase the need for better and more obvious documentation, even if that documentation is "keep off the grass." :-)
Separating the devs from the builders
The issues you mention are exactly why I now lean towards the following solution:
A five star widget for projects
A block of 'project health' information. Number of open issues, number of downloads, etc.
A dedicated forum for discussion by users of each project (NOT support).
The five star widget offers a simple 'at a glance' view. The 'project health' tracker is the kind of information that devs or experienced users probably want to use to evaluate a project. And the forums are where one goes to discuss things in more detail. All those approaches have their advantages and disadvantages, but together they could help.
Admittedly, these things won't necessarily help the actual mechanics of learning Drupal and working with it, day-in, day-out. It just helps streamline the process of evaluating third-party plugins when one hasn't been around long enough to learn which ones have a good reputation. That's an important thing, but only one small slice of the metaphorical pie.
I agree with your plan wholeheartedly
And I'll bet it makes a bigger difference in productivity than you think. :-) (I especially like the idea of a forum for each module -- cool!) Maybe use organic groups in conjunction with the project page? This new emphasis on the modules would heighten their importance and perhaps motivate project developers to provide more complete solutions, more like the core devs do.
Rating systems provide good positive and negative feedback. This can help rookie project devs learn what it means to build a truly robust application. So the rating system may help the developers as much as it helps users.
popularity != quality
Popular modules are not always good. Think tinymce. Also, there are no two modules that fit the absolute same role. As my goals with a website is fixed, what good is it to me whether a module is popular or not? If it's not enough quality, I can polish it (or hire someone to polish) it's likely to be easier and cleaner, than bending another module to fit my needs.
Programmer bias
Sure, but there goes the same programmer bias again and is *exactly* what Jeff was talking about.
Of the three or so WYSIWYG editors, if TinyMCE has 4.5 stars out of 5 compared to 3 out of 5 for all other ones, that's useful information.
A basic star rating system will collapse all ratings, from quality to popularity to scalability, etc., since EVERYONE rates it.
Per-module forum?
The problem with ratings is everyone is rating against their own expectations, each of which are completely different. There are many ways (all imperfect to some extent) to help set people's expectations more appropriately. One would be an over simplistic 5 star rating system (+1/-1 works for ebay). Another would be to allow users to rate a module against several things like ease of installation, end user usablity, maintainer responsivenes, How well it performs its stated task, feature richness.
Yet another would be allow module tags on forum posts. This would aid in searching the massive "Post-installation Forum"
Wow!
Module tags on forum posts! Now, what a concept. Sometimes the simplest ideas are the best. I have wondered why Drupal.org doesn't use some of Drupal's own features more heavily.
I also put in a vote for the ability to edit one's own forum post. :-)
Post new comment