Dangerous Territory: The CMS of the Gaps
One of the recurring discussions in the Drupal community these days revolves around the issue of specialization. Today, Drupal's core download ships as a sort of generic "anything-builder."
What does that mean? Drupal has tools to create things like multi-user blogs, discussion forums, documentation archives, and so on. Thousands of additional plugins can add additional features, but by default the user experience of Drupal is pretty generic. On one level, that's a real strength: the flexibility that the 'generic' approach forces on the developer community means that lots of solutions can be built on top of Drupal. The downside is that those solutions have to be built on top of Drupal.
The alternatives
Contrast that with specialized, focused tools that are tailored for a particular use case. OpenTape is a turnkey tool for setting up web-streamable music playlists. That's all it does, and it does it really really well. WordPress? Its biggest strength lies in the relentless focus on improving the blogging experience, baking in specific features that single and multi-user blogs require, and leaving the rest as an exercise for the tinkerers. Moodle is an educational tool that inspires wails of pain from my developer friends. But it's successful because it delivers tailored functionality for classrooms that would otherwise take a pile of work in more generic tools.
On the opposite side of the spectrum lie dedicated frameworks like Django, Rails, Cake PHP, and so on. These are tools for individuals or teams that don't want assumptions about workflow and database schemas -- they want control, and they don't want to worry about working around someone else's crap ideas. For those teams, choosing an existing piece of software is about maximizing the amount of work that is already done, while minimizing the work-arounds they'll have to perform to circumvent a framework's assumptions.
Enter the heretic
I've made waves in the Drupal community for pointing people to those different tools, recommending them in favor of Drupal when I think it's appropriate. To me, it's an indication of a troubling blind spot in the community.
For the first category of software (specialized tools) the argument is often made that they lack Drupal's flexibility, scalability, or robustness. That may be true, but for someone who needs a blog or a playlist streaming tool and so on and so forth, the hypothetical benefits of Drupal are vastly outweighed by the work they would put into assembling the tailored tool they desire out of Drupal's building blocks. If I have to spend two months of my life learning Drupal, and the big payoff is being able to sort my blog posts in reverse chronological order, why not just use Wordpress?
For the second category of software (low level frameworks) the argument is inverted: why spend all the time implementing features on top of a framework when Views is there? (Insert Organic Groups, UberCart, FormAPI, CCK, or any other popular Drupal tool.) The answer is obvious if we're willing to look at things from the other side. While Drupal is more flexible than a tool like Wordpress, it's less flexible than something like Django. It offers more functionality out of the box, but also bakes in a laundry list of assumptions that don't exist in pure frameworks. If building your web app in Drupal means working around the assumptions of Drupal core, implementing dozens of custom modules and sidestepping common patterns... why not just write it in straight PHP, or some other lower-level framework?
Today, the answer for Drupal is easy. If you're trying to roll some new and novel combination of functionality into a custom site, and the assumptions inherent in Drupal core are a reasonable match your workflow and requirements... go for it! If you have to do too much customization, though... you might as well use a dedicated framework that will give you more flexibility. And if a dedicated, use-case tailored tool emerges to do what you're trying to accomplish? Odds are, it will meet your needs better. That spells danger.
And now, let's talk religion
In arguments between Christians and Atheists (always a popular staple on the Internet), there's a catch phrase that gets tossed around: The God Of The Gaps.
It's shorthand for a complicated idea, and the rebuttals that go along with it. God, some might say, is evidenced by all of the things we don't understand in the world. It's a quick, tempting answer to the challenges of unbelievers: after all, their beliefs can't explain how geckos stick to ceilings, or where love comes from! But the increasing reach of human knowledge, others argue, shrinks that pool of mystery every day. New mysteries may emerge, but relegating God to those "gaps" in human knowledge is a losing game. Over the long term the domain of His influence in human affairs will shrink and shrink to exotic, esoteric subjects that few even care about.
That's the danger of The God of the Gaps: it is a tempting rhetorical escape that works in the short term. Over time, though, as knowledge expands, (or in the case of software, as new specialized tools emerge...) the argument undercuts itself. Now, a skeptic says, we do understand that mystery, so there's no need for hand-waving. Extending the analogy, imagine that a specialized tool does emerge for some use case that previously required a custom solution! Why bother with the mystery of faith (or the generic nature of Drupal)?
A fine question indeed. Solving the issues of faith, mystery, and belief are above my pay grade. When it comes to Drupal, though, I see a profound danger in assuming that the software will always thrive as "The CMS of the Gaps."
Another way forward
For quite a few months I've been talking enthusiastically about a couple of initiatives in the Drupal community. One is SmallCore, a loose affiliation of developers working to ensure Drupal makes fewer assumptions, and offloads more of the user-facing workflow and presentation decisions to custom-tailored tools. Other fun projects like Features allow developers to bundle up configuration and code into packages that can be easily re-used by site builders. Nerd toys like Aegir and Drush allow devs to manage their projects better. And advances like pre-packaged Drupal Install Profiles allow tailored tools to be built with Drupal -- then distributed as turnkey solutions rather than long series of blog posts on "How to build X in Drupal."
On the flip side, I've also expressed deep concern about certain aspects of Drupal 7 and the #d7ux project, intended to make Drupal's user interface slicker and more polished. Why? A lot of the attention went towards making Drupal a better generic tool for generic users. It's easy to arrive at the worst of all worlds with that approach, baking in lots of assumptions but refusing to commit to a single kind of problem to solve.
The common thread connecting all of those interests and concerns is concern about the gaps. By working towards a world where Drupal's useful framework chunks (like FormAPI, session management, URL routing, localization, and so on) can be re-used cleanly, we allow teams that would otherwise jump to a framework to take what they want from Drupal and ignore the unecessary stuff. By separating the underlying nerd-tools from the tailored stuff on top, we ensure that the focus on one use case doesn't screw over every other use case. And finally, by providing tools for packaging, distributing, and maintaining pre-customized distributions of Drupal, we ensure that end users can treat it like a specialized tool, rather than a box of parts with cryptic assembly instructions.
Not all puppies and ponies
There are dangers, of course. It's possible that Drupal developers could just nerd out, and the hoped-for "focused solutions" remain vaporware. Tools like Open Atrium and Managing News (an intranet management tool, and a news feed monitoring tool) are both built in Drupal, but they took lots of work. It remains to be seen whether smaller teams and less focused will be able to deliver equivalent results.
But the alternative is (in my eyes) just as dangerous: Drupal as "the tool you use when a better tool doesn't exist"... AKA, the CMS of the Gaps. Today, Drupal is thriving. Tomorrow, as the reach of the web publishing continues to grow, specialized tools could easily undercut its advantages. Are we, as a community, ready to tackle challenges more subtle than feature lists? I think we are -- and I hope others agree.
Thank you
Eaton,
if you are ever bored with programming, you can easily make a leaving with writing :) Excellent post. It was a pleasure to read.
Like you said, there have been tons of discussions about Smallcore. In my not-so-humble opinion, though, a lot of debaters were struggling to even explain what #smallcore is, or more correctly: where is the problem with the current state of Drupal that makes us eager to debate something called #smallcore. You nailed it - we are trying to save Drupal from trying to become an ephemeral CMS of Gaps :)
As for the solution: I think Drupal is moving in the right direction and we will get there. No single blog post, yours, mine or anybody else's will be able to provide a ready formula for the "cure", it will be a process, but I have no doubt that we are moving in the right direction.
The strongest thing that Drupal has going for itself is the amazing community around it. I think it's safe to say that Drupal has gathered some of the brightest, talented and innovative minds in the industry. Drupal is not a copy-cat or a linear progression of anything - it is a disruptive innovation, that is why so many people love it.
You are on-spot about many things in your post and in particular that it's not a small task to build a solid distribution on top of Drupal, but the good news is - it has been done, is being done and seems like is becoming an increasing trend.
I don't know any other "CMS" that has been used as successfully to create great "specialized tools" and I don't know any other "framework" that is so feature-rich with a lot of "just use it" tools. So Drupal is unique and it will succeed as far as its biggest asset - talented and innovative developers enjoy working with it.
It's all about community.
I'll be the first one to
I'll be the first one to admit that my affair with Drupal is a bit on the skids. This is going to be another one of those rambling comments that I'm probably going to make into a blog post.
At its heart Drupal is a hobbyist tool. I don't think there's no better CMS for tinkering. It has everything you'd want: a huge community, tons of modules, a very elegant core, a challenging, but rewarding design. It's the LEGO of CMS. I haven't touched Wordpress since I migrated off it, but I'm not sure if it's better for building a blog than Drupal is today (if you are a web developer, of course).
In the professional arena Drupal presents a different set of strengths. "Uh, we need a [youtube|flickr|del.icio.us|google reader|linkedin|piratebay|etc|etc|etc] clone demo in two weeks. And by demo I mean a site we can launch in another week." Drupal is what's enabling this kind of business strategy. Can we have X? Yes, there's a module for that.
Most of the money in Drupal world is made in a) educating PHP developers to use Drupal for a big new redesign and b) fixing the performance on the said redesign a few month later.
Performance tuning is painful, but it's doable once people forget the "it's gonna run on two gigs of ram in a 5 yo machine that we have".
Unfortunately Drupal is obese. It takes hundreds of mysql queries to do anything. The elegance of Drupal's core is hurting it, and people are usually turning a blind eye to the fact that every single page runs dozens if not hundreds of queries that don't belong there.
The solution that is practiced in the Drupal community is what I call the "Mall Cop": you put the fat guy on a Segway (memcached + squid + CDN + a lot of hardware). You can make a mall cop that way, but this would not fly in NYPD or in the military.
This isn't the worst part. The worst part is that Drupal is not staying true to the Unix way: have a solid core + a lot of tools that are very good at what they do. The problem is not granularity, and the core is pretty decent. It's the tools.
A lot of them are horrible. Views and Panels are murderously bloated. Drupal's build in search is a nightmare (you just have to use Lucene or Google). The interfaces for content and user managemen when you have a lot of content and userst? Third rate (and these are baked into the core). There are modules that are just UGH - quickstats, xmlsitemps. Comments? Drupal comments are slow and bad. Managing comments is very tough, and fighting spam is even tougher.
Once you replace all this functionality you start asking: why do I want the rest of Drupal? Why would I want to run 500 queries per page when it could be done in 10 since I'm already bringing in so many plugins that could work with anything else?
Drupal core is evolving by leaps and bound, paying down design debt as it goes. Except it's not Drupal's maintainers that are paying. It's people stuck with old installations with hundreds of modules and zero chance of upgrading. It's the site maintainers who pay the price.
There's no focus on the upgrade path. There's no focus on making fewer queries. There's no focus on the CMS fundamentals.
Strong writing, although I have some reservations
This is a really eloquently written and concise essay, by far the best I've read on the topic. I agree with most of it, particularly the advantage of specialized tools over generalized ones.
However, I disagree about the resolution. I feel like just because Drupal doesn't have a clear vertical application doesn't mean we should make it a bunch of APIs. If I was writing a framework there is NO WAY I would write it in the Drupal style. I am not a fan of the Drupal architecture, and the Not Invented Here culture which spawns it. I am a fan of the momentum in the community and the contributed modules that people create.
While I can relate to Adrian's post as a site builder, I think the premise is way off. I don't think projects like Managing News should be built in Drupal. I think they should integrate with Drupal, Intranets (Open Atrium) I'm on the fence about. In short, I think Drupal wins being more generic than WP, but strongly being a CMS. It wins in the *business* world (which is something we're not even discussing yet) when it is a brand. To be a brand, it has to provide real value as a product. Profiles can and should exists, but they should be profiles to build something similar to what core is, a social publishing application for interactive websites. Anything else dilutes its potential.
I admin that "CMS" is too broad, but making Drupal a framework to build accounting systems to aircraft control to MMORPG is like re-writing it. The contributed modules mean nothing, nor do the themes. If you want to do that, use J2EE, it's much better for that type of thing.
Okay, other than that, you are absolutely right on about too much dogfood eating and the fallacy of the Gap argument. Beautiful!
- Jacob
Thanks for the feedback,
Thanks for the feedback, Jacob!
I'm curious about the reasoning behind it. What's the cutoff line? What kinds of projects should we be facilitating, and what kinds of projects should we be discouraging? That's a discussion few in the community are willing to have at this point, at least in my experience.
I've worked with a lot of large businesses who've attempted to build solutions on Drupal, a number of which abandoned it for the reasons outlined in this article. I'm not sure that 'business' is a category specific enough to draw too many conclusions! For quite a few businesses, Wordpress is itself overkill. For others, Drupal's baked in assumptions about content, workflows, schemas, and so on are an exercise in frustration.
Of course! By 'specialized' I don't mean things that dramatic. The examples I mentioned (blogging, learning assistance tools, online playlist management) are IMO not odd use cases for Drupal. They are square in the middle of what it claims to do well, but there are specialized tools that do just one of those things much better.
One of my personal skunk works projects -- a Drupal based platform for webcomics publishing -- is an attempt to build something 'focused' on top of Drupal's generic approach. It's tough, but I think we need projects that are willing to focus. Enabling that is, at least in my opinion, a worthy goal.
Its OK
I'll agree with most everything thats said here. Drupal does indeed live in the slushy world between focused tools and Django/Rails. As you say, this gap may decrease over time. But its OK. No software is meant to last forever. Drupal is on its 9th year and it is still not even on the fast end of its hocky stick according to most observers. It feels like its exploding. So, we've got another 9 years IMO. I'm not going to sweat it much. The future is very hard to predict.
In summary, your diagnosis is right but I don't know that Drupal should do anything about it.
So, I guess my problem here
So, I guess my problem here is that I don't understand the disagreements... it makes no sense to me. From my standpoint, it seems like a really easy logical jump to say that drupal "core" is essentially already doing a bit of this with the new minimal installation profile and of course the new and improved drupal 7 install profile. I've made mention of this point before, but Drupal 7 is essentially just the first "distribution" of the new code base.
Accepting that point (which, hopefully we all can) I'm not sure I understand the opposition to allowing other "distributions" to limit what they want from core (as long as it's in the core optional list, it'd be nice if drush_make could pull these items individually as needed, but I digress) This approach essentially turns drupal core into the "mythical" framework, allows Drupal the product (i.e. distribution) to continue existing, and give other distribution developers the best of all worlds since most of the "assumptions" that are so painful to work around are simply missing.
I'll cut short at this point, but I must say, I find the push back from the community, very confusing on this point. I'd happily listen if someone could explain this point to me. :-)
Eclipse
As resident Devil's Advocate...
...I think I do understand some of the opposition to this very idea.
Two issues have long plagued the Drupal world. First, the "let the themer solve it" approach to UX and integration. Second, the tendency to rewrite large swaths of core, pulling the rug out from under the contrib world and the site-builder community, sometimes with negligible payoff.
To someone who feels those points of pain on a daily basis, "Make Drupal more frameworky, and layer UX/workflow solutions on top of it!" sounds like something that will make those problems far worse, not far better. It's a valid concern. But, as I said, there are dangers just as serious in the status quo. At the very least, I'm hoping that those who are troubled by that are also able to see the validity of my concern about the "gaps." It's something the community needs to iron out together, and I hope that we aren't sucked into negativism or booster-ism while we're doing it.
Not the same
I think that's easily the most common misconception about the framework-split/smallcore/whatnot discussion. It really has nothing to do with what modules are or are not enabled. It has to do with how modules interact with the core system, and how the core system is built.
For example, if you have *any* even remotely non-trivial logic in your form submit handlers then You're Doing It Wrong(tm), because it means that logic cannot be accessed from a non-web-form-human-user workflow.
And don't even get me started on the hoops that are involved in hooking Field API up to Form API right now. I don't actually understand them, and there are open issues to try and fix them because even the people wrote it don't fully understand it. :-)
A more frameworky approach has a complete data object lifecycle that works purely from code, with code that is readable and comprehensible. Then you build a form structure on top of that which acts purely as glue.
The assumption here that is flawed is that people will be interacting with nodes only through node/add and node/X/edit. Look at hook_nodeapi op update (or hook_node_update in D7) sometime and the way that a bare object structure has to map to BOTH the database AND the form structure for it to work properly, something that is not actually documented. That is a critical architectural flaw, and exactly the sort of thing we need to get rid of.
that logic cannot be accessed
Indeed -- a lot of modules need to fix this before they can get, say, drush integration.
So what is the solution? I
So what is the solution? I have now heard it many times we are not really great as framework but neither as a product.
While I have been very open to putting many of the interface assumptions we do into profiles. There is actually very few people working on it. What is the actionable work for Durpal 8 - to make this distinction as you see it happen?
An excellent question.
At the end of the day, most of it is un-glamorous "plumbing" work. Making sure that entities we use in core (taxonomy, node type definitions, user profile information, permission presets, imagecache presets, forum structures, etc. etc.) are easy to export and import, or at the very least define in a programmatic fashion as well as via the web UI. Making sure that 'heavy' aspects of our UI, like admin toolbars and cluttered administration forms, can actually be turned off rather than just papered over. (The admin toolbar in D7 is a good example -- if I don't like it, I can kill it and use 'Admin' module instead.)
On top of that work, we can build cleaner and smoother solutions in the form of bundled install profiles, prepackaged "features", and so on. A lot of work has been done in this area in D7, which is good, but we have yet to hit a point where these kinds of things are actually shared community standards, or explicitly articulated goals.
Creating the best CMS of the Gaps
Thank you Jeff for drawing attention to one of the most definitive characteristics of Drupal.
I'm less worried about disappearing gaps than Drupal loosing its edge as a CMS of the Gaps.
The fact that Drupal is the swiss army knife among the CMSs and frameworks (the one with the magnifying glass and saw) makes it so attractive as a weapon of choice for diverse projects. If you earn bread and butter with building out custom solutions for your clients, if you build a blog today, a e-commerce site tomorrow and an intranet the day after that, the CMS of the Gaps is what you want.
From a business perspective it's smart to work on a consolidated stack, especially if you're small to mid-size - your team has a real chance to become fast and efficient in solving problems thrown at it. I'd venture to say that most contributors in the Drupal community come from this background (and Jacob - this is also the reason why it does make sense for us to build Managing News on Drupal).
This is exactly why I am not worried about disappearing gaps. I think even as more specialized tools are popping up, there will continue to be a strong need for the swiss army knife.
However, what I *am* worried about is Drupal loosing the edge when it comes to being the CMS of the Gaps, the swiss army knife.
*A lot* of improvement has gone into Drupal 7, there is *much* more that needs to be done and you have mentioned some of it. To my mind come exportables, cleaner APIs, cleaner theming layer, UI patterns, performance, libraries, plugins, deployability, default variables, Form API etc.
Are we in a good shape for this work? I think we could do *much* better.
We seem to try to grow Drupal's reach by growing Drupal core. The result is an ever heavier project with slower release cycles and more people talking at the same time in an ever more crowded room.
One of the solutions in my mind - and here we are in agreement - is to work towards a better defined framework portion of Drupal while improving the out of the box experience. This would allow different speeds in release cycles and makes it more attractive to work on Drupal as a product.
I am well aware though that structuring Drupal core better into framework and CMS isn't the silver bullet nor is there universal agreement for it.
The question I remain with is: what else can we improve to make Drupal the best CMS of the Gaps?
Pingback
[...] Dangerous Territory: The CMS of the Gaps | via positiva jeff.viapositiva.net/archives/2010/01/dangerous-territory-cms-gaps – view page – cached One of the recurring discussions in the Drupal community these days revolves around the issue of specialization. Today, Drupal's core download ships as a sort of generic "anything-builder." [...]
Mind the Gap...but not too much
I think you're very right about Drupal roosting comfortably in the gaps, but I think (like Alex) that there's more to the gaps than you give them credit for. They're not just a hasn't-yet-been-filled-in-space. The gaps that Drupal is really filling are the ones that arise from the need or want to take individual web phenomenon (e.g., Twitter, YouTube, etc.) and hybridize them together into something really new. And at a much deeper level than what "mash-up" typically refers to. It's kinda like a super mash-up (Smash-up, anybody? :P). In essence, there are a lot of Drupal modules which aim to take some whole internet phenomenon and package it as a building block.
Personally, this is why I got into Drupal in the first place. I'd spent a couple years passively toying with wikis, forums, groupware, CRMs, etc., and eventually reached the conclusion that there's no way I'd be able to stitch all those systems together into something usable if I was using separate providers for all of them. I jumped on Drupal once I discovered that it could act as the connective tissue between these parts.
To be clear - I don't think the current economy around Drupal could be sustained exclusively by (s)mash-up work. Not even close. And I also agree _wholeheartedly_ that we need to hurl effort at encouraging a trend towards repackaging Drupal for specific cases. But I'd still encourage you to revisit what it really means to be "the tool you use when a better tool doesn't exist." Those "better tools" need time and space to evolve - and as long as Drupal has a huge community hell-bent on transforming whole internet phenomenon into LEGO blocks for your site, we're going to be very fertile ground for necessary experimentation.
I typed this up before
I typed this up before reading Alex B's comment, but I think that pretty much nailed my disagreements with the opening post here.
The advantage of Drupal is that many people end up building more than one type of site within the space of a few months or years, and they don't need to learn a new CMS/Application/Framework each time they do that. No matter how many gaps get closed, that's not going to change. This counts for individual websites/organisations, individual site builders (whether hobbyist or professional), and large development teams.
Specialized tools might look nifty when viewed in isolation, but they tend to wane a fair bit when you need to be familar with three or four of them to get your job done.
I've worked on at least two sites where a single public facing site was made out of a combination of 6-7 different technologies - (including but not limited to custom-PHP, Drupal, Wordpress, geeklog, phpbb, gallery2, Silva/Zope - three of these were on both sites). Both could have had their entire feature set and more with Drupal alone, with much less maintenance overhead. One of them now does.
I'm also really, really glad for my own sanity that I only have to deal with Drupal these days instead of the 10-15 CMS / blogging / ecommerce / forum / gallery applications I've used to a greater or lesser extent in the past 8-odd years. Even if this was updated and limited to Wordpress, Drupal and Django it'd be a headache to keep up with.
On smallcore / framework vs. product etc. I'm still concerned that people are even talking about Drupal core as a 'product'. Products are things like Acquia Gardens and distributions. If core become targeted, then it ends up competing with services and distributions - they'd be built parallel, rather than on top of.
On Drupal 7 / d7ux - Drupal 7 is as much the release of the new database layer (much more framework-y than nearly any other Drupal API), and thousands of lines of refactoring, as it is d7ux, which only existed for the latter 25% of the release and you'll barely notice if you do a straight upgrade from D6 to D7 since it's 99% contained in optional modules anyway.
Most of the baked-in assumptions about administrative interface and feature-set are inherited from Drupal 6, Drupal 5 and earlier. To me those are the much bigger problem to deal with.
So for me, I don't feel like it's a blind spot in the community at all - we're quite happy to say that Drupal does almost nothing useful out of the box, that it's a framlication, that it being generic can be annoying, that these are strengths as well as weaknesses, and that it can look crappy when compared to targeted apps which do a subset of its potential functionality really well.
What worries me most is that two years of API refactoring and improvements in Drupal 7, in many places, still feel half done - or inconsistently applied (I felt the same way about a lot of changes in Drupal 6 though, probably why I spent so much time on 7...). And much of this is due to lack of manpower rather than any other reason - see http://drupal.org/node/621618
But that also means a lot of the work is already mapped out for Drupal 8 - building out the huge missing pieces in the entity API, getting exportables into core, making FAPI happier doing all the thing it's currently unwillingly forced into now, more pluggabillity. I hope posts like this encourage more people who usually stay away from core to get involved in Drupal 8, but I also hope they turn into Drupal 8 battleplans rather than gloomy predictions soon - because while we're worrying about being a framework or an application, it's quite possible someone is cooking up a new framlication without ten years of legacy code somewhere, and it's that (or a meaningful fork) which would be the real risk over the next 5-10 years.
Post new comment