Framework, Platform, Application, Features, Product, Workload

This is an attempt to summarize some recent discussion about Drupal 8 direction and development, it is heavily biased towards my own opinions and is not intended to be either neutral or comprehensive. There is still plenty to discuss around all of this, while there has also been a significant move towards a consensus on some areas as well.

First, framework vs. product is a false dichotomy. Drupal core is both a crap framework, and a crap product, either of them on their own would have died a very quick and lonely death some time ago.

This has described this as the platypus problem. While it’s a problem, it’s also a strength. Let’s replace ‘platypus’ with ‘human’:

  • can’t run fast
  • can’t jump high
  • looks naked without any clothes on
  • wastes resources
  • opposable thumbs
  • self conscious
  • good at adapting to environments and altering those environments to suit its needs.

Sounds like Drupal right?

Various Drupal ‘competitors’ might be described as cheetahs, peacocks, blue whales, ants - they are generally more single purpose, monolithic, lower level, and/or prettier - whether more numerous or less.

If core is neither framework nor product now, what is it? And how might it look in 2-3 years?

As well as a platypus, Drupal is often described as a framlication, for me this has always been it’s core strength as well as a cause for confusion and difficult choices.

It contains a mixture of APIs and features at all levels. Currently these are all mixed together, often with bizarre and unexpected dependencies (mostly on system.module), but in general they provide a good basis from which to build other stuff.

Here’s a table that attempts to conceptually differentiate how things break down in terms of core feature sets, how this relates to specific components, and some desired outcomes for what things might look like in Drupal 8.

Level Description APIs and dependencies Implementation/features
Framework Very low level functionality, usually included in most PHP 5.3 frameworks. Minimal interdependencies within this layer, or very clearly defined.

This is used to build platform functionality.

Database layer, error handling, configuration system, session, cache, queue, lock, module/hook system, theme system, form handling, logging, router.

These should have no dependencies except possibly on other internal APIs.

It should be possible to add and remove APIs at this level via contrib.

At the moment this includes both /includes and modules. It might do in Drupal 8 too.

MySQL, PostgreSQL support. Memcache, Redis

These usually have external dependencies which may or may not be available on different environments.

It should be easy to add and remove implementations at this level via contrib.

Stuff at this level could potentially be used outside Drupal. Or by drush.
You might need to have one implementation for each of these things on your site, but you usually don't need both MySQL and PostgreSQL or both database caching and Memcache.

Platform
(this is still a framework for any definition other than ‘micro-framework’).
Low level functionality that provides elements used in web siteunctionality that provides elements used in web sites or other web applications.

These are used to build application and product level functionality.

Users, entities, fields, vertical tabs, views, panels, blocks, ajax, states, actions, tokens, voting API.

Can freely depend on framework level components, but should have minimal or clearly defined dependencies on each other.

Should be easy to add, remove and replace APIs at this level from contrib.

nodes, comments, file and image uploads, path aliases, menu, taxonomy, relation, stark, blocks again, rules, views, services.

It should be trivial to add and remove implementations at this level via contrib.

Some implementations in turn expose APIs.

Application
(this is still a platform, for any definition other than ‘framework’).
Higher level functionality that allows platform and framework functionality to be combined together and used as a coherent platform.

It is built using the framework and platform.

Core site building experience / site construction kit

APIs start to thin out at this level, except for ones inherited from the platform or framework level.

Can freely depend on framework and platform level components, but should have minimal or clearly defined dependencies on each other.

/admin, module and theme management, status report, web installer, log viewer, update status, toolbar, overlay, Seven, contextual links, field UI, Views UI.

It should be easy to add and remove implementations at this level. via contrib.

Features (not features module as such) High level functionality that is tailored to particular use cases. Mainly dependencies and configuration, bits of glue code.

Shared and re-used on multiple levels, may also depend on others.

Polls, forums, Bartik, image gallery, shopping cart, workbench.
Product Packaged functionality designed to meet specific use cases.

It is built on all the previous levels.

Install profiles.

N/A.

It’s OK to have lots of sideways dependencies here, these will mostly be created via setup and configuration though.

Some of these are just conveniently collected sets of modules, some also give you a head start setting those modules up.

Setup happens here, but not runtime code.

Standard profile, Drupress Blogging profile, Drupal Commons, Commerce

Download, install, fully-fledged ready to rumble products. (Compare with Wordpress.)

To get to this point there are several major pieces of work that need to be done:

  • Improve the low level framework via refactoring or replacing APIs.
  • Improve the platform and application layers
    • move APIs downwards into the lower level framework and platform for shared dependencies
    • bring platform/application-level features that are buried in /includes and system module up to the level they belong.
    • Fix long standing bugs, architectural and usability issues here.

    These two are loosely covered by the unofficial framework, CMI and WSCCI initiatives.

  • This is a lot of work and is going to require touching tens if not hundreds of thousands of lines of code. To make some space for this to happen,make that easier, we need to move some legacy features (at any of the four levels, but starting with ‘product’) that add more maintenance burden than they’re worth out of core. Either explicitly state that we’re deprecating some old features, or find maintainers for them. If we can’t do either, work on outreach to fix that and refactor them to bring them up to date.
  • Ensure that new features that are added (at all levels) match the longer term goals and make things easier rather than harder, and that people will continue to maintain them after the initial work to get them in.
  • This is a lot of work (so much I’m saying it twice), and we will also be maintaining Drupal 7 for at least four years. Nurture and develop ‘short tail’ contributors via office hours and MAINTAINERS.txt both to help with Drupal 8 development and continue maintenance of Drupal 7.
  • Define the product or products that is/are essential for Drupal to provide on the day that 8.0 ships (is it only a site building platform, do we need use-case specific examples, how are these presented?). This is the territory of the snowman project. This may result in new features being added at the platform and application levels, as well as potentially removing some too. (I am completely ignoring the packaging issue here, deliberately - if it holds up an 8.0 release, simply moving things around doesn’t make anything move faster, there’s a lot of other stuff that needs to be done regardless of that, and changing packaging could make things worse, especially while all our code is interdependent).

Comments

There have been a lot of blog posts along these lines the past couple years. What needs to happen to turn all this talk into action? Is one or more people holding it up? If so, who?

One of the biggest challenges we face at this point in Drupal's life is that ad-hoc tactical responses to these issues are insufficient, because the community is not on the same page about either the underlying problems or the solutions. These posts are -- at their heart -- about clarifying and refining that shared vision so that the tactical steps can be executed and evaluated consistently.

In several of the meta-issues on Drupal.org (http://drupal.org/node/1224666, http://drupal.org/node/1260214, http://drupal.org/node/1242448, etc) there are a LOT of specific issues that interested developers can work on to make these broader ideas a reality. Making sure that we'er working towards a consistent goal -- one that solves the really critical pain points of the developer and site builder communities -- is a higher level question that desperately needs discussions like this one to clarify.

The idea that there can be a shared vision is a poor one. There are multiple interests that are competing. In some of the avenues there can be agreement, but in others there never will be any. For instance, developers want one thing -- the corporations wants something diametrically opposed. Acquia has this idea that Drupal should replace everything that a corporation has... and that end users should be able to control much of Drupal. The developers want something completely different. Figuring out which group benefits from what decision goes a long way to clarifying who should be working on what.

What features should be put into modules, and paid to be maintained by corporations... and what stuff should stay in core. How core can meet all the various interests the best way possible is one thing that we should answer before trying to agreement mass agreement... as there is no such thing. Certain voices are being heard because of their position. Other voices are being ignored.

In terms of low level cleanup, there is nothing holding it up and a lot of consensus around what needs to happen. There are both immediately actionable issues, and several details to work out. http://drupal.org/node/1224666, http://drupal.org/node/1239644 and http://drupal.org/node/1240138 have most of the action, or links to other issues. This is where to get stuck in if you are sick of reading blog posts. I started opening many of these issues because I also was getting sick of reading blog posts. No one is blocking this, but since core is extremely interdependent, a lot of the issues are interdependent too. If you read those issues and don't know where to start, find me on #drupal-contribute in irc.

You can also contact Crell and/or heyrocker to get involved with the WSCCI and CMI initiatives - these have been in planning stages for a long time, but they are moving into writing code now (UUID support just went in, the first entity API clean-up patch is RTBC and there is active work going on for the next steps, there is work on the $context object and converting some core globals to use it). So between all of this it should be very close to an all hands on deck situation very soon - since there is consensus on the fundamental issues between all those different areas.

The other thing that is immediately actionable is helping out with Drupal 7 bug fixing. This week is the first time that all critical bugs, major bugs and major tasks have had any headroom under the thresholds. The further down they go, the more space there is to work on refactoring Drupal 8 at a faster pace.

Things that are blocked:

For ripping out legacy unmaintained features, we need to reach a consensus on what's a "feature", what's "legacy", and what's "unmaintained". Almost immediately after Drupal 7 opened for development, I opened a tonne of issues to remove some features from core, one example would be http://drupal.org/node/233301. It took more than three years for that patch to be committed (some other issues did get committed though), and that removal was precipitated by the recent shitstorm.

Over the past couple of years I had become increasingly disillusioned that we would ever be able to get rid of any of that cruft and was resigning myself to chipping small pieces of it off occasionally, and trying to ignore it the rest of the time. Apparently I was wrong to give up, but it required some of this drama to force the issue. Figuring out heuristics to decide exactly what does and doesn't belong in core (in terms of the current feature set and things we might add) is the job of http://drupal.org/node/1255674.

While there is some low hanging fruit in terms of what to remove, whenever this comes up, there is always a tonne of people who say things like "let's separate the framework from the product and maintain them as separate distributions then this would all be so easy!".

This is wrong for two reasons. First, very few people have ever laid out all the features in core and tried to put them in framework vs. product categories - that is what the big table in this blog post tries to do, but since I disagree with the binary distinction in the first place, there are five categories of stuff there, not two. Depending on your definition, you might draw the line in different places down that chart.

In my view, the issue there is we need to reach a basic minimum agreement around what the heuristics are and how they apply to the worst examples in core, then keep working on that iteratively. I was not at DrupalCon so I missed a lot of the in-person discussion around these issues, this feels like it could quickly become actionable rather than an endless bikeshed, but we'll see.

This kind of informations give a lot of overview understanding to Drupal. Higher level and meta issues discussions do help a lot in understanding our orientations, goals and how to work it out.

I wish I can voted for a session about this for DrupalCON, or they should be better exposed for the community on Drupal.org.

This is also being worked on, the http://groups.drupal.org/core is an announcement-only group, piped to both Drupal Planet and twitter: http://twitter.com/#!/drupalcore - which is specifically for directing people to meta issues and higher level discussions. It is very early days for that group so what's in there currently is a bit of a random mix.

Thanks for this post. I'm really enjoying the way you're synthesizing some of these big ideas and conversations into something with enough specificity for people to agree or disagree with: a very important step towards reaching a shared vision.

Possibly a minor nit, but I'm not sure the words platform and application for describing your levels 2 and 3 capture the meaning you're trying to express. Paraphrasing from Jeff Eaton's Drupalcon talk:

  • A product lets you do something (e.g., Wordpress lets you blog, Ubercart lets you sell stuff online).
  • A framework lets you build something (e.g., hammer, nails, and wood).
  • A platform lets you use a product (e.g., the internet and wireless telecom network let you use an iPhone).

With these definitions in mind, I think:

  • What you call Framework in your table, can be called a web application framework: the stuff that lets programmers build a web application.
  • What you call a Platform in your table, can be called a CMS framework or social website framework: the stuff that lets programmers build a particular kind of web application. So what this level represents is an increase in specificity of purpose, but no change in audience (it's still targeted to programmers).
  • What you call an Application in your table, can be called a site builder's CMS or social website framework: the stuff that lets people build a particular website to suit their (or their client's) needs without writing code. This level represents no change in specificity of purpose, but a broadening of audience.
  • I agree with the word Features for the 4th level. I think what this level represents is another jump in specificity of purpose, but no change in audience (still targeted to site builders).
  • I also agree with the word Product for the 5th level, and I think this level represents no change in specificity of purpose (relative to the Features level), but another broadening of audience (at this level, you're no longer targeting someone who has the hobby or profession of building websites, but someone who wants to accomplish some other goal, like Bob, the cheese shop owner from Jeff's presentation).

I think there's something aesthetically pleasing about this staggered movement from level to level: 2 jumps in specificity of purpose, and 2 jumps in broadening of audience. I hope you do too. I also think the words/phrases I suggest above can be further improved, but I hope they're steps in the right direction, and that getting the terminology to capture more precise meaning can further help with getting to a shared vision.

Thanks Alex. Yeah I agree that if we use the colloquial definition of framework (as opposed to 'a PHP framework) then levels 1-3 are all included - that is also the hard line (in terms of types of functionality, not specific stuff in core right now) I would personally draw where things do not get farmed off any time soon (and likely long term as well).

The points on audience and specificity of purpose are also great.

Doing the cleanup and seperation into Framework/Product Layers sounds like a good plan.
So good I would actually see it more important than building new features.

Watching Dries Keynote and knowing the scope of the Core initiatives that are already on the way is kinda opposed to that plan. What is in the planning for Drupal 8 (adding the initiatives that Dries listed to be still started) is just too much to do in one release.

I wonder if it is possible to see the initiatives as "can go in, but must not". Some will have to be delayed to D9 if not going to be half-baked. I can only imagine the amount of cleanup and bugfixing these big changes to core induce. So maybe it needs to be prioritized more and think about what is realistic to be done with in say two years to a release.

As the cleanup catch, sun and many others suggest are often quite low-level, this is the foundation a lot of the rest builds upon.
So my two cents - do less, but do it better and thus more quickly. Having D8 more on time and less pain than the D7 armageddon.

Drupal is like a coral reef, built up over time via accretion. It is likely that this 'crisis' is a common stage in the development of a large software project over time. While there is pain, it might be 'growing pains' that are a sign of good progress. Hard thinking like this is crucial to forward movement. Thank you.

I like the distinctions effulgentsia suggested in labeling / describing the layers. Terms like "framework", "platform", "product", and "features" are all general terms with lots of uses in other programming / software contexts. Before this discussion, when I heard "framework" I thought Symphony or Zend and when I heard "platform" I thought Drupal. I suspect that the more clearly the function of each layer is defined the easier it will become to architect changes.

This is what I got from reading your post:

1. The Web Application Framework ("Framework") is the general infrastructure functionality.
2. The CMS Framework ("Platform") is the CMS specific functionality.

These two are what module builders work with.

3. The Site Builder's CMS ("Application") becomes the packaged "Drupal Core".
4. Contrib.
5. What people do with 1 through 4.

Do I have it right or am I lost?

There's nothing like coming up with an idea, and finding out your thinking is on the right track from the get-go. It's a rush, and I got that rush reading these posts.

More than that, I find I've found a home in Drupal, and am looking forward to contributing on a regular basis.

Keep up the great work, catch. These were great posts.