Drupal 8 performance and scalability - audience priorities

Submitted by catch on Sun, 02/06/2011 - 04:57

This is the first of what may be (or may not, let's see how it goes) a series of blog posts looking at performance and scalability in Drupal 8.

I'm posting this to continue some of the discussions already begun by Eaton and Dries, which have touched on performance and scalability in amongst other thoughts about the Drupal 8 release cycle. It's also to large extent a followup on Larry Garfield's post about Drupal's audience priorities.

There seems to be quite a lot of energy at the moment to improve performance in Drupal 8 (Drupal 7 already scales very well if you take advantage of those features, although we can undoubtedly improve things there too). But while there are always patches in the queue, I don't feel like we have a good framework for discussing what the overall issues are, or for who we want to fix things for. In many discussions, the most common distinction made is between 'high end', which I would describe as "has a hardware and development budget", and "low-end" - "has minimal hardware and no development budget" - however this really comes down to resources rather than needs.

I've talked about some of what I perceive as the issues with Drupal 7 performance - both from a technical and development process standpoint in the comments on those blog posts, let alone the issue queue, and don't intend to discuss that here, yet.

Instead I think we should step back a level, and discuss the types of sites, the performance and scalability issues those face, and then who we want to fix things for. This will allow us to begin to identify areas which will affect the vast majority of Drupal users, and those which only affect a subsection. While we might disagree on the priorities, we'll at least know what trade-offs we're making up front.

Everyone reading this probably already knows that Drupal runs a massive variety of sites, from this blog (literally a few hours old as I type this since I couldn't be bothered to upgrade the old one from Drupal 6), to whitehouse.gov, economist.com and examiner.com, with a whole spectrum in-between. The following will by necessity be incomplete, but can hopefully begin to set up terms of reference that we can expand on later.

First of all we need to look at what kinds of performance and scalability profiles various kinds of sites have. It's not possible to identify particular roles for particular site types, instead we need to take a set of different requirements, and these will apply more or less to various kinds of sites.

Features
This is pretty simple, you either don’t have many features, or you have a lot, or somewhere in-between.

Traffic

Low traffic
High anonymous traffic, low authenticated traffic
High authenticated traffic, low anonymous traffic - social networking, forums, intranets
High everything - community, media/news, sites that try to combine all these
Cutting across these - regular traffic vs. spikes

Content

dozens or hundreds of posts
thousands or tens of thousands of posts
hundreds of thousands or millions of posts
twitter

Hardware resources
(prices are off the top of my head, obviously there's massive variation depending on support levels etc.)

'Random Arse Shared hosting' - $5-15/month
Premium shared hosting / small VPS - $15-40/month
Larger VPS/single dedicated server $40-300/month
More than one dedicated server - $300/month+
Cutting across these - various cloud services that claim to scale for you and may have usage-based pricing

Developer resources

No experience
Web savvy but no coding
Lone professional developer/enthusiast
Dedicated development team(s)
Dedicated infrastructure team(s)

If we look at these, we can see that this isn't really about big and small sites except at the two extreme ends. Someone's personal blog might get thousands of unique hits per day, a forum site set up by someone on their weekend with no budget could end up with thousands of users and millions of posts, a very complex ecommerce site might only get a few hundred pages requested per day but still be viable financially. All of these have wildly different needs, and the approach taken to solve performance issues could be as simple as a configuration change, or require architectural refactoring.

If we look at the issues that sites run into, there's a similar pattern:

Database - complex features often result in un-indexed queries. Lots of content makes those un-indexed queries take longer and longer. Some setups have issues with latency to the database server as well.
Memory - some things (rebuilding all caches, for example saving the modules page) take a lot of memory. Normal run-time memory usage affects how many threads you can run at once.
CPU - whatever kind of site you have, CPU time is finite per thread, and no-one wants their site to be slow
i/o - both shared hosting and multiple-server setups often run with either shared disks, or network file storage where both read and write operations can be very expensive

Setting priorities
In term of anonymous user performance, apart from a few tweaks this has worked well for as many Drupal releases as I can remember with minimal changes. One thing we should look at changing is enabling page and block caching, and css/js aggregation by default, some sites never, ever end up with these enabled, and the minimal profile exists for people who definitely don’t want these things when they first install Drupal. in fact I only just remembered to go off and enable these before posting this!

Where we fall down is in two main areas:

Run-time authenticated user performance - both raw speed and resource usage
Peak authenticated user performance, and cache misses for anonymous users - this boils down slow performance and high resources usage (related but not always the same) building site-wide, page, and block-level caches

The first leads to a general feeling of sluggishness, and a need to upgrade hardware when authenticated traffic increases.

The second leads some variation of ‘site down’: you exceed your PHP memory limit and your site stops working on that page; you have pages that hang for several seconds but eventually load; or your entire server falls over and no-one can see any pages - whether temporarily or for an extended period.

I think we need to look at both of these for Drupal 8, and again they cut across various types of site.

In terms of where things fit into site profiles, there are currently several steps that sites take as they begin to scale up:

Enable various kinds of built-in application-level caching (page, block, views etc.)
Install an opcode cache like APC
Swap out core systems to use alternative storage or technologies, whether that’s varnish for page caching, memcache, mongodb
Upgrade hardware
Tune database queries or other application logic to use less resources

Going back to resources, many of these options are only really available to people running on premium hosting, a VPS or higher - this assumes hosting costs of at least $15/month. Below that you have only lowest common denominator software available to you, which often won’t include an opcode cache. Once you’re on a VPS, then you’re mainly limited by your own ability to pick up sysadmin skills (or pay someone to do it for you), and there’s quite a big stretch in front of you, even running a couple of servers, before you get into serious hardware maintenance or wildly divergent configurations beyond a few different choices.

In terms of where Drupal core should perform best, I would aim it precisely at high-end shared hosting/VPS upwards. This assumes an opcode cache, and some degree of flexibility on installing other packages - although not necessarily that people have the expertise to do so, or to take advantage of them properly.

For sites on shared hosting, performance improvements in core (not just swappable subsystems) will benefit those too. Some issues can’t be dealt with without moving up from $5/month to $15/month hosting (even if we take steps to load less code, it still can’t replace an opcode cache - look at benchmarks with page caching enabled to see that in action). And while $15/month is three times as expensive as $5/month. It pales in comparison to paying someone to look at performance issues on your site (where $120 for the year might instead pay for an hour or so of development work).

However once you past a VPS or single dedicated server, things get a lot more expensive and a lot more complicated very quickly. It’s at that level that I think we really need to allow people to stay for as long as possible. Or at a minimum, if we introduce changes that only benefit sites on shared hosting, we need to make sure they don’t penalise sites once they get to the next stage.

For ‘big’ sites, we need to continue the trend of making as many subsystems as swappable as possible. But also recognise that only a small fraction of sites will actually do this - some sites become ‘big’ before they have any time to plan for it. The combined usage for the memcache, cacherouter and boost projects for Drupal 6 is less than 15,000, or 5% of installs that ping back to Drupal.org.

So in summary:

Page caching is good, we should keep it
The target for performance improvements should be cold starts and ‘normal’ authenticated traffic’
The archetype/baseline of Drupal hosting should be where most sites can move to (i.e. small VPS or premium shared hosting), not necessarily where they are now
We should avoid deferring performance and scalability to contrib except for the more extreme cases
For the very biggest sites, we mainly need to ensure that core can get out of the way without too many workarounds and avoid the need for core patches

Comments

Marcos (not verified)

Wed, 10/05/2011 - 01:51

Main menu

You are here

Drupal 8 performance and scalability - audience priorities

Tags:

Comments

Add new comment

Filtered HTML

Plain text