Drupal 8 performance and scalability - module bloat, part 1

UPDATE: while preparing the stats for this blog post, I confirmed a regression in page caching that had previously been reported by Heine, there's now an issue at http://drupal.org/node/1064212 which fixes a good proportion of that regression

Following the comments on audience priorities, since a good chunk of the discussion was around actual performance issues, I’m going to go straight into one of the main issues that we didn’t manage to do much with in Drupal 7 - module bloat.

Note that Drupal 8 development isn’t open yet, so all of the numbers here are from Drupal 7 HEAD. So this post is really about analysing issues in Drupal 7, and looking at how we might fix them both from the perspective of smaller fixes (possibly backportable) and larger architectural changes.

The main focus of this post will be memory usage, although by necessity it’ll go into CPU and other areas as well.

I’ll be comparing the following Drupal installs. In all cases profiling the default front page with one node, and with no configuration changes apart from css and javascript aggregation enabled.

Drupal 6, all optional modules disabled.
Drupal 6, all optional modules enabled, one node posted to the front page.
Drupal 7, (minimal profile) all optional modules disabled.
Drupal 7, (standard profile) all optional modules enabled, one node posted to the front page.

All of these with and without APC.

This is only a very rough comparison - even with only required modules, there are huge changes between Drupal 6 and 7 (for example Field module is required in Drupal 7), and things will diverge even more once you start adding modules (there’s no equivalents to RDF, contextual links, Fields etc. in D6 core). As sites add more modules, they usually add more functionality, content and configuration too, however in this case the only functionality we’ll be adding is from the different install profiles, one node on the bigger installs, and what you get just from having the modules enabled. But hopefully it’s enough variation to spot some patterns.

To get an understanding of ‘bloat’, there are a lot of different metrics. We can’t just compare the download size of Drupal 6 vs. 7 (as many people have) because that ignores things like javascript, css, tests - while this in some cases is extra code that needs to be maintained, it’s not run-time PHP code that affects your site’s normal operation.

Before going into the results we need some background:

First, Drupal isn’t only modules, it also has a lot of code loaded from /includes - things like the database, module and theme system, and the theme itself. However apart from some very rare exceptions, once you have a site up and running, you will only be adding to or changing your theme, and adding modules. So there’s a baseline of Drupal framework (including required modules) that you’re more or less stuck with, then everything else you can add on top.

Second, while this will probably get covered in the results, it’s worth going over what adding a module to a Drupal install actually means:

  • Any module in the Drupal install will be part of system scans such as on admin/build/modules
  • Any module that’s ever been enabled (even if currently disabled) is in the system table, may have some database tables and content associated with it.
  • During full bootstrap and page building/rendering:
    • Any module that’s enabled needs to have a certain amount of metadata loaded in module_list()/system_list()
    • All enabled modules are loaded with require_once
    • Loading a module will execute any define() statements. Functions and classes will need to be copied into local PHP memory (whether using an opcode cache or not), if not using an opcode cache the code itself has to be parsed and compiled.
    • If the module defines hooks such as hook_menu(), hook_theme() hook_field_info() and similar, the output of those will make itself into global caches such as the menu router and theme registry, which are loaded on most or every request
    • If the module implements hooks that are called as part of page building (hook_init() hook_form_alter(), hook_node_view()), this code will be executed during the request

This covers most of what happens, I may have missed some so corrections welcome. This means that depending on the module, its effect on baseline performance and memory usage, even before doing any configuration, can vary widely.

To compare the 5 installs at a high level, I collected the following, all as an anonymous user, no page caching:

  • Number of modules
  • xhprof summary: cpu, memory usage, number of function calls.
  • Number of constants, user functions and classes defined (collected at the end of full bootstrap).
  • Number of files cached in APC (after an apache restart and two requests to the front page)
  • Size of the APC cache (gives some indication of code weight)
  • Requests per second when running ab -c1 -n1000 - all on localhost so very rough, but better than nothing

Let’s get the metrics first, then we can see what’s behind them. Note there are 1484 constants and 129 classes defined by my PHP installation in a straight PHP file, before Drupal even gets involved. So take the raw numbers in that context.

 

No. modules

No. consts

 

No.user  funcs

No. classes

APC files

APC size(mb)

6-minimal

8

1581

873

129

39

7.5

6-all-optional

33

1598

1298

1129

64

11.1

7-minimal

9

1608

1334

169

55

12

7-optional

43

1655

2271

172

90

18.6

 

And some more:

 

 

 

Wall time(us)

CPU time (us)

Incl. mem use (bytes)

Peak mem use (bytes)

No. function calls

reqs/sec (no page cache)

reqs/sec (normal page cache)

6-minimal - APC

20,096

20,000

2,953,496

3,132,456

3,194

40.90

214.20

6-all-optional  - APC

55,173

50,000

5,147,808

5,429,072

6,318

24.54

203.06

7-minimal -APC

58,053

50,000

4,864,704

5,057,616

5,730

36.31

142.19

7-optional -  APC

93,170

80,000

7,654,864

7,935,488

9,503

24.88

129.12

6-minimal - no APC

62,146

60,000

10,393,440

10,574,432

3,194

18.74

89.24

6-all-optional  - no APC

104,422

100,000

16,734,328

16,992,464

6,318

12.28

72.94

7-minimal -no APC

113,013

100,000

17,246,288

17,424,248

5,730

11.45

51.47

7-optional - no APC

137,489

130,000

28,031,040

28,299,336

9,503

7.48

44.50

 

 

Since the number of modules are completely different between D6 and 7, we can try to divide things up to mean average extra added per module. Note that classes are lazy loaded in Drupal 7, so this only counts classes that were loaded, not all that might be found in includes.

 

 

 

Optional modules

funcs/module

classes/module

files/module

mem/module w apc

mem/module w/o apc

6-optional

25

17

0

1

87772

253635

7-optional

34

21.8

0.08

1.03

82063

317198

Just to re-iterate, this is not an apples to apples comparison. Drupal 6 optional modules are a very different set to what you get in Drupal 7, and some of the new modules in D7 are a lot more feaure-rich than modules that were removed from Drupal 6 (remember ping module?), so these numbers actually don’t mean very much. The interesting results will be looking at the actual xhprof output to see where the distribution is within the different requests. However while I was planning to post profiling results here too, this post is long enough that that’ll go in a part 2.

Immediate thoughts though:

- APC is great!
- Page caching has taken a serious hit in Drupal 7. I benchmarked it myself around a year ago and some regressions were fixed around that time so that it got back up even with Drupal 6, so presumably something has changed in the meantime during code ‘freeze’.
- Out of the box Drupal 7 performance really is slower in Drupal 7 than Drupal 6.
- Code weight accounts for some of this, and it’s not just optional modules, the minimal profile includes this too, however even just the minimal profile contains a tonne of new features, like Fields, so it’s not a fair comparison unless you take this into account.
- 43 modules isn’t all that many for a Drupal site. It would be good to add a Drupal 7 install with around 100 modules from the high usage contrib projects to the installs.

For me there are two main things to worry about:

1. What happened to page caching? (issue forthcoming)
2. Why do we use an additional 80kb of memory (on average) for every module even when APC is enabled? (Note there are already some answers to that question in the issue queue: http://bit.ly/eYkk6t).

Comments

intramuscular injection needle gauge pain pediatric seroquel and depersonalization hawthorn office chairs synagis injection phil yum tpn in florida university resorcinol glue keflex antibiotic information caffeine resonance structure ony days sareen sarna structural difference oxytocin vasopressin transderm scope patch jason perrigo wellbutrin quit smoking lumigan p slippery winds fish camp ontario trailer parking chestnut mountain galena lisinopril losartan coadministration hydroxyzine popped transformer resonance and switching surges when dose del mar fair open mtx amp power wire slave husband licks my pussy pictures of soy candles caplets recipe soft boiled egg cost of benicar hct usa pharmacy soma latrodectus species which lav mic clip lotronex attorneys san diego associated side effects of furosemide fluids that dissolve paper anticonvulsants and d o t law little apri collagen stimulating hormone ionamin 30 mg novolin n nph maxalt migrain open capsule macrobid in food lumigan eye drops where does goldenseal grow protein-bound paclitaxel phyl foster red 30mg oxymorphone how to make lsd with ergot emulsions science and technology zonisamide for seizures in canine joshua williams trafficing cocaine

[url=http://tyuitu.com/2011/05/21/chelate-gluconic-gluconates/]silver sulfadiazine chlorhexidine gluconate scabies[/url] [url=http://tyuitu.com/2011/05/21/dissolving-calculus/]secret to dissolving cocoa[/url] [url=http://tyuitu.com/2011/05/22/dihydrocodeine/]dihydrocodeine addiction[/url] [url=http://tyuitu.com/2011/05/22/bubbles-amphetamine/]herbal amphetamines work[/url] [url=http://tyuitu.com/2011/05/23/cyclosporine-aq-2/]order cyclosporine[/url] [url=http://tyuitu.com/2011/05/24/cmv-virus-symptoms/]cmv status[/url] [url=http://tyuitu.com/2011/05/24/fruity-2007-jelsoft-enterprises-ltd/]crack fruity loops[/url] [url=http://tyuitu.com/2011/05/25/muscle-relaxants-used-in-electroconvulsive-... relaxants[/url] [url=http://tyuitu.com/2011/05/26/luminal-a/]luminal ph[/url] [url=http://tyuitu.com/2011/05/26/portable-douche/]make your own anal douche[/url] [url=http://tyuitu.com/2011/05/27/pediatric-comfy-splints/]nodular mass clitoral hood pediatric[/url] [url=http://tyuitu.com/2011/05/27/alta-fab/]danelectro fab metal power supply[/url] [url=http://tyuitu.com/2011/05/27/atrovent-albuterol-wheezing-emergency-room/... nasal[/url] [url=http://tyuitu.com/2011/05/28/rancho-bella-rocas/]jaime roca[/url] [url=http://tyuitu.com/2011/05/28/how-to-jack-off-perfectly/]how to finger perfectly[/url] [url=http://tyuitu.com/2011/05/29/sulfur-springs-usa/]sulfur powder buy[/url] [url=http://tyuitu.com/2011/05/29/entex-semen/]reliant energy entex[/url] [url=http://tyuitu.com/2011/05/29/technetium-hmpao/]technetium element[/url] [url=http://tyuitu.com/2011/05/30/vicoprofen-withdrawal/]vicoprofen sales 2006[/url] [url=http://tyuitu.com/2011/05/30/rituximab-mabthera/]rituximab producing indian companies[/url] [url=http://tyuitu.com/2011/05/30/drug-lansoprazole/]information on lansoprazole medication[/url] [url=http://tyuitu.com/2011/05/31/la-mina-de-oro/]la monedita de oro[/url] [url=http://tyuitu.com/2011/06/01/anita-hertz-hbc/]hbc cable[/url] [url=http://tyuitu.com/2011/06/01/gill-sans-download-free-torrent/]sans souci llc[/url] [url=http://tyuitu.com/2011/06/02/getting-pregnant-with-progesterone-vaginal-... suppositories side effects[/url] [url=http://tyuitu.com/2011/06/02/carbodopa-levo/]carmen levo construction[/url] [url=http://tyuitu.com/2011/06/03/lisinopril-diabetic-nephropathy/]lisinopril and erectile disfunction[/url] [url=http://tyuitu.com/2011/06/03/watson-carisoprodol/]carisoprodol cpd[/url] [url=http://tyuitu.com/2011/06/04/do-vaccines-have-any-side-effects/]flue vaccine dangers[/url] [url=http://tyuitu.com/2011/06/05/callus-eliminator-solution/]skin lesion looks like callus[/url] [url=http://tyuitu.com/2011/06/05/hurricaine-shields/]hurricaine shutters[/url] [url=http://tyuitu.com/2011/06/05/alternatives-to-ritalin/]ritalin causing small digits[/url] [url=http://tyuitu.com/2011/06/06/gu-gels/]sticky gels[/url] [url=http://tyuitu.com/2011/06/06/guide-to-ease-anal-sex/]wall ease duct insulation[/url] [url=http://tyuitu.com/2011/06/07/estro-2009/]estro cappuccino espresso[/url] [url=http://tyuitu.com/2011/06/07/nubain-without-prescription/]order nubain[/url] [url=http://tyuitu.com/2011/06/08/inh-tb/]tetralogy of fallot inh propranolol[/url] [url=http://tyuitu.com/2011/06/09/daily-limits-for-cogentin/]cogentin in dystonic reactions[/url] [url=http://tyuitu.com/2011/06/09/stability-of-salts/]benefits epson salts[/url] [url=http://tyuitu.com/2011/06/10/calcium-oxalate-crystals/]low calcium oxalate diet[/url] [url=http://tyuitu.com/2011/06/10/feverfew-seed/]is feverfew safe[/url] [url=http://tyuitu.com/2011/06/11/ninja-250-nitrous/]fleetwood nitrous hyperlite[/url] [url=http://tyuitu.com/2011/06/11/phenothiazine-reaction/]phenothiazine antipsychotic drugs[/url] [url=http://tyuitu.com/2011/06/12/the-drowsy-chaperone-seattle/]the drowsy chaperone musical[/url] [url=http://tyuitu.com/2011/06/13/side-effects-of-phenergan/]can phenergan be given with methotrexate[/url] [url=http://tyuitu.com/2011/06/13/pdms-to-cadworx/]pdm file sharing service[/url] [url=http://tyuitu.com/2011/06/14/opana-picture/]opana er price[/url] [url=http://tyuitu.com/2011/06/15/naltrexone-vs-campral-giving/]buy campral online[/url] [url=http://tyuitu.com/2011/06/15/animated-nephron/]physiology of nephron[/url] [url=http://tyuitu.com/2011/06/16/does-ambrotose-contain-gsh/]side affects ambrotose[/url]