2. Mathew Beane
@aepod
Director of Systems Engineering - Robofirm
Zend Z-Team Volunteer – Magento Division
Family member – 3 Kids and a Wife
Magento Certified Developer
Scaling Magento – Mathew Beane - php[world] 20152
3.
4. Scaling Magento – Mathew Beane - php[world] 2015
Magento: What is it?
• Open-source e-commerce platform
• PHP based application
• Core utilizes Zend Framework 1
• Very flexible, built to modify
• Extremely scalable, supports huge stores
• Market leader and still growing
• Magento 2 is right around the corner
• 2.0.0-rc2 is out now
• http://magento.com/developers/magento2
5. Scaling Magento – Mathew Beane - php[world] 2015
Cluster Configuration Overview
https://github.com/aepod/magento-digitalocean-sandbox
• Uses Puppet to build out cluster
• To setup the initial test checkout readme in project
• Initial release standard (n)webnodes + single db config
• Monolith server has Web, PHP, DB, Redis all-in-one
• We will come back to this later
6. Scaling Magento – Mathew Beane - php[world] 2015
Using the Magento Vagrant Puppet Sandbox
1) Install Vagrant
2) Install Digital Ocean provider
3) Grab the project source from github
git clone https://github.com/aepod/Magento-Vagrant-
Puppet-Sandbox.git
4) Add your Token to Vagrantfile
5) Ensure you have a ssh key
6) Stand up some boxes
vagrant up puppet
* Add puppet IP from output to Vagrantfile
vagrant up monolith
Vagrant
https://www.vagrantup.com/
Digital Ocean Provider
https://github.com/smdahlen/vagrant-digitalocean
Magento Vagrant Puppet Sandbox
https://github.com/aepod/magento-digitalocean-sandbox
7. Scaling Magento - Todays Plan
• Introduction
• Magento Application: Preparing and Optimizing
• Magento Cluster Architecture: Examine Typical Layouts
• Magento Digital Ocean Sandbox: Demonstrating Clustering
• Magento Performance Toolkit: Measuring Performance
Benchmarks
• Magento Digital Ocean Sandbox: Guided experiments
• Conclusion: Open Q & A
9. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
10. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Overview
• Optimize sysctl.conf: Increase limits for the servers
• Modify Nginx settings:
• Setup PHP FPM with TCP properly
• Optimize Worker Connections
• Tune up caching and other settings
• Optimize PHP FPM
• Process Manager Optimization
• APC / Zend Opcode Cache
• Tune up other settings
• Optimize Redis
• Optimize MySQL
The goal is to optimize flow.
11. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Linux Kernel
• Sysctl is an interface for configuring the Linux Kernel.
• The following slides show some examples of the
settings you may change in the sysctl.conf.
• Remember to backup your configurations.
• After changing you run: sysctl –p to update your
kernel settings.
• Always test after changing values, quantify your
results against previous tests.
See also: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html
12. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Linux Kernel
# Controls the maximum shared segment size, in bytes (64gb)
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages (16gb)
kernel.shmall = 4294967296
#controls the number of process identifers that can be created
kernel.pid_max = 2097152
# Number of open files handles
fs.file-max = 2097152
# Defaults to 65536 – Deals with malloc maps.
vm.max_map_count = 262144
You can find a lot more tuning information by looking up the parameters and learning more about them.
13. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Linux Kernel
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.netdev_max_backlog = 65536
net.core.somaxconn = 65536
net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.tcp_wmem = 4096 1048576 16777216
net.ipv4.tcp_max_syn_backlog = 65536
net.core.somaxconn is the least trivial, and typically set very low causing the following:
“apr_socket_recv: Connection reset by peer” and “connect() … failed“.
Queuing up packets efficiently, make sure you
have enough space in your buffers.
15. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Nginx
Battle Hardened Magento Nginx Config:
https://gist.github.com/gwillem/cd5ae6845fa33aa0d481
worker considerations:
Worker Processes = Total # CPU Cores
Worker Connections: ½ of ulimit –n
Max Connections = worker processes * worker connections
Worker_rlimit_nofile safe value = 2 * Max Connections
keepalive_timeout mitigates overage by trading for latency
16. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Nginx
# Media: images, icons, video, audio, HTC
location ~* .(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
expires 1M;
access_log off;
add_header Cache-Control "public";
}
Set long expires dates on media assets.
Bonus: Use a CDN to offset all Media and CSS calls.
17. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Nginx
curl -X GET -I www.idealphp.com:80/skin/frontend/rwd/default/images/logo.gif
HTTP/1.1 200 OK
Server: nginx/1.0.15
Date: Tue, 14 Apr 2015 17:56:32 GMT
Content-Type: image/gif
Content-Length: 2320
Last-Modified: Tue, 14 Apr 2015 00:30:08 GMT
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Cache-control: public
Test your headers using curl or other tools
18. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – PHP-FPM
• PHP 5.4 - Minimum
• PHP 5.5 – Great, better performance if you have compatibility
• PHP 5.6/5.7 - Even Better, although not officially supported
• PHP 7 / HHVM - No official support until Magento 2.
Magento 1 HHVM: https://gist.github.com/tegansnyder/96d1be1dd65852d3e576
- Lots of work went into this by Tegan Snyder (@tegansnyder)
- Thanks to Daniel Sloof (@daniel_sloof) for doing MUCH of the groundwork allowing this to work
• Use TCP instead of sockets
• Test with Zend OpCache instead of APC
• Turn off APC Stat/Zend Revalidate
apc.stat = 0
opcache.revalidate_freq=0
19. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – PHP-FPM
Tune Process Manager
• pm: ondemand
• pm.max_children: about 6-8 per CPU *more later
• pm.max_requests: 10000 *php threads should die for garbage collection
• listen.backlog: set to 64000 (typically set to -1 or 128)
Listen backlog is limited by net.ipv4.tcp_max_syn_backlog
*only do after sytsctl.conf optimization
20. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – PHP-FPM
pm.max_children will limit the amount of memory php uses overall
• Calculate the memory a thread is using while under load:
ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"M") }'
• Use the total amount of memory you want php to consume:
pm.max_children = Total RAM dedicated to the web server / Max child process size
Based on this, you can set your max_children based on memory instead of the CPU metric used earlier. This will allow for
more memory consumption in production environment. Be sure to test with higher values, more is not always better.
Further Reading: http://myshell.co.uk/index.php/adjusting-child-processes-for-php-fpm-nginx/
21. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Redis
• Redis is the preferred cache for Magento
• sysctl ulimit values affect the redis maxclients
• Use three separate Redis instances with a single DB in each, on different ports
• tcp-backlog can be increased, adjust sysctl somaxconn and tcp_max_syn_backlog
• maxmemory can be set very low typically.
• Using Sentinel and twemproxy you can horizontally scale
• Clustering redis to eliminate all single threaded bottlenecks and single points of failure
• Requires orchestration, and Magento application changes
• Don’t forget to update Cm_Redis to newest version (Magento lags behind)
• At Rackspace or AWS: Use Object Rocket https://objectrocket.com/ for Redis SaaS
22. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – MySQL
• Turn off MySQL Query Caching – This is single threaded and not of use with Magento
(* Before MySQL 5.7)
• Innodb settings can be tweaked to consume more memory
Selected Sample Configuration Settings for a 32GB Dual Hex-Core Server
myisam_sort_buffer_size = 64M
key_buffer_size = 256M
join_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 4M
sort_buffer_size = 8M
table_open_cache = 8192
thread_cache_size = 512
tmp_table_size = 384M
max_heap_table_size = 384M
max_allowed_packet = 1024M
query_cache_type=0
query_cache_size=0
innodb_thread_concurrency = 24
innodb_buffer_pool_size = 16G
innodb_log_file_size = 384M
innodb_log_buffer_size=24M
innodb_additional_mem_pool_size =
16M
innodb_io_capacity = 800
innodb_concurrency_tickets = 900
innodb_flush_neighbor_pages=cont
innodb_lock_wait_timeout=75
innodb_flush_method=O_DIRECT
23. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – MySQL
• Clustering MySQL for Magento Requires all tables to be converted to InnoDB
• catalog_full_text_search requires MyISAM, unless your running MySQL 5.7
• When clustering always use identical hardware for each server
• Master/Master is not supported by Magento
• You can specify read servers and even load balance them
• It is unusual for Magento to require a true MySQL Cluster, as it is typically not the
bottleneck
24. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Hints – Other
• Varnish is great, however your application must be suitable
• Adding Varnish to a slow site is like putting a bandage on a neck wound
• Varnish complicates logging and request tracking throughout the application
• Lesti_FPC is a good redis ready FPC for Community Edition.
• https://gordonlesti.com/projects/lestifpc/
• CDNs can have a great impact on end-user performance
• Reduction in requests is also beneficial to the web server
• Typically one of the cheapest/best moves to make
• Always test any changes you make
25. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
26. Scaling Magento – Mathew Beane - php[world] 2015
What You Don’t Want To Scale
A very good un-cached response, good ready to scale.
A typical bad response, scaling this will only cause you heartache.
28. Scaling Magento – Mathew Beane - php[world] 2015
Magento – “Clean Core”
Keep a clean core. If you inherit a project, one of the first
things you should do is clean up core. This means you
have not changed any default files including templates,
layouts and code.
Ben Marks, Magento Community Evangelist @benmarks
(Protector of the Core)
29. Scaling Magento – Mathew Beane - php[world] 2015
How to Keep Core Clean
Source: https://joind.in/talk/view/16204
• Magento allows class overrides on nearly
everything.
• Magento’s Event/Observer mechanism is pre-
built into most business logic. You can add
more if you need.
• Designs/Templates have a fallback system that
allows you to utilize any of the core layouts and
templates, or replace them in your current
design.
• Untouched index.php. Outside of development
and very rare implementations you should not
“need” to edit this.
30. Scaling Magento – Mathew Beane - php[world] 2015
Magento Project Mess Detector
• Find and eliminate your core changes with:
https://github.com/AOEpeople/mpmd
Magento Project Mess Detector by Fabrizio Branca
31. Scaling Magento – Mathew Beane - php[world] 2015
Refactoring Bad Magento Projects – Further Study
Fabrizio Branca’s work on cleaning up Magento projects:
http://fbrnc.net/blog/2014/12/magento-update
In this article he goes into depth on the process of separating the core files,
and using Modman and composer to package everything.
Tegan Snyder's work on building modman configurations for extensions out of
existing source:
https://github.com/tegansnyder/meff
This will allow you to generate modman files for your extensions which is
handy in the process of cleaning up.
32. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Your Application with Zend Z-Ray
• Use Z-Ray to profile and tune your Magento Application
• Z-Ray allows you to see every request, and many details about the
request.
• It’s easy to install, free to try and really amazing.
• It can be used to debug Mobile browsers through the Zend Server
backend
• Z-Ray No longer requires Zend Server:
http://www.zend.com/en/products/z-ray/z-ray-preview
33. Scaling Magento – Mathew Beane - php[world] 2015
Optimizing Your Application with Zend Z-Ray
Demo: http://serverdemo.zend.com/magento/
Blog Article: http://aepod.com/debug-magento-using-zend-zray/
34. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
35. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle
• Should apply equally to the
hardware architecture as to
the software application.
• Every project should have a
continuous development
cycle in place, and ensure
that it is well documented
and easy to follow for all of
the people involved.
36. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle - Simplified
Design: Plan our your sprint, stick to Agile.
Develop: Toward a release on a schedule
• Test: Build PHPUnit and Selenium tests into your process
• Code Review: Pull requests and code review EVERYTHING.
• User Acceptance Tests: Make sure the client is happy first.
Deploy: Package, test and deploy.
• Deployments: Automate, and make the production servers hands off.
37. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle – Agile / Scrum
• Goal based system that encourages teamwork.
• Daily communications with all involved in project.
• Flexible and focused on delivery.
• Encourages Teamwork and Accountability.
38. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle – Branching
• Keep a clean core
• Never edit core files. Ensure that you can see any core changes in
production
.
• Release / Feature Branching
• Choose a branching methodology and build around it.
• Use Pull Request
• Code review increases the quality of your code. Make pull requests part of
your workflow.
39. Scaling Magento – Mathew Beane - php[world] 2015
Successful Git Branching Model
http://nvie.com/posts/a-successful-git-branching-model/
• Use Tagging as you release to master
• Types of branches:
• Master: Tagged versions, what's on production.
• Develop: What will be merged into master with
the next release.
• Release: Preparing for release. Allows for last
minute fixes, testing and prepping.
• Feature: Usually checked out by developers to
work on features.
• Hotfix: Fixes aimed for unplanned production
releases.
40. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle – Testing
Build testing into all of your development cycles.
Developers are working on testing with Magento 1
BDD: https://speakerdeck.com/alistairstead/bdd-with-magento
BeHat: https://github.com/MageTest/BehatMage
PHPSpec: https://github.com/MageTest/MageSpec
Magento 2 has a built in Test Automation Framework ( PHPUnit,Selenium )
https://wiki.magento.com/display/MAGE2DOC/Magento+Automated+Testing+Guide
https://github.com/magento/taf
https://alankent.wordpress.com/2014/06/28/magento-2-test-automation/
41. Scaling Magento – Mathew Beane - php[world] 2015
Do Not Edit on Production
• Use a deployment tool, and LOCK DOWN
production. No one should ever have to
touch an application file manually for any
reason.
• If you can extend this practice to your OS
and server stack you now have fully
engaged DevOps.
42. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle – Testing (Example of Fail)
After introducing SOLR indexing, testing
showed serious issues as it indexed.
43. Scaling Magento – Mathew Beane - php[world] 2015
Development Cycle – Application Packaging
• You must be able to package your application in order to deploy it to
a cluster.
• Deployment infrastructure may determine what packaging you will need to
use, for instance one of the following:
• Zend Server offers Application Packaging and Deployment
• A simple “golden copy” (tgz or flat) that is pushed to web nodes.
• RPM or other proprietary packaging tool.
• Use Composer and/or modman to help maintain your application
packages.
• Composer can even be used to install Magento Core
• Make sure your build process is repeatable and testable.
44. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
45. Scaling Magento – Mathew Beane - php[world] 2015
Deployment Tools – Typical Software
• When deploying an application to a cluster of application
servers, a deployment tool is a requirement.
• There are many choice, here are a couple typical deployment tools.
• Capistrano: Written in Ruby but well accepted and great to work with
• Jenkins: Another example of Deployment Automation
• Bamboo: Part of the Atlassian stack also includes testing and other features.
• Roll Your Own: This is more common, using bash scripts and other tools you can build a project
deployment tool fairly easily.
46. Scaling Magento – Mathew Beane - php[world] 2015
Deployment Tools – Some Requirements
• Integrated with code versioning
• Supports Multi-Server
• Supports the Application
• Handles Maintenance Mode automatically
• Runs Installers
• Clears Caches
• Low Downtime
• Rollback Procedure (a/b switch)
• Trigger events such as shell scripts, or CDN clearing
• Nice to have – May not be required
• Integrated testing
• Packaging
• Integration to GUI Interfaces
Deploying applications manually is a lot of work.
47. Scaling Magento – Mathew Beane - php[world] 2015
Deployment Tools – Further Study
I highly suggest researching Fabrizio Branca’s work on the subject:
http://www.slideshare.net/aoepeople/rock-solid-magento
Fabrizio has a ton of articles and slideshares on the subject.
Also check out Joshua Warren’s slides, he has several really
informative presentations on the subject.
http://www.slideshare.net/joshuaswarren/
48. Scaling Magento – Mathew Beane - php[world] 2015
Deployment Tools - Server Deployments
• Automate server deployments:
• Puppet
• Chef
• Ansible
• Salt
• Centralize configurations
• Automated updates and server onlining process
• Take out the human factor!
49. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
50. Scaling Magento – Mathew Beane - php[world] 2015
Load Testing and Metrics
"Don't rate potential over performance."
- Jim Fassel
Blaze Meter
Using Blazemeter you can
easily build repeatable tests,
with very nice graphs.
(based on JMeter)
Gatling
http://gatling.io/
On par with Blazemeter.
JMeter
Very effective, without
having to purchase a SaaS
Siege
Can be used minimally to
simulate some types of
load.
51. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit
https://github.com/magento/magento-performance-toolkit
The Magento Performance Toolkit is a set of automated tools that enables you to quickly and
consistently measure the application performance of Magento using Apache JMeter. The
toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for
the performance measurement.
From: Magento Performance Toolkit Users Guide
More Reading: http://aepod.com/using-the-magento-performance-toolkit/
52. Scaling Magento – Mathew Beane - php[world] 2015
JMeter Tests for “Normal Sites”
The Magento Performance Toolkit can be crafted to pull data from production sites,
although its complicated and you may want to cover your site using your own JMeter
tests.
Gathering the steps for a client site typically takes a day or more depending on the
complexity of the site design, and the checkout.
Developing and debugging this can be a real time sink depending on how far down the
rabbit hole you want to go.
53. Scaling Magento – Mathew Beane - php[world] 2015
Using Siege
siege –t 5M –b –c 5 http://www.idealphp.com/
This will start up 5 threads and hammer the URL for 5
minutes.
Siege is very useful, but limited. It’s easy to put a lot of
traffic on a box with it, but the traffic will be limited to
GET requests.
You can pull a list of URL’s from the Magento sitemap
and run through random requests, this can even be
used to do some pseudo-warming of the cache.
54. Scaling Magento – Mathew Beane - php[world] 2015
New Relic - Introduction
• Analyze software in Real-time
• Collect Data
• Application Data
• Server Data
• Browser Data
• Magento Data *
• Synthetics provides NrSQL
• Alerts
* Magento data provided via Magento Extension:
http://www.magentocommerce.com/magento-connect/new-relic-reporting-29133.html
55. Scaling Magento – Mathew Beane - php[world] 2015
New Relic – Production
57. Scaling Magento – Mathew Beane - php[world] 2015
Load Testing Production
1. Prepare a downtime:
Forward site to a non-magento maintenance page
Exclude Testing IP – So tests can still proceed
Disable Cron
2. Copy Production Database
3. Point Magento at the new copied DB
4. Modify Magento Admin Settings in copied DB
Use Test Payment Gateway
Disable Transaction Emails
5. Test against Copied DB
6. After Testing is complete:
Point site at original DB
Put site back up on live
Enable Cron
Using this method, you can test against production on
production. You will need to bring the site down during the
time period, but if this is done at a low point in traffic it
causes little to no stress to the customers.
58. Before You Start Clustering
Optimized Server Settings
Magento application is optimized and clean
Development pipeline in place
Deployment infrastructure is solid
Rigorous testing protocols are in place
Pre-Scaling Checklist
Next Up: Discussion about typical cluster architecture and components
60. Scaling Magento – Mathew Beane - php[world] 2015
Magento Cluster Architecture – Typical Layout
• Load Balanced – Front End Webservers
• Admin / Backend Server Separated
• Single MySQL Database Master, with Slaves
61. Scaling Magento – Mathew Beane - php[world] 2015
Magento Cluster Architecture - Components
Web Servers
• Apache / Nginx
• PHP 5.4 + (depends on Magento Version)
• Multiple Webservers – Easy to cluster
• Varnish Can be used as a Reverse Proxy
• Nexcess Turpentine is the standard Magento Extension
• Magento2 has Varnish support out of the box
62. Scaling Magento – Mathew Beane - php[world] 2015
Magento Cluster Architecture - Components
Database
• MySQL / Percona
• Master / Slave
• Slaves are typically used for reporting or Hotswap for
failure
• Can be set to read servers for application, but it
requires some work
• Solr / Elastic Search
• Typically run on the Database or Backend Server
• Can power Search or other Magento business logic
63. Scaling Magento – Mathew Beane - php[world] 2015
Magento Cluster Architecture - Components
Other Components
• File Server
• NFS – This is by far the most common
• NAS – Can be used on high end clients
• Redis / Memcache
• Deployment Tools
• Monitoring
• ELK – Log aggregation across servers
• Puppet / Chef
64. Scaling Magento – Mathew Beane - php[world] 2015
Magento Cluster Architecture - Bottlenecks
Breaking Points, where you need to adjust first.
1. PHP – More web nodes, easy to do and very performant.
2. Filesystem – Use a CDN to offset static asset traffic.
3. Redis – Cluster Redis using sentinel/nutcracker.
4. Database – Look at read server clustering
At some point adding Varnish should be considered as well. Varnish requires a clean
application with some thought as to which blocks can be loaded dynamically via AJAX.
65. Scaling Magento – Mathew Beane - php[world] 2015
Scaling Implementation – A Scoring System
Concern Score (1-5)
What is the simplicity of the solution?
How difficult is it to support?
Is it extensible, how broadly does it apply?
Will it break often?
How easy is it to update?
Is it Expensive?
Concern Yes / No
Does it require specialized services? (os level services, not typically installed)
Does it require Magento application changes?
Does it create a single point of failure?
Yes = 5 Points, No = 0
66. Scaling Magento – Mathew Beane - php[world] 2015
Scaling – Web Server Concerns
• Simple to Load Balance, most sites start here
• Challenges include the following:
• Session Management: should be a no brainer if your using redis for sessions
• Shared file System: Use NFS for media, keep all code local to web servers
• Fencing: easily solved with either software or hardware
• Log Collection: ELK, Rsyslog or Splunk
• Keep the system builds simple, repeatable and automate if possible
67. Scaling Magento – Mathew Beane - php[world] 2015
Scaling – Web Server Concerns (Automation)
How to automate:
Create an complete image to work from - include puppet so it can pull from a puppet
master
The puppet master spins up your webserver stack
You have your deployment process in place, so deploy the application and spin up the
node
Things of Note:
• Be prepared to lose nodes, the more you have the more likely failure is
• When a node runs amok, you must be prepared to kill it dead
• Webservers typically wont fail, and fencing is more of an issue with db/redis.
68. Scaling Magento – Mathew Beane - php[world] 2015
Scaling - File systems
Best practice for sharing files: Use NFS
• Most other options are not viable. (glusterfs, iscsi, and so on)
• NFS will not be the bottleneck if your application is configured correctly
• Share only media/ to avoid bottlenecks
• Client requirements may drive cluster toward a NAS
Network Attached Storage (NAS)
• Typically only on very large sites
• Expensive but very reliable and robust
• Entry level has very high bar (typically 20+ drives)
69. Scaling Magento – Mathew Beane - php[world] 2015
Scaling – File Systems and Symbolic Links
• Share specific folders via symbolic links
• Maintaining these when deploying is easy, just remake symbolic links or copy them
• Mount /var/magento/shared/[directory] via NFS, typically served from backend server
• Example: [docroot]/media -> /var/magento/shared/media
• Can also be used for local content
• Not mounted via NFS, writes local var files for this server only
• Example: [docroot]/var -> /var/media/local/var
70. Scaling Magento – Mathew Beane - php[world] 2015
Scaling – Redis Clustering
“Redis clustering using sentinel is easy to set up. Adding twemproxy allows for a highly
scalable Redis cluster and you get auto fail over and a ton of other benefits with this
configuration. This arrangement can also remove your Redis single point of failure.”
http://aepod.com/clustering-magento-redis-caching-with-sentinel-keepalived-twemproxy-and-twemproxy-agent/
• Easy to setup… well not really
• Very Complex, requires orchestration
• Sentinel Monitors Redis clusters
• twemproxy handles sharding
• twemproxy agent monitors sentinel
• Very robust when setup, nothing is single
threaded, everything is HA and the speed….
• Can be transparent to Magento despite the
complexity
71. Scaling Magento – Mathew Beane - php[world] 2015
Redis (Sentinel / twemproxy)
Requires several layers of applications
• Load Balancer: Load balances redis traffic to the twemproxy servers
• Twemproxy: proxy server, typically setup as a cluster of 2 servers (can be mixed on
other servers)
• Nutcracker Agent: Connects twemproxy to sentinel to keep track of shards of servers
• Sentinel: Monitors the redis instances and maintains availability
• Redis: Dozens of redis servers, sharded, maintained and fenced by Sentinel and
nutcracker
• VERY Complex
72. Scaling Magento – Mathew Beane - php[world] 2015
Scaling - MySQL
• Should be considered a low value target, except for High Availability concerns because
Magento typically is not bottlenecked by MySQL
• Using Percona XtraDB you can add read slaves
• Magento only supports a single Master/Write server, with multiple read servers
• Setting up load balanced READ db servers is not overly difficult, but doesn’t offer much
if any performance benefits
• Cluster DB will require more advanced fencing
• Typical production sites do not use load balanced or clustered MySQL
73. Scaling Magento – Mathew Beane - php[world] 2015
Database Servers (Percona)
• Percona XtraDB to cluster MySQL
• Percona XtraBackup for duplicating and rebuilding nodes
• Percona Toolkit to help debug any issues your running into
• Difficult to scale Write Servers
• Scale out your read servers as needed, but MySQL reads are rarely the bottleneck
• Typically Slave server is used for backup and hot swap, NOT clustering.
A couple quick tips:
• Not all tables in Magento are InnoDB, converting the MyISAM and Memory tables is OK
• You will need to be able to kill read servers and refresh them. (Percona XtraBackup)
• Use your Master server as a read server in the load balancer pool, when you kill all your read servers, it can fall back to
master.
75. Scaling Magento – Mathew Beane - php[world] 2015
Load Balancers – Software / HAProxy
• HAProxy: a free fast and reliable solution featuring high availability, load
balancing and proxy for TCP and HTTP-based applications.
• HAProxy can be used for web servers, database servers, Redis and
any other TCP based application.
• Easy to configure using a puppet module
• Can be clustered easily for high availability concerns
• Handles L4 Load Balancing if you want to do fancy multi-application
environments
• Built in stats interface
SOFTWARE
76. Scaling Magento – Mathew Beane - php[world] 2015
Hardware Load Balancers
• F5 Hardware load balancers are a standard
• Brocade is another common brand
• Rackspace offers a very easy to use web
interface to maintain a hybrid infrastructure
• Hardware load balancers offer a turn-key
mature solution.
• Typically Managed Service, with very hands
off approach.
HYBRID
Hybrid Load Balancers
• AWS Elastic Load Balancers
• Rackspace Cloud Load Balancers
• Cloudflare
• Many others…
Load Balancers – Hardware & Hybrid
77. Scaling Magento – Mathew Beane - php[world] 2015
Load Balancers – Making a choice
• Budget concerns will drive this decision
• Hosting Choices will affect availability, costs and toolsets
• Start locally with HAProxy and build test clusters using vagrant
• HAProxy can still be used, even with a hardware load balancer in
place.
78. Scaling Magento – Mathew Beane - php[world] 2015
Scaling - Others
• Reverse Proxy: Varnish or Nginx you can and should cluster these
• Puppet likes to be clustered: Mastering Puppet by Thomas Uphill is a great reference
• Monitoring Tools: You’re running these single threaded, what if the monitoring server
fails?
• Warehouse Integrations, ERP Layer Stuff, everything may need to be multi-threaded
• Search back ends such as solr or elasticsearch
• Consider all of the parts of the business that surrounds the Magento Application and try
to encapsulate them
79. Scaling Magento – Mathew Beane - php[world] 2015
Reverse Proxy (Varnish) Scaling Concerns
Turpentine is the standard go to extension for Magento to tie to Varnish
• Handles ESI
• Handles Dealing with updating VCL
• http://www.magentocommerce.com/magento-connect/turpentine-varnish-cache.html
• Varnish will replace your FPC, toggle it FPC off if you are using varnish
• https://github.com/nexcess/magento-turpentine/wiki/Installation
• Typically Varnish will run on each render box, and proxy to the local nginx.
• Issues will typically show up as over caching of blocks or other bits of front end page
• SSL is NOT supported by varnish, terminate it at your load balancer
80. Scaling Magento – Mathew Beane - php[world] 2015
Reverse Proxy (Nginx) Scaling Concerns
• NGINX is a great reverse proxy and load balancer
• http://nginx.com/resources/admin-guide/reverse-proxy/
• Can be used to effectively buffer and split requests to different cluster servers
81. Scaling Magento – Mathew Beane - php[world] 2015
Auto-Scaling Strategy
• Insert puzzle building analogy joke here: http://www.wikihow.com/Assemble-Jigsaw-
Puzzles
• Each hosting environment has its own quirks and add on top of that the business logic
requirements you will almost always have a unique infrastructure for every client
• Build small pieces and work them into the larger picture, you can get a lot of
performance with a few minor changes.
• Test everything you do, keep detailed notes on the configurations and compare against
the previous tests
83. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit
https://github.com/magento/magento-performance-toolkit
The Magento Performance Toolkit is a set of automated tools that enables you to quickly and
consistently measure the application performance of Magento using Apache JMeter. The
toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for
the performance measurement.
From: Magento Performance Toolkit Users Guide
More Reading: http://aepod.com/using-the-magento-performance-toolkit/
Magento CE 1.9.1.0 Ready Version: https://github.com/aepod/magento-performance-toolkit
84. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – generate.php
• Importing can take a long time, and may require
changes to php and MySQL settings
• If it fails, it fails hard. Restart with a fresh DB
• You can modify the profiles to better suite your
testing requirements
• Using the toolkit requires putting it in the
dev/tools/performance_toolkit/ directory in the
Magento install you want to use it on
• The scripts requires this exact directory, due to
includes and autoloading
• Oftentimes will require you to remove the tmp dir
inside the toolkit
85. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – Generate Profiles
• You can create your own profiles by copying and modifying the XML in profiles/
• Large and X-Large take extremely long times to import and require changes to
the PHP and MySQL settings and it still likes to fail hard.
• After it completes import, double check the following
• Front end still up and running and shows the framework categories
• Admin and Frontend logs in correctly
• Indexes and Caches rebuild correctly
86. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – jMeter GUI
Modify the tests, learn how they work and dig into errors you may be having running them.
jMeter GUI
• Easy to use
• Much easier than editing
XML for the tests.
• Debugging requirement
• Load/Save XML
• Tends to crash
when on windows
87. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – JMeter Configuration
./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=30 -Jramp_period=30 -
Jreport_save_path=./ -Jloops=2000 -Jadmin_user="admin" -Jadmin_password=“adminpass";
• Host: host to check against
• Base Path: for building the url
• User: Roughly the # of threads (more on next slid)
• Ramp Period: How long to take starting up
• Report Save Path: Where it saves the summary
• Loops: How many times it will try to loop through (except if it runs out of customers)
• Admin User/Pass: The admin user and pass for the admin site.
• This is expected to be at http://www.idealphp.com/admin/
• Any JMeter Attribute from top level can be set from command line using –Jattribute=
89. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit –Users Demystified
The JMeter tests use 4 thread groups to
accommodate the testing.
• Category Product Browsing(30%): Opens the
homepage, goes to a category, opens up 2 simple
products and then a configurable.
• View Product Add To Cart(62%): Same as Category
browsing however it adds each of the items to the cart
in addition to viewing them.
• Guest Checkout (4%): Same as View/Add to Cart, in
addition it runs through the checkout as a guest user.
• Customer Checkout (4%): Same as View/Add to Cart, in
addition it runs through the checkout as a logged in
customer.
Thread groups create threads based on the number
of users and the above percentages(by default).
users * group_percent / 100
• This result is rounded down, and can be 0 for
checkouts, making no checkout threads.
• You can set EACH group independently, and they do
not need to add up to 100% or less in totality.
90. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – More Attributes
Any JMeter Attribute from top level can be
set from command line using –Jattribute=
Customer and Admin Password must match
what was used during the generate.
Orders are tricky and used to calculate
“users” which are used within the thread
groups.
See jmeter test setup Thread Group -> validate properties and count users
91. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit –JMeter Test Plan
Setup Thread Groups
1. Extracts Categories from Top Menu
2. Searches for Simple Products, builds list
3. Searches for Config Products, builds list
4. Logs in on the Admin
5. Grabs a list of customers from the customer grid
6. Builds users count which is used by the threads.
Loops through Each Thread Group
1. Category Product Browsing
2. Product Browsing and add items to cart
3. Guest Checkout
4. Customer Checkout
Teardown
1. Just removes some stats, but you could extend this
spot for some killer stats.
jMeter test, building the current test. Notice it takes quite a
few requests to get started.
jMeter tests running, you can see the “customer count”
decrementing as it runs through the tests.
92. Scaling Magento – Mathew Beane - php[world] 2015
Using Newrelic During JMeter tests
93. Scaling Magento – Mathew Beane - php[world] 2015
Using Newrelic During JMeter tests
95. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – Maxed Orders
./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10
-Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password"
-Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0
-Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100
(This can be done in the JMX as well)
Non-Trivial Settings:
View Catalog : 0%
View Product Add to Cart : 0%
Customer Checkout: 0%
Guest Checkout: 100%
This test will slam orders through as fast as it can. Using it is as simple as recording the number of orders at the start and finish
of a test, and then calculating the number of orders per hour on that.
Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders)
96. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – In Production
To use the Magento Performance Toolkit on a existing built out production website:
1.Follow the testing on a production site plan from earlier.
2.Ensure your top-menu matches the JMeter tests search for top_menu, you can switch the jmeter test or
try to modify the template to match.
3.Modify the generate script to not create categories
4.Generate your data
5.Run your tests
Other Issues:
Not a trivial issue, it will take some time to setup.
Checkout and cart parts probably will fail and require changes to tests
Not the cleanest methodology, but it does work if you make the effort
97. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – Benchmarking
• Find metrics to base any decisions you make, and quantify the results
• 30 Threads per web head is a good spot to start (except in the VM)
• Orders Per Hour
• Collect test start and end orders
• (end orders – start orders) * (3600 / time to complete)
• Customers Per Day
• (86400 / time to complete) * Online Customers
98. Scaling Magento – Mathew Beane - php[world] 2015
Magento Performance Toolkit – Benchmarking
• Max Response Time: Should stay below 10 seconds in a good test
• Average Response Time: Should be below 500ms in a good test
• Error Rate: 0.00% is what your looking for, some glitches may happen
• Collect New Relic Stats or other stats and use them to supplement the
tests
100. Scaling Magento – Mathew Beane - php[world] 2015
Using the Magento Vagrant Puppet Sandbox
1) Install Vagrant
2) Install Digital Ocean provider
3) Grab the project source from github
git clone https://github.com/aepod/magento-
digitalocean-sandbox
4) Add your Token to Vagrantfile
5) Ensure you have a ssh key
6) Stand up some boxes
vagrant up puppet
* Add puppet IP from output to Vagrantfile
vagrant up monolith
Vagrant
https://www.vagrantup.com/
Digital Ocean Provider
https://github.com/smdahlen/vagrant-digitalocean
Magento Vagrant Puppet Sandbox
https://github.com/aepod/magento-digitalocean-sandbox
101. Scaling Magento – Mathew Beane - php[world] 2015
Adding Digital Ocean API Token
1) Navigate to API -> Tokens -> Generate
2) Copy token that is generated
3) Add Token to Vagrantfile
102. Scaling Magento – Mathew Beane - php[world] 2015
Windows Git / Rsync / SSH Key
1. Install git: https://git-scm.com/download/win
2. Install Rsync: http://sourceforge.net/projects/backuppc/files/cygwin-rsyncd/3.0.9.0/
3. Double check git is in path: C:Program Files (x86)Gitbin
4. Add c:rsyncdbin to your path
5. Run ssh-keygen
103. Scaling Magento – Mathew Beane - php[world] 2015
Using the Magento Sandbox - Newrelic
Newrelic has provided the following signup:
http://newrelic.com/robofirm/
Free 30 day lite account
• Add your newrelic key to the Vagrantfile
• Puppet will handle the rest, you will see test results in New Relic
104. Scaling Magento – Mathew Beane - php[world] 2015
Sandbox – Stand Up
• All nodes run Centos 6.5 and have firewalls disabled (NOT SECURED)
• Puppet does a lot of the base system configuration
• Some shell scripting triggered from vagrant happens before/after puppet.
Available cluster nodes:
• Puppet: All other boxes require it to stand up.
• Monolith: Fully functional single webserver
• DB: MySQL(Percona), Redis and Shared Filesystem(NFS)
• Web: The public facing haproxy server.
• Render[1-4]: Nginx, PHP-fpm. These can be added/removed and scaled.
• jMeter: Used for testing, simple java enabled box.
105. Scaling Magento – Mathew Beane - php[world] 2015
Sandbox – vagrant up (puppet)
1. vagrant up puppet
Some time later…. After a wall of text.
106. Scaling Magento – Mathew Beane - php[world] 2015
Sandbox – vagrant up (monolith)
1. Add puppet IP from output to Vagrantfile
2. vagrant up monolith
Some time later…. After another wall of text.
107. Scaling Magento – Mathew Beane - php[world] 2015
Config - Switching Hosts (monolith)
To use the tests locally you need to “poison your DNS”
On Mac: /private/etc/hosts
http://www.tekrevue.com/tip/edit-hosts-file-mac-os-x/
On Windows: c:windowsSystem32driversetchosts
http://www.thewindowsclub.com/hosts-file-in-windows
Add the following for monolith:
192.168.200.21 web www.idealphp.com
108. Scaling Magento – Mathew Beane - php[world] 2015
Sandbox – vagrant up (jmeter)
1. vagrant up jmeter
A short time later…. After yet another wall of text.
109. Scaling Magento – Mathew Beane - php[world] 2015
Test -Monolith
1. Login on JMeter box
2. Add Monolith IP to the hosts file on jmeter
3. Switch to the jmeter directory:
cd /usr/local/apache-jmeter-2.13/bin/
./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com
DON’T FORGET TO NOTE THE RESULTS
110. Scaling Magento – Mathew Beane - php[world] 2015
Sandbox – vagrant up (cluster)
1. vagrant up puppet
2. Add puppet IP from output to Vagrantfile
3. Vagrant up db
4. Vagrant up web
5. Vagrant up render1 *
* render2,render3,render4 can also be brought up at anytime.
111. Scaling Magento – Mathew Beane - php[world] 2015
Config – Switching Hosts (cluster/web)
Remove or change the entry for monolith
On Mac: /private/etc/hosts
On Windows: c:windowsSystem32driversetchosts
Change the hosts on the JMeter box, in /etc/hosts
Modify the hosts file:
192.168.200.12 web www.idealphp.com
116. Scaling Magento – Mathew Beane - php[world] 2015
Config – Add a Web Node
Vagrant up render3
Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats
117. Scaling Magento – Mathew Beane - php[world] 2015
Test – Cluster (3 web)
Repeat Tests:
./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com
Note Results
You can start to experiment with more threads if you wish.
118. Scaling Magento – Mathew Beane - php[world] 2015
Config – Add Another Web Node
Vagrant up render4
Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats
120. Scaling Magento – Mathew Beane - php[world] 2015
Results – What does it all mean?
We are testing on small cloud servers, the results will not be good.
Doing some compare between the setups you should see throughput going up as you add
boxes.
In production you will start to find bottlenecks easily using this method
121. Scaling Magento – Mathew Beane - php[world] 2015
Test – Cluster (Max Orders)
./jmeter -n -t ../tests/benchmark.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10
-Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password"
-Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0
-Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100
This test will slam orders through as fast as it can. Using it is as simple as recording the
number of orders at the start and finish of a test, and then calculating the number of orders per
hour on that.
Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders)
Try against less web nodes and compare results
122. Scaling Magento – Mathew Beane - php[world] 2015
Config – PHP FPM Tuning
Adjust /etc/php-fpm.d/www.conf change max_children from 8 to 16
Edit on puppet /etc/puppet/modules/daemons/manifests/nginx/php.pp
Line 42: Change to 16
Run on web: /root/deploy_application.sh
Retest with original 4 server test:
./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com -Jramp_period=30
124. Thank You!
Mathew Beane <mbeane@robofirm.com>
Twitter: @aepod
Blog: http://aepod.com/
Rate this talk:
http://joind.in/talk/view/15501
(Slides will be available)
Thanks to :
My Family
The Magento Community
Robofirm
Fabrizo Branca (deployments)
Thjis Feryn (sentinel)
Ben Rose (puppet)
Digital Ocean
New Relic
Zend and the PHP Community
125. Scaling Magento – Mathew Beane - php[world] 2015
Logo graphics:
http://payload.cargocollective.com/1/3/107799/1402304/Puppet%20Hand.jpeg
http://images.clipartpanda.com/sandbox-clipart-black-and-white-
lemmling_sandbox_black_white_line_art_scalable_vector_graphics_svg_clip_art.svg
Digital Ocean and Magento Logo
Editor's Notes
Robofirm is a Magento Solutions provider and a Magento Partner focused on Mid-level to Enterprise clients. Based out of New York City, however the bulk of our developers are in Dallas or Minneapolis.
Zteam – Magento Zray and other Zend Server stuff
Look at the notes in readme or on gist for more details on standup
Test everything, just because we try it doesn’t mean it always works. Sysctl.conf settings are modified based on CPU and memory and server architecture.
Tuning this sort of thing is a presentation in itself, but I will try and cover some of the territory and explain a few of the common tweaks we make.
Rmem: size of receiving socket buffersWmem: size of sending socket buffers
netdev_max_backlog: processor input queue size
Somaxconn: size limit on listen queue TCPtcp_max_syn_backlog: max # of remembered backlogged tcp requests
Optimizing fastcgi correctly, Adding buffers will help a lot with the odd socket errors.Nginx is easier to use in a clustered environment
The configurations are cleaner
Built in Load Balancer and Reverse Proxy, very extensible
Apache Performance just as well in almost all cases
Optimizing fastcgi correctly, Adding buffers will help a lot with the odd socket errors.
Tune Client Caching using expires
When tuning Nginx cache use curl to fetch headers:
Tune Client Caching using expires
When tuning Nginx cache use curl to fetch headers:
All of these settings are very resource dependent
You cannot extend the following classes: Mage_Core_Block_Template, Mage_Core_Model_Abstract, Varien_Object, Mage_Core_Block_Abstract, Mage_Customer_Model_Address_Abstract
Diff your core against known sources to be able to get the junk out. AOE extensions are some of my favorite.
- Scheduler
- Core Store Creation was actually an AOE module
“And I never test.”
CCC transplanting tree
Needs to highlight using metrics to prove performance as you go. Applications will vary, so you need to methodically use metrics to judge performance.
Fassel = NY Giants coach
We will go into a lot more detail on this later in the presentation.
Basis for almost all the performance tests we are doing.
Examples of these are rare and typically proprietary because the time to develop them and the uniqueness of the specific Application
Larger clients are willing to pay for the time to develop them, or may even have internal resources.
Easy solutions for partial test coverage can easily be achieved.
Use Newrelic to supplement your JMeter tests.
This test was on 2 web servers, and a single DB instance.
This assumes you already have working jmeter or other load testing ready.
Gluster – Distributed filesystems don’t perform well
Breaking Points, where you need to adjust first.
php-fpm on it's own nodes, proxied by nginx.
Varnish / Turpentine
NFS becomes bottleneck
Redis Clustering pairing twemproxy (nutcracker) with redis is a no-brainer as it multiplexes the redis queries, even if you just put it on the same node as redis is running. and
Database
Rinse and Repeat
Partial list, full list hasn’t been published yet. Expect a blog article about this “soon”
The demo later focuses on this type of load balancing
The demo later focuses on this type of load balancing
Scalearc and write-ahead-logging as a solution for single writes
Only for EE magento, but I have added some code and have it available on my github to run in CE
Pro-Tip, use the jmeter gui to edit and test your changes to the xml
I have typically been just leaving Orders as 0
Giant Trivial Pursuit @ St Vincent College
Look at the notes in readme or on gist for more details on standup