Magento 2 has been released, so we are trying to dig more useful materials related to the new version of the popular ecommerce platform. This time, our attention has been drawn to Alan Kent’s article related to Magento 2 testing, so we’ve also decided to cover this topic on the Firebear blog. For those developers willing to improve the quality of their Magento 2 projects, testing is mandatory; therefore, we propose you to examine the following Magento 2 guide, which is based on the aforementioned article, the official documentation, as well as our own experience related to both the Magento 2 platform and testing.

UPD (11.02.16): Magento 2 code katas and Getting your hands dirty testing Magento 2

According to Alan Kent, one of the key Magento 2 goals is better test automation, since it can dramatically improve the code quality by finding bugs at early stages of development which leads to more frequent releases. Thus, we should start with discussing different types of tests available in Magento 2:

• Unit Tests (dev/tests/unit). With the aid of unit tests developers check isolated individual classes or functions in order to find logic bugs. Such tests are often run along with integration and functional tests. Highly focused unit tests make it easy to identify the root cause of a problem, but at the same time turn writing highly coupled code into a headache.
• Integration Tests (dev/tests/integration). As for integration tests, they are helpful when you try to prevent problems in interactions between components, layers, and the environment. This group of tests is designed to examine multiple real classes in combination. Use them in addition to unit tests to improve the quality of your MAgento 2 testing.
• JavaScript Tests (dev/tests/js). You probably use tons of JavaScript in your Magento 2 projects, so JS tests are equally as valid as others.
• Performance Tests (dev/tests/performance). Tests for tracking and maintaining certain performance level per build. Don’t confuse such tests with full Load and Performance tests.
• Static Code Analysis Tests (dev/tests/static). Such tests analyze code instead of running it. Their major goal is to check whether the code conforms to coding standards or not.
• Static integrity tests (dev/tests/static/testsuite/Magento/Test/Integrity). They examine how the application is linked, as you might have guessed, without running the code.
• Integrity Tests (dev/tests/integration/testsuite/Magento/Test/Integrity). As for integrity tests, they also check how the app is linked, but deal with more advanced scenarios, so the application have to be run.
• Legacy Tests. These tests are extremely helpful for porting modules to Magento 2, since they examine the usage of methods/classes/constants and show their substitutes.
• Functional Tests (dev/tests/functional). With the help of functional tests, developers find out if product features work correctly, since such tests imitate user’s interaction with the system.

@benmarks explaining the necessity of testing in #magento2 during #MM15GR pic.twitter.com/HCl6wECzeP

— Yannis Develekos (@YannisDevelekos) 9 октября 2015

You can check all these Magento 2 tests running publically here. Before introducing you to the Magento Testing Framework, we’d like to draw your attention to some popular testing solutions:

• JMetrix  brings an aspect oriented approach into the measurement of application performance. With this solution, developers can easily add specific measurement features into the existing projects.
• Selenium automates web applications for testing purposes. It supports major browsers, browser automation tools, APIs, and frameworks. Selenium is a perfect solution for creating robust regression automation suites and tests.
• PHPUnit is a testing framework for PHP and at the same time an xUnit instance for unit testing frameworks.
• Various JS Unit tests are also important for every Magento 2 project, since they help to improve the quality of your JavaScript.

Magento 2 Testing Framework

The Magento Testing Framework (MTF) provides the ability to create and run functional tests, simplifying acceptance, smoke, regression, or other testing. At the same time you should remember that the framework itself does not contain tests; all functional tests are available in the Magento code.

Besides being open source, MTF is a cross-platform solution that enables the development and usage of functional tests. The platform is capable for running multiple tests simultaneously, but you can still perform a single test.

Below, you will find a list of official posts that describe the Magento Testing Framework and a way you should use it. Then, there is a section related to core actions necessary for running Magento 2 tests.

• INSTALLATION – this post sheds light on the installation of MTF entities. Start your journey into the world of Magento 2 testing here.
• QUICK START section leads you to the following posts:

1. First of all, you should adjust MTF configuration. Check the tutorial before going any further.

2. When your MTF is configured, it’s time to prepare environment for running tests.

3. And only then you can perform the test run.

4. Chances are, you will get failed tests, but you can easily see their logs.

• ENTITIES is another essential section of the official MTF guide. It includes 10 posts about various useful details related to Magento 2 testing.

1. A list of properties of the Magento entity under test is called Fixture. The appropriate post describes how to create, use, and update fixtures.

2. The Fixture Repository tutorial provides a repository overview, describes how to create a repo for the entire fixture, shows how to configure the repository, and so on.

3. Handler includes overview, configuration, and usage.

4. The same is about Blocks, they are also described in details.

5. The Typified element post sheds light on required classes for a typified element, and the usage of typified elements.

6. Page discusses page objects. Acting as containers for blocks, they interact with the Magento 2 page under test.

7. Constraint covers the following topics: the overview, structure, creation, and usage of constraints.

8. Data set – the overview and structure of data set utilized by test case and constraints.

9. Test case teaches how to create a new test case from scratch.

10. How to create a Scenario test step by step.

• The UPDATE section teaches how to update MTF by installing its new version or updating components from dependencies
• CHANGELOG provides the ability to check current state of the MTF and illustrates the history of its changes.

Adjust configuration

1. First of all, you should configure PHPUnit by editing phpunit.xml:

• Go to  magento2/dev/tests/functional, find your phpunit.xml file, and open it in editor. Alternatively, you can create it from phpunit.xml.dist (use this approach only if you cannot find the file).
• Find the <php> element to set URL for the Magento storefront you are going to test. Value for name=”app_frontend_url” is your actual URL.
• The same is about backend: you should set value that is your admin URL for name=”app_backend_url”. For instance

If you are testing your Magento 2 store on Windows, you should set a full path for basedir and credentials_file_path.  For example

and

2. Then, configure Magento Testing Framework by editing config.xml:

Please note that for further changes the default configuration set in magento2/dev/tests/functional/etc/config.xml.dist should be copied as config.xml.

• Go to magento2/dev/tests/functional/etc, find config.xml, and open it in editor.
• Set credentials in your <application> element: change content of <backendLogin> and <backendPassword> to your actual credentials.
• Choose browser. Mozilla Firefox is the default solution, but you can easily specify another browser in config.xml. You have to copy your <server> element from magento2/dev/tests/functional/vendor/magento/mtf/etc/config.xml> and paste it in magento2/dev/tests/functional/config.xml after the <install> element. Now, set actual data in the following attributes: <browserName> and <browser>. The Google Chrome example looks as follows:

3. The following step requires editing the credentials.xml file, since some Magento modules require credentials:

• Go to magento2/dev/tests/functional, find credentials.xml, and open it in editor. You can create credentials.xml from credentials.xml.dist if you cannot find it.

To set credentials for a Magento 2 module you are going to test, in credentials.xml find the block of elements related to this module and enter actual data in the value attribute of the corresponding field element.

Preparations for test run

The Selenium Server

The Selenium Server utilizes your browser to execute tests. You can download it here. Please note that:

• the tool requires installed Java;
• specific Selenium Server versions are compatible with specific browser versions.

To run the Selenium Server, enter the following command in terminal:

Non default browser

To run tests on a non default browser (chances are the Selenium Server will not work directly with it) utilize the corresponding WebDriver.

Set a browser that MTF will use for testing Magento 2 in the config.xml file. Check the Google Chrome example above. Now, you can run the Selenium Server with an additional argument. Again, we provide an example for Google Chrome:

Generator

This tool generates fixtures, repositories, as well as page objects. Thus, After initializing MTF, all classes will be pre-generated. As a result, you facilitate creating and running tests for your Magento 2 project. To run the generator, enter the following command in terminal:

Test run

You can find all tests classified by categories in magento2/dev/tests/functional/tests/app/Magento. As for the testing tool, the framework (MTF) utilizes PHPUnit from magento2/dev/tests/functional/vendor/bin.

Please note that before running any tests, you should be confident that

• Magento 2 is ready for running different tests;
• MTF is installed: MTF installation guide;
• The framework is configured;
• Environment is ready for running a test.

How to run all tests in Magento 2

To run all tests from magento2/dev/tests/functional/tests/app/Magento/, enter the below command in terminal:

How to run a particular test in Magento 2

To run a particular test in Magento 2, enter the following command in terminal

where <name of test> is the name of PHP file with the test.

How to get logs for failed tests

All failed tests are logged in magento2/dev/tests/functional/var/log, while all Magento 2 errors – in magento2/var/log.

___

We expect that extensions in the new version of Magento Connect will have a complete test coverage. Check our blog posts related to Magento testing and TDD for further information.

Our team also recommends you to check the Entities of the Magento Testing Framework post on the official devdoc. The guide provides a complete data set overview and structure as well as describes how to merge data set. Check the full tutorial here: Entities of the Magento Testing Framework.

A simple and productive article about Custom Module Unit Tests in Magento 2 by Creare is also among our favourite materials related to Magento 2 testing. Don’t waste your chance to get a deep insight into this complex procedure without any headaches. For further information, follow this link: Beginners Guide to Magento 2 Custom Module Unit Tests.

Magento 2 code katas

Vinai Kopp creates a lot of amazing Screencasts of TDD practice sessions on the appropriate YouTube channel here: Mage2Katas. You get a great opportunity for learning TDD in Magento 2 via code katas – basic coding patterns practiced until you can use them automatically. You can read more about the term here: What is a Code Kata?

Getting your hands dirty testing Magento 2

Magento 2 Demo