How to import 100,000 products to Magento 2 in less than an hour
Magento 2 can easily handle more than 1 million products. However, the more products you need to import, the more time it will take to update your catalog. The amount of server resources also impacts the import performance. And it may take hours to update your catalog which consists of less than 100k items. However, there is a solution that lets you import 100,000 products to Magento 2 in less than an hour. Meet the Improved Import & Export extension. Below, we talk about the Magento 2 import speed that can be achieved with this module. But let’s take a look at average productivity issues associated with the default Magento 2 import.
Magento 2 import gets stuck after 400 SKUs
shares his Magento 2 . His goal was to import a price list that contained 19000 SKUs. The task was simplified since the import file was split into several files with a maximum of 5k SKUs in each. However, only 400 SKUs had been updated after every import process.
The default Magento 2 data transfer got stuck in an endless loop: No error had been displayed and the status was in progress.
Does it mean that Magento’s import capacity is only 400 SKUs?
Of course, not. As Narendra explained, they had Cloudfare CDN that timed out after 1.5 minutes. This time was enough to import only 400 records. Bypassing Cloudflare fixed the issue. But let’s calculate the import speed for this case.
If it takes 1,5 minutes to import 400 products, then for 100,000 products the import process should take 375 minutes. What if you can do that much faster in less than 60 minutes?
How to import 100,000 products to Magento 2 in less than an hour
As we’ve just mentioned above, you can dramatically increase your import speed with the help of the Improved Import & Export Magento 2 extension. Below, we compare the module’s performance with the speed of the default import solution described in Narendra’s case:
| Tool | Behavior | Entity count | File size | Minutes | Ent/Min |
| Improved Import | Product import | 100,000 | 127 Mb | 40 | 2500 |
| Standard Magento 2 import | Price list update | 100,000 | N/A | 375 | 267 |
Thus, the Improved Import & Export Magento 2 extension lets you import 100,000 products in less than an hour, transferring 2500 entities per minute, while the default import solution may increase time expenditures by 10 times – to 375 minutes, as Narendra’s case show.
However, everything also depends on two more factors: how well your store is configured and what server resources are available.
In our case, the server specs that allow for importing 100,000 products in less than 60 minutes include:
- Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz, 4-core CPU;
- 64 Gb RAM;
- Crucial MX550 SSD x 2;
- MariaDB 10.1.26 (MySQL 5.7);
- PHP 7.2.6.
Unfortunately, we don’t know about server specs in Narendra’s case. However, we may assume that even the same server specs won’t be able to increase the import speed from 267 entities per minute to 2500.
Other import advantages
The Improved Import & Export extension for Magento 2 offers lots of other advantages in comparison to the default import process. Let’s see its features that can help you improve Narendra’s case.
First of all, the module can help avoid the necessity to split the import file into several parts since it can handle huge import tables that contain hundreds of thousands of records.
In addition to that, the entire import process can be automated. The Improved Import & Export Magento 2 extension lets you create update schedules. Then, the module follows them, importing update files automatically.
And if an update file contains unsupported or missing attributes, the Improved Import & Export plugin can replace them with the information your Magento 2 website requires. The extension’s mapping interface contains numerous features that let you modify third-party data or create missing attributes on the fly.
In addition to that you can import files of various formats (CSV, XML, JSON, ODS, etc.) from different sources (FTP/SFTP, Google Drive, Dropbox, etc.). If file transfers are not the preferred way to import data, the Improved Import & Export offers Google Sheets support and API transfers.
Follow these links for more information about the extension:
Get Improved Import & Export Magento 2 Extension
Improved Import & Export Manual
And don’t forget about our Magento 2 Developer’s Cookbook!