How to import data from external HBase database to Magento 2

The following article sheds light on how to import data from an external HBase database into Magento 2. Below, we discuss and compare two different ways of doing that. First of all, you will find out how to apply a standard algorithm of moving data from a database to Magento 2. Next, we will provide you with a reliable tool that dramatically improves the community-proven approach. Let’s start the exploration.

The Common Way of Moving HBase Bases into Magento 2

The standard way of moving data from HBase to Magento 2 consists of three steps. The procedure starts with creating a CSV output from your HBase database. Why CSV you will ask? Because Magento supports only this format. Of course, you can create a document in another file format, but it would be necessary to convert it into CSV. Do you need this additional step?

The second stage of moving data from HBase to Magento 2 requires reviewing the document created in the previous step. Since different platforms use different data standards, you need to edit the CSV file making it suitable for Magento 2 requirements. When the document is ready, you can import it into Magento 2, which is the third stage of the standard way of moving HBase bases into Magento 2. Let’s take a look at each step individually.

CSV Export from HBase Database

The basic syntax of the HBase Export utility looks as follows:

Note that you should call the command the (Linux) shell, not the HBase shell. In the directory identified as the targeted export location, a part-m-00000 file represents the results of the map job created by the utility. Besides, you can see an empty file that indicates the success or failure of the map job.

By using this simple syntax, you export the latest version of each row in the table. For instance, you can use the following code snippet:

The procedure looks quite complicated, isn’t it? And it doesn’t produce a CSV file directly, so it’s just the beginning of your headaches. Now, it is necessary to transform the output into a CSV file.

For further information, check this post: Working with the HBase Import and Export Utility. Next, follow this link: Export all data from HBase database to CSV.

Map Third-Party Attributes to Magento 2 Attributes

Now, when you’ve managed the HBase CSV export, you can edit the file. The more attributes the table contain, the more work should be done. And if some attributes are fixed incorrect, you won’t move them with the corresponding data into Magento 2.

And the worst thing about manual editing is the inability to save the matching scheme for further updates: you should start everything from the very beginning on each new data import.

Import HBase Table into Magento 2

Finally, you can import a remote HBase table into Magento 2. Note that the process cannot be automated. Furthermore, you have to repeat all the steps described above on every update. Read this article for further information: The Complete Guide to Magento 2 Product Import / Export. Here, you will discover what attributes to use and how to adopt an external output to the needs of Magento 2 making the import procedure more efficient.

Pros & Cons

Now, let’s look at the pros and cons of the standard way of moving HBase databases into Magento 2

Pros

1. You can take the external output and transfer it into your Magento 2 website.

Cons

1. CSV export is complicated and requires coding skills.
2. You should map attributes manually.
3. It is necessary to repeat the whole procedure every time an update is required.

What do we offer instead?

The Easy Way of Moving HBase Bases into Magento 2

As you might have already guessed, it is necessary to install a third-party tool to simplify the Magento 2 HBase integration. Meet the Improved Import & Export extension – your number one time-saver when it comes to data migrations between your e-commerce website and any third-party systems. Below, we explore critical differences between the easy way of moving HBase bases into Magento 2 and the common approach we’ve just described.

HBase Database Connection

While the community-proven approach requires creating a CSV output, the Improved Import & Export extension allows eliminating this step entirely. Our module connects to HBase directly allowing you to transfer data from the database into your e-commerce store without any intermediaries. As a result, you no longer need to work with commands or look for unsuitable attributes in a data file. Furthermore, all processes take place in a single interface of the Improved Import & Export extension!

Magento 2 HBase Data Import

Although the Improved Import & Export extension allows importing data from a remote Hive base to Magento 2 directly, you still need to solve the problem of different attribute standards. The module lets you either apply a mapping presets or match the external designations to the corresponding internal ones in your Magento admin! We will describe both opportunities lately.

Another vital improvement provided by the Improved Import & Export extension is the ability to automate the data import process. You can either create a schedule of updates or choose events that will trigger the procedure. These features are described in the corresponding section of the article as well.

How to Import Data From HBase to Magento 2

Let’s take a look at how to import data from a remote HBase table to Magento 2 with the help of the Improved Import & Export Magento 2 extension.

1. Go to Magento 2 backend -> extension admin -> import section and create a new import profile there.
2. Configure the job: add a schedule (or choose a trigger), select an imported entity, match attributes, etc.
3. Now, you can save the job and launch it afterward.

Our extension will transfer data from HBase to Magento 2 performing all updates automatically. But what are the pros and cons of this approach?

Pros & Cons

Do you remember that the only positive aspect of the traditional approach is the ability to move data from HBase to Magento 2? At the same time, there are lots of drawbacks. Let’s see how the Improved Import & Export Magento 2 extension changes the situation.

Pros

1. You don’t need to export data files.
2. No data files should be edited.
3. The whole import process is automated.
4. A direct connection between Magento 2 and MariaDB is established.

Cons

1. Installing a third-party extension is necessary.

If you have any difficulties with the installation process and its further configuration, our specialists will help you. Thus, the Improved Import & Export extension offers an entirely revamped process of transferring data from HBase to Magento 2. Let’s take a look at the core HBase features and then proceed to the more in-depth exploration of our import/export extension.

Get Improved Import & Export Magento 2 Extension

HBase Facts

HBase is often compared with Cassandra, MongoDB, and Hive. Being a wide-column store, it is based on Apache Hadoop and BigTable concepts. Other features of HBase are:

• The database developer is Apache Software Foundation;
• The initial release is dated back to 2008; the current version is 1.4.3, April 2018;
• The license model is Open Source;
• No cloud-based options are offered;
• The implementation language of the database is Java;
• Its server operating systems are Linux, Unix, and Windows;
• HBase is schema-free and provides no support for typing, XML, secondary indexes, and SQL;
• APIs and other access methods include Java API, RESTful HTTP API, and Thrift;
• Supported programming languages are C, C#, C++, Groovy, Java, PHP, Python, and Scala;
• Server-side scripts and triggers are supported;
• Partitioning methods are represented by sharding;
• Replication methods – by selectable replication;
• MapReduce is available;
• Immediate Consistency describes consistency concepts;
• No foreign keys and transaction concepts are available;
• But HBase offers concurrency and durability features;
• In-memory capabilities are not available;
• User concepts are represented by Access Control Lists (ACL).

Improved Import & Export Magento 2 Extension

Now, it is necessary to shed light on the core features of the Improved Import & Export extension. Below, you will discover what exact tools it offers to simplify the Magento 2 HBase integration. Let’s begin with the automation.

Automated Import & Export Processes

The Improved Import & Export Magento 2 extension offers two ways to automate data migrations. Firstly, you can create a schedule of updates. Secondly, it lets you specify events that can be utilized as triggers. Let’s explore each approach separately.

Cron Schedule

Since our module supports crone, you can fully leverage this technology while automating your data transfers. Use the default cron syntax to create any custom schedule of updates. Alternatively, you can select one of the predefined intervals. Note that these intervals can be tailored according to your business needs as well.

And you can freely create import and export jobs that have no schedule. They are useful for asynchronous updates or one-time migrations when a schedule cannot be established. Also, note that you always have an opportunity to launch any scheduled process whenever you want. Let’s assume that your partner has provided you with a new vital update but the corresponding job is initiated for tomorrow. You don’t need to wait – start the necessary process within a few clicks.

Below, we show how to configure various schedule types:

Events

In addition to schedules, you can also leverage system or custom events to launch import/export processes. Just specify a trigger and create specific conditions to make the launch more precise. For further information about event-based automation, read this article:  How to Run Magento 2 Import or Export After Specific System Event or Process.

Now, when you know how to automate data import from HBase to Magento 2, we can proceed to the next block of the article that describes various mapping features.

Advanced Mapping Features

The Improved Import & Export extension offers lots of tools designed to solve problems caused by different attributes and their values.

Mapping Presets

The easiest way of improving your daily routine related to attributes matching is represented by mapping presets. Our team has created multiple mapping schemes for you so that it is only necessary to select and apply one that satisfies your integration.

The Improved Import & Export Magento 2 extension will analyze the input file, discover third-party attributes, and match them to ones used internally. You only need to click a few buttons.

Below, you can see how to establish the connection between Magento 1 and 2.x solving the problem of different attributes:

You must admit that it is the easiest way of attributes matching, but it is always possible to do the same procedure manually within a convenient interface.

Matching Interface

Is certain presets are not available, you should match attributes manually, but it is not a reason for panic. The Improved Import & Export Magento 2 extension offers a user-friendly mapping interface where you can effortlessly select all third-party attributes and specify the corresponding designations used in Magento 2 in front of them just as shown below:

And don’t forget about the opportunity to set hardcoded values – attribute values that will be added automatically to all items in the table.

Attribute Values Mapping

In addition to attributes matching, the Improved Import & Export Magento 2 extension provides the ability to map their values. The process is based on the same principle as the one above. You need to select an external attribute, specify its values, and link them to ones used internally. It is possible to apply this tool when you import data from HBase to Magento 2.

For instance, there is an external attribute “color” in a remote HBase table. It has two values: “lime-green” and “avocado-green”. In your Magento 2 database, there is only one corresponding value – “green”. Match both external designations to the one used internally. You can find more information regarding attribute values mapping here: Attribute Values Mapping.

Attribute Values Editing

If the aforementioned procedure is not enough to satisfy your business needs, the Improved Import & Export Magento 2 extension also lets you edit multiple attribute values in bulk. Our module not only supports numerous commands but also allows combining them to increase the efficiency. And to apply changes more precisely, the plugin provides the ability to create specific conditions.

The following commands are at your disposal:

• Add prefix – add a prefix to multiple attribute values in bulk;
• Add suffix – add a suffix to multiple attribute values in bulk;
• Split data – split data related to different values;
• Merge data – merge data related to the same value.

For further information, follow this link:  How to Modify Attribute Values During Import and Export in Magento 2.

Category Mapping

Additionally to attributes and their values, the Improved Import & Export Magento 2 extension allows mapping product categories. If you import something from an external catalog that differs from one used in your database, it is no longer a problem due to robust mapping opportunities. Select a third-party category and match it to the one used internally as shown below:

Also, note that you can even create new categories during the import process. Specify a parent category and a name of a new one. Both features are described here:  Category Mapping.

Attributes On The Fly

The last feature that simplifies your routine related to the data import is the ability to create attributes on the fly. If an imported table lacks any data, you can easily add parameters necessary to generate it, and the Improved Import & Export Magento 2 extension will add missing attributes automatically. Use the following general format to achieve this goal:

Attribute|attribute_property_name:attribute_property_value|…

You can discover the full algorithm here: Product attributes import.

Extended Connectivity Options

Another important aspect of the Improved Import & Export Magento 2 extension is various connectivity options. You can either use data files, online, sheets or connect to external systems directly to transfer data.

Multiple File Standards

Our module supports multiple file standards. While Magento 2 works with CSV only, the Improved Import & Export Magento 2 extension adds  XML, JSON, ODS, and Excel. You can freely import and export data using these file types. Thus, there is no need to convert everything to CSV and vice versa like in case of standard Magento tools.

Multiple File Sources

Of course, you need a point of connection where two systems can exchange data files. The Improved Import & Export Magento 2 extension offers three different options.

First of all, you can fully leverage FTP/SFTP connections. It means that your local server can become the point of connection. At the same time, a remote server can be utilized for the same purpose.

Another option is represented with Dropbox support. Create a new account or use an existing one to transfer data files between Magento 2 and an external system through the cloud.

And if you need to establish a one-way connection, the Improved Import & Export module offers the ability to import data via a direct URL. Specify a file address, and the extension will import files automatically.

It is also necessary to mention that compressed data is no longer a problem since our plugin can unzip/untar it automatically.  

Alternative Ways of Import & Export

In addition to file transfers, the Improved Import & Export Magento 2 extension provides several alternative ways of establishing a connection. The module leverages two different approaches. First of all, you can use some intermediaries, such as Google Sheets, Office 365 Excel, or Zoho Sheet. The Google Sheets Magento 2 integration looks as follows:

The second way is represented by the support for various API connections: REST, SOAP, and GraphQL. Note that you can freely apply all the features described above to any connection type or entity.

Watch the video below to see the extension in action:

Final Words

As you can see, the Improved Import & Export module dramatically improves the traditional approach to the data import from HBase to Magento 2. With our extension, you can fully automate this process and avoid multiple headaches caused by different stages of the common import. For further information, follow the link below and contact our support:

Get Improved Import & Export Magento 2 Extension