How to import data from Couchbase to Magento 2

The following article discusses two different approaches to importing data from Couchbase to Magento 2. First of all, we shed light on a common way of achieving this goal which is based on a universal algorithm suitable for all remote databases. Next, a more user-friendly and efficient approach is described. Let’s start our journey to the world of Magento 2 Couchbase data import.

The Common Way of Moving Couchbase Tables into Magento 2

Every common way of importing data from a remote database to Magento 2 is based on three pillars: CSV export, file editing, and data import. Thus, you have to do some work in three different environments. Depending on various conditions, this routine can become very time-consuming, but let’s take a look at each step of moving Couchbase tables into Magento 2 to illustrate what we mean.

CSV Export from Couchbase Database

To export a CSV file from a remote Couchbase database, you should use the cbtransfer tool. It transfers data to Couchbase Server as documents. Next, they are exported from the server into comma-separated values. Note that design documents associated with vBuckets are not included.

Let’s explore an example with a document ID re-fdeea652a89ec3e9, 0 flags, 0 expiration, and the CAS value 4271152681275955. The actual value is the hash starting with “{“”key””. Check the following snippet:

Now, we have to export these items to a CSV file. Let’s transfer items from the default bucket, -b default available at the node http://host:8091 and put them into the /data.csv file. Note that in case of a different bucket for the -b option, all items are exported from that bucket. If you are going to export items from a bucket in the cluster, provide credentials. Use the following command to launch the export process:

As a result, we get the following example response:

What we have here is 1053 batches of data transferred at 550.8 batches per second. You can also see the “cannot save bucket design….” warning that warns you about the inability to export design documents. For further information, read this article: Couchbase Server cbtransfer.

Map Third-Party Attributes to Magento 2 Attributes

Now, it is necessary to edit the created CSV file to make it suitable for the further import process. First of all, review all attributes available in the document. You should make them suitable for Magento 2 requirements. Otherwise, no data will be imported. The process consumes the more time; the more attributes are available in the document.

Besides, you can edit attribute values. There are no mandatory requirements that influence the import process, but some designations may be not available in your database or have a slightly different appearance. To make your database well-ordered and organized, it is recommended to edit attribute values as well.

As you might have already guessed, this routine requires even more time, since one attribute usually has multiple values. And it is necessary to do everything manually. Check the following guide to discover native attributes of Magento 2 and other nuances of the successful import: The Complete Guide to Magento 2 Product Import / Export. Note that all requirements regarding attribute values are based on your particular business case.

Import Couchbase Table into Magento 2

Now, when your CSV file is edited correctly, you can start the import process. The link mentioned above leads to an article that fully describes it. Sot, to make the long story short, we can only add that the default Magento 2 tools are enough to complete the process. Note that you won’t automate data updates with their help, but it is not the goal of the common approach to importing data from Couchbase to Magento 2.

Pros & Cons

What are the pros and cons of the standard way of moving data from one system to another? Let’s see.

Pros

The only positive aspect of the default approach (which is universal as well) is the ability to move data from Couchbase to Magento 2. Unfortunately, it is not the most efficient one, but what did you want from the algorithm that can be applied to so many different systems? Nobody promised you that it would be a funny and entertaining walk. And there are many negative aspects.

Cons

• You have to work in three different environments.
• Three different sets of tasks are involved.
• It is impossible to automate any of them.
• You should work with CLI commands.
• It is necessary to edit data files manually modifying both attributes and their values.
• The process is time-consuming and cannot be automated.

It seems that now you have enough reasons to look for a better way of importing data from Couchbase into Magento 2. Let’s discussed the promised one.  

The Easy Way of Moving Couchbase Bases into Magento 2

It’s evident that the better way of moving data from the database to Magento 2 is based on a third-party extension. Therefore, meet the Improved Import & Export module – your number one time-saver when it comes to data transfers and integrations with external systems. Our plugin completely revamps the process described above and lets you automate all its steps.

Couchbase Database Connection

Instead of file transfers, the Improved Import & Export Magento 2 extension establishes a direct connection between your e-commerce store and a remote database and provides the ability to replace the work in multiple environments with a single-interface configuration. You need to create a new import profile (we describe the process below), configure it, and launch the transfer.

Magento 2 Couchbase Data Import

Unfortunately, the direct connection doesn’t solve the problem of different attribute standards, but the Improved Import & Export Magento 2 extension offers two ways to fix it in a user-friendly manner. You can either apply presets or match external attributes to ones used internally in an intuitive interface which also lets you map attribute values. All these features are described further in this article.

Besides, our module provides the ability to automate data transfers between Couchbase and Magento 2 in several ways. Firstly, you can create a schedule of updates. Secondly, a trigger can be applied. And don’t forget that the established mapping scheme is saved and used automatically.

As a result, there is no need to manage further data updates – the Improved Import & Export Magento 2 extension will do everything for you.

How to Import Data From Couchbase to Magento 2

Now, let’s take a look at how to import data from Couchbase to Magento 2 with the help of the Improved Import & Export extension. The following steps are involved:

1. In the Magento admin, go to the Improved Import & Export extension, open its import section, and create a new job (profile).
2. Specify parameters necessary to connect to your database, set what attributes should be mapped, create an update schedule, and configure other options if necessary;
3. Save and launch the profile.

That’s it! These three tiny steps are the replacement for the common way of moving data from a remote database into Magento 2. You’ve just saved tons of time and effort connected your e-commerce store to Couchbase. Moreover, the process is fully automated so that you prevent further time expenditures! You must admit that this approach is much easier than the default one.

Pros & Cons

Let’s see what pros and cons the Improved Import & Export offers then it comes to the Magento 2 Couchbase integration.

Pros

1. Couchbase to Magento 2 data transfers;
2. The direct connection between the two systems with no data files, CSV output, manual attribute editing;
3. All updates are automated entirely.

Cons

The only negative aspect is the necessity to install a third-party module. But in case of any difficulties, you can freely contact our support, and the Firebear team will help you. Furthermore, you can propose your specific changes to the module’s functionality, and we will discuss their implementation!

Consequently, the need to install the Improved Import & Export Magento 2 extension is not a drawback – it is a necessity. Especially if you compare the user-friendly algorithm with the universal one, you understand the difference and see the real disadvantages in the second one.

Now, let’s take a look at some features of Couchbase and then proceed to a more in-depth exploration of our extension.

Get Improved Import & Export Magento 2 Extension

Couchbase Facts

Couchbase is often compared with MongoDB, Cassandra, and Redis. It is an open source NoSQL database engineered to support the agile development and scalable deployment of various projects in such areas as enterprise web, mobile, and IoT. You can freely deploy Couchbase as a:

• distributed cache for low latency reads;
• key/value store for high performance reads and writes;
• document database for powerful querying and lightweight analytics.

The database is designed to supports agile development with a flexible data model. Besides, you can leverage a powerful query language, scalable deployment with a single node type.

The powerful querying functionality is represented by an expressive query language that extends SQL and supports joins. The database lets you achieve consistently high performance due to a managed object cache and streaming replication, persistence, and indexing.

As for global deployment, there is an optimized replication protocol to support multiple data centers in active/active configurations. And don’t forget about extensive mobile support represented by an embedded database for iOS, Android, and other mobile environments with automatic synchronization. Other features are:

• JSON-based document store;
• The database is derived from CouchDB;
• The interface is Memcached-compatible;
• Spark integration: Spark, Spark SQL, and Spark Streaming;
• The database exists since 2011;
• Its current release is 5.5.0, July 2018;
• Couchbase operates under an open source license;
• It is available only as a cloud service;
• Implementation languages are C, C++, Go, and Erlang;
• Server operating systems include Linux, OS X, and Windows;
• Data scheme is schema-free
• Typing is fully supported;
• The same is about secondary indexes;
• SQL-like query language (N1QL) is also among the features of the database;
• APIs and other access methods are represented by the Memcached protocol and RESTful HTTP API;
• Supported programming languages are .Net, C, ColdFusion, Erlang, Java, Perl, PHP, Python, Ruby, etc;
• Server-side scripts include view functions in JavaScript;
• Triggers are supported;
• Partitioning methods include Sharding;
• Replication methods are represented by master-master and master-slave replication;
• Couchbase offers an API for user-defined Map/Reduce methods;
• Consistency concepts are represented by Eventual and Immediate Consistency;
• Foreign keys and transaction concepts are represented partly;
• Concurrency and durability features are a part of the database;
• User concepts include User and Administrator separation with password-based and LDAP integrated authentication.

Improved Import & Export Magento 2 Extension

The following section of this article is fully dedicated to the Improved Import & Export extension. Below, we describe all the module’s features that simplify the Magento 2 Couchbase synchronization. Let’s start the exploration from the automation features.

Automated Import & Export Processes

Improved Import & Export offers two ways to automated your daily routine related to data transfers. You can use either schedules or triggers.

Schedules

Since our module supports cron, you can create any custom update schedule using its default syntax. Alternatively, it is possible to select one of the predefined intervals which are fully customizable as well.

At the same time, the Improved Import & Export Magento 2 extension allows creating import and export profiles that have no schedule. You can use them for one-time data transfers or non-periodical updates.

And every scheduled profile can be launched asynchronously whenever you want. If your partner delayed the important update, you don’t need to wait for the next planned event – relaunch import manually.

Below, you can see how to configure the appropriate section of the extension:

Events

Event-based import and export is another opportunity provided by the Improved Import & Export Magento 2 extension. You can specify triggers on the basis of native Magento 2 events or create custom ones to launch data transfers when a certain process takes place. You can find more information about this feature here: How to Run Magento 2 Import or Export After Specific System Event or Process.

Advanced Mapping Features

As mentioned above, the Improved Import & Export extension provides the ability to solve multiple problems related to different attributes and their values. Below, we describe the main features of this area.

Mapping Presets

The easiest way to solve the problem of different attributes is represented by mapping presets. The Improved Import & Export Magento 2 extension lets you match external designations to ones used in Magento 2 in a matter of a few clicks. You only need to select and apply a preset suitable for a particular integration.

Next, the plugin will analyze the input data mapping all external attributes to the corresponding internal ones. Below, you can see how to import data from Magento 1 to Magento 2 using a corresponding preset:

Matching Interface

If there is no preset suitable for your integration or you want to take the full control over the process, use the mapping interface of the Improved Import & Export extension. It is possible to match attributes there manually. Indeed, the procedure is very straightforward. You select an external attribute and specify an internal one in front of it. That’s all. Also, it is possible to add a hardcoded attribute value that will be applied to all items simultaneously. The interface is illustrated below:

Attribute Values Mapping

There is also an opportunity to map attribute values. You no longer have to import unnecessary data and edit it via the Magento 2 backend. The Improved Import & Export extension offers a more efficient way to do so. You can apply the same logic as we’ve just described using the same interface.

Let’s assume that a remote Couchbase table contains an external attribute “color” with two values: “lime-green” and “avocado-green”. In its turn, your Magento 2 database includes only one suitable value: “green”. Match both external designations to the one used internally in the extension interface. There is no need to import the unnecessary data and edit it manually. You can find more information regarding the feature here: Attribute Values Mapping.

Attribute Values Editing

In addition to the previous feature, the Improved Import & Export Magento 2 extension provides the ability to edit multiple attribute values in bulk. You can add a prefix or suffix. At the same time, our module allows merging data related to the same value or doing the opposite thin – splitting data of different values. You can combine rules as well as create conditions to apply them more precisely. The feature is described here in more detail:  How to Modify Attribute Values During Import and Export in Magento 2.

Category Mapping

As for third-party categories, they can be easily mapped to ones used internally. The process is similar to attributes mapping. You select an external category and match it to the corresponding internal one, as shown below:

At the same time, the Improved Import & Export Magento 2 extension provides the ability to create categories on the fly. Specify a parent category and type a name of a new one to do that. Both features are described here: Category Mapping.

Attributes On The Fly

And if a file lacks some vital data preventing the import process, you can create attributes on the fly. The Improved Import & Export Magento 2 extension provides the ability to specify parameters necessary for the automatic attribute generation. Use the following general form to achieve this goal:

Attribute|attribute_property_name:attribute_property_value|…

The full algorithm of the attributes on the fly creation is described here: Product attributes import.

Extended Connectivity Options

Another important aspect of the Improved Import & Export Magento 2 extension is its extended connectivity. Although the Magento 2 Couchbase integration is based on a direct connection, there are multiple other options that can be used to transfer data between the two systems.

Multiple File Standards

Firstly, the module significantly extends the list of supported file formats. While Magento works with CSV only, our plugin allows freely importing and exporting XML, JSON, ODS, and Excel files. You no longer need to convert them to CSV and back due to the standard limitations of the Magento platform.  

Multiple File Sources

Of course, numerous file sources can be turned to the point of data exchange. Due to FTP/SFTP support, it is possible to leverage a local or remote server to transfer data. A Dropbox account can be utilized for the same goal. And when it comes to import processes only, our extension lets you specify a direct file URL. Note that all compressed data is unpacked automatically.

Alternative Ways of Import & Export

And there are multiple ways to import/export data avoiding data files. First of all, you can use intermediaries such as Google Sheets, Office 365 Excel, or Zoho Sheet. The import process from Google Sheets is illustrated below:

The second way is based on APIs. The Improved Import & Export extension allows transferring data via REST, SOAP, and GraphQL. There is only one requirement: an external system should support one of these connection types.

Note that all features and instruments of our module can be applied to data file transfers and connections established directly or with the help of the intermediary services.

Watch this video to see the extension in action:

Final Words

As you can see, there are two ways of importing data from Couchbase to Magento 2: the long one and the short one. Depending on your preferences, you can select any of them, but it is obvious which approach is more user-friendly and less time-consuming. If you still have questions regarding the Improved Import & Export Magento 2 extension and its role in the connection of two systems, feel free to contact our support. Use the following link to do that:

Get Improved Import & Export Magento 2 Extension