The Omni Connector is an universal solution that allows you to ingest your catalog data from a raw file (or multiple files), as long as you have a supported file format and are able to send the data via a supported data transfer method.

Why use it?

Constructor's Omni Connector accelerates the path to value for any customer through a ready made product catalog integration. By sending your raw files, you'll be able to map and transform the data using our powerful connector ecosystem.

Data mapping and transformation

The entire data mapping and transformation processes happen inside our connector ecosystem. You can leverage the power of our Connect CLI to specify how your data is handled across all of your indexes.

With the Omni Connector, you'll be able to map each object of your file (for example a line inside your JSONL file) to any number of Constructor items, variations and item groups.

Supported file formats

📘

Note that we're working to bring support to new file formats to expand the reach of this connector. If you have a specific request, please reach out to [email protected]

Currently, the Omni Connector supports the following file formats:

• JSONL
• NDJSON
• CSV

File compression

While the connector can work with raw files, it's generally recommended that you compress your files using gzip. This will ensure that you are sending a much smaller file so the entire process is more performant on storage and network usage.

When compressing files, please pay attention to the filename with the rules described below.

Supported data transfer methods

📘

Note that we're working to bring support to new data transfer methods to expand the reach of this connector. If you have a specific request, please reach out to [email protected]

Currently, the Omni Connector works with the following data transfer methods:

• SFTP

Note that whenever the connector receives a new file, this will trigger a new ingestion which will cause your catalog to be updated.

Sending files via SFTP

To send a file via SFTP to the Omni Connector, you'll need to connect to following URL at port 22: http://connect-sftp.cnstrc.com.

The authentication method is password, where you'll use your connect auth token (the token used as the credential on the connector ecosystem) to log in. Once authenticated, you'll be able to upload files via put. Note that you can only upload files to the SFTP root directory and the filename dictates how you want to perform the ingestion.

Overall, this is how the process should look like if you're using a raw sftp session:

# establish the sftp connection
> sftp connect-sftp.cnstrc.com

# input your password to authenticate
> constructor_connect_auth_token_xxxxxxxxxx

# upload the file
> put con_xxxxxxxxxxxxxxxx.ndjson

# disconnect
> bye

In production, you'll likely want to automate the process of transferring those files. As an example, take a look at some popular ways to send SFTP files programmatically:

• NodeJS: ssh2-sftp-client
• Python: Paramiko
• Java: JSch

It's important to note that the SFTP filename will define how your ingestion is processed. It consists of a few parameters:

• Connection slug: required, used to identify which connection / index to ingest to.
  • It is formatted as con_xxxxxxxxxxxxxxxx (20-character long string)
• Force: optional, if we should force the ingestion or not
  • It defaults to false
  • If true, the ingestion will not fail even if it updates a large part of the catalog (for example > 50%)
• Ingestion strategy: optional, defines how the feed should be ingested
  • It defaults to full for a full, which will replace your catalog
  • Valid values are: full, delta, patch_delta_fail, patch_delta_create, patch_delta_ignore
• Any other identifier: after this piece of the filename, you can include whatever else to identify your files. For example, a timestamp or a unique id.

On top of that, the filename should follow this structure:

• [name].[extension], or
• [name].[extension].gz if compressed

According to the preceding rules, these would be valid filenames:

• All arguments: con_xxxxxxxxxxxxxxxx_force_patch_delta_ignore.ndjson.gz
• No force flag: con_xxxxxxxxxxxxxxxx_patch_delta_ignore.ndjson.gz
• No strategy: con_xxxxxxxxxxxxxxxx_force.ndjson.gz
• No optional arguments: con_xxxxxxxxxxxxxxxx.ndjson.gz

Feel free to include any information after all arguments. These would be valid filenames:

• With a timestamp: con_xxxxxxxxxxxxxxxx_force_20241212T124000.ndjson.gz
• With a unique id: con_xxxxxxxxxxxxxxxx_my_unique_id.jsonl.gz

And these would be invalid filenames:

• Does not have the file extension: con_xxxxxxxxxxxxxxxx
• Has a wrong extension: con_xxxxxxxxxxxxxxxx.txt
• Does not have the slug: file.ndjson.gz

Best practices

To make the best use of the Omni Connector, you should keep a few best practices in mind. This will ensure that your integration will be easy, performant and scalable for any new requirements in the future.

Ensure your files are properly structured

We rely on the template mapping layer to identify what are items, variations, and item groups inside your file.

While you can group variations together and nest them inside your items, you need to ensure that:

• Items are defined in record in your files
• Item groups are defined in record in your files

For instance, this means that if you're using JSONL, you'll want to make sure that all data mapped to an item or item group is present on one line of your file, and not scattered across many lines.

If mapped entity is duplicated, the duplicates will be ignored and only the first version will be ingested.

On the other hand, it's fine to have variations scattered across the file. You can group them together or nest them under your items easily with the connector ecosystem.

Ensure ingestion order

The files are ingested by the order that they're received via your preferred data transfer method.

To ensure that your catalog files will be ingested in the correct order, it's a good practice to make sure that the feeds you're sending are properly scheduled in a way that won't cause conflicts.

For instance, let's say that you are sending a full feed daily plus a pricing feed hourly. Make sure that the pricing feed is sent after the full feed, so that once you replace the catalog you still keep the latest pricing data.

Ensure force flags are properly set

Be careful when using the force flag in production, as it can wipe out your data if you use it without care.

❗️

Caution

Both forcedelta and forcesync are powerful tools intended for manual use under specific circumstances. Including these keywords in a catalog file can bypass certain safeguards designed to protect the integrity of your product catalog.

For example, both of these operations ignore the 50% safeguard, which prevents a sync from overwriting an existing catalog. If used carelessly, you could accidentally delete more than half of your products. If a significant number of products are unintentionally removed, it could affect the availability of items for customers, leading to a poor user experience and potential loss of sales.

If you wish to use these keywords, we recommend you first reach out to your Customer Success Manager to discuss your specific needs before proceeding.