Database Writers with Configuration Rows support

We're happy to announce the arrival of Configuration Rows, our new powerful configuration format, to database writers.

From now on, you'll see a migration button in the configuration detail of each database writer (Snowflake, MySQL, SQL Server, Oracle, PostgreSQL, Impala, Hive, and Redshift).

Just click Migrate Configuration and the configuration will be migrated to the new format.

After the migration, you'll see more information about each table. All tables can be easily reordered, so you can move more important tables to the top and they will be uploaded first.

Also, you will be able to see information about each table on a new table detail page, with Last runs and Versions in a sidebar.

Underlying Important Changes

While there were certain limitations in the old configuration format, this is no longer true in the new "rows format".

The following features are worth mentioning:

  • Disabled tables will no longer be exported from Storage (previously, they were exported with limit=1 and not used in the database writer).
  • Each table has its own state with information about the date/time of the last import (previously, an upload of a single table cleared the state for other tables).


MongoDB Extractor

There's a new extractor available in our group of Docker extractors - MongoDB Extractor.

This extractor allows you fetch data from your MongoDB databases. By specifying collection, query, sort, limit and mapping you're able to extract exactly which parts of your data you want.

On the other hand, MongoDB extractor is very similar to our new set of standard database extractors, so there's a chance you are already familiar with some parts of its UI.

Main features of the MongoDB extractor:

  • you can specify query, sort and limit to filter your data (as in mongoexport commnad)
  • each export has to be named, which helps you identify your exports better
  • there's a mapping section (actually most important) through which you can specify how your data will be processed, how they'll be split to multiple tables and which columns will be exported (thus you can join them with ease)
  • as always, there's an option to have multiple exports in one configuration

Here's a sneak peak of sample configuration:

For more information about its configuration follow the guide at our help site.