Database Writers with Configuration Rows support

We're happy to announce the arrival of Configuration Rows, our new powerful configuration format, to database writers.

From now on, you'll see a migration button in the configuration detail of each database writer (Snowflake, MySQL, SQL Server, Oracle, PostgreSQL, Impala, Hive, and Redshift).

Just click Migrate Configuration and the configuration will be migrated to the new format.

After the migration, you'll see more information about each table. All tables can be easily reordered, so you can move more important tables to the top and they will be uploaded first.

Also, you will be able to see information about each table on a new table detail page, with Last runs and Versions in a sidebar.

Underlying Important Changes

While there were certain limitations in the old configuration format, this is no longer true in the new "rows format".

The following features are worth mentioning:

  • Disabled tables will no longer be exported from Storage (previously, they were exported with limit=1 and not used in the database writer).
  • Each table has its own state with information about the date/time of the last import (previously, an upload of a single table cleared the state for other tables).


Fixed IP Address Ranges

It is our pleasure to announce that as of today, all our outgoing network connections in the US region are using a pool of four static IP addresses. 

This can help you meet your company security standards. To find out more, please visit the IP Addresses page in our documentation.

Be aware that IP addresses can change in the future. For your convenience, you can programatically fetch and parse the list of existing IP addresses in JSON format at https://help.keboola.com/extractors/ip-addresses/kbc-public-ip.json. 

PLEASE NOTE: 

Some of you were employing a very old concept where we performed an on-demand network source routing which allowed us to force the source IP under syrup-out.keboola.com. This was deprecated almost a year ago. Today, we are also announcing that old source routing is deprecated and will be turned off at the end of this month. If you rely on source IP, please move all your existing firewall rules to our new addresses before June 30, 2017.


Shared buckets

We're happy to announce the release of the Shared Buckets  feature.  It's an easy way to share data between projects in Keboola Connection.

    This will help you:

    • Have greater organizational control over your data
    • Speedup your data workflow
    • Reduce your project usage totals

    For more details about Shared Buckets, see our User documentation. Developers can find more info in our API documentation in Apiary.



      Deleting Buckets and Snapshots

      To simplify cleanup tasks in KBC storage we've added support for deleting non-empty buckets.  You no longer have to delete any tables before deleting your bucket, so this should save some time when cleaning up your workspace.




      We've also added support for deleting table snapshots, which should be helpful when working with tables and snapshots.




      Both functions are available through the KBC Storage API, documented here: "drop bucket"  and "delete snapshot"

      Deleting Projects

      To simplify cleanup when a KBC project no longer serves its purpose, we added simple method for deleting it. You'll find that in the "Users & Settings" tab.

      We maintain a recoverable backup copy of each project for 60 days following the deletion. After that period, it's gone for good.


      KBC as a Data Science Brain Interface

      The Keboola Data App Store has a fresh new addition. That brings us to total of 16 currently available apps, three of which provided by development partners.

      This new one is called “aLook Analytics”, and technically it is a clone of our development project, a “Custom Science” app (not available yet, but soon!). It facilitates connection to a GitHub/Bitbucket repository of a specific data science shop, which you can “hire” via the app and enable them to safely work on your project.

      This first instance is connected to Adam Votava’s company aLook Analytics (check them out at http://www.alookanalytics.com/).

      How does it work?

      Let’s imagine you want to build something data-science-complex in your project. You get in touch with aLook and agree on what it is you want them to do for you. You exchange some data, the boys there will do some testing on their side, set up the environment and once they’re done, they’ll give you a short configuration script that you will enter into their app in KBC. Any business agreement regarding their work is to be made directly between you and aLook, Keboola stays on the sidelines for this one.


      When you run the app, your data gets served to aLook’s prepared model and scripts, saved in aLooks repository get executed on Keboola servers. All the complex stuff happens and the resulting data gets returned into your project. The app can be (like any other) included in your Orchestrations, which means it can run automatically as a part of your regular workflow.

      The user of KBC does not have direct access to the script, protecting aLook’s IP (of course, if you agree with them otherwise, we do not put up any barriers).

      Very soon we will enable the generic “Custom Science” app mentioned above. That means that any data science pro can connect their GitHub/Bitbucket themselves - that gives you, our user, the freedom to find the best brain in the world for your job.

      Why people and not just machines?

      No “Machine Learning Drag&Drop” app provides the same quality as a bit of thought by a seasoned data scientist. We’re talking business analytics here! People can put things in context and be creative, while all machines can do is to adjust (sometimes thousands of) parameters and tests the results against a training set. That may be awesome for facial recognition or self-driving car AI, but in any specific business application, a trained brain will beat the machine. Often you don’t even have enough of a test sample so a bit of abstract thinking is critical and irreplaceable.

      Data Takeout

      As a part of our commitment to openness and total, utter and complete aversion to "customer lock in" tactics, we introduced a "Data Takeout" functionality. It's been around for awhile actually, but now it is right there in the UI. This means that shall our customer become less than completely satisfied with us, there's no technical barrier for them to collect all their data AND the project structure (including all transformation scripts and queries etc.) at a push of a button. (And yeah, we took a hint from Google on how to call the service.)

      What this button does is to export everything into AWS Simple Storage Service (S3).

      Several files will be created there, which will contain:

      • All bucket and table metadata (attributes, names, columns, settings)
      • All table data exported to gzipped CSV files
      • All component configurations (e.g. transformation configuration with all queries, database extractor settings and queries, etc.)

      Export can be limited only to metadata and configurations and the "Data Take Out" button can be found at Users & Settings page.