Week In Review -- January 16, 2017

Our last week was filled with some bug fixes, performance improvements and old infrastructure deprecation.

Shared Buckets

Shared bucket loading was tweaked to load data directly on Snowflake backend. (available to everyone)

Loading to Sanboxes

Loading data to sandboxes (mainly used by Transformations and Sandboxes themselves) has been improved, which means multiple tables can be loaded at once.

You should notice speed improvement when using more tables in input mapping and loading larger amount of data. (testing phase, available on request)

Also we would like to mention:

Detail of a job with enabled "Multi Load"

Google Drive Extractor Migration

A couple of months ago we launched a new version of Google Drive Extractor and promised to release a migration tool.

Well it took a bit more time then we expected, but as promised, here it is. This tool will help you migrate your configurations to the new version of the extractor.

How to

  1. Click "Proceed to Migration" button in any configuration of the old Google Drive Extractor
  2. A dialog window will open with a list of all configurations in the project
  3. Click the "Migrate" button and a migration job will be created
  4. You can watch the progress of the migration job either in the migration dialog window or in the Jobs list
  5. When the job has finished, you can see the result of the migration for every configuration and also any affected orchestrations will be listed in the "Affected Orchestrations" tab

Backward compatibility

Please be aware that in the new extractor, sheets in one configuration can only be downloaded into the same bucket. In the migration process, the destination bucket is determined from the last sheet of the "old" configuration.

If you have any questions or troubles, please contact support@keboola.com.

Snowflake Database Writer

The much anticipated Snowflake Database Writer is now available!

Key features:

  • container based architecture
  • upload data to your own Snowflake database or use a Keboola provisioned database. 
  • SSH tunnel
  • incremental load

Provisioned database limits:

You can use a Snowflake database provisioned by us, and this database is included in the price of your Keboola Connection project.  Please note, however, that there is a 15 second timeout per query, and an additional fee will be required in order to increase this timeout.

If you have any questions or troubles setting up this writer, please contact our support at support@keboola.com.

Week In Review -- January 9, 2017

Happy New Year everyone! Some good news here, some long awaited features finally made it.

  • Shared buckets allow you to share data between projects. Making RESTbox deprecated one feature at a time.
  • We decided to scrap all Project Power stats older than Jan 5th. We hope this is the last adjustment.
  • Bugfixes in GetStat and Qualtrics extractors.
  • Redshift bloating tables popped up again. We were not able to detect certain cases. Fixed.

Also the UI got some love

  • Link to documentation in the left pane, so you don't have to guess it
  • Long Run Ids wrap correctly in the job detail, if you're into deep job nesting
  • Google Drive Extractor does not delete the input for custom header names every 5 seconds

What's coming up next?

  • New Facebook and Salesforce extractors are getting really close.
  • Developer Portal is almost ready. There is no UI so far, we're API first.
  • Keboola Connection in AWS EU region.

Job failures

There were  jobs failures between 1:30 AM - 6:15 AM UTC caused by flooded disk on one of our servers. We are making steps to avoid this issue happening again.

We're sorry for this inconvenience.

Shared buckets

We're happy to announce the release of the Shared Buckets  feature.  It's an easy way to share data between projects in Keboola Connection.

    This will help you:

    • Have greater organizational control over your data
    • Speedup your data workflow
    • Reduce your project usage totals

    For more details about Shared Buckets, see our User documentation. Developers can find more info in our API documentation in Apiary.

      Project Power Reset

      Since the Project Power feature has been developed/improved in a continuous manner, and the last thing we want to do is provide partial (or worse, inaccurate) results, we have decided to begin measuring credit consumption from the beginning of this year, 2017.

      So all Project Power statistics from 2016 will no longer be shown, and the Project Power era officially begins for everybody from Jan 5, 2017 (as in image bellow).

      Thanks for your understanding.