Weeks in review -- May 20, 2019

Improvements

  • Transformation buckets can now run in parallel. That means when you put multiple transformation buckets in one orchestration phase, they will execute in parallel - the same way extractors or writers work. Previously the transformation jobs would be queued to execute sequentially.
  • The editor of transformation queries now uses the official Snowflake syntax highlighter, so the syntax highlighting should now be consistent with the Snowflake Sandbox.

  • The organization settings now offer a new feature to require MFA (Multi-Factor Authentication) for all organization members.

  • A new GoodData User Management Application was released. It replaces part of the functionality of the old GoodData writer and allows to manage users in the GoodData project, invite new users to the project, change roles and assign data permissions for each user.

Fixes

  • A more readable Roboto font is now used in Keboola Connection and Developer Portal UI.
  • Fixed error when adding metadata to columns with special characters in name.
  • Column metadata are now displayed even when they cannot be edited (due to insufficient permissions, table being an alias or table being hosted on Redshift).

Developers

  • Removed support for deprecated private Quay.io and private Docker Hub repositories (no active components were using it).


Week in Review -- October 9th, 2017

The longstanding issue with slow SHOW DATABASES query executions on Snowflake has finally been resolved. We will now gradually enforce snowflake query execution limits where applicable by contract. Before turning the limits back on, we will check the query history and notify you if you may be affected.
(Query times in milliseconds)

New Features

Minor Improvements

We continue with cleaning up the KBC UI. So far, we have fixed many little annoying issues:

  • List of configurations now shows when the configuration was created (with exact date on hover) 
  • Cleaned up Job list (removed unused stuff).
  • Backend in Transformations is now preselected to default project backend.
  • CSV importer now supports .TSV and .GZ files too.
  • "Test Credentials" button and result polished.
  • Removed 'sticky' buttons when editing configurations.

Fixes

  • Fixed files input mapping not showing in certain configurations.
  • Fixed a couple of errors with the Save button when editing transformation queries.
  • Fixed bug in Redshift writer with mixed case table names.

Developers

  • The encryption API is simplified to a single endpoint. If you manually encrypt values for KBC, please follow the docs. We will be contacting you if you are using the deprecated API calls.


New Generic Extractor Documentation

Long neglected, long awaited, it's here!

We are done with a major overhaul of our Generic Extractor documentation. It is now completely moved to developers.keboola.com. The new documentation contains over 100 runnable examples. We have also created a tutorial which guides you through how to configure the Generic Extractor for a new API.

We have also a configuration map with links to documentation of all parts of the configuration.

In case you haven't heard of the Generic Extractor, it is our universal REST API client. It reads data (in JSON format) from an API according to how you configure it, then converts the results to CSV files and imports them into KBC Storage. This way you can extract data from new APIs in tens of minutes. You can use the Generic Extractor to create a new extractor or perform ad-hoc extractions from an API. 

If you are interested, give a try to the Tutorial.

Week in Review -- March 6, 2017

Minor Improvements

  • We have upgraded the Google Sign-in to the latest most secure version.

Fixes

  • Dropping a empty primary key in Storage no longer causes an error.

  • Creating a Storage table with a missing S3 file now returns a proper error message.

  • Handle correctly a situation when a primary key is supplied as an array to a Storage table.


AWS S3 Outage Resolved

Amazon services are almost restored. Our infrastructure recovered too, everything should be running now. Though the waiting times are expected to be longer. 

We're sorry for this inconvenience and thanks a lot for your understanding and patience.

Call for testers: RStudio and Jupyter Sandboxes

We have been working on RStudio and Jupyter Notebook sandboxes (these are the missing sandboxes for R and Python transformations) for some time and we now feel that it is time to move to the next phase. 

We can now enable these on a per-user basis for early adopters. If you are interested in playing with these Sandboxes please contact us at support@keboola.com (don't forget to tell us your KBC email).

Limitations:

  • Sandboxes are available only as plain sandboxes (not in a transformation bucket yet)
  • HTTPS is not yet supported.
  • Sandbox disk space is limited to 10GB.
  • Memory is limited to 8GB.
  • The UI allows loading only tables to the Sandbox. Loading input files and transformation script is supported only by the API.
  • Sandboxes will be deleted after 5 days.
  • Adding data to existing sandboxes is not supported.

Otherwise, it's like RStudio connected to your KBC project:

The CSV file locations are the same as in transformations (/data/in/tables/). In the meantime, we will work on:

  • HTTPS Support.
  • Creating better UI and integration with transformations.
  • Allowing tables and files to be added to a running sandbox.


Week in review -- December 27, 2016

Hello everyone,

Last week we have updated the R environment for both custom science and transformations to version 3.3.2.

Apart from that not much has happened last week because our office has been seized:


If everything goes as planned, this should be our last post of the year. So we wish you Happy New Year!

Job failures

Job Failures

There were jobs failures between 4 AM - 9 AM UTC caused by one of our servers dying in a horrible slow and painful death.

We're sorry for this inconvenience, we have restarted the affected orchestrations.


New Currency Extractor

We have created a new currency exchange rate extractor. This component allows you to extract currency exchange rates as published by European Central Bank (ECB). Its configuration is dead simple, all you need to do is select the source currency (USD and EUR are currently supported).


Migration

This extractor provides the same data in the same format as the old extractor (which had to be configured through support). We encourage you to switch to the new one - set it up as any other extractor. We will keep running the old extractor till January 2017. Just let us know once you don't need it, so that we can deactivate it and stop magically pushing data to your project.

Like in the old currency extractor, the resulting data contains gaps for bank holidays (including weekends). If you would like to fill the gaps with last known value, you can use a little SQL script we hacked together.

Weeks in Review -- October 19, 2016

Even though the weekly status has been a little weak recently, it does not mean that we're dawdling. We are working on some quite big internal things, which take much more than a week. For example (in no particular order):

  • Developer Portal, which will enable 3rd party developers to manage their applications.
  • Keboola Connection in EU Region.
  • OpenRefine transformations.
  • Validation and simplification of input and output mapping.
  • Internal changes to transformations and sandboxes using so called Workspaces.
  • RStudio and Jupyter Sandboxes.
  • Collecting more statistics about running jobs:
  • New Redshift, Oracle and MSSQL Writers and DynamoDB, Facebook and S3 Extractors.
  • Support of LuckyGuess applications (e.g. Anomaly Detection) on Snowflake.
  • Shared buckets between projects (to replace copying data via Restbox).
  • Resolving problem with ever growing tables on Redshift.
  • Plus we fixed the annoying problem that yelled "Table already exists" when loading data into sandbox.

All of the above are in various state of completeness (except the last item, which is complete). So stay tuned for more announcements when these features are finished.