Keboola Status

New Features and UI Improvements

• There is a new ui picker that allows you to easily switch between projects and their organizations.
  

  

• Detail of a transformation bucket was tweaked. Button for adding transformations is reachable better, phases and backends are better arranged and design of the list of transformations was united with similar lists in other components.
  

  

• Sidebars across the whole application were united and polished.
  

  

• Editing descriptions across the whole application was polished too.
  

  

• Input mappings in a transformation detail can be reordered using drag & drop.
  

  

Updated Components

• There is a new template for getting projects and tasks in detail within Asana Extractor.
• Updated Typeform Extractor  to download data from Responses API instead of Data API

New Components

• Microsoft Dynamics 365 Extractor and Writer by our KDS team allows downloading and sending data to this CRM tool.
• Microsoft SharePoint Lists Extractor by KDS team allows downloading collections of data from SharePoint lists.
• Azure Blob Storage Extractor and Writer by KDS team allows reading and writing data to Azure Storage similarly to our S3 Writer.
  
• OneDrive Writer by Jakub Bartel which uploads files from Storage to OneDrive or SharePoint account.

We are investigating problems in Storage which could lead to inappropriate processing of incremental loading and related functionality. We will keep you updated.

UPDATE Jan 29 9:45 CET - Problem is that incremental load from transformations is not setting _timestamp column. We had identified a release that was causing trouble and finished its revert. Unfortunately, already created table rows having the _timestamp with null values still won't be processed by incremental loading. You can fix this situation manually by running a full load. Now we are running tests to isolate the root cause and preparing global correction of its side effects.

UPDATE Jan 29 14:00 CET - The _timestamp values are written correctly for all newly added table rows since the morning release revert. So now the problem persists only with the rows added previously (ca during yesterday and today's night) which still contain empty timestamp values and so won't be processed e.g. by writers configured to incremental loads. (You can overcome it temporarily by setting them to full load.) Now we are working on backfilling missing timestamp values and so fixing the incremental functionality for these affected data.

UPDATE Jan 29 18:00 CET - The backfill of missing timestamps is being prepared and should be ready tomorrow.

UPDATE Jan 30 15:00 CET - The backfill mechanism is released. Now we are performing last checks before applying it to all affected data. 

UPDATE Jan 30 16:30 CET - Unfortunately, we are forced to leave the backfill for tomorrow. There is an incident in Travis CI which is slowing down our work the whole day and is still not resolved and we decided not to risk running it rashly during the night to be able to support it properly if anything goes wrong.

UPDATE Jan 31 10:30 CET - All preparations are done and preliminary test finished. We are just starting with the backfill.

UPDATE Jan 31 12:00 CET - The backfill is running and ca. 10 % of affected data is fixed.

UPDATE Jan 31 14:00 CET - The backfill is running and ca. 20 % of affected data is fixed.

UPDATE Jan 31 15:30 CET - The backfill is still running and ca. 25 % of affected data in US region and ca. 40 % in EU region are fixed.

UPDATE Jan 31 17:00 CET - The backfill is still running and ca. 70 % of affected data in US region and ca. 100 % in EU region are fixed.

UPDATE Jan 31 17:30 CET - The backfill is finally finished. All the affected data are fixed and have _timestamp column set to "2020-01-29 00:00:00". So that you should update your settings if you have incremental loads set to lower value then 3 days.

SQLdep API suddenly stopped working and because of that transformations validation and SQL analysis are not working properly. (You will probably get false negatives for all validation requests.)

We are investigating the issue and will let you know once we know more.

UPDATE: We've resolved the issue and the integration with the service is working again. It lied in a change of SQLdep API address which we had unfortunately missed.

Updated Components

Tableau TDE writer

• will now retry when a request failed - this should greatly improve reliability as network glitch or temporary unavailability of remote server won't fail the job. Instead it will wait a little and retry the request. 

New Features

• Every new Keboola Connection user will now get Guide mode, regardless if they were invited by current user or registered themselves. It will guide them hands-on though the basics of using Keboola Connection

Minor Improvements

• In Transformation detail, Backend and Phase can be changed from the sidebar
  

  

There was increased amount of server errors in US region that resulted in issues in Keboola Connection UI. No jobs were affected. 

Jan 22 15:50 UTC - increased amount of server errors

Jan 22 16:08 UTC - the issue is mostly resolved

Jan 22 16:19 UTC - the issue is completely resolved

Jan 22 18:28 UTC - the post was amended, original version mistakenly mentioned EU region was affected while actually this was an issue in the US region.

New Features and Updates

Incremental Fetching

The following extractors now support Incremental Fetching:

• Snowflake Extractor
• Oracle Extractor
• Redshift Extractor
  
• Firebird Extractor
  
• Cloudera Impala Extractor 

Job Performance

When a job is finished, we show you how long it took. If a job takes 10% longer (or less) than the average, we'll show you the difference.

Transformation Buckets

The list of Transformation Buckets is no longer collapsible and looks like other lists of configurations in an application.

Bucket details in Data Catalog

In a source project (a project which shares a bucket and "owns" data), you are able to see the bucket detail.

• You can see how the bucket is shared (to organization members, projects members, selected projects, etc.).
• You can edit the description of the bucket — it will be available for those users who want to link the bucket to their project.
• You can see a list of tables and bucket properties (size, row count, etc.).
• Also, if someone uses the bucket, you can see which project it is (Used by section).

In a destination project (where you can see and use a bucket):

• You can see the description of the bucket and decide if you want to use it.
• You can see a list of tables and bucket properties (size, row count, etc.).
• If you decide to use the bucket, you will also see the link to Storage.

Show tables searched by metadata in Writers

In case you use our new feature Scaffolds, related writers (e.g., the Snowflake database writer) will show you all input tables that are searched by metadata.

Environment updates

• R updated to 3.6.2
• Python updated to 3.7.5

New Components

• Deal Prediction application by Bootiq predicts deal outcomes based on metrics from CRM software of your choice.
• Google My Business extractor gets reviews and media related information from your Google My Business account.
  

There will be a maintenance period of Developer Portal on Sunday, January 12, 2020 from 7:00am UTC and should take around 2 hours.

This maintenance won't affect Connection nor any other Keboola service.

We are investigating VPN connection issues for extractors and writers in both regions. 

UPDATE Jan 01, 10:03 AM (UTC) - Our VPN provider is investigating the issue

UPDATE Jan 01, 11:54 AM (UTC) - Our VPN provider has found the root cause and fixed the issue. All VPN connections should be working again.

New Components

TimeDoctor extractor 

• Time Doctor is a time tracking software which allows employees to track time per project and task. Team managers are then able to monitor their computer activity and evaluate effectiveness.
• The Time Doctor extractor for Keboola Connection allows you to download Time Doctor data about users and their activity. 
• This component was developed by KDS Team.

Snapchat Marketing extractor

• The Snapchat marketing extractor allows users to download data about marketing campaigns and statistics via the Snapchat Marketing API.
• It allows to download statistics for campaigns, adsquads, and ads with all supported metrics from Snapchat with either hourly or daily granularity.
  
• This component was developed by KDS Team.

Updated Components

• All database extractors now retry on all the SQL queries they perform.
• Firebird and Impala extractors now support simple table & column select mode and incremental fetching.

New Features

New type of sharing buckets

• From now on, you can share buckets to specified users. Any listed project users in the organization can use the shared data bucket.

  

Minor Improvements

• Increased speed of name, description and metadata editing in components.

• Latest Activity shown on the Dashboard also for configurations. We also added grouping by events.

  

• Added button for creating a bucket and file in the Storage console.

  
• Tables can no longer be created as direct text input. 

  
• Redesigned error display for jobs; now errors have their own box and details can be shown in the modal window.

  
• Orchestration detail now shows the trendline for previous jobs.

  
• Already seen last time was added to Dashboard Latest Activity.

  

We are experiencing snowflake jobs being processed too slow in EU region. We keep on investigating the issue.

UPDATE Dec 11, 10:57 AM (UTC) - We increased computing power of our Snowflake warehouse and there should be no slow jobs for now, furthermore we keep on monitoring the jobs and investigating the root cause.