New Features

Partial Label

Transformation description, Last Runs, and Updates

• There's a new option to save transformation with description.
  

• The Last Runs section and Updates section have been added also to the Transformation Detail page and all components which support "single runs" (e.g. when HTTP extractor or AWS S3 extractor is extracting one file)

Input Mapping data types

In transformations with Snowflake backend, data types should be populated automatically for tables created with the components which set data types - MySQL extractor, Oracle extractor, MSSQL Server extractor, PostgreSQL extractor, DB2 extractor, and Snowflake extractor.

Input Mapping load type

For Snowflake backend, you can set Clone Table load type which leads to an ultra-fast load to a workspace. Most of your tables should be loaded under 10 seconds.

You can find more about this feature in our documentation.

Enhancements

• You can see a component type in Orchestration Tasks, thus having both MySQL writer and MySQL extractor in task won't be confusing anymore
• API Token can be Refreshed, Sent or Deleted directly from the token detail page

• We improved how data is loaded to Storage from our components - compression is used by default. Loads to Storage are about 40% faster for sliced files. So it impacts Credit Consumption too.
• MSSQL Server Extractor has new option WITH(NOLOCK) and supports incremental fetching with smalldatetime column
• Orchestration Notifications page has been improved, and notifications can be set easier

New components and component updates

• In RStudio/Jupyter sandboxes there's a hard memory limit set to 8GB. This limit is equal to memory limit in Transformations. With this change, you can be sure that what worked in Sandbox will work in Transformation too.
• Rename Files processor to rename files using regular expressions: https://components.keboola.com/components/jakub-bartel.processor-rename-files
• XLS processor to convert .xls(x) files to .csv files: https://components.keboola.com/components/jakub-bartel.processor-xls2csv
• Snowflake writer uses atomic swap of tables during full load (prevents empty table issue)

From around 21:10 CET today, Snowflake began having availability issues in the EU region.

https://community.snowflake.com/s/group/0F90Z000000U8dJ/alerts-awseu-frankfurt

[Update 21:33 CET -- They have discovered the cause and are working on a remedy] 

[Update 21:42 CET -- Service is now fully restored in the EU region]

The orchestration section is one of the last parts of Keboola Connection which does not yet have a unified user interface. We believe that users should feel comfortable using the different parts of our UI, so unifying the interface elements is very important to us.

We have launched the new Orchestration Detail page

What has changed?

• There is no longer a sidebar on the left side with a list of orchestrations
• Tasks, Schedule and Notifications now have their own place in the detail page - they're no longer combined in that table
• We added new sidebar on the right side that you should feel familiar with because it is the same as other components
• Orchestration Action buttons (Run, Enable/Disable and Delete) have been moved to the right sidebar
• Also, information about creation date and updates now also have their own place in the right sidebar

This is the beginning of the Orchestration Interface tuning and there are more things to come.

---

Just for completeness, this is the previous version:

During deployment of Keboola Connection in the EU region at around 13:30 CET today there was a timing issue that resulted in some failed storage jobs.

The problem occurred because of the asynchronous timing between the rollover of a worker and the API server.  A small number of jobs that were created with the previous version then failed when executed against the new API.  These jobs failed with one of the following error messages "`Invalid source type" or "Workspace not found".

There was no corruption of data by these failures as no jobs that were in progress were affected.

 

We're sorry for this inconvenience.

New Features

Updated Components

SQL Server Extractor now supports incremental fetching on DateTime and Identity columns

Unexpected Events

Many of you have noticed that we added validation of SQL queries to the Transformation Detail page last week.

Due to some problems in availability of this feature we decided to revert this change.

For now you can still use the "standard" validation button on the right.

We love this feature too and the plan is to bring it back after an additional round of improvements.

Today between 10 and 11 a.m. (UTC) SQL validations and SQLdep functionality in Transformations were returning errors.

The errors were caused by temporary outage of external SQLdep service (https://status.sqldep.com/779807590). Transformation jobs were not affected.

We're experiencing a high number of errors in Storage Jobs in the US region. The errors are caused by invalid SSL connection to Snowflake and we're investigating the issue.

Update 01:02 am CET: Snowflake issues an official statement.

Update 07:00 am CET: Snowflake updated an issue. We are working on a workaround.

Update 07:56 am CET: Issue is resolved. Snowflake moved the Keboola account to a new certificate provider. Now we are processing backlog of jobs.

Update 09:20 am CET: Backlog is cleared. All systems are operating normally.

Revised Project Invitations

Inviting users to KBC projects has been reworked. Previously, users were directly added to the project via Users & Settings -> Users menu. Now, when adding a new user to the project, an invitation email is sent to the user.

The invited user can accept or decline the pending invitations in the Account Settings.

See the docs for more details.

New Features

• We have added some predefined searches to the job list. See the docs for a more detailed description of the search queries.

Minor Improvements

• Jobs related to Python/R sandboxes now show the proper link to a configuration:

• Improved description of components in events

• Configurations in trash are now sorted by the date of deletion:

• Component configuration state is reset on copy:

UI Improvements and fixes

• Dialog for adding a new task to orchestration now contains configuration description 

  
• Every job detail events log now contains component info

  
• Improved orchestration notifications ui

  
• Improved brief summary of orchestration tasks

  
• Unified view of data mappings in the job detail page with respect to the running component as follows: 
• Input - represents all the data flowing from the storage into the component - typically this would apply for transformation or writer component type. 
• Output - represents all the data flowing from the component into the storage, typically this would apply for extractor component type.

  

• Code editor support for JSON editing of component setup in Developer portal 

• When creating a new extractor in Developer portal  the default bucket is automatically turned on
• Improved naming of new phase and phase names capitalisation