The pre-announced maintenance period for the US and EU regions has just started. The estimated downtime is 5 hours. 

There will be a maintenance period on Saturday, October 6th, 2018 from 8:00am CEST and should take less than 5 hours.

We will be upgrading component job indices and metadata databases.

All projects in connection.keboola.com and connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

Since September 25 we're experiencing degraded Snowflake performance affecting all Snowflake operations. 

We're sorry for this inconvenience, we're working with Snowflake to fix this issue.

Update, October 8

Snowflake Engineering team has discovered and fixed the issue (waiting for an official statement from Snowflake). We're seeing operation times going back to normal.

We have encountered an increased error rate in MySQL Extractor after a new version release on Friday, Sept 14, 16:00 UTC.

These errors happened only when the job had a connection issue and had to reconnect to the MySQL server.

The previous version is now rolled back and we're working on a fix.

We're sorry for this inconvenience.

We have encountered an increased number of Snowflake connection errors between 04:11 UTC and 04:21 UTC. This may have caused failed storage and component jobs.

Furthermore Snowflake announced possible SQL query failures between 02:24 UTC and 03:30 UTC. 

We're sorry for this inconvenience, all systems are operational now. 

We're currently investigating an incident in the US region that may have caused component job and orchestration failures today, Tuesday, Sep 11, between 02:30 UTC and 06:30 UTC.

Update 08:30 UTC

The DB storing job locks (preventing jobs from running multiple times on multiple workers) was restarted at 02:33 UTC. All connections were terminated and all component jobs and transformations running at that time were disconnected. This could lead to any or both of the following situations:

• Failure at any time later during the job execution
• A new parallel job execution

Any jobs started after the DB restart were not affected by this issue.

We apologize for this inconvenience. We're planning an infrastructure change to prevent such huge impact during similar situations. 

If you have any further questions, please use the support button in your project.

Between 4:00 pm CEST and 7:30 pm CEST the new version of Create Manifest processor was failing with an application exception. This change was rolled back and the processor is functional now. 

We're sorry for this inconvenience.

The time has come and the Restbox component as we know it is now being deprecated. But do not worry, its functionality will not disappear.

The Restbox component was once a great tool to collect data from various sources that were not complex enough to deserve a dedicated extractor (or writer) and shared similar processing aspects, such as decompression, audit tools, and CSV formatting. This has now been broken up into multiple separate components and processors to achieve or exceed the required functionality.

Today we have the following components to replace the Restbox component:

• S3 extractor
• S3 writer
• HTTP extractor
• Keboola Connection Storage extractor
• Keboola Connection Storage writer

Detailed documentation for most of them can be found at help.keboola.com.

The Restbox component is now available in the US region only, and, as of this announcement, no new configurations can be created. The component will be supported until November 1, 2018 when it will be finally shut down. Please migrate your configurations to one of the above-mentioned components. 

If you have any trouble migrating your configurations, please do not hesitate to contact us using the support button in your project.

Updated Components

• Dark Sky extractor supports future dates (and gets a forecast for up to 7 days ahead).

Minor Improvements

• Setting up MFA in Keboola Connection can display the code in plain text where QR is not available.
• We have changed Docker storage driver for Docker Runner job workers. We hope this will stabilize docker response times and minimize startup and shutdown overhead. This will be most significant when running short jobs.
• Editing components in Keboola Developer Portal got a facelift.

Unfortunately tonight there were a few more unexpected Application errors and delayed or longer running jobs between 1:25am–4:55am CEST (4:25pm–7:55pm) in the US region.

We have experimented with different storage drives (swapping from SSD to throughput optimized HDD) which lead to initial issues with building Custom Science apps. Attempts to provision further resources lead to too many running jobs at once (you could see "SQLSTATE[HY000] [1040] Too many connections" in the failed app events) and removing some of the additional resources could have yielded some other Application errors too. 

Currently we're running SSD drives again with enough resources to process all workloads. Please restart your failed jobs.

We hope we'll be able to stabilize this whole unfortunate situation as soon as possible and we're very sorry for inconvenience.