Sudden Jobs Failures on July 6th, 2017

On July 6th, 2017, between 9:16-9:18am CET one of our internal databases was forced to update and restart by AWS. Result of this action was Application error failure of all running components jobs. The error may show up significantly later than the restart.  We recommend to review your orchestrations jobs and take action if needed. We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

Jobs queues overload (resolved)

since 3:50am CET we are experiencing server queues overload. We are still investigating the issue and will inform about the progress.

UPDATE 5:45am CET- We found the possible root of the cause, Snowflake queries are being unusually queued, furthermore we are unable to raise power or number of cluster of our Snowflake warehouses and we waiting for Snowflake Support findings.

UPDATE  6:10am CET from Snowflake support: Engineering has started mitigation steps to address this issue, we will provide another update shortly.

UPDATE 7:00am CET: Snowflake queries queues has got empty and everything looks to be back to normal.

UPDATE RESOLVED 9:00am CET:  We confirm the issue has been resolved, the cause was the unexpected queueing of Snowflake queries. Snowflake confirmed and rolled back their latest release. However, as a consequence our waiting jobs reached backoff threshold and timeout, which resulted in orchestration failures. We recommend to review your project orchestrations jobs and take action if needed.

Apifier extractor (Public Beta)

We are pleased to announce the public beta release of the Apifier extractor. The Apifier extractor runs a web crawler and stores the results in Keboola storage. The crawler is created and configured in Apifier and can also be adjusted in extractor configuration. This way you can extract structured data from any website and import it directly do Keboola.

Apifier already contains many community web crawlers that can be reused, so you can seamlessly download restaurant reviews from Yelp, hotels from TripAdvisor, or request the creation of a custom crawler for an additional fee.

Week in Review -- June 28, 2017

RStudio and Jupyter Sandboxes

RStudio and Jupyter sandboxes are now in public beta, you can try them right now in your projects.
You can now also see and extend your sandbox expiration.

We are working on:

  • HTTPS Support.
  • Configurable (pay as you go) memory and disk limits.
  • Creating better UI and integration with transformations.
  • Allowing tables and files to be added to a running sandbox.

Keboola Connection in EU

We have launched Keboola Connection in a new region outside of the USA. You can now create your projects in the EU-Frankfurt region, please contact support@keboola.com if you are interested. This new region is now in beta.

Minor improvements

  • Improved performance of PostgreSQL extractor.  

Loading Events Application Errors (Updated)

Friday June 23rd, 20:30 PST

We're experiencing a large number of application errors when loading events from Storage in the UI. 

We're trying to debug and fix this issue, no other operations should be affected.

Friday June 23rd, 20:48 PST

We have rolled out new servers which seems to mitigate the issue. All operations back to normal.

We're sorry for this inconvenience.

Redshift Writer Job Failures

Today (Jun 21, 2017) between 16:00 and 18:30 CEST all Redshift Writer jobs created in this period failed with an application error. We have identified and fixed the issue.

We're sorry for this inconvenience.

GA Extractor Quota Limits

Recently (this week) we've begun hitting request quota limits with our Google Analytics Extractor.  The process of increasing the quota has begun, but until that happens we will unfortunately be required to put a hard cap on the extractor runtime of 30 minutes.  This means that only extractors that finish their jobs in under 30 minutes will complete successfully.

If you are affected by this limit, please try adjusting your configuration to reduce the amount of data being extracted.

Sorry for this inconvenience, we'll update here when/if the quota gets increased and we can remove this limitation.

Fixed IP Address Ranges - temporarily reverted

Some database extractor timeout errors have occurred in a few projects due to yesterday's switch to static IPs. 

We have temporarily reverted back from from new Fixed IPs as we attempt to resolve these issues.

We will post an update when the fixed IPs return and the timeout issues resolved.
 

Temporarily Out of Order Component Synchronous Actions

A release yesterday has broken some synchronous actions for some Keboola Connection components.

No jobs are affected by this, the components run as before, but creating and updating components may be affected.

We are currently working on a patch and will update here with more information.

Thank you for your patience.

Update: 2017-05-17 11:38

All synchronous actions should now be working as expected.  

Job failures

There were few job failures and the UI encountered a brief outage in some API calls on May 3 between 16:07 and 16:14 GMT+2/CEST

Affected jobs and API calls returned an Application error. 

The outage was caused by an automatic upgrade of one of our metadata DB servers. We will be turning off automatic upgrades to prevent this in the future.

We're sorry for this inconvenience.