Tables with _tmp suffix might be dropped

We are experiencing an issue with possibility of tables with _tmp suffix might be dropped. These tables have to be in the same bucket, and there have to exist a table with the same name, just without the _tmp suffix. We found the root cause and we are working on a fix which might be deployed in 4 hours.

2022-07-27 11:17 UTC - We have deployed the fix of this issue. Affected customers were contacted directly. We'll undrop affected tables.

2022-07-27 16:34 UTC - We have fixed affected tables. The issue is now resolved. We are sorry for the inconvenience.


Migration to the new Job Queue in North Europe (Continuation)

Starting on August 8th we will resume migrations to the new Job Queue in our Azure North Europe stack.

This is a follow up to the previous round of migration, and all relevant information can be found in our previous post.

If you're not sure if your project is already on the new Job Queue, go to orchestrations in your project and if the url ends with "orchestrations-v2" then your project is already on the new Job Queue.

If you have any questions regarding the migration, please contact us via support@keboola.com



Stricter use of parameters in the Create File Resource API endpoint

We recently updated the API endpoint for Create File Resource to make the validation of the parameters stricter. In doing so, we discovered a misuse in our code examples (Python Example, Manually Uploading a File).

If your Create File Resource stops working, please modify the Create File Resource API endpoint call to insert the parameter federationToken=1 into the body of the API call.

If you need any more information, please visit our documentation.

We apologize for any inconvenience.

UI Bug in the transformations input mapping

We are aware of a bug in the UI that prevents tables in the input mapping list from being visible. We are in the process of reverting the release causing it and will find a fix afterward.

UPDATE 18:15 CET: The release introducing the bug was reverted so it does not happen again. You may need to reload the browser. The bug was introduced today at 16:04 CET.

 

Higher jobs error rate in AWS eu-central-1 stack

2022-06-20 06:40 UTC - We are investigating higher rate of jobs ending with internal error Job "XYZ" terminated unexpectedly in connection.eu-central-1.keboola.com stack since June 18th. At the moment the situation is stabilised and we don't see any more internal errors, we are investigating the root causes. Next update when new information will be available.

2022-06-20 14:36 UTC - Incident is resolved, last occurence of the error was at 05:35 UTC. We have found the jobs causing the errors and notified the project owners.

Job queue high error rate on Azure stacks

2022-06-16  13:09 UTC - We are investigating high job queue error rates on Azure stacks. Next update when new information will be available or in 30 minutes.

2022-06-16  13:28 UTC - We have found a root cause and implemented a fix. API does not longer returns the errors and platform should stabilized. We continue to monitor the situation.  Next update when new information will be available or in 30 minutes.

2022-06-16  13:51 UTC - Incident has been resolved and since 13:28 UTC  we don't see an increased api error rate and platform is stable.

Increased API Error rate in AWS us-east-1 connection.keboola.com stack

2022-06-10 03:40 UTC Today since 2:50 UTC we are experiencing increased Connection api internal error rates as well as increased latency due to an incident in AWS us-east-1 region. The job processing shouldn't be affected, however there might be slow down in the UI responsiveness.

2022-06-10 04:20 UTC - AWS still haven't resolved the incident. Furthermore we are noticing some jobs processing might be affected ending in internal error due to failed connections retries to AWS API. 

2022-06-10 08:15 UTC - the issue has been resolved and since 04:30 UTC we don't see an increased api error rate.

Stuck jobs in AWS us-east-1 connection.keboola.com stack

2022-06-06 21:00 UTC We are investigating a problem with not processing jobs in AWS US stack https://connection.keboola.com/. Some jobs will probably end up with an internal error.

2022-06-06 21:25 UTC Correction - jobs are only delayed and all jobs ends successfully.

2022-06-06 21:40 UTC Unfortunately the problem still persists, we are trying to find out the cause.

2022-06-06 22:30 UTC We will update this post when we have more information.

2022-06-06 23:25 UTC We have managed to improve the situation. We will continue monitoring the situation and tomorrow we will provide more information about the problem.

2022-06-07 12:40 UTC The situation is now under control and our services are running normally.

We are sorry for the inconvenience.

Increased error rate in Pay-as-you-go projects

Since 2022-06-03 23:30 UTC we're seeing a higher error rate in Pay As You Go projects in Azure Stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. You may encounter higher loading times of the main and billing page and occasional errors when running jobs. Non Pay As You Go projects are not affected.

Update 2022-06-04 7:20 UTC: The Azure service performance was restored back to normal at approximately 2022-06-04 2:40. Our services closely followed. The issue is now resolved.

Next update in 8 hours.