Migration to the new Job Queue in North Europe (Continuation)

Starting on August 8th we will resume migrations to the new Job Queue in our Azure North Europe stack.

This is a follow up to the previous round of migration, and all relevant information can be found in our previous post.

If you're not sure if your project is already on the new Job Queue, go to orchestrations in your project and if the url ends with "orchestrations-v2" then your project is already on the new Job Queue.

If you have any questions regarding the migration, please contact us via support@keboola.com



PostgreSQL data destination writing no data

Today at 07:38 (UTC) a new version (1.9.1) of PostgreSQL data destination was released which introduced a bug causing some configurations to write no data. The component version was rolled back to 1.9.0 at 13:48 (UTC) and everything should work as expected.

If you still encounter similar symptoms please let us know.

We're sorry for this inconvenience. 

Stricter use of parameters in the Create File Resource API endpoint

We recently updated the API endpoint for Create File Resource to make the validation of the parameters stricter. In doing so, we discovered a misuse in our code examples (Python Example, Manually Uploading a File).

If your Create File Resource stops working, please modify the Create File Resource API endpoint call to insert the parameter federationToken=1 into the body of the API call.

If you need any more information, please visit our documentation.

We apologize for any inconvenience.

UI Bug in the transformations input mapping

We are aware of a bug in the UI that prevents tables in the input mapping list from being visible. We are in the process of reverting the release causing it and will find a fix afterward.

UPDATE 18:15 CET: The release introducing the bug was reverted so it does not happen again. You may need to reload the browser. The bug was introduced today at 16:04 CET.

 

Delayed orchestrations on Azure North Europe stack

2022-07-02 7:45 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-07-02 8:40 UTC  After our investigation, problem starts at 6:40 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:45 UTC and all orchestrations are scheduled properly.

We are sorry for the inconvenience.

Stuck Jobs in US Stack

2022-06-23 22:30 UTC: We're investigating cases of jobs not processing in the US stack (https://connection.keboola.com/). Next update in 30 minutes.

2022-06-23 07 23:00 UTC: The situation is now under control and our services are running normally. Processing jobs will start to gradually speed up.

We are sorry for the inconvenience.

Increased Error Rate of Components Using Specific Processor

There was an increased error rate for components using keboola.processor-create-manifest in configurations. This error affected only component configurations which were using empty enclosure in CSV Settings.

Affected component may include data sources like AWS S3, FTP, etc.

We are sorry for any inconvenience. Please feel free to restart your jobs manually.

2022-06-20 16:44 UTC - We are investigating increased error rate of some component using keboola.processor-create-manifest.

2022-06-21 07:22 UTC - We identified a root cause and continue working on a fix.

2022-06-21 12:30 UTC - Incident is resolved, last occurrence of the error was at 12:13 UTC.

Higher jobs error rate in AWS eu-central-1 stack

2022-06-20 06:40 UTC - We are investigating higher rate of jobs ending with internal error Job "XYZ" terminated unexpectedly in connection.eu-central-1.keboola.com stack since June 18th. At the moment the situation is stabilised and we don't see any more internal errors, we are investigating the root causes. Next update when new information will be available.

2022-06-20 14:36 UTC - Incident is resolved, last occurence of the error was at 05:35 UTC. We have found the jobs causing the errors and notified the project owners.

Job queue high error rate on Azure stacks

2022-06-16  13:09 UTC - We are investigating high job queue error rates on Azure stacks. Next update when new information will be available or in 30 minutes.

2022-06-16  13:28 UTC - We have found a root cause and implemented a fix. API does not longer returns the errors and platform should stabilized. We continue to monitor the situation.  Next update when new information will be available or in 30 minutes.

2022-06-16  13:51 UTC - Incident has been resolved and since 13:28 UTC  we don't see an increased api error rate and platform is stable.