tag:status.keboola.com,2013:/posts Keboola Status 2022-10-02T18:50:22Z Keboola Connection - Platform Status tag:status.keboola.com,2013:Post/1885865 2022-10-02T18:49:45Z 2022-10-02T18:50:22Z Stuck jobs in EU stack

We are investigating stucked jobs since Oct 2 11:00 UTC on https://connection.eu-central-1.keboola.com/ stack.

18:47 UTC - Stucked jobs were unblocked. We continue to monitor the situation. Everything should be operational now and running as expected.

]]>
tag:status.keboola.com,2013:Post/1884518 2022-09-29T14:10:04Z 2022-09-29T15:15:59Z Stuck jobs in AWS US and EU stack

14:08 UTC: We are investigating created jobs stuck to start processing in AWS US and EU stacks. The next update in 15 minutes.

14:23 UTC: We found the root cause, rolled back the previous working version and newly created job now should process as expected. We are still trying to push previously created jobs (created within 1 hour ago) to the processing state. The next update in 15 minutes.

14:50 UTC: We have released the stuck jobs in US stack, we continue to release the stuck jobs in EU stack. Next update in 30 minutes.

15:15 UTC: All stuck jobs in EU have been released. Everything is operational now and running as expected.

]]>
tag:status.keboola.com,2013:Post/1884473 2022-09-29T10:11:02Z 2022-09-29T10:31:47Z Newly created transformations run failing on unrecognized option

10:10 UTC: We are investigating newly created transformations (in last 24 hours) that if run fails with an error `Unrecognized option "blocks" under "container"`. Next update in 15 minutes.

10:30 UTC [resolved]: We have identified the root cause and revert the latest changes of UI. From now on if a user creates a new transformation it will run without an error. However already created transformation are still affected, and manual update of the transformation is needed. In such case on the transformation detail page go the raw config editing page by appending "/raw" to the transformation detail page url. Then remove the topmost "blocks" array from the configuration and save it. 

]]>
tag:status.keboola.com,2013:Post/1884453 2022-09-29T07:37:34Z 2022-09-29T07:53:40Z Azure North Europe stack - jobs API higher error rate

07:36 UTC: We are experiencing higher error rate of Jobs API, we are investigating it. Next update in 15 minutes. 

07:52 UTC [resolved]: We identified the root cause - the projects migration running in the background caused higher load to our metadata database. This led to slower api response and in some cases timeout of the api request. However this didn't affect any running jobs nor running new ones. Everything is running as expected and fully operational now.

]]>
tag:status.keboola.com,2013:Post/1882464 2022-09-24T07:14:05Z 2022-09-24T12:33:29Z Scheduled Maintenance of AWS EU Stack starting now As announced previously, we’re starting a planned maintenance on the AWS EU stack. One of our internal databases will be upgraded.

Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. 

We'll update this post when the maintenance is finished. 

UPDATE 09:06 UTC: The maintenance is taking longer than expected. Next update in 30 minutes or when new information is available.

UPDATE 09:33 UTC: The maintenance has successfully finished and the stack is now fully operational.

UPDATE 10:30 UTC: Unfortunately the database needs another immediate upgrade. We'll be turning on maintenance back on shortly. Next update in 30 minutes.

We're sorry for any inconvenience and thanks for your understanding. 

UPDATE 11:00 UTC: We are starting second round of maintenance on the AWS EU stack. Stack should be back up and running within 90 minutes.

UPDATE 12:33 UTC: The maintenance has successfully finished and the stack is now fully operational.
]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1881848 2022-09-22T17:51:14Z 2022-09-22T18:21:24Z Azure North Europe stack - Pay-as-you-go projects - Billing api problems

2022-09-22 17:50 UTC - We are investigating issue with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). Non Pay As You Go projects should not be affected.

Next update in 30 minutes.

2022-09-22 18:20 UTC - The issue was caused by heavy load on the underlying database during parallel migration of some projects to new Queue. Services are now returning to normal operations. It is possible that some orchestrations will have their schedule delayed. We are very sorry for any inconvenience.

]]>
tag:status.keboola.com,2013:Post/1879355 2022-09-14T12:50:57Z 2022-09-14T12:50:57Z Update on Scheduled Maintenance of AWS EU Stack on 2022-09-24

We’re sorry to let you know that the previously announced maintenance on September 24th will have more of impact on the platform operations than previously thought. Due to unforeseen circumstances, we’re unable to ensure the previously announced scope of the maintenance, and we need to turn off the platform completely for about 90 minutes. 

The AWS EU maintenance will start at 9:30 AM CET, and we expect it to finish by 11 AM CET. 

During the maintenance, you will not be able to access your data and projects. All network connections will be terminated by the "HTTP 503 – down for maintenance" status message.

All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. However, feel free to re-schedule your Saturday orchestrations to avoid this maintenance window.

We apologize for any inconvenience and thank you for your understanding.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1877213 2022-09-07T11:36:59Z 2022-09-07T17:13:30Z Delayed orchestrations on Azure North Europe stack

2022-09-07 11:35 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-09-07 12:25 UTC - After our investigation, problem starts at 10:00 UTC. It is caused by outage of our Billing Api. Orchestrations that should have be scheduled after this time, are delayed in the order of minutes or tens of minutes. This applies to standard projects.

Orchestrations of Pay As You Go projects are not scheduled at the moment.

Next update in 60 minutes.

2022-09-07 14:00 UTC - We implemented a hotfix and the situation is returning back to normal. Orchestrations should be scheduled properly now. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)

We are monitoring a situation, next update in two hours.

2022-09-07 17:10 UTC - All orchestrations are executed without delay. We apologize for any inconvenience.


]]>
Erik Žigo
tag:status.keboola.com,2013:Post/1877196 2022-09-07T10:57:37Z 2022-09-07T17:12:10Z Azure North Europe stack - Pay-as-you-go projects - Billing api problems

2022-09-07 10:55 UTC - We are investigating problems with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. Non Pay As You Go projects should not be affected.

Next update in 30 minutes.

2022-09-07 12:10 UTC - Problems with one of the Azure services still persist and our Billing Api is still unavailable. We are waiting for service recovery by Azure Support.

All of Pay As You Go projects are affected:

  • Your credit balance is temporary on zero value
  • Orchestration jobs are not scheduled at the moment
  • You cannot manually execute any component

Next update in 60 minutes.

2022-09-07 13:15 UTC - After our investigation, problem starts at 10:00 UTC. Azure service is still not working properly, but we implemented fix to avoid this situation until Azure Support resolve the root cause.

We keep monitoring the situation closely. At the moment Billing Api service is available, jobs and orchestrations are running.

Next update in 120 minutes.

2022-09-07 17:10 UTC - All Keboola Connection services are running normally. The incident is resolved. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)

We apologize for any inconvenience.

]]>
Erik Žigo
tag:status.keboola.com,2013:Post/1874724 2022-08-30T09:10:55Z 2022-08-30T10:26:25Z Jobs Failing on Internal Errors in Azure North Europe stack

2022-08-30 9:10 UTC - We are investigating some jobs failing on internal error since 6:30 UTC. Next update in 30 minutes.

2022-08-30 9:20 UTC - We identified the root cause in some DNS resolving errors in Azure VMs, see Azure status for details: https://status.azure.com/en-us/status. We are restarting the running nodes which should solve the problem. Next update in 30 minutes.

2022-08-30 9:40 UTC - The Kubernetes nodes that are running containers with jobs were restarted and no error is visible in the logs since then.

]]>
tag:status.keboola.com,2013:Post/1872026 2022-08-23T13:36:04Z 2022-08-23T13:36:39Z Failing Synchronous Actions

Synchronous actions, a mechanism behind some UI features, like testing credentials or listing available databases on a distant server, were affected by a bug and were not working properly from 12:50 UTC until 13:25 UTC when the revert of defective release was finished and the functionality is back to normal since then.

]]>
tag:status.keboola.com,2013:Post/1870910 2022-08-20T14:58:52Z 2022-08-20T15:31:05Z Jobs Failing on Internal Errors in Azure North Europe stack

2022-08-20 14:57 UTC - we are investigating some jobs failing on internal error. Next update in 30 minutes.

2022-08-20 15:30 UTC - we see the internal error is caused by one of the internal components failing to call Azure API, however don't know the root cause. We restarted one of the instances and see no error for now. We continue monitoring the issue.

]]>
tag:status.keboola.com,2013:Post/1868981 2022-08-16T09:49:07Z 2022-08-16T09:49:26Z Scheduled Maintenance of AWS EU Stack 2022-09-24

On 24th September 2022 between 10:00 CET and 10:30 CET, https://connection.eu-central-1.keboola.com will undergo planned maintenance during which one of our internal databases will be upgraded. As a result, any loads or unloads to storage will be paused during that time for up to 30 minutes. Any currently running jobs will continue to run. However, they may be delayed by up to 30 minutes as well.

]]>
tag:status.keboola.com,2013:Post/1868426 2022-08-15T10:43:19Z 2022-08-15T16:01:13Z Delayed processing of jobs in AWS US stack

2022-08-15 10:40 UTC - We are investigating component job delays in connection.keboola.com. Next update when new information will be available or in hour.
2022-08-15 11:14 UTC - We have identified the root cause of the problem and we are working on a solution. Next update when new information will be available or in hour.
2022-08-15 14:04 UTC - We've added another worker to help with the workload and the processing times has returned to normal

2022-08-15 15:32 UTC - We are investigating reoccurrence of the issue causing jobs stuck in Created state. Next update when new information will be available or in hour.

2022-08-15 15:59 UTC - The reoccurrence was resolved. We keep monitoring the situation closely, but at the moment job runtimes should be back to the normal.

]]>
tag:status.keboola.com,2013:Post/1864989 2022-08-06T16:36:30Z 2022-08-06T17:46:30Z Azure north europe higher error rates and latencies

2022-08-06 16:35 UTC - We are investigating higher error rates and latencies in connection.north-europe.azure.keboola.com stack. Next update in 30 minutes.

2022-08-06 17:20 UTC - Higher errors rates are resolved and latencies are back to normal. Schedules might be delayed, the backlog is processed. 

2022-08-06 17:46 UTC - Backlogs is cleared. This incident has been resolved.

]]>
tag:status.keboola.com,2013:Post/1864329 2022-08-05T00:48:10Z 2022-08-05T01:01:04Z Delayed processing of jobs in AWS US stack

2022-08-05 00:40 UTC - We are investigating component job delays in connection.keboola.com.  Next update when new information will be available or in hour.

UPDATE 2022-08-05 00:55 UTC - We have identified the root cause and replaced the faulty instances.

UPDATE 2022-08-05 01:00 UTC - All operations are back to normal. We're sorry for this inconvenience. 

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1863993 2022-08-04T12:07:21Z 2022-08-04T13:08:07Z AWS us-east-1 internal job errors [resolved]

2022-08-04 12:05 UTC We are investigating internal job errors. Next update in 30 minutes.

2022-08-04 12:36 UTC We experienced two waves of internal errors, between 11:50-11:57 and 12:00-12:07 UTC. The root cause has been removed and I don't see any other errors in the reporting. We continue to monitoring the situation. Next update in 30 minutes.

2022-08-04 13:06 UTC All Keboola Connection services are running normally. The incident is resolved. 

We apologize for any inconvenience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/1863975 2022-08-04T09:42:56Z 2022-08-04T09:54:29Z HTML source code visible on error pages

2022-08-04 09:40 UTC We are investigating bug, which causing error pages rendered as html source code.


2022-08-04 09:55 UTC Issue is now resolved all systems are working normally.


We are sorry for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/1863069 2022-08-02T14:21:17Z 2022-08-02T17:03:34Z Tables with _tmp suffix might be dropped

We are experiencing an issue with possibility of tables with _tmp suffix might be dropped.

If you manipulate with a table <X> and there also exist a table with name <X>_tmp, the table <X>_tmp could be accidentally dropped.

These tables (<X> and <X>_tmp) have to be in the same bucket and your project has to have Snowflake backend. We found the root cause and fix will be deployed in 2 hours.

Affected tables will be restored.

2022-08-02 17:00 UTC - We have deployed the fix of this issue and fixed affected tables. The issue is now resolved. We contact affected customers directly.

We are sorry for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/1862968 2022-08-02T06:59:55Z 2022-08-02T07:28:25Z Delayed orchestrations on Azure North Europe stack

2022-08-02 7:00 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-08-02 7:30 UTC  After our investigation, problem starts at 6:00 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:15 UTC and all orchestrations are scheduled properly.


We are sorry for the inconvenience.]]>
Erik Žigo
tag:status.keboola.com,2013:Post/1860568 2022-07-27T11:14:19Z 2022-07-27T17:38:59Z Slower Snowflake transformation loads for dynamic backends

2022-07-27 11:12 UTC - Snowflake transformations does not respect selected warehouse size during input mapping which might lead to slower processing times. We are working on a  fix which might be deployed in 4 hours.

2022-07-27 14:59 UTC - The fix was not yet deployed due to unexpected complications. We are working on it and the fix should be deployed in 4 hours.

2022-07-27 17:34 UTC - The fix was deployed. All Snowflake transformations are now using correct Warehouse size for execution. The issue is now resolved.

]]>
tag:status.keboola.com,2013:Post/1860536 2022-07-27T09:17:06Z 2022-07-27T16:35:09Z Tables with _tmp suffix might be dropped

We are experiencing an issue with possibility of tables with _tmp suffix might be dropped. These tables have to be in the same bucket, and there have to exist a table with the same name, just without the _tmp suffix. We found the root cause and we are working on a fix which might be deployed in 4 hours.

2022-07-27 11:17 UTC - We have deployed the fix of this issue. Affected customers were contacted directly. We'll undrop affected tables.

2022-07-27 16:34 UTC - We have fixed affected tables. The issue is now resolved. We are sorry for the inconvenience.


]]>
tag:status.keboola.com,2013:Post/1860159 2022-07-26T14:24:08Z 2022-07-26T14:24:09Z Migration to the new Job Queue in North Europe (Continuation)

Starting on August 8th we will resume migrations to the new Job Queue in our Azure North Europe stack.

This is a follow up to the previous round of migration, and all relevant information can be found in our previous post.

If you're not sure if your project is already on the new Job Queue, go to orchestrations in your project and if the url ends with "orchestrations-v2" then your project is already on the new Job Queue.

If you have any questions regarding the migration, please contact us via support@keboola.com



]]>
tag:status.keboola.com,2013:Post/1859903 2022-07-25T14:09:26Z 2022-07-25T14:34:33Z PostgreSQL data destination writing no data

Today at 07:38 (UTC) a new version (1.9.1) of PostgreSQL data destination was released which introduced a bug causing some configurations to write no data. The component version was rolled back to 1.9.0 at 13:48 (UTC) and everything should work as expected.

If you still encounter similar symptoms please let us know.

We're sorry for this inconvenience. 

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1854924 2022-07-15T09:16:44Z 2022-07-15T09:16:45Z Stricter use of parameters in the Create File Resource API endpoint
We recently updated the API endpoint for Create File Resource to make the validation of the parameters stricter. In doing so, we discovered a misuse in our code examples (Python Example, Manually Uploading a File).

If your Create File Resource stops working, please modify the Create File Resource API endpoint call to insert the parameter federationToken=1 into the body of the API call.

If you need any more information, please visit our documentation.

We apologize for any inconvenience.

]]>
tag:status.keboola.com,2013:Post/1854081 2022-07-12T15:41:39Z 2022-07-12T16:18:51Z UI Bug in the transformations input mapping

We are aware of a bug in the UI that prevents tables in the input mapping list from being visible. We are in the process of reverting the release causing it and will find a fix afterward.

UPDATE 18:15 CET: The release introducing the bug was reverted so it does not happen again. You may need to reload the browser. The bug was introduced today at 16:04 CET.

 

]]>
tag:status.keboola.com,2013:Post/1849885 2022-07-02T07:45:47Z 2022-07-02T08:40:45Z Delayed orchestrations on Azure North Europe stack

2022-07-02 7:45 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-07-02 8:40 UTC  After our investigation, problem starts at 6:40 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:45 UTC and all orchestrations are scheduled properly.

We are sorry for the inconvenience.

]]>
Erik Žigo
tag:status.keboola.com,2013:Post/1846089 2022-06-23T22:37:27Z 2022-06-23T23:01:30Z Stuck Jobs in US Stack

2022-06-23 22:30 UTC: We're investigating cases of jobs not processing in the US stack (https://connection.keboola.com/). Next update in 30 minutes.

2022-06-23 07 23:00 UTC: The situation is now under control and our services are running normally. Processing jobs will start to gradually speed up.

We are sorry for the inconvenience.

]]>
Ondřej Popelka
tag:status.keboola.com,2013:Post/1845827 2022-06-23T14:16:22Z 2022-06-23T14:16:22Z Non-functional use cases tool in the Flow Builder on Azure North Europe stack

2022-06-23 16:15 CET We detected a problem with non-function use cases in the Flow Builder tool at https://connection.north-europe.azure.keboola.com between 13:00 CET and 15:30 CET

The situation is now under control and our service is running normally.


]]>
tag:status.keboola.com,2013:Post/1844869 2022-06-21T13:38:14Z 2022-06-21T13:38:14Z Increased Error Rate of Components Using Specific Processor

There was an increased error rate for components using keboola.processor-create-manifest in configurations. This error affected only component configurations which were using empty enclosure in CSV Settings.

Affected component may include data sources like AWS S3, FTP, etc.

We are sorry for any inconvenience. Please feel free to restart your jobs manually.

2022-06-20 16:44 UTC - We are investigating increased error rate of some component using keboola.processor-create-manifest.

2022-06-21 07:22 UTC - We identified a root cause and continue working on a fix.

2022-06-21 12:30 UTC - Incident is resolved, last occurrence of the error was at 12:13 UTC.

]]>
Vladimír Kriška