2021-10-19 23:07 UTC - We are investigating  job processing delays in connection.eu-central-1.keboola.com.  Next update when new information will be available or in hour.

Update 2021-10-20 00:01 UTC We have identified the root cause and working on a fix. Next update when new information will be available or in hour.

Update 2021-10-20 00:45 UTC Everything should be running without any issues now. We're sorry for this inconvenience. 

Some jobs of keboola.ex-google-adwords-reports-v201809 in the Azure North Europe stack fail immediately with an error. The job details is missing the job start date and for PAYG customers the billing consumes all available credits.

We're investigating this issue and will update this status in 60 minutes or when an update is available. 

UPDATE 18:43 UTC: We have found the root cause and we're working on a fix. Next update in 60 minutes or when an update is available. 

UPDATE 19:45 UTC: We have fixed billing stats in all affected projects. The root cause has not been fixed yet that means a new job of the keboola.ex-google-adwords-reports-v201809 component will again consume invalid number of credits. We'll update the billing stats later tonight and tomorrow early morning to keep the projects running smoothly. Next update as soon as we have any news or at 07:00 UTC. 

UPDATE Oct 11 06:55 UTC: Unfortunately we're still seeing failing jobs of the affected component after releasing the fix. We're further investigating the issue and preparing a new fix. Next update in 6 hours (13:00 UTC) or when new information is available. 

UPDATE Oct 11 11:40 UTC: We have deployed and verified the fix. We'll continue monitoring jobs closely to see double check for any re-occurrences. 

Oct 9, 2021, 10:00 CET - Azure EU is down for scheduled maintenance: https://status.keboola.com/azure-eu-maintenance-announcement-1

Oct 9, 2021, 10:38 CET - Azure EU is back and fully operational. Thanks for your patience.

We have added new outbound IPs for connection.north-europe.azure.keboola.com stack.

Please make sure you have whitelisted all these IP addresses in your firewalls to allow Keboola Connection to successfully connect to your system:

• 40.127.144.42
• 20.82.252.94 (new)
• 20.82.252.129 (new)
• 20.82.252.124 (new)

For your convenience, you can process programmatically fetch and process the list of existing IP addresses in JSON format.

Read more about outbound IP addresses in documentation.

On 07 Oct 2021 we experienced failing Google Drive extractor jobs run between 12:22 UTC and 12:49 UTC in all regions. The issue is now resolved. 

04 Oct 2021 07:27 UTC - We are investigating increased backlog of jobs in connection.eu-central-1.keboola.com stack. Next update in 60 minutes or as new information becomes available.

04 Oct 2021 07:45 UTC - We have temporarily boosted the Snowflake capacity and backlog is cleared. All operations are back to normal and everything is fully working.

2021-09-29 08:11 UTC - We are investigating higher error rate and increased latencies in https://connection.north-europe.azure.keboola.com stack. Next update when new information will be available or in hour.

2021-09-29 09:15 UTC - The error rate has dropped, but we are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.

2021-09-29 10:12 UTC - We are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.

2021-09-29 12:00 UTC - We have identified one of 3rd party services to be the possible root cause of this issue and we are working with support team of this service to fix it.

2021-09-29 13:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.

2021-09-29 14:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.

2021-09-29 16:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or  tomorrow morning.

2021-09-29 19:05 UTC - Still working with 3rd party service support on a solution. Job execution times also might be longer due to this incident. Next update when new information will be available or tomorrow morning.

2021-09-29 22:01 UTC - We've applied a fix which reduced the latencies, job processing times should be back to normal. We are  investigating occasional error responses and we are still working with 3rd party service support on a solution. Next update tomorrow morning.

2021-09-30 09:20 UTC - Platform is stabilized after yesterday's fix, however we still see occasional errors. We are still working with 3rd party service support on a solution. Next update when new information will be available or in four hours.

2021-09-30 17:57 UTC - We have implemented another fix which decreased the error rate. We continue to monitor the situation but unless the situation escalates we consider this incident resolved.

A maintenance of Azure EU Keboola Connection (connection.north-europe.azure.keboola.com) will take place on Saturday, Oct 9th, 2021 from 10:00 CET and should take less than two hours.

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

Orchestrations and running component jobs should be generally delayed, but not interrupted.  However, feel free to re-schedule your saturday orchestrations to avoid this maintenance window.

2021-09-21 08:27 UTC -  We have noticed that emails from our platforms are being sent with empty body (email..title, email..body etc.).  Next update in one hour.

2021-09-21 11:14 UTC  - we successfully rolled back the previous working version. 

We're sorry for this inconvenience.

2021-09-13 08:15:10 UTC:  there was a release of the UI which broke the parsing of transformations into separate queries.  Queries that were not edited this morning were not affected.  

2021-09-13 11:48:29 UTC: We identified broken version and applied a fix

We advise all users to reload their browsers and resave any affected transformations

In order to provide a better experience and support we are taking over the maintenance of Salesforce components from the 3rd party developer. Due to this, the original Salesforce (Bulk API) extractor (htns.ex-salesforce component) is being deprecated and replaced with the new Keboola maintained Salesforce extractor.

The old extractor configuration will continue to work, however you won’t be able to create new configurations. It is highly recommended to migrate to the new extractor version. We have recently added an option to perform most of the migration process automatically.

What’s new?

The new extractor comes with improved functionality:

1. Improved dynamic UI

2. Incremental fetching and primary key support

Now you have a better control of how the data is loaded. In the previous version users only had option to switch the `incremental` flag which caused addition of a hardcoded primary key. This caused issues with incremental loading of objects like *History which do not have an Id column at all. Now it is possible to define the primary key manually (defaults to Id) and choose between several options:

• Full Load: Overwrite all data at destination

• Incremental: Upsert data in destination

• Incremental with Incremental fetching: Bring in only data that have changed since previous execution. The date column used for this comparison can be specified. This allows you to use for instance CreatedDate for History objects where the LastModifiedDate is not available.

1. Fetch deleted records

Now it is possible to control whether to include also the deleted records

Migration process

We have released a migration script that will help you migrate any existing configurations directly from the project. 

Migration Behaviour:

• Encrypted values (credentials) are NOT TRANSFERRED - because the component has a new ID users must set up credentials in the new configuration again manually.

• Migration transfers component state so any incremental configurations will start from where they left off. 

• The output bucket is maintained  - there is no need to change any downstream input mappings

• Orchestrations are not updated - this step is left for the user so it can be done safely

Steps to migrate:

1. Each affected configuration will display the following message:

1. Click on the “PROCEED TO MIGRATION”

2. You’ll be displayed a list of affected configurations and orchestrations:

1. Click the “Migrate” button.

2. After successful migration you will find all new configurations in your project

3. Replace the credentials

4. Inspect configurations and test if needed.

Change affected Orchestrations manually and replace the old configuration with the new one.

2021-08-28 10:25 UTC - We have noticed increase rate of failed jobs when importing data into Storage. They are failing with user errors Some columns are missing in the csv file.

2021-08-28 10:50 UTC - We identified the problem and rolled back the previous version of our Storage.

2021-08-28 11:50 UTC - Failed jobs was related with minor Storage changes released on 2021-08-27 between 8-9 AM UTC.

Table imports started after that could finished with User Errors "Some columns are missing in the csv file." when columns names in the source file was ending with non-alphanumeric characters.

On 2021-08-17 since 7:30 UTC we are experiencing some Jupyter based workspaces failing to create or restore in all regions. We are now rolling back previous working version.

2021-08-17 9:35 UTC  - we successfully rolled back the previous working version and creation/restore of Jupyter workspaces works now as expected. As a root cause we identified mishandled process when scaling up server instances, so only creation/restore that triggered instances scale up would fail. We work on the fix and will be released soon after.

On 2021-08-15, between 22:40 and 11:10 CET we experienced increased rate of Storage API errors. We identified the problem and resolved it. As a consequence some jobs may have failed to start. We're very sorry for this inconvenience. 

2021-08-13 07:20 UTC - We are investigating  job processing delays in connection.eu-central-1.keboola.com.  Next update when new information will be available or in hour.

2021-08-13 07:53 UTC - Backlog is cleared and there shouldn't be any delays in jobs processing. We are going to monitor the situation and keep you posted.

2021-08-02 07:40 UTC - We are investigating component job delays in connection.keboola.com.  Next update when new information will be available or in hour.

UPDATE 2021-08-02 08:13 UTC - We have identified the root cause of overload and added more capacity. Backlog is cleared and new jobs are processed immediately. Jobs started before the incident might be still running longer than usually. We are going to monitor the situation and keep you posted.

UPDATE 2021-08-02 10:14 UTC - Incident is resolved. All operations are back to normal. We're sorry for this inconvenience.

We are currently investigating an issue regarding corrupted data obtained via our Telemetry Data component (keboola.ex-telemetry-data).

We have most probably identified the issue and we're working on a fix.

We are very sorry for any inconvenience that this might caused you.

Next update in 15:00 UTC.

Update: We have modified the component so that it now loads data using full loads only. To ensure that you have the correct telemetry data, all you need to do is run the extractor (or wait for your pipeline to run it). We will re-implement the incremental fetching in the following months.

Since 2021-07-20 17:00 UTC some jobs processing may be delayed and job termination requests may be unsuccessful in all Azure stacks. The total number of affected jobs or requests is very small.

This bug was introduced due to a network settings change. The change has been reverted and is currently being deployed to all Azure stacks. If you experience any of the mentioned symptoms please get in touch with our support so we can mitigate the issue faster.

We're very sorry for this inconvenience. 

Since 2021-07-21 07:00 UTC We are experiencing number of jobs in waiting state more than usual. We are going to monitor the situation and keep you posted.

UPDATE 2021-07-21 08:00 UTC We replaced an unhealthy instance with a new one and the issue has been resolved. We're sorry for this inconvenience.

We are experiencing timeouts during communication with Elasticsearch running in the AWS EU Central region. It affects the ability of Connection to create Storage events. 

We are investigating the problem and let you know about its progress in an hour.

UPDATE 2021-07-13 13:30 UTC: We confirmed that the problems are caused by an incident with EC2 instances in AWS (see their status page for more details. We are monitoring the situation. Next update in an hour.

UPDATE 2021-07-13 14:30 UTC: AWS is working on resolving the connectivity issues and our monitoring seems to be calming down too so probably the problems will be fixed soon. Next update in an hour.

UPDATE 2021-07-13 15:30 UTC: AWS resolved the connectivity issues and we are replacing all affected instances preventively to eliminate other potential problems. 

We are experiencing troubles with loading data to Snowflake, probably related to null values in columns set as PK. The issue causes some jobs to fail with the error "NULL result in a non-nullable column" or similar. The issue affects different snowflake components (transformations, writer)

We're investigating the causes for this issue. Next update in one hour.

UPDATE 8:40 UTC: We are still investigating the problem. We managed to reproduce the problem in a test and are performing other steps to find the root cause. Next update in one hour.

UPDATE 9:00 UTC: The root cause of the problem is the fact that Snowflake did not implement NOT NULL constraint on columns set as primary keys in the past but started to enforce it in the latest release. See this announcement for further details: https://docs.snowflake.com/en/release-notes/2021-06-bcr.html#create-table-command-change-to-enforcement-of-primary-keys-created-by-command 

You can fix the problem immediately by removing the null values from columns set as primary keys. We are talking to Snowflake support if it is possible to put back the behavior for selected tables or databases for customers who cannot fix it easily now (as it would e.g. break some incremental loading settings) so please reach us at support@keboola.com if you are in such situation. 

UPDATE 10:00 UTC: We were able to roll back this change with Snowflake support and postpone it for four weeks, giving you time to prepare your data. If you are running your own Snowflake warehouse you will need to disable this change on your own.

UPDATE 11:02 UTC: We've verified in multiple jobs that the issue has been resolved on all our Snowflake accounts.
Unfortunately, there were cases where the issue was not mitigated. The reason is that the change has been released worldwide to all Snowflake clients. So if you're using your own Snowflake account in Keboola (either as backend or in Snowflake writer), you need to rollback the release bundle on your Snowflake account as well.
You need to run the following query on your Snowflake instance

SELECT SYSTEM$DISABLE_BEHAVIOR_CHANGE_BUNDLE('2021_05');

You will need ACCOUNTADMIN role to do that. This will temporarily rollback the problematic behavior. We will either provide a fix on platform level if possible or publish a migration guide to mitigate this once this change will be reenabled in approximately 4 weeks. 

Since 2021-06-30 18:00 UTC we are experiencing increased errors on Snowflake backend in all stacks due an incident in Snowflake. We are going to monitor the situation and keep you posted. 

Update 19:07 UTC
Snowflake identified the root cause and they continue working on mitigating the problem.

Update 20:07 UTC
Snowflake claims that infrastructure is up and running. 

Update 2021-07-02 07:30 UTC
Snowflake has published report about the incident 

Since 02:23 CET 2021-06-25 we're seeing a delay in processing of job events in Azure North Europe stack.

We are monitoring a situation, next update in one hour.

UPDATE 09:15 CET 2021-06-25 We increased the capacity of infrastructure and issue has been resolved. All job events have been processed. We're sorry for this inconvenience.

According to the announcement published two weeks ago we switched default Python environments in Python Transformations and Workspaces. The old Sandboxes will be switched during today. See the previous post for more information.

Today, June 14, 2021, we released a number of new features on our AWS stacks.

Unfortunately we have noticed an issue that when loading larger datasets to the new Python or R workspaces the job will fail.

We're investigating the root cause and will update here when we know more or the issue gets resolved. 

UPDATE 17:59 UTC  - We've deployed fix which should resolve the loading issues. We continue to monitor the situation.

Beginning with 4:05 UTC 2021-06-11 we're seeing slowdown processing of transformation jobs in AWS EU stack. We're working on a fix. Next update in 30 minutes.

UPDATE 7:02 UTC We increased capacity of infrastructure and the situation is better. We continue to monitor the situation.

UPDATE 7:30 UTC  The situation is stable. All operations are back to normal. We're sorry for this inconvenience.

The current environment for transformations is fairly outdated (due to compatibility concerns) and needs an update. New transformation environments are now available with an update plan. You may choose to leave your environment in auto update mode or lock your environment to either the current or the future version.

Action needed:
If you want to prevent the new transformations from being updated automatically later this month, you must select the version you wish to use. This update is planned for June 22, 2021.

The following options are available:

• Leave your setting as is: “1.4.1 - Python 3.8.5 + Pandas 0.25.3 (updates automatically)”. The transformations will update automatically to Python 3.9.5 + Pandas 1.2.4 on June 22, 2021. We recommend choosing this option if you are not relying on Pandas, as it is very likely that your transformations will work without any changes.
• Lock your transformation environment to the current version by selecting “0.3.3 - Python 3.8.5 + Pandas 0.25.3 (locked)”. The update on June 22 will not affect the transformation.
• Lock your transformation to the latest future version: “Python 3.9.5 + Pandas 1.2.4 (locked)”. The update on June 22 will not affect the transformation.
• If you do not want to jump versions, you can choose to lock your transformation to an intermediate version: “Python 3.8.10 + Pandas 1.1.5 (locked)”. The update on June 22 will not affect the transformation.

The Sandbox and Workspace environments will keep running the current version until the switch date on June 22, 2021, at which point they will switch to the latest version (we're jumping the intermediate Python 3.8.10 + Pandas 1.1.5). If you want to continue to use the current (outdated) version of Pandas in the Sandbox/Workspace after this date, you'll have to install it manually.

In the future, we will also be offering at least two versions of the Python transformations: one that updates automatically (the default and current behavior) and one that does not auto-update. That way you can always choose whether your transformation environment is auto-updating or not.

We're seeing increased user error rate in Python transformation jobs since May 25 16:08 UTC. Most common error message is "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed." but also other errors may appear.

We're sorry for this inconvenience, we're actively investigating the issue. Next update in 30 minutes

Update 07:28 UTC: We have rolled back Python transformations from version 1.4.1 to version 1.4.0 and everything seems to be working again. If you encounter further issues please get in touch with us. We'll be monitoring this issue and post an update in one hour. 

Update 08:36 UTC: The situation is stable, we're not seeing any further errors since the rollback. Please accept our apology for the inconvenience. 

From 2021-05-18 21:13 UTC to 2021-05-19 05:06 UTC we experienced increased jobs failures in the Azure North Europe region.

These errors were caused by a faulty deploy on a single node. The deploy has been fixed and the situation is back to normal and you can restart the failed jobs.

We're sorry for this inconvenience.