Increased "Obsolete output mapping" transformation user errors

Since October 10, 3:00 am UTC we are experiencing higher rate of "Obsolete output mapping" user errors in transformation jobs. This issue is caused by a recent update of the transformation component. We'll be rolling back the release as soon as possible and keep you updated on further events.

UPDATE 8:45 am UTC: We have successfully deployed a previous version and all affected configurations should be working now. 

UPDATE October 13, 6:11 PM UTC: A new version with a fix has been released and deployed.

We apologize for the inconvenience.

Client Webinar (Q4/2019)

Keboola is hosting our Q4/2019 client webinar and we want you there!

Date: Tue, Oct 8, 2019
Time: 6:00 PM - 7:00 PM CET

If you haven't done so already, please register now.  

Learn about new features, functionalities, and components that we have built over the first half of the year.  

Additionally, see the exciting plans that we have for the product for the next 3-12 months.

We hope that you can join us, but if not the webinar will be available on video by request afterward as well.


Your Keboola Team

End of life: old OAuth Broker

The deprecated OAuth Broker won't be supported after September 30, 2019

When executing configurations in not-yet-migrated projects, you will start getting errors with the end of life message. The migration will be available for at least one week after the deprecated Broker is shut down.

Please migrate your projects if you haven't done so already. If you have any questions, contact Keboola Support.

End of life: GoodData Writer v2

The support for the deprecated GoodData Writer v2  ends today, September 30, 2019.

The deprecated writer will start returning errors with the end of life message. You will be able to migrate for at least one week after the deprecated GoodData Writer is shut down.

If you are still using the deprecated GoodData Writer, you should see a deprecation notice in Keboola Connection. Please proceed to the migration.

If you need any help migrating the writer, contact Keboola Support.

Email notification failures in EU region

On Saturday night, September 28-29th between 11:00pm - 8:20am we experienced failures of sending automated emails in EU region. Our monitoring shows only few jobs were affected, we recommend to double-check status of your recent orchestrations run. We apologize for the inconvenience.

GoodData writer internal errors

A new release of the GoodData writer published on September 24, 2019, 3:17 pm UTC contained a bug affecting certain jobs in all regions. The bug caused the jobs to finish with an internal error.

We rolled it back to a previous version on September 25, 5:57 am UTC and all jobs are working now. We apologize for the inconvenience.

Jupyter sandbox not starting

We have a problem with starting Jupyter sandboxes started at 12:58 UTC. 

We will provide an update when we'll have more information.

UPDATE 13:15 UTC - Problem is fixed. All services are running.

Week in Review — September 23, 2019

New Features

Julia sandboxes and transformations

We are happy to introduce a new Julia sandbox and Julia transformations. Both are in Beta preview for now; to try either of them, write a ticket to the support, and we will enable them for you. You can find more information about the Julia transformations in our documentation.

User roles

User roles are now available in all projects. We have added a new guest role that has only limited access to projects' resources. Learn more about roles in our documentation.

New Components

Brightloom Extractor (previously known as Eatsa) — downloads all your transactions and item details from the Brightloom POS system.

Job errors in the EU region [post-mortem]


On Sunday, September 15th at 01:32 UTC, orchestrator and other component jobs started failing in the EU region. In the following hours, our worker servers weren't able to handle the workload, and the job backlog started to increase. We manually resolved the incident, and the platform was in full operation with a clean backlog at 08:26 UTC.

What happened?

One of the MySQL instances was automatically restarted and patched on September 15th at 01:32 UTC.
The instance is required for the lock mechanism for job processing, and it also stores information about queues for the worker servers. The 2-minute downtime of the database instance caused a failure of the jobs that were running at the moment. Additionally, the running workers weren't able to fetch the information about the queues, and some of them gave up restarts and stopped. With only half of the processing capacity left, the workload could not be processed.

Once we discovered the incident, we replaced all our worker servers and added more capacity to clean up the backlog faster.

What are we doing about this?

We have implemented notifications about upcoming instance patches and are going to perform updates during scheduled and announced maintenance windows.

We are also working on a completely new job processing and scheduling mechanism that will prevent similar issues from occurring down the road. We sincerely apologize for the inconvenience caused.