Since October 15, 2019 5:16PM UTC the Docker Hub container registry is experiencing a complete service disruption. Some of Keboola Connection 3rd party components' images are historically hosted on Docker Hub. Such components' jobs will fail during the outage. You can subscribe to the outage mitigation progress on Docker Hub status page.
All the new components and Keboola-maintained components are using AWS ECR as a registry, so they are not affected by the outage.
We're very sorry about any problems this may cause.
There will be a maintenance period of Developer Portal on Saturday, October 12, 2019 from 6:00am UTC and should take around 1 hour.
We will be upgrading database infrastructure. This maintenance won't affect Connection nor any other Keboola service.
Keboola is hosting our Q4/2019 client webinar and we want you there!
Date: Tue, Oct 8, 2019
Time: 6:00 PM - 7:00 PM CET
Date: Tue, Oct 8, 2019
Time: 6:00 PM - 7:00 PM CET
If you haven't done so already, please register now.
Learn about new features, functionalities, and components that we have built over the first half of the year.
Additionally, see the exciting plans that we have for the product for the next 3-12 months.
Learn about new features, functionalities, and components that we have built over the first half of the year.
Additionally, see the exciting plans that we have for the product for the next 3-12 months.
We hope that you can join us, but if not the webinar will be available on video by request afterward as well.
Cheers,
Your Keboola Team
The deprecated OAuth Broker won't be supported after September 30, 2019.
When executing configurations in not-yet-migrated projects, you will start getting errors with the end of life message. The migration will be available for at least one week after the deprecated Broker is shut down.
Please migrate your projects if you haven't done so already. If you have any questions, contact Keboola Support.
The support for the deprecated GoodData Writer v2 ends today, September 30, 2019.
The deprecated writer will start returning errors with the end of life message. You will be able to migrate for at least one week after the deprecated GoodData Writer is shut down.
If you are still using the deprecated GoodData Writer, you should see a deprecation notice in Keboola Connection. Please proceed to the migration.
If you need any help migrating the writer, contact Keboola Support.
On Saturday night, September 28-29th between 11:00pm - 8:20am we experienced failures of sending automated emails in EU region. Our monitoring shows only few jobs were affected, we recommend to double-check status of your recent orchestrations run. We apologize for the inconvenience.
A new release of the GoodData writer published on September 24, 2019, 3:17 pm UTC contained a bug affecting certain jobs in all regions. The bug caused the jobs to finish with an internal error.
We rolled it back to a previous version on September 25, 5:57 am UTC and all jobs are working now. We apologize for the inconvenience.
We have a problem with starting Jupyter sandboxes started at 12:58 UTC.
We will provide an update when we'll have more information.
UPDATE 13:15 UTC - Problem is fixed. All services are running.
New Features
Julia sandboxes and transformations
We are happy to introduce a new Julia sandbox and Julia transformations. Both are in Beta preview for now; to try either of them, write a ticket to the support, and we will enable them for you. You can find more information about the Julia transformations in our documentation.
User roles
User roles are now available in all projects. We have added a new guest role that has only limited access to projects' resources. Learn more about roles in our documentation.
New Components
Brightloom Extractor (previously known as Eatsa) — downloads all your transactions and item details from the Brightloom POS system.
Summary
On Sunday, September 15th at 01:32 UTC, orchestrator and other component jobs started failing in the EU region. In the following hours, our worker servers weren't able to handle the workload, and the job backlog started to increase. We manually resolved the incident, and the platform was in full operation with a clean backlog at 08:26 UTC.
What happened?
One of the MySQL instances was automatically restarted and patched on September 15th at 01:32 UTC.The instance is required for the lock mechanism for job processing, and it also stores information about queues for the worker servers. The 2-minute downtime of the database instance caused a failure of the jobs that were running at the moment. Additionally, the running workers weren't able to fetch the information about the queues, and some of them gave up restarts and stopped. With only half of the processing capacity left, the workload could not be processed.
Once we discovered the incident, we replaced all our worker servers and added more capacity to clean up the backlog faster.
What are we doing about this?
We have implemented notifications about upcoming instance patches and are going to perform updates during scheduled and announced maintenance windows.We are also working on a completely new job processing and scheduling mechanism that will prevent similar issues from occurring down the road. We sincerely apologize for the inconvenience caused.