Short Quay.io outage

On April 11 between 4:15pm and 4:31pm CEST our connection to quay.io was down. All Docker images stored there were inaccessible. That caused failures in running some components. If you encountered an application error during that period, this outage was likely the culprit. 

Unfortunately status.quay.io remains silent about this outage. 

We are sorry for this inconvenience and to mitigate similar issues in the future, we will move all used Docker images from public repositories (DockerHub and Quay.io) to our private repository in AWS ECR.

Updated UI for S3 Extractor and CSV Import

We have updated the UI for the S3 Extractor and the CSV Import tool. We're keen on trying new things to improve your user experience with Keboola Connection and hope that these improvements offer a simpler and easier approach to editing your component configurations.

There is no difference between editing and non-editing state, everything is directly editable and the state of the Save button indicates if the configuration has been changed or not. 

If this approach proves useful and stands the test of time, we'll carry it on to other components as well.

Let us know what you think!

Week in Review -- March 13, 2017

New Components

Snowflake Extractor
  • Snowflake Extractor is now in private beta. For an early access please contact us at support@keboola.com.

Minor Improvements

  • Next Event application now estimates the date of the next event

Fixes

  • Fixed all dead links on status.keboola.com and help.keboola.com

  • Google Sign In correctly redirects after authorization

  • Keboola Connection cookies use the secure flag

  • Password reset does not leak information about (non)existing user

  • Orchestrator trims notification email addresses, no more whitespace mess

  • GoodData Writer properly propagates errors when creating a new writer

  • GoodData Writer does not remove our domain admin from a project which is shared among more writers


AWS S3 Outage

Keboola Connection is suffering very hard from the ongoing AWS S3 outage. All workers are down, lots of functionality is broken. For more details please see https://status.aws.amazon.com/

Once the AWS team fixes the issue, Keboola Connection will be up and working again. 

We're sorry for this inconvenience and thanks a lot for your understanding and patience.

Failed jobs

On Sat, Feb 25, 04:05 AM UTC+1 one of our AWS RDS server was restarted. This lead to loss of connection between our lock server and job processes. The outcome may be following

  • Orchestrations, that have all tasks successfully finished, but the orchestration itself failed
  • Failed tasks within orchestrator; the task execution finished successfully, only the job status was not saved correctly
  • Job was scheduled to execute again while it was still running, and especially in transformation jobs this lead to simultaneous execution in the same database/workspace

We will resume all orchestrations, that have failed tasks due to this outage (starting at the task following the failure).

We're sorry for this inconvenience, we're working towards mitigating this bug in the future.

Failing Snowflake Writer

Between Feb 02 22:40 UTC and Feb 03 17:25 UTC all Snowflake Writer jobs ended with the following error:

SQL compilation error: conflicting values file format parameter 'ESCAPE_UNENCLOSED_FIELD', SQL state 42601 in SQLPrepare.

We have identified the root cause of this issue and fixed it. All Snowflake writers should be fully operational now.

Please accept our sincere apologies for this inconvenience.

Week In Review -- January 9, 2017

Happy New Year everyone! Some good news here, some long awaited features finally made it.

  • Shared buckets allow you to share data between projects. Making RESTbox deprecated one feature at a time.
  • We decided to scrap all Project Power stats older than Jan 5th. We hope this is the last adjustment.
  • Bugfixes in GetStat and Qualtrics extractors.
  • Redshift bloating tables popped up again. We were not able to detect certain cases. Fixed.

Also the UI got some love

  • Link to documentation in the left pane, so you don't have to guess it
  • Long Run Ids wrap correctly in the job detail, if you're into deep job nesting
  • Google Drive Extractor does not delete the input for custom header names every 5 seconds

What's coming up next?

  • New Facebook and Salesforce extractors are getting really close.
  • Developer Portal is almost ready. There is no UI so far, we're API first.
  • Keboola Connection in AWS EU region.

Generic Extractor Failures

We're encountering a series of "Found orphaned table manifest" errors in Generic Extractor. We have identified the root cause and reverting last changes to get it back to fully working state. We will restart all affected orchestrations.

We'll update this post when the fix will be deployed.

We're sorry for this inconvenience.

UPDATE 7:25pm CEST: The fix has been deployed to production, we're restarting all failed orchestrations.

MySQL Transformation Job Failures

During last 24 hours some MySQL transformation jobs failed with the following error message:

SQLSTATE[HY093]: Invalid parameter number: no parameters were bound

This was caused by an update in the application and it only affects transformations, that contain empty queries (eg. just ";" character). The fix will be deployed shortly and we'll restart all affected orchestrations. 

We're sorry for this inconvenience.