Facebook Ads Extractor Failed Jobs

Between 1:00 - 8:20 CET January 4th, 2019 some Facebook extractor jobs failed after yesterday update of the extractor. We rolled back to the previous working version and continue investigating the issue. Please revise your facebook ads extractor latest jobs and restart if needed. 

We are sorry for this inconvenience.

Update 12:50 CET

The issue has been resolved. The problem was in configurations that used own its authorization token, i.e., not authorized under Keboola Facebook App.

Weeks in review -- January 2, 2019

New components and component updates

  • New FTP extractor is now in beta. It is packed with the same features as AWS S3 extractor and it is extensible with processors. We aim to gradually replace the current FTP extractor which is developed by 3rd party.
  • New fit-into-storage processor that allows to import non-csv files (TXT, JSON) into storage by wrapping them into CSV tables.
  • AWS S3 Extractor fixes bug with file matching. When the Subfolders option was off and Wildcard option was on, the extractor would erroneously download files contained in subfolders exactly matching the key.
  • MongoDB extractor export mode is now set to "raw" by default on new configurations.

Enhancements

Input and output mapping for components is now parallelized. Configurations with large number of tables in input and output mapping (except transformations) should now run considerably faster. This affects both extractors and writers. Most recently updated extractors (e.g. AWS S3, FTP, HTTP, MySQL, MSSQL, Storage) also load tables into storage during the extraction of other tables. Together, these make up to 40% run time reduction for some configurations.

Input mapping load type is now shown in input mapping details:

The Run Orchestration dialog now has the option to select/deselect all tasks:

Developers

The option to have gzipped CSV file on component output mapping (/data/out/tables/) was removed. Only plain CSV files are accepted now. To our best knowledge this was never used.


Failed Jobs

Between 23:16 - 23:22 UTC December 14th, 2018 some jobs failed with Application error due to spikes in infrastructure load. 

We are investigating the root cause and taking measures so that this does not repeat. 

We deeply apologize for the inconvenience caused.

Update [Dec 15, 2018, 08:19 CET]: This still happens occasionally.

Update [Dec 15, 2018, 10:57 CET]: We're still working on mitigating the issue. Comparing to historical hourly avg, actual job error rate is 6% bigger.

Update [Dec 15, 2018, 11:47 CET]: We're going to deploy patch into production. It will take about 2 hours. Expected resolution: 14:00 CET

Update [Dec 15, 2018, 12:45 CET]: Patch has been deployed.

We are experiencing momentary technical difficulties in US

As of around 23:30 CET Dec 13, 2018 we observed technical issues related to out job servers, we are continuing to investigate and will update as soon as we know more.

  • Update: [Dec 13, 2018 23:55 CET] - We're hot on the trail and expect to have the issue resolved shortly.
  • Update: [Dec 14, 2018 02:43 CET] - Service is now fully restored in US region

We're very sorry for any inconvenience.

To make recovery faster we boosted Snowflake performance for next 24 hours.

And you'll hear from us with "post mortem" soon.

28 November, 2018 -- Storage Job Failures in EU

During deployment of Keboola Connection in the EU region at around 13:30 CET today there was a timing issue that resulted in some failed storage jobs.

The problem occurred because of the asynchronous timing between the rollover of a worker and the API server.  A small number of jobs that were created with the previous version then failed when executed against the new API.  These jobs failed with one of the following error messages "`Invalid source type" or "Workspace not found".

There was no corruption of data by these failures as no jobs that were in progress were affected.

 

28 November, 2018 -- Stalled Jobs in the EU region

Around 5:00am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

At 9:30am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

We're sorry for this inconvenience.

Week in review -- November 27, 2018

New Features

You can now provide your project with a markdown description on your project's Overview page

Updated Components

SQL Server Extractor now supports incremental fetching on DateTime and Identity columns

Unexpected Events

Many of you have noticed that we added validation of SQL queries to the Transformation Detail page last week.

Due to some problems in availability of this feature we decided to revert this change.

For now you can still use the "standard" validation button on the right.

We love this feature too and the plan is to bring it back after an additional round of improvements.


Storage Jobs Errors - US region

We're experiencing a high number of errors in Storage Jobs in the US region. The errors are caused by invalid SSL connection to Snowflake and we're investigating the issue.

Update 01:02 am CET: Snowflake issues an official statement.

Update 07:00 am CET: Snowflake updated an issue. We are working on a workaround.

Update 07:56 am CET: Issue is resolved. Snowflake moved the Keboola account to a new certificate provider. Now we are processing backlog of jobs.

Update 09:20 am CET: Backlog is cleared. All systems are operating normally.

Week in Review -- November 14, 2018

Revised Project Invitations

Inviting users to KBC projects has been reworked. Previously, users were directly added to the project via Users & Settings -> Users menu. Now, when adding a new user to the project, an invitation email is sent to the user.

After clicking on the link in the email, the user is redirected to the login screen, where pending invitations to projects are displayed.

The invited user can accept or decline the pending invitations in the Account Settings.

See the docs for more details.

New Features

  • We have added some predefined searches to the job list. See the docs for a more detailed description of the search queries.


Minor Improvements

  • Jobs related to Python/R sandboxes now show the proper link to a configuration:

  • Improved description of components in events

  • Configurations in trash are now sorted by the date of deletion:

  • Component configuration state is reset on copy:
If you're making a copy of a component configuration (e.g. Zendesk Extractor), the state of the configuration is cleared for the new copy. For example, Zendesk Extractor downloads all tickets from Zendesk on the first run and stores the date and time into its state. On the subsequent runs the Extractor only downloads the tickets created after that time. Now, the new copy of this configuration will also download all tickets on its first run as there is nothing in its state.