Keboola Status

On Jun 2, 2016 3:40pm UTC+2 a new version of Storage API was released containing the following bug.

Incremental loads into Redshift tables with primary keys do not correctly deduplicate data - rows with duplicate primary key may exist in the table.

We'll be deploying the original version shortly and then we'll dedup all affected tables.

We're sorry for this inconvenience, we'll keep you updated in this post. 

UPDATE Jun 3, 2016, 2:30pm UTC+2

Original version deployed. We're starting recovery process, no service outage will be required.

UPDATE Jun 4, 2016, 9:15am UTC+2

Recovery process for all affected tables is finished, all duplicate records should be mitigated. We're now investigating the root cause of the issue to prevent similar incidents in the future.

We're experiencing errors in our legacy OAuth component. Affected components are

• Dropbox Writer
• TDE Writer (using Dropbox Writer)

Some configurations may not run. We're sorry for this inconvenience and investigating this issue. 

UPDATE 10pm UTC+2: The issue was resolved, everything should be running smoothly again. Please let us know if you still have errors.

Around 9pm UTC+2 we encountered 2 system wide issues.

1) One of our API servers ran out of disk space. Requests running on this server might have finished with an error. This influenced UI, orchestrations, listing jobs or worker jobs using our APIs.

2) AWS encountered increased API error rates. This might have influenced all components of Keboola Connection, from UI to orchestrations.

Both issues are now resolved and all operations are resumed. We're sorry for any inconvenience and thank you for your patience. We're currently going through failed orchestrations and restarting them. 

One of our metadata servers was restarted by AWS at 1:55pm UTC+2. 

This may have caused some jobs end with an application error. 

We're sorry for this inconvenience and we'll restart all affected orchestrations. 

We've encountered an issue running newly created transformations. Please stay with us, the fix currently being tested and will be deployed within the next hour. 

The symptom is a failing job with error Transformation 'x' not found. Only newly created transformations are affected by this bug, all old transformations should work.

We're sorry for this inconvenience!

EDIT 4.00 pm CEST: The bugfix has been deployed to production. 

As of now we're removing type input mapping attribute in Redshift transformations. 

All tables which were set to be created using CREATE VIEW from another schema will now be automatically created using CREATE TABLE and the data will be copied.

There are many reasons behind this update, the most important being a deadlock when deleting tables in Storage that had a view created in a transformation or in an orphaned sandbox.

You may experience slightly slower transformation times and heavier disk usage caused by physically copying data.

Should you experience any unexpected behaviour or a major slowdown, please let us know.

PS: No action required, we're glad you made it here safely.

Between 11:08am CEST and 2:03pm CEST under certain circumstances transformation jobs were forking from existing processing transformation jobs. We have found the root cause, fixed it, terminated all forks and restarted all failed orchestrations. If you're still seeing this issue, please let us know.

UPDATE: This issue occurred again between 4:49pm CEST and 8:05pm CEST.

We're sincerely sorry for this inconvenience.

On Friday April 8th we'll be switching our Facebook Ads extractor to Facebook Marketing API from v2.4 to v2.5. If your configuration is using a fixed API version (api_version bucket attribute), please update it manually.

Please see the Graph API changelog to verify your configuration won't break, here's a shortlist:

Marketing API Changes from v2.4 to v2.5

• Consistent naming for three-level campaign structure between API and UI. This changes naming at endpoint, params, fields, and enum level. For the full list of changes, refer here.

• Change /adcampaign_groups to /campaigns
• Change /adcampaigns to /adsets
• Change /adgroups to /ads
• In the write path, change campaign_group_status, campaign_status, adgroup_status to status

• Campaign

• Change naming of objectives
• WEBSITE_CLICKS to LINK_CLICKS
• WEBSITE_CONVERSIONS to CONVERSIONS

• Targeting

• Response for call to /search?type=adgeolocation will contain only the 'City' instead of 'City, State'
• Change {cpc, cpm, cpa}_{min, max, median}, such as cpc_min fields in /reachestimate to bid_amount_min, bid_amount_median, bid_amount_max. Change bid_for to optimize_for

Should you encounter any issues after the update, please let us know at support@keboola.com. 

In case you need a bit more time to migrate your configuration, use bucket attribute api_version and set it to v2.4. This will give you time until Facebook shuts down this version completely (Monday April 11th).

We're introducing a simple versions management for transformation buckets. You can quickly access some functions via a dropdown on transformation bucket page.

This dropdown shows only last 5 versions and allows to quickly rollback any of the previous versions or to copy the current version to a new bucket. All versions are available on a separate page via Show all versions menu option.

There you can see all versions, do a fulltext search within their descriptions, dates or authors, rollback to any version in history or copy any version to a new bucket.

Versions always work with the whole bucket, you cannot rollback or copy a single transformation. 

Please consider this as a first version - we're too excited to present you what we have so far we didn't want to wait to have everything ready. There are plenty of enhancements to come:

• better change descriptions - we'll be fixing the messages to be more accurate and descriptive
• versions for all components and their configurations
• access to versions from transformations and not only buckets
• diffing
• dev workspace and committing changes
• ability to set your own change description

Please let us know your thoughts, happy versioning! And of course, if you experience any unexpected behavior or find a bug, reach out to us at support@keboola.com.

• Truncating Redshift tables now correctly updates bucket stats

• Apache Impala extractor
  

  
• Bing Ads extractor, a 3rd party extractor provided by David Ešner
  

  
• S3 writer allows to specify the format of timestamp suffix and optionally compresses the files