tag:status.keboola.com,2013:/posts Keboola Status 2018-05-16T17:29:13Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1282404 2018-05-16T17:11:17Z 2018-05-16T17:29:13Z Week in Review -- May 16, 2018

Core

Components

Bugfixes

  • Component configuration state is not updated in case of attached processor failure. e.g in case of AWS S3 extractor and New Files Only option in leaves files as unprocessed in case of processor failure so the files can be processed again until the whole pipeline of processors is executed successfully.
  • MSSQL Writer - fixed support of unicode characters
  • Google Sheets Writer - fixed writing of large tables without performance issues
  • When specifying transformation output mapping, the bucket name is automatically webalized as being typed

Developers

We are happy to introduce the first version of Keboola Storage API Javascript client.


]]>
tag:status.keboola.com,2013:Post/1283926 2018-05-15T21:41:54Z 2018-05-16T06:44:16Z SQL Server Extractor Connection Issues

Between 17:00 May 15, 2018 and 8:00 on May 16, 2018 CET we experienced issues with the SQL Server database extractor. We recommend to review your orchestrations and take appropriate actions if needed.

If you were affected by this, please accept our sincere apologies.

]]>
tag:status.keboola.com,2013:Post/1282357 2018-05-11T07:53:53Z 2018-05-11T07:53:53Z SQL Server writer failures

Between 9 May 2018, 10:53 CEST and 11 May 2018, 09:30 CEST there were job failures for SQL Server Writer configurations that had nullable data types. The issue was created by a new version of the writer so we have rolled it back to the  previous version while we investigate the root cause.

We're sorry for any inconvenience. 

]]>
tag:status.keboola.com,2013:Post/1278617 2018-04-30T15:57:56Z 2018-05-01T08:44:04Z Week in Review -- April 30, 2018

Core

  • Improved generated configuration changes descriptions
  • Added configuration version to jobs results of Docker-based components (it is not yet available for legacy components like transformation and gooddata-writer)
  • Refreshed Manage API docs with working examples
  • Fixed loading of large tables for R-studio and Jupyter sandboxes
  • Fixed random CSV Import upload errors in EU region

Components

  • Improved "show details" experience for input and output mappings
  • Added visibility of columns non-existing in Storage to writers
  • Increased query timeout for all Keboola Provisioned Snowflake writers from 15 seconds to 15 minutes
  • Added support of unconventional column names to MySQL extractor
  • Removed static state from MongoDB extractor

Processors

  • Added support of snappy format to processor-decompress
  • Added processor filter-files
  • Added support for sanitization of invalid utf-8 in processor-iconv

Developers

New Debug API call is available (replaces very rarely used sandbox, dry-run and input-data calls). It creates a snapshot of the data directory used for running the component and stores it in your KBC project. To learn more, feel free to go through the API Docs or through the tutorial. In short the API call:

  • uses the same calling convention as the Run API,
  • filters encrypted values from the data directory,
  • works with all components (previously only those without encryption were supported),
  • works with Processors,
  • works with Configuration Rows,
  • works also with broken components and configurations (even if the run fails, you'll still get a snapshot of the data directory).

Python transformations

Pip version 10 was released recently which removes the pip.main method (more reading). The recommended way to install packages from within python is:

import subprocess
import sys
subprocess.call([sys.executable, '-m', 'pip', 'install', '--disable-pip-version-check', 'PACKAGE_NAME'])

Currently there are 70 transformations using the removed pip methods. If your projects are using them we'll be contacting you with a list of affected transformations. This breaking change introduced in pip is currently blocking us from upgrading python to 3.6.5 where pip 10 is used by default.

]]>
tag:status.keboola.com,2013:Post/1277785 2018-04-28T08:00:41Z 2018-04-28T08:00:41Z Unexpected Job Failures

Between April 28 2:30 and 3:15 UTC there was a high rate of application errors on one of our instances processing component jobs. 

The instance was under heavy load and we're investigating the root cause. Instance is now back to normal and is safe to restart the jobs.

We're sorry for any inconvenience. 

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1273307 2018-04-16T07:40:40Z 2018-04-16T07:40:41Z Degraded performance of Google Sheets Writer

On March 23, 2018 we released a new version of Google Sheets Writer to remove workaround which resized sheet's grid. Unfortunately this version caused a significant performance degradation for tables with larger number of rows.

We decided to revert this version to bring back original performance.

We are working on proper fix and it'll be released soon.

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1272211 2018-04-13T15:00:08Z 2018-04-13T15:00:08Z Orchestration Notification Updates Resulted in Deleted Tasks

There was an update to the orchestrator this week that had an unfortunate side-effect.  If you updated your orchestrations' notifications it would delete the orchestration's tasks.

Thankfully, the orchestrations are versioned, so if this happened to you, we will restore the tasks from the last version.
If you have any concerns about this please contact us at support@keboola.com.  

For what it's worth, updating notifications will no longer delete orchestration tasks, please accept our humble apologies if you were affected.

]]>
tag:status.keboola.com,2013:Post/1270520 2018-04-09T15:32:52Z 2018-04-09T15:32:52Z Week in Review -- April 09, 2018

Updated Components

Google AdWords Reports

  • This extractor is finally enabled also for customers using EU instance

Snowflake Writer

  • Added support of VARIANT data type

Google Drive Extractor/Writer, Google Sheets Writer

  • We added support for Team Drives

Impala Extractor

  • Added support for internal tables

Generic Components

We continue with removing so called "static state" from components. Few weeks ago we removed static state from Transformations, and there was a time for additional components. JSON configurations are also editable straight away. This includes configurations from templates (e.g. Youtube Extractor) and configurations for Custom Science Apps (e.g. Custom Science Python).

Fixes

  • Python/R transformation sandboxes correctly apply filters in input mappings, so input data will be loaded correctly
  • CSV Import uses server side encryption in S3 stage (before uploading to our storage) by default
  • Gmail Extractor supports "message parts" in more sections and there should no longer be messages without parts
  • ThoughtSpot writer correctly handles the "Test Credentials" action

Deprecations

We are deprecating direct import from URL into Storage. Please use the new Http Extractor instead which gives you much more flexibility.

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1270631 2018-04-09T13:43:27Z 2018-04-09T13:44:46Z SQLdep Graph Unavailability

Since Friday April 8, 2018, the SQLdep functionality in Transformations was returning an error. This was fixed today.

We're sorry for this inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1266471 2018-03-29T13:47:57Z 2018-03-29T14:18:19Z Jupyter and RStudio Sandboxes are not starting

3:45pm CEST: We're investigating the issue.

3:55pm CEST: Not only starting, existing sandboxes do not seem to respond.

4:10pm CEST: We're shutting down existing sandbox instances and spinning up new ones. This will take a couple of minutes before the sandboxes will be available again. All existing sandboxes will be unfortunately deleted.

4:20pm CEST: Sandboxes are starting again. All previous sandboxes are deleted. We're sorry for this inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1266213 2018-03-29T06:58:53Z 2018-03-29T06:58:53Z Week in Review -- March 29, 2018

New Components

New Features

  • We have released the Guide Mode, an interactive tutorial for Keboola Connection
  • "Sudo" mode - important changes are protected by requiring password

Updated Components

  • Google Drive Extractor, Google Drive Writer and Google Sheets Writer all support Team Drives
  • Google Sheets Writer preserves formatting when writing into an existing sheet
  • Generic Extractor supports arrays as properties in child jobs.

Minor Improvements

  • Encrypted values are now filtered from component events. This prevents accidental leak of credentials from a component, e.g. when it crashes as prints its stack trace or other internal logs to events
  • keboola.processor-orthogonal is now available to fix malformed CSVs. Handy if you encounter Load error: Line 1 - Extra column(s) found errors
]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1264698 2018-03-28T15:15:11Z 2018-03-28T15:15:11Z New HTTP(S) Extractor

Another one joins the band.

In our effort to replace Restbox with modern components, the next logical step was the HTTP(s) extractor. It allows you to download a single CSV file or a compressed, publicly available file and import them into a single table in Storage. In case you have more public files to download from a single domain, the UI allows you to reuse the same base URL for more files.

The UI of the new extractor supports many features out of the box, but the extractor is not limited by its UI: it's another component that supports processors. So your CSV file can be invalid, in a weird charset, pivoted or mutilated in some other way, and there's tooling to get that fixed.

The list of available processors will be kept and updated in the Developer Portal list of components. A full description of the extractor is available in our documentation.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1263047 2018-03-28T09:59:56Z 2018-03-28T09:59:56Z Introducing Guide Mode

We are happy to announce the immediate availability of Guide Mode. In Guide mode, the Keboola Connection user interface will switch to an interactive tutorial which will guide you through the basics of using Keboola Connection. 

Guide mode is designed for new users and works best on empty projects. Therefore, when you invite a new person to Keboola Connection, they will receive a special link in their invitation email:

The link leads to the try.keboola.com page. By following the link, they will receive a 15day demo project with the Guide mode activated. 

The Guide Mode is the very first step in creating a replacement the old Academy. We are gradually going to fill it with more advanced content, but in the mean time try it out and let us know what you think.


]]>
tag:status.keboola.com,2013:Post/1265698 2018-03-27T10:04:12Z 2018-03-27T10:04:12Z New version of AdWords Extractor

We have just released a new version of AdWords Extractor. It works with AdWords API v201802 (see the Release notes).

The previous version of the extractor is deprecated and you can use our migration tool which will migrate your AWQL queries. However, you have to reauthorize the extractor and give it access to your AdWords data again. The previous version uses AdWords API v201710 which will be switched off on 11 July 2018.

]]>
tag:status.keboola.com,2013:Post/1262840 2018-03-19T15:57:04Z 2018-03-19T15:57:04Z Week in Review -- March 19, 2018

New Components

Asana Extractor

We’re happy to welcome the Asana Extractor to our family. It can extract your projects and tasks from the Asana application which is designed to help teams track their work. This component was developed by Leo Chan.

Thoughtspot Writer

We're likewise delighted to announce a new writer to Thoughtspot that is now available for public use. 
Thoughtspot is a "search and AI-driven analytics platform".

DynamoDB Extractor

We also released a beta version of the DynamoDB extractor. It does not have any UI yet, and has to be configured via JSON. If you are feeling adventurous, please give it a try and let us know how it goes.

Marketing Miner Extractor

Lastly, but in no way least, we have a new extractor for Marketing Miner that allows you to fetch your project rank tracking data from Marketing Miner. 

New Features

  • The project API Tokens section now shows when a token was refreshed: 

Minor Improvements

  • We've modified the storage job polling to reduce component job run times.  The greatest speedups will be observable in small to medium sized data loads.  
  • Artificial limits were removed from CSV file import. Previously the upload had to go through in 10 minutes. Now it's left to the decission of your web browser. Please note that it still holds that large files should be uploaded through the API.

  • Further improvements to Output mapping. The destination bucket is now prefilled from the transformation name.

Fixes

  • The MSSQL extractor was updated to correctly handle databases with case-sensitive collations.

  • The Email Attachments extractor now supports incremental and addresses in angle brackets, ex: `Joe <email@example.com>`

  • Developer portal vendors can now approve requests to join via the request email.



]]>
tag:status.keboola.com,2013:Post/1257963 2018-03-13T09:33:13Z 2018-03-13T09:33:14Z New S3 Extractor

This one took us a while, but we believe it's worth it. We carefully gathered feedback and made the most commonly used features accessible through a new streamlined UI. And there's even more under the hood.

The original AWS S3 extractor was renamed to Simple AWS S3. It stays fully supported and is not being deprecated. There's no need to migrate your configurations.

There are several major differences between the original and the new extractor. The new AWS S3 extractor

  • can download multiple files/tables using a single set of credentials.
  • fully supports incremental loads.
  • is more flexible.

The UI of the new extractor supports many features, but the extractor is not limited by its UI: it is the first component that openly supports processors. Opening the JSON editor (aka Power User Mode) opens up the configuration to endless possibilities. The extractor itself does only a simple job – downloads a set of files from S3. All other jobs (decompression, CSV fixing, setting the manifest file, etc.) are delegated to processors. You can order and configure the processors so that they handle the files as required. You can even develop your own processor in case you're missing something. We're fully aware that this is not an easy concept to grasp, but it's intended for advanced users. Not advanced? Use the UI.

The list of available processors will be kept and updated in the Developer Portal list of components. A full description of the extractor is available in our documentation.

One step closer to replacing legacy Restbox. The HTTP extractor will follow shortly. 

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1255180 2018-03-09T13:32:30Z 2018-03-11T17:30:07Z Week in Review -- March 5, 2018

Improvements

  • Schema in Snowflake Extractor is no longer a required connection parameter. If not set, the table selector allows you select tables from the whole database.
  • Snowflake extractor now supports importing these semi-structured data types: `VARIANT`, `OBJECT`, and `ARRAY` 

  • Updated two factor authentication in Keboola Developer Portal. The SMS authentication is now deprecated. All new users will have to use either Google Authenticator or Duo Mobile app.
  • MySQL extractor now has an option to enable compression of data sent over network
  • Enhanced Output Mapping selector
  • R in Sandbox and Transformations is updated to 3.4.3, also the Tidyverse package is now installed by default.

Bug fixes

  • Data Takeout was randomly failing on backing up your data to S3.
  • Task editor in Orchestrator produced errors when orchestration had configured dozens of tasks.
  • In Twitter extractor template, if a user made a mention of your account, the details of that user account weren't downloaded. Edit and save existing configuration to remedy this issue.
]]>
Miroslav Čillík
tag:status.keboola.com,2013:Post/1257112 2018-03-07T13:14:13Z 2018-03-11T17:30:07Z New Email Attachments Extractor

There’s a new version of Email attachments extractor (previously known as Pigeon extractor) you can use from the Keboola Connection’s Extractors tab. It serves for importing csv files to the Storage by sending them as attachments to a generated email address.

Email address for sending csv attachments is generated automatically and the new extractor has a fresh and simpler UI.

The old version is deprecated and will be discontinued on April 6. Please migrate to the new version in upcoming weeks. There is no automatic migration script because you need to generate new email addresses but the switch should be very easy.

]]>
tag:status.keboola.com,2013:Post/1249492 2018-02-26T21:35:16Z 2018-03-11T17:30:06Z Farewell to Custom Science

Yes, we are going to deprecate the Custom Science application. We introduced it more than two years ago as an alternative to components. Unlike components, it was easy to implement and use. However, we've made a lot of progress in simplifying component development.

The latest additions are a simplified component creation workflow, a component generator tool, and a rewritten developer documentation. See a 10 minute video (or this one for Gitlab) on how to create a Hello World component. All of this means that creating a component is much easier that it was two years ago and is definitely worth the effort. 

At the same moment, Custom Science (CS) is producing more and more problems, specifically:

  • We have no trace of what code was actually executed. That means when something breaks, we don't know if the code was changed in the meantime or not. When something was successful, we don't know for sure which version it was. We can't run a configuration with a previous version of the code.
  • There is a direct dependency on the git repository, and while Github and Bitbucket outages are neither common nor long, they do account for dozens of failed jobs (last year).
  • Risk of loss: If you lose access to the git repository, the jobs immediately fail. There is nothing we can do about it. No grace period. No way back. This can easily happen when people change positions or leave their company.
  • Dependency: Typically, there is only one person which can fix broken CS. If an issue arises, we don't know who the person is and can't contact them. Even if we do know the person, they might not respond. In the meantime, we have no way for a workaround (i.e. reverting to the last working state).
  • Poor security: If the repository is private, we need credentials to it. These should be dedicated robot credentials, but most people use their own. Plus, it's your code repository, so why should you give us credentials to it?
  • Poor performance: CS can easily spend 1-2 minutes on the warm up. If it is installing packages, then it is even more because they are being installed on every run.

We are fully aware that there are some disadvantages of converting every CS into a component. Specifically:

  • It takes several minutes before the updated code is deployed in KBC.
  • The initial setup takes several minutes of your work.

The first issue is not going to change any time soon (we will work on shortening the delay, but there will always be some delay). We tried to minimize the second issue – you can follow our migration guide, or see a 10 minute video of migration (done manually and using our tool) or see the new Component development tutorial.

Overall, CS is great for experimenting. The problem is that we are unable to draw the line between experimenting and production use. And CS in production usually causes countless problems. We are aware that creating components is not ideal for ad hoc stuff, and we're going to improve that too before the final demise of Custom Science which will be October 1, 2018.

]]>
tag:status.keboola.com,2013:Post/1251168 2018-02-22T08:23:55Z 2018-03-11T17:30:06Z Facebook and Instagram extractors failures

Some of the configurations of Facebook and Instagram extractors are failing during import to Storage. 

We are working on a fix and we'll update this status when the issue is resolved.


UPDATE 09:56 AM UTC - The issue was resolved. All Facebook and Instagram extractors configurations should be working again.

]]>
tag:status.keboola.com,2013:Post/1250008 2018-02-19T17:10:29Z 2018-03-11T17:30:07Z Week in Review -- February 19, 2018

New components

Bug fixes and smaller improvements

  • Bug fix in Currency extractor - exchange rates for Danish Krone (DKK) and Icelandic Krona (ISK) were not updated for some time because of a bug in its configuration.
  • Snowflake extractor now offers views in the tables list too.
]]>
Jakub Matějka
tag:status.keboola.com,2013:Post/1243850 2018-02-13T15:38:19Z 2018-03-11T17:30:06Z Time Travel Restore

Snowflake has a wonderful feature that they call Time Travel.  It allows you to replicate your table from its state in the past.  We're happy to announce initial support for this great feature in Keboola Connection. 

To begin with, every project with a Snowflake backend has been set to retain data history for 7days. That means that you can restore a table to how it existed at any point within the last week.  It is possible to increase the data history retention period, so if you're interested in doing that please let us know by using the support button in your project


We've added this restoration method to the snapshots tab in the storage console:


Restoring a table is very simple, just use the calendar to pick the date and time, give the new table a name, and choose which bucket to put it in.


We plan on extending the use of this feature to be able to use time travel replicas as an input option for transformations and to create a "Storage Trash".  

Happy travelling!

]]>
tag:status.keboola.com,2013:Post/1246382 2018-02-12T10:28:18Z 2018-03-11T17:30:06Z Google BigQuery Extractor failures

There were Google BigQuery Extractor  jobs failures between 10th  and 12th February. It was caused by internal platform changes, the issue is resolved now. We are sorry for inconvenience.

]]>
tag:status.keboola.com,2013:Post/1244928 2018-02-12T07:20:59Z 2018-03-11T17:30:06Z Week in Review -- February 12, 2018

New Components

  • Google Trends extractor: this component, developed by Leo Chan (cleojanten@hotmail.com), allows to extract search trends for given keywords in a specified region.

Deprecations 

Indexed columns

With the deprecation/removal of the MySQL backend, we deprecated indexed columns because there is no more use for them. You can search/filter through any column now without the need to mark it as indexed.

The following attributes will be removed from manifest files by the end of March 2018:

  • indexed_columns – with the deprecation of the MySQL backend, there is no need to define indexes.
  • rows_count and data_size_bytes – these values are not (and never were) in sync with the input table data and are useless.
  • attributes – table attributes are replaced by table metadata.
  • is_alias – this is something that has nothing to do with the exported data.

Fixes

  • The Developer portal is now available under a new URL: components.keboola.com (instead of apps.keboola.com). The main reason is that we used the word application in two meanings, and that was confusing. For example, there were applications of type Extractor but also applications of type Application. From now on, everything is a Component. Components are of four types: Extractors (loading data from somewhere), Writers (writing data somewhere), Applications (manipulating data), and Processors (data processing helpers).
]]>
tag:status.keboola.com,2013:Post/1239746 2018-02-01T15:27:10Z 2018-03-11T17:30:06Z Week in Review -- January 30, 2018

Plantyst Extractor

To those who are collecting data from productions machines to Plantyst, you can employ new extractor made by BizzTreat and start doing complex data analysis.

Stories.BI writer

You can automatically push data to Stories.bi and get automatic insights instead of crunching business data by hand.


Updated Components

  • Sklik extractor has new variable accountID
  • YouTube extractor has new version. It is based on Generic Extractor. Old extractor will be deprecated on March 1, 2018
  • Snowflake extractor is now a bit faster and has better error handling
  • Geneea NLP App is now available in EU region
  • BingAds extractor is now available in EU region
  • Facebook extractor with new Page Tokens can newly fetch Page Reviews
  • Twitter extractor is now available in EU region
  • Snowflake and Redshift writers has fixed eventual columns mismatch.


Minor Improvements

  • Quick search in component list was improved - it has better accuracy
  • Component name can be finally submitted by pressing ENTER


]]>
Petr Šimeček
tag:status.keboola.com,2013:Post/1236293 2018-01-25T00:01:03Z 2018-03-11T17:30:06Z Week in Review -- January 22, 2018

Linked/Source Buckets

From now, you'll be able to find source/linked buckets information in Storage section in Keboola Connection. This is very helpful when you need to find out which projects are using (linking) your shared bucket. Also, vice versa, which bucket is the source for your bucket.

MFA required also for Google Login

If you have MFA (Multi-factor Authentication) enabled, a confirmation code is required if you use "Login with Google" functionality. Please contact us if you have problems with login.

Facebok Extractor uses page access token for page/posts insights retrieval

Due to breaking changes recently introduced by Facebok API our facebook extractor is updated so it uses page access token for page or posts insights retrieval instead of user access token. This leads to slower extraction if more pages are included in a query. User access token is still used for other data retrieval such as feed, likes, comments. Moreover, this change does not affect facebook ads extractor. 

Improvements

  • We slightly updated UI for the recently published Tokens page

Fixes

  • Project Power consumption is shown only for the current + 2 previous months. This is only a temporary limitation.  It will soon be solved and return to showing you more data


]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1235407 2018-01-21T09:50:38Z 2018-01-30T12:03:24Z Snowflake Outage in US Region

There was a short Snowflake outage between 10:30 and 10:35 CEST (09:30am and 09:35am UTC) in US region.

  • Sandboxes might have lost their data and worksheets
  • Transformation jobs might have finished with an error
  • Async data loads and exports were unaffected

We're investigating the impact and root cause and will update this post as soon as we know more. Snowflake is now back fully operational.

UPDATE Jan 30 2018: Snowflake released their RCA.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1228326 2018-01-10T13:29:46Z 2018-01-10T13:29:46Z New UI section for API Tokens

We are glad to introduce a new UI for Storage Api Tokens that can be now found under the Users & Settings section. We will be removing the old one found under Storage section. The new UI covers the same functionality as the old one.

As a security measure, the token itself will not be shown anymore, only once after its creation. The only way to see an existing token in the UI is to send it via email (temporary link to token is sent) or refresh it and get a new token string. On the backend, the token can still be seen in the response from the tokens list api call but will be removed in the near future.


]]>
tag:status.keboola.com,2013:Post/1229093 2018-01-10T07:54:54Z 2018-01-11T08:08:24Z Week in Review -- January 9, 2018

You haven't heard from us for a while. We're sorry. Here's what's new.

New Components

Updated Components

Minor Improvements

  • SSL configuration of all database extractors is on the same page with credentials and the SSH tunnel configuration
  • Database extractors give a warning if your connection is invalid
  • Apify Extractor accepts a list of urls from a table in Storage
  • New part about ad-hoc data exploration in Jupyter was added to the tutorial on Ad-hoc Data Analysis

Fixes

  • Database extractors automatically changes table names to lowercase
  • Fixed a bug affecting non-incremental import of sliced tables in the BigQuery and Snowflake extractors 

Blog

Our developers have published 2 blog posts

Community News

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1228744 2018-01-08T22:09:15Z 2018-01-08T22:09:15Z Failed Jobs

Today (8.1.) at 20:05 - 21:56 UTC a number of jobs failed with internal or encryption error. This was caused by a bug affecting OAuth configurations. We have reverted the internal release. We do apologize for this enormous mess up.


]]>