Week in Review -- January 22, 2018

Linked/Source Buckets

From now, you'll be able to find source/linked buckets information in Storage section in Keboola Connection. This is very helpful when you need to find out which projects are using (linking) your shared bucket. Also, vice versa, which bucket is the source for your bucket.

MFA required also for Google Login

If you have MFA (Multi-factor Authentication) enabled, a confirmation code is required if you use "Login with Google" functionality. Please contact us if you have problems with login.

Facebok Extractor uses page access token for page/posts insights retrieval

Due to breaking changes recently introduced by Facebok API our facebook extractor is updated so it uses page access token for page or posts insights retrieval instead of user access token. This leads to slower extraction if more pages are included in a query. User access token is still used for other data retrieval such as feed, likes, comments. Moreover, this change does not affect facebook ads extractor. 

Improvements

  • We slightly updated UI for the recently published Tokens page

Fixes

  • Project Power consumption is shown only for the current + 2 previous months. This is only a temporary limitation.  It will soon be solved and return to showing you more data


Snowflake Outage in US Region

There was a short Snowflake outage between 10:30 and 10:35 CEST (09:30am and 09:35am UTC) in US region.

  • Sandboxes might have lost their data and worksheets
  • Transformation jobs might have finished with an error
  • Async data loads and exports were unaffected

We're investigating the impact and root cause and will update this post as soon as we know more. Snowflake is now back fully operational.

UPDATE Jan 30 2018: Snowflake released their RCA.

New UI section for API Tokens

We are glad to introduce a new UI for Storage Api Tokens that can be now found under the Users & Settings section. We will be removing the old one found under Storage section. The new UI covers the same functionality as the old one.

As a security measure, the token itself will not be shown anymore, only once after its creation. The only way to see an existing token in the UI is to send it via email (temporary link to token is sent) or refresh it and get a new token string. On the backend, the token can still be seen in the response from the tokens list api call but will be removed in the near future.


Week in Review -- January 9, 2018

You haven't heard from us for a while. We're sorry. Here's what's new.

New Components

Updated Components

Minor Improvements

  • SSL configuration of all database extractors is on the same page with credentials and the SSH tunnel configuration
  • Database extractors give a warning if your connection is invalid
  • Apify Extractor accepts a list of urls from a table in Storage
  • New part about ad-hoc data exploration in Jupyter was added to the tutorial on Ad-hoc Data Analysis

Fixes

  • Database extractors automatically changes table names to lowercase
  • Fixed a bug affecting non-incremental import of sliced tables in the BigQuery and Snowflake extractors 

Blog

Our developers have published 2 blog posts

Community News

Failed Jobs

Today (8.1.) at 20:05 - 21:56 UTC a number of jobs failed with internal or encryption error. This was caused by a bug affecting OAuth configurations. We have reverted the internal release. We do apologize for this enormous mess up.


Year 2017 in Review

Since most of us are enjoying the winter holiday, the usual Week in Review would be half-empty. It's perhaps time to review the year with the benefit of hindsight. A lot of stuff happened last year which is easily lost in the daily commotion, so let's have a little recap.

The big things

  • Keboola Connection in EU region — Although this might not look exactly revolutionary, it required immense amount of work to make it so our backend is not tied to a specific region any more. This of course opens doors to other regions, which are going to be much easier to do.
  • Developer Portal — This allows anyone to create and deploy applications in Keboola Connection. Hosting of Docker images, automated deployment and testing integration — all inclusive.
  • Big files support and other performance improvements — We all have big data, don't we? There were a number of limitations in several places which didn't allow processing of files larger than 2GB or 5GB. Though there are new limits now (around 100GB), it's much harder to hit them. A lot of other performance improvements were achieved, among those the recent up to 80% speedup in workspace load is worth bragging about.
  • Component Dockerization — This is a ton of completely invisible work which has no immediate impact on you. Once we are done with this, we'll be able to rework job processing stuff and offer super flexible projects (ever wanted long running jobs? or jobs requiring XXX GBs of RAM?). There is still a lot to do, but we're not dawdling.
  • Shared buckets — Although they are not perfect, they hugely simplify sharing things in larger organizations.
  • Processors — Quite a hidden nerdy feature which is slowly making it into production. This, together with Developer portal opens the door for simplified implementations (especially in extractors) of those "Oh, it's perfect, I just need to convert this one little thing ...".
  • Trash of deleted configurations — A lifesaver for many.
  • RStudio and Jupyter Sandboxes — Do they count as a big thing too?
  • New database extractors — The database extractors were greatly simplified.  You no longer need to remember your database schema, and can make your configuration in just a couple of clicks.
  • UI/UX improvements — No shocking things happened (yet?).  but we certainly put much more focus on this area throughout the year. There are a various new features (markdown descriptions, merging table slices, finer input mapping granularity). But we are also putting a lot of effort in unifying the look and feel of different Keboola Connection parts and smoothing the flow. Hopefully this makes Keboola Connection more pleasurable to work with.

Security improvements

We're keeping an eye over security all the time, and constantly improving. To name a few things:

  • Project Access Approval
  • New Google Sign-in
  • CSP (Content Security Policy) used in entire Keboola Connection
  • Better secured Keboola Connection cookies and sessions
  • Display of all active account sessions

New components

There were literally a ton of new and updated components. Ok, maybe not a ton, but definitely a lot. To name a few:

  • Facebook Extractor, Facebook Ads Extractor, Azor Extractor, Snowflake DB Extractor, BigQuery Extractor, Papertrail Extractor, Dropbox extractor, Pipedrive extractor, CJAffiliate extractor, Dark Sky Extractor
  • GeoIP application, What3words Application, Data Health Application
  • SAS Writer, Google BigQuery Writer, GoogleSheets Writer, Snowflake Database Writer, Qlik writer, Looker writer, Salesforce Analytics Cloud Writer
  • Dozens of other connections thanks to Generic Extractor.

What's next?

  • We are working on replacing the Storage Console as the most ancient part of Keboola Connection UI. This is big, so we'll replace it in parts, but it's already in the works.
  • New GoodData Writer is being made. We want to get rid of its "special" behavior and make it a standard writer. Some of the not-exactly-writery features will go into separate tools.
  • New Pigeon "extractor" is in progress. Again, this is an old component with quirky behavior so it has to be replaced with something reliable.
  • Also a new S3 extractor is waiting behind the door. This will be in many ways a conceptually different extractor taking advantage of processors and having configurations organized in a different way.
  • We're working on making it easier to learn Keboola Connection. It's not quite finished yet, but we definitely want to make life easier for new users.
  • Removing MySQL backend. It's been there a long time and it's sad to see it go. Actually, no it isn't. We'll throw a party when it's gone for good.
  • Database writers are going to get a larger update.

And beside that?

Life is a bitch, so we can't make promises, but:

  • Better managed wish list is something both you and we wish for.
  • RStudio and Jupyter Sandbox have many improvements in queue.
  • We have quite a few (almost revolutionary) ideas about Input mapping.
  • Shared credentials (need three configurations connecting to the same database server?) would make a lot of situations easier to handle.
  • Transformations are going to get simplified (no phases, no dependencies, just transformations).
  • We'd like to support even bigger data (no big is too big), so thanks everyone for pushing the limits.
  • Orchestrator needs quite a few updates and so does Generic Extractor

We hope we meet those and many other goals but that's all for now. 

We wish you all a very happy and successful New Year.

Slower job processing

We're experiencing slower Docker components jobs processing, many jobs stalled in waiting state. Finding the root cause, hopefully we'll be back online soon.

UPDATE 9:40 AM CET: All operations are back to normal, the stalled jobs were caused by a misbehaving Redshift cluster. We're going to investigate the root cause. 

We're very sorry for this inconvenience.


Week in Review -- November 30, 2017

Database extractors

After recent announcement of new database extractors we are bringing other improvements to these extractors.

  • By default the new bucket is created for each extractor configuration. Previously extractor configurations were sharing one bucket which led to collisions.
  • You can reload the list of tables fetched from database. It is useful when you are tuning credentials permissions or you have just added some tables to the database. 
  • Primary key is now validated against table created in storage. Warning is thrown if the configured key is different than the key defined on the table.

Generic extractor

The new ignoreErrors option was introduced. This option allows you to force Generic Extractor to ignore certain extraction errors.  Read more in documentation.

Fixes

  • Google Drive Writer in append mode was overwriting rows instead of appending new rows under some circumstances
  • Google Big Query extractor was ignoring files exported to multiple slices
  • Tableau Writer is now available in the EU region


New version of AdWords Extractor

We have just released a new version of AdWords Extractor. It works with AdWords API v201710 (see the Release notes) and you no longer need your own developer token to use it.

The previous version of the extractor is deprecated and you can use our migration tool which will migrate your AWQL queries. However, you have to reauthorize the extractor and give it access to your AdWords data again. The previous version uses AdWords API v201705 which will be switched off on 28 March 2018.

Week in Review -- November 21, 2017

New Features

  • There is a new extractor for what3words service (https://what3words.com/)

Improvements

  • Configuration of MySQL Extractor allows parameter database to be optional
  • Loading data to Storage Workspace allows renaming of columns
  • Configuration errors from DarkSky Extractor are properly marked as user errors
  • Unified incremental load settings for all database writers