Today (8.1.) at 20:05 - 21:56 UTC a number of jobs failed with internal or encryption error. This was caused by a bug affecting OAuth configurations. We have reverted the internal release. We do apologize for this enormous mess up.

Since most of us are enjoying the winter holiday, the usual Week in Review would be half-empty. It's perhaps time to review the year with the benefit of hindsight. A lot of stuff happened last year which is easily lost in the daily commotion, so let's have a little recap.

The big things

• Keboola Connection in EU region — Although this might not look exactly revolutionary, it required immense amount of work to make it so our backend is not tied to a specific region any more. This of course opens doors to other regions, which are going to be much easier to do.
• Developer Portal — This allows anyone to create and deploy applications in Keboola Connection. Hosting of Docker images, automated deployment and testing integration — all inclusive.
• Big files support and other performance improvements — We all have big data, don't we? There were a number of limitations in several places which didn't allow processing of files larger than 2GB or 5GB. Though there are new limits now (around 100GB), it's much harder to hit them. A lot of other performance improvements were achieved, among those the recent up to 80% speedup in workspace load is worth bragging about.
• Component Dockerization — This is a ton of completely invisible work which has no immediate impact on you. Once we are done with this, we'll be able to rework job processing stuff and offer super flexible projects (ever wanted long running jobs? or jobs requiring XXX GBs of RAM?). There is still a lot to do, but we're not dawdling.
• Shared buckets — Although they are not perfect, they hugely simplify sharing things in larger organizations.
• Processors — Quite a hidden nerdy feature which is slowly making it into production. This, together with Developer portal opens the door for simplified implementations (especially in extractors) of those "Oh, it's perfect, I just need to convert this one little thing ...".
• Trash of deleted configurations — A lifesaver for many.
• RStudio and Jupyter Sandboxes — Do they count as a big thing too?
• New database extractors — The database extractors were greatly simplified.  You no longer need to remember your database schema, and can make your configuration in just a couple of clicks.
• UI/UX improvements — No shocking things happened (yet?).  but we certainly put much more focus on this area throughout the year. There are a various new features (markdown descriptions, merging table slices, finer input mapping granularity). But we are also putting a lot of effort in unifying the look and feel of different Keboola Connection parts and smoothing the flow. Hopefully this makes Keboola Connection more pleasurable to work with.

Security improvements

We're keeping an eye over security all the time, and constantly improving. To name a few things:

• Project Access Approval
• New Google Sign-in
• CSP (Content Security Policy) used in entire Keboola Connection
• Better secured Keboola Connection cookies and sessions
  
• Display of all active account sessions
  

New components

There were literally a ton of new and updated components. Ok, maybe not a ton, but definitely a lot. To name a few:

• Facebook Extractor, Facebook Ads Extractor, Azor Extractor, Snowflake DB Extractor, BigQuery Extractor, Papertrail Extractor, Dropbox extractor, Pipedrive extractor, CJAffiliate extractor, Dark Sky Extractor
• GeoIP application, What3words Application, Data Health Application
• SAS Writer, Google BigQuery Writer, GoogleSheets Writer, Snowflake Database Writer, Qlik writer, Looker writer, Salesforce Analytics Cloud Writer
• Dozens of other connections thanks to Generic Extractor.

What's next?

• We are working on replacing the Storage Console as the most ancient part of Keboola Connection UI. This is big, so we'll replace it in parts, but it's already in the works.
• New GoodData Writer is being made. We want to get rid of its "special" behavior and make it a standard writer. Some of the not-exactly-writery features will go into separate tools.
• New Pigeon "extractor" is in progress. Again, this is an old component with quirky behavior so it has to be replaced with something reliable.
• Also a new S3 extractor is waiting behind the door. This will be in many ways a conceptually different extractor taking advantage of processors and having configurations organized in a different way.
• We're working on making it easier to learn Keboola Connection. It's not quite finished yet, but we definitely want to make life easier for new users.
• Removing MySQL backend. It's been there a long time and it's sad to see it go. Actually, no it isn't. We'll throw a party when it's gone for good.
• Database writers are going to get a larger update.

And beside that?

Life is a bitch, so we can't make promises, but:

• Better managed wish list is something both you and we wish for.
• RStudio and Jupyter Sandbox have many improvements in queue.
• We have quite a few (almost revolutionary) ideas about Input mapping.
• Shared credentials (need three configurations connecting to the same database server?) would make a lot of situations easier to handle.
• Transformations are going to get simplified (no phases, no dependencies, just transformations).
• We'd like to support even bigger data (no big is too big), so thanks everyone for pushing the limits.
• Orchestrator needs quite a few updates and so does Generic Extractor

We hope we meet those and many other goals but that's all for now. 

We wish you all a very happy and successful New Year.

As part of our effort to make Keboola Connection an open platform we would like to announce the availability of the Keboola Developer Portal - https://apps.keboola.com/. Some of you may already be aware that it is possible to modify applications in KBC through the Developer Portal API. Now we are adding an application to the API so that you can get things done more easily.

The Keboola Developer Portal is a completely separate application from Keboola Connection. It is region-less because it has no access to customer projects. The Developer Portal provides an authoritative list of KBC applications for each region. Each Keboola Connection stack reads this list regularly. As usual in Keboola, the Developer Portal is done in an API first approach, so everything that the application does can be done programmatically too.

What does the Developer Portal do?

• It allows users registered to vendors to create and modify applications in KBC without Keboola Tech support.
• It provides service accounts for automating deployment and testing.
• It provides Docker AWS ECR repositories for applications (thus avoiding Quay and Dockerhub issues).
• It provides an authoritative list of publicly available approved applications.

Anyone can sign up for the Developer Portal. However until you join a vendor or create a new one, you will not be able to do anything. Both creating and joining a vendor must be approved. Joining a vendor must be approved by the vendor administrator, and creating a vendor must be approved by Keboola Tech. Once a member of a vendor, you can modify and add new applications to that vendor. There's no need to fill a checklist and go through our support. When you create an application, it becomes immediately available for you to use. Using the Developer Portal, you can also set up automated deployment for your application.

Keboola Tech still has some things to do:

• approve new vendors
• change some controlled properties of an application (e.g. memory limit)
• approve a new application before it can become public

Next steps:

• We still have a long list of things to be done on the Developer Portal, so we consider it as beta quality at the moment.
• We are soon going to hold a developer meet-up in the Czech Republic where we’d like to hear about your experience developing applications for KBC (in January probably).
• If you have previously used the Developer Portal API, we encourage you to try the application. If you are having trouble joining your vendor, please contact us on support (there may be some friction depending on how your account was created).
• If you have created complex applications using Custom Science, go ahead and check the Developer Portal, it may now be more convenient to turn them into full applications.
• If you have never created an application for Keboola Connection, we highly encourage you to try it - it's much easier now. In fact a component can be created from scratch in an hour.

New Features

• We are pleased to announce a significant speed boost to Snowflake load to workspace (also speeding up Input Mapping inTransformations). For tables with primary keys the speed difference is up to 80%.
• We have also simplified our database extractors, as we mentioned here last week.
• Fixed CSV Import. The component generates meta-data now.

Minor Imporvements

• We have updated our Terms and Conditions.
  Per section 14.3 of our Master Software Subscription Agreement, we would like to notify you of a change to the Agreement and its new version released November 10th. The purpose of this change was to provide further protection to Client Data and to comply with requirements of some third party providers (namely Google). We do not consider this change to be a material change, however we are obliged to provide you with this notification. There is no action required from you in relation to this change if you accept it. We are setting out the original and revised language for your review below. Please do not hesitate to contact us with any questions or concerns at legal@keboola.com.
  
  Original (version from March 16th, 2016):
  6.7.1. Permitted Use of Client Data. Keboola will only use and will cause all Affiliates and Keboola employees, agents, subcontractors and representatives to only use Client Data for purposes of performing the Services. Keboola shall follow Client instructions when processing Client Data and will not use Client Data other than as authorized by this Agreement. 
  
  Updated (version dated November 10th, 2017):
  6.7.1. Permitted Use of Client Data. Keboola will only use and will cause all Affiliates and Keboola employees, agents, subcontractors and representatives to only use Client Data for purposes of performing the Services. Keboola shall follow Client instructions when processing Client Data and will not use Client Data other than as authorized by this Agreement. Client Data which Keboola downloads (with permission granted by Client during component configuration) from Google, Facebook, Twitter or other third party APIs ("Third Party Data") are stored and handled the same as Client Data under this Agreement. Third Party Data will not be shared anywhere (it remains stored securely on Keboola systems) until Client decides to share that data (for example, by configuring Data writer to Client or third party systems). Client can delete all Third Party Data and data source configurations from Keboola storage anytime. This section shall be read along side with and be subject to the provisions of sections 6.9 (Retention and Deletion) and 6.10 (Compliance with Law) of this Agreement.

We're very pleased to announce the arrival of simplified database extractors.  (Available for MySQL, MSSQL, PGSQL, Redshift, Snowflake, Oracle and IBM DB2)

Configuring the new extractors can not be any easier.  All you need to do is select the tables you want to import and ... that's it! 

Facebook is deprecating Marketing API v2.9 on Monday, November 13, 2017 thus any api call to v2.9 will fail with error.

The latest version is now v2.10.

The migration should only require changing the api-version parameter in all existing configurations of our Facebook Ads Extractor. However, since there may be some breaking changes, we strongly recommend to change the api-version in your configurations manually, review any possible changes and take the appropriate actions.  For more details on the new version, please read Marketing API Changes in v2.10 in the Facebook API Changelog.

Possible breaking change might be renaming of action_link_click_destination breakdown types as follows:

• app_store_clicks => click_to_app_store
• deeplink_clicks => click_to_app_deeplink
• website_clicks => click_to_website

In other Facebook news, they have announced version v2.11 of Facebook Graph API and Facebook Marketing API, for more info read changelog here.

We have been experiencing temporary technical difficulties today since this morning (~ 3:00 AM CET)

Some jobs may have failed as a result.

We're working on resolving the problem and will update here with further information when we have it.

Thank you for your understanding.

Update 14:00 CET, October 22, 2017:  All systems have returned to normal operations and all jobs are now being processed normally.

New Features and Improvements

• Geneea NLP app now supports relations - e.g. My pizza was completely burned. > burn(OBJECT:pizza)
• Redshift transformations input mapping now supports ZSTD compression.
• Improved handling of events search errors

New Applications

GeoIP

GeoIP is latest transformation component power by MaxMind. It helps you to obtain location data from your IP address inputs.

Data Health Application

Data Health Application is an app designed to aid users to produce a clean data file. For more details, please check out Keboola Blog.

Today, September 25, 2017, we experienced jobs failures between 15:11 - 16:15 PM CET. One of our worker servers went down and while launching a new one automatic job restart mechanism didn't seem to work properly and this led to the jobs failures. Everything is working properly now and we will further investigate the cause of the issue. However we recommend to review your orchestrations and take appropriate action if needed. 

We are sorry for any inconvenience.