Week in Review — December 3, 2019

New Components

AppsFlyer Extractor

  • We’re happy to welcome the AppsFlyer Extractor to the family!

  • AppsFlyer is the market leader in mobile advertising attribution and analytics, helping marketers to pinpoint their targeting, optimize their ad spend, and boost ROI.

  • This component was developed by KDS Team.

DynamoDB Writer

  • Another great addition to our component collection is DynamoDB Writer!

  • This component lets you write data from our Storage into AWS' DynamoDB, which is a key-value, document-based database. It's fully managed, multiregion, multimaster, durable with built-in security, backup and restore, and in-memory caching for internet-scale applications.

  • This component was developed by KDS Team.

CSV to XLSX Processor

  • Highly demanded utility to convert data in CSV format to XLSX is here!

  • This component was also developed by KDS Team.


Updated Components

AWS S3 Extractor

  • Uses parallel download and it's more than 20 times faster!

Snowflake Extractor

  • Supports incremental load.

  • Supports config rows.

Oracle Extractor

  • Supports incremental load.

  • Supports configuration rows.

  • Implemented retry mechanism for all types of queries

Redshift Extractor

  • Supports incremental load.

MySQL Extractor

  • Implemented retry mechanism for all types of queries

PostgreSQL Extractor

  • Added support for PostgreSQL Server v12


Improvements

Transformations

  • Transformations are collapsed by default.

Data Catalog

  • Improved "empty" page

  • Type of sharing is now shown and will be improved further. 

  • Improved “initial" modal

  • Bucket description is shown where applicable. 

Storage

  • When creating an alias table, you can now filter data by the EMPTY_STRING or SPACE characters.


Fixes

  • Fixed a bug in the Darksky extractor date conversion. The bug caused dates in the date column for places in UTC+ timezones to be one day less than the actual forecast date.


Developers


Google components authorisation issue [Post-Mortem]

Since July 26 2019 there was an issue with authorisation of our Google related components using OAuth in the EU region.

We needed to undergo a verification process of our apps by Google. This process has been recently made more strict and we weren't able to finish it before the deadline.

We are very sorry about this, we should have paid more attention to this process and the consequences related to not complying with it.

We have now successfully verified most of our Google components. We are still waiting for the verification of GMail Extractor and GMail Attachments Extractor.

Weeks in review -- March 1, 2019

New Features

  • Oracle Writer - supports setting custom schema in credentials configuration
  • GitHub Extractor - adds organization to downloaded commits and issues
  • Storage API - supports multiple where filters and order by statement inside datapreview and asynchronous table export. You can quickly search for your data in datapreview with multiple conditions. We are going to add this feature to UI soon.
  • BigQuery Extractor - supports extracting data from the EU region


UI

  • full page table preview for Storage tables



Bug fixes

  • OAuth Broker API - fixed missing component credentials
  • Oracle Extractor - fixed manifest for exported tables if there were more than one table with the same name in different schemas
  • Transformations 
    • unpaired closing comment tag ( */ ) in SQL query is now properly identified as a user error
    • added additional retries when creating a workspace


    Troubles with new OAuth Broker API

    UPDATE 2019-02-25 9:00 AM UTC
    A fix has beed deployed and the problem now longer occurs. We suppose, that around 50 jobs suffered from this issue.
    We do sincerely apologize for the trouble this may have caused to you. Don't hesitate to contact our support for help.


    2019-02-24 10:00 PM UTC
    We're experiencing issues with new OAuth Broker API. 

    In some cases it might not return the authorised credentials for a component's job. Re-running the job might be successful.

    The fix will be deployed very soon.

    If you haven't migrated to the new version, please wait until the fix is deployed.

    We're terribly sorry for any inconvenience.

    Migrate to new version of OAuth Broker API

    We have just released a new version of our OAuth Broker API.

    OAuth Broker is a KBC service, which handles the authorisation flow for all KBC components (extractors, writers, ...) using OAuth authorisation and also stores the credentials (tokens) for them.

    The new version was needed to simplify integration with KBC and allows us to implement new features into this API more easily.
    The features we are preparing are for example automatic refreshing of OAuth tokens if needed, using more than one OAuth client id for better quota limits handling and so on.

    The old OAuth Broker is now deprecated, and we ask you to migrate affected configurations credentials before May 1, 2019. We can't migrate these credentials automatically because we cannot modify configurations in your project without your consent.

    In the project Overview, you can see whether your project contains any configurations needing migration:


    Proceed to the migration page where you can migrate all the affected configurations in one click:


    Some of the components - GitHub Extractor, Twitter Ads Extractor and ZOHO CRM Writer - need to be reauthorised manually in order to be migrated to the new version.

    New configurations will use the new version of OAuth API from now on. Also, if you reset the authorisation of an existing configuration, it will be created using the new version of the API.

    Week in Review -- August 22, 2018

    Updated Components

    Storage

    • Added Configuration Row detail API resource

    Slack Extractor

    • Fixed pagination. The extractor was only downloading 1000 records from a channel history.
      Existing configuration of the extractor needs to be re-saved to apply this fix.


    New Features

    • When creating new API tokens, you can now choose Orchestrator from the Custom Access components dropdown

    Minor Improvements

    • Added Incremental label to Output Mapping
    • Storage API and Manage API tokens are now generated from alphanumeric characters instead of hexadecimal
    • Added Documentation badge to component detail. Additional external documentation is available for component marked with this badge. 
    • Filter expression operators were renamed to be more intuitive

      EQ (equals) -> IN
      NE (not equal) -> NOT IN
    • It's no longer necessary to click the <-Back button when adding tasks to Orchestration. The New Task modal will reset its state and after reopening it shows the list of components. Previously, after reopening, it showed a list of configurations of the previously selected component

    Week in Review -- May 28, 2018

    New Components

    OneDrive Extractor (Beta)
    • You can now download your documents from OneDrive. The component was developed by Jakub Bartel.

    Google Big Query Writer (Beta)

    • Write data from KBC to Google Big Query with our new writer. The component was developed by us, Keboola s.r.o. 


    Updated Components

    Currency Extractor

    • Now offers exchange rates of GBP.

    Oracle Extractor

    • Now supports exporting columns of the LOB datatype family.

    Gmail Attachments Extractor

    • Added support for processors, see example.

    Create Manifest Processor 

    • Will test, if all the slices of a CSV file have the same header columns.

    Flatten Folders Processor 

    • Throws a user exception when the flattened filename is too long (longer than 255 characters).
    • Has now configurable flatten strategies. Added `hash-sha256` strategy, which solves an issue with 255 characters filename limit in default `concat` strategy.


    New Features

    • Component badges.
    • Validate your SQL in transformations with new Validate feature, provided by SQLDep.

    • MySQL extractor now supports incremental fetching. You can extract just the most recent records from a database table and write them incrementally into Storage.


      Minor Improvements

      • Show more / show less button added to the list of inputs and outputs of a job. It now shows all the tables and the view is more compact.

      Week in Review -- March 5, 2018

      Improvements

      • Schema in Snowflake Extractor is no longer a required connection parameter. If not set, the table selector allows you select tables from the whole database.
      • Snowflake extractor now supports importing these semi-structured data types: `VARIANT`, `OBJECT`, and `ARRAY` 

      • Updated two factor authentication in Keboola Developer Portal. The SMS authentication is now deprecated. All new users will have to use either Google Authenticator or Duo Mobile app.
      • MySQL extractor now has an option to enable compression of data sent over network
      • Enhanced Output Mapping selector
      • R in Sandbox and Transformations is updated to 3.4.3, also the Tidyverse package is now installed by default.

      Bug fixes

      • Data Takeout was randomly failing on backing up your data to S3.
      • Task editor in Orchestrator produced errors when orchestration had configured dozens of tasks.
      • In Twitter extractor template, if a user made a mention of your account, the details of that user account weren't downloaded. Edit and save existing configuration to remedy this issue.

      Google Analytics Extractor - Custom Authorization

      Recently we have been hitting quota limits of the Google Analytics API with our extractor. As these limits are shared between all users of the extractor, we decided to add a new authorization option - Custom Authorization.

      With Custom Authorization, you can provide your own API keys for the Google Analytics extractor, so you will have your own quota limits. This way you can monitor these limits as you need, and other users won't be affected.

      Please check out our step-by-step guide on how to set up Custom Authorization in the Google Analytics Extactor.

      In case of any problems or questions, please contact us on support@keboola.com.

      End of life of Keboola Academy

      All things have to come to an end and Keboola Academy’s GoodData training is no exception. But don’t mourn, we are working hard on a new education program. Its main focus will be Keboola Connection.

      Keboola Academy courses will be available for purchase until December 31, 2017. After that, you won’t be able to purchase new courses, but you’ll still be able to access your already purchased courses for another two months. Courses will be shut down completely on February 28, 2018.

      If you have any questions, please contact us via support@keboola.com.