Weeks in review -- March 1, 2019

New Features

  • Oracle Writer - supports setting custom schema in credentials configuration
  • GitHub Extractor - adds organization to downloaded commits and issues
  • Storage API - supports multiple where filters and order by statement inside datapreview and asynchronous table export. You can quickly search for your data in datapreview with multiple conditions. We are going to add this feature to UI soon.
  • BigQuery Extractor - supports extracting data from the EU region


UI

  • full page table preview for Storage tables



Bug fixes

  • OAuth Broker API - fixed missing component credentials
  • Oracle Extractor - fixed manifest for exported tables if there were more than one table with the same name in different schemas
  • Transformations 
    • unpaired closing comment tag ( */ ) in SQL query is now properly identified as a user error
    • added additional retries when creating a workspace


    Troubles with new OAuth Broker API

    UPDATE 2019-02-25 9:00 AM UTC
    A fix has beed deployed and the problem now longer occurs. We suppose, that around 50 jobs suffered from this issue.
    We do sincerely apologize for the trouble this may have caused to you. Don't hesitate to contact our support for help.


    2019-02-24 10:00 PM UTC
    We're experiencing issues with new OAuth Broker API. 

    In some cases it might not return the authorised credentials for a component's job. Re-running the job might be successful.

    The fix will be deployed very soon.

    If you haven't migrated to the new version, please wait until the fix is deployed.

    We're terribly sorry for any inconvenience.

    Migrate to new version of OAuth Broker API

    We have just released a new version of our OAuth Broker API.

    OAuth Broker is a KBC service, which handles the authorisation flow for all KBC components (extractors, writers, ...) using OAuth authorisation and also stores the credentials (tokens) for them.

    The new version was needed to simplify integration with KBC and allows us to implement new features into this API more easily.
    The features we are preparing are for example automatic refreshing of OAuth tokens if needed, using more than one OAuth client id for better quota limits handling and so on.

    The old OAuth Broker is now deprecated, and we ask you to migrate affected configurations credentials before May 1, 2019. We can't migrate these credentials automatically because we cannot modify configurations in your project without your consent.

    In the project Overview, you can see whether your project contains any configurations needing migration:


    Proceed to the migration page where you can migrate all the affected configurations in one click:


    Some of the components - GitHub Extractor, Twitter Ads Extractor and ZOHO CRM Writer - need to be reauthorised manually in order to be migrated to the new version.

    New configurations are created with the new version of OAuth API from now. Also, if you reset the authorisation of existing configuration, it will be created with the new version.

    Week in Review -- August 22, 2018

    Updated Components

    Storage

    • Added Configuration Row detail API resource

    Slack Extractor

    • Fixed pagination. The extractor was only downloading 1000 records from a channel history.
      Existing configuration of the extractor needs to be re-saved to apply this fix.


    New Features

    • When creating new API tokens, you can now choose Orchestrator from the Custom Access components dropdown

    Minor Improvements

    • Added Incremental label to Output Mapping
    • Storage API and Manage API tokens are now generated from alphanumeric characters instead of hexadecimal
    • Added Documentation badge to component detail. Additional external documentation is available for component marked with this badge. 
    • Filter expression operators were renamed to be more intuitive

      EQ (equals) -> IN
      NE (not equal) -> NOT IN
    • It's no longer necessary to click the <-Back button when adding tasks to Orchestration. The New Task modal will reset its state and after reopening it shows the list of components. Previously, after reopening, it showed a list of configurations of the previously selected component

    Week in Review -- May 28, 2018

    New Components

    OneDrive Extractor (Beta)
    • You can now download your documents from OneDrive. The component was developed by Jakub Bartel.

    Google Big Query Writer (Beta)

    • Write data from KBC to Google Big Query with our new writer. The component was developed by us, Keboola s.r.o. 


    Updated Components

    Currency Extractor

    • Now offers exchange rates of GBP.

    Oracle Extractor

    • Now supports exporting columns of the LOB datatype family.

    Gmail Attachments Extractor

    • Added support for processors, see example.

    Create Manifest Processor 

    • Will test, if all the slices of a CSV file have the same header columns.

    Flatten Folders Processor 

    • Throws a user exception when the flattened filename is too long (longer than 255 characters).
    • Has now configurable flatten strategies. Added `hash-sha256` strategy, which solves an issue with 255 characters filename limit in default `concat` strategy.


    New Features

    • Component badges.
    • Validate your SQL in transformations with new Validate feature, provided by SQLDep.

    • MySQL extractor now supports incremental fetching. You can extract just the most recent records from a database table and write them incrementally into Storage.


      Minor Improvements

      • Show more / show less button added to the list of inputs and outputs of a job. It now shows all the tables and the view is more compact.

      Week in Review -- March 5, 2018

      Improvements

      • Schema in Snowflake Extractor is no longer a required connection parameter. If not set, the table selector allows you select tables from the whole database.
      • Snowflake extractor now supports importing these semi-structured data types: `VARIANT`, `OBJECT`, and `ARRAY` 

      • Updated two factor authentication in Keboola Developer Portal. The SMS authentication is now deprecated. All new users will have to use either Google Authenticator or Duo Mobile app.
      • MySQL extractor now has an option to enable compression of data sent over network
      • Enhanced Output Mapping selector
      • R in Sandbox and Transformations is updated to 3.4.3, also the Tidyverse package is now installed by default.

      Bug fixes

      • Data Takeout was randomly failing on backing up your data to S3.
      • Task editor in Orchestrator produced errors when orchestration had configured dozens of tasks.
      • In Twitter extractor template, if a user made a mention of your account, the details of that user account weren't downloaded. Edit and save existing configuration to remedy this issue.

      Google Analytics Extractor - Custom Authorization

      Recently we have been hitting quota limits of the Google Analytics API with our extractor. As these limits are shared between all users of the extractor, we decided to add a new authorization option - Custom Authorization.

      With Custom Authorization, you can provide your own API keys for the Google Analytics extractor, so you will have your own quota limits. This way you can monitor these limits as you need, and other users won't be affected.

      Please check out our step-by-step guide on how to set up Custom Authorization in the Google Analytics Extactor.

      In case of any problems or questions, please contact us on support@keboola.com.

      End of life of Keboola Academy

      All things have to come to an end and Keboola Academy’s GoodData training is no exception. But don’t mourn, we are working hard on a new education program. Its main focus will be Keboola Connection.

      Keboola Academy courses will be available for purchase until December 31, 2017. After that, you won’t be able to purchase new courses, but you’ll still be able to access your already purchased courses for another two months. Courses will be shut down completely on February 28, 2018.

      If you have any questions, please contact us via support@keboola.com.

      Week in Review -- August 28, 2017

      New Features

      • List of deprecated components in the Project Overview:

        In the Project Overview you may now see a warning that you are using deprecated components in your project. 

        To prevent any inconveniences, please keep your components up-to-date. 

        Clicking on a component from the list will take you to a component configuration, from where you can start a migration process.

        Fixes

        • Fixed issue in Orchestrator, when you changed notification settings during orchestrators run, the settings will return to previous values after orchestrator has finished. (github issue in Czech language)

        Week in review -- July 19, 2017

        GoodData Writer

        • Upload Project job isn't executing reports anymore. This caused troubles on Vertica backend. You can add report execution API call into orchestrator if needed

        Python sandboxes

        • Updated to Jupyter 5.0.0 
        • Current directory is now */data/*

          Transformations (SQLdep)

          • Fixed issue with lineage on aliased cloumns

          Generic Extractor

          • Delimiter character can now be configured in Data Field, see documentation
          • Functions can now be used in Login Authentication in API Request, see documentation

          Storage API CLI

          • Added create-bucket command
          • It is now possible to run this tool as Docker container
          • Latest version is available on GitHub

          Storage API Client for Python

          • Alpha version of the client is available on GitHub.  
          • Call for contributions: any pythonistas out there want to contribute, please do so:)