Keboola Status

It is my tremendously big pleasure to announce an important update - Dropbox Extractor component was updated in Keboola Connection (KBC) and this new version is officially released and available for everybody.   

It's been quite a while since the original version was developed. So happened, it was one of my first components written in Node.js and also my first-ever KBC component where I didn't use a generic UI, but developed a custom one (I also used React for the first time). I loved that, but (on the other hand) I was also aware the first version was anything, but perfect.

The new version is here and the main reason for this update is the end of the support of the Dropbox API v1 (used in the original extractor). It became deprecated and (by the end of the June 2017) this old API stops working completely. 

Good news is that this new version of the Dropbox extractor brings a lot of important and valuable improvements. The main change is usage of the official Dropbox Chooser component. You can easily authorize and select csv files from your Dropbox. The Chooser component generates a link (you can paste already existing one) and all these links are encrypted in the KBC configuration, so the whole process is very fast (even with large files), convenient and secure.

Migration note: There are breaking changes between the previous (v1) and new (v2) versions and if you use the original Dropbox Extractor (v1) and want to keep using Dropbox extractor in KBC in the future (after June 28, 2017), you have to create a new configuration in this updated version (v2) and select all your required files again. Unfortunately, no automated migration tool between v1 and v2 is available as the architecture is completely different.

I would love to say big thanks to Keboola, Tomáš Kačur in particular. He provided very helpful feedback as well as helped me to fine-tune the UI part. 

This extractor is developed independently by Radek Tomášek. For more information on how to use this extractor, please refer to the documentation. In case you run into some issues or you have more questions about the usage of the component (e.g. migration between the old and new version), please don't hesitate to contact me directly.

It's my big pleasure to announce another major update on one of my first component I had ever build - YouTube Reporting API.

YouTube Reporting API offers a very simple way to download daily reports that belongs to a Content Owner of a Youtube Channel (in other words, your channel account must be authorised by Google if you want to download data from this API). These reports are generated by the defined jobs and all you need to do is to download these results. And this is something the extractor was build for. 

As the general process is very simple, the first version of this extractor was completed in such a short time. However, while we were using the extractor in the production, we found that (from time to time) Google triggers some background actions leading into generating reports which broke the original logic and produced incorrect result (caused by logic related to merge operations). And for that reason the first version of this extractor was not super useful for the production deployment.

However, based on that experience I really wanted to fix the problematic parts of the original version and turn this extractor into the project which is fun to use. And I simply believe I made it and I am extremely proud of what I achieved in this update.

You can read the full description in the documentation. In a nutshell, this extractor downloads reports generated by jobs. However, there are lots of extra features which help you to manage these downloads in a very convenient way. For example, the original configuration requirements implemented in the first version of this extractor was reduced significantly and there were also added several options for creating a backup (S3). But most importantly, all data should be downloaded correctly.

This writer is developed independently by Blue Sky Media. For more information on how to use the writer, please refer to the documentation. In case you run into some issues or you have more questions, please contact me directly. 

Have you ever used Sailthru and are you interested in analysis of your campaign data? If you answer is positive in both cases, you might be happy for this new extension to the Keboola Connection family - Sailthru Campaign Extractor. 

As title says, the extractor is able to download your campaign data and datasets like beacon times, devices, domains, engagement, just to name a few (you can see the full list in the documentation).

The configuration of this extractor is very simple and a lot of things are happening 'under the hood'. The output datasets are aggregated and some of them contain some additional information, just for your convenient to potentially simplify your analytics queries in Keboola Connection (at least a little bit). During the development there was a big focus on extensibility and for that reason there is a high probability there will be a new set of cool features in the future. 

The development was sponsored by Pond5 and developed independently by Radek Tomášek. I would love to thank Petr Havlik for his tremendous support during the whole development process. His experience made the whole development easier. For more information on how to use the extractor, please refer to the documentation.

It is my pleasure to announce a new extension to a Keboola Connection component family - Ooyala Video Analytics Extractor, which will help you with the extraction of the analytical data from Ooyala Analytical backend.

The component has quite flexible options for the configuration and you can get (aggregated) data based on various metrics and timeframes. 

There are also a lot of things for your convenient on the Ooyala backend. A good example are the error messages, if you put something unexpected into the configuration, the backend will raise a very specific response which helps you to debug the configuration easily.

This writer is developed independently by Blue Sky Media. For more information on how to use the writer, please refer to the documentation. In case you run into some issues or you have more questions, please contact me directly. 

Recently, there was an announcement from Apple about their plans to deprecate the iTunes Connect Autoingest tool by the end of November, 2016. They also mentioned the new way how to download financial and sales data from their APIs and for that reason they made available the new Reporter API.

In order to avoid any potential problems, it is my pleasure to announce that the iTunes Connect Extractor was updated and from now on it started using the new APIs which helps you to keep downloading data from the Apple reporting without any problem.

The new set of APIs is not the only change. I focused all my effort on simplicity and simplified the way how to pass the input params into Keboola. That means a lot of bigger things are now managed 'under the hood' and you don't to worry much about it. For instance, if you start selling the content in any new country, the extraction process will handle that and you will see the new region in your Keboola Storage object automatically. 

This writer is developed independently by Blue Sky Media. For more information on how to use the extractor, please refer to the documentation. In case you run into some issues or you have more questions, please contact me directly. 

It's my pleasure to introduce another component into the family of Keboola Connection ecosystem: IBM dashDB Writer.

IBM dashDB is a new generation analytical datawarehouse. Build on core of DB2, Powered by in-memory and columnar storage and enhanced by in-database analytical functions migrated from IBM Netezza. You will find more in the documentation.

The current version of the writer supports transfers of the objects from Keboola Storage into dashDB tables and at the moment just full loads are supported. But you should have no problem to transfer even bigger files in decent time.

This writer is developed independently by Radek Tomášek with huge support of IBM. For more information on how to use the writer, please refer to the documentation.

It's my pleasure to announce another addition to Keboola Connection - a FTP/FTPS Writer. 

I was asked to extend the SFTP/WebDAV Writer and add a support for FTP/FTPS protocols. However, after some discussion with Keboola developers, a decision of making a separated connector was made.

The use-case is simple, you can upload your data from Keboola Storage to either FTP or FTPS location. The great thing from my perspective is that this writer supports the new configuration schema that helps to pass the input configuration in a very convenient way.

This writer is developed independently by Blue Sky Media. For more information on how to use the writer, please refer to the documentation. In case you run into some issues or you have more questions, please contact me directly (radek@bluesky.pro). 

It's my pleasure to announce another addition to Keboola Connection - an Amazon S3 Writer. 

Many of you know have been already familiar with REST Box component with functionality related to S3 manipulation (reading to and/or writing from). However, there are certain situations where a custom component is a better fit and for that reason we developed one.

The main advantage is a speed. It's been written in Node.js and using AWS SDK library where the actual file upload is handled via stream processing. Even a bit larger files (+1GB) are uploaded in a very convenient timeframe.

Configuration is very straightforward as well (similar to SFTP/WebDAV Writer) and you should be able to start using this component very quickly.

This writer is developed independently by Blue Sky Media. For more information on how to use the writer, please refer to the documentation. In case you run into some issues or you have more questions, please contact me directly (radek@bluesky.pro). 

I am happy to announce another component for Keboola Connection, a SFTP/WebDAV Writer, created in our company - Blue Sky Media. For those of you who haven't met me yet, let me introduce myself. My name is Radek Tomasek and I want to write a quick summary about this extension.

A long story short, this component allows you to upload data from Keboola Storage to your SFTP/WebDAV location.

The configuration is very straightforward. There is a Table Input Mapping helper which helps you select tables from Keboola Storage and prepare them for the upload. The rest of the configuration is about adding a few parameters to the Input Parameters component. A complete specification is available in the documentation.

In case of any issue/question, don't hesitate to contact me (radek@bluesky.pro). I am more than happy to help you with the configuration.

It's difficult for me to write a quick summary about Dropbox Extractor - project that I had been developing and tuning for a while. But I'll try anyway and I am sure the wait for the result was worth it!

First of all, thank you for your patience! :-) The main reason for such a long development time was that I wanted to reach the best user experience I could. Of course, the first idea was to use a default GUI for Docker infrastructure. But I guess this approach would be enough only for specific connectors like iTunes Connect or Youtube Reporting Api, not for Dropbox, where a bigger adoption is expected (if the UX is good enough).

For that reason I was sure I wanted to write my own GUI that makes the work with the Dropbox very straightforward. The idea sounded great, but my problem was I had little (almost none, in other words) knowledge of frontend development and zero knowledge of React/Flux. However, my eagerness to reach a certain level of JavaScript full-stack development competence helped me to start with actual development and learn everything on the fly. And I never regret that decision, despite the fact, it took me a little bit more time to really understand what is going on, how to write a React/Flux code and how components in Keboola Connector are composed.

A month ago a production version was completed, but I still wanted to add some more features and improve the overall user-experience. Now I can proudly share the result of that effort. In a nutshell, you can simply select files from your Dropbox and upload them to your Keboola storage, but after a deeper look you will find features like:

- Selecting CSV files and assign them to various KBC buckets (one file can be assigned to multiple buckets).

- In case of selecting two files with the same name, they will either keep the actual name (if two different buckets had been selected) or distinguish between each other by adding a simple hash.

- Bulk or single file upload.

- Output table details in configuration list.

Well, I would love to say thanks to Keboola developers, to Martin Halamicek in particular. His hints and debugging skills helped to define what the Dropbox Extractor is now today and I just hope you enjoy it in the same way as I do! And of course, any feedback is welcome! Enjoy!