Deleting Projects

To simplify cleanup when a KBC project no longer serves its purpose, we added simple method for deleting it. You'll find that in the "Users & Settings" tab.

We maintain a recoverable backup copy of each project for 60 days following the deletion. After that period, it's gone for good.


Storage API Client For R

Want to play with your KBC data in your local R environment?

Install the keboola-sapi-r-client and you can.  
(The package is on GitHub so it is installed via the devtools package) 

install.packages("devtools")
library(devtools)

We need to install a github dependency
for aws request signature generation

devtools::install_github("cloudyr/aws.signature")

Now we can install the Storage api client and load it into our R session

devtools::install_github("keboola/sapi-r-client")
library(keboola.sapi.r.client)

Just like any other R package, once installed, it can be invoked in any future session with the library() command.

To instantiate the client just give it a KBC token.
We'll use the token for the currency exchange rates for demonstration purposes.

client <- SapiClient$new('452-33945-de5bb7fecb818901f0834b2431564003296a4b05')

Now we can import data to our R session

currencyData <- client$importTable('in.c-ex-currency.rates')

Just for fun, let's make a simple plot of EUR vs USD using the ggplot2 library
if not installed on your R use install.packages("ggplot2")

# prepare our data
eurVsUsd <- currencyData[which(currencyData$toCurrency == "USD"),]
eurVsUsd$date <- as.Date(eurVsUsd$date)

# load the libraries needed to make our plot
library(ggplot2)
library(scales) # for prettier x-axis labeling

p <- ggplot(eurVsUsd, aes_string(x="date", y="rate")) + geom_point()
# add x-axis scaling and title
p <- p + scale_x_date(breaks="1 year", labels=date_format("%Y"))
p <- p + ggtitle("EUR vs USD")
print(p)

The code for this sample is here in this gist. 

The Storage API client gives full read and write access to your KBC project within the comforting power of your local R environment.

Imagine the possibilities!

* small print *  This is a development tool in Beta, use at your own risk!

Table Aliases with your custom SQL

A few weeks ago, we silently launched ability to create Storage API Aliases by using your own SQL code. These Alias Tables with custom SQL can be created with Redshift backend only.

Create New Alias Table:

Define your own SQL code:

Why?

Alias Tables can help you structure your data. Imagine it as a "Transform on Demand" - everything is happening on-the-fly (aka real-time). Say we have business transactions in table "data". This is an example how to define "derived" table with weekly sum of all transactions, that can't be joined with our Customer (alarm, wrong data!! :-)

Raw Result of this simple alias table:

"year","week","total"
"2014","1","1314788.27"
"2014","2","3719694.16"
"2014","3","3907852.92"
"2014","4","4013945.26"
"2014","5","3884234.84"

Thanks to the almost unlimited power of the Redshift cluster, you can also create much more complex examples. For instance, this one creates a denormalised table of transactions that occur during the night at weekends, in EUR, outside of Czech Republic and not having one specific product code:

Transformations Redshift output mapping bug

There was a bug in import to Storage API from transformations, it was present only in the following conditions:

  • Redshift transformation with output to Redshift table
  • Output was incremental
  • Previously null value was changed to non null value

Null values were never updated in Storage API table. Bug is fixed now but the affected output tables have to recreated. The bug was present in Storage since the roll out of Redshift support.

Direct (r/o) Access to any Redshift Bucket

Today we're announcing new Storage API feature: Bucket Credentials (api here). 

If you're using Keboola Connection w/ Redshift backend, you can have read-only credentials (direct sql access) to any Redshift bucket. 

In Storage API Console, go to Bucket Detail > Credentials and press "Create new credentials" button:

Describe new credentials (you can have multiple credentials assigned to each bucket!):

When you create credentials, carefully copy&paste credentials to you SQL client or preferred remote service (jackdb.com, chartio.com, etc.). After closing displayed credentials, you can't display it's settings:

In case you need to re-use already created credentials, you have to delete it and create new combination of username and password. All existing credentials are listed under it's bucket:


Limits:
  1. credentials can be used for accessing just one bucket
  2. write access isn't supported 

WARNING: Always employ SSL when accessing your data. Generated credentials are opening your dedicated AWS Redshift Cluster. Please read "Configure Security Options for Connections". Redshift Cluster's CA certificate can be downloaded here.

Storage API outage

Our Storage API servers were offline from 2:33 UTC to 6:47 UTC of July 21th, 2014. We're still identifying main roots of this outage now, but servers are up and running. You can expect slow performance for next 1~2 hours. All nightly jobs failed. 

We let you down and we know it. We take our responsibilities — and the trust you place in us — very seriously. I cannot express how sorry I am to those of you who were inconvenienced. 

Please, contact us at support@keboola.com in case you have any trouble, we'd like to assist you!