Since most
of us are enjoying the winter holiday, the usual Week in Review would be
half-empty. It's perhaps time to review the year with the benefit of hindsight. A lot of
stuff happened last year which is easily lost in the daily commotion, so
let's have a little recap.
The big
things
- Keboola
Connection in EU region — Although this might not look exactly revolutionary, it
required immense amount of work to make it so our backend is not tied to a specific
region any more. This of course opens doors to other regions, which are going
to be much easier to do.
- Developer
Portal — This allows anyone to create and deploy applications in Keboola Connection. Hosting
of Docker images, automated deployment and testing integration — all inclusive.
- Big files
support and other performance improvements — We all have big data, don't we?
There were a number of limitations in several places which didn't allow
processing of files larger than 2GB or 5GB. Though there are new limits now (around 100GB), it's much harder to hit them. A lot of other
performance improvements were achieved, among those the recent up to 80% speedup in
workspace load is worth bragging about.
- Component
Dockerization — This is a ton of completely invisible work which has no
immediate impact on you. Once we are done with this, we'll be able to rework
job processing stuff and offer super flexible projects (ever wanted long
running jobs? or jobs requiring XXX GBs of RAM?). There is still a lot to do, but
we're not dawdling.
- Shared
buckets — Although they are not perfect, they hugely simplify sharing things in larger organizations.
- Processors — Quite a hidden nerdy feature which is slowly making it into
production. This, together with Developer portal opens the door for simplified
implementations (especially in extractors) of those "Oh, it's perfect, I just need to convert this one little thing ...".
- Trash of
deleted configurations — A lifesaver for many.
- RStudio
and Jupyter Sandboxes — Do they count as a big thing too?
- New database extractors — The database extractors were greatly simplified. You no longer need to remember your database schema, and can make your configuration in just a couple of clicks.
- UI/UX
improvements — No shocking things happened (yet?). but we certainly put much
more focus on this area throughout the year. There are a various new features
(markdown descriptions, merging table slices, finer input mapping granularity). But we are also putting a lot of effort in unifying
the look and feel of different Keboola Connection parts and smoothing the flow. Hopefully this
makes Keboola Connection more pleasurable to work with.
Security improvements
We're
keeping an eye over security all the time, and constantly improving. To name a
few things:
- Project
Access Approval
- New
Google Sign-in
- CSP
(Content Security Policy) used in entire Keboola Connection
- Better
secured Keboola Connection cookies and sessions
- Display
of all active account sessions
New
components
There were literally
a ton of new and updated components. Ok, maybe not a ton, but definitely a lot.
To name a few:
- Facebook
Extractor, Facebook Ads Extractor, Azor Extractor, Snowflake DB Extractor,
BigQuery Extractor, Papertrail Extractor, Dropbox extractor, Pipedrive
extractor, CJAffiliate extractor, Dark Sky Extractor
- GeoIP
application, What3words Application, Data Health Application
- SAS
Writer, Google BigQuery Writer, GoogleSheets Writer, Snowflake Database Writer,
Qlik writer, Looker writer, Salesforce Analytics Cloud Writer
- Dozens of other connections thanks to Generic Extractor.
What's
next?
- We are
working on replacing the Storage Console as the most ancient part of Keboola Connection UI.
This is big, so we'll replace it in parts, but it's already in the works.
- New
GoodData Writer is being made. We want to get rid of its "special" behavior and
make it a standard writer. Some of the not-exactly-writery features will go
into separate tools.
- New
Pigeon "extractor" is in progress. Again, this is an old component
with quirky behavior so it has to be replaced with something reliable.
- Also a
new S3 extractor is waiting behind the door. This will be in many ways a conceptually different extractor taking advantage of processors and having
configurations organized in a different way.
- We're
working on making it easier to learn Keboola Connection. It's not quite finished yet, but we definitely
want to make life easier for new users.
- Removing
MySQL backend. It's been there a long time and it's sad to see it go. Actually, no it
isn't. We'll throw a party when it's gone for good.
- Database
writers are going to get a larger update.
And beside
that?
Life is a
bitch, so we can't make promises, but:
- Better
managed wish list is something both you and we wish for.
- RStudio
and Jupyter Sandbox have many improvements in queue.
- We have
quite a few (almost revolutionary) ideas about Input mapping.
- Shared
credentials (need three configurations connecting to the same database server?)
would make a lot of situations easier to handle.
- Transformations are going to get simplified (no phases, no dependencies, just
transformations).
- We'd like
to support even bigger data (no big is too big), so thanks everyone for pushing
the limits.
- Orchestrator needs quite a few updates and so does Generic Extractor
We hope we
meet those and many other goals but that's all for now.
We wish you all a very happy and successful New
Year.
