End of life of VPNs and Fixed IPs

Many of current configurations of database extractors are set using fixed IPs and VPN connections. We think it's time to move to the 21st century and use SSH tunnels (yeah those were designed in 90s too). Therefore we won't be offering fixed IPs and VPN setups for new customers, effective from August 2016.

Why?

Because there are never-ending problems. IPs which are not supposed to change, do change. VPNs are prone to outages. Servers get restarted unexpectedly. It is sometimes very complicated to identify the source of connection problems and many other issues. Overall, these setups are not up to our reliability and traceability standards.

Migration

We will support all of you - existing customers - till August 2017. This should give you enough time to migrate to SSH tunnels. We recommend that you (or your system administrator) read our guide for setting up an SSH tunnel. If you have any questions, do not hesitate to contact us.

Occasional Job failures

We are experiencing some issues with our infrastructure which causes occasional job failures. The jobs fail with generic message "Internal error occurred ...". We have identified the issue and we applied a fix already.

Currently we are monitoring the situation closely and we restarted the failed orchestrations. In case you encounter the issue, the solution is to restart the failed job, because the failure is transient.


Job failures

Today between 12:13 and 13:43 we had an error in one of our compontents, which caused some jobs to fail with message: 

Running container exceeded the timeout of 3600 seconds.

even if the job did not in fact exceed a timeout. Failed orchestrations have been restared. We are really sorry about this.

Week In Review - May 16, 2016

Last week was very rich on new features and stuff, so this week is a little lighter for a change.

New features

- If you have a limited project, we will now send you an email notification a week before the project expires.

- We have updated R in Transformations and Custom Science to R version 3.2.5 (April 2016).


Bug fixes

- Snapshots of Redshift tables with non-lowercase primary keys are now working correctly.

- Project Backup and Takeout now exports configuration rows (e.g. transformation queries) and works for large projects too.


Other posts this week

Redshift Transformation Input Mapping Update

Orchestrator table deletion announcement

Python Transformations

We have added Python support to our Transformation engine. Python is a handy and versatile programming language. It also has a lot of useful libraries. Particularly interesting may be the SciPy stack. All Python transformations run in our public docker image with Python 3.5.1 and have an 8GB memory limit. 

The interface is highly similar to the existing R transformations. You can start by setting input and output mapping in the UI. The tables from input mapping will be created as CSV files in in/tables directory. Result CSV files from out/tables directory will be uploaded to your project Storage. All your python code has to do is read the CSV file, do some magic and then write a CSV file.

If you need some packages from PyPI, you can list them in the packages section of the UI. By the way, the SciPy stack is installed by default.

If you are interested in writing Python transformations, we have an introduction article in documentation with some examples that show how to work with the input and output files.