New components and component updates
- New FTP extractor is now in beta. It is packed with the same features as AWS S3 extractor and it is extensible with processors. We aim to gradually replace the current FTP extractor which is developed by 3rd party.
- New fit-into-storage processor that allows to import non-csv files (TXT, JSON) into storage by wrapping them into CSV tables.
- AWS S3 Extractor fixes bug with file matching. When the Subfolders option was off and Wildcard option was on, the extractor would erroneously download files contained in subfolders exactly matching the key.
- MongoDB extractor export mode is now set to "raw" by default on new configurations.
Enhancements
Input and output mapping for components is now parallelized. Configurations with large number of tables in input and output mapping (except transformations) should now run considerably faster. This affects both extractors and writers. Most recently updated extractors (e.g. AWS S3, FTP, HTTP, MySQL, MSSQL, Storage) also load tables into storage during the extraction of other tables. Together, these make up to 40% run time reduction for some configurations.
Input mapping load type is now shown in input mapping details:
The Run Orchestration dialog now has the option to select/deselect all tasks:
Developers
The option to have gzipped CSV file on component output mapping (/data/out/tables/) was removed. Only plain CSV files are accepted now. To our best knowledge this was never used.