Hello postgres users, Spark-postgres is designed for reliable and performant ETL in big-data workload and offers read/write/scd capability to better bridge spark and postgres. The version 3 introduces a datasource API. It outperforms sqoop by factor 8 and the apache spark core jdbc by infinity. Features: - use of pg COPY statements - parallel reads/writes - use of hdfs to store intermediary csv - reindex after bulk-loading - SCD1 computations done on the spark side - use unlogged tables when needed - handle arrays and multiline string columns - useful jdbc functions (ddl, updates...) The official repository: https://framagit.org/parisni/spark-etl/tree/master/spark-postgres And its mirror on microsoft github: https://github.com/EDS-APHP/spark-etl/tree/master/spark-postgres -- nicolas
