MarkLogic Database for Ontologies
Problem Statement: Customer is restarting PostgreSQL database every night. Long running jobs are failing from after DB restart.
Just to save the trouble of scanning through the PostgreSQL driver release notes, the Connection.setNetworkTimeout() was released in version 42.2.0 of the driver (https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.0)
Check when database was restarted
To get the start time, execute the query below:
psql -c “SELECT pg_postmaster_start_time();”
To get the uptime, execute the query below:
psql -c “SELECT now() – pg_postmaster_start_time();”
Exception Handling for guaranteed write:
boolean writeStatus = false;
write to database;
writeStatus = true;
}catch (Exception ex)
sleep for 30 seconds;
//Hope for db to recover.
Greenplum Database stores and processes large amounts of data by distributing the data and processing workload across several servers or hosts. Greenplum Database is an array of individual databases based upon PostgreSQL 8.2 working together to present a single database image. The master is the entry point to the Greenplum Database system. It is the database instance to which clients connect and submit SQL statements. The master coordinates its work with the other database instances in the system, called segments, which store and process the data.
Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.
Apache CouchDB™ lets you access your data where you need it by defining the Couch Replication Protocol that is implemented by a variety of projects and products that span every imaginable computing environment from globally distributed server-clusters, over mobile phones to web browsers. Software that is compatible with the Couch Replication Protocol include: PouchDB, Cloudant, and Couchbase Lite.
Store your data safely, on your own servers, or with any leading cloud provider. Your web- and native applications love CouchDB, because it speaks JSON natively and supports binary for all your data storage needs. The Couch Replication Protocol lets your data flow seamlessly between server clusters to mobile phones and web browsers, enabling a compelling, offline-first user-experience while maintaining high performance and strong reliability. CouchDB comes with a developer-friendly query language, and optionally MapReduce for simple, efficient, and comprehensive data retrieval.
ETL Design Pattern: http://www.leapfrogbi.com/2013/05/11/etl-design-patterns-the-foundation/
Standard ETL Scenarios: https://dwbi.org/etl/etl-design-pattern/57-etl-design-pattern