Stream names in insertion_method_tables, added logging #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hg-ramosaj
commented
May 22, 2023
hg-ramosaj
commented
- Added logging for deleted tables
- allowed insertion_method_tables to take stream names
hsyyid
requested changes
May 22, 2023
README.md
Outdated
| | validate_records | Boolean | | (Default: False) Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation. | | ||
| | temp_dir | String | | (Default: platform-dependent) Directory of temporary CSV files with RECORD messages. | | ||
| | insertion_method | String | | (Default: 'incremental') Currently only handles 'truncate' (drop and re-create table before adding rows) and 'incremental' (upsert new rows) | | ||
| <<<<<<< HEAD |
Member
There was a problem hiding this comment.
Looks like there's a conflict here
README.md
Outdated
| | insertion_method_tables | Array(String) | | (Default: []) Names of streams corresponding to tables to apply "insertion method" to. Has no effect if insertion method not specified. | | ||
| ======= | ||
| | insertion_method_tables | Array(String) | | (Default: []) Tables to apply "insertion method" to. Has no effect if insertion method not specified. | | ||
| >>>>>>> 4a18fba3c9a6c759f298e7b1bcc51640d95d2a76 |
target_postgres/db_sync.py
Outdated
| if len(insertion_method_tables) == 0: #if no insertion_method_tables specified, remove all tables and repopulate | ||
| self.query(self.drop_table()) | ||
| elif table_name[1:-1] in insertion_method_tables: #and if table name in insertion_method_tables | ||
| elif stream in insertion_method_tables: #and if table name in insertion_method_tables, truncate |
Member
There was a problem hiding this comment.
Does this mean it will always expect stream name instead of table name? Can we support both?
Author
There was a problem hiding this comment.
Yes! I figured it would be clearer in the config to have the stream names instead of the table names, since the table names for postgres are different (underscores, lowercase, etc). We could support both if needed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.