WIP: New hashing schema #59
Conversation
|
@abulte I would need a review here. I left several comments in the code after implementing the double cache feature. A work around would be to change the endpoint: This would work but would it make the URL hash useless and not relevant? |
abulte
left a comment
abulte
left a comment
There was a problem hiding this comment.
About upload view: I think we can rely on file hash only in this case, and store a url hash = None.
| else: | ||
| raise RuntimeError('Func get_db_info need at least one not none argument') | ||
|
|
||
| res = c.fetchone() |
There was a problem hiding this comment.
The code below can probably be made cleaner/shorter by getting column values in a dict instead of a list eg res[‘uuid’].
| logger=app.logger, | ||
| sniff_limit=app.config.get('CSV_SNIFF_LIMIT'), | ||
| max_file_size=app.config.get('MAX_FILE_SIZE') | ||
| ) |
| encoding = detect_encoding(filepath) if not encoding else encoding | ||
| table = from_csv(filepath, encoding=encoding, sniff_limit=sniff_limit) | ||
| return to_sql(table, urlhash, storage) | ||
| return to_sql(table, urlhash, filehash, storage) |
| filehash = X.hexdigest() | ||
| logger.debug('* Downloaded %s', filehash) | ||
| if not is_hash_relevant(urlhash, filehash): | ||
| print("HASH IS NOT RELEVANT") |
| except Exception as e: | ||
| raise APIError('Error parsing CSV: %s' % e) | ||
| else: | ||
| print("HASH IS RELEVANT") |
| raise APIError('Error parsing CSV: %s' % e) | ||
| else: | ||
| app.logger.info(f"{urlhash}.db already exists, skipping parse.") | ||
| print("AGE IS OK") |
2 participants
No description provided.