Skip to content

Improved GPU utilization #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
geertvandeweyer wants to merge 44 commits into
base: master
Choose a base branch
from
Open

Improved GPU utilization #2

geertvandeweyer wants to merge 44 commits into from

Conversation

@geertvandeweyer
Copy link

@geertvandeweyer geertvandeweyer commented Nov 16, 2022

This PR contains:

  • fixed implementation of the tensor-batch-size (was not used)
  • offload more code to CPU, preventing repeated np to tensor conversion of the batch during prediction
  • implemented a queue to always have prepped batches ready for analysis
  • a fix where invitae implementation crashed on missing predictions for some variants
  • a dockerfile & dockerhub entry based on this version

Updated benchmarks show ~ 2.5x performance gain over the current version.

wings-public and others added 30 commits November 12, 2022 13:30
@wings-public
reordered flow to only process full batches
dfb20f7
@wings-public
sporadic missing predictions handling
d72d9ee
@wings-public
print overall performance when batching
6f1d40e
@wings-public
fix tensorflow batch size to actually use the provided size
7662a76
@wings-public
revise logic to read parallel to prediction
ecd71f6
@wings-public
disabled performance output on predict in illumina implementation
5e64b96
@wings-public
furhter optimization of gpu usage through offloading np-tensor conver…
e20c586 
…sion to cpu
@wings-public
add gc collect to original code to prevent oom kills
ee16e00
@wings-public
add long variant for all arguments
3160793
@wings-public
add dockerfile
36a6659
@wings-public
moved docker directory
e6e06cc
@wings-public
corrected long argument access
d223e4c
@wings-public
updated benchmarks
580f5d5
@wings-public
added docker link
90a3f01
@wings-public
small fix to dockerfile
f7ec23c
@wings-public
small fix to dockerfile
7255331
@wings-public
fixed typo
b1f28d2
@wings-public
fixed code layout
2c5d052
@wings-public
fixed code layout
1b28d31
@wings-public
fixed table layout
2cf5ac5
@geertvandeweyer
Update README.md
af2e7f8
@geertvandeweyer
revised code to scale to multiple gpus
c7fd579
@geertvandeweyer
relocate annotation loader in worker
cbc9679
@geertvandeweyer
fix arguments
0754a37
@geertvandeweyer
fix arguments
e2ec04b
@geertvandeweyer
add small sleep investigating issues above 3 gpus
aad6bf9
@geertvandeweyer
looking into startup issues
fdbcf1d
@geertvandeweyer
There is an issue when going above 2 GPUs
31a2ad0
@geertvandeweyer
hide physical devices in batch workers to evaluate memory issues
0e89d63
@geertvandeweyer
pass nonmasked device to worker for correct shelf names
18c6dac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@geertvandeweyer @wings-public @barneyhill @matthiasblum