Skip to content

fixing overflow in GatkSparkTool.getRecommendedNumReducers() - #5586

Merged
lbergelson merged 4 commits into
masterfrom
lb_fix_overflow_issue
Jan 22, 2019
Merged

lbergelson merged 4 commits into
masterfrom
lb_fix_overflow_issue

Conversation

@lbergelson

Copy link
Copy Markdown
Contributor

No description provided.

@lbergelson
lbergelson requested a review from jamesemery January 17, 2019 16:56
@codecov-io

codecov-io commented Jan 17, 2019 •

Copy link
Copy Markdown

Codecov Report

Merging #5586 into master will increase coverage by 0.009%.
The diff coverage is 90.909%.

@@              Coverage Diff               @@
##             master     #5586       +/-   ##
==============================================
+ Coverage     87.04%   87.049%   +0.009%     
- Complexity    31490     31527       +37     
==============================================
  Files          1924      1928        +4     
  Lines        145189    145340      +151     
  Branches      16082     16089        +7     
==============================================
+ Hits         126373    126517      +144     
- Misses        12958     12963        +5     
- Partials       5858      5860        +2
Impacted Files Coverage Δ Complexity Δ
...org/broadinstitute/hellbender/utils/MathUtils.java 79.6% <100%> (+0.324%) 220 <2> (+4) ⬆️
...stitute/hellbender/engine/spark/GATKSparkTool.java 81.921% <75%> (-0.365%) 78 <1> (ø)
...dinstitute/hellbender/utils/MathUtilsUnitTest.java 92.418% <93.333%> (+0.018%) 148 <3> (+3) ⬆️
...dinstitute/hellbender/utils/mcmc/SliceSampler.java 87.5% <0%> (-8.418%) 8% <0%> (-10%)
...copynumber/models/AlleleFractionSegmentedData.java 94.444% <0%> (-2.614%) 8% <0%> (-1%)
...er/tools/copynumber/formats/records/CopyRatio.java 79.167% <0%> (-0.833%) 10% <0%> (ø)
...lotypecaller/readthreading/ReadThreadingGraph.java 88.608% <0%> (-0.253%) 144% <0%> (-1%)
...e/spark/datasources/VariantsSparkSinkUnitTest.java 82.171% <0%> (ø) 28% <0%> (ø) ⬇️
...ber/formats/collections/SimpleCountCollection.java 100% <0%> (ø) 12% <0%> (ø) ⬇️
...er/tools/copynumber/CollectAllelicCountsSpark.java 96% <0%> (ø) 10% <0%> (ø) ⬇️
... and 18 more

}
int size = readInputs.keySet().stream().mapToInt(k -> (int) BucketUtils.dirSize(k)).sum();
return 1 + (size / getTargetPartitionSize());
long size = readInputs.keySet().stream().mapToLong(k -> BucketUtils.dirSize(k)).sum();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add an explicit check for overflow before downcasting the long?

@jamesemery jamesemery left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good

int size = readInputs.keySet().stream().mapToInt(k -> (int) BucketUtils.dirSize(k)).sum();
return 1 + (size / getTargetPartitionSize());
long size = readInputs.keySet().stream().mapToLong(k -> BucketUtils.dirSize(k)).sum();
return 1 + Math.toIntExact(size / getTargetPartitionSize());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are going to use a throwing method you should at least catch this exception and give a helpful error message, as some user/developer may want to increase the target partition size to compensate or some other such thing.

@lbergelson

Copy link
Copy Markdown
Contributor Author

@droazen @jamesemery I've made changes, could someone take a look if you get a chance.

@jamesemery jamesemery left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me now 👍 feel free to merge

@lbergelson
lbergelson dismissed droazen’s stale review January 22, 2019 19:10

responded to these comments and James did a re-review

@lbergelson
lbergelson merged commit d3e45c7 into master Jan 22, 2019
@lbergelson
lbergelson deleted the lb_fix_overflow_issue branch January 22, 2019 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants