Repository navigation
- fix issues with training with external memory on cpu - #4487
Conversation
- use the batch size to determine the correct number of rows in a batch
- use the right number of threads in omp parallalization if the batch size
is less than the default omp max threads (applicable for the last batch)
| size_t prev_sum = 0; | ||
|
|
||
| for (const auto &batch : p_fmat->GetRowBatches()) { | ||
| auto bsize = static_cast<omp_ulong>(batch.Size()); |
| cut.Init(p_fmat, max_num_bins); | ||
| const int32_t nthread = omp_get_max_threads(); | ||
| // const int nthread = 1; | ||
| size_t nthread = omp_get_max_threads(); |
There was a problem hiding this comment.
it is being changed later
There was a problem hiding this comment.
See comment below. Probably safer to declare another variable there.
|
|
||
| for (const auto &batch : p_fmat->GetRowBatches()) { | ||
| auto bsize = static_cast<omp_ulong>(batch.Size()); | ||
| nthread = std::min(bsize, nthread); |
There was a problem hiding this comment.
Wouldn't this make nthread too small if batch size fluctuates? Probably should declare a new variable.
There was a problem hiding this comment.
i think this is going to be pertinent only for the last batch. for the ones before that, nthread will be the default omp max. for the last batch, if it happens to be lesser (typically ~ 64), then it scales it down. the nthread is mostly pegged to the batch size in this block.
There was a problem hiding this comment.
From tracing through some tests, the batch size doesn't seem to be constant and can fluctuate up and down. Seems safer to not make nthread a running min.
There was a problem hiding this comment.
i have left the number of threads as is and adjusted the partial sums if the batch size is smaller than the number of threads.
the test should also now test all possible scenarios with batch size and thread pool size.
|
|
||
| for (const auto &batch : p_fmat->GetRowBatches()) { | ||
| auto bsize = static_cast<omp_ulong>(batch.Size()); | ||
| nthread = std::min(bsize, nthread); |
There was a problem hiding this comment.
From tracing through some tests, the batch size doesn't seem to be constant and can fluctuate up and down. Seems safer to not make nthread a running min.
| cut.Init(p_fmat, max_num_bins); | ||
| const int32_t nthread = omp_get_max_threads(); | ||
| // const int nthread = 1; | ||
| size_t nthread = omp_get_max_threads(); |
There was a problem hiding this comment.
See comment below. Probably safer to declare another variable there.
…eads - augment tests such that we can test all scenarios (batch size <, >, = number of threads)
…ation fails on win64
…re than a single batch of data; vary threads so that it ends up being <, >, = to the last batch size
| p_part[0] = prev_sum; | ||
| for (int32_t i = 1; i < nthread; ++i) { | ||
| p_part[i] = p_part[i - 1] + row_ptr[rbegin + i*block_size]; | ||
| if (batch_size < nthread) { |
There was a problem hiding this comment.
Would this also solve your problem?
const size_t nthread = std::min(batch_size, omp_get_max_threads());
There was a problem hiding this comment.
sure - that is how i'd it initially ca904a8. then, i realized that not all thread blocks are pegged to the batch size. hence, i left the desired number of threads as-is.
There was a problem hiding this comment.
So the PR does not work if you do this, or there are performance side-effects? I am asking because I prefer simple solutions.
There was a problem hiding this comment.
the original intent was that the number of threads do not change, and i tried to preserve that intent. if the number of threads is pegged to the batch size everywhere, then this change would have preserved that intent. however, it is also pegged to the number of bins later. hence, if you'd a large number of bins and a smaller batch size, then that could be impacted with a change like this. hence, i decided to leave it as-is.
There was a problem hiding this comment.
So can't you just uniquely define the number of threads before each parallel block?
There was a problem hiding this comment.
sure, there could be multiple ways to solve this. doing this would also mean that someone would have to remember to readjust the number of threads every time a block here is added/moved; when the block logic changes to parallelize on something other than batch_size or number of bins etc.
| MemStackAllocator<size_t, 128> partial_sums(nthread); | ||
| size_t* p_part = partial_sums.Get(); | ||
|
|
||
| size_t block_size = batch.Size() / nthread; |
There was a problem hiding this comment.
Why not use batch.Size()? Is this expected to change?
There was a problem hiding this comment.
since it is used in multiple places, i only invoke this once and use that variable elsewhere.
There was a problem hiding this comment.
+1 to @RAMitchell's suggestion. batch.Size() is a simple property getter (https://github.com/dmlc/xgboost/blob/master/include/xgboost/data.h#L280), which will most likely be inlined by the compiler.
There was a problem hiding this comment.
batch.Size() is the "most true" source of this information so you should use it directly if there is no performance penalty. This may seem pedantic but adding layers between affects readability and creates assumptions. For example that batch.Size() has not changed since you first called it - in this case maybe a reasonable assumption, but in general this as a reason not to do this.
Hope this makes sense.
There was a problem hiding this comment.
i do not want to make assumptions on performance implications without profiling first. hence, this wasn't done due to performance reasons.
i'm trying to understand how using a variable named batch_size as opposed to using batch.Size() impacts readability, given that this is used in multiple places in the code block. by using the variable here, we are also conveying the intent that a new value isn't used at the call site and that a constant value computed at the beginning of the block is used. further, this also localizes the means to compute the batch size in a single place, such that if the way to compute the batch size changes later, the change is limited to a single place.
canonizer
left a comment
There was a problem hiding this comment.
This pull request contains a large number of changes not related to fixing the bug, such as replacing batch.Size() with batch_size. Please limit it to fixing the bug and testing it.
| // in the thread pool | ||
| TEST_HIST_IDX_MATRIX(20) | ||
| TEST_HIST_IDX_MATRIX(30) | ||
| TEST_HIST_IDX_MATRIX(40) |
There was a problem hiding this comment.
Please replace the macro with a function or a loop. This will make the code more readable.
There was a problem hiding this comment.
i'll change this to a function
| } | ||
|
|
||
| index.resize(row_ptr.back()); | ||
| index.resize(row_ptr[rbegin + batch_size]); |
There was a problem hiding this comment.
Is this change necessary?
There was a problem hiding this comment.
this is the most important change in this pr. the index and the row_ptr is incrementally built with every batch. without this index will be an empty vector, if it has multiple batches.
| cut.Init(p_fmat, max_num_bins); | ||
| const int32_t nthread = omp_get_max_threads(); | ||
| // const int nthread = 1; | ||
| const size_t nthread = omp_get_max_threads(); |
There was a problem hiding this comment.
Is the change of the type of nthread necessary?
There was a problem hiding this comment.
since the max_thread count cannot be a negative number, i made the return type reflect that.
| MemStackAllocator<size_t, 128> partial_sums(nthread); | ||
| size_t* p_part = partial_sums.Get(); | ||
|
|
||
| size_t block_size = batch.Size() / nthread; |
There was a problem hiding this comment.
+1 to @RAMitchell's suggestion. batch.Size() is a simple property getter (https://github.com/dmlc/xgboost/blob/master/include/xgboost/data.h#L280), which will most likely be inlined by the compiler.
|
w.r.t. this: the batch size is referenced in multiple places in this block. the change wasn't done to somehow make it performant. it was simply done to localize getting the batch size in a single place, such that it is easy to change in the future, in case the means to retrieve it changes later. |
|
close and reopen to fix this transient error. |
|
@sriramch Let's ignore the Mac OSX tests for the time being. Somehow we are getting the error |
|
@sriramch Mac OSX tests are no longer required for merging. Please ignore Mac OSX tests until I find time to fix them |
|
@hcho3 the issue seems to be with python tests. see here - https://travis-ci.org/dmlc/xgboost/builds/536902957 it has happened twice - similar issue. wouldn't this be deemed a build failure then? |
|
@sriramch No, because the failure is due to failed installation of Miniconda Python environment in Mac OSX. Look for the error message |
- recalibrate the number of threads based on the batch size like the initial commit
is less than the default omp max threads (applicable for the last batch)
@canonizer @RAMitchell @rongou please review.