Skip to content

- fix issues with training with external memory on cpu - #4487

Merged
RAMitchell merged 10 commits into
dmlc:masterfrom
sriramch:cpu_extmem
May 29, 2019
Merged

RAMitchell merged 10 commits into
dmlc:masterfrom
sriramch:cpu_extmem

Conversation

@sriramch

Copy link
Copy Markdown
Contributor
  • use the batch size to determine the correct number of rows in a batch
  • use the right number of threads in omp parallalization if the batch size
    is less than the default omp max threads (applicable for the last batch)

@canonizer @RAMitchell @rongou please review.

   - use the batch size to determine the correct number of rows in a batch
   - use the right number of threads in omp parallalization if the batch size
     is less than the default omp max threads (applicable for the last batch)
Comment thread src/common/hist_util.cc Outdated
size_t prev_sum = 0;

for (const auto &batch : p_fmat->GetRowBatches()) {
auto bsize = static_cast<omp_ulong>(batch.Size());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batch_size?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

Comment thread src/common/hist_util.cc Outdated
cut.Init(p_fmat, max_num_bins);
const int32_t nthread = omp_get_max_threads();
// const int nthread = 1;
size_t nthread = omp_get_max_threads();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this const?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is being changed later

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment below. Probably safer to declare another variable there.

Comment thread src/common/hist_util.cc Outdated

for (const auto &batch : p_fmat->GetRowBatches()) {
auto bsize = static_cast<omp_ulong>(batch.Size());
nthread = std::min(bsize, nthread);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this make nthread too small if batch size fluctuates? Probably should declare a new variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is going to be pertinent only for the last batch. for the ones before that, nthread will be the default omp max. for the last batch, if it happens to be lesser (typically ~ 64), then it scales it down. the nthread is mostly pegged to the batch size in this block.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From tracing through some tests, the batch size doesn't seem to be constant and can fluctuate up and down. Seems safer to not make nthread a running min.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have left the number of threads as is and adjusted the partial sums if the batch size is smaller than the number of threads.

the test should also now test all possible scenarios with batch size and thread pool size.

Comment thread src/common/hist_util.cc Outdated

for (const auto &batch : p_fmat->GetRowBatches()) {
auto bsize = static_cast<omp_ulong>(batch.Size());
nthread = std::min(bsize, nthread);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From tracing through some tests, the batch size doesn't seem to be constant and can fluctuate up and down. Seems safer to not make nthread a running min.

Comment thread src/common/hist_util.cc Outdated
cut.Init(p_fmat, max_num_bins);
const int32_t nthread = omp_get_max_threads();
// const int nthread = 1;
size_t nthread = omp_get_max_threads();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment below. Probably safer to declare another variable there.

sriramch added 2 commits May 22, 2019 14:04
…eads

- augment tests such that we can test all scenarios (batch size <, >, = number of threads)
Comment thread src/common/hist_util.cc Outdated
p_part[0] = prev_sum;
for (int32_t i = 1; i < nthread; ++i) {
p_part[i] = p_part[i - 1] + row_ptr[rbegin + i*block_size];
if (batch_size < nthread) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this also solve your problem?

const size_t nthread = std::min(batch_size, omp_get_max_threads());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure - that is how i'd it initially ca904a8. then, i realized that not all thread blocks are pegged to the batch size. hence, i left the desired number of threads as-is.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the PR does not work if you do this, or there are performance side-effects? I am asking because I prefer simple solutions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the original intent was that the number of threads do not change, and i tried to preserve that intent. if the number of threads is pegged to the batch size everywhere, then this change would have preserved that intent. however, it is also pegged to the number of bins later. hence, if you'd a large number of bins and a smaller batch size, then that could be impacted with a change like this. hence, i decided to leave it as-is.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So can't you just uniquely define the number of threads before each parallel block?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, there could be multiple ways to solve this. doing this would also mean that someone would have to remember to readjust the number of threads every time a block here is added/moved; when the block logic changes to parallelize on something other than batch_size or number of bins etc.

Comment thread src/common/hist_util.cc
MemStackAllocator<size_t, 128> partial_sums(nthread);
size_t* p_part = partial_sums.Get();

size_t block_size = batch.Size() / nthread;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use batch.Size()? Is this expected to change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since it is used in multiple places, i only invoke this once and use that variable elsewhere.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to @RAMitchell's suggestion. batch.Size() is a simple property getter (https://github.com/dmlc/xgboost/blob/master/include/xgboost/data.h#L280), which will most likely be inlined by the compiler.

@RAMitchell RAMitchell May 25, 2019 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batch.Size() is the "most true" source of this information so you should use it directly if there is no performance penalty. This may seem pedantic but adding layers between affects readability and creates assumptions. For example that batch.Size() has not changed since you first called it - in this case maybe a reasonable assumption, but in general this as a reason not to do this.

Hope this makes sense.

@sriramch sriramch May 25, 2019 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not want to make assumptions on performance implications without profiling first. hence, this wasn't done due to performance reasons.

i'm trying to understand how using a variable named batch_size as opposed to using batch.Size() impacts readability, given that this is used in multiple places in the code block. by using the variable here, we are also conveying the intent that a new value isn't used at the call site and that a constant value computed at the beginning of the block is used. further, this also localizes the means to compute the batch size in a single place, such that if the way to compute the batch size changes later, the change is limited to a single place.

@canonizer canonizer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a large number of changes not related to fixing the bug, such as replacing batch.Size() with batch_size. Please limit it to fixing the bug and testing it.

Comment thread tests/cpp/common/test_column_matrix.cc Outdated
// in the thread pool
TEST_HIST_IDX_MATRIX(20)
TEST_HIST_IDX_MATRIX(30)
TEST_HIST_IDX_MATRIX(40)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace the macro with a function or a loop. This will make the code more readable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll change this to a function

Comment thread src/common/hist_util.cc Outdated
}

index.resize(row_ptr.back());
index.resize(row_ptr[rbegin + batch_size]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change necessary?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the most important change in this pr. the index and the row_ptr is incrementally built with every batch. without this index will be an empty vector, if it has multiple batches.

Comment thread src/common/hist_util.cc
cut.Init(p_fmat, max_num_bins);
const int32_t nthread = omp_get_max_threads();
// const int nthread = 1;
const size_t nthread = omp_get_max_threads();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the change of the type of nthread necessary?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the max_thread count cannot be a negative number, i made the return type reflect that.

Comment thread src/common/hist_util.cc
MemStackAllocator<size_t, 128> partial_sums(nthread);
size_t* p_part = partial_sums.Get();

size_t block_size = batch.Size() / nthread;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to @RAMitchell's suggestion. batch.Size() is a simple property getter (https://github.com/dmlc/xgboost/blob/master/include/xgboost/data.h#L280), which will most likely be inlined by the compiler.

@sriramch

Copy link
Copy Markdown
Contributor Author

w.r.t. this:

#4487 (comment)

the batch size is referenced in multiple places in this block. the change wasn't done to somehow make it performant. it was simply done to localize getting the batch size in a single place, such that it is easy to change in the future, in case the means to retrieve it changes later.

@sriramch

Copy link
Copy Markdown
Contributor Author

close and reopen to fix this transient error.

@sriramch sriramch closed this May 24, 2019
@sriramch sriramch reopened this May 24, 2019
@sriramch sriramch closed this May 24, 2019
@sriramch sriramch reopened this May 24, 2019
@chyunsu3

Copy link
Copy Markdown
Collaborator

@sriramch Let's ignore the Mac OSX tests for the time being. Somehow we are getting the error

MD5MismatchError: Conda detected a mismatch between the expected content and downloaded content

@chyunsu3

Copy link
Copy Markdown
Collaborator

@sriramch Mac OSX tests are no longer required for merging. Please ignore Mac OSX tests until I find time to fix them

@sriramch

Copy link
Copy Markdown
Contributor Author

@hcho3 the issue seems to be with python tests. see here - https://travis-ci.org/dmlc/xgboost/builds/536902957

it has happened twice - similar issue. wouldn't this be deemed a build failure then?

@chyunsu3

chyunsu3 commented May 24, 2019 •

Copy link
Copy Markdown
Collaborator

@sriramch No, because the failure is due to failed installation of Miniconda Python environment in Mac OSX. Look for the error message

MD5MismatchError: Conda detected a mismatch between the expected content and downloaded content

sriramch added 2 commits May 28, 2019 15:22
- recalibrate the number of threads based on the batch size like the initial commit
@RAMitchell
RAMitchell merged commit a3fedbe into dmlc:master May 29, 2019
@sriramch
sriramch deleted the cpu_extmem branch June 27, 2019 18:58
@lock lock Bot locked as resolved and limited conversation to collaborators Sep 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants