Skip to content

Add Json integer, remove specialization. - #4739

Merged
trivialfis merged 1 commit into
dmlc:masterfrom
trivialfis:json-integer
Aug 6, 2019
Merged

trivialfis merged 1 commit into
dmlc:masterfrom
trivialfis:json-integer

Conversation

@trivialfis

Copy link
Copy Markdown
Member

Extracted from #4732, which in turns is extracted from #4683 to reduce PR size for quicker review.

@trivialfis

Copy link
Copy Markdown
Member Author

Integer is used to read pointer value

@chyunsu3 chyunsu3 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Thanks for your hard work for implementing JSON serialization so far. I was able to review this pull request (and #4708) with confidence because I previously read your JSON implementation at #3930.

Can you address my comments?

Comment thread src/common/json.cc
Json JsonReader::ParseNumber() {
std::string substr = raw_str_.substr(cursor_.Pos(), kMaxNumLength);
size_t pos = 0;
// Adopted from sajson with some simplifications and small optimizations.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered using dmlc::strtof() to parse floating-point numbers?
https://github.com/dmlc/dmlc-core/blob/808f485387f9a03f78fa9f1159f387d0d91b7a28/include/dmlc/strtonum.h#L99

Some differences:

I think your implementation is reasonable. I'll defer to you to choose between your implementation or dmlc::strtof().

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 The problem is we also support integer parsing. It's not in json specification but added by most of implementations. Also we have lots of integers in tree node index. So I need a function that can determine whether the string is an integer or a float.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trivialfis Got it. In that case, let's just keep JsonReader::ParseNumber() as it is.

In the future, when we decide to move your JSON serializer into dmlc-core, we can discuss how to combine the logic in JsonReader::ParseNumber() and dmlc::ParseFloat() for code de-duplication.

Comment thread src/common/json.cc
#if !defined(__GNUC__) || defined(_WIN32) || defined(__APPLE__)
#define exp10(val) std::pow(10, (val))
#endif // !defined(__GNUC__) || defined(_WIN32) || defined(__APPLE__)
f *= exp10(exponent);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is exp10 faster?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 Sorry I never benchmark this particular function knowing that the newly revised ParseNumber is no longer a bottle neck.

Comment thread src/common/json.cc
Json Json::Load(StringView str, bool ignore_specialization) {
Json Json::Load(StringView str) {
GlobalCLocale guard;
LOG(WARNING) << "Json serialization is still experimental."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing this warning?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 Will put it into Learner::Load and Save. This implementation will also be used to consume __array_interface__ from numpy and __cuda_array_interface__ from cudf and numba and friends.

std::string str = "31.8892";
auto json = Json::Load(StringView{str.c_str(), str.size()});
ASSERT_NEAR(get<JsonNumber>(json), 31.8892f, kRtEps);
{

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Json::Dump(origin, &fout);
fout.close();

buffer = common::LoadSequentialFile(path);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice that common::LoadSequentialFile() has a special logic for POSIX target. Is there significant speed-up when you use posix_fadvise() followed by POSIX read()?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 I will remove it in next PR as it's not really useful for loading model from memory buffer, and conflicts with rabit check point logic.

But to answer your question, yes, it's a huge speed up.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 Looking into those tests, JSON number is much simpler. Like not having f suffix, no inf or NAN.

number

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trivialfis Interesting, so NAN and INF are not allowed in JSON. That's awesome

Comment thread include/xgboost/json.h
@@ -256,15 +262,16 @@ class JsonBoolean : public Value {
};

struct StringView {

@chyunsu3 chyunsu3 Aug 6, 2019 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside: we may want to contribute this class into dmlc-core (dmlc::StringView) in the future.

@trivialfis trivialfis Aug 6, 2019 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcho3 There are quite a lot of requirements to make it really useful. One of them is {from, to}_chars from c++17. I tried to adopt it from libcxx from llvm project, but a few thousands lines of code is unavoidable.

@trivialfis

Copy link
Copy Markdown
Member Author

I previously read your JSON implementation at #3930.

Ah, that was almost a thousand tickets ago. ;-)

@trivialfis

Copy link
Copy Markdown
Member Author

Merging as no additional logic change to original code base.

@trivialfis
trivialfis merged commit 2a4df8e into dmlc:master Aug 6, 2019
@trivialfis

Copy link
Copy Markdown
Member Author

@hcho3 Thanks for the review.

@trivialfis trivialfis mentioned this pull request Aug 8, 2019
5 of 10 tasks
@trivialfis
trivialfis deleted the json-integer branch August 21, 2019 12:58
@lock lock Bot locked as resolved and limited conversation to collaborators Nov 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants