Repository navigation
Add Json integer, remove specialization. - #4739
Conversation
|
Integer is used to read pointer value |
| Json JsonReader::ParseNumber() { | ||
| std::string substr = raw_str_.substr(cursor_.Pos(), kMaxNumLength); | ||
| size_t pos = 0; | ||
| // Adopted from sajson with some simplifications and small optimizations. |
There was a problem hiding this comment.
Have you considered using dmlc::strtof() to parse floating-point numbers?
https://github.com/dmlc/dmlc-core/blob/808f485387f9a03f78fa9f1159f387d0d91b7a28/include/dmlc/strtonum.h#L99
Some differences:
dmlc::strtof()contains more checks for overflow and underflow conditionsdmlc::strtof()supports NAN and INFdmlc::strtof()avoids the use ofexp10andexpby repeating*10:
https://github.com/dmlc/dmlc-core/blob/808f485387f9a03f78fa9f1159f387d0d91b7a28/include/dmlc/strtonum.h#L184-L199
I think your implementation is reasonable. I'll defer to you to choose between your implementation or dmlc::strtof().
There was a problem hiding this comment.
@hcho3 The problem is we also support integer parsing. It's not in json specification but added by most of implementations. Also we have lots of integers in tree node index. So I need a function that can determine whether the string is an integer or a float.
There was a problem hiding this comment.
@trivialfis Got it. In that case, let's just keep JsonReader::ParseNumber() as it is.
In the future, when we decide to move your JSON serializer into dmlc-core, we can discuss how to combine the logic in JsonReader::ParseNumber() and dmlc::ParseFloat() for code de-duplication.
| #if !defined(__GNUC__) || defined(_WIN32) || defined(__APPLE__) | ||
| #define exp10(val) std::pow(10, (val)) | ||
| #endif // !defined(__GNUC__) || defined(_WIN32) || defined(__APPLE__) | ||
| f *= exp10(exponent); |
There was a problem hiding this comment.
@hcho3 Sorry I never benchmark this particular function knowing that the newly revised ParseNumber is no longer a bottle neck.
| Json Json::Load(StringView str, bool ignore_specialization) { | ||
| Json Json::Load(StringView str) { | ||
| GlobalCLocale guard; | ||
| LOG(WARNING) << "Json serialization is still experimental." |
There was a problem hiding this comment.
Why are we removing this warning?
There was a problem hiding this comment.
@hcho3 Will put it into Learner::Load and Save. This implementation will also be used to consume __array_interface__ from numpy and __cuda_array_interface__ from cudf and numba and friends.
| std::string str = "31.8892"; | ||
| auto json = Json::Load(StringView{str.c_str(), str.size()}); | ||
| ASSERT_NEAR(get<JsonNumber>(json), 31.8892f, kRtEps); | ||
| { |
There was a problem hiding this comment.
Take a look at https://github.com/dmlc/dmlc-core/blob/808f485387f9a03f78fa9f1159f387d0d91b7a28/test/unittest/unittest_param.cc#L18 and see if we should adopt some of the test cases.
| Json::Dump(origin, &fout); | ||
| fout.close(); | ||
|
|
||
| buffer = common::LoadSequentialFile(path); |
There was a problem hiding this comment.
I notice that common::LoadSequentialFile() has a special logic for POSIX target. Is there significant speed-up when you use posix_fadvise() followed by POSIX read()?
There was a problem hiding this comment.
@hcho3 I will remove it in next PR as it's not really useful for loading model from memory buffer, and conflicts with rabit check point logic.
But to answer your question, yes, it's a huge speed up.
There was a problem hiding this comment.
@trivialfis Interesting, so NAN and INF are not allowed in JSON. That's awesome
| @@ -256,15 +262,16 @@ class JsonBoolean : public Value { | |||
| }; | |||
|
|
|||
| struct StringView { | |||
There was a problem hiding this comment.
Aside: we may want to contribute this class into dmlc-core (dmlc::StringView) in the future.
There was a problem hiding this comment.
@hcho3 There are quite a lot of requirements to make it really useful. One of them is {from, to}_chars from c++17. I tried to adopt it from libcxx from llvm project, but a few thousands lines of code is unavoidable.
Ah, that was almost a thousand tickets ago. ;-) |
|
Merging as no additional logic change to original code base. |
|
@hcho3 Thanks for the review. |

Extracted from #4732, which in turns is extracted from #4683 to reduce PR size for quicker review.