Skip to content

[EPIC] Allow failed worker retry in distributed training #4753

Closed

Description

@chenqin

Fault recovery in native distributed XGB training (not xgb-spark) have been broken for more than a year. Community have seen various of issue like this dmlc/rabit#63

tracking backlogs work from merged prs

Activity

  1. trivialfis commented on Aug 12, 2019

    @trivialfis
    Member

    Looks interesting. ;-)

  2. trams commented on Aug 20, 2019

    @trams
    Contributor

    This looks interesting indeed. One question: "What is native distributed XGB training?" How can I use it? Is it the one which involves a python package + dask?

  3. chenqin commented on Aug 23, 2019

    @chenqin
    ContributorAuthor

    This looks interesting indeed. One question: "What is native distributed XGB training?" How can I use it? Is it the one which involves a python package + dask?

    If you try xgboost with more than one machine. You are already using rabit, we added feature and track changes in progress to xgb layer

    https://www.slideshare.net/ChenQin1/scaling-xgboost

  4. trams commented on Aug 23, 2019

    @trams
    Contributor

    @chenqin, I understand that xgboost uses rabit as AllReduce implementation. I still don't really know what is a native distributed XGB training?
    Does xgboost-spark run native distributed XGB training? If yes then why do we add "native" here? What other distributed XGB training do we have?

    I am new to the project and I am still discovering different ways to use xgboost so this is an honest question.

  5. chenqin commented on Aug 23, 2019

    @chenqin
    ContributorAuthor

    @chenqin, I understand that xgboost uses rabit as AllReduce implementation. I still don't really know what is a native distributed XGB training?
    Does xgboost-spark run native distributed XGB training? If yes then why do we add "native" here? What other distributed XGB training do we have?

    I am new to the project and I am still discovering different ways to use xgboost so this is an honest question.

    In some use cases where user don't have complex feature generating needs, user can launch xgboost native worker (c++) without data processing
    framework. https://github.com/kubeflow/xgboost-operator

  6. chenqin commented on Aug 29, 2019

    @chenqin
    ContributorAuthor

    from description in this thread, yes, dask-xgboost leverage rabit as well.
    #2032

  7. changed the title [-]Allow failed worker retry in distributed training[/-] [+][EPIC] Allow failed worker retry in distributed training[/+] on Sep 19, 2019
  8. chyunsu3 commented on Sep 19, 2019

    @chyunsu3
    Collaborator

    @chenqin It's epic :)

  9. nateagr commented on Dec 11, 2019

    @nateagr
    Contributor

    Hi there! Any update on this epic ? Thanks.

  10. chenqin commented on Dec 14, 2019

    @chenqin
    ContributorAuthor

    Hi there! Any update on this epic ? Thanks.

    Yes, it's going (holiday session slow), still miss the final piece of patch XGBoost-spark after we can get current open pr landed.

  11. trivialfis commented on Sep 27, 2020

    @trivialfis
    Member

    I don't think this is possible in short term. Let's stick with fail all strategy for now. We @hcho3 are thinking about redesigning RABIT from scratch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions