Skip to content

[jvm-package] jvm test should clean up after themselfs - #4706

Merged
CodingCat merged 1 commit into
dmlc:masterfrom
criteo-forks:clean_jvm_tests
Aug 4, 2019
Merged

CodingCat merged 1 commit into
dmlc:masterfrom
criteo-forks:clean_jvm_tests

Conversation

@trams

@trams trams commented Jul 25, 2019

Copy link
Copy Markdown
Contributor

I noticed that some tests do not fully clean up their tmp files so I fixed that.
Nothing else

@CodingCat

Copy link
Copy Markdown
Member

which test doesn't clean? the PR looks like extracting temp dir creation to a class instead of adding anything?

}

test("test persistence of MLlib pipeline with XGBoostClassificationModel") {
val tempDir = createTmpFolder("MLLib").toFile

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need to create tempDir in each test manually?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know in scalatest different tests are not isolated.
I create a temp folder once per suite. So one need to create temp folders in each test explicitly to make them isolated.

Alternative solution is to change a trait to create a temp folder per test

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but what's the problem with reusing the created dir here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this particular case it is overkill (I am being paranoid). Each test create files with different filenames so tests can reuse the tmpDir. I can change it to reuse the dir

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer keeping writing test easy by forcing devs to remember call createTmpFolder manually,

if reusing the directory doesn't bring any issue, can we just reuse the created one in beforeAll()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trams would you please address this so that we can merge it?

import org.apache.spark.network.util.JavaUtils
import org.scalatest.{BeforeAndAfterAll, FunSuite}

trait TmpFolderPerSuite extends BeforeAndAfterAll { self: FunSuite =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the issue to put it in PerTest?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to have an extra flexibility if we want to have a test which create temp files but we do not need SparkSession

@trams trams left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CheckpointManagerSuite did not clean after itself.
The other modified test were ok but I wanted all tests to use the same trait

}

test("test persistence of MLlib pipeline with XGBoostClassificationModel") {
val tempDir = createTmpFolder("MLLib").toFile

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know in scalatest different tests are not isolated.
I create a temp folder once per suite. So one need to create temp folders in each test explicitly to make them isolated.

Alternative solution is to change a trait to create a temp folder per test

import org.apache.spark.network.util.JavaUtils
import org.scalatest.{BeforeAndAfterAll, FunSuite}

trait TmpFolderPerSuite extends BeforeAndAfterAll { self: FunSuite =>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to have an extra flexibility if we want to have a test which create temp files but we do not need SparkSession

import org.apache.hadoop.fs.{FileSystem, Path}

class CheckpointManagerSuite extends FunSuite with PerTest with BeforeAndAfterAll {
class CheckpointManagerSuite extends FunSuite with TmpFolderPerSuite with PerTest {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may miss something, why this change can address the issue?

is it just moving the beforeAll&afterAll to somewhere else?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change in CheckpointManagerSuite each test created the temp directory in /tmp (on linux) and never removed it.

(Note that PerTest neither did not nor does not now create any temp directories. It just create a SparkSession to use)

After this change CheckpointManagerSuite will include TmpFolderPerSuite .
TmpFolderPerSuite will create a temp directory and then it will be removed.
Also after this change all temp folders are created as children of this root temp folder (see line 35 in this file)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh. yes, we manually created "test" dir there....

@trams

trams commented Jul 28, 2019 via email

Copy link
Copy Markdown
Contributor Author

@trams
trams force-pushed the clean_jvm_tests branch 2 times, most recently from ae9fd46 to da4046b Compare July 31, 2019 02:15
@trams
trams force-pushed the clean_jvm_tests branch from da4046b to a1d9cc3 Compare August 1, 2019 00:31
@trams

trams commented Aug 1, 2019

Copy link
Copy Markdown
Contributor Author

@CodingCat , I updated review as you asked. I also rebased on a fresh master and CI gives a green light.

@CodingCat

Copy link
Copy Markdown
Member

thanks, merged!

@CodingCat
CodingCat merged commit b68de01 into dmlc:master Aug 4, 2019
@trams
trams deleted the clean_jvm_tests branch August 16, 2019 01:28
@lock lock Bot locked as resolved and limited conversation to collaborators Nov 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants