Repository navigation
[jvm-package] jvm test should clean up after themselfs - #4706
Conversation
|
which test doesn't clean? the PR looks like extracting temp dir creation to a class instead of adding anything? |
| } | ||
|
|
||
| test("test persistence of MLlib pipeline with XGBoostClassificationModel") { | ||
| val tempDir = createTmpFolder("MLLib").toFile |
There was a problem hiding this comment.
why we need to create tempDir in each test manually?
There was a problem hiding this comment.
As far as I know in scalatest different tests are not isolated.
I create a temp folder once per suite. So one need to create temp folders in each test explicitly to make them isolated.
Alternative solution is to change a trait to create a temp folder per test
There was a problem hiding this comment.
but what's the problem with reusing the created dir here?
There was a problem hiding this comment.
In this particular case it is overkill (I am being paranoid). Each test create files with different filenames so tests can reuse the tmpDir. I can change it to reuse the dir
There was a problem hiding this comment.
I would prefer keeping writing test easy by forcing devs to remember call createTmpFolder manually,
if reusing the directory doesn't bring any issue, can we just reuse the created one in beforeAll()?
There was a problem hiding this comment.
@trams would you please address this so that we can merge it?
| import org.apache.spark.network.util.JavaUtils | ||
| import org.scalatest.{BeforeAndAfterAll, FunSuite} | ||
|
|
||
| trait TmpFolderPerSuite extends BeforeAndAfterAll { self: FunSuite => |
There was a problem hiding this comment.
what's the issue to put it in PerTest?
There was a problem hiding this comment.
I wanted to have an extra flexibility if we want to have a test which create temp files but we do not need SparkSession
trams
left a comment
There was a problem hiding this comment.
CheckpointManagerSuite did not clean after itself.
The other modified test were ok but I wanted all tests to use the same trait
| } | ||
|
|
||
| test("test persistence of MLlib pipeline with XGBoostClassificationModel") { | ||
| val tempDir = createTmpFolder("MLLib").toFile |
There was a problem hiding this comment.
As far as I know in scalatest different tests are not isolated.
I create a temp folder once per suite. So one need to create temp folders in each test explicitly to make them isolated.
Alternative solution is to change a trait to create a temp folder per test
| import org.apache.spark.network.util.JavaUtils | ||
| import org.scalatest.{BeforeAndAfterAll, FunSuite} | ||
|
|
||
| trait TmpFolderPerSuite extends BeforeAndAfterAll { self: FunSuite => |
There was a problem hiding this comment.
I wanted to have an extra flexibility if we want to have a test which create temp files but we do not need SparkSession
| import org.apache.hadoop.fs.{FileSystem, Path} | ||
|
|
||
| class CheckpointManagerSuite extends FunSuite with PerTest with BeforeAndAfterAll { | ||
| class CheckpointManagerSuite extends FunSuite with TmpFolderPerSuite with PerTest { |
There was a problem hiding this comment.
I may miss something, why this change can address the issue?
is it just moving the beforeAll&afterAll to somewhere else?
There was a problem hiding this comment.
Before this change in CheckpointManagerSuite each test created the temp directory in /tmp (on linux) and never removed it.
(Note that PerTest neither did not nor does not now create any temp directories. It just create a SparkSession to use)
After this change CheckpointManagerSuite will include TmpFolderPerSuite .
TmpFolderPerSuite will create a temp directory and then it will be removed.
Also after this change all temp folders are created as children of this root temp folder (see line 35 in this file)
There was a problem hiding this comment.
oh. yes, we manually created "test" dir there....
|
Yes, I will address it
…On Sun, Jul 28, 2019, 1:39 PM Nan Zhu ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In
jvm-packages/xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/PersistenceSuite.scala
<#4706 (comment)>:
> @@ -124,6 +98,8 @@ class PersistenceSuite extends FunSuite with PerTest with BeforeAndAfterAll {
}
test("test persistence of MLlib pipeline with XGBoostClassificationModel") {
+ val tempDir = createTmpFolder("MLLib").toFile
@trams <https://github.com/trams> would you please address this so that
we can merge it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4706?email_source=notifications&email_token=AABZ2PAJAAWCZ7YOIX4T4QTQBX7XJA5CNFSM4IGXIEV2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB7ZA4KI#discussion_r308017204>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABZ2PCIS32N7OQGTWLOPYTQBX7XJANCNFSM4IGXIEVQ>
.
|
ae9fd46 to
da4046b
Compare
|
@CodingCat , I updated review as you asked. I also rebased on a fresh master and CI gives a green light. |
|
thanks, merged! |
I noticed that some tests do not fully clean up their tmp files so I fixed that.
Nothing else