FileAlreadyExistException when writing tachyon table #339
+85
−31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
June 10, 2014 09:04
|
Can one of the admins verify this patch? |
|
yes. This fix is quite important in the speculative run. Otherwise, 2 conflict writes are working on the same target. any idea? |
added 2 commits
September 3, 2014 12:01
459d030 to
90def52
Compare
90def52 to
d159007
Compare
|
@aarondav can you take a look at this? |
|
LGTM, @rxin please merge! |
rxin
added a commit
that referenced
this pull request
Oct 2, 2014
FileAlreadyExistException when writing tachyon table
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
currently Shark write data into Tachyon table directly by RAWTable interface in Tachyon.
when speculation execution is on, it throws FileAlreadyExistException because two tasks attempts to write same partition into Tachyon table
the approach to solve the problem is first write output of each task into a temp folder, then commit the result when the task ends. just like write to HDFS.