Use netty maxDirectMemory instead of DirectMemoryUtils #2989
Conversation
3394d6e to
ed3f7d9
Compare
dlg99
left a comment
There was a problem hiding this comment.
I don't think this is the right change. As I understand you have a situation when direct memory usage results in problems for other processes, possibly linux's OOM killer kicking in.
https://netty.io/4.1/xref/io/netty/util/internal/PlatformDependent.html#L153 describes how PlatformDependent.maxDirectMemory is controlled and used in netty. It doesn't limit direct memory for the JVM, -XX:MaxDirectMemorySize jvm parameter does.
The idea is to set -XX:MaxDirectMemorySize for the JVM and possibly limit (up to that max) for netty,
I suggest taking a look at the shell scripts for BK for examples of -XX:MaxDirectMemorySize usage and setting it to the appropriate value.
Please let me know if I misunderstood something.
.
|
@dlg99 Hi, We have a bookkeeper server, which is conifugred as "-Xmx6G -XDirectMemory4G". And we are using huawei jdk11. |
|
Just switching to PlatformDependent.maxDirectMemory() feels like a misuse of it plus it might not work for some people if they e.g. set |
ed3f7d9 to
008a7ff
Compare
|
@dlg99 You are right. set |
1729509 to
d67d2ed
Compare
|
rerun failure checks |
eolivelli
left a comment
There was a problem hiding this comment.
Did you copy paste the code from Netty?
Yes, I did it because that these are more genernal, and netty is under apache license. |
|
What about sending a PR to Netty in order to expose such value? I would prefer to not copy paste, because in Netty they are maintaining that code and adapt it to new Java versions, and we will keep this version forever |
|
@eolivelli Thanks for your advice, I will try |
|
@eolivelli @dlg99 netty/netty#12118 is merged. I will reopen this PR when we update to netty next version. |
d67d2ed to
4b27d0e
Compare
4b27d0e to
7058a17
Compare
|
@eolivelli @dlg99 @merlimat @pkumar-singh PTAL again. Netty has exposed a util method for user to get maxDirectoryMemory. See netty/netty#12118 |
|
@eolivelli @dlg99 @merlimat @pkumar-singh @nicoloboschi |
### Motivation - `PlatformDependent.maxDirectMemory()` can be inaccurate if `io.netty.maxDirectMemory` are setted - Bookkeeper's `DirectMemoryUtils` is not worked within some jdk releases. In netty 4.1.75, they introduced a new method `PlatformDependent.estimateMaxDirectMemory` to help users get maxDirectMemory. Since netty's this method works well on many jdk releases, use this to replace below two. PS: `DirectMemoryUtils` has been removed from bookkeeper in apache/bookkeeper#2989 ### Modifications - use `PlatformDependent.estimateMaxDirectMemory` instead
### Motivation #2989 ### Modification Use estimateMaxDirectMemory instead of maxDirectMemory
### Motivation Our `DirectMemoryUtils` has huge limit, it can't work well with other jvm. The Netty `PlatformDependent.maxDirectMemory();` is more generic. ### Changes Use `PlatformDependent.maxDirectMemory();` instead of `DirectMemoryUtils` Reviewers: Enrico Olivelli <[email protected]>, Andrey Yegorov <None>, Matteo Merli <[email protected]>, Nicolò Boschi <[email protected]> This closes apache#2989 from Shoothzj/direct-memory
### Motivation apache#2989 ### Modification Use estimateMaxDirectMemory instead of maxDirectMemory
Motivation
Our
DirectMemoryUtilshas huge limit, it can't work well with other jvm. The NettyPlatformDependent.maxDirectMemory();is more generic.Changes
Use
PlatformDependent.maxDirectMemory();instead ofDirectMemoryUtils