Race condition in org.bytedeco.javacpp.Loader with multiple JVMs on the same machine
#608 opened on Sep 23, 2022
Repository metrics
- Stars
- (4,279 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
Hi,
We recently hit this race condition issue with javacpp.Loader class. The issue happens when you have multiple JVMs running on the same machine, like in Spark.
It manifests as errors like below when trying to load models using the library.
Caused by: java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
at java.lang.Runtime.loadLibrary0(Runtime.java:872)
at java.lang.System.loadLibrary(System.java:1124)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1738)
at org.bytedeco.javacpp.Loader.load(Loader.java:1345)
at org.bytedeco.javacpp.Loader.load(Loader.java:1157)
at org.bytedeco.javacpp.Loader.load(Loader.java:1133)
at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.bytedeco.javacpp.Loader.load(Loader.java:1212)
at org.bytedeco.javacpp.Loader.load(Loader.java:1157)
at org.bytedeco.javacpp.Loader.load(Loader.java:1149)
at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:140)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.tensorflow.Graph.<clinit>(Graph.java:1341)
....
Caused by: java.lang.UnsatisfiedLinkError: /user/home/.javacpp/cache/jarname/org/tensorflow/internal/c_api/linux-x86_64/libjnitensorflow.so: libtensorflow_cc.so.2: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
at java.lang.Runtime.load0(Runtime.java:811)
at java.lang.System.load(System.java:1088)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1685)
... 104 more
Looking at the Loader code, the file lock only lock within cacheResource method: https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Loader.java#L571-L697
And there are logics like deleting the file within the cacheResource method. (although I still don't understand why it enters into the code block at all if others already cached the file)
file.delete();
extractResource(resourceURL, file, null, null, true);
file.setLastModified(timestamp);
If another JVM on the same machine using the same cacheDir (by default /user/home/.javacpp/cache) try to loadLibrary, it may find the file got deleted by another JVM since there is no file lock here.