bytedeco/javacpp

Race condition in org.bytedeco.javacpp.Loader with multiple JVMs on the same machine

Open

#608 opened on Sep 23, 2022

View on GitHub
 (10 comments) (0 reactions) (0 assignees)Java (620 forks)batch import
bughelp wanted

Repository metrics

Stars
 (4,279 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Hi,

We recently hit this race condition issue with javacpp.Loader class. The issue happens when you have multiple JVMs running on the same machine, like in Spark.

It manifests as errors like below when trying to load models using the library.

Caused by: java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
	at java.lang.Runtime.loadLibrary0(Runtime.java:872)
	at java.lang.System.loadLibrary(System.java:1124)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1738)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1345)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1157)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1133)
	at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1212)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1157)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1149)
	at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
	at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:140)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.tensorflow.Graph.<clinit>(Graph.java:1341)
        ....
Caused by: java.lang.UnsatisfiedLinkError: /user/home/.javacpp/cache/jarname/org/tensorflow/internal/c_api/linux-x86_64/libjnitensorflow.so: libtensorflow_cc.so.2: cannot open shared object file: No such file or directory
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:811)
	at java.lang.System.load(System.java:1088)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1685)
	... 104 more

Looking at the Loader code, the file lock only lock within cacheResource method: https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Loader.java#L571-L697 And there are logics like deleting the file within the cacheResource method. (although I still don't understand why it enters into the code block at all if others already cached the file)

                        file.delete();
                        extractResource(resourceURL, file, null, null, true);
                        file.setLastModified(timestamp);

If another JVM on the same machine using the same cacheDir (by default /user/home/.javacpp/cache) try to loadLibrary, it may find the file got deleted by another JVM since there is no file lock here.

Contributor guide