LdaMulticore and OpenMP BLAS

Many python installations these days use a version of numpy linked with a highly optimized BLAS that, by default, uses as many cores as are available. This means that running LdaMulticore with workers=5 on a six-core machine ends up trying to use 30 cores, with a pretty serious hit to performance.

Take this test program:

from gensim.corpora import Dictionary
from gensim.models import LdaModel, LdaMulticore
from nltk.corpus import brown
from time import perf_counter

vocab = Dictionary(brown.words(f) for f in brown.fileids())
vocab.filter_extremes(no_below=50, no_above=0.5)
corpus = [vocab.doc2bow(brown.words(f)) for f in brown.fileids()]

start = perf_counter()
lda = LdaModel(corpus=corpus, id2word=vocab, num_topics=100, chunksize=50, passes=5)
print('%15s %5.2f'%('LdaModel', perf_counter()-start))

start = perf_counter()
lda = LdaMulticore(corpus=corpus, id2word=vocab, num_topics=100, chunksize=50, passes=5,
                                     workers=5)
print('%15s %5.2f'%('LdaMulticore', perf_counter()-start))

Run on a six-core CPU using the Intel Python distribution, I get:

       LdaModel 10.38
   LdaMulticore 14.24

But, with MKL_NUM_THREADS set to 1 to disable parallelism inside numpy, I get:

       LdaModel 20.46
   LdaMulticore  4.85

Using Intel's parallel MKL BLAS, LdaModel is actually faster than LdaMulticore, though LdaMulticore with a single-threaded BLAS is by far the fastest.

As far as I can tell there isn't any portable and reliable way to control how many threads numpy's numeric core might be using, so for now maybe the best thing to do is to add a warning about this to the documentation.