piskvorky/gensim

MemoryError when using ldaseqmodel.doc_topics

Open

#2,499 opened on May 23, 2019

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Python (4,349 forks)batch import
Hacktoberfestbugimpact LOWneed inforeach LOW

Repository metrics

Stars
 (15,144 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

I am trying to create a matrix/data frame with the topic distribution over documents for all documents, since these are needed to find what documents have the highest prevalence of a given topic. However, I get a MemoryError when I use ldaseqmodel.doc_topics().

I am using a list comprehension to get the topic distributions for each document.

prop = [ldaseq.doc_topics(n) for n in np.arange(start,end,1)]

I get the following error:

 File "/home/jnava/utilities/anaconda3/lib/python3.7/site-packages/gensim/models/ldaseqmodel.py", line 575, in doc_topics
    doc_topic = np.copy(self.gammas)
 File "/home/jnava/utilities/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 733, in copy
    return array(a, order=order, copy=True)
MemoryError 

I have broken up the task into chunks such that I am only doing a batch of n documents in a list comprehension, say n=500, instead of all documents in a single list comprehension. This works sometimes, but not with all data sets/models.

I have also tried a for loop and get the same error.

Contributor guide