Repository metrics
- Stars
- (15,144 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
I am trying to create a matrix/data frame with the topic distribution over documents for all documents, since these are needed to find what documents have the highest prevalence of a given topic. However, I get a MemoryError when I use ldaseqmodel.doc_topics().
I am using a list comprehension to get the topic distributions for each document.
prop = [ldaseq.doc_topics(n) for n in np.arange(start,end,1)]
I get the following error:
File "/home/jnava/utilities/anaconda3/lib/python3.7/site-packages/gensim/models/ldaseqmodel.py", line 575, in doc_topics
doc_topic = np.copy(self.gammas)
File "/home/jnava/utilities/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py", line 733, in copy
return array(a, order=order, copy=True)
MemoryError
I have broken up the task into chunks such that I am only doing a batch of n documents in a list comprehension, say n=500, instead of all documents in a single list comprehension. This works sometimes, but not with all data sets/models.
I have also tried a for loop and get the same error.