Ambiguous `docvecs` indexing, documentation missing
#2,097 opened on Jun 20, 2018
Repository metrics
- Stars
- (15,144 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
Description
The tagging system for training corpora appears to result in ambiguous indexing on model.docvecs
Steps/Code/Corpus to Reproduce
This example is a lightly modified version of code snippets from the main demo.
import os
import smart_open
import gensim
# Set file names for train and test data
test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])
lee_train_file = test_data_dir + os.sep + 'lee_background.cor'
def read_corpus(fname, offset = 0):
with smart_open.smart_open(fname, encoding="iso-8859-1") as f:
for i, line in enumerate(f):
yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), [i + offset])
def build_model(corpus):
model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
model.build_vocab(corpus)
model.train(corpus, total_examples=model.corpus_count, epochs=model.epochs)
return model
corpus = list(read_corpus(lee_train_file))
offset_corpus = list(read_corpus(lee_train_file, offset = 100))
model = build_model(corpus)
offset_model = build_model(offset_corpus)
Expected Results
The only difference between the two models is that the training corpora use different sets of tags.
-
The number of training documents is the same for both, so I would expect
len(model.docvecs) == len(offset_model.docvecs) -
offset_corpus.docvecs[0]should throw an error since0is not among its tags. -
OR ... if not (2), then
offset_corpus.docvecs[0]should correspond to the first document, in which caseoffset_corpus.docvecs[0] == offset_corpus.docvecs[100]. Specifically, note thatoffset_corpus.docvecs[399]appears to return a valid result.
Actual Results
None of (1), (2), or (3) hold.
Versions
>>> import platform; print(platform.platform())
Darwin-16.7.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.14.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.0.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.4.0
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0