piskvorky/gensim

Ambiguous `docvecs` indexing, documentation missing

Open

#2,097 opened on Jun 20, 2018

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (4,349 forks)batch import
Hacktoberfestdifficulty easydocumentation

Repository metrics

Stars
 (15,144 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Description

The tagging system for training corpora appears to result in ambiguous indexing on model.docvecs

Steps/Code/Corpus to Reproduce

This example is a lightly modified version of code snippets from the main demo.

import os
import smart_open
import gensim

# Set file names for train and test data
test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])
lee_train_file = test_data_dir + os.sep + 'lee_background.cor'

def read_corpus(fname, offset = 0):
    with smart_open.smart_open(fname, encoding="iso-8859-1") as f:
        for i, line in enumerate(f):
            yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), [i + offset])

def build_model(corpus):
    model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
    model.build_vocab(corpus)
    model.train(corpus, total_examples=model.corpus_count, epochs=model.epochs)
    return model

corpus = list(read_corpus(lee_train_file))
offset_corpus = list(read_corpus(lee_train_file, offset = 100))

model = build_model(corpus)
offset_model = build_model(offset_corpus)

Expected Results

The only difference between the two models is that the training corpora use different sets of tags.

  1. The number of training documents is the same for both, so I would expect len(model.docvecs) == len(offset_model.docvecs)

  2. offset_corpus.docvecs[0] should throw an error since 0 is not among its tags.

  3. OR ... if not (2), then offset_corpus.docvecs[0] should correspond to the first document, in which case offset_corpus.docvecs[0] == offset_corpus.docvecs[100]. Specifically, note that offset_corpus.docvecs[399] appears to return a valid result.

Actual Results

None of (1), (2), or (3) hold.

Versions

>>> import platform; print(platform.platform())
Darwin-16.7.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.14.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.0.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.4.0
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0

Contributor guide