piskvorky/gensim

bug in gensim.summarization.mz_entropy.mz_keywords

Open

#2,523 opened on Jun 10, 2019

View on GitHub
 (1 comment) (1 reaction) (0 assignees)Python (4,349 forks)batch import
Hacktoberfestbugdifficulty easygood first issueimpact LOWreach LOW

Repository metrics

Stars
 (15,144 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Problem statement:

It seems to be a bug if the text is too short and number of words is lower than blocksize. In my case the values were: n_words (232.0) and blocksize (1024).

Log:

gensim\summarization\mz_entropy.py:127: RuntimeWarning: invalid value encountered in double_scalars
  - __log_combinations(n_words, blocksize)

Dirty solution:

Override blocksize value from the default 1024 to something lower:

mz_keywords(text, blocksize=128)

Contributor guide

Research direction
Inspect the code in gensim/summarization/mz entropy.py around line 127. The issue occurs when n words is less than blocksize, causing a runtime warning. Modify the logic to handle short texts gracefully, e.g., by adjusting blocksize dynamically or returning an empty result.
Tech stack
python
Domain
machine learning
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75