Skip to content

Add --min-word-length option to ignore short words#3971

Open
ekanshul wants to merge 1 commit into
codespell-project:mainfrom
ekanshul:min-word-length-2211
Open

Add --min-word-length option to ignore short words#3971
ekanshul wants to merge 1 commit into
codespell-project:mainfrom
ekanshul:min-word-length-2211

Conversation

@ekanshul

Copy link
Copy Markdown

Closes #2211

Adds a --min-word-length LENGTH option so that words shorter than LENGTH are not checked. This helps with codebases full of 2-3 letter variable names that would otherwise each need an entry in --ignore-words-list.

Implementation: entries shorter than LENGTH are dropped from the misspelling dictionary right after it is built, so short words are ignored consistently everywhere (file contents, stdin and --check-filenames) with no per-word cost during scanning. The default (0) keeps current behavior.

Example:

$ codespell file.txt
file.txt:1: nwe ==> new
file.txt:1: abandonned ==> abandoned
$ codespell --min-word-length 4 file.txt
file.txt:1: abandonned ==> abandoned

Tests included (test_min_word_length) covering the length boundaries, case-insensitivity and --check-filenames.

Words shorter than the given length are dropped from the misspelling
dictionary up front, so they are ignored both in file contents and
with --check-filenames. This avoids having to list every short
variable name in --ignore-words-list.

Closing: codespell-project#2211
@chuenchen309

Copy link
Copy Markdown

Applied this branch and exercised it rather than just reading it — it works, and filtering at the dictionary level is the right layer. Sharing the checks in case they're useful to a maintainer:

$ cat short.txt
The aas value and the abandonned config.

$ codespell short.txt
short.txt:1: aas ==> ass, as
short.txt:1: abandonned ==> abandoned

$ codespell --min-word-length 4 short.txt
short.txt:1: abandonned ==> abandoned          # short key dropped, long one still caught

The one thing I went looking for was whether len(key) could mean something other than "word length" — a multi-word dictionary key would make --min-word-length 4 compare against the whole phrase rather than a word. It can't: no builtin dictionary has a key containing a space (I checked all of codespell_lib/data/dictionary*.txt — zero). So len(key) is unambiguously the word length here. Worth knowing it holds by data rather than by construction, in case custom -D dictionaries are ever expected to carry phrases.

Filtering the dictionary once after build_dict rather than at each match site also means --check-filenames and the -w path get the same treatment for free, which matches what the PR description claims.

For context on the size of what it buys: 162 of the default dictionary's keys are ≤3 characters, so --min-word-length 4 is a meaningful reduction for a codebase full of short identifiers — the use case in #2211.

(Not a maintainer, just a contributor who was recently in this file.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Whishlist: Optionally ignore short words

3 participants