Skip to content

Keep --ignore-regex from shifting match offsets#3980

Open
chuenchen309 wants to merge 1 commit into
codespell-project:mainfrom
chuenchen309:fix/ignore-regex-offset-shift
Open

Keep --ignore-regex from shifting match offsets#3980
chuenchen309 wants to merge 1 commit into
codespell-project:mainfrom
chuenchen309:fix/ignore-regex-offset-shift

Conversation

@chuenchen309

Copy link
Copy Markdown

_ignore_word_sub() replaces each ignored region with a single space, which changes the length of the text. Matches are then found in that substituted text, but several checks index back into the original line:

  • the escape-sequence check reads line[match.start() - 1] (_codespell.py:1055)
  • --ignore-sic matches the line from match.end() (:1067)
  • the interactive context display slices the line (:1079:825)

So any ignored region longer than one character shifts every offset after it, and those checks read the wrong place. The visible result is a false positive caused by an --ignore-regex that doesn't overlap the reported word at all.

Reproducing

$ printf "var = 'IGNOREME \\\\nwe are here'\n" > esc.txt
$ codespell esc.txt                          # quiet: the \n escape suppresses "nwe"
$ echo $?
0
$ codespell --ignore-regex 'IGNOREME' esc.txt
esc.txt:1: nwe ==> new                       # false positive
$ echo $?
65

IGNOREME is 8 characters and becomes 1, so nwe is found at offset 10 of the substituted text while it really lives at 17. The escape check reads line[9] ('N') instead of line[16] ('\').

--ignore-sic breaks the same way when the ignored region precedes the word:

$ printf "IGNOREME padding wrod [sic]\n" > sic.txt
$ codespell --ignore-sic sic.txt                             # quiet, as expected
$ codespell --ignore-sic --ignore-regex 'IGNOREME' sic.txt
sic.txt:1: wrod ==> word                     # false positive

The change

Pad each ignored region to its own length. This also lines up with what the option is documented to do — ignore by treating the region as whitespace — and the extra spaces don't affect [\w\-'’]+ word splitting, so detection is unchanged. --ignore-regex=donn on abandonned donn still reports 0, \bdonn\b still reports 1.

On performance

@DimitriPapadopoulos — you raised this on #3978 and it's a fair thing to ask here too, since this one is on the per-line path rather than the -w path. So I measured rather than guessed:

  • Without --ignore-regex: literally unchanged. The if ignore_word_regex: guard means the substitution never runs, so there is no new cost for anyone not using the option.
  • With --ignore-regex, end-to-end: 62,280 lines / 2.1 MB, best-of-5, pattern SKIP:[a-z]*|IGNOREME0.296s → 0.308s, about +4%.
  • On the sub() call alone: a line containing two matches goes 0.96 µs → 1.18 µs; a line with no matches is identical, because the callable is never invoked. Most lines have no matches.

I used m.end() - m.start() rather than len(m.group()) — same result, measurably cheaper (1.18 µs vs 1.38 µs) since it skips materializing the matched string.

If +4% on that path is still too much, the alternative is to drop the masking entirely and filter matches by ignored span instead, but that's a much larger change and I didn't want to reach for it uninvited.

Testing

  • New test_ignore_regex_does_not_shift_offsets covers both the escape-sequence and [sic] paths. It fails on main (assert 1 == 0) and passes with the fix. Existing test_ignore_regex_option is untouched and still passes.
  • Full suite: 124 passed, 3 failed, 16 skipped. The 3 failures (test_command, test_stdin, test_args_from_file) are pre-existing FileNotFoundError for the codespell executable in my sandbox — I confirmed the failing set is byte-identical with and without my change by stashing it.
  • ruff check, ruff format --check, and codespell --toml pyproject-codespell.precommit-toml are all clean.

Note there's an ancient open PR, #2049, that touches this same line for an unrelated reason (consecutive matches). It's independent of this fix, but flagging it since it would conflict textually if it were ever revived.

Disclosure: I used an AI coding assistant (Claude Code) to find this and help write the fix. The repros and benchmarks above are ones I ran myself; happy to answer anything about the change.

_ignore_word_sub() replaced each ignored region with a single space, which
changes the length of the text. Matches are found in that substituted text but
several later checks index back into the original line, so every offset after
an ignored region was wrong:

  - the escape-sequence check reads line[match.start() - 1]
  - --ignore-sic matches the line from match.end()
  - the interactive context display slices the line

The result was false positives from an --ignore-regex that does not overlap the
reported word at all. With `var = 'IGNOREME \nwe are here'`, codespell is quiet
by default (the \n escape suppresses "nwe"), but --ignore-regex=IGNOREME made it
report `nwe ==> new`.

Pad each ignored region to its own length instead. This also matches what the
option is documented to do -- ignore by treating the region as whitespace -- and
the extra spaces do not affect word splitting, so detection is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant