Skip to content

Proposed alternate prompting scheme #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
gvanrossum wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Proposed alternate prompting scheme #241

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions python/src/typechat/_internal/translator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,14 @@ async def translate(self, input: str, *, prompt_preamble: str | list[PromptSecti

messages: list[PromptSection] = []

messages.append({"role": "user", "content": input})
messages.append({"role": "system", "content": self._create_request_prompt(input)})

if prompt_preamble:
if isinstance(prompt_preamble, str):
prompt_preamble = [{"role": "user", "content": prompt_preamble}]
else:
messages.extend(prompt_preamble)

@gvanrossum gvanrossum Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that the original code had a bug, where if prompt_preamble was a string, it would be ignored? Was that bug reported in any issue?

@robgruen robgruen Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was your change from a while ago so I can't speak to that. I think you mean to have the "intent" to be a separate user message after the JSON translation string which you're proposing as the system prompt.

@robgruen robgruen Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here are no live tests at the moment, that's on the TODO list.  The AI generated report I sent you compares the existing prompting scheme in comparison with the one proposed here. Since both the C# and TypeScript implementations have the same prompting scheme I wanted to ensure that we see the same results across implementations.

The net takeaway is that the net token count goes down by a few percentage points as does the overall retry count. With the current set of models this approach is marginally better given the sample schemas we have in the repo at this time. I would like to flush it out a bit more before declaring victory and making a final determination. I favor this approach since there is clear separation between the messages whereas the existing schema is slightly more coupled.

messages.extend(prompt_preamble)

messages.append({"role": "user", "content": self._create_request_prompt(input)})
messages.append({"role": "user", "content": input})

num_repairs_attempted = 0
while True:
Expand Down Expand Up @@ -103,11 +103,7 @@ def _create_request_prompt(self, intent: str) -> str:
```
{self._schema_str}
```
The following is a user request:
'''
{intent}
'''
The following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:
You translate each user request into a JSON object with 2 spaces of indentation and no properties with the value undefined.
"""
return prompt

Expand Down