Skip to content

fix(chatgpt): use data-turn to detect upload previews vs generated images #2292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jesuisjohan wants to merge 1 commit into
base: main
Choose a base branch
from
+70 −3
Open

fix(chatgpt): use data-turn to detect upload previews vs generated images #2292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions clis/chatgpt/utils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2560,12 +2560,28 @@ export async function getChatGPTVisibleImageUrls(page) {
return /avatar|profile|logo|icon/.test(text);
};
const isUserUploadPreview = (img) => {
const alt = (img.getAttribute('alt') || '').toLowerCase();
const turn = img.closest('section[data-testid^="conversation-turn"]');
const heading = (turn?.querySelector('h4')?.innerText || '').toLowerCase();
// Authoritative signal first: ChatGPT stamps data-turn directly on the
// turn section as soon as the turn mounts, well before an attached
// image's alt text/aria-label finish populating. Racing multiple
// uploads against that async metadata (the old behaviour here) let
// still-generic upload-preview thumbnails pass as "new" images for a
// poll or two, which is long enough to satisfy the stability check in
// waitForChatGPTImages and return the wrong (uploaded, not generated)
// images when 2+ files were attached.
const turnRole = turn?.getAttribute('data-turn') || '';
if (turnRole === 'user') return true;
if (turnRole === 'assistant') return false;
// Fallback for markup without data-turn. innerText reads as empty
// on a visually-hidden heading in real Chrome (jsdom has no layout and
// never exposed this gap) - use textContent instead.
const heading = (turn?.querySelector('h4')?.textContent || '').toLowerCase();
if (/you said|你说/.test(heading)) return true;
if (/chatgpt|assistant|助手/.test(heading)) return false;
const openButtonLabel = (img.closest('button[aria-label^="Open image:"]')?.getAttribute('aria-label') || '').toLowerCase();
const alt = (img.getAttribute('alt') || '').toLowerCase();
// ChatGPT's multi-image "Open image" button label now reads
// "Open image N of M: name", not the older "Open image: name".
const openButtonLabel = (img.closest('button[aria-label*="Open image"]')?.getAttribute('aria-label') || '').toLowerCase();
const previewText = [alt, openButtonLabel].join(' ');
return /\.(png|jpe?g|webp|gif|heic|heif)(?:\b|$)/i.test(previewText)
|| /ref-|reference|参考|upload|uploaded|attachment/.test(previewText);
Expand Down
51 changes: 51 additions & 0 deletions clis/chatgpt/utils.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1548,6 +1548,39 @@ describe('chatgpt generated image detection', () => {
]);
});

it('ignores multiple upload-preview thumbnails via data-turn before their alt/aria-label metadata populate', async () => {
// Reproduces a real regression: uploading 2+ reference images made
// waitForChatGPTImages return the just-uploaded thumbnails instead of
// the actual generated image. Right after upload, a thumbnail's alt
// text and "Open image N of M: <name>" aria-label haven't populated
// yet, so the old alt/aria-label-only fallback couldn't tell them
// apart from a real result during that window. `data-turn` on the
// turn <section> is set immediately and must be checked first.
const page = createDomPage(`
<!doctype html>
<section data-testid="conversation-turn-1" data-turn="user">
<h4>You said:</h4>
<img alt="" src="https://chatgpt.com/backend-api/uploaded/ref-1.png">
<img alt="" src="https://chatgpt.com/backend-api/uploaded/ref-2.png">
<img alt="" src="https://chatgpt.com/backend-api/uploaded/ref-3.png">
</section>
<section data-testid="conversation-turn-2" data-turn="assistant">
<h4>ChatGPT said:</h4>
<img alt="Generated image: result" src="https://chatgpt.com/backend-api/generated/foo.webp">
</section>
`, (window) => {
for (const img of window.document.querySelectorAll('img')) {
Object.defineProperty(img, 'naturalWidth', { configurable: true, value: 512 });
Object.defineProperty(img, 'naturalHeight', { configurable: true, value: 512 });
img.getBoundingClientRect = () => ({ width: 512, height: 512 });
}
});

await expect(getChatGPTVisibleImageUrls(page)).resolves.toEqual([
'https://chatgpt.com/backend-api/generated/foo.webp',
]);
});

it('keeps assistant generated images even when they are inside an open-image button', async () => {
const page = createDomPage(`
<!doctype html>
Expand All @@ -1569,6 +1602,24 @@ describe('chatgpt generated image detection', () => {
]);
});

it('recognizes the "Open image N of M: name" aria-label ChatGPT uses for multi-attachment uploads', async () => {
const page = createDomPage(`
<!doctype html>
<section data-testid="conversation-turn-1">
<button aria-label="Open image 1 of 3: reference.png">
<img alt="" src="https://chatgpt.com/backend-api/uploaded/reference.png">
</button>
</section>
`, (window) => {
const img = window.document.querySelector('img');
Object.defineProperty(img, 'naturalWidth', { configurable: true, value: 512 });
Object.defineProperty(img, 'naturalHeight', { configurable: true, value: 512 });
img.getBoundingClientRect = () => ({ width: 512, height: 512 });
});

await expect(getChatGPTVisibleImageUrls(page)).resolves.toEqual([]);
});

it('exports assets for generated CSS background images', async () => {
const imageUrl = 'https://chatgpt.com/backend-api/generated/foo.webp';
const page = createDomPage(`
Expand Down