Skip to content

Added target to playground link #1189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Added target to playground link #1189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4abcc55
Added target to playground link
damianr13 Feb 11, 2025
97cdd0b
Improved colors in the tokenizer playground
damianr13 Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions examples/tokenizer-playground/src/App.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { Token } from './components/Token'
import './App.css'

Expand Down Expand Up @@ -93,12 +93,16 @@ function App() {
worker.current.postMessage({ model_id, text: textareaRef.current.value });
}, []);

const hasColoredBackground = useMemo(() => {
return outputOption === 'text';
}, [outputOption]);

return (
<div className='w-full max-w-[720px] flex flex-col gap-4 items-center'>

<div>
<h1 className='text-5xl font-bold mb-2'>The Tokenizer Playground</h1>
<h2 className='text-lg font-normal'>Experiment with different tokenizers (running <a className="text-gray-900 underline" href="https://github.com/huggingface/transformers.js">locally</a> in your browser).</h2>
<h2 className='text-lg font-normal'>Experiment with different tokenizers (running <a className="text-blue-500 underline" href="https://github.com/huggingface/transformers.js" target="_blank">locally</a> in your browser).</h2>
</div>

<div>
Expand Down Expand Up @@ -146,7 +150,7 @@ function App() {
</div>
</div>

<div ref={outputRef} className='font-mono text-lg p-2.5 w-full bg-gray-100 rounded-lg border border-gray-200 whitespace-pre-wrap text-left h-[200px] overflow-y-auto'>
<div ref={outputRef} className={`font-mono text-lg p-2.5 w-full bg-gray-100 rounded-lg border border-gray-200 whitespace-pre-wrap text-left h-[200px] overflow-y-auto text-gray-600`}>
{outputOption === 'text' ? (
decodedTokens.map(
(token, index) => <Token key={index} text={token} position={index} margin={margins[index]} />
Expand Down
2 changes: 1 addition & 1 deletion examples/tokenizer-playground/src/components/Token.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export function Token({ text, position, margin }) {
));
return (<span
style={{ marginLeft: margin }}
className={`leading-5 ${textWithLineBreaks.length === 1 ? 'inline-block ' : ''}${COLOURS[position % COLOURS.length]}`}>
className={`leading-5 text-white/[0.87] ${textWithLineBreaks.length === 1 ? 'inline-block ' : ''}${COLOURS[position % COLOURS.length]}`}>
{textWithLineBreaks}
</span>)
}