Skip to content

feat(ui): add Token Estimator link to footer #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

feat(ui): add Token Estimator link to footer #337

wants to merge 17 commits into from

Conversation

HmbleCreator
Copy link

Summary

This PR adds a "Token Estimator" link to the footer of the Gitingest web frontend. The link points to https://gitingest.com/tokencount, allowing users to easily access a tool for estimating token counts in pasted text.

Details

  • UI Change:

    • Added a new link labeled Token Estimator (with a counter icon) to the left column of the footer, alongside the Extension and Python package links.
    • The link opens in a new tab and is styled consistently with the other resource links.
    • No changes were made to the CLI or Python package.

  • Why:

  • How to test:

    1. Start the server (cd src && uvicorn server.main:app --reload).
    2. Open the app in your browser.
    3. Scroll to the footer and verify the "Token Estimator" link appears and opens https://gitingest.com/tokencount in a new tab.

Screenshot

Screenshot 2025-06-29 205833

Closes #318

@HmbleCreator
feat(ui): add Token Estimator link to footer
a3eef1a 
- Adds a "Token Estimator" link (with icon) to the left column of the site footer.
- Link points to https://gitingest.com/tokencount and opens in a new tab.
- Styled consistently with other resource links.
- No changes to CLI or Python package.

Closes coderamp-labs#318
@cyclotruc
Copy link
Contributor

@HmbleCreator Thanks,
But this is just the link, it seems to me that #318 would require to actually implement the backend route for this new function, do you think you can work on this?

@HmbleCreator
Copy link
Author

HmbleCreator commented Jun 30, 2025 •
edited
Loading

@HmbleCreator Thanks, But this is just the link, it seems to me that #318 would require to actually implement the backend route for this new function, do you think you can work on this?

Thanks for pointing that out! Yes, I’d be happy to implement the backend route for token estimation as part of #318. I can add an endpoint (e.g. /api/estimate-tokens) that accepts raw text, estimates the token count using tiktoken or a similar utility, and returns the result. Let me know if you have a preferred response schema or want it namespaced differently; I'd be happy to get started.

@HmbleCreator
Copy link
Author

HmbleCreator commented Jun 30, 2025 •
edited
Loading

@HmbleCreator Thanks, But this is just the link, it seems to me that #318 would require to actually implement the backend route for this new function, do you think you can work on this?

Thanks for pointing that out! Yes, I’d be happy to implement the backend route for token estimation as part of #318. I can add an endpoint (e.g. /api/estimate-tokens) that accepts raw text, estimates the token count using tiktoken or a similar utility, and returns the result. Let me know if you have a preferred response schema or want it namespaced differently; I'd be happy to get started.

Are we only gonna have support for OpenAI models or other models also, like open source models available via HF?

@HmbleCreator
feat: Add token estimator documentation page and simplify footer link…
6247f92 
… text

- Add GET endpoint for /api/tokencount with interactive documentation
- Include API usage examples and interactive form for testing
@HmbleCreator
feat: Add token estimator documentation page and simplify footer link…
cef47a2 
… text

- Add GET endpoint for /api/tokencount with interactive documentation
- Include API usage examples and interactive form for testing
@HmbleCreator
Copy link
Author

The token counting functionality uses two efficient tokenizer libraries:

  • tiktokenizer for OpenAI models - provides fast and accurate token counting for GPT models
  • autotiktokenizer for non-OpenAI models - enables efficient token counting across a wide range of open-source models

This dual approach ensures we can provide fast and accurate token counting regardless of the model being used, while keeping the implementation lightweight and efficient.

ix-56h
ix-56h reviewed Jun 30, 2025
View reviewed changes
@HmbleCreator
feat: use Jinja template for token estimator API docs
48328ab 
- Add tokencount_api.jinja for styled token estimator documentation
- Update /api/tokencount GET endpoint to render the new template
- Documentation now matches the look and feel of other API docs
@ix-56h
Copy link
Contributor

ix-56h commented Jun 30, 2025 •
edited
Loading

I'll test it out in few hours.

I see that both logic and routing are defined and implemented within the entrypoint of the server, maybe move this to the "routers/" folder.

ix-56h
ix-56h requested changes Jun 30, 2025
View reviewed changes
filipchristiansen and others added 4 commits June 30, 2025 22:11
@filipchristiansen
Merge branch 'main' into main
2ed86f6
@HmbleCreator
Merge branch 'main' into main
3f0c1a5
@HmbleCreator
Refactor token count API and routing, enforce API-only for /api/token…
98e5c45 
…count, and backend cleanup

- Removed GET /api/tokencount route that rendered a Jinja template; /api/tokencount is now POST-only and returns JSON.
- Renamed and refactored tokencount_api.jinja to tokencount.jinja, using a select dropdown for model selection and matching git_form.jinja UX.
- Updated footer link to point to /tokencount (user-facing form) instead of /api/tokencount.
- Ensured /tokencount is the user-facing form (GET/POST, Jinja) and /api/tokencount is API-only (POST, JSON).
- Updated dependency management: added autotiktokenizer to requirements.txt and pyproject.toml, and added a guideline to CONTRIBUTING.md.
- Cleaned up unused imports and dead code in backend.
@HmbleCreator
Merge branch 'main' of https://github.com/HmbleCreator/gitingest
5c271b5
@HmbleCreator HmbleCreator requested a review from ix-56h July 1, 2025 03:09
@HmbleCreator
Copy link
Author

Hi!
I wanted to ask—does the UI theme/design you use for Gitingest have a name or is it based on a particular design system? I liked the style; it’s clean, modern, and playful!
Additionally, I wanted to express my gratitude for your guidance and patience over the past two to three days. As a newbie, it was a great experience learning about FastAPI, backend/frontend separation, API best practices, and collaborative open-source workflows. I learned a great deal and truly appreciate the opportunity to contribute!
Thanks again!

@ix-56h
Copy link
Contributor

ix-56h commented Jul 1, 2025 •
edited
Loading

Hi! I wanted to ask—does the UI theme/design you use for Gitingest have a name or is it based on a particular design system?

Yes, neobrutalism, you can find it at neobrutalism.com.

I wanted to express my gratitude for your guidance and patience over the past two to three days. As a newbie, it was a great experience learning about FastAPI, backend/frontend separation, API best practices, and collaborative open-source workflows. I learned a great deal and truly appreciate the opportunity to contribute! Thanks again!

You're welcome :)

@HmbleCreator
Copy link
Author

hey, is there any issue with the merging, any issue with code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ix-56h ix-56h Awaiting requested review from ix-56h

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

feat: include a token estimator utility
4 participants
@HmbleCreator @cyclotruc @ix-56h @filipchristiansen