[docs]: refine README for dpo updates

[mrhaoxx]

What does this PR do?

Fixes # (issue)

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

[gemini-code-assist]

Summary of Changes

Hello @mrhaoxx, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on refining the documentation to accurately reflect recent updates regarding DPO (Direct Preference Optimization) fine-tuning capabilities. It ensures that users have up-to-date information on DPO support, correct model references, and clear guidance for training and inference, enhancing the overall usability and accuracy of the project's documentation.

Highlights

• README Update: Added a new entry to the main README.md under the "Updates" section, announcing support for RL-DPO fine-tuning with LLaMA-Factory and linking to its tutorial.
• DPO Tutorial Refinement: Updated the DPO_tutorial.md to use the correct model identifier (deepseek-ai/DeepSeek-V2-Lite) and adjusted the num_train_epochs from 0.1 to 3 in the example YAML configuration.
• Inference Clarification: Improved the clarity of the DPO tutorial by adding instructions on how to use the saved LoRA adapter for inference after DPO training.
• kt-sft README Link: Included a new line in the kt-sft/README.md to highlight the support for RL DPO training and provide a direct link to the DPO tutorial.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the documentation to include information about the new DPO fine-tuning support. It adds an entry to the main README, creates a new DPO tutorial, and links to it from the kt-sft README. The changes are mostly good, but I've found a couple of minor issues in the new tutorial file (DPO_tutorial.md): a grammatical error and an inconsistent model name in one of the examples. My review includes suggestions to fix these for clarity and correctness.

[gemini-code-assist]

While this clarification is helpful, the YAML configuration example that follows (starting on line 135) appears to have an inconsistency. The model_name_or_path on line 136 is set to DeepSeek-V2-Lite-Chat, but earlier in this document (lines 64 and 83), this was updated to deepseek-ai/DeepSeek-V2-Lite. Please update line 136 as well for consistency.