Clarify repository and data package licensing structure

[ekaf]

Context

There are long-standing open issues in nltk_data regarding unclarified licenses for certain data packages—most notably the Punkt tokenizer models. Despite community efforts spanning years, no clarification for these licenses appears forthcoming.

Proposal

To improve transparency and compliance, resolve this issue by creating and maintaining three files:

1. LICENSE-OVERVIEW.md: A short, human-readable overview of the repository’s licensing structure, summarizing the main repository license and the situation with individual data packages.
2. LICENSE (Apache 2.0): The full text of the Apache 2.0 license, which governs the repository as a whole.
3. DATASET-LICENSES.md: A comprehensive list of all individual data packages, grouped by license type. For packages with unclarified or unknown licenses (e.g., Punkt), group them under an “unclarified” heading and clearly indicate the lack of clarity.

Rationale

• This approach increases transparency for users and downstream projects.
• It provides clear documentation of which packages have ambiguous licensing, and the project’s ongoing efforts in this area.
• It is a practical solution given the persistent lack of licensing clarification, ensuring responsible stewardship of data and code.

Tasks

• Draft the three files as described above.
• Keep the DATASET-LICENSES.md up to date as packages are added or removed.
• Clearly highlight and prioritize resolving unclarified licensing cases wherever possible.

---

This issue focuses on the licensing documentation and transparency, especially regarding long-standing ambiguities such as those with Punkt.

This is a sub-issue of #240.

[ekaf]

How to "clearly indicate the lack of clarity"?

To “clearly indicate the lack of clarity” about a data package’s license in documentation, you should:

1. Explicitly State Unclarified Status:
   Use clear language such as:

   • “The license for this package is unclarified.”
   • “No official license has been provided for this package.”
   • “License status: Unknown / Pending clarification.”

2. Highlight the Issue:
   Use a dedicated section or heading, for example:

   ### Unclarified or Unknown License Packages
   

   Then list those packages, and for each, provide a brief statement, e.g.:

   - Punkt Tokenizer Models: License unclarified. The original author has not provided a license statement. Use with caution.
   

3. Explain the Consequences (Optional but Helpful):
   Add a short note about the implications:

   • “As the license is unclarified, redistribution or commercial use may be restricted. Users are advised to exercise caution.”

4. Document Efforts (Optional):
   You can also mention any attempts to resolve the issue:

   • “We have attempted to contact the original author(s) for clarification but have not received a response.”

Example Table Row:

Dataset Name	License	Notes
Punkt Tokenizer Models	Unclarified	No license provided; use with caution.

Summary:
Use unambiguous wording (“unclarified,” “unknown,” “pending”), and provide a direct note explaining the lack of clarity and any risks. This ensures transparency for users and downstream projects.