Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: export utf8proc Unicode transformation functionality in Julia #5576

Merged
merged 4 commits into from
Feb 3, 2014
Merged

RFC: export utf8proc Unicode transformation functionality in Julia #5576

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6039a46
export utf8proc functionality in Julia (followup to #5462 and #5434)
stevengj Jan 27, 2014
0f36f0f
added normalize_string test cases
stevengj Feb 1, 2014
76c99f3
added jiahao's tests from unicode SA15
stevengj Feb 1, 2014
a500b83
make compose=true the default in normalize_string
stevengj Feb 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added normalize_string test cases
  • Loading branch information
@stevengj
stevengj committed Feb 1, 2014
commit 0f36f0f36653da1e0626c9b7eedcd3b71541cc82
20 changes: 20 additions & 0 deletions test/strings.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -855,3 +855,23 @@ for T = (Uint8,Int8,Uint16,Int16,Uint32,Int32,Uint64,Int64,Uint128,Int128,BigInt
n = T != BigInt ? rand(T) : BigInt(rand(Int128))
@test parseint(T,base(b,n),b) == n
end

# normalize_string (Unicode normalization etc.):
@test normalize_string("\u006e\u0303", :NFC) == "\u00f1"
@test "\u006e\u0303" == normalize_string("\u00f1", :NFD)
@test normalize_string("\ufb00", :NFC) != "ff"
@test normalize_string("\ufb00", :NFKC) == "ff"
@test normalize_string("\u006e\u0303\ufb00", :NFKC) == "\u00f1"*"ff"
@test normalize_string("\u00f1\ufb00", :NFKD) == "\u006e\u0303"*"ff"
@test normalize_string("\u006e\u0303", compose=true) == "\u00f1"
@test "\u006e\u0303" == normalize_string("\u00f1", decompose=true)
@test normalize_string("\u006e\u0303\u00b5",compat=true) == "\u00f1\u03bc"
@test normalize_string("Σσς",casefold=true) == "σσσ"
@test normalize_string("∕⁄", lump=true) == "//"
@test normalize_string("\ua\n\r\r\ua", newline2lf=true) == "\ua\ua\ua\ua"
@test normalize_string("\ua\n\r\r\ua", newline2ls=true) == "\u2028\u2028\u2028\u2028"
@test normalize_string("\ua\n\r\r\ua", newline2ps=true) == "\u2029\u2029\u2029\u2029"
@test normalize_string("\u00f1", stripmark=true) == "n"
@test isempty(normalize_string("\u00ad", stripignore=true))
@test normalize_string("\t\r", stripcc=true) == " "
@test normalize_string("\t\r", stripcc=true, newline2ls=true) == " \u2028"