Hello,
I just wanted to flag that Qdap doesn't work with Chinese text unless it has been pre-segmented. It ends up picking up extra text because Chinese text doesn't include word spaces. Folks who work with Chinese text typically run a word segmentation program on the text before using it, but you need to remove the URLs first before running the word segmenter because sometimes the segmenter will break up the URLs.
I used the following regular expression work around. The only cases where it doesn't work are when English text direct follows the URL with no spaces. So it wouldn't be suitable for a mixed-language corpus:
"http://[a-zA-Z/.0-9$–\-+!*‘()]{1,}|www[.]{1}[a-zA-Z/.0-9$–\-+!*‘()]{1,}"
Hello,
I just wanted to flag that Qdap doesn't work with Chinese text unless it has been pre-segmented. It ends up picking up extra text because Chinese text doesn't include word spaces. Folks who work with Chinese text typically run a word segmentation program on the text before using it, but you need to remove the URLs first before running the word segmenter because sometimes the segmenter will break up the URLs.
I used the following regular expression work around. The only cases where it doesn't work are when English text direct follows the URL with no spaces. So it wouldn't be suitable for a mixed-language corpus:
"http://[a-zA-Z/.0-9$–\-+!*‘()]{1,}|www[.]{1}[a-zA-Z/.0-9$–\-+!*‘()]{1,}"