TfIdf can use chinese ?

[c941010623]

my code is :

var natural = require('natural'),
TfIdf = natural.TfIdf,
tfidf = new TfIdf();

tfidf.addDocument('中文測試', 's1');
var s = JSON.stringify(tfidf);
console.log(s)

[kkoch986]

I havent personally tried tfidf with chinese, first glance it doesnt seem to work.

You probably need to change the tokenizer but i dont think we have a chinese tokenizer yet. I'll leave this open for awhile and see if anyone else has any experience with tfidf this way.

[mike820324]

Is it possible to use something like the following libraries to first tokenize the Chinese sentences or document?
https://github.com/dotSlashLu/nodescws
https://github.com/yanyiwu/nodejieba

which should separate a Chinese sentences into several Chinese tokens.

for example the following string
"中文測試", which means "Chinese test", will become the following list,
["中文", "測試"], which means ["Chinese", "test"].

[dcsan]

@mike820324 did you get any further with this? I'm also using nodejieba on some Chinese NLP projects, but not sure if i should move to python for the project for NLTK etc.

[anton-bot]

I have no problem with the Chinese tokenizer, but the code still doesn't work. When I checked listTerms(), it assigns the tfidf of zero to all terms:

我: 0
搵: 0
緊: 0
游泳池: 0
你們: 0
喺邊度: 0

Is this a problem? How to fix that?

[dcsan]

what do the two values mean in list of terms? Is this a basic frequency or the inverse frequency related to the text?

FWIW frequency word lists are a mixed bag for chinese. I think that Jieba has its own built in, which while trained on not the most representative material, would at least match the same tokens...

[titanism]

You can use https://github.com/yishn/chinese-tokenizer for tokenization. Perhaps @Hugo-ter-Doest would like to add this directly to the package? Similar to the port done for Japanese tokenizer.