org.ai_heuristic.eval.text

Class BowHeuristic

java.lang.Object

org.ai_heuristic.eval.text.BowHeuristic

Direct Known Subclasses:

Idf

public class BowHeuristic
extends java.lang.Object

This class can be used as a base class for bag-of-words analysis heuristics.

Author:

Kieran Greer

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.HashMap<java.lang.String,java.lang.Integer>	termCount Count for every term in any document
protected java.util.ArrayList<java.lang.String>	termList List of all terms in all documents

Constructor Summary

Constructors

Constructor and Description
BowHeuristic() Create a new instance of BowHeuristic.

Method Summary

Modifier and Type	Method and Description
java.util.ArrayList<BagOfWords>	createBoW(java.util.ArrayList<java.lang.String> filePaths, java.util.ArrayList<java.lang.String> wordList, boolean wordStem) Create a bag-of-words for each file.
BagOfWords	createBoW(java.lang.String theText, java.util.ArrayList<java.lang.String> wordList, boolean wordStem) Create a bag-of-words for the input text sequence.
protected java.util.HashMap<java.lang.String,java.lang.Integer>	getTermCount(java.util.ArrayList<BagOfWords> bowList) Count all terms over all bag of words.
java.util.ArrayList<java.lang.String>	sortTermOrder(java.util.HashMap<java.lang.String,java.lang.Float> termScore) Sort the terms in descending order of the term score.
java.util.ArrayList<java.lang.String>	sortTopTermOrder(int termNumber, java.util.HashMap<java.lang.String,java.lang.Float> termScore) Sort the top number of terms to use.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

termList

protected java.util.ArrayList<java.lang.String> termList

List of all terms in all documents

termCount

protected java.util.HashMap<java.lang.String,java.lang.Integer> termCount

Count for every term in any document

Constructor Detail

BowHeuristic

public BowHeuristic()

Create a new instance of BowHeuristic.

Method Detail

createBoW

public java.util.ArrayList<BagOfWords> createBoW(java.util.ArrayList<java.lang.String> filePaths,
                                                 java.util.ArrayList<java.lang.String> wordList,
                                                 boolean wordStem)
                                          throws java.lang.Exception

Create a bag-of-words for each file. No other processing of the text is performed, for example, removing character sequences.

Parameters:

filePaths - list of files to read.

wordList - list of common words to remove from each bow.

wordStem - if true use word stemming.

Returns:

the created bag of words lists.

Throws:

java.lang.Exception - any error.

createBoW

public BagOfWords createBoW(java.lang.String theText,
                            java.util.ArrayList<java.lang.String> wordList,
                            boolean wordStem)
                     throws java.lang.Exception

Create a bag-of-words for the input text sequence. No other processing of the text is performed, for example, removing character sequences.

Parameters:

theText - the text to read.

wordList - list of common words to remove from each bow. Can be null.

wordStem - if true use word stemming.

Returns:

the created bag of words list.

Throws:

java.lang.Exception - any error.

getTermCount

protected java.util.HashMap<java.lang.String,java.lang.Integer> getTermCount(java.util.ArrayList<BagOfWords> bowList)
                                                                      throws java.lang.Exception

Count all terms over all bag of words. If the term exists in the bag of words then increment the count by 1.

Parameters:

bowList - list of bag of words to process. Of type BagOfWords.

Returns:

the list of terms with counts.

Throws:

java.lang.Exception - any error.

sortTermOrder

public java.util.ArrayList<java.lang.String> sortTermOrder(java.util.HashMap<java.lang.String,java.lang.Float> termScore)
                                                    throws java.lang.Exception

Sort the terms in descending order of the term score.

Parameters:

termScore - a score (floating point value) for each term.

Returns:

the terms in descending order of score.

Throws:

java.lang.Exception - any error.

sortTopTermOrder

public java.util.ArrayList<java.lang.String> sortTopTermOrder(int termNumber,
                                                              java.util.HashMap<java.lang.String,java.lang.Float> termScore)
                                                       throws java.lang.Exception

Sort the top number of terms to use.

Parameters:

termNumber - the maximum number of terms to add to the sorted list.

termScore - a score (float value) for each term. Key is term name, value is the score.

Returns:

the list of top term names.

Throws:

java.lang.Exception - any error.