org.licas_text

Class SourceText

java.lang.Object

org.licas_text.SourceText

public class SourceText
extends java.lang.Object

This class can be used to parse a text source, to extract only specific types of content. For example, if the source is an HTML file, then maybe only the resource links are required. It is at a slightly higher level than the FilterHtml class that it uses to do the parsing.

Author:

Kieran Greer

Constructor Summary

Constructors

Constructor and Description
SourceText() Create a new instance of SourceText

Method Summary

Modifier and Type	Method and Description
java.util.ArrayList<java.net.URL>	getHyperLinks() Return the list of parsed hyperlinks.
java.lang.String	parseHTML(java.lang.String htmlSource, SourceConfig parseConfig) Parse the html source as defined by parseConfig.
java.lang.String	parseHTML(java.net.URL htmlUri, SourceConfig parseConfig) Read the source URL and parse the html source as defined by parseConfig.
java.util.ArrayList<java.net.URL>	parseHyperLinks(java.lang.String theText) Parse all hyperlinks in the text if any exist and keep all found.
java.util.ArrayList<java.net.URL>	parseHyperLinks(java.lang.String theText, java.util.ArrayList<java.lang.String> toKeep, java.lang.String formatType, boolean resetHyperList) Parse all hyperlinks in the text if any exist, but only keep certain types of link.
java.lang.String	readableContent(java.lang.String htmlSource) Get only the readable content of the html document.
java.lang.String	readableContent(java.net.URL htmlUri) Get only the readable content of the html document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SourceText

public SourceText()

Create a new instance of SourceText

Method Detail

parseHTML

public java.lang.String parseHTML(java.net.URL htmlUri,
                                  SourceConfig parseConfig)
                           throws java.lang.Exception

Read the source URL and parse the html source as defined by parseConfig. To automatically remove all of the markup, you can try readableContent instead.

Parameters:

htmlUri - uri of the html file.

parseConfig - defines what parsing should be done. Can be null when no text removal is performed.

Returns:

the parsed text contents of the file.

Throws:

java.lang.Exception - any error.

parseHTML

public java.lang.String parseHTML(java.lang.String htmlSource,
                                  SourceConfig parseConfig)
                           throws java.lang.Exception

Parse the html source as defined by parseConfig. To automatically remove all of the markup, you can try readableContent instead.

Parameters:

htmlSource - the source text in html format.

parseConfig - defines what parsing should be done. Can be null for default settings.

Returns:

the parsed text contents of the file.

Throws:

java.lang.Exception - any error.

readableContent

public java.lang.String readableContent(java.net.URL htmlUri)
                                 throws java.lang.Exception

Get only the readable content of the html document. This will perform some pre-defined operations and also remove all of the xml formatting or markup tags. This content might be more useful for analysing, for example.

Parameters:

htmlUri - uri of the html file.

Returns:

the text where only the content between the readable tags has been kept.

Throws:

java.lang.Exception - any error.

readableContent

public java.lang.String readableContent(java.lang.String htmlSource)
                                 throws java.lang.Exception

Get only the readable content of the html document. This will perform some pre-defined operations and also remove all of the xml formatting or markup tags. This content might be more useful for analysing, for example.

Parameters:

htmlSource - the source text in html format.

Returns:

the text where only the content between the readable tags has been kept.

Throws:

java.lang.Exception - any error.

parseHyperLinks

public java.util.ArrayList<java.net.URL> parseHyperLinks(java.lang.String theText)
                                                  throws java.net.MalformedURLException,
                                                         java.lang.Exception

Parse all hyperlinks in the text if any exist and keep all found. These are also stored in the hyperLinks list, which is reset first and a html format is assumed.

Parameters:

theText - the text with the hyperlink descriptions.

Returns:

the list of hyperlinks.

Throws:

java.net.MalformedURLException - hyperlink cannot be created.

java.lang.Exception - any error.

parseHyperLinks

public java.util.ArrayList<java.net.URL> parseHyperLinks(java.lang.String theText,
                                                         java.util.ArrayList<java.lang.String> toKeep,
                                                         java.lang.String formatType,
                                                         boolean resetHyperList)
                                                  throws java.net.MalformedURLException,
                                                         java.lang.Exception

Parse all hyperlinks in the text if any exist, but only keep certain types of link. These are also stored in the hyperLinks list.

Parameters:

theText - the text with the hyperlink descriptions.

toKeep - a list of descriptions of link types to keep. If null or empty, then all link types are kept. This can currently be HtmlConst.HTTPLINKTYPE or IMAGELINKTYPE.

formatType - the format of the text. Can be VendorEngine.HTML or JSON.

resetHyperList - true if reset the hyperlink list first.

Returns:

the list of hyperlinks.

Throws:

java.net.MalformedURLException - hyperlink cannot be created.

java.lang.Exception - any error.

getHyperLinks

public java.util.ArrayList<java.net.URL> getHyperLinks()

Return the list of parsed hyperlinks. Need to parse the text first to extract the hyperlinks. A call to parseHyperlinks does this.

Returns:

the value of hyperLinks.