5. EXPERIMENTS
We ran experiments using both synthetic and real data. We first experiment with the synthetic data to evaluate the efficiency and performance of our algorithms, following it with the qualitative evaluation of correlations detected on a real dataset.
We implemented our algorithms in Java, and ran the experiments using Java JRE1.7.0 on a machine with dual core2.53 GHz CPU and 1.5 Gb of main memory.
5.1 Datasets
Synthetic Dataset
In order to accurately measure the precision of our system in identifying sentiment correlations, we conduct a series of experiments on synthetic data, containing time series of sentiments with artificially added positive and negative correlations, level biases and sentiment noise as demonstrated in Figure5. We describe the layout of our dataset below.
Hierarchies. As a preliminary step, we generated a set of demographics hierarchies, for such attributes as age, gender, occupation, and location, containing 8, 3, 4 and 65 nodes respectively. Each node in every hierarchy was randomly assigned with a weight and a bias probability. Weights of nodes are distributed according to a Zipf’s distribution, and normalized to add up to 1 at every level of a hierarchy. Nodes from different hierarchies, when combined, form a lattice of 6240 demographic groups as shown in Figure1. The sentiment volume of each demographic group was taken as a multiplication of weights of attribute nodes, and its bias as a weighted sum of their individual biases. That way, we achieve natural regularity in the demographics lattice, providing a natural distribution of sentiments, which is necessary for a proper evaluation of extracting maximal groups and pruning.
Topic Sentiment. The dataset itself contains time series of sentiments generated independently for multiple topics over a time span of 8 years. Each topic is represented by a unique topic time series, produced using a random walk method, which aggregates uniformly distributed sentiments, whose timestamps follow Poisson distribution. We vary the parameter of rate for timestamps to produce faster or slower changing time series for each topic. Since sentiments for the topic time series are sampled uniformly, its mean value is close to zero in a long run, meaning that it crosses the zero line a few times (for example, at points t1 and t2 in Figure5). The original topic time series is stored for a randomly chosen demographic node, and we generate time series for other lattice nodes using individual bias templates for each of them. Bias templates contain sentiment bias levels and intervals of correlation with the topic time series (positive, negative or zero, randomly changed at “zero sentiment” points). The goal of our evaluation is then to correctly extract these correlation intervals between topic and biased time series, generated as described below.
Biased Sentiment. We produce a biased time series in a correspondence with the template, by copying (inverting) the topic time series in the case of positive (negative) correlation, or outputting randomized data otherwise (as seen in Figure5, bottom). Following that, we add a certain positive or negative bias to the whole time series (shifting all the values) and the uniformly-distributed noise (for each value). Finally, we scale the time series to make sure that sentiments lay within the boundaries of [-1,1]. After generating a biased time series for the node, we insert raw sentiment data into the index in a proportion corresponding to node’s volume. Then, we proportionally distribute these sentiments for all node’s children (Figure1), ensuring the maximality of that node.