Gensim Word2Vec Tutorial

In this tutorial, we will learn how to use python gensim word2vec to train our word embedding. You can click here for more info about this python library https://radimrehurek.com/gensim/.

This tutorial focuses on the use of the Word2Vec package from the Gensim libray. For more conceptual details, you can check my lecture slides. For more conceptual details, you can check my lecture slides. You can check here for more gensim word2vec model intro https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#sphx-glr-download-auto-examples-tutorials-run-word2vec-py.

For the understanding of basic concepts of neural network on nlp, you can check Stanford CS224d: Deep Learning for NLP http://web.stanford.edu/class/cs224n/. Dr. Manning has a very intuitive, useful intro to word2vec.

You will use 2000 NYT news article related to dynamic of collective action as raw texts. Please do not share this data set publicly. You can click here for more information about DOCA project https://web.stanford.edu/group/collectiveaction/cgi-bin/drupal/.

This tutorial is adapted from https://www.kaggle.com/pierremegret/gensim-word2vec-tutorial and https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#sphx-glr-download-auto-examples-tutorials-run-word2vec-py. You can check here for more details.

Briefing about Word2Vec

drawing

Word2Vec was introduced in two papers, including Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality between September and October 2013, by a team of researchers at Google. You can read these papers for more tech details. The Python implementation was in Gensim.

Word2Vec assumes that two words sharing similar contexts also share a similar meaning and consequently a similar vector representation from the model. Word2vec is a widely featured as a member of the “new wave” of machine learning algorithms based on neural networks, commonly referred to as “deep learning” (though word2vec itself is rather shallow).

Using large amounts of unannotated plain text, word2vec learns relationships between words automatically. The output are vectors, one vector per word, with remarkable linear relationships that allow us to do things like:

vec(“king”) - vec(“man”) + vec(“woman”) =~ vec(“queen”)

You should be familiar with the bag-of-words model. This model transforms each document to a fixed-length vector of integers. For example, given the sentence:

John likes to watch movies. Mary likes movies too.

John also likes to watch football games. Mary hates football.

The model outputs the vectors:

[1, 2, 1, 1, 2, 1, 1, 0, 0, 0, 0]

[1, 1, 1, 1, 0, 1, 0, 1, 2, 1, 1]

Each vector has 10 elements, where each element counts the number of times a particular word occurred in the document. The order of elements is arbitrary. In the example above, the order of the elements corresponds to the words: ["John", "likes", "to", "watch", "movies", "Mary", "too", "also", "football", "games", "hates"].

Bag-of-words models are surprisingly effective, but have several weaknesses.

First, they lose all information about word order: “John likes Mary” and “Mary likes John” correspond to identical vectors.

Second, the model does not attempt to learn the meaning of the underlying words, and as a consequence, the distance between vectors doesn’t always reflect the difference in meaning. The Word2Vec model addresses this second problem.

Word2Vec is a more recent model that embeds words in a lower-dimensional vector space using a shallow neural network. The result is a set of word-vectors where vectors close together in vector space have similar meanings based on context, and word-vectors distant to each other have differing meanings. For example, strong and powerful would be close together and strong and Paris would be relatively far.

The are two versions of this model and Word2Vec class implements them both: Skip-grams (SG) and Continuous-bag-of-words (CBOW)

The Word2Vec Skip-gram model, for example, takes in pairs (word1, word2) generated by moving a window across text data, and trains a 1-hidden-layer neural network based on the synthetic task of given an input word, giving us a predicted probability distribution of nearby words to the input. A virtual one-hot encoding of words goes through a ‘projection layer’ to the hidden layer; these projection weights are later interpreted as the word embeddings. So if the hidden layer has 300 neurons, this network will give us 300-dimensional word embeddings.

Continuous-bag-of-words Word2vec is very similar to the skip-gram model. It is also a 1-hidden-layer neural network. The synthetic training task now uses the average of multiple input context words, rather than a single word as in skip-gram, to predict the center word. Again, the projection weights that turn one-hot words into averageable vectors, of the same width as the hidden layer, are interpreted as the word embeddings.

Getting Started

Libraries used:

If your python environment does not have these libraries, you should use pip or conda to install these modules.

In [42]:
import re  # For preprocessing
import pandas as pd  # For data handling
from time import time  # To time our operations
from collections import defaultdict  # For word frequency

import spacy  # For preprocessing

import logging  # Setting up the loggings to monitor gensim
logging.basicConfig(format="%(levelname)s - %(asctime)s: %(message)s", datefmt= '%H:%M:%S', level=logging.INFO)

The data

The DoCA NYT news articles raw texts were collected from ProQuest by matching news article titles with doca news article title. We only provide you 2000 articles for this tutorial. If your mac has limited RAM, you can then randomly select 1000 articles, but you can try to use 2K articles.

Again you can access the data set via this link. You can also access doca data set here: https://web.stanford.edu/group/collectiveaction/cgi-bin/drupal/node/21 You can match article data with doca main dataset to retrieve more covariates. DOCA coded all news artcles. If you have any research ideas regarding using the DOCA raw NYT news articles, plz let me know. We can think about some research ideas maybe in the future.

Preprocessing

We keep only two variables:

  • title_doca: the news article title for doca in case that you want to match the data with doca_main data.
  • text: the raw text for news articles related to protest
In [43]:
import os
os.getcwd() # check the working directory...
# in python, we use pandas to read and manipulate data...it is similar to R dataframe...
df = pd.read_csv('data_doca.csv').sample(n=2000).reset_index() # large text data
df.shape
Out[43]:
(2000, 3)
In [44]:
df.head() # check the top 5 rows for a look
Out[44]:
index title_doca text
0 1432 FIRE RAZES SCHOOL THAT HELPED MARCH FIRE RAZES SCHOOL THAT HELPED MARCH\nNew York ...
1 614 Antiwar Protests Erupt Across US ANTIWAR PROTESTS ERUPT ACROSS U.S.: COLUMBIA R...
2 1461 3 NEGRO STUDENTS GET POLICE ESCORT 3 NEGRO STUDENTS GET POLICE ESCORT: PARENTS TE...
3 681 NEW ROCHELLE HIGH SHUT New Rochelle High Shut\n\nSpecial to The New Y...
4 350 SPLIT AMONG JERSEY CUBANS IS POINTED UP BY BOM... Split Among, Jersey Cubans Is Pointed Up by Bo...

Let us dive into one example. See raw text

In [45]:
df["text"][0]
Out[45]:
"FIRE RAZES SCHOOL THAT HELPED MARCH\nNew York Times (1923-Current fle); Jun 25, 1966; ProQuest Historical Newspapers: The New York Times\n\nFIRE RAZES SCHOOL\nTHAT HELPED MARCH\n\nCARTHAGE, Miss., June 24\n(AP)—A rural Roman Catholic\nmission school for Negroes near\nhere was destroyed by fire early\ntoday. The Federal Bureau of\nInvestigation said it was looking into possible violations of\nFederal law.\n\nA staff member at the school,\nthe Rev. Luke Mikschl of New\nYork, said dynamite had been\nthrown on the school grounds\n“on several occasions” since it\nbegan classes for Negro adults\nlast February, No one was hurt\nby the blasts, he said.\n\nThe school, St. Joachim, was\noperated by the Trinitarian Missionaries. It was situated six\nmiles northwest of Carthage-—\na town half way between Mississippi's two cities of current\nracial tensions, Canton and Philadelphia, and had given refuge\nto civil rights marchers.\n\nLeake County Sheriff Russell\nEdwards could not be reached,\nbut a spokesman in his office\nhere said no record of the fire\nhad been received. .\n\nThe FBI. said an explosive\nhad been detonated at the\nschool “on at least one occasion\npreviously.” Antipoverty program literacy classes for adults\nwere begun at the school in\nFebruary.\n\nReproduced with permission of the copyright owner. Further reproduction prohibited without permission."

Removing the missing values:

Cleaning:

We need to remove those urgly punctuations We are lemmatizing and removing the stopwords and non-alphabetic characters for each line of dialogue. We use spacy library to do text processing. You can check here for more details https://spacy.io/usage/processing-pipelines

In [46]:
nlp = spacy.load('en_core_web_sm', disable=['ner', 'parser']) # disabling Named Entity Recognition for speed
## Josh comments: you need to download en_core_web_sm using "python -m spacy download en_core_web_sm"
def cleaning(doc):
    # Lemmatizes and removes stopwords
    # doc needs to be a spacy Doc object
    txt = [token.lemma_ for token in doc if not token.is_stop]
    return ' '.join(txt)

Let us removes non-alphabetic characters:

In [47]:
brief_cleaning = (re.sub("[^A-Za-z']+", ' ', str(row)).lower() for row in df['text'])

Taking advantage of spaCy .pipe() attribute to speed-up the cleaning process:

In [48]:
t = time()

txt = [cleaning(doc) for doc in nlp.pipe(brief_cleaning, batch_size=5000, n_threads=-1)]

print('Time to clean up everything: {} mins'.format(round((time() - t) / 60, 2)))
Time to clean up everything: 0.46 mins

Put the results in a DataFrame to remove missing values and duplicates:

In [54]:
df_clean = pd.DataFrame({'clean': txt})
df_clean = df_clean.dropna().drop_duplicates()
df_clean.shape
Out[54]:
(1980, 1)

Bigrams:

We are using Gensim Phrases package to automatically detect common phrases (bigrams) from a list of sentences. https://radimrehurek.com/gensim/models/phrases.html

The main reason we do this is to catch words like "new york" or "social movement" !

In [55]:
from gensim.models.phrases import Phrases, Phraser

As Phrases() takes a list of list of words as input:

In [56]:
article = [row.split() for row in df_clean['clean']]

Creates the relevant phrases from the list of articles:

In [57]:
phrases = Phrases(article, min_count=3)
INFO - 10:30:37: collecting all words and their counts
INFO - 10:30:37: PROGRESS: at sentence #0, processed 0 words and 0 word types
INFO - 10:30:38: collected 453893 word types from a corpus of 662842 words (unigram + bigrams) and 1980 sentences
INFO - 10:30:38: using 453893 counts as vocab in Phrases<0 vocab, min_count=3, threshold=10.0, max_vocab_size=40000000>

Phrases function Parameters

sentences (iterable of list of str, optional) – The sentences iterable can be simply a list, but for larger corpora, consider a generator that streams the sentences directly from disk/network, See BrownCorpus, Text8Corpus or LineSentence for such examples.

min_count (float, optional) – Ignore all words and bigrams with total collected count lower than this value.

threshold (float, optional) – Represent a score threshold for forming the phrases (higher means fewer phrases). A phrase of words a followed by b is accepted if the score of the phrase is greater than threshold. Heavily depends on concrete scoring-function, see the scoring parameter.

max_vocab_size (int, optional) – Maximum size (number of tokens) of the vocabulary. Used to control pruning of less common words, to keep memory under control. The default of 40M needs about 3.6GB of RAM. Increase/decrease max_vocab_size depending on how much available memory you have.

delimiter (str, optional) – Glue character used to join collocation tokens, should be a byte string (e.g. b’_’).

scoring ({'default', 'npmi', function}, optional) –

Specify how potential phrases are scored. scoring can be set with either a string that refers to a built-in scoring function, or with a function with the expected parameter names. Two built-in scoring functions are available by setting scoring to a string:

”default” - original_scorer().

”npmi” - npmi_scorer().

common_terms (set of str, optional) – List of “stop words” that won’t affect frequency count of expressions containing them. Allow to detect expressions like “bank_of_america” or “eye_of_the_beholder”.

The goal of Phraser() is to cut down memory consumption of Phrases(), by discarding model state not strictly needed for the bigram detection task:

In [58]:
bigram = Phraser(phrases)
INFO - 10:30:40: source_vocab length 453893
INFO - 10:30:43: Phraser built with 6551 phrasegrams

Transform the corpus based on the bigrams detected:

In [59]:
articles = bigram[article]

Most Frequent Words:

Mainly a sanity check of the effectiveness of the lemmatization, removal of stopwords, and addition of bigrams.

In [60]:
word_freq = defaultdict(int)
for article in articles:
    for i in article:
        word_freq[i] += 1
len(word_freq)
Out[60]:
45255
In [61]:
sorted(word_freq, key=word_freq.get, reverse=True)[:10]
Out[61]:
['say',
 's',
 'new_york',
 'student',
 'mr',
 'group',
 'york_times',
 'school',
 'city',
 'black']

Training the model

Gensim Word2Vec Implementation:

We use Gensim implementation of word2vec: https://radimrehurek.com/gensim/models/word2vec.html

BEFORE WE DIVE INTO TRAINING OUR OWN MODEL, LET US SEE A DEMO FIRST

To see what Word2Vec can do, let’s download a pre-trained model and play around with it. We will fetch the Word2Vec model trained on part of the Google News dataset, covering approximately 3 million words and phrases. Downloading and loading it with Gensim takes minutes. The model is approximately 2GB, so you’ll need a decent network connection and enough storage to proceed. Otherwise, skip ahead to the “Training Our Own Model” section below.

You can download google news dataset here https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing

In [62]:
from gensim.models import KeyedVectors
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin.gz', binary=True)
INFO - 10:30:54: loading projection weights from GoogleNews-vectors-negative300.bin.gz
INFO - 10:31:24: loaded (3000000, 300) matrix from GoogleNews-vectors-negative300.bin.gz

A common operation is to retrieve the vocabulary of a model. That is trivial:

In [63]:
for i, word in enumerate(model.vocab):
    if i == 10:
        break
    print(word)
</s>
in
for
that
is
on
##
The
with
said

Word2Vec supports several word similarity tasks out of the box. You can see how the similarity intuitively decreases as the words get less and less similar.

In [64]:
pairs = [
    ('car', 'minivan'),   # a minivan is a kind of car
    ('car', 'bicycle'),   # still a wheeled vehicle
    ('car', 'airplane'),  # ok, no wheels, but still a vehicle
    ('car', 'cereal'),    # ... and so on
    ('car', 'communism'),
]
for w1, w2 in pairs:
    print('%r\t%r\t%.2f' % (w1, w2, model.similarity(w1, w2)))
'car'	'minivan'	0.69
'car'	'bicycle'	0.54
'car'	'airplane'	0.42
'car'	'cereal'	0.14
'car'	'communism'	0.06

Check here for a description of cosine similairty https://en.wikipedia.org/wiki/Cosine_similarity

In [65]:
print(model.most_similar(positive=['car', 'minivan'], topn=5)) #Print the 5 most similar words to “car” or “minivan”
print(model.doesnt_match(['fire', 'water', 'land', 'sea', 'air', 'car'])) #Print the one that does not belong in the sequence?
INFO - 10:31:48: precomputing L2-norms of word weight vectors
[('SUV', 0.853219211101532), ('vehicle', 0.8175784349441528), ('pickup_truck', 0.7763689160346985), ('Jeep', 0.7567334175109863), ('Ford_Explorer', 0.756571888923645)]
car
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gensim/models/keyedvectors.py:877: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)
In [66]:
import multiprocessing

from gensim.models import Word2Vec

Train our OWN model

  1. Word2Vec(): In this first step, set up the parameters of the model one-by-one.
  2. .build_vocab(): Here it builds the vocabulary from a sequence of sentences and thus initialized the model.
  3. .train(): Finally, trains the model.
In [67]:
cores = multiprocessing.cpu_count() # Count the number of cores in a computer
cores
Out[67]:
16

The parameters:

sentences (iterable of iterables, optional) – The sentences iterable can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec module for such examples. See also the tutorial on data streaming in Python. If you don’t supply sentences, the model is left uninitialized – use if you plan to initialize it in some other way.

corpus_file (str, optional) – Path to a corpus file in LineSentence format. You may use this argument instead of sentences to get performance boost. Only one of sentences or corpus_file arguments need to be passed (or none of them, in that case, the model is left uninitialized).

size (int, optional) – Dimensionality of the word vectors.

window (int, optional) – Maximum distance between the current and predicted word within a sentence.

min_count (int, optional) – Ignores all words with total frequency lower than this.

workers (int, optional) – Use these many worker threads to train the model (=faster training with multicore machines).

sg ({0, 1}, optional) – Training algorithm: 1 for skip-gram; otherwise CBOW.

hs ({0, 1}, optional) – If 1, hierarchical softmax will be used for model training. If 0, and negative is non-zero, negative sampling will be used.

negative (int, optional) – If > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). If set to 0, no negative sampling is used.

ns_exponent (float, optional) – The exponent used to shape the negative sampling distribution. A value of 1.0 samples exactly in proportion to the frequencies, 0.0 samples all words equally, while a negative value samples low-frequency words more than high-frequency words. The popular default value of 0.75 was chosen by the original Word2Vec paper. More recently, in https://arxiv.org/abs/1804.04212, Caselles-Dupré, Lesaint, & Royo-Letelier suggest that other values may perform better for recommendation applications.

cbow_mean ({0, 1}, optional) – If 0, use the sum of the context word vectors. If 1, use the mean, only applies when cbow is used.

alpha (float, optional) – The initial learning rate.

min_alpha (float, optional) – Learning rate will linearly drop to min_alpha as training progresses.

seed (int, optional) – Seed for the random number generator. Initial vectors for each word are seeded with a hash of the concatenation of word + str(seed). Note that for a fully deterministically-reproducible run, you must also limit the model to a single worker thread (workers=1), to eliminate ordering jitter from OS thread scheduling. (In Python 3, reproducibility between interpreter launches also requires use of the PYTHONHASHSEED environment variable to control hash randomization).

max_vocab_size (int, optional) – Limits the RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM. Set to None for no limit.

max_final_vocab (int, optional) – Limits the vocab to a target vocab size by automatically picking a matching min_count. If the specified min_count is more than the calculated min_count, the specified min_count will be used. Set to None if not required.

sample (float, optional) – The threshold for configuring which higher-frequency words are randomly downsampled, useful range is (0, 1e-5).

hashfxn (function, optional) – Hash function to use to randomly initialize weights, for increased training reproducibility.

iter (int, optional) – Number of iterations (epochs) over the corpus.

trim_rule (function, optional) –

Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to keep_vocab_item()), or a callable that accepts parameters (word, count, min_count) and returns either gensim.utils.RULE_DISCARD, gensim.utils.RULE_KEEP or gensim.utils.RULE_DEFAULT. The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part of the model.

The input parameters are of the following types: word (str) - the word we are examining

count (int) - the word’s frequency count in the corpus

min_count (int) - the minimum count threshold.

sorted_vocab ({0, 1}, optional) – If 1, sort the vocabulary by descending frequency before assigning word indexes. See sort_vocab().

batch_words (int, optional) – Target size (in words) for batches of examples passed to worker threads (and thus cython routines).(Larger batches will be passed if individual texts are longer than 10000 words, but the standard cython code truncates to that maximum.)

compute_loss (bool, optional) – If True, computes and stores loss value which can be retrieved using get_latest_training_loss().

callbacks (iterable of CallbackAny2Vec, optional) – Sequence of callbacks to be executed at specific stages during training.

In [76]:
w2v_model = Word2Vec(min_count=3,
                     window=2,
                     size=300,
                     sample=6e-5, 
                     alpha=0.03, 
                     min_alpha=0.0007, 
                     negative=20,
                     sg=0,
                     workers=cores-1)

Building the Vocabulary Table:

Word2Vec requires us to build the vocabulary table (simply digesting all the words and filtering out the unique words, and doing some basic counts on them):

In [77]:
t = time()

w2v_model.build_vocab(articles, progress_per=10000)

print('Time to build vocab: {} mins'.format(round((time() - t) / 60, 2)))
INFO - 10:34:08: collecting all words and their counts
INFO - 10:34:08: PROGRESS: at sentence #0, processed 0 words, keeping 0 word types
INFO - 10:34:09: collected 45255 word types from a corpus of 576654 raw words and 1980 sentences
INFO - 10:34:09: Loading a fresh vocabulary
INFO - 10:34:09: effective_min_count=3 retains 19177 unique words (42% of original 45255, drops 26078)
INFO - 10:34:09: effective_min_count=3 leaves 545256 word corpus (94% of original 576654, drops 31398)
INFO - 10:34:09: deleting the raw counts dictionary of 45255 items
INFO - 10:34:09: sample=6e-05 downsamples 1099 most-common words
INFO - 10:34:09: downsampling leaves estimated 346596 word corpus (63.6% of prior 545256)
INFO - 10:34:09: estimated required memory for 19177 words and 300 dimensions: 55613300 bytes
INFO - 10:34:09: resetting layer weights
Time to build vocab: 0.07 mins

Training of the model:

Parameters of the training:

  • total_examples = int - Count of articles;
  • epochs = int - Number of iterations (epochs) over the corpus - [10, 20, 30]
In [79]:
t = time()

w2v_model.train(articles, total_examples=w2v_model.corpus_count, epochs=100, report_delay=1)

print('Time to train the model: {} mins'.format(round((time() - t) / 60, 2)))
WARNING - 10:35:12: Effective 'alpha' higher than previous training cycles
INFO - 10:35:12: training model with 15 workers on 19177 vocabulary and 300 features, using sg=0 hs=0 sample=6e-05 negative=20 window=2
INFO - 10:35:13: EPOCH 1 - PROGRESS: at 66.01% examples, 219724 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:13: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:13: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:14: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:14: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:14: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:14: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:14: EPOCH - 1 : training on 576654 raw words (346388 effective words) took 1.5s, 229116 effective words/s
INFO - 10:35:15: EPOCH 2 - PROGRESS: at 66.01% examples, 222419 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:15: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:15: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:15: EPOCH - 2 : training on 576654 raw words (346961 effective words) took 1.5s, 231863 effective words/s
INFO - 10:35:16: EPOCH 3 - PROGRESS: at 66.01% examples, 223446 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:17: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:17: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:17: EPOCH - 3 : training on 576654 raw words (346449 effective words) took 1.5s, 231445 effective words/s
INFO - 10:35:18: EPOCH 4 - PROGRESS: at 66.01% examples, 226647 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:18: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:18: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:18: EPOCH - 4 : training on 576654 raw words (346938 effective words) took 1.5s, 229390 effective words/s
INFO - 10:35:19: EPOCH 5 - PROGRESS: at 66.01% examples, 225178 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:20: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:20: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:20: EPOCH - 5 : training on 576654 raw words (346357 effective words) took 1.5s, 231422 effective words/s
INFO - 10:35:21: EPOCH 6 - PROGRESS: at 66.01% examples, 219672 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:21: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:21: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:21: EPOCH - 6 : training on 576654 raw words (346492 effective words) took 1.5s, 228288 effective words/s
INFO - 10:35:22: EPOCH 7 - PROGRESS: at 66.01% examples, 222345 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:23: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:23: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:23: EPOCH - 7 : training on 576654 raw words (346518 effective words) took 1.5s, 227298 effective words/s
INFO - 10:35:24: EPOCH 8 - PROGRESS: at 66.01% examples, 224710 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:24: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:24: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:24: EPOCH - 8 : training on 576654 raw words (346670 effective words) took 1.5s, 228494 effective words/s
INFO - 10:35:25: EPOCH 9 - PROGRESS: at 52.22% examples, 179635 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:26: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:26: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:26: EPOCH - 9 : training on 576654 raw words (346789 effective words) took 1.5s, 228870 effective words/s
INFO - 10:35:27: EPOCH 10 - PROGRESS: at 66.01% examples, 218570 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:27: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:27: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:27: EPOCH - 10 : training on 576654 raw words (346723 effective words) took 1.5s, 226451 effective words/s
INFO - 10:35:28: EPOCH 11 - PROGRESS: at 66.01% examples, 218184 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:29: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:29: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:29: EPOCH - 11 : training on 576654 raw words (346608 effective words) took 1.5s, 229934 effective words/s
INFO - 10:35:30: EPOCH 12 - PROGRESS: at 66.01% examples, 222528 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:30: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:30: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:30: EPOCH - 12 : training on 576654 raw words (346862 effective words) took 1.5s, 231319 effective words/s
INFO - 10:35:31: EPOCH 13 - PROGRESS: at 66.01% examples, 226397 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:32: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:32: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:32: EPOCH - 13 : training on 576654 raw words (346795 effective words) took 1.5s, 232184 effective words/s
INFO - 10:35:33: EPOCH 14 - PROGRESS: at 64.39% examples, 221329 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:33: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:33: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:33: EPOCH - 14 : training on 576654 raw words (346619 effective words) took 1.5s, 230914 effective words/s
INFO - 10:35:34: EPOCH 15 - PROGRESS: at 66.01% examples, 225389 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:35: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:35: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:35: EPOCH - 15 : training on 576654 raw words (346878 effective words) took 1.5s, 230032 effective words/s
INFO - 10:35:36: EPOCH 16 - PROGRESS: at 66.01% examples, 222093 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:36: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:36: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:36: EPOCH - 16 : training on 576654 raw words (346867 effective words) took 1.5s, 231456 effective words/s
INFO - 10:35:37: EPOCH 17 - PROGRESS: at 67.78% examples, 229327 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:38: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:38: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:38: EPOCH - 17 : training on 576654 raw words (346487 effective words) took 1.5s, 231104 effective words/s
INFO - 10:35:39: EPOCH 18 - PROGRESS: at 66.01% examples, 222306 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:39: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:39: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:39: EPOCH - 18 : training on 576654 raw words (346228 effective words) took 1.5s, 230697 effective words/s
INFO - 10:35:40: EPOCH 19 - PROGRESS: at 67.78% examples, 223383 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:41: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:41: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:41: EPOCH - 19 : training on 576654 raw words (346753 effective words) took 1.5s, 231415 effective words/s
INFO - 10:35:42: EPOCH 20 - PROGRESS: at 66.01% examples, 228117 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:42: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:42: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:42: EPOCH - 20 : training on 576654 raw words (346752 effective words) took 1.5s, 231847 effective words/s
INFO - 10:35:43: EPOCH 21 - PROGRESS: at 66.16% examples, 218830 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:44: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:44: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:44: EPOCH - 21 : training on 576654 raw words (346464 effective words) took 1.5s, 231744 effective words/s
INFO - 10:35:45: EPOCH 22 - PROGRESS: at 63.08% examples, 215473 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:45: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:45: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:45: EPOCH - 22 : training on 576654 raw words (346580 effective words) took 1.5s, 230082 effective words/s
INFO - 10:35:46: EPOCH 23 - PROGRESS: at 66.01% examples, 223349 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:47: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:47: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:47: EPOCH - 23 : training on 576654 raw words (346383 effective words) took 1.5s, 229432 effective words/s
INFO - 10:35:48: EPOCH 24 - PROGRESS: at 66.01% examples, 224155 words/s, in_qsize 1, out_qsize 0
INFO - 10:35:48: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:48: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:48: EPOCH - 24 : training on 576654 raw words (346412 effective words) took 1.5s, 231271 effective words/s
INFO - 10:35:49: EPOCH 25 - PROGRESS: at 66.01% examples, 223409 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:50: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:50: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:50: EPOCH - 25 : training on 576654 raw words (346462 effective words) took 1.5s, 229012 effective words/s
INFO - 10:35:51: EPOCH 26 - PROGRESS: at 64.39% examples, 216982 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:51: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:51: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:51: EPOCH - 26 : training on 576654 raw words (346426 effective words) took 1.5s, 227487 effective words/s
INFO - 10:35:52: EPOCH 27 - PROGRESS: at 64.39% examples, 221124 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:53: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:53: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:53: EPOCH - 27 : training on 576654 raw words (345923 effective words) took 1.5s, 229331 effective words/s
INFO - 10:35:54: EPOCH 28 - PROGRESS: at 64.39% examples, 220714 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:54: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:54: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:54: EPOCH - 28 : training on 576654 raw words (346340 effective words) took 1.5s, 229510 effective words/s
INFO - 10:35:55: EPOCH 29 - PROGRESS: at 64.39% examples, 218201 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:56: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:56: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:56: EPOCH - 29 : training on 576654 raw words (346593 effective words) took 1.5s, 226718 effective words/s
INFO - 10:35:57: EPOCH 30 - PROGRESS: at 62.53% examples, 215126 words/s, in_qsize 0, out_qsize 1
INFO - 10:35:57: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:57: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:57: EPOCH - 30 : training on 576654 raw words (346628 effective words) took 1.5s, 231022 effective words/s
INFO - 10:35:59: EPOCH 31 - PROGRESS: at 66.01% examples, 222629 words/s, in_qsize 0, out_qsize 0
INFO - 10:35:59: worker thread finished; awaiting finish of 14 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 13 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 12 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 11 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 10 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 9 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 8 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 7 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 6 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 5 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 4 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 3 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 2 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 1 more threads
INFO - 10:35:59: worker thread finished; awaiting finish of 0 more threads
INFO - 10:35:59: EPOCH - 31 : training on 576654 raw words (346503 effective words) took 1.5s, 231205 effective words/s
INFO - 10:36:00: EPOCH 32 - PROGRESS: at 66.01% examples, 221861 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:00: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:00: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:01: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:01: EPOCH - 32 : training on 576654 raw words (346701 effective words) took 1.5s, 230520 effective words/s
INFO - 10:36:02: EPOCH 33 - PROGRESS: at 66.01% examples, 226700 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:02: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:02: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:02: EPOCH - 33 : training on 576654 raw words (346258 effective words) took 1.5s, 229817 effective words/s
INFO - 10:36:03: EPOCH 34 - PROGRESS: at 62.53% examples, 215257 words/s, in_qsize 0, out_qsize 1
INFO - 10:36:04: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:04: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:04: EPOCH - 34 : training on 576654 raw words (346660 effective words) took 1.5s, 225227 effective words/s
INFO - 10:36:05: EPOCH 35 - PROGRESS: at 66.16% examples, 214834 words/s, in_qsize 0, out_qsize 2
INFO - 10:36:05: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:05: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:05: EPOCH - 35 : training on 576654 raw words (346424 effective words) took 1.5s, 229834 effective words/s
INFO - 10:36:06: EPOCH 36 - PROGRESS: at 64.39% examples, 220167 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:07: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:07: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:07: EPOCH - 36 : training on 576654 raw words (346713 effective words) took 1.5s, 228267 effective words/s
INFO - 10:36:08: EPOCH 37 - PROGRESS: at 66.01% examples, 222391 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:08: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:08: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:08: EPOCH - 37 : training on 576654 raw words (346188 effective words) took 1.5s, 229191 effective words/s
INFO - 10:36:09: EPOCH 38 - PROGRESS: at 66.01% examples, 225179 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:10: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:10: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:10: EPOCH - 38 : training on 576654 raw words (346865 effective words) took 1.5s, 230119 effective words/s
INFO - 10:36:11: EPOCH 39 - PROGRESS: at 64.39% examples, 217452 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:11: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:11: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:11: EPOCH - 39 : training on 576654 raw words (346205 effective words) took 1.5s, 227468 effective words/s
INFO - 10:36:12: EPOCH 40 - PROGRESS: at 66.01% examples, 227320 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:13: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:13: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:13: EPOCH - 40 : training on 576654 raw words (346229 effective words) took 1.5s, 228203 effective words/s
INFO - 10:36:14: EPOCH 41 - PROGRESS: at 66.01% examples, 227298 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:14: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:14: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:14: EPOCH - 41 : training on 576654 raw words (346873 effective words) took 1.5s, 230811 effective words/s
INFO - 10:36:15: EPOCH 42 - PROGRESS: at 64.39% examples, 220490 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:16: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:16: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:16: EPOCH - 42 : training on 576654 raw words (346923 effective words) took 1.5s, 230743 effective words/s
INFO - 10:36:17: EPOCH 43 - PROGRESS: at 67.78% examples, 227466 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:17: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:17: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:17: EPOCH - 43 : training on 576654 raw words (346534 effective words) took 1.5s, 230406 effective words/s
INFO - 10:36:18: EPOCH 44 - PROGRESS: at 67.78% examples, 223357 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:19: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:19: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:19: EPOCH - 44 : training on 576654 raw words (346661 effective words) took 1.5s, 231403 effective words/s
INFO - 10:36:20: EPOCH 45 - PROGRESS: at 66.01% examples, 227310 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:20: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:20: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:20: EPOCH - 45 : training on 576654 raw words (346641 effective words) took 1.5s, 231315 effective words/s
INFO - 10:36:21: EPOCH 46 - PROGRESS: at 66.01% examples, 220868 words/s, in_qsize 0, out_qsize 1
INFO - 10:36:22: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:22: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:22: EPOCH - 46 : training on 576654 raw words (346503 effective words) took 1.5s, 227774 effective words/s
INFO - 10:36:23: EPOCH 47 - PROGRESS: at 66.01% examples, 225489 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:23: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:23: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:23: EPOCH - 47 : training on 576654 raw words (346367 effective words) took 1.5s, 227114 effective words/s
INFO - 10:36:24: EPOCH 48 - PROGRESS: at 66.01% examples, 226557 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:25: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:25: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:25: EPOCH - 48 : training on 576654 raw words (346853 effective words) took 1.5s, 231672 effective words/s
INFO - 10:36:26: EPOCH 49 - PROGRESS: at 66.01% examples, 226344 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:26: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:26: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:26: EPOCH - 49 : training on 576654 raw words (346175 effective words) took 1.5s, 230116 effective words/s
INFO - 10:36:27: EPOCH 50 - PROGRESS: at 67.78% examples, 228378 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:28: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:28: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:28: EPOCH - 50 : training on 576654 raw words (346305 effective words) took 1.5s, 230473 effective words/s
INFO - 10:36:29: EPOCH 51 - PROGRESS: at 64.39% examples, 222270 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:29: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:29: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:29: EPOCH - 51 : training on 576654 raw words (346808 effective words) took 1.5s, 230783 effective words/s
INFO - 10:36:30: EPOCH 52 - PROGRESS: at 66.01% examples, 222677 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:31: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:31: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:31: EPOCH - 52 : training on 576654 raw words (346341 effective words) took 1.5s, 231351 effective words/s
INFO - 10:36:32: EPOCH 53 - PROGRESS: at 64.39% examples, 220743 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:32: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:32: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:32: EPOCH - 53 : training on 576654 raw words (346977 effective words) took 1.5s, 230813 effective words/s
INFO - 10:36:33: EPOCH 54 - PROGRESS: at 66.01% examples, 221519 words/s, in_qsize 1, out_qsize 0
INFO - 10:36:34: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:34: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:34: EPOCH - 54 : training on 576654 raw words (346676 effective words) took 1.5s, 228242 effective words/s
INFO - 10:36:35: EPOCH 55 - PROGRESS: at 66.01% examples, 224560 words/s, in_qsize 1, out_qsize 0
INFO - 10:36:35: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:35: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:35: EPOCH - 55 : training on 576654 raw words (346603 effective words) took 1.5s, 231948 effective words/s
INFO - 10:36:36: EPOCH 56 - PROGRESS: at 66.01% examples, 225708 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:37: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:37: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:37: EPOCH - 56 : training on 576654 raw words (346557 effective words) took 1.5s, 230710 effective words/s
INFO - 10:36:38: EPOCH 57 - PROGRESS: at 67.78% examples, 228231 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:38: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:38: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:38: EPOCH - 57 : training on 576654 raw words (346347 effective words) took 1.5s, 231810 effective words/s
INFO - 10:36:39: EPOCH 58 - PROGRESS: at 66.01% examples, 227979 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:40: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:40: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:40: EPOCH - 58 : training on 576654 raw words (346537 effective words) took 1.5s, 230530 effective words/s
INFO - 10:36:41: EPOCH 59 - PROGRESS: at 66.01% examples, 221892 words/s, in_qsize 0, out_qsize 1
INFO - 10:36:41: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:41: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:41: EPOCH - 59 : training on 576654 raw words (346379 effective words) took 1.5s, 230506 effective words/s
INFO - 10:36:42: EPOCH 60 - PROGRESS: at 64.39% examples, 220430 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:43: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:43: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:43: EPOCH - 60 : training on 576654 raw words (346751 effective words) took 1.5s, 229590 effective words/s
INFO - 10:36:44: EPOCH 61 - PROGRESS: at 66.01% examples, 226418 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:44: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:44: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:45: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:45: EPOCH - 61 : training on 576654 raw words (346904 effective words) took 1.5s, 228909 effective words/s
INFO - 10:36:46: EPOCH 62 - PROGRESS: at 66.01% examples, 224760 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:46: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:46: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:46: EPOCH - 62 : training on 576654 raw words (346388 effective words) took 1.5s, 228948 effective words/s
INFO - 10:36:47: EPOCH 63 - PROGRESS: at 64.39% examples, 221248 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:48: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:48: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:48: EPOCH - 63 : training on 576654 raw words (346257 effective words) took 1.5s, 230281 effective words/s
INFO - 10:36:49: EPOCH 64 - PROGRESS: at 66.01% examples, 226667 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:49: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:49: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:49: EPOCH - 64 : training on 576654 raw words (347025 effective words) took 1.5s, 230666 effective words/s
INFO - 10:36:50: EPOCH 65 - PROGRESS: at 66.01% examples, 226991 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:51: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:51: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:51: EPOCH - 65 : training on 576654 raw words (346579 effective words) took 1.5s, 231350 effective words/s
INFO - 10:36:52: EPOCH 66 - PROGRESS: at 67.78% examples, 225406 words/s, in_qsize 1, out_qsize 0
INFO - 10:36:52: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:52: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:52: EPOCH - 66 : training on 576654 raw words (346713 effective words) took 1.5s, 231773 effective words/s
INFO - 10:36:53: EPOCH 67 - PROGRESS: at 66.01% examples, 226614 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:54: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:54: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:54: EPOCH - 67 : training on 576654 raw words (346593 effective words) took 1.5s, 231512 effective words/s
INFO - 10:36:55: EPOCH 68 - PROGRESS: at 66.01% examples, 227988 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:55: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:55: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:55: EPOCH - 68 : training on 576654 raw words (346948 effective words) took 1.5s, 231553 effective words/s
INFO - 10:36:56: EPOCH 69 - PROGRESS: at 66.01% examples, 222773 words/s, in_qsize 0, out_qsize 1
INFO - 10:36:57: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:57: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:57: EPOCH - 69 : training on 576654 raw words (346770 effective words) took 1.5s, 231610 effective words/s
INFO - 10:36:58: EPOCH 70 - PROGRESS: at 66.01% examples, 226835 words/s, in_qsize 0, out_qsize 0
INFO - 10:36:58: worker thread finished; awaiting finish of 14 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 13 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 12 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 11 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 10 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 9 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 8 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 7 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 6 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 5 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 4 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 3 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 2 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 1 more threads
INFO - 10:36:58: worker thread finished; awaiting finish of 0 more threads
INFO - 10:36:58: EPOCH - 70 : training on 576654 raw words (346156 effective words) took 1.5s, 231313 effective words/s
INFO - 10:36:59: EPOCH 71 - PROGRESS: at 66.01% examples, 224788 words/s, in_qsize 1, out_qsize 0
INFO - 10:37:00: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:00: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:00: EPOCH - 71 : training on 576654 raw words (346584 effective words) took 1.5s, 231362 effective words/s
INFO - 10:37:01: EPOCH 72 - PROGRESS: at 66.01% examples, 224821 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:01: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:01: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:01: EPOCH - 72 : training on 576654 raw words (346499 effective words) took 1.5s, 231469 effective words/s
INFO - 10:37:02: EPOCH 73 - PROGRESS: at 66.01% examples, 222985 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:03: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:03: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:03: EPOCH - 73 : training on 576654 raw words (346722 effective words) took 1.5s, 231734 effective words/s
INFO - 10:37:04: EPOCH 74 - PROGRESS: at 66.01% examples, 227078 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:04: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:04: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:04: EPOCH - 74 : training on 576654 raw words (346970 effective words) took 1.5s, 231343 effective words/s
INFO - 10:37:05: EPOCH 75 - PROGRESS: at 66.01% examples, 226665 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:06: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:06: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:06: EPOCH - 75 : training on 576654 raw words (346881 effective words) took 1.5s, 231361 effective words/s
INFO - 10:37:07: EPOCH 76 - PROGRESS: at 66.01% examples, 226353 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:07: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:07: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:07: EPOCH - 76 : training on 576654 raw words (346798 effective words) took 1.5s, 231466 effective words/s
INFO - 10:37:08: EPOCH 77 - PROGRESS: at 66.01% examples, 226832 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:09: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:09: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:09: EPOCH - 77 : training on 576654 raw words (346681 effective words) took 1.5s, 231456 effective words/s
INFO - 10:37:10: EPOCH 78 - PROGRESS: at 67.78% examples, 229622 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:10: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:10: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:10: EPOCH - 78 : training on 576654 raw words (346155 effective words) took 1.5s, 231244 effective words/s
INFO - 10:37:11: EPOCH 79 - PROGRESS: at 67.78% examples, 223118 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:12: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:12: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:12: EPOCH - 79 : training on 576654 raw words (346513 effective words) took 1.5s, 232112 effective words/s
INFO - 10:37:13: EPOCH 80 - PROGRESS: at 52.58% examples, 170724 words/s, in_qsize 2, out_qsize 0
INFO - 10:37:13: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:13: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:13: EPOCH - 80 : training on 576654 raw words (346953 effective words) took 1.4s, 243692 effective words/s
INFO - 10:37:14: EPOCH 81 - PROGRESS: at 67.78% examples, 223173 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:15: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:15: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:15: EPOCH - 81 : training on 576654 raw words (346565 effective words) took 1.5s, 231417 effective words/s
INFO - 10:37:16: EPOCH 82 - PROGRESS: at 67.78% examples, 230158 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:16: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:16: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:16: EPOCH - 82 : training on 576654 raw words (346642 effective words) took 1.5s, 231848 effective words/s
INFO - 10:37:17: EPOCH 83 - PROGRESS: at 66.01% examples, 223074 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:18: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:18: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:18: EPOCH - 83 : training on 576654 raw words (346566 effective words) took 1.5s, 231996 effective words/s
INFO - 10:37:19: EPOCH 84 - PROGRESS: at 64.39% examples, 222453 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:19: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:19: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:19: EPOCH - 84 : training on 576654 raw words (347025 effective words) took 1.5s, 231523 effective words/s
INFO - 10:37:20: EPOCH 85 - PROGRESS: at 67.78% examples, 230134 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:21: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:21: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:21: EPOCH - 85 : training on 576654 raw words (346488 effective words) took 1.5s, 231842 effective words/s
INFO - 10:37:22: EPOCH 86 - PROGRESS: at 66.01% examples, 224292 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:22: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:22: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:22: EPOCH - 86 : training on 576654 raw words (346950 effective words) took 1.5s, 233018 effective words/s
INFO - 10:37:23: EPOCH 87 - PROGRESS: at 67.78% examples, 222901 words/s, in_qsize 0, out_qsize 1
INFO - 10:37:24: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:24: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:24: EPOCH - 87 : training on 576654 raw words (346752 effective words) took 1.5s, 231788 effective words/s
INFO - 10:37:25: EPOCH 88 - PROGRESS: at 66.01% examples, 226511 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:25: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:25: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:25: EPOCH - 88 : training on 576654 raw words (346489 effective words) took 1.5s, 231056 effective words/s
INFO - 10:37:26: EPOCH 89 - PROGRESS: at 64.39% examples, 221847 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:27: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:27: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:27: EPOCH - 89 : training on 576654 raw words (346782 effective words) took 1.5s, 230811 effective words/s
INFO - 10:37:28: EPOCH 90 - PROGRESS: at 66.01% examples, 226319 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:28: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:28: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:28: EPOCH - 90 : training on 576654 raw words (346788 effective words) took 1.5s, 231174 effective words/s
INFO - 10:37:29: EPOCH 91 - PROGRESS: at 67.78% examples, 228637 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:30: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:30: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:30: EPOCH - 91 : training on 576654 raw words (346814 effective words) took 1.5s, 231552 effective words/s
INFO - 10:37:31: EPOCH 92 - PROGRESS: at 64.39% examples, 222218 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:31: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:31: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:31: EPOCH - 92 : training on 576654 raw words (346531 effective words) took 1.5s, 231595 effective words/s
INFO - 10:37:32: EPOCH 93 - PROGRESS: at 64.39% examples, 221733 words/s, in_qsize 0, out_qsize 1
INFO - 10:37:33: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:33: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:33: EPOCH - 93 : training on 576654 raw words (346789 effective words) took 1.5s, 232038 effective words/s
INFO - 10:37:34: EPOCH 94 - PROGRESS: at 66.01% examples, 222710 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:34: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:34: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:34: EPOCH - 94 : training on 576654 raw words (346464 effective words) took 1.5s, 230568 effective words/s
INFO - 10:37:35: EPOCH 95 - PROGRESS: at 66.01% examples, 227891 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:36: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:36: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:36: EPOCH - 95 : training on 576654 raw words (346551 effective words) took 1.5s, 231283 effective words/s
INFO - 10:37:37: EPOCH 96 - PROGRESS: at 64.39% examples, 221835 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:37: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:37: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:37: EPOCH - 96 : training on 576654 raw words (346366 effective words) took 1.5s, 230772 effective words/s
INFO - 10:37:38: EPOCH 97 - PROGRESS: at 66.01% examples, 222641 words/s, in_qsize 0, out_qsize 1
INFO - 10:37:39: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:39: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:39: EPOCH - 97 : training on 576654 raw words (346318 effective words) took 1.5s, 231733 effective words/s
INFO - 10:37:40: EPOCH 98 - PROGRESS: at 66.01% examples, 226153 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:40: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:40: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:40: EPOCH - 98 : training on 576654 raw words (346317 effective words) took 1.5s, 231332 effective words/s
INFO - 10:37:41: EPOCH 99 - PROGRESS: at 66.01% examples, 228255 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:42: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:42: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:42: EPOCH - 99 : training on 576654 raw words (346593 effective words) took 1.5s, 232220 effective words/s
INFO - 10:37:43: EPOCH 100 - PROGRESS: at 66.01% examples, 223644 words/s, in_qsize 0, out_qsize 0
INFO - 10:37:43: worker thread finished; awaiting finish of 14 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 13 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 12 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 11 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 10 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 9 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 8 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 7 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 6 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 5 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 4 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 3 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 2 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 1 more threads
INFO - 10:37:43: worker thread finished; awaiting finish of 0 more threads
INFO - 10:37:43: EPOCH - 100 : training on 576654 raw words (346709 effective words) took 1.5s, 232300 effective words/s
INFO - 10:37:43: training on a 57665400 raw words (34659490 effective words) took 151.2s, 229240 effective words/s
Time to train the model: 2.52 mins

As we do not plan to train the model any further, we are calling init_sims(), which will make the model much more memory-efficient:

In [80]:
w2v_model.init_sims(replace=True)
INFO - 10:46:31: precomputing L2-norms of word weight vectors

Storing and loading models¶

You’ll notice that training non-trivial models can take time. Once you’ve trained your model and it works as expected, you can save it to disk. That way, you don’t have to spend time training it all over again later.

You can store/load models using the standard gensim methods:

In [40]:
import tempfile
import gensim

with tempfile.NamedTemporaryFile(prefix='gensim-model-', delete=False) as tmp:
    temporary_filepath = tmp.name
    w2v_model.save(temporary_filepath)
    #
    # The model is now safely stored in the filepath.
    # You can copy it to other machines, share it with others, etc.
    #
# To load a saved model:

new_model = gensim.models.Word2Vec.load(temporary_filepath)
INFO - 10:26:30: saving Word2Vec object under /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm, separately None
INFO - 10:26:30: not storing attribute vectors_norm
INFO - 10:26:30: not storing attribute cum_table
INFO - 10:26:30: saved /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm
INFO - 10:26:30: loading Word2Vec object from /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm
INFO - 10:26:30: loading wv recursively from /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm.wv.* with mmap=None
INFO - 10:26:30: setting ignored attribute vectors_norm to None
INFO - 10:26:30: loading vocabulary recursively from /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm.vocabulary.* with mmap=None
INFO - 10:26:30: loading trainables recursively from /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm.trainables.* with mmap=None
INFO - 10:26:30: setting ignored attribute cum_table to None
INFO - 10:26:30: loaded /var/folders/cj/pyr99bvn3xd5qjy_m40ld9kc0000gn/T/gensim-model-qrmx7gdm

Exploring the model

The main part of the model is w2v_model.wv, where “wv” stands for “word vectors”.

Most similar to:

Here, we will ask our model to find the word most similar to "new"

Let's see what we get for the "new":

In [81]:
w2v_model.wv.most_similar(positive=["new"])
Out[81]:
[('state', 0.3105401396751404),
 ('plan', 0.30601766705513),
 ('city', 0.2809823751449585),
 ('year', 0.2775835692882538),
 ('court', 0.2735474109649658),
 ('s', 0.2733498811721802),
 ('require', 0.2732350826263428),
 ('decision', 0.26876622438430786),
 ('contract', 0.26309239864349365),
 ('school', 0.2575671672821045)]

A Note here:
We only use 2000 articles to train our word embeddings.

Let's see what the bigram "new_york":

In [82]:
w2v_model.wv.most_similar(positive=["new_york"])
Out[82]:
[('times_current', 0.7055099010467529),
 ('york_times', 0.6473978757858276),
 ('times', 0.6473656892776489),
 ('proquest_historical', 0.6095507144927979),
 ('newspaper_new', 0.6021746397018433),
 ('ps_bs', 0.5700998902320862),
 ('ew_york', 0.5552119016647339),
 ('current_fle', 0.5476503968238831),
 ('pg', 0.5311660766601562),
 ('proqu', 0.5239004492759705)]
In [83]:
w2v_model.wv.most_similar(positive=["activist"])
Out[83]:
[('joseph_b', 0.27729183435440063),
 ('vienna', 0.2686992883682251),
 ('treaster', 0.2660790681838989),
 ('group', 0.24379265308380127),
 ('align', 0.23536896705627441),
 ('farm_union', 0.22481276094913483),
 ('support', 0.22475281357765198),
 ('laymen', 0.21873614192008972),
 ('policie', 0.2175137996673584),
 ('consumer_advocate', 0.2151505947113037)]
In [75]:
w2v_model.wv.most_similar(positive=["movement"])
Out[75]:
[('belligerent', 0.2227466106414795),
 ('delaying', 0.20663030445575714),
 ('foreclosure', 0.20028670132160187),
 ('anonymously', 0.19974614679813385),
 ('vietcong', 0.1975529044866562),
 ('turnaround', 0.19336067140102386),
 ('inside_outside', 0.1912272870540619),
 ('sivertsen', 0.18977245688438416),
 ('burling', 0.18529903888702393),
 ('health_safety', 0.18120959401130676)]

Looks like it is making sense!

Similarities:

Here, we will see how similar are two words to each other :

In [84]:
w2v_model.wv.similarity("activist", 'movement')
Out[84]:
0.18649663
In [85]:
w2v_model.wv.similarity('activist', 'protest')
Out[85]:
0.106794365

Analogy difference:

Which word is to protest as activist is to movement?

In [86]:
w2v_model.wv.most_similar(positive=["activist", "movement"], negative=["protest"], topn=3)
Out[86]:
[('reconstruction', 0.23501867055892944),
 ('goff', 0.23463329672813416),
 ('equate', 0.23000328242778778)]
In [87]:
w2v_model.wv.most_similar(positive=["black", "activist"], negative=["white"], topn=3)
Out[87]:
[('laymen', 0.2714104652404785),
 ('palestine_liberation', 0.2692701816558838),
 ('group', 0.2545587718486786)]

Visualising the Word Embeddings

The word embeddings made by the model can be visualised by reducing dimensionality of the words to 2 dimensions using tSNE.t-SNE is a non-linear dimensionality reduction algorithm that attempts to represent high-dimensional data and the underlying relationships between vectors in a lower-dimensional space.Here is a good tutorial on it: https://medium.com/@luckylwk/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b

Visualisations can be used to notice semantic and syntactic trends in the data.

Example:

Semantic: words like cat, dog, cow, etc. have a tendency to lie close by

Syntactic: words like run, running or cut, cutting lie close together.

Vector relations like vKing - vMan = vQueen - vWoman can also be noticed.

In [90]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
 
import seaborn as sns
sns.set_style("darkgrid")

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

Our goal in this section is to plot our 300 dimensions vectors into 2 dimensional graphs, and see if we can spot interesting patterns.
For that we are going to use t-SNE implementation from scikit-learn.

To make the visualizations more relevant, we will look at the relationships between a query word (in **red**), its most similar words in the model (in **blue**), and other words from the vocabulary (in **green**).

In [91]:
def tsnescatterplot(model, word, list_names):
    """ Plot in seaborn the results from the t-SNE dimensionality reduction algorithm of the vectors of a query word,
    its list of most similar words, and a list of words.
    """
    arrays = np.empty((0, 300), dtype='f')
    word_labels = [word]
    color_list  = ['red']

    # adds the vector of the query word
    arrays = np.append(arrays, model.wv.__getitem__([word]), axis=0)
    
    # gets list of most similar words
    close_words = model.wv.most_similar([word])
    
    # adds the vector for each of the closest words to the array
    for wrd_score in close_words:
        wrd_vector = model.wv.__getitem__([wrd_score[0]])
        word_labels.append(wrd_score[0])
        color_list.append('blue')
        arrays = np.append(arrays, wrd_vector, axis=0)
    
    # adds the vector for each of the words from list_names to the array
    for wrd in list_names:
        wrd_vector = model.wv.__getitem__([wrd])
        word_labels.append(wrd)
        color_list.append('green')
        arrays = np.append(arrays, wrd_vector, axis=0)
        
    # Reduces the dimensionality from 300 to 50 dimensions with PCA
    reduc = PCA(n_components=5).fit_transform(arrays)
    
    # Finds t-SNE coordinates for 2 dimensions
    np.set_printoptions(suppress=True)
    
    Y = TSNE(n_components=2, random_state=0, perplexity=15).fit_transform(reduc)
    
    # Sets everything up to plot
    df = pd.DataFrame({'x': [x for x in Y[:, 0]],
                       'y': [y for y in Y[:, 1]],
                       'words': word_labels,
                       'color': color_list})
    
    fig, _ = plt.subplots()
    fig.set_size_inches(9, 9)
    
    # Basic plot
    p1 = sns.regplot(data=df,
                     x="x",
                     y="y",
                     fit_reg=False,
                     marker="o",
                     scatter_kws={'s': 40,
                                  'facecolors': df['color']
                                 }
                    )
    
    # Adds annotations one by one with a loop
    for line in range(0, df.shape[0]):
         p1.text(df["x"][line],
                 df['y'][line],
                 '  ' + df["words"][line].title(),
                 horizontalalignment='left',
                 verticalalignment='bottom', size='medium',
                 color=df['color'][line],
                 weight='normal'
                ).set_size(15)

    
    plt.xlim(Y[:, 0].min()-50, Y[:, 0].max()+50)
    plt.ylim(Y[:, 1].min()-50, Y[:, 1].max()+50)
            
    plt.title('t-SNE visualization for {}'.format(word.title()))
    

10 Most similar words vs. 5 Random words:

Let's compare where the vector representation of activist, the 10 most similar words from the model, as well as 5 random ones, lies in a 2D graph:

In [92]:
tsnescatterplot(w2v_model, 'activist', ['new', 'times', 'today', 'great', 'black'])

Interestingly, the 10 most similar words to Homer ends up around him, so does Apu and (sideshow) Bob, two recurrent characters.

10 Most similar words vs. 10 Most dissimilar

This time, let's compare where the vector representation of Maggie and her 10 most similar words from the model lies compare to the vector representation of the 10 most dissimilar words to Maggie:

In [94]:
tsnescatterplot(w2v_model, 'protest', [i[0] for i in w2v_model.wv.most_similar(negative=["protest"])])

Lab problem set released

Replicate our tutorial using NYT news article TITLE_DOCA instead of TEXT. Please send me a screenshot for your extra credits.

Materials for more in depths understanding:

  • Word Embeddings introduction: https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/
  • Another Word2Vec introduction: http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
  • Le and Mikolov in 2014 introduced the Doc2Vec algorithm, which usually outperforms such simple-averaging of Word2Vec vectors. The basic idea is: act as if a document has another floating word-like vector, which contributes to all training predictions, and is updated like other word-vectors, but we will call it a doc-vector. Gensim’s Doc2Vec class implements this algorithm. There are two implementations: Paragraph Vector - Distributed Memory (PV-DM) and Paragraph Vector - Distributed Bag of Words (PV-DBOW). PV-DM is analogous to Word2Vec CBOW. The doc-vectors are obtained by training a neural network on the synthetic task of predicting a center word based an average of both context word-vectors and the full document’s doc-vector. PV-DBOW is analogous to Word2Vec SG. The doc-vectors are obtained by training a neural network on the synthetic task of predicting a target word just from the full document’s doc-vector. (It is also common to combine this with skip-gram testing, using both the doc-vector and nearby word-vectors to predict a single target word, but only one at a time.) You can check here for more details. https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html#sphx-glr-auto-examples-tutorials-run-doc2vec-lee-py

Acknowledgements

End

In [ ]: