Self Training

• Research questions:
  • can we improve performance by training on self labelled data?
  • how to train the supertagger using the parser

• Earlier work
  • the final report of the 2002 CLSP workshop team working on semi-supervised methods for statistical parsing: http://www.clsp.jhu.edu/ws2002/groups/wide/
  • David McClosky et al.(2006) Effective Self-Training for Parsing http://cs.brown.edu/~dmcc/papers/dmcc-naacl-2006.pdf
  • McClosky et al.(2008) When is Self-Training Effective for Parsing? http://cs.brown.edu/~dmcc/papers/dmcc-coling-2008.pdf

In Progress

• Implementing a map-reduce based combining of the features

Progress Updates

• parallelised the feature extraction process
• broke up the wikipedia dump and parsed it, producing ccgbank derivations and grs
• parsed sections 2-21 of WSJ and retrained the supertagger on the parsed output
• added features to the supertagger
  • extending the window to three words on either side rather than two
  • trigram tags (up to 3 tags both directions)
  • bigram and trigram words (up to 3 words both directions)
  • added flags for all extended features (all false by default except bigram tags up to 2 in both directions)
• added MapReduce MPI implementation to supertagger training process
  • http://www.sandia.gov/~sjplimp/mapreduce/doc/mapreduce.html

Stages of Parallelization of Supertagger Training

• Stage One: no parallelization

• Stage Two: Jono's MPI implementation of weight estimation

• Stage Three: Jono & Jessi's python script for parallelizing feature extraction

• Stage Four: James & Jessi's MapReduce MPI implementaiton of entire training process

• Stage Five: complete parallelization, including initial training data

Evaluating the Parser

• to evaluate parser output (grs):
  • run the output of the parser through the post-processing script (in candc/candc-1.00/src/scripts/ccg/grs2depbank)
  • run evalParse.pl (attached)
    • usage: perl evalParse.pl <TEST_GRS> <GOLD_GRS>
    • gold grs attached (Steve's 300 annotated sentences)

Experiments

• retraining the POS tagger on WSJ plus 60 gold standard Wikipedia sentences from Steve's annoated data, duplicated 10 times and added to WSJ 02-21
  • evaluated on Wikipedia data
  • results
    • trained on WSJ only: 97.87%
    • trained on WSJ + Wikipedia: 97.68%
• retraining the supertagger on WSJ plus 60 gold standard Wikipedia sentences, duplicated 10 times and added to WSJ 02-21

training	Gold POS tags	Auto POS tags
WSJ only	91.05%	89.95%
WSJ+Wiki	91.26%	90.29%

• evaluating the parser using the newly trained POS tagger and supertagger models

training	Gold POS tags	Auto POS tags
WSJ only	P = 82.27; R = 82.72; F = 82.48	P = 81.01; R = 80.85; F = 80.90
WSJ+wiki	P = 82.21; R = 82.82; F = 82.50	P = 80.15; R = 82.18; F = 81.15