Changes between Version 4 and Version 5 of FactoidSteveIdeas

Show
Ignore:
Author:
jamesh (IP: 128.220.117.40)
Timestamp:
07/14/09 01:54:14 (4 months ago)
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • FactoidSteveIdeas

    v4 v5  
    1 == 2nd Run == 
     1= Steve's thoughts on experiments = 
     2 
     3== 1st Thoughts == 
     4 
     5View the task as one of domain adaptation, with constraints providing higher quality training data than self-training alone. Of course the methods are more generally applicable than domain adaptation, but this is one useful and compelling application of the techniques. 
     6 
     7The trick now is to find a domain which isn't too artificial but one for which we're likely to get accuracy gains. The domain generated by the original queries is an obvious one to use, eg the "Alexander Graham Bell inventing the telephone domain". In order to widen this domain a little, probably better to have the "Alexander Graham Bell telephone" domain, where the inventing is implicit; this also allows the creation, pioneering, etc of the telephone. In fact, one could even imagine an application of this domain. Suppose we have a QA system for which users often ask about Alexander Graham Bell; here it would be useful to have a parser which is accurate on these sentences. The ultimate application would be real-time learning of the parser when the user enters the query. 
     8 
     9=== Suggested datasets === 
     10 
     115 different domains, based on pairs of named entities. Eg (Alexander Graham Bell, telephone). 
     12 
     1350 sentences for each from the relevant query annotated with gold-standard GRs for testing. The advantage in having these marked up is that we can also use this test data to measure the accuracy of the parses after the constraints have been applied. 
     14 
     15300+ sentences for training for each domain. 300 might be enough given the domain is so constrained. 
     16 
     17=== Experiments (for each domain) === 
     18 
     19 1. Measure the accuracy of the parser on the test sentences before and after applying the constraints. 
     20 1. Train the supertagger on the 1-best parse from the constrained training sentences. Do the same on the unconstrained 1-best parse. Add the training sentences, say, 10 times to the CCGbank data. Measure performance of the supertagger and parser using: 
     21   * wsj model 
     22   * wsj+wiki model (no constraints) 
     23   * wsj+wiki model (with constraints)  
     24 1. Repeat the above for training the full parsing model. Measure accuracy of the wsj model, the wsj+wiki model without constraints, and the wsj+wiki model with constraints. The hope is that the performance will increase in each case.  
     25 
     26== 2nd Thoughts == 
    227 
    328We want to demonstrate the potential utility of the constrained data. Problem: we're not going to improve on the full wsj parsing task, or indeed the performance of the parser on wikipedia with a full model, just because this sets the baseline too high at this stage. So we need an alternative experimental framework which isn't too noddy or artificial -- so that people will be convinced that the method might generalise to the full case and is worth pursuing -- but on the other hand is noddy enough that we can get some +ve results in the next 4 weeks. 
    732 1. start out with a model trained on a subset of CCGbank (eg 5,000 sentences?) and try and improve performance on the wikipedia test data. (we can also use the wsj test data of course). This mirrors the situation where training data is sparse and where we're using the bootstrapping as a cheap way of getting more data. The big advantage with this one is that we already have all the data. It would also demonstrate that the constraints can provide broadly applicable data which is better because of a knock-on effect of the constraints applying in the training data, rather than just acquiring specific knowledge about inventor sentences, say. 
    833 
    9  1. create around 3 new "domains" defined by some manually chosen queries (ones which appear to provide useful constraints). Eg the "inventor" domain. Extract N sentences using the query, where ideally N is around 1000; take 50 for manual annotation with GRs; use the rest for training. Then we can see this as a kind of domain adaptation exercise, albeit one with rather tightly constrained domains. Note that the constraints can generlise across the inventors, eg a constraint of the form INVENTOR pioneered INVENTION can be extracted from, and apply to, both Edison and Dyson. The big disadvantage with this one is the need to annotate test data. 
     34 2. create around 3 new "domains" defined by some manually chosen queries (ones which appear to provide useful constraints). Eg the "inventor" domain. Extract N sentences using the query, where ideally N is around 1000; take 50 for manual annotation with GRs; use the rest for training. Then we can see this as a kind of domain adaptation exercise, albeit one with rather tightly constrained domains. Note that the constraints can generlise across the inventors, eg a constraint of the form INVENTOR pioneered INVENTION can be extracted from, and apply to, both Edison and Dyson. The big disadvantage with this one is the need to annotate test data. 
    1035 
    1136Hence proposal is to try 1) next week.