Optional apply logprob computation at call site instead of construction site

[jerinphilip]

This issue is meant to track the possibility of some workaround to get QE to be optional at run time, as opposed to construction time (skip-cost= true).

Trace:

skip-cost:

marian-dev/src/translator/scorers.cpp

Lines 41 to 43 in 08b1544

	bool skipCost = options->get<bool>("skip-cost");
	auto encdec = models::createModelFromOptions(
	options, skipCost ? models::usage::raw : models::usage::translation);

createModelFromOptions:

marian-dev/src/models/model_factory.cpp

Lines 370 to 376 in 08b1544

	// add (log)softmax if requested
	if (use == usage::translation) {
	if(std::dynamic_pointer_cast<EncoderDecoder>(baseModel)) {
	if(options->get<bool>("output-sampling", false))
	return New<Stepwise>(std::dynamic_pointer_cast<EncoderDecoder>(baseModel), New<GumbelSoftmaxStep>());
	else
	return New<Stepwise>(std::dynamic_pointer_cast<EncoderDecoder>(baseModel), New<LogSoftmaxStep>());

StepWise:

marian-dev/src/models/costs.h

Lines 309 to 313 in 08b1544

	// class to wrap an IEncoderDecoder and a ILogProbStep that are executed in sequence,
	// wrapped again in the IEncoderDecoder interface
	// @TODO: seems we are conflating an interface defition with its implementation?
	// @TODO: needs a better name. Stepwise is an adjective. Classes are things=nouns. StepwiseWhat?
	class Stepwise : public IEncoderDecoder {

StepWise Relevant call site:

marian-dev/src/models/costs.h

Lines 360 to 369 in 08b1544

	virtual Ptr<DecoderState> step(Ptr<ExpressionGraph> graph,
	Ptr<DecoderState> state,
	const std::vector<IndexType>& hypIndices, // [beamIndex * activeBatchSize + batchIndex]
	const Words& words, // [beamIndex * activeBatchSize + batchIndex]
	const std::vector<IndexType>& batchIndices, // [batchIndex]
	int beamSize) override {
	auto nextState = encdec_->step(graph, state, hypIndices, words, batchIndices, beamSize);
	return cost_->apply(nextState);
	}

If I insert a bool skipCost defaulting to false as part of the arguments here and ignore the cost operation if skipCost=true and trigger the param via beamsearch (see below), there is a possibility?

Call-site:

marian-dev/src/translator/beam_search.cpp

Line 421 in 08b1544

states[i] = scorers_[i]->step(graph, states[i], hypIndices, prevWords, batchIndices, (int)maxBeamSize);

marian-dev/src/translator/scorers.h

Lines 129 to 138 in 08b1544

	virtual Ptr<ScorerState> step(Ptr<ExpressionGraph> graph,
	Ptr<ScorerState> state,
	const std::vector<IndexType>& hypIndices,
	const Words& words,
	const std::vector<IndexType>& batchIndices,
	int beamSize) override {
	graph->switchParams(getName());
	auto wrapperState = std::dynamic_pointer_cast<ScorerWrapperState>(state);
	auto newState = encdec_->step(graph, wrapperState->getState(), hypIndices, words, batchIndices, beamSize);
	return New<ScorerWrapperState>(newState);