Panics in return value encoding crash the VM

[ethanwu10]

If the implementation of rustler::Encoder::encode panics during encoding of a return value from a #[rustler::nif] method, the panic is not caught and brings down the whole VM.

In the implementation of the nif macro, return value conversion is handled outside of the catch_unwind block:

rustler/rustler_codegen/src/nif.rs

Lines 83 to 91 in 566cc7c

	std::panic::catch_unwind(move || {
	#decoded_terms
	#function
	Ok(#name(#argument_names))
	});
	rustler::codegen_runtime::handle_nif_result(
	result, env
	)

While this is necessary to convert a caught panic from within the NIF function itself (which is rustler code and thus can be assumed to not panic), this also puts encoding for normal return values with library-user-controlled Encoder implementations outside the catch_unwind.

As an example, I happened to run into this when using a malformed SerdeTerm as a return value; the serde code panicked and brought down the VM.

[filmor]

You are right, this is an oversight.