Remove some as casts

[jonasbb]

This removes some potential arithmetic issues

See #771 for reference

[codecov]

Codecov Report

Attention: Patch coverage is 58.62069% with 12 lines in your changes missing coverage. Please review.

Project coverage is 66.40%. Comparing base (abb13ca) to head (c910fa1).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
serde_with/src/utils/duration.rs	57.14%	6 Missing ⚠️
serde_with/src/chrono_0_4.rs	50.00%	4 Missing ⚠️
serde_with/src/de/impls.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #809      +/-   ##
==========================================
- Coverage   66.55%   66.40%   -0.16%     
==========================================
  Files          40       40              
  Lines        2613     2625      +12     
==========================================
+ Hits         1739     1743       +4     
- Misses        874      882       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.