Skip to content

Conversation

@savagedev
Copy link
Contributor

@savagedev savagedev commented Jul 2, 2022

This implements #55 to support local file providers with a JSON or CSV file format.

  • Larry now has a new contentfile provider which supports files in the CSV and JSON formats
  • This provider requires a command line arg --content-file which provides the local file and the file extension indicates the format of the file
  • The provider expects the Content struct fields on each record where only the Title field is required
  • The Provider uses the reservoir sampling method to efficiently (one pass) find a random record in the file
  • The provider also has an optional command line arg --skip-csv-header to skip past the header record of the CSV file
  • The CSV file can have an "unlimited" number of ExtraData fields by just including them at the end of each record (note: every CSV record needs the same number of fields.
  • The CSV file can have fields surrounded by double quotes or not
  • The readme has been updated to include the new features

Update: I updated this comment to reflect the refactoring changes I made

@savagedev
Copy link
Contributor Author

@ezeoleaf While working on the S3 file source I see where this code can be refactored quite a bit to eliminate redundant code. You can either close this PR and I'll create a new one or just wait and I'll update this one with my changes.

@savagedev
Copy link
Contributor Author

@ezeoleaf My refactoring is complete!

Copy link
Owner

@ezeoleaf ezeoleaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@ezeoleaf ezeoleaf merged commit 9e95c6d into ezeoleaf:main Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants