Note parsing!

[CuddleBear92]

Finish up the Note side of things and bring in Note Parsing so it can be more useful! <3

[ShadowJonathan]

@CuddleBear92 sorry, what do you mean with this exactly? Do you mean a nicer Note UI (or something), or do you mean to allow parsers to store metadata within notes?

[CuddleBear92]

Note Content Parser in the download system ofc!
Want to download notes and post info and many many other thinghs that isnt just a tag.
Lets not get into password for zips!

[bbappserver]

Basically the following workflow
downloader -> descriptions-> notes -> note parse -> tags.
But given hydrus's limited capacity for programatic note parsing I'm not sure how good it would be. I would prefer some kind of subprocess delegation fothat task.

[Zweibach]

@bbappserver Why are you involving tags in note parsing? And what "limited capacity for programatic note parsing"? Far as I've understood it the purpose of parsing notes is to get content that is not necessarily tags but might still be of interest such as uploader comment and similar information. Stuff that is usually available on the page in the same way tags are.

[DonaldTsang]

So notes will be metadata that cannot be put into tags?

[CuddleBear92]

@DonaldTsang Notes is already split apart from tags.
It needs more expansions on its feature set though. Have its own service, make the service mappings exportable and importable.
And ofc parsed from sites which this issue covers.
ATM you can have multiple notes and parsing could enable chunks of text into notes from any site or have special parsers for passwords for archives for example.

[DonaldTsang]

What would be the basic usage of notes compared to tags? Importance? Non-standard data?

[pozieuto]

The most common use case would be to preserve the descriptions that artists post with their works.

[Zweibach]

What would be the basic usage of notes compared to tags? Importance? Non-standard data?

Why are you asking questions answered in discussions you've participated in? Notes are for non-tag stuff people might want to save. Post descriptions, uploader comments, translation notes from boorus, zip passwords, ANYTHING!

[DonaldTsang]

The basic property of the notes, in this case, would be:

1. does not have critically identifying characteristics of the media it is attached to
2. have metadata or context properties that aid in searching in rare or personal cases

[Mengmoshu]

@DonaldTsang No.

The key property of the notes is that they are arbitrary, possibly bulk, information attached to a file in the database. Anything else you've said is you projecting or constraining for no useful reason. Attempting to define things further the way you seem to be doing completely misses the point that just like tags it's a flexible system capable of being adapted to many uses.

This issue is definitely about being able to create notes via parsers. It could also be considered a reminder that there might be further features in Hydev's todo list for notes.

[bbappserver]

@Zweibach

Why are you involving tags in note parsing?

There is descriptive information in post bodies that can supplement tag data, but is not in a machine readable. It is more prevalent on gallery services than boorus. Doing preliminary parsing of the structure of a page allows you to stuff a dictionary of data that hydrus either does not presently support, or which is not fully automatic and needs additional processing and/or user intervention to convert to useful metadata. At a later time you could read this back by extenral program and complete the processing.

Often times OG gallery services as opposed to boorus are limited in their capabilities to represent the following machine readable relationships.

• Creator vs poster (commissioned works)
• Franchise

Additionally hydrus cannot currently represent in a proper searchable way

• Posted timestamp
• Collection membership
• External links
• Supported platforms

And sometimes extra data can be derived or corrected from parsed data such as
-keywords in description
-correcting tags based on order (split because of space instead of underscore) (e.g. elf ears ->["elf","ears"] but "elf_ears"->"elf ears")