[Q] Configuring the metadatas

[Galewin]

I'm not used that much to fidget with config files and command lines softwares so sorry for the question, I tried my best to find the answer myself but I'm unable to for these questions.

I mainly use the program to download deviantart galleries. I managed to use one fo the template on the github to make some changes like the save folder or the filename, but I can't figure out how to change the parameters of the metadata (template I used as a base -> https://github.com/mikf/gallery-dl/blob/master/docs/gallery-dl.conf)

What I want to do is :

1. Basicly downloading metadata automaticly, without specifiying "--write-metadata" to make it more simple, I want them pretty much every single time
2. Putting them inside a subfolder (pictures in "deviantart/[artist_name]/pictures.format" and metadata in "deviantart/[artist_name]/Metadata/pictures.json")

I have no idea of what are the lines of codes I need, and more importantly, where to write them in the template I used.

Thank you in advance for your help !

[mikf]

You need to set up a post processor inside the deviantart block of your config file.

To replicate the behavior of --write-metadata, the following is enough:

    "postprocessors": [
        {"name": "metadata"}
    ]

To also set a separate output directory, use the directory option

    "postprocessors": [
        {
            "name": "metadata",
            "directory": "Metadata"
        }
    ]

All in all this should look something like

"deviantart":
{
    "-- other options --": "",
    "postprocessors": [
        {
            "name": "metadata",
            "directory": "Metadata"
        }
    ]
}

[Galewin]

Thank you very much for your answer, it's working exactly how I'd like it to do now !

I'm being curious to learn a bit more. Where I originally tried to add instructions was at the very top, where there is a "postprocessors": null, . Is it like a global block, where I can replicate the code you gave me in example to apply parameters to every website ?

edit: sorry, it seems I have another problem.

I set "extra": true, since the artist I download have alternates in the description. It seems it manages to generate the metadata, but not download the picture itself. I have like 500 metadatas for only 200 actual pictures that correspond to the gallery of the artist.
Maybe it's what is causing the issue, I also added "filename": "{filename}.{extension}", since I only wanted the name of the file and get rid of the ID before to get like "picturename_by_artist.format" and not "91537287_picturename_by_artist.format", but I don't know since the alts have different names anyway, It shouldn't conflict.

I also zipped it in case it's a known issue

edit2: okay, I think it's that I'm an idiot and links from sta.sh are considered scraps, so I used the "include" and it seems to work.

Also one last question. I also included journals to be downloaded, but right now, pictures downloaded from the journal end up in the artist folder, can I change that for journals picture only to be sent into the subfolder journal with the htm files ?

[hellupline]

I was wondering something similar:

save files in "gallery-dl/media/deviantart/artist_name/filename.ext"
and save metadata in "gallery-dl/metadata/deviantart/artist_name/filename.ext.json"

my current set up is "gallery-dl/deviantart/artist_name/filename.ext{,.json}"

( it would help a lot my indexing script )

would that be possible ?

[Hrxn]

By using the exec postprocessor, I think, and with setting the event to "after" like this, maybe?

Use it by setting the postprocessors value to an array containing the desired postprocessor-objects, like this:

[
    {
        "name": "ugoira",
        "ffmpeg-twopass": true,
        "ffmpeg-args": ["-c:v", "libvpx", "-crf", "4", "-b:v", "5000k", "-an"]
    },
    {
        "name": "zip" ,
        "compression": "store"
    },
    {
        "name": "exec",
        "command": ["/home/foobar/script", "{category}", "{image_id}"]
    }
]

(This is just an example, you only need the third object here - the one containing "name": "exec"

You just need to set "postprocessors" at the right level in the extractor.category.subcategory hierarchy.
For example, in "deviantart.journal", i.e. like this:

{
    "extractor":
    {
   
     .....

        "deviantart": {
            "journal": {
                "postprocessors": [
                    {
                        "name"   : "exec",
                        "async"  : false,
                        "command": ["mv", "{_file}", "gallery-dl/media/deviantart/artist_name/{_filename}"],
                        "event"  : "after"
                     }
                 ]
            }
       }
}

[mikf]

I'm being curious to learn a bit more. Where I originally tried to add instructions was at the very top, where there is a "postprocessors": null, . Is it like a global block, where I can replicate the code you gave me in example to apply parameters to every website ?

Yes, exactly.

I set "extra": true, since the artist I download have alternates in the description. It seems it manages to generate the metadata, but not download the picture itself. I have like 500 metadatas for only 200 actual pictures that correspond to the gallery of the artist.

That shouldn't be possible, since it only writes a metadata file after having successfully downloaded the real file.

Maybe it's what is causing the issue, I also added "filename": "{filename}.{extension}", since I only wanted the name of the file and get rid of the ID before to get like "picturename_by_artist.format" and not "91537287_picturename_by_artist.format", but I don't know since the alts have different names anyway, It shouldn't conflict.

That shouldn't be causing this issue. The downloads for files with duplicate filenames get skipped and no post processors are run.

Also one last question. I also included journals to be downloaded, but right now, pictures downloaded from the journal end up in the artist folder, can I change that for journals picture only to be sent into the subfolder journal with the htm files ?

The category-transfer option might be useful for that

    "deviantart": {
        "journal": {
            "category-transfer": true,
            "extra": true
        }
    }

This should cause any stash files coming from journals to use the same settings, including the target directory, as journals themselves.

@hellupline the directory option for metadata files can, at the moment, only be a static value. This can maybe be done with what Hrxn posted, but that is rather complicated as you can see.

[Galewin]

Thank you for your answers, I really appreciate it ! And for that :

That shouldn't be possible, since it only writes a metadata file after having successfully downloaded the real file.

I think I might have found something. It seems the problem was with the --zip command. I tried without it, and I got every file correctly. I could see the files I missed in the cmd command line as if it was downloading them when I used the --zip command, but it looked like it didn't download in the zip archive.

Turns out it did as you said and it actually downloaded. I could see the artist archive size increasing with the stash files that "wouldn't download" in real time, so it was added to the archive. I got an uncorrputed archive I could open and extract, but I got an error from 7-zip warning me there were still "useful datas after the end of datas" or something like that.

7-zip shows 202 files, exactly the number of pictures in the gallery, while the default windows explorer shows 304, wich is a bit better but not the full 500 metadata files and actual pictures I get when I don't use the --zip command.

Sorry if it is a bit unclear or I forgot some infos, I'm not used to send rapports like that, let me know if you still want to know something else !