Need help with config file

[sighhhguy]

I've been trying to do as much of this on my own and have solved some problems on my own but I've hit a few road blocks. I'm gonna organize this post by each extractor I'm having trouble with. Also I'm using the .exe, idk if that gets in the way of anything but I couldn't get powershell to recognize the gallery-dl commands.

edit: I looked at the config file outline in the wiki (https://github.com/mikf/gallery-dl/wiki/Config-File-Outline) and realized the answer was in there the whole time. I had looked through there quite a bit but just missed that piece. In case anybody was having issues setting up an extractor to download posts you need to log in for, you just need to use the "auth_token" option for the "cookies" subcategory. You can find your auth_token in the cookies.txt file generated from one of the cookie.txt extensions found in the README.rst. I don't know if this is a problem anyone else has had or if I was just being a ding dong but I thought I should post my solution.

this is how my extractor looks after I fixed it

    "twitter":
    {
        "username": "xxxxxx",
        "password": "xxxxxx",
        "cards": false,
		"cookies": {
			"auth_token":"xxxxxx"
		},
        "conversations": false,
		"include": "tweets, media",
		"locked": "abort",
        "pinned": false,
		"postprocessors": ["content"],
        "quoted": false,
		"ratelimit": "wait",
        "replies": "self",
        "retweets": false,
		"size": "orig",
        "strategy": null,
        "text-tweets": true,
		"transform": true,
        "twitpic": false,
        "unique": true,
        "videos": true
    },

tumblr

I had found some issues and discussions that had most of what I needed for this extractor but the format of the txt files look like:

content:
None
None
None
whatever
['whatever']

URL: https://tumblr.com/whatever/0000000

It looks like the only content being downloaded is the summary. That may be the cause of the other problem being that only the question and tags from answer posts are being saved.

this is how my extractor looks like now

    "tumblr":
    {
        "avatar": false,
        "external": false,
        "inline": true,
        "posts": "all",
        "offset": 0,
        "original": true,
        "reblogs": "same-blog",
		"api-key": "xxxxx",
		"api-secret": "xxxxx",
		            "metadata": true,
        "ratelimit": "wait",
        "postprocessors":[{
            "directory": "posts",
            "name": "metadata",
            "filename": "{blog_name}-{date:%Y%m%d}-{id}_{num:>02}.{extension}.txt",
            "event": "file",
            "mode": "custom",
            "format": "content: \n{text} \n{full_ask} \n{caption} \n{summary} \n{tags} \n \nURL: https://tumblr.com/{blog_name}/{id}"
        }],
        "filename": "{blog_name}-{date:%Y%m%d}-{id}_{num:>02}.{extension}",
        "directory": { 
            "": ["{category}","{blog_name}"]
        }
    },

I don't have much prior experience with other coding type stuff, but I have learned a lot and I am getting more proficient. It been pretty fun trying to do as much as I can by myself but I would appreciate a little guidance. I've tried to be as detailed and concise as possible with my questions so I hope I've made it easier to answer. I don't expect answers to all of this but I'd appreciate any response. Thanks for giving this post a look!

[13luwish]

Thanks for posting about the cookies and auth_token part, I finally managed to make my customization work

Edit: working well, issues ironed out in my file

[sighhhguy]

I don't know anything about setting up directories yet, I would recommend opening your own issue or discussion

[13luwish]

Forgot to add an update (did so now in the edit), but managed to do it yesterday while tinkering around

[mikf]

twitter

Most likely broken due to twitter.com -> x.com changes: #5597

It looks like the only content being downloaded is the summary.

The field names containing text content are different for each post type.
For example, the text content of a text post is inside body.

[sighhhguy]

Sorry, missed this for a few days. I think that will work but the formatting conflicts with !H:?//.

Heres the error:

[error] An unexpected error occurred: ValueError - Invalid format specifier '|body!H:?//' for object of type 'str'.

This is the updated format for the post processor

        "postprocessors":[{
            "directory": "posts",
            "name": "metadata",
            "filename": "{blog_name}-{date:%Y%m%d}-{id}_{num:>02}.{extension}.txt",
            "event": "file",
            "mode": "custom",
            "format": "content: \nBlog:\n{blog_name} \n{blog_title} \n{blog['url']} \nQuestion:\n{question} \nText:\n{answer!H:?//|body!H:?//} \nTags:\n{tags} \nURL:\nhttps://tumblr.com/{blog_name}/{id}"
        }],
        "filename": "{blog_name}-{date:%Y%m%d}-{id}_{num:>02}.{extension}",
        "directory": { 
            "": ["{category}","{blog_name}"]
        }
    },

also this processor hasn't been downloading regular text posts but is downloading the text for other posts so idk whats wrong there. Thanks again!

[mikf]

{answer!H:?//|body!H:?//}

{answer|body!H:?//}

You might want to include a few other possible field names:

{answer|body|description|excerpt|summary!H:?//}

[sighhhguy]

Dropped that in and did a little more tweaking, its working great now. I'd also like some help with my twitter extractor if that's alright. I'm trying to collect the text content from tweets but nothing I've done has worked, and I'd like to filter media from quote tweets into a subfolder.

Here's the extractor

    "twitter":
    {
        "username": "xxxxx",
        "password": "xxxxx",
        "cards": "ytdl",
		"cookies": {
			"auth_token":"xxxxxx"
			},
        "filename": "{author[name]!t:R.//}-{date:%Y-%m-%d}-{tweet_id}-{count:?.//>04}.{extension}",
        "conversations": false,
		"include": "tweets, media",
		"locked": "abort",
        "pinned": false,
		"postprocessor":
			{
			"name": "metadata",
			"directory": "tweets",
			"#": "write data for every post instead of each individual file",
			"event": "post",
			"filename": "{author[name]!t:R.//}-{date:%Y-%m-%d}-{tweet_id}-{count:?.//>04}.txt",
			"mode" : "custom",
			"format": "content: \nUser: {author['name']} \n{author['nick']} \n{date} \nText: {content} \nURL: {https://x.com/{author['name']}/status/{tweet_id} \n{author['url']}"
			},
        "quoted": true,
		"ratelimit": "wait",
        "replies": "self",
        "retweets": false,
		"size": "orig",
        "strategy": null,
        "text-tweets": true,
		"transform": true,
        "twitpic": false,
        "unique": true,
        "videos": true
    },

[mikf]

The name is postprocessors with an s at the end, and its value must be a list of objects:

"postprocessors": [ {…}, {…}, ... ]

    "postprocessors": [
        {
        "name": "metadata",
        "directory": "tweets",
        "#": "write data for every post instead of each individual file",
        "event": "post",
        "filename": "{author[name]!t:R.//}-{date:%Y-%m-%d}-{tweet_id}-{count:?.//>04}.txt",
        "mode" : "custom",
        "format": "content: \nUser: {author['name']} \n{author['nick']} \n{date} \nText: {content} \nURL: https://x.com/{author['name']}/status/{tweet_id} \n{author['url']}"
        }
    ],

---

Setting a static auth_token cookie value makes it so your username & password get completely ignored. You should remove one or the other.

---

"filename": "{author[name]!t:R.//}-{date:%Y-%m-%d}-{tweet_id}-{count:?.//>04}.{extension}",

This'll give the same filename to each file in a Tweet with more than one file.
Change {count} to {num} to avoid this, for example.

[sighhhguy]

This works great, thanks for your help! And thanks for making gallery-dl!