GTK3 - WLX - gtk_socket - Web

[advice2020]

DEPENDENCIES USED
build-essential libgtk3.0-dev libmagic-dev
The webkit2gtk "Addition" mentions
libwebkit2gtk-4.0-dev
My repo had slightly newer version (hope this is okay)
libwebkit2gtk-4.1-dev
I did not install any of the other Additions dependencies

---

MHTML QUESTIONS / POSSIBLE ISSUES
I am assuming that the libwebkit2gtk-4.1-dev dependency is the actual library used for the opening of mhtml files.

In the GTK2 version, I remember having to add mhtml to the settings.ini file which I did again.
(I left the plugin registered for/detect string field blank as it is by default)

Not sure if some of this issue is plugin or DC itself, because I reported a similar error to DC about plugins QPP retaining/locking focus, but I experience some slight differences with this plugin compared to when this occurs in a QT version of DC.

When I tried to use QPP to view mhtml file
-The QPP pane first showed a gear/almost chrome style icon in the center of the pane
-Then it went blank black panel
-Then loaded up mhtml file.
-After it loaded though, you can scroll up and down in the QPP pane, but DC itself gets frozen, cannot click on anything.
-When this DC issue occurs in QT versions, I can minimize the DC window and then open it again and DC works.
However with DC GTK3 + using this plugin, I can minimize the DC window but then the window does not open, and I have to terminate to close DC completely.

So not sure what of this is DC and what if anything of this is due to this plugin?

---

If you have any ideas, please let me know if I can test anything

Thank You

[j2969719]

besides the python script for gtk_socket, theres already wlxwebkit_gtk3 plugin for gtk3. gtk_socket itself is slow compared to native plugins and served more as a workaround for inserting modern stuff somehow into the ancient gtk2 (theres currently a test plugin, gtk_socket_tst, in the repository as an attempt to make it a bit faster, but its future is uncertain)

[j2969719]

i've also recently noticed a issue with capturing focus after clicking in ctrl+q with qt and gtk3, but not with gtk2. no idea whats going on or what might be causing it. dont see hang with dc gtk3 described above. in my case, right-clicking in the file panel or opening the properties window works as a workaround. wlxwebkit_gtk3 plugin displays pdf and mht files (at least for me), but you need to add extensions to detectstring in the dc options window

[Skif-off]

wlxwebkit_gtk3 plugin displays pdf and mht files (at least for me), but you need to add extensions to detectstring in the dc options window

I have a problem, I see empty white window only (GTK2, wlxwebkit & gtk_socket + webkit2gtk.py).
Any ideas?

[Skif-off]

@j2969719,
it works. Not without problems (for example, 4 HTML files (*.html) were extracted from one MHT file, but in reality, by content: 1 HTML file, 1 CSS file and 2 JS files; there were also two files that were extracted normally, but the HTML files were not displayed), but it works.

I'm a little confused by this line

html_content = re.sub(r'charset=["\']?[\w-]+["\']?', 'charset="utf-8"', html_content, flags=re.I)

charset is not a separate attribute, but the part of the content attribute: content="text/html; charset=utf-8". But I'm no sure.

I remembered about python-unmht, maybe it will be useful to you?

[j2969719]

looks pretty similar, it just doesnt try to do shenanigans with encoding sub and replacing links with unpacked files and use contentid. ["\']? buuuuuut yep it always replaces with double quotes, should have used capture groups. can you upload a few problematic files to poke around in pm if thers nothing private in them?

[j2969719]

.. also itseems like webkitgtk2 displays some of my extracted html files better than gtk3 one (though maybe thats just a difference due to js being disabled with my webkitgtk2 dunno) which makes this crap kinda pointless

[Skif-off]

@j2969719

About

for example, 4 HTML files (*.html) were extracted from one MHT file, but in reality, by content: 1 HTML file, 1 CSS file and 2 JS files; there were also two files that were extracted normally, but the HTML files were not displayed

Files themselves are problematic:

Content-Disposition: inline; filename=jscript.htm
Content-Type: text/html; charset=windows-1251; name=jscript.htm
Content-Location: http://example.com/includes/jscript.js
...
Content-Disposition: inline; filename=styles.htm
Content-Type: text/html; charset=windows-1251; name=styles.htm
Content-Location: http://example.com/themes//styles.css

It seems to me that these files were saved incorrectly.

There is also a problem with encoding: the file is encoded in cp1251, but the script uses UTF-8 fo reading. Rare files, I found two.
I don't know if it makes sense to add an encoding detector, because it seems to me that these files were saved incorrectly.

can you upload a few problematic files

If you meant charset, then just open any HTML file and replace

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

with

<meta http-equiv="Content-Type" content="text/html; charset="utf-8">

[j2969719]

it was about mht with wrong file exts; thought there was something wrong with mimetypes.guess_extension and reinvented the wheel already
about encoding might have misunderstood something as usual, thought that email thing itself handles the reading and text is already utf in memory, so script writes text back as utf and changes the tags with that regex. but the people from your link dont do such nonsense (and part.get_filename() too)..
in theory, even previous script, with an existing Content-Location, should use its basename for the filename, unless that part.get_filename() thing actually returned something lol. did you see unicorn with ./test ld /path/to/that.mht in the term?

#!/usr/bin/env python3

import os
import re
import sys
import email
import mimetypes
from email import policy

script_name = os.path.basename(__file__)
extract_filelist = None
out_dir = './'
is_dont_write='l' in sys.argv[1]
is_debug='d' in sys.argv[1]
args = len(sys.argv)
if args < 3:
	print(f"usage: {script_name} <command> <mht> <files...> <-Odir> <@filelist>\n<commands>\n  l[d]\tlist archive contents[debug]\n  x\textract files", file=sys.stderr, flush=True)
	sys.exit(1)
elif args > 3:
	for i in range(3, args):
		if sys.argv[i].startswith("@"):
			try:
				with open(sys.argv[i][1:], 'r', encoding='utf-8') as f:
					extract_filelist = {line.strip() for line in f}
			except Exception as e:
				print(f"{script_name}: error {sys.argv[2]}: {e}", file=sys.stderr, flush=True)
		elif sys.argv[i].startswith("-O"):
			out_dir = sys.argv[i][2:]
			os.makedirs(out_dir, exist_ok=True)
		else:
			if not extract_filelist:
				extract_filelist = set()
			extract_filelist.add(sys.argv[i])

def extract_mht(path, tmpdir):
	ext_map = {
		'text/html':              '.html',
		'text/css':                '.css',
		'text/javascript':          '.js',
		'application/x-javascript': '.js',
		'application/javascript':   '.js',
		'application/octet-stream':'.bin',
		'image/x-png':             '.png',
		'image/pjpeg':             '.jpg',
		'image/jpeg':              '.jpg',
		'image/gif':               '.gif',
		'image/bmp':               '.bmp',
		'image/png':               '.png',
		'audio/midi':              '.mid',
		'audio/x-midi':            '.mid',
		'audio/mpeg':              '.mp3',
		'audio/wav':               '.wav',
		'audio/x-wav':             '.wav',
		'video/mp4':               '.mp4',
		'video/mpeg':             '.mpeg',
		'application/x-ms-wmv':    '.wmv',
		'message/rfc822':          '.eml',
	}
	try:
		with open(path, 'rb') as f:
			msg = email.message_from_binary_file(f, policy=policy.default)
		first_path = None
		localhost = {}
		org_name = os.path.splitext(os.path.basename(path))[0]
		for i, part in enumerate(msg.walk()):
			if part.is_multipart():
				continue
			content_type = part.get_content_type()
			filename = part.get_filename()
			if is_debug and filename:
				print(f"UNICORN: get_filename {filename}", file=sys.stderr)
			if not filename:
				location = part.get('Content-Location')
				if content_type in ext_map:
					ext = ext_map[content_type]
				else:
					ext = mimetypes.guess_extension(content_type) or '.???'
				if is_debug:
					print(f"location: {location}\ncontent_type\t{content_type}\text\t{ext}\tContent-ID\t{part.get("Content-ID")}", file=sys.stderr)
				if not location or location.endswith('/'):
					if ext == '.html' and first_path is None:
						filename = f"{org_name}{ext}"
					else:
						filename = f"{org_name}_part_{i}{ext}"
				else:
					filename = os.path.basename(location.split('?')[0])
					if content_type != 'application/octet-stream' and os.path.splitext(filename)[1].lower() != ext:
						filename = f"{filename}{ext}"
					if "://" in location:
						localhost[location] = filename
			filepath = os.path.join(tmpdir, filename)
			filedata = part.get_content()
			is_text = isinstance(filedata, str)
			size = len(filedata.encode('utf-8')) if is_text else len(filedata)
			print(f"{size}\t{filename}")
			if is_dont_write:
				continue
			elif extract_filelist and not (filename in extract_filelist):
				continue
			mode = 'w' if is_text else 'wb'
			encoding = 'utf-8' if is_text else None
			try:
				with open(filepath, mode, encoding=encoding) as out:
					out.write(filedata)
			except Exception as e:
				print(f"{script_name}: write error {filepath}: {e}", file=sys.stderr, flush=True)
			if content_type == 'text/html' and first_path is None:
				first_path = filepath

		#if first_path and os.path.exists(first_path):
		if not is_dont_write and first_path and os.path.exists(first_path):
			with open(first_path, 'r', encoding='utf-8', errors='ignore') as f:
				html_content = f.read()
			sorted_urls = sorted(localhost.items(), key=lambda x: len(x[0]), reverse=True)
			for url, filename in sorted_urls:
				html_content = html_content.replace(url, filename)
			html_content = re.sub(r'(charset=["\']?)([\w-]+)(["\']?)', r'\1utf-8\3', html_content, flags=re.I)
			html_content = re.sub(r'(encoding=["\']?)([\w-]+)(["\']?)', r'\1utf-8\3', html_content, flags=re.I)
			try:
				with open(first_path, 'w', encoding='utf-8') as out:
					out.write(html_content)
				#return GLib.filename_to_uri(first_path)
			except Exception as e:
				print(f"{script_name}: write error {first_path}: {e}", file=sys.stderr, flush=True)
	except Exception as e:
		print(f"{script_name}: extract_mht {path}: {e}", file=sys.stderr, flush=True)
	return None

extract_mht(sys.argv[2], out_dir)

[advice2020]

Thank you for your responses.
gtk_socket
As mentioned in my other post, I will avoid using this plugin.

wlxwebkit_gtk3
I had just read the the readme.md for this plugin and it only says it supports "HTML/XHTML files", I was not aware this could also be used for MHTML and also PDF, thank you.

I was able to compile this and test this plugin out.
I noticed some things so I will be making a separate post about this plugin if anyone is interested.

Thanks again

[advice2020]

@Skif-off
Before reading your reply, I also was able to get wlxwebkit_gtk3 to work but experienced the same thing you did.
At first I did not realize that the pages were blank as you said, because I experienced a trickier result.
In QPP, if you view an mhtml page that does not work FIRST, then yes it shows up as a blank white page.
However if you happen to view an mhtml file that DOES work correctly BEFORE you end up on an mhtml file that does not work, the previous mhtml files page loaded will remain visible, giving you an inaccurate filename under cursor = QPP.

I tried your digging deeper approach but I did not seem to have the same consistency that you may have had?
Meaning I have pages work and not work with both
Content-Type: text/html
Content-Type: multipart/related
Unless I am not following the same approach you did?

If interested see my post about wlxwebkit_gtk3 not sure if it will help you in any way?
Also let me know if there is maybe anything I can test/verify for you if needed.
Thank You

Up to you if want to move your wlxwebkit_gtk3 results to that post. If you do then I will close this gtk_socket post.

---

Just saw new responses you guys made.
-So this wlxwebkit_gtk3 issue cannot be fixed?
-I have not tried hx_webkit_gtk3_crap :) plugin, I thought it was only GTK2? Are you saying that this one works better then wlxwebkit_gtk3 plugin?

Thank you for any information shared