Towards functioning expl3 #1188

dginev · 2019-08-13T19:31:53Z

Combines #1182 and Bruce's expl3 branch with additional fixes from my debugging.

…s in expansion

define CharDef->equals; fix \unless support pdf version & \pdfstrcmp respect \endlinechar in Mouth more careful token comparisions for space, \relax, etc especially in <one optional space> after reading numbers \edef and friends read the body *while* expanding LaTeX \@onefilewithoptions This allows texlive 2016's l3kernel to be read; (there are still issues with 2019). Test cases still need to be added.

…add minor revision for pdf proc

dginev · 2019-08-13T19:34:50Z

In need of further work:

XUntil parameter needs to be more careful not to expand the final match marker
Alignments need to be adapted to the more strict readXToken and the third parameter should be removed
\lipsum should work fluently, and a test case added, ideally with basic xparse support

brucemiller · 2019-08-15T15:25:16Z

lib/LaTeXML/Core/Definition/Register.pm

+    && Equals($$self{getter}, $$other{getter})
+##    && Equals($$self{beforeDigest}, $$other{beforeDigest})
+##    && Equals($$self{afterDigest},  $$other{afterDigest})
+    ; }
 #===============================================================================
 1;



Can we checkout this back to master (pre my pointless commit); it's a no-op.

Confused by the phrasing "Check it out back to master". Do you mean remove the code block entirely, or do you mean "it's in master, let's rebase"?

I must meant like git checkout ...\Register: ie, no change to the file at all, since the only changes have no effect.

Alright, I've removed the changes. Sadly "reverting" to the previous Register isn't simple this many commits later, I just papered over with a new commit.

brucemiller · 2019-08-15T15:27:27Z

lib/LaTeXML/Core/Gullet.pm

 sub neutralizeTokens {
  my ($self, @tokens) = @_;
  my @result = ();
  foreach my $token (@tokens) {
    if ($$token[1] == CC_PARAM) {    # Inline ->getCatcode!
      push(@result, $token); }
-    elsif (defined(my $defn = LaTeXML::Core::State::lookupDefinition($STATE, $token))) {
+    elsif (!defined(my $meaning = LaTeXML::Core::State::lookupMeaning($STATE, $token)) ||


Isn't both meaning & defintion redundant?

What I intended to add here, was also neutralizing undefined command sequences, which is what a negative lookupMeaning achieves.

If your point is that every time lookupDefinition is true, then lookupMeaning is also true, that may be.

The question is, are there cases where lookupMeaning is true, but lookupDefinition is false, where we do not want to add a \noexpand. An example of that is e.g. any T_LETTER token, and a variety of others. I think adding a \noexpand before those is simply a no-op, so it may be a "distinction without a difference"... Unsure what is cleanest.

If we could get away with adding a \noexpand unconditionally, before every element of the input @tokens, that would certainly feel most elegant to me.

Doh; missed the "!".

brucemiller · 2019-08-16T17:15:12Z

Probably all looks good, other than the debugging turds that I left in --- is that worth removing?
Otherwise, I assume this should be squashed, since there're so many fiddly commits, right?

dginev · 2019-08-16T17:16:05Z

We can squash, certainly. Would you like to leave comments on the bits that need removing?

brucemiller · 2019-08-16T17:17:34Z

Incidentally, with regard to XUntil, I may have finally come around to your POV: that it is legitimate to leave it as is, and require that if you use it, you should probably \let the delimiting token to \relax or something. As it stands, you'll get an error message (right?) which should be relatively decipherable.

dginev · 2019-08-16T17:25:08Z

As to XUntil: yes to your question, the last token will shoot an error if it's expanded without being defined. And if you let it to relax, or some such, XUntil will succeed properly.

brucemiller · 2019-08-16T17:26:37Z

Mostly I was thinking of removing the commented out debugging code (eg. prints or otherwise dead code)

dginev · 2019-08-16T17:41:03Z

There is still the issue with readXToken having 3 arguments in t his PR, which as you mentioned is something to eliminate. So maybe we should make sure alignments work with the new expansion errors and transfer them to the new behaviour before merging?

dginev · 2019-08-16T17:42:04Z

I've removed the debug comments, thanks for mentioning.

brucemiller · 2019-08-16T18:09:36Z

Oh, you mean backing off the extra args to readXToken in \@@open@inner@column? Yeah, if it works without, then we can go ahead & remove that mod to readXToken. Better earlier than later :>

dginev · 2019-08-16T18:14:14Z

Indeed - I am trying that change now, but I have a subtle failure in the siunitx tests, need a few more mins...

brucemiller · 2019-08-16T18:16:57Z

take your time; I gotta run errands :>

…elaxed when undefined

dginev · 2019-08-16T19:11:14Z

Alright, feeling very good about the refactor of readXToken now! The method signature remains the same as in master, and the only remaining change is the raised error when an undefined token is expanded.

No issues with alignment are visible from the tests - the once encountered with siunits in tables were related to DeclaredUnits not being defined -- let-ing them to relax when needed allowed for smooth passing of the tests. I experimented a little with literal strings and discovered siunitx isn't actually loading some of its dependencies (amsmath and array) so I threw those in for good measure.

All good on my end!

dginev · 2019-08-16T19:16:10Z

lib/LaTeXML/Package.pm

@@ -430,9 +430,14 @@ my @rmletters = ('i', 'v', 'x', 'l', 'c', 'd', 'm');    # [CONSTANT]

 sub roman_aux {


Side comment, the roman functions really feel like Util-level pieces of code, rather than Package.pm bits.

dginev · 2019-08-16T19:19:33Z

lib/LaTeXML/Package/TeX.pool.ltxml

@@ -148,7 +148,7 @@ DefParameterType('GeneralText', sub {
    my ($gullet) = @_;
    my $open = $gullet->readXToken;
    if ($open->equals(T_BEGIN)) {
-      return $gullet->readBalanced; }
+      return scalar($gullet->readBalanced); }


I am a bit stumped at this line. Returning the number of tokens instead of the tokens themselves? From within a parameter type ?!?!? Is this some sort of intermediate code that slipped by or am I missing something extremely obvious?

Good news is that all tests pass with the scalar removed. I'm tempted to just push in the change since it looks so horrendous.

dginev · 2019-08-16T19:22:42Z

lib/LaTeXML/Package/TeX.pool.ltxml

+    my $token  = $gullet->readXToken(0);
+    my @tokens = ();
+    if ($token->getCatcode == CC_BEGIN) {
+      return scalar($gullet->readBalanced(1)); }


same scalar here...

dginev · 2019-08-16T19:25:59Z

lib/LaTeXML/Core/Gullet.pm

@@ -473,7 +482,7 @@ sub readArg {
  if (!defined $token) {
    return; }
  elsif ($$token[1] == CC_BEGIN) {    # Inline ->getCatcode!
-    return $self->readBalanced; }
+    return scalar($self->readBalanced); }


And another scalar. Maybe I should wait until I hear a reply, but I just have the strong urge of removing all of these guys.

dginev · 2019-08-16T19:33:22Z

Please ignore all scalar comments. Perl is way too overloaded for my normal mortal mind. After a test finally failed when I removed all scalar() calls, I went and re-read wantarray, and it hit me that you've used scalar to force the evaluation of a function to return in scalar context. I had - way too simply - memorized that if you are returning an array and invoke scalar() you get back the number of elements, so I was expecting the readBalanced calls to return the scalar 2.

But of course when you apply the scalar on the function call that ends with wantarray it would not return the array output and count it, but would return the scalar output... Sigh. It's a headscratcher when you see it unprepared. But all good, I restored the PR to the state before I sunk into the rabbit hole.

brucemiller · 2019-08-16T21:49:49Z

TeeHee! I knew it'd hit you shortly :>

dginev and others added 16 commits August 9, 2019 10:56

report error for undefined token expansion; neutralize undefined tokens

0ccd538

missing macro stubs

7b1af78

ensure all binding command sequences have reasonable/expected meaning…

e3ab1ec

…s in expansion

also enforce defined second token for \expandafter use

9239b56

attempt to streamline TeX-compliant expansions, back off from Alignment

0cab65f

progress with bookkeeping @currnamestack, more expl3 macros

2345f77

guard roman numerals from negative ints; cover CharDefs in \meaning; …

b258186

…add minor revision for pdf proc

test: tricky number+tilde interaction in expl3

bda3c35

Gullet::readOptionalSigns should accept aliases of space

7877967

pdftexrevision is an expandable macro

85f9748

cleaner messages for \expandafter and \message

9de0ae4

strange oversight, \ExplSyntaxOn needs to be in test case

826a17b

stub aux method which isnt presently needed

71f2c57

use Object::Equals for space comparison in Number reads

4f636e9

finer touches

caf85bb

dginev mentioned this pull request Aug 13, 2019

[WIP] TeX-faitfhul errors on undefined command sequence expansion #1182

Closed

brucemiller reviewed Aug 15, 2019

View reviewed changes

undo Register experimental changes

ac8dbfe

remove debug comments

12e15a5

fully enforce new readXToken; ensure siunitx declared units are let r…

6585726

…elaxed when undefined

Equals instead of explicit meaning check

9df53b6

dginev commented Aug 16, 2019

View reviewed changes

dginev force-pushed the further-expl3 branch from d5bd1b9 to 0777c96 Compare August 16, 2019 19:24

dginev commented Aug 16, 2019

View reviewed changes

dginev force-pushed the further-expl3 branch from 0777c96 to 9df53b6 Compare August 16, 2019 19:30

brucemiller merged commit f3a72b6 into brucemiller:master Aug 17, 2019

dginev mentioned this pull request Aug 17, 2019

Discrepancy in expansion of undefined command sequences #1180

Closed

dginev deleted the further-expl3 branch August 17, 2019 13:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Towards functioning expl3 #1188

Towards functioning expl3 #1188

dginev commented Aug 13, 2019

dginev commented Aug 13, 2019 •

edited

Loading

brucemiller Aug 15, 2019

dginev Aug 15, 2019

brucemiller Aug 15, 2019

dginev Aug 15, 2019

brucemiller Aug 15, 2019

dginev Aug 15, 2019

dginev Aug 15, 2019

brucemiller Aug 15, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019 •

edited

Loading

dginev Aug 16, 2019

dginev Aug 16, 2019

dginev Aug 16, 2019

dginev Aug 16, 2019

dginev Aug 16, 2019

dginev Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

		@@ -430,9 +430,14 @@ my @rmletters = ('i', 'v', 'x', 'l', 'c', 'd', 'm'); # [CONSTANT]

		sub roman_aux {

Towards functioning expl3 #1188

Towards functioning expl3 #1188

Conversation

dginev commented Aug 13, 2019

dginev commented Aug 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dginev commented Aug 16, 2019

brucemiller commented Aug 16, 2019

dginev commented Aug 13, 2019 •

edited

Loading

dginev commented Aug 16, 2019 •

edited

Loading