Re: Command substitution in here-documents

[Herbert Xu <herbert@xxxxxxxxxxxxxxxxxxx>]

On Fri, Aug 17, 2018 at 09:08:35PM +0800, Herbert Xu wrote:
> Ron Yorston <rmy@xxxxxxxxxxxx> wrote:
> > I've been continuing to worry about this.
> > 
> > In the old code (prior to commit 7c245aa) the TNL token that was
> > detected in list() had actually been read.  The current code uses
> > peektoken().
> > 
> > When parseheredoc() is called in interactive mode the last call to
> > readtoken() is in the code to set the prompt.  Even though the text
> > of that token is long gone it's used anyway.  Exactly what then happens
> > on a given platform depends on how memory allocation is handled.
> > 
> > I can make the error go away by explicitly reading the TNL token when
> > peektoken() detects a newline and we're about to call parseheredoc().
> > 
> > Of course, I may be wrong.
> 
> The real problem is the fact that we call expandstr which reenters
> the parser and the parser is not reentrant.
> 
> We need to save the parser state and restore it around expandstr.
> I'm working on a fix.

This patch should fix the problem:

---8<---
parser: Do not push token back before parseheredoc

When we read the first token in list() we use peektoken instead
of readtoken as the following code needs to use the same token
again.  However, this is wrong when we're in a here-document as
it will clobber the saved token without resetting the tokpushback
flag.

This patch fixes it by doing the tokpushback after parseheredoc
and setting lasttoken again if parseheredoc was called.

Reported-by: Ron Yorston <rmy@xxxxxxxxxxxx>
Fixes: 7c245aa8ed33 ("[PARSER] Simplify EOF/newline handling in...")
Fixes: ee5cbe9fd6bc ("[SHELL] Optimize dash -c "command" to avoid a fork")
Signed-off-by: Herbert Xu <herbert@xxxxxxxxxxxxxxxxxxx>

diff --git a/src/parser.c b/src/parser.c
index c4e6378..1f9e8ec 100644
--- a/src/parser.c
+++ b/src/parser.c
@@ -166,7 +166,7 @@ list(int nlflag)
 
 	n1 = NULL;
 	for (;;) {
-		switch (peektoken()) {
+		switch (readtoken()) {
 		case TNL:
 			if (!(nlflag & 1))
 				break;
@@ -177,9 +177,12 @@ list(int nlflag)
 			if (!n1 && (nlflag & 1))
 				n1 = NEOF;
 			parseheredoc();
+			tokpushback++;
+			lasttoken = TEOF;
 			return n1;
 		}
 
+		tokpushback++;
 		checkkwd = CHKNL | CHKKWD | CHKALIAS;
 		if (nlflag == 2 && tokendlist[peektoken()])
 			return n1;
-- 
Email: Herbert Xu <herbert@xxxxxxxxxxxxxxxxxxx>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt