make(1): add tests for :!...! parse errors

(rillig)

make(1): add test for missing input validation in :[123] modifier

(rillig)

make(1): add Buf_AddInt to make the calling code simpler

(rillig)

make(1): add basic tests for the :S modifier

(rillig)

make(1): make code in the :S modifier simpler

This code path is covered by the "sequences of letters" in modmisc.mk.

(rillig)

make(1): fix bug in :S modifier from 2020-07-19

(rillig)

make(1): demonstrate bug in the :S modifier with the 1 modifier

The bug has been introduced in var.c r1.268 on 2020-07-19.

(rillig)

make(1): use consistent test names in modmisc.mk

(rillig)

make(1): add test for prefix mismatch in SysV substitution

(rillig)

make(1): add test for appending and exporting a variable

(rillig)

make(1): add tests for appending to a variable

(rillig)

make(1): add test for .MAKE.SAVE_DOLLARS

(rillig)

make(1): fix missing error message about missing @ delimiter

This had been broken since 2020-07-03, during a "refactoring".

(rillig)

make(1): add test about missing delimiters in :@ modifier

These error conditions have been broken since var.c 1.236 on 2020-07-03.

(rillig)

share/mk/bsd.README: fix typo

(rillig)

style: fix typo about enum constants being capitalized

(rillig)

make(1): remove references to the removed lst.lib

(rillig)

make(1): replace macros with functions

(rillig)

make(1): condense the list library into a single file

The list library is only used in make(1). Having it spread out over 28
files made it look more complex than it really is. In fact, it's just a
versatile generic data type like in hash.c.

Having all the list functions in a single file reduces the code size,
both by omitting the many RCS Ids and by inlining commonly used code.

(rillig)

make(1): add newly added unit tests

(rillig)

make(1): add tests for previously uncovered code

(rillig)

make(1): make it easy to run gcov on the code

There seems to be no gcov support in share/mk yes, but it's still good
to have, especially for a complex program like make(1).

(rillig)

make(1): indent Makefile directives like in pkgsrc

This makes it easier to see the corresponding .if/.endif, even without
comments.

(rillig)

make(1): format Makefile consistently

(rillig)

make(1): regroup source files

(rillig)

make(1): add test for undefined (or empty) variable in :? modifier

(rillig)

make(1): add test for brk_string with more than 50 words

(rillig)

make(1): remove redundant condition

If there are multiple words from brk_string, none of them can be empty.
And if there is oneSingleWord that is empty, it doesn't hurt to add 0
bytes.

(rillig)

make(1): prefer direct jump over indirect variable assignment

(rillig)

make(1): make condition in ModifyWord_SubstRegex simpler

The generated code is still the same, at least with Clang.

(rillig)

make(1): reduce lifetime of regmatch_t for :C modifier

(rillig)

make(1): rename local variable

(rillig)

make(1): allocate capturing groups for :C modifier statically

Since there are at most 10 capturing groups, it's easier to allocate them
on the stack.

To avoid growing the code size on x86_64, the matches have been moved to
the end of the struct, to keep the other offsets small.

(rillig)

make(1): clean up Var_Export

(rillig)

make(1): inline Var_Parse_State into ApplyModifiersState

There was no need to have this as a separate struct.  Especially none of
the modifiers used by ModifyWords modify this state, therefore it was
not necessary to pass it by reference.  This makes the code easier to
understand since there's less modifiable state.

(rillig)

make(1): document ApplyModifiersState.cp

(rillig)

make(1): eliminate ApplyModifiersState.tstr

This field always pointed to the beginning of the current modifier.
Therefore a better name is "mod", and there is no need to store it in the
shared state.

(rillig)

make(1): only set missing_delim if it is actually missing

Just to make it easier to understand the code.

(rillig)

make(1): fix wrong warning about missing delimiter

(rillig)

make(1): demonstrate wrong error message about missing delimiter

The current practice of setting st->delim at the beginning of each
modifier and resetting it again at the end produces too many error
messages.  In this case, there is no missing delimiter.

(rillig)

make(1): reduce code size on x86_64

(rillig)

make(1): remove parameter from ModifyWordCallback

Most modifiers don't need a variable context, and those that do already
get all their parameters in a custom Args struct.

(rillig)

make(1): make implementation of the :C modifier simpler

(rillig)

make(1): add test for :S,,sep

The empty string does not match anywhere.  In other implementations of
string replacement, an empty string matches at every position of the
source string.

This only works for the :S modifier.  The :C modifier does not allow an
empty search pattern.

(rillig)

make(1): add test for :Q and .newline

(rillig)

make(1): only ask for newline if :Q actually needs it

This simple change reduces the binary size of var.o by 96 bytes (Clang,
x86_64). Given that the code is still the same, just in a different
order, this is a bit surprising.

(rillig)

make(1): clean up various things in var.c

(rillig)

make(1): use consistent and descriptive names for callback functions

(rillig)

make(1): remove unnecessary VarSelectWords_t struct

(rillig)

make(1): extract :ts<separator> modifier into separate function

(rillig)

make(1): clean up implementation of :[...] and other modifiers

(rillig)

make(1): fix edge cases with $ at the end of the :@ modifier

In both parts of the :@ modifier, by passing &pflags to
ParseModifierPart, a final $ was interpreted as an anchor, which only
makes sense in the :S and :C modifiers.

This edge case is neither used by src nor by pkgsrc, except for the unit
tests that have been adjusted.

(rillig)

make(1): add test for nested assignments

This test demonstrates that in ApplyModifier_Assign, the call to
ParseModifierPart must get its pflags.

(rillig)

make(1): make conditions work without pattern flags

The "pattern flag" VARP_NOSUBST was used to prevent evaluating
irrelevant conditions.  Omitting the evaluation flag VARE_WANTRES has the
same effect.

(rillig)

make(1): fix obscure edge case for ${VAR:e$=x}

The characters "$=" do not occur in either src or pkgsrc, therefore this
case is unlikely to occur in practice.

(rillig)

make(1): demonstrate obscure edge case in SysV modifier

(rillig)

make(1): move documentation for assignment modifiers

(rillig)

make(1): add test for :tu modifier with spaces

(rillig)

make(1): prefer memcmp over strncmp

(rillig)

make(1): fix type of VarPattern.lhsLen and rhsLen

String length is measured in size_t, not int.

(rillig)

make(1): fix undefined behavior in :S modifier

The expression word + wordLen - leftLen had resulted in an out-of-bounds
pointer before.  Luckily the heap addresses were high enough in typical
applications to prevent a wrap-around.

(rillig)

make(1): make implementation of the :S modifier simpler

(rillig)

make(1): make modifier handling simpler

Implementing a modifier such as :S or :M should not be concerned with
separating the words of the resulting string.  Ideally this should be
done in the same way by all modifiers.

Before, the :R (filename root) modifier added a separator even if the
resulting filename root was an empty string.  The chances that this
change in behavior breaks anything are epsilon.

The :@ modifier, if it appeared after a :ts modifier, did not use the
word separator from the :ts modifier (which all other modifiers do) but
always added a space.  This behavior has been preserved for now.  It's an
unnecessary inconsistency though.

In contrast to Buffer, the newly added SepBuf uses size_t for memory
sizes and also uses the conventional parameter order (mem, memsize)
instead of the unusual (memsize, mem).

(rillig)

make(1): remove unnecessary UNCONST

(rillig)

make(1): make ampersand in ${VAR:from=to&} an ordinary character

In SysV substitutions, wildcards are expressed with % instead of &.  The
& is not mentioned in the manual page, and having another wildcard for
the whole word would be such an obscure feature that not even pkgsrc uses
it.  The easiest way to discover this feature had been to read the source
code of make(1) or to use a fuzzer and accidentally stumble upon this
edge case.

(rillig)

make(1): don't call VarSubstitute unnecessarily

(rillig)

make(1): remove wrong VarPattern from ApplyModifier_Exclam

(rillig)

make(1): remove wrong VarPattern from ApplyModifier_Assign

(rillig)

make(1): add test for assignment operators in conditions

(rillig)

make(1): flatten ApplyModifier_Assign

(rillig)

make(1): add tests for the ::= modifiers

(rillig)

make(1): fix date in comment in unit test

(rillig)

make(1): add tests for maximum number of capturing groups in :C

(rillig)

make(1): fix off-by-one error in :C modifier

Previously this off-by-one error had invoked undefined behavior.
Until today there was no corresponding unit test though.

(rillig)

make(1): fix double free introduced this morning in r1.258

(rillig)

make(1): remove redundant loop control variable

(rillig)

make(1): rename ApplyModifiersState.flags to eflags

There are just too many variables called "flags" in this file, and their
constants must not accidentally overlap.

(rillig)

make(1): demonstrate off-by-one bug in :C modifier

(rillig)

make(1): clean up VarLoop structure

(rillig)

make(1): remove unused parameter from ParseModifierPart

(rillig)

make(1): rename VarGetPattern to ParseModifierPart

(rillig)

make(1): eliminate unused local variables

(rillig)

make(1): eliminate macro from VarGetPattern

(rillig)

make(1): add test for stray dollar signs in :@ modifier

(rillig)

make(1): add test for ampersand in SysV substitution modifier

(rillig)

make(1): replace ApplyModifiersState field with local variable

(rillig)

make(1): rename flags for :S and :C pattern matching

Renaming these flags makes it obvious that VarPattern is used in a few
places where it doesn't belong.  These will be fixed in follow-up
commits.

(rillig)

make(1): convert VarFind flags to typed enum

(rillig)

make(1): clean up unnecessary snprintf and multi-line function calls

(rillig)

make(1): rename Varf_Flags to VarEvalFlags

In var.c there are lots of different flag types.  To make any accidental
mixture obvious, each flag group gets its own prefix.

The only flag group that is visible outside of var.c is concerned with
evaluating variables, therefore the "e", which replaces the former "f"
that probably just meant "flag".

(rillig)

make(1): clean up handling of :? modifier

The VarPattern type had nothing to do with parsing conditions.

(rillig)

make(1): clean up code in var.c

* use canonical comparison to zero in more cases
* omit unnecessary snprintf for string literal concatenation
* omit a few redundant parentheses and braces

(rillig)

make(1): move SysV string matching to var.c

This kind of string matching is only used in variable modifiers, and only
if this feature is enabled by SYSVVARSUB.

(rillig)

make(1): add more tests for :ts modifier

(rillig)

make(1): demonstrate inconsistent :ts behavior among modifiers

(rillig)

make(1): ignore system-specific error message from regcomp in test

NetBSD 8 has: repetition-operator operand invalid
Ubuntu has: Invalid preceding regular expression

(rillig)

make(1): add post-processing for tests

By defining SED_CMDS.${test}, a test can do additional post-processing
on its output, for example to normalize paths or error messages.

The Makefile for the tests now follows the common style to have all
relevant definitions at the top of the file, and all the implementation
details at the bottom.

By using the += instead of backslashed lists, it becomes easier to
comment out a single test, either temporarily or permanently (such as
the POSIX tests).

The :C modifier has been replaced with the simpler :S, some :S modifiers
have been replaced with simpler := modifiers.

The postprocessing commands have been extracted from the actual command.
This makes editing and commenting them easier.

(rillig)

make(1): improve documentation about adding new tests

(rillig)

make(1): add tests for dollar characters in modifiers

(rillig)

make(1): expand the test for variable expansion in the :@ modifier

(rillig)

make(1): add debugging output for :@ modifier

The inner working of the :@ modifier is quite tricky.  To understand what
really happens, it helps to look at each word as it is being modified.

(rillig)

make(1): constify VarModify parameter

Since var.c r1.238 and r1.239, the callbacks for the :H :T :E :R modifiers
don't modify the word anymore, and the others didn't modify it at all.

(rillig)

make(1): add test for an obscure edge case of using the :@ modifier

(rillig)

make(1): fix :hash modifier on 16-bit platforms

On platforms where int has only 16 bits the shifts would have been in
16-bit arithmetic, which would invoke undefined behavior for "ustr[3] <<
24" as well as "ustr[2] << 16" (C99, 6.5.7p3).

(rillig)

make(1): remove unnecessary _t suffix from VarLoop type

(rillig)

make(1): fix documentation of Var_Subst, clean up other documentation

(rillig)

make(1): remove unnecessary forward declarations, fix indentation

(rillig)

make(1): remove unnecessary forward declarations

(rillig)

make(1): add tests for the :S and :C modifiers

(rillig)

make(1): reduce line length in var.c

(rillig)

make(1): refactor ApplyModifier_SysV to have less indentation

(rillig)

make(1): clean up documentation, small refactorings for variables

- document the callback for VarModify
- clearly mark the callbacks
- shorten the documentation for some callback functions
- fix the documentation of VarSYSVMatch
- remove unnecessary null check from VarLoopExpand
- add test for applying modifiers to empty strings

(rillig)

make(1): add another test for the SysV :%=% modifier

(rillig)

make(1): add expected test result for modmisc

(rillig)

make(1): fix bug in :E modifier, introduced in r1.239 today

That case was not covered by the tests before.

(rillig)

make(1): fix indentation in var.c

Fixing the indentation manually is a tedious task.  indent(1) was no help
since it has several bugs and is not flexible enough for the original
make code.

(rillig)

make(1): add Hash_ForEach to avoid duplicate code

(rillig)

make(1): fix indentation and alignment

(rillig)

make(1): make code for the :E and :R modifiers simpler

(rillig)

make(1): add test for :R modifier

(rillig)

make(1): add more tests for :H, :T and :E modifiers

(rillig)

make(1): clean up code for the :H and :T modifiers

(rillig)

make(1): add test for :H and :T modifiers with several words

(rillig)

make(1): add missing test for the :H and :T modifiers

(rillig)

make(1): refactor ApplyModifiers

Previously, ApplyModifiers was 1200 lines long and pretty complicated.
Now each of the complicated modifiers has its own function, which reduces
the indentation and makes it possible to use early returns.

(rillig)

make(1): remove redundant parentheses around return value

(rillig)

make(1): remove trailing whitespace

(rillig)

make(1): remove redundant parentheses around return values

(rillig)

make(1): refactor Str_Match

- use shorter variable names to avoid line breaks
- eliminate c2 variable
- change return type to Boolean

(rillig)

make(1): clean up Str_Match

- removed unnecessary parentheses
- uniform test for end of string

(rillig)

make(1): clean up documentation of Str_Match

(rillig)

cgram(6): use standard cursor keys, use standard shuffle algorithm

The previous shuffle algorithm asked for 100 random numbers, on average.
The new algorithm asks exactly for 26 random numbers.

Curses predefines numeric constants for keys, and there is no apparent
reason not to use these standard keys for cursor movement.

(rillig)

make(1): fix unnecessary evaluation of variables in :@var@ modifier

(rillig)

make(1): fix unnecessary evaluation for :S :C := modifiers

(rillig)

make(1): demonstrate some more cases for unintended evaluation

(rillig)

make(1): clean up code in var.c

- remove redundant parentheses in return statements
- add missing space after comma
- remove some redundant parentheses around comparisons
- use hexadecimal numbers for bit mask constants
- remove Var_GetTail and Var_GetHead, which had been unused for 19 years

(rillig)

make(1): remove useless parameter from Var_Set

The enum corresponding to this int parameter is only defined in var.c,
which makes it impractical for the outside to set this parameter to
anything but 0.

On x86_64, this reduces the size of the resulting executable by 5 kB.

(rillig)

make(1): use enum instead of #define for VarPattern flags

(rillig)

make(1): add more type safety for enums

There are several types of flags for variables, and these cannot be
mixed.  To prevent accidental typos, these are defined in separate enum
types.  Clang warns about direct assignments between distinct types, but
not about mixing distinct types in binary expressions like A | B.  GCC
does not warn at all.

(rillig)

make(1): fix unnecessary evaluation of :M and :N modifiers

(rillig)

make(1): demonstrate another unintended variable evaluation

(rillig)

make(1): fix edge case for evaluating unnecessary conditions

(rillig)

make(1): demonstrate another unnecessary condition evaluation

(rillig)

make(1): document the actual cause for "recursive variable"

That wrong error message had nothing to do with the .for loop, it was
only caused because the .for loop uses the :U modifier internally.

(rillig)

make(1): fix evaluation of unreachable conditions

Since 2015-10-11, make had evaluated unreachable conditions even though
the manual page said it didn't.

(rillig)

make(1): demonstrate bug when evaluating conditions

(rillig)

make(1): fix performance problem in specially crafted :M modifier

This fix was previously suspected to make the vax build fail.  The next
build succeeded though, and it started 2 hours before this fix was
reverted.

(rillig)

usr.bin/make: revert performance improvement

That change might be the cause of a build failure for vax.
http://releng.netbsd.org/builds/HEAD/202006131940Z/ builds fine.
http://releng.netbsd.org/builds/HEAD/202006141020Z/ doesn't.

The build fails with:
/home/source/ab/HEAD/src/external/gpl3/gcc/dist/gcc/machmode.h:524:28:
error: 'mode_size_inline' was not declared in this scope

(rillig)

usr.bin/make: make Str_Match faster for repeated asterisks

Conceptually related to https://en.wikipedia.org/wiki/ReDoS.

(rillig)

usr.bin/make: remove redundant parentheses around return

(rillig)

usr.bin/make: consistently use ++ for incrementing pointers

(rillig)

usr.bin/make: fix typo in comment

(rillig)

usr.bin/make: fix test failure

The two files include-sub.mk and include-subsub.mk were never intended to
be test cases on their own.  They belong to include-main.mk.

Fixes PR bin/55360.

(rillig)

usr.bin/make: demonstrate actual behavior of .INCLUDEDFROMFILE

(rillig)

usr.bin/make: fix test for dollar and backslash at eol

The previous version of this test relied on the way how the shell
interprets a lonely backslash at the end of the line.  The NetBSD and
FreeBSD shells print the backslash, while Bash doesn't.

While here, make the escaping a bit simpler and align the test
descriptions with the actual test data.

(rillig)

usr.bin/make: fix typo in dollar test

(rillig)

usr.bin/make: sort test cases alphabetically

(rillig)

usr.bin/make: add tests for surprising dollar removal

(rillig)

usr.bin/make: refactor brk_string

The variables are renamed to reflect to which memory region each pointer
belongs.

The variable "curlen" was always zero.

The type of "ch" has changed to char, and its scope is now limited to
its actual use.

Comparisons of pointers are now consistently written as p != NULL
instead of !p, like character comparisons are written as ch != '\0'.

The "store_words_buf" is updated when the function returns, not before.

(rillig)

usr.bin/make: add test demonstrating that .for stops at newline

(rillig)

usr.bin/make: fix test for late evaluated condition

(rillig)

usr.bin/make: add test case for lazy conditions

(rillig)

t_glob.c: clean up test code

In struct vfs_file, using an int as a boolean is an anachronism and has
been replaced with a single-character file type, like in ls(1).

Some other redundant test code has been removed as well since it was
either unreachable or existed only for performance reasons.

(rillig)

t_glob.c: add test cases for hidden directory and file

The existing test code was geared towards every little bit of
performance. It even duplicated the file definitions in vfs_stat in order
to avoid a few strcmp calls. This made the test code fragile. Therefore,
vfs_stat has been rewritten completely to not duplicate any information
from the vfs.

In vfs_stat, the returned st_mode is now more realistic. It had been 0
before. The file mode is only logged when it makes sense. In the ENOENT
case it is not logged anymore.

The debug logging for opendir/closedir now logs the same pointer, so that
the corresponding calls can be matched easily. Failed vfs_opendir calls
are logged as well, to get a more complete picture of which callbacks are
called.

(rillig)

t_glob.c: use distinct names for test structures

Before, the structures and functions defined by the test used the same
prefix as the code to be tested. This made it difficult to draw a line
between these parts.

(rillig)

t_glob.c: move expected globbing result directly into the test cases

This makes the tests more self-contained. The example directory tree that
is common to all the tests is still defined elsewhere, but in the same
file. Setting up the example directory structure in each test would make
the tests even more independent and read.

(rillig)

usr.bin/make: document probabilities for random test failures

Side node: this test will never fail between 2024-04-15 and 2024-07-06.

(rillig)

usr.bin/make: remove dead code

The preprocessor conditions contradicted each other: __hpux__ or __hpux
would need to be defined, and at the same time none of them would need to
be defined.

(rillig)

usr.bin/make: fix typos in comment

(rillig)

Add more tests for variable modifiers in make.

(rillig)

Fix out-of-bounds read in Str_Match.

(rillig)

Demonstrate some more edge cases for the :M modifier

(rillig)

Fix unit test for edge case in :M modifier

(rillig)

Ignore temporary files from running the unit tests

(rillig)

Add another unit test for the :M modifier

(rillig)

Add unit tests for variable modifiers like :M and :N

(rillig)

When listing aliases, sort them alphabetically.

(rillig)

Fix "every" typo in quote from The Mythical Man-Month

(rillig)