make: fix memory leak in IncludeFile

Previously, each .include leaked a copy of the file name.

(rillig)

make: in Arch_ParseArchive, rename libName to lib

For symmetry with mem.

No binary change.

(rillig)

make: remove unused type MFStr

The only binary change is the line number of the assertion in
Substring_Sub.

(rillig)

make: in Arch_ParseArchive, replace MFStr with FStr

This was the only use of the type MFStr, which produced the same code as
FStr.

No binary change.

(rillig)

Add a bunch of missing food items.

(dholland)

make: use simpler memory management for parsing archive members

No functional change.

(rillig)

fix various typos in comments.

(andvar)

tests/make: test and document the newly added "variable" .SUFFIXES

(rillig)

Add .SUFFIXES as read-only variable.

References to ${.SUFFIXES} are handled dynamically in
ParseVarnameLong by calling Suff_NamesStr.

The variable cannot be set normally.

Reviewed by: rillig

(sjg)

Add amdgpu firmware build goo (disabled)

(maya)

ParseVarnameLong: minimize calls to LazyBuf_Get

Reviewed by: rillig

(sjg)

make: rename Parse_SetInput to Parse_PushInput

The word 'set' sounded too much like it would replace the current file,
but instead the file is pushed to the stack, and the previous file is
continued later.

No functional change.

(rillig)

make: remove unused loadedfile.path

(rillig)

Fix copypasto in comment.

(nia)

make: rename ForLoop.sub_next to nextItem

This matches the naming style of the other ForLoop members.

No functional change.

(rillig)

make: fix memory leak in LazyBuf

This affects many operations on variable expressions.  Those using
LazyBuf_Done are affected, those using LazyBuf_DoneGet aren't.

$ cat <<'EOF' > lazybuf-memleak.mk
.for i in ${:U:range=5}
.  for j in ${:U:range=1000}
.    for k in ${:U:range=1000}
.      if 0 && ${VAR:Dpattern\: that needs unescaping}
.      endif
.    endfor
.  endfor
.endfor

all:
@ps -o vsz -p ${.MAKE.PID} | sed 1d
EOF

Before using LazyBuf for the modifier ':D':

    $ make-2021.04.14.16.12.26 -r -f lazybuf-memleak.mk
        VSZ    RSZ
    1357136 1336432

Using LazyBuf for the modifier ':D':

    $ make-2021.04.14.16.59.34 -r -f lazybuf-memleak.mk
        VSZ    RSZ
    1590864 1574164

This commit alone allocates around 150 MB more data, which matches
5_000_000 strings * 30 bytes/string.

It looks very wrong that the above simple makefile uses 1.3 GB of RAM at
all, and it had already done this in 2017, long before LazyBuf was
introduced.  Before 2017.01.30.02.46.20, the above test makefile reports
way smaller numbers, but that's because the modifier ':range' did not
exist back then.

(rillig)

fix various typos, mainly in comments.

(andvar)

s/Miscelaneous/Miscellaneous/ and s/slahes/slashes/ in comments.

(andvar)

s/Miscellanous/Miscellaneous/ in copypasta comments.

(andvar)

make: remove enum.o from Makefile.boot

On 2021-11-18, enum.h and enum.c have been removed.

(rillig)

make: do not report '?' for fully covered header files

(rillig)

make: only define target report-coverage if USE_COVERAGE is yes

Without USE_COVERAGE, GCOV was undefined, the '2>&1' passed all error
messages to GCOV_PERL, when then discarded them.  If the error messages
had been left on stderr, the error message 'sh: arch.o.gcda: not found'
would have been a clear indicator of the actual cause of an empty
coverage report.

(rillig)

tests/make: add tests for irrelevant function calls

(rillig)

tests/make: refine comments for parsing and evaluating conditions

(rillig)

fix typos in word "request(s)".

(andvar)

make: rename ParseFuncArg to ParseWord, update comments

No functional change.

(rillig)

fix typos in word "operation(s)".

(andvar)

make: merge duplicate code in CondParser_FuncCall

No functional change.

(rillig)

make: generate more specific error message on malformed conditional

(rillig)

crashme(9): Touch up man page markup.

Better consistency with sysctl(7), less random widthing and
offsetting.

(riastradh)

Document vchiq@acpi

(wiz)

remove clause 3 from all my licenses that aren't conflicting with
another copyright claim line.  again.  (i did this in 2008 and then
did not update all of my personal templates.)

(mrg)

Update crashme(9) man page for recent additions.  Reorganize slightly.

(riastradh)

Typo in comment

(skrll)

make: remove dead code in CondParser_FuncCall

The return value (size_t)-1 was only used by the function 'empty', which
moved to CondParser_FuncCallEmpty.

No functional change.

(rillig)

make: remove unused function parameter in conditional handling

No functional change.

(rillig)

make: clean up CondParser_FuncCallEmpty

No functional change.

(rillig)

make: inline ParseEmptyArg into CondParser_FuncCallEmpty

No functional change.

(rillig)

make: simplify evaluation of function 'empty'

No functional change.

(rillig)

make: merge duplicate code in ParseEmptyArg

No functional change.

(rillig)

make: use proper return type for ParseEmptyArg

Now that ParseEmptyArg is no longer bound to have the same prototype as
ParseFuncArg, it can use a more appropriate return type instead of
encoding everything in a size_t.

Since ParseEmptyArg never returned 0, that code path was unused.  It was
only used for all other functions.

No functional change.

(rillig)

make: remove unused arguments from CondParser_FuncCallEmpty

No functional change.

(rillig)

make: separate handling of 'empty' and the other functions

No functional change.

(rillig)

tests/make: proofread and clean up comments for function 'empty'

This prepares a refactoring for handling the function 'empty' in
conditionals like '.if'.

The function 'empty' is fundamentally different from all other functions
since it is parsed differently and passes its result on different path
than the other functions.  Splitting up these code paths will untangle
the control flow of parsing a condition like 'empty(VARNAME)'.  It will
also remove several ARGSUSED and MAKE_ATTR_UNUSED that make the current
code smell.

(rillig)

make: merge duplicate code in parsing conditions

No functional change.

(rillig)

make: remove recursion from CondParser_And

No functional change intended.  Before cond.c 1.286 from today, there
would have been the functional change that in malformed conditions, the
extra expression would not be evaluated.  Now that CondParser_Token is
always called with doEval == false, there is no change in behavior to be
expected.

(rillig)

make: do not expand operator token in CondParser_Or

At the point where CondParser_Or calls CondParser_Token, there was a
previous call to CondParser_And.  Due to this, the next token is already
stored in par->curr, and the parameter doEval is ignored.

Changing the argument from doEval to false makes the code similar to the
corresponding code in CondParser_And.

No functional change.

(rillig)

make: do not evaluate misplaced expressions in malformed conditions

This change only affects the behavior for parse errors.  Syntactically
well-formed conditions work exactly as before.

(rillig)

break down large, unaligned copies into chunks that CG14Copy8_short_*()
can handle, make sure destinations are aligned whenever possible
now we only fall back to byte-wise access for really small copies, which
we probably shouldn't bother throwing at SX in the first place

(macallan)

s/occured/occurred/ in comments, log messages and man pages.

(andvar)

tests/make: extend test for parse errors in conditions

(rillig)

useradd(8): Recognize Argon2 passwords as "valid" when they are given
on the command line.

Reported by Robert Nestor on current-users@.

(nia)

make: simplify parsing of '||' in conditions

Previously, the grammar said 'Or -> Or || And', while the code looked
more like 'Or -> And || Or'.  Make the code look like the grammar and
keep track of the resulting value of the condition explicitly.

No functional change intended.

(rillig)

skip reading destination fb if we're going to overwrite it, only read words
we need to partially write
while there write mask registers only if we're actually going to need them
another 20% speedup

(macallan)

s/unaccessible/inaccessible/

(andvar)

tests/make: add more comprehensive tests for short-circuit evaluation

(rillig)

CG14Copy8_short_rop(): skip the funnel shifter if source and destination are
aligned. Small but measurable speedup.

(macallan)

add another Copy8() variant:
- supports unaligned source and destination
- uses all 32bit accesses
- supports copies up to 124 pixels wide so an entire line fits into registers
  and we can ignore x direction
... mostly an exercise in learning how to use the funnel shifter
TODO:
- skip the funnel shifter if source and destination are aligned
- skip fb reads where possible, like straight GXcopy

(macallan)

Move PF mailbox initialization from ixgbe_attach() to ixgbe_init_iov().

  From FreeBSD ix-3.3.18. No functional change.

(msaitoh)

No functional change.

  - Sync with FreeBSD ix-3.3.18.
    - Rename ixgbe_get_advertise() to ixgbe_get_default_advertise.
    - Sort lines, modify comment and whitespace to reduce diff against FreeBSD.

(msaitoh)

Whitespace fix. Sync with FreeBSD ix-3.3.14.

(msaitoh)

Add some unused macros. Sync with FreeBSD ix-3.3.14.

(msaitoh)

Move increments after evaluations.

  FreeBSD: dc11ba4eb3fe5cce615f361de83e85e07005ca24 or ix-3.3.14
  DPDK:    390445ec30b4c52a3d2887c3d2a202d9cf37ea8e

    The retry variable was being incremented before it was evaluated by the
    subsequent conditional against the maximum retries to figure out which
    message to print.  So we'll move the increment op to the end.

(msaitoh)

ifdef D(ebug)? From FreeBSD ix-3.3.14.

(msaitoh)

Fix infinite recursion on PCIe link down.

  - FreeBSD: 8270b7174c48417a4d5f3effa4a4f4588205e687 or ix-3.3.14
  - DPDK:    2d04b9e856125197ec8e967471426d56ab7efcf0

    In some corner cases the functions ixgbe_clear_rar_generic and
    ixgbe_clear_vmdq_generic may call one another leading to infinite
    recursion.

    When ixgbe_clear_vmdq_generic is called with IXGBE_CLEAR_VMDQ_ALL
    flag, it's going to clear MPSAR registers, and proceed to call
    ixgbe_clear_rar_generic, which in turn will clear the RAR registers,
    and recursively call back ixgbe_clear_vmdq_generic. Normally, the
    latter would detect that MPSAR registers have already been cleared
    and terminate the recursion.

    However, when PCIe link is down, and before the driver has had the
    opportunity to shut itself down, all register reads return 0xFFFFFFFF,
    and all register writes fail silently. In such case, because
    ixgbe_clear_vmdq_generic blindly assumes that clearing MPSAR registers
    succeeded, it's going to always call ixgbe_clear_rar_generic, which
    in turn will always call back ixgbe_clear_vmdq_generic, creating
    infinite recursion.

    This patch re-reads MPSAR register values after they had been cleared.
    In case of PCIe link failure, the values read will be non-zero, which
    will terminate the recursion. On the other hand, under normal
    circumstances the value read from MPSAR registers is going to be equal
    to the value previously written, so this patch is expected not to cause
    any regressions.

  - Note that NetBSD doesn't support VMDQ.

(msaitoh)

Remove debug error message.

(msaitoh)

Change argument. No functional change. This file is not used in NetBSD.

(msaitoh)

Remove unnecessary return value check.

  FreeBSD: 3a89005394bc5d82ce9b6baa9e7f8dee362354ae
  DPDK:    4b0ee6529b7897c2a08dd56669f07ac1f46a8474

(msaitoh)

Remove unused argument. Note that this file is not used in NetBSD.

(msaitoh)

Add typecast for type mismatch.

  FreeBSD: 994dd6328c66fc277438ad51ed074f3c52096147
  DPDK:    d8e52b2cf771c31b523b46852fd86225b5a2c721

(msaitoh)

Initialize data field in struct buffer.

  FreeBSD: b3ebe337ffa06b0f1f460bf8f1e42fb55db77d0b or ix-3.3.14
  DPDK:    40543be5376ca415b2a7e196315d0555725b8bdf

    While sending request using ixgbe_hic_unlocked() the data field in
    buffer struct is not used. It is set when the struct is overwritten by
    FW to deliver the response. To not pass random data to FW the whole
    structure should be zeroed before use.

(msaitoh)

s/OCCURED/OCCURRED/ in a debug message.

(msaitoh)

s/premissions/permissions/

(msaitoh)

Fix previous, don't copy up if the underlying node is unreadable.

(hannken)

Simplify.  Same code before and after.

(skrll)

Trailing whitespace

(skrll)

Add comment to clarify.

(msaitoh)

tests/make: prevent the bug from cond.c 1.283 from happening again

(rillig)

make: revert previous commit to CondParser_Or

Even though the unit tests for make cover a lot of cases, they don't
cover all cases.  After the previous commit, the NetBSD build failed
with:

bsd.sys.mk line 247: \
    Malformed conditional (!defined(NOPIE) && \
    (!defined(LDSTATIC) || ${LDSTATIC} != "-static"))

(rillig)

make: avoid recursion in CondParser_Or

Previously, a long chain of '1 || 1 || 1 || 1 || ...' led to a deep
recursion.  Furhermore, the code didn't match the grammar on superficial
reading: the grammar said "or || and", the code said "and || or".

No functional change.

(rillig)

s/refernce/reference/

(andvar)

Few typos slipped in the last review, fixing them.

(andvar)

s/atached/attached/ and removing double s in state.

(andvar)

make: add back support for jemalloc 100

The ports sun2 and vax default to jemalloc 100, as opposed to the rest
of the ports, which use jemalloc 510.  Some developers choose to use
jemalloc 100, so let them run the tests with memory debugging as well.

Suggested by mrg on source-changes-d.

(rillig)

make: in parse errors, mark whitespace more clearly

This prevents any trailing whitespace from going unnoticed.  It also
marks leading whitespace more clearly, as in the examples with the time
value " 1".

(rillig)

make: remove period from end of error messages and warnings

The majority of the existing error messages and warnings does not
include a period at the end.  Follow this style consistently.

(rillig)

Pixel == unsigned long, fix format.

(christos)

Print TLB message consistently to improve readability.

Example:
cpu0: L2 cache: 256KB 64B/line 4-way
cpu0: L3 cache: 4MB 64B/line 16-way
cpu0: 64B prefetching
-cpu0: ITLB: 64 4KB entries 8-way, 2M/4M: 8 entries
+cpu0: ITLB: 64 4KB entries 8-way, 8 2M/4M entries
cpu0: DTLB: 64 4KB entries 4-way, 4 1GB entries 4-way
cpu0: L2 STLB: 1536 4KB entries 6-way
cpu0: Initial APIC ID 0

(msaitoh)

Print 1GB TLB entry at the same leve's line.

Example:
  before:
    cpu0: ITLB: 128 4KB entries 8-way, 2M/4M: 8 entries
    cpu0: DTLB: 64 4KB entries 4-way
    cpu0: L2 STLB: 4K/2M: 1024 entries
    cpu0: L1 1GB page DTLB: 4 1GB entries 4-way

  after:
    cpu0: ITLB: 128 4KB entries 8-way, 2M/4M: 8 entries
    cpu0: DTLB: 64 4KB entries 4-way, 4 1GB entries 4-way
    cpu0: L2 STLB: 4K/2M: 1024 entries

(msaitoh)

PR misc/56541 ctwm complains about TitleFocus line in system.ctwmrc

TitleFocus is the default so it's likely safe to remove, it being
kept here was only left over from us previously having NoTitleFocus.

(nia)

tests/t_magic_symlinks: remove duplicate word in test description

(rillig)

Omit RCS Ids from test input and expected output files.

(agc)

Elex - an embeddable regexp-based lexer
=======================================

I have found myself fairly often needing a lexer utility to tokenise
input (for configuration files, for various file-scanning utilities,
and for other applciations), but using full-blown lex(1) program to do
this is overkill, or designed for a separate process, which doesn't fit
well with the design - syntax-coloring editors, for example.

This utility, elex, is a regexp-based tokenizer, an embedded lexer,
which can be used for various uses.  It usually takes a lexer file
(similar to lex input files), although the API allows lexers to be
built on the fly by just issuing the calls to make new rules.

Implementation
==============

Normal lex(1) is implemented (usually) as a conglomeration of all the
regular expressions for a start state - if multiple matches are found,
the largest match is the one used.  I've found that, in practice, this
constrains the way a number of things are done.  So I've implemented
elex, using a multiple, prioritised multiple regexp matching scheme.
This supercedes the usual way of distinguishing reserved words and
identifiers in the lexer - recognising all "words" first, and
searching for each word through a number of tables; if a match is not
found, then the word recognized is an identifier.  elex works around
this by using a regexp to match reserved words first, and then to
recognise the word as an identifier after that.  Since normal regular
expressions usually progress through the input trying to find a match,
the regular expressions used in elex are constrained by anchoring the
search, not allowing progression through the input.  In practice, this
makes for more efficient matching.

another side effect is the ability to use more modern regexp features,
such as perl escapes, UTF-8 matching, in-subexpression ignore case, etc.

elex implements start states, similar to flex.  These are useful for
recognising multiline comments (almost any language), or multi-line
strings (perl, python, lua etc).

elex dynamically sizes the regmatch arrays used to accommodate the
largest regexp in the input, and matching subexpressions can be
returned to the caller.  The 0'th subexpression is the whole matching
expression, and is the same as "yytext".

And so on to an elex definition which recognises C and some C++:

# start state
%state COMMENT

# the types we define
%type IDENT 0xdb8ea4d
%type PUNCT 0xe454e3a
%type NUMBER 0xca1edaec
%type COMMENT 0xee5ae423
%type CONSTANT 0xd497741f
%type PREPROC 0xdcf9b98d
%type RESWORD1 0xb5ac6a6a
%type RESWORD2 0xb5ac6a6b

# and finally... the rules

<INITIAL>(auto|char|class|const|double|enum|extern|float|friend|inline|int|long|mutable|namespace|new|private|protected|public|register|requires|short|signed|static|this|struct|this|typedef|union|unsigned|void|volatile)\></> { return RESWORD1; }
<INITIAL>(asm|break|case|catch|continue|default|do|else|for|goto|if|return|switch|throw|try|while)\></> { return RESWORD2; }
<INITIAL>[a-zA-Z_][0-9a-zA-Z_]*</> { return IDENT; }

<INITIAL>([1-9][0-9]*|0x[0-9a-f]|0X[0-9A-F]+|0[0-7]*|'(\\.|[^'])*')</> { return NUMBER; }
<INITIAL>[ \t\n\r]+</> { return PUNCT; }

<INITIAL>/\*</> { BEGIN(COMMENT); return COMMENT; }
<COMMENT>[^\n]*\*/</> { BEGIN(INITIAL); return COMMENT; }
<COMMENT>\n|[^\n]+</> { return COMMENT; }

<INITIAL>//[^\n]*</> { return COMMENT; }

<INITIAL>"(\\.|[^"])*"</> { return CONSTANT; }

<INITIAL>(==|[-]>|!=|<=|>=|~=|%=|&=|[*]=|[-]=|[+]=|[|]=|(<<|>>)=?)</> { return PUNCT; }
<INITIAL>[\u005b;(){}\u005d*<>,+/%~!\u005e&=|.?:\u002d]</> { return PUNCT; }

<INITIAL>#[ \t]*(define|el(se|if)|endif|error|if|ifn?def|include|line|pragma|undef)[^\n]*</> { return PREPROC; }

Start states are explicitly used for rules, since it is easier to read
in practice.  Elex comments are eol-style comments, beginning '#' and
ending with '\n'.  Types can be defined using the "%type" directive,
and the unsigned 32bit value they take will be returned.  This is more
work than using magic constants, but much more readable in practice -
see the example calling program below.

A rule is of the form:

<startstate>regexp</> { BEGIN(newstate); return TYPE; }

(the BEGIN() is optional.  </> terminates the regexp - it's clearer
than wrapping everything in double quotes or some other normal
character which would then have to be escaped.  The return is also
optional, but is usually present in practice.)

Hex codepoints can be used to specify wildcard characters in character
classes - the ones enclosed in [] above).

Start states can be defined using the %state (or %x) directive, and
transitioned to using the BEGIN() action, in the same way as standard
lex(1).

Elex provides bookmarks, which are numbered numerically from 0.
Assuming a mark has already been successfully created using
"set-mark", the bookmark offset can be retrieved by using its index
using "get-mark", and the user can then seek to that offset.  This
makes it possible to set up, and re-visit sync points, in case there
was ever a programming language or other input devised which was
ambiguous to some degree.

A heavier-weight alternative to bookmarks is to simply clone the elex
structure at the point of possible divergence, and perform the
alternatives on the clones.  A deep copy is made during the clone
operation.

elex is called from a C (or other language) program in much the same
way as lex is:

int
main(int argc, char **argv)
{
uint64_t len;
elex_t *elex;
size_t size;
char *lexfile;
char *input;
char *text;
char *s;
int graphic;
int type;
int i;

input = lexfile = NULL;
s = NULL;
size = 0;
graphic = 0;
while ((i = getopt(argc, argv, "Vf:gi:")) != -1) {
switch(i) {
case 'V':
printf("elex version " ELEX_VERSION_S(ELEX_VERSION_NUM) "\n");
exit(EXIT_SUCCESS);
case 'f':
lexfile = optarg;
break;
case 'g':
graphic = 1;
break;
case 'i':
input = optarg;
break;
}
}
if ((elex = elex_new()) == NULL) {
errx(EXIT_FAILURE, "can't create new elex structure");
}
if (!elex_exec(elex, "read-defs-file", lexfile, 0)) {
errx(EXIT_FAILURE, "can't read defs file '%s'", lexfile);
}
if (input == NULL) {
input = argv[optind];
}
if ((s = readfile(input, &size)) == NULL || size == 0) {
errx(EXIT_FAILURE, "can't read input file '%s'", input);
}
if (!elex_exec(elex, "insert", s, size)) {
errx(EXIT_FAILURE, "can't set input");
}
while (elex_exec(elex, "next-token", NULL, 0) != 0) {
type = elex_exec(elex, "get-yytype", NULL, 0);
text = elex_exec_str(elex, "get-yytext", 0, &len);
if (graphic) {
switch(type) {
case /* "IDENT" */ 0xdb8ea4d:
case /* "PUNCT" */ 0xe454e3a:
case 1:
/* no color, standard */
printf("%.*s", (int)len, text);
break;
case /* "NUMBER" */ 0xca1edaec:
case 2:
p(CYAN, len, text);
break;
case /* "COMMENT" */ 0xee5ae423:
case 3:
p(BRIGHT_RED, len, text);
break;
case /* "CONSTANT" */ 0xd497741f:
case 4:
p(MAGENTA, len, text);
break;
case /* "PREPROC" */ 0xdcf9b98d:
case 5:
p(GREEN, len, text);
break;
case /* "RESWORD1" */ 0xb5ac6a6a:
case 6:
p(BRIGHT_GREEN, len, text);
break;
case /* "RESWORD2" */ 0xb5ac6a6b:
case 7:
p(BRIGHT_YELLOW, len, text);
break;
default:
printf("%.*s", (int)len, text);
break;
}
} else {
printf("[%d] %d '%s'\n", type, (int)len, text);
}
free(text);
}
elex_dispose(&elex);
exit(EXIT_SUCCESS);
}

This program, when invoked with -g, and using the elex definition above,
will perform the same syntax coloring on C programs as is performed in
my editor.

All access to elex is through an opaque elex_t structure pointer.

There are 5 functions in the API:

/* create and destroy */
elex_t *elex_new(void);
int elex_dispose(elex_t **/*elex*/);

/* these functions do ALL the work */
int64_t elex_exec(elex_t */*elex*/, const char */*info*/, const char */*s*/, int64_t /*n*/);
void *elex_exec_str(elex_t */*elex*/, const char */*info*/, uint64_t /*n*/, uint64_t */*size*/);

/* with one exeception - deal with states */
int elex_make_new_rule(elex_t */*elex*/, const char */*startstate*/,
const char */*pat*/, const char */*newstate*/, int /*ret*/);

An editable input buffer is held in the elex structure.  This can be
set, additional input inserted, and bytes deleted from it.  Callers
can seek within the input buffer, although it is the caller's
responsibility to set the start states correctly after any seeking.

Usually a single file is used (which can be re-used by other programs),
although new start states, types and rules can be done manually.

Note that the type returned on EOF is 0, as in lex, so it is best to
start your types at 1; an alternative is to use stringswitch methods
by returning a hash value of the state name. See the elex file above.

There are 2 main accessor functions:  elex_exec(), which executes an
action and returns a numeric result, and elex_exec_str() which
executes an action and returns a string (allocated using calloc(1),
and it is the caller's responsibility to free this storage).  Within
each of these functions, the second, string argument gives the action
to be performed.  These actions are:

elex_exec:
"delete" - delete n chars from the input
"get-input-length" - return the length of the input buffer
"get-mark" - return bookmark number n (see set-mark)
"get-yyleng" - get the length of the pattern matched
"get-yyline" - get the current line number in the input buffer
"get-yytype" - return the type of the pattern matched
"hash" - return djbhash value for a string
"insert" - insert bytes into the input buffer
"new-state" - make a new start state
"new-type" - make a new type
"next-token" - perform the equivalent of yylex(). Return the type of
the pattern matched.  If no pattern is matched, print the
current byte on stdout and advance 1 byte.
"read-defs" - read the elex definitions from memory
"read-defs-file" - read the elex definitions from a file
"seek" - move to an ofset in the input buffer. It is the caller's
responsibility to set the start state after using this action
"tell" - return the current offset in the input buffer
"set-mark" - set a bookmark at the current offset, and return the index of
the mark
"set-start-state" - set the current start state

elex_exec_str:
"clone" - make a deep copy of the elex struct
"get-rule" - return the full rule for the n'th rule
"get-subexpr" - for the current match, get the n'th matching subexpression in allocated space
"get-yyrule-pat" - get the input pattern for the matching regexp in
allocated space
"get-yystate" - get the name of the current state in allocated space
"get-yytext" - get the full text of the current match in allocated space

There is also a function to add a new rule, elex_make_new_rule. In future,
this will be extended to allow changing rules on the fly (an example use
of this would be to use a regexp to recognise identifiers in the language).

Designing elex definition files
===============================

Firstly, any start states are defined. Start states are states in elex
which are different from the initial state, aka INITIAL. They are generally
used for complex recognition, like multi-line strings or comments - they
work more effectively, and the definitions are clearer.

After start states, the next section to be defined is the types
section.  This allows us to define the return types that elex will
return to the calling program.  Since a 0 type usually denotes end of
input, it is advised to define return types starting at 1.
Historically, in lex definitions, the user-defined types started at
256, and it was common to return ASCII values for single characters up
to 256.  Since this is no longer acceptable in a world with multibyte
characters, and because we tend to tokenise based on types of input
tokens, hopefully this practice will never be used again.

Current start states are specified at the start of the rule.

<INITIAL>/\*</> { BEGIN(COMMENT); return COMMENT; }
<COMMENT>[^\n]*\*/</> { BEGIN(INITIAL); return COMMENT; }
<COMMENT>\n|[^\n]+</> { return COMMENT; }

To use the start state above as an example, if we are in the initial state, and
get a "/*" pattern in the input, we transition to the COMMENT state using the
"BEGIN" action, and return the COMMENT type.

Once we are in the COMMENT state, if we receive any text terminated by
a "*/" pattern, we transition to the INITIAL state, and return a
COMMENT type.  Having specified the termination of the COMMENT state,
if we receive any other text, we simply return a COMMENT at the end of
each line.

It is also possible just to specify a COMMENT action as

<INITIAL>/\*.*\*/</>

but any special actions to be taken on a newline character would be missed
in this.

Usually, when tokenising programming language, there would be a number
of definitions for reserved words, and standard identifiers. There would
also be definitions for punctuation, and numeric and string constants.
Some languages have definitions for multiline strings.

Alistair Croooks
Thu Nov 18 16:57:44 PST 2021

(agc)

fix typo, pointed out by Andras Farkas

(snj)

fix various typos in comments and log messages.

(andvar)

s/desireable/desirable/ in comments.

(andvar)

s/efficent/efficient/ in comments.

(andvar)

s/speficication/specification/

(andvar)

s/recomended/recommended/ and s/unknonwn/unknown/

(andvar)

be more consistent with underscores in instruction names

(macallan)

add SX_ROPB instruction

(macallan)

Ticket #1720

(martin)

Pull up the following, requested by msaitoh in ticket #1720:

sys/arch/x86/include/specialreg.h 1.146, 1.171,
1.173-1.178 via patch
sys/arch/x86/x86/identcpu.c 1.106, 1.117,
1.122 via patch
sys/arch/x86/x86/pmap.c patch
sys/external/bsd/drm2/drm/drm_cache.c 1.14
usr.sbin/cpuctl/arch/i386.c 1.114-1.117

- Add PT, PKRU, HDC, LA57, PKE, PKS, CET, CET_U, CET_S, HWP, KL,
  AVX512_BF16, TME_EN and PCONFIG.
- Rename some macros to match the x86 specification and the other OSes.
- Print CPUID 0x8000008 %ebx on Intel, too.
- Print CPUID leaf 7 subleaf 1.
- Identify Tiger Lake, 3rd gen Xeon Scalable (Ice Lake), Elkhart Lake
  and Jasper Lake.
- Remove a few unused MSRs.
- Add comment.
- KNF. Whitespace fix.

(martin)

Tickets #1391 and #1392

(martin)

Pull up following revision(s) (requested by hauke in ticket #1392):

sys/dev/ipmi.c: revision 1.6
sys/dev/ipmi.c: revision 1.7
sys/dev/ipmi.c: revision 1.8
sys/dev/ipmi.c: revision 1.9

Fail open when ipmi device was found.
Clear sc_envsys on failure to prevent double-free.

ipmi(4): Use config_pending_incr/decr.
Don't abuse autoconf's private (and incoherent) dv_flags.

ipmi(4): Notify config_pending_decr when ready, not when dying.
Should fix hang at boot.

ipmi(4): Tidy up ipmi_thread a little.
- Join on detach -- don't free anything until thread has exited; thread
  may still be using stuff.
- Nix dead error branch -- malloc(M_WAITOK) cannot fail.
- x = malloc(sizeof(x[0]) * n), not x = malloc(sizeof(type_t) * n)

(martin)

Pull up the following revisions, requested by msaitoh in ticket #1391:

sys/arch/x86/include/specialreg.h 1.171, 1.173-1.178
sys/arch/x86/x86/identcpu.c 1.106, 1.117,
1.122 via patch
sys/dev/nvmm/x86/nvmm_x86.c 1.18
sys/external/bsd/drm2/drm/drm_cache.c 1.14
sys/external/bsd/drm2/include/asm/cpufeature.h 1.5
usr.sbin/cpuctl/arch/i386.c 1.114-1.117

- Add LA57, PKE, PKS, CET, CET_U, CET_S, HWP, KL, AVX512_BF16, TME_EN
  and PCONFIG.
- Rename some macros to match the x86 specification and the other OSes.
- Print CPUID 0x8000008 %ebx on Intel, too.
- Print CPUID leaf 7 subleaf 1.
- Identify Tiger Lake, 3rd gen Xeon Scalable (Ice Lake), Elkhart Lake
  and Jasper Lake.
- Add comment.
- KNF. Whitespace fix.

(martin)

Make .Nx \*V happy for current builds.

I'm not sure why mdoc complains here (not the technical reason, but
the pragmatic purpose).  My best guess is that it wants to catch
mistakes like .Fx 1.6.1 (instead of .Nx 1.6.1).  I'm not entirely
convinced this doesn't create more problems that it solves.

No objections in the 15 years since this was proposed netbsd-docs.

(uwe)

Punctuation in macro invocation must be a separate argument.

(uwe)

make: document that the input buffer is not null-terminated

(rillig)

make: inline common subexpression in ParseRawLine

There is no need to load the buf_end from memory each time a character
from a makefile is processed.  The end of the buffer only changes when
the file changes, not while reading a single line.

No functional change.

(rillig)

bsd.clean.mk: fix typo in comment

(rillig)

fix typos in word "instead", mainly in log messages.

(andvar)

s/listner/listener/

(andvar)

make: inline HashIter_Init

It is only used in non-critical code paths, but the generated code gets
smaller by inlining.

No functional change.

(rillig)

make: merge branches for modifiers ':D' and ':U'

These branches have the exact same code, without involving any macros,
nevertheless GCC 10 doesn't merge them.  Further, GCC thinks that the
function ApplyModifier_Defined were called from two different places,
thus creating an actual function definition.  Merging the case labels
inlines the function, saving a few instructions and a few hundred bytes
in the binary.

No functional change.

(rillig)

s/dissallow/disallow/

(andvar)

fix various typos, mainly in comments.

(andvar)

make: extract common code into Expr_Words

No functional change.

(rillig)

m4 manual - bring over the rest of the tmac.sU bits.

All tmac.sU does is sets wider text and defines the .MH abbreviation
for "Murray Hill" (.MH was brought over in the previous commit).

(uwe)

m4 manual - adapt to current -ms

Comment out macros related to the Bell Labs memorandum style.
Bring over .MH (abbreviation for "Murray Hill") from tmac.sU.

(uwe)

m4: fix typo in error message

To trigger the error message, compile m4 with -DREAL_FREEZE, then run
'./m4 -R /dev/null'.

(rillig)

m4 - bring back the m4 manual.

This is the version from the csrg-final tree by way of OpenBSD.
FreeBSD version is identical modulo slightly different license text
formatting and a few macro fixes.

(uwe)

G/C pckbc_attach_args

(skrll)

m4: remove PSD.doc

make: don't know how to make m4.ms. Stop

(rillig)

libgmp: remove trailing whitespace

(rillig)

libgmp: suppress lint warnings about int/size_t mismatch

The cases where numbers have 4 GB of precision are probably rare.

(rillig)

libgmp: clean up LINTFLAGS

Now that gmp-impl.h includes <stdlib.h>, defining alloca is no longer
necessary.

(rillig)

what: use numeric exit status, as indicated by the comment

ok christos

(rillig)

Mention the SCMD device

(brad)

A driver and user land utility for the Sparkfun Serial Controlled Motor
Driver module as illustrated here:

https://www.sparkfun.com/products/13911

A SCMD module is a ARM SOC simular to a Arduino in front of a motor
driver chip.  The single SCMD module can control two motors and up to
16 additional modules can be chained together using an internal I2C
bus.  One can interface with the SCMD using tty uart commands, SPI or
I2C.  The driver in this commit adds a kernel driver for the I2C and
SPI interfaces.  The command line utility provides a set of
convenience commands that support most of the functions of the SCMD
and is able to use the tty uart mode, SPI user land or the included
kernel driver in a uniform manor.

The use of the SCMD module is mostly for small robots and the like,
but it can control anything that is controllable by voltage.

(brad)

remove impossible test.

(christos)

Tickets #1386 - #1390

(martin)

Tickets #1716 - #1719

(martin)

Pull up following revision(s) (requested by msaitoh in ticket #1719):

sys/arch/x86/x86/identcpu.c: revision 1.121

make a numeric literal unsigned as it is bit-negated.

(martin)

Pull up following revision(s) (requested by msaitoh in ticket #1390):

sys/arch/x86/x86/identcpu.c: revision 1.121

make a numeric literal unsigned as it is bit-negated.

(martin)