doc/gawknotes.texi - master - niceSOFT/gawk

doc/gawknotes.texi
737 строк26 KB
Arnold D. Robbins
Rename notes.texi gawknotes.texi.
25 мар 2026, 20:25
25 мар 2026, 20:253854e92
\input texinfo   @c -*-texinfo-*-
@settitle GNU Awk Maintainer Notes
@syncodeindex pg cp

@set UPDATE-MONTH February, 2024
@set EDITION 0.5
@set TITLE GNU Awk Maintainer Notes

@copying
@c For HTML, spell out email addresses, to avoid problems with
@c address harvesters for spammers.
@ifhtml
@macro EMAIL{real,spelled}
``\spelled\''
@end macro
@end ifhtml
@ifnothtml
@macro EMAIL{real,spelled}
@email{\real\}
@end macro
@end ifnothtml

This manual is for maintainers of GNU @command{awk}.

Copyright @copyright{} 2024 Arnold Robbins.

This is Edition @value{EDITION} of @cite{@value{TITLE}}.

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation.
@end quotation
@end copying

@titlepage
@title @value{TITLE}
@subtitle Edition @value{EDITION}
@subtitle @value{UPDATE-MONTH}
@author Arnold Robbins
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@node Top
@top GNU Awk Maintainer Notes

This manual is for maintainers of GNU @command{awk}.

@menu
* GNU Awk Maintainership Background and Context:: GNU Awk Background and
                                                  Context.
* General Maintainership Background::             General Maintainership
                                                  Background.
* General Maintainership::                        General Maintainership.
* Index::
@end menu

@c first note
@node GNU Awk Maintainership Background and Context
@chapter GNU @command{Awk} Background and Context

The purpose of this document is to provide background on GNU @command{awk},
(@command{gawk}), it's code and it's development.  The idea is to have
something new maintainers can read in order to help them come up to
speed.

@section Goals
Here are my personal goals for bringing new maintainers on board.
Over the next few years, I want to:

@itemize @bullet
@item
Do a thorough review of the manual, and get it published again.

@item
Bring you two up to speed to the point that each of you would be able to:

@itemize ---
@item
Fix bugs as reported on the bug list

@item
Make releases

@item
Make improvements in the code base
@end itemize

@item
This will allow me to become ``Maintainer Emeritus''.
In that position I hope that I would still be contributing,
but no longer bearing the main responsibility.
@end itemize

This won't happen overnight.
The code base is large and there's a lot to come up to speed on, and I don't
expect miracles, we all have lives and day jobs.
``Slow but steady'' will be the watch word.


@section Some History

The manual states:

@quotation
@cindex Rubin, Paul
@cindex Fenlason, Jay
@cindex Stallman, Richard
@cindex Woods, John
@cindex Trueman, David
Paul Rubin wrote @command{gawk} in 1986.  Jay Fenlason completed it,
with advice from Richard Stallman.  John Woods contributed
parts of the code as well.  In 1988 and 1989, David Trueman,
with help from me, thoroughly reworked @command{gawk} for compatibility
with the newer @command{awk}.  Circa 1994, I became the primary
maintainer.  Current development focuses on bug fixes,
performance improvements, standards compliance, and,
occasionally, new features.
@end quotation

@cindex Aho, Alfred
@cindex Weinberger, Peter
@cindex Kernighan, Brian
In October of 1987, I picked up a copy of the Aho, Kernighan and Weinberger
book on @command{awk}.
Up until then, the documentation on @command{awk} was so skimpy that I'd stayed away
from it; the book made things much clearer, and also more exciting!
I wanted to give (new) @command{awk} a try.

New @command{awk} wasn't easily available at the time, so I went to see what
the GNU project had, and lo and behold, there was indeed a GNU @command{awk}.
This very first version was a (buggy!) clone of the original Unix
@command{awk}. Having an interest in compiler and interpreter technology, I
volunteered to work on @command{gawk} and help bring it into compatibility with
the new version. (I was single and owned my own Unix system...)

@cindex Trueman, David
David Trueman had already volunteered to do that, so I joined his effort.
We set up UUCP links and exchanged diffs via email; he was the primary
maintainer.

@cindex Barlow Close, Diane
At some point, I asked if there was a manual; a 90 page Texinfo draft existed
(written by Diane Barlow Close) and I started adding to and improving it.

@cindex Trueman, David
@cindex Robbins, Arnold
David and I met a time or two in person at USENIX conferences; I may even still have some
notes on our feature plans, most of which eventually made their way into
@command{gawk}.

@cindex Trueman, David
Circa 1994 (I don't remember if it was '93 or '94) David had to pull out of
the project, and I became the sole maintainer.
We had a port to MS-DOS and also to VMS around then, and @command{mawk} had
also been written and released.
There were also various ports to different kinds of then-common Unix systems.

@cindex Adams, Rick
In the mid-90s @command{gawk} got some heavy-duty use (Rick Adams as UUNET
started to use it for his accounting because of the ``no arbitrary limits''
principle), but people were still using the Unix version mostly.

As Linux systems started to grow in popularity, @command{gawk} became
@emph{the} installed version of @command{awk}, and it really began to stabilize
and get many bugs worked out.
When it was just ``also'' available, it wasn't as heavily stressed as when it
was @emph{the} @command{awk}.
So, this was good.

The manual has a section listing in detail the order in which features were
added to @command{gawk} over time.
That list is very long, so there's no need to repeat it here.

The major versions of @command{gawk} through 3.x all built a parse tree
representing the program, and then recursively evaluated that tree in order to
execute the program.

@cindex Haque, John
At version 4.0, that changed, when an outside developer (John Haque) changed
@command{gawk} to generate and interpret ``byte codes'' representing the
program. As part of that work, John also added an @command{awk}-level debugger, which is
really useful.
This structure remains in place today.

@cindex Haque, John
John Haque left the project in December of 2012 over a disagreement with me
as to how the project should be run.
He wanted to see things being done in a certain way, which I did not have the
cycles to do, but he himself did not want to help make those changes.
(I had offered to promote him to co-maintainer, but he declined).
Effectively, ``here is what you should do, but I'm not going to help.''
Since that didn't suit me, he left in a bit of huff, but that's life.

Today, I feel like I have stretched the @command{awk} language as much as it
should be (more really).
I think future facilities should be added via the extension mechanism and not
via language changes.

There remains plenty of interesting work to do in terms of the code
structure, data structures, and performance, which I no longer have the
motivation to really tackle;
I'm hoping new blood will be able to do some of these things. :-)

For what it's worth, as many versions as I could find are in the
@command{git} repository and tagged appropriately, so it's possible to
go back and see the earlier versions.


@section First Steps

Here are some first steps for every new co-maintainer candidate:

@enumerate
@item
Start the process to sign paperwork to assign copyright in your changes
to @command{gawk} and its documentation to the Free Software Foundation.
Please send an email to
@email{copyright-clerk@@fsf.org}
asking for paperwork to assign copyright in all current and future changes to
@command{gawk} and its documentation to the FSF.
You will have to do this before any patches can be integrated into the code
base.

@item
Check out the @command{gawk} documentation online at the GNU project.
In particular, look at the @command{gawk} workflow document for how things
will start out for your contributions.

@item
Also, if you haven't read the @command{gawk} manual, please do.
You will probably know a lot of what's there, but maybe not everything.
There's a lot there.
The section on contributing code is most relevant in terms of how patches
need to be done, but you'll need to learn the language itself as well.
@end enumerate

@c second note
@node General Maintainership Background
@chapter General Maintainership Background

This chapter describes the processes and tools you shold
be familiar with, with general pointers to specific
documents to read.  All of them can be found easily on
the Internet, so specific URLs are not included.

@section Workflow and Tooling Outline

This section provides some background introductory material and pointers.
@c You should be able to find some of these documents just by searching.

@subsection GNU Coding Standards
This document describes how to write code for the GNU Project.
If you haven't read it, you should.
The most important point it makes is the ``no arbitrary limits'' principle:
GNU programs should not fall over due to fixed-size buffers, for example.

The document also describes code formatting standards for GNU programs.
Note though, that with one exception, @command{gawk} does not follow the coding
style, instead using standard ``K & R'' code layout, with tabs, not spaces. The
one exception is that function names in function definitions are placed at the
beginning of the line.

These days the document lives in GNULIB:

@example
git clone https://git.savannah.gnu.org/git/gnulib.git
cd gnulib/doc
texi2pdf standards.texi
@end example

The @command{gawk} manual has a detailed section on C coding style; please read
that after reading the coding standards.

@subsection maintain.texi
This document describes high level general stuff to know if you're going to
be a GNU maintainer.
These days it also lives in GNULIB:

@example
cd gnulib/doc
texi2pdf maintain.texi
@end example

@noindent
It's necessary, if not exciting, reading.

@subsection Savannah
The GNU project has its own software hosting, mainly for Git and CVS,
called
@uref{https://savannah.gnu.org, Savannah}.
This is where the @command{gawk} Git repository is maintained.
It's worth poking around there.
Eventually you'll want to create a user account so that you'll be able to
commit directly to the repository, but this is not urgent at this point.

@subsection Mailing list or bug tracker
@command{gawk} has two mailing lists:
@email{bug-gawk@@gnu.org}
for bug reports, and
@email{help-gawk@@gnu.org}
for people looking for help.
I sometimes skim the latter via the web archive but do not receive it.
I leave it to other folks to answer questions there, or not, as it happens.

I don't really wish to change off of mailing list usage.
On the other hand, if one of you wants to set up and manage a bug tracker, then let's
discuss.

@subsection Release management
@command{gawk} use X.Y.Z versioning.
The X is the major version, currently 5.
It changes rarely.
The Y is the minor version, changing whenever there are enough new features
to make it worth doing a major release.
The Z is the patch level.
Those patch releases happen 2-3 times a year.

Releases used to be more frequent, but I have purposely slowed down, trying
to avoid adding new features or making major changes.

You can look at the @file{NEWS*} files to get a sense of the kinds of things
that change between releases.
You can skim the @file{ChangeLog*} files to see descriptions of the kinds of
changes that happen.
Most are small, incremental changes.


@section Technical Skills

Here are some the technical skills you'll need as a maintainer.
These all also apply pretty much across the board, not just to @command{gawk}.

@subsection Git skills
@command{gawk}, and just about all the other GNU projects, use Git for source
code management.
Using the @command{git} command line is @strong{strongly} recommended.
I have seen things messed up too often when relying on graphical Git tools.

The @command{gawk} workflow document provides an extremely basic introduction
to using Git.
That's a starting point, but you should also invest time in going beyond
what's presented there.

@subsection C (and maybe also C++)
I doubt this is an issue, but you should definitely have C skills.
@command{gawk} is a very large C program.

The code expects at least a C 99 capable compiler.
I try to avoid being bleeding edge in the use of C features in order to allow
for maximum portability.
In particular, the OpenVMS C compiler isn't any further advanced than that,
so as long as we want to continue to support OpenVMS we can't use the latest
features.
Personally, this doesn't bother me; I don't feel a need to use the latest
features.

It's an interesting question as to whether knowing C++ is necessary.
Moving @command{gawk} to a simple subset of C++ in order to get away from its
use of unions might not be a bad idea.
On the other hand, it's a big chunk of work and I think that most of the rest of the
@command{gawk} developers would @emph{not} be happy working in C++.
On the third hand, the parts of @command{gawk} they deal with are generally
both small and localized, and they could probably manage.

@subsection Make
You should be comfortable with @command{make} and @file{Makefile}s.
I try to keep the @file{Makefile}s portable to more than just GNU Make.
@command{make} is the foundation for the ``Autotools.''

@subsection Autoconf, Automake, libtool, GNU @command{m4}
The ``Autotools.''
You should read the manuals for each of these, at least.
I like to use the latest @emph{released} version of these tools, not whatever
may be in Git on a particular day.

These tools are what create the @command{configure} script. @command{autoconf}
uses @command{m4} macros to create @command{configure}; you should have a basic
understanding of @command{m4} also, although @command{autoconf} hides most of
it from you.

The files of interest are @file{configure.ac}, @file{extension/configure.ac},
and files in the @file{m4} directory.

@subsection Texinfo

The various manuals are all written in Texinfo, the GNU Project's markup
language.
Texinfo can be used to produce Info files, HTML and (with @TeX{}) PDF.
I don't care for Info myself, but it's the GNU standard format, so the files
are created during build for inclusion in releases.

Texinfo has grown a lot over the years.
I don't use every last feature in it.
You may want to skim @file{doc/gawk.texi} to get a feel for it and then read
some of the ``basics'' on Texinfo in the Texinfo documentation.

Texinfo is important since, if you add a user-visible feature, you will have
to document it in the manual as well.

@subsection @command{troff} @option{-man} --- semi-optional

The man pages are written in @command{troff}.
A basic knowledge of how to write man pages is helpful.
For the 5.3.0 release, I cut the
@file{gawk.1} man page down in size by a lot, but it's still very large.

The four-color reference card, @file{awkcard.pdf}, is written in
@command{troff} and @command{tbl}. That's a much harder thing to deal with. I suspect
that once I ``retire'' it can be dropped from the distribution, or marked as
unmaintained.

@subsection GDB, @command{valgrind}, and @command{gprof}

Debugging and profiling skills.  I've been using @command{gdb} for
decades, to the point where I can't really use any other debugger.
Both Vim and Emacs have excellent integration for @command{gdb}.

@command{valgrind} is invaluable for dealing with memory leak issues.

@command{gprof} is occasionally necessary if you think there's a performance
bottleneck somewhere.

@subsection The GCC Compile Farm

Google this, and get yourself an account on it.
It's a resource that I only recently started to leverage, but they have a
number of machines that it's unlikely that you will have.
So if you want to test @command{gawk} (or something else) on a particular
architecture that they have, this is one way to go.

@c third note
@node General Maintainership
@chapter General Maintainership


@section Compiler / Interpreter Skills

@cindex Levine, John

You should be generally familiar with the terminology and components of
compilers and interpreters. Things like: parsing, scanning, and symbol tables.  In
particular, @command{gawk} uses a hand-written scanner and a @command{Bison}
parser (LALR, based on YACC).  The @cite{Flex and Bison} book by John
Levine is probably a good resource if you have no background in this.

As @command{gawk} parses a program, it builds up a linked list of so-called
``byte codes,'' representing the program to be run. This is ``compiling''
and ``code generation'' of a sort, but the instructions generated are
simply structs representing operations defined by @command{gawk} itself, not
code for a real machine or for any standard virtual machine such
as the Java JVM or the .Net CLR.

Once the progam is built, @command{gawk} then interprets the byte codes, by
way of a loop around a gigantic switch statement on each type of
instruction.

The bytes codes are the basis also for the debugger and the pretty
printer.  The pretty printer interprets the byte codes to reconstitute
@command{awk} source code.  The pretty printer also handles line count
profiling.

The debugger remains something of a black box to me; I'm not sure
I can explain how it works off the top of my head.  (This is sad.)

Future notes will talk about some of these things in more detail.

@section Learn The Language

To maintain @command{gawk}, you have to know @command{awk}, as well as what
@command{gawk} provides in addition to standard @command{awk}.  There are three
major resources for learning the language and for knowing how it's supposed to
work.

@cindex Aho, Alfred
@cindex Kernighan, Brian
@cindex Weinberger, Peter
@enumerate
@item
@cite{The AWK Programming Language} by Aho, Kernighan and Weinberger.
The first edition of this book is the classic reference for @command{awk},
despite it being 35 years old.
The second edition came out in 2023, and it has been completely
rewritten to reflect the changes in BWK @command{awk}.
It is compact, but very worthwhile.

@item
The manual, @cite{GAWK: Effective AWK Programming}, by yours truly.
The manual has grown over the years, as @command{gawk} has
grown. I've done my best to make it both tutorial and reference for
both standard @command{awk} and for @command{gawk}'s capabilities.
If you want to print it out, here are the steps:

@example
cd /path/to/gawk/repo
./bootstrap.sh && ./configure && make
cd doc
make pdf
@end example

@noindent
The @file{gawk.pdf} file will be the manual. You'll need all the right @TeX{}
and Texinfo bits installed for this.
Alternatively, you can find it online in various formats at
@uref{https://www.gnu.org/software/gawk/manual/}.

Eventually y'all will be maintaining this manual, so you should
become familiar with it in any case.

You should read the other Texinfo files too, at your leisure:
@file{gawkinet.pdf} for networking with @command{gawk}, and
@file{gawkworkflow.pdf}. The workflow document is most relevant for getting started
working with the team.

@item
The POSIX standard for @command{awk}.
This can be found online at:
@uref{https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html}.

It is the official standard for @command{awk}. It's written
in more formal prose than the AK&W book or the @command{gawk} manual.  It's
handy to have a copy of this to refer to, to know what POSIX specifies
and what it doesn't.  In particular, @command{gawk} @option{--posix} is
supposed to be (mostly) POSIX compliant, meaning that all @command{gawk}
extensions are disabled.

There are a few (minor) areas where @command{gawk} @option{--posix} doesn't
follow the standard, since doing so makes it too painful to use.  See the
@file{POSIX.STD} file in the distribution (or Git).
@end enumerate


@section Test Suite Structure

The test suite is fairly formalized. For each test @var{foo}, there is at least a
@command{@var{foo}.awk} and a @file{@var{foo}.ok} file. Optionally, there may be a
@file{@var{foo}.in} file, which is input data for @file{@var{foo}.awk} to process.
@file{@var{foo}.ok} is the combined standard output and standard error from running
@command{gawk} on @command{@var{foo}.awk} (with @file{@var{foo}.in}, if relevant).

Adding new tests generally does not require actually writing the test stanzas.
Instead the @file{Makefile} process takes care of this, using an @command{awk}
script that reads @file{Makefile.am} and generates the test stanzas for any
tests that have not been written manually.

To add a new test, first be sure that @command{configure} and @command{make}
have already been run.  You should also have the ``Autotools'' installed, using
the same versions as I do.  In that case, the steps to add a test involve
editing @file{Makefile.am}:

@enumerate
@item
In the list of files for @code{EXTRA_DIST}, add @file{@var{foo}.awk} and
@file{@var{foo}.ok}. Add @file{@var{foo}.in} also if relevant. Keep this list sorted
alphabetically.

@item
Add the new test name in the relevant list:

@multitable @columnfractions .30 .70
@item @code{ARRAYDEBUG_TESTS} @tab Array debugging tests.
@item @code{BASIC_TESTS} @tab Tests for standard @command{awk} functionality.
@item @code{EXTRA_TESTS} @tab Tests we don't usually run.  (I should rethink these, maybe.)
@item @code{GAWK_EXT_TESTS} @tab Tests of @command{gawk}-specific extensions.
@item @code{INET_TESTS} @tab Networking facility tests.
@item @code{LOCALE_CHARSET_TESTS} @tab Tests whose results depend on different locales.
@item @code{MACHINE_TESTS} @tab Tests that may vary based on machine architecture.
@item @code{SHLIB_TESTS} @tab Tests of the extension facility.
@item @code{UNIX_TESTS} @tab Tests that require *nix.
@end multitable

@item
In addition, there are a number of @code{NEED_@var{xxx}} groups listing tests that
need additional @command{gawk} options, such as @option{--lint} or
@option{--debug}. The @file{Makefile} doesn't use them directly. Instead, the
@command{test/Gentests} script, which generates the test stanzas, reads
@file{Makefile.am} and processes these lists in order to generate the test
stanzas.  I recommend reviewing the @command{test/Gentests} script as well.

@item
After saving @file{Makefile.am}, just run @command{make}, and everything
will be regenerated.  This should be done in the top level directory. This will
also update @file{pc/Makefile.tst}, which is used for running the tests on
MinGW.
@end enumerate


@section Documentation

The @file{doc} directory contains all of @command{gawk}'s documentation.  This
includes manuals in Texinfo as well as man pages, and a four-color
reference card, written in @command{tbl} and @command{troff}.

The various files are:

@table @code
@item ChangeLog*
ChangeLog files tracking changes.

@item Makefile.am
@itemx Makefile.in
@file{Makefile} stuff via the Autotools.

@item README.card
@itemx ad.block
@itemx awkcard.in
@itemx cardfonts
@itemx colors
@itemx macros
@itemx no.colors
@itemx setter.outline
The files for making the reference card.

@item api-figure*.fig
@itemx api-figure*.*
@itemx array-elements.fig
@itemx array-elements.*
@itemx general-program.fig
@itemx general-program.*
@itemx lflashlight-small.xpic
@itemx lflashlight.*
@itemx process-flow.fig
@itemx process-flow.*
@itemx rflashlight-small.xpic
@itemx rflashlight.*
@itemx statist.*
Image files used in the manual, both the original
from the drawing programs (@file{.fig} and @file{.xpic}) and the
generated files.

@item awkforai.txt
A draft of an article for ACM SIGPLAN Notices on why
@command{awk} is good for AI.

@item bc_notes
@itemx implementation-notes.txt
Some implementation notes.
(I should review these.)

@item gawk.1
@itemx gawkbug.1
@itemx igawk.1
@itemx pm-gawk.1
Manual pages. @file{igawk.1} is no longer installed.

@item gawkinet.texi
@itemx gawk.texi
@itemx gawkworkflow.texi
@itemx pm-gawk.texi
@itemx texinfo.tex
The Texinfo manuals.

@cindex Colombo, Antonio
@item it/*
The Italian translation of the manual and of the man
pages@footnote{The Italian translation is maintained by Antonio Colombo
@EMAIL{azc100@@gmail.com, azc100 at gmail dot com}. He has commit access
to the repository.}.

@item wordlist*
Lists of valid words that my @i{spell}(1) doesn't think are
valid. Used for spell checking the various documentation files.
@end table

@section Portability: POSIX and Non-POSIX

There are places in @command{gawk} where it needs to find out OS-specific
information or do an OS-specific task.  These are abstracted into the following
routines:

@smallexample
os_arg_fixup(int *argcp, char ***argvp)
os_devopen(const char *name, int flag)
os_close_on_exec(int fd, const char *name, const char *what, const char *dir)
os_isdir(int fd)
os_isreadable(const awk_input_buf_t *iobuf, bool *isdir)
os_is_setuid()
os_setbinmode(int fd, int mode)
os_restore_mode(int fd)
os_isatty(int fd)
os_maybe_set_errno(void)
@end smallexample

There are three different implementations of these routines, one for
OpenVMS, one for Windows with MinGW, and one for everything else (POSIX).

In the top level directory, the @file{gawkmisc.c} file includes a subsidiary
file for each environment, depending upon the system where @command{gawk} is
being compiled.

On POSIX systems, this is @file{posix/gawkmisc.c}.
On OpenVMS, this is @file{vms/gawkmisc.vms},
and on Windows with MinGW, this is @file{pc/gawkmisc.pc}.
Each of these files are C code that define the above routines.
On some of the systems, these routines return default or error values, as
appropriate.

In this way, the mainline @command{gawk} code avoids lots of @code{#ifdef}
statements, yet the @samp{os_} prefix provides an indication that the operation
being performed is OS-specific.

If you look at the files, you will see that they are very similar to each
other.


@section GNULIB

In the support directory are files that @command{gawk} uses as essentially
``library'' or support routines. For example, the @code{regex} and @code{dfa} matchers. (Why
@command{gawk} has two regular expression matchers is another story.)
Almost all these files come from GNULIB.

Unlike most other projects that take files from GNULIB, all I do is take the
source files. I don't use the GNULIB infrastructure (@code{gnulibtoolize}, or
whatever it's called) to bring in all their @command{m4} files. The reason is
that they tend to pull in the transitive closure of everything any particular
item depends upon.  Thus the supporting infrastructure ends up overwhelming the
actual code in the project.  That's both silly and wasteful of CPU time, person
time, and disk space.

The script @file{helpers/update-support.sh} can be used to update the support
directory's files, although it assumes that the local @command{git} clone of
the GNULIB repository lives in a certain directory.

I am the only one who runs it, although we can look into making it more
general going forward.

@node Index
@unnumbered Index

@printindex cp

@bye