 @@ -1,24 +1,24 @@
-<!-- $NetBSD: bulk.xml,v 1.12 2013/12/31 19:27:06 dholland Exp $ -->
+<!-- $NetBSD: bulk.xml,v 1.13 2014/05/31 20:49:16 asau Exp $ -->
 <chapter id="bulk">
 <title>Creating binary packages for everything in pkgsrc (bulk
 builds)</title>
 <para>When you have multiple machines that should run the same packages,
 it is wasted time if they all build their packages themselves from
-source. There are two ways of getting a set of binary packages: The old
+source. There is a ways of getting a set of binary packages:
-bulk build system, or the new (as of 2007) parallel bulk build (pbulk)
+The bulk build system, or pbulk ("p" stands for "parallel).
-system. This chapter describes how to set them up so that the packages
+This chapter describes how to set it up so that the packages
 are most likely to be usable later.</para>
 <sect1 id="bulk.pre">
 <title>Think first, build later</title>
 <para>Since a bulk build takes several days or even weeks to finish, you
 should think about the setup before you start everything. Pay attention
 to at least the following points:</para>
 <itemizedlist>
 <listitem><para>If you want to upload the binary packages to
 ftp.NetBSD.org, make sure the setup complies to the requirements for binary
 @@ -72,548 +72,26 @@ temporary filesystems, others must survi
 <listitem><para>1 GB for the pkgsrc tree (read-only, remote, permanent)</para></listitem>
 <listitem><para>5 GB for <filename>LOCALBASE</filename> (read-write, local, temporary for pbulk, permanent for old-bulk)</para></listitem>
 <listitem><para>10 GB for the log files (read-write, remote, permanent)</para></listitem>
 <listitem><para>5 GB for temporary files (read-write, local, temporary)</para></listitem>
 </itemizedlist>
 </sect1>
 <sect1 id="bulk.old">
 <title>Running an old-style bulk build</title>
 <note><para>There are two ways of doing a bulk build. The old-style
 one and the new-style <quote>pbulk</quote>. The latter is the recommended
 way.</para></note>
 <sect2 id="binary.configuration">
 <title>Configuration</title>
 <!-- begin old -->
       <sect3 id="binary.bulk.build.conf">
 	<title><filename>build.conf</filename></title>
 	<para>The <filename>build.conf</filename> file is the main
 	configuration file for bulk builds. You can configure how your
 	copy of pkgsrc is kept up to date, how the distfiles are
 	downloaded, how the packages are built and how the report is
 	generated. You can find an annotated example file in
 	<filename>pkgsrc/mk/bulk/build.conf-example</filename>. To use
 	it, copy <filename>build.conf-example</filename> to
 	<filename>build.conf</filename> and edit it, following the
 	comments in that file.</para>
       </sect3>
       <sect3 id="binary.mk.conf">
 	<title>&mk.conf;</title>
 	<para>You may want to set variables in &mk.conf;.
 	Look at <filename>pkgsrc/mk/defaults/mk.conf</filename> for
 	details of the default settings. You will want to ensure that
 	<varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
 	As used in this example, <varname>SKIP_LICENSE_CHECK=yes</varname>
 	completely bypasses the license check.</para>
 <programlisting>
 PACKAGES?=      ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
 WRKOBJDIR?=     /usr/tmp/pkgsrc   # build here instead of in pkgsrc
 BSDSRCDIR=      /usr/src
 BSDXSRCDIR=     /usr/xsrc         # for x11/xservers
 OBJHOSTNAME?=   yes               # use work.`hostname`
 FAILOVER_FETCH= yes               # insist on the correct checksum
 PKG_DEVELOPER?= yes
 SKIP_LICENSE_CHECK=    yes
 </programlisting>
 	<para>Some options that are especially useful for bulk builds
 	can be found at the top lines of the file
 	<filename>mk/bulk/bsd.bulk-pkg.mk</filename>. The most useful
 	options of these are briefly described here.</para>
 	<itemizedlist>
 	  <listitem><para>If you are on a slow machine, you may want to
 	  set <varname>USE_BULK_BROKEN_CHECK</varname> to
 	  <quote>no</quote>.</para></listitem>
 	  <listitem><para>If you are doing bulk builds from a read-only
 	  copy of pkgsrc, you have to set <varname>BULKFILESDIR</varname>
 	  to the directory where all log files are created. Otherwise the
 	  log files are created in the pkgsrc directory.</para></listitem>
 	  <listitem><para>Another important variable is
 	  <varname>BULK_PREREQ</varname>, which is a list of packages that
 	  should be always available while building other
 	  packages.</para></listitem>
 	</itemizedlist>
 	<para>Some other options are scattered in the pkgsrc
 	infrastructure:</para>
 	<itemizedlist>
 	  <listitem><para><varname>ALLOW_VULNERABLE_PACKAGES</varname>
 	  should be set to <literal>yes</literal>. The purpose of the
 	  bulk builds is creating binary packages, no matter if they
 	  are vulnerable or not. Leaving this variable unset would
 	  prevent the bulk build system from even trying to build
 	  them, so possible building errors would not show
 	  up.</para></listitem>
 	  <listitem><para><varname>CHECK_FILES</varname>
 	  (<filename>pkgsrc/mk/check/check-files.mk</filename>) can be set to
 	  <quote>yes</quote> to check that the installed set of files
 	  matches the <filename>PLIST</filename>.</para></listitem>
 	  <listitem><para><varname>CHECK_INTERPRETER</varname>
 	  (<filename>pkgsrc/mk/check/check-interpreter.mk</filename>) can be set to
 	  <quote>yes</quote> to check that the installed
 	  <quote>#!</quote>-scripts will find their
 	  interpreter.</para></listitem>
 	  <listitem><para><varname>PKGSRC_RUN_TEST</varname> can be
 	  set to <quote><literal>yes</literal></quote> to run each
 	  package's self-test before installing it. Note that some
 	  packages make heavy use of <quote>good</quote> random
 	  numbers, so you need to assure that the machine on which you
 	  are doing the bulk builds is not completely idle. Otherwise
 	  some test programs will seem to hang, while they are just
 	  waiting for new random data to be
 	  available.</para></listitem>
 	</itemizedlist>
       </sect3>
       <sect3 id="pre-build.local">
 	<title><filename>pre-build.local</filename></title>
 	<para>It is possible to configure the bulk build to perform
 	certain site-specific tasks at the end of the pre-build
 	stage.  If the file
 	<filename>pre-build.local</filename> exists in
 	<filename>/usr/pkgsrc/mk/bulk</filename>, it will be executed
 	(as a &man.sh.1; script) at the end of the usual pre-build
 	stage.  An example use of
 	<filename>pre-build.local</filename> is to have the line:</para>
 	<screen>echo "I do not have enough disk space to build this pig." \
 	&gt; misc/openoffice/$BROKENF</screen>
 	<para>to prevent the system from trying to build a particular package
 	which requires nearly 3 GB of disk space.</para>
       </sect3>
     </sect2>
     <sect2 id="other-environmental-considerations">
       <title>Other environmental considerations</title>
       <para>As <filename>/usr/pkg</filename> will be completely
       deleted at the start of bulk builds, make sure your login
       shell is placed somewhere else. Either drop it into
       <filename>/usr/local/bin</filename> (and adjust your login
       shell in the passwd file), or (re-)install it via
       &man.pkg.add.1; from <filename>/etc/rc.local</filename>, so
       you can login after a reboot (remember that your current
       process won't die if the package is removed, you just can't
       start any new instances of the shell any more).  Also, if you
       use &os; earlier than 1.5, or you still want to use the pkgsrc
       version of ssh for some reason, be sure to install ssh before
       starting it from <filename>rc.local</filename>:</para>
 <programlisting>
 (cd /usr/pkgsrc/security/ssh && make bulk-install)
 if [ -f /usr/pkg/etc/rc.d/sshd ]; then
   /usr/pkg/etc/rc.d/sshd
 fi
 </programlisting>
       <para>Not doing so will result in you being not able to log in
       via ssh after the bulk build is finished or if the machine
       gets rebooted or crashes. You have been warned! :)</para>
     </sect2>
     <sect2 id="operation">
       <title>Operation</title>
       <para>Make sure you don't need any of the packages still
       installed.</para>
       <warning>
 	<para>During the bulk build, <emphasis>all packages, their
 	configuration files and some more files from
 	<filename>/var</filename>, <filename>/home</filename> and
 	possibly other locations will be removed! So don't run a bulk
 	build with privileges that might harm your
 	system.</emphasis></para>
       </warning>
       <para>Be sure to remove all other things that might
       interfere with builds, like some libs installed in
       <filename>/usr/local</filename>, etc. then become root and type:</para>
       <screen>
 &rprompt; <userinput>cd /usr/pkgsrc</userinput>
 &rprompt; <userinput>sh mk/bulk/build</userinput>
       </screen>
       <para>If for some reason your last build didn't complete (power
       failure, system panic, ...), you can continue it by
       running:</para>
       <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
       <para>At the end of the bulk build, you will get a summary via mail,
       and find build logs in the directory specified by
       <varname>FTP</varname> in the <filename>build.conf</filename>
       file.</para>
     </sect2>
     <sect2 id="what-it-does">
       <title>What it does</title>
       <para>The bulk builds consist of three steps:</para>
       <variablelist>
 	<varlistentry>
 	  <term>1. pre-build</term>
 	  <listitem>
 	    <para>The script updates your pkgsrc tree via (anon)cvs, then
 	    cleans out any broken distfiles, and removes all
 	    packages installed.</para>
 	  </listitem>
 	</varlistentry>
 	<varlistentry>
 	  <term>2. the bulk build</term>
 	  <listitem>
 	    <para>This is basically <quote>make bulk-package</quote> with
 	    an optimised order in which packages will be
 	    built. Packages that don't require other packages will
 	    be built first, and packages with many dependencies will
 	    be built later.</para>
 	  </listitem>
 	</varlistentry>
 	<varlistentry>
 	  <term>3. post-build</term>
 	  <listitem>
 	    <para>Generates a report that's placed in the directory
 	    specified in the <filename>build.conf</filename> file
 	    named <filename>broken.html</filename>, a short version
 	    of that report will also be mailed to the build's
 	    admin.</para>
 	  </listitem>
 	</varlistentry>
       </variablelist>
       <para>During the build, a list of broken packages will be compiled
       in <filename>/usr/pkgsrc/.broken</filename> (or
       <filename>.../.broken.${MACHINE}</filename> if
       <varname>OBJMACHINE</varname> is set), individual build logs
       of broken builds can be found in the package's
       directory. These files are used by the bulk-targets to mark
       broken builds to not waste time trying to rebuild them, and
       they can be used to debug these broken package builds
       later.</para>
     </sect2>
     <sect2 id="disk-space-requirements">
       <title>Disk space requirements</title>
       <para>Currently, roughly the following requirements are valid for
       NetBSD 6.99/amd64:</para>
       <itemizedlist>
 	<listitem>
 	  <para>40 GB - distfiles (NFS ok)</para>
 	</listitem>
 	<listitem>
 	  <para>30 GB - full set of all binaries (NFS ok)</para>
 	</listitem>
 	<listitem>
 	  <para>5 GB - temp space for compiling (local disk recommended)</para>
 	</listitem>
       </itemizedlist>
       <para>Note that all pkgs will be de-installed as soon as they are
       turned into a binary package, and that sources are removed,
       so there is no excessively huge demand to disk
       space. Afterwards, if the package is needed again, it will
       be installed via &man.pkg.add.1; instead of building again, so
       there are no cycles wasted by recompiling.</para>
     </sect2>
     <sect2 id="setting-up-a-sandbox">
       <title>Setting up a sandbox for chrooted builds</title>
       <para>If you don't want all the packages nuked from a machine
       (rendering it useless for anything but pkg compiling), there
       is the possibility of doing the package bulk build inside a
       chroot environment.</para>
       <para>The first step is to set up a chroot sandbox,
       e.g. <filename>/usr/sandbox</filename>.  This can be done by
       using null mounts, or manually.</para>
       <para>There is a shell script called
       <filename>mksandbox</filename> installed by the <filename
       role="pkg">pkgtools/mksandbox</filename> package, which will set
       up the sandbox environment using null mounts. It will also
       create a script called <filename>sandbox</filename> in the
       root of the sandbox environment, which will allow the null
       mounts to be activated using the <command>sandbox
       mount</command> command and deactivated using the
       <command>sandbox umount</command> command.</para>
       <para>To set up a sandbox environment by hand, after extracting all
       the sets from a &os; installation or doing a <command>make
       distribution DESTDIR=/usr/sandbox</command> in
       <filename>/usr/src/etc</filename>, be sure the following items
       are present and properly configured:</para>
       <procedure>
 	<step>
 	  <para>Kernel</para>
 	  <screen>&rprompt; <userinput>cp /netbsd /usr/sandbox</userinput></screen>
 	</step>
 	<step>
 	  <para><filename>/dev/*</filename></para>
 	  <screen>&rprompt; <userinput>cd /usr/sandbox/dev ; sh MAKEDEV all</userinput></screen>
 	</step>
 	<step>
 	  <para><filename>/etc/resolv.conf</filename> (for <filename
 	  role="pkg">security/smtpd</filename> and mail):</para>
 	  <screen>&rprompt; <userinput>cp /etc/resolv.conf /usr/sandbox/etc</userinput></screen>
 	</step>
 	<step>
 	  <para>Working(!) mail config (hostname, sendmail.cf):</para>
 	  <screen>&rprompt; <userinput>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</userinput></screen>
 	</step>
 	<step>
 	  <para><filename>/etc/localtime</filename> (for <filename
 	  role="pkg">security/smtpd</filename>):</para>
 	  <screen>&rprompt; <userinput>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</userinput></screen>
 	</step>
 	<step>
 	  <para><filename>/usr/src</filename> (system sources,
           rarely used by packages if at all:</para>
 	  <screen>&rprompt; <userinput>ln -s ../disk1/cvs .</userinput>
 	  &rprompt; <userinput>ln -s cvs/src-2.0 src</userinput></screen>
 	</step>
 	<step>
 	  <para>Create <filename>/var/db/pkg</filename> (not part of default install):</para>
 	  <screen>&rprompt; <userinput>mkdir /usr/sandbox/var/db/pkg</userinput></screen>
 	</step>
 	<step>
 	  <para>Create <filename>/usr/pkg</filename> (not part of default install):</para>
 	  <screen>&rprompt; <userinput>mkdir /usr/sandbox/usr/pkg</userinput></screen>
 	</step>
 	<step>
 	  <para>Checkout pkgsrc via cvs into
 	  <filename>/usr/sandbox/usr/pkgsrc</filename>:</para>
 	  <screen>
 &rprompt; <userinput>cd /usr/sandbox/usr</userinput>
 &rprompt; <userinput>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</userinput>
 	  </screen>
 	  <para>Do not mount/link this to the copy of your pkgsrc tree
 	  you do development in, as this will likely cause problems!</para>
 	</step>
 	<step>
 	  <para>Make
 	  <filename>/usr/sandbox/usr/pkgsrc/packages</filename> and
 	  <filename>.../distfiles</filename> point somewhere
 	  appropriate. NFS- and/or nullfs-mounts may come in handy!</para>
 	</step>
 	<step>
 	  <para>Edit &mk.conf;, see <xref linkend="binary.mk.conf"/>.</para>
 	</step>
 	<step>
 	  <para>Adjust <filename>mk/bulk/build.conf</filename> to suit your needs.</para>
 	</step>
       </procedure>
       <para>When the chroot sandbox is set up, you can start
       the build with the following steps:</para>
       <screen>
 &rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
 &rprompt; <userinput>sh mk/bulk/do-sandbox-build</userinput>
       </screen>
       <para>This will just jump inside the sandbox and start building.  At
 	the end of the build, mail will be sent with the results of
 	the build. Created binary pkgs will be in
 	<filename>/usr/sandbox/usr/pkgsrc/packages</filename>
       (wherever that points/mounts to/from).</para>
     </sect2>
     <sect2 id="building-a-partial-set">
       <title>Building a partial set of packages</title>
       <para>In addition to building a complete set of all packages in
       pkgsrc, the <filename>pkgsrc/mk/bulk/build</filename> script
       may be used to build a subset of the packages contained in
       pkgsrc.  By setting <varname>SPECIFIC_PKGS</varname>
       in &mk.conf;, the variables</para>
       <itemizedlist>
 	<listitem><para>SITE_SPECIFIC_PKGS</para></listitem>
 	<listitem><para>HOST_SPECIFIC_PKGS</para></listitem>
 	<listitem><para>GROUP_SPECIFIC_PKGS</para></listitem>
 	<listitem><para>USER_SPECIFIC_PKGS</para></listitem>
       </itemizedlist>
       <para>will define the set of packages which should be built.
       The bulk build code will also include any packages which are
       needed as dependencies for the explicitly listed packages.</para>
       <para>One use of this is to do a bulk build with
       <varname>SPECIFIC_PKGS</varname> in a chroot sandbox
       periodically to have a complete set of the binary packages
       needed for your site available without the overhead of
       building extra packages that are not needed.</para>
     </sect2>
     <sect2 id="bulk-upload">
       <title>Uploading results of a bulk build</title>
       <para>This section describes how pkgsrc developers can upload binary
       pkgs built by bulk builds to ftp.NetBSD.org.</para>
       <para>If you would like to automatically create checksum files for the
       binary packages you intend to upload, remember to set
       <varname>MKSUMS=yes</varname> in your
       <filename>mk/bulk/build.conf</filename>.</para>
       <para>If you would like to PGP sign the checksum files (highly
       recommended!), remember to set
       <varname>SIGN_AS=username@NetBSD.org</varname> in your
       <filename>mk/bulk/build.conf</filename>.  This will prompt you for
       your GPG password to sign the files before uploading everything.</para>
       <para>Then, make sure that you have <varname>RSYNC_DST</varname>
       set properly in your <filename>mk/bulk/build.conf</filename>
       file, i.e. adjust it to something like one of the following:</para>
       <screen>RSYNC_DST=ftp.NetBSD.org:/pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload</screen>
       <para>Please use appropriate values for "20xxQy" (the branch),
       "a.b.c" (the OS version) and "arch" here. If your login on ftp.NetBSD.org
       is different from your local login, write your login directly
       into the variable, e.g. my local account is "feyrer", but for my
       login "hubertf", I use:</para>
       <screen>RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload</screen>
       <para>A separate <filename>upload</filename> directory is used
       here to allow "closing" the directory during upload. To do
       so, run the following command on ftp.NetBSD.org next:</para>
       <screen>nbftp% <userinput>mkdir -p -m 750 /pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy/upload</userinput></screen>
       <para>Before uploading the binary pkgs, ssh authentication needs
       to be set up. This example shows how to set up temporary keys
       for the root account <emphasis>inside the sandbox</emphasis>
       (assuming that no keys should be present there usually):</para>
       <screen>
 &rprompt; <userinput>chroot /usr/sandbox</userinput>
 chroot-&rprompt; <userinput>rm $HOME/.ssh/id-dsa*</userinput>
 chroot-&rprompt; <userinput>ssh-keygen -t rsa</userinput>
 chroot-&rprompt; <userinput>cat $HOME/.ssh/id-rsa.pub</userinput>
       </screen>
       <para>Now take the output of <filename>id-rsa.pub</filename> and
       append it to your <filename>~/.ssh/authorized_keys</filename>
       file on ftp.NetBSD.org. You should remove the key after the
       upload is done!</para>
       <para>Next, test if your ssh connection really works:</para>
       <screen>chroot-&rprompt; <userinput>ssh ftp.NetBSD.org date</userinput> </screen>
       <para>Use "-l yourNetBSDlogin" here as appropriate!</para>
       <para>Now after all this works, you can exit the sandbox and start
       the upload:</para>
       <screen>
 chroot-&rprompt; <userinput>exit</userinput>
 &rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
 &rprompt; <userinput>sh mk/bulk/do-sandbox-upload</userinput>
       </screen>
       <para>The upload process may take quite some time. Use &man.ls.1; or
       &man.du.1; on the FTP server to monitor progress of the
       upload. The upload script will take care of not uploading
       restricted packages.</para>
       <para>After the upload has ended, first thing is to revoke ssh access:</para>
       <screen>nbftp% <userinput>vi ~/.ssh/authorized_keys</userinput>
       Gdd:x! </screen>
       <para>Use whatever is needed to remove the key you've entered
       before! Last, move the uploaded packages out of the
       <filename>upload</filename> directory to have them accessible
       to everyone:</para>
       <screen>
 nbftp% <userinput>cd /pub/pkgsrc/packages/NetBSD/arch/a.b.c-20xxQy</userinput>
 nbftp% <userinput>mv upload/* .</userinput>
 nbftp% <userinput>rmdir upload</userinput>
 nbftp% <userinput>chgrp -R netbsd .</userinput>
 nbftp% <userinput>find . -type d | xargs chmod 775</userinput>
       </screen>
 <!-- end old -->
 </sect2>
 </sect1>
 <sect1 id="bulk.pbulk">
 <title>Running a pbulk-style bulk build</title>
 <para>Running a pbulk-style bulk build works roughly as follows:</para>
 <itemizedlist>
 <listitem><para>First, build the pbulk infrastructure in a fresh pkgsrc location.</para></listitem>
 <listitem><para>Then, build each of the packages from a clean installation directory using the infrastructure.</para></listitem>
 </itemizedlist>
 <sect2 id="bulk.pbulk.prepare">
 <title>Preparation</title>