 @@ -1,190 +1,249 @@
-<!-- $NetBSD: bulk.xml,v 1.13 2014/05/31 20:49:16 asau Exp $ -->
+<!-- $NetBSD: bulk.xml,v 1.14 2014/07/15 23:41:55 asau Exp $ -->
 <chapter id="bulk">
 <title>Creating binary packages for everything in pkgsrc (bulk
 builds)</title>
 <para>When you have multiple machines that should run the same packages,
 it is wasted time if they all build their packages themselves from
 source. There is a ways of getting a set of binary packages:
 The bulk build system, or pbulk ("p" stands for "parallel).
 This chapter describes how to set it up so that the packages
 are most likely to be usable later.</para>
 <sect1 id="bulk.pre">
 <title>Think first, build later</title>
 <para>Since a bulk build takes several days or even weeks to finish, you
 should think about the setup before you start everything. Pay attention
 to at least the following points:</para>
 <itemizedlist>
 <listitem><para>If you want to upload the binary packages to
 ftp.NetBSD.org, make sure the setup complies to the requirements for binary
 packages:</para>
 <itemizedlist>
 <listitem><para>To end up on ftp.NetBSD.org, the packages must be built
 by a NetBSD developer on a trusted machine (that is, where you and only
 you have root access).</para></listitem>
 <listitem><para>Packages on ftp.NetBSD.org should only be created from
 the stable branches (like 2009Q1), so that users browsing the available
 collections can see at a glance how old the packages
 are.</para></listitem>
 <listitem><para>The packages must be built as root, since some packages
 require set-uid binaries at runtime, and creating those packages as
 unprivileged user doesn't work well at the moment.</para></listitem>
 </itemizedlist>
 </listitem>
 <listitem><para>Make sure that the bulk build cannot break anything in
 your system. Most bulk builds run as root, so they should be run at least
 in a chroot environment or something even more restrictive, depending on
 what the operating system provides. There have been numerous cases where
 certain packages tried to install files outside the
 <filename>LOCALBASE</filename> or wanted to edit some files in
 <filename>/etc</filename>. Furthermore, the bulk builds install and
 deinstall packages in <filename>/usr/pkg</filename> (or whatever
 <filename>LOCALBASE</filename> is) during their operation, so be sure
 that you don't need any package during the build.</para></listitem>
 </itemizedlist>
 </sect1>
 <sect1 id="bulk.req">
 <title>Requirements of a bulk build</title>
 <para>A complete bulk build requires lots of disk space. Some of the
 disk space can be read-only, some other must be writable. Some can be on
 remote filesystems (such as NFS) and some should be local. Some can be
 temporary filesystems, others must survive a sudden reboot.</para>
 <itemizedlist>
 <listitem><para>40 GB for the distfiles (read-write, remote, temporary)</para></listitem>
 <listitem><para>30 GB for the binary packages (read-write, remote, permanent)</para></listitem>
 <listitem><para>1 GB for the pkgsrc tree (read-only, remote, permanent)</para></listitem>
 <listitem><para>5 GB for <filename>LOCALBASE</filename> (read-write, local, temporary for pbulk, permanent for old-bulk)</para></listitem>
 <listitem><para>10 GB for the log files (read-write, remote, permanent)</para></listitem>
 <listitem><para>5 GB for temporary files (read-write, local, temporary)</para></listitem>
 </itemizedlist>
 </sect1>
 <sect1 id="bulk.pbulk">
 <title>Running a pbulk-style bulk build</title>
 <para>Running a pbulk-style bulk build works roughly as follows:</para>
 <itemizedlist>
 <listitem><para>First, build the pbulk infrastructure in a fresh pkgsrc location.</para></listitem>
 <listitem><para>Then, build each of the packages from a clean installation directory using the infrastructure.</para></listitem>
 </itemizedlist>
 <sect2 id="bulk.pbulk.prepare">
 <title>Preparation</title>
 <para>First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory <filename>/usr/pbulk</filename> or <filename>$HOME/pbulk</filename> for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there.</para>
 <screen>
 $ <userinput>cd /usr/pkgsrc</userinput>
 $ <userinput>./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap</userinput>
 $ <userinput>rm -rf /tmp/pbulk-bootstrap</userinput>
 </screen>
 <para>Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file <filename>/usr/pbulk/etc/mk.conf</filename> to fit your needs. Typical things you might set now are:</para>
 <itemizedlist>
 <listitem><para><literal><varname>PKG_DEVELOPER</varname>=yes</literal>, to enable many consistency checks,</para></listitem>
 <listitem><para><literal><varname>WRKOBJDIR</varname>=/tmp/pbulk-outer</literal>, to keep <filename>/usr/pkgsrc</filename> free from any modifications,</para></listitem>
 <listitem><para><literal><varname>DISTDIR</varname>=/distfiles</literal>, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded,</para></listitem>
 <listitem><para><literal><varname>ACCEPTABLE_LICENSES</varname>+=...</literal>, to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you,</para></listitem>
 <listitem><para><literal><varname>SKIP_LICENSE_CHECK</varname>=yes</literal>, to bypass the license checks.</para></listitem>
 </itemizedlist>
 <para>Now you are ready to build the rest of the pbulk infrastructure.</para>
 <screen>
 $ <userinput>cd pkgtools/pbulk</userinput>
 $ <userinput>/usr/pbulk/bin/bmake install</userinput>
 $ <userinput>rm -rf /tmp/pbulk-outer</userinput>
 </screen>
 <para>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</para>
 </sect2>
 <sect2 id="bulk.pbulk.conf">
 <title>Configuration</title>
-<para>TODO; see pkgsrc/doc/HOWTO-pbulk for more information.</para>
+<para>To simplify configuration we provide helper script <filename>mk/pbulk/pbulk.sh</filename>.</para>
-<para>TODO: continue writing</para>
+<para>In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine).
 Configure network access to fetch distribution files.
 Create user with name "pbulk".</para>
 <para>Fetch and extract pkgsrc. Use a command like one of these:</para>
 <screen>
 &rprompt; <userinput>(cd /usr &amp;&amp; ftp -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
 &rprompt; <userinput>(cd /usr &amp;&amp; fetch -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput>
 &rprompt; <userinput>(cd /usr &amp;&amp; cvs -Q -z3 -d anoncvs@anoncvs.netbsd.org:/cvsroot get -P pkgsrc)</userinput>
 </screen>
 <para>Or any other way that fits (e.g., curl, wget).</para>
 <para>Deploy and configure pbulk tools, e.g.:</para>
 <screen>
 &rprompt; <userinput>sh pbulk.sh -n # native (NetBSD)</userinput>
 &rprompt; <userinput>sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment</userinput>
 &rprompt; <userinput>sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build</userinput>
 </screen>
 <note><para><filename>mk.conf.frag</filename> is a fragment of
 <filename>mk.conf</filename> that contains settings you want to
 apply to packages you build. For instance,</para>
 <programlisting>
 PKG_DEVELOPER=		yes		# perform more checks
 X11_TYPE=		modular		# use pkgsrc X11
 SKIP_LICENSE_CHECK=	yes		# accept all licences (useful when building all packages)
 </programlisting>
 </note>
 <para>If configured for limited list, replace the list in <filename>/usr/pbulk/etc/pbulk.list</filename>
 with your list of packages one per line without empty lines or comments. E.g.:</para>
 <programlisting>
 www/firefox
 mail/thunderbird
 misc/libreoffice4
 </programlisting>
 <para>Start it:</para>
 <screen>
 &rprompt; <userinput>/usr/pbulk/bin/bulkbuild</userinput>
 </screen>
 <para>After it finishes, you'll have <filename>/mnt</filename> filled with distribution files, binary packages, and reports,
 plain text summary in <filename>/mnt/bulklog/meta/report.txt</filename>
 </para>
 <note><para>The <filename>pbulk.sh</filename> script does not cover all possible use cases.
 While being ready to run, it serves as a good starting point to understand and build more complex setups.
 The script is kept small enough for better understanding.</para>
 </note>
 <note><para>The <filename>pbulk.sh</filename> script supports running
 unprivileged bulk build and helps configuring distributed bulk builds.</para>
 </note>
 </sect2>
 </sect1>
   <sect1 id="creating-cdroms">
     <title>Creating a multiple CD-ROM packages collection</title>
     <para>After your pkgsrc bulk-build has completed, you may wish to
     create a CD-ROM set of the resulting binary packages to assist
     in installing packages on other machines.  The
     <filename role="pkg">pkgtools/cdpack</filename> package provides
     a simple tool for creating the ISO 9660 images.
     <command>cdpack</command> arranges the packages on the CD-ROMs in a
     way that keeps all the dependencies for a given package on the same
     CD as that package.</para>
     <sect2 id="cdpack-example">
       <title>Example of cdpack</title>
       <para>Complete documentation for cdpack is found in the cdpack(1)
       man page. The following short example assumes that the binary
       packages are left in
       <filename>/usr/pkgsrc/packages/All</filename> and that
       sufficient disk space exists in <filename>/u2</filename> to
       hold the ISO 9660 images.</para>
       <screen>
 &rprompt; <userinput>mkdir /u2/images</userinput>
 &rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
 &rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput>
       </screen>
       <para>If you wish to include a common set of files
       (<filename>COPYRIGHT</filename>, <filename>README</filename>,
       etc.) on each CD in the collection, then you need to create a
       directory which contains these files. e.g.</para>
       <screen>
 &rprompt; <userinput>mkdir /tmp/common</userinput>
 &rprompt; <userinput>echo "This is a README" &gt; /tmp/common/README</userinput>
 &rprompt; <userinput>echo "Another file" &gt; /tmp/common/COPYING</userinput>
 &rprompt; <userinput>mkdir /tmp/common/bin</userinput>
 &rprompt; <userinput>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</userinput>
 &rprompt; <userinput>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</userinput>
 &rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput>
       </screen>
       <para>Now create the images:</para>
       <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
       <para>Each image will contain <filename>README</filename>,
       <filename>COPYING</filename>, and <filename>bin/myscript</filename>
       in their root directories.</para>
     </sect2>
   </sect1>
 </chapter>