| @@ -1,14 +1,14 @@ | | | @@ -1,14 +1,14 @@ |
| 1 | <!-- $NetBSD: bulk.xml,v 1.13 2014/05/31 20:49:16 asau Exp $ --> | | 1 | <!-- $NetBSD: bulk.xml,v 1.14 2014/07/15 23:41:55 asau Exp $ --> |
| 2 | | | 2 | |
| 3 | <chapter id="bulk"> | | 3 | <chapter id="bulk"> |
| 4 | <title>Creating binary packages for everything in pkgsrc (bulk | | 4 | <title>Creating binary packages for everything in pkgsrc (bulk |
| 5 | builds)</title> | | 5 | builds)</title> |
| 6 | | | 6 | |
| 7 | <para>When you have multiple machines that should run the same packages, | | 7 | <para>When you have multiple machines that should run the same packages, |
| 8 | it is wasted time if they all build their packages themselves from | | 8 | it is wasted time if they all build their packages themselves from |
| 9 | source. There is a ways of getting a set of binary packages: | | 9 | source. There is a ways of getting a set of binary packages: |
| 10 | The bulk build system, or pbulk ("p" stands for "parallel). | | 10 | The bulk build system, or pbulk ("p" stands for "parallel). |
| 11 | This chapter describes how to set it up so that the packages | | 11 | This chapter describes how to set it up so that the packages |
| 12 | are most likely to be usable later.</para> | | 12 | are most likely to be usable later.</para> |
| 13 | | | 13 | |
| 14 | <sect1 id="bulk.pre"> | | 14 | <sect1 id="bulk.pre"> |
| @@ -117,29 +117,88 @@ $ rm -rf /tmp/pbulk-bootstrap | | | @@ -117,29 +117,88 @@ $ rm -rf /tmp/pbulk-bootstrap |
| 117 | | | 117 | |
| 118 | <screen> | | 118 | <screen> |
| 119 | $ <userinput>cd pkgtools/pbulk</userinput> | | 119 | $ <userinput>cd pkgtools/pbulk</userinput> |
| 120 | $ <userinput>/usr/pbulk/bin/bmake install</userinput> | | 120 | $ <userinput>/usr/pbulk/bin/bmake install</userinput> |
| 121 | $ <userinput>rm -rf /tmp/pbulk-outer</userinput> | | 121 | $ <userinput>rm -rf /tmp/pbulk-outer</userinput> |
| 122 | </screen> | | 122 | </screen> |
| 123 | | | 123 | |
| 124 | <para>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</para> | | 124 | <para>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</para> |
| 125 | </sect2> | | 125 | </sect2> |
| 126 | | | 126 | |
| 127 | <sect2 id="bulk.pbulk.conf"> | | 127 | <sect2 id="bulk.pbulk.conf"> |
| 128 | <title>Configuration</title> | | 128 | <title>Configuration</title> |
| 129 | | | 129 | |
| 130 | <para>TODO; see pkgsrc/doc/HOWTO-pbulk for more information.</para> | | 130 | <para>To simplify configuration we provide helper script <filename>mk/pbulk/pbulk.sh</filename>.</para> |
| 131 | | | 131 | |
| 132 | <para>TODO: continue writing</para> | | 132 | <para>In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). |
| | | 133 | Configure network access to fetch distribution files. |
| | | 134 | Create user with name "pbulk".</para> |
| | | 135 | |
| | | 136 | <para>Fetch and extract pkgsrc. Use a command like one of these:</para> |
| | | 137 | |
| | | 138 | <screen> |
| | | 139 | &rprompt; <userinput>(cd /usr && ftp -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput> |
| | | 140 | &rprompt; <userinput>(cd /usr && fetch -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</userinput> |
| | | 141 | &rprompt; <userinput>(cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.netbsd.org:/cvsroot get -P pkgsrc)</userinput> |
| | | 142 | </screen> |
| | | 143 | |
| | | 144 | <para>Or any other way that fits (e.g., curl, wget).</para> |
| | | 145 | |
| | | 146 | <para>Deploy and configure pbulk tools, e.g.:</para> |
| | | 147 | |
| | | 148 | <screen> |
| | | 149 | &rprompt; <userinput>sh pbulk.sh -n # native (NetBSD)</userinput> |
| | | 150 | &rprompt; <userinput>sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment</userinput> |
| | | 151 | &rprompt; <userinput>sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build</userinput> |
| | | 152 | </screen> |
| | | 153 | |
| | | 154 | <note><para><filename>mk.conf.frag</filename> is a fragment of |
| | | 155 | <filename>mk.conf</filename> that contains settings you want to |
| | | 156 | apply to packages you build. For instance,</para> |
| | | 157 | |
| | | 158 | <programlisting> |
| | | 159 | PKG_DEVELOPER= yes # perform more checks |
| | | 160 | X11_TYPE= modular # use pkgsrc X11 |
| | | 161 | SKIP_LICENSE_CHECK= yes # accept all licences (useful when building all packages) |
| | | 162 | </programlisting> |
| | | 163 | </note> |
| | | 164 | |
| | | 165 | <para>If configured for limited list, replace the list in <filename>/usr/pbulk/etc/pbulk.list</filename> |
| | | 166 | with your list of packages one per line without empty lines or comments. E.g.:</para> |
| | | 167 | |
| | | 168 | <programlisting> |
| | | 169 | www/firefox |
| | | 170 | mail/thunderbird |
| | | 171 | misc/libreoffice4 |
| | | 172 | </programlisting> |
| | | 173 | |
| | | 174 | <para>Start it:</para> |
| | | 175 | |
| | | 176 | <screen> |
| | | 177 | &rprompt; <userinput>/usr/pbulk/bin/bulkbuild</userinput> |
| | | 178 | </screen> |
| | | 179 | |
| | | 180 | <para>After it finishes, you'll have <filename>/mnt</filename> filled with distribution files, binary packages, and reports, |
| | | 181 | plain text summary in <filename>/mnt/bulklog/meta/report.txt</filename> |
| | | 182 | </para> |
| | | 183 | |
| | | 184 | <note><para>The <filename>pbulk.sh</filename> script does not cover all possible use cases. |
| | | 185 | While being ready to run, it serves as a good starting point to understand and build more complex setups. |
| | | 186 | The script is kept small enough for better understanding.</para> |
| | | 187 | </note> |
| | | 188 | |
| | | 189 | <note><para>The <filename>pbulk.sh</filename> script supports running |
| | | 190 | unprivileged bulk build and helps configuring distributed bulk builds.</para> |
| | | 191 | </note> |
| 133 | | | 192 | |
| 134 | </sect2> | | 193 | </sect2> |
| 135 | | | 194 | |
| 136 | </sect1> | | 195 | </sect1> |
| 137 | | | 196 | |
| 138 | <sect1 id="creating-cdroms"> | | 197 | <sect1 id="creating-cdroms"> |
| 139 | <title>Creating a multiple CD-ROM packages collection</title> | | 198 | <title>Creating a multiple CD-ROM packages collection</title> |
| 140 | | | 199 | |
| 141 | <para>After your pkgsrc bulk-build has completed, you may wish to | | 200 | <para>After your pkgsrc bulk-build has completed, you may wish to |
| 142 | create a CD-ROM set of the resulting binary packages to assist | | 201 | create a CD-ROM set of the resulting binary packages to assist |
| 143 | in installing packages on other machines. The | | 202 | in installing packages on other machines. The |
| 144 | <filename role="pkg">pkgtools/cdpack</filename> package provides | | 203 | <filename role="pkg">pkgtools/cdpack</filename> package provides |
| 145 | a simple tool for creating the ISO 9660 images. | | 204 | a simple tool for creating the ISO 9660 images. |