https://wiki.fai-project.org/api.php?action=feedcontributions&user=Schuldei&feedformat=atomFAIWiki - User contributions [en]2024-03-29T11:35:55ZUser contributionsMediaWiki 1.39.5https://wiki.fai-project.org/index.php?title=Development_with_quilt&diff=2799Development with quilt2008-08-10T12:29:20Z<p>Schuldei: </p>
<hr />
<div>Quilt is a patch management system that keeps its patches in a seperate subdirectory. The patches get applied at package build time if desired. <br />
<br />
== Getting started with quilt ==<br />
<br />
You need quilt to be installed on the system you use to do your development work.<br />
<br />
aptitude install quilt<br />
<br />
Then you need to set two environment variables that quilt need to work. In order to use quilt for different packages you best do that on a per-package base, like so:<br />
<br />
cd ~/src/fai-source<br />
export QUILT_PC=$(pwd)/debian/.pc<br />
export QUILT_PATCHES=$(pwd)/debian/patches<br />
<br />
This could go even into debian/rules as a manual target.<br />
<br />
== Normal Workflow ==<br />
<br />
This describes the usual steps one does again and again when working with quilt.<br />
<br />
=== Applying existing patches ===<br />
<br />
You can apply existing changesets from the stack either one by one with <br />
<br />
quilt push<br />
<br />
or apply the whole stack of changesets with<br />
<br />
quilt push -a<br />
<br />
This also happens in debian/rules before you start building the package.<br />
<br />
=== Removing patches again ===<br />
<br />
In order to get rid of all the changesets again (and start over clean) you can do <br />
<br />
quilt pop -a<br />
<br />
or force-remove patches with<br />
<br />
quilt -f pop -a <br />
<br />
=== Starting a new changeset ===<br />
<br />
(If you are the first person using quilt on a package, quilt will create the directories it needs on the fly.)<br />
<br />
If you want to start a new changeset at that level within the patch stack that you are at after a couple of pushes and pops you can type<br />
<br />
quilt new ipv6<br />
<br />
to start a changeset that does cool ipv6 stuff. Next you should specify what file you want to patch.<br />
<br />
=== Adding files to a quilt changeset ===<br />
<br />
This is necessary in order to tell quilt which files are part of a quilt changeset. <br />
<br />
quilt add bin/fai-ipv6<br />
<br />
Then you can edit that file to your hearts content. Afterwards you need to refresh the current changeset to make that change stick. You can also move around the tree and add files without the path.<br />
<br />
With <br />
<br />
quilt remove fai-ipv6<br />
<br />
you disaccosiate files from your changeset again.<br />
<br />
=== Adding a change to a changeset ===<br />
<br />
quilt refresh<br />
<br />
will add new changes to all files that you added to the current changeset. <br />
<br />
quilt refresh --strip-trailing-whitespace<br />
<br />
is usefull to clean up and get rid of trailing whitespaces in the changeset.<br />
<br />
<br />
=== Getting help ===<br />
<br />
With <br />
<br />
quilt help<br />
<br />
you get a short but usefull command overview. The manpage is helpfull, too.</div>Schuldeihttps://wiki.fai-project.org/index.php?title=Developers_information&diff=2797Developers information2008-08-10T11:24:35Z<p>Schuldei: /* documentation */</p>
<hr />
<div><br />
= coordination =<br />
* [http://faiwiki.informatik.uni-koeln.de/rails/todos/list todo list] of things most important needed to be done next: <br />
* [[etch-doc-todo | List of BUGS important for correct documentation for FAI in Etch]]<br />
* [[development plan]] - long term plan, notes and status of some bigger developments<br />
* [[people branches]] - people branches and what they're aimed at<br />
* [[UbuntuTeam | Ubuntu's FAI Team]] - FAI packages for Kubuntu/Ubuntu/... flavors<br />
<br />
= documentation =<br />
<br />
* [[fai subversion howto]] - what you need to know to develop fai within the subversion repository on alioth<br />
* [[development with quilt]] - how you can add features etc to svn trunk in a quilt patch without messing up trunk<br />
* [[testing]] - general information on FAI development testing, and documentation of [[testcases]] and a [[test plan]] for the development - who tests what and how. Eventually we want to have a system so developers can report the results of their tests, as long as we don#t have that fully automated, which will take a while.<br />
<br />
= communication =<br />
* [[linux-fai-devel mailing list info]] - netiquette, how to subscribe and use<br />
<br />
= FAI pages in the Debian Bug Tracking System(BTS) =<br />
* [http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=pkg&data=fai&archive=no&version=&dist=unstable unstable] <br />
* [http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=pkg&data=fai&archive=no&version=&dist=testing testing]<br />
* [http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=pkg&data=fai&archive=no&version=&dist=stable stable] <br />
* http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=fai;pri0=tag:svn-trunk;pri1=severity:critical,grave,serious,important,normal,minor,wishlist - display fixed in svn-trunk bugs first, then open bugs ordered by priority<br />
<br />
= FAI pages in the Debian Quality Assurance (QA) system =<br />
* [http://packages.qa.debian.org/f/fai.html source package fai]<br />
* [http://packages.qa.debian.org/f/fai-kernels.html source package fai-kernels]<br />
<br />
= Subversion repository access =<br />
* svn co svn://svn.debian.org/svn/fai/trunk fai-sources<br />
* http://svn.debian.org/wsvn/fai<br />
<br />
= other stuff =<br />
* [[FAI Wiki Administration Coordination]]<br />
* [http://changelogs.debian.net/fai Changelog file]<br />
<br />
----<br />
* TODO: fill pages linked here with content:<br />
** [[developer FAQ]] from the mailing list and irc channel</div>Schuldeihttps://wiki.fai-project.org/index.php?title=Setup-storage&diff=2722Setup-storage2008-02-03T23:17:25Z<p>Schuldei: adding smart lvm volume rease to the hook</p>
<hr />
<div>= Introduction =<br />
As setup_harddisks is not portable due to its dependency on sfdisk and the lack of support for RAID and LVM, it has been re-implemented from scratch. However, it is not yet integrated with FAI and deserves rigorous testing. <br />
<br />
To test the current implementation, keep reading. There are some TODOs left and<br />
probably a lot more will arise as soon as people actually start testing it. Before you<br />
start, just let me clarify that this is a very dangerous piece of software that<br />
has not been sufficiently tested for any cases where you want to retain any of<br />
your data. It will definitely destroy any data on any of your disks in your<br />
system. You have been warned.<br />
<br />
If you're still interested, the following steps are required next:<br />
<br />
* install libparse-recdescent-perl, liblinux-lvm-perl ('''as of 2007-11-18, this is only available in Debian unstable''', download today's version [http://http.us.debian.org/debian/pool/main/libl/liblinux-lvm-perl/liblinux-lvm-perl_0.13-1_all.deb here]), parted, lvm2 within your NFSROOT <b>and add lvm2 to one of your client classes as well</b> (it must be available on the target system, if you define LVMs in your disk_config). In FAI >= 3.2, these packages are installed into the NFSROOT by adding them to /etc/fai/NFSROOT.<br />
<br />
* grab setup-storage from the SVN: [http://svn.debian.org/wsvn/fai/people/michael/features/setup_harddisks_2/implementation/] (get setup-storage and the entire lib/ directory)<br />
<br />
* for a first shot copy the files somewhere into your NFSROOT and adapt the @INC in setup-storage to the path of the lib/ directory. Alternatively, move the files to your config space to avoid copying again in case you run make-fai-nfsroot. For the time being, it is assumed that you checked out the files to <CONFIGSPACE>/store/, that is you have<br />
<br />
<pre><br />
michael@demo[16:31]:~$ ls -R /srv/fai/config/store/<br />
/srv/fai/config/store/:<br />
lib setup-storage<br />
<br />
/srv/fai/config/store/lib:<br />
commands.pm exec.pm fstab.pm init.pm parser.pm sizes.pm volumes.pm<br />
</pre><br />
<br />
and in /srv/fai/config/store/setup-storage you need to have line 77 as follows<br />
<pre><br />
unshift @INC, "/var/lib/fai/config/store/lib";<br />
</pre><br />
<br />
* create a disk_config for some class MY_TEST_CLASS from the old one apart from (at least) the following changes:<br />
** the filesystem is now given as the 4th column, the mount options are now in column 5<br />
** there is no ;-hack anymore -- anything given after the mount options is passed on to mkfs.<filesystem><br />
** Partitions are marked as bootable using the bootable:<nr> option in the disk_config line<br />
** The other things should at least work like they did before, but there are some new things as well, like RAID support (which is surely incomplete), LVM support, mounting by label or UUID. <br />
** '''Please take a look at the [[Setup-storage#Some_examples|attached examples]] to get an idea of the new format.'''<br />
** All details of the implemented syntax and examples are given below.<br />
<br />
* create a hook partition.MY_TEST_CLASS.source as follows (make sure you don't forget the .source). In case you checked out setup-storage to some other path, please adapt the paths below.<br />
<pre><br />
#!/bin/sh<br />
<br />
# load the device mapper module for LVM support<br />
modprobe dm_mod<br />
<br />
# if you want lvm volumes, and dont want to keep them this can help you to create them cleanly.<br />
# it will erase your volumes before new ones are set up.<br />
# this is necessary since setup-storage does not handle existing <br />
# lvm volumes properly and fails if it runs into existing ones.<br />
#for i in $( pvdisplay -C --noheadings -o PV_Name ); do pvremove -y -ff $i; done<br />
#udevsettle --timeout=10<br />
<br />
<br />
debug=1 /var/lib/fai/config/setup-storage<br />
# if you really brave, you can get things written to disk instead<br />
# by added -X<br />
# debug=1 /var/lib/fai/config/setup-storage -X<br />
# now define variable for root and boot partition and boot device<br />
# this is necessary because we skip the original task<br />
. $LOGDIR/disk_var.sh<br />
# skip the original parititioning<br />
skiptask partition<br />
<br />
</pre><br />
* Give it a go (you have backed up all data, don't you?). There will be lots of debug output; if it finishes with the printout of an fstab file that is to your liking, it should have succeeded.<br />
<br />
<br />
Some further infos may also be found on [http://wiki.debian.org/DebianEdu/HowTo/FaiInstallDebianEdu].<br />
<br />
= New configuration file syntax =<br />
<br />
In the following, we present a complete EBNF description of a modified configuration file syntax, as well as some examples.<br />
<br />
<pre><br />
file ::= <lines> EOF <br />
<br />
lines ::= EOL <br />
/* empty lines or whitespace only */<br />
| <comment> EOL <br />
| <config> EOL <br />
<br />
comment ::= #.* <br />
<br />
config ::= disk_config lvm <br />
| disk_config raid<br />
| disk_config end <br />
| disk_config disk[[:digit:]]+( <option>)*<br />
| disk_config [^[:space:]]+( <option>)*<br />
/* fully qualified device-path or short form, like hda, whereby full<br />
* path is assumed to be /dev/hda */<br />
| <volume><br />
<br />
option ::= /* empty */<br />
| preserve_always:[[:digit:]]+(,[[:digit:]]+)*<br />
/* preserve partitions -- always */<br />
| preserve_reinstall:[[:digit:]]+(,[[:digit:]]+)*<br />
/* preserve partitions -- unless the system is installed for the <br />
first time */<br />
| resize:[[:digit:]]+(,[[:digit:]]+)*<br />
/* attempt to resize partitions */<br />
| disklabel:(msdos|gpt)<br />
/* write a disklabel - default is msdos */<br />
| bootable:[[:digit:]]+<br />
/* mark a partition bootable, default is / */<br />
| virtual<br />
/* do not assume the disk to be a physical device, use with xen */<br />
| fstabkey:(device|label|uuid)<br />
/* when creating the fstab, the key used for defining the device<br />
may be the device (/dev/xxx), a label given using -L, or the uuid<br />
*/ <br />
<br />
volume ::= <type> <mountpoint> <size> <filesystem> <mount_options> <fs_options><br />
| vg <name> <size><br />
/* lvm vg */<br />
<br />
type ::= primary<br />
/* for physical disks only */<br />
| logical<br />
/* for physical disks only */<br />
| raid[0156]<br />
/* raid level */<br />
| [^/[:space:]]+-[^/[:space:]]+<br />
/* lvm logical volume: vg name and lv name*/<br />
<br />
mountpoint ::= -<br />
/* do not mount */<br />
| swap<br />
/* swap space */<br />
| /[^[:space:]]*<br />
/* fully qualified path */<br />
<br />
name ::= [^/[:space:]]+<br />
/* lvm volume group name */<br />
<br />
size ::= [[:digit:]]+[kMGTP%]?(-([[:digit:]]+[kMGTP%]?)?)?(:resize)?<br />
/* size in kilo, mega (default), giga, tera or petabytes or %,<br />
* possibly given as a range; physical<br />
* partitions or lvm logical volumes only; */<br />
| -[[:digit:]]+[kMGTP%]?(:resize)?<br />
/* size in kilo, mega (default), giga, tera or petabytes or %,<br />
* given as upper limit; physical partitions<br />
* or lvm logical volumes only */<br />
| [^,:[:space:]]+(:(spare|missing))*(,[^,:[:space:]]+(:(spare|missing))*)*<br />
/* devices and options for a raid or lvm vg */<br />
<br />
mount_options ::= [^[:space:]]+<br />
<br />
filesystem ::= -<br />
| swap<br />
| [^[:space:]]<br />
/* mkfs.xxx must exist */<br />
<br />
fs_options ::= .*<br />
/* options appended to mkfs.xxx call */<br />
</pre><br />
<br />
The major differences to the prior format:<br />
<br />
* the disk_config ... line allows for the keywords lvm and raid<br />
* options may need to be appended to the disk_config line<br />
* the ";" is not used anymore, the options that were given there have now been split up<br />
** the filesystem is now an explicit parameter; note, that the order of filesystem/mount-options is the same /etc/fstab, as opposed to the previous format of disk_config<br />
** any options to mkfs.xxx may be given<br />
** the "preserveX" and "boot" options are one of the options now given on the disk_config line<br />
* support for LVM and RAID is completely new :-)<br />
* resizing partitions is supported<br />
<br />
= Some examples =<br />
<br />
<pre><br />
<br />
# Configure the device /dev/hda<br />
disk_config hda preserve:6,7 disklabel:msdos bootable:3<br />
# preserve the 6th and the 7th partition. The disklabel is msdos, which is the default<br />
# for x86. Furthermore the 3rd partition is made bootable. <br />
primary /boot 20-100 ext3 rw<br />
# create a primary partition /dev/hda1 with a size between 20 and 100 MB and mount it<br />
# read-write as /boot; it is formatted using ext3 filesystem<br />
primary swap 1000 swap sw<br />
# /dev/hda2 will be a swap space of 1000 MB<br />
primary / 12000 ext3 rw -b 2048<br />
# /dev/hda3 should be formatted using ext3 filesystem; when calling mkfs.ext3<br />
# the option "-b 2048" is appended.<br />
logical /tmp 1000 ext3 rw,nosuid<br />
# create the logical partition /dev/hda5<br />
logical /usr preserve6 ext3 rw<br />
logical /var 10%- ext3 rw<br />
# make /dev/hda7 at least 10% of the disk size<br />
logical /nobackup 0- xfs rw<br />
# use mkfs.xfs to format the partition<br />
<br />
</pre><br />
<br />
<pre><br />
<br />
# Configure the virtual device /dev/sda (for, e.g., Xen setups), all<br />
# partitions are primary and sizes are ignored (so one could as well specify<br />
# anything other than 0)<br />
disk_config sda virtual<br />
primary /boot 0 ext3 rw<br />
primary / 0 ext3 rw<br />
primary /tmp 0 ext3 rw<br />
primary /usr 0 ext3 rw<br />
primary /var 0 ext3 rw<br />
<br />
# resizing partitions is possible by appending :resize to any given size<br />
disk_config /dev/scsi/host0/bus0/target1/lun0 <br />
primary / 3000-6000:resize ext3 rw<br />
# resize to any value between 3000 and 6000 MB, as limited by the disk size,<br />
# the desired sizes of the other partitions, and the data on the partition<br />
primary /tmp 1000 ext3 rw<br />
<br />
</pre><br />
<br />
<pre><br />
<br />
# Create a softRAID<br />
disk_config raid<br />
raid1 / sda1,sdd1 ext2 rw,errors=remount-ro<br />
# create a RAID-1 on /dev/sda1 and /dev/sdd1, format using mkfs.ext2 and mount<br />
# it as /<br />
raid0 - disk2.2,sdc1,sde1:spare:missing ext2 default<br />
# create a RAID-0 on the second partition of the second disk, /dev/sdc1, and<br />
# /dev/sde1 as a spare partition, which (may?) me missing<br />
<br />
<br />
</pre><br />
<br />
<pre><br />
# Simple LVM example<br />
disk_config sda bootable:1<br />
primary /boot 500 ext3 rw<br />
primary - 4096- - -<br />
<br />
disk_config lvm<br />
vg my_pv sda2<br />
my_pv-_swap swap 2048 swap sw<br />
my_pv-_root / 2048 ext3 rw<br />
</pre><br />
<br />
A pretty complex setup looks as follows:<br />
<pre><br />
disk_config sda<br />
<br />
primary - 256 - - <br />
primary swap 1024 swap sw<br />
primary - 0- - -<br />
<br />
disk_config sdb<br />
primary - 0- - -<br />
<br />
disk_config sdc<br />
primary - 0- - -<br />
<br />
disk_config sdd<br />
primary - 256 - -<br />
primary - 1024 - -<br />
primary - 0- - -<br />
<br />
disk_config sde<br />
primary - 0- - -<br />
<br />
disk_config sdf<br />
primary - 0- - -<br />
<br />
disk_config raid<br />
<br />
raid1 / sda1,sdd1 ext2 rw,errors=remount-ro <br />
raid1 swap sda2,sdd2 swap rw <br />
raid1 - sda3,sdd3 ext2 default <br />
<br />
raid0 - sdb1,sde1 ext2 default<br />
raid0 - sdc1,sdf1 ext2 default<br />
<br />
# config the LVM<br />
disk_config lvm<br />
vg my_pv md2,md3 <br />
my_pv-_usr /usr 2048 ext3 rw -O dir_index,resize_inode<br />
my_pv-_var /var 600 ext3 rw -O dir_index,resize_inode<br />
my_pv-_e_h /export/home 10240 reiser rw,notail<br />
my_pv-_e_s /export/sites 2048 reiser rw,notail<br />
my_pv-_v /vservers 2048 ext3 rw <br />
</pre><br />
<br />
= Implementation =<br />
The current implementation is found at [http://svn.debian.org/wsvn/fai/people/michael/features/setup_harddisks_2/implementation/] and has still some TODOs left:<br />
<br />
== Missing features and regressions ==<br />
<br />
* Missing features<br />
** lazyformat<br />
** some auto mode (something like auto:server, auto:desktop?) might be desirable<br />
** man page<br />
** crypto support would be nice to have; we could implement it by an additional (optional) argument to the mount point specification. A nice tutorial on crypt setups is found at [http://www.saout.de/tikiwiki/tiki-index.php?page=EncryptedDeviceUsingLUKS]<br />
** there needs to be a proper order of events. you need to tear down lvm, then raid and finally partitions and build them up the other way around. Right now things are just sorted in the disk, raid, lvm order and both teardown and buildup take place right after each other within each section. <br />
** it should be possible to preserve logical volumes, etc. using preserve instead of giving a size<br />
** once everything is done, some checks should be performed, e.g., test that all partitions were created<br />
** Disks must be selectable by their ID or the like [http://www.mail-archive.com/linux-fai-devel@uni-koeln.de/msg00023.html]<br />
** Having /boot on a SW-RAID or LVM should be tested, I think it doesn't work<br />
** implement disklabels other than msdos and gpt<br />
<br />
* Implementation details<br />
** The code of create_volume_group for the case of an existing vg must be thoroughly tested and probably improved<br />
** mdadm --misc --zero-superblock /dev/hdx may be necessary<br />
** preserve must retain the flags (bootable, etc.)<br />
** LVMs require resizing the filesystem<br />
** the RAID commands are surely incomplete and lack any management of unanticipated situations<br />
** try to get libparted-swig-perl and use that one instead of the manual parsing<br />
** volume groups may need to be cleaned, which could require some magic, see partman [http://svn.debian.org/wsvn/d-i/trunk/packages/partman/partman-auto/auto-shared.sh?op=file&rev=0&sc=0], dm_wipe_lvm<br />
** evaluate the return code of external commands in exec.pm<br />
** more error messages must be caught by exec.pm<br />
<br />
== Changes required for integration with main-line FAI ==<br />
* libparse-recdescent-perl and liblinux-lvm-perl must get installed in nfsroot (add Depends: to fai-nfsroot package)<br />
* closes #380629, #330915, #277045, #356862, #416633<br />
* no-bug: #364763<br />
<br />
== Documentation of internal data structures ==<br />
<br />
The hash of all configurations specified in the disk_config file<br />
<pre><br />
PHY_<DEVICE><br />
virtual (0|1)<br />
disklabel STRING<br />
bootable -1..n<br />
partitions<br />
<1..n><br />
size<br />
extended (0|1)<br />
preserve (0|1)<br />
resize (0|1)<br />
range<br />
eff_size<br />
number 1..n<br />
maps_to_existing 1..n<br />
start_byte<br />
end_byte<br />
mountpoint<br />
mount_options<br />
filesystem<br />
fs_options<br />
label<br />
VG_<NAME><br />
devices<br />
estimated_size<br />
volumes<br />
<logical-volume-name><br />
size<br />
preserve (0|1)<br />
resize (0|1)<br />
range<br />
eff_size<br />
mountpoint<br />
mount_options<br />
filesystem<br />
fs_options<br />
label<br />
RAID<br />
volumes<br />
<0..n><br />
mode<br />
devices<br />
/dev/<device-name><br />
options<br />
spare (0|1)<br />
missing (0|1)<br />
mountpoint<br />
mount_options<br />
filesystem<br />
fs_options<br />
label<br />
</pre><br />
<br />
The current disk configuration<br />
<pre><br />
<DEVICE><br />
bios_cylinders<br />
bios_heads<br />
bios_sectors_per_track<br />
sector_size<br />
disklabel<br />
begin_byte<br />
end_byte<br />
partitions<br />
<1..n><br />
begin_byte<br />
end_byte<br />
count_byte<br />
is_extended<br />
filesystem<br />
</pre><br />
<br />
The current LVM configuration<br />
<pre><br />
<VG><br />
physical_volumes<br />
size<br />
volumes<br />
<lv-name><br />
size<br />
</pre><br />
<br />
The current RAID configuration<br />
<pre><br />
<0..n><br />
devices<br />
mode<br />
</pre><br />
<br />
= Other implementations for RAID and LVM = <br />
<br />
Meanwhile you might also want to look at<br />
<br />
* [[User:Svamberg/Software RAID|Software RAID hook]] with mdadm<br />
* [[User:Gervai/Raid and LVM hook|SWRaid and LVM hook]] (with mdadm)<br />
* [http://www.informatik.uni-koeln.de/fai/download/developers/setup_harddisks patched version of setup_harddisks]: Code refactoring, LVM/RAID support and POD manual page: Copyright (c) 2003, Sam Vilain ''(not updated with recent changes in FAI)''<br />
* [http://www.linuxma.com/fai_raid/ LVM & RAID]</div>Schuldeihttps://wiki.fai-project.org/index.php?title=Setup-storage&diff=2570Setup-storage2007-10-19T20:59:26Z<p>Schuldei: /* Implementation */</p>
<hr />
<div>= Introduction =<br />
As setup_harddisks is not portable due to its dependency on sfdisk and the lack of support for RAID and LVM, it has been re-implemented from scratch. However, it is not yet integrated with FAI and deserves rigorous testing. <br />
<br />
To test my current implementation, keep reading. There are some TODOs left and<br />
probably a lot more will arise as soon as people actually start testing it. Before you<br />
start, just let me clarify that this is a very dangerous piece of software that<br />
has not been sufficiently tested for any cases where you want to retain any of<br />
your data. It will definitely destroy any data on any of your disks in your<br />
system. You have been warned.<br />
<br />
If you're still interested, the following steps are required next:<br />
<br />
* install libparse-recdescent-perl, parted, lvm2 within your NFSROOT <b>and add lvm2 to FAIBASE or DEFAULT as well</b> (it must be available on the target system, if you define LVMs in your disk_config)<br />
<br />
* grab setup_harddisks_2 from the SVN: [http://svn.debian.org/wsvn/fai/people/michael/features/setup_harddisks_2/implementation/] (get all the shdd2* files)<br />
<br />
* For a first shot copy the files somewhere into your NFSROOT<br />
<br />
* create a disk_config for <CLASS> from the old one apart from (at least) the following changes:<br />
** the filesystem is now given as the 4th column, the mount options are now in column 5<br />
** there is no ;-hack anymore -- anything given after the mount options is passed on to mkfs.<filesystem><br />
** Partitions are marked as bootable using the bootable:<nr> option in the disk_config line<br />
** The other things should at least work like they did before, but there are some new things as well, like RAID support (which is surely incomplete), LVM support, mounting by label or UUID. Please take a look at the attached examples to get an idea of the new format.<br />
All details of the implemented syntax and examples are given below.<br />
<br />
* create a hook partition.<CLASS>.source as follows (make sure you don't forget the .source)<br />
<pre><br />
#!/bin/sh<br />
<br />
# load the device mapper module for LVM support<br />
modprobe dm_mod<br />
<br />
cd <somewhere><br />
debug=1 ./shdd2<br />
# if you really brave, you can get things written to disk as well<br />
debug=1 ./shdd2 -X<br />
# now define variable for root and boot partition and boot device<br />
# this is necessary because we skip the original task<br />
. $LOGDIR/disk_var.sh<br />
# skip the original parititioning<br />
skiptask partition<br />
<br />
</pre><br />
* Give it a go and hope... There will be lots of debug output; if it finishes with the printout of an fstab file that is to your liking, it should have succeeded.<br />
<br />
<br />
Some further infos may also be found on [http://wiki.debian.org/DebianEdu/HowTo/FaiInstallDebianEdu].<br />
<br />
= New configuration file syntax =<br />
<br />
In the following, we present a complete EBNF description of a modified configuration file syntax, as well as some examples.<br />
<br />
<pre><br />
file ::= <lines> EOF<br />
<br />
lines ::= EOL<br />
/* empty lines or whitespace only */<br />
| <comment> EOL<br />
| <config> EOL<br />
<br />
comment ::= #.*<br />
<br />
config ::= disk_config lvm<br />
| disk_config raid<br />
| disk_config end<br />
| disk_config disk[[:digit:]]+( <option>)*<br />
| disk_config [^[:space:]]+( <option>)*<br />
/* fully qualified device-path or short form, like hda, whereby full<br />
* path is assumed to be /dev/hda */<br />
| <volume><br />
<br />
option ::= /* empty */<br />
| preserve:[[:digit:]]+(,[[:digit:]]+)*<br />
/* preserve partitions */<br />
| resize:[[:digit:]]+(,[[:digit:]]+)*<br />
/* attempt to resize partitions */<br />
| disklabel:(msdos|gpt)<br />
/* write a disklabel - default is msdos */<br />
| bootable:[[:digit:]]+<br />
/* mark a partition bootable, default is / */<br />
| virtual<br />
/* do not assume the disk to be a physical device, use with xen */<br />
| fstabkey:(device|label|uuid)<br />
/* when creating the fstab, the key used for defining the device<br />
may be the device (/dev/xxx), a label given using -L, or the uuid<br />
*/<br />
<br />
volume ::= <type> <mountpoint> <size> <filesystem> <mount_options> <fs_options><br />
| vg <name> <size><br />
/* lvm vg */<br />
<br />
type ::= primary<br />
/* for physical disks only */<br />
| logical<br />
/* for physical disks only */<br />
| raid[0156]<br />
/* raid level */<br />
| [^/[:space:]]+-[^/[:space:]]+<br />
/* lvm logical volume: vg name and lv name*/<br />
<br />
mountpoint ::= -<br />
/* do not mount */<br />
| swap<br />
/* swap space */<br />
| /[^[:space:]]*<br />
/* fully qualified path */<br />
<br />
name ::= [^/[:space:]]+<br />
/* lvm volume group name */<br />
<br />
size ::= [[:digit:]]+[kMGTP%]?(-([[:digit:]]+[kMGTP%]?)?)?(:resize)?<br />
/* size in kilo, mega (default), giga, tera or petabytes or %,<br />
* possibly given as a range; physical<br />
* partitions or lvm logical volumes only; */<br />
| -[[:digit:]]+[kMGTP%]?(:resize)?<br />
/* size in kilo, mega (default), giga, tera or petabytes or %,<br />
* given as upper limit; physical partitions<br />
* or lvm logical volumes only */<br />
| preserve[[:digit:]]+<br />
/* do not modify this partition */<br />
| [^,:[:space:]]+(:(spare|missing))*(,[^,:[:space:]]+(:(spare|missing))*)*<br />
/* devices and options for a raid or lvm vg */<br />
<br />
mount_options ::= [^[:space:]]+<br />
<br />
filesystem ::= -<br />
| swap<br />
| [^[:space:]]<br />
/* mkfs.xxx must exist */<br />
<br />
fs_options ::= .*<br />
/* options appended to mkfs.xxx call */<br />
<br />
<br />
<br />
</pre><br />
<br />
The major differences to the prior format:<br />
<br />
* the disk_config ... line allows for the keywords lvm and raid<br />
* options may be appended to the disk_config line as well, but this may be subject to change<br />
* the ";" is not used anymore, the options that were given there have now been split up<br />
** the filesystem is now an explicit parameter; note, that the order of filesystem/mount-options is the same /etc/fstab, as opposed to the previous format of disk_config<br />
** any options to mkfs.xxx may be given<br />
** the "boot"-option is one of the options now given on the disk_config line<br />
* support for LVM and RAID is completely new :-)<br />
* resizing partitions is supported<br />
<br />
= Some examples =<br />
<br />
<pre><br />
<br />
# Configure the device /dev/hda<br />
disk_config hda preserve:6,7 disklabel:msdos bootable:3<br />
# preserve the 6th and the 7th partition. Alternatively, <br />
# one could say preserve7 below. The disklabel is msdos, which is the default<br />
# for x86. Furthermore the 3rd partition is made bootable, which would have<br />
# happened as well by default as it is mounted as /<br />
primary /boot 20-100 ext3 rw<br />
# create a primary partition /dev/hda1 with a size between 20 and 100 MB and mount it<br />
# read-write as /boot; it is formatted using ext3 filesystem<br />
primary swap 1000 swap sw<br />
# /dev/hda2 will be a swap space of 1000 MB<br />
primary / 12000 ext3 rw -b 2048<br />
# /dev/hda3 should be formatted using ext3 filesystem; when calling mkfs.ext3<br />
# the option "-b 2048" is appended.<br />
logical /tmp 1000 ext3 rw,nosuid<br />
# create the logical partition /dev/hda5<br />
logical /usr preserve6 ext3 rw<br />
logical /var 10%- ext3 rw<br />
# make /dev/hda7 at least 10% of the disk size<br />
logical /nobackup 0- xfs rw<br />
# use mkfs.xfs to format the partition<br />
<br />
</pre><br />
<br />
<pre><br />
<br />
# Configure the virtual device /dev/sda (for, e.g., Xen setups), all<br />
# partitions are primary and sizes are ignored (so one could as well specify<br />
# anything other than 0)<br />
disk_config sda virtual<br />
primary /boot 0 ext3 rw<br />
primary / 0 ext3 rw<br />
primary /tmp 0 ext3 rw<br />
primary /usr 0 ext3 rw<br />
primary /var 0 ext3 rw<br />
<br />
# resizing partitions is possible by appending :resize to any given size<br />
disk_config /dev/scsi/host0/bus0/target1/lun0 <br />
primary / 3000-6000:resize ext3 rw<br />
# resize to any value between 3000 and 6000 MB, as limited by the disk size,<br />
# the desired sizes of the other partitions, and the data on the partition<br />
primary /tmp 1000 ext3 rw<br />
<br />
</pre><br />
<br />
<pre><br />
<br />
# Create a softRAID<br />
disk_config raid<br />
raid1 / sda1,sdd1 ext2 rw,errors=remount-ro<br />
# create a RAID-1 on /dev/sda1 and /dev/sdd1, format using mkfs.ext2 and mount<br />
# it as /<br />
raid0 - disk2.2,sdc1,sde1:spare:missing ext2 default<br />
# create a RAID-0 on the second partition of the second disk, /dev/sdc1, and<br />
# /dev/sde1 as a spare partition, which (may?) me missing<br />
<br />
<br />
</pre><br />
<br />
A pretty complex setup looks as follows:<br />
<pre><br />
disk_config sda<br />
<br />
primary - 256 - - <br />
primary swap 1024 swap sw<br />
primary - 0- - -<br />
<br />
disk_config sdb<br />
primary - 0- - -<br />
<br />
disk_config sdc<br />
primary - 0- - -<br />
<br />
disk_config sdd<br />
primary - 256 - -<br />
primary - 1024 - -<br />
primary - 0- - -<br />
<br />
disk_config sde<br />
primary - 0- - -<br />
<br />
disk_config sdf<br />
primary - 0- - -<br />
<br />
disk_config raid<br />
<br />
raid1 / sda1,sdd1 ext2 rw,errors=remount-ro <br />
raid1 swap sda2,sdd2 swap rw <br />
raid1 - sda3,sdd3 ext2 default <br />
<br />
raid0 - sdb1,sde1 ext2 default<br />
raid0 - sdc1,sdf1 ext2 default<br />
<br />
# config the LVM<br />
disk_config lvm<br />
vg my_pv md2,md3 <br />
my_pv-_usr /usr 2048 ext3 rw -O dir_index,resize_inode<br />
my_pv-_var /var 600 ext3 rw -O dir_index,resize_inode<br />
my_pv-_e_h /export/home 10240 reiser rw,notail<br />
my_pv-_e_s /export/sites 2048 reiser rw,notail<br />
my_pv-_v /vservers 2048 ext3 rw <br />
</pre><br />
<br />
= Implementation =<br />
The current implementation is found at [http://svn.debian.org/wsvn/fai/people/michael/features/setup_harddisks_2/implementation/] and has still some TODOs left:<br />
<br />
* Missing features<br />
** some auto mode (something like auto:server, auto:desktop?) might be desirable<br />
** no man page<br />
** crypto support would be nice to have; we could implement it by an additional (optional) argument to the mount point specification. A nice tutorial on crypt setups is found at [http://www.saout.de/tikiwiki/tiki-index.php?page=EncryptedDeviceUsingLUKS]<br />
** does preserve work the way lazyformat used to work?<br />
** there needs to be a proper order of events. you need to tear down lvm, then raid and finally partitions and build them up the other way around. Right now things are just sorted in the disk, raid, lvm order and both teardown and buildup take place right after each other within each section. <br />
* Implementation details<br />
** resize should imply resizing the filesystem as well (unless this is done by parted already, needs to be checked)<br />
** LVMs definitely require resizing the filesystem<br />
** how to detect old-style config files? migration strategies?<br />
** implement disklabels other than msdos and gpt<br />
** the RAID commands are surely incomplete and lack any management of unanticipated situations<br />
** try to get libparted-swig-perl and use that one instead of the manual parsing<br />
** (done) setup_harddisks_2 writes the UUIDs or labels to fstab, but grub's menu.lst must be adapted as well<br />
** volume groups may need to be cleaned, which could require some magic, see partman [http://svn.debian.org/wsvn/d-i/trunk/packages/partman/partman-auto/auto-shared.sh?op=file&rev=0&sc=0], dm_wipe_lvm<br />
** more error messages must be caught by shdd2-exec, e.g.,<br />
<pre><br />
(CMD) parted -s /dev/sdc unit TiB print 1> /tmp/xkxZndMHZ7 2> /tmp/qaYSzz9BLr<br />
(STDERR) No Implementation: Partition 1 isn't aligned to cylinder boundaries. This is still unsupported.<br />
</pre><br />
** create a proper perl namespace<br />
** chop the long functions into many smaller ones<br />
** try to use modules like LINUX::Lvm for lvm handling. Unfortunatly there seems to be no raid or partitioning module.<br />
<br />
* Ideas that came up on IRC<br />
** it should be possible to preserve logical volumes, etc. using preserve instead of giving a size<br />
** (done) volume groups and RAID configs should refer to disk1.2 etc.<br />
** once everything is done, some checks should be performed, e.g., test that all paritions were created<br />
** (done) test and use udevsettle instead of sleep commands<br />
** Disks must be selectable by their ID or the like [http://www.mail-archive.com/linux-fai-devel@uni-koeln.de/msg00023.html]<br />
<br />
= Other implementations for RAID and LVM = <br />
<br />
Meanwhile you might also want to look at<br />
<br />
* [[User:Svamberg/Software RAID|Software RAID hook]] with mdadm<br />
* [[User:Gervai/Raid and LVM hook|SWRaid and LVM hook]] (with mdadm)<br />
* [http://www.informatik.uni-koeln.de/fai/download/developers/setup_harddisks patched version of setup_harddisks]: Code refactoring, LVM/RAID support and POD manual page: Copyright (c) 2003, Sam Vilain ''(not updated with recent changes in FAI)''<br />
* [http://www.linuxma.com/fai_raid/ LVM & RAID]</div>Schuldeihttps://wiki.fai-project.org/index.php?title=MAC_based_Booting_PXE&diff=1046MAC based Booting PXE2007-03-27T17:10:57Z<p>Schuldei: /* MAC Based booting */</p>
<hr />
<div>= MAC Based booting =<br />
<br />
Installing with PXE and FAI MAC Address based.<br />
<br />
With this hooks and some small changes in dhcpd.conf (based on FAI-CD) every computer that boots fai gets a MAC based config file installed in the pxelinux.cfg/ dir. May be in the future a GUI can use/change this information. <br />
<br />
== File dhcpd.conf ==<br />
<pre><br />
....<br />
# deny unknown-clients<br />
....<br />
subnet 192.168.1.0 netmask 255.255.255.0<br />
{<br />
range 192.168.1.200 192.168.1.249;<br />
....<br />
</pre><br />
<br />
If you press <shift> key during pxe booting you can get a prompt. <br />
<br />
Possible commands are: <br />
mkboot HOST=local <br />
mkboot HOST=remove <br />
.....<br />
mkboot HOST=demohost<br />
.....<br />
<br />
== Hook mkboot ==<br />
<br />
<pre><br />
#! /bin/bash <br />
# hook mkboot for fai.<br />
# If you start this script on a fai client it copy, create or delete a pxe bootfile. It use the ethernet mac<br />
# address of the fai client. <br />
<br />
cd /tmp <br />
TFTPCONFDIR="/boot/fai/pxelinux.cfg"<br />
SSHUSER=fai<br />
TFTPSERVER=faiserver<br />
<br />
HOSTNAME="default"<br />
if [ "$HOST" != "" ]; then HOSTNAME=$HOST; fi ;<br />
BOOTMAC=`/sbin/ifconfig "$netdevices_up" |grep HWaddr|cut -d " " -f11|tr ':' '-'|tr '[:upper:]' '[:lower:]'`<br />
BOOTMAC="01-"$BOOTMAC<br />
echo Next Boot will boot or install: $HOSTNAME on $BOOTMAC.<br />
echo "!!! OK - Press ctrl-c to interrupt FAI and to get a shell"<br />
<br />
case $HOSTNAME in<br />
disable)<br />
ssh $SSHUSER@$TFTPSERVER "mv $TFTPCONFDIR/$BOOTMAC $TFTPCONFDIR/$BOOTMAC\.disable" ;;<br />
local)<br />
ssh $SSHUSER@$TFTPSERVER "mv $TFTPCONFDIR/$BOOTMAC $TFTPCONFDIR/$BOOTMAC\.disable ; cp $TFTPCONFDIR/local $TFTPCONFDIR/$BOOTMAC " ;;<br />
enable)<br />
ssh $SSHUSER@$TFTPSERVER "mv $TFTPCONFDIR/$BOOTMAC\.disable $TFTPCONFDIR/$BOOTMAC" ;;<br />
remove)<br />
ssh $SSHUSER@$TFTPSERVER "rm $TFTPCONFDIR/$BOOTMAC" ;;<br />
*)<br />
ssh $SSHUSER@$TFTPSERVER "cp $TFTPCONFDIR/$HOSTNAME $TFTPCONFDIR/$BOOTMAC" ;;<br />
esac<br />
<br />
task_faiend<br />
<br />
# End of mkboot<br />
</pre><br />
<br />
== Hook chboot.default ==<br />
<br />
<pre> <br />
#! /bin/sh <br />
# hook for fai. Set boot to local <br />
set -xv<br />
<br />
SCRIPTNAME=`basename $0`<br />
skiptask=`echo $SCRIPTNAME | sed "s/\..*$//"`<br />
<br />
cd /tmp <br />
TFTPCONFDIR="/boot/fai/pxelinux.cfg"<br />
SSHUSER=fai<br />
TFTPSERVER=faiserver<br />
<br />
HOSTNAME="local"<br />
<br />
BOOTMAC=`/sbin/ifconfig "$netdevices_up" |grep HWaddr|cut -d " " -f11|tr ':' '-'|tr '[:upper:]' '[:lower:]'`<br />
BOOTMAC="01-"$BOOTMAC<br />
echo Next Boot will boot or install: $HOSTNAME on $BOOTMAC.<br />
<br />
case $HOSTNAME in<br />
local)<br />
ssh $SSHUSER@$TFTPSERVER "mv $TFTPCONFDIR/$BOOTMAC $TFTPCONFDIR/$BOOTMAC\.disable ; cp $TFTPCONFDIR/local $TFTPCONFDIR/$BOOTMAC " ;;<br />
*)<br />
ssh $SSHUSER@$TFTPSERVER "cp $TFTPCONFDIR/$HOSTNAME $TFTPCONFDIR/$BOOTMAC" ;;<br />
esac<br />
<br />
skiptask $skiptask <br />
<br />
# End of hook chboot.DEFAULT<br />
</pre><br />
<br />
== File: /boot/fai/pxelinux.cfg/default ==<br />
<pre><br />
default sysinfo<br />
<br />
label sysinfo<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=mkboot HOST=sysinfo root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt,reboot<br />
<br />
label mkboot<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=mkboot root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt,reboot<br />
</pre><br />
<br />
== File: /boot/fai/pxelinux.cfg/local ==<br />
<pre><br />
default local<br />
<br />
label local<br />
localboot 0<br />
<br />
label mkboot<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=mkboot root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt,reboot<br />
</pre><br />
<br />
== File: /boot/fai/pxelinux.cfg/demohost ==<br />
<pre><br />
default demohost<br />
<br />
label demohost<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=install HOST=demohost root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt<br />
<br />
label mkboot<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=mkboot root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt,reboot<br />
<br />
</pre><br />
<br />
== File: /boot/fai/pxelinux.cfg/sysinfo ==<br />
<br />
<pre><br />
default sysinfo<br />
<br />
label sysinfo<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=sysinfo root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt<br />
<br />
label mkboot<br />
kernel vmlinuz-install<br />
append ip=dhcp devfs=nomount FAI_ACTION=mkboot root=/dev/nfs nfsroot=/usr/lib/fai/nfsroot,v2,rsize=32768,wsize=32768 FAI_FLAGS=verbose,sshd,createvt,reboot<br />
</pre></div>Schuldei