Monikop (and Pokinom)

Installation

We assume Debian GNU/Linux here, but any distribution should work. Adapt installation instructions accordingly.

Debian Packages needed:

to run Monikop (on Rover) or Pokinom (in office): rsync, mingetty, sudo, libcurses-perl, libfile-rsync-perl;
to install from a git repository: git-core;
to run the tests: bc, time.
Both Monikop and Pokinom run on text console; you don't need anything like Gnome, KDE or even X.

Prepare Removable Disks

Put sticker labels with disk names on your removable disks.

File Systems

Create labelled file systems on the removable disks. Example (suppose a removable disk with a sticker label "disk_10" on its case is attached to /dev/sdg1):

# mke2fs -j -L disk_10 /dev/sdg1

On both Monikop's and Pokinom's host, label the system root partition. If it were on /dev/sda1/, that's e.g.:

# e2label /dev/sda1 root

On both Monikop's and Pokinom's host, label the swap partition. If it happens to be on /dev/sda5, e.g.:

# swapoff

# mkswap -L swap /dev/sda5

# swapon

Mount Points

On both Monikop's and Pokinom's host, create mount points, one for each removable disk:

# mkdir -p /media/disk_{01,02,03,04...}

# chmod a+rx /media/disk_{01,02,03,04...}

In /etc/fstab on both Monikop's and Pokinom's host, make use of the disk labels:

## System partitions ###
LABEL=root     /               ext3  defaults,errors=remount-ro 0  1
LABEL=swap     none            swap  sw                         0  0
## Removable disks
LABEL=disk_01  /media/disk_01  ext3  rw,user,auto               0  0
LABEL=disk_02  /media/disk_02  ext3  rw,user,auto               0  0
LABEL=disk_03  /media/disk_03  ext3  rw,user,auto               0  0
LABEL=disk_04  /media/disk_04  ext3  rw,user,auto               0  0
# etc.

Put each removable disk in and make it writable; e.g.:

# mount /media/disk_01

# chmod a+rwx /media/disk_01

Maintain Bootability

On both Monikop's and Pokinom's host, make sure the operating system boots actually from its system disk rather than from some of the removable ones. Change /boot/grub/menu.lst where it says # kopt=root=...:

### BEGIN AUTOMAGIC KERNELS LIST
## lines between the AUTOMAGIC KERNELS LIST markers will be modified
## by the debian update-grub script except for the default options below

## DO NOT UNCOMMENT THEM, Just edit them to your needs

## ## Start Default Options ##
## default kernel options
## default kernel options for automagic boot options
## If you want special options for specific kernels use kopt_x_y_z
## where x.y.z is kernel version. Minor versions can be omitted.
## e.g. kopt=root=/dev/hda1 ro
##      kopt_2_6_8=root=/dev/hdc1 ro
##      kopt_2_6_8_2_686=root=/dev/hdc2 ro
# kopt=root=/dev/disk/by-label/root noresume ro

and call

# update-grub

Configure Monikop and Pokinom

Create a user on both Monikop's and Pokinom's machine. For description's sake, we assume they're called m-operator.

Inside m-operator's home directory, get Monikop (and Pokinom); unpack the tarball:

$ tar -xzf monikop-<version>.tar.gz

$ mv monikop-<version> monikop

$ cd monikop

Copy monikop.config.example to monikop.config and pokinom.config.example to pokinom.config, respectively, and adapt them according to your needs. Both are perl code, so be careful and keep the punctuation in place.

`monikop.config`

# (This is -*- perl -*- code.)
##############################################################################
# Monikop's configuration file.
#
# Copy this file to `monikop.config' and adapt it to your needs.
##############################################################################

# Possible data Sources, and by what directory name to represent them in
# Destination.
# When the latter is not unique, care must be taken that all pathnames in the 
# respective Sources are unique, or files will overwrite each other in
# unpredictable ways.
%sources = (
    'data_producer1::data' => 'p1_dir',
    'data_producer2::data' => 'p2_dir',
    'data_producer3::data' => '',
    'data_producer4::data' => '',
    );

# Possible mount points of the removable disks.
@usable_mount_points = (
    '/media/disk_1',
    '/media/disk_2',
    '/media/disk_3',
    );

# Common directory (under a mount point) to put new data in.
# Must agree with Pokinom's setting.
$path_under_mount_point =
    'measuring_data'
    ;

# Directories (under any mount point) of this name will be deleted by Monikop.
# Must agree with Pokinom's setting.
$path_under_mount_point_backed_up =
    'backed_up'
    ;

# Directory name (under a mount point) while being deleted.
# Must agree with Pokinom's setting.
$path_under_mount_point_being_deleted =
    'being_deleted'
    ;

# Path and file name prefix for rsync's raw logs:
$rsync_log_prefix =
    '~/log/monikop/log.'
    ;

# Path and file name prefix for the list of successfully rsynced files:
$finished_prefix =
    '~/log/monikop/finished.'
    ; 

# Safe files are supposed to survive power cuts during write operations.
# How to suffix the name of the duplicate of a safe file:
$safe_file_backup_suffix =
    '.bak'
    ; 

# How to suffix the name of an unfinished safe file:
$safe_file_unfinished_suffix =
    '.unfinished'
    ; 

# What to do (shutdown) when F3 has been pressed:
#$key_f3_action =
#    "sudo halt -p"
#    ;

# What to do (shutdown) when F3 has been pressed (on a systemd-based system):
$key_f3_action =
    "systemctl poweroff"
    ;

# What to do (reboot) when F6 has been pressed:
#$key_f6_action =
#    "sudo reboot"
#    ;

# What to do (reboot) when F6 has been pressed (on a systemd-based system):
$key_f6_action =
    "systemctl reboot"
    ;

# Rsync's time (in seconds) to wait for a response. This is roughly the time
# Monikop needs to notice the disappearance of a Source. Must not be 0.
$rsync_timeout =
    30
    ;

# Rsync's directory (relative to mount point of removable disk) for partially
# transferred files.
# Must agree with Pokinom's setting. Make sure your payload data does not
# contain an equally-named directory.
$rsync_partial_dir_name =
    '.rsync_partial'
    ;

For Monikop, change in monikop.config at least:

%sources: Data producing Sources on Rover in one of the formats Rsync understands, together with a source-specific directory name where data of the respective Source goes. Those directory names can be equal for several Sources as long as all filenames in the payload are certain to be unique.
@usable_mount_points: Mount points (directories) you set up earlier for your removable disks.

`pokinom.config`

# (This is -*- perl -*- code.)
########################################################################
# Pokinom's configuration file.
#
# Copy this file to `pokinom.config' and adapt it to your needs.
########################################################################

# Possible mount points of the removable disks:
@usable_mount_points = (
    '/media/disk_1',
    '/media/disk_2',
    '/media/disk_3',
    );

# Directory relative to a mount point where new data resides.
# Must agree with Monikop's setting.
$path_under_mount_point =
    'measuring_data';

# Directories of this name will be deleted.
# Must agree with Monikop's setting.
$path_under_mount_point_backed_up =
    'backed_up'
    ;

# Directory name while being deleted by monikop.
# Must agree with Monikop's setting.
$path_under_mount_point_being_deleted =
    'being_deleted'
    ;

# Data Destination:
$destination =
    'big-server::incoming/NEW_DATA'
    ;

# Credentials of the rsync server on Destination. String, or 0 if not used:
$rsync_username =
    'm-operator'
    ;
$rsync_password =
    'sEcReT'
    ;

# Path and file name prefix to rsync's raw log:
$rsync_log_prefix =
    '~/log/pokinom/log.'
    ;

# Path and file name prefix to a file where a list of rsync's incompletely
# transferred files is kept:
$interrupted_prefix =
    '~/log/pokinom/interrupted.'
    ;

# Shut down when finished? (Default, can be toggled by user by pressing F9.)
# 1 = yes; 0 = stay on.
$shut_down_when_done =
    0
    ;

# What to do (shutdown) when F3 has been pressed:
#$shut_down_action =
#    "sudo halt -p"
#    ;

# What to do (shutdown) when F3 has been pressed (on a systemd-based system):
$key_f3_action =
    "systemctl poweroff"
    ;

# Rsync's directory (relative to mount point of removable disk) for partially
# transferred files.
# Must agree with Monikop's setting. Make sure your payload data does not
# contain an equally-named directory.
$rsync_partial_dir_name =
    '.rsync_partial'
    ;

For Pokinom you should edit in pokinom.config at least:

@usable_mount_points (as with monikop.config)
$destination: Data destination in one of the formats Rsync understands; cf. setup of Data Destination.
$rsync_username, $rsync_password: credentials of (and only known to) the Rsync server; cf. setup of Data Destination.

Automatic Program Start

Append to /home/m-operator/.profile (create it if necessary):

/home/m-operator/monikop/monikop

/home/m-operator/monikop/pokinom,

respectively.

If necessary, specify path to config file, e.g.

/home/m-operator/monikop/monikop /home/m-operator/monikop/monikop.config

Setup Sudo

(Not necessary on a systemd-based system.)

On both Monikop's and Pokinom's host authorise m-operator to shut down computer. Use visudo to change /etc/sudoers; add:

m-operator ALL=(ALL) NOPASSWD: /sbin/halt -p
m-operator ALL=(ALL) NOPASSWD: /sbin/reboot

Automatic Login (under systemd)

On both Monikop's and Pokinom's host, change the line in /etc/inittab that looks like

1:2345:respawn:/sbin/getty 38400 tty1

into

1:2345:respawn:/sbin/mingetty --autologin m-operator --noclear tty1

Automatic Login (under systemd)

On both Monikop's and Pokinom's host, create the file (and the containing directory) /etc/systemd/system/getty@tty1.service.d/autologin.conf:

[Service]
ExecStart=
ExecStart=-/sbin/agetty --autologin m-operator --noclear %I 38400 linux

Configure Rsync on Sources

Install package rsync.

Example for /etc/rsyncd.conf:

pid file=/var/run/rsyncd.pid
[data]
    path = /mnt/hdd_0
    use chroot = false
    lock file = /var/lock/rsyncd
    read only = yes
    list = yes
    transfer logging = false

In /etc/default/rsync, change the line

RSYNC_ENABLE = false

RSYNC_ENABLE = true

Start rsync server:

# /etc/initd/rsync start

or reboot.

On Windows, install Cygwin for x86 or amd64 including package rsync. Start Cygwin as Administrator.

Inside Cygwin, edit /etc/rsyncd.conf:

use chroot = false
strict modes = false
hosts allow = *
logfile = rsyncd.log
[data]
#   /cygdrive/e/log stands for E:\log
    path = /cygdrive/e/log
    read only = false
    transfer logging = false

Configure rsync as a service:

$ cygrunsrv --install "rsyncd" --path /usr/bin/rsync \
  --args "--daemon --no-detach" \
  --desc "Start rsync daemon for accepting incoming rsync connections" \
  --disp "Rsync Daemon" \
  --type auto

Start the rsync service (or just reboot):

$ net start rsyncd

Network Setup

Depending on the amount of data to transfer, consider putting a dedicated NIC for each Source into Monikop's machine. In this case, you should provide for non-overlapping subnets. IP-Calculator may be helpful.

Monikop

Name the Sources

Example for /etc/hosts:

127.0.0.1      localhost
192.168.200.10 data-producer1
192.168.200.20 data-producer2
192.168.200.30 data-producer3
192.168.200.50 data-producer4
192.168.178.1  monikop

Configure NICs

Example for /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# Net of smaller Sources
allow-hotplug eth1
iface eth1 inet static
   address 192.168.178.1
   netmask 255.255.255.0

# Dedicated NIC for data-producer1
allow-hotplug eth2
iface eth2 inet static
   address 192.168.200.9
   netmask 255.255.255.248

# Dedicated NIC for data_producer2
allow-hotplug eth3
iface eth3 inet static
   address 192.168.200.19
   netmask 255.255.255.248

# Dedicated NIC for data_producer3
allow-hotplug eth4
iface eth4 inet static
   address 192.168.200.29
   netmask 255.255.255.248

# Dedicated NIC for data_producer4
allow-hotplug eth5
iface eth5 inet static
   address 192.168.200.49
   netmask 255.255.255.248

Data Sources

Use /etc/hosts as with Monikop. For Windows, it's %SystemRoot%\system32\drivers\etc\hosts.

Source's NIC

Example for /etc/network/interfaces:

auto lo
iface lo inet loopback

# service (not relevant for Monikop)
allow-hotplug eth0
iface eth0 inet static
   address 192.168.178.2
   netmask 255.255.255.0

# Monikop's dedicated NIC
allow-hotplug eth1
iface eth1 inet static
   address 192.168.200.10
   netmask 255.255.255.248

For Windows, configure your network settings accordingly.

Pokinom

Pokinom's network settings don't need any special treatment. Just integrate it into the office LAN Destination is connected to.

Data Destination

Rsync Server on Destination

Install package rsync.

Adapt /etc/rsyncd.conf, e.g.:

gid = data_receiving_group
use chroot = yes
max connections = 0
pid file = /var/run/rsyncd.pid

[incoming]
        path = /mnt/./raid_0
        list = no
        comment = Pokinom only; requires authentication
        read only = no
        incoming chmod = g+r,g+w,g+X
        write only = yes
        # Pokinom's IP:
        hosts allow = 192.168.180.120
        auth users = m-operator
        secrets file = /etc/rsyncd.secrets

/etc/rsyncd.secrets contains Rsync's credentials which must correspond to settings $rsync_passwd and $rsync_username in pokinom.config:

m-operator:sEcReT

/etc/rsyncd.secrets must not be world-readable.

In /etc/default/rsync, change the line

RSYNC_ENABLE = false

RSYNC_ENABLE = true

Start rsync server:

# /etc/initd/rsync start

or reboot.

With the above, rsync puts the payload it receives into /mnt/raid_0/NEW_DATA/. ("NEW_DATA" was set with $destination in pokinom.config.)

NEW_DATA/ and everything inside belongs to user nobody and group data_receiving_group.

If on Destination you can't do without Windows, install rsync under Cygwin as described above.

Monikop (and Pokinom)

rsync between unconnected hosts

Installation

Prepare Removable Disks

File Systems

Mount Points

Maintain Bootability

Configure Monikop and Pokinom

monikop.config

pokinom.config

Automatic Program Start

Setup Sudo

Automatic Login (under systemd)

Automatic Login (under systemd)

Configure Rsync on Sources

Network Setup

Monikop

Name the Sources

Configure NICs

Data Sources

Source's NIC

Pokinom

Data Destination

Rsync Server on Destination

`monikop.config`

`pokinom.config`