Digging in to the Linux install on a NAS
In this post I'm going to describe my adventures in trying to bend an ASUSTOR AS3102T to my will. It all started with the built in webserver for personal pages blocking my ability to use the usual HTTP ports for docker images.
What is a NAS?
A NAS is a Network Attached Storage Device. These come in many forms from a variety of manufacturers from small single drive units with limited functionality, slightly larger 2-4 disk units which can run third party apps and/or container images, to large U1 industrial rack hardware. Most of them run Linux, as such they are fairly pliable if you have root or escalated privileges.
ASUSTOR Linux
The unit I'm using has a very very minimal Linux installed on it (had, I boot openSUSE on it now), and the base is backed up to a second partition. The disk used is a USB2 DOM (disk on module):
Device Start End Sectors Size Type
/dev/sdd1 2048 6143 4096 2M EFI System
/dev/sdd2 6144 505855 499712 244M Microsoft basic data
/dev/sdd3 505856 1005567 499712 244M Microsoft basic data
sdd1
is the EFI boot, it contains only a grub loader, config, and a few modules required.
sdd2
and sdd3
are clones of each other, so if one fails then grub will boot the second one.
Booting
On first boot the minimal Linux is loaded and a lighthttp
server is started which allows the user to configure disks and install the rest of the system proper.
The contents of sdd2
are:
dom2/:
total 152M
-rw-r--r-- 1 root root 134M Oct 19 00:35 builtin.tgz
-rw-r--r-- 1 root root 84 Oct 19 00:35 builtin.tgz.cksum
-rw-r--r-- 1 root root 97 Oct 19 00:35 builtin.tgz.md5sum
-rw-r--r-- 1 root root 3.9M Oct 19 00:35 bzImage
-rw-r--r-- 1 root root 78 Oct 19 00:35 bzImage.cksum
-rw-r--r-- 1 root root 93 Oct 19 00:35 bzImage.md5sum
-rw-r--r-- 1 root root 14M Oct 19 00:35 initramfs
-rw-r--r-- 1 root root 81 Oct 19 00:35 initramfs.cksum
-rw-r--r-- 1 root root 95 Oct 19 00:35 initramfs.md5sum
bzImage
and initramfs
are your usual suspects - Linux kernel and initial ramdisk. The one of interest is builtin.tgz
which we will cover soon. First let's peek in initramfs
, this can easily be extracted using Gnome Nautilus, there's no need to use a terminal for absolutely everything. It's pretty usual with a run of the mill directory layout using lib
and lib64
.
bin dev etc init lib lib64 mnt opt proc root run sbin sys tmp usr var
But what is init
? It's a script run on boot with the following contents:
#!/bin/sh
# set global environment (command, apkg path)
source /etc/script/lib/command.sh
source /etc/script/lib/apkg_path.sh
# replaces the current process image with a new process image.
exec /sbin/init
and /sbin/init
is in-fact a symlink to /bin/busybox
... So what about the scripts?
/etc/script/lib/apkg_path.sh
exports a series of globals that are used for the ASUSTOR apkg package management. command.sh
does a lot of exporting of CLI tools under AS_
prefixes such as export AS_UPTIME=/usr/bin/uptime
.
What else is in /etc/script
?
-rwxr-xr-x 1 luke users 1.2K Apr 22 2014 apkg_script_ldr.sh
-rwxr-xr-x 1 luke users 157 Nov 20 2015 dhcpd-script.sh
-rwxr-xr-x 1 luke users 214 Aug 11 2016 FU41lighttpd.sh
-rwxr-xr-x 1 luke users 874 Jan 3 2017 FU50ssh.sh
-rwxr-xr-x 1 luke users 89 Apr 17 2017 FU99login_template.sh
-rwxr-xr-x 1 luke users 517 Aug 23 2013 kernel_log_backup.sh
drwxr-xr-x 2 luke users 4.0K Oct 27 14:20 lib
-rwxr-xr-x 1 luke users 1012 Oct 9 20:37 rcHotPlug
-rwxr-xr-x 1 luke users 927 Apr 1 2014 rcK.builtinfs
-rwxr-xr-x 1 luke users 1.6K Apr 14 2015 rcK.pluginsfs
-rwxr-xr-x 1 luke users 1.6K Apr 23 2014 rcK.raminitfs
-rwxr-xr-x 1 luke users 1.1K Aug 21 2014 rcNetwork
-rwxr-xr-x 1 luke users 492 Apr 3 2013 rcOnce
-rwxr-xr-x 1 luke users 532 Oct 29 2012 rcReset
-rwxr-xr-x 1 luke users 881 Feb 6 2014 rcS.builtinfs
-rwxr-xr-x 1 luke users 1.3K Apr 14 2015 rcS.pluginsfs
-rwxr-xr-x 1 luke users 3.6K Oct 3 22:51 rcS.raminitfs
-rwxr-xr-x 1 luke users 533 May 13 2013 rcSysEvent
-rwxr-xr-x 1 luke users 997 Jul 25 2012 rcTime
-rwxr-xr-x 1 luke users 1.1K May 6 2014 rcUserChange
-rwxr-xr-x 1 luke users 370 Apr 1 2014 script_trap.sh
-rwxr-xr-x 1 luke users 173 Nov 7 2012 sysinit.sh
-rwxr-xr-x 1 luke users 210 Oct 9 20:37 T11nasmand
-rwxr-xr-x 1 luke users 1.3K Sep 11 2015 vpn.sh
-rwxr-xr-x 1 luke users 176 Oct 9 20:37 Y11nasmand
-rwxr-xr-x 1 luke users 159 Oct 9 20:37 Y12netmand
-rwxr-xr-x 1 luke users 180 Oct 9 20:37 Y13emboardmand
The ones of note here are the *mand
scripts as this correlate to the proprietary daemons. The rest of the init process is through old-school init.d
, and again the ones of note are *mand
. As you can see there are a basic set of services started each and every boot
-rwxr-xr-x 1 luke users 3.0K May 11 2017 NASsuspend
-rwxr-xr-x 1 luke users 223 Oct 9 20:37 P01emboardmand
-rwxr-xr-x 1 luke users 212 Oct 9 20:37 P02lcmd
-rwxr-xr-x 1 luke users 260 Oct 9 20:37 P99hostmand
lrwxrwxrwx 1 luke users 25 Oct 27 14:20 rcK -> /etc/script/rcK.raminitfs
lrwxrwxrwx 1 luke users 19 Oct 27 14:20 rcR -> /etc/script/rcReset
lrwxrwxrwx 1 luke users 25 Oct 27 14:20 rcS -> /etc/script/rcS.raminitfs
-rwxr-xr-x 1 luke users 510 Oct 9 20:37 S11nasmand
-rwxr-xr-x 1 luke users 856 Oct 9 20:37 S12netmand
-rwxr-xr-x 1 luke users 339 Oct 9 20:37 S14lcmd
-rwxr-xr-x 1 luke users 139 Oct 9 20:36 S15fan
-rwxr-xr-x 1 luke users 1.4K Jan 10 2012 S20urandom
-rwxr-xr-x 1 luke users 647 Oct 9 20:37 S21stormand
-rwxr-xr-x 1 luke users 525 May 25 2012 S41crond
-rwxr-xr-x 1 luke users 646 Aug 17 2017 S41lighttpd
-rwxr-xr-x 1 luke users 1.7K Oct 5 16:26 S41ssh
-rwxr-xr-x 1 luke users 350 Oct 9 20:37 S42hostmand
-rwxr-xr-x 1 luke users 277 Oct 9 20:37 S44httpredir
-rwxr-xr-x 1 luke users 972 Oct 9 20:37 S93emboardmand
-rwxr-xr-x 1 luke users 480 Nov 7 2012 S99chk_config
-rwxr-xr-x 1 luke users 324 Oct 9 20:37 S99crontab_check
-rwxr-xr-x 1 luke users 305 Oct 9 20:37 S99watchmand
A little further detail here:
S15fan
starts a proprietary fan controllerS41lighttpd
is the initial boot HTTP server which serves compiled CGI (and is actually very fast)S44httpredir
redirects ports 80 and 443 to 8000 and 8001 respectivelyS99chk_config
checks if two volumes exist and and symlinks init script directories on them to base init.
Setup
Once booted for the first time the user is led through a GUI to set up disks in raid. Part of this process is also the creation of some extra volumes: a second stage boot, swap partition, and the actual storage partition.
Device Start End Sectors Size Type
/dev/sda1 2048 524287 522240 255M Linux filesystem
/dev/sda2 524288 4718591 4194304 2G Linux RAID
/dev/sda3 4718592 8912895 4194304 2G Linux RAID
/dev/sda4 8912896 3907028991 3898116096 1.8T Linux RAID
sda1
contains raid infosda2
is mount at/volume0
-builtin.tgz
is unpacked heresda3
is swapsda4
is mounted at/volume1
The boot process is actually initramfs
-> /volume0/usr/builtin/etc/init.d/
-> /volume1/.@plugins/etc/init.d/
. The last directory .@plugins
is where user-installed packages put their init scripts eg, Docker uses this to start its daemon. With three levels it's a bit of a pain to navigate through everything, and even with just the two base ones this looks like a maintenance nightmare.
The binaries in each stage
One
Lets look at stage one, the initramfs. The /bin
contents are minimal:
bsdiff busybox iostat mpstat pidstat xz
bspatch cifsiostat lz4 nfsiostat sh
as is /sbin
, only what's required to get a very minimal Linux up and running with a shell:
badblocks e2fsck fsck.ntfs mdev.sh mkfs.msdos
blkid ethtool fsck.vfat mkdosfs mkfs.ntfs
dhclient fsck.ext2 gdisk mke2fs mkfs.vfat
dhclient-script fsck.ext3 init mkfs.ext2 parted
dmsetup fsck.ext4 ip mkfs.ext3 resize2fs
dosfsck fsck.ext4dev ldconfig mkfs.ext4 sgdisk
dosfslabel fsck.hfsplus mdadm mkfs.ext4dev tune2fs
dumpe2fs fsck.msdos mdev_init.sh mkfs.hfsplus wpa_action.sh
/usr/bin
aggregate_js.sh gpasswd rsync ssh-add tripplite_usb
bcmxcp_usb groups scp ssh-agent upsc
blazer_usb grub-editenv sftp ssh-keygen upsdrvctl
confutil iconv sftp-server ssh-keyscan usbhid-ups
dbus-daemon ldd slogin ssh-keysign wget
FU99login_template lighttpdutil snmp-ups ssh-pkcs11-helper
getent richcomm_usb ssh toolbox
Of note is confutil
which is a compiled binary whose sole task is to manage a few base configurations.
/usr/sbin
apkg ezrouterd iwlist ntpdate stormand
applog fanctrl iwpriv pppd syslog
avahi-daemon ftpbackupd iwspy pppoe upsctrl
badblockctrl fwupdate lcmctrl pppoe-connect upsd
bios_update_util groupadd lcmd pppoe-relay upsmon
buzzctrl groupdel ledctrl pppoe-setup upssched
chknasmode groupmod lighttpd pppoe-start useradd
chpasswd hdparm lighttpd-angel pppoe-status userdel
crontab_check hostmand logmand pppoe-stop usermanutil
dhcp6c httpredir logrotate raidmand usermod
dhcp6ctl ifrename mcu-update recybind volmand
dhcpserverctrl iodumpman myhttpd sftpmand watchmand
dmidecode iwconfig nasmand shutdownctrl wpa_cli
ecctrl iwevent netman sm-notify wpa_supplicant
emboardmand iwgetid netmand sshd zic
I want to call out myhttpd
here, this little fucker was hard to kill, there (was, I think they changed it after complaints) an init script in the initramfs
which would start this without fail, and its sole purpose is to redirect ports 80 and 443 to 8000 and 8001 unless the personal apache server is being used. I hated it! I ended up making a single script to run before Docker with the sole purpose of finding its PID and KILLING IT WITH FIRE (-9)! You little shitbag. DIE!
cough
So yeah, see those *mand
files? Those are the proprietary ones, there's a few others also: fanctrl
, bios_update_util
(supplied by Intel?), and ezrouterd
.
Two
/volume0/usr/builtin/bin
also contains a few proprietary tools, mostly to do with system registration, updates, and maybe some non-obvious ones.
2-step-verify l2ping php
7z ldapmodify php-cgi
adm-online-update ldapsearch php-fpm
certificate locate pptp
convert lprm pushbullet
cryptsetup lpstat quota
curl memcached resolveip
dbd msmtp rfcomm
dbus-send mutt rsync
dcraw my_print_defaults rsyncd
effectiveperm mysql sdptool
exiv2 mysqladmin set-used-server
ffmpeg-mini mysqlbackup simple-mtpfs
file mysqlcheck smbclient
ftpscrub mysqld_safe smbpasswd
ftpwho mysqldump smbstatus
FU50-22888 mysql_install_db sudo
gearadmin mysql_secure_installation sudoedit
gearman mysql_upgrade sudoreplay
google-authenticator net thumbnail
hcitool nmblookup tree
healthrecord notification_add_column udevadm
htop ntlm_auth ufraw-batch
htpasswd openssl updatedb
identify openvpn wbinfo
internalbackup pdbedit webservconf
ip pdftops zip
I won't list /volume0/usr/builtin/sbin
, it mimics the above with yet more userland tools, and a few more proprietary ones.
Three
Wow this is tiring. Let's not list things, there's nothing of importance except the it is yet another layer of:
bin etc lib lib64 sbin tmp usr
nestled in the /volume1/.@plugins
directory.
Those proprietary binaries
I'm keen to see what's in those pesky files, aren't you? This entire exercise stemmed from me trying to work out how to kill the port hogger above. All the *mand
are:
emboardmand logmand netmand sftpmand volmand
hostmand nasmand raidmand stormand watchmand
strings stormand
, a snippet of strings:
_Z20Lock_System_Usb_Diski
_Z20Lookup_Nas_Disk_InfoiiiR17_T_NAS_DISK_STAT_
_Z14Probe_Nas_DiskPKcR17_T_NAS_DISK_STAT_
_Z20Lookup_Nas_Bank_InfoiiiR17_T_NAS_BANK_STAT_
_Z18Get_Nas_Disk_AliasRK17_T_NAS_DISK_STAT_RiPc
_Z34Lookup_Nas_Archive_Id_By_Device_IdiRi
_Z29Reactivate_Nas_Storage_SystemPFvvE
_Z14Unload_Nas_Boxii
_Z17Get_Nas_Box_AliasiiiRiPc
...
[Volume]
SYSTEM
/sys/bus/usb/devices/%d-%d
/sys/bus/usb/devices/%d-%d.%d
%s/bDeviceClass
stormand: Reboot now!
/dev/sd
/mnt/booting
MyArchive
>/dev/null 2>&1
/usr/sbin/fwupdate
%s recovery %s &
%s %s &
/proc/mounts
/volume
/archive
/usr/sbin/volmand
%s %s%d %s &
/sys/bus/usb/devices/%d-%d.
[Expansion Box]
%s %s ejected.
Ejected
[USB]
[eSATA]
[Disk]
DISK %d
Detected
satadisk
%s %s detected.
...
GCC: (crosstool-NG crosstool-ng-1.22.0 - x86_64 64-bit toolchain - ASUSTOR Inc.) 4.6.4
I've only grabbed some of the more interesting strings here. Error codes and lots of hard-coded paths. Browsing through the strings it looks like stormand
is indeed responsible for all storage, and is passed args via the base webserver. The rest of the Mandy's follow the same routine. Browsing the error codes it looks like ASUSTOR catches everything possible too (but don't quote me).
The initial webserver (lighthttp)
lighthttp serves up content from /usr/webman
once it's booted. The server side of this contains compiled cgi, and a client side app
which is a whole lot of minified but not obscured js, and all the assets required. At his point I was just getting curious about things and wondering how the overall NAS structure was done.
A quick strings
of /usr/webman/index.cgi
[luke@linux-cq7k]$ strings index.cgi
/lib64/ld-linux-x86-64.so.2
_fini
__gmon_start__
_Jv_RegisterClasses
_init
libnasman.so
_Z21Is_System_Initializedv
libdl.so.2
libc.so.6
puts
time
<snipped>
libnasman.so
eh? Let's look at that later. The only other insteresting thing in webman is jslibs/ext-5.0.1/
- this is an interesting library from Sencha, I think it's used for a variety of things in the user facing UI such as fancy graphs of the system stats, the windowing system, themes etc. Overall it's quite nice but overkill - the library size is 1.9MB, gedit shits the bed on files that size, vim doesn't though (yay vim!) but I didn't fancy scrolling a very long line. This is also a very expensive library at $1895USD for the enterpri.... I'm in the wrong line of work, I should be writing JS libraries to sell to NAS makers.
At this point I've entirely forgotten the point of this post and I'm still waiting for gedit to close after trying to load that library. Maybe I should crack open that wine. I bought a bottle of wine once but didn't have a corkscrew, I thought I'd be clever and push the cork in. I mean it worked... But there was wine fucking everywhere - I guess that pressure had to go somewhere eventually. And whenever I tried to pour it in to the glass the cork would exhibit this uncanny ability to go straight up the neck and block the flow.
I ended up drinking wine from the bottle with a straw.
Right, so, libnasman.so
-rw-r--r-- 1 luke users 534K Oct 9 20:36 libnasman.so.0.0
Do the strings thing, and, well this library seems to be a jack of all trades and hands duties off to everyone. It looks to handle core functionality such as image loading and conversions, video playing, video conversions, thumbnailing, networks, firmware, drives and partitioning, encryption and more. So something of a orchestrator of many of the base tasks a user would like to do. It's linked in to the webman cgi stuff, and in to many of the Mandy daemons.
Putting it all together
Heck of a ride. I actually did a lot more than what I've covered in this post - I tracked library links, mapped init scripts to run-levels to daemons to libraries etc - but hopefully the main gist of it comes through. The ASUSTOR NAS is really quite good, they created a very small and minimal software package that is extremely robust, with a front and back-end that is fast and attractive (oh... now I'm a product reviewer?).
The overall boot flow is something like this:
- Boot initramfs
- Does
/volume0
exist? No -> Go to step 4, else - Does
/volume1
exist? Yes -> Go to step 6, else - Load initial http server and direct user through setup
- Unpack
builtin.tgz
to/volume0
- Run init scripts in
/volume0/usr/builtin/etc/init.d/
- Run init scripts in
/volume1/.@plugins/etc/init.d/
- If any error, push cork in to bottle and attempt futile task of deleting
myhttpd
And the strong points of the way ASUSTOR have created this are:
- Uses lighthttp to serve the user UI, also uses a flash JS library for windows you can cover other windows with.
- Minimal Linux is fairly fast to boot, but then so is openSUSE
- The dual boot partitions are a good strategy to have ones behind
- Does not use any sort of interpreted environment anywhere. Nothing. Nadda. Everything is compiled to be fast.
- And so uses bugger all resources - less than 100MB
What's bad though, is you as a user of this product don't have full control over it.
- It uses apkg, which is a pain to create and is limited
- Lots of proprietary stuff, including control of the disk activity LED
- Tracking
This is just one way to build a NAS. It works, and works well. But in the end it wasn't what I wanted and the available OSS options weren't to my taste either so...
I'm going to be writing my own NAS front and back-end, using Rust, and based on openSUSE (targetting Kubic as the base). Keep an eye out for hardbytes.co.nz in the coming weeks.
Oh and I'll probably move this blog to rustacean.co.nz some-time soon.
Edit #1
I dug up the old posts I made on the ASUSTOR forums where I tried to find info on how to disable the myhttpd
daemon - the majority of those posts are duplicated below.
I started with running netstat, which I had to scp
over as it isn't available in the base install:
admin@nas:/etc/init.d $ sudo netstat -ntulp | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 4100/myhttpd
admin@nas:/etc/init.d $ cat /proc/4100/cmdline
/usr/sbin/myhttpd-t1-p8001
And running myhttpd
on its own results in:
admin@nas:/etc/init.d $ myhttpd
Usage:
httpd -t type -p port
type : 0(http), 1(https)
So the process that called "myhttpd -t1 -p8001" is what we need to find and kill...
If I start apache with ports 80/443 enabled then that process is killed, but now apache is using those ports. Even if apache is running, but on different ports, then something starts that process as soon as those ports are freed from apache.
I thought it was /etc/init.d/S44httpredir
controlling it, but because this is in the initramfs
it is restored on reboot each time, so any modification is practically useless (unless you edit the initramfs, which can be done).
The library libservice
calls myhttpd
in a few ways, and that lib is linked to by various things in the webman.
./lib/libservice.so.0:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./lib/libservice.so:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./lib/libservice.so.0.0:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so.0.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so.0.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./bin/lighttpdutil:1078:libservice.so
./webman/initial/sysreset.cgi:1402:libservice.so
./webman/initial/initial.cgi:3658:libservice.so
./lib/libservice.so.0:33733:libservice.so
./lib/libplugin.so:2384:libservice.so
./lib/libbuiltin.so.0.0:11025:libservice.so
./lib/libnasman.so.0:32123:libservice.so
./lib/libnasman.so.0.0:32123:libservice.so
./lib/libplugin.so.0:2384:libservice.so
./lib/libservice.so:33733:libservice.so
./lib/libbuiltin.so:11025:libservice.so
./lib/libplugin.so.0.0:2384:libservice.so
./lib/libbuiltin.so.0:11025:libservice.so
./lib/libnasman.so:32123:libservice.so
./lib/security/pam_google_authenticator.so:1759:libservice.so
./lib/libservice.so.0.0:33733:libservice.so
./sbin/httpredir:840:libservice.so
./sbin/recybind:1755:libservice.so
./sbin/nasmand:3057:libservice.so
./sbin/sftpmand:3425:libservice.so
./sbin/stormand:4836:libservice.so
./sbin/hostmand:3991:libservice.so
./sbin/logmand:2375:libservice.so
./sbin/netmand:2925:libservice.so
./sbin/dhcpserverctrl:1142:libservice.so
Because the whole point of this exercise in frustration was to claim the 80 and 443 ports for use with docker images, so I could then use nginx for routing to various containers, it turned out the easiest way to deal with this was to create either a startup script in the /volume1/.@plugins/etc/init.d/
or append to the docker init script the following:
PID1="$(fuser 80/tcp)"
PID2="$(fuser 443/tcp)"
echo "Killing useless port 80 hog, PID=${PID1}"
kill -9 ${PID1}
echo "Killing useless port 443 hog, PID=${PID2}"
kill -9 ${PID2}
and hope that docker claimed those ports before myhttpd
could be started again.