Saturday, 19 August 2017

Machine Learning resources

How To Become A Machine Learning Engineer

Useful for more advanced IoT applications?

Saturday, 31 December 2016

Coding for IoT

Wednesday, 31 December 2014

First Mail of the New Year

Happy New Year.  Welcome to 2015

Ok, so now we have a hardened server in play, the next step is to find some collaboration-ware to put on it.  To my mind, there are several options, and there are also a couple of different purposes that I am trying to fulfill.

Private Cloud

For this purpose, my decision is already made.  I've been playing with ownCloud for some time, and I do like it.  It is simplistic.  Its uses extend beyond just a file share in a cloud.  There is options for tracking tasks and sharing other information.

Private Email Server

Email servers have been a challenge.  In this field I have found three options; Citadel, Xmail and Exim.

I have watched Citadel mature over the last while (even had it running on a box for a while), and it seems to be coming together quite nicely, and from what I can read in their online instructions, setting it up should be reasonably straight forward - it comes as one install package.  It has a nice clear UI, and it handles lots of different email/calendar tools.  It also integrates with other email servers.  Nice.

Xmail on the other hand, at first glance does not appear as such a good option.   It is down-loadable as source - that's not too much of an issue.  However, there are so many third party components, and the basic configuration information and details are so vast, from the short skim-read I've had, it feels like I'd need to a complete degree to wrap my head around it properly. Also, the most recent release of Xmail was 4 years ago.  In the longer run I think Xmail could be more powerful, but given that I want this collaborative functionality up and running soon, Xmail is not my first choice.  Citadel, I have installed and set up before, and I know it will be quicker.

Exim I have only just found.  At this stage I do not know enough about it, and their web pages seem to have too little information up front, and possibly too many links to buy the book about Exim.  Although it does seem to be a single install package like Citadel, which is good, it is something that I'd like more information to be readily available before I commit to it.

When Too Many Tools Do the Same Job - A First World Problem

In looking at ownCloud alongside Citadel, there are a lot of features in both product sets.  A lot of these features are common to both products.  To set up collaboration-ware for any size group, even a family, there is a requirement to keep the options simple and straight forward.  If you give people two different task lists - which one should they obey.  How do they manage two different lists easily.  File repositories - if you have too many, there is always the question of 'Where did I put that file?'. Appointments that appear on one calendar - can they be synced to the other.  If not, can we constrain the feature set of one or both products, so that the overlap offered is minimised? For my part, successful delivery of a service does not simply mean a successful install - it has to be usable and efficient for the end users as well.

Plan of Attack

So, my first plan for the new year is simple.  Get a mail server working.  This will allow individuals to get their own calendars, tasks, contacts and emails functional.  The quickest solution here is obviously Citadel.  Once it is up an d running, it will then be possible to see how much the latest version of Citadel supports our required collaboration, find what gaps need to be filled, and see if ownCloud is actually needed.

Tuesday, 30 December 2014

Tuning it Up and Tightening the Screws

Before we put our RasPi server live to the world, there are some further steps that we need to take to improve its overall security, and keep in mind areas where we may improve its performance.  What we have really is a vanilla install of Raspbian Wheezy, with a few extra services such as Apache 2, PHP 5 and MySQL added in, and if we connect to it with our web browser, we will see Apache's vanilla test index page.

If anyone was to attempt to hack a standard install of any system, the plan of attack that is most likely to lead to success, is to utilise what they know or reasonably expect to be true, and that which they can easily find out.  For example, most people who have any experience with Raspberry Pis will know that the default user login is 'pi', and the default password is 'raspberry'.  Fortunately we were smart enough to update the default password to one of our own choosing (You did do his?  Didn't you?). We were also smart enough to set a new password for root.  This means that we have control over both the pi account, and the root account (sudo passwd root).  We keep the root account for times only when we desperately need it.  The pi account is more of an everyday account, and it has the ability to sudo if needed.

However, whilst we can use configuration of services such as SSHd to deny permission for the root account to remote in, we want the pi account, or an account like it to be able to remote in for our administrative needs.  Since we will deny root this privileged access, the pi account becomes the greatest weakness.  The authentication of every login requires two pieces of information - the user login, and the password.  If we continue using the pi account, any potential hacker will assume that the RasPi will by standard have such account, and we would be effectively giving them one of those pieces of information, making it easier for them to utilise hacking methods such as brute-force attacks to break only the password of the account.  Therefore, our first step must be to replace the pi account, with a new user account.

Clean up User Accounts

First, we need to find out which user groups the pi login is a member of:

groups pi

Replace Pi

From this list we need to choose which groups you want the new account to be a part of, and set up the new user with the following command

sudo useradd -m -G <list of groups> -s /bin/bash replace_pi

Depending upon how you are going to use your RasPi, you may not need your new account to be part of all of them. I left out dialout (not using these protocols), cdrom (no cd drive connected – don't need to access the drivers), games and spi.  Also I did not give it root.

Next we need to remove the pi user account.  It is a little bit like an old passport.  Just because we don't use it any more, we shouldn't leave it lying around where others might be able to use it for the wrong purposes.

sudo deluser --remove-home pi

Remove Unnecessary Accounts

Remove any guest accounts, or other unnecessary accounts.  To aid in this you may like to open the following files and review the contents:


I found that there were several accounts whose name signified a certain purpose, that did not fit with how I plan to use my RasPi.  These were 'games', 'lp' and 'nobody'.  For now, this RasPi will not be used to run games, nor will it be a print server, and I don't like accounts named 'guest' or 'nobody'.  Removal of these accounts requires the removal of users in each group, then the deletion of the group:

sudo deluser games
sudo groupdel games
sudo deluser lp

sudo groupdel lp
sudo deluser nobody

sudo groupdel nogroup

Consider what access you want the new account to have. If you are deploying websites to /var/www, do you want the new account to have access to write to that folder? Or, if you use Visual Studio or Dreamweaver, or a version control tool to manage such deployments, do you want/need to create another account with the specific access permissions so it can do just that job? What protocols will your deployment tools use? FTP? Does the account have the right access?

Configure SSH

Returning to our thoughts about hackers having a higher chance of success by using what they know about common services, lets change what they know.

sudo nano /etc/ssh/sshd_config

Secure Ports

Within this config file we need to make a few changes.  Lets start with the most obvious.  Every one knows that SSH connections use port 22.  But they don't have to.  Lets change the port number that SSH listens to, to another port number - say 5022.  Given that we currently have two IP addresses operational for our RasPi at the moment, we should also restrict SSH access to just one of those addresses, to limit the opportunity for discovery.

# What ports, IPs and protocols we listen for
Port 5022
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::

Check the Protocol

SSH has gone through a number of iterations, and the service is into it's second major release.  This second release improves upon the security of the first, and is the version we should be using.  We need to configure our daemon to make sure Protocol 2 is used.

Look for the line in the file indicating which protocol is in use.  Should appear as:

Protocol 2
# HostKeys for protocol version 2

If you are seeing this in your config, you are right to proceed with using SSH 2.

Allow Users

Within the same config file, we should also specify which users are permitted to use (remotely login) the SSH service.  To do this, add in the following lines

AllowUsers replace_pi (use a comma separated list for every user account to be allowed access)

Another appropriate measure for securing SSH is to restrict the number of concurrent connection attempts - these are attempts where a login is in progress, but not yet succeeded.  This is done through the MaxStartUps property.  This property has three values that represent start:rate:full.  The start value tells the daemon how many connection attempts to allow at once without restriction.  Think about how many users you are likely to have attempting to connect at the same time via SSH - probably not many.  By default, this value is 10.  If this is definitely too many for your circumstances, then reduce that value.  Do not set the value below 1 or you are likely to have troubles connecting.  The rate value dictates the rate of connection refusals that should occur once the number of attempted connections meets or exceeds the start value.  So in the set of values expressed below, we allow a start value of 2 attempted connections, and after the second attempt, we will automatically fail 50 percent of any further connection attempts.  The last value, the full value, tells our daemon the absolute maximum number of connection attempts that are allowed before all further connection attempts will automatically fail.  Please note - these are connection attempts only.  Once the remote user has successfully logged in, their session is authenticated, and does not count towards the number of connection attempts anymore.  By reducing these properties to values that make sense for your own situation, you are limiting the impact of brute-force and DOS attacks against the daemon.

MaxStartUps 2:50:6

Minimise Unattended Session Risk

Next we want to minimise the unattended session risk.  To do this we are going to shorten timeout settings.  ClientAliveInterval tells the daemon how many seconds after the last activity from a connected client to send the Client Alive message.  This is a system level message that polls the client for activity.  If the client system does not respond, the daemon will send another client alive message after another period the length of the interval value.  The ClientAliveCountMax value tells the daemon how many un-responded ClientAlive messages to tolerate before closing a client connection.  With the following settings below, the daemon will wait for 4 minutes (240 seconds) after the last client connection activity, poll the client, and if the poll is not responded to, close the connection.

ClientAliveInterval 240
ClientAliveCountMax 0

Block root Access

We also do not want the root account using SSH.  Check that the following line is part of the config file :

PermitRootLogin no

Warning Banner

Within the sshd_config file, there is also a line for setting a banner message through a third file. Uncomment the line '#Banner /etc/issue':

Banner /etc/issue

Banner messages can be used in a couple of ways: a legal warning or notice, a statement of purpose or a terms and conditions of use. As a legal notice or terms and conditions, you can make users aware of the ownership of the server and attempt to scare them away with the threat of legal action. A good hacker however knows that for legal action to be successful, you need to be able to identify them. The threats are not that scary. A statement of purpose might be more useful - particularly one that indicates that there is not likely to be anything important on the server. Never reveal what equipment, systems or software are available unless you are providing public access, and try not to expose private information..

A bad banner:

# This is the home sever of Joe Bloggs. It provides FTP, WWW,  #

# and MySql services, with authentication via Active Directory.#

This banner tells potential hackers who the owner is, which ports should be open, what types of vulnerabilities to expect based upon the services exposed, and upon the fact that this would likely be a Windows server running Active Directory.  The only details it does not offer up in plain sight is the user's passwords.

A better banner:

# This is the server for the internal use of the Bloggs family. No files of any value or            #
# external importance are maintained within. If you are not part of our family, please         #
# cease in your actions and disconnect from this server immediately.                                      #

Use TCPIP Wrappers and edit /etc/hosts.allow file

Another important restriction that should be applied to SSH, is where on the network/Internet will your RasPi accept connections from.  The TCPIP Wrappers work by default to allow connections from everywhere, and this will occur if you do not modify the /etc/hosts.allow file and the /etc/hosts.deny file.

In the /etc/hosts.allow file, you need to explicitly state which IP addresses, or address ranges can connect to the daemon.

sshd:     # allows one host to connect to the daemon

sshd:,     # allows two hosts to connect

sshd:     # allows any host from the specified network segment to connect

If our host IP has not been found in the hosts.allow file, the daemon will next look for it in the hosts.deny file.  If the daemon finds it here, the connection will be denied.  Edit the /etc/hosts.deny file

sshd: *     # denies access to all hosts, except those that have already been grated access in the hosts.allow file

If you do not include a line such as this in your hosts.deny file, the daemon will:
- allow all hosts it finds listed in the hosts.allow file to connect
- deny connections to all other hosts if they are listed in the hosts.deny file (so, if you list none, then any host can ssh to your Pi)
- allow connections from all hosts not already listed in the hosts.allow and hosts.deny file, by default.

Verify the Configuration

At his point, we have made some significant changes.  Before proceeding any further, we should verify the integrity of the changes we have made by running the following command from the command line:

sudo /usr/sbin/sshd -t

Clean up Apache

Finally, let's look at the apache2.conf and ports.conf files, to improve security around our web server components.

sudo nano /etc/apache2/apache2.conf

For now, just review the values in the # prefork MPM section - may assist with performance and security

sudo nano /etc/apache2/ports.conf

In the ports.conf file, you can tell Apache to only listen on port 443 by commenting our the line for listening to port 80.  This would force all incoming connections through SSL ports (443).  Depending upon what you want to do with your RasPi, this may be an option to work with at a later point in time.

Make your own Index.html

Finally, change the default index page. The default index page is an indicator of a web server that is not fully configured. Search engines will index such pages making it easy for potential hackers to find such pages.  You don't need to change much - only the text that appears.

Saturday, 6 December 2014

Up and Away - Setting up a LAMPi

For guidance I found a great article by Etel Sverdlov on Digital Ocean.  Written for Ubuntu 12.04 - but none the less still valid

Installing a LAMP (server) on your Raspberry Pi

This is something that I have done a number of times now, but I've always dashed in and just done it in whatever order worked.  Having looked at this in a bit more of a planned manner, I read Etel Sverdlov's article first, and that made a lot of sense.  What I'm attempting to present here is based upon that, but with a few updates to call out some important points.

Step 1 - Latest repositories

First of all, need to update the repositories for aptitude (I prefer aptitude to apt-get - use apt-get if you wish)

sudo aptitude update

Step 2 - Install Apache2

Apache2 is a nice web server.  Get used to it and it can be really friendly to use and administer.  To install:

sudo aptitude install apache2

It's pretty straight forward.  You'll be asked to confirm the installation of new files - just answer 'y'.  Once that is done, then it is time to test your Apache2 installation using your browser.  If you are running the Pi GUI on your RasPi, point your browser at  If like me, you are setting up your RasPi to be a headless server, with only the command line interface, you'll have to point another machine on your network at the RasPi's address.  You should see a web page like this:

During the Apache2 installation, there was one message - "Could not reliably determine the server's fully qualified domain name, using for ServerName".  This possibly happened on your install too.  Depending on what you want to do with your RasPi, this may need to be addressed later on.  For now, I'm letting it slide.  Let's call it a Priority 2 TODO

Now you may have read my previous post, where I setup multiple IP addresses on the RasPi NIC at eth0 through aliases.  So when I point my browser at the root address of the alias, I still get Apache's default index page.  So, the web server is running, on both aliases.  This is a bit more urgent - let's say a Priority 1 TODO.

I'll come back to these points in future posts.  Moving on for now.

Step 3 - Install MySQL

MySQL is quite a nice bit of kit.  Haven't used it enough.  And full credit to Etel, I've followed her instructions quite closely this time.  In the past I was quite nervous about attempting an install without the test DB, and without other features, but as Etel rightly point's out, these aren't necessary, and fixing them now will lead to a tighter installation, better for security.  Keeping in mind that this RasPi is due to be effectively a production level machine, that's what we need.

Installing MySQL is a bit more involved, with a few choices to be made.  To kick off the process:

sudo aptitude install mysql-server libapache2-mod-auth-mysql php5-mysql

From this command, your system may detect some conflicts in the packages, particularly apache2-mpm-prefork vs apache2-mpm-worker. Where we go from here depends upon what you want to do with your box.

For mine, I'm going to accept removing apache2-mpm-worker module. The apache2-mpm-worker is a more memory efficient module for handling multiple calls using thread processing.  However it constrains then that you should only use thread safe calls, and this module is known to have issues with PHP.  If we were building a dedicated MySQL box, without the Apache and PHP components, for a large, highly scalable commercial system, I would probably recommend differently.  But for a low volume server, where system performance is non-critical, where some code used may not be thread-safe, I am choosing apache2-mpm-prefork.

Set the MySQL Root password when prompted.  Just get it done and dusted.  You can do it later, but doing it now will help avoid forgetting later.

Next, set-up and configure the database with the following commands:

sudo mysql_install_db

sudo /usr/bin/mysql_secure_installation

The system should prompt you for the MySQL Root password:

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Now come the important security choices in the MySQL installation.  The first will be the choice to remove anonymous users.  For me, this is a no-brainer.  Unless you want to setup a publicly accessible free-to-play database server, there is no reason.  To support a web server, your install should only have known users with roughly three levels of access; root for admin, developer accounts for developing (or deploying/troubleshooting), and web app/service accounts for the applications that are going to consume the database.  So, that's a yes to removing anonymous users.

Remove anonymous users? [Y/n] y                                           
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y

... Success!

HOLD THE FORT!  Don't let root login remotely?  How will we administer this?  Don't stress, before putting the RasPi to it's final config, we will create the other user accounts.  If however, you are already running your RasPi through SSH, you may wish to choose 'n' for this option for now. (Just note down for later that you need to change it)

Removing the default test database is another no-brainer.  For this to be a production level box, we should not have any test artifacts.  If this was for a development server, or a classroom education server, I'd say keep it, but for this one, it's going.

Remove test database and access to it? [Y/n] y

 - Dropping test database...

 ... Success!

 - Removing privileges on test database...

 ... Success!

Reloading the privilege tables will ensure that all changes made so far will take effect immediately.

Reload privilege tables now? [Y/n] y

 ... Success!

Step 4 - Install PHP

Installing PHP is pretty straight forward, but there will be some more choices to be made along the way.  Let's start with:

sudo aptitude install php5 libapache2-mod-php5 php5-mcrypt

Then as the chosen packages are identified, we have another choice to make, a conflict between libapache2-mod-php5filter and libapache2-mod-php5.  "What's the difference?" you ask - well here goes:

  • libapache2-mod-php5filter is not thread safe, so if you opted for apache2-mpm-worker in MySQL (see above), do not use libapache2-mod-php5filter
  • libapache2-mod-php5filter does not pass PUT and OPTIONS calls to the PHP processor - sends them straight to Apache and can break WebDAV - possibly not good for collaboration platforms

So, with this reasoning together, my selection was easy - libapache2-mod-php5 stays.

Next we want to make sure that if we have a PHP coded index page for a site, that it will get recognised as the index page.  This may, or may not have been put into the dir.conf file during installation.  Double check by using sudo to open dir.conf in your preferred editor.

sudo nano /etc/apache2/mods-enabled/dir.conf

If necessary add index.php to the beginning of index files. The page should now look like this:

<IfModule mod_dir.c>

          DirectoryIndex index.php index.html index.cgi index.xhtml index.htm



Next, you may want to add some extra modules to extend the PHP functionality.  This does not need to be done right now, but if you want to, check which modules are available in the repositories first:

apt-cache search php5-

Choose your modules and install.  Finally test the PHP install.  Use your preferred text editor to create the following file: /var/www/info.php


Then restart Apache

sudo service apache2 restart

Now test the page in your browser:
If you see a page like the one below, all has been successful.

Thursday, 4 December 2014

Change Brings Challenges

For some time now I've been running a #RasPi off our router, providing the private cloud based options for file-sharing, collaboration, and getting to those important documents we forgot to bring.  So far I'm very happy with ownCloud.  I'm also happy with Apache as my base web server, though I have not done enough with it.  A basic home page is all really.

Plus, that image has been up and running now for some time.  It's image base is old (2013??) and it probably needs a spruce-up of its firm-ware.  I want to put an email server on it.  So far I'm considering Citadel as my front runner there.

The other problem is that I fear it may have been hacked.  Either that, or with my increasing age, and the extended periods of time since I last accessed that box, I have forgotten the passwords.  Ooopps!  Can't have a box that has been rooted.  The kid in me smells a counter hack in the making. The old bloke recognises that there was nothing on the box of any grand importance. A #rebuild with a new SD card and properly implemented security would probably avoid the potential of reclaiming a box with unknown backdoors.

Beyond this, what I'd also like is to setup the opportunity for some web services that can interface with some home-made mobile apps.

What I am uncomfortable with is our router.  Yes, you can set it up to forward connections for different services and applications.  It will forward incoming common protocols to the correct port on a specified IP address.  I am not certain however, if you have more than one website, differentiated by port, that the router's port forwarding works that well.  The interface for setting up the port forwarding is not user friendly.  So, I am considering using multiple IP addresses on eth0 (#aliases), to ease the pain.

To set up for using multiple IP addresses, there are two methods.  One is to a running change that is non-persistent  An example of this can be found at Penguin Tutor.  The method I am going to use will persist the changes when the RasPi is rebooted:

pi@exampleberry ~ $ sudo vi /etc/network/interfaces
Edit the interfaces file with vi to add the eth0:0 alias

iface eth0 inet static
    auto eth0

iface eth0:0 inet static
    auto eth0:0


iface eth0:9 inet static

Repeat as necessary for aliases eth0:1.. to eth0:x, depending upon how many aliases your box requires. Save the changes to your interfaces file and restart your network services.

pi@exampleberry ~ $ sudo /etc/init.d/networking restart

Check the results by running ifconfig.

Of course, a potential headache that I will create by doing this is that I will simply increase the exposure of my RasPi box to further intrusions.  So before I go any further, its time to research how to harden my RasPi and monitor for intrusion.

Narad Shrestha's article on Tecmint (, "Create Multiple IP Addresses to One Single Network Interface"

Friday, 20 June 2014

Moving Beyond the QNAP Concepts

It's been some time since I drooled over a friend's set up of Plex on a QNAP NAS, and wondered how I could efficiently manifest something similar on a RasPi.  Further analysis of the circumstances have reshaped my thoughts on that matter.  But as for any of the technical challenges I seem to take on around our home, the journey is rarely straight forward, and the direction constantly changes.  And for this project, so has the prospect of a budget.

Previously I had identified that running the Plex services of the RasPi was not an option.  And the reason I wanted to use Plex was that it would work natively with our lounge room TV.  Further research and trials has since blown that concept out of the water - LG does not have the smart TV apps for our model that work with Plex.  This does not mean our ideas for using Plex are dead - far from it.  Allow me to elaborate.

The media centre solution that I need to apply now needs to facilitate the following:
Access from our lounge room TV (LG) - Wirelessly connected to our internal network
Access from our bedroom TV with smart Blu-ray (Samsung) - Wirelessly connected to our internal network
Limited access from our son's playroom TV (Medion) - not currently connected to our networks
RasPi does not run Plex, but can run XBMC
I have canabalised an old 2nd hand machine, and setup an Ubuntu server with Plex, CouchPotato, Sickbeard and other apps.  It's in our DMZ.  This means that the Samsung smart Blu-ray can access it, but our LG TV cannot as its network features do not seem to allow it to discover appliances outside its subnet.

So, the Plex server in the DMZ, works. Samsung has a Plex app. Works on the bedroom blu-ray. Lounge TV can't use Plex, and can't connect to the DMZ. But the Plex server is capturing content, and I don't want to put it in the internal subnet.  The Plex server has a fair but of content on it already, but I want to be able to control the access our son has to that via his playroom TV.  And at the same time, whilst I'd like to be as power conservative as possible, our current situation dictates that conserving money is the more important criteria.

Limiting the connectivity of the playroom TV would also be advisable - helps with regulating the young boy's Internet access.  Plus, it would be beneficial family suitable content could also be made available on our portable DVD-player when we travel.  It accepts USB connections, as does the playroom TV.  So, my thoughts are that the easiest solution there, is a portable USB drive.  I can refresh the content every once in a while to provide variety, but otherwise it is low-cost, simplistic, restricts access, and is portable.

That means that there is only the lounge room TV to satisfy.  My current thinking is that my RasPi with XBMC, could serve movies to it directly through HDMI, and the content for the RasPi could be migrated from the Plex server either by some kind of scheduled task, or by establishing some code that will detect when new movies and TV shows have arrived on the Plex server, and will cause an update to the RasPi.