Multi-user Screen With SSH for Supervised Remote Support

This is a guide on how to configure systems to allow supervised shell access to remote support engineers. It is intended to:

• Allow multiple individuals shared access to one shell session
• Allow the PC operator to supervise and view commands
• Reassure the PC operator the remote engineer is not doing anything unauthorised
• Provide for several experts to share investigation of a remote problem
• Allow novices to see experts in action and learn from them

Quick Start

The preferred and quickest way to get going is Indirect Relay Connection. This uses an SSH relay-server to accept SSH client connections from the supported PC and the engineer, and then allows the engineer to connect to and share the user's command line terminal terminal on the supported PC.

Indirect Relay Connection

If you are a client wanting to configure an Indirect Relay Connection follow these steps:

1. Software installation and SSH client configuration
2. Configure Supported PC
3. Shared Session

or you can simply download the attached shell script to configure the Supported PC.

wget http://tjworld.net/raw-attachment/wiki/Howto/MultiuserScreenWithSshForSupervisedRemoteSupport/remoterelay.sh
chmod +x remoterelay.sh
./remoterelay.sh

Direct Remote Connection

If you are a client wanting to configure a Direct Remote Connection follow these steps:

1. Port Forwarding and software installation
2. Configure Supported PC
3. Shared Session

Requirements

1. The screen utility
2. SSH server (openssh-server)
3. Access to supported PC using SSH from outside the supported PC's network:
   1. Direct Remote connection to the SSH server port (directly or via port-forwarding in a gateway router) or
   2. Indirect Relay connection using an external SSH relay server to avoid port-forwarding requirement

Warning: If the supported PC is running an older version of Ubuntu that is not a  Long Term Support release (for example: 7.10 Feisty) it may not be possible to install optional packages such as openshh-server (if not already installed) for which Ubuntu no longer provides support. This is because the Ubuntu archives may no longer list the packages for that release, and may also have deleted them from the archives. It is possible to locate the packages through the  Ubuntu old releases archive.

Decide on Access Strategy

Because there are two distinctly different methods to provide SSH access to the supported PC from outside its home network, the first decision is which to use since the configuration steps thereafter are significantly different.

If the operator of the supported PC has control of, and access to, their local Internet gateway/router and the knowledge to add a port-forwarding rule that will allow SSH client connections from the Internet to be forwarded to the supported PC, the recommended strategy is Direct Remote Connection.

If the operator of the supported PC has no control of their local internet gateway/router and it doesn't currently provide SSH port-forwarding to the supported PC, the recommended strategy is Indirect Relay Connection.

The following two sections describe the configuration steps required for each access strategy in detail.

Direct Remote Connection

Supported PC

Port Forwarding

It is beyond the scope of this guide to give instructions on configuring any routers for port-forwarding. The process is usually obvious in web-based router control panels. The key part is to forward the public Internet interface's TCP port 22 to port 22 on the local PC. Usually that requires setting the IP address or name of the local PC as a destination.

If you want to avoid using the well-known port 22 on the public interface to avoid brute-force password cracking attempts by casual scanners, configure the public Internet interface (listening) port to be some suitably random high port number (1024 to 65535) and give that port number to remote operators; for example: port 12345. The destination port on the local PC should still be 22.

Install required software

Ensure screen and SSH server are installed. On Ubuntu that requires:

sudo apt-get install screen openssh-server openssh-client

Usually screen and the SSH client (openssh-client) are already installed.

The next step is Configure Supported PC.

Indirect Relay Connection

In this scenario a third computer (accessible on the Internet) is used as a relay between the supported PC and the remote operator. It avoids the need for the supported PC's operator to reconfigure their internet gateway/router to port-forward SSH. It requires that either the remote support operator or the supported PC's operator has administrative control of an internet server.

Relay Server

The administrator should connect to the relay-server. Usually this will be via SSH which implies the server already has the SSH server package installed. In case it doesn't it should be installed along with the netcat utility that is used to relay the connection:

sudo apt-get install netcat openssh-server

Ensure the server's firewall rules allow both the remote operator and supported PC to connect via SSH (some firewalls will limit SSH access to specific IP addresses or sub-nets).

sudo iptables -nL

Create a user account that the SSH relay connections can connect to:

sudo useradd remoterelay

Communicate the assigned password for the remoterelay account to the operators of the relay clients so they can transfer their public keys to the server.

Note: After the support client operator has transfered their public key to the server it is recommended that the sessions be blocked for 48 hours (or more, or less) so that anyone attempting to use the credentials to access the relay-server will have a very long wait before the log-in shell is available to them. This won't affect the relay connection since it does not connect to a terminal. Do this by prefixing the public key in ~/.ssh/authorized_keys with a sleep command:

command="sleep 172800"

so the support client entry begins similar to this:

command="sleep 172800" ssh-rsa AAAAB3

Warning: If the remote support operator also transfers their public key to the relay-server in the same way, do not prefix their key with this command otherwise they will have to wait before their commands can be executed!

Supported PC

To enable the relay to succeed the operator's SSH configuration needs to be configured so that it makes an outbound connection to the external relay, then forwards that connection to the local SSH server.

First, ensure the SSH server is installed and running on the PC that requires remote support:

sudo apt-get install openssh-server

Create a new SSH public/private key-pair specifically for use by the relay-server that doesn't require a pass-phrase:

ssh-keygen -t rsa -N "" -f ~/.ssh/id_relay

Create a pre-defined host entry in the operator's SSH configuration. Some of the values will be specific to the particular support incident so adjust these as appropriate. This command appends the configuration to an existing ~/.ssh/config file. You may want to use a text-editor to add or amend the settings:

cat <<EOF >> ~/.ssh/config
Host relay-server
User remoterelay
HostName relay.tjworld.net
RemoteForward 22222 localhost:22
IdentityFile  ~/.ssh/id_relay
EOF

In this example SSH will expect to connect to the user remoterelay on the server relay.tjworld.net. The relay-server administrator should ensure the remoterelay account is created and the password made known to the operator before the next step, which is to transfer the operator's public key to the relay-server:

ssh-copy-id -i ~/.ssh/id_relay.pub remoterelay@relay.tjworld.net

From now on, any connections to the relay server will not require the password since the client will use its private key and the server will verify using the public key now stored.

At this point the supported PC's operator can connect to the relay-server and have their local SSH server wait for a connection from the remote support operator.

The following command connects to the host 'relay-server' (defined in ~/.ssh/config) and waits for remote connections to be successfully established. It is prevented from reading stdin and is put into the background:

ssh -n relay-server &

The supported PC is now ready to accept connections from the remote support operator, via the relay-server.

When the operator wants to stop the server

The next step is to configure screen in the Configure Supported PC section.

Remote Support Operator PC

The remote support operator makes a connection to the relay server and executes commands that will forward the connection to the supported PC. To keep commands simple the options can be added to the remote operator's SSH configuration:

cat <<EOF >> ~/.ssh/config
Host support-client
HostName supported-pc.somedomain.net
ProxyCommand ssh remoterelay@relay.tjworld.net /bin/nc localhost 22222
EOF

The HostName should be replaced with the DNS hostname or IP address of the supported PC's public internet connection.

The ProxyCommand will be executed when the connection is established on the relay-server. It will run the command "/bin/nc localhost 22222". The port (22222) specified here is the one specified in the supported PC's operator's SSH configuration. These must always be the same.

To avoid having to provide a password each time the remote support operator connects to the relay-server, the remote support operator's public SSH key can be transfered to the relay-server. If using their regular key (from ~/.ssh/id_rsa.pub):

ssh-copy-id -i remoterelay@relay.tjworld.net

If using a key just for the relay-server:

ssh-keygen -t rsa -N "" -f ~/.ssh/id_relay
ssh-copy-id -i ~/.ssh/id_relay.pub remoterelay@relay.tjworld.net

Configure Supported PC

There are two stages to preparing the supported PC. The first is to ensure that the screen utility is able to perform multi-user functions, which are a one-time requirement (unless using a live-CD in which case they need performing each time the live-CD is started). The second are the steps performed for each support incident.

Configure screen for Multi-User Access

To provide multi-user access the screen executable binary file requires setuid root permission. This is a potential security issue in some circumstances if left enabled permanently. Fortunately it is simple to set and reset the 'setuid' bit.

All except Jaunty 9.04

To setuid root:

sudo chmod 4755 /usr/bin/screen

To reset (after the support incident is completed):

sudo chmod 0755 /usr/bin/screen

Jaunty 9.04

To setuid root:

sudo chmod 4755 /usr/bin/screen.real

To reset (after the support incident is completed):

sudo chmod 0755 /usr/bin/screen.real

Set permissions for screen's working directory:

All except Jaunty 9.04

sudo mkdir -p /var/run/screen
sudo chmod 755 /var/run/screen

Jaunty 9.04

sudo mkdir -p /var/run/screen
sudo chmod 775 /var/run/screen

Add a Remote User

The remote connections require a regular user account on the supported PC. Create the account 'remoteop' with a simple password (the password is given to the remote support operator):

sudo adduser remoteop

Screen Session

Start screen with a user-friendly session name and log all activity (logs are saved to the current directory with the name format screenlog.X where X is the window-number):

screen -L -S Multiuser

screen starts a new terminal and 'gets out of the way'.

In order to give screen commands an escape sequence (combination of key-presses) is used to tell screen to enter command-mode. The default is Ctrl+A (but Ubuntu has changed it to Ctrl+] on Jaunty 9.04 (that's a closing square bracket, not the letter 'Jay')).

You'll need the escape-sequence to control screen once it is running. The command you type will appear above the screen status line (bottom of the screen terminal window).

Note: We've found that when doing this from a live-CD the escape sequence is Ctrl+A, the default screen sequence.

Enter multi-user mode. Press the escape-sequence and then:

:multiuser on

Give permission for the 'remoteop' user. Press the escape-sequence and then:

:acladd remoteop

Now wait for the remote operator to connect and join your session.'

Note: the supported PC's operator's log-in username should be given to the remote operator (in addition to the 'remoteop' username used to connect). It is used by the remote operator's screen to identify which session to connect to.

Remote Operator

Direct Remote Connection

Connect to the supported PC as 'remoteop' (using the password given to you by the supported PC's operator) with ssh (replace 1.2.3.4 with the public IP address or DNS name of the target PC's gateway router):

SUPPORT_HOST=1.2.3.4
ssh remoteop@$SUPPORT_HOST

If the user has the SSH daemon listening (or port-forwarding) on a port other than 22 add the -p <port> option, for example:

ssh -p 12345 remoteop@$SUPPORT_HOST

Indirect Relay Connection

Using the settings previously added to ~/.ssh/config makes connecting to the client's PC easy. If you transfered your public key to the relay-server you will not need to enter a password for remoterelay@relay-server - only for the supported client. If you haven't, then you will be prompted first for the password for remoterelay@relay-server, and once the relay-connection reaches the supported PC, for the password for remoteop@support-client. Try not to get confused by which host is asking for a password!

ssh support-client

Connect to Client Session

Once the log-in to remoteop@support-client is established you should connect to the client's already-running multi-user screen session. The connection string is 'username/session'. The username is the account on the supported PC that started the multi-user screen session (that of the supported PC's operator). For a live-CD that will usually be ubuntu:

USERNAME=ubuntu
screen -x $USERNAME/Multiuser

Shared Session

The shared session is now running. Both the supported PC's operator and the remote support operator can type to the same command-line and therefore some discipline is required so that both don't try to type at once. An IRC or other instant-messenger session might be helpful for discussing what to do.

End Screen Session

To remove permissions for the remote support operator, press the escape-sequence and then:

:acldel remoteop

To close the screen session, press the escape sequence and then:

\

Screen Commands

There are a larger number of key-sequences and commands in screen. To discover them use the manual page:

man screen

Locking and Unlocking Remote Account

Obviously it would not be a good idea to leave the 'remoteop' account enabled. It is simple to lock and unlock it as needed.

To lock the account (prevents access but doesn't reset the password):

sudo passwd remoteop -l

To unlock the account (access enabled):

sudo passwd remoteop -u