Docker Installation/Setup in Ubuntu

Installation
————-

Update your droplet:

$ sudo apt-get update
$ sudo apt-get -y upgrade

Make sure aufs support is available:

$ sudo apt-get install linux-image-extra-`uname -r`

Add docker repository key to apt-key for package verification:

$ sudo apt-key adv –keyserver hkp://pgp.mit.edu:80 –recv-keys 58118E89F3A912897C070ADBF76221572C52609D

Add the docker repository to Apt sources:

$ echo “deb https://apt.dockerproject.org/repo ubuntu-trusty main” | sudo tee /etc/apt/sources.list.d/docker.list

Update the repository with the new addition:

$ sudo apt-get update

Finally, download and install docker:

$ sudo apt-get install docker-engine

Edit UFW (Uncomplicated Firewall) configuration using the nano text editor.

$ sudo nano /etc/default/ufw

Replace: DEFAULT_FORWARD_POLICY=”DROP”
With: DEFAULT_FORWARD_POLICY=”ACCEPT”

Finally, reload the UFW:

$ sudo ufw reload

Using Docker
————

To run the docker daemon:

$ sudo docker -d &

For system-wide information on docker:

$ sudo docker info

For docker version:

$ sudo docker version

Working with Images
——————-

Searching for a docker image:*

$ sudo docker search ubuntu

Downloading (PULLing) an image:

# Usage: sudo docker pull [image name]

$ sudo docker pull ubuntu

Listing images:

$ sudo docker images

INFO[4160] GET /v1.21/images/json
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu              latest              e9ae3c220b23        10 days ago         187.9 MB
hello-world         latest              0a6ba66e537a        5 weeks ago         960 B

Committing changes to an image:

# Usage: sudo docker commit [container ID] [image name]

$ sudo docker commit 8dbd9e392a96 my_img

Sharing (PUSHing) images:

$ sudo docker push my_username/my_first_image

Working with Containers
———————–

Use the following command to list all running containers:

$ sudo docker ps

To have a list of both running and non-running ones, use:

$ sudo docker ps -l

Creating a New Container

To create a new container, you need to use a base image and specify a command to run.

# Usage: sudo docker run [image name] [command to run]

$ sudo docker run ubuntu echo “hello”

# Usage: sudo docker run –name [name] [image name] [comm.]

$ sudo docker run –name=test ubuntu echo “hello”

Running a container:

# Usage: sudo docker run [container ID]

$ sudo docker run c629b7d70666

Stopping a container:

# Usage: sudo docker stop [container ID]

$ sudo docker stop c629b7d70666

Removing / Deleting a container:

# Usage: sudo docker rm [container ID]

$ sudo docker rm c629b7d70666

Upgrade Docker
————–

$ curl -sSL https://get.docker.com/ | sh

Uninstallation
————–

To uninstall the Docker package:

$ sudo apt-get purge docker-engine

To uninstall the Docker package and dependencies that are no longer needed:

$ sudo apt-get autoremove –purge docker-engine

If you wish to delete all images, containers, and volumes run the following command:

$ rm -rf /var/lib/docker

You must delete the user created configuration files manually.

Ref: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-getting-started

HAProxy setup for mysql load balancing

Available Servers
—————–
HAProxy: 10.0.0.100
mysql1 : 10.0.0.1
mysql2 : 10.0.0.2

Before setup, make sure that mysql servers are working fine and database replication is proper.

HAProxy Installation
——————–
Create 2 mysql users in any of the mysql server for HAProxy, one for checking status and another one with root privileges.

$ mysql -u root -p -e “INSERT INTO mysql.user (Host,User) values (‘10.0.0.100′,’haproxy_check’); FLUSH PRIVILEGES;”
$ mysql -u root -p -e “GRANT ALL PRIVILEGES ON *.* TO ‘haproxy_root’@’10.0.0.100’ IDENTIFIED BY ‘password’ WITH GRANT OPTION; FLUSH PRIVILEGES”

Install mysql client in haproxy server.

$ apt-get install mysql-client

Try accessing databases from haproxy server.

$ mysql -h 10.0.0.1 -u haproxy_root -p -e “SHOW DATABASES”

Install haproxy package.

$ apt-get install haproxy

Enable haproxy startup in init script.

$ sed -i “s/ENABLED=0/ENABLED=1/” /etc/default/haproxy

HAProxy Configuration
———————

Add the below entry to configuration file.

$ vi /etc/haproxy/haproxy.cfg

global
log 127.0.0.1 local0 notice
user haproxy
group haproxy

defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000

listen mysql-cluster
bind 127.0.0.1:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.0.0.1:3306 check
server mysql-2 10.0.0.2:3306 check

If you need to see statistics of load balancing, add the below entry too to the configuration.

listen 0.0.0.0:8080
mode http
stats enable
stats uri /
stats realm Strictly\ Private
stats auth A_Username:YourPassword
stats auth Another_User:passwd

Replace the usernames and passwords in “stats auth”. This will make HAProxy listen on port 8080 for HTTP requests and the statistics will be protected with HTTP Basic Authentication. So you can access stats at http://<Public IP of Load Balancer>:8080/

Once you’re done configuring start the HAProxy service.

$ service haproxy start

Use the mysql client to query HAProxy.

$ mysql -h 127.0.0.1 -u haproxy_root -p -e “SHOW DATABASES”

Ref : https://www.digitalocean.com/community/tutorials/how-to-use-haproxy-to-set-up-mysql-load-balancing–3

Hbase Installation

Installing Hbase
———————
$ su hadoop
$ wget http://www.eu.apache.org/dist/hbase/hbase-1.0.2/hbase-1.0.2-bin.tar.gz
$ tar -zvxf hbase-1.0.2-bin.tar.gz
$ mv hbase-1.0.2 hbase

Configuration
————–
$ cd /home/hadoop/hbase/conf/

I. Edit hbase-env.sh and set the java path
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/

II. Edit hbase-site.xml
Inset the following contents between <configuration></configuration> tags

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://<hadoop_master_IP>:9000/hbase</value>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
</property>

Start Hbase
————
start-hbase.sh

Multi-node Hadoop Cluster Setup

Installing Java 8 (Both Master and Slave)
—————–
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

root@test1:~# java -version
java version “1.8.0_66”
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

Configuring Java Environment (Both Master and Slave)
—————————-
$ sudo apt-get install oracle-java8-set-default

Creating hadoop User (Both master and Slave)
——————–
$ adduser hadoop
$ passwd hadoop

Switch to hadoop, generate RSA key and add to its authorized_keys (Both Master and Slave)
—————————————————————–
$ su – hadoop
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

$ ssh localhost
$ exit

Disable IPV6 – Hadoop does not support IPV6 (Both Master and Slave)
——————————————-
vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

sysctl -p

Download Hadoop (Both Master and Slave)
—————

$ cd /home/hadoop/
$ wget http://apache.claz.org/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
$ tar xzf hadoop-2.7.1.tar.gz
$ mv hadoop-2.7.1 hadoop
$ chown -R hadoop. hadoop

Configuration (Both Master and Slave)
————-

a. Edit the ~/.bashrc file and add contents as below

export HADOOP_HOME=/home/hadoop/hadoop
export HBASE_HOME=/home/hadoop/hbase
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HBASE_HOME/bin

After updating the ~./bashrc run

source ~/.bashrc

b. Edit hadoop configuration files(except hadoop-env.sh) and add the given contents between <configuration> </configuration> tags at the end of each file

$ cd $HADOOP_HOME/etc/hadoop

I. Edit core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://<hadoop_master_IP>:9000</value> //9000 will be set as hdfs port
</property>

II. Edit hdfs-site.xml

<property>
<name>dfs.replication</name>
<value>2</value> // Give the total number of slaves
</property>

<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>

III. Edit mapred-site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value> //mapreduce is handled by yarn in newer versions
</property>

IV. Edit  yarn-site.xml

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Now make the following changes.

Edit hadoop-env.sh and changed JAVA_HOME variable.
i.e. export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Edit yarn-env.sh and changed JAVA_HOME variable.
i.e.   JAVA_HOME=/usr/lib/jvm/java-8-oracle

Edit the file $HADOOP_HOME/etc/hadoop/slaves in master server and add the IP of master and slave
Edit the file $HADOOP_HOME/etc/hadoop/masters in slave server and add the IP address of master

Format the namenode (Only on Master)
——————-
$ hdfs namenode -format

Start the hadoop Cluster (Only on Master)
————————
start-dfs.sh
start-yarn.sh

Git : adding, merging and deleting branches

Creating branch and pushing changes.

First clone a repository to your local machine.

$ git clone test@<IP>:repo.git

You can see all branches using:

$ git branch

To create and switch to a new branch:

$ git checkout -b <mybranch>

This is shorthand for

$ git branch <mybranch>
$ git checkout <mybranch>

Now make changes to any file and add it.

$ git add .

Commit the changes.

$ git commit -m “commit_name”

Push changes to branch now.

$ git push origin <mybranch>

Updating and merging changes.

To update your local repository to the newest commit and to fetch and merge remote changes.

$ git pull # pulling details of new branch

$ git checkout <mybranch> # switching to new branch to see files

$ git merge origin/<mybranch> # merging files in branch to master

$ git add .

$ git commit -m <commit_name>

$ git push origin master

Deleting branch.

$ git checkout master # Switch to master first

$ git branch -d <mybranch> # Delete the branch

locate: can not stat () `/var/lib/mlocate/mlocate.db’

Got the below error while trying to use the locate command.

locate: can not stat () `/var/lib/mlocate/mlocate.db’: No such file or directory

The fix was pretty simple. Just ran the updatedb command. Locate started working just fine.

root@test1:~# updatedb
root@test1:~# locate jdk
/etc/profile.d/jdk.csh
/etc/profile.d/jdk.sh

Thanks.

 

Git Local Repository Setup

#Create git user account and add to appropriate group.

sudo adduser –shell $(which git-shell) –gecos ‘git version control’ –disabled-password git
sudo usermod -a -G www-data git
sudo usermod -a -G developers git

#Setup authorized_keys file for access and fix permissions.

sudo mkdir -p /home/git/.ssh
sudo touch /home/git/.ssh/authorized_keys
sudo chmod 600 /home/git/.ssh/authorized_keys
sudo chmod 700 /home/git/.ssh

#Copy the git-shell-commands to get limited shell access and correct permission.

sudo cp -r /usr/share/doc/git/contrib/git-shell-commands /home/git/
sudo chmod 750 /home/git/git-shell-commands/*
sudo chown -R git:git /home/git/

#Create and add your SSH key to the authorized key list.

ssh-keygen
cat ~/.ssh/id_rsa.pub | sudo tee -a /home/git/.ssh/authorized_keys
echo “AllowUsers git” | sudo tee -a /etc/ssh/sshd_config
sudo service ssh restart

#Create a location to store repositories.

sudo mkdir -p /home/repo
sudo chown -R git:www-data /home/repo

#Install gitweb and update the gitweb configuration file.

sudo apt-get install gitweb
sudo cp /etc/gitweb.conf /etc/gitweb.conf.$(date +%Y-%m-%d)
sudo sed -i ‘s/^\$projectroot.*/\$projectroot = \\”\\/home\\/repo\\”;/’ /etc/gitweb.conf
echo -e “\n\n# User additions” | sudo tee -a /etc/gitweb.conf
echo “\\$feature{‘blame’}{‘default’} = [1];” | sudo tee -a /etc/gitweb.conf
echo “\\$feature{‘search’}{‘default’} = [1];” | sudo tee -a /etc/gitweb.conf
echo “\\$feature{‘highlight’}{‘default’} = [1];” | sudo tee -a /etc/gitweb.conf
echo “\\$feature{‘grep’}{‘default’} = [1];” | sudo tee -a /etc/gitweb.conf

#Setup GitWeb executable in /home/git and use Kogakure’s gitweb-theme.

sudo mkdir /home/git/gitweb
sudo ln -s /usr/share/gitweb/gitweb.cgi /home/git/gitweb/index.cgi
git clone https://github.com/kogakure/gitweb-theme.git
sudo mv gitweb-theme /home/git/gitweb/static
sudo chown -R git:www-data /home/git/gitweb

#Edit Lighttpd configuration for localhost to include:

$HTTP[“url”] =~ “^/gitweb” {
server.document-root = “/home/git/”
server.indexfiles = (“index.cgi”)
cgi.assign = (“.cgi” => “”)
}

#Create a directory and initialize a bare repository.

cd /home/repo/helloworld.git
sudo git –bare init
echo “Hello World Repository. Testing system configuration” | sudo tee /home/repo/helloworld.git/description
echo “[gitweb]” | sudo tee -a /home/repo/helloworld.git/config
sudo chown -R git:www-data /home/repo/helloworld.git

sudo apt-get install lighttpd
/etc/init.d/lighttpd status
vim /etc/lighttpd/lighttpd.conf

#Clone the empty repository and add some code.

cd /home/git/gitweb/
git clone git@$(hostname):/home/repo/helloworld.git
cd helloworld

#Propose change to the working directory.

wget https://github.com/phi-jp/helloworld.cpp
git add helloworld.cpp
git config –global user.email “name@your_email.com”
git config –global user.name “your_name”

#Commit the changes made.

git commit -m “Initial commit with helloworld in C++”

#Push the changes to the remote repository.

git push origin master

Check the web interface now. The URL should be: http://localhost/gitweb and you should see the helloworld repository. To check the code-base, follow the tree link. For any new repository you want to create, follow the steps you did in “Create a directory and initialize a bare repository”.

Install memcached for CentOS6/CloudLinux

 

Step 1) Enable the EPEL repo for CentOS6.

Step 2) Install the memcached, memcached-devel, and alt-libmemcached (the alt- is only for CloudLinux 6).

Step 3) Setup the configuration:

/etc/sysconfig/memcached

Contents:

PORT=”11211″ USER=”memcached” MAXCONN=”256″ CACHESIZE=”1024″ OPTIONS=”-l 127.0.0.1″

The CACHESIZE is measured in megabytes so adjust accordingly.

Step 4) Enable via chkconfig, start service, and make sure it is running.

Step 5) Install memcache with

pecl install memcache pecl install memcached

Check the result php -m | grep memecache

ModSecurity: Audit log: Failed to create subdirectories: /var/asl/data/audit/nobody/

Noticed the below error in apache error log

[Wed Jan 07 11:19:54.786501 2015] [:error] [pid 24001:tid 140170514261760] [client 1.2.3.4] ModSecurity: Audit log: Failed to create subdirectories: /var/asl/data/audit/nobody/20150107/20150107-1119 (Permission denied) [hostname “test.com”] [uri “/index.php”] [unique_id “VKyJukgiKxYAAF3BqvIAAABA”]

Fix:

mkdir /var/asl /var/asl/data /var/asl/data/audit /var/asl/data/msa /var/asl/data/security

confirm which user apache uses to run     ps -ef | grep http

Normally it will be nobody.

chown -R nobody.nobody /var/asl/data/

/etc/init.d/httpd restart

Installing and Uninstalling cagefs

To install CageFS:

$ yum install cagefs
$ /usr/sbin/cagefsctl –init

That last command will create skeleton directory that might be around 7GB in size. If you don’t have enough disk space in /usr/share, use following commands to have cagefs-skeleton being placed in a different location:
$ mkdir /home/cagefs-skeleton
$ ln -s /home/cagefs-skeleton /usr/share/cagefs-skeleton

To uninstall CageFS,

start by disabling & removing all directories
$ /usr/sbin/cagefsctl –remove-all

That command will: Disable CageFS for all customers, unmount CageFS for all users, removes /usr/share/cagefs-skeleton & /var/cagefs directories. It will not remove /etc/cagefs directory

Remove CageFS RPM:
$ yum remove cagefs