Simple life, Complicated mind

Sunday, June 18, 2017

Install and Configure Elasticsearch on Ubuntu 16.04

Install and Configure Elasticsearch on Ubuntu 16.04

Installing the Oracle JDK:

# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java9-installer

Multiple Java installations can be installed on one server. You can use the following command to configure which version is the default for use:

# update-alternatives --config java

Install Elasticsearch:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

# apt-get update && apt-get install elasticsearch

# systemctl daemon-reload
# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service

# vim ~/.bashrc

export JAVA_HOME="/usr/lib/jvm/java-9-oracle"

# echo $JAVA_HOME

/usr/lib/jvm/java-9-oracle

To tail the journal:

# journalctl -f

To list journal entries for the elasticsearch service:

# journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

# journalctl --unit elasticsearch --since "2017-06-17 15:09:11"

To test Elasticsearch is running:

# curl http://localhost:9200/

Configure Elasticsearch:

# cd /etc/elasticsearch

To check the cluster health:

# curl -X GET 'localhost:9200/_cat/health?v&pretty'

Get a list of nodes in the cluster:

# curl -X GET 'localhost:9200/_cat/nodes?v'

Create the index named "customer":

# curl -X PUT 'localhost:9200/customer?pretty'

List all indices:

# curl -X GET 'localhost:9200/_cat/indices?v'

Add a customer document into the customer index, "external type, with an ID of 1:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "John Doe"
}
'

Retrieve the document we just added:

# curl -X GET 'localhost:9200/customer/external/1?pretty&pretty'

Replace the document:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "Jane Doe"
}
'

Update the document:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"doc": { "name": "Jane Doe", "age": 20 }
}
'

Updates can also be performed by using simple scripts:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"script" : "ctx._source.age += 5"
}
'

Note: ctx._source refers to the current source document that is about to be updated.

Bulk insert the multiple documents:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"index":{"_id":"1"}}
{"name": "John Doe 3" }
{"index":{"_id":"2"}}
{"name": "Jane Doe 3" }
'

Note: the existing documents will be replaced instead of updated.

Update the first document and delete the second document in one bulk operation:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"update":{"_id":"1"}}
{"doc": { "name": "John Doe becomes Jane Doe" } }
{"delete":{"_id":"2"}}
'

Delete a document:

# curl -X DELETE 'localhost:9200/customer/external/1?pretty&pretty'

Delete the customer index:

# curl -X DELETE 'localhost:9200/customer?pretty&pretty'

Load data into the cluster:

# curl -H "Content-Type: application/json" -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"
# curl 'localhost:9200/_cat/indices?v'

Note: the sample data can be generated from http://www.json-generator.com/

Query all documents in the index:

# curl -X GET 'localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty&pretty'

Alternative query method:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" }
]
}
'

Returns all accounts containing the term "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match": { "address": "mill lane" } }
}
'

Returns all accounts containing the phrase "mill lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_phrase": { "address": "mill lane" } }
}
'

Returns all accounts containing "mill" and "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts containing "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts that contain neither "mill" nor "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must_not": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/_delete_an_index.html

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-16-04

Saturday, June 17, 2017

Install MySQL, Apache, PHP on Ubuntu 16.04

Install MySQL, Apache, PHP on Ubuntu 16.04

Install VMware tools:

VM > Guest > Install/Upgrade VMware Tools

# su -
# df -h
# cd /media/jun/VMware\ Tools/
# ls -la
# tar zxvf VMwareTools-9.4.0-1280544.tar.gz -C /tmp/
# cd /tmp
# ls
# cd vmware-tools-distrib/
# ls
# ./vmware-install.pl -d
# reboot

Note: For more info https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022525

Move Ubuntu launcher to the bottom:

# gsettings set com.canonical.Unity.Launcher launcher-position Bottom

Update the package repository:

# apt-get update

Install SSH server:

# apt-get install openssh-server
# systemctl status sshd.service
# systemctl restart sshd.service

Compile and install the latest Git 2.13.1:

# apt-get install dh-autoreconf libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev
# apt-get install curl
# cd /usr/local/src/
# curl -L https://github.com/git/git/archive/v2.13.1.tar.gz -o git.tar.gz
# tar zxvf git.tar.gz
# cd git-2.13.1/
# make configure
# ./configure --prefix=/usr
# make all
# make install

Compile and install the latest Vim 8:

# apt-get install libncurses5-dev python-dev python3-dev ruby-dev libperl-dev ruby-dev liblua5.3-dev exuberant-ctags cscope

// Fix liblua paths
# ln -s /usr/include/lua5.3 /usr/include/lua
# ln -s /usr/lib/x86_64-linux-gnu/liblua5.3.so /usr/local/lib/liblua.so

# cd /usr/local/src
# git clone https://github.com/vim/vim.git

# cd vim
# ./configure --prefix=/usr --with-features=huge --enable-multibyte --enable-pythoninterp -enable-python3interp --enable-rubyinterp --enable-perlinterp --enable-luainterp --enable-cscope
# make
# make install

# hash -r
# vim --version

Install MTA mail server:

# apt-get install postfix

Note: select "Internet site".

Note: If you need to reconfigure the postfix setting, run either one of the following:

# dpkg-reconfigure -plow postfix

or

# apt-get purge postfix

For other mail related packages:

# apt-get install mailutils

Install mail client:

# apt-get install bsd-mailx
# echo "test message" | mailx -s 'test subject' myemail@mydomain.com

Install Git:

# add-apt-repository ppa:git-core/ppa
# apt-get update

# apt-cache policy git
# apt-cache madison git

# apt-get install git=1:2.11.0-2~ppa0~ubuntu16.04.1

# git --version

Install MySQL5.7:

# apt-get install mysql-server

Install Apache2.4:

# apt-get install apache2

Enable the following Apache modules if you are connecting to PHP through a unix socket (/run/php/php7.0-fpm.sock):

# a2enmod proxy proxy_fcgi rewrite

Install and enable the following Apache modules if you are connecting to PHP through a TCP socket (127.0.0.1:9000):

# apt-get install libapache2-mod-fastcgi
# a2enmod fastcgi rewrite

Edit apache2.conf:

# vim /etc/apache2/apache2.conf

AllowOverride All

Set up a virtual host:

# cd /etc/apache2/sites-available
# cp 000-default.conf mag2.local.conf
# vim mag2.local

Check the configuration:

# apache2ctl -V
# apache2ctl -t
# apache2ctl -M
# apache2ctl configtest

Enable the site:

# a2ensite mag2.local

Install PHP7.0:

# apt-get install php7.0-fpm php-xdebug
# apt-get install php7.0-xml php7.0-curl php7.0-zip php7.0-gd php7.0-bcmath php7.0-intl php7.0-mbstring php7.0-mcrypt php7.0-mysql
# apt-get install php7.0-json php7.0-opcache

# vim /etc/php/7.0/fpm/php.ini

date.timezone = America/Vancouver
display_errors = On
display_startup_errors = On
error_reporting = E_ALL
error_log = /var/log/php_errors.log

# a2enconf php7.0-fpm.conf

Start MySQL, PHP, and Apache:

# systemctl restart mysql.service
# systemctl restart php7.0-fpm.service
# systemctl restart apache2.service

# ps auxww | grep -i mysql
# ps auxww | grep -i php-fpm
# ps auxww | grep -i apache2

List all the installed PHP packages:

# dpkg -l | grep php| awk '{print $2}' |tr "\n" " "

Show the available package version:

# apt-cache search php
# apt-cache policy php

Install the specific package version:

# apt-get install php7=7.0+35ubuntu6

Note: You can look up old versions of packages at their site http://www.debian.org/distrib/packages

Install older version of PHP (PHP5.6):

# add-apt-repository ppa:ondrej/php

# apt-get update

# apt-get install php5.6-fpm

# apt-get install php5.6-gd php5.6-intl php5.6-json php5.6-mbstring php5.6-mcrypt php5.6-mysql php5.6-opcache php5.6-xml

# a2disconf php7.0-fpm.conf
# a2enconf php5.6-fpm.conf

# systemctl restart apache2.service

Install PHPStorm:

# cd ~jun/Downloads/
# tar xf PhpStorm-*.tar.gz -C /opt/
# cd /opt/PhpStorm-163.10504.2/
# ./bin/phpstorm.sh

Generate a self-signed SSL certificate:

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -subj "/C=CA/ST=British Columbia/L=Vancouver/O=My Company Name/CN=erp.local" -keyout /etc/ssl/private/test.local.key -out /etc/ssl/certs/test.local.crt

Thursday, May 25, 2017

Redirect both stdout and stderr to a file:

Redirect both stdout and stderr to a file:
# ls &> filename

Note: This operator is now functional, as of Bash 4, final release.

Redirect stderr to stdout (&1), and then redirect stdout to a file:
# ls > filename 2>&1

Note: 2>&1 redirects file descriptor 2 (stderr) to file descriptor 1 (stdout).

Reference:

https://stackoverflow.com/questions/7526971/how-to-redirect-both-stdout-and-stderr-to-a-file

Wednesday, May 17, 2017

gpg: agent_genkey failed: Permission denied

$ gpg2 --gen-key

// On Ubuntu
gpg: agent_genkey failed: Permission denied
Key generation failed: Permission denied

// On CentOS
gpg: cancelled by user
gpg: Key generation canceled.

Solution:

$ ls -la $(tty)

crw--w----. 1 someone tty 136, 9 May 17 20:47 /dev/pts/9

$ sudo chown MyUserName /dev/pts/9

$ gpg2 --gen-key

Monday, May 15, 2017

sign_and_send_pubkey: signing failed: agent refused operation

sign_and_send_pubkey: signing failed: agent refused operation

Try to add the private key identities to the authentication agent:

# ssh-add

To see a list of fingerprints of all identities:

# ssh-add -l

Reference:

https://askubuntu.com/questions/762541/ubuntu-16-04-ssh-sign-and-send-pubkey-signing-failed-agent-refused-operation

Friday, May 12, 2017

Use rpmbuild to build a custom RPM package on CentOS 7

Use rpmbuild to build a custom RPM package on CentOS 7

Install the necessary tools:

# yum install rpm-build rpmdevtools rpmlint

Create the necessary directories:

# rpmdev-setuptree

// or create them manually
# mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,tmp}

# ls -l ~/rpmbuild

Set up the temporary directory:

# vim ~/.rpmmacros

%_tmppath %(echo $HOME)/rpmbuild/tmp

Prepare some program/files for the RPM package:

# cd ~/rpmbuild
# mkdir -p ~/rpmbuild/SOURCES/test-repo-1.0.0
# echo -e '#!/bin/sh\necho "Hello World"' > ~/rpmbuild/SOURCES/test-repo-1.0.0/hello
# chmod +x ~/rpmbuild/SOURCES/test-repo-1.0.0/hello
# tar zcvf ~/rpmbuild/SOURCES/test-repo-1.0.0.tar.gz -C ~/rpmbuild/SOURCES/ test-repo-1.0.0/

Edit the spec file:

# vim ~/rpmbuild/SPECS/test-repo.spec

%define NAME test-repo
%define VERSION 1.0.0
%define INSTALL_DIR /usr/local
%define OWNER root

# Disable creating debuginfo RPM
%define debug_package %{nil}

# Strip debug symbols (possibly making the program not funtion properly)
#%define __strip /bin/true

Name: %NAME
Version: %VERSION

# Refer to:
# https://fedoraproject.org/wiki/How_to_create_an_RPM_package
# https://fedoraproject.org/wiki/Packaging:DistTag?rd=Packaging/DistTag
Release: 1%{?dist}

Summary: Package foo summary
Source: %NAME-%VERSION.tar.gz
License: MIT

# Refer to:
# cat /usr/share/doc/rpm-4.11.3/GROUPS
Group: Development/Tools

%description
Package foo description.

%prep

# Start uncompressing
%setup -q

%build

%install

# Turn this on to find out the available environment variables.
#env

install -m 0755 -d ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME
#mkdir -p ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME
cp -r * ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME/

#make install DESTDIR=$RPM_BUILD_ROOT

%clean
rm -rf ${RPM_BUILD_DIR}/*
rm -rf ${RPM_BUILD_ROOT}
rm -rf %_tmppath/*

%post
echo . .
echo .Wring some descripton here to show after package installation!.

%files
%INSTALL_DIR
%defattr(-,%OWNER,%OWNER)

%changelog

Validate the spec:

# cd ~/rpmbuild
# rpmlint SPECS/test-repo.spec

SPECS/test-repo.spec:10: W: macro-in-comment %define
SPECS/test-repo.spec:42: W: macro-in-comment %INSTALL_DIR
SPECS/test-repo.spec:42: W: macro-in-comment %NAME
SPECS/test-repo.spec: W: invalid-url Source0: test-repo-1.0.0.tar.gz
0 packages and 1 specfiles checked; 0 errors, 4 warnings.

Note: Do not worry if you see the warnings.

Build RPM without the source:

# cd ~/rpmbuild
# rpmbuild -v -bb SPECS/test-repo.spec

Build RPM with the source:

# rpmbuild -v -ba SPECS/test-repo.spec

Install the RPM:

# rpm -ivh RPMS/x86_64/test-repo-1.0.0-1.el7.centos.x86_64.rpm

Check the RPM installation:

# ls -la /usr/local/test-repo/

List the installed RPM:

# rpm -qa | grep test-repo

test-repo-1.0.0-1.el7.centos.x86_64

List the files in the installed RPM:

# rpm -q --filesbypkg test-repo

test-repo                 /usr/local
test-repo                 /usr/local/test-repo
test-repo                 /usr/local/test-repo/hello

To find out which RPM a file belongs to:

# rpm -qf /usr/local/test-repo/hello

test-repo-1.0.0-1.el7.centos.x86_64

Remove the RPM:

# rpm -e test-repo

Reference:

https://fedoraproject.org/wiki/How_to_create_an_RPM_package

Wednesday, May 10, 2017

gpg --gen-key hangs at gaining enough entropy on CentOS 7

gpg --gen-key hangs at gaining enough entropy on CentOS 7

Solution 1:

Install random number generator:

# yum install rng-tools

# systemctl enable rngd

# systemctl restart rngd

Solution 2:

# dd if=/dev/sda of=/dev/zero

Reference:

https://serverfault.com/questions/471412/gpg-gen-key-hangs-at-gaining-enough-entropy-on-centos-6#