Setup A Centralized Log Server Using Rsyslog on Ubuntu 16.04 LTS

Setup A Centralized Log Server Using Rsyslog on Ubuntu 16.04 LTS

Rsyslog Server:

# vim /etc/rsyslog.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# vim /etc/rsyslog.d/tmpl.conf

$template TmplAuth, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/client_logs/%HOSTNAME%/%PROGRAMNAME%.log"

authpriv.* ?TmplAuth
*.info;mail.none;authpriv.none;cron.none ?TmplMsg

# systemctl restart rsyslog

Rsyslog Client:

# vim /etc/rsyslog.conf

##RULES## 
*.* @192.168.1.200:514

Note: The @ symbol before the IP address tells rsyslog to use UDP to send the messages. Change this to @@ to use TCP.

# systemctl restart rsyslog

# logger -s " This is my Rsyslog client "

# logger --server 127.0.0.1 --port 9000 --udp --rfc3164 "my testing msg"

# tree /var/log/client_logs/

The following is a list of RFCs that define the Syslog protocol:

RFC 3195 Reliable Delivery for syslog
RFC 5424 The Syslog Protocol
RFC 5425 TLS Transport Mapping for Syslog
RFC 5426 Transmission of Syslog Messages over UDP
RFC 5427 Textual Conventions for Syslog Management
RFC 5848 Signed Syslog Messages
RFC 6012 Datagram Transport Layer Security (DTLS) Transport Mapping for Syslog

Reference:

http://yallalabs.com/linux/how-to-setup-a-centralized-log-server-using-rsyslog-on-ubuntu-16-04-lts/

https://success.trendmicro.com/solution/TP000086250-What-are-Syslog-Facilities-and-Levels

https://en.wikipedia.org/wiki/Syslog

https://www.elastic.co/blog/how-to-centralize-logs-with-rsyslog-logstash-and-elasticsearch-on-ubuntu-14-04

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-syslog.html

Install MySQL 5.7, Apache 2.4, PHP 7.1 on Ubuntu 16.04

Install MySQL 5.7, Apache 2.4, PHP 7.1 on Ubuntu 16.04

Install VMware tools:

VM > Guest > Install/Upgrade VMware Tools

# su -
# df -h
# cd /media/jun/VMware\ Tools/
# ls -la
# tar zxvf VMwareTools-9.4.0-1280544.tar.gz -C /tmp/
# cd /tmp
# ls
# cd vmware-tools-distrib/
# ls
# ./vmware-install.pl -d
# reboot

Note: For more info https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022525

Move Ubuntu launcher to the bottom:

# gsettings set com.canonical.Unity.Launcher launcher-position Bottom

Update the package repository:

# apt-get update

Upgrades packages with auto-handling of dependencies:

# apt-get dist-upgrade

or

# apt full-upgrade

Install SSH server:

# apt-get install openssh-server
# systemctl status sshd.service
# systemctl restart sshd.service

Compile and install the latest Git 2.20.1 from source code:

# apt-get install make gcc dh-autoreconf libcurl4-gnutls-dev libexpat1-dev gettext zlib1g-dev libssl-dev \
&& apt-get install curl \
&& cd /usr/local/src/ \
&& curl -L https://github.com/git/git/archive/v2.20.1.tar.gz -o git.tar.gz \
&& tar zxvf git.tar.gz \
&& cd git-2.20.1/ \
&& make configure \
&& ./configure --prefix=/usr \
&& make all \
&& make install

# git --version

git version 2.20.1

Install Git from ppa:

# add-apt-repository ppa:git-core/ppa
# apt-get update

# apt-cache policy git
# apt-cache madison git

# apt-get install git=1:2.11.0-2~ppa0~ubuntu16.04.1

# git --version

Compile and install the latest Vim 8:

# apt-get install libncurses5-dev python-dev python3-dev ruby-dev libperl-dev ruby-dev liblua5.3-dev exuberant-ctags cscope

// Fix liblua paths
# ln -s /usr/include/lua5.3 /usr/include/lua \
&& ln -s /usr/lib/x86_64-linux-gnu/liblua5.3.so /usr/local/lib/liblua.so

# cd /usr/local/src \
&& git clone https://github.com/vim/vim.git --depth 1 \
&& cd vim \
&& ./configure \
--prefix=/usr \
--with-features=huge \
--enable-multibyte \
--enable-pythoninterp \
--enable-python3interp \
--enable-rubyinterp \
--enable-perlinterp \
--enable-luainterp \
--enable-cscope \
&& make \
&& make install \
&& hash -r \
&& vim --version | head

Install MTA mail server:

# apt-get install postfix

Note: select "Internet site".

Note: If you need to reconfigure the postfix setting, run either one of the following:

# dpkg-reconfigure -plow postfix

or

# apt-get purge postfix

For other mail related packages:

# apt-get install mailutils

Install mail client:

# apt-get install bsd-mailx
# echo "test message" | mailx -s 'test subject' myemail@mydomain.com

For hexdump command:

# apt-get install bsdmainutils

# hexdump -c test.log

Install MySQL5.7:

# apt-cache policy mysql-server
# apt-cache search mysql-server
# apt-cache show mysql-server | less
# apt show mysql-server

# apt-get install mysql-server

# vim /etc/mysql/mysql.conf.d/mysqld.cnf

bind-address = 0.0.0.0

# vim ~/.my.cnf

[client]
host = localhost
port = 3306
user = root
password = MyPassword

# chmod 400 ~/.my.cnf

# mysql -e "SHOW variables WHERE variable_name REGEXP 'open_files_limit|table_open_cache|max_connections';"

+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| max_connections            | 151   |
| open_files_limit           | 1024  |
| table_open_cache           | 431   |
| table_open_cache_instances | 16    |
+----------------------------+-------+

Note: You will see the following error message in the error.log file if you did not change the open files limit:

[Warning] Changed limits: max_open_files: 1024 (requested 5000)
[Warning] Changed limits: table_open_cache: 431 (requested 2000)

# mkdir /etc/systemd/system/mysql.service.d
# vim /etc/systemd/system/mysql.service.d/override.conf

[Service]
#LimitNOFILE=infinity
LimitNOFILE=5000

#LimitMEMLOCK=infinity

# systemctl daemon-reload
# systemctl restart mysql

# mysql -e "SHOW variables WHERE variable_name REGEXP 'open_files_limit|table_open_cache|max_connections';"

+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| max_connections            | 151   |
| open_files_limit           | 5000  |
| table_open_cache           | 2000  |
| table_open_cache_instances | 16    |
+----------------------------+-------+

To check MySQL process's limit:

# cat /proc/$(pgrep mysqld$)/limits

Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15614                15614                processes
Max open files            5000                 5000                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15614                15614                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Note: https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html

Note: https://stackoverflow.com/questions/30901041/can-not-increase-max-open-files-for-mysql-max-connections-in-ubuntu-15

Note: https://serverfault.com/questions/821695/mysqld-service-for-systemd-failed-to-parse-resource-value-ignoring-40000-l

To move a MySQL data directory to another directory:

# mysql -e "SELECT @@datadir;"

+-----------------+
| @@datadir       |
+-----------------+
| /var/lib/mysql/ |
+-----------------+

# systemctl stop mysql
# systemctl status mysql

# vim /etc/mysql/mysql.conf.d/mysqld.cnf

datadir         = /home/mysql

# vim /etc/apparmor.d/tunables/alias

alias /var/lib/mysql/ -> /home/mysql/,

Note: We need to tell AppArmor to let MySQL write to the new directory by creating an alias between the default directory and the new location.

Note: If you skipped the AppArmor configuration step, you would see the following error message:

Job for mysql.service failed because the control process 
exited with error code. See "systemctl status mysql.service" 
and "journalctl -xe" for details.

# systemctl restart apparmor
# systemctl restart mysql

To move the existing to MySQL directory to /home:

# rsync -av /var/lib/mysql /home

Or, you can run the following commands to initialize the MySQL data directory:

# mkdir /home/mysql \
&& chown mysql:mysql /home/mysql \
&& chmod 700 /home/mysql \
&& mysqld --initialize-insecure

Note: This option is used to initialize a MySQL installation by creating the data directory and populating the tables in the mysql system database.

Note: If you use --initialize, the random initial password is stored at: tail -n 1 /var/log/mysql/error.log.

Note: You can also start mysqld with --skip-grant-tables to access the database and change the password.

# systemctl start mysql && systemctl status mysql

Login MySQL with the above commands if you initialized MySQL data directory with --initialize-insecure option.
# mysql -u root --skip-password
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'new_password';

Check the current MySQL data directory:

# mysql -e "SELECT @@datadir;"

+--------------+
| @@datadir    |
+--------------+
| /home/mysql/ |
+--------------+

To change the root password if you did not know the current root password:

# vim /root/tmp/mysql-init.txt

GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY 'YourPassWordHere' WITH GRANT OPTION;
GRANT SUPER ON *.* TO 'root'@'localhost';
FLUSH PRIVILEGES;

# mysqld --init-file=/root/tmp/mysql-init.txt

Use RAM-DISK for tmpdir:

# mysql -e "SHOW GLOBAL STATUS LIKE 'Created_tmp%tables';"
# mysql -e "SHOW GLOBAL VARIABLES LIKE '%table_size';"

# mkdir -p /mnt/ramdisk
# mount -t tmpfs -o size=512M tmpfs /mnt/ramdisk
# chown mysql:mysql /mnt/ramdisk

# id mysql

uid=123(mysql) gid=130(mysql) groups=130(mysql)

# vim /etc/fstab

tmpfs           /mnt/ramdisk     tmpfs   rw,uid=123,gid=130,mode=1770,size=512M    0       0

Note: You need to change the uid and gid of MySQL.

# mysql -e "SHOW GLOBAL VARIABLES LIKE 'tmpdir';"

# vim /etc/apparmor.d/local/usr.sbin.mysqld

/mnt/ramdisk rw,
owner /mnt/ramdisk/** rwkl,

Note: The first line gives read and write access to the directory, the second line gives read, write, lock(k) and link(l) access to all the files and the directories inside the directory owned by the mysql user.

# vim /etc/mysql/mysql.conf.d/mysqld.cnf

[mysqld]
tmpdir      = /mnt/ramdisk

# systemctl restart apparmor.service
# systemctl restart mysql.service

Install PHP7.1:

# command -v add-apt-repository >/dev/null 2>&1 \
|| { echo >&2 "add-apt-repository is not installed. I will install it for you"; apt-get install python-software-properties; }

# add-apt-repository -y ppa:ondrej/php
# apt-get update

# apt-cache policy php7.1

# apt-get install php7.1-fpm
# apt-get install php7.1-xml php7.1-curl php7.1-zip php7.1-gd php7.1-bcmath php7.1-intl php7.1-mbstring php7.1-mcrypt php7.1-mysql
# apt-get install php7.1-json php7.1-opcache
# apt-get install php-xdebug

# php -v

PHP 7.1.10-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Sep 29 2017 17:04:25) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.10-1+ubuntu16.04.1+deb.sury.org+1, Copyright (c) 1999-2017, by Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans

# vim /etc/php/7.1/fpm/pool.d/www.conf

;listen = /run/php/php7.1-fpm.sock
listen = 127.0.0.1:9000

Note: You can choose to use either a Unix socket (for local access only) or TCP socket (for the other server on the network to access).

Set up xdebug:

# vim /etc/php/7.1/fpm/php.ini

[Xdebug]
xdebug.default_enable=1
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_host=localhost
xdebug.remote_port=9009
xdebug.remote_log=/tmp/xdebug.log
xdebug.remote_connect_back=0
xdebug.remote_autostart=1
xdebug.remote_mode=req

xdebug.max_nesting_level=1000

xdebug.var_display_max_depth = 5
xdebug.var_display_max_children = 256
xdebug.var_display_max_data = 1024

# systemctl restart php7.1-fpm.service && systemctl status php7.1-fpm.service

# ss -an | grep :9000

tcp    LISTEN     0      128    127.0.0.1:9000                  *:*

# vim /etc/php/7.1/fpm/php.ini

date.timezone = America/Vancouver
display_errors = On
display_startup_errors = On
error_reporting = E_ALL
error_log = /var/log/php_errors.log

List all the installed PHP packages:

# dpkg -l | grep php| awk '{print $2}' |tr "\n" " "

Show the available package version:

# apt-cache search php
# apt-cache policy php

Install the specific package version:

# apt-get install php7=7.0+35ubuntu6

Note: You can look up old versions of packages at their site http://www.debian.org/distrib/packages

Install older version of PHP (PHP5.6):

# add-apt-repository ppa:ondrej/php

# apt-get update

# apt-get install php5.6-fpm

# apt-get install php5.6-gd php5.6-intl php5.6-json php5.6-mbstring php5.6-mcrypt php5.6-mysql php5.6-opcache php5.6-xml

# a2disconf php7.0-fpm.conf
# a2enconf php5.6-fpm.conf

# systemctl restart apache2.service

Install Apache2.4:

# command -v add-apt-repository >/dev/null 2>&1 \
|| { echo >&2 "add-apt-repository is not installed. I will install it for you"; apt-get install python-software-properties; }

# add-apt-repository -y ppa:ondrej/apache2
# apt-get update

# apt-cache policy apache2

# apt-get install apache2

Enable the following modules to talk to PHP:

# cat /etc/apache2/conf-available/php7.1-fpm.conf

# a2enmod proxy proxy_fcgi rewrite setenvif ssl
# a2enconf php7.1-fpm.conf

If your apache is talking to PHP through a TCP socket (127.0.0.1:9000) instead of a Unix socket (/run/php/php7.1-fpm.sock), you will need to modify the following line:

# vim /etc/apache2/conf-available/php7.1-fpm.conf

# Define a matching worker.
    # The part that is matched to the SetHandler is the part that
    # follows the pipe. If you need to distinguish, "localhost; can
    # be anything unique.
    <Proxy "fcgi://localhost/" enablereuse=on max=10>
    </Proxy>
    <FilesMatch ".+\.ph(ar|p|tml)$">
        #SetHandler "proxy:unix:/run/php/php7.1-fpm.sock|fcgi://localhost"
        SetHandler "proxy:fcgi://127.0.0.1:9000"
    </FilesMatch>

# apache2ctl configtest
# systemctl restart apache2 && systemctl status apache2

Install and enable the following Apache modules if you are connecting to PHP through a TCP socket (127.0.0.1:9000):

# apt-get install libapache2-mod-fastcgi
# a2enmod fastcgi rewrite setenvif

Edit apache2.conf:

# vim /etc/apache2/apache2.conf

AllowOverride All

Set up a virtual host:

# cd /etc/apache2/sites-available
# cp 000-default.conf mag2.local.conf
# vim mag2.local

Check the configuration:

# apache2ctl -V
# apache2ctl -t
# apache2ctl -M
# apache2ctl configtest

Enable the site:

# a2ensite mag2.local

Start MySQL, PHP, and Apache:

# systemctl restart mysql.service
# systemctl restart php7.0-fpm.service
# systemctl restart apache2.service

# ps auxww | grep -i mysql
# ps auxww | grep -i php-fpm
# ps auxww | grep -i apache2

Install PHPStorm:

# cd ~jun/Downloads/
# tar xf PhpStorm-*.tar.gz -C /opt/
# cd /opt/PhpStorm-163.10504.2/
# ./bin/phpstorm.sh

Generate a self-signed SSL certificate:

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -subj "/C=CA/ST=British Columbia/L=Vancouver/O=My Company Name/CN=erp.local" -keyout /etc/ssl/private/test.local.key -out /etc/ssl/certs/test.local.crt

Install node:

$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash

$ command -v nvm

nvm

$ nvm ls-remote
$ nvm install 8.9.3
$ nvm use 8.9.3
$ node -v
$ nvm ls

$ echo '{}' > package.json
$ npm install webpack eslint js-beautify --save-dev

Reference:

https://wiki.apache.org/httpd/PHP-FPM

http://httpd.apache.org/docs/2.4/mod/mod_proxy_fcgi.html

https://www.digitalocean.com/community/tutorials/how-to-move-a-mysql-data-directory-to-a-new-location-on-ubuntu-16-04

http://www.fromdual.com/mysql-tmpdir-on-ram-disk

http://www.victordodon.com/changing-mysql-tmpdir-in-ubuntu/

https://blog.remirepo.net/post/2014/03/28/PHP-FPM-and-HTTPD-2.4-improvement

Run multiple PHP-FPM versions on the same Ubuntu server

Run multiple PHP-FPM versions on the same Ubuntu server

# vim /etc/apache2/sites-enabled/symfony.local.conf

<VirtualHost *:80>
    ServerName symfony.local

    Include "conf-available/php7.2-fpm.conf"

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/symfony.local/curr/public

    <Directory /var/www/symfony.local/curr/web>
        AllowOverride All
    </Directory>
</VirtualHost>

Switch php versions on commandline ubuntu 16.04

Switch php versions on commandline ubuntu 16.04

# update-alternatives --set php /usr/bin/php7.1
# update-alternatives --set phar /usr/bin/phar7.1
# update-alternatives --set phar.phar /usr/bin/phar.phar7.1

Reference:

https://stackoverflow.com/questions/42619312/switch-php-versions-on-commandline-ubuntu-16-04

Build OpenJDK 8 on Ubuntu 14.04

# apt-get update \
&& apt-get install build-essential mercurial zip openjdk-7-jdk libX11-dev libxext-dev libxrender-dev libxtst-dev libxt-dev libcups2-dev libfreetype6-dev libasound2-dev ccache

# cd /usr/local/src \
&& hg clone http://hg.openjdk.java.net/jdk8u/jdk8u

# cd jdk8u/ \
&& bash ./get_source.sh

# bash ./configure -with-freetype-include=/usr/include/freetype2 -with-freetype-lib=/usr/lib/x86_64-linux-gnu/

# make all

# make install

Install and Configure Elasticsearch on Ubuntu 16.04

Install and Configure Elasticsearch on Ubuntu 16.04

Installing the Oracle JDK:

# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java9-installer

Multiple Java installations can be installed on one server. You can use the following command to configure which version is the default for use:

# update-alternatives --config java

Install Elasticsearch:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

# apt-get update && apt-get install elasticsearch

# systemctl daemon-reload
# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service

# vim ~/.bashrc

export JAVA_HOME="/usr/lib/jvm/java-9-oracle"

# echo $JAVA_HOME

/usr/lib/jvm/java-9-oracle

To tail the journal:

# journalctl -f

To list journal entries for the elasticsearch service:

# journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

# journalctl --unit elasticsearch --since "2017-06-17 15:09:11"

To test Elasticsearch is running:

# curl http://localhost:9200/

Configure Elasticsearch:

# cd /etc/elasticsearch

To check the cluster health:

# curl -X GET 'localhost:9200/_cat/health?v&pretty'

Get a list of nodes in the cluster:

# curl -X GET 'localhost:9200/_cat/nodes?v'

Create the index named "customer":

# curl -X PUT 'localhost:9200/customer?pretty'

List all indices:

# curl -X GET 'localhost:9200/_cat/indices?v'

Add a customer document into the customer index, "external type, with an ID of 1:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "John Doe"
}
'

Retrieve the document we just added:

# curl -X GET 'localhost:9200/customer/external/1?pretty&pretty'

Replace the document:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "Jane Doe"
}
'

Update the document:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"doc": { "name": "Jane Doe", "age": 20 }
}
'

Updates can also be performed by using simple scripts:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"script" : "ctx._source.age += 5"
}
'

Note: ctx._source refers to the current source document that is about to be updated.

Bulk insert the multiple documents:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"index":{"_id":"1"}}
{"name": "John Doe 3" }
{"index":{"_id":"2"}}
{"name": "Jane Doe 3" }
'

Note: the existing documents will be replaced instead of updated.

Update the first document and delete the second document in one bulk operation:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"update":{"_id":"1"}}
{"doc": { "name": "John Doe becomes Jane Doe" } }
{"delete":{"_id":"2"}}
'

Delete a document:

# curl -X DELETE 'localhost:9200/customer/external/1?pretty&pretty'

Delete the customer index:

# curl -X DELETE 'localhost:9200/customer?pretty&pretty'

Load data into the cluster:

# curl -H "Content-Type: application/json" -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"
# curl 'localhost:9200/_cat/indices?v'

Note: the sample data can be generated from http://www.json-generator.com/

Query all documents in the index:

# curl -X GET 'localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty&pretty'

Alternative query method:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" }
]
}
'

Returns all accounts containing the term "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match": { "address": "mill lane" } }
}
'

Returns all accounts containing the phrase "mill lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_phrase": { "address": "mill lane" } }
}
'

Returns all accounts containing "mill" and "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts containing "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts that contain neither "mill" nor "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must_not": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/_delete_an_index.html

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-16-04

gpg: agent_genkey failed: Permission denied

$ gpg2 --gen-key

// On Ubuntu

gpg: agent_genkey failed: Permission denied
Key generation failed: Permission denied

// On CentOS

gpg: cancelled by user
gpg: Key generation canceled.

Solution:

$ ls -la $(tty)

crw--w----. 1 someone tty 136, 9 May 17 20:47 /dev/pts/9

$ sudo chown MyUserName /dev/pts/9

$ gpg2 --gen-key

Changing Ubuntu full Screen Resolution in a Hyper-V VM

Changing Ubuntu full Screen Resolution in a Hyper-V VM

# vi /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash video=hyperv_fb:1920×1080"

# update-grub
# sync; reboot

Reference:

https://blogs.msdn.microsoft.com/virtual_pc_guy/2014/09/19/changing-ubuntu-screen-resolution-in-a-hyper-v-vm/

Check Ubuntu version

Check Ubuntu version

# cat /etc/os-release

# cat /etc/lsb-release

# lsb_release -a

Ubuntu Software (Software Center) Crash - Ubuntu 16.04

# apt-get update
# apt-get upgrade gnome-software

or

# rm ~/.local/share/gnome-software/*

Reference:

http://askubuntu.com/questions/783398/ubuntu-software-software-center-crash-ubuntu-16-04#

PhpStorm or IntelliJ suddenly hangs / freezes / keyboard not responsive/ unresponsive while editing in Linux / Ubuntu

PhpStorm or IntelliJ suddenly hangs / freezes / keyboard not responsive/ unresponsive while editing in Linux / Ubuntu

Try:

# ibus restart

or

Help > Diagnostic > Change memory settings

Reference:

https://blog.cppse.nl/phpstorm-intellij-hangs-freezes-unresponsive-keyboard

http://stackoverflow.com/questions/26524923/phpstorm-freezes-very-often

https://stackoverflow.com/questions/25546022/phpstorm-webstorm-increase-memory-to-more-than-512mb

How to create desktop shortcut (launcher icon) for PhpStorm?

Use IDE to create launcher. Open Tools -> Create Desktop Entry.

apt-get install does not ask for confirmation

apt-get install does not ask for confirmation

apt-get will not ask for the confirmation if every package (including dependencies) is provided on the command line; apt-get will ask for the confirmation if the installation required installing a package not specified on the command line.

Use dry run instead before installing:

# apt-get -s tree

How do I disable the blinking cursor in gnome-terminal?

How do I disable the blinking cursor in gnome-terminal?

Method 1:

System settings > Keyboard > uncheck "Cursor blink in text field"

Method 2:

# gsettings set org.gnome.desktop.interface cursor-blink false

Reference:

http://askubuntu.com/questions/49606/how-do-i-disable-the-blinking-cursor-in-gnome-terminal

Move Ubuntu launcher to the bottom

Move Ubuntu launcher to the bottom

Run this command in Terminal:

# gsettings set com.canonical.Unity.Launcher launcher-position Bottom

bash shell tab auto-completion and history command completion

If you have this in your /etc/inputrc or ~/.inputrc, you will no longer have to hit the <tab> key twice to produce a list of all possible completions. A single <tab> will suffice. This setting is highly recommended.

# vi ~/.bash_profile
#!/bin/bash
source ~/.bashrc # get aliases

# vi ~/.bashrc
### alias
alias ls='ls --color=auto'
alias ll='ls -l'
alias h='history'

### ls with color
export CLICOLOR=1 # Use colors (if possible)
export LSCOLORS="ExGxFxdxCxDxDxBxBxExEx"

### display history command with date and time
export HISTTIMEFORMAT="%m/%d/%y %T "

### Prompt
PS1='\e[0;32m\u@\h \w #\e[m '

#######
# Note: on Ubuntu, xterm-256color may be in different place, try this:
# find /lib/terminfo /usr/share/terminfo -name "*256*"
# Note: tmux respects screen-256color
#######
#if [ -e /usr/share/terminfo/x/xterm-256color ]; then
# export TERM='xterm-256color'
#else
# export TERM='xterm-color'
#fi

### Make bash check its window size after a process completes
#shopt -s checkwinsize

Bash tab auto-completion and history completion by using ~/.inputrc:
# vi ~/.inputrc
### enable filename tab auto-completion
set show-all-if-ambiguous on
set show-all-if-unmodified on

### if you don't want case-sensitivity
#set completion-ignore-case on

### bash history completion to complete what's already on the line
### arrow up
"\e[A": history-search-backward
### arrow down
"\e[B": history-search-forward

Note: A bit background explanation:
Bash is using readline to handle the prompt. ~/.inputrc is the configuration file for readline.
Read the bash manual for more information about readline. There you can also find more history related readline commands.

To get the escape codes for the arrow keys you can do the following:

1. Start cat in a terminal (just cat, no further arguments).
2. Type keys on keyboard, you will get things like ^[[A for up arrow and ^[[B for down arrow.
3. Replace ^[ with \e.

Note: Normally, Up and Down are bound to the Readline functions previous-history and next-history respectively. I prefer to bind PgUp/PgDn to these functions, instead of displacing the normal operation of Up/Down.

# vi ~/.inputrc
"\e[5~": history-search-backward
"\e[6~": history-search-forward

Note: use "bind -p" or "bind -P" to get a list of bash key bindings.

Note: .inputrc will get read whenever you start a new bash process.

Note: After you modify ~/.inputrc, restart your shell or use Ctrl+X, Ctrl+R to tell it to re-read ~/.inputrc.

Note: please notice that ~/.inputrc and /etc/inputrc are NOT bash scripts, so please don’t try to source them. So, restart your Terminal.

Bash tab auto-completion and history completion by using ~/.bashrc:
# vi ~/.bashrc
### bash history completion to complete what's already on the line
bind '"\e[A": history-search-backward'
bind '"\e[B": history-search-forward'

From the bash manpage:

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When a login shell exits, bash reads and executes commands from the file ~/.bash_logout, if it exists.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

Thus, if you want to get the same behavior for both login shells and interactive non-login shells, you should put all of your commands in either .bashrc or .bash_profile, and then have the other file source the first one.

===================================================
When working with Linux, Unix, and Mac OS X, I always forget which bash config file to edit when I want to set my PATH and other environmental variables for my shell. Should you edit .bash_profile or .bashrc in your home directory?
You can put configurations in either file, and you can create either if it doesn’t exist. But why two different files? What is the difference?
According to the bash man page, .bash_profile is executed for login shells, while .bashrc is executed for interactive non-login shells.

What is a login or non-login shell?

When you login (type username and password) via console, either sitting at the machine, or remotely via ssh: .bash_profile is executed to configure your shell before the initial command prompt.

But, if you’ve already logged into your machine and open a new terminal window (xterm) inside Gnome or KDE, then .bashrc is executed before the window command prompt. .bashrc is also run when you start a new bash instance by typing /bin/bash in a terminal.

Why two different files?

Say, you’d like to print some lengthy diagnostic information about your machine each time you login (load average, memory usage, current users, etc). You only want to see it on login, so you only want to place this in your .bash_profile. If you put it in your .bashrc, you’d see it every time you open a new terminal window.

Mac OS X — an exception

An exception to the terminal window guidelines is Mac OS X’s Terminal.app, which runs a login shell by default for each new terminal window, calling .bash_profile instead of .bashrc. Other GUI terminal emulators may do the same, but most tend not to.

Recommendation

Most of the time you don’t want to maintain two separate config files for login and non-login shells — when you set a PATH, you want it to apply to both. You can fix this by sourcing .bashrc from your .bash_profile file, then putting PATH and common settings in .bashrc.

To do this, add the following lines to .bash_profile:

if [ -f ~/.bashrc ]; then
   source ~/.bashrc
fi

Now when you login to your machine from a console .bashrc will be called.

Reference:
http://www.caliban.org/bash/index.shtml

http://stackoverflow.com/questions/902946/about-bash-profile-bashrc-and-where-should-alias-be-written-in

http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html

http://askubuntu.com/questions/59846/bash-history-search-partial-up-arrow

Checking your Ubuntu Version

# lsb_release -a

Show the List of Installed Packages on Ubuntu or Debian

Show a list of all the currently installed packages:
# dpkg --get-selections

Find the locations of the files within a package from the list by using the dpkg -L command, such as:
# dpkg -L tmux

how to start ssh server service

start: Unknown job: ssh
Before installing SSH server, it's a good idea to update apt to ensure we have the latest packages.

# sudo apt-get update
# sudo apt-get install openssh-server
# rehash

Note: Bash doesn't have the rehash builtin command. Use "hash -r" instead.

Start SSH service:
# sudo /etc/init.d/ssh start

OR

# sudo service ssh start

If you are using the latest version of Ubuntu such as 12.04 LTS or 13.04+, you need to use upstart job based commands:
# sudo stop ssh
# sudo start ssh
# sudo restart ssh
# sudo status ssh

To prevent login password prompt slowly:
// add following line on SSH server
# vi /etc/ssh/sshd_config
UseDNS no

// add following line on SSH client
# vi /etc/ssh/ssh_config
UseDNS no

List all services with the status for each of them:
# initctl list
# initctl help

Note initctl list shell command lists the contents of /etc/init rather than the suggested dbus-send command.

You can list all of the upstart jobs with by querying upstart over dbus:
# dbus-send --print-reply --system --dest=com.ubuntu.Upstart \
/com/ubuntu/Upstart com.ubuntu.Upstart0_6.GetAllJobs

You may have to change 0_6 to reflect the version of upstart you have; this command works on my lucid install.

Reference:
http://askubuntu.com/questions/218/command-to-list-services-that-start-on-startup

How to Install and Use tmux on Ubuntu 13.04

Before installing tmux, it's a good idea to update apt to ensure we have the latest packages.
# sudo apt-get update

Then install tmux:
# sudo apt-get install tmux

Run tmux:
# tmux

Run existing tmux:
# tmux a -d

tmux.conf configuration file:
As per dpkg -L tmux which shows you what files the package installed, there is no default tmux.conf included in the package. /etc/tmux.conf is just a location that you may use (only makes sense with multiple users using tmux) that will be evaluated before ~/.tmux.conf. You have to create your own .conf file

# dpkg -L
# ls /usr/share/doc/tmux/examples

# vi ~/.tmux.conf

### Note: key meaning
### C- means ctrl-, so C-x is ctrl-x.
### M- means meta (generally left-alt or escape)-, so M-x is left-alt-x.

### Set the prefix to ^A.
### Note: the default prefix is Ctrl-b.
unbind C-b
set -g prefix ^A
bind a send-prefix

### Set the maximum number of lines held in window history.
set -g history-limit 5000

### terminal 256 colors.
### As the tmux manual suggests: for tmux to work correctly, this must be set to "screen" or a derivative of it.
### Note: modify following line and set it to the terminal supported by your system (run cat /etc/termcap | egrep 'screen-256color|xterm-256color' to verify), and uncomment it.
###set -g default-terminal screen-256color
set -g default-terminal xterm-256color

### Instructs tmux to expect UTF-8 sequences to appear in this window.
setw -g utf8 on

### Instruct tmux to treat top-bit-set characters in the status-left and status-right strings as UTF-8;
set -g status-utf8 on

### date format: hostname weekday month day, hour:minute
set -g status-right '#h %a %b %d, %H:%M'

### Set status line background colour.
set -g status-bg black

### Set status line foreground colour.
set -g status-fg yellow

### Set status line background colour for the currently active window.
setw -g window-status-current-bg magenta

### Set status line foreground colour for the currently active window.
setw -g window-status-current-fg white

### Reload tmux.conf configuration file
### Note: alternative way to reload the configuration file:
### Method 1: Run from command line: tmux source-file ~/.tmux.conf
### Method 2: In any tmux sessions: [prefix Ctrl-b] : source-file /usr/local/etc/tmux.conf
### Method 3: bind the source-file command with a key like following line, then you type: ctrl-b-r
bind r source-file /usr/local/etc/tmux.conf

### [START] vi- and vim-like bindings
### split windows like vim
### vim's definition of a horizontal/vertical split is reversed from tmux's
bind s split-window -v
bind v split-window -h

### move around panes with hjkl, as one would in vim after pressing ctrl-w
bind h select-pane -L
bind j select-pane -D
bind k select-pane -U
bind l select-pane -R

### resize panes like vim
### feel free to change the "5" to however many lines you want to resize by, only
### one at a time can be slow
bind < resize-pane -L 5
bind > resize-pane -R 5
bind - resize-pane -D 5
bind + resize-pane -U 5

### bind : to command-prompt like vim
### this is the default in tmux already
bind : command-prompt

### vi-style controls for copy mode
setw -g mode-keys vi
### [END] vi- and vim-like bindings

Reference:
http://askubuntu.com/questions/192401/where-is-the-default-tmux-conf-file-located