Install and Configure Elasticsearch on Ubuntu 16.04

Install and Configure Elasticsearch on Ubuntu 16.04

Installing the Oracle JDK:

# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java9-installer

Multiple Java installations can be installed on one server. You can use the following command to configure which version is the default for use:

# update-alternatives --config java

Install Elasticsearch:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

# apt-get update && apt-get install elasticsearch

# systemctl daemon-reload
# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service

# vim ~/.bashrc

export JAVA_HOME="/usr/lib/jvm/java-9-oracle"

# echo $JAVA_HOME

/usr/lib/jvm/java-9-oracle

To tail the journal:

# journalctl -f

To list journal entries for the elasticsearch service:

# journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

# journalctl --unit elasticsearch --since "2017-06-17 15:09:11"

To test Elasticsearch is running:

# curl http://localhost:9200/

Configure Elasticsearch:

# cd /etc/elasticsearch

To check the cluster health:

# curl -X GET 'localhost:9200/_cat/health?v&pretty'

Get a list of nodes in the cluster:

# curl -X GET 'localhost:9200/_cat/nodes?v'

Create the index named "customer":

# curl -X PUT 'localhost:9200/customer?pretty'

List all indices:

# curl -X GET 'localhost:9200/_cat/indices?v'

Add a customer document into the customer index, "external type, with an ID of 1:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "John Doe"
}
'

Retrieve the document we just added:

# curl -X GET 'localhost:9200/customer/external/1?pretty&pretty'

Replace the document:

# curl -X PUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"name": "Jane Doe"
}
'

Update the document:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"doc": { "name": "Jane Doe", "age": 20 }
}
'

Updates can also be performed by using simple scripts:

# curl -X POST 'localhost:9200/customer/external/1/_update?pretty&pretty' -H 'Content-Type: application/json' -d'
{
"script" : "ctx._source.age += 5"
}
'

Note: ctx._source refers to the current source document that is about to be updated.

Bulk insert the multiple documents:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"index":{"_id":"1"}}
{"name": "John Doe 3" }
{"index":{"_id":"2"}}
{"name": "Jane Doe 3" }
'

Note: the existing documents will be replaced instead of updated.

Update the first document and delete the second document in one bulk operation:

# curl -X POST 'localhost:9200/customer/external/_bulk?pretty&pretty' -H 'Content-Type: application/json' -d'
{"update":{"_id":"1"}}
{"doc": { "name": "John Doe becomes Jane Doe" } }
{"delete":{"_id":"2"}}
'

Delete a document:

# curl -X DELETE 'localhost:9200/customer/external/1?pretty&pretty'

Delete the customer index:

# curl -X DELETE 'localhost:9200/customer?pretty&pretty'

Load data into the cluster:

# curl -H "Content-Type: application/json" -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"
# curl 'localhost:9200/_cat/indices?v'

Note: the sample data can be generated from http://www.json-generator.com/

Query all documents in the index:

# curl -X GET 'localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty&pretty'

Alternative query method:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" }
]
}
'

Returns all accounts containing the term "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match": { "address": "mill lane" } }
}
'

Returns all accounts containing the phrase "mill lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": { "match_phrase": { "address": "mill lane" } }
}
'

Returns all accounts containing "mill" and "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts containing "mill" or "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Returns all accounts that contain neither "mill" nor "lane" in the address:

# curl -X GET 'localhost:9200/bank/_search?pretty' -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must_not": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'

Reference:

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/_delete_an_index.html

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-16-04

Redirect both stdout and stderr to a file:

Redirect both stdout and stderr to a file:
# ls &> filename

Note: This operator is now functional, as of Bash 4, final release.

Redirect stderr to stdout (&1), and then redirect stdout to a file:
# ls > filename 2>&1

Note: 2>&1 redirects file descriptor 2 (stderr) to file descriptor 1 (stdout).

Reference:

https://stackoverflow.com/questions/7526971/how-to-redirect-both-stdout-and-stderr-to-a-file

gpg: agent_genkey failed: Permission denied

$ gpg2 --gen-key

// On Ubuntu

gpg: agent_genkey failed: Permission denied
Key generation failed: Permission denied

// On CentOS

gpg: cancelled by user
gpg: Key generation canceled.

Solution:

$ ls -la $(tty)

crw--w----. 1 someone tty 136, 9 May 17 20:47 /dev/pts/9

$ sudo chown MyUserName /dev/pts/9

$ gpg2 --gen-key

sign_and_send_pubkey: signing failed: agent refused operation

sign_and_send_pubkey: signing failed: agent refused operation

Try to add the private key identities to the authentication agent:

# ssh-add

To see a list of fingerprints of all identities:

# ssh-add -l

Reference:

https://askubuntu.com/questions/762541/ubuntu-16-04-ssh-sign-and-send-pubkey-signing-failed-agent-refused-operation

Use rpmbuild to build a custom RPM package on CentOS 7

Use rpmbuild to build a custom RPM package on CentOS 7

Install the necessary tools:

# yum install rpm-build rpmdevtools rpmlint

Create the necessary directories:

# rpmdev-setuptree

// or create them manually
# mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,tmp}

# ls -l ~/rpmbuild

Set up the temporary directory:

# vim ~/.rpmmacros

%_tmppath %(echo $HOME)/rpmbuild/tmp

Prepare some program/files for the RPM package:

# cd ~/rpmbuild
# mkdir -p ~/rpmbuild/SOURCES/test-repo-1.0.0
# echo -e '#!/bin/sh\necho "Hello World"' > ~/rpmbuild/SOURCES/test-repo-1.0.0/hello
# chmod +x ~/rpmbuild/SOURCES/test-repo-1.0.0/hello
# tar zcvf ~/rpmbuild/SOURCES/test-repo-1.0.0.tar.gz -C ~/rpmbuild/SOURCES/ test-repo-1.0.0/

Edit the spec file:

# vim ~/rpmbuild/SPECS/test-repo.spec

%define NAME test-repo
%define VERSION 1.0.0
%define INSTALL_DIR /usr/local
%define OWNER root

# Disable creating debuginfo RPM
%define debug_package %{nil}

# Strip debug symbols (possibly making the program not funtion properly)
#%define __strip /bin/true

Name: %NAME
Version: %VERSION

# Refer to:
# https://fedoraproject.org/wiki/How_to_create_an_RPM_package
# https://fedoraproject.org/wiki/Packaging:DistTag?rd=Packaging/DistTag
Release: 1%{?dist}

Summary: Package foo summary
Source: %NAME-%VERSION.tar.gz
License: MIT

# Refer to:
# cat /usr/share/doc/rpm-4.11.3/GROUPS
Group: Development/Tools

%description
Package foo description.

%prep

# Start uncompressing
%setup -q

%build

%install

# Turn this on to find out the available environment variables.
#env

install -m 0755 -d ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME
#mkdir -p ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME
cp -r * ${RPM_BUILD_ROOT}%INSTALL_DIR/%NAME/

#make install DESTDIR=$RPM_BUILD_ROOT

%clean
rm -rf ${RPM_BUILD_DIR}/*
rm -rf ${RPM_BUILD_ROOT}
rm -rf %_tmppath/*

%post
echo . .
echo .Wring some descripton here to show after package installation!.

%files
%INSTALL_DIR
%defattr(-,%OWNER,%OWNER)

%changelog

Validate the spec:

# cd ~/rpmbuild
# rpmlint SPECS/test-repo.spec

SPECS/test-repo.spec:10: W: macro-in-comment %define
SPECS/test-repo.spec:42: W: macro-in-comment %INSTALL_DIR
SPECS/test-repo.spec:42: W: macro-in-comment %NAME
SPECS/test-repo.spec: W: invalid-url Source0: test-repo-1.0.0.tar.gz
0 packages and 1 specfiles checked; 0 errors, 4 warnings.

Note: Do not worry if you see the warnings.

Build RPM without the source:

# cd ~/rpmbuild
# rpmbuild -v -bb SPECS/test-repo.spec

Build RPM with the source:

# rpmbuild -v -ba SPECS/test-repo.spec

Install the RPM:

# rpm -ivh RPMS/x86_64/test-repo-1.0.0-1.el7.centos.x86_64.rpm

Check the RPM installation:

# ls -la /usr/local/test-repo/

List the installed RPM:

# rpm -qa | grep test-repo

test-repo-1.0.0-1.el7.centos.x86_64

List the files in the installed RPM:

# rpm -q --filesbypkg test-repo

test-repo                 /usr/local
test-repo                 /usr/local/test-repo
test-repo                 /usr/local/test-repo/hello

To find out which RPM a file belongs to:

# rpm -qf /usr/local/test-repo/hello

test-repo-1.0.0-1.el7.centos.x86_64

Remove the RPM:

# rpm -e test-repo

Reference:

https://fedoraproject.org/wiki/How_to_create_an_RPM_package

gpg --gen-key hangs at gaining enough entropy on CentOS 7

gpg --gen-key hangs at gaining enough entropy on CentOS 7

Solution 1:

Install random number generator:

# yum install rng-tools

# systemctl enable rngd

# systemctl restart rngd

Solution 2:

# dd if=/dev/sda of=/dev/zero

Reference:

https://serverfault.com/questions/471412/gpg-gen-key-hangs-at-gaining-enough-entropy-on-centos-6#

Changing Ubuntu full Screen Resolution in a Hyper-V VM

Changing Ubuntu full Screen Resolution in a Hyper-V VM

# vi /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash video=hyperv_fb:1920×1080"

# update-grub
# sync; reboot

Reference:

https://blogs.msdn.microsoft.com/virtual_pc_guy/2014/09/19/changing-ubuntu-screen-resolution-in-a-hyper-v-vm/