Tuesday, October 17, 2017

Installing ClickHouse

Check System requirements

This is not a cross-platform system. It requires Linux Ubuntu Precise (12.04) or newer, x86_64 architecture with SSE 4.2 instruction set.
To test for SSE 4.2 support, do

# grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

We recommend using Ubuntu Trusty or Ubuntu Xenial or Ubuntu Precise.
The terminal must use UTF-8 encoding (the default in Ubuntu).

Installation

For testing and development, the system can be installed on a single server or on a desktop computer.

Installing from packages

In /etc/apt/sources.list (or in a separate /etc/apt/sources.list.d/clickhouse.list file), add the repository:

On Ubuntu Trusty (14.04):

deb http://repo.yandex.ru/clickhouse/trusty stable main

For other Ubuntu versions, replace trusty to xenial or precise.

Then run:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E0C56BD4 # optional
sudo apt-get update
sudo apt-get install clickhouse-client clickhouse-server-common

You can also download and install packages manually from here:

Trusty Xenial Precise

ClickHouse contains access restriction settings. They are located in the 'users.xml' file (next to 'config.xml').

By default, access is allowed from everywhere for the default user without a password. See 'user/default/networks'. For more information, see the section Configuration Files

Installing from source

Build following the instructions in build.md

You can compile packages and install them. You can also use programs without installing packages.

Client: src/dbms/src/Client/
Server: src/dbms/src/Server/

For the server, create a catalog with data, such as:

/opt/clickhouse/data/default/
/opt/clickhouse/metadata/default/

(Configured in the server config.)
Run 'chown' for the desired user.

Note the path to logs in the server config (src/dbms/src/Server/config.xml).

Other methods of installation

The Docker image is located here: ClickHouse Server

There is Gentoo overlay located here: ClickHouse Overlay

Launch

To start the server (as a daemon), run:

# sudo service clickhouse-server start

View the logs in the catalog

# /var/log/clickhouse-server/

If the server doesn't start, check the configurations in the file

# vim /etc/clickhouse-server/config.xml

You can also launch the server from the console:

# clickhouse-server --config-file=/etc/clickhouse-server/config.xml

In this case, the log will be printed to the console, which is convenient during development. If the configuration file is in the current directory, you don't need to specify the '--config-file' parameter. By default, it uses './config.xml'.

You can use the command-line client to connect to the server:

# clickhouse-client

The default parameters indicate connecting with localhost:9000 on behalf of the user 'default' without a password.

The client can be used for connecting to a remote server. For example:

clickhouse-client --host=example.com

For more information, see the section "Command-line client".

Checking the system:

root@localhost# ./clickhouse-client
ClickHouse client version 0.0.18749.
Connecting to localhost:9000.
Connected to ClickHouse server version 0.0.18749.
SELECT 1
SELECT 1
┌─1─┐
│ 1 │
└───┘
1 rows in set. Elapsed: 0.003 sec.

Congratulations, it works!

Increase Logical Volumes in Linux

I Have created the for video the reference link Click Here

Fist Check partitions by #fdisk -l

Partition the new disk

We now need to partition the new /dev/sdb disk so that it can be used, this is done by using fdisk.

# fdisk /dev/sdb

This should provide us with the below prompt, the inputs I have entered in are shown in bold.

‘n’ was selected for adding a new partition.

‘p’ is then selected as we are making a primary partition.

Command action
e extended
p primary partition (1-4)
p

As this is a new disk, we do not yet have any partitions on it so we will use partition 1 here.

Partition number (1-4): 1

Next, we press the enter key twice, as by default the first and last cylinders of the unallocated space should be correct.

First cylinder (1-2610, default 1): "enter"
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): "enter"
Using default value 2610

‘t’ is selected to change to a partitions system ID, in this case, we change to ’1′ automatically as this is currently our only partition.

Command (m for help): t
Selected partition 1

The hex code ’8e’ was entered as this is the code for a Linux LVM which is what we want this partition to be, as we will be joining it with the original Linux LVM which is currently using /dev/sda5.

Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

‘w’ is used to write the table to disk and exit, all changes that have
been done will be saved and then you will be the exit from fdisk.

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

By using “fdisk -l” now you will be able to see that /dev/sdb1 is listed, this is the new partition created on our newly added /dev/sdb disk and it is currently using all 20gb of space.

Increasing the logical volume

Next, we will use the pvcreate command to create a physical volume for later use by the LVM. In this case, the physical volume will be our new /dev/sdb1 partition.

#pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created

Now we need to confirm the name of the current volume group using the vgdisplay command. The name will vary depending on your setup, for me, it is the name of my test server. vgdisplay provides plenty of information on the volume group, I have only shown the name and the current size of it for this example.

# vgdisplay
--- Volume group ---
VG Name motadata
VG Size 13.97 GiB

Now using the vgextend command, we extend the ‘motadata’ volume group by adding in the physical volume of /dev/sdb1 which we created using the pvcreate command just before.

# vgextend motadata /dev/sdb1
Volume group "motadata" successfully extended

Using the pvscan command we scan all disks for physical volumes, this should confirm the original /dev/sda5 partition and the newly created physical volume /dev/sdb1

[root@centosdemo ~]# pvscan
PV /dev/sda5 VG centos lvm2 [9.31 GiB / 0 free]
PV /dev/sda2 VG motadata lvm2 [13.97 GiB / 0 free]
PV /dev/sdb1 VG motadata lvm2 [5.00 GiB / 5.00 GiB free]
Total: 3 [28.28 GiB] / in use: 3 [28.28 GiB] / in no VG: 0 [0 ]

Next, we need to increase the logical volume with the lvextend command (rather than the physical volume which we have already done). This means we will be taking our original logical volume and extending it over our new disk/partition/physical volume of /dev/sdb1.

Firstly confirm the name of the logical volume using lvdisplay. The name will vary depending on your setup.

[root@centosdemo ~]# lvdisplay
--- Logical volume ---
LV Path /dev/motadata/motadata
LV Name motadata
VG Name motadata
LV UUID NVvHXi-MvaQ-2zOB-vHVA-9poh-N6MD-PtD4gw
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2016-09-30 17:31:03 +0530
LV Status available
# open 1
LV Size 13.97 GiB
Current LE 3577
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos
LV UUID kQPUTY-RhUp-dxQV-NcDg-2R9x-MedH-wGdlPS
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2016-09-30 17:31:02 +0530
LV Status available
# open 1
LV Size 9.31 GiB
Current LE 2384
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0

The logical volume is then extended using the lvextend command. We are extending the original logical volume of /dev/motadata/motadata over the newer /dev/sdb1

[root@centosdemo ~]# lvextend /dev/motadata/motadata /dev/sdb1
Size of logical volume motadata/motadata changed from 13.97 GiB (3577 extents) to 18.97 GiB (4856 extents).
Logical volume motadata successfully resized.

If you like you can then run vgdisplay and lvdisplay again to confirm the size of the volume group and logical volume respectively, I have done this and I now have the following.

However, if you run a “df” command to see available disk space it will not have changed yet as there is one final step, we need to resize the file system using the resize2fs command in order to make use of this space.

[root@centosdemo ~]# xfs_growfs /dev/motadata/motadata
meta-data=/dev/mapper/motadata-motadata isize=256 agcount=4, agsize=915712 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=3662848, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 3662848 to 4972544

Alternatively, if you’re facing any error running above command you can also use below command “resize2fs /dev/motadata/motadata”.

Rather than resizing the file system manually, you could instead use the -r option of the lvextend command which will automatically resize the file system to make use of the additional disk space.

The resize took a minute or so to complete (it will depend on the disk speed and size), running the “df” command now shows the correct disk space for /dev/mapper/motadata-motadata.

Enable SSH for root user on Ubuntu

To enable root Login, first assign the password for root user.

#sudo passwd root -/enter new password for root user here.

Directly copy and paste this below lines in command prompt, to set ssh to allow root logins.

sed -i 's/Authentication:/#Authentication:/g' /etc/ssh/sshd_config sed -i 's/LoginGraceTime/#LoginGraceTime/g' /etc/ssh/sshd_config sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config sed -i 's/PermitRootLogin without-password/#PermitRootLogin without-password/g' /etc/ssh/sshd_config sed -i 's/StrictModes yes/#StrictModes yes' /etc/ssh/sshd_config service ssh restart

Or you can also configure it by manually by editing configuration file as mentioned below.

#vim /etc/ssh/sshd_config

# Authentication:
#LoginGraceTime 120
PermitRootLogin yes
#PermitRootLogin without-password
#StrictModes yes

Comment lines as mentioned above.

Save and exit the file.

Now restart ssh service

#service ssh restart

Check from the client machine.

Configure sflow in fortigate firewall

Login to Firewall CLI_Console

FG100E # config system sflow

FG100E (sflow) # set collector-ip 172.16.10.115

FG100E (sflow) # set collector-port 4738

FG100E (sflow) # end

FG100E # config system interface

FG100E (interface) # edit wan2

FG100E (wan2) # set sflow-sampler enable

FG100E (wan2) # set sample-rate 100

FG100E (wan2) # set sample-direction both

FG100E (wan2) # set polling-interval 60

FG100E (wan2) # next

FG100E (interface) # end

FG100E # cfg save
Unknown action 0

To remove configuration.

FG100E # config system sflow

FG100E (sflow) # unset collector-ip

FG100E (sflow) # unset collector-port

FG100E (sflow) # end

FG100E # config system interface

FG100E (interface) # edit wan2

FG100E (wan2) # unset sflow-sampler enable

FG100E (wan2) # unset sample-rate 100

FG100E (wan2) # unset sample-direction both

FG100E (wan2) # unset polling-interval 60

FG100E (wan2) # next

FG100E (interface) # end

FG100E # cfg save
Unknown action 0

Access Windows Shares from Linux Machine SSH Console

Mount Windows share using mount command

Use command line tool called smbclient or you can mount windows shares the mount command. Another option is use GUI tools. Please refer previous articles about access windows share from Linux

Here we are going to access windows share :\\192.168.1.1\share

# mkdir -p /mnt/share
# mount -t smbfs -o username=administrator,password=password //192.168.1.1/share /mnt/win

Note: If you face error related to file systems, smbfs filesystem not found, you can use cifs instead of smbfs.

# cd /mnt/share
# ls -l

For the share //192.168.1.1/share to be automatically mounted at every system start (after reboot), insert an option in the file /etc/fstab:

# vi /etc/fstab
Append following line (written in a single line)

//192.168.1.1/share /mnt/share smbfs auto,gid=users,fmask=0664,dmask=0775,iocharset=iso8859-15, credentials=/etc/sambapasswords 0 0

Next create the password file /etc/sambapasswords:

# vi /etc/sambapasswords
Now add following content:

username = Administrator
password = password

Save and close the file. Make sure only root can access your file:

# chown 0.0 /etc/sambapasswords
# chmod 600 /etc/sambapasswords

System Securities with Linux and Virtualization.!!