davidyin@minecraft:~$ sudo apt install ncdu
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  ncdu
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 50.7 kB of archives.
After this operation, 117 kB of additional disk space will be used.
Get:1 http://ca.archive.ubuntu.com/ubuntu noble/universe amd64 ncdu amd64 1.19-0                                                                                                                                                             .1 [50.7 kB]
Fetched 50.7 kB in 0s (124 kB/s)
Selecting previously unselected package ncdu.
(Reading database ... 128087 files and directories currently installed.)
Preparing to unpack .../ncdu_1.19-0.1_amd64.deb ...
Unpacking ncdu (1.19-0.1) ...
Setting up ncdu (1.19-0.1) ...
Processing triggers for man-db (2.12.0-4build2) ...
Scanning processes...
Scanning candidates...
Scanning linux images...

Pending kernel upgrade!
Running kernel version:
  6.8.0-45-generic
Diagnostics:
  The currently running kernel version is not the expected kernel version 6.8.0-48-generic.

Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting.

Restarting services...

Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

Then, run the command ncdu in the terminal window.

The program will scan the files and folders under the current prompt directory. The old school command window shows the space usage of the folders.

Now, I can use the arrow key to navigate. Press enter to go into the folder.

Press Q to exit.

It is very easy to use. I like the simple interface and quick view.

The post Check the folder size on Ubuntu Terminal Window appeared first on David Yin's Blog.

I have a VPS with Nginx as a web server and also a lot of websites on it. It is on Digital Ocean.

Here is the VPS basic information.

• 1GB VPS
• Ubuntu Linux 20.04LTS
• Premium-Intel, 1 cores
• Nginx 1.8.0

All sites use one access.log file. The following code is in the http block of /etc/nginx/nginx.conf file.

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

All access logs look like below:

Now it is time to show you how to use GoAccess to Generate Reports with Multiple Nginx Log files.

Install GoAccess

The GoAccess in the Ubuntu repo is too old. Version 1.3, released about 2016. So I decided to use the goaccess’s own repository as below to install the latest version 1.5.5.

$ wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
$ echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg] https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/goaccess.list
$ sudo apt-get update
$ sudo apt-get install goaccess

Modify the Config file of GoAccess

GoAccess config file is located at /etc/goaccess/

sudo nano /etc/goaccess/goaccess.conf

Add the following code or modify the code inside of the goaccess.conf file.

time-format %H:%M:%S

date-format %d/%b/%Y

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

geoip-database /var/GeoIP/dbip-city-lite-2022-02.mmdb

db-path /var/goaccess/cache

persist true

restore true

I downloaded the mmdb format GeoIP2 IP database from https://db-ip.com/db/download/ip-to-city-lite
Decompress it and put it at /var/GeoIP/

Then I made a folder at /var/goaccess/cache

Run GoAccess to generate the report

I would like to get a static report.

sudo goaccess /var/log/nginx/access.log -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The report looks like below:

Due to the log is not just one file. The log files are generated by date and also compressed.

Two log files are not compressed.

• access.log
• access.log.1

The older log files are compressed.

• access.log.2.gz
• access.log.3.gz
• access.log.4.gz
• access.log.5.gz

I want to generate all the access log files. So the command is changed to the way which can phrase multiple log files.

sudo zcat /var/log/nginx/access.log.*.gz | sudo goaccess /var/log/nginx/access.log /var/log/nginx/access.log.1 - -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The sample command above uses mywebsite.com. Please replace it with your own sites.

At the last, I add the command below into Cron jobs. Schedule it every hour.

sudo goaccess /var/log/nginx/access.log -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The post Use GoAccess 1.5.5 to Generate Report with Multiple Nginx Log Files appeared first on David Yin's Blog.

Yesterday I SSH to this VM and run the apt update and apt upgrade, found the error of the key signing of the gitlab repository.

The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F

Later I found the GPG key used to sign GitLab package repositories’ metadata is changed on Apr. 6, 2020. vi

OK, just update this key signing first.

Follow the command below to update the key.

# Download the new key
curl https://packages.gitlab.com/gpg.key -o /tmp/omnibus_gitlab_gpg.key

# Import the key
## Debian/Ubuntu/Raspbian
$ sudo apt-key add /tmp/omnibus_gitlab_gpg.key

# CentOS/OpenSUSE/SLES
$ sudo rpm --import /tmp/omnibus_gitlab_gpg.key

Then I run the apt update and apt upgrade commands to check the update.

I got the following error.

It said I can not upgrade my GitLab 12.9.2 to the latest 13.3.2. Because it found I am upgrading from major version 12 to major version 13.  It gives me a link to upgrade documentation.

https://docs.gitlab.com/ce/policy/maintenance.html#upgrade-recommendations

I studied it and decide on the path to upgrade.

My current version 12.9.2 ->  12.9.10 -> 12.10.14 -> 13.0.12 -> 13.1.8 -> 13.2.6 -> 13.3.2

I have to upgrade my GitLab step by step, from one version to another version. Until the latest one. It took me more than an hour to do all these upgrades. Each one has a 700+ MB upgrade package.  Downloading and installation take time.

The command to upgrade with a specific version. I use 13.1.8 as a sample version number.

sudo apt upgrade gitlab-ce=13.1.8-ce.0

Here is a link of GitLab release lists.

The post Upgrade Gitlab CE from 12.9.2 to 13.3.2 appeared first on David Yin's Blog.

Router:

D-Link DIR-880L, Firmware: DD-WRT v3.0-r42617 std (03/05/20)

LAN IP: 192.168.1.1

Server:

A VM on my DELL R710 machine.

OS: Ubuntu 20.04 LTS

LAN IP: 192.168.1.153

I would like to set up the Router Syslog to a remote server at the server.

At the DD-WRT web interface, find Service > Service, scroll down and find the system log section.

The Syslogd default setting is Disable. Now change it to Enable, and input the IP address of my server, 192.168.1.153. If you have different IP, use your own.

Then Save and Apply Settings.

On the Server, SSH to it. Enter the command as below:

sudo nano /etc/rsyslog.d/local-enable-ports.conf

Paste the following content into it and save and exit.

# enable TCP and UDP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

$ModLoad imudp
$UDPServerRun 514

This is to enable the ports for the remote log.

sudo nano /etc/rsyslog.d/10-custom.conf

Then paste the following content, save and exit.

#save messages from D-Link Dir-880L
if $fromhost-ip == '192.168.1.1' then /var/log/dd-wrt.log
& stop

After the above changes, I restart the Server, and the router in order.

I use webmin syslog tool to view the log file at /var/log/dd-wrt.log

The post Setup the DD-WRT router syslog to a remote server appeared first on David Yin's Blog.

There is no Running Processes information on the home page or System Information page.

Then, I google it and find the solution on the Webmin/Bugs page.

I did the fix as below:

1) SSH to the Ubuntu Server.

2) Backup the old file.

cd /usr/share/webmin/proc/

sudo mv linux-lib.pl linux-lib.pl.1.941

3) Add the new linux-lib.pl

sudo nano linux-lib.pl

Add the content from the link below into it and save it.

linux-lib.pl

4) Done. Check the webmin panel.

And If check the details of the running processes. It shows the list of the processes instead of the empty page.

The post Fix the problem of No info on the running processes, Webmin 1.941, Ubuntu 20.04LTS appeared first on David Yin's Blog.

I installed the Ubuntu 18.04 on a VM with 1CPU, 1024MB memory and 16GB storage room.

After the installation. Start the Ubuntu server and ssh to it.

Run the following command to keep it all updated.

sudo apt update
sudo apt upgrade

There is about 254 packages need to be upgraded.

Now it is time to start the installation.

Install Nginx web server, Mariadb server, php 7.2 support.

sudo apt install nginx
sudo apt install mariadb-server
sudo mysql-secure-installation
sudo apt install php7.2 php7.2-fpm php7.2-cli php7.2-common php7.2-mbstring php7.2-gd php7.2-intl php7.2-xml php7.2-mysql php7.2-zip php7.2-json php7.2-curl

First, regarding nginx.

davidyin@php-mdb:~$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0
libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip
libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common
nginx-core
Suggested packages:
libgd-tools fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0
libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip
libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter
libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx
nginx-common nginx-core
0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,461 kB of archives.
After this operation, 8,210 kB of additional disk space will be used.
Do you want to continue? [Y/n]

Regarding MariaDB installation

davidyin@php-mdb:~$ sudo apt install mariadb-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
galera-3 libaio1 libcgi-fast-perl libcgi-pm-perl libconfig-inifiles-perl
libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libio-html-perl libjemalloc1
liblwp-mediatypes-perl libmysqlclient20 libterm-readkey-perl
libtimedate-perl liburi-perl mariadb-client-10.1 mariadb-client-core-10.1
mariadb-common mariadb-server-10.1 mariadb-server-core-10.1 mysql-common
socat
Suggested packages:
libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl
libdata-dump-perl libipc-sharedcache-perl libwww-perl mailx mariadb-test
tinyca
The following NEW packages will be installed:
galera-3 libaio1 libcgi-fast-perl libcgi-pm-perl libconfig-inifiles-perl
libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libio-html-perl libjemalloc1
liblwp-mediatypes-perl libmysqlclient20 libterm-readkey-perl
libtimedate-perl liburi-perl mariadb-client-10.1 mariadb-client-core-10.1
mariadb-common mariadb-server mariadb-server-10.1 mariadb-server-core-10.1
mysql-common socat
0 upgraded, 29 newly installed, 0 to remove and 0 not upgraded.
Need to get 24.2 MB of archives.
After this operation, 184 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Add root user password to MariaDB

davidyin@php-mdb:~$ sudo mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n]
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n]
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n]
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n]
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n]
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
davidyin@php-mdb:~$

Install php7.2-fpm

davidyin@php-mdb:~$ sudo apt install php7.2 php7.2-fpm php7.2-cli php7.2-common php7.2-mbstring php7.2-gd php7.2-intl php7.2-xml php7.2-mysql php7.2-zip php7.2-json php7.2-curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libsodium23 libzip4 php-common php7.2-opcache php7.2-readline
Suggested packages:
  php-pear
The following NEW packages will be installed:
  libsodium23 libzip4 php-common php7.2 php7.2-cli php7.2-common php7.2-curl
  php7.2-fpm php7.2-gd php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql
  php7.2-opcache php7.2-readline php7.2-xml php7.2-zip
0 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.
Need to get 5,015 kB of archives.
After this operation, 21.3 MB of additional disk space will be used.
Do you want to continue? [Y/n]

After that, fix the potential problem of connection with phpmyadmin

Setup the Nginx site

cd /etc/nginx/conf.d/
sudo nano mysite.conf

Paste the these code into the file and save it.
Run the following command to test the Nginx configuration and restart the Nginx service.

sudo nginx -t
sudo service nginx restart

Add a phpinfo page to test

cd ~
mkdir mysite.com
cd mysite.com
nano phpinfo.php

Add the following php code.

phpinfo();

In the browser, enter the address http://mysite.com/phpinfo.php

Install mdbtools etc.

sudo apt install mdbtools unixodbc odbc-mdbtools unixodbc-bin php7.2-odbc
sudo service php7.2-fpm restart

Then I upload the a Microsoft Access MDB file, sample.mdb and a mdb.php file.
This database file has a table, pt_users, which has two fields of user’s first name and last name. I will retrieve them.
The content of mdb.php as below:

$db = './sample.mdb';

$dbName  =  $db  ; 
$driver  =  'MDBTools'; 

$dbh  =  new  PDO("odbc:Driver=$driver;DBQ=$dbName;");
	
// Table with column num, name test 
   $sql  =  "SELECT userfirstname, userlastname FROM pt_users" ;  // The rules are the same as above 
   $sth  =  $dbh ->  prepare($sql); 
   $sth ->  execute(); 
   // Bind to each column 
   $sth ->  bindColumn(1,$fname); 
   $sth ->  bindColumn(2,$lname);

   // Because of binding to the column, only the judgment result will be returned to $ flg 
   $values  =  [];  // Array to assign the value 
   while (  $flg  =  $sth ->  fetch(PDO::FETCH_BOUND)){ 
             $values[]  =  [$fname,$lname ]; 
	   	echo $fname,' :  ',$lname,'
';
   }	

I got blank page and check the Nginx error log file at /var/log/nginx/error.log

2019/11/15 06:55:53 [error] 28826#28826: *7 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught PDOException: SQLSTATE[01000] SQLDriverConnect: 0 [unixODBC][Driver Manager]Can't open cursor lib '/etc/libodbccr.so' : file no$
Stack trace:
#0 /home/davidyin/mysite.com/mdb.php(18): PDO->__construct('odbc:Driver=MDB...')
#1 {main}
thrown in /home/davidyin/mysite.com/mdb.php on line 18" while reading response header from upstream, client: 192.168.1.3, server: mysite.com, request: "GET /mdb.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", $

It said missing libodbccr.so.
Run the following command.

cd /usr/lib/x86_64-linux-gnu
sudo ln libodbccr.so.2 libodbccr.so

Then Reload the http://mysite.com/mdb.php
Got the correct result. Done.

At last, the environment:

• Ubuntu 18.04.3 LTS
• Nginx 1.14.0
• php 7.2.24
• MariaDB 10.1.41

The post Make the PHP support Microsoft Access MDB file on Ubuntu 18.04 appeared first on David Yin's Blog.

So the final decision is to migrate the Gitlab from the old server Ubuntu 16.04 to a new server with  Ubuntu 18.04.

This is how to do it.

First, I install the new server on my Proxmox VE environment with Ubuntu 18.04. 3CPU Cores and 4GB memory and 64GB storage. Meanwhile, the old server still uses Ubuntu 16.04. And the Gitlab version that I use is 12.4.2

On the Old Server

1.  Shutdown Gitlab service

   sudo gitlab-ctl stop unicorn

   sudo gitlab-ctl stop sidekiq

2. Backup Gitlab on the old server

   sudo gitlab-rake gitlab:bakcup:create

3. Create a folder named gitlab-old

   mkdir ~/gitlab-old

4. Copy the backup file to the folder ~/gitlab-old

   sudo cp ~/backups/xxxxxxxx_gitlab_backup.tar ~/gitlab-old

5. Copy the Gitlab configuration files

   sudo cp /etc/gitlab/gitlab.rb ~/gitlab-old

   sudo cp /etc/gitlab/gitlab-secrets.json ~/gitlab-old

   sudo cp -R /etc/gitlab/ssl ~/gitlab-old

Then all the important files are under folder ~/gitlab-old

My configuration of Gitlab changed the backup folder to ~/backups. The default path is /var/opt/gitlab/backups/

You can change it on the “Backup Settings” section in the file gitlab.rb

To verify the version of your gitlab installation, enter the following command.

sudo gitlab-rake gitlab:env:info

On the New Server

The server is a clean installation of Ubuntu 18.04. The only service is SSH  and postfix.

Before doing the migration, I transfer the whole folder ~/gitlab-old from the old server to this new server.

1.  Add Gitlab source

   curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

2.  Install Gitlab 12.4.2 CE

   sudo apt update

   sudo apt install gitlab-ce=12.4.2

3. Copy the configuration files to folder /etc/gitlab

   sudo cp ~/gitlab-old/gitlab* /etc/gitlab

4. Copy the ssl folder to folder /etc/gitlab

   sudo cp -R ~/gitlab-old/ssl /etc/gitlab

5. Run Gitlab service for the first time

   sudo gitlab-ctl reconfigure

6. Shutdown the Gitlab services

   sudo gitlab-ctl stop unicorn

   sudo gitlab-ctl stop sidekiq

7. Copy the backup file to ~/backups then change the permission

   sudo cp ~/gitlab-old/*.tar ~/backups

   sudo chown git:root ~/backups/*.tar

8. Restore the Gitlab backup file

   sudo gitlab-rake gitlab:backup:restore BACKUP=XXXXXX

9. Restart Gitlab and check

   sudo gitlab-ctl start

   sudo gitlab-rake gitlab:check SANITIZE=true

There are some notes for the above nine steps.

The backup file name looks like 1573175514_2019_11_07_12.4.2_gitlab_backup.tar. So the code in the step 8, BACKUP=1573175514_2019_11_07_12.4.2

The version of Gitlab I am using 12.4.2. I install the 12.4.2 on step 2. When I do the restore in step 8, it displays the error, the version does not match. The backup is 12.4.2, but the installation is 12.4.1. I have to do the apt update and apt upgrade to upgrade the Gitlab from 12.4.1 to 12.4.2. I am not sure where the problem came from. Just a reminder, double-check the version installed on the new server.

At the first step of adding the repo source, I always get the network error on my server. I believed it is the problem of guest OS on Proxmox VE. So I actually add the repo source and public key of gitlab source manually.

My current Gitlab information on the new server Ubuntu 18.04

davidyin@gitlab:~$ sudo gitlab-rake gitlab:env:info

System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.6.3p62
Gem Version: 2.7.9
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 3.2.12
Git Version: 2.22.0
Sidekiq Version:5.2.7
Go Version: unknown

GitLab information
Version: 12.4.2
Revision: 393a5bdafa2
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 10.9
URL: https://gitlab.g2soft.net
HTTP Clone URL: https://gitlab.g2soft.net/some-group/some-project.git
SSH Clone URL: git@gitlab.g2soft.net:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version: 10.2.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git

At the end, I should give the credit to Didiet A. Pambudiono. Most of the steps above is based on this post.

The post Migrating GitLab to another Server from Ubuntu 16.04 to Ubuntu 18.04 appeared first on David Yin's Blog.

I use it on my Linode VPS. It is a Nanon type of VPS at Fremont, CA, USA.

1GB RAM, 25GB storage, 1 CPU.

Ubuntu 18.04 LTS was installed on it.

Step 0, Build the system from Linode Dashboard.

Step 1, Update the system

I SSH to the server with user root.

apt updat

apt upgrade

Step 2, Enable TCP BBR to improve network speed

sysctl net.ipv4.tcp_available_congestion_control

The above command should report

net.ipv4.tcp_available_congestion_control = cubic reno

To change it to bbr, opent the file /etc/sysctl.conf, and add following lines intot the end of the file.

net.core.default_qdisc=fq

net.ipv4.tcp_congestion_control=bbr

Save the file and enter the following command

sysctl -p

Step 3, Install Webmin 1.9.0

I like to use webmin as my web panel on VPS.

Enter the following command to install the latest webmin version 1.9.0

apt-get install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl apt-show-versions python
wget http://prdownloads.sourceforge.net/webadmin/webmin_1.910_all.deb
dpkg --install webmin_1.910_all.deb

Reboot the server, the ssh connection is lost.

Step 4, Add a new user

Sign in the webmin as user root by entering the URL into the browser https://ip.address.of.the.vps:10000

Add a new user, for example: davidyin and add sudo as the second group of this user.

From this point, I will use davidyin to do all the ssh jobs. I will not use root in terminal anymore.

Step 5 Build Nginx with TLS 1.3 and brotli now

SSH the VPS with user davidyin.

First, list the version of the software.

• Openssl: openssl-1.1.1b
• Nginx: nginx-1.17.0
• Brotli: ngx-brotli-0.13rc

Install the related software packages.

sudo apt install build-essential

sudoapt install libpcre3 libpcre3-dev zlib1g zlib1g-dev openssl libssl-dev

Prepare the source code.

Nginx:

wget https://nginx/org/download/nginx-1.17.0.tar.gz

tar xvzfnginx-1.17.0.tar.gz

rm nginx-1.17.0.tar.gz

Openssl:

wget https://www.openssl.org/source/openssl-1.1.1b.tar.gz

tar xvzf openssl-1.1.1b.tar.gz

rm openssl-1.1.1b.tar.gz

Brotli:

git clone https://github.com/eustas/ngx_brotli.git

cd ngx_brotli

git submodule update --init --recursive

Compile Nginx

cd ~/nginx-1.17.0

./configure  --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/nginx.lock --user=www-data --group=www-data --with-openssl=../openssl-1.1.1b --with-openssl-opt=enable-tls1_3 --with-http_v2_module --with-http_ssl_module --with-debug --with-http_gunzip_module --with-http_realip_module --with-http_sub_module --with-http_gzip_static_module --with-threads --with-file-aio --add-module=../ngx_brotli

make

sudo make install

Make Nginx as a service

If you can not start Nginx as service by “sudo service nginx restart”, make a new file at /etc/systemd/system/nginx.service

cd /etc/systemd/system

sudo nano nginx.service

Paste the following content:

# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /var/run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
[Install]
WantedBy=multi-user.target


Ente the command to check the version:

nginx version: nginx/1.17.0
built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)
built with OpenSSL 1.1.1b 26 Feb 2019

Now the Nginx server is supporting TLS 1.3 and Brotli compression.

The post How to build an Nginx with Brotli and TLS 1.3 support appeared first on David Yin's Blog.

If you want to use IPv4 in one command, just run following:

$ sudo apt -o Acquire::ForceIPv4=true update

If you want to force to use IPv4 for all apt operations, add a new file under /etc/apt/apt.conf.d/

$ sudo nano /etc/apt/apt.conf.d/99force-ipv4

Then add a line:

Acquire::ForceIPv4 "true";

If you want to disable IPv4, change the IPv4 to IPv6.

The post Force IPv4 when using APT command in Ubuntu appeared first on David Yin's Blog.

The original image is youtube.jpg, size is 1024X768.

https://images.g2soft.net/image/Ai8

Install FFmpeg

sudo apt install ffmpeg

Resize the image

ffmpeg -i "youtube.jpg" -vf scale=480:360 "youtubex480.jpg"

The command above explains itself.

It resizes the youtube.jpg to a smaller picture, 480×360.

https://images.g2soft.net/image/FFM

It is the post of “How to resize the image by FFmpeg”.

The post How to resize an image through the Ubuntu terminal appeared first on David Yin's Blog.

Let me list the existing installation.

• My Desktop, Windows 10 Pro version 1809.
• Virtualbox 6.0.4 r128413
• Guest VM inside of Virtualbox is Ubuntu 18.04 with 1G memory and 20GB disk space.
• Proxmox VE 5.3-8, VM and LXC are stored in the local-lvm

Step 1, Located the vdi file and convert it to RAW format

Check the settings of the virtual machine, storage, and located the location of the vdi file. Mine is in the d:\Virtualbox VM\

Then open a command prompt window, enter the following command to convert the vdi file to raw image file.

d:
cd "program files\oracle\virtualbox"
vboxmanage clonehd --format RAW "d"\virtualbox vm\ubuntu 18.04 (122)\ubuntu 18.04.vdi" d:\testing.img

After a while the new image file is ready. The time depends on the size of your VM. Maybe 10 minutes or half an hour.

Step 2, Transfer the image file to Proxmox host

My Proxmox Ve is using local-lvm for the guest disk image, the type is LVM-Thin. I can only access the local folder only.

So transfer the file into /root/

I use FileZilla to transfer the testing.img to /root/

step 3,  Change the storage local to accept the “Disk image” content

This step is important.

Datacenter > Storage > local (edit)

Select  Disk image to be content type, click OK (Max backups 1, if needed).

Step 4, Creat a new VM in Proxmox

On the “OS” tab, choose “Do not use any media”

On the “Hard Disk” tab, choose “local” as Storage, Disk size is my testing.img size + 1GB, Format, choose “Raw disk image (raw)”. This will create a VM with an ID and puts its “img” file in “/var/lib/vz/images/ID”

On CPU and Memory, I choose the same as my Virtualbox settings.

Step 5, Replace the VM image file

Now it is time to remove the image file just created on step 4 and move the /root/testing.img to “/var/lib/vz/images/ID” folder and rename it.

Before entering the following command, make sure the machine is shutdown.

Here is the ID, 107.

cd /var/lib/vz/images/107
rm vm-107-disk-0.raw
mv /root/testing.img ./vm-107-disk-0.raw

Step 6, Move the image from local to local-lvm

Since my local has 100Gb only, local-lvm is about 800GB. So it is better to move the disk into local-lvm.

In the Proxmox web interface, select the virtual machine 107 > hardware > Hard Disk

Click Move disk button, select the Target Storage “local-lvm”, check the checkbox of “Delete source”.

Moving image file from local to local-lvm may take more time if the size of image is big.

Step 7, Start the machine, fix the networking problem

Start VM 107.

From Console, I can not access the Internet. I found there is only loop there.

So I check the file /etc/netplan/50-cloud-init.yaml

The ethernets are for enp0s3. But the virtual machine is using  ens18 as an interface.

Just change the content of this file and restart the machine. It works.

If yours is different, use command ip link to check the name of the interface.

PS.

The network configuration on Ubuntu 16.04 is different. Please check the file /etc/network/interfaces

The post Move Virtualbox VDI to Proxmox VE System appeared first on David Yin's Blog.

The solution is to use wkhtmltoimage, which is a part of wkhtmtopdf.

I used to use the old version wkhtmltopdf v0.12.3. Now it is not working anymore due to some reason I don’t know.

But the wkhtmltopdf v0.12.5  is OK.

The OS is Ubuntu 18.04.

Then I install wkhtmltopdf with the following command

sudo apt-get install -y software-properties-common
sudo apt-add-repository -y "deb http://security.ubuntu.com/ubuntu xenial-security main" 
sudo apt-get -yq update
sudo apt-get install -y libxrender1 libfontconfig1 libx11-dev libjpeg62 libxtst6 fontconfig xfonts-75dpi xfonts-base libpng12-0 libjpeg-turbo8
wget "https://downloads.wkhtmltopdf.org/0.12/0.12.5/wkhtmltox_0.12.5-1.xenial_amd64.deb" 
sudo dpkg -i wkhtmltox_0.12.5-1.xenial_amd64.deb 
sudo apt-get -f install

After installation, run the command to show the help of the program.

Now I am going to render an image from the most famous web site.
The URL is https://www.google.com
Run the following command to generate the screenshot.

wkhtmltoimage --javascript-delay 3500 --height 768 --quality 64 "https://www.google.com" /home/davidyin/google.jpg

Let’s see the result.

Later, I did it with Youtube.

The post How to take a webpage screenshot at terminal window appeared first on David Yin's Blog.

Here is the VPS basic information.

• SFO2
• 1GB Nanode
• Ubuntu Linux 18.04.2
• Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, 1 cores
• Nginx 1.15.9

All sites use one access.log file. The following code is in the http block of /etc/nginx/nginx.conf file.

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

All access logs look like below:

Now it is time to show you how to use GoAccess to Generate Report with Multiple Nginx Log files.

Install GoAccess

sudo install goaccess

Modify the Config file of GoAccess

GoAccess config file is located at /etc/

sudo nano /etc/goaccess.conf

Add the following code or modify the code inside of the goaccess.conf.

time-format %H:%M:%S

date-format %d/%b/%Y

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

Run GoAccess to generate the report

I would like to get a static report.

sudo goaccess /var/log/nginx/access.log -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The report looks like below:

Due to the log is not just one file. The log files are generated by date and also compressed.

Two log files are not compressed.

• access.log
• access.log.1

The older log files are compressed.

• access.log.2.gz
• access.log.3.gz
• access.log.4.gz
• access.log.5.gz

I want to generate all the access log files. So the command is changed to the way which can phrase multiple log files.

zcat /var/log/nginx/access.log.*.gz | goaccess /var/log/nginx/access.log /var/log/nginx/access.log.1 - -o /home/davidyin/mywebsite.com/report.html --log-format=COMBINED

The sample command above using mywebsite.com. Please replace it with your own sites.

At the last, I add this command into Cron jobs. Schedule it every hour.

The post Use GoAccess to Generate Report with Multiple Nginx Log Files appeared first on David Yin's Blog.

This is a follow-up to the End of Life warning sent earlier this month
to confirm that as of today (July 19, 2018), Ubuntu 17.10 is no longer
supported. No more package updates will be accepted to 17.10, and
it will be archived to old-releases.ubuntu.com in the coming weeks.

The original End of Life warning follows, with upgrade instructions:

Ubuntu announced its 17.10 (Artful Aardvark) release almost 9 months
ago, on October 19, 2017. As a non-LTS release, 17.10 has a 9-month
support cycle and, as such, the support period is now nearing its
end and Ubuntu 17.10 will reach end of life on Thursday, July 19th.

At that time, Ubuntu Security Notices will no longer include
information or updated packages for Ubuntu 17.10.

The supported upgrade path from Ubuntu 17.10 is via Ubuntu 18.04.
Instructions and caveats for the upgrade may be found at:

https://help.ubuntu.com/community/BionicUpgrades

Ubuntu 18.04 continues to be actively supported with security updates
and select high-impact bug fixes. Announcements of security updates
for Ubuntu releases are sent to the ubuntu-security-announce mailing
list, information about which may be found at:

https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce

Since its launch in October 2004 Ubuntu has become one of the most
highly regarded Linux distributions with millions of users in homes,
schools, businesses and governments around the world. Ubuntu is Open
Source software, costs nothing to download, and users are free to
customise or alter their software in order to meet their needs.

On behalf of the Ubuntu Release Team,

Adam Conrad

The post Ubuntu 17.10 (Artful Aardvark) End of Life reached on July 19 2018 appeared first on David Yin's Blog.

Now, my all VPS is running on the Ubuntu 16.04 LTS. LTS is an abbreviation for “Long Term Support”.  An LTS version is released every two years. So after Ubuntu 16.04 LTS, it is Ubuntu 18.04 LTS.  Now all LTS version has five years support.

I can use Ubuntu 16.04 LTS until 2021. Security updates will be released during the time period.

I am waiting for the release of Ubuntu 18.04 LTS. I would like to try it on my staging server, or a virtual machine. Try all my existing web programs on Ubuntu 18.04 LTS. Find the compatible problems then fix them.

Maybe after a few months testing. I am sure there are no issues with it. I will install a new VPS with Ubuntu 18.04 LTS and move the websites to it.

New Features in 18.04

There are a lot of new features since 16.04. I just mark my interested one here. Only server related.

• Linux kernel 4.15
• The default DNS resolver is systemd-resolved.
• The gpg binary is provided by gnupg2
• For new installs, a swap file will be used by default instead of a swap partition.
• Python 2 is no longer installed by default. Python 3 has been updated to 3.6. This is the last LTS release to include Python 2 in main.

• OpenSSH now refuses to use RSA keys smaller than 1024 bits. ssh-keygen -l -f /path/to/key.pub can report the length of a key.

• PHP was updated to version 7.2.x.

• Apache was updated to version 2.4.29. Additionally, HTTP/2 support is now enabled in 18.04.

I will install the beta version first, to have a look today. Then wait for two more days to try the official release by upgrading and new installation.

The post Waiting for the Ubuntu 18.04 LTS appeared first on David Yin's Blog.