Install nZEDb Ubuntu for Private Usenet Indexing

nZEDb is a fork of the popular newznab indexer. It is becoming increasingly popular because it is actively developed and has a plethora of features. This guide will show you how to get nZEDb up and running on a Ubuntu 14.04 machine so you can have your own private usenet indexer. nZEDb will delete fake and passworded releases from your database if you have unrar enabled. It can detect the media info of video files and create screenshots. It will also grab covers for media from various API sites. This guide is derived from the official nZEDb guide but has been expanded. When you're done you will have your own private indexer. I tested this on an old Penium M system and it updated 2 groups just fine, the more groups you add the more stress it will put on the system. I tested it on a 64 bit VPS with 10GB of RAM and it was able to handle 20 groups and backfill efficiently. Do be aware of the limitations of your hardware before you embark on this private indexer journey.

Install nZEDb Ubuntu for Personal Usenet Indexing

Install some basic stuff like python and git

sudo apt-get install software-properties-common python-software-properties git -y

If you are on a 32 bit system you must add this repo for PHP5.6 or your update binaries scripts will fail
You will get gzopen() errors, this solves them.

sudo add-apt-repository ppa:ondrej/php5-5.6
sudo apt-get update
sudo apt-get upgrade -y

Install MySQL

All of your nzbs and releases and stuff will be put into a MySQL database. You will install mysqltuner as well so you can get suggestions about how to tweak the database in the future.

sudo apt-get install mysql-server mysql-client libmysqlclient-dev mysqltuner -y

Edit the configuration file

sudo nano /etc/mysql/my.cnf

Max allowed packet should already be set

max_allowed_packet		= 16M

Add this line in the same section max_allowed_packet is

group_concat_max_len    = 8192

Change key_buffer to key_buffer_size to avoid future problems with your MySQL configuration.

Set the key buffer size to 256M under the [mysqld] section

key_buffer_size    = 256M

Hit Ctrl+X, Y and then Enter to exit the MySQL configuration

Run MySQL and enter your root password you chose during the MySQL installation

sudo mysql -p

Grant your username permission to access the MySQL database
Note: you do not paste the mysql> part for either of these commands.

mysql>GRANT FILE ON *.* TO 'username'@'localhost';

Exit MySQL

mysql>exit

Change AppArmor Behavior

AppArmor interferes with MySQL so we have to manipulate it so that nZEDb can work with MySQL. Disabling AppArmor may screw up how your system works, you have been warned – you can also edit the AppArmor policy though.

The secure way to do this is by editing the AppArmor policy

sudo nano /etc/apparmor.d/usr.sbin.mysqld

Above the last } in the file add these lines so it looks like this. Do not add an extra } sign.

/data/ r,
/data/** rwk,
}

Create the data folder

sudo mkdir /data

Change ownership of the data folder

sudo chown mysql:mysql /data

You should not have to reboot after editing the policy just reload Apparmor

sudo service apparmor reload

You can also just Remove Apparmor

Apparmor needs to be removed because it prevents proper functioning between nZEDb and MySQL
To disable Apparmor completely and uninstall, remove it from startup daemons

sudo update-rc.d apparmor disable
sudo /etc/init.d/apparmor stop
sudo /etc/init.d/apparmor teardown
sudo update-rc.d -f apparmor remove
sudo apt-get purge -yqq apparmor* apparmor-utils

You must reboot, this is not optional

sudo reboot

Install a Web Server

Since nZEDb runs on PHP, you need to host its php files and website files so you can access and manage the indexer

You have two options, Apache or nginx. nginx is great for lower end systems.

Install Apache

sudo apt-get install apache2 -y

Configure it to point to nZEDb

sudo nano /etc/apache2/sites-available/nZEDb.conf

paste this into the configuration

<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 ServerName localhost
 DocumentRoot "/var/www/nZEDb/www"
 LogLevel warn
 ServerSignature Off
 ErrorLog /var/log/apache2/error.log
 <Directory "/var/www/nZEDb/www">
 Options FollowSymLinks
 AllowOverride All
 Require all granted
 </Directory>
 Alias /covers /var/www/nZEDb/resources/covers
</VirtualHost>

Edit the Apache configuration

sudo nano /etc/apache2/apache2.conf

If you have AllowOverride None change it to to AllowOverride All

        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted

Disable the default site, enable the nZEDbsite, turn on rewrite and restart apache

sudo a2dissite 000-default
sudo a2ensite nZEDb.conf
sudo a2enmod rewrite
sudo service apache2 restart

Install nginx Web Server

Install nginx from the repository

sudo apt-get install nginx -y

Install PHP fast page for nginx

sudo apt-get install php5-fpm -y

Edit the configuration file

sudo nano /etc/nginx/sites-available/nZEDb

Paste this code, change HTPCGuides.com to your local ip address (localhost did not work for me, it had to be the local address 192.168.40.110). You can add multiple server names separated by spaces like
server_name 192.168.40.110 HTPCGuides.com;

server {
    # Change these settings to match your machine.
    listen 80 default_server;
    server_name 192.168.40.110 HTPCGuides.com;

    # These are the log locations, you should not have to change these.
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # This is the root web folder for nZEDb, you shouldn't have to change this.
    root /var/www/nZEDb/www/;
    index index.html index.htm index.php;

    # Everything below this should not be changed unless noted.
    location ~* \.(?:ico|css|js|gif|inc|txt|gz|xml|png|jpe?g) {
        expires max;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }

    location / {
        try_files $uri $uri/ @rewrites;
    }

    location ^~ /covers/ {
        # This is where the nZEDb covers folder should be in.
        root /var/www/nZEDb/resources;
    }

    location @rewrites {
        rewrite ^/([^/\.]+)/([^/]+)/([^/]+)/? /index.php?page=$1&id=$2&subpage=$3 last;
        rewrite ^/([^/\.]+)/([^/]+)/?$ /index.php?page=$1&id=$2 last;
        rewrite ^/([^/\.]+)/?$ /index.php?page=$1 last;
    }

    location /admin {
    }

    location /install {
    }

    location ~ \.php$ {
        include /etc/nginx/fastcgi_params;

        # Uncomment the following line and comment the .sock line if you want to use TCP.
        #fastcgi_pass 127.0.0.1:9000;
        fastcgi_pass unix:/var/run/php5-fpm.sock;

        # The next two lines should go in your fastcgi_params
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
    location /phpmyadmin {
        root /usr/share/;
        index index.php index.html index.htm;
        location ~ ^/phpmyadmin/(.+\.php)$ {
            try_files $uri =404;
            root /usr/share/;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            include /etc/nginx/fastcgi_params;
        }
        location ~* ^/phpmyadmin/(.+\.(jpg|jpeg|gif|css|png|js|ico|html|xml|txt))$ {
            root /usr/share/;
        }
    }
    location /phpMyAdmin {
        rewrite ^/* /phpmyadmin last;
    }
}

Create the nginx log files

sudo mkdir -p /var/log/nginx

Make the log file writeable

sudo chmod 755 /var/log/nginx

Disable the default nginx site

sudo unlink /etc/nginx/sites-enabled/default

Make nZEDb the default site for nginx

sudo ln -s /etc/nginx/sites-available/nZEDb /etc/nginx/sites-enabled/nZEDb

Restart php5-fpm

sudo service php5-fpm restart

Restart nginx

sudo service nginx restart

Add your user to the www-data group

sudo usermod -aG www-data username

Install PHP

sudo apt-get install php5 php5-cli php5-dev php5-json php-pear php5-gd php5-mysqlnd php5-curl -y

For some reason php5 includes Apache so if you want to use nginx, you will have to remove Apache2

sudo apt-get purge apache*
sudo apt-get remove apache*

Edit the main php file that both apache2 and nginx use

sudo nano /etc/php5/cli/php.ini

Change the following settings:

The default execution time is 30 seconds, change it to 120 seconds

max_execution_time = 120

Memory can be set to -1 (-1 means unlimited) or you can limit it if you are running other processes, nZEDb will run on as little memory as you give it.

memory_limit = 1024M

Change your timezone from a list of timezones here. Remove the preceding ;

List of Timezones

date.timezone = YourLocalTimezone

Enable error logging (Needed when reporting bugs) should be already set

error_reporting = E_ALL

log errors, should also be set to on

log_errors = On

Add this line under log_errors so you have a log file location

error_log = php-errors.log

Close and save this file.

Make the same changes in Apache or nginx

You need to make the same changes in the Apache or nginx php configuration

For log_errors you will need to add the lines shown above.

Open apache2 php file

sudo nano /etc/php5/apache2/php.ini

For nginx edit this file

sudo nano /etc/php5/fpm/php.ini

Install php_yenc_decode

PHP decode is used to decode the yenc posts on usenet, it is the fastest and most efficient

cd ~/
git clone https://github.com/kevinlekiller/simple_php_yenc_decode
cd simple_php_yenc_decode/
sh ubuntu.sh
cd ..
rm -rf simple_php_yenc_decode/

Install phpMyAdmin

This is optional but it is recommended, it gives you a nice interface to access your indexer databases

sudo apt-get install phpmyadmin -y

If you are running apache2 then let it reconfigure, press space to mark apache2 with an asterix.

If you are running nginx don't tell it to reconfigure anything and just press Enter.

Say ok to dbconfig-common

You will need to enter the root password for the MySQL database which you created earlier

I entered the same password I use for MySQL here, this is the same password you will use to log in to phpMyAdmin

Enter the same password as the previous screen here

Now secure phpMyAdmin with encryption

sudo php5enmod mcrypt

You can access phpMyAdmin at http://ip.address/phpmyadmin

Your login details for phpmyadmin should be same as for MySQL

Optional Stuff

These items are optional if you want to post process releases.

• create screenshots
• find covers for movies, games etc
• discard fake releases

Some post processing can be very resource intensive, so you should have appropriate hardware if you plan to have your indexer examine releases for authenticity.

If you want to look inside rar files and remove garbage releases that contain exe files, you will need unrar

sudo apt-get install build-essential -y
cd /tmp
RARVERSION=$(wget -q http://www.rarlab.com/rar_add.htm -O - | grep unrarsrc | awk -F "[\"]" ' NR==1 {print $2}')
wget $RARVERSION
tar -xvf unrarsrc*.tar.gz
cd unrar
sudo make -j$(nproc) -f makefile
sudo install -v -m755 unrar /usr/bin
cd ..
rm -R unrar*
rm unrarsrc-*.tar.gz

Pzip is also used for looking into releases

sudo apt-get install p7zip-full -y

avprobe (instead of ffmpeg) for looking at video files. This will enable your indexer to create screenshots and video samples

sudo apt-get install libav-tools -y

lame is used for audio processing and can create samples

sudo apt-get install lame -y

Media Info

This will analyze video sources and give your indexer information about the bitrate and resolution of video files

64 bit

Media Info Dependency

wget http://mediaarea.net/download/binary/libzen0/0.4.29/libzen0_0.4.29-1_amd64.xUbuntu_14.04.deb
sudo dpkg -i libzen0_0.4.29-1_amd64.xUbuntu_14.04.deb

Media Info Library

wget http://mediaarea.net/download/binary/libmediainfo0/0.7.70/libmediainfo0_0.7.70-1_amd64.xUbuntu_14.04.deb
sudo dpkg -i libmediainfo0_0.7.70-1_amd64.xUbuntu_14.04.deb

Media Info Install

wget http://mediaarea.net/download/binary/mediainfo/0.7.70/mediainfo_0.7.70-1_amd64.Debian_7.0.deb
sudo dpkg -i mediainfo_0.7.70-1_amd64.Debian_7.0.deb

32 bit

Media Info Dependency

wget http://mediaarea.net/download/binary/libzen0/0.4.29/libzen0_0.4.29-1_i386.xUbuntu_14.04.deb
sudo dpkg -i libzen0_0.4.29-1_i386.xUbuntu_14.04.deb

Media Info Library

wget http://mediaarea.net/download/binary/libmediainfo0/0.7.70/libmediainfo0_0.7.70-1_i386.xUbuntu_14.04.deb
sudo dpkg -i libmediainfo0_0.7.70-1_i386.xUbuntu_14.04.deb

Media Info Install

wget http://mediaarea.net/download/binary/mediainfo/0.7.70/mediainfo_0.7.70-1_i386.Debian_7.0.deb
sudo dpkg -i mediainfo_0.7.70-1_i386.Debian_7.0.deb

Install Memcache

Grab memcache to enable caching for php

sudo apt-get install memcached php5-memcached -y

You will enable this later in the Finishing up section

Install nZEDb

Git clone the latest nZEDb

sudo git clone https://github.com/nZEDb/nZEDb.git /var/www/nZEDb

Change ownership of the nZEDb installation, change username to your username

sudo chown -R username:www-data /var/www/nZEDb

Change permissions so nZEDb can put covers places, unpack rar files and execute stuff in general

sudo chmod 777 /var/www/nZEDb/libs/smarty/templates_c
sudo chmod -R 777 /var/www/nZEDb/resources/covers/ 
sudo chmod -R 777 /var/www/nZEDb/www/ 
sudo chmod -R 777 /var/www/nZEDb/resources/nzb/
sudo chmod -R 777 /var/www/nZEDb/resources/tmp/unrar/
sudo chmod -R 777 /var/lib/php5
sudo mkdir /var/lib/php5/sessions
sudo chmod 777 /var/lib/php5/sessions

Restart or you might get some preflight errors during the initial setup
I got php-json not supported Error and several Warnings , restarting made them disappear.
If you are using Apache2 restart that

sudo service apache2 restart

Or restart nginx

sudo service nginx restart

nZEDb Initial Setup Configuration

Point your browser to the box http://ip.address/install to start the Installation setup.

Time to do a pre-flight check

Everything should be fine, the warnings will usually disappear if you restart the webserver

Scroll down and click Set up the database

Database system is mysql
Hostname is localhost or 127.0.0.1
Port is 3306
Username is root
password is the MySQL password you chose during the MySQL installation
Database to connect to is nzedb

Click Setup Database

On to OpenSSL

Since this is a personal indexer we will skip over OpenSSL. Click Verify OpenSSL

It will say success, now it's usenet provider time. I use UsenetServer because they support compressed headers.

Click Setup news server connection

Enter your usenet provider server address, username and password.

Set the port to 563 or 443 if you are using SSL. Check off SSL.

Scroll down and click Test primary connection

Click Save Settings

Now it's time to set up an Admin. Enter a username and password, you also need an email address.

Click Create Admin User

Click Set file paths

Leave them as they are and click Set file paths

You succeeded! click admin home page to go to the administration page

You should see this login screen, enter the admin username and password you just created

Here is the admin page 🙂

Updating your Indexer

It is best to do a test to make sure the indexing scripts work correctly

In your admin page, click Settings -> Group and choose View

Enter teevee in the search field and click Go

In the Active column, click Activate, leave Backfill Deactivated for now

Back in Terminal, enter the scripts folder

cd /var/www/nZEDb/misc/update/

Run the script to download headers from the usenet group you enabled

php update_binaries.php

If everything looked normal, you can process the headers. This creates releases and nzb files from the headers you just downloaded

php update_releases.php 1 true

If you got any gzopen() errors you need to do the 32 bit fixes I mentioned in the beginning

You can now go browse your TV category by clicking TV then All TV in your indexer's web interface, you should see some releases added in list view.

If it's all working, let's automate the process using Tmux or Screen so you don't have to keep running these scripts manually.

Using nZEDb Tmux Scripts

Install Tmux

sudo apt-get install tmux time -y

Install more Tmux things so it can work with MySQL

sudo apt-get install python-setuptools python-pip -y
sudo easy_install cymysql pynntp socketpool

Install the python 3 modules as well or you will get a python 3 cymysql error

sudo apt-get install python3-setuptools python3-pip -y
sudo pip3 install cymysql pynntp socketpool

Change the settings in Site -> Tmux Settings

Set Tmux Scripts Running to Yes

Set Run Sequential to Basic

Update Binaries Simple Threaded

Update Releases set to Update Releases Simple

Now run the Tmux script

cd /var/www/nZEDb/misc/update/nix/tmux/
php start.php

Press Ctrl+A, release the buttons and press D
Tmux will index in the background

You can go back in and view it

tmux attach

To kill the tmux session

Press Ctrl+A, type :, you will see a yellow box pop up at the bottom of the screen

Type in kill-session and press Enter

If you get this error
Allowed memory size of 262144 bytes exhausted (tried to allocate 12288 bytes) in /var/www/nZEDb/misc/update/nix/tmux/start.php on line 141

Restart the web server or you made a mistake in the max_memory values in the php.ini files

Go back and double check your memory_limit values in both php.ini files

Screen Script Method

If you have issues with Tmux you can use the screen method. This can be optimal on lower end systems.

Install screen

sudo apt-get install screen -y

Go into the scripts folder

cd /var/www/nZEDb/misc/update/nix/screen/threaded

Screen is used to keep the shell running even after you have detached and exited from your SSH session

screen sh start.sh

To detach the screen so it continues running in its shell, press Ctrl+A, let go, then press D

To reattach the screen list the screen sessions

screen -r

You will see a list like this

14221.pts-0.HTPCGuides (11/11/14 13:30:14) (Attached)
6282.RELEASES (11/11/14 08:17:41) (Detached)
1786.POSTP (11/11/14 07:45:55) (Detached)

Then to reattach the syntax is screen -r ID

screen -r 6282

You can detach again using Ctrl+A, release and press D

If you want to kill the attached screen

List the screens

screen -r

Here is a screen list sample output

14221.pts-0.HTPCGuides (11/11/14 13:30:14) (Attached)
6282.RELEASES (11/11/14 08:17:41) (Detached)
1786.POSTP (11/11/14 07:45:55) (Detached)

Identify the one you wish to kill, the time stamp shows when it was started

The syntax is screen -X -S ID kill
The ID number is the numbers, so if I wanted to kill the top one

screen -X -S 14221 kill

If you want to reattach and view the script and how it's doing

screen -r

Identify the one you want to reattach and use this code

screen -r ID number

You can then detach again with Ctrl+A, release and press D.

Finishing touches

Enable Memcache

sudo nano /var/www/nZEDb/www/config.php

Add these lines at the bottom of the config file

// Enable memcache
define('MEMCACHE_ENABLED', 'TRUE');

Tuning MySQL

You installed mysqltuner which can give suggestions about how to optimize your database

It is best to run this after the indexer has been running for a few days

Run it using this command.

mysqltuner

You will get some suggestions and you can search how to implement them

Usually it will involve editing the mysql.conf file

Make sure you stop running your scripts first or they will freeze

sudo nano /etc/mysql/my.cnf

To save and exit the mysql configuration hit Ctrl+X, press Y and then enter

Restart MySQL for the configuration changes you made to take effect

sudo service mysql restart

Start your scripts again

Reinstall nZEDb

You may make some mistakes or want to start over with your nZEDb indexer, this is how I did it.

Change username to your username you use to log on with

sudo service apache2 stop
sudo service nginx stop
sudo rm -r /var/www/nZEDb/
cd /var/www/nZEDb
sudo git clone https://github.com/nZEDb/nZEDb.git /var/www/nZEDb
sudo chown username:www-data /var/www/nZEDb

You will need to change permissions again like you did for the initial nZEDb setup after git cloning.

My nZEDb configuration guide should follow in the future, but this initial setup should get you indexing a few groups for personal usage.