About MLFow
MLflow is an open source platform for managing the end-to-end machine learning lifecycle. It tackles four primary functions:
Tracking experiments to record and compare parameters and results (MLflow Tracking). Packaging ML code in a reusable, reproducible form in order to share with other data scientists or transfer to production (MLflow Projects). Managing and deploying models from a variety of ML libraries to a variety of model serving and inference platforms (MLflow Models). Providing a central model store to collaboratively manage the full lifecycle of an MLflow Model, including model versioning, stage transitions, and annotations (MLflow Model Registry). MLflow is library-agnostic. You can use it with any machine learning library, and in any programming language, since all functions are accessible through a REST API and CLI. For convenience, the project also includes a Python API, R API, and Java API.
(more details... https://mlflow.org/docs/latest/index.html)
Make Python3 default
Python2 and Python3 are already installed on Debian 10, but the default version is Python2. How to check it...
python --version
That will return Python 2.x.x
Check for Python3
python3 --version
That will return Python 3.x.x
We have to make Python3 as the default version for the distro.
Firstly we have to find the directories for each version...
ls /usr/bin/python* -la
In this example it returns a bunch of directories...
lrwxrwxrwx 1 root root 7 Mar 4 2019 /usr/bin/python -> python2
lrwxrwxrwx 1 root root 9 Mar 4 2019 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3604200 Oct 10 2019 /usr/bin/python2.7
lrwxrwxrwx 1 root root 9 Mar 26 2019 /usr/bin/python3 -> python3.7
-rwxr-xr-x 2 root root 4744392 Apr 3 2019 /usr/bin/python3.7
-rwxr-xr-x 2 root root 4744392 Apr 3 2019 /usr/bin/python3.7m
-rwxr-xr-x 1 root root 1018 Mar 3 2018 /usr/bin/python3-jsondiff
-rwxr-xr-x 1 root root 3661 Mar 3 2018 /usr/bin/python3-jsonpatch
-rwxr-xr-x 1 root root 1342 Apr 30 2016 /usr/bin/python3-jsonpointer
-rwxr-xr-x 1 root root 398 Sep 7 2018 /usr/bin/python3-jsonschema
lrwxrwxrwx 1 root root 10 Mar 26 2019 /usr/bin/python3m -> python3.7m
Of all these paths we need the ones which have only the name and the version of Pythons (2 and 3 version), without any other description.
In the example above the directories are...
/usr/bin/python2.7
# For Python2
/usr/bin/python3.7
# For Python3
Now we must update/create the "Python alternatives list" using the above paths.
sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.7 2
Finally we can switch between Python versions. Running the following command, a menu will appear in order to choose the Python version that we want as default.
sudo update-alternatives --config python
There are 2 choices for the alternative python (providing /usr/bin/python).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/bin/python3.7 2 auto mode
1 /usr/bin/python2.7 1 manual mode
2 /usr/bin/python3.7 2 manual mode
Press <enter> to keep the current choice[*], or type selection number: 2
Choosing the selection 2 then Python3 will become the default version.
We could check if Python3 has applied as default, running...
python --version
In my case it returns Python 3.7.3
Download and install Miniforge3
Miniforge is the community (conda-forge) driven minimalistic conda installer. Subsequent package installations come thus from conda-forge channel, and it started because miniconda didn't support aarch64,
But first check if there is the Download directory on your distro. If there is none, then create one...
mkdir ~/Download cd ~/Download sudo wget https://github.com/conda-forge/miniforge/releases/download/4.8.3-4/Miniforge3-Linux-aarch64.sh bash Miniforge3-Linux-aarch64.sh
* Miniforge3 will be installed at this location: ~/miniforge3/bin/ and add it to your PATH variable.
In order to check the location of executable "conda" file, type the following...
sudo find / -type f -name "conda"
In my case it returned "~/miniforge3/bin/" and the variable PATH must be changed to...
export PATH=$PATH:~/miniforge3/bin
Install MLFlow using Conda
conda install -c https://conda.anaconda.org/paulscherrerinstitute mflow conda update conda conda --version
In my case the version is 4.9.2
Create and activate a conda virtual environment and install MLFlow
A conda environment is a directory that contains a specific collection of conda packages that you have installed. For example, you may have one environment with NumPy 1.7 and its dependencies. The name of the environment is env_mlflow.
Use your Python's version to create an environment...
conda create --name env_mlflow python=3.7.3 conda activate env_mlflow
Reboot the machine and login again. If you are using a remote machine, reboot the remote instance and use ssh to login again.
After the reboot we can install MLFlow using pip3.
pip3 install mlflow
Install Sklearn using pip3 and other dependencies
We need GCC and G++ compilers as dependencies to build Sklearn.
sudo apt-get install gcc sudo apt-get install g++
This app helps us to save the point of the process, when the connection drops. As a result, when we reconnect, we can continue from the point where our process was disrupted and not from the very beginning.
(optional) Install tmux.
It takes a lot of time to build Sklearn, and there is a high possibility the internet access will be lost. So, tmux helps us to continue the proccess from where it left off.
sudo apt-get install tmux
To open tmux just type... tmux
Install Sklearn using pip3 and the dependencies.
pip3 install sklearn pip3 install Cython pip3 install --upgrade setuptools
Install Matplotlib
Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. It provides an object-oriented API for embedding plots into applications using general-purpose GUI toolkits like Tkinter, wxPython, Qt, or GTK+.
conda install -c conda-forge matplotlib
In order to check the installation...
pip3 install matplotlib
The return must be "Requirement already satisfied"
Install PostgreSQL
PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasising extensibility and SQL compliance.
sudo apt-get install postgresql sudo apt-get install postgresql-contrib sudo apt-get install postgresql-server-dev-all
Install Psycopg
Psycopg is the most popular PostgreSQL database adapter for the Python programming language. Its main features are the complete implementation of the Python DB API 2.0 specification and the thread safety (several threads can share the same connection).
pip3 install psycopg2
Download and run MLFlow examples
Check if Git is already installed...
git --version
If Git is not installed...
sudo apt update sudo apt install git
Clone MLFlow example code
cd ~ git clone https://github.com/mlflow/mlflow cd ~/mlflow/examples/sklearn_elasticnet_diabetes/linux
Run the example "train_diabetes"
python train_diabetes.py 0.1 0.9 python train_diabetes.py 0.5 0.5 python train_diabetes.py 0.9 0.1
For each example above the return must be something like this...
RMSE: 71.98302888908191
MAE: 60.5647520017933
R2: 0.2165516143465459
MLflow runs can be recorded either locally in files or remotely to a tracking server. By default, the MLflow Python API logs runs to files in an mlruns directory wherever you ran your program. You can then run "mlflow ui" to see the logged runs.
mlflow ui
The return must be something like that...
2020-12-03 14:17:36 +0000] [1267] [INFO] Starting gunicorn 20.0.4
[2020-12-03 14:17:36 +0000] [1267] [INFO] Listening at: http://127.0.0.1:5000 (1267)
[2020-12-03 14:17:36 +0000] [1267] [INFO] Using worker: sync
[2020-12-03 14:17:36 +0000] [1270] [INFO] Booting worker with pid: 1270
At the 2nd line there is the localhost's IP and the port (127.0.0.1:5000). Now we can use them to get the results.
Using a remote machine (OpenStack), the instances does not have a Desktop Environment (GNOME, XFCE, e.t.c.) in order to see the results. But we can check if it has created the "index.html" file at http://127.0.0.1:5000...
cd ~/Download wget 127.0.0.1:5000 cat index.html
But, using SSH Tunneling we can see the OpenStack's instance localhost on the local Web browser.
Syntax: ssh -L port:localhost:port userName@FloatingIP
Example: ssh -L 5000:localhost:5000 debian@214.152.131.68
(We can replace the localhost with the 127.0.0.1)