README

LIBMF is a library for large-scale sparse matrix factorization. For the
optimization problem it solves and the overall framework, please refer to [3].



Table of Contents
=================

- Installation
- Data Format
- Model Format
- Command Line Usage
- Examples
- Library Usage
- SSE, AVX, and OpenMP
- Building Windows and Mac Binaries
- References



Installation
============

- Requirements
 To compile LIBMF, a compiler which supports C++11 is required. LIBMF can
 use SSE, AVX, and OpenMP for acceleration. See Section SSE, AVX, and OpenMP
 if you want to disable or enable these features.

- Unix & Cygwin

 Type `make' to build `mf-train' and `mf-precict.'

- Windows & Mac

 See `Building Windows and Mac Binaries' to compile. For Windows, pre-built
 binaries are available in the directory `windows.'



Data Format
===========

LIBMF's command-line tool can be used to factorize matrices with real or binary
values. Each line in the training file stores a tuple,

 <row_idx> <col_idx> <value>

which records an entry of the training matrix. In the `demo' directory, the
files `real_matrix.tr.txt' and `real_matrix.te.txt' are the training and test
sets for a demonstration of real-valued matrix factorization (RVMF). For binary
matrix factorization (BMF), the set of <value> is {-1, 1} as shown in
`binary_matrix.tr.txt' and `binary_matrix.te.txt.' For one-class MF, all
<value>'s are positive. See `all_one_matrix.tr.txt' and `all_one_matrix.te.txt'
as examples.

Note: If the values in the test set are unknown, please put dummy zeros.



Model Format
============

 LIBMF factorizes a training matrix `R' into a k-by-m matrix `P' and a
 k-by-n matrix `Q' such that `R' is approximated by P'Q. After the training
 process is finished, the two factor matrices `P' and `Q' are stored into a
 model file. The file starts with a header including:

 `f': the loss function of the solved MF problem
 `m': the number of rows in the training matrix,
 `n': the number of columns in the training matrix,
 `k': the number of latent factors,
 `b': the average of all elements in the training matrix.

 From the 5th line, the columns of `P' and `Q' are stored line by line. In
 each line, there are two leading tokens followed by the values of a
 column. The first token is the name of the stored column, and the second
 word indicates the type of values. If the second word is `T', the column is
 a real vector. Otherwise, all values in the column are NaN. For example, if

 [1 NaN 2] [-1 -2]
 P = |3 NaN 4|, Q = |-3 -4|,
 [5 NaN 6] [-5 -6]

 and the value `b' is 0.5, the content of the model file is:

 --------model file--------
 m 3
 n 2
 k 3
 b 0.5
 p0 T 1 3 5
 p1 F 0 0 0
 p2 T 2 4 6
 q0 T -1 -3 -5
 q1 T -2 -4 -6
 --------------------------



Command Line Usage
==================

- `mf-train'

 usage: mf-train [options] training_set_file [model_file]

 options:
 -l1 <lambda>,<lambda>: set L1-regularization parameters for P and Q.
 (default 0) If only one value is specified, P and Q share the same
 lambda.
 -l2 <lambda>,<lambda>: set L2-regularization parameters for P and Q.
 (default 0.1) If only one value is specified, P and Q share the same
 lambda.
 -f <loss>: specify loss function (default 0)
 for real-valued matrix factorization
 0 -- squared error (L2-norm)
 1 -- absolute error (L1-norm)
 2 -- generalized KL-divergence (--nmf is required)
 for binary matrix factorization
 5 -- logarithmic error
 6 -- squared hinge loss
 7 -- hinge loss
 for one-class matrix factorization
 10 -- row-oriented pair-wise logarithmic loss
 11 -- column-oriented pair-wise logarithmic loss
 12 -- squared error (L2-norm)
 -k <dimensions>: set number of dimensions (default 8)
 -t <iter>: set number of iterations (default 20)
 -r <eta>: set initial learning rate (default 0.1)
 -a <alpha>: set coefficient of negative entries' loss (default 1)
 -c <c>: set value of negative entries (default 0.0001).
 Every positive entry is assumed to be 1.
 -s <threads>: set number of threads (default 12)
 -n <bins>: set number of bins (may be adjusted by LIBMF for speed)
 -p <path>: set path to the validation set
 -v <fold>: set number of folds for cross validation
 --quiet: quiet mode (no outputs)
 --nmf: perform non-negative matrix factorization
 --disk: perform disk-level training (will create a buffer file)

 `mf-train' is the main training command of LIBMF. At each iteration, the
 following information is printed.

 - iter: the index of iteration.
 - tr_*: * is the evaluation criterion on the training set.
 - tr_*+: * is the evaluation criterion on the positive entries in the
 training set.
 - tr_*-: * is the evaluation criterion on the negative entries in the
 training set.
 - va_*: the same criterion on the validation set if `-p' is set
 - va_*+: * is the evaluation criterion on the positive entries in the
 validation set.
 - va_*-: * is the evaluation criterion on the negative entries in the
 validation set.
 - obj: objective function value.
 - reg: regularization term.

 Here `tr_*' and `obj' are estimations because calculating true values
 can be time-consuming. Different solvers can print different combinations
 those values.

 For different losses, the criterion to be printed is listed below.

 <loss>: <evaluation criterion>
 - 0: root mean square error (RMSE)
 - 1: mean absolute error (MAE)
 - 2: generalized KL-divergence (KL)
 - 5: logarithmic loss
 - 6 & 7: accuracy
 - 10 & 11: pair-wise logarithmic loss in Bayesian personalized ranking
 - 12: sum of squared errors. The label of positive entries is 1
 - while negative entries' value is set using command line
 - option -c.

- `mf-predict'

 usage: mf-predict [options] test_file model_file output_file

 options:
 -e <criterion>: set the evaluation criterion (default 0)
 0: root mean square error
 1: mean absolute error
 2: generalized KL-divergence
 5: logarithmic loss
 6: accuracy
 10: row-oriented mean percentile rank (row-oriented MPR)
 11: colum-oriented mean percentile rank (column-oriented MPR)
 12: row-oriented area under ROC curve (row-oriented AUC)
 13: column-oriented area under ROC curve (column-oriented AUC)

 `mf-predict' outputs the prediction values of the entries specified in
 `test_file' to the `output_file.' The selected criterion will be printed
 as well.



Examples
========
This section gives example commands of LIBMF using the data sets in `demo'
directory. In `demo,' a shell script `demo.sh' can be run for demonstration.

> mf-train real_matrix.tr.txt model

train a model using the default parameters

> mf-train -l1 0.05 -l2 0.01 real_matrix.tr.txt model

train a model with the following regularization coefficients:

 coefficient of L1-norm regularization on P = 0.05
 coefficient of L1-norm regularization on Q = 0.05
 coefficient of L2-norm regularization on P = 0.01
 coefficient of L2-norm regularization on Q = 0.01

> mf-train -l1 0.015,0 -l2 0.01,0.005 real_matrix.tr.txt model

train a model with the following regularization coefficients:

 coefficient of L1-norm regularization on P = 0.05
 coefficient of L1-norm regularization on Q = 0
 coefficient of L2-norm regularization on P = 0.01
 coefficient of L2-norm regularization on Q = 0.03

> mf-train -f 5 -l1 0,0.02 -k 100 -t 30 -r 0.02 -s 4 binary_matrix.tr.txt model

train a BMF model using logarithmic loss and the following parameters:

 coefficient of L1-norm regularization on P = 0
 coefficient of L1-norm regularization on Q = 0.01
 latent factors = 100
 iterations = 30
 learning rate = 0.02
 threads = 4

> mf-train -p real_matrix.te.txt real_matrix.tr.txt model

use real_matrix.te.txt for hold-out validation

> mf-train -v 5 real_matrix.tr.txt

do five fold cross validation

> mf-train -f 2 --nmf real_matrix.tr.txt

do non-negative matrix factorization with generalized KL-divergence

> mf-train --quiet real_matrix.tr.txt

do not print message to screen

> mf-train --disk real_matrix.tr.txt

do disk-level training

> mf-predict real_matrix.te.txt model output

do prediction

> mf-predict -e 1 real_matrix.te.txt model output

do prediction and output MAE



Library Usage
=============

These structures and functions are declared in the header file `mf.h.' You need
to #include `mf.h' in your C/C++ source files and link your program with
`mf.cpp.' Users can read `mf-train.cpp' and `mf-predict.cpp' as usage examples.

Before predicting test data, we need to construct a model (`mf_model') using
training data which is either a C structure `mf_problem' or the path to the
training file. For the first case, the whole data set needs to be fitted into
memory. For the second case, a binary version of the training file will be
created, and only some parts of the binary file are loaded at one time. Note
that a model can also be saved in a file for later use. To evaluate the quality
of a model, users can call an evaluation function in LIBMF with a `mf_problem'
and a `mf_model.'


There are four public data structures in LIBMF.

- struct mf_node
 {
 mf_int u;
 mf_int v;
 mf_float r;
 };

 `mf_node' represents an element in a sparse matrix. `u' represents the row
 index, `v' represents the column index, and `r' represents the value.


- struct mf_problem
 {
 mf_int m;
 mf_int n;
 mf_long nnz;
 struct mf_node *R;
 };

 `mf_problem' represents a sparse matrix. Each element is represented by
 `mf_node.' `m' represents the number of rows, `n' represents the number of
 columns, `nnz' represents the number of non-zero elements, and `R' is an
 array of `mf_node' whose length is `nnz.'


- struct mf_parameter
 {
 mf_int fun;
 mf_int k;
 mf_int nr_threads;
 mf_int nr_bins;
 mf_int nr_iters;
 mf_float lambda_p1;
 mf_float lambda_p2;
 mf_float lambda_q1;
 mf_float lambda_q2;
 mf_float alpha;
 mf_float c;
 mf_float eta;
 bool do_nmf;
 bool quiet;
 bool copy_data;
 };

 `mf_parameter' represents the parameters used for training. The meaning of
 each variable is:

 variable meaning default
 ================================================================
 fun loss function 0
 k number of latent factors 8
 nr_threads number of threads used 12
 nr_bins number of bins 20
 nr_iters number of iterations 20
 lambda_p1 coefficient of L1-norm regularization on P 0
 lambda_p2 coefficient of L2-norm regularization on P 0.1
 lambda_q1 coefficient of L1-norm regularization on Q 0
 lambda_q2 coefficient of L2-norm regularization on Q 0.1
 eta learning rate 0.1
 alpha importance of negative entries 0.1
 c desired value of negative entries 0.0001
 do_nmf perform non-negative MF (NMF) false
 quiet no outputs to stdout false
 copy_data copy data in training procedure true

 There are two major algorithm categories in LIBMF. One is for stochastic
 gradient method and the other one is for coordinate descent method. Both
 of them support multi-threading. Currently, the only solver used
 coordinate descent method is implemented for fun=12. All other types of loss
 functions such as fun=0 may use stochastic gradient method. Notice that
 when a framework does support the parameters specified, LIBMF may ignore
 them or throw an error.

 LIBMF's framework for stochastic gradient method:

 In LIBMF, we parallelize the computation by griding the data matrix
 into nr_bins^2 blocks. According to our experiments, this parameter is
 not sensitive to both effectiveness and efficiency. In most cases the
 default value should work well.

 For disk-level training, `nr_bins' controls the memory usage of
 because one thread accesss an entire block at one time. If `nr_bins'
 is 4 and `nr_threads' is 1, the expected usage of memory is 25% of the
 memory to store the whole training matrix.

 Let the training data is a `mf_problem.' By default, at the beginning
 of the training procedure, the data matrix is copied because it would
 be modified in the training process. To save memory, `copy_data' can
 be set to false with the following effects.

 (1) The raw data is directly used without being copied.
 (2) The order of nodes may be changed.
 (3) The value in each node may become slightly different.

 Note that `copy_data' is invalid for disk-level training.

 To obtain a parameter with default values, use the function
 `get_default_parameter.'

 Note that parameter alpha and c are not ignored under this framework. 

 LIBMF's framework for coordinate descent method:

 Currently, only one solver is implemented under this framework. It
 minimizes the squared errors overall the whole training matrix. Its
 regularization function is Frobenius norm on the two factor matrices
 P and Q. Note that the the original training matrix R (m-by-n) is
 approximated by P^TQ. This solver requires two copies of the original
 positive entries if `copy_data' is false. That is, if your input data is
 50MB, LIBMF may need 150MB memory in total for data storage. By
 setting `copy_data' to false, LIBMF will only make one extra copy.
 Disk-level training is not supported.

 Parameters recognized by this framework are `fun,' `k,' `nr_threads,'
 `nr_iters,' `lambda_p2,' `lambda_q2,' `alpha,' `c,' `quiet,' and
 `copy_data.'

 Unlike the standard C++ thread class used in stochastic gradient
 method's framework, the parallel computation here relies on OpenMP, so
 please make sure your complier can support it.


- struct mf_model
 {
 mf_int fun;
 mf_int m;
 mf_int n;
 mf_int k;
 mf_float b;
 mf_float *P;
 mf_float *Q;
 };

 `mf_model' is used to store models learned by LIBMF. `fun' indicates the
 loss function of the solved MF problem. `m' represents the number of rows,
 `n' represents the number of columns, `k' represents the number of latent
 factors, and `b' is the average of all elements in the training matrix. `P'
 is used to store a kxm matrix in column oriented format. For example, if
 `P' stores a 3x4 matrix, then the content of `P' is:

 P11 P21 P31 P12 P22 P32 P13 P23 P33 P14 P24 P34

 `Q' is used to store a kxn matrix in the same manner.


Functions available in LIBMF include:


- mf_parameter mf_get_default_param();

 Get default parameters.

- mf_int mf_save_model(struct mf_model const *model, char const *path);

 Save a model. It returns 0 on sucess and 1 on failure.

- struct mf_model* mf_load_model(char const *path);

 Load a model. If the model could not be loaded, a nullptr is returned.

- void mf_destroy_model(struct mf_model **model);

 Destroy a model.

- struct mf_model* mf_train(
 struct mf_problem const *prob,
 mf_parameter param);

 Train a model. A nullptr is returned if fail.

- struct mf_model* mf_train_on_disk(
 char const *tr_path,
 mf_parameter param);

 Train a model while parts of data is put in disk to reduce memory usage. A
 nullptr is returned if fail.

 Notice: the model is still fully loaded during the training process.

- struct mf_model* mf_train_with_validation(
 struct mf_problem const *tr,
 struct mf_problem const *va,
 mf_parameter param);

 Train a model with training set `tr' and validation set `va.' The
 evaluation criterion of the validation set is printed at each iteration.

- struct mf_model* mf_train_with_validation_on_disk(
 char const *tr_path,
 char const *va_path,
 mf_parameter param);

 Train a model using the training file `tr_path' and validation file
 `va_path' for holdout validation. The same strategy is used to save memory
 as in `mf_train_on_disk.' It also printed the same information as
 `mf_train_with_validation.'

 Notice: LIBMF assumes that the model and the validation set can be fully
 loaded into the memory.

- mf_float mf_cross_validation(
 struct mf_problem const *prob,
 mf_int nr_folds,
 mf_parameter param);

 Do cross validation with `nr_folds' folds.

- mf_float mf_predict(
 struct mf_model const *model,
 mf_int p_idx,
 mf_int q_idx);

 Predict the value at the position (p_idx, q_idx). The predicted value is a
 real number for RVMF or OCMF. For BMF, the range of the prediction values
 are {-1, 1}. If `p_idx' or `q_idx' can not be found in the training set,
 the function returns the average (mode if BMF) of all values in the
 training matrix.

- mf_double calc_rmse(mf_problem *prob, mf_model *model);

 calculate the RMSE of the model on a test set `prob.' It can be used to
 evaluate the result of real-valued MF.

- mf_double calc_mae(mf_problem *prob, mf_model *model);

 calculate the MAE of the model on a test set `prob.' It can be used to
 evaluate the result of real-valued MF.

- mf_double calc_gkl(mf_problem *prob, mf_model *model);

 calculate the Generalized KL-divergence of the model on a test set `prob.'
 It can be used to evaluate the result of non-negative RVMF.

- calc_logloss(mf_problem *prob, mf_model *model);

 calculate the logarithmic loss of the model on a test `prob.' It can be
 used to evaluate the result of BMF.

- mf_double calc_accuracy(mf_problem *prob, mf_model *model);

 calculate the accuracy of the model on a test `prob.' It can be used to
 evaluate the result of BMF.

- mf_double calc_mpr(mf_problem *prob, mf_model *model, bool transpose)

 calculate the MPR of the model on a test `prob.' If `transpose' is `false
 row-oriented MPR is calculated and otherwise column-oriented MPR. It can be
 used to evaluate the result of OCMF.

- calc_auc(mf_problem *prob, mf_model *model, bool transpose);

 calculate the row-oriented AUC of the model on a test `prob' if `transpose'
 is `false.' For column-oriented AUC, set `transpose' to be 'true.' It can
 be used to evaluate the result of OCMF.



SSE, AVX, and OpenMP
====================

LIBMF utilizes SSE instructions to accelerate the computation. If you cannot
use SSE on your platform, then please comment out

 DFLAG = -DUSESSE

in Makefile to disable SSE.

Some modern CPUs support AVX, which is more powerful than SSE. To enable AVX,
please comment out

 DFLAG = -DUSESSE

and uncomment the following lines in Makefile.

 DFLAG = -DUSEAVX
 CFLAGS += -mavx

If OpenMP is not available on your platform, please comment out the following
lines in Makefile.

 DFLAG += -DUSEOMP
 CXXFLAGS += -fopenmp

Notice: Please always run `make clean all' if these flags are changed.



Building Windows and Mac and Binaries
=====================================

- Windows

 Windows binaries are in the directory `windows.' To build them via
 command-line tools of Microsoft Visual Studio, use the following steps:

 1. Open a DOS command box (or Developer Command Prompt for Visual Studio)
 and go to libmf directory. If environment variables of VC++ have not been
 set, type

 "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\vcvars64.bat"

 You may have to modify the above command according which version of VC++ or
 where it is installed.

 2. Type

 nmake -f Makefile.win clean all

 3. (optional) To build shared library mf_c.dll, type

 nmake -f Makefile.win lib

- Mac

 To complie LIBMF on Mac, a GCC complier is required, and users need to
 slightly modify the Makefile. The following instructions are tested with
 GCC 4.9.

 1. Set the complier path to your GCC complier. For example, the first
 line in the Makefile can be

 CXX = g++-4.9

 2. Remove `-march=native' from `CXXFLAGS.' The second line in the Makefile
 Should be

 CXXFLAGS = -O3 -pthread -std=c++0x

 3. If AVX is enabled, we add `-Wa,-q' to the `CXXFLAGS,' so the previous
 `CXXFLAGS' becomes

 CXXFLAGS = -O3 -pthread -std=c++0x -Wa,-q



References
==========

[1] W.-S. Chin, Y. Zhuang, Y.-C. Juan, and C.-J. Lin. A Fast Parallel
Stochastic Gradient Method for Matrix Factorization in Shared Memory Systems.
ACM TIST, 2015. (www.csie.ntu.edu.tw/~cjlin/papers/libmf/libmf_journal.pdf)

[2] W.-S. Chin, Y. Zhuang, Y.-C. Juan, and C.-J. Lin. A Learning-rate Schedule
for Stochastic Gradient Methods to Matrix Factorization. PAKDD, 2015.
(www.csie.ntu.edu.tw/~cjlin/papers/libmf/mf_adaptive_pakdd.pdf)

[3] W.-S. Chin, B.-W. Yuan, M.-Y. Yang, Y. Zhuang, Y.-C. Juan, and C.-J. Lin.
LIBMF: A Library for Parallel Matrix Factorization in Shared-memory Systems.
JMLR, 2015.
(www.csie.ntu.edu.tw/~cjlin/papers/libmf/libmf_open_source.pdf)

For any questions and comments, please email:

 cjlin@csie.ntu.edu.tw