The goal of this homework is to compare two different supervised machine learning models: knearest neighbors (Knn) and support vector machines (SVM). Knn is a hugely popular algorithm that is used only for small datasets, based solely on the proximity of the points. It is a lazy algorithm: this means that it only has a short training phase, or is not trained at all. The most important hyperparameter to set is "K", which represents the number of neighbors needed to determine which category the new sample belongs to. SVM, instead, is based on the "dual theory" and it’s made to solve classification problems using both linear and non-linear kernel. It’s planned to solve large scale problems, especially the " LinearSVC" class of the sklearn library, but we will deal with this later.
To read more about homework 1 click here.
The premise to the homework that will be carried out later, consists of a brief introduction relating to the environment and the tools that will be used to perform the assigned task. We must first consider that in this case a great computational effort is needed to train deep models that will have to be trained using GPUs. The Google project called Colab, therefore, fits perfectly in this case because it allows you to use (although with some limitations) Google GPUs to train the models for free. Colab also allows you to take advantage of 12 GB of DDR5 RAM and about 70 GB of SSD. It may seem that we have many resources but, as we will find out later, the memory required to handle this type of problem is very huge, therefore many arrangements are necessary. Deep neural models such as AlexNet, ResNet etc. are very well-known models, and many frameworks further implement them. In this homework, we should use the Pytorch library developed by Facebook. It’s not a common choice, since many developers prefer to work with Keras and TensorFlow which are developed by Google instead. The main advantage that derives from the use of Pytorch consists in the fact that, with this library, it is possible to work simultaneously at a high and low level. Vice versa, using the models produced by Google, if we want to work at a higher level we’d have to use Keras, otherwise it’d be necessary to use TensorFlow. The main drawback of the Facebook library is that, as it is not very well-known, it is difficult to find, online, feedback or suggestions if a problem arises. The dataset (ds) used in this homework is called Caltech101, as it was developed by some researchers from the Californian Institute of Technology. In this ds, as we can imagine, there are images of objects belonging to 101 different categories. Most of the categories catalog about 50 images. The size of each image is approximately 300 x 200 pixels. It’s not a huge dataset, so achieving a good accuracy score could be difficult. The model used at first is AlexNet: it is an old model designed designed by Alex Krizhevsky in 2012. AlexNet receives 224 X 224 X 3 images as input, and each of them passes through 8 different layers. The firsts are convolutional layers, followed by max-pooling layers, and the last three are fully connected layers. Non-linearity is introduced by the ReLU activation function. It is certainly not the state-of-art model with which we can deal with, but it will still represent our first baseline.
To read more about homework 2 click here.
As shown in the last homeworks, CNNs are very powerful tools and, wanting to make a classification task, for example, they prove very precise in the forecast if they have been trained with a lot of data, and the test set has a similar distribution. Even if we don’t have a large number of photos, pre-trained nets on large ds (like imagenet) still allow us to obtain extraordinary results. However, when using a CNN, we assume first that the train data distribution is representative of the test distribution. If this hypothesis fails, CNN can behave negatively. Recently, many researchers have tried to solve this problem and many different methods have been developed to do it. All of these methods improve performance, but none of them is the ultimately solution to the problem. This homework will be focused on the Domain-Adversarial neural network (Dann) developed by Ganin et al. in the Pacs dataset. The network is divided into two branches: the blue layers, fully connected, which predict the label of the image, while the pink ones provide for the domain of the photo. The idea behind this method is really simple. We want to create domain-independent features (f). This means that, when the network is trained, the domain predictor will no longer be able to tell if a photo belongs to one domain or another. At the same time, the f predictors will be good enough to allow for a better result. This method is developed using two different losses: the classification loss and the domain confusion loss. The first loss is minimized using gradient descent, while to create domain-independent predictors the confusion loss is maximized using gradient ascent and in particular a gradient reversal layer. One well-known dataset that can be used to try to test this method is the PACS dataset. This ds has four different domains: art painting, cartoon, sketch and photos. For each domain, we have the same seven classes. It is evident from the ds that, while artistic painting and cartoon are often similar, the other two classes are significantly different; the sketch will be the most difficult to predict because, while the other domains use colours, the sketches are in black and white. Moreover, the data set is not large enough because it contains less than ten thousand photos. So to increase the performance we use transfer learning in particular, we will use an AlexNet model trained on the ImageNet dataset.
To read more about homework 3 click here.
Written by MarcoChain.