LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

Tuan Dinh*, Yuchen Zeng*, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee
University of Wisconsin-Madison
NeurIPS 2022
*Denotes equal contribution. Author order is alphabetical.
Paper Code Video

A high-level illustration of the LIFT framework. LIFT has four steps: (i) converting the dataset into sentences, (ii) fine-tuning the pretrained LLMs (e.g., GPT) on the obtained sentences, (iii) generating predictions, and (iv) converting the predictions back to the original data format.

Abstract

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling "no-code machine learning with LMs." We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning.

One Model, Many Tasks

LIFT for Regression

MY ALT TEXT Approximating various functions with LIFT using GPT-J. We visualize the target functions (first row) and the predictor functions learned by LIFT on GPT-J (second row). Blue dots show the 1000 training samples. One can observe that LIFT well approximates the target functions.

LIFT for Tabular Classification

0%
100%

Accuracies (↑) on OpenML tabular classification datasets. We evaluate LIFT/GPTs on OpenML tabular datasets, varying number of features and data classes. Overall, LIFT/GPTs perform comparably well across tasks, and achieve co mpetitive performances with the best methods, e.g., XGBoost.

LIFT for Image Classification

MY ALT TEXT
0%
100%

Top: Illustration of using LIFT for image classification. Bottom: Accuracies (↑) on image classification datasets. LIFT/GPTs achieve competitive accuracies on this task.

LIFT for Image Generation

MY ALT TEXT

Top: Illustration of using LIFT for image generation. Bottom: Generating MNIST images using LIFT/GPTs. We observe that LIFT/GPTs can generate images of comparably high quality.

Understanding LIFT: A Deep Dive

Language-Interfaced Learning: Fine-Tuning versus In-Context Learning

Comparison of accuracies (↑) between Language-Interfaced ICL (LIICL) and Language-Interfaced Fine-Tuning (LIFT) on OpenML datasets.

Due to the context length limit of LMs, we use a subset of the training data for ICL. Here, the size of the subset is chosen to satisfy the LMs' context length. Overall, LIFT/GPTs on full data achieve the best performances. However, when using the same number of samples, LIFT and ICL are more comparable in most cases. Note that both methods may be worse than Majority Classifier due to the limited training data in some cases.

Yes, LIFT Needs Pretrained Knowledge on Natural Language Data

Comparison of accuracies (↑) between different models.

Accuracies (↑) of LIFT with different LMs. We compare variants of LIFT with different LMs: LIFT/GPTs using GPTs pretrained on natural language data (our models), LIFT/Rand-GPT-J using a randomly initialized GPT-J, LIFT/CodeGen and LIFT/CodeParrot using LMs pretrained on programming language data, and LIFT/Gibberish using GPT-J fine-tuned on gibberish data.

Robustness to Feature Corruption on Test Data

Results of PGD Attack on Different Models and Attack Strength.

Robustness results on MNIST classification under PGD attacks transferred from LeNet-5 with varying perturbation radius ε. LIFT/GPT-3 cannot tolerate transferred adversarial attack, implying that the adversarial attack on LeNet-5 is transferred to LIFT/GPT-3.

Effect of Context-Aware Prompting

Comparison of accuracies (↑) between models with and without correct feature names on OpenML datasets.

The effect of using feature names on LIFT. We compare classification accuracy (↑) of LIFT/GPT-3 when feature names provided in the target dataset are and are not incorporated into the prompts. We provide four versions of LIFT when feature names are correctly incorporated (Correct-Names columns) and when feature names are randomly shuffled (Shuffled-Names columns). We evaluate models on three OpenML datasets, including CMC, TAE, and Vehicle. We also compare our models with the majority classifier. As a result, all LIFT models achieve better performance than the majority classifier. Among the evaluated models, LIFTs with correct feature names achieve the best accuracies on all three datasets.

Beyond Basics: Improvements

Two-Stage Fine-Tuning

Accuracies (↑) of LIFT/GPT-J with/without two-stage fine-tuning on tabular datasets.

For any given dataset, we first generate two pretext tasks with simple synthetic Gaussian datasets sharing the same number of features and the label space (for classification tasks) or the range of responses’ values (for the regression tasks) to the actual data. We apply LIFT first on synthetic pretext data before the real datasets, outperforming fine-tuning when training data is small.

Improving Robustness via Data Augmentation

Accuracies (↑) of LIFT/GPT-J with/without data augmentation on MNIST.

Data augmentation means that we are using a noisy version of MNIST training data by adding Gaussian noise. Given an MNIST image having range [0,1], different levels of noise are added. One can confirm that the data augmentation significantly improves the tolerance of LIFT/GPT-J against perturbed test data in both Gaussian and signed constant noise.

Building on Our Research: Spotlights

Predictive Chemistry

Leveraging large language models for predictive chemistry.

Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, Berend Smit
Physics of Metamaterial

Can Large Language Models Learn the Physics of Metamaterials? An Empirical Study with ChatGPT.

Darui Lu, Yang Deng, Jordan M. Malof, Willie J. Padilla
Bayesian Inference

Large Language Models to Enhance Bayesian Optimization.

Tennison Liu*, Nicolás Astorga*, Nabeel Seedat, Mihaela van der Schaar

Cite Us


        @inproceedings{tuan_zeng_2022_lift,
          author = {Dinh, Tuan and Zeng, Yuchen and Zhang, Ruisu and Lin, Ziqian and Gira, Michael and Rajput, Shashank and Sohn, Jy-yong and Papailiopoulos, Dimitris and Lee, Kangwook},
          booktitle = {Advances in Neural Information Processing Systems},
          pages = {11763--11784},
          title = {LIFT: Language-Interfaced Fine-Tuning for Non-language Machine Learning Tasks},
          volume = {35},
          year = {2022}
         }