NeurlPS 2022 | Classification of NLP Related Papers

MLNLP community is a well-known machine learning and natural language processing community in China and abroad, covering NLP master’s and doctoral students, university teachers, and industry researchers. The vision of the community is to promote communication and progress between the academic and industrial fields of natural language processing and machine learning, especially for beginners.

Reprinted from | RUC AI Box

Author | Wang Xiaolei

Institution | Renmin University of China, Gaoling School of Artificial Intelligence

Research Direction | Conversational Information Retrieval

This article selects over 200 papers related to natural language processing from more than 2000 accepted papers at NeurIPS 2022 and categorizes them according to research themes for reference. The article is also published on the AI Box Zhihu column (search for AI Box column on Zhihu), and everyone is welcome to comment and discuss below the articles in the Zhihu column!

Introduction:

NeurIPS 2022 is a CCF Class A conference and one of the top international conferences in the field of artificial intelligence. The 36th Conference on Neural Information Processing Systems will be held from November 28 to December 9 this year. The official list of accepted papers is available at the following link: https://nips.cc/Conferences/2022/Schedule?type=Poster.

This article selects over 200 papers related to natural language processing from more than 2000 accepted papers and categorizes them according to research themes for reference. The paper list is also updated on GitHub, and everyone is welcome to follow and star: github.com/RUCAIBox/Top-conference-paper-list.

Table of Contents:

• Model 【模型】

• Interpretability, Analysis and Evaluation 【可解释性、分析、评测】

• Robustness and Safety 【鲁棒性与安全】

• Knowledge and Reasoning 【知识与推理】

• Information Extraction 【信息抽取】

• Information Retrieval 【信息检索】

• Text Classification 【文本分类】

• Text Generation 【文本生成】

• Machine Translation and Multilinguality 【机器翻译与多语言】

• Multimodality 【多模态】

• Special Tasks 【特殊任务】

1

『Model 【模型】』

1. Model Design 【模型设计】

• Recurrent Memory Transformer

• Jump Self-attention: Capturing High-order Statistics in Transformers

• Block-Recurrent Transformers

• Staircase Attention for Recurrent Processing of Sequences

• Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings

• Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling

• Mixture-of-Experts with Expert Choice Routing

• On the Representation Collapse of Sparse Mixture of Experts

• Improving Transformer with an Admixture of Attention Heads

• Your Transformer May Not be as Powerful as You Expect

• Confident Adaptive Language Modeling

• Decoupled Context Processing for Context Augmented Language Modeling

• Unsupervised Cross-Task Generalization via Retrieval Augmentation

• Revisiting Neural Scaling Laws in Language and Vision

• Learning to Scaffold: Optimizing Model Explanations for Teaching

2. Model Compression 【模型压缩】

• Information-Theoretic Generative Model Compression with Variational Energy-based Model

• Towards Efficient Post-training Quantization of Pre-trained Language Models

• Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

• Deep Compression of Pre-trained Transformer Models

• LiteTransformerSearch: Training-free On-device Search for Efficient Autoregressive Language Models

• GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale

• MorphTE: Injecting Morphology in Tensorized Embeddings

• Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models

• A Fast Post-Training Pruning Framework for Transformers

3. Model Training 【模型训练】

• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

• A Data-Augmentation Is Worth A Thousand Samples

• TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers

• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models

• Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction

• Training and Inference on Any-Order Autoregressive Models the Right Way

• Decentralized Training of Foundation Models in Heterogeneous Environment

4. Model Usage 【模型使用】

• The Unreliability of Explanations in Few-Shot In-Context Learning

• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

• Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning

• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

• Training language models to follow instructions with human feedback

• LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

• How to talk to your model: Instructions, descriptions, and learning

• Data Distributional Properties Drive Emergent In-Context Learning in Transformers

• Sparse Structure Search for Parameter-Efficient Tuning

• Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

• Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

• LIFT: Language-Interfaced FineTuning for Non-language Machine Learning Tasks

• Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts

2

『Interpretability, Analysis and Evaluation 【可解释性、分析、评测】』

• CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior

• Rule-Based but Flexible? Evaluating and Improving Language Models as Accounts of Human Moral Judgment

• Understanding the Failure of Batch Normalization for Transformers in NLP

• AttCAT: Explaining Transformers via Attentive Class Activation Tokens

• An empirical analysis of compute-optimal large language model training

• Why GANs are overkill for NLP

• Exploring Length Generalization in Large Language Models

• Capturing Failures of Large Language Models via Human Cognitive Biases

• Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning

• First is Better Than Last for Language Data Influence

• What are the best Systems? New Perspectives on NLP Benchmarking

• Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

• FETA: Towards Specializing Foundational Models for Expert Task Applications

• This is the way – lessons learned from designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish

• Rethinking Knowledge Graph Evaluation Under the Open-World Assumption

• A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction

3

『Robustness and Safety 【鲁棒性与安全】』

• Active Learning Helps Pretrained Models Learn the Intended Task

• Improving Certified Robustness via Statistical Learning with Logical Reasoning

• Moderate-fitting as a Natural Backdoor Defender for Pre-trained Language Models

• BadPrompt: Backdoor Attacks on Continuous Prompts

• A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models

• Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

• AD-DROP: Attribution Driven Dropout for Robust Language Model Finetuning

• Large (robust) models from computational constraints

• Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve

• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks

• Recovering Private Text in Federated Learning of Language Models

• LAMP: Extracting Text from Gradients with Language Model Priors

• SeqPATE: Differentially Private Text Generation via Knowledge Distillation

• Differentially Private Model Compression

• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach

4

『Knowledge and Reasoning 【知识与推理】』

• Learning to Sample and Aggregate: Few-shot Reasoning over Temporal Knowledge Graph

• Retaining Knowledge for Learning with Dynamic Definition

• Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer

• What Makes a “Good” Data Augmentation in Knowledge Distillation – A Statistical Perspective

• Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures

• Roadblocks for Temporarily Disabling Shortcuts and Learning New Knowledge

• PALBERT: Teaching ALBERT to Ponder

• Locating and Editing Factual Associations in GPT

• OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport

• Large Language Models are Zero-Shot Reasoners

• STaR: Bootstrapping Reasoning With Reasoning

• Chain of Thought Prompting Elicits Reasoning in Large Language Models

• ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler

• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

• Inductive Logical Query Answering in Knowledge Graphs

• Formalizing Coherence and Consistency Applied to Transfer Learning in Neuro-Symbolic Autoencoders

• CoNSoLe: Convex Neural Symbolic Learning

• Deep Bidirectional Language-Knowledge Pretraining

• Neurosymbolic Deep Generative Models for Sequence Data with Relational Constraints

• Instance-based Learning for Knowledge Base Completion

• LogiGAN: Learning Logical Reasoning via Adversarial Pre-training

• Learning robust rule representations for abstract reasoning via internal inferences

• Solving Quantitative Reasoning Problems with Language Models

• Towards Better Evaluation for Dynamic Link Prediction

• Predictive Querying for Autoregressive Neural Sequence Models

• Semantic Probabilistic Layers for Neuro-Symbolic Learning

• End-to-end Symbolic Regression with Transformers

• A Unified Framework for Deep Symbolic Regression

• ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time

5

『Information Extraction 【信息抽取】』

• Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model

• TweetNERD – End to End Entity Linking Benchmark for Tweets

• METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets

6

『Information Retrieval 【信息检索】』

• Transformer Memory as a Differentiable Search Index

• Autoregressive Search Engines: Generating Substrings as Document Identifiers

• A Neural Corpus Indexer for Document Retrieval

7

『Text Classification 【文本分类】』

• CascadeXML: End-to-end Multi-Resolution Learning for Extreme Multi-Label Text Classification

• Text Classification with Born’s Rule

• Public Wisdom Matters! Discourse-Aware Hyperbolic Fourier Co-Attention for Social Text Classification

8

『Text Generation 【文本生成】

• CoNT: Contrastive Neural Text Generation

• A Character-Level Length Control Algorithm for Non-Autoregressive Sentence Summarization

• Towards Improving Faithfulness in Abstractive Summarization

• QUARK: Controllable Text Generation with Reinforced Unlearning

• Teacher Forcing Recovers Reward Functions for Text Generation

• Retrieve, Reason, and Refine: Generating Accurate and Faithful Patient Instructions

• A Contrastive Framework for Neural Text Generation

• Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation

• COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics

• Diffusion-LM Improves Controllable Text Generation

• Factuality Enhanced Language Models for Open-Ended Text Generation

• Controllable Text Generation with Neurally-Decomposed Oracle

• InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model

• Relation-Constrained Decoding for Text Generation

• EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

• TGEA 2.0: A Large-Scale Diagnostically Annotated Dataset with Benchmark Tasks for Text Generation of Pretrained Language Models

9

『Machine Translation and Multilinguality 【机器翻译与多语言】』

• Exploring Non-Monotonic Latent Alignments for Non-Autoregressive Machine Translation

• A new dataset for multilingual keyphrase generation

• Less-forgetting Multi-lingual Fine-tuning

• Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing

• Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Translation Model

• OccGen: Selection of Real-world Multilingual Parallel Data Balanced in Gender within Occupations

• Multilingual Abusive Comment Detection at Scale for Indic Languages

• The BigScience Corpus A 1.6TB Composite Multilingual Dataset

• Addressing Resource Scarcity across Sign Languages with Multilingual Pretraining and Unified-Vocabulary Datasets

10

『Multimodality 【多模态】

• REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering

• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

• GLIPv2: Unifying Localization and Vision-Language Understanding

• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

• A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval

• Egocentric Video-Language Pretraining

• Flamingo: a Visual Language Model for Few-Shot Learning

• Language Conditioned Spatial Relation Reasoning for 3D Object Grounding

• Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

• Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies

• OmniVL: One Foundation Model for Image-Language and Video-Language Tasks

• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

• Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

• TVLT: Textless Vision-Language Transformer

• Divert More Attention to Vision-Language Tracking

• Flamingo: a Visual Language Model for Few-Shot Learning

• Self-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

• UniCLIP: Unified Framework for Contrastive Language-Image Pre-training

• Contrastive Language-Image Pre-Training with Knowledge Graphs

• PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

• Enhancing and Scaling Cross-Modality Alignment for Contrastive Multimodal Pre-Training via Gradient Harmonization

• Mutual Information Divergence: A Unified Metric for Multimodal Generative Models

• Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

• MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching

• HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes

• CyCLIP: Cyclic Contrastive Language-Image Pretraining

• S-Prompts Learning with Pre-trained Transformers: An Occam’s Razor for Domain Incremental Learning

• Delving into OOD Detection with Vision-Language Representations

• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

• Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

• DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

• Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

• CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation

• Relational Language-Image Pre-training for Human-Object Interaction Detection

• Fine-Grained Semantically Aligned Vision-Language Pre-Training

• Cross-Linked Unified Embedding for cross-modality representation learning

• Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP

• Kernel Multimodal Continuous Attention

• Paraphrasing Is All You Need for Novel Object Captioning

• Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning

• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders

• One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations

• LGDN: Language-Guided Denoising Network for Video-Language Modeling

• Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

• WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models

• VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

• ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

• LAION-5B: An open large-scale dataset for training next generation image-text models

• Towards Video Text Visual Question Answering: Benchmark and Baseline

• TaiSu: A 166M Large-scale High-Quality Dataset for Chinese Vision-Language Pre-training

• Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark

• Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment

• Multi-modal Robustness Analysis Against Language and Visual Perturbations

• CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

• OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression

11

『Special Tasks 【特殊任务】』

1. Code 【代码】

• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

• Fault-Aware Neural Code Rankers

• NS3: Neuro-symbolic Semantic Code Search

• Pyramid Attention For Source Code Summarization

2. Mathematics 【数学】

• HyperTree Proof Search for Neural Theorem Proving

• NaturalProver: Grounded Mathematical Proof Generation with Language Models

• Autoformalization with Large Language Models

• Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers

3. Others 【其他】

• Measuring and Reducing Model Update Regression in Structured Prediction for NLP

• Learning to Follow Instructions in Text-Based Games

• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

• LISA: Learning Interpretable Skill Abstractions from Language

• Inherently Explainable Reinforcement Learning in Natural Language

• Using natural language and program abstractions to instill human inductive biases in machines

• Semantic Exploration from Language Abstractions and Pretrained Representations

• Pre-Trained Language Models for Interactive Decision-Making

• Knowledge-Aware Bayesian Deep Topic Model

• Improving Intrinsic Exploration with Language Abstractions

• Improving Policy Learning via Language Dynamics Distillation

• Meta-Complementing the Semantics of Short Texts in Neural Topic Models

• Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

• BigBio: A Framework for Data-Centric Biomedical Natural Language Processing

Technical Group Chat Invitation

△ Long press to add the assistant

Scan the QR code to add the assistant WeChat

Please note: Name-School/Company-Research Direction (e.g., Xiao Zhang-Harbin Institute of Technology-Dialogue System) to apply to join the Natural Language Processing/Pytorch and other technical group chats

About Us

MLNLP community is a civil academic community jointly established by scholars in machine learning and natural language processing from home and abroad. It has now developed into a well-known machine learning and natural language processing community, aiming to promote progress between the academic and industrial fields of machine learning and natural language processing.The community can provide an open exchange platform for the further study, employment, and research of relevant practitioners. Everyone is welcome to pay attention to and join us.