My main goal is to teach machines how to understand and generate human language and vision. To do so, I have explored ways to improve the controllability of language and visual generation models, incorporate and assess their reasoning capabilities, and integrate structured, multimodal, and linguistic information to enhance them.
I worked with Eduard Hovy at CMU's Language Technologies Institute and Malihe Alikhani at the University of Pittsburgh on research projects involving language generation, semantics, and data augmentation. Earlier, I worked at the University of Waterloo with Jesse Hoey.
I am a co-instructor for the Stanford CS25 Transformers course, and mentor and advise several students. I also led the organization of CtrlGen, a controllable generation workshop at NeurIPS 2021, and was involved in the GEM benchmark and workshop for NLG evaluation.
Jan. 2023: Excited to be co-instructing Stanford's CS25 Transformers course! We feature amazing speakers each week. We also have a Discord server that is open to everybody - it's a more general "Transformers community".
Jan. 2023: Two papers accepted to EACL 2023! CHARD: clinical reasoning with text generation, and PANCETTA: automatically generating tongue twisters using language models!
May 2021: Our data augmentation survey paper was accepted to ACL 2021 Findings! It has received lots of attention on social media (e.g. this tweet, Sebastian Ruder's NLP Newsletter) and was one of the top 10 hottest machine learning papers in May 2021 (source: labml.ai).
Peer-Reviewed Publications and Conference Proceedings
We motivate and introduce CHARD: Clinical Health-Aware Reasoning across Dimensions, to investigate the capability of text generation models to act as implicit clinical knowledge bases and generate free-flow textual explanations about various health-related conditions across several dimensions. We collect and present an associated dataset, CHARDat, consisting of explanations about 52 health conditions across three clinical dimensions. We conduct extensive experiments using BART and T5 along with data augmentation, and perform automatic, human, and qualitative analyses. We show that while our models can perform decently, CHARD is very challenging with strong potential for further exploration.
@inproceedings{feng-etal-2023-CHARD,
title = "{CHARD}: Clinical Health-Aware Reasoning Across Dimensions for Text Generation Models",
author = "Feng, Steven Y. and Khetan, Vivek and Sacaleanu, Bogdan and Gershman, Anatole and Hovy, Eduard",
booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
month = may,
year = "2023",
url = "https://arxiv.org/abs/2210.04191"}
Tongue twisters are meaningful sentences that are difficult to pronounce. The process of automatically generating tongue twisters is challenging since the generated utterance must satisfy two conditions at once: phonetic difficulty and semantic meaning. Furthermore, phonetic difficulty is itself hard to characterize and is expressed in natural tongue twisters through a heterogeneous mix of phenomena such as alliteration and homophony. In this paper, we propose PANCETTA: Phoneme Aware Neural Completion to Elicit Tongue Twisters Automatically. We leverage phoneme representations to capture the notion of phonetic difficulty, and we train language models to generate original tongue twisters on two proposed task settings. To do this, we curate a dataset called PANCETTA, consisting of existing English tongue twisters. Through automatic and human evaluation, as well as qualitative analysis, we show that PANCETTA generates novel, phonetically difficult, fluent, and semantically meaningful tongue twisters.
@inproceedings{keh-etal-2023-PANCETTA,
title = "{PANCETTA}: Phoneme Aware Neural Completion to Elicit Tongue Twisters Automatically",
author = "Keh, Sedrick Scott and Feng, Steven Y. and Gangal, Varun and Alikhani, Malihe and Hovy, Eduard",
booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
month = may,
year = "2023",
url = "https://arxiv.org/abs/2209.06275"}
A personification is a figure of speech that endows inanimate entities with properties and actions typically seen as requiring animacy. In this paper, we explore the task of personification generation. To this end, we propose PINEAPPLE: Personifying INanimate Entities by Acquiring Parallel Personification data for Learning Enhanced generation. We curate a corpus of personifications called PersonifCorp, together with automatically generated de-personified literalizations of these personifications. We demonstrate the usefulness of this parallel corpus by training a seq2seq model to personify a given literal input. Both automatic and human evaluations show that fine-tuning with PersonifCorp leads to significant gains in personification-related qualities such as animacy and interestingness. A detailed qualitative analysis also highlights key strengths and imperfections of PINEAPPLE over baselines, demonstrating a strong ability to generate diverse and creative personifications that enhance the overall appeal of a sentence.
@inproceedings{keh-etal-2022-pineapple,
title = "{PINEAPPLE}: Personifying {IN}animate Entities by Acquiring Parallel Personification Data for Learning Enhanced Generation",
author = "Keh, Sedrick Scott and Lu, Kevin and Gangal, Varun and Feng, Steven Y. and Jhamtani, Harsh and Alikhani, Malihe and Hovy, Eduard",
booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
month = oct,
year = "2022",
address = "Gyeongju, Republic of Korea",
publisher = "International Committee on Computational Linguistics",
url = "https://aclanthology.org/2022.coling-1.547",
pages = "6270--6284"}
We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.
@article{Feng_Lu_Tao_Alikhani_Mitamura_Hovy_Gangal_2022,
title={Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models},
volume={36},
url={https://ojs.aaai.org/index.php/AAAI/article/view/21306},
DOI={10.1609/aaai.v36i10.21306},
number={10},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Feng, Steven Y. and Lu, Kevin and Tao, Zhuofu and Alikhani, Malihe and Mitamura, Teruko and Hovy, Eduard and Gangal, Varun},
year={2022},
month={Jun.},
pages={10618-10626}}
Many implicit inferences exist in text depending on how it is structured that can critically impact the text's interpretation and meaning. One such structural aspect present in text with chronology is the order of its presentation. For narratives or stories, this is known as the narrative order. Reordering a narrative can impact the temporal, causal, event-based, and other inferences readers draw from it, which in turn can have strong effects both on its interpretation and interestingness. In this paper, we propose and investigate the task of Narrative Reordering (NAREOR) which involves rewriting a given story in a different narrative order while preserving its plot. We present a dataset, NAREORC, with human rewritings of stories within ROCStories in non-linear orders, and conduct a detailed analysis of it. Further, we propose novel task-specific training methods with suitable evaluation metrics. We perform experiments on NAREORC using state-of-the-art models such as BART and T5 and conduct extensive automatic and human evaluations. We demonstrate that although our models can perform decently, NAREOR is a challenging task with potential for further exploration. We also investigate two applications of NAREOR: generation of more interesting variations of stories and serving as adversarial sets for temporal/event-related tasks, besides discussing other prospective ones, such as for pedagogical setups related to language skills like essay writing and applications to medicine involving clinical narratives.
@article{Gangal_Feng_Alikhani_Mitamura_Hovy_2022,
title={NAREOR: The Narrative Reordering Problem},
volume={36},
url={https://ojs.aaai.org/index.php/AAAI/article/view/21309},
DOI={10.1609/aaai.v36i10.21309},
number={10},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
author={Gangal, Varun and Feng, Steven Y. and Alikhani, Malihe and Mitamura, Teruko and Hovy, Eduard},
year={2022},
month={Jun.},
pages={10645-10653}}
We motivate and propose a suite of simple but effective improvements for concept-to-text generation called SAPPHIRE: Set Augmentation and Post-hoc PHrase Infilling and REcombination. We demonstrate their effectiveness on generative commonsense reasoning, a.k.a the CommonGen task, through experiments using both BART and T5 models. Through extensive automatic and human evaluation, we show that SAPPHIRE noticeably improves model performance. An in-depth qualitative analysis illustrates that SAPPHIRE effectively addresses many issues of the baseline model generations, including lack of commonsense, insufficient specificity, and poor fluency.
@inproceedings{feng-etal-2021-sapphire,
title = "{SAPPHIRE}: Approaches for Enhanced Concept-to-Text Generation",
author = "Feng, Steven and
Huynh, Jessica and
Narisetty, Chaitanya Prasad and
Hovy, Eduard and
Gangal, Varun",
booktitle = "Proceedings of the 14th International Conference on Natural Language Generation",
month = aug,
year = "2021",
address = "Aberdeen, Scotland, UK",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.inlg-1.21",
pages = "212--225"}
Data augmentation has recently seen increased interest in NLP due to more work in low-resource domains, new tasks, and the popularity of large-scale neural networks that require large amounts of training data. Despite this recent upsurge, this area is still relatively underexplored, perhaps due to the challenges posed by the discrete nature of language data. In this paper, we present a comprehensive and unifying survey of data augmentation for NLP by summarizing the literature in a structured manner. We first introduce and motivate data augmentation for NLP, and then discuss major methodologically representative approaches. Next, we highlight techniques that are used for popular NLP applications and tasks. We conclude by outlining current challenges and directions for future research. Overall, our paper aims to clarify the landscape of existing literature in data augmentation for NLP and motivate additional work in this area. We also present a GitHub repository with a paper list that will be continuously updated at this link.
@inproceedings{feng-etal-2021-survey,
title = "A Survey of Data Augmentation Approaches for {NLP}",
author = "Feng, Steven Y. and
Gangal, Varun and
Wei, Jason and
Chandar, Sarath and
Vosoughi, Soroush and
Mitamura, Teruko and
Hovy, Eduard",
booktitle = "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021",
month = aug,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.findings-acl.84",
doi = "10.18653/v1/2021.findings-acl.84",
pages = "968--988"}
In this paper, we investigate data augmentation for text generation, which we call GenAug. Text generation and language modeling are important tasks within natural language processing, and are especially challenging for low-data regimes. We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods, and that the quality of generations improves to a peak at approximately three times the amount of original data.
@inproceedings{feng-etal-2020-genaug,
title = "{G}en{A}ug: Data Augmentation for Finetuning Text Generators",
author = "Feng, Steven Y. and
Gangal, Varun and
Kang, Dongyeop and
Mitamura, Teruko and
Hovy, Eduard",
booktitle = "Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.deelio-1.4",
doi = "10.18653/v1/2020.deelio-1.4",
pages = "29--42"}
For conversational AI and virtual assistants to communicate with humans in a realistic way, they must exhibit human characteristics such as expression of emotion and personality. Current attempts toward constructing human-like dialogue agents have presented significant difficulties. We propose Human Level Attributes (HLAs) based on tropes as the basis of a method for learning dialogue agents that can imitate the personalities of fictional characters. Tropes are characteristics of fictional personalities that are observed recurrently and determined by viewers' impressions. By combining detailed HLA data with dialogue data for specific characters, we present a dataset, HLA-Chat, that models character profiles and gives dialogue agents the ability to learn characters' language styles through their HLAs. We then introduce a three-component system, ALOHA (which stands for Artificial Learning of Human Attributes), that combines character space mapping, character community detection, and language style retrieval to build a character (or personality) specific language model. Our preliminary experiments demonstrate that two variations of ALOHA, combined with our proposed dataset, can outperform baseline models at identifying the correct dialogue responses of chosen target characters, and are stable regardless of the character's identity, the genre of the show, and the context of the dialogue.
@article{Li_2020,
title={ALOHA: Artificial Learning of Human Attributes for Dialogue Agents},
volume={34},
ISSN={2159-5399},
url={http://dx.doi.org/10.1609/aaai.v34i05.6328},
DOI={10.1609/aaai.v34i05.6328},
number={05},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
publisher={Association for the Advancement of Artificial Intelligence (AAAI)},
author={Li, Aaron W. and Jiang, Veronica and Feng, Steven Y. and Sprague, Julia and Zhou, Wei and Hoey, Jesse},
year={2020},
month={Apr},
pages={8155–8163}}
In this paper, we present a novel method for measurably adjusting the semantics of text while preserving its sentiment and fluency, a task we call semantic text exchange. This is useful for text data augmentation and the semantic correction of text generated by chatbots and virtual assistants. We introduce a pipeline called SMERTI that combines entity replacement, similarity masking, and text infilling. We measure our pipeline’s success by its Semantic Text Exchange Score (STES): the ability to preserve the original text’s sentiment and fluency while adjusting semantic content. We propose to use masking (replacement) rate threshold as an adjustable parameter to control the amount of semantic change in the text. Our experiments demonstrate that SMERTI can outperform baseline models on Yelp reviews, Amazon reviews, and news headlines.
@inproceedings{feng-etal-2019-keep,
title = "Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange",
author = "Feng, Steven Y. and
Li, Aaron W. and
Hoey, Jesse",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
month = nov,
year = "2019",
address = "Hong Kong, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D19-1272",
doi = "10.18653/v1/D19-1272",
pages = "2701--2711"}
* Equal Contribution
Talks and Interviews
July 2021: Eduard Hovy and I were on The Data Exchange Podcast with Ben Lorica. We discuss data augmentation for NLP (inspired by our survey paper) and challenges + future directions in NLP and machine learning research. Audio and notes here.
Aug. 2021: Varun and I gave a talk (to over 100 attendees) for Google Research about data augmentation for NLP (inspired by our survey paper). We also touch upon NL-Augmenter and our CtrlGen Workshop at NeurIPS 2021.
Teaching and Instruction
Stanford's CS25: Transformers United - I am a co-instructor for Stanford's CS25 course! We feature in-depth discussion from exciting speakers each week about cutting-edge research in Transformers for various ML areas/applications. We have a Discord server open to everyone - a more general "Transformers community". Some speakers so far include Andrej Karpathy, Jan Leike (OpenAI), and Jason Wei (Google Brain). Some class photos with them below!
Mentorship and Advising
Sedrick Scott Keh [CMU Master's of Machine Learning (MSML), Class of 2022]
Mentoring several research projects on controllable and creative text generation [e.g. paper1, paper2].
Kevin Lu [University of Waterloo Undergrad, Computer Science, Class of 2026]
Mentored several research projects on controllable, creative, and visually-grounded text generation [e.g. paper1, paper2].
Zhuofu (Derek) Tao [UCLA Ph.D. in Electrical Engineering, Class of 2025]
Mentored a research project on controllable and visually-grounded text generation [paper].
Advised their software engineering capstone project on text simplification for ESL students.
Piano, Guitar, & Music
I have been playing the piano since I was 6 years old, and also some guitar (mainly self-taught starting May 2022). I play anime, classical, and other pieces. I sometimes upload and livestream on my YouTube and TikTok. I'm also the founder and president of the Stanford Piano Society. Follow our Instagram!
