Pretrained language models (LMs) have been shown to be capable of few-shot learning via in-context demonstrations. In this work, we examine if in-context learning generalizes out-of-distribution. On several text classification tasks and considering different kinds of realistic demonstration vs. test distribution shifts—domain, adversarial, and dialectal—we find a significant drop in test accuracy compared to in-distribution performance. Moreover, we find that the accuracy is inversely proportional to the number of demonstrations. To address this issue, we explore different zero-shot prompting approaches which do not rely on demonstrations. With six open-source language model families, we show that zero-shot prompting techniques which verbalize the target distribution in the prompt are able to close the gap to in-domain few-shot classification performance.