This paper proposes \textit{Semantic Inference} (SINF), a approach that creates semantic subgraphs in a neural network based on a new Discriminative Capability Score (DCS) to drastically reduce the neural network computational load with limited performance loss. SINF is based on the intuition that semantically similar inputs (e.g., classes in "animals" or "flowers") share a significant number of filter activations, which can help create semantic subgraphs. Each subgraph can be ``turned on" when inputs coming from the corresponding semantic cluster are present, while the rest of the neural network can be "turned off". We evaluate the performance of SINF on VGG16, VGG19, and ResNet50 trained on CIFAR100 and a subset of the ImageNet dataset. Moreover, we compare its performance against 6 state-of-the-art pruning approaches. Our results show that (i) on average, SINF reduces the inference time of VGG16, VGG19, and ResNet50 respectively by up to 29\%, 35\%, and 15\% with only 3.75\%, 0.17\%, and 6.75\% accuracy loss for CIFAR100 while for ImageNet benchmark, the reduction in inference time is 18\%, 22\%, and 9\% for accuracy drop of 3\%, 2.5\%, and 6\%; (ii) DCS achieves respectively up to 3.65\%, 4.25\%, and 2.36\% better accuracy with VGG16, VGG19, and ResNet50 with respect to existing discriminative scores for CIFAR100 and the same for ImageNet is 8.9\%, 5.8\%, and 5.2\% respectively. Through experimental evaluation on Raspberry Pi and NVIDIA Jetson Nano, we show SINF is about 51\% and 38\% more energy efficient and takes about 25\% and 17\% less inference time than the base model for CIFAR100 and ImageNet benchmarks.