With the escalating costs associated with fine-tuning large pre-trained models, the significance of parameter-efficient fine-tuning (PEFT) methods has become increasingly evident. Among these methods, we focus on LoRA, which introduces parallel trainable parameters in the multi-head attention component and has demonstrated promising results. However, previous research may have overlooked the mechanistic interpretability of the transformer architecture, especially since PEFT methods are built upon this framework. Drawing on this insight, we propose InterLoRA, which integrates LoRA with feature adaptation mechanism into both the attention layer, considering the varying importance of multiple heads, and the Feed-Forward Network (FFN) layer, acknowledging the memory storage characteristics. Experiments conducted on a variety of complex generation tasks highlight the effectiveness of InterLoRA in jointly fine-tuning both components while efficiently managing parameter memory.