Large language models (LLMs) are increasingly used as agents in agent-based modeling (ABM) to simulate complex social and economic behaviors. However, existing implementations often rely on minimal wrappers around LLMs, lacking structure, modularity, and support for reuse, hindering reproducibility, generalization and exploratory research. We introduce Shachi (鯱), a general and modular agent framework for building and evaluating LLM-driven agents in ABM. Shachi defines a standardized architecture comprising four core components (LLM, tools, memory, and configuration) and supports clear agent-environment and agent-agent interactions. To evaluate its effectiveness, we provide a benchmark suite of tasks spanning three levels of complexity, from single-agent behavior to multi-agent communication. Using Shachi, we reproduce prior LLM-based ABM studies, conduct systematic agent-task cross-evaluations, and demonstrate novel exploratory study results. Shachi enables scalable, reusable, and insightful research at the intersection of LLMs and ABM for social science, and is released as an open-source toolkit. Code: https://anonymous.4open.science/r/bench-F6E7