Spectral-based hashing (SpH) is the most used method for cross-view hash function learning (CVHFL). However, the following three problems are shared by many existing SpH methods. Firstly, preserving intra- and inter-similarity simultaneously increases models' complexity significantly. Secondly, linear model applied in many SpH methods is hard to handle multimodal data in cross-view scenarios. Thirdly, to learn irrelevant multiple bits, SpH imposes orthogonality constraints which decreases the mapping quality substantially with the increase of bit number. To address these challenges, we propose a novel SpH method for CVHFL in this paper, referred to as Kernel-based Supervised Hashing for Cross-view Similarity Search (KSH-CV). We prove that the intra-adjacency matrix is redundant given inter-adjacency matrix. Then we define our objective function in a supervised and k-ernelized way which just needs to preserve inter-similarity. Furthermore a novel Adaboost algorithm, which minimizes exponential mapping loss function for cross-view similarity search, is derived to solve the objective function efficiently while avoiding orthogonality constraints. Extensive experiments verifies that KSH-CV can significantly outperform several state-of-the-art methods on three cross-view datasets.