Traditional PFL approaches often expose significant privacy risks through direct parameter exchanges, making them vulnerable to privacy attacks such as model inversion and membership inference. SAFE-PFL is a novel framework that optimizes PFL by enhancing privacy without sacrificing model performance. SAFE-PFL incorporates three innovative components: a secure clustering module a novel heuristic for similarity analysis based on parameter identifiers, which eliminates the need for gradient transmission and thus enhances privacy, a cluster-based Multi-key Homomorphic Encryption scheme that allows individual nodes within a cluster to encrypt their data with unique keys, and a selective encryption strategy that targets only sensitive gradient components.