AlloyBert is a transformer-based model ... AlloyBert's foundational model is RoBERTa, a pre-existing encoder. RoBERTa was used due to its self-attention mechanism, a feature that allows the ...