Are you by-chance referring to the "Deathmatch" system that is on the PC version?

"Deathmatches are team battles that can be held anywhere. They do not have the area or zone restrictions that PKs have. Select Deathmatch from the Social menu and organize a custom Team vs Team battle, inviting your friends and/or your enemies. You can even put your money where your mouth is and bet gold and items while battling 10 versus 10"

If you are, I don't believe the Deathmatch system has been added for Console. I remember asking about it pretty early on, but never received confirmation that the system was going to be added.