Vito Scaletta is an Italian immigrant in the United States. Born into a poor family from Sicily, he grows up in the city of Empire Bay and soon becomes best friends with Joe Barbaro, who convinces him that crime is just a way to get easy money. After having attempted a robbery, Vito is arrested and offered a choice by the authorities: go to jail or join the army. Choosing the latter, he is drafted and sent to Italy to fight Mussolini and his followers. After being injured and discharged, he returns to Empire Bay and learns that his deceased father owed a large amount of money. Willing to protect his family from debt collectors, Vito contacts Joe, and the two gradually begin to steer their lives towards organized crime, unaware of the price they will have to pay for that.

Mafia II shares the theme and the basic gameplay concept with Mafia; though it takes place at a later time and follows the life story of a different protagonist, the game's plot has a few connections with that of the predecessor. The game's story begins in the 1940's and extends into the 1950's. The story follows a linear path and is divided into chapters, each telling a separate episode in Vito's life and his trials and tribulations after getting involved with the mafia.

Like its predecessor, the game combines free-roaming driving with third-person shooting stages, focusing on the latter. Though the player is free to explore the city (driving a car or on foot), most of the driving gameplay is confined to the missions, which involves getting to the destination, stopping to buy items, hijacking different cars, etc. There are noticeably less chases or violent encounters during the driving sections than in the previous game. Police may arrest Vito for speeding or violent behavior; the player has the option to bribe officers in exchange for freedom.

The third-person shooting mechanics have been overhauled, featuring a cover-based system similar to that of Gears of War. Vito's health regenerates over a short period of time and cannot be restored by other means. He can be easily killed when running in the open; therefore, taking cover and periodically shooting is the prevalent tactic. In addition to firearms from the previous game (Colt 1911, tommy gun, pump-action shotgun, etc.), there are World War II-era weapons in the game, such as the MP 40 and M3 submachine guns. Vito is sometimes assisted by Joe or other characters during the shooting stages.

One portion of the game, which takes place inside of a prison, has melee-focused brawls. Players can explore interiors to find collectibles, more specifically vintage Playboy magazines to unlock the erotic photos within. The PlayStation 3 version of the game includes exclusive access to the DLC, The Betrayal of Jimmy. It is available for free only to those who purchase the base game.