Some games default to windowed mode on their first launch. You can switch to fullscreen from within the game itself:

• Open the game menu or settings and look for Display or Graphics options
• Change the display mode to Fullscreen or Borderless Windowed
• Many games also support the Alt + Enter keyboard shortcut to toggle fullscreen

Once changed, the game will usually remember your preference on future launches.