How can I use the Visual Studio command line vars in a VS Code task?

I have configured the VS cli vars in VS Code like this:

{
    "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\cmd.exe",
    "terminal.integrated.shellArgs.windows": [
        "/k",
        "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat"
    ]
}

When I press Alt-`, it properly opens the command line with the vars enabled. However, I can't get this to work properly for my tasks.

This is my taskfile:

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "ninja",
            "type": "shell",
            "command": "ninja -C build",
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

When I run it, I get this error: 'C:\Program' is not recognized as an internal or external command, operable program or batch file.

If I set the path of the terminals shell args in quotes again to account for Windows needing them, it does not build, but the terminal stops working too.