VScode remote development: How can I run a build task to source environment variables before running the real build task?

I want to setup VScode so I can use build tasks to build my project. The project is build using make, en I have defined a build task that runs make. However, before running make, I normally would source a script that sets my environment variables correctly. If I add a new build task with the source command, and set my main build tasks to first execute the source command, the environment variables are not propagated properly. How can I make sure the enviroment variables are kept between build tasks?

My tasks.json file:

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558 
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "type": "shell",
            "label": "make",
            "command": "make",
            "args": [],
            "problemMatcher": [
                "$gcc"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "dependsOn": "Set environment"
        },
        {
            "label": "Set environment",
            "type": "shell",
            "command": "source path/to/source_me.sh",
            "group": "build",
   ]
}