Setting up devcontainer.json as non-root user in VSCode

I'm trying to setup VSCode for remote development with docker container, but keep having permission issues when trying to install vscode-server inside the container.

Setting up container for folder or workspace: /home/username/4NN3-Neural-Networks-and-Deep-Learning
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db /bin/sh -c (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Run: docker cp 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db:/etc/passwd -
Forking shutdown monitor: /home/username/.vscode/extensions/ms-vscode-remote.remote-containers-0.81.1/dist/shutdownMonitorProcess /run/user/1001/snap.code/vscode-remote-containers-a2996f7f87a405d00b1db59d2b1d8d9b31ef07dc.sock singleContainer no-trace
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db test -d /home/username/.vscode-server
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db test -d /home/username/.vscode-remote
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db /bin/sh -c set -o noclobber ; mkdir -p '/home/username/.vscode-server/data/Machine'&& { > '/home/username/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
mkdir: cannot create directory ‘/home/username/.vscode-server’: Permission denied
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db /bin/sh -c set -o noclobber ; mkdir -p '/home/username/.vscode-server/data/Machine'&& { > '/home/username/.vscode-server/data/Machine/.copyGitConfigMarker' ; } 2> /dev/null
mkdir: cannot create directory ‘/home/username/.vscode-server’: Permission denied
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db test -d /home/username/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390
Installing VS Code Server for commit 6ab598523be7a800d7f3eb4d92d7ab9a66069390
Run: docker exec 9181871e4be347ece95c47cce6ec717d8cce9d8e2febe3fd3e7ea98984a434db mkdir -p /home/username/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390_1572636263815
mkdir: cannot create directory ‘/home/username/.vscode-server’: Permission denied

My Dockerfile is bellow:

FROM tiangolo/python-machine-learning:python3.6

LABEL version="1.0"
LABEL description="Dev Environment for McMaster University, SFWRTECH 4NN3 Course"

# Update Conda
RUN conda update -n base -c defaults conda

# Install ML Libraries  
RUN conda install -y tensorflow
RUN conda install -y keras
RUN conda install -y numpy
RUN conda install -y scipy
RUN conda install -y scikit-learn
RUN conda install -y pandas
RUN conda install -y matplotlib
RUN conda install -y pylint

# Creating New User and Set Password
RUN useradd --create-home --shell /bin/bash username
RUN echo 'username:passw0rd' | chpasswd

# Set User & WORKDIR
USER username
ENV HOME /home/username
WORKDIR /home/username

The container created from the image in regular terminal runs just fine. This seems to be a recurring issue, I've tried a few solutions proposed in issue tracker in github here and here both propose to set the $HOME to /home/username.

I've set the $HOME in both the Dockerfile and devcontainer.json, but the issue persists, anybody a workaround for this?

The devcontainer.json is bellow:

{
    "name": "4NN3 Dev Container",
    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",
    // "dockerFile": "../Dockerfile",
    "image": "4nn3-mcmaster",

    "runArgs": [
        // See https://aka.ms/vscode-remote/containers/non-root-user
        // "-u", "username""--env", "HOME=/home/username/"
    ],

    "settings": {
        "terminal.integrated.shell.linux": null
    },

    "workspaceMount": "src=${localWorkspaceFolder}/,dst=/home/username/,type=bind,consistency=delegated",
    "workspaceFolder": "/home/username/",

    "extensions": ["ms-python.python", "eamodio.gitlens"]
}