I tried creating another custom domain to app.genlabs.dev but then app.genlabs.dev points to project.pages.dev instead of app.project.pages.dev so I’m not quite sure how to make it use app..
Add the subdomains as you did – yes, they point to the main branch.
Then, go into your DNS settings, find the entry for app.genlabs.dev, edit it, and change the target to app.project.pages.dev. Ignore the warning it gives you about changing an automatic CNAME.