It took some digging and discussion but here is some general info/rules I've put together for you (Some maybe obvious but bare with me)
Develop Corporate Guide lines with the proper business groups and have it published somewhere accessible (Maybe create an SPO Site 😊) this should include components of page, colors, Logos, Font, picture sizes, etc. If having trouble picking the right colors look here https://color.adobe.com/create/color-wheel/
These are your choices for Branding, the best practice is to stay as close to the top of this list while still being able to provide the look and feel that you want. As an example Communication/Publishing sites may require more customization then the other and require that you use Custom Master pages, but this would be a decision you need to make.
- Office 365 Themes - Inexpensive and simple with limited capabilities, this allows you to customize the Office 365 objects but not the SharePoint objects.
- SharePoint Themes - Inexpensive and simple with limited capabilities, this allows you to customize the SharePoint Colors in Site Collection.
- Composed looks (Color Schemes, font schemes) - Take some understanding to compose and create but provide a good level of capabilities.
- Alternate CSS - Provides an excellent level of capabilities requires web knowledge to create
- Custom MasterPages - Provides you with complete control of the look and feel, with excellent capability and flexibility, but requires web knowledge and developer skills. As well since you must create your own Custom Masterpage (Do not edit OOB masterpages) you take on the responsibility of deploying functionality to the page (Not necessarily a bad thing).
NOTE: See these articles for more details around particular customizations
Customize the Office 365 theme for your organization - https://support.office.com/en-us/article/Customize-the-Office-365-theme-for-your-organization-8275da91-7a48-4591-94ab-3123a3f79530
Composed looks overview for SharePoint - https://dev.office.com/sharepoint/docs/general-development/themes-overview-for-sharepoint
SharePoint Color Palette tool to create spcolor file - http://www.microsoft.com/en-us/download/details.aspx?id=38182
Deploying the customization is generally best done thru the remote provisioning pattern, except in the case of using Office 365 Themes as these can be provisioned via OOB functionality in the Tenant. When using Remote provisioning you are using your branding objects (Provisioning Template) from outside of SPO and a little bit of PowerShell sitting on a server (On-prem or in Azure) to push those templates to where you want them to go. See this articles for some basic information and sample scripts https://github.com/SharePoint/PnP-Guidance/blob/551b9f6a66cf94058ba5497e310d519647afb20c/articles/Introducing-the-PnP-Provisioning-Engine.md around how to provision the branding in your Site Collections in Office 365. That article covers very basic levels of branding but as it eludes to you can perform some advanced deployments of content-types, Columns and many other Site objects.Consider Page performance, this was true in SharePoint on-prem versions and this is still true in the Cloud. There are some simple rules to follow to make sure that your branding/customizations don't cause page performance issues
Keep add-ins and web parts on any page to a minimum, you will notice performance issues around 12 objects or so depending on the complexity of those add-ins.
Keep Images, CSS, JS, etc to a minimum
Minimize data requests, or only make data requests at the time of access
Do not make all your data central and then make multiple requests at the sametime
Next you need to worry about Navigation, and obviously if appropriate then it's best to use the OOB engine. But if you have a business need or the Site Collection is complicated enough to require it then you should consider the following possibilitie0
OOB Navigation Stores (MMS, Search Index, or Site Structure) these are simplest to manage or deploy.
Navigation Store API, while very flexible this requires Public Internet resources (SSL, DNS, etc) and particular dev skills
Considerations for Modern sites, while these new Team Sites are designed to be responsive and perceived as faster the customization of these sites is limited to the following:
Apply a custom theme/Change the logo
Apply an OOB theme
Create custom Site Columns and Content Types
Custom lists and libraries
Site Configurations (Regional Settings, languages, etc)
NOTE: There are changes coming to this space to allow for more elaborate customization