Your basic requirements are:

• Some kind of domain / subdomain payed or free;
• Preferably Home ISP that has provides public IP addresses - no CGNAT BS;
• Ideally a static IP at home, but you can do just fine with a dynamic DNS service such as https://freedns.afraid.org/.

Quick setup guide and checklist:

1. Create your subdomain for the dynamic DNS service https://freedns.afraid.org/ and install the daemon on the server - will update your domain with your dynamic IP when it changes;
2. List what ports you need remote access to;
3. Isolate the server from your main network as much as possible. If possible have then on a different public IP either using a VLAN or better yet with an entire physical network just for that - avoids VLAN hopping attacks and DDoS attacks to the server that will also take your internet down;
4. If you’re using VLANs then configure your switch properly. Decent switches allows you to restrict the WebUI to a certain VLAN / physical port - this will make sure if your server is hacked they won’t be able to access the Switch’s UI and reconfigure their own port to access the entire network. Note that cheap TP-Link switches usually don’t have a way to specify this;
5. Configure your ISP router to assign a static local IP to the server and port forward what’s supposed to be exposed to the internet to the server;
6. Only expose required services (nginx, game server, program x) to the Internet us. Everything else such as SSH, configuration interfaces and whatnot can be moved to another private network and/or a WireGuard VPN you can connect to when you want to manage the server;
7. Use custom ports with 5 digits for everything - something like 23901 (up to 65535) to make your service(s) harder to find;
8. Disable IPv6? Might be easier than dealing with a dual stack firewall and/or other complexities;
9. Use nftables / iptables / another firewall and set it to drop everything but those ports you need for services and management VPN access to work - 10 minute guide;
10. Configure nftables to only allow traffic coming from public IP addresses (IPs outside your home network IP / VPN range) to the Wireguard or required services port - this will protect your server if by some mistake the router starts forwarding more traffic from the internet to the server than it should;
11. Configure nftables to restrict what countries are allowed to access your server. Most likely you only need to allow incoming connections from your country and more details here.

Realistically speaking if you’re doing this just for a few friends why not require them to access the server through WireGuard VPN? This will reduce the risk a LOT and won’t probably impact the performance. Here a decent setup guide and you might use this GUI to add/remove clients easily.

Don’t be afraid to expose the Wireguard port because if someone tried to connect and they don’t authenticate with the right key the server will silently drop the packets.

Now if your ISP doesn’t provide you with a public IP / port forwarding abilities you may want to read this in order to find why you should avoid Cloudflare tunnels and how to setup and alternative / more private solution.

You’re going to get a lot of bad or basic advice with no reasoning (use a firewall) in here… And as you surmised this is a very big topic and you haven’t provided a lot of context about what you intend to do. I don’t have any specific links, but I do have some advice for you:

First - keep in mind that security is a process not a thing. 90% of your security will come from being diligent about applying patches, keeping software up-to-date, and paying attention to security news. If you’re not willing to apply regular patches then don’t expose anything to the internet. There are automated systems that simply scan for known vulnerabilities on the internet. Self-hosting is NOT “set it and forget it”. Figuring out ways to automate this help make it easy to do and thus more likely to be done. Checkout things like Ansible for that.

Second is good authentication hygiene. Choose good passwords. Better yet long passphrases. Or enable MFA and other additional protections. And BE SURE TO CHANGE ANY DEFAULT PASSWORDS for software you setup. Often there is some default ‘admin’ user.

Beyond that your approach is"security in depth" - you take a layered approach to security understanding what your exposure is and what will happen should one of your services / systems be hacked.

Examples of security in depth:

• Proper firewalling will ensure that you don’t accidentally expose services you don’t intend to expose (adds a layer of protection). Sometimes there are services running that you didn’t expect.
• Use things like “fail2ban” that will add IP addresses to temporary blocklists if they start trying user/passwords that don’t work. This could catch a bot from finding that “admin/password” user on your Nextcloud server that you haven’t changed yet…

Minimize your attack surface area. If it doesn’t need to be exposed to the internet then don’t expose it. VPNs can help with the “I want to connect to my home server while I’m away” problem and are easy to setup (tailscale and wireguard being two popular options). If your service needs to be “public” to the internet understand that this is a bigger step and that everything here should be taken more seriously.

Minimize your exposure. Think though the question of “if a malicious person got this password what would happen and how would I handle it?” Would they have access to files from other services running on the same server (having separation between services can help with this)? Would they have access to unencrypted files with sensitive data? It’s all theoretical, until it isn’t…

If you do expose services to the internet monitor your logs to see if there is anything “unusual” happening. Be prepared to see lots of bots attempting to hack services. It may be scary at first, but relatively harmless if you’ve followed the above recommendations. “Failed logins” by the thousands are fine. fail2ban can help cut that down a bit though.

Overall I’d say start small and start “internal” (nothing exposed to the internet). Get through a few update/upgrade cycles to see how things go. And ask questions! Especially about any specific services and how to deploy them securely. Some are more risky than others.

Check out the “Open Source Security Podcast” with Kurt Siegfried and Josh Bressers. It’s not about specifics so much as how to build a mindset around security for IOT and hosting, generally dealing with opensource offerings.

I’m not expert but for the sake of getting some discussion going:

Don’t open ports on your router to expose services to the open internet.

Use a vpn when torrenting and make sure your torrent client is set to only use the vpn’s network adapter. This way, if your vpn drops out the torrent client can’t reach the internet.

I keep everything local and use Tailscale to access things while I’m away from home.

Are you taking about security for your homelab? It essentially comes down to good key hygiene, network security and keeping everything updated.

Don’t open ports, use a good firewall at the border of the network, use a seedbox for torrenting. Use ACLs alongside VLANs in your network. Understand DNS in terms of how your requests are forwarded and how they are processed.