The Mars series by Kim Stanley Robinson definitely fits the bill. The Ministry of the Future does too but it is more about the coming climate change disaster.

All tomorrow’s by c.m koseman may be interesting to you. It’s a short story that examines the state of humanity several billion years in the future after they have evolved to be unrecognizable. Some civilizations thrived and became better, many devolved and live tortured existances. Quite a few lose the ability to speak or lose intelligence in general.

Maybe not “hard” enough for you (eg. it has absibles) but Becky Chambers’s Record of a Spaceborn Few is about life on a fleet of generation ships.

The Culture series novel, my favorite optimistic and hard sci fi that includes artificial intelligence (minds that have giant ships or habitats for bodies and humanoid avatars to interact with people).

They basically never live on planets because they are inefficient and “inelegant”. They live on gigantic ring orbitals that have a fraction of the mass of a planet but multiple times the surface area. No big take-off energy needed either. They also live on gigantic ships that endlessly cruise the milky way. Highly recommend!

Another thought about “colonizing planets” would be that it’s basically a form of genocide. Imagine someone had colonized earth half a billion years ago or just a few million years ago. Humanity would never have existed. Just stepping foot on a planet like they do on star trek is basically ecocide - with the introduction of completely foreign and possibly incredibly disruptive micro organisms. Besides the ethical aspect there would also be the loss of information - if you imagine a pristine planet to be a bio computer creating countless unique and new genetic variations and new forms of chemistry. Quite possible not something that can be covered with a computer. Or observing primitive planets as a source of entertainment. There are lots of reasons why outside of a few “home planets” advanced civilizations would never terraform existing biological systems, and would find artificial habitats far more efficient or practical.

Seveneves by Neal Stephenson is this.

Also Tau Zero by Poul Anderson, to a degree.

I’m going to go the other way and recommend The Fifth Season, which is technically a fantasy trilogy but which has won both the Hugo and Nebula awards, because (as if that wasn’t a spoiler) it’s got a ton of sci Fi in it.

It’s basically about people on a planet that keeps dying. They’ve had to deal with so many apocalyptic events that prepping for the next one defines the entirety of their civilization. If you want a window into the psychology of a society constantly on the verge of destruction, I can’t think of a better series.

Check out A Memory Called Empire by Arkady Martin and it’s sequel.

Children of Time series goes over this a little bit, especially in the first book. Colonists end up waking up early due to a malfunction and end up falling into a devolving tribalistic race to the bottom on their journey to the planet.

Surprised no one has mentioned The Expanse series. A ton of world building in very different kinds of environments. Space stations, small ships, big ships, generation ships, asteroids, moons, planets.

The environments are well thought out in how the residents would need to adapt

You might want to check out Record of a Spaceborn Few by Becky Chambers. The book is about the people of the Exodus Fleet, a group of multi-generation ships that left Earth years ago. Even though the fleet eventually found other planets for them to live on, many are content to continue living out in space. It’s a neat little slice of life book about this community doing their part to keep these ships going.

Ohhhhh boy, I get to nerd out. OK, super short story; reading and chatting about The Expanse book series got me pointed towards the work of Alastair Reynolds. The early parts of his universes arch aren’t really relevant for your purposes, but in the latter books, how humanity survives on lifeless rocks, is exactly what you’re looking for. Plus, he’s a astrophysicist doctor, iirc, and it is quite quite good hard Sci Fi.

The Children of Time books by Adrian Tchaikovsky have a lot of those themes. Half of the first book is about an ark ship sent out to find a habitable planet because earth is dying. It spans hundreds of years as key crew members go in and out of hyper sleep. Relationships and political factions form and dissolve as the ageing ship continues its mission to find a new home.

The second book focuses on a terraforming crew that was sent to another star system to prepare a planet for humans. However, the planet’s ecology is so alien it proves very difficult to gain a foothold.

Not quite what you’re after but I absolutely love Diaspora by Greg Egan.

It’s a different take on the same issues you’re asking about (not at first, but it’s not really a spoiler to say that it explores them whether or not it’s as necessary as your examples state), a take that leans more into different forms of existence rather than supporting our current existence in a different environment (but touches on aspects of that too, kind of). It’s mega-multi-generational while also not being that at all, depending on perspective.

People already mention the Mars series by Orson Scott Card, the Expanse series by Corey, and Seveneves by Stephenson, which are all fantastic and all fit your request well. Two others you might consider are:

• The Moon is a Harsh Mistress by Heinlein. Very old school classic that features a moon colony fighting with earth.
• Beggars in Spain by Kress. Most of it is on a near future earth, but the last hunk of it involves a segment of people relocating to a space station.

It’s a very non traditional story structure (at least to a western reader) but The Three Body Problem series has a lot of plot revolving around the lack of inhabitable worlds.