Docs: how to run services reliably and update service autorestart to service lifecycle.

[IronCore864]

According to our discussion (internal doc, sorry), this is the first piece of the next few how-to guides for Pebble.

In this PR:

• The “service autorestart” doc is renamed to “service lifecycle” and expanded to a more-detailed reference.
• A new how-to guide "how to run services reliably" detailing health checks is created, and placed after “how to install”, as the second how-to doc.
• Per David's suggestion, some of the causes of unreliability in the modern microservice world are listed, and which can be mitigated by health checks/Pebble are explained.
• Per David's suggestion, a few words on "what health checks are not" are added, so that the users won't misuse this feature to run cronjobs.

See more details and outline in the above discussion doc.

Preview: https://canonical-pebble--541.com.readthedocs.build/en/541/how-to/run-services-reliably/

[dwilding]

Thank you for doing this! Before I work on a more detailed review, I have a few ideas about the overall contents:

1. "Health checks" section - I think this section should have more prominence. So how about making each topic a separate subsection, instead of using bullets?

   Also in this section, I think we ought to make it obvious up-front that "health checks" are a combination of Pebble's feature + what the service author chooses to implement.

   Then within each topic we can say what we recommend the service to do. For example, in the topic "Identifying Dependent Service Failures", I really like what you wrote:

   A service’s health check can include checks to ensure it can connect to its database, message queue, or other required services.

   This is nice & clear advice about how to approach health checks on the service side.

2. "Configuring health checks in Pebble" and "Restarting service on health check failure" sections - I'd probably combine these into a single section that is fully focused on how to configure a health check of HTTP type and restart the service when the health check fails.

   Since this is a how-to doc, it's OK to omit the different options for health checks. I think it's better to give a specific scenario, then link to the reference pages for people to understand the different options.

3. "Demo service" and "Putting it all together" sections - I feel these are extremely helpful, and the health-check-sample-service idea is neat! Since these sections are more guided, I'd consider moving them to a tutorial instead.

[IronCore864]

Thank you for doing this! Before I work on a more detailed review, I have a few ideas about the overall contents:

1. "Health checks" section - I think this section should have more prominence. So how about making each topic a separate subsection, instead of using bullets?
   Also in this section, I think we ought to make it obvious up-front that "health checks" are a combination of Pebble's feature + what the service author chooses to implement.
   Then within each topic we can say what we recommend the service to do. For example, in the topic "Identifying Dependent Service Failures", I really like what you wrote:

   A service’s health check can include checks to ensure it can connect to its database, message queue, or other required services.

   This is nice & clear advice about how to approach health checks on the service side.

2. "Configuring health checks in Pebble" and "Restarting service on health check failure" sections - I'd probably combine these into a single section that is fully focused on how to configure a health check of HTTP type and restart the service when the health check fails.
   Since this is a how-to doc, it's OK to omit the different options for health checks. I think it's better to give a specific scenario, then link to the reference pages for people to understand the different options.

3. "Demo service" and "Putting it all together" sections - I feel these are extremely helpful, and the health-check-sample-service idea is neat! Since these sections are more guided, I'd consider moving them to a tutorial instead.

Refactored according to 1 and 2; for 3, I haven't done anything yet, mostly because those two parts are not long enough for a tutorial. Should we do that anyway?

[IronCore864]

Per discussion elsewhere, we decided to remove the "Demo service" and "Putting it all together" sections. We will add a tutorial in the future using content from these sections.