Automatically generate and commit site HTML

[nchammas]

Add workflows to automatically build the site and push the generated HTML. This eliminates the need for contributors to commit or review HTML changes.

Workflow changes:

• html-build.yml: Add a new workflow that runs on PRs and simply generates the docs. This confirms the Jekyll build is not obviously broken.
• html-push.yml: Add a new workflow that runs on new commits to asf-site. It generates the docs and pushes the generated HTML automatically as a new commit to asf-site.
• doc_gen.yml: Delete this workflow since it is subsumed by the new workflows.

Website source changes:

• Stabilize the sitemap so it's generated in a deterministic manner, eliminating repeated diff noise.

Repo documentation changes:

• Unify the build instructions so there are only two places to look: committers.md and the main repo README.
• Add important note about content/ so its critical role is documented clearly for everyone.

This is a second attempt at #697 (which was reverted in #698).

[nchammas]

cc @cloud-fan @holdenk

If you'd like to setup a staging site to test this out, I'm happy to help with that.

Otherwise, if/when we merge this, let's confirm the deployed production site is still working as expected so we can revert quickly if necessary.

[holdenk]

I am -1 on this change until the in flight releases are complete and distributed I don't think it's worth the risk. After those are done (hopefully middle of this month but you never know), happy to revisit. To be clear this is an explicit but temporary veto on changing the website build until after 4.2.0/4.1.3/4.0.4/3.5.9 are all released. I think this is a change we do really need (and greatly appreciate @nchammas for doing the work here), just the timing doesn't work in my mind.

[nchammas]

Yes, I agree, no rush. Let's wait until all these upcoming releases are complete.

Unless we invest in setting up a proper staging site (or per-PR staging sites, though it's not clear to me whether that's possible), the only way to fully test this new setup is with a production deployment.

[pan3793]

@nchammas, thank you for working on this area, I agree that the current spark-website repo is a disaster ... some ideas and experience from maintaining other ASF project website repos might be helpful:

1. use master branch to track only the source code (if we want to keep the history, might need to use the git command to rewrite the whole history and filter out the compiled output files)
2. leave the asf-site branch to track the compiled output (keep it orphaned, no need to track history to keep the size under control)
3. when pushing code to master, trigger a workflow to build and push the output to the asf-site

if we agree with this direction, then:

1. Spark PMC needs to ask the infra team to change the default branch to master
2. @nchammas you can debug that on your forked repo first, and compare the differences between the auto-generated asf-site (your forked repo) and the current asf-site (apache upstream)

[nchammas]

@pan3793 - This was exactly my original suggestion. However, @cloud-fan wanted to take a more conservative approach so as not to completely change the current workflow that committers are familiar with. I am happy to help with either approach.

Setting up a staging site (or multiple staging sites, one per branch) is also a good idea and orthogonal to splitting the source from the HTML output. Happy to help with that, too.

[pan3793]

@nchammas, thanks for the info, I wasn't aware of that.

@cloud-fan I don't think tracking the compiled output files in the source tree makes sense, for example, we don't track *.class files for a Java/Scala project, can we try with two branches?

• master for source code only;
• asf-site for generated content, keep history at the beginning to allow us to restore in case something goes wrong, and eventually only keep the last commit to reduce the repo size

[cloud-fan]

The commit history is already mixed with md and html files, so I don't think it's a must-have to separate the commit history now. It's orthogonal to the html auto generation.