[isabelle-dev] Jenkins maintenance

[Lars Hupel]

(Please read carefully!)

> over the weekend, there will be – once again – a scheduled maintenance
> on the build infrastructure. I will add some more jobs (which are
> running nightly) to test a wider variety of side conditions (mainly
> -j/-o threads). I will take this opportunity to tidy up the Jenkins home
> page, too. In the unlikely event that something goes wrong with my
> deployment scripts and I have to roll back changes,

Maintenance period is now over. Here are the most prominent changes
affecting users.

~~~

- Previously, in occasions where Jenkins is under high load, subjob runs
(e.g. "isabelle-repo-afp") might have used a different Mercurial
revision than the parent job. This situation will not occur anymore.
Now: as soon as a job gets kicked off, the Isabelle revision is fixed.

  - This means that we're getting close to be able to run concurrent
jobs. Currently, some pushes don't get built: when a job is already
running and more than one push happens in the meantime (only the last
push is built). Soon, all pushes will get built. I will enable this for
the AFP repository at some point and will monitor how much this
increases overall load.

  - This also means a potential increase in build notifications. To
alleviate this, I have changed the configuration such that all failure
mails will only be sent when the first failure in a job occurs. If it
keeps failing, no more emails. If it succeeds again and then fails
again, a new email.

- All "makeall" jobs now run in 64 bit mode to avoid spurious failures.
This is now uniform between "makeall" and "afp" jobs.

- The "isabelle-nightly-benchmark" job now runs more than just
"~~/src/HOL/Benchmarks", but also a variety of "critical" sessions to
obtain reliable timing information. It continues to run in 32 bit mode.

- The new job "isabelle-release-makeall" checks the Isabelle release
repository. This will become relevant after the fork point for
Isabelle2016-1. Currently, there is no job integrating that repository
and the AFP. I'm not sure yet how that should be organized.

- All builds now carry timestamp information in the console output.

- There are now four different views on the front page of Jenkins (these
are the newly-introduced tabs above the list of jobs).

  - "All": same as before; all jobs
  - "dashboard": graphical overview without testboard; this is like a
"mission control view"
  - "multijob": shows the hierarchical structure of the jobs*
  - "nightly": just the nightly jobs

I encourage all developers to try out the "dashboard" and "multijob"
views and give feedback:

  <https://ci.isabelle.systems/jenkins/view/multijob/>
  <https://ci.isabelle.systems/jenkins/view/dashboard/>

One of them is eventually going to become the default view.

~~~

As always, feedback is appreciated. There are more changes planned (new
jobs), but they can proceed without downtime.

Cheers
Lars




* The "multijob" plugin experienced some breaking changes recently. This
lead to nasty exceptions being displayed by Jenkins. I have now deployed
an unreleased, patched version which I believe is stable. If you see any
JVM stack traces or exceptions, please report them to me privately.