www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Thinktank: CI's, compiler lists, and project automation

reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
(I'm posting this here instead of D.learn because it isn't really a "How 
to do XYZ in D?", but rather to invite discussion on high-level 
solutions to a problem.)

Here's a common problem:

1. A project (ex, some library) uses travis-ci (and/or other) to ensure 
compatibility with a range of compiler versions. The travis 
configuration file includes a manually-curated list of compiler versions.

2. CI tests are triggered by new commits/merges/PRs in the project.

3. New compiler versions are released.

4. Compiler updates may or may not trigger new tests with the new 
compiler, depending on the project.

5. Project maintainer must manually update the list of compilers. (And 
this isn't one task, but rather O(n), where n=number of projects the 
given person is maintaining).



We do already have certain approaches to at least certain facets of the 
problem (ex: some third party libs are included in DMD's CI testing), 
but there are still limitations and downsides (ex: D tester has limited 
resources, and it doesn't address manually updating the travis config's 
list of compilers or notifying authors their project's compiler list 
needs updated.)

Another sample point of discussion: One possible approach is to have a 
bot generate PRs to update project's compiler lists. But that leads to 
other questions: How/when is the bot triggered? On what machine does it 
run? Is there an existing jumping-off point for creating such a bot? How 
does it approach the task of modifying `.travis.yml` (how smart/dumb 
does it need to be)? Any security concerns to be mindful of? Etc.
Mar 01 2018
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 2 March 2018 at 06:07:25 UTC, Nick Sabalausky 
(Abscissa) wrote:

 (I'm posting this here instead of D.learn because it isn't 
 really a "How to do XYZ in D?", but rather to invite discussion 
 on high-level solutions to a problem.)

 Here's a common problem:

 1. A project (ex, some library) uses travis-ci (and/or other) 
 to ensure compatibility with a range of compiler versions. The 
 travis configuration file includes a manually-curated list of 
 compiler versions.

 2. CI tests are triggered by new commits/merges/PRs in the 
 project.

 3. New compiler versions are released.

 4. Compiler updates may or may not trigger new tests with the 
 new compiler, depending on the project.

 5. Project maintainer must manually update the list of 
 compilers. (And this isn't one task, but rather O(n), where 
 n=number of projects the given person is maintaining).





Travis CI allows you to specify a D compiler in the following 
ways:

* <compiler> - the latest version of the specified compiler
* <compiler>-beta - the latest beta
* <compiler>-nightly - the nightly build
* <compiler>-<version> - a specific version of the compiler

Where <compiler> is "dmd", "ldc" or "gdc".

Combine the above with a scheduled build that builds once every 
day you're pretty well covered. For an example, see the Travis CI 
configuration for DStep [1]. I'm building the latest version of 
DMD and LDC on Linux and macOS, but the betas and nightly are 
only built on Linux, due the the limited resources of macOS 
builds on Travis CI.

[1] 
https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml

--
/Jacob Carlborg
Mar 01 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/02/2018 02:24 AM, Jacob Carlborg wrote:

 
 Travis CI allows you to specify a D compiler in the following ways:
 
 * <compiler> - the latest version of the specified compiler
 * <compiler>-beta - the latest beta
 * <compiler>-nightly - the nightly build
 * <compiler>-<version> - a specific version of the compiler
 
 Where <compiler> is "dmd", "ldc" or "gdc".
 
 Combine the above with a scheduled build that builds once every day 
 you're pretty well covered. For an example, see the Travis CI 
 configuration for DStep [1]. I'm building the latest version of DMD and 
 LDC on Linux and macOS, but the betas and nightly are only built on 
 Linux, due the the limited resources of macOS builds on Travis CI.
 
 [1] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml
 


Certainly a possible approach, but has downsides:

- Maybe there's a simple setting I've overlooked, but when a build job 
fails on travis, the author does not get proactively notified. The 
author only finds out next time they go into travis. (I've been 
surprised many times to discover failed builds that had occured several 
days ago, or more.)

- A project author will still need to: 1. Actively notice new compiler 
releases and 2. Manually update the .travis.yml files for each of their 
projects. Certainly there's room for more automation here.

- Except when "nightly" is desired, this leads to many unnecessary 
redundant builds/tests, especially across all the various D projects. 
(Though I don't know how much that would matter on travis. Maybe it'd be 
a drop in their bucket.)
Mar 02 2018
next sibling parent Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 March 2018 at 06:39:42 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 03/02/2018 02:24 AM, Jacob Carlborg wrote:

 
 Travis CI allows you to specify a D compiler in the following 
 ways:
 
 * <compiler> - the latest version of the specified compiler
 * <compiler>-beta - the latest beta
 * <compiler>-nightly - the nightly build
 * <compiler>-<version> - a specific version of the compiler
 
 Where <compiler> is "dmd", "ldc" or "gdc".
 
 Combine the above with a scheduled build that builds once 
 every day you're pretty well covered. For an example, see the 
 Travis CI configuration for DStep [1]. I'm building the latest 
 version of DMD and LDC on Linux and macOS, but the betas and 
 nightly are only built on Linux, due the the limited resources 
 of macOS builds on Travis CI.
 
 [1] 
 https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml
 


 Certainly a possible approach, but has downsides:

 - Maybe there's a simple setting I've overlooked, but when a 
 build job fails on travis, the author does not get proactively 
 notified. The author only finds out next time they go into 
 travis. (I've been surprised many times to discover failed 
 builds that had occured several days ago, or more.)


I receive emails from them, after it broke, each time it builds 
and that the build is not repaired and finally a last one when 
the build is repaired.
I've just checked and there is no settings to activate this.


 - A project author will still need to: 1. Actively notice new 
 compiler releases and 2. Manually update the .travis.yml files 
 for each of their projects. Certainly there's room for more 
 automation here.

 - Except when "nightly" is desired, this leads to many 
 unnecessary redundant builds/tests, especially across all the 
 various D projects. (Though I don't know how much that would 
 matter on travis. Maybe it'd be a drop in their bucket.)


I agree. "Nightly" is useful but "beta" not, or rather _not 
always_. "beta" is useful when there's a new beta and until it 
get released (usually 2-3 weeks), after what this wastes 
computing resources.
Mar 03 2018
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2018-03-03 07:39, Nick Sabalausky (Abscissa) wrote:


 Certainly a possible approach, but has downsides:
 
 - Maybe there's a simple setting I've overlooked, but when a build job 
 fails on travis, the author does not get proactively notified. The 
 author only finds out next time they go into travis. (I've been 
 surprised many times to discover failed builds that had occured several 
 days ago, or more.)


You should get an email notification about failed builds [1] [2],


 - A project author will still need to: 1. Actively notice new compiler 
 releases and 2. Manually update the .travis.yml files for each of their 
 projects. Certainly there's room for more automation here.


No. If you specify "dmd" (without any version) in the list of compilers 
[5], it will build the latest version. For example, this build [3], 
which ran 23 hours ago uses DMD 2.078.3, while this one [4], 29 days 
ago, uses 2.078.2. As you can see on the commit in Travis CI, it's the 
same commit.


 - Except when "nightly" is desired, this leads to many unnecessary 
 redundant builds/tests, especially across all the various D projects. 
 (Though I don't know how much that would matter on travis. Maybe it'd be 
 a drop in their bucket.)



[1] 
https://docs.travis-ci.com/user/notifications/#Default-notification-settings

[2] 
https://docs.travis-ci.com/user/notifications/#Missing-build-notifications

[3] https://travis-ci.org/jacob-carlborg/dstep/jobs/348275710
[4] https://travis-ci.org/jacob-carlborg/dstep/jobs/340527251

[5] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml#L4

-- 
/Jacob Carlborg
Mar 03 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/03/2018 10:22 AM, Jacob Carlborg wrote:

 On 2018-03-03 07:39, Nick Sabalausky (Abscissa) wrote:

 - Maybe there's a simple setting I've overlooked, but when a build job 
 fails on travis, the author does not get proactively notified. The 
 author only finds out next time they go into travis. (I've been 
 surprised many times to discover failed builds that had occured 
 several days ago, or more.)


 
 You should get an email notification about failed builds [1] [2],


Weird. Never worked for me. Will have to check into that.


 - A project author will still need to: 1. Actively notice new compiler 
 releases and 2. Manually update the .travis.yml files for each of 
 their projects. Certainly there's room for more automation here.


 
 No. If you specify "dmd" (without any version) in the list of compilers 
 [5], it will build the latest version. For example, this build [3], 
 which ran 23 hours ago uses DMD 2.078.3, while this one [4], 29 days 
 ago, uses 2.078.2. As you can see on the commit in Travis CI, it's the 
 same commit.
 


No. That is only sufficient *temporarily*, that is, until the next time 
the label "dmd" is updated to another new release once again. At *some* 
point, .travis.yml will still need to be manually updated...

Scenario 1-2-3:

1. Suppose a library supports DMD v2.071.0 and up. Because of this, 
.travis.yml includes:


   - dmd-2.079.0
   - dmd-2.078.2
   - dmd-2.078.1
   - dmd-2.078.0
   - dmd-2.077.1
   - dmd-2.077.0

Summary: CI now tests 2.076.1 through 2.079.0.

2. Release of DMD 2.080.0 occurs. "dmd" now points to 2.080.0 (and 
forget reproducibility: if 2.080.0 contains an unexpected regression 
then using travis's "Restart Job" on the "dmd" job will break overall 
reproducibility.)

Summary: CI now tests 2.076.1 through 2.080.0.

So far so good, right? Until...

3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless 
.travis.yml has been manually updated, it still contains:



   - dmd-2.079.0
   - dmd-2.078.2
   - dmd-2.078.1
   - dmd-2.078.0
   - dmd-2.077.1
   - dmd-2.077.0

Summary: CI *stops* testing dmd-2.080.0.

FAIL.
Mar 03 2018
next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote:

 
 No. That is only sufficient *temporarily*, that is, until the next time 
 the label "dmd" is updated to another new release once again. At *some* 
 point, .travis.yml will still need to be manually updated...
 
 Scenario 1-2-3:
 
 1. Suppose a library supports DMD v2.071.0 and up. Because of this, 
 .travis.yml includes:
 

    - dmd-2.079.0
    - dmd-2.078.2
    - dmd-2.078.1
    - dmd-2.078.0
    - dmd-2.077.1
    - dmd-2.077.0
 
 Summary: CI now tests 2.076.1 through 2.079.0.
 
 2. Release of DMD 2.080.0 occurs. "dmd" now points to 2.080.0 (and 
 forget reproducibility: if 2.080.0 contains an unexpected regression 
 then using travis's "Restart Job" on the "dmd" job will break overall 
 reproducibility.)
 
 Summary: CI now tests 2.076.1 through 2.080.0.
 
 So far so good, right? Until...
 
 3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless 
 .travis.yml has been manually updated, it still contains:
 


    - dmd-2.079.0
    - dmd-2.078.2
    - dmd-2.078.1
    - dmd-2.078.0
    - dmd-2.077.1
    - dmd-2.077.0
 
 Summary: CI *stops* testing dmd-2.080.0.
 
 FAIL.


Additionally, what happens when said project above doesn't receive 
commits/merges between steps 1 and 3 (ie, during the time period where 
2.080.0 is the latest release)? Any incompatibilities with 2.080.0 go 
completely unnoticed because CI never ends up testing with 2.080.0.

The "dmd" label does NOT solve the problem.
Mar 03 2018
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote:

 
 1. Suppose a library supports DMD v2.071.0 and up. Because of this, 
 .travis.yml includes:
 

    - dmd-2.079.0
    - dmd-2.078.2
    - dmd-2.078.1
    - dmd-2.078.0
    - dmd-2.077.1
    - dmd-2.077.0
 
 Summary: CI now tests 2.076.1 through 2.079.0.
 


Oops, proofread failures. Naturally, I meant this (fixes emphasized):

-----------------------------------------

1. Suppose a library supports ***DMD v2.077.0*** and up. Because of 
this, .travis.yml includes:


   - dmd-2.079.0
   - dmd-2.078.2
   - dmd-2.078.1
   - dmd-2.078.0
   - dmd-2.077.1
   - dmd-2.077.0

Summary: CI now tests ***2.077.1*** through 2.079.0.
Mar 03 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/04/2018 02:23 AM, Nick Sabalausky (Abscissa) wrote:


 Oops, proofread failures. Naturally, I meant this (fixes emphasized):
 
 -----------------------------------------
 
 1. Suppose a library supports ***DMD v2.077.0*** and up. Because of 
 this, .travis.yml includes:
 

    - dmd-2.079.0
    - dmd-2.078.2
    - dmd-2.078.1
    - dmd-2.078.0
    - dmd-2.077.1
    - dmd-2.077.0
 
 Summary: CI now tests ***2.077.1*** through 2.079.0.


Summary: CI now tests ***2.077.0*** through 2.079.0.

(Argh, maybe it's bedtime... ;))
Mar 03 2018
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2018-03-04 08:05, Nick Sabalausky (Abscissa) wrote:


 3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless 
 .travis.yml has been manually updated, it still contains:
 


    - dmd-2.079.0
    - dmd-2.078.2
    - dmd-2.078.1
    - dmd-2.078.0
    - dmd-2.077.1
    - dmd-2.077.0
 
 Summary: CI *stops* testing dmd-2.080.0.
 
 FAIL.


Aha, you mean like that. Seems a bit difficult to fix. Perhaps 
specifying a range of compiler versions would do?

-- 
/Jacob Carlborg
Mar 04 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/04/2018 03:42 AM, Jacob Carlborg wrote:

 
 Aha, you mean like that. Seems a bit difficult to fix. Perhaps 
 specifying a range of compiler versions would do?


Yea, there's really no two ways around it: Ultimately, each new compiler 
release will need to get added to .travis.yml (or any other CI's 
equivalent) either sooner or later. (Unless the project eschews anything 
beyond "latest versions ONLY", but that comes with its own downsides, 
especially for libraries.)

Luckily, this is definitely automatable, at least as far as 
auto-submitted PRs, if nothing else. The devil is just in the details.
Mar 04 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote:

 On 03/03/2018 10:22 AM, Jacob Carlborg wrote:

 You should get an email notification about failed builds [1] [2],


 
 Weird. Never worked for me. Will have to check into that.
 


Ahh, it looks like travis disregards the github email address (perhaps 
it can't access it, or indirectly tell github to email me?), and the 
only way to receive travis email notifications is to expose the desired 
email address to the general public via either .travis.yml or the git 
commit author.

(Un)fortunately, I have a far more effective, reliable method of 
filtering spam than the "state of the art" gmail heuristics: I run a 
mail server, create domain-specific email addresses, and delete any 
accounts that get spammed (knowing full well who it was that leaked my 
email). I was never stupid enough to configure git with a legitimate 
email address I actually check...I have better things to do than fight 
spam and regularly check some "spam folder".

Travis (not to mention Git itself) doesn't appear to support the 
post-1980 reality of email, ie, "Publicly-posted addressed WILL be 
spammed, hard"), thus, even though I get GitHub notifications just fine, 
I haven't been getting notifications from travis even though its linked 
to my github account.

</rant>
Mar 04 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 03/05/2018 02:42 AM, Nick Sabalausky (Abscissa) wrote:

 On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote:

 On 03/03/2018 10:22 AM, Jacob Carlborg wrote:

 You should get an email notification about failed builds [1] [2],


 Weird. Never worked for me. Will have to check into that.


 
 Ahh, it looks like travis disregards the github email address (perhaps 
 it can't access it, or indirectly tell github to email me?), and the 
 only way to receive travis email notifications is to expose the desired 
 email address to the general public via either .travis.yml or the git 
 commit author.
 
 (Un)fortunately, I have a far more effective, reliable method of 
 filtering spam than the "state of the art" gmail heuristics: I run a 
 mail server, create domain-specific email addresses, and delete any 
 accounts that get spammed (knowing full well who it was that leaked my 
 email). I was never stupid enough to configure git with a legitimate 
 email address I actually check...I have better things to do than fight 
 spam and regularly check some "spam folder".
 
 Travis (not to mention Git itself) doesn't appear to support the 
 post-1980 reality of email, ie, "Publicly-posted addressed WILL be 
 spammed, hard"), thus, even though I get GitHub notifications just fine, 
 I haven't been getting notifications from travis even though its linked 
 to my github account.
 
 </rant>


Hooray "web 2.0" :/
Mar 04 2018
prev sibling next sibling parent rjframe <dlang ryanjframe.com> writes:
On Fri, 02 Mar 2018 01:07:25 -0500, Nick Sabalausky (Abscissa) wrote:


 Another sample point of discussion: One possible approach is to have a
 bot generate PRs to update project's compiler lists. But that leads to
 other questions: How/when is the bot triggered? On what machine does it
 run? Is there an existing jumping-off point for creating such a bot? How
 does it approach the task of modifying `.travis.yml` (how smart/dumb
 does it need to be)? Any security concerns to be mindful of? Etc.


Once I catch up on the other things I've said I'm going to do, I'm 
planning on doing something like pyup-bot[1] for Dub projects.

pyup does this with project dependencies for python - you get a pull 
request with dependency updates, links to their changelog/documentation, 
etc.

Adding compiler updates might make sense, as the compiler is a dependency. 
Configuration would need to allow for the various decisions people make 
concerning what versions they support, so it's not a trivial addition, but 
doable.


--Ryan


[1]: pyup.io/
Mar 03 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Friday, 2 March 2018 at 06:07:25 UTC, Nick Sabalausky 
(Abscissa) wrote:

 (I'm posting this here instead of D.learn because it isn't 
 really a "How to do XYZ in D?", but rather to invite discussion 
 on high-level solutions to a problem.)

 Here's a common problem:

 1. A project (ex, some library) uses travis-ci (and/or other) 
 to ensure compatibility with a range of compiler versions. The 
 travis configuration file includes a manually-curated list of 
 compiler versions.

 2. CI tests are triggered by new commits/merges/PRs in the 
 project.

 3. New compiler versions are released.


Well, just for the record - imho adding your project to the 
project tester is the best way to prevent regressions a priori: 
add your project to the Project Tester (ci.dlang.io).

The Project Tester runs the testsuite of currently ~40 packages 
on every PR at dmd, druntime, phobos, tools and dub
An example: 
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6259/2/pipeline

Imho it's te best way as a regression is caught _before_ the 
change is merged and the need for the interaction is on the side 
of the person who would introduce the regression (i.e. the one 
understanding the change).

New projects can be added either by making a PR to the file to 
[1] or pinging me.

[1] 
https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy



 but there are still limitations and downsides (ex: D tester has 
 limited resources


FWIW it's not so limited and highly parallizable. It's already 
configured to automatically create new temporary worker machines 
in the Google Cloud [2] whenever there's more traffic/demand.
Considering the upsides of knowing that your project can _never 
ever_break again is so huge that imho even if more workers are 
needed, it's well-justified as there's quite a net gain.

[2] It currently uses the cheap preemptible VM machines in the 
Google Cloud 
(https://cloud.google.com/compute/docs/instances/preemptible), 
but Martin plans to migrate to Hetzner soon as there are too many 
problems with those VMs.
Mar 13 2018