Skip to content
Snippets Groups Projects

Remove sleep when starting tests

Closed Zeger-Jan van de Weg requested to merge zj-fix-startup-race into master
2 unresolved threads

This was introduced as, at that time, there was a race when serving pages. However this I couldn't reproduce. I've tried:

for i in `seq 1 100`; do
make acceptance > /dev/null
echo $?
;
done

However, this led to 100 times a 0 being printed. Giving me the idea this has been fixed in the past.

Fixes #31

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • @zj I rebased against master (just to be sure) and ran these tests locally a few times. It didn't take me long to get the following errors:

    --- FAIL: TestKnownHostReturns200 (0.15s)
    	helpers_test.go:134: Running ./gitlab-pages [-listen-http 127.0.0.1:37000 -listen-http [::1]:37000 -listen-https 127.0.0.1:37001 -listen-https [::1]:37001 -listen-proxy 127.0.0.1:37002 -listen-proxy [::1]:37002 -root-key /tmp/https-fixture319762395 -root-cert /tmp/https-fixture361286014]
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' http://127.0.0.1:37000/project/
            Location:       acceptance_test.go:61
    	Error:		Not equal: 200 (expected)
    			        != 404 (actual)
    
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' http://[::1]:37000/project/
            Location:       acceptance_test.go:61
    	Error:		Not equal: 200 (expected)
    			        != 404 (actual)
    
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' https://127.0.0.1:37001/project/
            Location:       acceptance_test.go:61
    	Error:		Not equal: 200 (expected)
    			        != 404 (actual)
    
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' https://[::1]:37001/project/
            Location:       acceptance_test.go:61
    	Error:		Not equal: 200 (expected)
    			        != 404 (actual)
    
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' http://127.0.0.1:37002/project/
    	helpers_test.go:196: curl -H'Host: group.gitlab-example.com' http://[::1]:37002/project/
    --- FAIL: TestPrometheusMetricsCanBeScraped (0.07s)
    	helpers_test.go:134: Running ./gitlab-pages [-listen-http 127.0.0.1:37003 -metrics-address :42345]
            Location:       acceptance_test.go:79
    	Error:		"# HELP gitlab_pages_domains_served_total The total number of sites served by this Pages app
    			# TYPE gitlab_pages_domains_served_total gauge
    			gitlab_pages_domains_served_total 0
    			# HELP gitlab_pages_domains_updated_total The total number of site updates processed since daemon start
    			# TYPE gitlab_pages_domains_updated_total counter
    			gitlab_pages_domains_updated_total 0
    			# HELP gitlab_pages_http_sessions_active The number of HTTP requests currently being processed
    			# TYPE gitlab_pages_http_sessions_active gauge
    			gitlab_pages_http_sessions_active 0
    			# HELP gitlab_pages_last_domain_update_seconds UNIX timestamp of the last update
    			# TYPE gitlab_pages_last_domain_update_seconds gauge
    			gitlab_pages_last_domain_update_seconds 0
    			# HELP go_gc_duration_seconds A summary of the GC invocation durations.
    			# TYPE go_gc_duration_seconds summary
    			go_gc_duration_seconds{quantile="0"} 0
    			go_gc_duration_seconds{quantile="0.25"} 0
    			go_gc_duration_seconds{quantile="0.5"} 0
    			go_gc_duration_seconds{quantile="0.75"} 0
    			go_gc_duration_seconds{quantile="1"} 0
    			go_gc_duration_seconds_sum 0
    			go_gc_duration_seconds_count 0
    			# HELP go_goroutines Number of goroutines that currently exist.
    			# TYPE go_goroutines gauge
    			go_goroutines 16
    			# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
    			# TYPE go_memstats_alloc_bytes gauge
    			go_memstats_alloc_bytes 498160
    			# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
    			# TYPE go_memstats_alloc_bytes_total counter
    			go_memstats_alloc_bytes_total 498160
    			# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
    			# TYPE go_memstats_buck_hash_sys_bytes gauge
    			go_memstats_buck_hash_sys_bytes 1.443406e+06
    			# HELP go_memstats_frees_total Total number of frees.
    			# TYPE go_memstats_frees_total counter
    			go_memstats_frees_total 0
    			# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
    			# TYPE go_memstats_gc_sys_bytes gauge
    			go_memstats_gc_sys_bytes 65536
    			# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
    			# TYPE go_memstats_heap_alloc_bytes gauge
    			go_memstats_heap_alloc_bytes 498160
    			# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
    			# TYPE go_memstats_heap_idle_bytes gauge
    			go_memstats_heap_idle_bytes 868352
    			# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
    			# TYPE go_memstats_heap_inuse_bytes gauge
    			go_memstats_heap_inuse_bytes 933888
    			# HELP go_memstats_heap_objects Number of allocated objects.
    			# TYPE go_memstats_heap_objects gauge
    			go_memstats_heap_objects 5868
    			# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
    			# TYPE go_memstats_heap_released_bytes_total counter
    			go_memstats_heap_released_bytes_total 0
    			# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
    			# TYPE go_memstats_heap_sys_bytes gauge
    			go_memstats_heap_sys_bytes 1.80224e+06
    			# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
    			# TYPE go_memstats_last_gc_time_seconds gauge
    			go_memstats_last_gc_time_seconds 0
    			# HELP go_memstats_lookups_total Total number of pointer lookups.
    			# TYPE go_memstats_lookups_total counter
    			go_memstats_lookups_total 27
    			# HELP go_memstats_mallocs_total Total number of mallocs.
    			# TYPE go_memstats_mallocs_total counter
    			go_memstats_mallocs_total 5868
    			# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
    			# TYPE go_memstats_mcache_inuse_bytes gauge
    			go_memstats_mcache_inuse_bytes 2416
    			# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
    			# TYPE go_memstats_mcache_sys_bytes gauge
    			go_memstats_mcache_sys_bytes 16384
    			# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
    			# TYPE go_memstats_mspan_inuse_bytes gauge
    			go_memstats_mspan_inuse_bytes 11648
    			# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
    			# TYPE go_memstats_mspan_sys_bytes gauge
    			go_memstats_mspan_sys_bytes 16384
    			# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
    			# TYPE go_memstats_next_gc_bytes gauge
    			go_memstats_next_gc_bytes 4.194304e+06
    			# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
    			# TYPE go_memstats_other_sys_bytes gauge
    			go_memstats_other_sys_bytes 819882
    			# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
    			# TYPE go_memstats_stack_inuse_bytes gauge
    			go_memstats_stack_inuse_bytes 294912
    			# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
    			# TYPE go_memstats_stack_sys_bytes gauge
    			go_memstats_stack_sys_bytes 294912
    			# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
    			# TYPE go_memstats_sys_bytes gauge
    			go_memstats_sys_bytes 4.458744e+06
    			# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
    			# TYPE process_cpu_seconds_total counter
    			process_cpu_seconds_total 0.02
    			# HELP process_max_fds Maximum number of open file descriptors.
    			# TYPE process_max_fds gauge
    			process_max_fds 1.048576e+06
    			# HELP process_open_fds Number of open file descriptors.
    			# TYPE process_open_fds gauge
    			process_open_fds 13
    			# HELP process_resident_memory_bytes Resident memory size in bytes.
    			# TYPE process_resident_memory_bytes gauge
    			process_resident_memory_bytes 7.847936e+06
    			# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
    			# TYPE process_start_time_seconds gauge
    			process_start_time_seconds 1.48897712762e+09
    			# HELP process_virtual_memory_bytes Virtual memory size in bytes.
    			# TYPE process_virtual_memory_bytes gauge
    			process_virtual_memory_bytes 1.18861824e+08
    			" does not contain "gitlab_pages_domains_served_total 7"

    The race is still there, and that doesn't surprise me since we haven't changed that area of code at all yet.

    We hook the handlers up to the listening sockets before the first update is run. This means that it returns 404 (and the prometheus metrics don't match) if the update - which is started in the background straight after - doesn't complete before we get around to making http requests in the test process.

  • assigned to @zj

  • Zeger-Jan van de Weg mentioned in merge request !20 (merged)

    mentioned in merge request !20 (merged)

  • added 24 commits

    • 5a346a7a...9851a84c - 23 commits from branch master
    • d74dfff9 - Block serving requests until first full update

    Compare with previous version

  • added 1 commit

    • fd2f1b09 - Block serving requests until first full update

    Compare with previous version

  • Author Developer

    @nick.thomas In theory this should work, but I'll test this tonight on my linux laptop.

  • Zeger-Jan van de Weg
  • Author Developer

    I can't get the current implementation to reliably pass on Linux again. Now its waiting to be listening on all the ports. Will debug further.

  • added 1 commit

    • 9a0d435d - Block serving requests until first full update

    Compare with previous version

  • Nick Thomas resolved all discussions

    resolved all discussions

  • 153 154 // for now. Without it, intermittent failures occur.
    154 155 //
    155 156 // TODO: replace this with explicit status from the pages binary
    156 // TODO: fix the first-request race
    • Sorry, I missed that most of the above comment block can be removed. Prune it down to just:

      // Wait for all TCP servers to be open
      //
      // TODO: replace this with explicit status from the pages binary
    • Please register or sign in to reply
  • assigned to @zj

  • removed assignee

  • Please register or sign in to reply
    Loading